2014-06-09 16:22:11.331248 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331250 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (1 active) 2014-06-09 16:22:11.331254 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0250 (2 active) 2014-06-09 16:22:11.331259 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc40dc20] 2014-06-09 16:22:11.331260 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0230 (1 active) 2014-06-09 16:22:11.331261 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.331262 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.331262 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331265 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331267 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331267 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331259 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:242, info:0.451( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.331269 7f869af78700 20 osd.26 243 _dispatch 0x10d0dc00 pg_notify(3.59f(10) epoch 242) v5 2014-06-09 16:22:11.331275 7f869af78700 5 -- op tracker -- , seq: 1486, time: 2014-06-09 16:22:10.835649, event: header_read, op: pg_notify(3.59f(10) epoch 242) 2014-06-09 16:22:11.331271 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.331282 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.331282 7f869af78700 5 -- op tracker -- , seq: 1486, time: 2014-06-09 16:22:10.835650, event: throttled, op: pg_notify(3.59f(10) epoch 242) 2014-06-09 16:22:11.331285 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.331290 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.331289 7f869af78700 5 -- op tracker -- , seq: 1486, time: 2014-06-09 16:22:10.835670, event: all_read, op: pg_notify(3.59f(10) epoch 242) 2014-06-09 16:22:11.331292 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.331296 7f869af78700 5 -- op tracker -- , seq: 1486, time: 2014-06-09 16:22:11.331265, event: dispatched, op: pg_notify(3.59f(10) epoch 242) 2014-06-09 16:22:11.331302 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.331303 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3034.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.331306 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.331303 7f869af78700 5 -- op tracker -- , seq: 1486, time: 2014-06-09 16:22:11.331301, event: waiting_for_osdmap, op: pg_notify(3.59f(10) epoch 242) 2014-06-09 16:22:11.331307 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (0 active) 2014-06-09 16:22:11.331308 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.331309 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331310 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x10d0dc00 2014-06-09 16:22:11.331314 7f868ef60700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331314 7f869af78700 5 -- op tracker -- , seq: 1486, time: 2014-06-09 16:22:11.331312, event: started, op: pg_notify(3.59f(10) epoch 242) 2014-06-09 16:22:11.331328 7f86a5e11700 10 _do_op 0x300603c0 seq 3034 r = 0, finisher 0x3061ffe0 0 2014-06-09 16:22:11.331332 7f86a5e11700 10 _finish_op 0x300603c0 seq 3034 osr(2.2da 0x2ed0c10)/0x2ed0c10 2014-06-09 16:22:11.331334 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc8580] 2014-06-09 16:22:11.331335 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0290 (1 active) 2014-06-09 16:22:11.331335 7f869af78700 5 -- op tracker -- , seq: 1486, time: 2014-06-09 16:22:11.331333, event: done, op: pg_notify(3.59f(10) epoch 242) 2014-06-09 16:22:11.331338 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc8580] 2014-06-09 16:22:11.331340 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.331340 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061ffe0] 2014-06-09 16:22:11.331341 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.331343 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331345 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331343 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.331341 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:3.59f( v 213'624 (0'0,213'624] local-les=208 n=117 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.331353 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331360 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331362 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331363 7f869af78700 20 osd.26 243 _dispatch 0x240f4400 pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.331357 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.331366 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0260 (2 active) 2014-06-09 16:22:11.331370 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.331370 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061ffe0] 2014-06-09 16:22:11.331372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.331373 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.331370 7f869af78700 5 -- op tracker -- , seq: 1487, time: 2014-06-09 16:22:10.999668, event: header_read, op: pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331376 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b7a0 (1 active) 2014-06-09 16:22:11.331376 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0290 (1 active) 2014-06-09 16:22:11.331370 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.331378 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d19e0 (2 active) 2014-06-09 16:22:11.331379 7f868e75f700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331380 7f86a5e11700 5 _do_op 0x308cb440 seq 3036 osr(0.1c8 0x2bd69a0)/0x2bd69a0 start 2014-06-09 16:22:11.331383 7f86a5e11700 10 _do_transaction on 0x390db00 2014-06-09 16:22:11.331381 7f869af78700 5 -- op tracker -- , seq: 1487, time: 2014-06-09 16:22:10.999668, event: throttled, op: pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331382 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.331389 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.331389 7f869af78700 5 -- op tracker -- , seq: 1487, time: 2014-06-09 16:22:10.999678, event: all_read, op: pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331392 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.331398 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0260 (0 active) 2014-06-09 16:22:11.331396 7f869af78700 5 -- op tracker -- , seq: 1487, time: 2014-06-09 16:22:11.331360, event: dispatched, op: pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331400 7f868ff62700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331404 7f869af78700 5 -- op tracker -- , seq: 1487, time: 2014-06-09 16:22:11.331401, event: waiting_for_osdmap, op: pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331409 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.331412 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.331414 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240f4400 2014-06-09 16:22:11.331420 7f869af78700 5 -- op tracker -- , seq: 1487, time: 2014-06-09 16:22:11.331417, event: started, op: pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331430 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3035.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.331435 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.331437 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0240 (1 active) 2014-06-09 16:22:11.331436 7f869af78700 5 -- op tracker -- , seq: 1487, time: 2014-06-09 16:22:11.331433, event: done, op: pg_log(2.37 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331445 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331447 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331442 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.331450 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.37( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.331457 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.331459 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331459 7f86a5610700 10 _do_op 0x300609c0 seq 3035 r = 0, finisher 0x1a002140 0 2014-06-09 16:22:11.331461 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331461 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.331462 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a1c0 pg_notify(0.1ca(23) epoch 242) v5 2014-06-09 16:22:11.331465 7f86a5610700 10 _finish_op 0x300609c0 seq 3035 osr(1.449 0x384d690)/0x384d690 2014-06-09 16:22:11.331468 7f869af78700 5 -- op tracker -- , seq: 1488, time: 2014-06-09 16:22:10.946542, event: header_read, op: pg_notify(0.1ca(23) epoch 242) 2014-06-09 16:22:11.331471 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e7e0 (1 active) 2014-06-09 16:22:11.331474 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5d40 (2 active) 2014-06-09 16:22:11.331473 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1a002140] 2014-06-09 16:22:11.331467 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6fee0] 2014-06-09 16:22:11.331476 7f86a5610700 5 _do_op 0x30061bc0 seq 3037 osr(0.3cf 0x2bd7270)/0x2bd7270 start 2014-06-09 16:22:11.331475 7f869af78700 5 -- op tracker -- , seq: 1488, time: 2014-06-09 16:22:10.946543, event: throttled, op: pg_notify(0.1ca(23) epoch 242) 2014-06-09 16:22:11.331479 7f86a5610700 10 _do_transaction on 0xc0d1380 2014-06-09 16:22:11.331479 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1a002140] 2014-06-09 16:22:11.331481 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.331482 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.331482 7f869af78700 5 -- op tracker -- , seq: 1488, time: 2014-06-09 16:22:10.946578, event: all_read, op: pg_notify(0.1ca(23) epoch 242) 2014-06-09 16:22:11.331485 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.331478 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.331489 7f869af78700 5 -- op tracker -- , seq: 1488, time: 2014-06-09 16:22:11.331459, event: dispatched, op: pg_notify(0.1ca(23) epoch 242) 2014-06-09 16:22:11.331491 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331495 7f869af78700 5 -- op tracker -- , seq: 1488, time: 2014-06-09 16:22:11.331493, event: waiting_for_osdmap, op: pg_notify(0.1ca(23) epoch 242) 2014-06-09 16:22:11.331500 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.331502 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2ea5a1c0 2014-06-09 16:22:11.331503 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.331508 7f869af78700 5 -- op tracker -- , seq: 1488, time: 2014-06-09 16:22:11.331505, event: started, op: pg_notify(0.1ca(23) epoch 242) 2014-06-09 16:22:11.331505 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331518 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6fee0] 2014-06-09 16:22:11.331520 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.331521 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.331521 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (2 active) 2014-06-09 16:22:11.331526 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331527 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.331537 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (3 active) 2014-06-09 16:22:11.331541 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.331538 7f869af78700 5 -- op tracker -- , seq: 1488, time: 2014-06-09 16:22:11.331535, event: done, op: pg_notify(0.1ca(23) epoch 242) 2014-06-09 16:22:11.331546 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331549 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331542 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 5 notify: (query_epoch:242, epoch_sent:242, info:0.1ca( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.331546 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.331556 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.331556 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.331565 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.331567 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.331570 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (2 active) 2014-06-09 16:22:11.331571 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331573 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331572 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331574 7f869af78700 20 osd.26 243 _dispatch 0x15998540 pg_notify(1.449(21) epoch 242) v5 2014-06-09 16:22:11.331578 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (1 active) 2014-06-09 16:22:11.331581 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331579 7f869af78700 5 -- op tracker -- , seq: 1489, time: 2014-06-09 16:22:10.746999, event: header_read, op: pg_notify(1.449(21) epoch 242) 2014-06-09 16:22:11.331587 7f869af78700 5 -- op tracker -- , seq: 1489, time: 2014-06-09 16:22:10.747000, event: throttled, op: pg_notify(1.449(21) epoch 242) 2014-06-09 16:22:11.331592 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331464 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.37( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.151506 7 0.000228 2014-06-09 16:22:11.331597 7f869af78700 5 -- op tracker -- , seq: 1489, time: 2014-06-09 16:22:10.747038, event: all_read, op: pg_notify(1.449(21) epoch 242) 2014-06-09 16:22:11.331610 7f869af78700 5 -- op tracker -- , seq: 1489, time: 2014-06-09 16:22:11.331570, event: dispatched, op: pg_notify(1.449(21) epoch 242) 2014-06-09 16:22:11.331622 7f869af78700 5 -- op tracker -- , seq: 1489, time: 2014-06-09 16:22:11.331620, event: waiting_for_osdmap, op: pg_notify(1.449(21) epoch 242) 2014-06-09 16:22:11.331607 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.37( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.331627 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.331629 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x15998540 2014-06-09 16:22:11.331582 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3036.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.331631 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.37( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.331640 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.331634 7f869af78700 5 -- op tracker -- , seq: 1489, time: 2014-06-09 16:22:11.331632, event: started, op: pg_notify(1.449(21) epoch 242) 2014-06-09 16:22:11.331641 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.331649 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.331659 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (2 active) 2014-06-09 16:22:11.331660 7f86a5e11700 10 _do_op 0x308cb440 seq 3036 r = 0, finisher 0xbd6e760 0 2014-06-09 16:22:11.331658 7f869af78700 5 -- op tracker -- , seq: 1489, time: 2014-06-09 16:22:11.331656, event: done, op: pg_notify(1.449(21) epoch 242) 2014-06-09 16:22:11.331665 7f86a5e11700 10 _finish_op 0x308cb440 seq 3036 osr(0.1c8 0x2bd69a0)/0x2bd69a0 2014-06-09 16:22:11.331667 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbebf0e0] 2014-06-09 16:22:11.331668 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331669 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbebf0e0] 2014-06-09 16:22:11.331670 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331671 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.331672 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.331673 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6e760] 2014-06-09 16:22:11.331664 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:1.449( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.331658 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.331677 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.331676 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 208 same_interval_since 242 2014-06-09 16:22:11.331687 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.331690 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331692 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331693 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (1 active) 2014-06-09 16:22:11.331693 7f869af78700 20 osd.26 243 _dispatch 0x240eaa80 pg_notify(0.f1(20) epoch 242) v5 2014-06-09 16:22:11.331688 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331697 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331700 7f869af78700 5 -- op tracker -- , seq: 1490, time: 2014-06-09 16:22:10.700801, event: header_read, op: pg_notify(0.f1(20) epoch 242) 2014-06-09 16:22:11.331702 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331707 7f869af78700 5 -- op tracker -- , seq: 1490, time: 2014-06-09 16:22:10.700802, event: throttled, op: pg_notify(0.f1(20) epoch 242) 2014-06-09 16:22:11.331715 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.331715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6e760] 2014-06-09 16:22:11.331717 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.331718 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.331720 7f8690f64700 5 queue_transactions existing osr(2.37 0x2ed05f0)/0x2ed05f0 2014-06-09 16:22:11.331721 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d19e0 (1 active) 2014-06-09 16:22:11.331719 7f869af78700 5 -- op tracker -- , seq: 1490, time: 2014-06-09 16:22:10.700836, event: all_read, op: pg_notify(0.f1(20) epoch 242) 2014-06-09 16:22:11.331723 7f8690f64700 10 op_submit_start 3083 2014-06-09 16:22:11.331723 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f79e0 (2 active) 2014-06-09 16:22:11.331724 7f8690f64700 5 queue_transactions (trailing journal) 3083 2014-06-09 16:22:11.331725 7f86a5e11700 5 _do_op 0x308ca360 seq 3038 osr(2.1fc 0x2ed0b90)/0x2ed0b90 start 2014-06-09 16:22:11.331726 7f8690f64700 5 queue_op 0x308ca480 seq 3083 osr(2.37 0x2ed05f0) 1880 bytes (queue has 47 ops and 54001 bytes) 2014-06-09 16:22:11.331728 7f8690f64700 10 op_submit_finish 3083 2014-06-09 16:22:11.331728 7f86a5e11700 10 _do_transaction on 0x3a86180 2014-06-09 16:22:11.331726 7f869af78700 5 -- op tracker -- , seq: 1490, time: 2014-06-09 16:22:11.331689, event: dispatched, op: pg_notify(0.f1(20) epoch 242) 2014-06-09 16:22:11.331732 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0240 (0 active) 2014-06-09 16:22:11.331731 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.331734 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.331735 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0240 (1 active) 2014-06-09 16:22:11.331734 7f869af78700 5 -- op tracker -- , seq: 1490, time: 2014-06-09 16:22:11.331732, event: waiting_for_osdmap, op: pg_notify(0.f1(20) epoch 242) 2014-06-09 16:22:11.331744 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.331738 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.331747 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240eaa80 2014-06-09 16:22:11.331748 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3037.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.331751 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.331752 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.331752 7f869af78700 5 -- op tracker -- , seq: 1490, time: 2014-06-09 16:22:11.331750, event: started, op: pg_notify(0.f1(20) epoch 242) 2014-06-09 16:22:11.331756 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.331766 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.331771 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (2 active) 2014-06-09 16:22:11.331775 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.331771 7f869af78700 5 -- op tracker -- , seq: 1490, time: 2014-06-09 16:22:11.331768, event: done, op: pg_notify(0.f1(20) epoch 242) 2014-06-09 16:22:11.331779 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0240 (1 active) 2014-06-09 16:22:11.331780 7f8690f64700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331780 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331782 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331784 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331777 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:0.f1( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.331790 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.331800 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.331803 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331805 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331806 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (0 active) 2014-06-09 16:22:11.331806 7f869af78700 20 osd.26 243 _dispatch 0x3004cc00 pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.331809 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331813 7f869af78700 5 -- op tracker -- , seq: 1491, time: 2014-06-09 16:22:11.053924, event: header_read, op: pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331821 7f869af78700 5 -- op tracker -- , seq: 1491, time: 2014-06-09 16:22:11.053925, event: throttled, op: pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331828 7f86a5610700 10 _do_op 0x30061bc0 seq 3037 r = 0, finisher 0xbef1480 0 2014-06-09 16:22:11.331828 7f869af78700 5 -- op tracker -- , seq: 1491, time: 2014-06-09 16:22:11.053940, event: all_read, op: pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331834 7f86a5610700 10 _finish_op 0x30061bc0 seq 3037 osr(0.3cf 0x2bd7270)/0x2bd7270 2014-06-09 16:22:11.331836 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe308c0] 2014-06-09 16:22:11.331840 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe308c0] 2014-06-09 16:22:11.331839 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5d40 (1 active) 2014-06-09 16:22:11.331841 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.331838 7f869af78700 5 -- op tracker -- , seq: 1491, time: 2014-06-09 16:22:11.331802, event: dispatched, op: pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331842 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.331841 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac5200 (2 active) 2014-06-09 16:22:11.331845 7f86a5610700 5 _do_op 0x308cbe60 seq 3039 osr(2.744 0x3ed0150)/0x3ed0150 start 2014-06-09 16:22:11.331847 7f86a5610700 10 _do_transaction on 0x2ed4f00 2014-06-09 16:22:11.331847 7f869af78700 5 -- op tracker -- , seq: 1491, time: 2014-06-09 16:22:11.331844, event: waiting_for_osdmap, op: pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331854 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.331853 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.331857 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004cc00 2014-06-09 16:22:11.331862 7f869af78700 5 -- op tracker -- , seq: 1491, time: 2014-06-09 16:22:11.331860, event: started, op: pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331867 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef1480] 2014-06-09 16:22:11.331872 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 201 same_interval_since 242 2014-06-09 16:22:11.331885 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331896 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.331901 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331910 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3038.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.331914 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.331913 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (1 active) 2014-06-09 16:22:11.331916 7f869af78700 5 -- op tracker -- , seq: 1491, time: 2014-06-09 16:22:11.331913, event: done, op: pg_log(0.663 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.331925 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331926 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331919 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.331923 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331932 7f86a5e11700 10 _do_op 0x308ca360 seq 3038 r = 0, finisher 0xbfce340 0 2014-06-09 16:22:11.331938 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.331939 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.331940 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef1480] 2014-06-09 16:22:11.331931 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.663( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.331940 7f869af78700 20 osd.26 243 _dispatch 0x2d733880 pg_notify(0.74a(23) epoch 242) v5 2014-06-09 16:22:11.331941 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbce6840] 2014-06-09 16:22:11.331942 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.331943 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.331941 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02f0 (2 active) 2014-06-09 16:22:11.331942 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.331944 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbce6840] 2014-06-09 16:22:11.331946 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.331947 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.331947 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331946 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.331945 7f869af78700 5 -- op tracker -- , seq: 1492, time: 2014-06-09 16:22:10.912321, event: header_read, op: pg_notify(0.74a(23) epoch 242) 2014-06-09 16:22:11.331949 7f86a5e11700 10 _finish_op 0x308ca360 seq 3038 osr(2.1fc 0x2ed0b90)/0x2ed0b90 2014-06-09 16:22:11.331951 7f869af78700 5 -- op tracker -- , seq: 1492, time: 2014-06-09 16:22:10.912322, event: throttled, op: pg_notify(0.74a(23) epoch 242) 2014-06-09 16:22:11.331954 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f79e0 (1 active) 2014-06-09 16:22:11.331955 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfce340] 2014-06-09 16:22:11.331947 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.331956 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac360 (2 active) 2014-06-09 16:22:11.331950 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.663( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] exit Started/Stray 1.058891 7 0.000169 2014-06-09 16:22:11.331957 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.331958 7f86a5e11700 5 _do_op 0x309a30e0 seq 3040 osr(2.5fb 0x2ed1d70)/0x2ed1d70 start 2014-06-09 16:22:11.331958 7f869af78700 5 -- op tracker -- , seq: 1492, time: 2014-06-09 16:22:10.912347, event: all_read, op: pg_notify(0.74a(23) epoch 242) 2014-06-09 16:22:11.331961 7f86a5e11700 10 _do_transaction on 0xc238780 2014-06-09 16:22:11.331962 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.331964 7f869af78700 5 -- op tracker -- , seq: 1492, time: 2014-06-09 16:22:11.331937, event: dispatched, op: pg_notify(0.74a(23) epoch 242) 2014-06-09 16:22:11.331960 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.663( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.331967 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.331958 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 212 same_interval_since 242 2014-06-09 16:22:11.331970 7f869af78700 5 -- op tracker -- , seq: 1492, time: 2014-06-09 16:22:11.331968, event: waiting_for_osdmap, op: pg_notify(0.74a(23) epoch 242) 2014-06-09 16:22:11.331966 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.331973 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3039.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.331975 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.331977 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.331970 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331970 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.663( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.331977 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d733880 2014-06-09 16:22:11.331976 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.331982 7f869af78700 5 -- op tracker -- , seq: 1492, time: 2014-06-09 16:22:11.331980, event: started, op: pg_notify(0.74a(23) epoch 242) 2014-06-09 16:22:11.331985 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.331980 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.331991 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02f0 (1 active) 2014-06-09 16:22:11.331984 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.331994 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02f0 (2 active) 2014-06-09 16:22:11.331989 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.332000 7f8692767700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.331998 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.332010 7f86a5610700 10 _do_op 0x308cbe60 seq 3039 r = 0, finisher 0x2d7382a0 0 2014-06-09 16:22:11.332012 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (3 active) 2014-06-09 16:22:11.332016 7f86a5610700 10 _finish_op 0x308cbe60 seq 3039 osr(2.744 0x3ed0150)/0x3ed0150 2014-06-09 16:22:11.332013 7f869af78700 5 -- op tracker -- , seq: 1492, time: 2014-06-09 16:22:11.332010, event: done, op: pg_notify(0.74a(23) epoch 242) 2014-06-09 16:22:11.332017 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbebe140] 2014-06-09 16:22:11.332019 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac5200 (1 active) 2014-06-09 16:22:11.332020 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbebe140] 2014-06-09 16:22:11.332022 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332021 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332021 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7818c0 (2 active) 2014-06-09 16:22:11.332023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332023 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332024 7f86a5610700 5 _do_op 0x2fc8c240 seq 3041 osr(0.4b3 0x2bd7350)/0x2bd7350 start 2014-06-09 16:22:11.332026 7f86a5610700 10 _do_transaction on 0xc2ab800 2014-06-09 16:22:11.332018 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 3 notify: (query_epoch:242, epoch_sent:242, info:0.74a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.332032 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfce340] 2014-06-09 16:22:11.332033 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.332034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d7382a0] 2014-06-09 16:22:11.332030 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.332040 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.332033 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332044 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.332044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332036 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 210 same_interval_since 242 2014-06-09 16:22:11.332045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332045 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (2 active) 2014-06-09 16:22:11.332047 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.332047 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332047 7f869af78700 20 osd.26 243 _dispatch 0x302a3000 pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.332049 7f8691f66700 5 queue_transactions existing osr(0.663 0x2bd7f30)/0x2bd7f30 2014-06-09 16:22:11.332051 7f8691f66700 10 op_submit_start 3084 2014-06-09 16:22:11.332052 7f8691f66700 5 queue_transactions (trailing journal) 3084 2014-06-09 16:22:11.332052 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332047 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332053 7f8691f66700 5 queue_op 0x308cbce0 seq 3084 osr(0.663 0x2bd7f30) 2048 bytes (queue has 45 ops and 53164 bytes) 2014-06-09 16:22:11.332056 7f8691f66700 10 op_submit_finish 3084 2014-06-09 16:22:11.332053 7f869af78700 5 -- op tracker -- , seq: 1493, time: 2014-06-09 16:22:11.019528, event: header_read, op: pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.332060 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (1 active) 2014-06-09 16:22:11.332063 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (2 active) 2014-06-09 16:22:11.332061 7f869af78700 5 -- op tracker -- , seq: 1493, time: 2014-06-09 16:22:11.019528, event: throttled, op: pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.332063 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d7382a0] 2014-06-09 16:22:11.332066 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.332067 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.332066 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.332068 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02f0 (1 active) 2014-06-09 16:22:11.332068 7f869af78700 5 -- op tracker -- , seq: 1493, time: 2014-06-09 16:22:11.019544, event: all_read, op: pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.332072 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02f0 (2 active) 2014-06-09 16:22:11.332068 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332074 7f869af78700 5 -- op tracker -- , seq: 1493, time: 2014-06-09 16:22:11.332044, event: dispatched, op: pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.332078 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332081 7f869af78700 5 -- op tracker -- , seq: 1493, time: 2014-06-09 16:22:11.332078, event: waiting_for_osdmap, op: pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.332082 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3041.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.332080 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.332087 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.332085 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.332089 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.332089 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x302a3000 2014-06-09 16:22:11.332083 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332094 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (1 active) 2014-06-09 16:22:11.332096 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332094 7f869af78700 5 -- op tracker -- , seq: 1493, time: 2014-06-09 16:22:11.332092, event: started, op: pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.332094 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.332103 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.332109 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0250 (2 active) 2014-06-09 16:22:11.332111 7f86a5610700 10 _do_op 0x2fc8c240 seq 3041 r = 0, finisher 0xc5a6660 0 2014-06-09 16:22:11.332110 7f869af78700 5 -- op tracker -- , seq: 1493, time: 2014-06-09 16:22:11.332107, event: done, op: pg_log(0.369 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.332116 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02f0 (1 active) 2014-06-09 16:22:11.332117 7f86a5610700 10 _finish_op 0x2fc8c240 seq 3041 osr(0.4b3 0x2bd7350)/0x2bd7350 2014-06-09 16:22:11.332118 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc90c0] 2014-06-09 16:22:11.332119 7f868b759700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332119 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332121 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332121 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc90c0] 2014-06-09 16:22:11.332121 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7818c0 (1 active) 2014-06-09 16:22:11.332123 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332124 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332123 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7440 (2 active) 2014-06-09 16:22:11.332126 7f86a5610700 5 _do_op 0x30606180 seq 3042 osr(1.1bc 0x384cab0)/0x384cab0 start 2014-06-09 16:22:11.332127 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a6660] 2014-06-09 16:22:11.332128 7f86a5610700 10 _do_transaction on 0xc0d6900 2014-06-09 16:22:11.332133 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332135 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332134 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.332136 7f869af78700 20 osd.26 243 _dispatch 0x2d094380 pg_notify(0.732(23) epoch 242) v5 2014-06-09 16:22:11.332130 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.332143 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.332142 7f869af78700 5 -- op tracker -- , seq: 1494, time: 2014-06-09 16:22:10.863265, event: header_read, op: pg_notify(0.732(23) epoch 242) 2014-06-09 16:22:11.332142 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332149 7f869af78700 5 -- op tracker -- , seq: 1494, time: 2014-06-09 16:22:10.863266, event: throttled, op: pg_notify(0.732(23) epoch 242) 2014-06-09 16:22:11.332156 7f869af78700 5 -- op tracker -- , seq: 1494, time: 2014-06-09 16:22:10.863283, event: all_read, op: pg_notify(0.732(23) epoch 242) 2014-06-09 16:22:11.332158 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (2 active) 2014-06-09 16:22:11.332159 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3040.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.332155 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332163 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.332163 7f869af78700 5 -- op tracker -- , seq: 1494, time: 2014-06-09 16:22:11.332133, event: dispatched, op: pg_notify(0.732(23) epoch 242) 2014-06-09 16:22:11.332170 7f868e75f700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332171 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a6660] 2014-06-09 16:22:11.332173 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.332174 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.332170 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332116 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.332185 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.332189 7f86a5e11700 10 _do_op 0x309a30e0 seq 3040 r = 0, finisher 0x3061f040 0 2014-06-09 16:22:11.332192 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (1 active) 2014-06-09 16:22:11.332194 7f86a5e11700 10 _finish_op 0x309a30e0 seq 3040 osr(2.5fb 0x2ed1d70)/0x2ed1d70 2014-06-09 16:22:11.332195 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061f020] 2014-06-09 16:22:11.332195 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (2 active) 2014-06-09 16:22:11.332198 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061f020] 2014-06-09 16:22:11.332198 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac360 (1 active) 2014-06-09 16:22:11.332200 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332200 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd8c0 (2 active) 2014-06-09 16:22:11.332201 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332202 7f86a5e11700 5 _do_op 0x308cb3e0 seq 3043 osr(3.491 0x3ed14f0)/0x3ed14f0 start 2014-06-09 16:22:11.332204 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061f040] 2014-06-09 16:22:11.332205 7f86a5e11700 10 _do_transaction on 0xc05d200 2014-06-09 16:22:11.332198 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332210 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.332207 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.332214 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.332209 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.332219 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.332218 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332224 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (1 active) 2014-06-09 16:22:11.332226 7f868ef60700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332230 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3042.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.332234 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.332236 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332246 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0260 (2 active) 2014-06-09 16:22:11.332253 7f86a5610700 10 _do_op 0x30606180 seq 3042 r = 0, finisher 0x18742440 0 2014-06-09 16:22:11.332257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061f040] 2014-06-09 16:22:11.332259 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.332259 7f86a5610700 10 _finish_op 0x30606180 seq 3042 osr(1.1bc 0x384cab0)/0x384cab0 2014-06-09 16:22:11.332260 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.332264 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3e560] 2014-06-09 16:22:11.332264 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7440 (1 active) 2014-06-09 16:22:11.332266 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3e560] 2014-06-09 16:22:11.332267 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7200 (2 active) 2014-06-09 16:22:11.332270 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332269 7f86a5610700 5 _do_op 0x30606120 seq 3044 osr(3.1bd 0x3ed0a50)/0x3ed0a50 start 2014-06-09 16:22:11.332271 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332272 7f86a5610700 10 _do_transaction on 0xc1f6600 2014-06-09 16:22:11.332274 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x18742440] 2014-06-09 16:22:11.332278 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.332278 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.332277 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 201 same_interval_since 242 2014-06-09 16:22:11.332288 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332293 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3043.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.332297 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.332301 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332315 7f86a5e11700 10 _do_op 0x308cb3e0 seq 3043 r = 0, finisher 0x3050b680 0 2014-06-09 16:22:11.332317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x18742440] 2014-06-09 16:22:11.332316 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (3 active) 2014-06-09 16:22:11.332319 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.332320 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.332320 7f86a5e11700 10 _finish_op 0x308cb3e0 seq 3043 osr(3.491 0x3ed14f0)/0x3ed14f0 2014-06-09 16:22:11.332323 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062a1a0] 2014-06-09 16:22:11.332324 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd8c0 (1 active) 2014-06-09 16:22:11.332326 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062a1a0] 2014-06-09 16:22:11.332327 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89ad80 (2 active) 2014-06-09 16:22:11.332327 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332328 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332328 7f86a5e11700 5 _do_op 0x30033b60 seq 3045 osr(2.d 0x2ed02b0)/0x2ed02b0 start 2014-06-09 16:22:11.332322 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332331 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050b680] 2014-06-09 16:22:11.332331 7f86a5e11700 10 _do_transaction on 0x2c80e40 2014-06-09 16:22:11.332336 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.332336 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.332333 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 208 same_interval_since 242 2014-06-09 16:22:11.332341 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332346 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332357 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332351 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.332361 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.332366 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (2 active) 2014-06-09 16:22:11.332170 7f869af78700 5 -- op tracker -- , seq: 1494, time: 2014-06-09 16:22:11.332168, event: waiting_for_osdmap, op: pg_notify(0.732(23) epoch 242) 2014-06-09 16:22:11.332370 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (3 active) 2014-06-09 16:22:11.332370 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.332373 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d094380 2014-06-09 16:22:11.332367 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332378 7f869af78700 5 -- op tracker -- , seq: 1494, time: 2014-06-09 16:22:11.332376, event: started, op: pg_notify(0.732(23) epoch 242) 2014-06-09 16:22:11.332384 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050b680] 2014-06-09 16:22:11.332386 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.332387 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.332391 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332384 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332398 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.332391 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332403 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.332404 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0260 (2 active) 2014-06-09 16:22:11.332403 7f869af78700 5 -- op tracker -- , seq: 1494, time: 2014-06-09 16:22:11.332401, event: done, op: pg_notify(0.732(23) epoch 242) 2014-06-09 16:22:11.332408 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0260 (3 active) 2014-06-09 16:22:11.332187 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 0.369( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.332412 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332412 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.332414 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332416 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.332413 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332419 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.369( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.088577 7 0.000177 2014-06-09 16:22:11.332423 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.332431 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.332436 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332438 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332432 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.369( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.332439 7f869af78700 20 osd.26 243 _dispatch 0x1607b180 pg_notify(1.1eb(20),2.4b5(20) epoch 242) v5 2014-06-09 16:22:11.332435 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 9 notify: (query_epoch:242, epoch_sent:242, info:0.732( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.332442 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.369( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.332445 7f869af78700 5 -- op tracker -- , seq: 1495, time: 2014-06-09 16:22:10.938158, event: header_read, op: pg_notify(1.1eb(20),2.4b5(20) epoch 242) 2014-06-09 16:22:11.332451 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332445 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.332454 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.332451 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.332455 7f869af78700 5 -- op tracker -- , seq: 1495, time: 2014-06-09 16:22:10.938158, event: throttled, op: pg_notify(1.1eb(20),2.4b5(20) epoch 242) 2014-06-09 16:22:11.332459 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0260 (2 active) 2014-06-09 16:22:11.332461 7f868ff62700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332462 7f869af78700 5 -- op tracker -- , seq: 1495, time: 2014-06-09 16:22:10.938188, event: all_read, op: pg_notify(1.1eb(20),2.4b5(20) epoch 242) 2014-06-09 16:22:11.332459 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.332469 7f869af78700 5 -- op tracker -- , seq: 1495, time: 2014-06-09 16:22:11.332436, event: dispatched, op: pg_notify(1.1eb(20),2.4b5(20) epoch 242) 2014-06-09 16:22:11.332467 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.332475 7f869af78700 5 -- op tracker -- , seq: 1495, time: 2014-06-09 16:22:11.332473, event: waiting_for_osdmap, op: pg_notify(1.1eb(20),2.4b5(20) epoch 242) 2014-06-09 16:22:11.332481 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.332483 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1607b180 2014-06-09 16:22:11.332488 7f869af78700 5 -- op tracker -- , seq: 1495, time: 2014-06-09 16:22:11.332486, event: started, op: pg_notify(1.1eb(20),2.4b5(20) epoch 242) 2014-06-09 16:22:11.332490 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.332506 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3045.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.332509 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.332513 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.332515 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (3 active) 2014-06-09 16:22:11.332518 7f8690763700 5 queue_transactions existing osr(0.369 0x2bd6ff0)/0x2bd6ff0 2014-06-09 16:22:11.332514 7f869af78700 5 -- op tracker -- , seq: 1495, time: 2014-06-09 16:22:11.332511, event: done, op: pg_notify(1.1eb(20),2.4b5(20) epoch 242) 2014-06-09 16:22:11.332520 7f8690763700 10 op_submit_start 3085 2014-06-09 16:22:11.332521 7f8690763700 5 queue_transactions (trailing journal) 3085 2014-06-09 16:22:11.332522 7f8690763700 5 queue_op 0x30607b60 seq 3085 osr(0.369 0x2bd6ff0) 1883 bytes (queue has 42 ops and 51195 bytes) 2014-06-09 16:22:11.332525 7f8690763700 10 op_submit_finish 3085 2014-06-09 16:22:11.332524 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332526 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332529 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (2 active) 2014-06-09 16:22:11.332520 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:242, info:1.1eb( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.332531 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332536 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (1 active) 2014-06-09 16:22:11.332536 7f86a5e11700 10 _do_op 0x30033b60 seq 3045 r = 0, finisher 0xbdf1d60 0 2014-06-09 16:22:11.332539 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (2 active) 2014-06-09 16:22:11.332541 7f86a5e11700 10 _finish_op 0x30033b60 seq 3045 osr(2.d 0x2ed02b0)/0x2ed02b0 2014-06-09 16:22:11.332542 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdbb100] 2014-06-09 16:22:11.332534 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.332544 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdbb100] 2014-06-09 16:22:11.332545 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.332545 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332545 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89ad80 (1 active) 2014-06-09 16:22:11.332547 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332546 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332547 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781320 (2 active) 2014-06-09 16:22:11.332548 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332549 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdf1d60] 2014-06-09 16:22:11.332549 7f86a5e11700 5 _do_op 0x30061b60 seq 3046 osr(2.4ba 0x2ed1740)/0x2ed1740 start 2014-06-09 16:22:11.332549 7f869af78700 20 osd.26 243 _dispatch 0x2d094700 pg_notify(2.102(20) epoch 242) v5 2014-06-09 16:22:11.332552 7f86a5e11700 10 _do_transaction on 0x336ae40 2014-06-09 16:22:11.332554 7f869af78700 5 -- op tracker -- , seq: 1496, time: 2014-06-09 16:22:10.845666, event: header_read, op: pg_notify(2.102(20) epoch 242) 2014-06-09 16:22:11.332557 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.332549 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:242, info:2.4b5( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.332552 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 212 same_interval_since 242 2014-06-09 16:22:11.332561 7f869af78700 5 -- op tracker -- , seq: 1496, time: 2014-06-09 16:22:10.845667, event: throttled, op: pg_notify(2.102(20) epoch 242) 2014-06-09 16:22:11.332561 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.332567 7f869af78700 5 -- op tracker -- , seq: 1496, time: 2014-06-09 16:22:10.845693, event: all_read, op: pg_notify(2.102(20) epoch 242) 2014-06-09 16:22:11.332564 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332571 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.332577 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (1 active) 2014-06-09 16:22:11.332580 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332573 7f869af78700 5 -- op tracker -- , seq: 1496, time: 2014-06-09 16:22:11.332546, event: dispatched, op: pg_notify(2.102(20) epoch 242) 2014-06-09 16:22:11.332579 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332589 7f8690f64700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332591 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.332585 7f869af78700 5 -- op tracker -- , seq: 1496, time: 2014-06-09 16:22:11.332583, event: waiting_for_osdmap, op: pg_notify(2.102(20) epoch 242) 2014-06-09 16:22:11.332597 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.332592 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.332599 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d094700 2014-06-09 16:22:11.332602 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.332603 7f869af78700 5 -- op tracker -- , seq: 1496, time: 2014-06-09 16:22:11.332601, event: started, op: pg_notify(2.102(20) epoch 242) 2014-06-09 16:22:11.332607 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (0 active) 2014-06-09 16:22:11.332606 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3046.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.332609 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332610 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.332618 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdf1d60] 2014-06-09 16:22:11.332622 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.332623 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.332626 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02b0 (1 active) 2014-06-09 16:22:11.332630 7f86a5e11700 10 _do_op 0x30061b60 seq 3046 r = 0, finisher 0xbd6fd60 0 2014-06-09 16:22:11.332637 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5b6a0] 2014-06-09 16:22:11.332638 7f86a5e11700 10 _finish_op 0x30061b60 seq 3046 osr(2.4ba 0x2ed1740)/0x2ed1740 2014-06-09 16:22:11.332640 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5b6a0] 2014-06-09 16:22:11.332642 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332633 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332643 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332642 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781320 (1 active) 2014-06-09 16:22:11.332645 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c480 (2 active) 2014-06-09 16:22:11.332646 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6fd60] 2014-06-09 16:22:11.332647 7f86a5e11700 5 _do_op 0x30033b00 seq 3047 osr(3.573 0x3ed18e0)/0x3ed18e0 start 2014-06-09 16:22:11.332650 7f86a5e11700 10 _do_transaction on 0x344cf00 2014-06-09 16:22:11.332646 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.332655 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.332648 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 208 same_interval_since 242 2014-06-09 16:22:11.332658 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.332663 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.332659 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332633 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332677 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332668 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:2.102( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.332682 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3044.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.332686 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.332681 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.332691 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.332694 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332697 7f868d75d700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02b0 (0 active) 2014-06-09 16:22:11.332700 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332709 7f86a5610700 10 _do_op 0x30606120 seq 3044 r = 0, finisher 0xbdf09a0 0 2014-06-09 16:22:11.332711 7f869af78700 5 -- op tracker -- , seq: 1496, time: 2014-06-09 16:22:11.332708, event: done, op: pg_notify(2.102(20) epoch 242) 2014-06-09 16:22:11.332715 7f86a5610700 10 _finish_op 0x30606120 seq 3044 osr(3.1bd 0x3ed0a50)/0x3ed0a50 2014-06-09 16:22:11.332708 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332718 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7200 (1 active) 2014-06-09 16:22:11.332719 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332721 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781e60 (2 active) 2014-06-09 16:22:11.332722 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332723 7f86a5610700 5 _do_op 0x30061b00 seq 3048 osr(3.1f4 0x3ed0970)/0x3ed0970 start 2014-06-09 16:22:11.332726 7f86a5610700 10 _do_transaction on 0xc1f0540 2014-06-09 16:22:11.332725 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (1 active) 2014-06-09 16:22:11.332727 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6fd60] 2014-06-09 16:22:11.332728 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdf09a0] 2014-06-09 16:22:11.332732 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdf09a0] 2014-06-09 16:22:11.332732 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.332734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.332734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.332730 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332738 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a6e00] 2014-06-09 16:22:11.332740 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332742 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332743 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.332743 7f869af78700 20 osd.26 243 _dispatch 0x240efc00 pg_notify(2.4e3(26) epoch 243) v5 2014-06-09 16:22:11.332740 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 212 same_interval_since 242 2014-06-09 16:22:11.332748 7f869af78700 5 -- op tracker -- , seq: 1497, time: 2014-06-09 16:22:10.999681, event: header_read, op: pg_notify(2.4e3(26) epoch 243) 2014-06-09 16:22:11.332747 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332755 7f869af78700 5 -- op tracker -- , seq: 1497, time: 2014-06-09 16:22:10.999682, event: throttled, op: pg_notify(2.4e3(26) epoch 243) 2014-06-09 16:22:11.332753 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332756 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.332764 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.332760 7f869af78700 5 -- op tracker -- , seq: 1497, time: 2014-06-09 16:22:10.999714, event: all_read, op: pg_notify(2.4e3(26) epoch 243) 2014-06-09 16:22:11.332768 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (2 active) 2014-06-09 16:22:11.332770 7f869af78700 5 -- op tracker -- , seq: 1497, time: 2014-06-09 16:22:11.332740, event: dispatched, op: pg_notify(2.4e3(26) epoch 243) 2014-06-09 16:22:11.332767 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332776 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (1 active) 2014-06-09 16:22:11.332776 7f869af78700 5 -- op tracker -- , seq: 1497, time: 2014-06-09 16:22:11.332774, event: waiting_for_osdmap, op: pg_notify(2.4e3(26) epoch 243) 2014-06-09 16:22:11.332779 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332780 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.332782 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240efc00 2014-06-09 16:22:11.332783 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332786 7f869af78700 5 -- op tracker -- , seq: 1497, time: 2014-06-09 16:22:11.332785, event: started, op: pg_notify(2.4e3(26) epoch 243) 2014-06-09 16:22:11.332794 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a6e00] 2014-06-09 16:22:11.332796 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332797 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332800 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.332796 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332808 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02a0 (2 active) 2014-06-09 16:22:11.332809 7f869af78700 5 -- op tracker -- , seq: 1497, time: 2014-06-09 16:22:11.332807, event: done, op: pg_notify(2.4e3(26) epoch 243) 2014-06-09 16:22:11.332813 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.332818 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332817 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3047.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.332820 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332820 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.332820 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (1 active) 2014-06-09 16:22:11.332813 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:2.4e3( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.332823 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (2 active) 2014-06-09 16:22:11.332827 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.332834 7f86a5e11700 10 _do_op 0x30033b00 seq 3047 r = 0, finisher 0x170018e0 0 2014-06-09 16:22:11.332826 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.332837 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.332836 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.332838 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.332839 7f869af78700 20 osd.26 243 _dispatch 0x19eb9500 pg_notify(1.1(20) epoch 242) v5 2014-06-09 16:22:11.332840 7f86a5e11700 10 _finish_op 0x30033b00 seq 3047 osr(3.573 0x3ed18e0)/0x3ed18e0 2014-06-09 16:22:11.332841 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd77920] 2014-06-09 16:22:11.332844 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd77920] 2014-06-09 16:22:11.332845 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.332846 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.332838 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x2e075f00,0x2e075e00,0x2e075d00,0x2e075c00 2014-06-09 16:22:11.332845 7f869af78700 5 -- op tracker -- , seq: 1498, time: 2014-06-09 16:22:10.985243, event: header_read, op: pg_notify(1.1(20) epoch 242) 2014-06-09 16:22:11.332848 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x170018e0] 2014-06-09 16:22:11.332852 7f869af78700 5 -- op tracker -- , seq: 1498, time: 2014-06-09 16:22:10.985244, event: throttled, op: pg_notify(1.1(20) epoch 242) 2014-06-09 16:22:11.332858 7f869af78700 5 -- op tracker -- , seq: 1498, time: 2014-06-09 16:22:10.985263, event: all_read, op: pg_notify(1.1(20) epoch 242) 2014-06-09 16:22:11.332851 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.332864 7f869af78700 5 -- op tracker -- , seq: 1498, time: 2014-06-09 16:22:11.332836, event: dispatched, op: pg_notify(1.1(20) epoch 242) 2014-06-09 16:22:11.332865 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332870 7f869af78700 5 -- op tracker -- , seq: 1498, time: 2014-06-09 16:22:11.332868, event: waiting_for_osdmap, op: pg_notify(1.1(20) epoch 242) 2014-06-09 16:22:11.332879 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.332881 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x19eb9500 2014-06-09 16:22:11.332882 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (3 active) 2014-06-09 16:22:11.332879 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.332886 7f869af78700 5 -- op tracker -- , seq: 1498, time: 2014-06-09 16:22:11.332884, event: started, op: pg_notify(1.1(20) epoch 242) 2014-06-09 16:22:11.332890 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0290 (4 active) 2014-06-09 16:22:11.339637 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02a0 (3 active) 2014-06-09 16:22:11.339647 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (4 active) 2014-06-09 16:22:11.339635 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.339659 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.339666 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.339678 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x1c66b300,0x257c5f00,0x257c5e00,0x270bfe00,0x270bfd00,0x270bfc00,0x270bfb00,0x270bfa00,0x270bf900 2014-06-09 16:22:11.332892 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x170018e0] 2014-06-09 16:22:11.339719 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.339719 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (5 active) 2014-06-09 16:22:11.339721 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.339734 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (6 active) 2014-06-09 16:22:11.339745 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (7 active) 2014-06-09 16:22:11.339756 7f8692f68700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (8 active) 2014-06-09 16:22:11.339766 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (9 active) 2014-06-09 16:22:11.339792 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.339796 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (10 active) 2014-06-09 16:22:11.339800 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0290 (9 active) 2014-06-09 16:22:11.339805 7f868e75f700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (10 active) 2014-06-09 16:22:11.339815 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (11 active) 2014-06-09 16:22:11.339802 7f868ef60700 10 osd.26 243 dequeue_op 0x1c66b300 prio 63 cost 450560 latency 0.621220 osd_op(client.4565.0:32981 rbd_data.11756b8b4567.000000000000159c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.96952573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.339823 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (12 active) 2014-06-09 16:22:11.339831 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (13 active) 2014-06-09 16:22:11.339827 7f868ef60700 5 -- op tracker -- , seq: 638, time: 2014-06-09 16:22:11.339821, event: reached_pg, op: osd_op(client.4565.0:32981 rbd_data.11756b8b4567.000000000000159c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.96952573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.339836 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.339847 7f869af78700 5 -- op tracker -- , seq: 1498, time: 2014-06-09 16:22:11.339844, event: done, op: pg_notify(1.1(20) epoch 242) 2014-06-09 16:22:11.339849 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1c66b300 2014-06-09 16:22:11.339859 7f868ef60700 10 osd.26 243 dequeue_op 0x1c66b300 finish 2014-06-09 16:22:11.339860 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.339863 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.339864 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (12 active) 2014-06-09 16:22:11.339867 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (13 active) 2014-06-09 16:22:11.339870 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 5 notify: (query_epoch:242, epoch_sent:242, info:1.1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.339887 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.339888 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.339890 7f869af78700 20 osd.26 243 _dispatch 0x280c5c00 pg_notify(1.246(21),3.51e(12) epoch 242) v5 2014-06-09 16:22:11.339883 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.339894 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.339896 7f869af78700 5 -- op tracker -- , seq: 1499, time: 2014-06-09 16:22:10.770224, event: header_read, op: pg_notify(1.246(21),3.51e(12) epoch 242) 2014-06-09 16:22:11.339900 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (12 active) 2014-06-09 16:22:11.339906 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (13 active) 2014-06-09 16:22:11.339903 7f869af78700 5 -- op tracker -- , seq: 1499, time: 2014-06-09 16:22:10.770224, event: throttled, op: pg_notify(1.246(21),3.51e(12) epoch 242) 2014-06-09 16:22:11.339910 7f869af78700 5 -- op tracker -- , seq: 1499, time: 2014-06-09 16:22:10.770267, event: all_read, op: pg_notify(1.246(21),3.51e(12) epoch 242) 2014-06-09 16:22:11.339916 7f869af78700 5 -- op tracker -- , seq: 1499, time: 2014-06-09 16:22:11.339886, event: dispatched, op: pg_notify(1.246(21),3.51e(12) epoch 242) 2014-06-09 16:22:11.339909 7f868ef60700 10 osd.26 243 dequeue_op 0x257c5f00 prio 63 cost 516096 latency 0.598584 osd_op(client.4565.0:32982 rbd_data.11756b8b4567.000000000000159c [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~516096] 3.96952573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.339923 7f869af78700 5 -- op tracker -- , seq: 1499, time: 2014-06-09 16:22:11.339920, event: waiting_for_osdmap, op: pg_notify(1.246(21),3.51e(12) epoch 242) 2014-06-09 16:22:11.339930 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.339932 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x280c5c00 2014-06-09 16:22:11.339936 7f868ef60700 5 -- op tracker -- , seq: 639, time: 2014-06-09 16:22:11.339924, event: reached_pg, op: osd_op(client.4565.0:32982 rbd_data.11756b8b4567.000000000000159c [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~516096] 3.96952573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.339946 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.339957 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x257c5f00 2014-06-09 16:22:11.339965 7f868ef60700 10 osd.26 243 dequeue_op 0x257c5f00 finish 2014-06-09 16:22:11.339929 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.339976 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.339986 7f869af78700 5 -- op tracker -- , seq: 1499, time: 2014-06-09 16:22:11.339936, event: started, op: pg_notify(1.246(21),3.51e(12) epoch 242) 2014-06-09 16:22:11.339976 7f868cf5c700 10 osd.26 243 dequeue_op 0x257c5e00 prio 63 cost 3227648 latency 0.597726 osd_op(client.4565.0:32983 rbd_data.11756b8b4567.000000000000159c [set-alloc-hint object_size 4194304 write_size 4194304,write 966656~3227648] 3.96952573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.340005 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (14 active) 2014-06-09 16:22:11.340017 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02b0 (15 active) 2014-06-09 16:22:11.340011 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:1.246( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.340024 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.340034 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.340022 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:3.51e( v 213'674 (0'0,213'674] local-les=191 n=127 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.340041 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (14 active) 2014-06-09 16:22:11.340043 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340039 7f868d75d700 7 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.340048 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.340050 7f868ef60700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.339998 7f868cf5c700 5 -- op tracker -- , seq: 670, time: 2014-06-09 16:22:11.339992, event: reached_pg, op: osd_op(client.4565.0:32983 rbd_data.11756b8b4567.000000000000159c [set-alloc-hint object_size 4194304 write_size 4194304,write 966656~3227648] 3.96952573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.340049 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.51e( v 213'674 (0'0,213'674] local-les=191 n=127 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.340067 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.340060 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.340073 7f868d75d700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02b0 (12 active) 2014-06-09 16:22:11.340076 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340071 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x257c5e00 2014-06-09 16:22:11.340080 7f868cf5c700 10 osd.26 243 dequeue_op 0x257c5e00 finish 2014-06-09 16:22:11.340085 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (11 active) 2014-06-09 16:22:11.340087 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340083 7f869af78700 5 -- op tracker -- , seq: 1499, time: 2014-06-09 16:22:11.340014, event: done, op: pg_notify(1.246(21),3.51e(12) epoch 242) 2014-06-09 16:22:11.340095 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340097 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340092 7f8690f64700 10 osd.26 243 dequeue_op 0x270bfe00 prio 63 cost 237568 latency 0.546210 osd_op(client.4570.0:34184 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.340120 7f8690f64700 5 -- op tracker -- , seq: 734, time: 2014-06-09 16:22:11.340115, event: reached_pg, op: osd_op(client.4570.0:34184 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.340127 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340128 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340129 7f869af78700 20 osd.26 243 _dispatch 0x240ebc00 pg_notify(2.368(23) epoch 242) v5 2014-06-09 16:22:11.340128 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.340134 7f869af78700 5 -- op tracker -- , seq: 1500, time: 2014-06-09 16:22:10.707319, event: header_read, op: pg_notify(2.368(23) epoch 242) 2014-06-09 16:22:11.340142 7f869af78700 5 -- op tracker -- , seq: 1500, time: 2014-06-09 16:22:10.707319, event: throttled, op: pg_notify(2.368(23) epoch 242) 2014-06-09 16:22:11.340140 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bfe00 2014-06-09 16:22:11.340150 7f8690f64700 10 osd.26 243 dequeue_op 0x270bfe00 finish 2014-06-09 16:22:11.340148 7f869af78700 5 -- op tracker -- , seq: 1500, time: 2014-06-09 16:22:10.707344, event: all_read, op: pg_notify(2.368(23) epoch 242) 2014-06-09 16:22:11.340156 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (10 active) 2014-06-09 16:22:11.340158 7f8690f64700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340157 7f868e75f700 10 osd.26 243 dequeue_op 0x270bfd00 prio 63 cost 516096 latency 0.538779 osd_op(client.4570.0:34185 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 237568~516096] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.340176 7f869af78700 5 -- op tracker -- , seq: 1500, time: 2014-06-09 16:22:11.340126, event: dispatched, op: pg_notify(2.368(23) epoch 242) 2014-06-09 16:22:11.340189 7f868e75f700 5 -- op tracker -- , seq: 756, time: 2014-06-09 16:22:11.340174, event: reached_pg, op: osd_op(client.4570.0:34185 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 237568~516096] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.340196 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.340207 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bfd00 2014-06-09 16:22:11.340216 7f868e75f700 10 osd.26 243 dequeue_op 0x270bfd00 finish 2014-06-09 16:22:11.340221 7f868e75f700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (9 active) 2014-06-09 16:22:11.340223 7f868e75f700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340228 7f8689755700 10 osd.26 243 dequeue_op 0x270bfc00 prio 63 cost 1032192 latency 0.526399 osd_op(client.4570.0:34186 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 753664~1032192] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.340248 7f8689755700 5 -- op tracker -- , seq: 763, time: 2014-06-09 16:22:11.340243, event: reached_pg, op: osd_op(client.4570.0:34186 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 753664~1032192] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.340254 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.340265 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bfc00 2014-06-09 16:22:11.340273 7f8689755700 10 osd.26 243 dequeue_op 0x270bfc00 finish 2014-06-09 16:22:11.340278 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (8 active) 2014-06-09 16:22:11.340280 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340285 7f868bf5a700 10 osd.26 243 dequeue_op 0x270bfb00 prio 63 cost 1036288 latency 0.521945 osd_op(client.4570.0:34187 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1785856~1036288] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.340305 7f868bf5a700 5 -- op tracker -- , seq: 784, time: 2014-06-09 16:22:11.340300, event: reached_pg, op: osd_op(client.4570.0:34187 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1785856~1036288] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.340312 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.340324 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bfb00 2014-06-09 16:22:11.340332 7f868bf5a700 10 osd.26 243 dequeue_op 0x270bfb00 finish 2014-06-09 16:22:11.340337 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (7 active) 2014-06-09 16:22:11.340339 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340345 7f868f761700 10 osd.26 243 dequeue_op 0x270bfa00 prio 63 cost 516096 latency 0.519924 osd_op(client.4570.0:34188 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2822144~516096] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.340364 7f868f761700 5 -- op tracker -- , seq: 800, time: 2014-06-09 16:22:11.340359, event: reached_pg, op: osd_op(client.4570.0:34188 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2822144~516096] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.340371 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.340381 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bfa00 2014-06-09 16:22:11.340390 7f868f761700 10 osd.26 243 dequeue_op 0x270bfa00 finish 2014-06-09 16:22:11.340394 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (6 active) 2014-06-09 16:22:11.340397 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340401 7f868c75b700 10 osd.26 243 dequeue_op 0x270bf900 prio 63 cost 856064 latency 0.516684 osd_op(client.4570.0:34189 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3338240~856064] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.340420 7f868c75b700 5 -- op tracker -- , seq: 816, time: 2014-06-09 16:22:11.340415, event: reached_pg, op: osd_op(client.4570.0:34189 rbd_data.117a6b8b4567.00000000000016f7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3338240~856064] 3.f5aba573 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.340426 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.340437 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bf900 2014-06-09 16:22:11.340446 7f868c75b700 10 osd.26 243 dequeue_op 0x270bf900 finish 2014-06-09 16:22:11.340450 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (5 active) 2014-06-09 16:22:11.340452 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340459 7f869af78700 5 -- op tracker -- , seq: 1500, time: 2014-06-09 16:22:11.340457, event: waiting_for_osdmap, op: pg_notify(2.368(23) epoch 242) 2014-06-09 16:22:11.340465 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.340467 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240ebc00 2014-06-09 16:22:11.340472 7f869af78700 5 -- op tracker -- , seq: 1500, time: 2014-06-09 16:22:11.340470, event: started, op: pg_notify(2.368(23) epoch 242) 2014-06-09 16:22:11.340490 7f869af78700 5 -- op tracker -- , seq: 1500, time: 2014-06-09 16:22:11.340488, event: done, op: pg_notify(2.368(23) epoch 242) 2014-06-09 16:22:11.340504 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340506 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340524 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340525 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340526 7f869af78700 20 osd.26 243 _dispatch 0x3004c600 pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.340533 7f869af78700 5 -- op tracker -- , seq: 1501, time: 2014-06-09 16:22:11.053975, event: header_read, op: pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340540 7f869af78700 5 -- op tracker -- , seq: 1501, time: 2014-06-09 16:22:11.053975, event: throttled, op: pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340547 7f869af78700 5 -- op tracker -- , seq: 1501, time: 2014-06-09 16:22:11.053996, event: all_read, op: pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340553 7f869af78700 5 -- op tracker -- , seq: 1501, time: 2014-06-09 16:22:11.340523, event: dispatched, op: pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340560 7f869af78700 5 -- op tracker -- , seq: 1501, time: 2014-06-09 16:22:11.340558, event: waiting_for_osdmap, op: pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340565 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.340568 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004c600 2014-06-09 16:22:11.340574 7f869af78700 5 -- op tracker -- , seq: 1501, time: 2014-06-09 16:22:11.340571, event: started, op: pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340589 7f869af78700 5 -- op tracker -- , seq: 1501, time: 2014-06-09 16:22:11.340587, event: done, op: pg_log(1.691 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340597 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340599 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340612 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340613 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340614 7f869af78700 20 osd.26 243 _dispatch 0x240ea1c0 pg_notify(0.498(20),2.75d(20) epoch 242) v5 2014-06-09 16:22:11.340620 7f869af78700 5 -- op tracker -- , seq: 1502, time: 2014-06-09 16:22:10.947230, event: header_read, op: pg_notify(0.498(20),2.75d(20) epoch 242) 2014-06-09 16:22:11.340627 7f869af78700 5 -- op tracker -- , seq: 1502, time: 2014-06-09 16:22:10.947231, event: throttled, op: pg_notify(0.498(20),2.75d(20) epoch 242) 2014-06-09 16:22:11.340633 7f869af78700 5 -- op tracker -- , seq: 1502, time: 2014-06-09 16:22:10.947287, event: all_read, op: pg_notify(0.498(20),2.75d(20) epoch 242) 2014-06-09 16:22:11.340640 7f869af78700 5 -- op tracker -- , seq: 1502, time: 2014-06-09 16:22:11.340611, event: dispatched, op: pg_notify(0.498(20),2.75d(20) epoch 242) 2014-06-09 16:22:11.340646 7f869af78700 5 -- op tracker -- , seq: 1502, time: 2014-06-09 16:22:11.340644, event: waiting_for_osdmap, op: pg_notify(0.498(20),2.75d(20) epoch 242) 2014-06-09 16:22:11.340650 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.340652 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240ea1c0 2014-06-09 16:22:11.340657 7f869af78700 5 -- op tracker -- , seq: 1502, time: 2014-06-09 16:22:11.340655, event: started, op: pg_notify(0.498(20),2.75d(20) epoch 242) 2014-06-09 16:22:11.340677 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (6 active) 2014-06-09 16:22:11.340682 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340684 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:2.368( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.340692 7f869af78700 5 -- op tracker -- , seq: 1502, time: 2014-06-09 16:22:11.340689, event: done, op: pg_notify(0.498(20),2.75d(20) epoch 242) 2014-06-09 16:22:11.340701 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340702 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340697 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.340708 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.340712 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.340727 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340729 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340730 7f869af78700 20 osd.26 243 _dispatch 0x2fe33200 pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.340723 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 1.691( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.340734 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.340738 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.340737 7f869af78700 5 -- op tracker -- , seq: 1503, time: 2014-06-09 16:22:11.019550, event: header_read, op: pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340745 7f869af78700 5 -- op tracker -- , seq: 1503, time: 2014-06-09 16:22:11.019550, event: throttled, op: pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340742 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.691( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.075598 7 0.000205 2014-06-09 16:22:11.340753 7f869af78700 5 -- op tracker -- , seq: 1503, time: 2014-06-09 16:22:11.019562, event: all_read, op: pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340755 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.691( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.340760 7f869af78700 5 -- op tracker -- , seq: 1503, time: 2014-06-09 16:22:11.340727, event: dispatched, op: pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340767 7f869af78700 5 -- op tracker -- , seq: 1503, time: 2014-06-09 16:22:11.340764, event: waiting_for_osdmap, op: pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340766 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.691( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.340771 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.340775 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe33200 2014-06-09 16:22:11.340776 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.340781 7f869af78700 5 -- op tracker -- , seq: 1503, time: 2014-06-09 16:22:11.340778, event: started, op: pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340786 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.340794 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.340797 7f869af78700 5 -- op tracker -- , seq: 1503, time: 2014-06-09 16:22:11.340794, event: done, op: pg_log(2.372 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.340805 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340806 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340819 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.340820 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.340821 7f869af78700 20 osd.26 243 _dispatch 0x1003a000 pg_notify(0.1a7(20),1.46e(20),1.479(20) epoch 242) v5 2014-06-09 16:22:11.340827 7f869af78700 5 -- op tracker -- , seq: 1504, time: 2014-06-09 16:22:10.899204, event: header_read, op: pg_notify(0.1a7(20),1.46e(20),1.479(20) epoch 242) 2014-06-09 16:22:11.340835 7f869af78700 5 -- op tracker -- , seq: 1504, time: 2014-06-09 16:22:10.899205, event: throttled, op: pg_notify(0.1a7(20),1.46e(20),1.479(20) epoch 242) 2014-06-09 16:22:11.340841 7f869af78700 5 -- op tracker -- , seq: 1504, time: 2014-06-09 16:22:10.899248, event: all_read, op: pg_notify(0.1a7(20),1.46e(20),1.479(20) epoch 242) 2014-06-09 16:22:11.340844 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.340849 7f8691f66700 5 queue_transactions existing osr(1.691 0x384de30)/0x384de30 2014-06-09 16:22:11.340848 7f869af78700 5 -- op tracker -- , seq: 1504, time: 2014-06-09 16:22:11.340818, event: dispatched, op: pg_notify(0.1a7(20),1.46e(20),1.479(20) epoch 242) 2014-06-09 16:22:11.340853 7f8691f66700 10 op_submit_start 3086 2014-06-09 16:22:11.340854 7f8691f66700 5 queue_transactions (trailing journal) 3086 2014-06-09 16:22:11.340856 7f8691f66700 5 queue_op 0x308cbc80 seq 3086 osr(1.691 0x384de30) 2009 bytes (queue has 39 ops and 49362 bytes) 2014-06-09 16:22:11.340856 7f869af78700 5 -- op tracker -- , seq: 1504, time: 2014-06-09 16:22:11.340853, event: waiting_for_osdmap, op: pg_notify(0.1a7(20),1.46e(20),1.479(20) epoch 242) 2014-06-09 16:22:11.340861 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.340863 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1003a000 2014-06-09 16:22:11.340863 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (7 active) 2014-06-09 16:22:11.340868 7f869af78700 5 -- op tracker -- , seq: 1504, time: 2014-06-09 16:22:11.340866, event: started, op: pg_notify(0.1a7(20),1.46e(20),1.479(20) epoch 242) 2014-06-09 16:22:11.340867 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 3 notify: (query_epoch:242, epoch_sent:242, info:2.75d( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.340879 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.340888 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.340892 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.340900 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0290 (8 active) 2014-06-09 16:22:11.340902 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.372( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.340912 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.340905 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 9 notify: (query_epoch:242, epoch_sent:242, info:0.1a7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.340916 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.340918 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.340919 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.372( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.804830 7 0.000225 2014-06-09 16:22:11.340928 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.340934 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340929 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.372( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.340939 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.372( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.340941 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 9 notify: (query_epoch:242, epoch_sent:242, info:1.46e( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.340948 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.340952 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.340962 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.340957 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.340968 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0290 (7 active) 2014-06-09 16:22:11.340971 7f868e75f700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.340965 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.340989 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0200 (8 active) 2014-06-09 16:22:11.340997 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 9 notify: (query_epoch:242, epoch_sent:242, info:1.479( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.341011 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.341009 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.341015 7f868ef60700 5 queue_transactions existing osr(2.372 0x2ed1150)/0x2ed1150 2014-06-09 16:22:11.341018 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.341022 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341027 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0200 (7 active) 2014-06-09 16:22:11.341030 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341038 7f869af78700 5 -- op tracker -- , seq: 1504, time: 2014-06-09 16:22:11.341025, event: done, op: pg_notify(0.1a7(20),1.46e(20),1.479(20) epoch 242) 2014-06-09 16:22:11.341047 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341083 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341084 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341085 7f869af78700 20 osd.26 243 _dispatch 0x19fda8c0 pg_notify(3.4d7(10) epoch 242) v5 2014-06-09 16:22:11.341090 7f869af78700 5 -- op tracker -- , seq: 1505, time: 2014-06-09 16:22:10.983890, event: header_read, op: pg_notify(3.4d7(10) epoch 242) 2014-06-09 16:22:11.341097 7f869af78700 5 -- op tracker -- , seq: 1505, time: 2014-06-09 16:22:10.983891, event: throttled, op: pg_notify(3.4d7(10) epoch 242) 2014-06-09 16:22:11.341103 7f869af78700 5 -- op tracker -- , seq: 1505, time: 2014-06-09 16:22:10.983911, event: all_read, op: pg_notify(3.4d7(10) epoch 242) 2014-06-09 16:22:11.341109 7f869af78700 5 -- op tracker -- , seq: 1505, time: 2014-06-09 16:22:11.341082, event: dispatched, op: pg_notify(3.4d7(10) epoch 242) 2014-06-09 16:22:11.341115 7f869af78700 5 -- op tracker -- , seq: 1505, time: 2014-06-09 16:22:11.341113, event: waiting_for_osdmap, op: pg_notify(3.4d7(10) epoch 242) 2014-06-09 16:22:11.341119 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.341121 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x19fda8c0 2014-06-09 16:22:11.341125 7f869af78700 5 -- op tracker -- , seq: 1505, time: 2014-06-09 16:22:11.341124, event: started, op: pg_notify(3.4d7(10) epoch 242) 2014-06-09 16:22:11.341144 7f869af78700 5 -- op tracker -- , seq: 1505, time: 2014-06-09 16:22:11.341142, event: done, op: pg_notify(3.4d7(10) epoch 242) 2014-06-09 16:22:11.341152 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341153 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341166 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341168 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341169 7f869af78700 20 osd.26 243 _dispatch 0x1405d340 pg_notify(0.138(20) epoch 242) v5 2014-06-09 16:22:11.341174 7f869af78700 5 -- op tracker -- , seq: 1506, time: 2014-06-09 16:22:10.852700, event: header_read, op: pg_notify(0.138(20) epoch 242) 2014-06-09 16:22:11.341180 7f869af78700 5 -- op tracker -- , seq: 1506, time: 2014-06-09 16:22:10.852701, event: throttled, op: pg_notify(0.138(20) epoch 242) 2014-06-09 16:22:11.341186 7f869af78700 5 -- op tracker -- , seq: 1506, time: 2014-06-09 16:22:10.852736, event: all_read, op: pg_notify(0.138(20) epoch 242) 2014-06-09 16:22:11.341192 7f869af78700 5 -- op tracker -- , seq: 1506, time: 2014-06-09 16:22:11.341166, event: dispatched, op: pg_notify(0.138(20) epoch 242) 2014-06-09 16:22:11.341198 7f869af78700 5 -- op tracker -- , seq: 1506, time: 2014-06-09 16:22:11.341196, event: waiting_for_osdmap, op: pg_notify(0.138(20) epoch 242) 2014-06-09 16:22:11.341202 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.341205 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1405d340 2014-06-09 16:22:11.341210 7f869af78700 5 -- op tracker -- , seq: 1506, time: 2014-06-09 16:22:11.341208, event: started, op: pg_notify(0.138(20) epoch 242) 2014-06-09 16:22:11.341228 7f869af78700 5 -- op tracker -- , seq: 1506, time: 2014-06-09 16:22:11.341226, event: done, op: pg_notify(0.138(20) epoch 242) 2014-06-09 16:22:11.341238 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341240 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341256 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341257 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341258 7f869af78700 20 osd.26 243 _dispatch 0x2a072c00 pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.341265 7f869af78700 5 -- op tracker -- , seq: 1507, time: 2014-06-09 16:22:11.003820, event: header_read, op: pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.341272 7f869af78700 5 -- op tracker -- , seq: 1507, time: 2014-06-09 16:22:11.003820, event: throttled, op: pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.341280 7f869af78700 5 -- op tracker -- , seq: 1507, time: 2014-06-09 16:22:11.003829, event: all_read, op: pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.341286 7f869af78700 5 -- op tracker -- , seq: 1507, time: 2014-06-09 16:22:11.341255, event: dispatched, op: pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.341293 7f869af78700 5 -- op tracker -- , seq: 1507, time: 2014-06-09 16:22:11.341290, event: waiting_for_osdmap, op: pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.341298 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.341301 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a072c00 2014-06-09 16:22:11.341307 7f869af78700 5 -- op tracker -- , seq: 1507, time: 2014-06-09 16:22:11.341304, event: started, op: pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.341321 7f869af78700 5 -- op tracker -- , seq: 1507, time: 2014-06-09 16:22:11.341318, event: done, op: pg_log(1.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.341330 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341332 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341344 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341345 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341346 7f869af78700 20 osd.26 243 _dispatch 0x3002cd80 osd_map(243..243 src has 1..243) v3 2014-06-09 16:22:11.341353 7f869af78700 3 osd.26 243 handle_osd_map epochs [243,243], i have 243, src has [1,243] 2014-06-09 16:22:11.341355 7f869af78700 10 osd.26 243 no new maps here, dropping 2014-06-09 16:22:11.341357 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341358 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341372 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341373 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341375 7f869af78700 20 osd.26 243 _dispatch 0x240ef500 pg_notify(0.411(21) epoch 242) v5 2014-06-09 16:22:11.341380 7f869af78700 5 -- op tracker -- , seq: 1508, time: 2014-06-09 16:22:10.784846, event: header_read, op: pg_notify(0.411(21) epoch 242) 2014-06-09 16:22:11.341387 7f869af78700 5 -- op tracker -- , seq: 1508, time: 2014-06-09 16:22:10.784847, event: throttled, op: pg_notify(0.411(21) epoch 242) 2014-06-09 16:22:11.341393 7f869af78700 5 -- op tracker -- , seq: 1508, time: 2014-06-09 16:22:10.784876, event: all_read, op: pg_notify(0.411(21) epoch 242) 2014-06-09 16:22:11.341399 7f869af78700 5 -- op tracker -- , seq: 1508, time: 2014-06-09 16:22:11.341372, event: dispatched, op: pg_notify(0.411(21) epoch 242) 2014-06-09 16:22:11.341405 7f869af78700 5 -- op tracker -- , seq: 1508, time: 2014-06-09 16:22:11.341403, event: waiting_for_osdmap, op: pg_notify(0.411(21) epoch 242) 2014-06-09 16:22:11.341409 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.341411 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240ef500 2014-06-09 16:22:11.341416 7f869af78700 5 -- op tracker -- , seq: 1508, time: 2014-06-09 16:22:11.341414, event: started, op: pg_notify(0.411(21) epoch 242) 2014-06-09 16:22:11.341432 7f869af78700 5 -- op tracker -- , seq: 1508, time: 2014-06-09 16:22:11.341430, event: done, op: pg_notify(0.411(21) epoch 242) 2014-06-09 16:22:11.341440 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341442 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341458 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341460 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341461 7f869af78700 20 osd.26 243 _dispatch 0x230baa80 pg_notify(3.67f(10) epoch 242) v5 2014-06-09 16:22:11.341466 7f869af78700 5 -- op tracker -- , seq: 1509, time: 2014-06-09 16:22:10.707650, event: header_read, op: pg_notify(3.67f(10) epoch 242) 2014-06-09 16:22:11.341473 7f869af78700 5 -- op tracker -- , seq: 1509, time: 2014-06-09 16:22:10.707650, event: throttled, op: pg_notify(3.67f(10) epoch 242) 2014-06-09 16:22:11.341479 7f869af78700 5 -- op tracker -- , seq: 1509, time: 2014-06-09 16:22:10.708212, event: all_read, op: pg_notify(3.67f(10) epoch 242) 2014-06-09 16:22:11.341485 7f869af78700 5 -- op tracker -- , seq: 1509, time: 2014-06-09 16:22:11.341458, event: dispatched, op: pg_notify(3.67f(10) epoch 242) 2014-06-09 16:22:11.341492 7f869af78700 5 -- op tracker -- , seq: 1509, time: 2014-06-09 16:22:11.341490, event: waiting_for_osdmap, op: pg_notify(3.67f(10) epoch 242) 2014-06-09 16:22:11.341496 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.341498 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x230baa80 2014-06-09 16:22:11.341502 7f869af78700 5 -- op tracker -- , seq: 1509, time: 2014-06-09 16:22:11.341501, event: started, op: pg_notify(3.67f(10) epoch 242) 2014-06-09 16:22:11.341520 7f869af78700 5 -- op tracker -- , seq: 1509, time: 2014-06-09 16:22:11.341518, event: done, op: pg_notify(3.67f(10) epoch 242) 2014-06-09 16:22:11.341528 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341529 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341543 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341545 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341546 7f869af78700 20 osd.26 243 _dispatch 0x3046bc00 pg_notify(3.3e6(10),0.6b3(20) epoch 243) v5 2014-06-09 16:22:11.341551 7f869af78700 5 -- op tracker -- , seq: 1510, time: 2014-06-09 16:22:11.054040, event: header_read, op: pg_notify(3.3e6(10),0.6b3(20) epoch 243) 2014-06-09 16:22:11.341558 7f869af78700 5 -- op tracker -- , seq: 1510, time: 2014-06-09 16:22:11.054041, event: throttled, op: pg_notify(3.3e6(10),0.6b3(20) epoch 243) 2014-06-09 16:22:11.341565 7f869af78700 5 -- op tracker -- , seq: 1510, time: 2014-06-09 16:22:11.054079, event: all_read, op: pg_notify(3.3e6(10),0.6b3(20) epoch 243) 2014-06-09 16:22:11.341571 7f869af78700 5 -- op tracker -- , seq: 1510, time: 2014-06-09 16:22:11.341543, event: dispatched, op: pg_notify(3.3e6(10),0.6b3(20) epoch 243) 2014-06-09 16:22:11.341577 7f869af78700 5 -- op tracker -- , seq: 1510, time: 2014-06-09 16:22:11.341575, event: waiting_for_osdmap, op: pg_notify(3.3e6(10),0.6b3(20) epoch 243) 2014-06-09 16:22:11.341582 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.341584 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046bc00 2014-06-09 16:22:11.341589 7f869af78700 5 -- op tracker -- , seq: 1510, time: 2014-06-09 16:22:11.341587, event: started, op: pg_notify(3.3e6(10),0.6b3(20) epoch 243) 2014-06-09 16:22:11.341612 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0290 (8 active) 2014-06-09 16:22:11.341616 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:242, info:3.4d7( v 214'562 (0'0,214'562] local-les=212 n=109 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.341629 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.341639 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.341643 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:0.138( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.341653 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.341662 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.341665 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.341675 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.5e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.341685 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.341688 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.341692 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.149532 7 0.000183 2014-06-09 16:22:11.341702 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.341712 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.341721 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.341729 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.341737 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.341782 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.341786 7f868e75f700 5 queue_transactions existing osr(1.5e 0x384c5d0)/0x384c5d0 2014-06-09 16:22:11.341806 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (9 active) 2014-06-09 16:22:11.341806 7f869af78700 5 -- op tracker -- , seq: 1510, time: 2014-06-09 16:22:11.341803, event: done, op: pg_notify(3.3e6(10),0.6b3(20) epoch 243) 2014-06-09 16:22:11.341815 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341817 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341812 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:0.6b3( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.341825 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.341835 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.341836 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.341838 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.341839 7f869af78700 20 osd.26 243 _dispatch 0x240ef6c0 pg_notify(3.4d1(10) epoch 242) v5 2014-06-09 16:22:11.341841 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (8 active) 2014-06-09 16:22:11.341843 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341844 7f869af78700 5 -- op tracker -- , seq: 1511, time: 2014-06-09 16:22:10.984430, event: header_read, op: pg_notify(3.4d1(10) epoch 242) 2014-06-09 16:22:11.341848 7f8690f64700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341851 7f869af78700 5 -- op tracker -- , seq: 1511, time: 2014-06-09 16:22:10.984431, event: throttled, op: pg_notify(3.4d1(10) epoch 242) 2014-06-09 16:22:11.341857 7f869af78700 5 -- op tracker -- , seq: 1511, time: 2014-06-09 16:22:10.984450, event: all_read, op: pg_notify(3.4d1(10) epoch 242) 2014-06-09 16:22:11.341897 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341903 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341906 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341910 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341914 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341918 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.341922 7f869af78700 5 -- op tracker -- , seq: 1511, time: 2014-06-09 16:22:11.341836, event: dispatched, op: pg_notify(3.4d1(10) epoch 242) 2014-06-09 16:22:11.341931 7f869af78700 5 -- op tracker -- , seq: 1511, time: 2014-06-09 16:22:11.341929, event: waiting_for_osdmap, op: pg_notify(3.4d1(10) epoch 242) 2014-06-09 16:22:11.341936 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.341938 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240ef6c0 2014-06-09 16:22:11.341943 7f869af78700 5 -- op tracker -- , seq: 1511, time: 2014-06-09 16:22:11.341941, event: started, op: pg_notify(3.4d1(10) epoch 242) 2014-06-09 16:22:11.341963 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0200 (9 active) 2014-06-09 16:22:11.341972 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 3 notify: (query_epoch:242, epoch_sent:242, info:3.4d1( v 213'606 (0'0,213'606] local-les=201 n=120 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.341984 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.341993 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.341999 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0200 (8 active) 2014-06-09 16:22:11.342002 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.342035 7f869af78700 5 -- op tracker -- , seq: 1511, time: 2014-06-09 16:22:11.342032, event: done, op: pg_notify(3.4d1(10) epoch 242) 2014-06-09 16:22:11.342044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.342045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.342061 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.342062 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.342064 7f869af78700 20 osd.26 243 _dispatch 0x2fe32c00 pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.342096 7f869af78700 5 -- op tracker -- , seq: 1512, time: 2014-06-09 16:22:11.044375, event: header_read, op: pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.342104 7f869af78700 5 -- op tracker -- , seq: 1512, time: 2014-06-09 16:22:11.044375, event: throttled, op: pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.342112 7f869af78700 5 -- op tracker -- , seq: 1512, time: 2014-06-09 16:22:11.044388, event: all_read, op: pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.342119 7f869af78700 5 -- op tracker -- , seq: 1512, time: 2014-06-09 16:22:11.342061, event: dispatched, op: pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.342127 7f869af78700 5 -- op tracker -- , seq: 1512, time: 2014-06-09 16:22:11.342124, event: waiting_for_osdmap, op: pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.342131 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.342135 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe32c00 2014-06-09 16:22:11.342140 7f869af78700 5 -- op tracker -- , seq: 1512, time: 2014-06-09 16:22:11.342137, event: started, op: pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.342155 7f869af78700 5 -- op tracker -- , seq: 1512, time: 2014-06-09 16:22:11.342152, event: done, op: pg_log(1.38d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.342163 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.342164 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.342176 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.342178 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.342179 7f869af78700 20 osd.26 243 _dispatch 0x48fca00 pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.342185 7f869af78700 5 -- op tracker -- , seq: 1513, time: 2014-06-09 16:22:10.909545, event: header_read, op: pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.342192 7f869af78700 5 -- op tracker -- , seq: 1513, time: 2014-06-09 16:22:10.909546, event: throttled, op: pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.342199 7f869af78700 5 -- op tracker -- , seq: 1513, time: 2014-06-09 16:22:10.909574, event: all_read, op: pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.342206 7f869af78700 5 -- op tracker -- , seq: 1513, time: 2014-06-09 16:22:11.342176, event: dispatched, op: pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.342213 7f869af78700 5 -- op tracker -- , seq: 1513, time: 2014-06-09 16:22:11.342211, event: waiting_for_osdmap, op: pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.342218 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) v4 from osd.9 2014-06-09 16:22:11.342222 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x48fca00 2014-06-09 16:22:11.342227 7f869af78700 5 -- op tracker -- , seq: 1513, time: 2014-06-09 16:22:11.342225, event: started, op: pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.342244 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (9 active) 2014-06-09 16:22:11.342243 7f869af78700 5 -- op tracker -- , seq: 1513, time: 2014-06-09 16:22:11.342240, event: done, op: pg_log(3.94 epoch 242 log log((214'567,214'571], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.342251 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.342253 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.342250 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.342265 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.342267 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.342268 7f869af78700 20 osd.26 243 _dispatch 0x2fe71500 pg_notify(3.1bd(10),3.750(10) epoch 242) v5 2014-06-09 16:22:11.342262 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.38d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.342274 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.342273 7f869af78700 5 -- op tracker -- , seq: 1514, time: 2014-06-09 16:22:10.991826, event: header_read, op: pg_notify(3.1bd(10),3.750(10) epoch 242) 2014-06-09 16:22:11.342277 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.342280 7f869af78700 5 -- op tracker -- , seq: 1514, time: 2014-06-09 16:22:10.991827, event: throttled, op: pg_notify(3.1bd(10),3.750(10) epoch 242) 2014-06-09 16:22:11.342281 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.070521 7 0.000188 2014-06-09 16:22:11.342287 7f869af78700 5 -- op tracker -- , seq: 1514, time: 2014-06-09 16:22:10.991859, event: all_read, op: pg_notify(3.1bd(10),3.750(10) epoch 242) 2014-06-09 16:22:11.342294 7f869af78700 5 -- op tracker -- , seq: 1514, time: 2014-06-09 16:22:11.342265, event: dispatched, op: pg_notify(3.1bd(10),3.750(10) epoch 242) 2014-06-09 16:22:11.342291 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.342300 7f869af78700 5 -- op tracker -- , seq: 1514, time: 2014-06-09 16:22:11.342298, event: waiting_for_osdmap, op: pg_notify(3.1bd(10),3.750(10) epoch 242) 2014-06-09 16:22:11.342305 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.342301 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.342307 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2fe71500 2014-06-09 16:22:11.342312 7f869af78700 5 -- op tracker -- , seq: 1514, time: 2014-06-09 16:22:11.342309, event: started, op: pg_notify(3.1bd(10),3.750(10) epoch 242) 2014-06-09 16:22:11.342309 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.342319 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.342326 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.342371 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.342376 7f868cf5c700 5 queue_transactions existing osr(1.38d 0x384d570)/0x384d570 2014-06-09 16:22:11.342383 7f8690f64700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.342768 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.343407 7f868ff62700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.345060 7f868b759700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332894 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.345298 7f868a757700 10 osd.26 243 dequeue_op 0x2e075f00 prio 63 cost 221184 latency 0.403973 osd_op(client.4577.0:37964 rbd_data.11816b8b4567.000000000000198d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.82d661bd RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.345328 7f868a757700 5 -- op tracker -- , seq: 1042, time: 2014-06-09 16:22:11.345322, event: reached_pg, op: osd_op(client.4577.0:37964 rbd_data.11816b8b4567.000000000000198d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~221184] 3.82d661bd RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.345340 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.345355 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x2e075f00 2014-06-09 16:22:11.345366 7f868a757700 10 osd.26 243 dequeue_op 0x2e075f00 finish 2014-06-09 16:22:11.345373 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (8 active) 2014-06-09 16:22:11.345376 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.345384 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (7 active) 2014-06-09 16:22:11.345387 7f8692767700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.332895 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c480 (1 active) 2014-06-09 16:22:11.332900 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.345395 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4ea0 (2 active) 2014-06-09 16:22:11.345402 7f8691f66700 10 op_submit_finish 3086 2014-06-09 16:22:11.345408 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 3 notify: (query_epoch:242, epoch_sent:242, info:0.498( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.345423 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3048.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.345428 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.345423 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.345434 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.345442 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (6 active) 2014-06-09 16:22:11.345445 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.345450 7f868ef60700 10 op_submit_start 3087 2014-06-09 16:22:11.345451 7f868ef60700 5 queue_transactions (trailing journal) 3087 2014-06-09 16:22:11.345453 7f868ef60700 5 queue_op 0x309a3020 seq 3087 osr(2.372 0x2ed1150) 1883 bytes (queue has 42 ops and 55008 bytes) 2014-06-09 16:22:11.345457 7f868ef60700 10 op_submit_finish 3087 2014-06-09 16:22:11.345464 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (5 active) 2014-06-09 16:22:11.345467 7f868ef60700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.345471 7f86a5e11700 5 _do_op 0x308cbe00 seq 3049 osr(2.74b 0x3ed0110)/0x3ed0110 start 2014-06-09 16:22:11.345476 7f86a5e11700 10 _do_transaction on 0x2eab980 2014-06-09 16:22:11.345475 7f86a5610700 10 _do_op 0x30061b00 seq 3048 r = 0, finisher 0x189aa9e0 0 2014-06-09 16:22:11.345483 7f86a5610700 10 _finish_op 0x30061b00 seq 3048 osr(3.1f4 0x3ed0970)/0x3ed0970 2014-06-09 16:22:11.345484 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.345489 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781e60 (1 active) 2014-06-09 16:22:11.345491 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca48360 (2 active) 2014-06-09 16:22:11.345494 7f86a5610700 5 _do_op 0x30033aa0 seq 3050 osr(3.4d1 0x3ed17a0)/0x3ed17a0 start 2014-06-09 16:22:11.345496 7f86a5610700 10 _do_transaction on 0x39db380 2014-06-09 16:22:11.345504 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.345509 7f868e75f700 10 op_submit_start 3088 2014-06-09 16:22:11.345510 7f868e75f700 5 queue_transactions (trailing journal) 3088 2014-06-09 16:22:11.345512 7f868e75f700 5 queue_op 0x2fc8cb40 seq 3088 osr(1.5e 0x384c5d0) 1880 bytes (queue has 41 ops and 54045 bytes) 2014-06-09 16:22:11.345514 7f868e75f700 10 op_submit_finish 3088 2014-06-09 16:22:11.345520 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:0.411( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.345532 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.345542 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.345547 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:3.67f( v 214'639 (0'0,214'639] local-les=203 n=124 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.345559 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.345569 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.345572 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.3e6( v 213'623 (0'0,213'623] local-les=210 n=121 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.345588 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.345584 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.345594 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.345600 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0290 (4 active) 2014-06-09 16:22:11.345603 7f868e75f700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.345609 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aa9e0] 2014-06-09 16:22:11.345608 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3049.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.345613 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.345618 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aa9e0] 2014-06-09 16:22:11.345620 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.345621 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.345667 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.345684 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3050.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.345689 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.345708 7f86a5610700 10 _do_op 0x30033aa0 seq 3050 r = 0, finisher 0xbd69140 0 2014-06-09 16:22:11.345712 7f86a5610700 10 _finish_op 0x30033aa0 seq 3050 osr(3.4d1 0x3ed17a0)/0x3ed17a0 2014-06-09 16:22:11.345718 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca48360 (1 active) 2014-06-09 16:22:11.345719 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd69140] 2014-06-09 16:22:11.345720 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77ac60 (2 active) 2014-06-09 16:22:11.345722 7f86a5610700 5 _do_op 0x308cacc0 seq 3051 osr(3.4b3 0x3ed1480)/0x3ed1480 start 2014-06-09 16:22:11.345725 7f86a5610700 10 _do_transaction on 0xbf45740 2014-06-09 16:22:11.345727 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd69140] 2014-06-09 16:22:11.345729 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.345730 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.345731 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.345764 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.345781 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3051.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.345785 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.345808 7f86a5610700 10 _do_op 0x308cacc0 seq 3051 r = 0, finisher 0xbd67ea0 0 2014-06-09 16:22:11.345812 7f86a5610700 10 _finish_op 0x308cacc0 seq 3051 osr(3.4b3 0x3ed1480)/0x3ed1480 2014-06-09 16:22:11.345818 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77ac60 (1 active) 2014-06-09 16:22:11.345820 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd67ea0] 2014-06-09 16:22:11.345821 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc000 (2 active) 2014-06-09 16:22:11.345823 7f86a5610700 5 _do_op 0x30606b40 seq 3052 osr(2.5d3 0x2ed1ab0)/0x2ed1ab0 start 2014-06-09 16:22:11.345826 7f86a5610700 10 _do_transaction on 0xc24f8c0 2014-06-09 16:22:11.345831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd67ea0] 2014-06-09 16:22:11.345833 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.345832 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.345834 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.345866 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.345882 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3052.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.345886 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.345910 7f86a5610700 10 _do_op 0x30606b40 seq 3052 r = 0, finisher 0x1a002860 0 2014-06-09 16:22:11.345914 7f86a5610700 10 _finish_op 0x30606b40 seq 3052 osr(2.5d3 0x2ed1ab0)/0x2ed1ab0 2014-06-09 16:22:11.345920 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc000 (1 active) 2014-06-09 16:22:11.345921 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1a002860] 2014-06-09 16:22:11.345922 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781680 (2 active) 2014-06-09 16:22:11.345925 7f86a5610700 5 _do_op 0x2c06ed80 seq 3053 osr(3.77c 0x3ed1dd0)/0x3ed1dd0 start 2014-06-09 16:22:11.345928 7f86a5610700 10 _do_transaction on 0x3488540 2014-06-09 16:22:11.345930 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1a002860] 2014-06-09 16:22:11.345932 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.345933 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.345934 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.345966 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.345983 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3053.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.345987 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346005 7f86a5610700 10 _do_op 0x2c06ed80 seq 3053 r = 0, finisher 0x1322c400 0 2014-06-09 16:22:11.346012 7f86a5610700 10 _finish_op 0x2c06ed80 seq 3053 osr(3.77c 0x3ed1dd0)/0x3ed1dd0 2014-06-09 16:22:11.346019 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322c400] 2014-06-09 16:22:11.346022 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781680 (1 active) 2014-06-09 16:22:11.346024 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322c400] 2014-06-09 16:22:11.346025 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f0e0 (2 active) 2014-06-09 16:22:11.346026 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346027 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346027 7f86a5610700 5 _do_op 0x306060c0 seq 3054 osr(3.750 0x3ed1eb0)/0x3ed1eb0 start 2014-06-09 16:22:11.346030 7f86a5610700 10 _do_transaction on 0xbf45d40 2014-06-09 16:22:11.346037 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.346072 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.346089 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3054.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.346093 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346113 7f86a5610700 10 _do_op 0x306060c0 seq 3054 r = 0, finisher 0xbd6f980 0 2014-06-09 16:22:11.346116 7f86a5610700 10 _finish_op 0x306060c0 seq 3054 osr(3.750 0x3ed1eb0)/0x3ed1eb0 2014-06-09 16:22:11.346122 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f0e0 (1 active) 2014-06-09 16:22:11.346124 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6f980] 2014-06-09 16:22:11.346124 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89fc20 (2 active) 2014-06-09 16:22:11.346127 7f86a5610700 5 _do_op 0x30032240 seq 3055 osr(0.479 0x2bd7490)/0x2bd7490 start 2014-06-09 16:22:11.346130 7f86a5610700 10 _do_transaction on 0xc142240 2014-06-09 16:22:11.346131 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6f980] 2014-06-09 16:22:11.346132 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346136 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.346168 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.346184 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3055.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.346188 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346212 7f86a5610700 10 _do_op 0x30032240 seq 3055 r = 0, finisher 0x1a002880 0 2014-06-09 16:22:11.346215 7f86a5610700 10 _finish_op 0x30032240 seq 3055 osr(0.479 0x2bd7490)/0x2bd7490 2014-06-09 16:22:11.346221 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89fc20 (1 active) 2014-06-09 16:22:11.346223 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1a002880] 2014-06-09 16:22:11.346224 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89ed80 (2 active) 2014-06-09 16:22:11.346226 7f86a5610700 5 _do_op 0x30606000 seq 3056 osr(3.1c7 0x3ed0a10)/0x3ed0a10 start 2014-06-09 16:22:11.346228 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1a002880] 2014-06-09 16:22:11.346229 7f86a5610700 10 _do_transaction on 0x3338300 2014-06-09 16:22:11.346230 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346231 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346231 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346234 7f86a5610700 10 _do_op 0x30606000 seq 3056 r = 0, finisher 0x18743800 0 2014-06-09 16:22:11.346237 7f86a5610700 10 _finish_op 0x30606000 seq 3056 osr(3.1c7 0x3ed0a10)/0x3ed0a10 2014-06-09 16:22:11.346242 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89ed80 (1 active) 2014-06-09 16:22:11.346243 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x18743800] 2014-06-09 16:22:11.346244 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6480 (2 active) 2014-06-09 16:22:11.346246 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x18743800] 2014-06-09 16:22:11.346247 7f86a5610700 5 _do_op 0x30032180 seq 3057 osr(3.1aa 0x3ed0ac0)/0x3ed0ac0 start 2014-06-09 16:22:11.346248 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346249 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346249 7f86a5610700 10 _do_transaction on 0x3a07380 2014-06-09 16:22:11.346251 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346254 7f86a5610700 10 _do_op 0x30032180 seq 3057 r = 0, finisher 0xbd6bbe0 0 2014-06-09 16:22:11.346257 7f86a5610700 10 _finish_op 0x30032180 seq 3057 osr(3.1aa 0x3ed0ac0)/0x3ed0ac0 2014-06-09 16:22:11.346261 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6480 (1 active) 2014-06-09 16:22:11.346263 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6bbe0] 2014-06-09 16:22:11.346264 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac47e0 (2 active) 2014-06-09 16:22:11.346266 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6bbe0] 2014-06-09 16:22:11.346267 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346266 7f86a5610700 5 _do_op 0x2c06ed20 seq 3058 osr(0.1f1 0x2bd68a0)/0x2bd68a0 start 2014-06-09 16:22:11.346269 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346270 7f86a5610700 10 _do_transaction on 0x2e62a80 2014-06-09 16:22:11.346277 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.346310 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.346326 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3058.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.346330 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346347 7f86a5610700 10 _do_op 0x2c06ed20 seq 3058 r = 0, finisher 0xbd691c0 0 2014-06-09 16:22:11.346351 7f86a5610700 10 _finish_op 0x2c06ed20 seq 3058 osr(0.1f1 0x2bd68a0)/0x2bd68a0 2014-06-09 16:22:11.346356 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac47e0 (1 active) 2014-06-09 16:22:11.346358 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca48240 (2 active) 2014-06-09 16:22:11.346358 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd691c0] 2014-06-09 16:22:11.346360 7f86a5610700 5 _do_op 0x30033a40 seq 3059 osr(2.b 0x2ed02f0)/0x2ed02f0 start 2014-06-09 16:22:11.346363 7f86a5610700 10 _do_transaction on 0x3934c00 2014-06-09 16:22:11.346360 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5a1a0,0xbef0f00,0xbce7d00,0xbd6bb00,0x30629b80,0x4eb89c0,0xc169680,0xcd0f820,0x1322d480,0xc411da0] 2014-06-09 16:22:11.346365 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd691c0] 2014-06-09 16:22:11.346366 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346367 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346368 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.346366 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 208 same_interval_since 242 2014-06-09 16:22:11.346389 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346399 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.346409 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (5 active) 2014-06-09 16:22:11.346406 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346414 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3059.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.346418 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346423 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.346432 7f86a5610700 10 _do_op 0x30033a40 seq 3059 r = 0, finisher 0xbd70c80 0 2014-06-09 16:22:11.346424 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 201 same_interval_since 242 2014-06-09 16:22:11.346435 7f86a5610700 10 _finish_op 0x30033a40 seq 3059 osr(2.b 0x2ed02f0)/0x2ed02f0 2014-06-09 16:22:11.346428 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346440 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca48240 (1 active) 2014-06-09 16:22:11.346441 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c07e0 (2 active) 2014-06-09 16:22:11.346437 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346443 7f86a5610700 5 _do_op 0x309a3920 seq 3060 osr(2.755 0x3ed0100)/0x3ed0100 start 2014-06-09 16:22:11.346445 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.346446 7f86a5610700 10 _do_transaction on 0xc3035c0 2014-06-09 16:22:11.346452 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.346452 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (4 active) 2014-06-09 16:22:11.346455 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (5 active) 2014-06-09 16:22:11.346450 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346458 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346466 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.346470 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x21967900,0x21967800,0x21967700,0x21967600,0x16bd6e00,0x16bd6d00,0x1c54ae00,0x1c54ad00 2014-06-09 16:22:11.346481 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.346478 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346495 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3060.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.346499 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346506 7f868cf5c700 10 op_submit_start 3089 2014-06-09 16:22:11.346509 7f868cf5c700 5 queue_transactions (trailing journal) 3089 2014-06-09 16:22:11.346510 7f868cf5c700 5 queue_op 0x300339e0 seq 3089 osr(1.38d 0x384d570) 1883 bytes (queue has 31 ops and 45897 bytes) 2014-06-09 16:22:11.346513 7f868cf5c700 10 op_submit_finish 3089 2014-06-09 16:22:11.346515 7f86a5610700 10 _do_op 0x309a3920 seq 3060 r = 0, finisher 0x30509d20 0 2014-06-09 16:22:11.346519 7f86a5610700 10 _finish_op 0x309a3920 seq 3060 osr(2.755 0x3ed0100)/0x3ed0100 2014-06-09 16:22:11.346521 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c07e0 (1 active) 2014-06-09 16:22:11.346523 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6120 (2 active) 2014-06-09 16:22:11.346526 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.346525 7f86a5610700 5 _do_op 0x30032120 seq 3061 osr(2.1ae 0x2ed08f0)/0x2ed08f0 start 2014-06-09 16:22:11.346528 7f86a5610700 10 _do_transaction on 0xc0e43c0 2014-06-09 16:22:11.346517 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'567 (0'0,214'567] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MLogRec from 9 2014-06-09 16:22:11.346533 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.346528 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346530 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346531 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'567 (0'0,214'567] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd9 2014-06-09 16:22:11.346545 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.346550 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (4 active) 2014-06-09 16:22:11.346543 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'567 (0'0,214'567] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.346555 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (5 active) 2014-06-09 16:22:11.346553 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'567 (0'0,214'567] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.346546 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.346563 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.346561 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'567 (0'0,214'567] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.9: log((214'567,214'571], crt=0'0) missing(0) 2014-06-09 16:22:11.346565 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346559 7f868bf5a700 10 osd.26 243 dequeue_op 0x16bd6e00 prio 63 cost 450560 latency 0.956373 osd_op(client.4568.0:35127 rbd_data.11786b8b4567.0000000000001709 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.5f5b89f4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.346572 7f868cf5c700 10 merge_log log((214'567,214'571], crt=0'0) from osd.9 into log((0'0,214'567], crt=0'0) 2014-06-09 16:22:11.346576 7f868cf5c700 10 merge_log extending head to 214'571 2014-06-09 16:22:11.346577 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3061.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.346581 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346579 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346585 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd70c80,0x30509d20] 2014-06-09 16:22:11.346587 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0240 (6 active) 2014-06-09 16:22:11.346497 7f868df5e700 10 osd.26 243 dequeue_op 0x2e075e00 prio 63 cost 2064384 latency 0.404771 osd_op(client.4577.0:37965 rbd_data.11816b8b4567.000000000000198d [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~2064384] 3.82d661bd RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.346594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd70c80,0x30509d20] 2014-06-09 16:22:11.346596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346597 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346604 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (7 active) 2014-06-09 16:22:11.346594 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346609 7f86a5610700 10 _do_op 0x30032120 seq 3061 r = 0, finisher 0xc411ea0 0 2014-06-09 16:22:11.346598 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 210 same_interval_since 242 2014-06-09 16:22:11.346612 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (8 active) 2014-06-09 16:22:11.346613 7f86a5610700 10 _finish_op 0x30032120 seq 3061 osr(2.1ae 0x2ed08f0)/0x2ed08f0 2014-06-09 16:22:11.346609 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x14052000,0x14053300,0x14053200,0x14053100,0x14053000 2014-06-09 16:22:11.346618 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6120 (1 active) 2014-06-09 16:22:11.346613 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346621 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89fb00 (2 active) 2014-06-09 16:22:11.346621 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (9 active) 2014-06-09 16:22:11.346623 7f86a5610700 5 _do_op 0x300320c0 seq 3062 osr(2.1b3 0x2ed0890)/0x2ed0890 start 2014-06-09 16:22:11.346625 7f86a5610700 10 _do_transaction on 0xc14d5c0 2014-06-09 16:22:11.346628 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc411ea0] 2014-06-09 16:22:11.346631 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.346626 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346636 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc411ea0] 2014-06-09 16:22:11.346638 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346639 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346642 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.346643 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (10 active) 2014-06-09 16:22:11.346639 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 212 same_interval_since 242 2014-06-09 16:22:11.346647 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346646 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346661 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.346662 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.346654 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346661 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.346664 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346666 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346666 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346676 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3062.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.346680 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346675 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x19fe1f00,0x1b0d6700,0x1b0d6600,0x1b0d6500,0x1b0d6400,0x1b0d6300 2014-06-09 16:22:11.346677 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x306b700,0x306b200,0x306b100,0xe032f00,0xe032e00 2014-06-09 16:22:11.346680 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.346694 7f86a5610700 10 _do_op 0x300320c0 seq 3062 r = 0, finisher 0xbef14e0 0 2014-06-09 16:22:11.346691 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346696 7f86a5610700 10 _finish_op 0x300320c0 seq 3062 osr(2.1b3 0x2ed0890)/0x2ed0890 2014-06-09 16:22:11.346701 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89fb00 (1 active) 2014-06-09 16:22:11.346703 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407b00 (2 active) 2014-06-09 16:22:11.346705 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.346706 7f86a5610700 5 _do_op 0x230b94a0 seq 3063 osr(3.21b 0x3ed0ce0)/0x3ed0ce0 start 2014-06-09 16:22:11.346701 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346708 7f86a5610700 10 _do_transaction on 0x2facd80 2014-06-09 16:22:11.346710 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (9 active) 2014-06-09 16:22:11.346714 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.346715 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (10 active) 2014-06-09 16:22:11.346722 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef14e0] 2014-06-09 16:22:11.346721 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0250 (11 active) 2014-06-09 16:22:11.346728 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef14e0] 2014-06-09 16:22:11.346729 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.346730 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.346731 7f868e75f700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (12 active) 2014-06-09 16:22:11.346738 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (13 active) 2014-06-09 16:22:11.346745 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.346745 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] flushed 2014-06-09 16:22:11.346760 7f86a5610700 15 _omap_rmkeyrange meta/1fe27a8b/pglog_3.21b/0//-1 [0000000214.00000000000000000610,4294967295.18446744073709551615] 2014-06-09 16:22:11.346744 7f868d75d700 10 osd.26 243 dequeue_op 0x14052000 prio 63 cost 1548288 latency 1.177780 osd_op(client.4574.0:31681 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1548288] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.346763 7f86a5610700 15 get_omap_iterator meta/1fe27a8b/pglog_3.21b/0//-1 2014-06-09 16:22:11.346759 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] flushed 2014-06-09 16:22:11.346769 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0230 (14 active) 2014-06-09 16:22:11.346774 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.346774 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346784 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346788 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] requeue_ops 0x21967d00,0x21967c00,0x21967b00,0x21967a00,0x270bf800,0x2c06bc00,0x2c06bb00 2014-06-09 16:22:11.346794 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346805 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5a1a0,0xbef0f00,0xbce7d00,0xbd6bb00,0x30629b80,0x4eb89c0,0xc169680,0xcd0f820,0x1322d480,0xc411da0] 2014-06-09 16:22:11.346809 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd747c0,0xbd7bd00,0x230c00e0,0xbef1080] 2014-06-09 16:22:11.346819 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.346812 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.346825 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0230 (13 active) 2014-06-09 16:22:11.346822 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346830 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.346838 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346837 7f8691765700 10 osd.26 243 dequeue_op 0x306b700 prio 63 cost 450560 latency 1.427524 osd_op(client.4565.0:32565 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.346849 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.346859 7f86a5610700 15 _omap_rmkeys meta/1fe27a8b/pglog_3.21b/0//-1 2014-06-09 16:22:11.346860 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346871 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346882 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.346892 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346898 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe27a8b/pglog_3.21b/0//-1 2014-06-09 16:22:11.346902 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346907 7f86a5610700 10 _omap_rmkeys meta/1fe27a8b/pglog_3.21b/0//-1 r = 0 2014-06-09 16:22:11.346912 7f86a5610700 15 _omap_rmkeys meta/1fe27a8b/pglog_3.21b/0//-1 2014-06-09 16:22:11.346917 7f86a5610700 10 _omap_rmkeys meta/1fe27a8b/pglog_3.21b/0//-1 r = 0 2014-06-09 16:22:11.346913 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.346924 7f86a5610700 15 _omap_setkeys meta/1fe27a8b/pglog_3.21b/0//-1 2014-06-09 16:22:11.346923 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346933 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.346940 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3063.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.346943 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd747c0,0xbd7bd00,0x230c00e0,0xbef1080] 2014-06-09 16:22:11.346945 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.346945 7f86a5610700 10 cid: oid: 1fe27a8b/pglog_3.21b/0//-1 skipping op, spos 3063.0.0 <= header.spos 150432.0.0 2014-06-09 16:22:11.346946 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.346948 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.346738 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346962 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.346966 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.346983 7f86a5610700 10 _do_op 0x230b94a0 seq 3063 r = 0, finisher 0xbd71220 0 2014-06-09 16:22:11.346990 7f86a5610700 10 _finish_op 0x230b94a0 seq 3063 osr(3.21b 0x3ed0ce0)/0x3ed0ce0 2014-06-09 16:22:11.346991 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd70880] 2014-06-09 16:22:11.346993 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd70880] 2014-06-09 16:22:11.346995 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.346994 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407b00 (1 active) 2014-06-09 16:22:11.346996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.346996 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3be60 (2 active) 2014-06-09 16:22:11.346996 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd71220] 2014-06-09 16:22:11.346977 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x270bf500,0x22009b00,0x22009e00,0x1b0b7b00,0x257c4400,0x1804c700,0xe2ac000,0xe032a00,0xe032900,0xe2ad000,0x270be300,0x270be200,0x270be100,0x270be000,0x29f5bf00,0x29f5be00,0x4679f00 2014-06-09 16:22:11.346998 7f86a5610700 5 _do_op 0x2c06ecc0 seq 3064 osr(2.75d 0x3ed00c0)/0x3ed00c0 start 2014-06-09 16:22:11.347001 7f86a5610700 10 _do_transaction on 0xc3ec240 2014-06-09 16:22:11.347007 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.347000 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 210 same_interval_since 242 2014-06-09 16:22:11.347017 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.347040 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.347045 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.347042 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.347055 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3064.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.347050 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.347059 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.347063 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.347063 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd71220] 2014-06-09 16:22:11.347065 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.347066 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.347069 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (13 active) 2014-06-09 16:22:11.346581 7f868bf5a700 5 -- op tracker -- , seq: 300, time: 2014-06-09 16:22:11.346576, event: reached_pg, op: osd_op(client.4568.0:35127 rbd_data.11786b8b4567.0000000000001709 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.5f5b89f4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347074 7f86a5610700 10 _do_op 0x2c06ecc0 seq 3064 r = 0, finisher 0x306cf660 0 2014-06-09 16:22:11.347076 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0250 (14 active) 2014-06-09 16:22:11.347080 7f86a5610700 10 _finish_op 0x2c06ecc0 seq 3064 osr(2.75d 0x3ed00c0)/0x3ed00c0 2014-06-09 16:22:11.347081 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd67780] 2014-06-09 16:22:11.347084 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3be60 (1 active) 2014-06-09 16:22:11.347085 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd67780] 2014-06-09 16:22:11.347087 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.347086 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780c60 (2 active) 2014-06-09 16:22:11.347088 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.347078 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347088 7f86a5610700 5 _do_op 0x309a38c0 seq 3065 osr(1.43b 0x384d6f0)/0x384d6f0 start 2014-06-09 16:22:11.347091 7f86a5610700 10 _do_transaction on 0x2eaa780 2014-06-09 16:22:11.347096 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.347090 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16bd6e00 2014-06-09 16:22:11.347099 7f868bf5a700 10 osd.26 243 dequeue_op 0x16bd6e00 finish 2014-06-09 16:22:11.347097 7f868df5e700 5 -- op tracker -- , seq: 1058, time: 2014-06-09 16:22:11.346715, event: reached_pg, op: osd_op(client.4577.0:37965 rbd_data.11816b8b4567.000000000000198d [set-alloc-hint object_size 4194304 write_size 4194304,write 221184~2064384] 3.82d661bd RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347104 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347109 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347108 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347113 7f868bf5a700 10 osd.26 243 dequeue_op 0x21967d00 prio 63 cost 442368 latency 0.787194 osd_op(client.4567.0:35150 rbd_data.11776b8b4567.00000000000017b4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~442368] 3.584859c7 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.347128 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.347121 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x2e075e00 2014-06-09 16:22:11.347131 7f868df5e700 10 osd.26 243 dequeue_op 0x2e075e00 finish 2014-06-09 16:22:11.347133 7f868bf5a700 5 -- op tracker -- , seq: 522, time: 2014-06-09 16:22:11.347128, event: reached_pg, op: osd_op(client.4567.0:35150 rbd_data.11776b8b4567.00000000000017b4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~442368] 3.584859c7 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347136 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347143 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347143 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3065.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.347146 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.347140 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347151 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x21967d00 2014-06-09 16:22:11.347146 7f868df5e700 10 osd.26 243 dequeue_op 0x21967900 prio 63 cost 630784 latency 0.639567 osd_op(client.4567.0:35175 rbd_data.11776b8b4567.00000000000017b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~630784] 3.6498b9f4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.347160 7f868bf5a700 10 osd.26 243 dequeue_op 0x21967d00 finish 2014-06-09 16:22:11.347163 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347166 7f86a5610700 10 _do_op 0x309a38c0 seq 3065 r = 0, finisher 0x3061fbc0 0 2014-06-09 16:22:11.347168 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347165 7f868df5e700 5 -- op tracker -- , seq: 617, time: 2014-06-09 16:22:11.347161, event: reached_pg, op: osd_op(client.4567.0:35175 rbd_data.11776b8b4567.00000000000017b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~630784] 3.6498b9f4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347172 7f86a5610700 10 _finish_op 0x309a38c0 seq 3065 osr(1.43b 0x384d6f0)/0x384d6f0 2014-06-09 16:22:11.347175 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780c60 (1 active) 2014-06-09 16:22:11.347177 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780b40 (2 active) 2014-06-09 16:22:11.347178 7f86a5610700 5 _do_op 0x309a3860 seq 3066 osr(0.708 0x384c130)/0x384c130 start 2014-06-09 16:22:11.347181 7f86a5610700 10 _do_transaction on 0x2dcf140 2014-06-09 16:22:11.347173 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347172 7f868bf5a700 10 osd.26 243 dequeue_op 0x270bf800 prio 63 cost 479232 latency 0.519277 osd_op(client.4570.0:34425 rbd_data.117a6b8b4567.0000000000001713 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.33969c7 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.347186 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.347184 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x21967900 2014-06-09 16:22:11.347193 7f868df5e700 10 osd.26 243 dequeue_op 0x21967900 finish 2014-06-09 16:22:11.347196 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347192 7f868bf5a700 5 -- op tracker -- , seq: 819, time: 2014-06-09 16:22:11.347187, event: reached_pg, op: osd_op(client.4570.0:34425 rbd_data.117a6b8b4567.0000000000001713 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~479232] 3.33969c7 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347202 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347200 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347216 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.347211 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x270bf800 2014-06-09 16:22:11.347206 7f868df5e700 10 osd.26 243 dequeue_op 0x270be300 prio 63 cost 450560 latency 0.532959 osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.347220 7f868bf5a700 10 osd.26 243 dequeue_op 0x270bf800 finish 2014-06-09 16:22:11.347223 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347228 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347225 7f868df5e700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:11.347221, event: reached_pg, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347232 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3066.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.347236 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.347233 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347235 7f8689f56700 10 osd.26 243 dequeue_op 0x2e075d00 prio 63 cost 516096 latency 0.402721 osd_op(client.4577.0:37966 rbd_data.11816b8b4567.000000000000198d [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.82d661bd RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.347244 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270be300 2014-06-09 16:22:11.347252 7f868df5e700 10 osd.26 243 dequeue_op 0x270be300 finish 2014-06-09 16:22:11.347253 7f86a5610700 10 _do_op 0x309a3860 seq 3066 r = 0, finisher 0xc169ac0 0 2014-06-09 16:22:11.347255 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347256 7f86a5610700 10 _finish_op 0x309a3860 seq 3066 osr(0.708 0x384c130)/0x384c130 2014-06-09 16:22:11.347258 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780b40 (1 active) 2014-06-09 16:22:11.347260 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7819e0 (2 active) 2014-06-09 16:22:11.347260 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347256 7f8689f56700 5 -- op tracker -- , seq: 1069, time: 2014-06-09 16:22:11.347251, event: reached_pg, op: osd_op(client.4577.0:37966 rbd_data.11816b8b4567.000000000000198d [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.82d661bd RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347262 7f86a5610700 5 _do_op 0x2c06ec60 seq 3067 osr(1.1eb 0x384ca00)/0x384ca00 start 2014-06-09 16:22:11.347264 7f86a5610700 10 _do_transaction on 0x32fb5c0 2014-06-09 16:22:11.347269 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.347264 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347264 7f868df5e700 10 osd.26 243 dequeue_op 0x270bf500 prio 63 cost 614400 latency 0.571140 osd_op(client.4577.0:37199 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~614400] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.347275 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x2e075d00 2014-06-09 16:22:11.347284 7f8689f56700 10 osd.26 243 dequeue_op 0x2e075d00 finish 2014-06-09 16:22:11.347282 7f868df5e700 5 -- op tracker -- , seq: 709, time: 2014-06-09 16:22:11.347277, event: reached_pg, op: osd_op(client.4577.0:37199 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~614400] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347290 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347289 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347300 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.347301 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bf500 2014-06-09 16:22:11.347309 7f868df5e700 10 osd.26 243 dequeue_op 0x270bf500 finish 2014-06-09 16:22:11.347295 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347316 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347314 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3067.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.347318 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.347321 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347327 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061e520,0xbd5bf60] 2014-06-09 16:22:11.347316 7f8689f56700 10 osd.26 243 dequeue_op 0x1b0b7b00 prio 63 cost 0 latency 0.788194 osd_op(client.4564.0:36207 rbd_header.11746b8b4567 [watch add cookie 1 ver 0] 3.552750 ondisk+read+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.347331 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061e520,0xbd5bf60] 2014-06-09 16:22:11.347332 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.347333 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.347342 7f86a5610700 10 _do_op 0x2c06ec60 seq 3067 r = 0, finisher 0x30508360 0 2014-06-09 16:22:11.347078 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.347337 7f8689f56700 5 -- op tracker -- , seq: 507, time: 2014-06-09 16:22:11.347332, event: reached_pg, op: osd_op(client.4564.0:36207 rbd_header.11746b8b4567 [watch add cookie 1 ver 0] 3.552750 ondisk+read+write e242) 2014-06-09 16:22:11.347348 7f86a5610700 10 _finish_op 0x2c06ec60 seq 3067 osr(1.1eb 0x384ca00)/0x384ca00 2014-06-09 16:22:11.347349 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061fa60] 2014-06-09 16:22:11.347351 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7819e0 (1 active) 2014-06-09 16:22:11.347351 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061fa60] 2014-06-09 16:22:11.347353 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.347353 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781560 (2 active) 2014-06-09 16:22:11.347354 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.347355 7f86a5610700 5 _do_op 0x309a3f80 seq 3068 osr(2.4b5 0x2ed1770)/0x2ed1770 start 2014-06-09 16:22:11.347357 7f86a5610700 10 _do_transaction on 0xbf37140 2014-06-09 16:22:11.347363 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.347353 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347357 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.347367 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.347365 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0b7b00 2014-06-09 16:22:11.347374 7f8689f56700 10 osd.26 243 dequeue_op 0x1b0b7b00 finish 2014-06-09 16:22:11.347378 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347371 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.347383 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347393 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.347384 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] requeue_ops 0x1c66be00,0x1c66bd00,0x1c66bc00,0x1c66bb00,0x21966400,0x21966300,0x1804c300,0x1804c200,0x1804c100,0x1804c000,0x19fe1d00,0x19fe1c00 2014-06-09 16:22:11.347387 7f8689f56700 10 osd.26 243 dequeue_op 0xe2ac000 prio 63 cost 1032192 latency 1.415716 osd_op(client.4566.0:33765 rbd_data.11766b8b4567.00000000000016aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~1032192] 3.4c8eb750 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.347407 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3068.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.347405 7f8689f56700 5 -- op tracker -- , seq: 8, time: 2014-06-09 16:22:11.347401, event: reached_pg, op: osd_op(client.4566.0:33765 rbd_data.11766b8b4567.00000000000016aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1392640~1032192] 3.4c8eb750 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347411 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.347412 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347423 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.347427 7f86a5610700 10 _do_op 0x309a3f80 seq 3068 r = 0, finisher 0x19940600 0 2014-06-09 16:22:11.347423 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xe2ac000 2014-06-09 16:22:11.347433 7f8689f56700 10 osd.26 243 dequeue_op 0xe2ac000 finish 2014-06-09 16:22:11.347433 7f86a5610700 10 _finish_op 0x309a3f80 seq 3068 osr(2.4b5 0x2ed1770)/0x2ed1770 2014-06-09 16:22:11.347434 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc410fa0] 2014-06-09 16:22:11.347436 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347436 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781560 (1 active) 2014-06-09 16:22:11.347438 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1b00 (2 active) 2014-06-09 16:22:11.347439 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc410fa0] 2014-06-09 16:22:11.347440 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.347440 7f86a5610700 5 _do_op 0x30061aa0 seq 3069 osr(0.759 0x384c370)/0x384c370 start 2014-06-09 16:22:11.347441 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.347441 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347442 7f86a5610700 10 _do_transaction on 0xc20ba40 2014-06-09 16:22:11.347427 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.347447 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.347452 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.347456 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.347444 7f8689f56700 10 osd.26 243 dequeue_op 0x21967c00 prio 63 cost 1032192 latency 0.775709 osd_op(client.4567.0:35151 rbd_data.11776b8b4567.00000000000017b4 [set-alloc-hint object_size 4194304 write_size 4194304,write 442368~1032192] 3.584859c7 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.347466 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.347475 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.347471 7f8689f56700 5 -- op tracker -- , seq: 551, time: 2014-06-09 16:22:11.347466, event: reached_pg, op: osd_op(client.4567.0:35151 rbd_data.11776b8b4567.00000000000017b4 [set-alloc-hint object_size 4194304 write_size 4194304,write 442368~1032192] 3.584859c7 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.347478 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.347479 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.347490 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.347492 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3069.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.347478 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347496 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.347494 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.347497 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x21967c00 2014-06-09 16:22:11.347507 7f8689f56700 10 osd.26 243 dequeue_op 0x21967c00 finish 2014-06-09 16:22:11.347503 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.347510 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.347511 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.347515 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.347516 7f86a5610700 10 _do_op 0x30061aa0 seq 3069 r = 0, finisher 0x13964aa0 0 2014-06-09 16:22:11.347521 7f86a5610700 10 _finish_op 0x30061aa0 seq 3069 osr(0.759 0x384c370)/0x384c370 2014-06-09 16:22:11.347523 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x306cf760] 2014-06-09 16:22:11.347525 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x306cf760] 2014-06-09 16:22:11.347525 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1b00 (1 active) 2014-06-09 16:22:11.347527 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.347527 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d18c0 (2 active) 2014-06-09 16:22:11.347528 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.347529 7f86a5610700 5 _do_op 0x308cbda0 seq 3070 osr(1.1c9 0x384ca90)/0x384ca90 start 2014-06-09 16:22:11.347531 7f86a5610700 10 _do_transaction on 0xc141200 2014-06-09 16:22:11.347519 7f8689f56700 10 osd.26 243 dequeue_op 0x2c06bc00 prio 63 cost 3612672 latency 0.518632 osd_op(client.4570.0:34426 rbd_data.117a6b8b4567.0000000000001713 [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~3612672] 3.33969c7 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.351743 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.351793 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.351810 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3070.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.351814 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.351828 7f86a5610700 10 _do_op 0x308cbda0 seq 3070 r = 0, finisher 0x2d7398c0 0 2014-06-09 16:22:11.351836 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7a780] 2014-06-09 16:22:11.351837 7f86a5610700 10 _finish_op 0x308cbda0 seq 3070 osr(1.1c9 0x384ca90)/0x384ca90 2014-06-09 16:22:11.351841 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7a780] 2014-06-09 16:22:11.351840 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d18c0 (1 active) 2014-06-09 16:22:11.351842 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.351842 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd9e0 (2 active) 2014-06-09 16:22:11.351843 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.351844 7f86a5610700 5 _do_op 0x308cbd40 seq 3071 osr(1.75b 0x2ed0010)/0x2ed0010 start 2014-06-09 16:22:11.347551 7f86a5e11700 10 _do_op 0x308cbe00 seq 3049 r = 0, finisher 0x3050bee0 0 2014-06-09 16:22:11.351847 7f86a5610700 10 _do_transaction on 0xbebcc00 2014-06-09 16:22:11.347544 7f8692f68700 10 osd.26 243 dequeue_op 0x2e075c00 prio 63 cost 1392640 latency 0.402110 osd_op(client.4577.0:37967 rbd_data.11816b8b4567.000000000000198d [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~1392640] 3.82d661bd RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.351854 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.347570 7f8689755700 10 osd.26 243 dequeue_op 0x16bd6d00 prio 63 cost 2580480 latency 0.947002 osd_op(client.4568.0:35128 rbd_data.11786b8b4567.0000000000001709 [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~2580480] 3.5f5b89f4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.351857 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050bd80] 2014-06-09 16:22:11.347596 7f8691765700 5 -- op tracker -- , seq: 2, time: 2014-06-09 16:22:11.346853, event: reached_pg, op: osd_op(client.4565.0:32565 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.351861 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050bd80] 2014-06-09 16:22:11.351863 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.351864 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.351867 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.351874 7f868d75d700 5 -- op tracker -- , seq: 158, time: 2014-06-09 16:22:11.346765, event: reached_pg, op: osd_op(client.4574.0:31681 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1548288] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.351888 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.351882 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x306b700 2014-06-09 16:22:11.351892 7f8691765700 10 osd.26 243 dequeue_op 0x306b700 finish 2014-06-09 16:22:11.351891 7f8689f56700 5 -- op tracker -- , seq: 876, time: 2014-06-09 16:22:11.351740, event: reached_pg, op: osd_op(client.4570.0:34426 rbd_data.117a6b8b4567.0000000000001713 [set-alloc-hint object_size 4194304 write_size 4194304,write 479232~3612672] 3.33969c7 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.346578 7f868cf5c700 20 ? 214'571 (214'570) modify 87496894/rbd_data.117f6b8b4567.000000000000151c/head//3 by client.4575.0:32103 2014-06-09 10:18:55.620813 2014-06-09 16:22:11.351903 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3071.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.351907 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.351901 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.351909 7f8692f68700 5 -- op tracker -- , seq: 1072, time: 2014-06-09 16:22:11.351856, event: reached_pg, op: osd_op(client.4577.0:37967 rbd_data.11816b8b4567.000000000000198d [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~1392640] 3.82d661bd RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.351915 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x2c06bc00 2014-06-09 16:22:11.351924 7f8689f56700 10 osd.26 243 dequeue_op 0x2c06bc00 finish 2014-06-09 16:22:11.351918 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.351930 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x2e075c00 2014-06-09 16:22:11.351938 7f8692f68700 10 osd.26 243 dequeue_op 0x2e075c00 finish 2014-06-09 16:22:11.351940 7f86a5610700 10 _do_op 0x308cbd40 seq 3071 r = 0, finisher 0x3050ab20 0 2014-06-09 16:22:11.351931 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.347913 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306cf660,0x3061fbc0,0xc169ac0,0x30508360,0x19940600,0x13964aa0] 2014-06-09 16:22:11.351946 7f86a5610700 10 _finish_op 0x308cbd40 seq 3071 osr(1.75b 0x2ed0010)/0x2ed0010 2014-06-09 16:22:11.351949 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd9e0 (1 active) 2014-06-09 16:22:11.351943 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14052000 2014-06-09 16:22:11.351952 7f868d75d700 10 osd.26 243 dequeue_op 0x14052000 finish 2014-06-09 16:22:11.351955 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383e6c0 (2 active) 2014-06-09 16:22:11.351958 7f86a5610700 5 _do_op 0x2fc8c120 seq 3072 osr(1.4db 0x384d850)/0x384d850 start 2014-06-09 16:22:11.351948 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 201 same_interval_since 242 2014-06-09 16:22:11.351960 7f86a5610700 10 _do_transaction on 0xc05a780 2014-06-09 16:22:11.351966 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.351961 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.351960 7f868bf5a700 10 osd.26 243 dequeue_op 0x14053300 prio 63 cost 1032192 latency 1.172308 osd_op(client.4574.0:31682 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1548288~1032192] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.351978 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e8700] 2014-06-09 16:22:11.351979 7f8689755700 5 -- op tracker -- , seq: 364, time: 2014-06-09 16:22:11.351860, event: reached_pg, op: osd_op(client.4568.0:35128 rbd_data.11786b8b4567.0000000000001709 [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~2580480] 3.5f5b89f4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.351984 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e8700] 2014-06-09 16:22:11.351986 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.351987 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.351989 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.351999 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.351995 7f868bf5a700 5 -- op tracker -- , seq: 176, time: 2014-06-09 16:22:11.351975, event: reached_pg, op: osd_op(client.4574.0:31682 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1548288~1032192] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352001 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16bd6d00 2014-06-09 16:22:11.352010 7f8689755700 10 osd.26 243 dequeue_op 0x16bd6d00 finish 2014-06-09 16:22:11.352003 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352016 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3072.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352019 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352016 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053300 2014-06-09 16:22:11.352025 7f868bf5a700 10 osd.26 243 dequeue_op 0x14053300 finish 2014-06-09 16:22:11.352018 7f868f761700 10 osd.26 243 dequeue_op 0x21967800 prio 63 cost 516096 latency 0.633653 osd_op(client.4567.0:35176 rbd_data.11776b8b4567.00000000000017b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 630784~516096] 3.6498b9f4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352038 7f86a5610700 10 _do_op 0x2fc8c120 seq 3072 r = 0, finisher 0xc6e9340 0 2014-06-09 16:22:11.352038 7f868f761700 5 -- op tracker -- , seq: 627, time: 2014-06-09 16:22:11.352033, event: reached_pg, op: osd_op(client.4567.0:35176 rbd_data.11776b8b4567.00000000000017b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 630784~516096] 3.6498b9f4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352044 7f86a5610700 10 _finish_op 0x2fc8c120 seq 3072 osr(1.4db 0x384d850)/0x384d850 2014-06-09 16:22:11.352045 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7b7c0] 2014-06-09 16:22:11.352048 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7b7c0] 2014-06-09 16:22:11.352047 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383e6c0 (1 active) 2014-06-09 16:22:11.352049 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.352049 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383fb00 (2 active) 2014-06-09 16:22:11.352050 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.352051 7f86a5610700 5 _do_op 0x30607bc0 seq 3073 osr(3.17 0x3ed03b0)/0x3ed03b0 start 2014-06-09 16:22:11.352054 7f86a5610700 10 _do_transaction on 0xc014b40 2014-06-09 16:22:11.352045 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352034 7f868df5e700 10 osd.26 243 dequeue_op 0x306b200 prio 63 cost 516096 latency 1.405652 osd_op(client.4565.0:32566 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~516096] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352059 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352057 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x21967800 2014-06-09 16:22:11.352066 7f868f761700 10 osd.26 243 dequeue_op 0x21967800 finish 2014-06-09 16:22:11.352066 7f868df5e700 5 -- op tracker -- , seq: 14, time: 2014-06-09 16:22:11.352060, event: reached_pg, op: osd_op(client.4565.0:32566 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~516096] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352073 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352074 7f868c75b700 10 osd.26 243 dequeue_op 0x1c54ae00 prio 63 cost 516096 latency 0.908756 osd_op(client.4568.0:35129 rbd_data.11786b8b4567.0000000000001709 [set-alloc-hint object_size 4194304 write_size 4194304,write 3031040~516096] 3.5f5b89f4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352092 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352084 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x306b200 2014-06-09 16:22:11.352095 7f868df5e700 10 osd.26 243 dequeue_op 0x306b200 finish 2014-06-09 16:22:11.352096 7f868c75b700 5 -- op tracker -- , seq: 368, time: 2014-06-09 16:22:11.352090, event: reached_pg, op: osd_op(client.4568.0:35129 rbd_data.11786b8b4567.0000000000001709 [set-alloc-hint object_size 4194304 write_size 4194304,write 3031040~516096] 3.5f5b89f4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352100 7f86a5e11700 10 _finish_op 0x308cbe00 seq 3049 osr(2.74b 0x3ed0110)/0x3ed0110 2014-06-09 16:22:11.352103 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4ea0 (1 active) 2014-06-09 16:22:11.352105 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fca20 (2 active) 2014-06-09 16:22:11.352108 7f86a5e11700 5 _do_op 0x308ca060 seq 3074 osr(3.308 0x3ed0da0)/0x3ed0da0 start 2014-06-09 16:22:11.352108 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3073.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352110 7f86a5e11700 10 _do_transaction on 0xc20b680 2014-06-09 16:22:11.352111 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352103 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352116 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352114 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1c54ae00 2014-06-09 16:22:11.352122 7f868c75b700 10 osd.26 243 dequeue_op 0x1c54ae00 finish 2014-06-09 16:22:11.347539 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352131 7f868af58700 10 osd.26 243 dequeue_op 0x1c54ad00 prio 63 cost 647168 latency 0.907310 osd_op(client.4568.0:35130 rbd_data.11786b8b4567.0000000000001709 [set-alloc-hint object_size 4194304 write_size 4194304,write 3547136~647168] 3.5f5b89f4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352150 7f868af58700 5 -- op tracker -- , seq: 394, time: 2014-06-09 16:22:11.352145, event: reached_pg, op: osd_op(client.4568.0:35130 rbd_data.11786b8b4567.0000000000001709 [set-alloc-hint object_size 4194304 write_size 4194304,write 3547136~647168] 3.5f5b89f4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352140 7f8691f66700 10 osd.26 243 dequeue_op 0x270be200 prio 63 cost 516096 latency 0.533515 osd_op(client.4573.0:34235 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~516096] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352159 7f86a5610700 10 _do_op 0x30607bc0 seq 3073 r = 0, finisher 0x189aa0e0 0 2014-06-09 16:22:11.352164 7f86a5610700 10 _finish_op 0x30607bc0 seq 3073 osr(3.17 0x3ed03b0)/0x3ed03b0 2014-06-09 16:22:11.352156 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352166 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383fb00 (1 active) 2014-06-09 16:22:11.352164 7f8691f66700 5 -- op tracker -- , seq: 780, time: 2014-06-09 16:22:11.352158, event: reached_pg, op: osd_op(client.4573.0:34235 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~516096] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352168 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f200 (2 active) 2014-06-09 16:22:11.352170 7f86a5610700 5 _do_op 0x2fc8cba0 seq 3075 osr(2.4f4 0x2ed16b0)/0x2ed16b0 start 2014-06-09 16:22:11.352173 7f86a5610700 10 _do_transaction on 0xbe89080 2014-06-09 16:22:11.352167 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1c54ad00 2014-06-09 16:22:11.352176 7f868af58700 10 osd.26 243 dequeue_op 0x1c54ad00 finish 2014-06-09 16:22:11.352178 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352179 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352171 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352186 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352183 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270be200 2014-06-09 16:22:11.352192 7f8691f66700 10 osd.26 243 dequeue_op 0x270be200 finish 2014-06-09 16:22:11.352195 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352201 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352190 7f868af58700 10 osd.26 243 dequeue_op 0x14053200 prio 63 cost 516096 latency 1.115294 osd_op(client.4574.0:31683 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2580480~516096] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352209 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352209 7f868af58700 5 -- op tracker -- , seq: 180, time: 2014-06-09 16:22:11.352204, event: reached_pg, op: osd_op(client.4574.0:31683 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2580480~516096] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352205 7f8691f66700 10 osd.26 243 dequeue_op 0x1804c300 prio 63 cost 958464 latency 1.045695 osd_op(client.4575.0:32138 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~958464] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.352224 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3075.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352216 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352228 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352226 7f8691f66700 5 -- op tracker -- , seq: 257, time: 2014-06-09 16:22:11.352222, event: reached_pg, op: osd_op(client.4575.0:32138 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~958464] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352229 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053200 2014-06-09 16:22:11.352237 7f868af58700 10 osd.26 243 dequeue_op 0x14053200 finish 2014-06-09 16:22:11.352240 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352234 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352245 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352248 7f86a5610700 10 _do_op 0x2fc8cba0 seq 3075 r = 0, finisher 0xbd70da0 0 2014-06-09 16:22:11.352251 7f86a5610700 10 _finish_op 0x2fc8cba0 seq 3075 osr(2.4f4 0x2ed16b0)/0x2ed16b0 2014-06-09 16:22:11.352246 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x1804c300 2014-06-09 16:22:11.352253 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f200 (1 active) 2014-06-09 16:22:11.352255 7f8691f66700 10 osd.26 243 dequeue_op 0x1804c300 finish 2014-06-09 16:22:11.352255 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc7e0 (2 active) 2014-06-09 16:22:11.352257 7f86a5610700 5 _do_op 0x2c06e840 seq 3076 osr(3.7d4 0xb7f0170)/0xb7f0170 start 2014-06-09 16:22:11.352259 7f86a5610700 10 _do_transaction on 0xc161ec0 2014-06-09 16:22:11.352265 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352273 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352269 7f869af78700 5 -- op tracker -- , seq: 1514, time: 2014-06-09 16:22:11.352267, event: done, op: pg_notify(3.1bd(10),3.750(10) epoch 242) 2014-06-09 16:22:11.352277 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (15 active) 2014-06-09 16:22:11.352281 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352283 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352280 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:242, info:3.1bd( v 214'661 (0'0,214'661] local-les=212 n=131 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.352296 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352294 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.352307 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352305 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352308 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352310 7f869af78700 20 osd.26 243 _dispatch 0x2d095dc0 pg_notify(2.6dd(20),3.224(10) epoch 242) v5 2014-06-09 16:22:11.352311 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3076.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352315 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352316 7f869af78700 5 -- op tracker -- , seq: 1515, time: 2014-06-09 16:22:10.859947, event: header_read, op: pg_notify(2.6dd(20),3.224(10) epoch 242) 2014-06-09 16:22:11.352313 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:242, info:3.750( v 214'518 (0'0,214'518] local-les=212 n=98 ec=118 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.352324 7f869af78700 5 -- op tracker -- , seq: 1515, time: 2014-06-09 16:22:10.859948, event: throttled, op: pg_notify(2.6dd(20),3.224(10) epoch 242) 2014-06-09 16:22:11.352323 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.352331 7f869af78700 5 -- op tracker -- , seq: 1515, time: 2014-06-09 16:22:10.859986, event: all_read, op: pg_notify(2.6dd(20),3.224(10) epoch 242) 2014-06-09 16:22:11.352338 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352338 7f869af78700 5 -- op tracker -- , seq: 1515, time: 2014-06-09 16:22:11.352306, event: dispatched, op: pg_notify(2.6dd(20),3.224(10) epoch 242) 2014-06-09 16:22:11.352344 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (14 active) 2014-06-09 16:22:11.352346 7f869af78700 5 -- op tracker -- , seq: 1515, time: 2014-06-09 16:22:11.352344, event: waiting_for_osdmap, op: pg_notify(2.6dd(20),3.224(10) epoch 242) 2014-06-09 16:22:11.352350 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352350 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.352353 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d095dc0 2014-06-09 16:22:11.352359 7f869af78700 5 -- op tracker -- , seq: 1515, time: 2014-06-09 16:22:11.352357, event: started, op: pg_notify(2.6dd(20),3.224(10) epoch 242) 2014-06-09 16:22:11.352353 7f8691f66700 10 osd.26 243 dequeue_op 0x306b100 prio 63 cost 1548288 latency 1.391718 osd_op(client.4565.0:32567 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 966656~1548288] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352372 7f8691f66700 5 -- op tracker -- , seq: 24, time: 2014-06-09 16:22:11.352368, event: reached_pg, op: osd_op(client.4565.0:32567 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 966656~1548288] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352384 7f869af78700 5 -- op tracker -- , seq: 1515, time: 2014-06-09 16:22:11.352381, event: done, op: pg_notify(2.6dd(20),3.224(10) epoch 242) 2014-06-09 16:22:11.352379 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352389 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x306b100 2014-06-09 16:22:11.352398 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352398 7f8691f66700 10 osd.26 243 dequeue_op 0x306b100 finish 2014-06-09 16:22:11.352399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352402 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352405 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (15 active) 2014-06-09 16:22:11.352408 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:2.6dd( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.352417 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352418 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352420 7f869af78700 20 osd.26 243 _dispatch 0x1609d6c0 pg_notify(0.3e(21) epoch 243) v5 2014-06-09 16:22:11.352419 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.352424 7f869af78700 5 -- op tracker -- , seq: 1516, time: 2014-06-09 16:22:11.003852, event: header_read, op: pg_notify(0.3e(21) epoch 243) 2014-06-09 16:22:11.352429 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352432 7f869af78700 5 -- op tracker -- , seq: 1516, time: 2014-06-09 16:22:11.003853, event: throttled, op: pg_notify(0.3e(21) epoch 243) 2014-06-09 16:22:11.352438 7f869af78700 5 -- op tracker -- , seq: 1516, time: 2014-06-09 16:22:11.003870, event: all_read, op: pg_notify(0.3e(21) epoch 243) 2014-06-09 16:22:11.352433 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:3.224( v 214'739 (0'0,214'739] local-les=208 n=135 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.352444 7f869af78700 5 -- op tracker -- , seq: 1516, time: 2014-06-09 16:22:11.352417, event: dispatched, op: pg_notify(0.3e(21) epoch 243) 2014-06-09 16:22:11.352445 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.352450 7f869af78700 5 -- op tracker -- , seq: 1516, time: 2014-06-09 16:22:11.352449, event: waiting_for_osdmap, op: pg_notify(0.3e(21) epoch 243) 2014-06-09 16:22:11.352455 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352455 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.352457 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1609d6c0 2014-06-09 16:22:11.352460 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (14 active) 2014-06-09 16:22:11.352461 7f869af78700 5 -- op tracker -- , seq: 1516, time: 2014-06-09 16:22:11.352459, event: started, op: pg_notify(0.3e(21) epoch 243) 2014-06-09 16:22:11.352466 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352472 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189aa440,0x2d7399a0] 2014-06-09 16:22:11.352477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189aa440,0x2d7399a0] 2014-06-09 16:22:11.352478 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.352479 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.352477 7f869af78700 5 -- op tracker -- , seq: 1516, time: 2014-06-09 16:22:11.352475, event: done, op: pg_notify(0.3e(21) epoch 243) 2014-06-09 16:22:11.352470 7f8691f66700 10 osd.26 243 dequeue_op 0xe032a00 prio 63 cost 516096 latency 1.393007 osd_op(client.4566.0:33766 rbd_data.11766b8b4567.00000000000016aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.4c8eb750 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352486 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352487 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352491 7f86a5610700 10 _do_op 0x2c06e840 seq 3076 r = 0, finisher 0x305091c0 0 2014-06-09 16:22:11.352490 7f8691f66700 5 -- op tracker -- , seq: 23, time: 2014-06-09 16:22:11.352485, event: reached_pg, op: osd_op(client.4566.0:33766 rbd_data.11766b8b4567.00000000000016aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2424832~516096] 3.4c8eb750 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352496 7f86a5610700 10 _finish_op 0x2c06e840 seq 3076 osr(3.7d4 0xb7f0170)/0xb7f0170 2014-06-09 16:22:11.352499 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc7e0 (1 active) 2014-06-09 16:22:11.352501 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc5a0 (2 active) 2014-06-09 16:22:11.352503 7f86a5610700 5 _do_op 0x308cac60 seq 3077 osr(2.312 0x2ed0f30)/0x2ed0f30 start 2014-06-09 16:22:11.352504 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352505 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352506 7f86a5610700 10 _do_transaction on 0xc280780 2014-06-09 16:22:11.352498 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352506 7f869af78700 20 osd.26 243 _dispatch 0x1405c1c0 pg_info(1 pgs e243:3.4fa(2)) v4 2014-06-09 16:22:11.352511 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352512 7f869af78700 5 -- op tracker -- , seq: 1517, time: 2014-06-09 16:22:11.003645, event: header_read, op: pg_info(1 pgs e243:3.4fa(2)) 2014-06-09 16:22:11.352509 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xe032a00 2014-06-09 16:22:11.352518 7f8691f66700 10 osd.26 243 dequeue_op 0xe032a00 finish 2014-06-09 16:22:11.352519 7f869af78700 5 -- op tracker -- , seq: 1517, time: 2014-06-09 16:22:11.003645, event: throttled, op: pg_info(1 pgs e243:3.4fa(2)) 2014-06-09 16:22:11.352521 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352524 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (15 active) 2014-06-09 16:22:11.352525 7f869af78700 5 -- op tracker -- , seq: 1517, time: 2014-06-09 16:22:11.003664, event: all_read, op: pg_info(1 pgs e243:3.4fa(2)) 2014-06-09 16:22:11.352531 7f869af78700 5 -- op tracker -- , seq: 1517, time: 2014-06-09 16:22:11.352503, event: dispatched, op: pg_info(1 pgs e243:3.4fa(2)) 2014-06-09 16:22:11.352527 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:0.3e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.352537 7f869af78700 5 -- op tracker -- , seq: 1517, time: 2014-06-09 16:22:11.352536, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4fa(2)) 2014-06-09 16:22:11.352542 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4fa(2)) v4 from osd.5 2014-06-09 16:22:11.352544 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352538 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.352545 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405c1c0 2014-06-09 16:22:11.352548 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352550 7f869af78700 5 -- op tracker -- , seq: 1517, time: 2014-06-09 16:22:11.352548, event: started, op: pg_info(1 pgs e243:3.4fa(2)) 2014-06-09 16:22:11.352553 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (14 active) 2014-06-09 16:22:11.352559 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352559 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3077.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352563 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352565 7f869af78700 5 -- op tracker -- , seq: 1517, time: 2014-06-09 16:22:11.352563, event: done, op: pg_info(1 pgs e243:3.4fa(2)) 2014-06-09 16:22:11.352573 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352575 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352563 7f8691f66700 10 osd.26 243 dequeue_op 0x21967b00 prio 63 cost 516096 latency 0.728160 osd_op(client.4567.0:35152 rbd_data.11776b8b4567.00000000000017b4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.584859c7 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.352584 7f86a5610700 10 _do_op 0x308cac60 seq 3077 r = 0, finisher 0xbd72e60 0 2014-06-09 16:22:11.352582 7f8691f66700 5 -- op tracker -- , seq: 582, time: 2014-06-09 16:22:11.352578, event: reached_pg, op: osd_op(client.4567.0:35152 rbd_data.11776b8b4567.00000000000017b4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.584859c7 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352588 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352588 7f86a5610700 10 _finish_op 0x308cac60 seq 3077 osr(2.312 0x2ed0f30)/0x2ed0f30 2014-06-09 16:22:11.352589 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352590 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc5a0 (1 active) 2014-06-09 16:22:11.352591 7f869af78700 20 osd.26 243 _dispatch 0x2a070380 pg_notify(0.2c2(21),3.582(12) epoch 242) v5 2014-06-09 16:22:11.352592 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2870e0 (2 active) 2014-06-09 16:22:11.352594 7f86a5610700 5 _do_op 0x300321e0 seq 3078 osr(3.7ce 0xb7f01a0)/0xb7f01a0 start 2014-06-09 16:22:11.352597 7f86a5610700 10 _do_transaction on 0xbf4f080 2014-06-09 16:22:11.352589 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352596 7f869af78700 5 -- op tracker -- , seq: 1518, time: 2014-06-09 16:22:10.803365, event: header_read, op: pg_notify(0.2c2(21),3.582(12) epoch 242) 2014-06-09 16:22:11.352602 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352604 7f869af78700 5 -- op tracker -- , seq: 1518, time: 2014-06-09 16:22:10.803365, event: throttled, op: pg_notify(0.2c2(21),3.582(12) epoch 242) 2014-06-09 16:22:11.352601 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x21967b00 2014-06-09 16:22:11.352611 7f8691f66700 10 osd.26 243 dequeue_op 0x21967b00 finish 2014-06-09 16:22:11.352614 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352611 7f869af78700 5 -- op tracker -- , seq: 1518, time: 2014-06-09 16:22:10.803392, event: all_read, op: pg_notify(0.2c2(21),3.582(12) epoch 242) 2014-06-09 16:22:11.352616 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (15 active) 2014-06-09 16:22:11.352618 7f869af78700 5 -- op tracker -- , seq: 1518, time: 2014-06-09 16:22:11.352587, event: dispatched, op: pg_notify(0.2c2(21),3.582(12) epoch 242) 2014-06-09 16:22:11.352625 7f869af78700 5 -- op tracker -- , seq: 1518, time: 2014-06-09 16:22:11.352623, event: waiting_for_osdmap, op: pg_notify(0.2c2(21),3.582(12) epoch 242) 2014-06-09 16:22:11.352619 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=206 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.352629 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.352631 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2a070380 2014-06-09 16:22:11.352634 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352636 7f869af78700 5 -- op tracker -- , seq: 1518, time: 2014-06-09 16:22:11.352633, event: started, op: pg_notify(0.2c2(21),3.582(12) epoch 242) 2014-06-09 16:22:11.352632 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=206 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.352648 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3078.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352652 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352645 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=206 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.183819 7 0.000165 2014-06-09 16:22:11.352661 7f869af78700 5 -- op tracker -- , seq: 1518, time: 2014-06-09 16:22:11.352658, event: done, op: pg_notify(0.2c2(21),3.582(12) epoch 242) 2014-06-09 16:22:11.352658 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=206 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.352668 7f86a5610700 10 _do_op 0x300321e0 seq 3078 r = 0, finisher 0xbd682a0 0 2014-06-09 16:22:11.352668 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352670 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352671 7f86a5610700 10 _finish_op 0x300321e0 seq 3078 osr(3.7ce 0xb7f01a0)/0xb7f01a0 2014-06-09 16:22:11.352673 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2870e0 (1 active) 2014-06-09 16:22:11.352675 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383fe60 (2 active) 2014-06-09 16:22:11.352677 7f86a5610700 5 _do_op 0x309a3f20 seq 3079 osr(1.30 0x384c280)/0x384c280 start 2014-06-09 16:22:11.352671 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=206 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.352679 7f86a5610700 10 _do_transaction on 0x2c815c0 2014-06-09 16:22:11.352684 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352689 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352682 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=206 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.352690 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352691 7f869af78700 20 osd.26 243 _dispatch 0x15458700 pg_notify(3.573(10) epoch 242) v5 2014-06-09 16:22:11.352696 7f869af78700 5 -- op tracker -- , seq: 1519, time: 2014-06-09 16:22:10.735743, event: header_read, op: pg_notify(3.573(10) epoch 242) 2014-06-09 16:22:11.352692 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'628 -> 213'628 2014-06-09 16:22:11.352703 7f869af78700 5 -- op tracker -- , seq: 1519, time: 2014-06-09 16:22:10.735743, event: throttled, op: pg_notify(3.573(10) epoch 242) 2014-06-09 16:22:11.352709 7f869af78700 5 -- op tracker -- , seq: 1519, time: 2014-06-09 16:22:10.735762, event: all_read, op: pg_notify(3.573(10) epoch 242) 2014-06-09 16:22:11.352715 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352715 7f869af78700 5 -- op tracker -- , seq: 1519, time: 2014-06-09 16:22:11.352689, event: dispatched, op: pg_notify(3.573(10) epoch 242) 2014-06-09 16:22:11.352722 7f869af78700 5 -- op tracker -- , seq: 1519, time: 2014-06-09 16:22:11.352720, event: waiting_for_osdmap, op: pg_notify(3.573(10) epoch 242) 2014-06-09 16:22:11.352726 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.352727 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x15458700 2014-06-09 16:22:11.352729 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3079.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352733 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352732 7f869af78700 5 -- op tracker -- , seq: 1519, time: 2014-06-09 16:22:11.352730, event: started, op: pg_notify(3.573(10) epoch 242) 2014-06-09 16:22:11.352734 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.352749 7f869af78700 5 -- op tracker -- , seq: 1519, time: 2014-06-09 16:22:11.352746, event: done, op: pg_notify(3.573(10) epoch 242) 2014-06-09 16:22:11.352753 7f86a5610700 10 _do_op 0x309a3f20 seq 3079 r = 0, finisher 0xc40d720 0 2014-06-09 16:22:11.352756 7f86a5610700 10 _finish_op 0x309a3f20 seq 3079 osr(1.30 0x384c280)/0x384c280 2014-06-09 16:22:11.352757 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352758 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383fe60 (1 active) 2014-06-09 16:22:11.352759 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352760 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0000 (2 active) 2014-06-09 16:22:11.352762 7f86a5610700 5 _do_op 0x309a2720 seq 3080 osr(1.603 0x384dc90)/0x384dc90 start 2014-06-09 16:22:11.352764 7f86a5610700 10 _do_transaction on 0xc014300 2014-06-09 16:22:11.352769 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352774 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.352776 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.352776 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352777 7f869af78700 20 osd.26 243 _dispatch 0x3004c000 pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.352781 7f8691f66700 5 queue_transactions existing osr(3.4fa 0x3ed1720)/0x3ed1720 2014-06-09 16:22:11.352784 7f8691f66700 10 op_submit_start 3090 2014-06-09 16:22:11.352785 7f8691f66700 5 queue_transactions (trailing journal) 3090 2014-06-09 16:22:11.352783 7f869af78700 5 -- op tracker -- , seq: 1520, time: 2014-06-09 16:22:11.059575, event: header_read, op: pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.352787 7f8691f66700 5 queue_op 0x308cbc20 seq 3090 osr(3.4fa 0x3ed1720) 1335 bytes (queue has 12 ops and 22088 bytes) 2014-06-09 16:22:11.352789 7f8691f66700 10 op_submit_finish 3090 2014-06-09 16:22:11.352790 7f869af78700 5 -- op tracker -- , seq: 1520, time: 2014-06-09 16:22:11.059576, event: throttled, op: pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.352794 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (14 active) 2014-06-09 16:22:11.352797 7f869af78700 5 -- op tracker -- , seq: 1520, time: 2014-06-09 16:22:11.059584, event: all_read, op: pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.352799 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352799 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352803 7f869af78700 5 -- op tracker -- , seq: 1520, time: 2014-06-09 16:22:11.352774, event: dispatched, op: pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.352811 7f869af78700 5 -- op tracker -- , seq: 1520, time: 2014-06-09 16:22:11.352808, event: waiting_for_osdmap, op: pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.352814 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3080.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352803 7f8691f66700 10 osd.26 243 dequeue_op 0x2c06bb00 prio 63 cost 102400 latency 0.486034 osd_op(client.4570.0:34427 rbd_data.117a6b8b4567.0000000000001713 [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.33969c7 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.352818 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352815 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.352819 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004c000 2014-06-09 16:22:11.352824 7f8691f66700 5 -- op tracker -- , seq: 897, time: 2014-06-09 16:22:11.352819, event: reached_pg, op: osd_op(client.4570.0:34427 rbd_data.117a6b8b4567.0000000000001713 [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~102400] 3.33969c7 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352838 7f86a5610700 10 _do_op 0x309a2720 seq 3080 r = 0, finisher 0x13964460 0 2014-06-09 16:22:11.352827 7f868af58700 10 osd.26 243 dequeue_op 0x1c66be00 prio 63 cost 860160 latency 0.841441 osd_op(client.4577.0:36986 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 0~860160] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.352841 7f86a5610700 10 _finish_op 0x309a2720 seq 3080 osr(1.603 0x384dc90)/0x384dc90 2014-06-09 16:22:11.352843 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0000 (1 active) 2014-06-09 16:22:11.352834 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352845 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62eb40 (2 active) 2014-06-09 16:22:11.352848 7f86a5610700 5 _do_op 0x309a26c0 seq 3081 osr(2.364 0x2ed11b0)/0x2ed11b0 start 2014-06-09 16:22:11.352850 7f86a5610700 10 _do_transaction on 0xc0d0600 2014-06-09 16:22:11.352847 7f868af58700 5 -- op tracker -- , seq: 472, time: 2014-06-09 16:22:11.352843, event: reached_pg, op: osd_op(client.4577.0:36986 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 0~860160] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352847 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x2c06bb00 2014-06-09 16:22:11.352855 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352856 7f8691f66700 10 osd.26 243 dequeue_op 0x2c06bb00 finish 2014-06-09 16:22:11.352859 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352862 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (15 active) 2014-06-09 16:22:11.352853 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352865 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x1c66be00 2014-06-09 16:22:11.352865 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:0.2c2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.352874 7f868af58700 10 osd.26 243 dequeue_op 0x1c66be00 finish 2014-06-09 16:22:11.352878 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352876 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.352883 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352886 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352885 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352890 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:3.582( v 213'598 (0'0,213'598] local-les=191 n=118 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.352887 7f868af58700 10 osd.26 243 dequeue_op 0x270be100 prio 63 cost 1548288 latency 0.532798 osd_op(client.4573.0:34236 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 966656~1548288] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352900 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3081.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.352903 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.352907 7f868af58700 5 -- op tracker -- , seq: 795, time: 2014-06-09 16:22:11.352903, event: reached_pg, op: osd_op(client.4573.0:34236 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 966656~1548288] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352904 7f8691f66700 7 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.352914 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352926 7f86a5610700 10 _do_op 0x309a26c0 seq 3081 r = 0, finisher 0xbdbb960 0 2014-06-09 16:22:11.352929 7f86a5610700 10 _finish_op 0x309a26c0 seq 3081 osr(2.364 0x2ed11b0)/0x2ed11b0 2014-06-09 16:22:11.352914 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.582( v 213'598 (0'0,213'598] local-les=191 n=118 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.352931 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62eb40 (1 active) 2014-06-09 16:22:11.352932 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352925 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270be100 2014-06-09 16:22:11.352933 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339320 (2 active) 2014-06-09 16:22:11.352934 7f868af58700 10 osd.26 243 dequeue_op 0x270be100 finish 2014-06-09 16:22:11.352935 7f86a5610700 5 _do_op 0x11fcfc80 seq 3082 osr(3.3b4 0x3ed1300)/0x3ed1300 start 2014-06-09 16:22:11.352938 7f86a5610700 10 _do_transaction on 0x2fb7e00 2014-06-09 16:22:11.352937 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.352943 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352943 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.352936 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:3.573( v 213'607 (0'0,213'607] local-les=203 n=120 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.352948 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.352946 7f868af58700 10 osd.26 243 dequeue_op 0x14053100 prio 63 cost 1032192 latency 1.115239 osd_op(client.4574.0:31684 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.352960 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.352966 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (14 active) 2014-06-09 16:22:11.352965 7f868af58700 5 -- op tracker -- , seq: 183, time: 2014-06-09 16:22:11.352960, event: reached_pg, op: osd_op(client.4574.0:31684 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.352972 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352973 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.352971 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.352976 7f8691f66700 10 osd.26 243 dequeue_op 0x1804c200 prio 63 cost 1032192 latency 1.037686 osd_op(client.4575.0:32139 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 958464~1032192] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.352988 7f86a5610700 15 _omap_rmkeyrange meta/1fe09d15/pglog_3.3b4/0//-1 [0000000214.00000000000000000673,4294967295.18446744073709551615] 2014-06-09 16:22:11.352982 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053100 2014-06-09 16:22:11.352991 7f86a5610700 15 get_omap_iterator meta/1fe09d15/pglog_3.3b4/0//-1 2014-06-09 16:22:11.352991 7f868af58700 10 osd.26 243 dequeue_op 0x14053100 finish 2014-06-09 16:22:11.352995 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.353000 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.352996 7f8691f66700 5 -- op tracker -- , seq: 263, time: 2014-06-09 16:22:11.352991, event: reached_pg, op: osd_op(client.4575.0:32139 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 958464~1032192] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.353008 7f869af78700 5 -- op tracker -- , seq: 1520, time: 2014-06-09 16:22:11.352822, event: started, op: pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353004 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.353015 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x1804c200 2014-06-09 16:22:11.353022 7f869af78700 5 -- op tracker -- , seq: 1520, time: 2014-06-09 16:22:11.353019, event: done, op: pg_log(1.92 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353025 7f8691f66700 10 osd.26 243 dequeue_op 0x1804c200 finish 2014-06-09 16:22:11.353030 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353031 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353031 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.353034 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (15 active) 2014-06-09 16:22:11.353036 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.353044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353046 7f869af78700 20 osd.26 243 _dispatch 0x2d737600 pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.353046 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 1.92( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.353053 7f869af78700 5 -- op tracker -- , seq: 1521, time: 2014-06-09 16:22:10.986752, event: header_read, op: pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353057 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.353062 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.353060 7f869af78700 5 -- op tracker -- , seq: 1521, time: 2014-06-09 16:22:10.986752, event: throttled, op: pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353067 7f869af78700 5 -- op tracker -- , seq: 1521, time: 2014-06-09 16:22:10.986771, event: all_read, op: pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353065 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.92( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.039217 7 0.000161 2014-06-09 16:22:11.353073 7f869af78700 5 -- op tracker -- , seq: 1521, time: 2014-06-09 16:22:11.353043, event: dispatched, op: pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353075 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.92( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.353081 7f869af78700 5 -- op tracker -- , seq: 1521, time: 2014-06-09 16:22:11.353078, event: waiting_for_osdmap, op: pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353085 7f86a5610700 15 _omap_rmkeys meta/1fe09d15/pglog_3.3b4/0//-1 2014-06-09 16:22:11.353085 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) v4 from osd.3 2014-06-09 16:22:11.353089 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d737600 2014-06-09 16:22:11.353085 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.92( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.353095 7f869af78700 5 -- op tracker -- , seq: 1521, time: 2014-06-09 16:22:11.353092, event: started, op: pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353095 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.353104 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.353108 7f869af78700 5 -- op tracker -- , seq: 1521, time: 2014-06-09 16:22:11.353105, event: done, op: pg_log(3.c1 epoch 242 log log((214'607,214'609], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353117 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353112 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.353118 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353118 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe09d15/pglog_3.3b4/0//-1 2014-06-09 16:22:11.353127 7f86a5610700 10 _omap_rmkeys meta/1fe09d15/pglog_3.3b4/0//-1 r = 0 2014-06-09 16:22:11.353132 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353131 7f86a5610700 15 _omap_rmkeys meta/1fe09d15/pglog_3.3b4/0//-1 2014-06-09 16:22:11.353133 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353136 7f86a5610700 10 _omap_rmkeys meta/1fe09d15/pglog_3.3b4/0//-1 r = 0 2014-06-09 16:22:11.353134 7f869af78700 20 osd.26 243 _dispatch 0x2fe32600 pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.353142 7f869af78700 5 -- op tracker -- , seq: 1522, time: 2014-06-09 16:22:11.044407, event: header_read, op: pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353146 7f86a5610700 15 _omap_setkeys meta/1fe09d15/pglog_3.3b4/0//-1 2014-06-09 16:22:11.353149 7f869af78700 5 -- op tracker -- , seq: 1522, time: 2014-06-09 16:22:11.044408, event: throttled, op: pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353156 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.353156 7f869af78700 5 -- op tracker -- , seq: 1522, time: 2014-06-09 16:22:11.044420, event: all_read, op: pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353161 7f8691f66700 5 queue_transactions existing osr(1.92 0x384c410)/0x384c410 2014-06-09 16:22:11.353163 7f8691f66700 10 op_submit_start 3091 2014-06-09 16:22:11.353164 7f8691f66700 5 queue_transactions (trailing journal) 3091 2014-06-09 16:22:11.353162 7f869af78700 5 -- op tracker -- , seq: 1522, time: 2014-06-09 16:22:11.353131, event: dispatched, op: pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353165 7f8691f66700 5 queue_op 0x308cbbc0 seq 3091 osr(1.92 0x384c410) 1873 bytes (queue has 11 ops and 20104 bytes) 2014-06-09 16:22:11.353168 7f8691f66700 10 op_submit_finish 3091 2014-06-09 16:22:11.353167 7f86a5610700 10 cid: oid: 1fe09d15/pglog_3.3b4/0//-1 skipping op, spos 3082.0.0 <= header.spos 150433.0.0 2014-06-09 16:22:11.353172 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (14 active) 2014-06-09 16:22:11.353169 7f869af78700 5 -- op tracker -- , seq: 1522, time: 2014-06-09 16:22:11.353167, event: waiting_for_osdmap, op: pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353172 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3082.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.353175 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.353174 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.353177 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.353178 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe32600 2014-06-09 16:22:11.353183 7f869af78700 5 -- op tracker -- , seq: 1522, time: 2014-06-09 16:22:11.353181, event: started, op: pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353180 7f8691f66700 10 osd.26 243 dequeue_op 0xe032f00 prio 63 cost 516096 latency 1.380440 osd_op(client.4565.0:32568 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 2514944~516096] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.353195 7f869af78700 5 -- op tracker -- , seq: 1522, time: 2014-06-09 16:22:11.353193, event: done, op: pg_log(1.67a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353206 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353208 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353214 7f86a5610700 10 _do_op 0x11fcfc80 seq 3082 r = 0, finisher 0xbd6ac60 0 2014-06-09 16:22:11.353217 7f86a5610700 10 _finish_op 0x11fcfc80 seq 3082 osr(3.3b4 0x3ed1300)/0x3ed1300 2014-06-09 16:22:11.353220 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353221 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353221 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339320 (1 active) 2014-06-09 16:22:11.353223 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1150e0 (2 active) 2014-06-09 16:22:11.353222 7f869af78700 20 osd.26 243 _dispatch 0x48fc400 pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.353225 7f86a5610700 5 _do_op 0x308ca480 seq 3083 osr(2.37 0x2ed05f0)/0x2ed05f0 start 2014-06-09 16:22:11.353227 7f86a5610700 10 _do_transaction on 0xc0d4d80 2014-06-09 16:22:11.353230 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (16 active) 2014-06-09 16:22:11.353229 7f869af78700 5 -- op tracker -- , seq: 1523, time: 2014-06-09 16:22:10.922124, event: header_read, op: pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353233 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.353237 7f869af78700 5 -- op tracker -- , seq: 1523, time: 2014-06-09 16:22:10.922125, event: throttled, op: pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353235 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'607 (0'0,214'607] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MLogRec from 3 2014-06-09 16:22:11.353244 7f869af78700 5 -- op tracker -- , seq: 1523, time: 2014-06-09 16:22:10.922146, event: all_read, op: pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353251 7f869af78700 5 -- op tracker -- , seq: 1523, time: 2014-06-09 16:22:11.353219, event: dispatched, op: pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353249 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'607 (0'0,214'607] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd3 2014-06-09 16:22:11.353259 7f869af78700 5 -- op tracker -- , seq: 1523, time: 2014-06-09 16:22:11.353256, event: waiting_for_osdmap, op: pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353264 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.353264 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) v4 from osd.9 2014-06-09 16:22:11.353261 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'607 (0'0,214'607] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.353268 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x48fc400 2014-06-09 16:22:11.353270 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'607 (0'0,214'607] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.353274 7f869af78700 5 -- op tracker -- , seq: 1523, time: 2014-06-09 16:22:11.353271, event: started, op: pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353278 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3083.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.353282 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.353279 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'607 (0'0,214'607] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.3: log((214'607,214'609], crt=0'0) missing(0) 2014-06-09 16:22:11.353287 7f869af78700 5 -- op tracker -- , seq: 1523, time: 2014-06-09 16:22:11.353285, event: done, op: pg_log(3.6d3 epoch 242 log log((214'630,214'632], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.353289 7f868ef60700 10 merge_log log((214'607,214'609], crt=0'0) from osd.3 into log((0'0,214'607], crt=0'0) 2014-06-09 16:22:11.353293 7f868ef60700 10 merge_log extending head to 214'609 2014-06-09 16:22:11.353294 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353296 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353294 7f868ef60700 20 ? 214'609 (0'0) modify f30230c1/rbd_data.11686b8b4567.00000000000016f3/head//3 by client.4552.0:34361 2014-06-09 10:19:29.917513 2014-06-09 16:22:11.353302 7f868ef60700 20 ? 214'608 (214'607) modify 217f88c1/rbd_data.11686b8b4567.00000000000016d2/head//3 by client.4552.0:34215 2014-06-09 10:19:18.782297 2014-06-09 16:22:11.353306 7f86a5610700 10 _do_op 0x308ca480 seq 3083 r = 0, finisher 0x170011c0 0 2014-06-09 16:22:11.353308 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353309 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353309 7f86a5610700 10 _finish_op 0x308ca480 seq 3083 osr(2.37 0x2ed05f0)/0x2ed05f0 2014-06-09 16:22:11.353311 7f869af78700 20 osd.26 243 _dispatch 0x30420480 osd_map(243..243 src has 1..243) v3 2014-06-09 16:22:11.353307 7f868ef60700 20 merge_log 214'608 (214'607) modify 217f88c1/rbd_data.11686b8b4567.00000000000016d2/head//3 by client.4552.0:34215 2014-06-09 10:19:18.782297 2014-06-09 16:22:11.353312 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1150e0 (1 active) 2014-06-09 16:22:11.353314 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbc20 (2 active) 2014-06-09 16:22:11.353316 7f86a5610700 5 _do_op 0x308cbce0 seq 3084 osr(0.663 0x2bd7f30)/0x2bd7f30 start 2014-06-09 16:22:11.353318 7f86a5610700 10 _do_transaction on 0xbebd140 2014-06-09 16:22:11.353318 7f869af78700 3 osd.26 243 handle_osd_map epochs [243,243], i have 243, src has [1,243] 2014-06-09 16:22:11.353321 7f869af78700 10 osd.26 243 no new maps here, dropping 2014-06-09 16:22:11.353322 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353318 7f868ef60700 20 merge_log 214'609 (0'0) modify f30230c1/rbd_data.11686b8b4567.00000000000016f3/head//3 by client.4552.0:34361 2014-06-09 10:19:29.917513 2014-06-09 16:22:11.353324 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353323 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.353336 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353337 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353338 7f869af78700 20 osd.26 243 _dispatch 0x2d7321c0 pg_notify(3.428(10) epoch 242) v5 2014-06-09 16:22:11.353343 7f869af78700 5 -- op tracker -- , seq: 1524, time: 2014-06-09 16:22:10.886366, event: header_read, op: pg_notify(3.428(10) epoch 242) 2014-06-09 16:22:11.353351 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.353350 7f869af78700 5 -- op tracker -- , seq: 1524, time: 2014-06-09 16:22:10.886366, event: throttled, op: pg_notify(3.428(10) epoch 242) 2014-06-09 16:22:11.353356 7f869af78700 5 -- op tracker -- , seq: 1524, time: 2014-06-09 16:22:10.886386, event: all_read, op: pg_notify(3.428(10) epoch 242) 2014-06-09 16:22:11.353361 7f869af78700 5 -- op tracker -- , seq: 1524, time: 2014-06-09 16:22:11.353335, event: dispatched, op: pg_notify(3.428(10) epoch 242) 2014-06-09 16:22:11.353364 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3084.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.353368 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.353368 7f869af78700 5 -- op tracker -- , seq: 1524, time: 2014-06-09 16:22:11.353366, event: waiting_for_osdmap, op: pg_notify(3.428(10) epoch 242) 2014-06-09 16:22:11.353372 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.353374 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d7321c0 2014-06-09 16:22:11.353380 7f86a5610700 10 _do_op 0x308cbce0 seq 3084 r = 0, finisher 0xbd6c3e0 0 2014-06-09 16:22:11.353379 7f869af78700 5 -- op tracker -- , seq: 1524, time: 2014-06-09 16:22:11.353377, event: started, op: pg_notify(3.428(10) epoch 242) 2014-06-09 16:22:11.353383 7f86a5610700 10 _finish_op 0x308cbce0 seq 3084 osr(0.663 0x2bd7f30)/0x2bd7f30 2014-06-09 16:22:11.353385 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbc20 (1 active) 2014-06-09 16:22:11.353387 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497680 (2 active) 2014-06-09 16:22:11.353389 7f86a5610700 5 _do_op 0x30607b60 seq 3085 osr(0.369 0x2bd6ff0)/0x2bd6ff0 start 2014-06-09 16:22:11.353391 7f86a5610700 10 _do_transaction on 0xc0cb140 2014-06-09 16:22:11.353396 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.353395 7f869af78700 5 -- op tracker -- , seq: 1524, time: 2014-06-09 16:22:11.353393, event: done, op: pg_notify(3.428(10) epoch 242) 2014-06-09 16:22:11.353403 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353405 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353423 7f869af78700 20 osd.26 243 _dispatch 0x2a072600 pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.353424 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.353429 7f869af78700 5 -- op tracker -- , seq: 1525, time: 2014-06-09 16:22:11.003895, event: header_read, op: pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353436 7f869af78700 5 -- op tracker -- , seq: 1525, time: 2014-06-09 16:22:11.003895, event: throttled, op: pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353438 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3085.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.353442 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.353443 7f869af78700 5 -- op tracker -- , seq: 1525, time: 2014-06-09 16:22:11.003903, event: all_read, op: pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353450 7f869af78700 5 -- op tracker -- , seq: 1525, time: 2014-06-09 16:22:11.353420, event: dispatched, op: pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353457 7f869af78700 5 -- op tracker -- , seq: 1525, time: 2014-06-09 16:22:11.353454, event: waiting_for_osdmap, op: pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353461 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.353465 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a072600 2014-06-09 16:22:11.353466 7f86a5610700 10 _do_op 0x30607b60 seq 3085 r = 0, finisher 0x19940c00 0 2014-06-09 16:22:11.353469 7f86a5610700 10 _finish_op 0x30607b60 seq 3085 osr(0.369 0x2bd6ff0)/0x2bd6ff0 2014-06-09 16:22:11.353471 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497680 (1 active) 2014-06-09 16:22:11.353470 7f869af78700 5 -- op tracker -- , seq: 1525, time: 2014-06-09 16:22:11.353468, event: started, op: pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353473 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358c60 (2 active) 2014-06-09 16:22:11.353475 7f86a5610700 5 _do_op 0x308cbc80 seq 3086 osr(1.691 0x384de30)/0x384de30 start 2014-06-09 16:22:11.353477 7f86a5610700 10 _do_transaction on 0xc128a80 2014-06-09 16:22:11.353482 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.353483 7f869af78700 5 -- op tracker -- , seq: 1525, time: 2014-06-09 16:22:11.353480, event: done, op: pg_log(2.73 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353490 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353492 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353503 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353504 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353505 7f869af78700 20 osd.26 243 _dispatch 0x302a2a00 pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.353512 7f869af78700 5 -- op tracker -- , seq: 1526, time: 2014-06-09 16:22:11.017464, event: header_read, op: pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353519 7f869af78700 5 -- op tracker -- , seq: 1526, time: 2014-06-09 16:22:11.017464, event: throttled, op: pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353526 7f869af78700 5 -- op tracker -- , seq: 1526, time: 2014-06-09 16:22:11.017477, event: all_read, op: pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353532 7f869af78700 5 -- op tracker -- , seq: 1526, time: 2014-06-09 16:22:11.353503, event: dispatched, op: pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353533 7f868ef60700 10 merge_log result log((0'0,214'609], crt=214'609) missing(2) changed=1 2014-06-09 16:22:11.353540 7f869af78700 5 -- op tracker -- , seq: 1526, time: 2014-06-09 16:22:11.353537, event: waiting_for_osdmap, op: pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353544 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.353548 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x302a2a00 2014-06-09 16:22:11.353538 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 peering m=2] peer osd.3 now 3.c1( v 214'609 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) missing(0) 2014-06-09 16:22:11.353554 7f869af78700 5 -- op tracker -- , seq: 1526, time: 2014-06-09 16:22:11.353551, event: started, op: pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353557 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 peering m=2] exit Started/Primary/Peering/GetLog 0.370352 5 0.000220 2014-06-09 16:22:11.353567 7f869af78700 5 -- op tracker -- , seq: 1526, time: 2014-06-09 16:22:11.353564, event: done, op: pg_log(1.82 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353574 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353575 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353570 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 peering m=2] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.353587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353588 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353580 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 peering m=2] state: osd.3 has no missing, identical log 2014-06-09 16:22:11.353589 7f869af78700 20 osd.26 243 _dispatch 0x29ec4000 pg_notify(1.5e1(26) epoch 242) v5 2014-06-09 16:22:11.353594 7f869af78700 5 -- op tracker -- , seq: 1527, time: 2014-06-09 16:22:10.833312, event: header_read, op: pg_notify(1.5e1(26) epoch 242) 2014-06-09 16:22:11.353593 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 peering m=2] exit Started/Primary/Peering/GetMissing 0.000023 0 0.000000 2014-06-09 16:22:11.353601 7f869af78700 5 -- op tracker -- , seq: 1527, time: 2014-06-09 16:22:10.833312, event: throttled, op: pg_notify(1.5e1(26) epoch 242) 2014-06-09 16:22:11.353608 7f869af78700 5 -- op tracker -- , seq: 1527, time: 2014-06-09 16:22:10.833347, event: all_read, op: pg_notify(1.5e1(26) epoch 242) 2014-06-09 16:22:11.353605 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 peering m=2] state: Leaving Peering 2014-06-09 16:22:11.353614 7f869af78700 5 -- op tracker -- , seq: 1527, time: 2014-06-09 16:22:11.353587, event: dispatched, op: pg_notify(1.5e1(26) epoch 242) 2014-06-09 16:22:11.353620 7f869af78700 5 -- op tracker -- , seq: 1527, time: 2014-06-09 16:22:11.353618, event: waiting_for_osdmap, op: pg_notify(1.5e1(26) epoch 242) 2014-06-09 16:22:11.353616 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 peering m=2] exit Started/Primary/Peering 1.090485 0 0.000000 2014-06-09 16:22:11.353624 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.353622 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509280,0xbef1fa0,0x13965500,0x33a65e0,0xcd0ffe0,0xc5a6b80,0xbd68120,0x1322dc40,0x30509da0,0xc6e9880] 2014-06-09 16:22:11.353626 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x29ec4000 2014-06-09 16:22:11.353631 7f869af78700 5 -- op tracker -- , seq: 1527, time: 2014-06-09 16:22:11.353629, event: started, op: pg_notify(1.5e1(26) epoch 242) 2014-06-09 16:22:11.353635 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509280,0xbef1fa0,0x13965500,0x33a65e0,0xcd0ffe0,0xc5a6b80,0xbd68120,0x1322dc40,0x30509da0,0xc6e9880] 2014-06-09 16:22:11.353638 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.353631 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] enter Started/Primary/Active 2014-06-09 16:22:11.353639 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.353640 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] state: In Active, about to call activate 2014-06-09 16:22:11.353646 7f869af78700 5 -- op tracker -- , seq: 1527, time: 2014-06-09 16:22:11.353644, event: done, op: pg_notify(1.5e1(26) epoch 242) 2014-06-09 16:22:11.353654 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353655 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353652 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.353669 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.353662 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] activate - snap_trimq [] 2014-06-09 16:22:11.353672 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353673 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353674 7f869af78700 20 osd.26 243 _dispatch 0x1ac94380 pg_notify(2.186(20) epoch 242) v5 2014-06-09 16:22:11.353671 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] activate - not complete, missing(2) 2014-06-09 16:22:11.353679 7f869af78700 5 -- op tracker -- , seq: 1528, time: 2014-06-09 16:22:10.748319, event: header_read, op: pg_notify(2.186(20) epoch 242) 2014-06-09 16:22:11.353684 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3086.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.353688 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.353687 7f869af78700 5 -- op tracker -- , seq: 1528, time: 2014-06-09 16:22:10.748320, event: throttled, op: pg_notify(2.186(20) epoch 242) 2014-06-09 16:22:11.353693 7f869af78700 5 -- op tracker -- , seq: 1528, time: 2014-06-09 16:22:10.748359, event: all_read, op: pg_notify(2.186(20) epoch 242) 2014-06-09 16:22:11.353699 7f869af78700 5 -- op tracker -- , seq: 1528, time: 2014-06-09 16:22:11.353671, event: dispatched, op: pg_notify(2.186(20) epoch 242) 2014-06-09 16:22:11.353705 7f869af78700 5 -- op tracker -- , seq: 1528, time: 2014-06-09 16:22:11.353704, event: waiting_for_osdmap, op: pg_notify(2.186(20) epoch 242) 2014-06-09 16:22:11.353710 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.353711 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1ac94380 2014-06-09 16:22:11.353711 7f86a5610700 10 _do_op 0x308cbc80 seq 3086 r = 0, finisher 0xbebe380 0 2014-06-09 16:22:11.353716 7f869af78700 5 -- op tracker -- , seq: 1528, time: 2014-06-09 16:22:11.353714, event: started, op: pg_notify(2.186(20) epoch 242) 2014-06-09 16:22:11.353721 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6bf40] 2014-06-09 16:22:11.353724 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6bf40] 2014-06-09 16:22:11.353725 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.353726 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.353729 7f86a5610700 10 _finish_op 0x308cbc80 seq 3086 osr(1.691 0x384de30)/0x384de30 2014-06-09 16:22:11.353731 7f869af78700 5 -- op tracker -- , seq: 1528, time: 2014-06-09 16:22:11.353728, event: done, op: pg_notify(2.186(20) epoch 242) 2014-06-09 16:22:11.353733 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358c60 (1 active) 2014-06-09 16:22:11.353736 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3586c0 (2 active) 2014-06-09 16:22:11.353737 7f86a5610700 5 _do_op 0x309a3020 seq 3087 osr(2.372 0x2ed1150)/0x2ed1150 start 2014-06-09 16:22:11.353739 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353740 7f86a5610700 10 _do_transaction on 0x2eaafc0 2014-06-09 16:22:11.353740 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353745 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.353756 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353757 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353758 7f869af78700 20 osd.26 243 _dispatch 0x304f9600 pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.353764 7f869af78700 5 -- op tracker -- , seq: 1529, time: 2014-06-09 16:22:11.061403, event: header_read, op: pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353772 7f869af78700 5 -- op tracker -- , seq: 1529, time: 2014-06-09 16:22:11.061404, event: throttled, op: pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353775 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.353779 7f869af78700 5 -- op tracker -- , seq: 1529, time: 2014-06-09 16:22:11.061414, event: all_read, op: pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353785 7f869af78700 5 -- op tracker -- , seq: 1529, time: 2014-06-09 16:22:11.353755, event: dispatched, op: pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353789 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3087.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.353793 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.353792 7f869af78700 5 -- op tracker -- , seq: 1529, time: 2014-06-09 16:22:11.353790, event: waiting_for_osdmap, op: pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353797 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.353801 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f9600 2014-06-09 16:22:11.353793 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] activate - starting recovery 2014-06-09 16:22:11.353806 7f869af78700 5 -- op tracker -- , seq: 1529, time: 2014-06-09 16:22:11.353804, event: started, op: pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353811 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x5552000 (1 active) 2014-06-09 16:22:11.353808 7f868ef60700 10 osd.26 243 queue_for_recovery queued pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.353815 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.353819 7f86a5610700 10 _do_op 0x309a3020 seq 3087 r = 0, finisher 0x170001c0 0 2014-06-09 16:22:11.353819 7f869af78700 5 -- op tracker -- , seq: 1529, time: 2014-06-09 16:22:11.353816, event: done, op: pg_log(1.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.353827 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd67500] 2014-06-09 16:22:11.353828 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353830 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353830 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd67500] 2014-06-09 16:22:11.353831 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.353833 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.353821 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] activate peer osd.3 3.c1( v 214'609 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) 2014-06-09 16:22:11.353836 7f86a5610700 10 _finish_op 0x309a3020 seq 3087 osr(2.372 0x2ed1150)/0x2ed1150 2014-06-09 16:22:11.353840 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3586c0 (1 active) 2014-06-09 16:22:11.353834 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] activate peer osd.3 is up to date, queueing in pending_activators 2014-06-09 16:22:11.353841 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353842 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4240 (2 active) 2014-06-09 16:22:11.353844 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353844 7f86a5610700 5 _do_op 0x2fc8cb40 seq 3088 osr(1.5e 0x384c5d0)/0x384c5d0 start 2014-06-09 16:22:11.353845 7f869af78700 20 osd.26 243 _dispatch 0x3002c900 osd_map(243..243 src has 1..243) v3 2014-06-09 16:22:11.353846 7f86a5610700 10 _do_transaction on 0xc138180 2014-06-09 16:22:11.353849 7f869af78700 3 osd.26 243 handle_osd_map epochs [243,243], i have 243, src has [1,243] 2014-06-09 16:22:11.353851 7f869af78700 10 osd.26 243 no new maps here, dropping 2014-06-09 16:22:11.353852 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.353853 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353854 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353850 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] activate peer osd.3 3.c1( v 214'609 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) uptodate 2014-06-09 16:22:11.353870 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353871 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353872 7f869af78700 20 osd.26 243 _dispatch 0x15999880 pg_notify(3.308(12),3.7d4(12),1.327(21) epoch 243) v5 2014-06-09 16:22:11.353864 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2 u=2] needs_recovery primary has 2 missing 2014-06-09 16:22:11.353882 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.353879 7f869af78700 5 -- op tracker -- , seq: 1530, time: 2014-06-09 16:22:11.044432, event: header_read, op: pg_notify(3.308(12),3.7d4(12),1.327(21) epoch 243) 2014-06-09 16:22:11.353877 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2 u=2] search_for_missing f30230c1/rbd_data.11686b8b4567.00000000000016f3/head//3 214'609 also missing on osd.26 2014-06-09 16:22:11.353888 7f869af78700 5 -- op tracker -- , seq: 1530, time: 2014-06-09 16:22:11.044432, event: throttled, op: pg_notify(3.308(12),3.7d4(12),1.327(21) epoch 243) 2014-06-09 16:22:11.353894 7f869af78700 5 -- op tracker -- , seq: 1530, time: 2014-06-09 16:22:11.044491, event: all_read, op: pg_notify(3.308(12),3.7d4(12),1.327(21) epoch 243) 2014-06-09 16:22:11.353896 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3088.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.353889 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2 u=2] search_for_missing 217f88c1/rbd_data.11686b8b4567.00000000000016d2/head//3 214'608 also missing on osd.26 2014-06-09 16:22:11.353900 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.353901 7f869af78700 5 -- op tracker -- , seq: 1530, time: 2014-06-09 16:22:11.353870, event: dispatched, op: pg_notify(3.308(12),3.7d4(12),1.327(21) epoch 243) 2014-06-09 16:22:11.353900 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2 u=2] needs_recovery_map missing {f30230c1/rbd_data.11686b8b4567.00000000000016f3/head//3=214'609,217f88c1/rbd_data.11686b8b4567.00000000000016d2/head//3=214'608(214'607)} 2014-06-09 16:22:11.353908 7f869af78700 5 -- op tracker -- , seq: 1530, time: 2014-06-09 16:22:11.353906, event: waiting_for_osdmap, op: pg_notify(3.308(12),3.7d4(12),1.327(21) epoch 243) 2014-06-09 16:22:11.353913 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.353915 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15999880 2014-06-09 16:22:11.353916 7f86a5610700 10 _do_op 0x2fc8cb40 seq 3088 r = 0, finisher 0x1a0022e0 0 2014-06-09 16:22:11.353911 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2 u=2] activate: adding 3 as a source 2014-06-09 16:22:11.353920 7f869af78700 5 -- op tracker -- , seq: 1530, time: 2014-06-09 16:22:11.353918, event: started, op: pg_notify(3.308(12),3.7d4(12),1.327(21) epoch 243) 2014-06-09 16:22:11.353926 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e8b20] 2014-06-09 16:22:11.353929 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e8b20] 2014-06-09 16:22:11.353931 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.353921 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2 u=2] search_for_missing f30230c1/rbd_data.11686b8b4567.00000000000016f3/head//3 214'609 is on osd.3 2014-06-09 16:22:11.353932 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.353935 7f86a5610700 10 _finish_op 0x2fc8cb40 seq 3088 osr(1.5e 0x384c5d0)/0x384c5d0 2014-06-09 16:22:11.353937 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4240 (1 active) 2014-06-09 16:22:11.353939 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359560 (2 active) 2014-06-09 16:22:11.353941 7f86a5610700 5 _do_op 0x300339e0 seq 3089 osr(1.38d 0x384d570)/0x384d570 start 2014-06-09 16:22:11.353943 7f86a5610700 10 _do_transaction on 0xc265e00 2014-06-09 16:22:11.353934 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2 u=1] search_for_missing 217f88c1/rbd_data.11686b8b4567.00000000000016d2/head//3 214'608 is on osd.3 2014-06-09 16:22:11.353949 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.353946 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] needs_recovery_map missing {f30230c1/rbd_data.11686b8b4567.00000000000016f3/head//3=214'609,217f88c1/rbd_data.11686b8b4567.00000000000016d2/head//3=214'608(214'607)} 2014-06-09 16:22:11.353953 7f869af78700 5 -- op tracker -- , seq: 1530, time: 2014-06-09 16:22:11.353950, event: done, op: pg_notify(3.308(12),3.7d4(12),1.327(21) epoch 243) 2014-06-09 16:22:11.353960 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] build_might_have_unfound 2014-06-09 16:22:11.353971 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353973 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353978 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.353971 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] _calc_past_interval_range: already have past intervals back to 201 2014-06-09 16:22:11.353983 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] build_might_have_unfound: built 3 2014-06-09 16:22:11.353992 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3089.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.353995 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.353997 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.353998 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.353999 7f869af78700 20 osd.26 243 _dispatch 0x1590f4a0 pg_query(3.450 epoch 242) v3 2014-06-09 16:22:11.353994 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] state: Activate Finished 2014-06-09 16:22:11.354004 7f869af78700 5 -- op tracker -- , seq: 1531, time: 2014-06-09 16:22:10.924112, event: header_read, op: pg_query(3.450 epoch 242) 2014-06-09 16:22:11.354011 7f869af78700 5 -- op tracker -- , seq: 1531, time: 2014-06-09 16:22:10.924113, event: throttled, op: pg_query(3.450 epoch 242) 2014-06-09 16:22:11.354006 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.354017 7f869af78700 5 -- op tracker -- , seq: 1531, time: 2014-06-09 16:22:10.924128, event: all_read, op: pg_query(3.450 epoch 242) 2014-06-09 16:22:11.354021 7f86a5610700 10 _do_op 0x300339e0 seq 3089 r = 0, finisher 0x1322c680 0 2014-06-09 16:22:11.354023 7f869af78700 5 -- op tracker -- , seq: 1531, time: 2014-06-09 16:22:11.353996, event: dispatched, op: pg_query(3.450 epoch 242) 2014-06-09 16:22:11.354029 7f869af78700 5 -- op tracker -- , seq: 1531, time: 2014-06-09 16:22:11.354027, event: waiting_for_osdmap, op: pg_query(3.450 epoch 242) 2014-06-09 16:22:11.354031 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x230c0e80] 2014-06-09 16:22:11.354033 7f869af78700 7 osd.26 243 handle_pg_query from osd.9 epoch 242 2014-06-09 16:22:11.354036 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x230c0e80] 2014-06-09 16:22:11.354036 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1590f4a0 2014-06-09 16:22:11.354037 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.354038 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.354041 7f86a5610700 10 _finish_op 0x300339e0 seq 3089 osr(1.38d 0x384d570)/0x384d570 2014-06-09 16:22:11.354041 7f869af78700 5 -- op tracker -- , seq: 1531, time: 2014-06-09 16:22:11.354039, event: started, op: pg_query(3.450 epoch 242) 2014-06-09 16:22:11.354043 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359560 (1 active) 2014-06-09 16:22:11.354043 7f868ef60700 10 write_log with: dirty_to: 0'0, dirty_from: 214'607, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.354046 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f440 (2 active) 2014-06-09 16:22:11.354047 7f86a5610700 5 _do_op 0x308cbc20 seq 3090 osr(3.4fa 0x3ed1720)/0x3ed1720 start 2014-06-09 16:22:11.354049 7f86a5610700 10 _do_transaction on 0x3859440 2014-06-09 16:22:11.354054 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.354053 7f869af78700 10 osd.26 pg_epoch: 243 pg[3.450( v 213'620 (0'0,213'620] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_query query(log 213'0 ec=118 les/c 213/213 242/242/212) from replica 9 2014-06-09 16:22:11.354074 7f869af78700 5 -- op tracker -- , seq: 1531, time: 2014-06-09 16:22:11.354072, event: done, op: pg_query(3.450 epoch 242) 2014-06-09 16:22:11.354082 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.354081 7f868ef60700 5 queue_transactions existing osr(3.c1 0x3ed05c0)/0x3ed05c0 2014-06-09 16:22:11.354084 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.354085 7f868ef60700 10 op_submit_start 3092 2014-06-09 16:22:11.354084 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.354086 7f868ef60700 5 queue_transactions (trailing journal) 3092 2014-06-09 16:22:11.354087 7f868ef60700 5 queue_op 0x288f9b60 seq 3092 osr(3.c1 0x3ed05c0) 1937 bytes (queue has 4 ops and 6198 bytes) 2014-06-09 16:22:11.354089 7f868ef60700 10 op_submit_finish 3092 2014-06-09 16:22:11.354096 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.354096 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x5552000 (0 active) 2014-06-09 16:22:11.354098 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.354099 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.354099 7f869af78700 20 osd.26 243 _dispatch 0x2d734600 pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.354095 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.354105 7f869af78700 5 -- op tracker -- , seq: 1532, time: 2014-06-09 16:22:11.013056, event: header_read, op: pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354109 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3090.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.354112 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.354106 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.67a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.354113 7f869af78700 5 -- op tracker -- , seq: 1532, time: 2014-06-09 16:22:11.013057, event: throttled, op: pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354117 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.354120 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.354120 7f869af78700 5 -- op tracker -- , seq: 1532, time: 2014-06-09 16:22:11.013071, event: all_read, op: pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354129 7f86a5610700 10 _do_op 0x308cbc20 seq 3090 r = 0, finisher 0x18742a00 0 2014-06-09 16:22:11.354127 7f869af78700 5 -- op tracker -- , seq: 1532, time: 2014-06-09 16:22:11.354096, event: dispatched, op: pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354123 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.081178 7 0.000182 2014-06-09 16:22:11.354135 7f869af78700 5 -- op tracker -- , seq: 1532, time: 2014-06-09 16:22:11.354132, event: waiting_for_osdmap, op: pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354138 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf1b20] 2014-06-09 16:22:11.354140 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf1b20] 2014-06-09 16:22:11.354134 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.354140 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.354143 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d734600 2014-06-09 16:22:11.354146 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.354147 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.354144 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.354150 7f86a5610700 10 _finish_op 0x308cbc20 seq 3090 osr(3.4fa 0x3ed1720)/0x3ed1720 2014-06-09 16:22:11.354149 7f869af78700 5 -- op tracker -- , seq: 1532, time: 2014-06-09 16:22:11.354146, event: started, op: pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354153 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f440 (1 active) 2014-06-09 16:22:11.354154 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2879e0 (2 active) 2014-06-09 16:22:11.354156 7f86a5610700 5 _do_op 0x308cbbc0 seq 3091 osr(1.92 0x384c410)/0x384c410 start 2014-06-09 16:22:11.354158 7f86a5610700 10 _do_transaction on 0xc2109c0 2014-06-09 16:22:11.354153 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.354163 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.354162 7f869af78700 5 -- op tracker -- , seq: 1532, time: 2014-06-09 16:22:11.354160, event: done, op: pg_log(0.4da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354161 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.354170 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.354172 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.354170 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.354182 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.354184 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.354185 7f869af78700 20 osd.26 243 _dispatch 0x240ebdc0 pg_notify(1.1b1(20) epoch 242) v5 2014-06-09 16:22:11.354189 7f869af78700 5 -- op tracker -- , seq: 1533, time: 2014-06-09 16:22:10.897877, event: header_read, op: pg_notify(1.1b1(20) epoch 242) 2014-06-09 16:22:11.354196 7f869af78700 5 -- op tracker -- , seq: 1533, time: 2014-06-09 16:22:10.897878, event: throttled, op: pg_notify(1.1b1(20) epoch 242) 2014-06-09 16:22:11.354202 7f869af78700 5 -- op tracker -- , seq: 1533, time: 2014-06-09 16:22:10.897896, event: all_read, op: pg_notify(1.1b1(20) epoch 242) 2014-06-09 16:22:11.354208 7f869af78700 5 -- op tracker -- , seq: 1533, time: 2014-06-09 16:22:11.354182, event: dispatched, op: pg_notify(1.1b1(20) epoch 242) 2014-06-09 16:22:11.354214 7f869af78700 5 -- op tracker -- , seq: 1533, time: 2014-06-09 16:22:11.354212, event: waiting_for_osdmap, op: pg_notify(1.1b1(20) epoch 242) 2014-06-09 16:22:11.354216 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.354218 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.354220 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240ebdc0 2014-06-09 16:22:11.354221 7f868ef60700 5 queue_transactions existing osr(1.67a 0x384de90)/0x384de90 2014-06-09 16:22:11.354224 7f868ef60700 10 op_submit_start 3093 2014-06-09 16:22:11.354225 7f868ef60700 5 queue_transactions (trailing journal) 3093 2014-06-09 16:22:11.354226 7f868ef60700 5 queue_op 0x309a2f00 seq 3093 osr(1.67a 0x384de90) 1883 bytes (queue has 4 ops and 6746 bytes) 2014-06-09 16:22:11.354224 7f869af78700 5 -- op tracker -- , seq: 1533, time: 2014-06-09 16:22:11.354222, event: started, op: pg_notify(1.1b1(20) epoch 242) 2014-06-09 16:22:11.354229 7f868ef60700 10 op_submit_finish 3093 2014-06-09 16:22:11.354232 7f868ef60700 20 osd.26 243 Sending info 3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) to shard 3 2014-06-09 16:22:11.354239 7f868ef60700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.354241 7f869af78700 5 -- op tracker -- , seq: 1533, time: 2014-06-09 16:22:11.354239, event: done, op: pg_notify(1.1b1(20) epoch 242) 2014-06-09 16:22:11.354248 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.354250 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.354259 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (15 active) 2014-06-09 16:22:11.354264 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354267 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.354268 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.354270 7f869af78700 20 osd.26 243 _dispatch 0x2a072000 pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.354276 7f869af78700 5 -- op tracker -- , seq: 1534, time: 2014-06-09 16:22:11.003912, event: header_read, op: pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354268 7f868ef60700 10 osd.26 243 dequeue_op 0xe032900 prio 63 cost 516096 latency 1.381759 osd_op(client.4566.0:33767 rbd_data.11766b8b4567.00000000000016aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~516096] 3.4c8eb750 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.354284 7f869af78700 5 -- op tracker -- , seq: 1534, time: 2014-06-09 16:22:11.003912, event: throttled, op: pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354293 7f869af78700 5 -- op tracker -- , seq: 1534, time: 2014-06-09 16:22:11.003922, event: all_read, op: pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354300 7f869af78700 5 -- op tracker -- , seq: 1534, time: 2014-06-09 16:22:11.354266, event: dispatched, op: pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354308 7f869af78700 5 -- op tracker -- , seq: 1534, time: 2014-06-09 16:22:11.354305, event: waiting_for_osdmap, op: pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.354312 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.354316 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a072000 2014-06-09 16:22:11.354325 7f868ef60700 5 -- op tracker -- , seq: 41, time: 2014-06-09 16:22:11.354284, event: reached_pg, op: osd_op(client.4566.0:33767 rbd_data.11766b8b4567.00000000000016aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2940928~516096] 3.4c8eb750 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354331 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.354336 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354348 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3091.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.354352 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.354349 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xe032900 2014-06-09 16:22:11.354358 7f868ef60700 10 osd.26 243 dequeue_op 0xe032900 finish 2014-06-09 16:22:11.354361 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354369 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (16 active) 2014-06-09 16:22:11.354376 7f86a5610700 10 _do_op 0x308cbbc0 seq 3091 r = 0, finisher 0x306cf140 0 2014-06-09 16:22:11.354372 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'630 (0'0,214'630] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MLogRec from 9 2014-06-09 16:22:11.354386 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2d739d40] 2014-06-09 16:22:11.354388 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2d739d40] 2014-06-09 16:22:11.354390 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.354391 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.354384 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'630 (0'0,214'630] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd9 2014-06-09 16:22:11.354393 7f86a5610700 10 _finish_op 0x308cbbc0 seq 3091 osr(1.92 0x384c410)/0x384c410 2014-06-09 16:22:11.354396 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2879e0 (1 active) 2014-06-09 16:22:11.354398 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3018c0 (2 active) 2014-06-09 16:22:11.354400 7f86a5610700 5 _do_op 0x288f9b60 seq 3092 osr(3.c1 0x3ed05c0)/0x3ed05c0 start 2014-06-09 16:22:11.354402 7f86a5610700 10 _do_transaction on 0xbf8ab40 2014-06-09 16:22:11.354395 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'630 (0'0,214'630] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.354407 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.354403 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'630 (0'0,214'630] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.354402 7f868af58700 10 osd.26 243 dequeue_op 0x1c66bd00 prio 63 cost 1032192 latency 0.841601 osd_op(client.4577.0:36987 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~1032192] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.354412 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'630 (0'0,214'630] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.9: log((214'630,214'632], crt=0'0) missing(0) 2014-06-09 16:22:11.354422 7f868ef60700 10 merge_log log((214'630,214'632], crt=0'0) from osd.9 into log((0'0,214'630], crt=0'0) 2014-06-09 16:22:11.354426 7f868ef60700 10 merge_log extending head to 214'632 2014-06-09 16:22:11.354425 7f868af58700 5 -- op tracker -- , seq: 490, time: 2014-06-09 16:22:11.354420, event: reached_pg, op: osd_op(client.4577.0:36987 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 860160~1032192] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354427 7f868ef60700 20 ? 214'632 (214'631) modify 86520ed3/rbd_data.116f6b8b4567.0000000000001817/head//3 by client.4559.0:35762 2014-06-09 10:19:07.540927 2014-06-09 16:22:11.354438 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.354435 7f868ef60700 20 ? 214'631 (214'630) modify 86520ed3/rbd_data.116f6b8b4567.0000000000001817/head//3 by client.4559.0:35761 2014-06-09 10:19:07.369223 2014-06-09 16:22:11.354432 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354441 7f868ef60700 20 merge_log 214'631 (214'630) modify 86520ed3/rbd_data.116f6b8b4567.0000000000001817/head//3 by client.4559.0:35761 2014-06-09 10:19:07.369223 2014-06-09 16:22:11.354446 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x1c66bd00 2014-06-09 16:22:11.354449 7f868ef60700 20 merge_log 214'632 (214'631) modify 86520ed3/rbd_data.116f6b8b4567.0000000000001817/head//3 by client.4559.0:35762 2014-06-09 10:19:07.540927 2014-06-09 16:22:11.354455 7f868af58700 10 osd.26 243 dequeue_op 0x1c66bd00 finish 2014-06-09 16:22:11.354459 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354464 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354468 7f868af58700 10 osd.26 243 dequeue_op 0x21967a00 prio 63 cost 2203648 latency 0.673870 osd_op(client.4567.0:35153 rbd_data.11776b8b4567.00000000000017b4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~2203648] 3.584859c7 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.354488 7f868af58700 5 -- op tracker -- , seq: 600, time: 2014-06-09 16:22:11.354483, event: reached_pg, op: osd_op(client.4567.0:35153 rbd_data.11776b8b4567.00000000000017b4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~2203648] 3.584859c7 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354494 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354505 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x21967a00 2014-06-09 16:22:11.354514 7f868af58700 10 osd.26 243 dequeue_op 0x21967a00 finish 2014-06-09 16:22:11.354517 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354522 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354525 7f868af58700 10 osd.26 243 dequeue_op 0x270be000 prio 63 cost 516096 latency 0.531848 osd_op(client.4573.0:34237 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 2514944~516096] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.354543 7f868af58700 5 -- op tracker -- , seq: 799, time: 2014-06-09 16:22:11.354539, event: reached_pg, op: osd_op(client.4573.0:34237 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 2514944~516096] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354550 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354560 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270be000 2014-06-09 16:22:11.354569 7f868af58700 10 osd.26 243 dequeue_op 0x270be000 finish 2014-06-09 16:22:11.354572 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354577 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354581 7f868af58700 10 osd.26 243 dequeue_op 0x14053000 prio 63 cost 65536 latency 1.104765 osd_op(client.4574.0:31685 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.354600 7f868af58700 5 -- op tracker -- , seq: 184, time: 2014-06-09 16:22:11.354596, event: reached_pg, op: osd_op(client.4574.0:31685 rbd_data.117e6b8b4567.00000000000014b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.3046a4d1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354606 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354617 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053000 2014-06-09 16:22:11.354626 7f868af58700 10 osd.26 243 dequeue_op 0x14053000 finish 2014-06-09 16:22:11.354628 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354633 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354637 7f868af58700 10 osd.26 243 dequeue_op 0x1804c100 prio 63 cost 516096 latency 1.032492 osd_op(client.4575.0:32140 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.354656 7f868ef60700 10 merge_log result log((0'0,214'632], crt=214'632) missing(1) changed=1 2014-06-09 16:22:11.354655 7f868af58700 5 -- op tracker -- , seq: 265, time: 2014-06-09 16:22:11.354651, event: reached_pg, op: osd_op(client.4575.0:32140 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354662 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354661 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 peering m=1] peer osd.9 now 3.6d3( v 214'632 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) missing(0) 2014-06-09 16:22:11.354672 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x1804c100 2014-06-09 16:22:11.354681 7f868af58700 10 osd.26 243 dequeue_op 0x1804c100 finish 2014-06-09 16:22:11.354684 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354679 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.437577 5 0.003820 2014-06-09 16:22:11.354689 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354692 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.354692 7f868af58700 10 osd.26 243 dequeue_op 0x1c66bc00 prio 63 cost 516096 latency 0.818618 osd_op(client.4577.0:36988 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 1892352~516096] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.354702 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 peering m=1] state: osd.9 has no missing, identical log 2014-06-09 16:22:11.354713 7f868af58700 5 -- op tracker -- , seq: 493, time: 2014-06-09 16:22:11.354708, event: reached_pg, op: osd_op(client.4577.0:36988 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 1892352~516096] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354714 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000022 0 0.000000 2014-06-09 16:22:11.354720 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354725 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.354732 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x1c66bc00 2014-06-09 16:22:11.354741 7f868af58700 10 osd.26 243 dequeue_op 0x1c66bc00 finish 2014-06-09 16:22:11.354735 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 0.649675 0 0.000000 2014-06-09 16:22:11.354744 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354749 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354748 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.354756 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.354760 7f8691f66700 5 -- op tracker -- , seq: 25, time: 2014-06-09 16:22:11.353195, event: reached_pg, op: osd_op(client.4565.0:32568 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 2514944~516096] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354767 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.354768 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354776 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.354779 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xe032f00 2014-06-09 16:22:11.354788 7f8691f66700 10 osd.26 243 dequeue_op 0xe032f00 finish 2014-06-09 16:22:11.354785 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'630 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.354793 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354799 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354802 7f8691f66700 10 osd.26 243 dequeue_op 0xe2ad000 prio 63 cost 737280 latency 1.343635 osd_op(client.4566.0:33768 rbd_data.11766b8b4567.00000000000016aa [set-alloc-hint object_size 4194304 write_size 4194304,write 3457024~737280] 3.4c8eb750 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.354821 7f8691f66700 5 -- op tracker -- , seq: 46, time: 2014-06-09 16:22:11.354817, event: reached_pg, op: osd_op(client.4566.0:33768 rbd_data.11766b8b4567.00000000000016aa [set-alloc-hint object_size 4194304 write_size 4194304,write 3457024~737280] 3.4c8eb750 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354828 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354839 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xe2ad000 2014-06-09 16:22:11.354847 7f8691f66700 10 osd.26 243 dequeue_op 0xe2ad000 finish 2014-06-09 16:22:11.354850 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354856 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354860 7f8691f66700 10 osd.26 243 dequeue_op 0x21967700 prio 63 cost 1032192 latency 0.631265 osd_op(client.4567.0:35177 rbd_data.11776b8b4567.00000000000017b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1146880~1032192] 3.6498b9f4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.354879 7f8691f66700 5 -- op tracker -- , seq: 641, time: 2014-06-09 16:22:11.354875, event: reached_pg, op: osd_op(client.4567.0:35177 rbd_data.11776b8b4567.00000000000017b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1146880~1032192] 3.6498b9f4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354886 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354897 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x21967700 2014-06-09 16:22:11.354905 7f8691f66700 10 osd.26 243 dequeue_op 0x21967700 finish 2014-06-09 16:22:11.354908 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354904 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.354913 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354917 7f868ef60700 10 osd.26 243 queue_for_recovery queued pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.354917 7f8691f66700 10 osd.26 243 dequeue_op 0x29f5bf00 prio 63 cost 516096 latency 0.531335 osd_op(client.4573.0:34238 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 3031040~516096] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.354935 7f8691f66700 5 -- op tracker -- , seq: 803, time: 2014-06-09 16:22:11.354930, event: reached_pg, op: osd_op(client.4573.0:34238 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 3031040~516096] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354930 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 3.6d3( v 214'632 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) 2014-06-09 16:22:11.354941 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 is up to date, queueing in pending_activators 2014-06-09 16:22:11.354941 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.354952 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x29f5bf00 2014-06-09 16:22:11.354961 7f8691f66700 10 osd.26 243 dequeue_op 0x29f5bf00 finish 2014-06-09 16:22:11.354954 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 3.6d3( v 214'632 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242) uptodate 2014-06-09 16:22:11.354964 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.354970 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.354973 7f86a5610700 15 _omap_rmkeyrange meta/516ef296/pglog_3.c1/0//-1 [0000000214.00000000000000000607,4294967295.18446744073709551615] 2014-06-09 16:22:11.354967 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.354976 7f86a5610700 15 get_omap_iterator meta/516ef296/pglog_3.c1/0//-1 2014-06-09 16:22:11.354977 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 86520ed3/rbd_data.116f6b8b4567.0000000000001817/head//3 214'632 also missing on osd.26 2014-06-09 16:22:11.354974 7f8691f66700 10 osd.26 243 dequeue_op 0x1804c000 prio 63 cost 516096 latency 1.031871 osd_op(client.4575.0:32141 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.354994 7f8691f66700 5 -- op tracker -- , seq: 280, time: 2014-06-09 16:22:11.354989, event: reached_pg, op: osd_op(client.4575.0:32141 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.354988 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {86520ed3/rbd_data.116f6b8b4567.0000000000001817/head//3=214'632(214'630)} 2014-06-09 16:22:11.354999 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 9 as a source 2014-06-09 16:22:11.355000 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.355011 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x1804c000 2014-06-09 16:22:11.355009 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 86520ed3/rbd_data.116f6b8b4567.0000000000001817/head//3 214'632 is on osd.9 2014-06-09 16:22:11.355020 7f8691f66700 10 osd.26 243 dequeue_op 0x1804c000 finish 2014-06-09 16:22:11.355023 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.355028 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.355022 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {86520ed3/rbd_data.116f6b8b4567.0000000000001817/head//3=214'632(214'630)} 2014-06-09 16:22:11.355034 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.355032 7f8691f66700 10 osd.26 243 dequeue_op 0x1c66bb00 prio 63 cost 516096 latency 0.815313 osd_op(client.4577.0:36989 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.355043 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 213 2014-06-09 16:22:11.355051 7f8691f66700 5 -- op tracker -- , seq: 513, time: 2014-06-09 16:22:11.355046, event: reached_pg, op: osd_op(client.4577.0:36989 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 2408448~516096] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.355058 7f86a5610700 15 _omap_rmkeys meta/516ef296/pglog_3.c1/0//-1 2014-06-09 16:22:11.355054 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 9 2014-06-09 16:22:11.355057 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.355064 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.355067 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x1c66bb00 2014-06-09 16:22:11.355076 7f8691f66700 10 osd.26 243 dequeue_op 0x1c66bb00 finish 2014-06-09 16:22:11.355079 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.355076 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.355084 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.355088 7f86a5610700 10 lookup_strip_header done cid meta oid 516ef296/pglog_3.c1/0//-1 2014-06-09 16:22:11.355088 7f8691f66700 10 osd.26 243 dequeue_op 0x21967600 prio 63 cost 2015232 latency 0.610607 osd_op(client.4567.0:35178 rbd_data.11776b8b4567.00000000000017b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2179072~2015232] 3.6498b9f4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.355110 7f868ef60700 10 write_log with: dirty_to: 0'0, dirty_from: 214'630, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.362944 7f868ef60700 5 queue_transactions existing osr(3.6d3 0x3ed1c50)/0x3ed1c50 2014-06-09 16:22:11.362951 7f868ef60700 10 op_submit_start 3094 2014-06-09 16:22:11.362953 7f868ef60700 5 queue_transactions (trailing journal) 3094 2014-06-09 16:22:11.362955 7f868ef60700 5 queue_op 0x16096720 seq 3094 osr(3.6d3 0x3ed1c50) 1943 bytes (queue has 4 ops and 6816 bytes) 2014-06-09 16:22:11.362958 7f868ef60700 10 op_submit_finish 3094 2014-06-09 16:22:11.362962 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:3.428( v 214'563 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.362981 7f868ef60700 7 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.6 2014-06-09 16:22:11.362993 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.6 info 3.428( v 214'563 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242), identical to ours 2014-06-09 16:22:11.363014 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.363019 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.363031 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.73( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.363043 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.363047 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.363052 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.73( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.127047 7 0.004472 2014-06-09 16:22:11.363065 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.73( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.363077 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.73( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.363087 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.363097 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.363107 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.363158 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.363164 7f868ef60700 5 queue_transactions existing osr(2.73 0x2ed0510)/0x2ed0510 2014-06-09 16:22:11.363167 7f868ef60700 10 op_submit_start 3095 2014-06-09 16:22:11.363168 7f868ef60700 5 queue_transactions (trailing journal) 3095 2014-06-09 16:22:11.363169 7f868ef60700 5 queue_op 0x309a2e40 seq 3095 osr(2.73 0x2ed0510) 1880 bytes (queue has 5 ops and 8696 bytes) 2014-06-09 16:22:11.363172 7f868ef60700 10 op_submit_finish 3095 2014-06-09 16:22:11.363175 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.363187 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 1.82( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.363199 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.363202 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.363206 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.82( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.141711 7 0.000178 2014-06-09 16:22:11.363216 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.82( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.363227 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.82( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.363237 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.363246 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.363255 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.363304 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.363309 7f868ef60700 5 queue_transactions existing osr(1.82 0x384c4d0)/0x384c4d0 2014-06-09 16:22:11.363311 7f868ef60700 10 op_submit_start 3096 2014-06-09 16:22:11.363313 7f868ef60700 5 queue_transactions (trailing journal) 3096 2014-06-09 16:22:11.363314 7f868ef60700 5 queue_op 0x309a2de0 seq 3096 osr(1.82 0x384c4d0) 2006 bytes (queue has 6 ops and 10702 bytes) 2014-06-09 16:22:11.363316 7f868ef60700 10 op_submit_finish 3096 2014-06-09 16:22:11.363319 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:1.5e1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.363333 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.363344 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.363349 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:2.186( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.363361 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.363370 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.363375 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.363385 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 1.631( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.363397 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.363400 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.363404 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.631( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 1.120440 7 0.000187 2014-06-09 16:22:11.363418 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.631( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.363428 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.631( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.363438 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.363447 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.363455 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.363506 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.363511 7f868ef60700 5 queue_transactions existing osr(1.631 0x384dfb0)/0x384dfb0 2014-06-09 16:22:11.363513 7f868ef60700 10 op_submit_start 3097 2014-06-09 16:22:11.363514 7f868ef60700 5 queue_transactions (trailing journal) 3097 2014-06-09 16:22:11.363516 7f868ef60700 5 queue_op 0x309a2d80 seq 3097 osr(1.631 0x384dfb0) 2099 bytes (queue has 7 ops and 12801 bytes) 2014-06-09 16:22:11.363519 7f868ef60700 10 op_submit_finish 3097 2014-06-09 16:22:11.363522 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.308( v 213'540 (0'0,213'540] local-les=191 n=105 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.363536 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.363551 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.363556 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.7d4( v 213'526 (0'0,213'526] local-les=191 n=106 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.363569 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.363580 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.363584 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:1.327( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.363596 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.363606 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.355115 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0xab0e000 (1 active) 2014-06-09 16:22:11.355122 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (17 active) 2014-06-09 16:22:11.355105 7f868af58700 10 osd.26 243 dequeue_op 0xe032e00 prio 63 cost 1163264 latency 1.381379 osd_op(client.4565.0:32569 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 3031040~1163264] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.363841 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.363871 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0xab0e000 (0 active) 2014-06-09 16:22:11.363852 7f868a757700 10 osd.26 243 dequeue_op 0x29f5be00 prio 63 cost 516096 latency 0.539159 osd_op(client.4573.0:34239 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 3547136~516096] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.363874 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.355127 7f869af78700 5 -- op tracker -- , seq: 1534, time: 2014-06-09 16:22:11.354319, event: started, op: pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.347514 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.363610 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.450( v 213'620 (0'0,213'620] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MQuery from 9 query_epoch 242 query: query(log 213'0 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.363892 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.363890 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.450( v 213'620 (0'0,213'620] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] log request from 9 2014-06-09 16:22:11.363897 7f869af78700 5 -- op tracker -- , seq: 1534, time: 2014-06-09 16:22:11.363893, event: done, op: pg_log(1.604 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.363899 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.450( v 213'620 (0'0,213'620] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] sending info+missing+log since 213'0 2014-06-09 16:22:11.363898 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.363911 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.363914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.363911 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.363920 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.363931 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.363924 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.363932 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.363936 7f869af78700 20 osd.26 243 _dispatch 0x1a039dc0 pg_notify(3.319(10),2.5e(23) epoch 243) v5 2014-06-09 16:22:11.363939 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.363945 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02f0 (18 active) 2014-06-09 16:22:11.363944 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.363951 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.363954 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.363964 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.363963 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.604( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.363975 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.363968 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.363979 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.363983 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.363982 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.604( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.091279 7 0.006632 2014-06-09 16:22:11.363987 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.363993 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.604( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.363997 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.364006 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.364005 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.604( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.364008 7f8691f66700 5 -- op tracker -- , seq: 655, time: 2014-06-09 16:22:11.362893, event: reached_pg, op: osd_op(client.4567.0:35178 rbd_data.11776b8b4567.00000000000017b9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2179072~2015232] 3.6498b9f4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364015 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.364009 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.364027 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.364019 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364024 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.364032 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.364031 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.364044 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x1b0b6100,0x19fe1b00,0x1b0b6000,0x19fe1a00,0x1b0b7a00,0x19fe1900,0x19fe1800,0x19fe1700,0x19fe1600,0x1c769c00,0x1b0b7900 2014-06-09 16:22:11.364070 7f868af58700 5 -- op tracker -- , seq: 47, time: 2014-06-09 16:22:11.363851, event: reached_pg, op: osd_op(client.4565.0:32569 rbd_data.11756b8b4567.0000000000001565 [set-alloc-hint object_size 4194304 write_size 4194304,write 3031040~1163264] 3.8b1e77c RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364084 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.364080 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364088 7f868a757700 5 -- op tracker -- , seq: 811, time: 2014-06-09 16:22:11.363875, event: reached_pg, op: osd_op(client.4573.0:34239 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 3547136~516096] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364095 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (17 active) 2014-06-09 16:22:11.364093 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xe032e00 2014-06-09 16:22:11.364101 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (18 active) 2014-06-09 16:22:11.364102 7f868af58700 10 osd.26 243 dequeue_op 0xe032e00 finish 2014-06-09 16:22:11.364106 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (17 active) 2014-06-09 16:22:11.364099 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364111 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (18 active) 2014-06-09 16:22:11.364104 7f8690763700 10 osd.26 243 dequeue_op 0x19fe1b00 prio 63 cost 294912 latency 0.955083 osd_op(client.4575.0:32282 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.364113 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x29f5be00 2014-06-09 16:22:11.364122 7f868a757700 10 osd.26 243 dequeue_op 0x29f5be00 finish 2014-06-09 16:22:11.364126 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (17 active) 2014-06-09 16:22:11.364115 7f868af58700 10 osd.26 243 dequeue_op 0x21966400 prio 63 cost 516096 latency 0.800986 osd_op(client.4577.0:36990 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.364125 7f8690763700 5 -- op tracker -- , seq: 305, time: 2014-06-09 16:22:11.364121, event: reached_pg, op: osd_op(client.4575.0:32282 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~294912] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364132 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (18 active) 2014-06-09 16:22:11.364139 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.364132 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364144 7f868b759700 5 queue_transactions existing osr(1.604 0x384dc50)/0x384dc50 2014-06-09 16:22:11.364147 7f868b759700 10 op_submit_start 3098 2014-06-09 16:22:11.364148 7f868b759700 5 queue_transactions (trailing journal) 3098 2014-06-09 16:22:11.364149 7f868b759700 5 queue_op 0x308cac00 seq 3098 osr(1.604 0x384dc50) 1883 bytes (queue has 8 ops and 14684 bytes) 2014-06-09 16:22:11.364152 7f868b759700 10 op_submit_finish 3098 2014-06-09 16:22:11.364146 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x19fe1b00 2014-06-09 16:22:11.364156 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02f0 (17 active) 2014-06-09 16:22:11.364157 7f8690763700 10 osd.26 243 dequeue_op 0x19fe1b00 finish 2014-06-09 16:22:11.364033 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x21967600 2014-06-09 16:22:11.364161 7f8691f66700 10 osd.26 243 dequeue_op 0x21967600 finish 2014-06-09 16:22:11.355168 7f868cf5c700 20 ? 214'570 (214'568) modify 87496894/rbd_data.117f6b8b4567.000000000000151c/head//3 by client.4575.0:32102 2014-06-09 10:18:55.580206 2014-06-09 16:22:11.364166 7f868cf5c700 20 ? 214'569 (214'567) modify 87738094/rbd_data.11696b8b4567.0000000000001690/head//3 by client.4553.0:33703 2014-06-09 10:19:09.923751 2014-06-09 16:22:11.364175 7f868cf5c700 20 ? 214'568 (214'566) modify 87496894/rbd_data.117f6b8b4567.000000000000151c/head//3 by client.4575.0:32101 2014-06-09 10:18:55.569039 2014-06-09 16:22:11.364165 7f868a757700 10 osd.26 243 dequeue_op 0x1b0b6100 prio 63 cost 679936 latency 0.874655 osd_op(client.4572.0:30512 rbd_data.117c6b8b4567.0000000000001463 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.1045021b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.364181 7f868cf5c700 20 merge_log 214'568 (214'566) modify 87496894/rbd_data.117f6b8b4567.000000000000151c/head//3 by client.4575.0:32101 2014-06-09 10:18:55.569039 2014-06-09 16:22:11.364161 7f868b759700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (18 active) 2014-06-09 16:22:11.364191 7f868cf5c700 20 merge_log 214'569 (214'567) modify 87738094/rbd_data.11696b8b4567.0000000000001690/head//3 by client.4553.0:33703 2014-06-09 10:19:09.923751 2014-06-09 16:22:11.364191 7f868b759700 10 osd.26 243 dequeue_op 0x4679f00 prio 63 cost 131072 latency 0.536482 osd_op(client.4573.0:34240 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 4063232~131072] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.364201 7f868cf5c700 20 merge_log 214'570 (214'568) modify 87496894/rbd_data.117f6b8b4567.000000000000151c/head//3 by client.4575.0:32102 2014-06-09 10:18:55.580206 2014-06-09 16:22:11.364213 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.364210 7f868cf5c700 20 merge_log 214'571 (214'570) modify 87496894/rbd_data.117f6b8b4567.000000000000151c/head//3 by client.4575.0:32103 2014-06-09 10:18:55.620813 2014-06-09 16:22:11.364169 7f868af58700 5 -- op tracker -- , seq: 518, time: 2014-06-09 16:22:11.364131, event: reached_pg, op: osd_op(client.4577.0:36990 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 2924544~516096] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364225 7f869af78700 5 -- op tracker -- , seq: 1535, time: 2014-06-09 16:22:11.032732, event: header_read, op: pg_notify(3.319(10),2.5e(23) epoch 243) 2014-06-09 16:22:11.364220 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364234 7f869af78700 5 -- op tracker -- , seq: 1535, time: 2014-06-09 16:22:11.032732, event: throttled, op: pg_notify(3.319(10),2.5e(23) epoch 243) 2014-06-09 16:22:11.364233 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x21966400 2014-06-09 16:22:11.364241 7f868af58700 10 osd.26 243 dequeue_op 0x21966400 finish 2014-06-09 16:22:11.364244 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.364241 7f869af78700 5 -- op tracker -- , seq: 1535, time: 2014-06-09 16:22:11.032775, event: all_read, op: pg_notify(3.319(10),2.5e(23) epoch 243) 2014-06-09 16:22:11.364248 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.364248 7f869af78700 5 -- op tracker -- , seq: 1535, time: 2014-06-09 16:22:11.363930, event: dispatched, op: pg_notify(3.319(10),2.5e(23) epoch 243) 2014-06-09 16:22:11.364256 7f869af78700 5 -- op tracker -- , seq: 1535, time: 2014-06-09 16:22:11.364253, event: waiting_for_osdmap, op: pg_notify(3.319(10),2.5e(23) epoch 243) 2014-06-09 16:22:11.364261 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.364252 7f868af58700 10 osd.26 243 dequeue_op 0x21966300 prio 63 cost 753664 latency 0.796365 osd_op(client.4577.0:36991 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~753664] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.364263 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a039dc0 2014-06-09 16:22:11.364270 7f868af58700 5 -- op tracker -- , seq: 526, time: 2014-06-09 16:22:11.364266, event: reached_pg, op: osd_op(client.4577.0:36991 rbd_data.11816b8b4567.00000000000018ee [set-alloc-hint object_size 4194304 write_size 4194304,write 3440640~753664] 3.f0e71aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364277 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364285 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.364278 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364289 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364289 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x21966300 2014-06-09 16:22:11.364297 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.364298 7f868af58700 10 osd.26 243 dequeue_op 0x21966300 finish 2014-06-09 16:22:11.364305 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364313 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.364318 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364328 7f868b759700 5 -- op tracker -- , seq: 817, time: 2014-06-09 16:22:11.364207, event: reached_pg, op: osd_op(client.4573.0:34240 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 4063232~131072] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364337 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364350 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x4679f00 2014-06-09 16:22:11.364359 7f868b759700 10 osd.26 243 dequeue_op 0x4679f00 finish 2014-06-09 16:22:11.364366 7f868af58700 10 osd.26 243 dequeue_op 0x22009b00 prio 63 cost 1548288 latency 0.587330 osd_op(client.4577.0:37200 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 614400~1548288] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.364371 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.450( v 213'620 (0'0,213'620] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] sending log((0'0,213'620], crt=0'0) missing(0) 2014-06-09 16:22:11.364384 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.364387 7f868af58700 5 -- op tracker -- , seq: 714, time: 2014-06-09 16:22:11.364382, event: reached_pg, op: osd_op(client.4577.0:37200 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 614400~1548288] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364400 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.364394 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364404 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.364406 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x22009b00 2014-06-09 16:22:11.364415 7f868af58700 10 osd.26 243 dequeue_op 0x22009b00 finish 2014-06-09 16:22:11.364419 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.364421 7f868cf5c700 10 merge_log result log((0'0,214'571], crt=214'571) missing(2) changed=1 2014-06-09 16:22:11.364424 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364414 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.4da( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.364428 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.364431 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.364431 7f869af78700 5 -- op tracker -- , seq: 1535, time: 2014-06-09 16:22:11.364267, event: started, op: pg_notify(3.319(10),2.5e(23) epoch 243) 2014-06-09 16:22:11.364426 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 peering m=2] peer osd.9 now 3.94( v 214'571 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) missing(0) 2014-06-09 16:22:11.364434 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive NOTIFY] exit Started/Stray 1.207166 7 0.000202 2014-06-09 16:22:11.364443 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.364446 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 peering m=2] exit Started/Primary/Peering/GetLog 0.467211 5 0.000232 2014-06-09 16:22:11.364452 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.364460 7f869af78700 5 -- op tracker -- , seq: 1535, time: 2014-06-09 16:22:11.364458, event: done, op: pg_notify(3.319(10),2.5e(23) epoch 243) 2014-06-09 16:22:11.364461 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.364459 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 peering m=2] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.364469 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.364471 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.364469 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.364469 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 peering m=2] state: osd.9 has no missing, identical log 2014-06-09 16:22:11.364476 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.364482 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 peering m=2] exit Started/Primary/Peering/GetMissing 0.000023 0 0.000000 2014-06-09 16:22:11.364498 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.364499 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.364493 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 peering m=2] state: Leaving Peering 2014-06-09 16:22:11.364501 7f869af78700 20 osd.26 243 _dispatch 0x1c039180 pg_notify(1.591(26) epoch 242) v5 2014-06-09 16:22:11.364506 7f869af78700 5 -- op tracker -- , seq: 1536, time: 2014-06-09 16:22:10.837590, event: header_read, op: pg_notify(1.591(26) epoch 242) 2014-06-09 16:22:11.364503 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 peering m=2] exit Started/Primary/Peering 1.085506 0 0.000000 2014-06-09 16:22:11.364516 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.364513 7f869af78700 5 -- op tracker -- , seq: 1536, time: 2014-06-09 16:22:10.837591, event: throttled, op: pg_notify(1.591(26) epoch 242) 2014-06-09 16:22:11.364520 7f868ef60700 5 queue_transactions existing osr(0.4da 0x2bd75f0)/0x2bd75f0 2014-06-09 16:22:11.364522 7f868ef60700 10 op_submit_start 3099 2014-06-09 16:22:11.364520 7f869af78700 5 -- op tracker -- , seq: 1536, time: 2014-06-09 16:22:10.837611, event: all_read, op: pg_notify(1.591(26) epoch 242) 2014-06-09 16:22:11.364523 7f868ef60700 5 queue_transactions (trailing journal) 3099 2014-06-09 16:22:11.364516 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] enter Started/Primary/Active 2014-06-09 16:22:11.364524 7f868ef60700 5 queue_op 0x309a2d20 seq 3099 osr(0.4da 0x2bd75f0) 1621 bytes (queue has 9 ops and 16305 bytes) 2014-06-09 16:22:11.364527 7f868ef60700 10 op_submit_finish 3099 2014-06-09 16:22:11.364526 7f869af78700 5 -- op tracker -- , seq: 1536, time: 2014-06-09 16:22:11.364497, event: dispatched, op: pg_notify(1.591(26) epoch 242) 2014-06-09 16:22:11.364526 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] state: In Active, about to call activate 2014-06-09 16:22:11.364533 7f869af78700 5 -- op tracker -- , seq: 1536, time: 2014-06-09 16:22:11.364531, event: waiting_for_osdmap, op: pg_notify(1.591(26) epoch 242) 2014-06-09 16:22:11.364538 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.364540 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1c039180 2014-06-09 16:22:11.364538 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.364545 7f869af78700 5 -- op tracker -- , seq: 1536, time: 2014-06-09 16:22:11.364543, event: started, op: pg_notify(1.591(26) epoch 242) 2014-06-09 16:22:11.364548 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] activate - snap_trimq [] 2014-06-09 16:22:11.364560 7f869af78700 5 -- op tracker -- , seq: 1536, time: 2014-06-09 16:22:11.364558, event: done, op: pg_notify(1.591(26) epoch 242) 2014-06-09 16:22:11.364530 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:1.1b1( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.364558 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'567 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] activate - not complete, missing(2) 2014-06-09 16:22:11.364573 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.364575 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.364569 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.364579 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.364583 7f868ef60700 20 osd.26 243 Sending info 3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) to shard 9 2014-06-09 16:22:11.364589 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.364593 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.364595 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.364596 7f869af78700 20 osd.26 243 _dispatch 0x280c5a40 pg_notify(1.238(20) epoch 242) v5 2014-06-09 16:22:11.364601 7f869af78700 5 -- op tracker -- , seq: 1537, time: 2014-06-09 16:22:10.770443, event: header_read, op: pg_notify(1.238(20) epoch 242) 2014-06-09 16:22:11.364607 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (19 active) 2014-06-09 16:22:11.364608 7f869af78700 5 -- op tracker -- , seq: 1537, time: 2014-06-09 16:22:10.770443, event: throttled, op: pg_notify(1.238(20) epoch 242) 2014-06-09 16:22:11.364610 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (20 active) 2014-06-09 16:22:11.364613 7f869af78700 5 -- op tracker -- , seq: 1537, time: 2014-06-09 16:22:10.770470, event: all_read, op: pg_notify(1.238(20) epoch 242) 2014-06-09 16:22:11.364620 7f869af78700 5 -- op tracker -- , seq: 1537, time: 2014-06-09 16:22:11.364593, event: dispatched, op: pg_notify(1.238(20) epoch 242) 2014-06-09 16:22:11.364613 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.319( v 214'607 (0'0,214'607] local-les=206 n=114 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.364626 7f869af78700 5 -- op tracker -- , seq: 1537, time: 2014-06-09 16:22:11.364624, event: waiting_for_osdmap, op: pg_notify(1.238(20) epoch 242) 2014-06-09 16:22:11.364630 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.364632 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x280c5a40 2014-06-09 16:22:11.364625 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.364635 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.364637 7f869af78700 5 -- op tracker -- , seq: 1537, time: 2014-06-09 16:22:11.364635, event: started, op: pg_notify(1.238(20) epoch 242) 2014-06-09 16:22:11.364638 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.5e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.364652 7f869af78700 5 -- op tracker -- , seq: 1537, time: 2014-06-09 16:22:11.364650, event: done, op: pg_notify(1.238(20) epoch 242) 2014-06-09 16:22:11.364660 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.364662 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.364648 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.364681 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.364681 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.364683 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.364684 7f869af78700 20 osd.26 243 _dispatch 0x3046ba40 pg_info(2 pgs e243:3.62a(2),3.9b(2)) v4 2014-06-09 16:22:11.364679 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] activate - starting recovery 2014-06-09 16:22:11.364690 7f869af78700 5 -- op tracker -- , seq: 1538, time: 2014-06-09 16:22:11.061501, event: header_read, op: pg_info(2 pgs e243:3.62a(2),3.9b(2)) 2014-06-09 16:22:11.364697 7f869af78700 5 -- op tracker -- , seq: 1538, time: 2014-06-09 16:22:11.061501, event: throttled, op: pg_info(2 pgs e243:3.62a(2),3.9b(2)) 2014-06-09 16:22:11.364694 7f868cf5c700 10 osd.26 243 queue_for_recovery queued pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.364704 7f869af78700 5 -- op tracker -- , seq: 1538, time: 2014-06-09 16:22:11.061519, event: all_read, op: pg_info(2 pgs e243:3.62a(2),3.9b(2)) 2014-06-09 16:22:11.364711 7f869af78700 5 -- op tracker -- , seq: 1538, time: 2014-06-09 16:22:11.364681, event: dispatched, op: pg_info(2 pgs e243:3.62a(2),3.9b(2)) 2014-06-09 16:22:11.364708 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] activate peer osd.9 3.94( v 214'571 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) 2014-06-09 16:22:11.364718 7f869af78700 5 -- op tracker -- , seq: 1538, time: 2014-06-09 16:22:11.364715, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.62a(2),3.9b(2)) 2014-06-09 16:22:11.364716 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:1.591( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.364722 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.62a(2),3.9b(2)) v4 from osd.7 2014-06-09 16:22:11.364726 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046ba40 2014-06-09 16:22:11.364721 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] activate peer osd.9 is up to date, queueing in pending_activators 2014-06-09 16:22:11.364727 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.364732 7f869af78700 5 -- op tracker -- , seq: 1538, time: 2014-06-09 16:22:11.364729, event: started, op: pg_info(2 pgs e243:3.62a(2),3.9b(2)) 2014-06-09 16:22:11.364736 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.364741 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (19 active) 2014-06-09 16:22:11.364736 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] activate peer osd.9 3.94( v 214'571 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242) uptodate 2014-06-09 16:22:11.364747 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364755 7f868a757700 5 -- op tracker -- , seq: 444, time: 2014-06-09 16:22:11.364183, event: reached_pg, op: osd_op(client.4572.0:30512 rbd_data.117c6b8b4567.0000000000001463 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~679936] 3.1045021b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364750 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2 u=2] needs_recovery primary has 2 missing 2014-06-09 16:22:11.364763 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2 u=2] search_for_missing 87738094/rbd_data.11696b8b4567.0000000000001690/head//3 214'569 also missing on osd.26 2014-06-09 16:22:11.364762 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364775 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2 u=2] search_for_missing 87496894/rbd_data.117f6b8b4567.000000000000151c/head//3 214'571 also missing on osd.26 2014-06-09 16:22:11.364778 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x1b0b6100 2014-06-09 16:22:11.364789 7f868a757700 10 osd.26 243 dequeue_op 0x1b0b6100 finish 2014-06-09 16:22:11.364785 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2 u=2] needs_recovery_map missing {87738094/rbd_data.11696b8b4567.0000000000001690/head//3=214'569(214'567),87496894/rbd_data.117f6b8b4567.000000000000151c/head//3=214'571(214'566)} 2014-06-09 16:22:11.364795 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.364797 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (20 active) 2014-06-09 16:22:11.364793 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2 u=2] activate: adding 9 as a source 2014-06-09 16:22:11.364801 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:1.238( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.364804 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2 u=2] search_for_missing 87738094/rbd_data.11696b8b4567.0000000000001690/head//3 214'569 is on osd.9 2014-06-09 16:22:11.364813 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.364823 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.364818 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2 u=1] search_for_missing 87496894/rbd_data.117f6b8b4567.000000000000151c/head//3 214'571 is on osd.9 2014-06-09 16:22:11.364830 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (19 active) 2014-06-09 16:22:11.364828 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] needs_recovery_map missing {87738094/rbd_data.11696b8b4567.0000000000001690/head//3=214'569(214'567),87496894/rbd_data.117f6b8b4567.000000000000151c/head//3=214'571(214'566)} 2014-06-09 16:22:11.364835 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364837 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] build_might_have_unfound 2014-06-09 16:22:11.364843 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] _calc_past_interval_range: already have past intervals back to 213 2014-06-09 16:22:11.364839 7f868a757700 10 osd.26 243 dequeue_op 0x22009e00 prio 63 cost 516096 latency 0.579817 osd_op(client.4577.0:37201 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.364860 7f868a757700 5 -- op tracker -- , seq: 725, time: 2014-06-09 16:22:11.364855, event: reached_pg, op: osd_op(client.4577.0:37201 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~516096] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364854 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] build_might_have_unfound: built 9 2014-06-09 16:22:11.364867 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] state: Activate Finished 2014-06-09 16:22:11.364867 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364876 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.364879 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x22009e00 2014-06-09 16:22:11.364889 7f868a757700 10 osd.26 243 dequeue_op 0x22009e00 finish 2014-06-09 16:22:11.364892 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.364897 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364899 7f868cf5c700 10 write_log with: dirty_to: 0'0, dirty_from: 214'567, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.364900 7f868a757700 10 osd.26 243 dequeue_op 0x19fe1a00 prio 63 cost 1032192 latency 0.952831 osd_op(client.4575.0:32283 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~1032192] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.364922 7f868a757700 5 -- op tracker -- , seq: 307, time: 2014-06-09 16:22:11.364916, event: reached_pg, op: osd_op(client.4575.0:32283 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 294912~1032192] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364937 7f868cf5c700 5 queue_transactions existing osr(3.94 0x3ed0640)/0x3ed0640 2014-06-09 16:22:11.364929 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.364941 7f868cf5c700 10 op_submit_start 3100 2014-06-09 16:22:11.364942 7f868cf5c700 5 queue_transactions (trailing journal) 3100 2014-06-09 16:22:11.364943 7f868cf5c700 5 queue_op 0x16fd5500 seq 3100 osr(3.94 0x3ed0640) 2371 bytes (queue has 10 ops and 18676 bytes) 2014-06-09 16:22:11.364946 7f868cf5c700 10 op_submit_finish 3100 2014-06-09 16:22:11.364942 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x19fe1a00 2014-06-09 16:22:11.364952 7f868a757700 10 osd.26 243 dequeue_op 0x19fe1a00 finish 2014-06-09 16:22:11.364949 7f868cf5c700 20 osd.26 243 Sending info 3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) to shard 9 2014-06-09 16:22:11.364956 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.364956 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.364961 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364971 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (19 active) 2014-06-09 16:22:11.364977 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.364964 7f868a757700 10 osd.26 243 dequeue_op 0x1b0b6000 prio 63 cost 1548288 latency 0.860587 osd_op(client.4572.0:30513 rbd_data.117c6b8b4567.0000000000001463 [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~1548288] 3.1045021b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.364985 7f868a757700 5 -- op tracker -- , seq: 480, time: 2014-06-09 16:22:11.364980, event: reached_pg, op: osd_op(client.4572.0:30513 rbd_data.117c6b8b4567.0000000000001463 [set-alloc-hint object_size 4194304 write_size 4194304,write 679936~1548288] 3.1045021b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.364982 7f868cf5c700 10 osd.26 243 dequeue_op 0x257c4400 prio 63 cost 516096 latency 0.571498 osd_op(client.4577.0:37202 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~516096] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.364992 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365002 7f868cf5c700 5 -- op tracker -- , seq: 736, time: 2014-06-09 16:22:11.364999, event: reached_pg, op: osd_op(client.4577.0:37202 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 2678784~516096] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365008 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365005 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x1b0b6000 2014-06-09 16:22:11.365016 7f868a757700 10 osd.26 243 dequeue_op 0x1b0b6000 finish 2014-06-09 16:22:11.365016 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x257c4400 2014-06-09 16:22:11.365020 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.365021 7f868cf5c700 10 osd.26 243 dequeue_op 0x257c4400 finish 2014-06-09 16:22:11.365028 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x52d2000 (1 active) 2014-06-09 16:22:11.365028 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.365030 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.365032 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x52d2000 (0 active) 2014-06-09 16:22:11.365035 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.355097 7f86a5610700 10 _omap_rmkeys meta/516ef296/pglog_3.c1/0//-1 r = 0 2014-06-09 16:22:11.365037 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.365045 7f86a5610700 15 _omap_rmkeys meta/516ef296/pglog_3.c1/0//-1 2014-06-09 16:22:11.365034 7f868a757700 10 osd.26 243 dequeue_op 0x19fe1900 prio 63 cost 516096 latency 0.947747 osd_op(client.4575.0:32284 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~516096] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.365054 7f86a5610700 10 _omap_rmkeys meta/516ef296/pglog_3.c1/0//-1 r = 0 2014-06-09 16:22:11.365058 7f868a757700 5 -- op tracker -- , seq: 363, time: 2014-06-09 16:22:11.365052, event: reached_pg, op: osd_op(client.4575.0:32284 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 1327104~516096] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365064 7f86a5610700 15 _omap_setkeys meta/516ef296/pglog_3.c1/0//-1 2014-06-09 16:22:11.365069 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.365064 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365078 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x19fe1900 2014-06-09 16:22:11.365088 7f868a757700 10 osd.26 243 dequeue_op 0x19fe1900 finish 2014-06-09 16:22:11.365086 7f86a5610700 10 cid: oid: 516ef296/pglog_3.c1/0//-1 skipping op, spos 3092.0.0 <= header.spos 150425.0.0 2014-06-09 16:22:11.365076 7f8689755700 10 osd.26 243 dequeue_op 0x1804c700 prio 63 cost 999424 latency 0.563503 osd_op(client.4577.0:37203 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 3194880~999424] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.365091 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.365092 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3092.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.365095 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.365097 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.365097 7f8689755700 5 -- op tracker -- , seq: 751, time: 2014-06-09 16:22:11.365091, event: reached_pg, op: osd_op(client.4577.0:37203 rbd_data.11816b8b4567.000000000000191d [set-alloc-hint object_size 4194304 write_size 4194304,write 3194880~999424] 3.4a78ff50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365104 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365100 7f868a757700 10 osd.26 243 dequeue_op 0x1b0b7a00 prio 63 cost 516096 latency 0.832905 osd_op(client.4572.0:30514 rbd_data.117c6b8b4567.0000000000001463 [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~516096] 3.1045021b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.365115 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1804c700 2014-06-09 16:22:11.365124 7f8689755700 10 osd.26 243 dequeue_op 0x1804c700 finish 2014-06-09 16:22:11.365121 7f868a757700 5 -- op tracker -- , seq: 481, time: 2014-06-09 16:22:11.365116, event: reached_pg, op: osd_op(client.4572.0:30514 rbd_data.117c6b8b4567.0000000000001463 [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~516096] 3.1045021b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365128 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365137 7f86a5610700 10 _do_op 0x288f9b60 seq 3092 r = 0, finisher 0x3062ade0 0 2014-06-09 16:22:11.365141 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x1b0b7a00 2014-06-09 16:22:11.365151 7f868a757700 10 osd.26 243 dequeue_op 0x1b0b7a00 finish 2014-06-09 16:22:11.365151 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062adc0] 2014-06-09 16:22:11.365127 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.365155 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062adc0] 2014-06-09 16:22:11.365157 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.365158 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.365161 7f86a5610700 10 _finish_op 0x288f9b60 seq 3092 osr(3.c1 0x3ed05c0)/0x3ed05c0 2014-06-09 16:22:11.365163 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.365165 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3018c0 (1 active) 2014-06-09 16:22:11.365168 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114480 (2 active) 2014-06-09 16:22:11.365170 7f86a5610700 5 _do_op 0x309a2f00 seq 3093 osr(1.67a 0x384de90)/0x384de90 start 2014-06-09 16:22:11.365173 7f86a5610700 10 _do_transaction on 0x2eaa0c0 2014-06-09 16:22:11.365162 7f868ff62700 10 osd.26 243 dequeue_op 0x19fe1800 prio 63 cost 516096 latency 0.932638 osd_op(client.4575.0:32285 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 1843200~516096] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.365179 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.365187 7f868ff62700 5 -- op tracker -- , seq: 382, time: 2014-06-09 16:22:11.365181, event: reached_pg, op: osd_op(client.4575.0:32285 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 1843200~516096] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365194 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365168 7f8689755700 10 osd.26 243 dequeue_op 0x19fe1d00 prio 63 cost 516096 latency 1.017127 osd_op(client.4575.0:32142 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~516096] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.365207 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x19fe1800 2014-06-09 16:22:11.365216 7f868ff62700 10 osd.26 243 dequeue_op 0x19fe1800 finish 2014-06-09 16:22:11.365214 7f8689755700 5 -- op tracker -- , seq: 294, time: 2014-06-09 16:22:11.365210, event: reached_pg, op: osd_op(client.4575.0:32142 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~516096] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365221 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365223 7f8691765700 10 osd.26 243 dequeue_op 0x1b0b7900 prio 63 cost 1449984 latency 0.831281 osd_op(client.4572.0:30515 rbd_data.117c6b8b4567.0000000000001463 [set-alloc-hint object_size 4194304 write_size 4194304,write 2744320~1449984] 3.1045021b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.365233 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x19fe1d00 2014-06-09 16:22:11.365242 7f8689755700 10 osd.26 243 dequeue_op 0x19fe1d00 finish 2014-06-09 16:22:11.365246 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.365248 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365246 7f8691765700 5 -- op tracker -- , seq: 497, time: 2014-06-09 16:22:11.365240, event: reached_pg, op: osd_op(client.4572.0:30515 rbd_data.117c6b8b4567.0000000000001463 [set-alloc-hint object_size 4194304 write_size 4194304,write 2744320~1449984] 3.1045021b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365255 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.365257 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365252 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365265 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (17 active) 2014-06-09 16:22:11.365268 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365266 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x1b0b7900 2014-06-09 16:22:11.365276 7f8691765700 10 osd.26 243 dequeue_op 0x1b0b7900 finish 2014-06-09 16:22:11.365276 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (16 active) 2014-06-09 16:22:11.365278 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365282 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.365284 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365291 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.365294 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365300 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.365302 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365292 7f8689f56700 10 osd.26 243 dequeue_op 0x19fe1700 prio 63 cost 1032192 latency 0.901563 osd_op(client.4575.0:32286 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~1032192] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.365313 7f8692f68700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (12 active) 2014-06-09 16:22:11.365317 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (13 active) 2014-06-09 16:22:11.365313 7f8689f56700 5 -- op tracker -- , seq: 391, time: 2014-06-09 16:22:11.365309, event: reached_pg, op: osd_op(client.4575.0:32286 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~1032192] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365320 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365327 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365332 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x19fe1700 2014-06-09 16:22:11.365342 7f8689f56700 10 osd.26 243 dequeue_op 0x19fe1700 finish 2014-06-09 16:22:11.365347 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (14 active) 2014-06-09 16:22:11.365351 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.365353 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365349 7f868af58700 10 osd.26 243 dequeue_op 0x19fe1600 prio 63 cost 516096 latency 0.885679 osd_op(client.4575.0:32287 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 3391488~516096] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.365367 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.365364 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365370 7f868af58700 5 -- op tracker -- , seq: 430, time: 2014-06-09 16:22:11.365365, event: reached_pg, op: osd_op(client.4575.0:32287 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 3391488~516096] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365371 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365376 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365375 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365381 7f868f761700 15 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.365390 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.365395 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (13 active) 2014-06-09 16:22:11.365388 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x19fe1600 2014-06-09 16:22:11.365398 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (14 active) 2014-06-09 16:22:11.365398 7f868af58700 10 osd.26 243 dequeue_op 0x19fe1600 finish 2014-06-09 16:22:11.365391 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.365404 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.365406 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365402 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365401 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365411 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (12 active) 2014-06-09 16:22:11.365413 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365412 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.365418 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (11 active) 2014-06-09 16:22:11.365420 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365416 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365426 7f868b759700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (10 active) 2014-06-09 16:22:11.365428 7f868b759700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365416 7f868ef60700 10 osd.26 243 dequeue_op 0x1c769c00 prio 63 cost 286720 latency 0.874806 osd_op(client.4575.0:32288 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 3907584~286720] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.365425 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.365432 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (9 active) 2014-06-09 16:22:11.365433 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.365435 7f868ff62700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365438 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (8 active) 2014-06-09 16:22:11.365441 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365438 7f868ef60700 5 -- op tracker -- , seq: 442, time: 2014-06-09 16:22:11.365433, event: reached_pg, op: osd_op(client.4575.0:32288 rbd_data.117f6b8b4567.0000000000001535 [set-alloc-hint object_size 4194304 write_size 4194304,write 3907584~286720] 3.c490ea1b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365445 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365440 7f8692767700 10 osd.26 243 dequeue_op 0x19fe1c00 prio 63 cost 655360 latency 0.968601 osd_op(client.4575.0:32143 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 3538944~655360] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] 2014-06-09 16:22:11.365445 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365461 7f8692767700 5 -- op tracker -- , seq: 303, time: 2014-06-09 16:22:11.365457, event: reached_pg, op: osd_op(client.4575.0:32143 rbd_data.117f6b8b4567.0000000000001523 [set-alloc-hint object_size 4194304 write_size 4194304,write 3538944~655360] 3.284fd1aa RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.365459 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x1c769c00 2014-06-09 16:22:11.365470 7f868ef60700 10 osd.26 243 dequeue_op 0x1c769c00 finish 2014-06-09 16:22:11.365474 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (7 active) 2014-06-09 16:22:11.365476 7f868ef60700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365468 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.365481 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (6 active) 2014-06-09 16:22:11.365484 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365489 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (5 active) 2014-06-09 16:22:11.365492 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365482 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] can_handle_while_inactive: 0x19fe1c00 2014-06-09 16:22:11.365503 7f8692767700 10 osd.26 243 dequeue_op 0x19fe1c00 finish 2014-06-09 16:22:11.365494 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=210 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.365505 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (4 active) 2014-06-09 16:22:11.365507 7f8692767700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365507 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=210 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.365520 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=210 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.087025 7 0.002961 2014-06-09 16:22:11.365530 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=210 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.365541 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=210 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.365551 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=210 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.365561 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'708 -> 213'708 2014-06-09 16:22:11.365613 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.365654 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.365659 7f8692f68700 5 queue_transactions existing osr(3.62a 0x3ed1aa0)/0x3ed1aa0 2014-06-09 16:22:11.365661 7f8692f68700 10 op_submit_start 3101 2014-06-09 16:22:11.365662 7f8692f68700 5 queue_transactions (trailing journal) 3101 2014-06-09 16:22:11.365664 7f8692f68700 5 queue_op 0x30606300 seq 3101 osr(3.62a 0x3ed1aa0) 1335 bytes (queue has 10 ops and 18074 bytes) 2014-06-09 16:22:11.365666 7f8692f68700 10 op_submit_finish 3101 2014-06-09 16:22:11.365670 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (3 active) 2014-06-09 16:22:11.365673 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365681 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365698 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 212 same_interval_since 242 2014-06-09 16:22:11.365709 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365726 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (4 active) 2014-06-09 16:22:11.365723 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365732 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365747 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.365740 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 212 same_interval_since 242 2014-06-09 16:22:11.365750 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365752 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365762 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365762 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.365772 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.365775 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365777 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.365788 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.365788 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365794 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (3 active) 2014-06-09 16:22:11.365798 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (4 active) 2014-06-09 16:22:11.365802 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365808 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365803 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365812 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.365821 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.365815 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365830 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (5 active) 2014-06-09 16:22:11.365825 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365837 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.365835 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365840 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365850 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.365849 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.365858 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.365854 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.365862 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (4 active) 2014-06-09 16:22:11.365865 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365870 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365865 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.365874 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.365875 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365873 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306cf660,0x3061fbc0,0xc169ac0,0x30508360,0x19940600,0x13964aa0] 2014-06-09 16:22:11.365878 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (3 active) 2014-06-09 16:22:11.365881 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365877 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d7398c0,0x3050ab20,0xc6e9340,0x3050bee0,0x189aa0e0,0xbd70da0,0x305091c0,0xbd72e60,0xbd682a0,0xc40d720,0x13964460,0xbdbb960,0xbd6ac60,0x170011c0,0xbd6c3e0,0x19940c00,0xbebe380,0x170001c0,0x1a0022e0,0x1322c680,0x18742a00,0x306cf140,0x3062ade0] 2014-06-09 16:22:11.365887 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365892 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365884 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.365896 7f868b759700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365900 7f868ff62700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.365896 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365908 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365923 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.365933 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.365996 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (2 active) 2014-06-09 16:22:11.366001 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (3 active) 2014-06-09 16:22:11.366004 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366018 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.366022 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366031 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.366097 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.366103 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (2 active) 2014-06-09 16:22:11.366106 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366185 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.366227 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02a0 (3 active) 2014-06-09 16:22:11.366235 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366229 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.4db( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.366234 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366243 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366245 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.366253 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.366257 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366267 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.366277 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.366283 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02a0 (2 active) 2014-06-09 16:22:11.366286 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366288 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.4db( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.366332 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (3 active) 2014-06-09 16:22:11.366333 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 210 same_interval_since 242 2014-06-09 16:22:11.366338 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4db( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366346 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.366351 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.4db( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.366361 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.366367 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (2 active) 2014-06-09 16:22:11.366370 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (3 active) 2014-06-09 16:22:11.366362 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.17( v 213'672 (0'0,213'672] local-les=243 n=137 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.366378 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.366373 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366394 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.17( v 213'672 (0'0,213'672] local-les=243 n=137 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.366414 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.366427 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.366385 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.366434 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.366444 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.366463 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (2 active) 2014-06-09 16:22:11.366468 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (3 active) 2014-06-09 16:22:11.366462 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.366477 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.366498 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (4 active) 2014-06-09 16:22:11.366496 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.312( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.366510 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.366502 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366516 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x16058e00,0x16058d00,0x16058c00,0x16058b00,0x16058a00,0x1804d000 2014-06-09 16:22:11.366525 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.312( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.366560 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.366563 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (5 active) 2014-06-09 16:22:11.366574 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (6 active) 2014-06-09 16:22:11.366583 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (7 active) 2014-06-09 16:22:11.366590 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (8 active) 2014-06-09 16:22:11.366578 7f868f761700 10 osd.26 243 dequeue_op 0x16058e00 prio 63 cost 835584 latency 1.116551 osd_op(client.4574.0:31715 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~835584] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.366599 7f8692f68700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (9 active) 2014-06-09 16:22:11.366603 7f868f761700 5 -- op tracker -- , seq: 197, time: 2014-06-09 16:22:11.366598, event: reached_pg, op: osd_op(client.4574.0:31715 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~835584] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.366608 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (10 active) 2014-06-09 16:22:11.366614 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366621 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (11 active) 2014-06-09 16:22:11.366611 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.366625 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16058e00 2014-06-09 16:22:11.366624 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7ce( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.366633 7f868f761700 10 osd.26 243 dequeue_op 0x16058e00 finish 2014-06-09 16:22:11.366625 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.312( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366638 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (10 active) 2014-06-09 16:22:11.366638 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.366641 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366638 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.312( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.366648 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.366654 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (9 active) 2014-06-09 16:22:11.366656 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366646 7f8691765700 10 osd.26 243 dequeue_op 0x16058d00 prio 63 cost 516096 latency 1.082827 osd_op(client.4574.0:31716 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 835584~516096] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.366652 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7ce( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.366661 7f868b759700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366471 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.17( v 213'672 (0'0,213'672] local-les=243 n=137 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366665 7f8691765700 5 -- op tracker -- , seq: 199, time: 2014-06-09 16:22:11.366661, event: reached_pg, op: osd_op(client.4574.0:31716 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 835584~516096] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.366666 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.17( v 213'672 (0'0,213'672] local-les=243 n=137 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.366678 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.366672 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.366682 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366684 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16058d00 2014-06-09 16:22:11.366694 7f8691765700 10 osd.26 243 dequeue_op 0x16058d00 finish 2014-06-09 16:22:11.366700 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (8 active) 2014-06-09 16:22:11.366703 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0230 (9 active) 2014-06-09 16:22:11.366707 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7ce( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366702 7f8689755700 10 osd.26 243 dequeue_op 0x16058c00 prio 63 cost 516096 latency 1.082153 osd_op(client.4574.0:31717 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 1351680~516096] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.366721 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.7ce( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.366725 7f8689755700 5 -- op tracker -- , seq: 212, time: 2014-06-09 16:22:11.366720, event: reached_pg, op: osd_op(client.4574.0:31717 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 1351680~516096] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.366732 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.366737 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (8 active) 2014-06-09 16:22:11.366740 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366734 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.366747 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0230 (7 active) 2014-06-09 16:22:11.366750 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366746 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16058c00 2014-06-09 16:22:11.366755 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366755 7f8689755700 10 osd.26 243 dequeue_op 0x16058c00 finish 2014-06-09 16:22:11.366670 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.30( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.366761 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (6 active) 2014-06-09 16:22:11.366764 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366766 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.366693 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.4f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.366772 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.366785 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (5 active) 2014-06-09 16:22:11.366788 7f8692767700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366782 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.30( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.366802 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (6 active) 2014-06-09 16:22:11.366800 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.603( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.366807 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.30( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.366815 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.366819 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.30( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.366828 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.366834 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (5 active) 2014-06-09 16:22:11.366836 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.366866 7f8692f68700 10 osd.26 243 dequeue_op 0x16058b00 prio 63 cost 1032192 latency 1.075963 osd_op(client.4574.0:31718 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 1867776~1032192] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.366989 7f8692f68700 5 -- op tracker -- , seq: 233, time: 2014-06-09 16:22:11.366909, event: reached_pg, op: osd_op(client.4574.0:31718 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 1867776~1032192] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.367002 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.367016 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16058b00 2014-06-09 16:22:11.367024 7f8692f68700 10 osd.26 243 dequeue_op 0x16058b00 finish 2014-06-09 16:22:11.367022 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.603( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.367030 7f8692f68700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (4 active) 2014-06-09 16:22:11.367032 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367037 7f868cf5c700 10 osd.26 243 dequeue_op 0x16058a00 prio 63 cost 1032192 latency 1.068301 osd_op(client.4574.0:31719 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~1032192] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.367083 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40a20 (5 active) 2014-06-09 16:22:11.367089 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.603( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367102 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.603( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367111 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.367084 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.364( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367117 7f868c75b700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40a20 (4 active) 2014-06-09 16:22:11.367119 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367122 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.367137 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.364( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.367154 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02a0 (5 active) 2014-06-09 16:22:11.367159 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.364( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367155 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] _activate_committed 243 peer_activated now 26 last_epoch_started 212 same_interval_since 242 2014-06-09 16:22:11.367169 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.364( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367178 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.367171 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] flushed 2014-06-09 16:22:11.367183 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02a0 (4 active) 2014-06-09 16:22:11.367186 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367194 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] flushed 2014-06-09 16:22:11.367215 7f868cf5c700 5 -- op tracker -- , seq: 241, time: 2014-06-09 16:22:11.367206, event: reached_pg, op: osd_op(client.4574.0:31719 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~1032192] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.367224 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367222 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.367236 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.367234 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16058a00 2014-06-09 16:22:11.367243 7f868cf5c700 10 osd.26 243 dequeue_op 0x16058a00 finish 2014-06-09 16:22:11.367248 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (3 active) 2014-06-09 16:22:11.367252 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (4 active) 2014-06-09 16:22:11.367249 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.367255 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367265 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (5 active) 2014-06-09 16:22:11.367270 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.367265 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367278 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.367270 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367275 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367282 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.37( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367292 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.367288 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] flushed 2014-06-09 16:22:11.367297 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (4 active) 2014-06-09 16:22:11.367300 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367288 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] requeue_ops 0x10c54200,0x10c54100,0x13023c00,0x13023b00,0x13023a00,0x13023900,0x13023800,0x1c768800,0x13023700,0x1804d900,0x1c768700,0x1804ca00,0x1c54bf00,0x14052f00,0x1c54be00,0xe2ac600,0x1c54bd00,0x1c54bc00,0x1b0b7100,0x1b0b7000,0x1b0b6f00 2014-06-09 16:22:11.367316 7f868b759700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (5 active) 2014-06-09 16:22:11.367304 7f8691f66700 10 osd.26 243 dequeue_op 0x1804d000 prio 63 cost 262144 latency 1.066521 osd_op(client.4574.0:31720 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 3932160~262144] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.367325 7f8691f66700 5 -- op tracker -- , seq: 252, time: 2014-06-09 16:22:11.367320, event: reached_pg, op: osd_op(client.4574.0:31720 rbd_data.117e6b8b4567.00000000000014c0 [set-alloc-hint object_size 4194304 write_size 4194304,write 3932160~262144] 3.131917d4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.367333 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (6 active) 2014-06-09 16:22:11.367329 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367332 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.367345 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.367344 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1804d000 2014-06-09 16:22:11.367353 7f8691f66700 10 osd.26 243 dequeue_op 0x1804d000 finish 2014-06-09 16:22:11.367357 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (5 active) 2014-06-09 16:22:11.367360 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367365 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367370 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367339 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367368 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.367376 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.663( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367385 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0230 (6 active) 2014-06-09 16:22:11.367389 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.367395 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (5 active) 2014-06-09 16:22:11.367388 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367397 7f868ef60700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367390 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367401 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367403 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.367399 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.369( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367408 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.367413 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0230 (4 active) 2014-06-09 16:22:11.367416 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367414 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.367431 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (5 active) 2014-06-09 16:22:11.367434 7f868ff62700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367431 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367435 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.367444 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.367446 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.367454 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.051095 7 0.000202 2014-06-09 16:22:11.367454 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.367461 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.367469 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.367471 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367484 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.367480 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.367490 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'568 -> 213'568 2014-06-09 16:22:11.367498 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.367521 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367529 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.367535 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.367546 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.367558 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.367561 7f868bf5a700 5 queue_transactions existing osr(3.9b 0x3ed0630)/0x3ed0630 2014-06-09 16:22:11.367565 7f868bf5a700 10 op_submit_start 3102 2014-06-09 16:22:11.367564 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (6 active) 2014-06-09 16:22:11.367565 7f868bf5a700 5 queue_transactions (trailing journal) 3102 2014-06-09 16:22:11.367567 7f868bf5a700 5 queue_op 0x308cb380 seq 3102 osr(3.9b 0x3ed0630) 1332 bytes (queue has 11 ops and 19406 bytes) 2014-06-09 16:22:11.367570 7f868bf5a700 10 op_submit_finish 3102 2014-06-09 16:22:11.367562 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367569 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367572 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367578 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.367580 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.372( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367581 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.691( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367589 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.367589 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.367594 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (5 active) 2014-06-09 16:22:11.367596 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367592 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.367431 7f869af78700 5 -- op tracker -- , seq: 1538, time: 2014-06-09 16:22:11.367427, event: done, op: pg_info(2 pgs e243:3.62a(2),3.9b(2)) 2014-06-09 16:22:11.367611 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.367628 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.367631 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.367630 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.367593 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367637 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367643 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (6 active) 2014-06-09 16:22:11.367646 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.367647 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.367648 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.367642 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.367649 7f869af78700 20 osd.26 243 _dispatch 0x29ec5180 pg_notify(0.4e3(23) epoch 243) v5 2014-06-09 16:22:11.367657 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (7 active) 2014-06-09 16:22:11.367663 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367676 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.367678 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.367688 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.367687 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.367693 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (6 active) 2014-06-09 16:22:11.367696 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.367698 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (7 active) 2014-06-09 16:22:11.367706 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (8 active) 2014-06-09 16:22:11.367712 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (7 active) 2014-06-09 16:22:11.367715 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367714 7f869af78700 5 -- op tracker -- , seq: 1539, time: 2014-06-09 16:22:11.003050, event: header_read, op: pg_notify(0.4e3(23) epoch 243) 2014-06-09 16:22:11.367725 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367723 7f869af78700 5 -- op tracker -- , seq: 1539, time: 2014-06-09 16:22:11.003050, event: throttled, op: pg_notify(0.4e3(23) epoch 243) 2014-06-09 16:22:11.367740 7f869af78700 5 -- op tracker -- , seq: 1539, time: 2014-06-09 16:22:11.003071, event: all_read, op: pg_notify(0.4e3(23) epoch 243) 2014-06-09 16:22:11.367747 7f869af78700 5 -- op tracker -- , seq: 1539, time: 2014-06-09 16:22:11.367646, event: dispatched, op: pg_notify(0.4e3(23) epoch 243) 2014-06-09 16:22:11.367752 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367757 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367755 7f869af78700 5 -- op tracker -- , seq: 1539, time: 2014-06-09 16:22:11.367752, event: waiting_for_osdmap, op: pg_notify(0.4e3(23) epoch 243) 2014-06-09 16:22:11.367759 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.367761 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367762 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec5180 2014-06-09 16:22:11.367784 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (8 active) 2014-06-09 16:22:11.367794 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (9 active) 2014-06-09 16:22:11.367801 7f8692767700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.367768 7f869af78700 5 -- op tracker -- , seq: 1539, time: 2014-06-09 16:22:11.367766, event: started, op: pg_notify(0.4e3(23) epoch 243) 2014-06-09 16:22:11.367824 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (10 active) 2014-06-09 16:22:11.367839 7f869af78700 5 -- op tracker -- , seq: 1539, time: 2014-06-09 16:22:11.367837, event: done, op: pg_notify(0.4e3(23) epoch 243) 2014-06-09 16:22:11.367848 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.367850 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.367873 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.367875 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.367876 7f869af78700 20 osd.26 243 _dispatch 0x304dd200 pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.367883 7f869af78700 5 -- op tracker -- , seq: 1540, time: 2014-06-09 16:22:11.053838, event: header_read, op: pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.367889 7f869af78700 5 -- op tracker -- , seq: 1540, time: 2014-06-09 16:22:11.053839, event: throttled, op: pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.367891 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.367893 7f869af78700 5 -- op tracker -- , seq: 1540, time: 2014-06-09 16:22:11.053859, event: all_read, op: pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.367896 7f869af78700 5 -- op tracker -- , seq: 1540, time: 2014-06-09 16:22:11.367873, event: dispatched, op: pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.367900 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (9 active) 2014-06-09 16:22:11.367900 7f869af78700 5 -- op tracker -- , seq: 1540, time: 2014-06-09 16:22:11.367899, event: waiting_for_osdmap, op: pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.367904 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (10 active) 2014-06-09 16:22:11.367903 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.367906 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304dd200 2014-06-09 16:22:11.367910 7f869af78700 5 -- op tracker -- , seq: 1540, time: 2014-06-09 16:22:11.367908, event: started, op: pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.367907 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.4e3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.367918 7f869af78700 5 -- op tracker -- , seq: 1540, time: 2014-06-09 16:22:11.367916, event: done, op: pg_log(1.91 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.367923 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.367924 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.367920 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.367930 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.367932 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.367932 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.367933 7f869af78700 20 osd.26 243 _dispatch 0x1ae7c380 pg_notify(1.4d3(20) epoch 242) v5 2014-06-09 16:22:11.367936 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (9 active) 2014-06-09 16:22:11.367936 7f869af78700 5 -- op tracker -- , seq: 1541, time: 2014-06-09 16:22:10.983872, event: header_read, op: pg_notify(1.4d3(20) epoch 242) 2014-06-09 16:22:11.367941 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (10 active) 2014-06-09 16:22:11.367941 7f869af78700 5 -- op tracker -- , seq: 1541, time: 2014-06-09 16:22:10.983872, event: throttled, op: pg_notify(1.4d3(20) epoch 242) 2014-06-09 16:22:11.367947 7f869af78700 5 -- op tracker -- , seq: 1541, time: 2014-06-09 16:22:10.983898, event: all_read, op: pg_notify(1.4d3(20) epoch 242) 2014-06-09 16:22:11.367953 7f869af78700 5 -- op tracker -- , seq: 1541, time: 2014-06-09 16:22:11.367931, event: dispatched, op: pg_notify(1.4d3(20) epoch 242) 2014-06-09 16:22:11.367945 7f868cf5c700 10 osd.26 243 dequeue_op 0x10c54200 prio 63 cost 24576 latency 1.229408 osd_op(client.4568.0:34992 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.367960 7f869af78700 5 -- op tracker -- , seq: 1541, time: 2014-06-09 16:22:11.367958, event: waiting_for_osdmap, op: pg_notify(1.4d3(20) epoch 242) 2014-06-09 16:22:11.367968 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40a20 (11 active) 2014-06-09 16:22:11.367968 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.367971 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1ae7c380 2014-06-09 16:22:11.367976 7f869af78700 5 -- op tracker -- , seq: 1541, time: 2014-06-09 16:22:11.367974, event: started, op: pg_notify(1.4d3(20) epoch 242) 2014-06-09 16:22:11.367973 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.367985 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.91( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.367996 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.367995 7f869af78700 5 -- op tracker -- , seq: 1541, time: 2014-06-09 16:22:11.367992, event: done, op: pg_notify(1.4d3(20) epoch 242) 2014-06-09 16:22:11.368000 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.368003 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.368005 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.368004 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.91( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.119452 7 0.000171 2014-06-09 16:22:11.368014 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.91( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.368025 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.368026 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.368027 7f869af78700 20 osd.26 243 _dispatch 0x2fe34400 pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.368025 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.91( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.368031 7f869af78700 5 -- op tracker -- , seq: 1542, time: 2014-06-09 16:22:11.021183, event: header_read, op: pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368036 7f869af78700 5 -- op tracker -- , seq: 1542, time: 2014-06-09 16:22:11.021184, event: throttled, op: pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368040 7f869af78700 5 -- op tracker -- , seq: 1542, time: 2014-06-09 16:22:11.021198, event: all_read, op: pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368035 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.368044 7f869af78700 5 -- op tracker -- , seq: 1542, time: 2014-06-09 16:22:11.368025, event: dispatched, op: pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368048 7f869af78700 5 -- op tracker -- , seq: 1542, time: 2014-06-09 16:22:11.368047, event: waiting_for_osdmap, op: pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368051 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.368044 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.368053 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe34400 2014-06-09 16:22:11.368057 7f869af78700 5 -- op tracker -- , seq: 1542, time: 2014-06-09 16:22:11.368055, event: started, op: pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368054 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.368065 7f869af78700 5 -- op tracker -- , seq: 1542, time: 2014-06-09 16:22:11.368063, event: done, op: pg_log(0.301 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368069 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.368070 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.368080 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.368082 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.368083 7f869af78700 20 osd.26 243 _dispatch 0x2d733340 pg_notify(2.18e(20) epoch 242) v5 2014-06-09 16:22:11.368088 7f869af78700 5 -- op tracker -- , seq: 1543, time: 2014-06-09 16:22:10.923513, event: header_read, op: pg_notify(2.18e(20) epoch 242) 2014-06-09 16:22:11.368095 7f869af78700 5 -- op tracker -- , seq: 1543, time: 2014-06-09 16:22:10.923514, event: throttled, op: pg_notify(2.18e(20) epoch 242) 2014-06-09 16:22:11.368101 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.368101 7f869af78700 5 -- op tracker -- , seq: 1543, time: 2014-06-09 16:22:10.923539, event: all_read, op: pg_notify(2.18e(20) epoch 242) 2014-06-09 16:22:11.368105 7f869af78700 5 -- op tracker -- , seq: 1543, time: 2014-06-09 16:22:11.368080, event: dispatched, op: pg_notify(2.18e(20) epoch 242) 2014-06-09 16:22:11.368106 7f868c75b700 5 queue_transactions existing osr(1.91 0x384c430)/0x384c430 2014-06-09 16:22:11.368109 7f868c75b700 10 op_submit_start 3103 2014-06-09 16:22:11.368110 7f868c75b700 5 queue_transactions (trailing journal) 3103 2014-06-09 16:22:11.368111 7f868c75b700 5 queue_op 0x2c06e8a0 seq 3103 osr(1.91 0x384c430) 1880 bytes (queue has 12 ops and 21286 bytes) 2014-06-09 16:22:11.368114 7f868c75b700 10 op_submit_finish 3103 2014-06-09 16:22:11.368118 7f868c75b700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40a20 (10 active) 2014-06-09 16:22:11.368123 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (11 active) 2014-06-09 16:22:11.368130 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0230 (12 active) 2014-06-09 16:22:11.368139 7f8692f68700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (13 active) 2014-06-09 16:22:11.368135 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 9 notify: (query_epoch:242, epoch_sent:242, info:1.4d3( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.368150 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (14 active) 2014-06-09 16:22:11.368148 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.368158 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.368162 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.368172 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.301( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.368183 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.368109 7f869af78700 5 -- op tracker -- , seq: 1543, time: 2014-06-09 16:22:11.368108, event: waiting_for_osdmap, op: pg_notify(2.18e(20) epoch 242) 2014-06-09 16:22:11.368187 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.368191 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.368194 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d733340 2014-06-09 16:22:11.368190 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.301( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.178047 7 0.009285 2014-06-09 16:22:11.368200 7f869af78700 5 -- op tracker -- , seq: 1543, time: 2014-06-09 16:22:11.368198, event: started, op: pg_notify(2.18e(20) epoch 242) 2014-06-09 16:22:11.368200 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.301( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.368210 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.301( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.368218 7f869af78700 5 -- op tracker -- , seq: 1543, time: 2014-06-09 16:22:11.368216, event: done, op: pg_notify(2.18e(20) epoch 242) 2014-06-09 16:22:11.368219 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.368226 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.368229 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.368227 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.368235 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.368249 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.368251 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.368253 7f869af78700 20 osd.26 243 _dispatch 0x2a074400 pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.368260 7f869af78700 5 -- op tracker -- , seq: 1544, time: 2014-06-09 16:22:11.004621, event: header_read, op: pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368265 7f869af78700 5 -- op tracker -- , seq: 1544, time: 2014-06-09 16:22:11.004622, event: throttled, op: pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368269 7f869af78700 5 -- op tracker -- , seq: 1544, time: 2014-06-09 16:22:11.004630, event: all_read, op: pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368273 7f869af78700 5 -- op tracker -- , seq: 1544, time: 2014-06-09 16:22:11.368249, event: dispatched, op: pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368277 7f869af78700 5 -- op tracker -- , seq: 1544, time: 2014-06-09 16:22:11.368275, event: waiting_for_osdmap, op: pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368280 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.368282 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a074400 2014-06-09 16:22:11.368282 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.368285 7f869af78700 5 -- op tracker -- , seq: 1544, time: 2014-06-09 16:22:11.368284, event: started, op: pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368287 7f8691765700 5 queue_transactions existing osr(0.301 0x2bd6e10)/0x2bd6e10 2014-06-09 16:22:11.368290 7f8691765700 10 op_submit_start 3104 2014-06-09 16:22:11.368291 7f8691765700 5 queue_transactions (trailing journal) 3104 2014-06-09 16:22:11.368292 7f8691765700 5 queue_op 0x30606f60 seq 3104 osr(0.301 0x2bd6e10) 1954 bytes (queue has 13 ops and 23240 bytes) 2014-06-09 16:22:11.368294 7f8691765700 10 op_submit_finish 3104 2014-06-09 16:22:11.368293 7f869af78700 5 -- op tracker -- , seq: 1544, time: 2014-06-09 16:22:11.368292, event: done, op: pg_log(1.76 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.368298 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.368299 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.368299 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0230 (13 active) 2014-06-09 16:22:11.368302 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0230 (14 active) 2014-06-09 16:22:11.368311 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.368313 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.368305 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:2.18e( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.368314 7f869af78700 20 osd.26 243 _dispatch 0x3043ac40 pg_info(1 pgs e243:3.31c(2)) v4 2014-06-09 16:22:11.368320 7f869af78700 5 -- op tracker -- , seq: 1545, time: 2014-06-09 16:22:11.032784, event: header_read, op: pg_info(1 pgs e243:3.31c(2)) 2014-06-09 16:22:11.368317 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.368327 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.368327 7f869af78700 5 -- op tracker -- , seq: 1545, time: 2014-06-09 16:22:11.032784, event: throttled, op: pg_info(1 pgs e243:3.31c(2)) 2014-06-09 16:22:11.368331 7f869af78700 5 -- op tracker -- , seq: 1545, time: 2014-06-09 16:22:11.032798, event: all_read, op: pg_info(1 pgs e243:3.31c(2)) 2014-06-09 16:22:11.368334 7f869af78700 5 -- op tracker -- , seq: 1545, time: 2014-06-09 16:22:11.368311, event: dispatched, op: pg_info(1 pgs e243:3.31c(2)) 2014-06-09 16:22:11.368338 7f869af78700 5 -- op tracker -- , seq: 1545, time: 2014-06-09 16:22:11.368337, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.31c(2)) 2014-06-09 16:22:11.368332 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.368348 7f868cf5c700 5 -- op tracker -- , seq: 119, time: 2014-06-09 16:22:11.367964, event: reached_pg, op: osd_op(client.4568.0:34992 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.368342 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.76( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.374897 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.374903 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.374894 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.374908 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.76( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.051566 7 0.000199 2014-06-09 16:22:11.374916 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x10c54200 2014-06-09 16:22:11.374928 7f868cf5c700 10 osd.26 243 dequeue_op 0x10c54200 finish 2014-06-09 16:22:11.374923 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.76( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.374935 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.76( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.374945 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.374939 7f8690763700 10 osd.26 243 dequeue_op 0x10c54100 prio 63 cost 516096 latency 1.227641 osd_op(client.4568.0:34993 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.374955 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.374963 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.374969 7f8690763700 5 -- op tracker -- , seq: 131, time: 2014-06-09 16:22:11.374963, event: reached_pg, op: osd_op(client.4568.0:34993 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 24576~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.374977 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.374992 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x10c54100 2014-06-09 16:22:11.375004 7f8690763700 10 osd.26 243 dequeue_op 0x10c54100 finish 2014-06-09 16:22:11.375012 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.375019 7f8691765700 5 queue_transactions existing osr(1.76 0x384c530)/0x384c530 2014-06-09 16:22:11.375022 7f8691765700 10 op_submit_start 3105 2014-06-09 16:22:11.375023 7f8691765700 5 queue_transactions (trailing journal) 3105 2014-06-09 16:22:11.375025 7f8691765700 5 queue_op 0x30606f00 seq 3105 osr(1.76 0x384c530) 1880 bytes (queue has 14 ops and 25120 bytes) 2014-06-09 16:22:11.375028 7f8691765700 10 op_submit_finish 3105 2014-06-09 16:22:11.375012 7f868f761700 10 osd.26 243 dequeue_op 0x13023c00 prio 63 cost 516096 latency 1.223306 osd_op(client.4568.0:34994 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.368361 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.375036 7f868f761700 5 -- op tracker -- , seq: 141, time: 2014-06-09 16:22:11.375031, event: reached_pg, op: osd_op(client.4568.0:34994 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.368341 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.31c(2)) v4 from osd.5 2014-06-09 16:22:11.375044 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043ac40 2014-06-09 16:22:11.375043 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.375052 7f869af78700 5 -- op tracker -- , seq: 1545, time: 2014-06-09 16:22:11.375049, event: started, op: pg_info(1 pgs e243:3.31c(2)) 2014-06-09 16:22:11.375058 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x13023c00 2014-06-09 16:22:11.375070 7f868f761700 10 osd.26 243 dequeue_op 0x13023c00 finish 2014-06-09 16:22:11.375077 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.375074 7f869af78700 5 -- op tracker -- , seq: 1545, time: 2014-06-09 16:22:11.375072, event: done, op: pg_info(1 pgs e243:3.31c(2)) 2014-06-09 16:22:11.375080 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (15 active) 2014-06-09 16:22:11.375091 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375084 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=206 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.375093 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375096 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=206 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.375109 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375111 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375112 7f869af78700 20 osd.26 243 _dispatch 0x1abd4000 pg_notify(2.12d(27) epoch 242) v5 2014-06-09 16:22:11.375108 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=206 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.143925 7 0.000208 2014-06-09 16:22:11.375118 7f869af78700 5 -- op tracker -- , seq: 1546, time: 2014-06-09 16:22:10.852230, event: header_read, op: pg_notify(2.12d(27) epoch 242) 2014-06-09 16:22:11.375119 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=206 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.375126 7f869af78700 5 -- op tracker -- , seq: 1546, time: 2014-06-09 16:22:10.852231, event: throttled, op: pg_notify(2.12d(27) epoch 242) 2014-06-09 16:22:11.375132 7f869af78700 5 -- op tracker -- , seq: 1546, time: 2014-06-09 16:22:10.852274, event: all_read, op: pg_notify(2.12d(27) epoch 242) 2014-06-09 16:22:11.375132 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=206 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.375139 7f869af78700 5 -- op tracker -- , seq: 1546, time: 2014-06-09 16:22:11.375109, event: dispatched, op: pg_notify(2.12d(27) epoch 242) 2014-06-09 16:22:11.375146 7f869af78700 5 -- op tracker -- , seq: 1546, time: 2014-06-09 16:22:11.375144, event: waiting_for_osdmap, op: pg_notify(2.12d(27) epoch 242) 2014-06-09 16:22:11.375143 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=206 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.375151 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.375153 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1abd4000 2014-06-09 16:22:11.375158 7f869af78700 5 -- op tracker -- , seq: 1546, time: 2014-06-09 16:22:11.375156, event: started, op: pg_notify(2.12d(27) epoch 242) 2014-06-09 16:22:11.375153 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'500 -> 213'500 2014-06-09 16:22:11.375175 7f869af78700 5 -- op tracker -- , seq: 1546, time: 2014-06-09 16:22:11.375173, event: done, op: pg_notify(2.12d(27) epoch 242) 2014-06-09 16:22:11.375183 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375185 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375189 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.375206 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375208 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375209 7f869af78700 20 osd.26 243 _dispatch 0x240ef180 pg_notify(3.6cd(10) epoch 242) v5 2014-06-09 16:22:11.375214 7f869af78700 5 -- op tracker -- , seq: 1547, time: 2014-06-09 16:22:10.795365, event: header_read, op: pg_notify(3.6cd(10) epoch 242) 2014-06-09 16:22:11.375221 7f869af78700 5 -- op tracker -- , seq: 1547, time: 2014-06-09 16:22:10.795366, event: throttled, op: pg_notify(3.6cd(10) epoch 242) 2014-06-09 16:22:11.375229 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.375228 7f869af78700 5 -- op tracker -- , seq: 1547, time: 2014-06-09 16:22:10.795384, event: all_read, op: pg_notify(3.6cd(10) epoch 242) 2014-06-09 16:22:11.375233 7f868f761700 5 queue_transactions existing osr(3.31c 0x3ed0d40)/0x3ed0d40 2014-06-09 16:22:11.375236 7f868f761700 10 op_submit_start 3106 2014-06-09 16:22:11.375234 7f869af78700 5 -- op tracker -- , seq: 1547, time: 2014-06-09 16:22:11.375206, event: dispatched, op: pg_notify(3.6cd(10) epoch 242) 2014-06-09 16:22:11.375237 7f868f761700 5 queue_transactions (trailing journal) 3106 2014-06-09 16:22:11.375239 7f868f761700 5 queue_op 0x309a3ec0 seq 3106 osr(3.31c 0x3ed0d40) 1335 bytes (queue has 15 ops and 26455 bytes) 2014-06-09 16:22:11.375242 7f868f761700 10 op_submit_finish 3106 2014-06-09 16:22:11.375241 7f869af78700 5 -- op tracker -- , seq: 1547, time: 2014-06-09 16:22:11.375239, event: waiting_for_osdmap, op: pg_notify(3.6cd(10) epoch 242) 2014-06-09 16:22:11.375245 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.375245 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (14 active) 2014-06-09 16:22:11.375247 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240ef180 2014-06-09 16:22:11.375252 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (15 active) 2014-06-09 16:22:11.375252 7f869af78700 5 -- op tracker -- , seq: 1547, time: 2014-06-09 16:22:11.375250, event: started, op: pg_notify(3.6cd(10) epoch 242) 2014-06-09 16:22:11.375255 7f868f761700 10 osd.26 243 dequeue_op 0x13023b00 prio 63 cost 516096 latency 1.220445 osd_op(client.4568.0:34995 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 1056768~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.375269 7f869af78700 5 -- op tracker -- , seq: 1547, time: 2014-06-09 16:22:11.375267, event: done, op: pg_notify(3.6cd(10) epoch 242) 2014-06-09 16:22:11.375280 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (16 active) 2014-06-09 16:22:11.375280 7f868f761700 5 -- op tracker -- , seq: 162, time: 2014-06-09 16:22:11.375274, event: reached_pg, op: osd_op(client.4568.0:34995 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 1056768~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.375287 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:2.12d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.375290 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.375301 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.375312 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.375305 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x13023b00 2014-06-09 16:22:11.375316 7f868f761700 10 osd.26 243 dequeue_op 0x13023b00 finish 2014-06-09 16:22:11.375319 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.375324 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.375316 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:3.6cd( v 214'577 (0'0,214'577] local-les=203 n=111 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.375329 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.375340 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.375328 7f868f761700 10 osd.26 243 dequeue_op 0x1c768800 prio 63 cost 303104 latency 0.879468 osd_op(client.4574.0:31823 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~303104] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.375346 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (15 active) 2014-06-09 16:22:11.375352 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (16 active) 2014-06-09 16:22:11.375351 7f868f761700 5 -- op tracker -- , seq: 438, time: 2014-06-09 16:22:11.375346, event: reached_pg, op: osd_op(client.4574.0:31823 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~303104] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.375361 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (17 active) 2014-06-09 16:22:11.375358 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.375371 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (18 active) 2014-06-09 16:22:11.375370 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1c768800 2014-06-09 16:22:11.375380 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375381 7f868f761700 10 osd.26 243 dequeue_op 0x1c768800 finish 2014-06-09 16:22:11.375382 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375387 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (17 active) 2014-06-09 16:22:11.375392 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (18 active) 2014-06-09 16:22:11.375398 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375400 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375401 7f869af78700 20 osd.26 243 _dispatch 0x26000000 pg_notify(2.3fb(20),1.3fa(20) epoch 243) v5 2014-06-09 16:22:11.375396 7f868f761700 10 osd.26 243 dequeue_op 0x13023a00 prio 63 cost 516096 latency 1.173740 osd_op(client.4568.0:34996 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 1572864~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.375407 7f869af78700 5 -- op tracker -- , seq: 1548, time: 2014-06-09 16:22:11.062164, event: header_read, op: pg_notify(2.3fb(20),1.3fa(20) epoch 243) 2014-06-09 16:22:11.375422 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.375418 7f868f761700 5 -- op tracker -- , seq: 165, time: 2014-06-09 16:22:11.375413, event: reached_pg, op: osd_op(client.4568.0:34996 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 1572864~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.375431 7f869af78700 5 -- op tracker -- , seq: 1548, time: 2014-06-09 16:22:11.062165, event: throttled, op: pg_notify(2.3fb(20),1.3fa(20) epoch 243) 2014-06-09 16:22:11.375428 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.375439 7f869af78700 5 -- op tracker -- , seq: 1548, time: 2014-06-09 16:22:11.062194, event: all_read, op: pg_notify(2.3fb(20),1.3fa(20) epoch 243) 2014-06-09 16:22:11.375445 7f869af78700 5 -- op tracker -- , seq: 1548, time: 2014-06-09 16:22:11.375398, event: dispatched, op: pg_notify(2.3fb(20),1.3fa(20) epoch 243) 2014-06-09 16:22:11.375443 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x13023a00 2014-06-09 16:22:11.375452 7f869af78700 5 -- op tracker -- , seq: 1548, time: 2014-06-09 16:22:11.375450, event: waiting_for_osdmap, op: pg_notify(2.3fb(20),1.3fa(20) epoch 243) 2014-06-09 16:22:11.375455 7f868f761700 10 osd.26 243 dequeue_op 0x13023a00 finish 2014-06-09 16:22:11.375457 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.375459 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26000000 2014-06-09 16:22:11.375461 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.375466 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.375464 7f869af78700 5 -- op tracker -- , seq: 1548, time: 2014-06-09 16:22:11.375461, event: started, op: pg_notify(2.3fb(20),1.3fa(20) epoch 243) 2014-06-09 16:22:11.375470 7f868f761700 10 osd.26 243 dequeue_op 0x1804d900 prio 63 cost 450560 latency 0.978536 osd_op(client.4569.0:38600 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.375489 7f869af78700 5 -- op tracker -- , seq: 1548, time: 2014-06-09 16:22:11.375486, event: done, op: pg_notify(2.3fb(20),1.3fa(20) epoch 243) 2014-06-09 16:22:11.375502 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (20 active) 2014-06-09 16:22:11.375503 7f868f761700 5 -- op tracker -- , seq: 361, time: 2014-06-09 16:22:11.375487, event: reached_pg, op: osd_op(client.4569.0:38600 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.375511 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.375524 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1804d900 2014-06-09 16:22:11.375521 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] _activate_committed 243 peer_activated now 26 last_epoch_started 201 same_interval_since 242 2014-06-09 16:22:11.375536 7f868f761700 10 osd.26 243 dequeue_op 0x1804d900 finish 2014-06-09 16:22:11.375543 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.375548 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.375540 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] flushed 2014-06-09 16:22:11.375555 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] flushed 2014-06-09 16:22:11.375551 7f868f761700 10 osd.26 243 dequeue_op 0x1c768700 prio 63 cost 516096 latency 0.878973 osd_op(client.4574.0:31824 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 303104~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.375572 7f868f761700 5 -- op tracker -- , seq: 441, time: 2014-06-09 16:22:11.375567, event: reached_pg, op: osd_op(client.4574.0:31824 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 303104~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.375576 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d7398c0,0x3050ab20,0xc6e9340,0x3050bee0,0x189aa0e0,0xbd70da0,0x305091c0,0xbd72e60,0xbd682a0,0xc40d720,0x13964460,0xbdbb960,0xbd6ac60,0x170011c0,0xbd6c3e0,0x19940c00,0xbebe380,0x170001c0,0x1a0022e0,0x1322c680,0x18742a00,0x306cf140,0x3062ade0] 2014-06-09 16:22:11.375583 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.375585 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.375579 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.375591 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375593 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375594 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1c768700 2014-06-09 16:22:11.375605 7f868f761700 10 osd.26 243 dequeue_op 0x1c768700 finish 2014-06-09 16:22:11.375609 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.375627 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (20 active) 2014-06-09 16:22:11.375632 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.375641 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375643 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375644 7f869af78700 20 osd.26 243 _dispatch 0x240f5600 pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.375646 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.92( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.375651 7f869af78700 5 -- op tracker -- , seq: 1549, time: 2014-06-09 16:22:11.003102, event: header_read, op: pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.375657 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.375660 7f869af78700 5 -- op tracker -- , seq: 1549, time: 2014-06-09 16:22:11.003102, event: throttled, op: pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.375667 7f869af78700 5 -- op tracker -- , seq: 1549, time: 2014-06-09 16:22:11.003115, event: all_read, op: pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.375662 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.375674 7f869af78700 5 -- op tracker -- , seq: 1549, time: 2014-06-09 16:22:11.375640, event: dispatched, op: pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.375679 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.375681 7f869af78700 5 -- op tracker -- , seq: 1549, time: 2014-06-09 16:22:11.375678, event: waiting_for_osdmap, op: pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.375686 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.375690 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240f5600 2014-06-09 16:22:11.375683 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.375696 7f869af78700 5 -- op tracker -- , seq: 1549, time: 2014-06-09 16:22:11.375694, event: started, op: pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.375695 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] requeue_ops 0xe2ad300,0xe2ad200,0xe2ad100,0x10c54500,0x10c54400,0x10c54300 2014-06-09 16:22:11.375710 7f869af78700 5 -- op tracker -- , seq: 1549, time: 2014-06-09 16:22:11.375708, event: done, op: pg_log(2.606 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.375718 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375720 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375723 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.375729 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (19 active) 2014-06-09 16:22:11.375732 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375734 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375734 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.375735 7f869af78700 20 osd.26 243 _dispatch 0x3046ae00 pg_notify(0.633(26) epoch 243) v5 2014-06-09 16:22:11.375740 7f869af78700 5 -- op tracker -- , seq: 1550, time: 2014-06-09 16:22:11.053877, event: header_read, op: pg_notify(0.633(26) epoch 243) 2014-06-09 16:22:11.375747 7f869af78700 5 -- op tracker -- , seq: 1550, time: 2014-06-09 16:22:11.053878, event: throttled, op: pg_notify(0.633(26) epoch 243) 2014-06-09 16:22:11.375738 7f868f761700 10 osd.26 243 dequeue_op 0x13023900 prio 63 cost 1032192 latency 1.173296 osd_op(client.4568.0:34997 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 2088960~1032192] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.375754 7f869af78700 5 -- op tracker -- , seq: 1550, time: 2014-06-09 16:22:11.053929, event: all_read, op: pg_notify(0.633(26) epoch 243) 2014-06-09 16:22:11.375763 7f868f761700 5 -- op tracker -- , seq: 191, time: 2014-06-09 16:22:11.375757, event: reached_pg, op: osd_op(client.4568.0:34997 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 2088960~1032192] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.375775 7f869af78700 5 -- op tracker -- , seq: 1550, time: 2014-06-09 16:22:11.375732, event: dispatched, op: pg_notify(0.633(26) epoch 243) 2014-06-09 16:22:11.375773 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.375783 7f869af78700 5 -- op tracker -- , seq: 1550, time: 2014-06-09 16:22:11.375781, event: waiting_for_osdmap, op: pg_notify(0.633(26) epoch 243) 2014-06-09 16:22:11.375788 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.375790 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046ae00 2014-06-09 16:22:11.375787 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x13023900 2014-06-09 16:22:11.375795 7f869af78700 5 -- op tracker -- , seq: 1550, time: 2014-06-09 16:22:11.375793, event: started, op: pg_notify(0.633(26) epoch 243) 2014-06-09 16:22:11.375799 7f868f761700 10 osd.26 243 dequeue_op 0x13023900 finish 2014-06-09 16:22:11.375803 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.375806 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (20 active) 2014-06-09 16:22:11.375811 7f869af78700 5 -- op tracker -- , seq: 1550, time: 2014-06-09 16:22:11.375809, event: done, op: pg_notify(0.633(26) epoch 243) 2014-06-09 16:22:11.375809 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.375820 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375822 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375819 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.606( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.375830 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.375833 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.375841 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375843 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375837 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.606( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 1.110075 7 0.000212 2014-06-09 16:22:11.375844 7f869af78700 20 osd.26 243 _dispatch 0x19ffdb00 osd_map(243..243 src has 1..243) v3 2014-06-09 16:22:11.375846 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.606( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.375851 7f869af78700 3 osd.26 243 handle_osd_map epochs [243,243], i have 243, src has [1,243] 2014-06-09 16:22:11.375854 7f869af78700 10 osd.26 243 no new maps here, dropping 2014-06-09 16:22:11.375856 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375857 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375857 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.606( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.375870 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375871 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375867 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.375873 7f869af78700 20 osd.26 243 _dispatch 0x2ea5b6c0 pg_info(1 pgs e243:3.7e2(2)) v4 2014-06-09 16:22:11.375879 7f869af78700 5 -- op tracker -- , seq: 1551, time: 2014-06-09 16:22:11.021214, event: header_read, op: pg_info(1 pgs e243:3.7e2(2)) 2014-06-09 16:22:11.375876 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.375886 7f869af78700 5 -- op tracker -- , seq: 1551, time: 2014-06-09 16:22:11.021214, event: throttled, op: pg_info(1 pgs e243:3.7e2(2)) 2014-06-09 16:22:11.375885 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.375893 7f869af78700 5 -- op tracker -- , seq: 1551, time: 2014-06-09 16:22:11.021234, event: all_read, op: pg_info(1 pgs e243:3.7e2(2)) 2014-06-09 16:22:11.375899 7f869af78700 5 -- op tracker -- , seq: 1551, time: 2014-06-09 16:22:11.375870, event: dispatched, op: pg_info(1 pgs e243:3.7e2(2)) 2014-06-09 16:22:11.375905 7f869af78700 5 -- op tracker -- , seq: 1551, time: 2014-06-09 16:22:11.375903, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7e2(2)) 2014-06-09 16:22:11.375909 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7e2(2)) v4 from osd.8 2014-06-09 16:22:11.375912 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5b6c0 2014-06-09 16:22:11.375917 7f869af78700 5 -- op tracker -- , seq: 1551, time: 2014-06-09 16:22:11.375915, event: started, op: pg_info(1 pgs e243:3.7e2(2)) 2014-06-09 16:22:11.375936 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.375934 7f869af78700 5 -- op tracker -- , seq: 1551, time: 2014-06-09 16:22:11.375931, event: done, op: pg_info(1 pgs e243:3.7e2(2)) 2014-06-09 16:22:11.375941 7f868f761700 5 queue_transactions existing osr(2.606 0x2ed1d30)/0x2ed1d30 2014-06-09 16:22:11.375942 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375943 7f868f761700 10 op_submit_start 3107 2014-06-09 16:22:11.375944 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375944 7f868f761700 5 queue_transactions (trailing journal) 3107 2014-06-09 16:22:11.375946 7f868f761700 5 queue_op 0x309a3e60 seq 3107 osr(2.606 0x2ed1d30) 2099 bytes (queue has 16 ops and 28554 bytes) 2014-06-09 16:22:11.375948 7f868f761700 10 op_submit_finish 3107 2014-06-09 16:22:11.375952 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (19 active) 2014-06-09 16:22:11.375957 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.375958 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.375960 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.375961 7f869af78700 20 osd.26 243 _dispatch 0x15999a40 pg_notify(0.77a(23),2.4ba(20) epoch 242) v5 2014-06-09 16:22:11.375967 7f869af78700 5 -- op tracker -- , seq: 1552, time: 2014-06-09 16:22:10.937627, event: header_read, op: pg_notify(0.77a(23),2.4ba(20) epoch 242) 2014-06-09 16:22:11.375961 7f868f761700 10 osd.26 243 dequeue_op 0xe2ad300 prio 63 cost 385024 latency 1.346000 osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.375975 7f869af78700 5 -- op tracker -- , seq: 1552, time: 2014-06-09 16:22:10.937628, event: throttled, op: pg_notify(0.77a(23),2.4ba(20) epoch 242) 2014-06-09 16:22:11.375763 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:2.3fb( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.375984 7f868f761700 5 -- op tracker -- , seq: 55, time: 2014-06-09 16:22:11.375978, event: reached_pg, op: osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.375997 7f869af78700 5 -- op tracker -- , seq: 1552, time: 2014-06-09 16:22:10.937664, event: all_read, op: pg_notify(0.77a(23),2.4ba(20) epoch 242) 2014-06-09 16:22:11.375994 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.376004 7f869af78700 5 -- op tracker -- , seq: 1552, time: 2014-06-09 16:22:11.375958, event: dispatched, op: pg_notify(0.77a(23),2.4ba(20) epoch 242) 2014-06-09 16:22:11.376012 7f869af78700 5 -- op tracker -- , seq: 1552, time: 2014-06-09 16:22:11.376009, event: waiting_for_osdmap, op: pg_notify(0.77a(23),2.4ba(20) epoch 242) 2014-06-09 16:22:11.376016 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.376018 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x15999a40 2014-06-09 16:22:11.376008 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0xe2ad300 2014-06-09 16:22:11.376023 7f869af78700 5 -- op tracker -- , seq: 1552, time: 2014-06-09 16:22:11.376021, event: started, op: pg_notify(0.77a(23),2.4ba(20) epoch 242) 2014-06-09 16:22:11.376027 7f868f761700 10 osd.26 243 dequeue_op 0xe2ad300 finish 2014-06-09 16:22:11.376031 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.376034 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (20 active) 2014-06-09 16:22:11.376036 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:0.633( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.376047 7f869af78700 5 -- op tracker -- , seq: 1552, time: 2014-06-09 16:22:11.376044, event: done, op: pg_notify(0.77a(23),2.4ba(20) epoch 242) 2014-06-09 16:22:11.376048 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.376056 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.376058 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.376059 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.376064 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.376075 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.376085 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.376087 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.376088 7f869af78700 20 osd.26 243 _dispatch 0x2a073e00 pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.376088 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.184369 7 0.000175 2014-06-09 16:22:11.376095 7f869af78700 5 -- op tracker -- , seq: 1553, time: 2014-06-09 16:22:11.004670, event: header_read, op: pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.376103 7f869af78700 5 -- op tracker -- , seq: 1553, time: 2014-06-09 16:22:11.004671, event: throttled, op: pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.376100 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.376110 7f869af78700 5 -- op tracker -- , seq: 1553, time: 2014-06-09 16:22:11.004683, event: all_read, op: pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.376111 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.376117 7f869af78700 5 -- op tracker -- , seq: 1553, time: 2014-06-09 16:22:11.376085, event: dispatched, op: pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.376120 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.376124 7f869af78700 5 -- op tracker -- , seq: 1553, time: 2014-06-09 16:22:11.376122, event: waiting_for_osdmap, op: pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.376129 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.376133 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a073e00 2014-06-09 16:22:11.376130 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'680 -> 213'680 2014-06-09 16:22:11.376139 7f869af78700 5 -- op tracker -- , seq: 1553, time: 2014-06-09 16:22:11.376136, event: started, op: pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.376152 7f869af78700 5 -- op tracker -- , seq: 1553, time: 2014-06-09 16:22:11.376149, event: done, op: pg_log(2.75 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.376161 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.376163 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.376175 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.376177 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.376178 7f869af78700 20 osd.26 243 _dispatch 0x15c94e00 pg_notify(3.38c(10),1.65d(23) epoch 243) v5 2014-06-09 16:22:11.376176 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.376184 7f869af78700 5 -- op tracker -- , seq: 1554, time: 2014-06-09 16:22:11.035677, event: header_read, op: pg_notify(3.38c(10),1.65d(23) epoch 243) 2014-06-09 16:22:11.376192 7f869af78700 5 -- op tracker -- , seq: 1554, time: 2014-06-09 16:22:11.035678, event: throttled, op: pg_notify(3.38c(10),1.65d(23) epoch 243) 2014-06-09 16:22:11.376198 7f869af78700 5 -- op tracker -- , seq: 1554, time: 2014-06-09 16:22:11.035718, event: all_read, op: pg_notify(3.38c(10),1.65d(23) epoch 243) 2014-06-09 16:22:11.376205 7f869af78700 5 -- op tracker -- , seq: 1554, time: 2014-06-09 16:22:11.376175, event: dispatched, op: pg_notify(3.38c(10),1.65d(23) epoch 243) 2014-06-09 16:22:11.376212 7f869af78700 5 -- op tracker -- , seq: 1554, time: 2014-06-09 16:22:11.376209, event: waiting_for_osdmap, op: pg_notify(3.38c(10),1.65d(23) epoch 243) 2014-06-09 16:22:11.376216 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.376216 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.376218 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15c94e00 2014-06-09 16:22:11.376220 7f868f761700 5 queue_transactions existing osr(3.7e2 0xb7f0120)/0xb7f0120 2014-06-09 16:22:11.376223 7f868f761700 10 op_submit_start 3108 2014-06-09 16:22:11.376224 7f868f761700 5 queue_transactions (trailing journal) 3108 2014-06-09 16:22:11.376223 7f869af78700 5 -- op tracker -- , seq: 1554, time: 2014-06-09 16:22:11.376221, event: started, op: pg_notify(3.38c(10),1.65d(23) epoch 243) 2014-06-09 16:22:11.376226 7f868f761700 5 queue_op 0x309a3e00 seq 3108 osr(3.7e2 0xb7f0120) 1335 bytes (queue has 17 ops and 29889 bytes) 2014-06-09 16:22:11.376228 7f868f761700 10 op_submit_finish 3108 2014-06-09 16:22:11.376233 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (19 active) 2014-06-09 16:22:11.376238 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.375988 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.376248 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.376244 7f868f761700 10 osd.26 243 dequeue_op 0x1804ca00 prio 63 cost 1032192 latency 0.944148 osd_op(client.4569.0:38601 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~1032192] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.376252 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.3fa( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.376266 7f868f761700 5 -- op tracker -- , seq: 374, time: 2014-06-09 16:22:11.376261, event: reached_pg, op: osd_op(client.4569.0:38601 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 450560~1032192] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.376265 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.376274 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.376281 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (19 active) 2014-06-09 16:22:11.376284 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (20 active) 2014-06-09 16:22:11.376274 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.376287 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:0.77a( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.376287 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1804ca00 2014-06-09 16:22:11.376299 7f868f761700 10 osd.26 243 dequeue_op 0x1804ca00 finish 2014-06-09 16:22:11.376303 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.376299 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.376308 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.376308 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.376312 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:2.4ba( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.376312 7f868f761700 10 osd.26 243 dequeue_op 0xe2ad200 prio 63 cost 516096 latency 1.345588 osd_op(client.4568.0:34736 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 385024~516096] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.376323 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.376332 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.376336 7f868f761700 5 -- op tracker -- , seq: 56, time: 2014-06-09 16:22:11.376330, event: reached_pg, op: osd_op(client.4568.0:34736 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 385024~516096] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.376336 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.376345 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.75( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.376343 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.376357 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.376360 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.376357 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0xe2ad200 2014-06-09 16:22:11.376369 7f868f761700 10 osd.26 243 dequeue_op 0xe2ad200 finish 2014-06-09 16:22:11.376364 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.75( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.029663 7 0.003169 2014-06-09 16:22:11.376373 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.376378 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.376379 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.75( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.376390 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.75( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.376381 7f868f761700 10 osd.26 243 dequeue_op 0x1c54bf00 prio 63 cost 516096 latency 0.878974 osd_op(client.4574.0:31825 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 819200~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.376399 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.376403 7f868f761700 5 -- op tracker -- , seq: 462, time: 2014-06-09 16:22:11.376398, event: reached_pg, op: osd_op(client.4574.0:31825 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 819200~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.376408 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.376409 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.376416 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.376423 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1c54bf00 2014-06-09 16:22:11.376436 7f868f761700 10 osd.26 243 dequeue_op 0x1c54bf00 finish 2014-06-09 16:22:11.376439 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.376444 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.376447 7f868f761700 10 osd.26 243 dequeue_op 0xe2ad100 prio 63 cost 1032192 latency 1.344925 osd_op(client.4568.0:34737 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 901120~1032192] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.376464 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.376469 7f8692767700 5 queue_transactions existing osr(2.75 0x2ed04f0)/0x2ed04f0 2014-06-09 16:22:11.376472 7f8692767700 10 op_submit_start 3109 2014-06-09 16:22:11.376468 7f868f761700 5 -- op tracker -- , seq: 63, time: 2014-06-09 16:22:11.376463, event: reached_pg, op: osd_op(client.4568.0:34737 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 901120~1032192] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.376473 7f8692767700 5 queue_transactions (trailing journal) 3109 2014-06-09 16:22:11.376474 7f8692767700 5 queue_op 0x309a2660 seq 3109 osr(2.75 0x2ed04f0) 1880 bytes (queue has 18 ops and 31769 bytes) 2014-06-09 16:22:11.376477 7f8692767700 10 op_submit_finish 3109 2014-06-09 16:22:11.376482 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (19 active) 2014-06-09 16:22:11.376474 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.376487 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.376487 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0xe2ad100 2014-06-09 16:22:11.376499 7f868f761700 10 osd.26 243 dequeue_op 0xe2ad100 finish 2014-06-09 16:22:11.376505 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.376514 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.376520 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.376526 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.376533 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0230 (19 active) 2014-06-09 16:22:11.376536 7f8691765700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.376517 7f8692767700 10 osd.26 243 dequeue_op 0x10c54500 prio 63 cost 516096 latency 1.333940 osd_op(client.4568.0:34738 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1933312~516096] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.376556 7f8692767700 5 -- op tracker -- , seq: 72, time: 2014-06-09 16:22:11.376550, event: reached_pg, op: osd_op(client.4568.0:34738 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1933312~516096] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.376564 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.376579 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x10c54500 2014-06-09 16:22:11.376590 7f8692767700 10 osd.26 243 dequeue_op 0x10c54500 finish 2014-06-09 16:22:11.376597 7f868f761700 10 osd.26 243 dequeue_op 0x10c54400 prio 63 cost 516096 latency 1.293182 osd_op(client.4568.0:34739 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2449408~516096] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.376620 7f868f761700 5 -- op tracker -- , seq: 83, time: 2014-06-09 16:22:11.376615, event: reached_pg, op: osd_op(client.4568.0:34739 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2449408~516096] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.376627 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.376641 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x10c54400 2014-06-09 16:22:11.376652 7f868f761700 10 osd.26 243 dequeue_op 0x10c54400 finish 2014-06-09 16:22:11.376651 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.376657 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.376660 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.376676 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3093.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.376666 7f868cf5c700 10 osd.26 243 dequeue_op 0x10c54300 prio 63 cost 1228800 latency 1.281624 osd_op(client.4568.0:34740 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2965504~1228800] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.376681 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.376688 7f868cf5c700 5 -- op tracker -- , seq: 95, time: 2014-06-09 16:22:11.376683, event: reached_pg, op: osd_op(client.4568.0:34740 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2965504~1228800] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.376696 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.376710 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x10c54300 2014-06-09 16:22:11.376721 7f868cf5c700 10 osd.26 243 dequeue_op 0x10c54300 finish 2014-06-09 16:22:11.376724 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (17 active) 2014-06-09 16:22:11.376727 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.376728 7f86a5610700 10 _do_op 0x309a2f00 seq 3093 r = 0, finisher 0xc5a7c80 0 2014-06-09 16:22:11.376732 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (16 active) 2014-06-09 16:22:11.376735 7f8692767700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.376736 7f86a5610700 10 _finish_op 0x309a2f00 seq 3093 osr(1.67a 0x384de90)/0x384de90 2014-06-09 16:22:11.376742 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114480 (1 active) 2014-06-09 16:22:11.376744 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a7c80] 2014-06-09 16:22:11.376745 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34978c0 (2 active) 2014-06-09 16:22:11.376748 7f86a5610700 5 _do_op 0x16096720 seq 3094 osr(3.6d3 0x3ed1c50)/0x3ed1c50 start 2014-06-09 16:22:11.376750 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a7c80] 2014-06-09 16:22:11.376751 7f86a5610700 10 _do_transaction on 0xc130fc0 2014-06-09 16:22:11.376751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.376753 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.376759 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.376809 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.376827 7f86a5610700 15 _omap_rmkeyrange meta/1fe79515/pglog_3.6d3/0//-1 [0000000214.00000000000000000630,4294967295.18446744073709551615] 2014-06-09 16:22:11.376830 7f86a5610700 15 get_omap_iterator meta/1fe79515/pglog_3.6d3/0//-1 2014-06-09 16:22:11.376916 7f86a5610700 15 _omap_rmkeys meta/1fe79515/pglog_3.6d3/0//-1 2014-06-09 16:22:11.376950 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe79515/pglog_3.6d3/0//-1 2014-06-09 16:22:11.376960 7f86a5610700 10 _omap_rmkeys meta/1fe79515/pglog_3.6d3/0//-1 r = 0 2014-06-09 16:22:11.376965 7f86a5610700 15 _omap_rmkeys meta/1fe79515/pglog_3.6d3/0//-1 2014-06-09 16:22:11.376969 7f86a5610700 10 _omap_rmkeys meta/1fe79515/pglog_3.6d3/0//-1 r = 0 2014-06-09 16:22:11.376978 7f86a5610700 15 _omap_setkeys meta/1fe79515/pglog_3.6d3/0//-1 2014-06-09 16:22:11.376996 7f86a5610700 10 cid: oid: 1fe79515/pglog_3.6d3/0//-1 skipping op, spos 3094.0.0 <= header.spos 150427.0.0 2014-06-09 16:22:11.377002 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3094.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377005 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377033 7f86a5610700 10 _do_op 0x16096720 seq 3094 r = 0, finisher 0x3050b4c0 0 2014-06-09 16:22:11.377037 7f86a5610700 10 _finish_op 0x16096720 seq 3094 osr(3.6d3 0x3ed1c50)/0x3ed1c50 2014-06-09 16:22:11.377042 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34978c0 (1 active) 2014-06-09 16:22:11.377044 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52fe60 (2 active) 2014-06-09 16:22:11.377045 7f86a5610700 5 _do_op 0x309a2e40 seq 3095 osr(2.73 0x2ed0510)/0x2ed0510 start 2014-06-09 16:22:11.377045 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050b4c0] 2014-06-09 16:22:11.377048 7f86a5610700 10 _do_transaction on 0xbfe6240 2014-06-09 16:22:11.377054 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.377065 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050b4c0] 2014-06-09 16:22:11.377067 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.377068 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.377087 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.377105 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3095.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377109 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377133 7f86a5610700 10 _do_op 0x309a2e40 seq 3095 r = 0, finisher 0x18743260 0 2014-06-09 16:22:11.377137 7f86a5610700 10 _finish_op 0x309a2e40 seq 3095 osr(2.73 0x2ed0510)/0x2ed0510 2014-06-09 16:22:11.377143 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52fe60 (1 active) 2014-06-09 16:22:11.377145 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300ea0 (2 active) 2014-06-09 16:22:11.377145 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x18743260] 2014-06-09 16:22:11.377147 7f86a5610700 5 _do_op 0x309a2de0 seq 3096 osr(1.82 0x384c4d0)/0x384c4d0 start 2014-06-09 16:22:11.377150 7f86a5610700 10 _do_transaction on 0xc244d80 2014-06-09 16:22:11.377152 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x18743260] 2014-06-09 16:22:11.377153 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.377155 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.377155 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.377188 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.377203 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3096.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377208 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377226 7f86a5610700 10 _do_op 0x309a2de0 seq 3096 r = 0, finisher 0xc6e8040 0 2014-06-09 16:22:11.377230 7f86a5610700 10 _finish_op 0x309a2de0 seq 3096 osr(1.82 0x384c4d0)/0x384c4d0 2014-06-09 16:22:11.377235 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300ea0 (1 active) 2014-06-09 16:22:11.377237 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114000 (2 active) 2014-06-09 16:22:11.377237 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8040] 2014-06-09 16:22:11.377240 7f86a5610700 5 _do_op 0x309a2d80 seq 3097 osr(1.631 0x384dfb0)/0x384dfb0 start 2014-06-09 16:22:11.377242 7f86a5610700 10 _do_transaction on 0xc3eccc0 2014-06-09 16:22:11.377245 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8040] 2014-06-09 16:22:11.377247 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.377248 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.377248 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.377279 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.377296 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3097.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377300 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377326 7f86a5610700 10 _do_op 0x309a2d80 seq 3097 r = 0, finisher 0x3061f180 0 2014-06-09 16:22:11.377331 7f86a5610700 10 _finish_op 0x309a2d80 seq 3097 osr(1.631 0x384dfb0)/0x384dfb0 2014-06-09 16:22:11.377336 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114000 (1 active) 2014-06-09 16:22:11.377338 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff59e0 (2 active) 2014-06-09 16:22:11.377338 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061f180] 2014-06-09 16:22:11.377341 7f86a5610700 5 _do_op 0x308cac00 seq 3098 osr(1.604 0x384dc50)/0x384dc50 start 2014-06-09 16:22:11.377343 7f86a5610700 10 _do_transaction on 0xc2d9140 2014-06-09 16:22:11.377344 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061f180] 2014-06-09 16:22:11.377345 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.377346 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.377349 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.377382 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.377399 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3098.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377404 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377424 7f86a5610700 10 _do_op 0x308cac00 seq 3098 r = 0, finisher 0xc6e8aa0 0 2014-06-09 16:22:11.377427 7f86a5610700 10 _finish_op 0x308cac00 seq 3098 osr(1.604 0x384dc50)/0x384dc50 2014-06-09 16:22:11.377433 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff59e0 (1 active) 2014-06-09 16:22:11.377435 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f320 (2 active) 2014-06-09 16:22:11.377435 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8aa0] 2014-06-09 16:22:11.377437 7f86a5610700 5 _do_op 0x309a2d20 seq 3099 osr(0.4da 0x2bd75f0)/0x2bd75f0 start 2014-06-09 16:22:11.377440 7f86a5610700 10 _do_transaction on 0xc0ec540 2014-06-09 16:22:11.377444 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8aa0] 2014-06-09 16:22:11.377445 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.377446 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.377446 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.377478 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.377495 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3099.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377499 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377520 7f86a5610700 10 _do_op 0x309a2d20 seq 3099 r = 0, finisher 0xc6e8060 0 2014-06-09 16:22:11.377524 7f86a5610700 10 _finish_op 0x309a2d20 seq 3099 osr(0.4da 0x2bd75f0)/0x2bd75f0 2014-06-09 16:22:11.377530 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f320 (1 active) 2014-06-09 16:22:11.377533 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406a20 (2 active) 2014-06-09 16:22:11.377532 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8060] 2014-06-09 16:22:11.377535 7f86a5610700 5 _do_op 0x16fd5500 seq 3100 osr(3.94 0x3ed0640)/0x3ed0640 start 2014-06-09 16:22:11.377538 7f86a5610700 10 _do_transaction on 0x39dbd40 2014-06-09 16:22:11.377538 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8060] 2014-06-09 16:22:11.377540 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.377541 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.377544 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.377575 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.377591 7f86a5610700 15 _omap_rmkeyrange meta/5168319b/pglog_3.94/0//-1 [0000000214.00000000000000000567,4294967295.18446744073709551615] 2014-06-09 16:22:11.377594 7f86a5610700 15 get_omap_iterator meta/5168319b/pglog_3.94/0//-1 2014-06-09 16:22:11.377681 7f86a5610700 15 _omap_rmkeys meta/5168319b/pglog_3.94/0//-1 2014-06-09 16:22:11.377714 7f86a5610700 10 lookup_strip_header done cid meta oid 5168319b/pglog_3.94/0//-1 2014-06-09 16:22:11.377724 7f86a5610700 10 _omap_rmkeys meta/5168319b/pglog_3.94/0//-1 r = 0 2014-06-09 16:22:11.377730 7f86a5610700 15 _omap_rmkeys meta/5168319b/pglog_3.94/0//-1 2014-06-09 16:22:11.377735 7f86a5610700 10 _omap_rmkeys meta/5168319b/pglog_3.94/0//-1 r = 0 2014-06-09 16:22:11.377747 7f86a5610700 15 _omap_setkeys meta/5168319b/pglog_3.94/0//-1 2014-06-09 16:22:11.377767 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3100.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377774 7f86a5610700 10 cid: oid: 5168319b/pglog_3.94/0//-1 skipping op, spos 3100.0.0 <= header.spos 150430.0.0 2014-06-09 16:22:11.377778 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377817 7f86a5610700 10 _do_op 0x16fd5500 seq 3100 r = 0, finisher 0x3062b7a0 0 2014-06-09 16:22:11.377821 7f86a5610700 10 _finish_op 0x16fd5500 seq 3100 osr(3.94 0x3ed0640)/0x3ed0640 2014-06-09 16:22:11.377826 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406a20 (1 active) 2014-06-09 16:22:11.377829 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb320 (2 active) 2014-06-09 16:22:11.377829 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062b7a0] 2014-06-09 16:22:11.377831 7f86a5610700 5 _do_op 0x30606300 seq 3101 osr(3.62a 0x3ed1aa0)/0x3ed1aa0 start 2014-06-09 16:22:11.377834 7f86a5610700 10 _do_transaction on 0xc3fcb40 2014-06-09 16:22:11.377840 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.377844 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062b7a0] 2014-06-09 16:22:11.377846 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.377847 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.377874 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.377889 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3101.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377893 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377908 7f86a5610700 10 _do_op 0x30606300 seq 3101 r = 0, finisher 0xbd74240 0 2014-06-09 16:22:11.377914 7f86a5610700 10 _finish_op 0x30606300 seq 3101 osr(3.62a 0x3ed1aa0)/0x3ed1aa0 2014-06-09 16:22:11.377919 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb320 (1 active) 2014-06-09 16:22:11.377921 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4066c0 (2 active) 2014-06-09 16:22:11.377921 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd74240] 2014-06-09 16:22:11.377923 7f86a5610700 5 _do_op 0x308cb380 seq 3102 osr(3.9b 0x3ed0630)/0x3ed0630 start 2014-06-09 16:22:11.377926 7f86a5610700 10 _do_transaction on 0xbf39a40 2014-06-09 16:22:11.377928 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd74240] 2014-06-09 16:22:11.377930 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.377931 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.377931 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.377963 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.377979 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3102.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.377983 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.377998 7f86a5610700 10 _do_op 0x308cb380 seq 3102 r = 0, finisher 0x33a6680 0 2014-06-09 16:22:11.378001 7f86a5610700 10 _finish_op 0x308cb380 seq 3102 osr(3.9b 0x3ed0630)/0x3ed0630 2014-06-09 16:22:11.378007 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4066c0 (1 active) 2014-06-09 16:22:11.378009 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f8c0 (2 active) 2014-06-09 16:22:11.378009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a6680] 2014-06-09 16:22:11.378011 7f86a5610700 5 _do_op 0x2c06e8a0 seq 3103 osr(1.91 0x384c430)/0x384c430 start 2014-06-09 16:22:11.378014 7f86a5610700 10 _do_transaction on 0xc30da40 2014-06-09 16:22:11.378014 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a6680] 2014-06-09 16:22:11.378015 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.378016 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.378020 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.378052 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.378067 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3103.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.378072 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.378096 7f86a5610700 10 _do_op 0x2c06e8a0 seq 3103 r = 0, finisher 0x2d738e40 0 2014-06-09 16:22:11.378100 7f86a5610700 10 _finish_op 0x2c06e8a0 seq 3103 osr(1.91 0x384c430)/0x384c430 2014-06-09 16:22:11.378104 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f8c0 (1 active) 2014-06-09 16:22:11.378107 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114ea0 (2 active) 2014-06-09 16:22:11.378106 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d738e40] 2014-06-09 16:22:11.378109 7f86a5610700 5 _do_op 0x30606f60 seq 3104 osr(0.301 0x2bd6e10)/0x2bd6e10 start 2014-06-09 16:22:11.378111 7f86a5610700 10 _do_transaction on 0x3085440 2014-06-09 16:22:11.378112 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d738e40] 2014-06-09 16:22:11.378114 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.378115 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.378117 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.378151 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.378167 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3104.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.378171 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.378188 7f86a5610700 10 _do_op 0x30606f60 seq 3104 r = 0, finisher 0x3062b240 0 2014-06-09 16:22:11.378190 7f86a5610700 10 _finish_op 0x30606f60 seq 3104 osr(0.301 0x2bd6e10)/0x2bd6e10 2014-06-09 16:22:11.378195 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114ea0 (1 active) 2014-06-09 16:22:11.378197 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339c20 (2 active) 2014-06-09 16:22:11.378198 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062b240] 2014-06-09 16:22:11.378199 7f86a5610700 5 _do_op 0x30606f00 seq 3105 osr(1.76 0x384c530)/0x384c530 start 2014-06-09 16:22:11.378201 7f86a5610700 10 _do_transaction on 0x30d12c0 2014-06-09 16:22:11.378206 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062b240] 2014-06-09 16:22:11.378207 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.378206 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.378208 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.378239 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.378255 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3105.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.378259 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.378283 7f86a5610700 10 _do_op 0x30606f00 seq 3105 r = 0, finisher 0xc2d6800 0 2014-06-09 16:22:11.378286 7f86a5610700 10 _finish_op 0x30606f00 seq 3105 osr(1.76 0x384c530)/0x384c530 2014-06-09 16:22:11.378294 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339c20 (1 active) 2014-06-09 16:22:11.378294 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d6800] 2014-06-09 16:22:11.378297 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358120 (2 active) 2014-06-09 16:22:11.378299 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d6800] 2014-06-09 16:22:11.378299 7f86a5610700 5 _do_op 0x309a3ec0 seq 3106 osr(3.31c 0x3ed0d40)/0x3ed0d40 start 2014-06-09 16:22:11.378301 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.378301 7f86a5610700 10 _do_transaction on 0xbeba6c0 2014-06-09 16:22:11.378302 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.378307 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.378340 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.378356 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3106.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.378359 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.378380 7f86a5610700 10 _do_op 0x309a3ec0 seq 3106 r = 0, finisher 0xbd698c0 0 2014-06-09 16:22:11.378383 7f86a5610700 10 _finish_op 0x309a3ec0 seq 3106 osr(3.31c 0x3ed0d40)/0x3ed0d40 2014-06-09 16:22:11.378388 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358120 (1 active) 2014-06-09 16:22:11.378390 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52fc20 (2 active) 2014-06-09 16:22:11.378391 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd698c0] 2014-06-09 16:22:11.378392 7f86a5610700 5 _do_op 0x309a3e60 seq 3107 osr(2.606 0x2ed1d30)/0x2ed1d30 start 2014-06-09 16:22:11.378395 7f86a5610700 10 _do_transaction on 0xc245d40 2014-06-09 16:22:11.378399 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.378400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd698c0] 2014-06-09 16:22:11.378402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.378403 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.378431 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.378447 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3107.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.378451 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.378472 7f86a5610700 10 _do_op 0x309a3e60 seq 3107 r = 0, finisher 0xbdf0b00 0 2014-06-09 16:22:11.378475 7f86a5610700 10 _finish_op 0x309a3e60 seq 3107 osr(2.606 0x2ed1d30)/0x2ed1d30 2014-06-09 16:22:11.378481 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52fc20 (1 active) 2014-06-09 16:22:11.378483 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286c60 (2 active) 2014-06-09 16:22:11.378482 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdf0b00] 2014-06-09 16:22:11.378485 7f86a5610700 5 _do_op 0x309a3e00 seq 3108 osr(3.7e2 0xb7f0120)/0xb7f0120 start 2014-06-09 16:22:11.378487 7f86a5610700 10 _do_transaction on 0xc05e000 2014-06-09 16:22:11.378488 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdf0b00] 2014-06-09 16:22:11.378490 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.378491 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.378492 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.378525 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.378540 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3108.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.378545 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.378559 7f86a5610700 10 _do_op 0x309a3e00 seq 3108 r = 0, finisher 0x33a74e0 0 2014-06-09 16:22:11.378561 7f86a5610700 10 _finish_op 0x309a3e00 seq 3108 osr(3.7e2 0xb7f0120)/0xb7f0120 2014-06-09 16:22:11.378566 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286c60 (1 active) 2014-06-09 16:22:11.378568 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd0e0 (2 active) 2014-06-09 16:22:11.378568 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a74e0] 2014-06-09 16:22:11.378570 7f86a5610700 5 _do_op 0x309a2660 seq 3109 osr(2.75 0x2ed04f0)/0x2ed04f0 start 2014-06-09 16:22:11.378572 7f86a5610700 10 _do_transaction on 0x3a07080 2014-06-09 16:22:11.378574 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a74e0] 2014-06-09 16:22:11.378575 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.378576 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.378577 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.378610 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.378625 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3109.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.378630 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.378633 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (17 active) 2014-06-09 16:22:11.378654 7f86a5610700 10 _do_op 0x309a2660 seq 3109 r = 0, finisher 0x1a0030a0 0 2014-06-09 16:22:11.378658 7f86a5610700 10 _finish_op 0x309a2660 seq 3109 osr(2.75 0x2ed04f0)/0x2ed04f0 2014-06-09 16:22:11.378663 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd0e0 (1 active) 2014-06-09 16:22:11.378665 7f86a5610700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.378660 7f869af78700 5 -- op tracker -- , seq: 1554, time: 2014-06-09 16:22:11.378657, event: done, op: pg_notify(3.38c(10),1.65d(23) epoch 243) 2014-06-09 16:22:11.378670 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.378672 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.378671 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcdb35a0,0x3050b820,0xbd6c320,0x1b0c1e60,0x2d738140,0xbd6e760,0x3061e920,0x3062b700,0x30628b20,0x30508780,0x30628320,0x305086a0,0x13965760,0xbd6e820,0xbccc400,0xcd9e680,0x33a62a0] 2014-06-09 16:22:11.378677 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.378695 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.378695 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.378697 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.378698 7f869af78700 20 osd.26 243 _dispatch 0x230bb880 pg_notify(3.458(12),3.72b(12),2.5ad(22) epoch 242) v5 2014-06-09 16:22:11.378705 7f869af78700 5 -- op tracker -- , seq: 1555, time: 2014-06-09 16:22:10.863773, event: header_read, op: pg_notify(3.458(12),3.72b(12),2.5ad(22) epoch 242) 2014-06-09 16:22:11.378714 7f869af78700 5 -- op tracker -- , seq: 1555, time: 2014-06-09 16:22:10.863774, event: throttled, op: pg_notify(3.458(12),3.72b(12),2.5ad(22) epoch 242) 2014-06-09 16:22:11.378716 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.378721 7f869af78700 5 -- op tracker -- , seq: 1555, time: 2014-06-09 16:22:10.863808, event: all_read, op: pg_notify(3.458(12),3.72b(12),2.5ad(22) epoch 242) 2014-06-09 16:22:11.378729 7f869af78700 5 -- op tracker -- , seq: 1555, time: 2014-06-09 16:22:11.378695, event: dispatched, op: pg_notify(3.458(12),3.72b(12),2.5ad(22) epoch 242) 2014-06-09 16:22:11.378736 7f869af78700 5 -- op tracker -- , seq: 1555, time: 2014-06-09 16:22:11.378734, event: waiting_for_osdmap, op: pg_notify(3.458(12),3.72b(12),2.5ad(22) epoch 242) 2014-06-09 16:22:11.378741 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.378743 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x230bb880 2014-06-09 16:22:11.378732 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 213 same_interval_since 242 2014-06-09 16:22:11.378748 7f869af78700 5 -- op tracker -- , seq: 1555, time: 2014-06-09 16:22:11.378746, event: started, op: pg_notify(3.458(12),3.72b(12),2.5ad(22) epoch 242) 2014-06-09 16:22:11.378752 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.378768 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.378782 7f869af78700 5 -- op tracker -- , seq: 1555, time: 2014-06-09 16:22:11.378779, event: done, op: pg_notify(3.458(12),3.72b(12),2.5ad(22) epoch 242) 2014-06-09 16:22:11.378782 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.378791 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.378793 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.378796 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.378817 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.378819 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.378820 7f869af78700 20 osd.26 243 _dispatch 0x2a0701c0 pg_notify(2.587(20) epoch 242) v5 2014-06-09 16:22:11.378821 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.378825 7f869af78700 5 -- op tracker -- , seq: 1556, time: 2014-06-09 16:22:10.803359, event: header_read, op: pg_notify(2.587(20) epoch 242) 2014-06-09 16:22:11.378833 7f869af78700 5 -- op tracker -- , seq: 1556, time: 2014-06-09 16:22:10.803360, event: throttled, op: pg_notify(2.587(20) epoch 242) 2014-06-09 16:22:11.378839 7f869af78700 5 -- op tracker -- , seq: 1556, time: 2014-06-09 16:22:10.803386, event: all_read, op: pg_notify(2.587(20) epoch 242) 2014-06-09 16:22:11.378835 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.378845 7f869af78700 5 -- op tracker -- , seq: 1556, time: 2014-06-09 16:22:11.378817, event: dispatched, op: pg_notify(2.587(20) epoch 242) 2014-06-09 16:22:11.378847 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.378852 7f869af78700 5 -- op tracker -- , seq: 1556, time: 2014-06-09 16:22:11.378850, event: waiting_for_osdmap, op: pg_notify(2.587(20) epoch 242) 2014-06-09 16:22:11.378856 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.378859 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2a0701c0 2014-06-09 16:22:11.378863 7f869af78700 5 -- op tracker -- , seq: 1556, time: 2014-06-09 16:22:11.378862, event: started, op: pg_notify(2.587(20) epoch 242) 2014-06-09 16:22:11.378869 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.378879 7f869af78700 5 -- op tracker -- , seq: 1556, time: 2014-06-09 16:22:11.378877, event: done, op: pg_notify(2.587(20) epoch 242) 2014-06-09 16:22:11.378887 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.378889 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.378882 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.378895 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.378908 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.378909 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.378910 7f869af78700 20 osd.26 243 _dispatch 0x304f9000 pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.378911 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.378917 7f869af78700 5 -- op tracker -- , seq: 1557, time: 2014-06-09 16:22:11.077803, event: header_read, op: pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.378926 7f869af78700 5 -- op tracker -- , seq: 1557, time: 2014-06-09 16:22:11.077804, event: throttled, op: pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.378932 7f869af78700 5 -- op tracker -- , seq: 1557, time: 2014-06-09 16:22:11.077822, event: all_read, op: pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.378923 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.378939 7f869af78700 5 -- op tracker -- , seq: 1557, time: 2014-06-09 16:22:11.378907, event: dispatched, op: pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.378941 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.378947 7f869af78700 5 -- op tracker -- , seq: 1557, time: 2014-06-09 16:22:11.378944, event: waiting_for_osdmap, op: pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.378951 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.378955 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f9000 2014-06-09 16:22:11.378953 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.378961 7f869af78700 5 -- op tracker -- , seq: 1557, time: 2014-06-09 16:22:11.378958, event: started, op: pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.378966 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.378975 7f869af78700 5 -- op tracker -- , seq: 1557, time: 2014-06-09 16:22:11.378971, event: done, op: pg_log(1.676 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.378980 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.378982 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.378984 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.378996 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.378997 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.378998 7f869af78700 20 osd.26 243 _dispatch 0x16064380 pg_notify(0.37e(23),3.c1(10) epoch 243) v5 2014-06-09 16:22:11.378994 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 active] flushed 2014-06-09 16:22:11.379004 7f869af78700 5 -- op tracker -- , seq: 1558, time: 2014-06-09 16:22:11.051025, event: header_read, op: pg_notify(0.37e(23),3.c1(10) epoch 243) 2014-06-09 16:22:11.379012 7f869af78700 5 -- op tracker -- , seq: 1558, time: 2014-06-09 16:22:11.051026, event: throttled, op: pg_notify(0.37e(23),3.c1(10) epoch 243) 2014-06-09 16:22:11.379006 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] _activate_committed 243 peer_activated now 26 last_epoch_started 213 same_interval_since 242 2014-06-09 16:22:11.379018 7f869af78700 5 -- op tracker -- , seq: 1558, time: 2014-06-09 16:22:11.051056, event: all_read, op: pg_notify(0.37e(23),3.c1(10) epoch 243) 2014-06-09 16:22:11.379025 7f869af78700 5 -- op tracker -- , seq: 1558, time: 2014-06-09 16:22:11.378995, event: dispatched, op: pg_notify(0.37e(23),3.c1(10) epoch 243) 2014-06-09 16:22:11.379022 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] flushed 2014-06-09 16:22:11.379033 7f869af78700 5 -- op tracker -- , seq: 1558, time: 2014-06-09 16:22:11.379030, event: waiting_for_osdmap, op: pg_notify(0.37e(23),3.c1(10) epoch 243) 2014-06-09 16:22:11.379037 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.379039 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16064380 2014-06-09 16:22:11.379044 7f869af78700 5 -- op tracker -- , seq: 1558, time: 2014-06-09 16:22:11.379042, event: started, op: pg_notify(0.37e(23),3.c1(10) epoch 243) 2014-06-09 16:22:11.379037 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] flushed 2014-06-09 16:22:11.379062 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.379068 7f869af78700 5 -- op tracker -- , seq: 1558, time: 2014-06-09 16:22:11.379066, event: done, op: pg_notify(0.37e(23),3.c1(10) epoch 243) 2014-06-09 16:22:11.379077 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379079 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379077 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.379090 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.379101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379103 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379104 7f869af78700 20 osd.26 243 _dispatch 0x3004ea00 pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.379103 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.379111 7f869af78700 5 -- op tracker -- , seq: 1559, time: 2014-06-09 16:22:11.071015, event: header_read, op: pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379117 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.379119 7f869af78700 5 -- op tracker -- , seq: 1559, time: 2014-06-09 16:22:11.071016, event: throttled, op: pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379126 7f869af78700 5 -- op tracker -- , seq: 1559, time: 2014-06-09 16:22:11.071028, event: all_read, op: pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379128 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.379133 7f869af78700 5 -- op tracker -- , seq: 1559, time: 2014-06-09 16:22:11.379101, event: dispatched, op: pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379141 7f869af78700 5 -- op tracker -- , seq: 1559, time: 2014-06-09 16:22:11.379138, event: waiting_for_osdmap, op: pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379146 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.379141 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.379150 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004ea00 2014-06-09 16:22:11.379153 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.379155 7f869af78700 5 -- op tracker -- , seq: 1559, time: 2014-06-09 16:22:11.379153, event: started, op: pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379169 7f869af78700 5 -- op tracker -- , seq: 1559, time: 2014-06-09 16:22:11.379166, event: done, op: pg_log(2.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379169 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.379176 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379178 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379182 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.379191 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379192 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379194 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.379193 7f869af78700 20 osd.26 243 _dispatch 0x2a075600 pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.379200 7f869af78700 5 -- op tracker -- , seq: 1560, time: 2014-06-09 16:22:11.007316, event: header_read, op: pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379204 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.379208 7f869af78700 5 -- op tracker -- , seq: 1560, time: 2014-06-09 16:22:11.007316, event: throttled, op: pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379215 7f869af78700 5 -- op tracker -- , seq: 1560, time: 2014-06-09 16:22:11.007331, event: all_read, op: pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379216 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.379222 7f869af78700 5 -- op tracker -- , seq: 1560, time: 2014-06-09 16:22:11.379190, event: dispatched, op: pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379227 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.379230 7f869af78700 5 -- op tracker -- , seq: 1560, time: 2014-06-09 16:22:11.379227, event: waiting_for_osdmap, op: pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379234 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.379238 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a075600 2014-06-09 16:22:11.379239 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.379244 7f869af78700 5 -- op tracker -- , seq: 1560, time: 2014-06-09 16:22:11.379241, event: started, op: pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379251 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.379257 7f869af78700 5 -- op tracker -- , seq: 1560, time: 2014-06-09 16:22:11.379254, event: done, op: pg_log(2.27 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379266 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.379266 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379268 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379281 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379282 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379277 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.379283 7f869af78700 20 osd.26 243 _dispatch 0x2fe33e00 pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.379290 7f869af78700 5 -- op tracker -- , seq: 1561, time: 2014-06-09 16:22:11.021244, event: header_read, op: pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379290 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.379298 7f869af78700 5 -- op tracker -- , seq: 1561, time: 2014-06-09 16:22:11.021244, event: throttled, op: pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379301 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.379305 7f869af78700 5 -- op tracker -- , seq: 1561, time: 2014-06-09 16:22:11.021256, event: all_read, op: pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379312 7f869af78700 5 -- op tracker -- , seq: 1561, time: 2014-06-09 16:22:11.379280, event: dispatched, op: pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379315 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.379320 7f869af78700 5 -- op tracker -- , seq: 1561, time: 2014-06-09 16:22:11.379317, event: waiting_for_osdmap, op: pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379324 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.379328 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe33e00 2014-06-09 16:22:11.379327 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.379334 7f869af78700 5 -- op tracker -- , seq: 1561, time: 2014-06-09 16:22:11.379331, event: started, op: pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379339 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.379347 7f869af78700 5 -- op tracker -- , seq: 1561, time: 2014-06-09 16:22:11.379344, event: done, op: pg_log(2.4e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379355 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379357 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379350 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.379365 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcdb35a0,0x3050b820,0xbd6c320,0x1b0c1e60,0x2d738140,0xbd6e760,0x3061e920,0x3062b700,0x30628b20,0x30508780,0x30628320,0x305086a0,0x13965760,0xbd6e820,0xbccc400,0xcd9e680,0x33a62a0] 2014-06-09 16:22:11.379369 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.379369 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379370 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.379371 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379372 7f869af78700 20 osd.26 243 _dispatch 0x240ea000 pg_notify(3.1db(10),3.491(10) epoch 242) v5 2014-06-09 16:22:11.379378 7f869af78700 5 -- op tracker -- , seq: 1562, time: 2014-06-09 16:22:10.945800, event: header_read, op: pg_notify(3.1db(10),3.491(10) epoch 242) 2014-06-09 16:22:11.379385 7f869af78700 5 -- op tracker -- , seq: 1562, time: 2014-06-09 16:22:10.945801, event: throttled, op: pg_notify(3.1db(10),3.491(10) epoch 242) 2014-06-09 16:22:11.379391 7f869af78700 5 -- op tracker -- , seq: 1562, time: 2014-06-09 16:22:10.945825, event: all_read, op: pg_notify(3.1db(10),3.491(10) epoch 242) 2014-06-09 16:22:11.379398 7f869af78700 5 -- op tracker -- , seq: 1562, time: 2014-06-09 16:22:11.379369, event: dispatched, op: pg_notify(3.1db(10),3.491(10) epoch 242) 2014-06-09 16:22:11.379405 7f869af78700 5 -- op tracker -- , seq: 1562, time: 2014-06-09 16:22:11.379402, event: waiting_for_osdmap, op: pg_notify(3.1db(10),3.491(10) epoch 242) 2014-06-09 16:22:11.379409 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.379411 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240ea000 2014-06-09 16:22:11.379416 7f869af78700 5 -- op tracker -- , seq: 1562, time: 2014-06-09 16:22:11.379414, event: started, op: pg_notify(3.1db(10),3.491(10) epoch 242) 2014-06-09 16:22:11.379443 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (16 active) 2014-06-09 16:22:11.379442 7f869af78700 5 -- op tracker -- , seq: 1562, time: 2014-06-09 16:22:11.379439, event: done, op: pg_notify(3.1db(10),3.491(10) epoch 242) 2014-06-09 16:22:11.379451 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379453 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379457 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0250 (17 active) 2014-06-09 16:22:11.379469 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379471 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379461 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.38c( v 213'596 (0'0,213'596] local-les=206 n=114 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.379472 7f869af78700 20 osd.26 243 _dispatch 0x3046a700 pg_notify(1.86(26) epoch 243) v5 2014-06-09 16:22:11.379477 7f869af78700 5 -- op tracker -- , seq: 1563, time: 2014-06-09 16:22:11.029886, event: header_read, op: pg_notify(1.86(26) epoch 243) 2014-06-09 16:22:11.379476 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.379485 7f869af78700 5 -- op tracker -- , seq: 1563, time: 2014-06-09 16:22:11.029886, event: throttled, op: pg_notify(1.86(26) epoch 243) 2014-06-09 16:22:11.379490 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379491 7f869af78700 5 -- op tracker -- , seq: 1563, time: 2014-06-09 16:22:11.029907, event: all_read, op: pg_notify(1.86(26) epoch 243) 2014-06-09 16:22:11.379497 7f869af78700 5 -- op tracker -- , seq: 1563, time: 2014-06-09 16:22:11.379469, event: dispatched, op: pg_notify(1.86(26) epoch 243) 2014-06-09 16:22:11.379495 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:1.65d( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.379503 7f869af78700 5 -- op tracker -- , seq: 1563, time: 2014-06-09 16:22:11.379501, event: waiting_for_osdmap, op: pg_notify(1.86(26) epoch 243) 2014-06-09 16:22:11.379507 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.379509 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046a700 2014-06-09 16:22:11.379514 7f869af78700 5 -- op tracker -- , seq: 1563, time: 2014-06-09 16:22:11.379512, event: started, op: pg_notify(1.86(26) epoch 243) 2014-06-09 16:22:11.379508 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.379527 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379529 7f869af78700 5 -- op tracker -- , seq: 1563, time: 2014-06-09 16:22:11.379527, event: done, op: pg_notify(1.86(26) epoch 243) 2014-06-09 16:22:11.379530 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.379541 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379543 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379541 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.379551 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379554 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:3.458( v 213'610 (0'0,213'610] local-les=191 n=125 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.379564 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379566 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379567 7f869af78700 20 osd.26 243 _dispatch 0x2a074a00 pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.379568 7f8690763700 7 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.379573 7f869af78700 5 -- op tracker -- , seq: 1564, time: 2014-06-09 16:22:11.058900, event: header_read, op: pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379581 7f869af78700 5 -- op tracker -- , seq: 1564, time: 2014-06-09 16:22:11.058901, event: throttled, op: pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379588 7f869af78700 5 -- op tracker -- , seq: 1564, time: 2014-06-09 16:22:11.058910, event: all_read, op: pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379578 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.458( v 213'610 (0'0,213'610] local-les=191 n=125 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.379593 7f869af78700 5 -- op tracker -- , seq: 1564, time: 2014-06-09 16:22:11.379564, event: dispatched, op: pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379597 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379600 7f869af78700 5 -- op tracker -- , seq: 1564, time: 2014-06-09 16:22:11.379597, event: waiting_for_osdmap, op: pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379603 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.379607 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a074a00 2014-06-09 16:22:11.379601 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.379611 7f869af78700 5 -- op tracker -- , seq: 1564, time: 2014-06-09 16:22:11.379609, event: started, op: pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379622 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379629 7f869af78700 5 -- op tracker -- , seq: 1564, time: 2014-06-09 16:22:11.379626, event: done, op: pg_log(2.6cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379626 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:3.72b( v 213'642 (0'0,213'642] local-les=191 n=121 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.379636 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379637 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379638 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.379647 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379649 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379649 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379650 7f869af78700 20 osd.26 243 _dispatch 0x1fc04c40 pg_notify(1.16d(22) epoch 242) v5 2014-06-09 16:22:11.379655 7f869af78700 5 -- op tracker -- , seq: 1565, time: 2014-06-09 16:22:10.886358, event: header_read, op: pg_notify(1.16d(22) epoch 242) 2014-06-09 16:22:11.379660 7f869af78700 5 -- op tracker -- , seq: 1565, time: 2014-06-09 16:22:10.886359, event: throttled, op: pg_notify(1.16d(22) epoch 242) 2014-06-09 16:22:11.379666 7f869af78700 5 -- op tracker -- , seq: 1565, time: 2014-06-09 16:22:10.886386, event: all_read, op: pg_notify(1.16d(22) epoch 242) 2014-06-09 16:22:11.379653 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:2.5ad( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.379672 7f869af78700 5 -- op tracker -- , seq: 1565, time: 2014-06-09 16:22:11.379647, event: dispatched, op: pg_notify(1.16d(22) epoch 242) 2014-06-09 16:22:11.379672 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.379678 7f869af78700 5 -- op tracker -- , seq: 1565, time: 2014-06-09 16:22:11.379676, event: waiting_for_osdmap, op: pg_notify(1.16d(22) epoch 242) 2014-06-09 16:22:11.379682 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379682 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.379684 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1fc04c40 2014-06-09 16:22:11.379689 7f869af78700 5 -- op tracker -- , seq: 1565, time: 2014-06-09 16:22:11.379687, event: started, op: pg_notify(1.16d(22) epoch 242) 2014-06-09 16:22:11.379685 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.379698 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 2014-06-09 16:22:11.379704 7f869af78700 5 -- op tracker -- , seq: 1565, time: 2014-06-09 16:22:11.379702, event: done, op: pg_notify(1.16d(22) epoch 242) 2014-06-09 16:22:11.379709 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379711 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379713 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379713 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.379730 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379732 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379733 7f869af78700 20 osd.26 243 _dispatch 0x1ac941c0 pg_notify(3.2ac(10) epoch 242) v5 2014-06-09 16:22:11.379723 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.73( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.379738 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379738 7f869af78700 5 -- op tracker -- , seq: 1566, time: 2014-06-09 16:22:10.803407, event: header_read, op: pg_notify(3.2ac(10) epoch 242) 2014-06-09 16:22:11.379745 7f869af78700 5 -- op tracker -- , seq: 1566, time: 2014-06-09 16:22:10.803408, event: throttled, op: pg_notify(3.2ac(10) epoch 242) 2014-06-09 16:22:11.379742 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:2.587( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.379751 7f869af78700 5 -- op tracker -- , seq: 1566, time: 2014-06-09 16:22:10.803427, event: all_read, op: pg_notify(3.2ac(10) epoch 242) 2014-06-09 16:22:11.379757 7f869af78700 5 -- op tracker -- , seq: 1566, time: 2014-06-09 16:22:11.379730, event: dispatched, op: pg_notify(3.2ac(10) epoch 242) 2014-06-09 16:22:11.379754 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.379763 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379763 7f869af78700 5 -- op tracker -- , seq: 1566, time: 2014-06-09 16:22:11.379761, event: waiting_for_osdmap, op: pg_notify(3.2ac(10) epoch 242) 2014-06-09 16:22:11.379767 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.379769 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1ac941c0 2014-06-09 16:22:11.379767 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.379773 7f869af78700 5 -- op tracker -- , seq: 1566, time: 2014-06-09 16:22:11.379771, event: started, op: pg_notify(3.2ac(10) epoch 242) 2014-06-09 16:22:11.379777 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.82( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.379785 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379789 7f869af78700 5 -- op tracker -- , seq: 1566, time: 2014-06-09 16:22:11.379787, event: done, op: pg_notify(3.2ac(10) epoch 242) 2014-06-09 16:22:11.379788 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.379796 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379798 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379798 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.631( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.379806 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379812 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379814 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379809 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.379815 7f869af78700 20 osd.26 243 _dispatch 0x304f8a00 pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.379819 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.604( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.379822 7f869af78700 5 -- op tracker -- , seq: 1567, time: 2014-06-09 16:22:11.079627, event: header_read, op: pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379827 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379829 7f869af78700 5 -- op tracker -- , seq: 1567, time: 2014-06-09 16:22:11.079628, event: throttled, op: pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379830 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.379836 7f869af78700 5 -- op tracker -- , seq: 1567, time: 2014-06-09 16:22:11.079641, event: all_read, op: pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379843 7f869af78700 5 -- op tracker -- , seq: 1567, time: 2014-06-09 16:22:11.379812, event: dispatched, op: pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379840 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 1.676( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.379851 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.379851 7f869af78700 5 -- op tracker -- , seq: 1567, time: 2014-06-09 16:22:11.379848, event: waiting_for_osdmap, op: pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379855 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.379856 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.379860 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f8a00 2014-06-09 16:22:11.379859 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.676( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.114495 7 0.000199 2014-06-09 16:22:11.379866 7f869af78700 5 -- op tracker -- , seq: 1567, time: 2014-06-09 16:22:11.379863, event: started, op: pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379870 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.676( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.379880 7f869af78700 5 -- op tracker -- , seq: 1567, time: 2014-06-09 16:22:11.379877, event: done, op: pg_log(0.521 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.379881 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.676( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.379888 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379890 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379890 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.379903 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379904 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379899 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.379906 7f869af78700 20 osd.26 243 _dispatch 0x10d0d180 pg_notify(1.a3(23) epoch 243) v5 2014-06-09 16:22:11.379907 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.379911 7f869af78700 5 -- op tracker -- , seq: 1568, time: 2014-06-09 16:22:11.051084, event: header_read, op: pg_notify(1.a3(23) epoch 243) 2014-06-09 16:22:11.379918 7f869af78700 5 -- op tracker -- , seq: 1568, time: 2014-06-09 16:22:11.051084, event: throttled, op: pg_notify(1.a3(23) epoch 243) 2014-06-09 16:22:11.379925 7f869af78700 5 -- op tracker -- , seq: 1568, time: 2014-06-09 16:22:11.051107, event: all_read, op: pg_notify(1.a3(23) epoch 243) 2014-06-09 16:22:11.379931 7f869af78700 5 -- op tracker -- , seq: 1568, time: 2014-06-09 16:22:11.379902, event: dispatched, op: pg_notify(1.a3(23) epoch 243) 2014-06-09 16:22:11.379939 7f869af78700 5 -- op tracker -- , seq: 1568, time: 2014-06-09 16:22:11.379936, event: waiting_for_osdmap, op: pg_notify(1.a3(23) epoch 243) 2014-06-09 16:22:11.379943 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.379946 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d0d180 2014-06-09 16:22:11.379952 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379951 7f869af78700 5 -- op tracker -- , seq: 1568, time: 2014-06-09 16:22:11.379949, event: started, op: pg_notify(1.a3(23) epoch 243) 2014-06-09 16:22:11.379956 7f8690763700 5 queue_transactions existing osr(1.676 0x384dee0)/0x384dee0 2014-06-09 16:22:11.379959 7f8690763700 10 op_submit_start 3110 2014-06-09 16:22:11.379960 7f8690763700 5 queue_transactions (trailing journal) 3110 2014-06-09 16:22:11.379961 7f8690763700 5 queue_op 0x30607b00 seq 3110 osr(1.676 0x384dee0) 1876 bytes (queue has 2 ops and 2929 bytes) 2014-06-09 16:22:11.379967 7f8690763700 10 op_submit_finish 3110 2014-06-09 16:22:11.379967 7f869af78700 5 -- op tracker -- , seq: 1568, time: 2014-06-09 16:22:11.379965, event: done, op: pg_notify(1.a3(23) epoch 243) 2014-06-09 16:22:11.379974 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379976 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.379969 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.379980 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.379996 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.379997 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.379999 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.380000 7f869af78700 20 osd.26 243 _dispatch 0x304dde00 pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.380007 7f869af78700 5 -- op tracker -- , seq: 1569, time: 2014-06-09 16:22:11.071047, event: header_read, op: pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.380000 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.380016 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.380015 7f869af78700 5 -- op tracker -- , seq: 1569, time: 2014-06-09 16:22:11.071047, event: throttled, op: pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.380023 7f869af78700 5 -- op tracker -- , seq: 1569, time: 2014-06-09 16:22:11.071055, event: all_read, op: pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.380019 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.37e( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.37e( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.380031 7f869af78700 5 -- op tracker -- , seq: 1569, time: 2014-06-09 16:22:11.379996, event: dispatched, op: pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.380031 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.37e( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.380043 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.380039 7f869af78700 5 -- op tracker -- , seq: 1569, time: 2014-06-09 16:22:11.380036, event: waiting_for_osdmap, op: pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.380045 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.380049 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304dde00 2014-06-09 16:22:11.380055 7f869af78700 5 -- op tracker -- , seq: 1569, time: 2014-06-09 16:22:11.380052, event: started, op: pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.380046 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.380065 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] requeue_ops 0x16059800,0x16059700,0x16059600,0x16059500,0x1804c600,0x1804c500,0x1804c400 2014-06-09 16:22:11.380150 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (18 active) 2014-06-09 16:22:11.389352 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.389374 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.389387 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.389393 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.389406 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.389418 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.389422 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.389427 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.768539 7 0.000172 2014-06-09 16:22:11.389440 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.389451 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.389461 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.389470 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.389477 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.389528 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.389535 7f868cf5c700 5 queue_transactions existing osr(2.3f0 0x2ed1320)/0x2ed1320 2014-06-09 16:22:11.389539 7f868cf5c700 10 op_submit_start 3111 2014-06-09 16:22:11.389540 7f868cf5c700 5 queue_transactions (trailing journal) 3111 2014-06-09 16:22:11.389541 7f868cf5c700 5 queue_op 0x300338c0 seq 3111 osr(2.3f0 0x2ed1320) 1883 bytes (queue has 3 ops and 4812 bytes) 2014-06-09 16:22:11.389549 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.380208 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1a0030a0] 2014-06-09 16:22:11.389561 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.389577 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.389595 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.389612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1a0030a0] 2014-06-09 16:22:11.389615 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.389616 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.380554 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f0e0 (2 active) 2014-06-09 16:22:11.389623 7f86a5610700 5 _do_op 0x30607b00 seq 3110 osr(1.676 0x384dee0)/0x384dee0 start 2014-06-09 16:22:11.389628 7f86a5610700 10 _do_transaction on 0x2f29440 2014-06-09 16:22:11.389637 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.380563 7f868ef60700 10 osd.26 243 dequeue_op 0x13023800 prio 63 cost 516096 latency 1.096484 osd_op(client.4568.0:34998 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 3121152~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.380634 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.389729 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3074.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.389734 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.389762 7f86a5e11700 10 _do_op 0x308ca060 seq 3074 r = 0, finisher 0xbd6e9c0 0 2014-06-09 16:22:11.389769 7f86a5e11700 10 _finish_op 0x308ca060 seq 3074 osr(3.308 0x3ed0da0)/0x3ed0da0 2014-06-09 16:22:11.389777 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fca20 (1 active) 2014-06-09 16:22:11.389779 7f86a5e11700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.389784 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd78100] 2014-06-09 16:22:11.389791 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd78100] 2014-06-09 16:22:11.389792 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.389793 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.389826 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.389844 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3110.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.389848 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.389865 7f86a5610700 10 _do_op 0x30607b00 seq 3110 r = 0, finisher 0xcd0ffa0 0 2014-06-09 16:22:11.389870 7f86a5610700 10 _finish_op 0x30607b00 seq 3110 osr(1.676 0x384dee0)/0x384dee0 2014-06-09 16:22:11.389873 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f0e0 (0 active) 2014-06-09 16:22:11.389875 7f86a5610700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.389879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e9200] 2014-06-09 16:22:11.389882 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e9200] 2014-06-09 16:22:11.389884 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.389885 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.389911 7f868cf5c700 10 op_submit_finish 3111 2014-06-09 16:22:11.389917 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.389930 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.91( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.389938 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.389943 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.389953 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.301( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.389962 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.389966 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.389977 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.76( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.389985 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.389988 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.389998 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.27( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390009 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390013 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.390016 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.27( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.227493 7 0.000194 2014-06-09 16:22:11.390025 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.27( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.390035 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.27( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.390043 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.390051 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.390058 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.390100 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390104 7f868cf5c700 5 queue_transactions existing osr(2.27 0x2ed0250)/0x2ed0250 2014-06-09 16:22:11.390107 7f868cf5c700 10 op_submit_start 3112 2014-06-09 16:22:11.390108 7f868cf5c700 5 queue_transactions (trailing journal) 3112 2014-06-09 16:22:11.390109 7f868cf5c700 5 queue_op 0x30033860 seq 3112 osr(2.27 0x2ed0250) 1873 bytes (queue has 2 ops and 3756 bytes) 2014-06-09 16:22:11.390114 7f868cf5c700 10 op_submit_finish 3112 2014-06-09 16:22:11.390116 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.390131 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.390143 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390147 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.390160 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.606( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.390169 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390174 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.390185 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.4e0( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390197 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390201 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.390205 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] exit Started/Stray 1.230642 7 0.000228 2014-06-09 16:22:11.390215 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.390225 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.390234 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.390244 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.390253 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.390305 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390310 7f868cf5c700 5 queue_transactions existing osr(2.4e0 0x2ed16f0)/0x2ed16f0 2014-06-09 16:22:11.390313 7f868cf5c700 10 op_submit_start 3113 2014-06-09 16:22:11.390314 7f868cf5c700 5 queue_transactions (trailing journal) 3113 2014-06-09 16:22:11.390316 7f868cf5c700 5 queue_op 0x30033800 seq 3113 osr(2.4e0 0x2ed16f0) 2048 bytes (queue has 3 ops and 5804 bytes) 2014-06-09 16:22:11.390318 7f868cf5c700 10 op_submit_finish 3113 2014-06-09 16:22:11.390321 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.390335 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.390346 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390350 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:3.1db( v 214'696 (0'0,214'696] local-les=208 n=139 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.390365 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.390377 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390381 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:3.491( v 214'640 (0'0,214'640] local-les=208 n=123 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.390394 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.390405 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390408 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:1.86( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.390420 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.390431 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390435 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.390445 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.6cd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390456 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390459 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.390462 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.711430 7 0.000192 2014-06-09 16:22:11.390472 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.390481 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.390490 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.390498 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.390505 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.390551 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390556 7f868cf5c700 5 queue_transactions existing osr(2.6cd 0x2ed1e70)/0x2ed1e70 2014-06-09 16:22:11.390559 7f868cf5c700 10 op_submit_start 3114 2014-06-09 16:22:11.390560 7f868cf5c700 5 queue_transactions (trailing journal) 3114 2014-06-09 16:22:11.390562 7f868cf5c700 5 queue_op 0x300337a0 seq 3114 osr(2.6cd 0x2ed1e70) 1954 bytes (queue has 4 ops and 7758 bytes) 2014-06-09 16:22:11.390564 7f868cf5c700 10 op_submit_finish 3114 2014-06-09 16:22:11.390567 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:1.16d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.390578 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.390587 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390591 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:3.2ac( v 213'663 (0'0,213'663] local-les=203 n=133 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.390604 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.390615 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390620 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.390630 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.521( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390642 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390646 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.390650 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.521( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.093921 7 0.000200 2014-06-09 16:22:11.390660 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.521( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.390671 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.521( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.390680 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.390689 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.390697 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.390743 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390747 7f868cf5c700 5 queue_transactions existing osr(0.521 0x2bd7570)/0x2bd7570 2014-06-09 16:22:11.390750 7f868cf5c700 10 op_submit_start 3115 2014-06-09 16:22:11.390751 7f868cf5c700 5 queue_transactions (trailing journal) 3115 2014-06-09 16:22:11.390752 7f868cf5c700 5 queue_op 0x30033740 seq 3115 osr(0.521 0x2bd7570) 2001 bytes (queue has 5 ops and 9759 bytes) 2014-06-09 16:22:11.390755 7f868cf5c700 10 op_submit_finish 3115 2014-06-09 16:22:11.390758 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.a3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.390771 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.390781 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390784 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.390794 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.6ec( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390804 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.390807 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.390811 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.603698 7 0.005358 2014-06-09 16:22:11.390821 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.390831 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.390840 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.390849 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.390858 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.390901 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390905 7f868cf5c700 5 queue_transactions existing osr(2.6ec 0x2ed1e10)/0x2ed1e10 2014-06-09 16:22:11.390908 7f868cf5c700 10 op_submit_start 3116 2014-06-09 16:22:11.390909 7f868cf5c700 5 queue_transactions (trailing journal) 3116 2014-06-09 16:22:11.390910 7f868cf5c700 5 queue_op 0x300336e0 seq 3116 osr(2.6ec 0x2ed1e10) 1883 bytes (queue has 6 ops and 11642 bytes) 2014-06-09 16:22:11.390912 7f868cf5c700 10 op_submit_finish 3116 2014-06-09 16:22:11.390922 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (17 active) 2014-06-09 16:22:11.390925 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (18 active) 2014-06-09 16:22:11.390928 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.390938 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.75( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.390947 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.390953 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (17 active) 2014-06-09 16:22:11.390955 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.390961 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301320 (1 active) 2014-06-09 16:22:11.390964 7f86a5e11700 5 _do_op 0x300338c0 seq 3111 osr(2.3f0 0x2ed1320)/0x2ed1320 start 2014-06-09 16:22:11.390967 7f86a5e11700 10 _do_transaction on 0xc05ef00 2014-06-09 16:22:11.390973 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.391013 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.391030 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3111.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.391034 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.391078 7f86a5e11700 10 _do_op 0x300338c0 seq 3111 r = 0, finisher 0xbccc340 0 2014-06-09 16:22:11.391085 7f86a5e11700 10 _finish_op 0x300338c0 seq 3111 osr(2.3f0 0x2ed1320)/0x2ed1320 2014-06-09 16:22:11.391088 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301320 (0 active) 2014-06-09 16:22:11.391091 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383ec60 (1 active) 2014-06-09 16:22:11.391093 7f86a5e11700 5 _do_op 0x30033860 seq 3112 osr(2.27 0x2ed0250)/0x2ed0250 start 2014-06-09 16:22:11.391096 7f86a5e11700 10 _do_transaction on 0xc3f0fc0 2014-06-09 16:22:11.391101 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.391136 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.391152 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3112.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.391156 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.391175 7f86a5e11700 10 _do_op 0x30033860 seq 3112 r = 0, finisher 0x189abac0 0 2014-06-09 16:22:11.391179 7f86a5e11700 10 _finish_op 0x30033860 seq 3112 osr(2.27 0x2ed0250)/0x2ed0250 2014-06-09 16:22:11.391181 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383ec60 (0 active) 2014-06-09 16:22:11.391184 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301200 (1 active) 2014-06-09 16:22:11.391186 7f86a5e11700 5 _do_op 0x30033800 seq 3113 osr(2.4e0 0x2ed16f0)/0x2ed16f0 start 2014-06-09 16:22:11.391188 7f86a5e11700 10 _do_transaction on 0x2ed4a80 2014-06-09 16:22:11.391193 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.391225 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.391241 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3113.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.391245 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.391272 7f86a5e11700 10 _do_op 0x30033800 seq 3113 r = 0, finisher 0x30629ce0 0 2014-06-09 16:22:11.391276 7f86a5e11700 10 _finish_op 0x30033800 seq 3113 osr(2.4e0 0x2ed16f0)/0x2ed16f0 2014-06-09 16:22:11.391279 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301200 (0 active) 2014-06-09 16:22:11.391282 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497b00 (1 active) 2014-06-09 16:22:11.391284 7f86a5e11700 5 _do_op 0x300337a0 seq 3114 osr(2.6cd 0x2ed1e70)/0x2ed1e70 start 2014-06-09 16:22:11.391286 7f86a5e11700 10 _do_transaction on 0xc3ed5c0 2014-06-09 16:22:11.391292 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.391323 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.391338 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3114.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.391341 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.391358 7f86a5e11700 10 _do_op 0x300337a0 seq 3114 r = 0, finisher 0xbd5a820 0 2014-06-09 16:22:11.391361 7f86a5e11700 10 _finish_op 0x300337a0 seq 3114 osr(2.6cd 0x2ed1e70)/0x2ed1e70 2014-06-09 16:22:11.391363 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497b00 (0 active) 2014-06-09 16:22:11.391365 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc240 (1 active) 2014-06-09 16:22:11.391366 7f86a5e11700 5 _do_op 0x30033740 seq 3115 osr(0.521 0x2bd7570)/0x2bd7570 start 2014-06-09 16:22:11.391369 7f86a5e11700 10 _do_transaction on 0x3489e00 2014-06-09 16:22:11.391373 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.391588 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7aae0,0x30509060,0x30629d80,0xcd9e680] 2014-06-09 16:22:11.391596 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7aae0,0x30509060,0x30629d80,0xcd9e680] 2014-06-09 16:22:11.391599 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.391600 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.391604 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338360 (2 active) 2014-06-09 16:22:11.391608 7f86a5610700 5 _do_op 0x300336e0 seq 3116 osr(2.6ec 0x2ed1e10)/0x2ed1e10 start 2014-06-09 16:22:11.391611 7f86a5610700 10 _do_transaction on 0xc139440 2014-06-09 16:22:11.391621 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.391632 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.391648 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3115.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.391653 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.401269 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.401281 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6e9c0,0xcd0ffa0,0xbccc340,0x189abac0,0x30629ce0,0xbd5a820] 2014-06-09 16:22:11.401288 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.401305 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.401324 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.401340 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.401358 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.401377 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.401391 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.401404 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.401418 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.401436 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.401450 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.401464 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.401477 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.401490 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] flushed 2014-06-09 16:22:11.401511 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.401523 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.401537 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.401552 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6e9c0,0xcd0ffa0,0xbccc340,0x189abac0,0x30629ce0,0xbd5a820] 2014-06-09 16:22:11.401554 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.401556 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.401606 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (18 active) 2014-06-09 16:22:11.401612 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.401628 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x16bd7700,0x16bd7600,0x16bd7500,0x16bd7400 2014-06-09 16:22:11.401656 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.401661 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.401673 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.401681 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.401685 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.401694 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.27( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.401702 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.401705 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.401715 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.401723 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.401726 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.401736 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.401744 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.401750 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (17 active) 2014-06-09 16:22:11.401756 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (18 active) 2014-06-09 16:22:11.401760 7f868cf5c700 10 osd.26 243 dequeue_op 0x16bd7700 prio 63 cost 409600 latency 1.102839 osd_op(client.4565.0:32863 rbd_data.11756b8b4567.000000000000158d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.6a13db08 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.401803 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.401808 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.c1( v 214'609 (0'0,214'609] local-les=201 n=123 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.401826 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.401839 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.401843 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.401857 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.401870 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.401884 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (17 active) 2014-06-09 16:22:11.401887 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0250 (18 active) 2014-06-09 16:22:11.401889 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.401899 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.676( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.401907 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.401912 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (17 active) 2014-06-09 16:22:11.401917 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (18 active) 2014-06-09 16:22:11.401926 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.401930 7f868f761700 10 osd.26 243 dequeue_op 0x16059800 prio 63 cost 385024 latency 1.137906 osd_op(client.4575.0:32099 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.380069 7f869af78700 5 -- op tracker -- , seq: 1569, time: 2014-06-09 16:22:11.380065, event: done, op: pg_log(2.6ec epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.401964 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.401966 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.401983 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.401985 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.401986 7f869af78700 20 osd.26 243 _dispatch 0x2fe71340 pg_notify(2.18(20),3.1e(10) epoch 243) v5 2014-06-09 16:22:11.401992 7f869af78700 5 -- op tracker -- , seq: 1570, time: 2014-06-09 16:22:11.007389, event: header_read, op: pg_notify(2.18(20),3.1e(10) epoch 243) 2014-06-09 16:22:11.402002 7f869af78700 5 -- op tracker -- , seq: 1570, time: 2014-06-09 16:22:11.007389, event: throttled, op: pg_notify(2.18(20),3.1e(10) epoch 243) 2014-06-09 16:22:11.402009 7f869af78700 5 -- op tracker -- , seq: 1570, time: 2014-06-09 16:22:11.007414, event: all_read, op: pg_notify(2.18(20),3.1e(10) epoch 243) 2014-06-09 16:22:11.402016 7f869af78700 5 -- op tracker -- , seq: 1570, time: 2014-06-09 16:22:11.401982, event: dispatched, op: pg_notify(2.18(20),3.1e(10) epoch 243) 2014-06-09 16:22:11.402023 7f869af78700 5 -- op tracker -- , seq: 1570, time: 2014-06-09 16:22:11.402021, event: waiting_for_osdmap, op: pg_notify(2.18(20),3.1e(10) epoch 243) 2014-06-09 16:22:11.402028 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.402031 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe71340 2014-06-09 16:22:11.402037 7f869af78700 5 -- op tracker -- , seq: 1570, time: 2014-06-09 16:22:11.402035, event: started, op: pg_notify(2.18(20),3.1e(10) epoch 243) 2014-06-09 16:22:11.402062 7f869af78700 5 -- op tracker -- , seq: 1570, time: 2014-06-09 16:22:11.402060, event: done, op: pg_notify(2.18(20),3.1e(10) epoch 243) 2014-06-09 16:22:11.402070 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402072 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402095 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402097 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402098 7f869af78700 20 osd.26 243 _dispatch 0x198056c0 pg_notify(3.300(10) epoch 243) v5 2014-06-09 16:22:11.402103 7f869af78700 5 -- op tracker -- , seq: 1571, time: 2014-06-09 16:22:11.021266, event: header_read, op: pg_notify(3.300(10) epoch 243) 2014-06-09 16:22:11.402110 7f869af78700 5 -- op tracker -- , seq: 1571, time: 2014-06-09 16:22:11.021266, event: throttled, op: pg_notify(3.300(10) epoch 243) 2014-06-09 16:22:11.402116 7f869af78700 5 -- op tracker -- , seq: 1571, time: 2014-06-09 16:22:11.021286, event: all_read, op: pg_notify(3.300(10) epoch 243) 2014-06-09 16:22:11.402121 7f869af78700 5 -- op tracker -- , seq: 1571, time: 2014-06-09 16:22:11.402095, event: dispatched, op: pg_notify(3.300(10) epoch 243) 2014-06-09 16:22:11.402128 7f869af78700 5 -- op tracker -- , seq: 1571, time: 2014-06-09 16:22:11.402126, event: waiting_for_osdmap, op: pg_notify(3.300(10) epoch 243) 2014-06-09 16:22:11.402132 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.402134 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x198056c0 2014-06-09 16:22:11.402139 7f869af78700 5 -- op tracker -- , seq: 1571, time: 2014-06-09 16:22:11.402137, event: started, op: pg_notify(3.300(10) epoch 243) 2014-06-09 16:22:11.402155 7f869af78700 5 -- op tracker -- , seq: 1571, time: 2014-06-09 16:22:11.402153, event: done, op: pg_notify(3.300(10) epoch 243) 2014-06-09 16:22:11.402162 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402164 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402178 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402180 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402181 7f869af78700 20 osd.26 243 _dispatch 0x2fe70380 pg_notify(3.48a(10) epoch 242) v5 2014-06-09 16:22:11.402186 7f869af78700 5 -- op tracker -- , seq: 1572, time: 2014-06-09 16:22:10.991070, event: header_read, op: pg_notify(3.48a(10) epoch 242) 2014-06-09 16:22:11.402192 7f869af78700 5 -- op tracker -- , seq: 1572, time: 2014-06-09 16:22:10.991070, event: throttled, op: pg_notify(3.48a(10) epoch 242) 2014-06-09 16:22:11.402198 7f869af78700 5 -- op tracker -- , seq: 1572, time: 2014-06-09 16:22:10.991084, event: all_read, op: pg_notify(3.48a(10) epoch 242) 2014-06-09 16:22:11.402204 7f869af78700 5 -- op tracker -- , seq: 1572, time: 2014-06-09 16:22:11.402178, event: dispatched, op: pg_notify(3.48a(10) epoch 242) 2014-06-09 16:22:11.402210 7f869af78700 5 -- op tracker -- , seq: 1572, time: 2014-06-09 16:22:11.402208, event: waiting_for_osdmap, op: pg_notify(3.48a(10) epoch 242) 2014-06-09 16:22:11.402215 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.402216 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2fe70380 2014-06-09 16:22:11.402221 7f869af78700 5 -- op tracker -- , seq: 1572, time: 2014-06-09 16:22:11.402219, event: started, op: pg_notify(3.48a(10) epoch 242) 2014-06-09 16:22:11.402236 7f869af78700 5 -- op tracker -- , seq: 1572, time: 2014-06-09 16:22:11.402234, event: done, op: pg_notify(3.48a(10) epoch 242) 2014-06-09 16:22:11.402244 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402246 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402274 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402276 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402277 7f869af78700 20 osd.26 243 _dispatch 0x2a073800 pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.402284 7f869af78700 5 -- op tracker -- , seq: 1573, time: 2014-06-09 16:22:11.038765, event: header_read, op: pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402292 7f869af78700 5 -- op tracker -- , seq: 1573, time: 2014-06-09 16:22:11.038765, event: throttled, op: pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402299 7f869af78700 5 -- op tracker -- , seq: 1573, time: 2014-06-09 16:22:11.038775, event: all_read, op: pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402305 7f869af78700 5 -- op tracker -- , seq: 1573, time: 2014-06-09 16:22:11.402274, event: dispatched, op: pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402312 7f869af78700 5 -- op tracker -- , seq: 1573, time: 2014-06-09 16:22:11.402310, event: waiting_for_osdmap, op: pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402317 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.402321 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a073800 2014-06-09 16:22:11.402333 7f869af78700 5 -- op tracker -- , seq: 1573, time: 2014-06-09 16:22:11.402324, event: started, op: pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402347 7f869af78700 5 -- op tracker -- , seq: 1573, time: 2014-06-09 16:22:11.402344, event: done, op: pg_log(0.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402358 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402370 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402371 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402372 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a540 pg_info(1 pgs e243:3.6d5(2)) v4 2014-06-09 16:22:11.402378 7f869af78700 5 -- op tracker -- , seq: 1574, time: 2014-06-09 16:22:11.058975, event: header_read, op: pg_info(1 pgs e243:3.6d5(2)) 2014-06-09 16:22:11.402384 7f869af78700 5 -- op tracker -- , seq: 1574, time: 2014-06-09 16:22:11.058975, event: throttled, op: pg_info(1 pgs e243:3.6d5(2)) 2014-06-09 16:22:11.402390 7f869af78700 5 -- op tracker -- , seq: 1574, time: 2014-06-09 16:22:11.058987, event: all_read, op: pg_info(1 pgs e243:3.6d5(2)) 2014-06-09 16:22:11.402396 7f869af78700 5 -- op tracker -- , seq: 1574, time: 2014-06-09 16:22:11.402370, event: dispatched, op: pg_info(1 pgs e243:3.6d5(2)) 2014-06-09 16:22:11.402403 7f869af78700 5 -- op tracker -- , seq: 1574, time: 2014-06-09 16:22:11.402401, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6d5(2)) 2014-06-09 16:22:11.402407 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6d5(2)) v4 from osd.5 2014-06-09 16:22:11.402410 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5a540 2014-06-09 16:22:11.402414 7f869af78700 5 -- op tracker -- , seq: 1574, time: 2014-06-09 16:22:11.402413, event: started, op: pg_info(1 pgs e243:3.6d5(2)) 2014-06-09 16:22:11.402430 7f869af78700 5 -- op tracker -- , seq: 1574, time: 2014-06-09 16:22:11.402428, event: done, op: pg_info(1 pgs e243:3.6d5(2)) 2014-06-09 16:22:11.402438 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402439 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402452 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402453 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402455 7f869af78700 20 osd.26 243 _dispatch 0x2d732fc0 pg_notify(3.15c(12) epoch 242) v5 2014-06-09 16:22:11.402459 7f869af78700 5 -- op tracker -- , seq: 1575, time: 2014-06-09 16:22:10.886409, event: header_read, op: pg_notify(3.15c(12) epoch 242) 2014-06-09 16:22:11.402466 7f869af78700 5 -- op tracker -- , seq: 1575, time: 2014-06-09 16:22:10.886409, event: throttled, op: pg_notify(3.15c(12) epoch 242) 2014-06-09 16:22:11.402472 7f869af78700 5 -- op tracker -- , seq: 1575, time: 2014-06-09 16:22:10.886429, event: all_read, op: pg_notify(3.15c(12) epoch 242) 2014-06-09 16:22:11.402478 7f869af78700 5 -- op tracker -- , seq: 1575, time: 2014-06-09 16:22:11.402452, event: dispatched, op: pg_notify(3.15c(12) epoch 242) 2014-06-09 16:22:11.402484 7f869af78700 5 -- op tracker -- , seq: 1575, time: 2014-06-09 16:22:11.402482, event: waiting_for_osdmap, op: pg_notify(3.15c(12) epoch 242) 2014-06-09 16:22:11.402488 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.402490 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d732fc0 2014-06-09 16:22:11.402495 7f869af78700 5 -- op tracker -- , seq: 1575, time: 2014-06-09 16:22:11.402493, event: started, op: pg_notify(3.15c(12) epoch 242) 2014-06-09 16:22:11.402511 7f869af78700 5 -- op tracker -- , seq: 1575, time: 2014-06-09 16:22:11.402509, event: done, op: pg_notify(3.15c(12) epoch 242) 2014-06-09 16:22:11.402521 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402523 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402538 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402539 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402541 7f869af78700 20 osd.26 243 _dispatch 0x4680700 pg_notify(0.594(20) epoch 242) v5 2014-06-09 16:22:11.402545 7f869af78700 5 -- op tracker -- , seq: 1576, time: 2014-06-09 16:22:10.835807, event: header_read, op: pg_notify(0.594(20) epoch 242) 2014-06-09 16:22:11.402552 7f869af78700 5 -- op tracker -- , seq: 1576, time: 2014-06-09 16:22:10.835808, event: throttled, op: pg_notify(0.594(20) epoch 242) 2014-06-09 16:22:11.402558 7f869af78700 5 -- op tracker -- , seq: 1576, time: 2014-06-09 16:22:10.835834, event: all_read, op: pg_notify(0.594(20) epoch 242) 2014-06-09 16:22:11.402563 7f869af78700 5 -- op tracker -- , seq: 1576, time: 2014-06-09 16:22:11.402538, event: dispatched, op: pg_notify(0.594(20) epoch 242) 2014-06-09 16:22:11.402570 7f869af78700 5 -- op tracker -- , seq: 1576, time: 2014-06-09 16:22:11.402568, event: waiting_for_osdmap, op: pg_notify(0.594(20) epoch 242) 2014-06-09 16:22:11.402574 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.402576 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x4680700 2014-06-09 16:22:11.402581 7f869af78700 5 -- op tracker -- , seq: 1576, time: 2014-06-09 16:22:11.402579, event: started, op: pg_notify(0.594(20) epoch 242) 2014-06-09 16:22:11.402596 7f869af78700 5 -- op tracker -- , seq: 1576, time: 2014-06-09 16:22:11.402594, event: done, op: pg_notify(0.594(20) epoch 242) 2014-06-09 16:22:11.402607 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402608 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402625 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402626 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402627 7f869af78700 20 osd.26 243 _dispatch 0x2fef4700 pg_notify(3.de(10) epoch 243) v5 2014-06-09 16:22:11.402631 7f869af78700 5 -- op tracker -- , seq: 1577, time: 2014-06-09 16:22:11.102141, event: header_read, op: pg_notify(3.de(10) epoch 243) 2014-06-09 16:22:11.402638 7f869af78700 5 -- op tracker -- , seq: 1577, time: 2014-06-09 16:22:11.102141, event: throttled, op: pg_notify(3.de(10) epoch 243) 2014-06-09 16:22:11.402644 7f869af78700 5 -- op tracker -- , seq: 1577, time: 2014-06-09 16:22:11.102158, event: all_read, op: pg_notify(3.de(10) epoch 243) 2014-06-09 16:22:11.402650 7f869af78700 5 -- op tracker -- , seq: 1577, time: 2014-06-09 16:22:11.402624, event: dispatched, op: pg_notify(3.de(10) epoch 243) 2014-06-09 16:22:11.402656 7f869af78700 5 -- op tracker -- , seq: 1577, time: 2014-06-09 16:22:11.402654, event: waiting_for_osdmap, op: pg_notify(3.de(10) epoch 243) 2014-06-09 16:22:11.402660 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.402662 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4700 2014-06-09 16:22:11.402666 7f869af78700 5 -- op tracker -- , seq: 1577, time: 2014-06-09 16:22:11.402665, event: started, op: pg_notify(3.de(10) epoch 243) 2014-06-09 16:22:11.402682 7f869af78700 5 -- op tracker -- , seq: 1577, time: 2014-06-09 16:22:11.402680, event: done, op: pg_notify(3.de(10) epoch 243) 2014-06-09 16:22:11.402689 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402691 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402704 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402706 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402707 7f869af78700 20 osd.26 243 _dispatch 0x15a29340 pg_info(1 pgs e243:3.359(2)) v4 2014-06-09 16:22:11.402712 7f869af78700 5 -- op tracker -- , seq: 1578, time: 2014-06-09 16:22:11.051148, event: header_read, op: pg_info(1 pgs e243:3.359(2)) 2014-06-09 16:22:11.402719 7f869af78700 5 -- op tracker -- , seq: 1578, time: 2014-06-09 16:22:11.051148, event: throttled, op: pg_info(1 pgs e243:3.359(2)) 2014-06-09 16:22:11.402724 7f869af78700 5 -- op tracker -- , seq: 1578, time: 2014-06-09 16:22:11.051156, event: all_read, op: pg_info(1 pgs e243:3.359(2)) 2014-06-09 16:22:11.402730 7f869af78700 5 -- op tracker -- , seq: 1578, time: 2014-06-09 16:22:11.402704, event: dispatched, op: pg_info(1 pgs e243:3.359(2)) 2014-06-09 16:22:11.402737 7f869af78700 5 -- op tracker -- , seq: 1578, time: 2014-06-09 16:22:11.402735, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.359(2)) 2014-06-09 16:22:11.402741 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.359(2)) v4 from osd.3 2014-06-09 16:22:11.402744 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a29340 2014-06-09 16:22:11.402748 7f869af78700 5 -- op tracker -- , seq: 1578, time: 2014-06-09 16:22:11.402747, event: started, op: pg_info(1 pgs e243:3.359(2)) 2014-06-09 16:22:11.402763 7f869af78700 5 -- op tracker -- , seq: 1578, time: 2014-06-09 16:22:11.402761, event: done, op: pg_info(1 pgs e243:3.359(2)) 2014-06-09 16:22:11.402771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402772 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402786 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402788 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402789 7f869af78700 20 osd.26 243 _dispatch 0x2fe32000 pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.402796 7f869af78700 5 -- op tracker -- , seq: 1579, time: 2014-06-09 16:22:11.071060, event: header_read, op: pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402803 7f869af78700 5 -- op tracker -- , seq: 1579, time: 2014-06-09 16:22:11.071060, event: throttled, op: pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402810 7f869af78700 5 -- op tracker -- , seq: 1579, time: 2014-06-09 16:22:11.071067, event: all_read, op: pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402816 7f869af78700 5 -- op tracker -- , seq: 1579, time: 2014-06-09 16:22:11.402786, event: dispatched, op: pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402823 7f869af78700 5 -- op tracker -- , seq: 1579, time: 2014-06-09 16:22:11.402820, event: waiting_for_osdmap, op: pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402827 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.402831 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe32000 2014-06-09 16:22:11.402836 7f869af78700 5 -- op tracker -- , seq: 1579, time: 2014-06-09 16:22:11.402834, event: started, op: pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402848 7f869af78700 5 -- op tracker -- , seq: 1579, time: 2014-06-09 16:22:11.402846, event: done, op: pg_log(2.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402855 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402857 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402869 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402870 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402871 7f869af78700 20 osd.26 243 _dispatch 0x2a075000 pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.402877 7f869af78700 5 -- op tracker -- , seq: 1580, time: 2014-06-09 16:22:11.021796, event: header_read, op: pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402884 7f869af78700 5 -- op tracker -- , seq: 1580, time: 2014-06-09 16:22:11.021797, event: throttled, op: pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402891 7f869af78700 5 -- op tracker -- , seq: 1580, time: 2014-06-09 16:22:11.021808, event: all_read, op: pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402898 7f869af78700 5 -- op tracker -- , seq: 1580, time: 2014-06-09 16:22:11.402868, event: dispatched, op: pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402905 7f869af78700 5 -- op tracker -- , seq: 1580, time: 2014-06-09 16:22:11.402902, event: waiting_for_osdmap, op: pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402909 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.402912 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a075000 2014-06-09 16:22:11.402918 7f869af78700 5 -- op tracker -- , seq: 1580, time: 2014-06-09 16:22:11.402915, event: started, op: pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402930 7f869af78700 5 -- op tracker -- , seq: 1580, time: 2014-06-09 16:22:11.402927, event: done, op: pg_log(1.56 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.402937 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402938 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402949 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.402950 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.402951 7f869af78700 20 osd.26 243 _dispatch 0x2a070a80 pg_info(1 pgs e243:3.19(2)) v4 2014-06-09 16:22:11.402956 7f869af78700 5 -- op tracker -- , seq: 1581, time: 2014-06-09 16:22:11.021294, event: header_read, op: pg_info(1 pgs e243:3.19(2)) 2014-06-09 16:22:11.402963 7f869af78700 5 -- op tracker -- , seq: 1581, time: 2014-06-09 16:22:11.021294, event: throttled, op: pg_info(1 pgs e243:3.19(2)) 2014-06-09 16:22:11.402969 7f869af78700 5 -- op tracker -- , seq: 1581, time: 2014-06-09 16:22:11.021307, event: all_read, op: pg_info(1 pgs e243:3.19(2)) 2014-06-09 16:22:11.402975 7f869af78700 5 -- op tracker -- , seq: 1581, time: 2014-06-09 16:22:11.402948, event: dispatched, op: pg_info(1 pgs e243:3.19(2)) 2014-06-09 16:22:11.402982 7f869af78700 5 -- op tracker -- , seq: 1581, time: 2014-06-09 16:22:11.402980, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.19(2)) 2014-06-09 16:22:11.402986 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.19(2)) v4 from osd.8 2014-06-09 16:22:11.402989 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a070a80 2014-06-09 16:22:11.402993 7f869af78700 5 -- op tracker -- , seq: 1581, time: 2014-06-09 16:22:11.402991, event: started, op: pg_info(1 pgs e243:3.19(2)) 2014-06-09 16:22:11.403008 7f869af78700 5 -- op tracker -- , seq: 1581, time: 2014-06-09 16:22:11.403006, event: done, op: pg_info(1 pgs e243:3.19(2)) 2014-06-09 16:22:11.403015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.403017 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.403028 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.403029 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.403031 7f869af78700 20 osd.26 243 _dispatch 0x15998700 pg_notify(0.1c8(20) epoch 242) v5 2014-06-09 16:22:11.403035 7f869af78700 5 -- op tracker -- , seq: 1582, time: 2014-06-09 16:22:10.991392, event: header_read, op: pg_notify(0.1c8(20) epoch 242) 2014-06-09 16:22:11.403042 7f869af78700 5 -- op tracker -- , seq: 1582, time: 2014-06-09 16:22:10.991393, event: throttled, op: pg_notify(0.1c8(20) epoch 242) 2014-06-09 16:22:11.403048 7f869af78700 5 -- op tracker -- , seq: 1582, time: 2014-06-09 16:22:10.991426, event: all_read, op: pg_notify(0.1c8(20) epoch 242) 2014-06-09 16:22:11.403052 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403054 7f869af78700 5 -- op tracker -- , seq: 1582, time: 2014-06-09 16:22:11.403028, event: dispatched, op: pg_notify(0.1c8(20) epoch 242) 2014-06-09 16:22:11.403061 7f869af78700 5 -- op tracker -- , seq: 1582, time: 2014-06-09 16:22:11.403059, event: waiting_for_osdmap, op: pg_notify(0.1c8(20) epoch 242) 2014-06-09 16:22:11.403065 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.403068 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x15998700 2014-06-09 16:22:11.403061 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.403073 7f869af78700 5 -- op tracker -- , seq: 1582, time: 2014-06-09 16:22:11.403071, event: started, op: pg_notify(0.1c8(20) epoch 242) 2014-06-09 16:22:11.403077 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403085 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0240 (18 active) 2014-06-09 16:22:11.403094 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0240 (19 active) 2014-06-09 16:22:11.403099 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.403109 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:11.403118 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403121 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:2.18( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.403133 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.403142 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403145 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.1e( v 214'592 (0'0,214'592] local-les=213 n=110 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.403156 7f868ef60700 5 -- op tracker -- , seq: 201, time: 2014-06-09 16:22:11.389704, event: reached_pg, op: osd_op(client.4568.0:34998 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 3121152~516096] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403157 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.403168 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403171 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.300( v 213'576 (0'0,213'576] local-les=212 n=119 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.403168 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403183 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.403194 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403186 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x13023800 2014-06-09 16:22:11.403198 7f868ef60700 10 osd.26 243 dequeue_op 0x13023800 finish 2014-06-09 16:22:11.403204 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403198 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:3.48a( v 214'648 (0'0,214'648] local-les=208 n=120 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.403209 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403211 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.403220 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403224 7f868cf5c700 5 -- op tracker -- , seq: 239, time: 2014-06-09 16:22:11.401779, event: reached_pg, op: osd_op(client.4565.0:32863 rbd_data.11756b8b4567.000000000000158d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~409600] 3.6a13db08 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403224 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.403234 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.8c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.403235 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403246 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.403250 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.403248 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16bd7700 2014-06-09 16:22:11.403257 7f868cf5c700 10 osd.26 243 dequeue_op 0x16bd7700 finish 2014-06-09 16:22:11.403262 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403255 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.172929 7 0.003903 2014-06-09 16:22:11.403267 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403266 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.403277 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.403270 7f868cf5c700 10 osd.26 243 dequeue_op 0x19fe1f00 prio 63 cost 172032 latency 1.006834 osd_op(client.4570.0:33856 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.403288 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.403291 7f868cf5c700 5 -- op tracker -- , seq: 301, time: 2014-06-09 16:22:11.403286, event: reached_pg, op: osd_op(client.4570.0:33856 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~172032] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403296 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.403297 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403305 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.403309 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x19fe1f00 2014-06-09 16:22:11.403318 7f868cf5c700 10 osd.26 243 dequeue_op 0x19fe1f00 finish 2014-06-09 16:22:11.403321 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403326 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403331 7f8690763700 10 osd.26 243 dequeue_op 0x16bd7600 prio 63 cost 516096 latency 1.103548 osd_op(client.4565.0:32864 rbd_data.11756b8b4567.000000000000158d [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.6a13db08 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.403353 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403351 7f8690763700 5 -- op tracker -- , seq: 248, time: 2014-06-09 16:22:11.403346, event: reached_pg, op: osd_op(client.4565.0:32864 rbd_data.11756b8b4567.000000000000158d [set-alloc-hint object_size 4194304 write_size 4194304,write 409600~516096] 3.6a13db08 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403358 7f8690f64700 5 queue_transactions existing osr(0.8c 0x2bd6480)/0x2bd6480 2014-06-09 16:22:11.403362 7f8690f64700 10 op_submit_start 3117 2014-06-09 16:22:11.403363 7f8690f64700 5 queue_transactions (trailing journal) 3117 2014-06-09 16:22:11.403365 7f8690f64700 5 queue_op 0x308ca3c0 seq 3117 osr(0.8c 0x2bd6480) 1880 bytes (queue has 3 ops and 5764 bytes) 2014-06-09 16:22:11.403357 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403368 7f8690f64700 10 op_submit_finish 3117 2014-06-09 16:22:11.403368 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16bd7600 2014-06-09 16:22:11.403377 7f8690763700 10 osd.26 243 dequeue_op 0x16bd7600 finish 2014-06-09 16:22:11.403370 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=206 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.403382 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403387 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403382 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=206 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.403390 7f8690763700 10 osd.26 243 dequeue_op 0x16bd7500 prio 63 cost 3096576 latency 1.096521 osd_op(client.4565.0:32865 rbd_data.11756b8b4567.000000000000158d [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~3096576] 3.6a13db08 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.403395 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=206 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.634210 7 0.000191 2014-06-09 16:22:11.403405 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=206 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.403408 7f8690763700 5 -- op tracker -- , seq: 282, time: 2014-06-09 16:22:11.403404, event: reached_pg, op: osd_op(client.4565.0:32865 rbd_data.11756b8b4567.000000000000158d [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~3096576] 3.6a13db08 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403416 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=206 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.403415 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403426 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16bd7500 2014-06-09 16:22:11.403426 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=206 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.403435 7f8690763700 10 osd.26 243 dequeue_op 0x16bd7500 finish 2014-06-09 16:22:11.403438 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403442 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403437 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'532 -> 213'532 2014-06-09 16:22:11.403448 7f868ef60700 10 osd.26 243 dequeue_op 0x16bd7400 prio 63 cost 172032 latency 1.043135 osd_op(client.4565.0:32866 rbd_data.11756b8b4567.000000000000158d [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.6a13db08 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.403467 7f868ef60700 5 -- op tracker -- , seq: 283, time: 2014-06-09 16:22:11.403463, event: reached_pg, op: osd_op(client.4565.0:32866 rbd_data.11756b8b4567.000000000000158d [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.6a13db08 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403472 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.403474 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403485 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16bd7400 2014-06-09 16:22:11.403494 7f868ef60700 10 osd.26 243 dequeue_op 0x16bd7400 finish 2014-06-09 16:22:11.403497 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403502 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403510 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403515 7f8690f64700 5 queue_transactions existing osr(3.6d5 0x3ed1c30)/0x3ed1c30 2014-06-09 16:22:11.403511 7f868f761700 5 -- op tracker -- , seq: 198, time: 2014-06-09 16:22:11.401950, event: reached_pg, op: osd_op(client.4575.0:32099 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403517 7f8690f64700 10 op_submit_start 3118 2014-06-09 16:22:11.403518 7f8690f64700 5 queue_transactions (trailing journal) 3118 2014-06-09 16:22:11.403519 7f8690f64700 5 queue_op 0x308ca300 seq 3118 osr(3.6d5 0x3ed1c30) 1335 bytes (queue has 4 ops and 7099 bytes) 2014-06-09 16:22:11.403522 7f8690f64700 10 op_submit_finish 3118 2014-06-09 16:22:11.403519 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403524 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:3.15c( v 213'624 (0'0,213'624] local-les=191 n=123 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.403533 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x16059800 2014-06-09 16:22:11.403544 7f868f761700 10 osd.26 243 dequeue_op 0x16059800 finish 2014-06-09 16:22:11.403539 7f8690f64700 7 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.403550 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403555 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403549 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.15c( v 213'624 (0'0,213'624] local-les=191 n=123 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.403568 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403558 7f868f761700 10 osd.26 243 dequeue_op 0x16059700 prio 63 cost 516096 latency 1.119279 osd_op(client.4575.0:32100 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 385024~516096] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.403572 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:0.594( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.403579 7f868f761700 5 -- op tracker -- , seq: 213, time: 2014-06-09 16:22:11.403574, event: reached_pg, op: osd_op(client.4575.0:32100 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 385024~516096] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403584 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.403594 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403585 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403598 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.de( v 214'581 (0'0,214'581] local-les=210 n=114 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.403598 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x16059700 2014-06-09 16:22:11.403610 7f868f761700 10 osd.26 243 dequeue_op 0x16059700 finish 2014-06-09 16:22:11.403618 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403623 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403610 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.403627 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403631 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=201 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.403626 7f868f761700 10 osd.26 243 dequeue_op 0x16059600 prio 63 cost 516096 latency 1.113865 osd_op(client.4575.0:32101 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 901120~516096] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.403629 7f868e75f700 10 osd.26 243 dequeue_op 0x1b0d6700 prio 63 cost 1032192 latency 1.001647 osd_op(client.4570.0:33857 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~1032192] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.403642 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=201 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.403648 7f868f761700 5 -- op tracker -- , seq: 214, time: 2014-06-09 16:22:11.403643, event: reached_pg, op: osd_op(client.4575.0:32101 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 901120~516096] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403654 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=201 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.147200 7 0.000150 2014-06-09 16:22:11.403657 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403665 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=201 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.403671 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x16059600 2014-06-09 16:22:11.403681 7f868f761700 10 osd.26 243 dequeue_op 0x16059600 finish 2014-06-09 16:22:11.403675 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=201 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.403684 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403689 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (19 active) 2014-06-09 16:22:11.403686 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=201 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.403696 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'669 -> 213'669 2014-06-09 16:22:11.403692 7f868f761700 10 osd.26 243 dequeue_op 0x16059500 prio 63 cost 516096 latency 1.112977 osd_op(client.4575.0:32102 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 1417216~516096] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.403713 7f868f761700 5 -- op tracker -- , seq: 223, time: 2014-06-09 16:22:11.403709, event: reached_pg, op: osd_op(client.4575.0:32102 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 1417216~516096] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403719 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403733 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x16059500 2014-06-09 16:22:11.403746 7f868f761700 10 osd.26 243 dequeue_op 0x16059500 finish 2014-06-09 16:22:11.403741 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.403749 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (18 active) 2014-06-09 16:22:11.403752 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.403758 7f868cf5c700 10 osd.26 243 dequeue_op 0x1804c600 prio 63 cost 516096 latency 1.111656 osd_op(client.4575.0:32103 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 1933312~516096] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.403778 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403783 7f8690f64700 5 queue_transactions existing osr(3.359 0x3ed10f0)/0x3ed10f0 2014-06-09 16:22:11.403785 7f8690f64700 10 op_submit_start 3119 2014-06-09 16:22:11.403786 7f8690f64700 5 queue_transactions (trailing journal) 3119 2014-06-09 16:22:11.403782 7f868cf5c700 5 -- op tracker -- , seq: 231, time: 2014-06-09 16:22:11.403777, event: reached_pg, op: osd_op(client.4575.0:32103 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 1933312~516096] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403787 7f8690f64700 5 queue_op 0x308ca2a0 seq 3119 osr(3.359 0x3ed10f0) 1335 bytes (queue has 5 ops and 8434 bytes) 2014-06-09 16:22:11.403790 7f8690f64700 10 op_submit_finish 3119 2014-06-09 16:22:11.403793 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.403789 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403803 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1804c600 2014-06-09 16:22:11.403804 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.530( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.403814 7f868cf5c700 10 osd.26 243 dequeue_op 0x1804c600 finish 2014-06-09 16:22:11.403816 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.403819 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (17 active) 2014-06-09 16:22:11.403819 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.403822 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.403822 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.530( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.106224 7 0.000192 2014-06-09 16:22:11.403832 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.530( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.403827 7f8690763700 10 osd.26 243 dequeue_op 0x1804c500 prio 63 cost 1548288 latency 1.106955 osd_op(client.4575.0:32104 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 2449408~1548288] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.403841 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.530( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.403847 7f8690763700 5 -- op tracker -- , seq: 244, time: 2014-06-09 16:22:11.403843, event: reached_pg, op: osd_op(client.4575.0:32104 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 2449408~1548288] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403851 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.403854 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403860 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.403868 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.403868 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1804c500 2014-06-09 16:22:11.403879 7f8690763700 10 osd.26 243 dequeue_op 0x1804c500 finish 2014-06-09 16:22:11.403884 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (16 active) 2014-06-09 16:22:11.403887 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.403892 7f868ef60700 10 osd.26 243 dequeue_op 0x1804c400 prio 63 cost 196608 latency 1.102098 osd_op(client.4575.0:32105 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 3997696~196608] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.403912 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.403916 7f8690f64700 5 queue_transactions existing osr(2.530 0x2ed19b0)/0x2ed19b0 2014-06-09 16:22:11.403913 7f868ef60700 5 -- op tracker -- , seq: 247, time: 2014-06-09 16:22:11.403908, event: reached_pg, op: osd_op(client.4575.0:32105 rbd_data.117f6b8b4567.000000000000151c [set-alloc-hint object_size 4194304 write_size 4194304,write 3997696~196608] 3.87496894 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.403918 7f8690f64700 10 op_submit_start 3120 2014-06-09 16:22:11.403919 7f8690f64700 5 queue_transactions (trailing journal) 3120 2014-06-09 16:22:11.403920 7f8690f64700 5 queue_op 0x308ca240 seq 3120 osr(2.530 0x2ed19b0) 1883 bytes (queue has 6 ops and 10317 bytes) 2014-06-09 16:22:11.403923 7f8690f64700 10 op_submit_finish 3120 2014-06-09 16:22:11.403920 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.403925 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.403933 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1804c400 2014-06-09 16:22:11.403943 7f868ef60700 10 osd.26 243 dequeue_op 0x1804c400 finish 2014-06-09 16:22:11.403936 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 1.56( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.403947 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.403949 7f868ef60700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.403948 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.403952 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.403955 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.56( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.243591 7 0.000206 2014-06-09 16:22:11.403960 7f869af78700 5 -- op tracker -- , seq: 1582, time: 2014-06-09 16:22:11.403957, event: done, op: pg_notify(0.1c8(20) epoch 242) 2014-06-09 16:22:11.403969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.403970 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.403966 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.56( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.403975 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.56( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.403987 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.403988 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.403984 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.403990 7f869af78700 20 osd.26 243 _dispatch 0x2a073200 pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.403992 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.403996 7f869af78700 5 -- op tracker -- , seq: 1583, time: 2014-06-09 16:22:11.038807, event: header_read, op: pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404000 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.404004 7f869af78700 5 -- op tracker -- , seq: 1583, time: 2014-06-09 16:22:11.038807, event: throttled, op: pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404011 7f869af78700 5 -- op tracker -- , seq: 1583, time: 2014-06-09 16:22:11.038816, event: all_read, op: pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404017 7f869af78700 5 -- op tracker -- , seq: 1583, time: 2014-06-09 16:22:11.403986, event: dispatched, op: pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404025 7f869af78700 5 -- op tracker -- , seq: 1583, time: 2014-06-09 16:22:11.404022, event: waiting_for_osdmap, op: pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404029 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.404032 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a073200 2014-06-09 16:22:11.404038 7f869af78700 5 -- op tracker -- , seq: 1583, time: 2014-06-09 16:22:11.404035, event: started, op: pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404042 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404046 7f8690f64700 5 queue_transactions existing osr(1.56 0x384c5f0)/0x384c5f0 2014-06-09 16:22:11.404048 7f8690f64700 10 op_submit_start 3121 2014-06-09 16:22:11.404049 7f8690f64700 5 queue_transactions (trailing journal) 3121 2014-06-09 16:22:11.404050 7f8690f64700 5 queue_op 0x308ca0c0 seq 3121 osr(1.56 0x384c5f0) 1873 bytes (queue has 7 ops and 12190 bytes) 2014-06-09 16:22:11.404052 7f8690f64700 10 op_submit_finish 3121 2014-06-09 16:22:11.404052 7f869af78700 5 -- op tracker -- , seq: 1583, time: 2014-06-09 16:22:11.404050, event: done, op: pg_log(1.a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404060 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404061 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404055 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=212 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.404074 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404076 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404067 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=212 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.404077 7f869af78700 20 osd.26 243 _dispatch 0x260001c0 pg_notify(3.310(10),3.4d(10) epoch 243) v5 2014-06-09 16:22:11.404083 7f869af78700 5 -- op tracker -- , seq: 1584, time: 2014-06-09 16:22:11.086819, event: header_read, op: pg_notify(3.310(10),3.4d(10) epoch 243) 2014-06-09 16:22:11.404080 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=212 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.241671 7 0.000203 2014-06-09 16:22:11.404090 7f869af78700 5 -- op tracker -- , seq: 1584, time: 2014-06-09 16:22:11.086820, event: throttled, op: pg_notify(3.310(10),3.4d(10) epoch 243) 2014-06-09 16:22:11.404091 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=212 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.404097 7f869af78700 5 -- op tracker -- , seq: 1584, time: 2014-06-09 16:22:11.086845, event: all_read, op: pg_notify(3.310(10),3.4d(10) epoch 243) 2014-06-09 16:22:11.404103 7f869af78700 5 -- op tracker -- , seq: 1584, time: 2014-06-09 16:22:11.404073, event: dispatched, op: pg_notify(3.310(10),3.4d(10) epoch 243) 2014-06-09 16:22:11.404102 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=212 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.404110 7f869af78700 5 -- op tracker -- , seq: 1584, time: 2014-06-09 16:22:11.404108, event: waiting_for_osdmap, op: pg_notify(3.310(10),3.4d(10) epoch 243) 2014-06-09 16:22:11.404114 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.404116 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x260001c0 2014-06-09 16:22:11.404112 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=212 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.404121 7f869af78700 5 -- op tracker -- , seq: 1584, time: 2014-06-09 16:22:11.404119, event: started, op: pg_notify(3.310(10),3.4d(10) epoch 243) 2014-06-09 16:22:11.404122 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'743 -> 213'743 2014-06-09 16:22:11.404149 7f869af78700 5 -- op tracker -- , seq: 1584, time: 2014-06-09 16:22:11.404146, event: done, op: pg_notify(3.310(10),3.4d(10) epoch 243) 2014-06-09 16:22:11.404157 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404158 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404174 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404176 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404177 7f869af78700 20 osd.26 243 _dispatch 0x2d732000 pg_notify(2.740(22) epoch 242) v5 2014-06-09 16:22:11.404172 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.404181 7f869af78700 5 -- op tracker -- , seq: 1585, time: 2014-06-09 16:22:10.898108, event: header_read, op: pg_notify(2.740(22) epoch 242) 2014-06-09 16:22:11.404188 7f869af78700 5 -- op tracker -- , seq: 1585, time: 2014-06-09 16:22:10.898108, event: throttled, op: pg_notify(2.740(22) epoch 242) 2014-06-09 16:22:11.404193 7f869af78700 5 -- op tracker -- , seq: 1585, time: 2014-06-09 16:22:10.898125, event: all_read, op: pg_notify(2.740(22) epoch 242) 2014-06-09 16:22:11.404199 7f869af78700 5 -- op tracker -- , seq: 1585, time: 2014-06-09 16:22:11.404174, event: dispatched, op: pg_notify(2.740(22) epoch 242) 2014-06-09 16:22:11.404206 7f869af78700 5 -- op tracker -- , seq: 1585, time: 2014-06-09 16:22:11.404204, event: waiting_for_osdmap, op: pg_notify(2.740(22) epoch 242) 2014-06-09 16:22:11.404209 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404210 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.404212 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d732000 2014-06-09 16:22:11.404213 7f8690f64700 5 queue_transactions existing osr(3.19 0x3ed03a0)/0x3ed03a0 2014-06-09 16:22:11.404216 7f8690f64700 10 op_submit_start 3122 2014-06-09 16:22:11.404217 7f8690f64700 5 queue_transactions (trailing journal) 3122 2014-06-09 16:22:11.404218 7f8690f64700 5 queue_op 0x30060960 seq 3122 osr(3.19 0x3ed03a0) 1332 bytes (queue has 8 ops and 13522 bytes) 2014-06-09 16:22:11.404216 7f869af78700 5 -- op tracker -- , seq: 1585, time: 2014-06-09 16:22:11.404214, event: started, op: pg_notify(2.740(22) epoch 242) 2014-06-09 16:22:11.404220 7f8690f64700 10 op_submit_finish 3122 2014-06-09 16:22:11.404228 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0240 (14 active) 2014-06-09 16:22:11.404232 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0240 (15 active) 2014-06-09 16:22:11.404235 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.404241 7f868ef60700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.404245 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.404237 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:242, info:0.1c8( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.404248 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.404249 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.404258 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404263 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.404255 7f868a757700 10 osd.26 243 dequeue_op 0x14052f00 prio 63 cost 516096 latency 0.950674 osd_op(client.4569.0:38602 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1482752~516096] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.404272 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.a2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.404279 7f868a757700 5 -- op tracker -- , seq: 375, time: 2014-06-09 16:22:11.404274, event: reached_pg, op: osd_op(client.4569.0:38602 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1482752~516096] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.404282 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.404286 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.404290 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.048452 7 0.000204 2014-06-09 16:22:11.404286 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.404300 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.404300 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x14052f00 2014-06-09 16:22:11.404311 7f868a757700 10 osd.26 243 dequeue_op 0x14052f00 finish 2014-06-09 16:22:11.404310 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.404317 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.404320 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.404319 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.404329 7f869af78700 5 -- op tracker -- , seq: 1585, time: 2014-06-09 16:22:11.404327, event: done, op: pg_notify(2.740(22) epoch 242) 2014-06-09 16:22:11.404328 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.404338 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404340 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404336 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.404357 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404358 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404359 7f869af78700 20 osd.26 243 _dispatch 0x11c67dc0 pg_notify(2.5d3(20) epoch 242) v5 2014-06-09 16:22:11.404364 7f869af78700 5 -- op tracker -- , seq: 1586, time: 2014-06-09 16:22:10.845796, event: header_read, op: pg_notify(2.5d3(20) epoch 242) 2014-06-09 16:22:11.404371 7f869af78700 5 -- op tracker -- , seq: 1586, time: 2014-06-09 16:22:10.845796, event: throttled, op: pg_notify(2.5d3(20) epoch 242) 2014-06-09 16:22:11.404377 7f869af78700 5 -- op tracker -- , seq: 1586, time: 2014-06-09 16:22:10.845822, event: all_read, op: pg_notify(2.5d3(20) epoch 242) 2014-06-09 16:22:11.404380 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404384 7f8690f64700 5 queue_transactions existing osr(1.a2 0x384c790)/0x384c790 2014-06-09 16:22:11.404383 7f869af78700 5 -- op tracker -- , seq: 1586, time: 2014-06-09 16:22:11.404356, event: dispatched, op: pg_notify(2.5d3(20) epoch 242) 2014-06-09 16:22:11.404387 7f8690f64700 10 op_submit_start 3123 2014-06-09 16:22:11.404388 7f8690f64700 5 queue_transactions (trailing journal) 3123 2014-06-09 16:22:11.404389 7f8690f64700 5 queue_op 0x30060900 seq 3123 osr(1.a2 0x384c790) 1880 bytes (queue has 9 ops and 15402 bytes) 2014-06-09 16:22:11.404391 7f8690f64700 10 op_submit_finish 3123 2014-06-09 16:22:11.404389 7f869af78700 5 -- op tracker -- , seq: 1586, time: 2014-06-09 16:22:11.404387, event: waiting_for_osdmap, op: pg_notify(2.5d3(20) epoch 242) 2014-06-09 16:22:11.404394 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.404396 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x11c67dc0 2014-06-09 16:22:11.404400 7f869af78700 5 -- op tracker -- , seq: 1586, time: 2014-06-09 16:22:11.404398, event: started, op: pg_notify(2.5d3(20) epoch 242) 2014-06-09 16:22:11.404394 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.310( v 213'668 (0'0,213'668] local-les=206 n=130 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.404407 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.404417 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404417 7f869af78700 5 -- op tracker -- , seq: 1586, time: 2014-06-09 16:22:11.404415, event: done, op: pg_notify(2.5d3(20) epoch 242) 2014-06-09 16:22:11.404425 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404426 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404420 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.4d( v 213'504 (0'0,213'504] local-les=206 n=102 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.404432 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.404441 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404446 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0240 (13 active) 2014-06-09 16:22:11.404446 7f869af78700 20 osd.26 243 _dispatch 0x3044ca80 pg_notify(2.e4(23) epoch 243) v5 2014-06-09 16:22:11.404450 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0240 (14 active) 2014-06-09 16:22:11.404452 7f869af78700 5 -- op tracker -- , seq: 1587, time: 2014-06-09 16:22:11.106844, event: header_read, op: pg_notify(2.e4(23) epoch 243) 2014-06-09 16:22:11.404453 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:2.740( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.404459 7f869af78700 5 -- op tracker -- , seq: 1587, time: 2014-06-09 16:22:11.106845, event: throttled, op: pg_notify(2.e4(23) epoch 243) 2014-06-09 16:22:11.404465 7f869af78700 5 -- op tracker -- , seq: 1587, time: 2014-06-09 16:22:11.106865, event: all_read, op: pg_notify(2.e4(23) epoch 243) 2014-06-09 16:22:11.404464 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.404471 7f869af78700 5 -- op tracker -- , seq: 1587, time: 2014-06-09 16:22:11.404443, event: dispatched, op: pg_notify(2.e4(23) epoch 243) 2014-06-09 16:22:11.404478 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404477 7f869af78700 5 -- op tracker -- , seq: 1587, time: 2014-06-09 16:22:11.404475, event: waiting_for_osdmap, op: pg_notify(2.e4(23) epoch 243) 2014-06-09 16:22:11.404481 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.404483 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044ca80 2014-06-09 16:22:11.404482 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:2.5d3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.404488 7f869af78700 5 -- op tracker -- , seq: 1587, time: 2014-06-09 16:22:11.404486, event: started, op: pg_notify(2.e4(23) epoch 243) 2014-06-09 16:22:11.404492 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.404501 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404505 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0240 (13 active) 2014-06-09 16:22:11.404505 7f869af78700 5 -- op tracker -- , seq: 1587, time: 2014-06-09 16:22:11.404503, event: done, op: pg_notify(2.e4(23) epoch 243) 2014-06-09 16:22:11.404508 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0240 (14 active) 2014-06-09 16:22:11.404513 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404515 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404511 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:2.e4( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.404522 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.404531 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.404532 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404533 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404534 7f869af78700 20 osd.26 243 _dispatch 0x1003a540 pg_notify(1.303(23) epoch 243) v5 2014-06-09 16:22:11.404535 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0240 (13 active) 2014-06-09 16:22:11.404537 7f8690f64700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.404539 7f869af78700 5 -- op tracker -- , seq: 1588, time: 2014-06-09 16:22:11.052466, event: header_read, op: pg_notify(1.303(23) epoch 243) 2014-06-09 16:22:11.404546 7f869af78700 5 -- op tracker -- , seq: 1588, time: 2014-06-09 16:22:11.052467, event: throttled, op: pg_notify(1.303(23) epoch 243) 2014-06-09 16:22:11.404552 7f869af78700 5 -- op tracker -- , seq: 1588, time: 2014-06-09 16:22:11.052492, event: all_read, op: pg_notify(1.303(23) epoch 243) 2014-06-09 16:22:11.404558 7f869af78700 5 -- op tracker -- , seq: 1588, time: 2014-06-09 16:22:11.404531, event: dispatched, op: pg_notify(1.303(23) epoch 243) 2014-06-09 16:22:11.404564 7f869af78700 5 -- op tracker -- , seq: 1588, time: 2014-06-09 16:22:11.404562, event: waiting_for_osdmap, op: pg_notify(1.303(23) epoch 243) 2014-06-09 16:22:11.404568 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.404570 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1003a540 2014-06-09 16:22:11.404574 7f869af78700 5 -- op tracker -- , seq: 1588, time: 2014-06-09 16:22:11.404573, event: started, op: pg_notify(1.303(23) epoch 243) 2014-06-09 16:22:11.404591 7f869af78700 5 -- op tracker -- , seq: 1588, time: 2014-06-09 16:22:11.404589, event: done, op: pg_notify(1.303(23) epoch 243) 2014-06-09 16:22:11.404599 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404601 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404620 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404621 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404622 7f869af78700 20 osd.26 243 _dispatch 0x305a7800 pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.404629 7f869af78700 5 -- op tracker -- , seq: 1589, time: 2014-06-09 16:22:11.071071, event: header_read, op: pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404636 7f869af78700 5 -- op tracker -- , seq: 1589, time: 2014-06-09 16:22:11.071071, event: throttled, op: pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404643 7f869af78700 5 -- op tracker -- , seq: 1589, time: 2014-06-09 16:22:11.071088, event: all_read, op: pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404649 7f869af78700 5 -- op tracker -- , seq: 1589, time: 2014-06-09 16:22:11.404619, event: dispatched, op: pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404656 7f869af78700 5 -- op tracker -- , seq: 1589, time: 2014-06-09 16:22:11.404653, event: waiting_for_osdmap, op: pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404660 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.404664 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a7800 2014-06-09 16:22:11.404670 7f869af78700 5 -- op tracker -- , seq: 1589, time: 2014-06-09 16:22:11.404667, event: started, op: pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404684 7f869af78700 5 -- op tracker -- , seq: 1589, time: 2014-06-09 16:22:11.404681, event: done, op: pg_log(1.560 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404693 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404704 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404706 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404707 7f869af78700 20 osd.26 243 _dispatch 0x159996c0 pg_notify(0.325(20) epoch 243) v5 2014-06-09 16:22:11.404711 7f869af78700 5 -- op tracker -- , seq: 1590, time: 2014-06-09 16:22:11.026888, event: header_read, op: pg_notify(0.325(20) epoch 243) 2014-06-09 16:22:11.404718 7f869af78700 5 -- op tracker -- , seq: 1590, time: 2014-06-09 16:22:11.026889, event: throttled, op: pg_notify(0.325(20) epoch 243) 2014-06-09 16:22:11.404723 7f869af78700 5 -- op tracker -- , seq: 1590, time: 2014-06-09 16:22:11.026918, event: all_read, op: pg_notify(0.325(20) epoch 243) 2014-06-09 16:22:11.404729 7f869af78700 5 -- op tracker -- , seq: 1590, time: 2014-06-09 16:22:11.404704, event: dispatched, op: pg_notify(0.325(20) epoch 243) 2014-06-09 16:22:11.404735 7f869af78700 5 -- op tracker -- , seq: 1590, time: 2014-06-09 16:22:11.404733, event: waiting_for_osdmap, op: pg_notify(0.325(20) epoch 243) 2014-06-09 16:22:11.404740 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.404741 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x159996c0 2014-06-09 16:22:11.404746 7f869af78700 5 -- op tracker -- , seq: 1590, time: 2014-06-09 16:22:11.404744, event: started, op: pg_notify(0.325(20) epoch 243) 2014-06-09 16:22:11.404763 7f869af78700 5 -- op tracker -- , seq: 1590, time: 2014-06-09 16:22:11.404761, event: done, op: pg_notify(0.325(20) epoch 243) 2014-06-09 16:22:11.404771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404773 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404790 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404791 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404792 7f869af78700 20 osd.26 243 _dispatch 0x2fe33800 pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.404799 7f869af78700 5 -- op tracker -- , seq: 1591, time: 2014-06-09 16:22:11.021322, event: header_read, op: pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404806 7f869af78700 5 -- op tracker -- , seq: 1591, time: 2014-06-09 16:22:11.021322, event: throttled, op: pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404812 7f869af78700 5 -- op tracker -- , seq: 1591, time: 2014-06-09 16:22:11.021338, event: all_read, op: pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404819 7f869af78700 5 -- op tracker -- , seq: 1591, time: 2014-06-09 16:22:11.404789, event: dispatched, op: pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404826 7f869af78700 5 -- op tracker -- , seq: 1591, time: 2014-06-09 16:22:11.404823, event: waiting_for_osdmap, op: pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404830 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.404834 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe33800 2014-06-09 16:22:11.404839 7f869af78700 5 -- op tracker -- , seq: 1591, time: 2014-06-09 16:22:11.404837, event: started, op: pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404851 7f869af78700 5 -- op tracker -- , seq: 1591, time: 2014-06-09 16:22:11.404849, event: done, op: pg_log(2.2b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404858 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404860 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404871 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404873 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404874 7f869af78700 20 osd.26 243 _dispatch 0x3002d440 osd_map(243..243 src has 1..243) v3 2014-06-09 16:22:11.404880 7f869af78700 3 osd.26 243 handle_osd_map epochs [243,243], i have 243, src has [1,243] 2014-06-09 16:22:11.404883 7f869af78700 10 osd.26 243 no new maps here, dropping 2014-06-09 16:22:11.404884 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404885 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404901 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404902 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404904 7f869af78700 20 osd.26 243 _dispatch 0x14dcd600 pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.404910 7f869af78700 5 -- op tracker -- , seq: 1592, time: 2014-06-09 16:22:11.038827, event: header_read, op: pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404918 7f869af78700 5 -- op tracker -- , seq: 1592, time: 2014-06-09 16:22:11.038827, event: throttled, op: pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404924 7f869af78700 5 -- op tracker -- , seq: 1592, time: 2014-06-09 16:22:11.038837, event: all_read, op: pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404931 7f869af78700 5 -- op tracker -- , seq: 1592, time: 2014-06-09 16:22:11.404901, event: dispatched, op: pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404938 7f869af78700 5 -- op tracker -- , seq: 1592, time: 2014-06-09 16:22:11.404936, event: waiting_for_osdmap, op: pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404943 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.404946 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dcd600 2014-06-09 16:22:11.404951 7f869af78700 5 -- op tracker -- , seq: 1592, time: 2014-06-09 16:22:11.404949, event: started, op: pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404964 7f869af78700 5 -- op tracker -- , seq: 1592, time: 2014-06-09 16:22:11.404961, event: done, op: pg_log(2.37a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.404971 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404972 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404984 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.404985 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.404986 7f869af78700 20 osd.26 243 _dispatch 0x305e4a00 pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.404992 7f869af78700 5 -- op tracker -- , seq: 1593, time: 2014-06-09 16:22:11.086866, event: header_read, op: pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.405000 7f869af78700 5 -- op tracker -- , seq: 1593, time: 2014-06-09 16:22:11.086867, event: throttled, op: pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.405007 7f869af78700 5 -- op tracker -- , seq: 1593, time: 2014-06-09 16:22:11.086880, event: all_read, op: pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.405013 7f869af78700 5 -- op tracker -- , seq: 1593, time: 2014-06-09 16:22:11.404983, event: dispatched, op: pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.405020 7f869af78700 5 -- op tracker -- , seq: 1593, time: 2014-06-09 16:22:11.405017, event: waiting_for_osdmap, op: pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.405024 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.405027 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e4a00 2014-06-09 16:22:11.405032 7f869af78700 5 -- op tracker -- , seq: 1593, time: 2014-06-09 16:22:11.405030, event: started, op: pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.405045 7f869af78700 5 -- op tracker -- , seq: 1593, time: 2014-06-09 16:22:11.405042, event: done, op: pg_log(0.54c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.405051 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405053 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405065 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405066 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405067 7f869af78700 20 osd.26 243 _dispatch 0x240e9500 pg_notify(0.722(27),3.18b(12),3.71f(12) epoch 242) v5 2014-06-09 16:22:11.405074 7f869af78700 5 -- op tracker -- , seq: 1594, time: 2014-06-09 16:22:10.924365, event: header_read, op: pg_notify(0.722(27),3.18b(12),3.71f(12) epoch 242) 2014-06-09 16:22:11.405082 7f869af78700 5 -- op tracker -- , seq: 1594, time: 2014-06-09 16:22:10.924365, event: throttled, op: pg_notify(0.722(27),3.18b(12),3.71f(12) epoch 242) 2014-06-09 16:22:11.405088 7f869af78700 5 -- op tracker -- , seq: 1594, time: 2014-06-09 16:22:10.924415, event: all_read, op: pg_notify(0.722(27),3.18b(12),3.71f(12) epoch 242) 2014-06-09 16:22:11.405095 7f869af78700 5 -- op tracker -- , seq: 1594, time: 2014-06-09 16:22:11.405064, event: dispatched, op: pg_notify(0.722(27),3.18b(12),3.71f(12) epoch 242) 2014-06-09 16:22:11.405102 7f869af78700 5 -- op tracker -- , seq: 1594, time: 2014-06-09 16:22:11.405099, event: waiting_for_osdmap, op: pg_notify(0.722(27),3.18b(12),3.71f(12) epoch 242) 2014-06-09 16:22:11.405106 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.405108 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240e9500 2014-06-09 16:22:11.405113 7f869af78700 5 -- op tracker -- , seq: 1594, time: 2014-06-09 16:22:11.405110, event: started, op: pg_notify(0.722(27),3.18b(12),3.71f(12) epoch 242) 2014-06-09 16:22:11.405146 7f869af78700 5 -- op tracker -- , seq: 1594, time: 2014-06-09 16:22:11.405143, event: done, op: pg_notify(0.722(27),3.18b(12),3.71f(12) epoch 242) 2014-06-09 16:22:11.405154 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405156 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405177 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405179 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405180 7f869af78700 20 osd.26 243 _dispatch 0x1ac94000 pg_notify(2.5fb(20) epoch 242) v5 2014-06-09 16:22:11.405185 7f869af78700 5 -- op tracker -- , seq: 1595, time: 2014-06-09 16:22:10.852685, event: header_read, op: pg_notify(2.5fb(20) epoch 242) 2014-06-09 16:22:11.405191 7f869af78700 5 -- op tracker -- , seq: 1595, time: 2014-06-09 16:22:10.852686, event: throttled, op: pg_notify(2.5fb(20) epoch 242) 2014-06-09 16:22:11.405197 7f869af78700 5 -- op tracker -- , seq: 1595, time: 2014-06-09 16:22:10.852713, event: all_read, op: pg_notify(2.5fb(20) epoch 242) 2014-06-09 16:22:11.405203 7f869af78700 5 -- op tracker -- , seq: 1595, time: 2014-06-09 16:22:11.405177, event: dispatched, op: pg_notify(2.5fb(20) epoch 242) 2014-06-09 16:22:11.405209 7f869af78700 5 -- op tracker -- , seq: 1595, time: 2014-06-09 16:22:11.405208, event: waiting_for_osdmap, op: pg_notify(2.5fb(20) epoch 242) 2014-06-09 16:22:11.405214 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.405215 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1ac94000 2014-06-09 16:22:11.405220 7f869af78700 5 -- op tracker -- , seq: 1595, time: 2014-06-09 16:22:11.405218, event: started, op: pg_notify(2.5fb(20) epoch 242) 2014-06-09 16:22:11.405235 7f869af78700 5 -- op tracker -- , seq: 1595, time: 2014-06-09 16:22:11.405233, event: done, op: pg_notify(2.5fb(20) epoch 242) 2014-06-09 16:22:11.405242 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405244 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405260 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405262 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405263 7f869af78700 20 osd.26 243 _dispatch 0x1abd5880 pg_info(1 pgs e243:3.e0(2)) v4 2014-06-09 16:22:11.405267 7f869af78700 5 -- op tracker -- , seq: 1596, time: 2014-06-09 16:22:11.106892, event: header_read, op: pg_info(1 pgs e243:3.e0(2)) 2014-06-09 16:22:11.405274 7f869af78700 5 -- op tracker -- , seq: 1596, time: 2014-06-09 16:22:11.106893, event: throttled, op: pg_info(1 pgs e243:3.e0(2)) 2014-06-09 16:22:11.405280 7f869af78700 5 -- op tracker -- , seq: 1596, time: 2014-06-09 16:22:11.106902, event: all_read, op: pg_info(1 pgs e243:3.e0(2)) 2014-06-09 16:22:11.405285 7f869af78700 5 -- op tracker -- , seq: 1596, time: 2014-06-09 16:22:11.405260, event: dispatched, op: pg_info(1 pgs e243:3.e0(2)) 2014-06-09 16:22:11.405292 7f869af78700 5 -- op tracker -- , seq: 1596, time: 2014-06-09 16:22:11.405290, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.e0(2)) 2014-06-09 16:22:11.405296 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.e0(2)) v4 from osd.7 2014-06-09 16:22:11.405299 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd5880 2014-06-09 16:22:11.405304 7f869af78700 5 -- op tracker -- , seq: 1596, time: 2014-06-09 16:22:11.405302, event: started, op: pg_info(1 pgs e243:3.e0(2)) 2014-06-09 16:22:11.405319 7f869af78700 5 -- op tracker -- , seq: 1596, time: 2014-06-09 16:22:11.405317, event: done, op: pg_info(1 pgs e243:3.e0(2)) 2014-06-09 16:22:11.405327 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405328 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405340 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405342 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405343 7f869af78700 20 osd.26 243 _dispatch 0x10d021c0 pg_notify(2.61b(23) epoch 243) v5 2014-06-09 16:22:11.405347 7f869af78700 5 -- op tracker -- , seq: 1597, time: 2014-06-09 16:22:11.057464, event: header_read, op: pg_notify(2.61b(23) epoch 243) 2014-06-09 16:22:11.405354 7f869af78700 5 -- op tracker -- , seq: 1597, time: 2014-06-09 16:22:11.057465, event: throttled, op: pg_notify(2.61b(23) epoch 243) 2014-06-09 16:22:11.405360 7f869af78700 5 -- op tracker -- , seq: 1597, time: 2014-06-09 16:22:11.057501, event: all_read, op: pg_notify(2.61b(23) epoch 243) 2014-06-09 16:22:11.405366 7f869af78700 5 -- op tracker -- , seq: 1597, time: 2014-06-09 16:22:11.405340, event: dispatched, op: pg_notify(2.61b(23) epoch 243) 2014-06-09 16:22:11.405372 7f869af78700 5 -- op tracker -- , seq: 1597, time: 2014-06-09 16:22:11.405371, event: waiting_for_osdmap, op: pg_notify(2.61b(23) epoch 243) 2014-06-09 16:22:11.405377 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.405378 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d021c0 2014-06-09 16:22:11.405383 7f869af78700 5 -- op tracker -- , seq: 1597, time: 2014-06-09 16:22:11.405381, event: started, op: pg_notify(2.61b(23) epoch 243) 2014-06-09 16:22:11.405397 7f869af78700 5 -- op tracker -- , seq: 1597, time: 2014-06-09 16:22:11.405395, event: done, op: pg_notify(2.61b(23) epoch 243) 2014-06-09 16:22:11.405404 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405406 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405426 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.405427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.405428 7f869af78700 20 osd.26 243 _dispatch 0x15a28c40 pg_notify(3.4e2(12) epoch 243) v5 2014-06-09 16:22:11.405433 7f869af78700 5 -- op tracker -- , seq: 1598, time: 2014-06-09 16:22:11.082304, event: header_read, op: pg_notify(3.4e2(12) epoch 243) 2014-06-09 16:22:11.405440 7f869af78700 5 -- op tracker -- , seq: 1598, time: 2014-06-09 16:22:11.082305, event: throttled, op: pg_notify(3.4e2(12) epoch 243) 2014-06-09 16:22:11.405446 7f869af78700 5 -- op tracker -- , seq: 1598, time: 2014-06-09 16:22:11.082324, event: all_read, op: pg_notify(3.4e2(12) epoch 243) 2014-06-09 16:22:11.405451 7f869af78700 5 -- op tracker -- , seq: 1598, time: 2014-06-09 16:22:11.405426, event: dispatched, op: pg_notify(3.4e2(12) epoch 243) 2014-06-09 16:22:11.405470 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (14 active) 2014-06-09 16:22:11.424317 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.303( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.424338 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.424349 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.424355 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.424366 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.560( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.424377 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.424381 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.424385 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.560( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.023391 7 0.000194 2014-06-09 16:22:11.424401 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.560( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.424413 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.560( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.424423 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.424432 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.424440 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.424489 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.424494 7f868ef60700 5 queue_transactions existing osr(1.560 0x384da90)/0x384da90 2014-06-09 16:22:11.424498 7f868ef60700 10 op_submit_start 3124 2014-06-09 16:22:11.424499 7f868ef60700 5 queue_transactions (trailing journal) 3124 2014-06-09 16:22:11.424501 7f868ef60700 5 queue_op 0x309a2cc0 seq 3124 osr(1.560 0x384da90) 1883 bytes (queue has 10 ops and 17285 bytes) 2014-06-09 16:22:11.424504 7f868ef60700 10 op_submit_finish 3124 2014-06-09 16:22:11.424506 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:0.325( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.424518 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.424527 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.424532 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.424541 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.2b( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.424552 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.424555 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.424558 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.260957 7 0.000240 2014-06-09 16:22:11.424566 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.424575 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.424584 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.424592 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.424599 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.424643 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.424647 7f868ef60700 5 queue_transactions existing osr(2.2b 0x2ed0230)/0x2ed0230 2014-06-09 16:22:11.424649 7f868ef60700 10 op_submit_start 3125 2014-06-09 16:22:11.424650 7f868ef60700 5 queue_transactions (trailing journal) 3125 2014-06-09 16:22:11.424651 7f868ef60700 5 queue_op 0x309a2c60 seq 3125 osr(2.2b 0x2ed0230) 1971 bytes (queue has 11 ops and 19256 bytes) 2014-06-09 16:22:11.424653 7f868ef60700 10 op_submit_finish 3125 2014-06-09 16:22:11.424656 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.424666 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.37a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.424677 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.424680 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.424683 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.181936 7 0.000195 2014-06-09 16:22:11.424692 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.424701 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.424709 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.424717 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.424724 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.424766 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.424770 7f868ef60700 5 queue_transactions existing osr(2.37a 0x2ed1160)/0x2ed1160 2014-06-09 16:22:11.424772 7f868ef60700 10 op_submit_start 3126 2014-06-09 16:22:11.424773 7f868ef60700 5 queue_transactions (trailing journal) 3126 2014-06-09 16:22:11.424774 7f868ef60700 5 queue_op 0x309a2c00 seq 3126 osr(2.37a 0x2ed1160) 1883 bytes (queue has 12 ops and 21139 bytes) 2014-06-09 16:22:11.424776 7f868ef60700 10 op_submit_finish 3126 2014-06-09 16:22:11.424779 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.424789 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.54c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.424799 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.424802 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.424805 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.829646 7 0.000207 2014-06-09 16:22:11.424814 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.424824 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.424832 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.424840 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.424847 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.424893 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.424897 7f868ef60700 5 queue_transactions existing osr(0.54c 0x2bd7850)/0x2bd7850 2014-06-09 16:22:11.424900 7f868ef60700 10 op_submit_start 3127 2014-06-09 16:22:11.424900 7f868ef60700 5 queue_transactions (trailing journal) 3127 2014-06-09 16:22:11.424901 7f868ef60700 5 queue_op 0x309a2ba0 seq 3127 osr(0.54c 0x2bd7850) 2001 bytes (queue has 13 ops and 23140 bytes) 2014-06-09 16:22:11.424904 7f868ef60700 10 op_submit_finish 3127 2014-06-09 16:22:11.424906 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:0.722( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.424918 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.424927 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.424931 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:3.18b( v 213'575 (0'0,213'575] local-les=191 n=113 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.424944 7f868ef60700 7 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.424954 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.18b( v 213'575 (0'0,213'575] local-les=191 n=113 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.424973 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.424976 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:3.71f( v 213'532 (0'0,213'532] local-les=191 n=101 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.424988 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.424998 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.425001 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:2.5fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.425012 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.425023 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.425027 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=210 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.425038 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=210 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.425049 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=210 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.943054 7 0.000187 2014-06-09 16:22:11.425059 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=210 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.425068 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=210 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.425077 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=210 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.425086 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'736 -> 213'736 2014-06-09 16:22:11.425134 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.411635 7f8689f56700 10 osd.26 243 dequeue_op 0x1c54be00 prio 63 cost 516096 latency 0.907054 osd_op(client.4574.0:31826 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1335296~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.416576 7f86a5e11700 10 _do_op 0x30033740 seq 3115 r = 0, finisher 0xbd6b7a0 0 2014-06-09 16:22:11.425168 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.425167 7f86a5e11700 10 _finish_op 0x30033740 seq 3115 osr(0.521 0x2bd7570)/0x2bd7570 2014-06-09 16:22:11.425173 7f868ef60700 5 queue_transactions existing osr(3.e0 0x3ed0500)/0x3ed0500 2014-06-09 16:22:11.425175 7f868ef60700 10 op_submit_start 3128 2014-06-09 16:22:11.425176 7f868ef60700 5 queue_transactions (trailing journal) 3128 2014-06-09 16:22:11.425177 7f868ef60700 5 queue_op 0x309a2b40 seq 3128 osr(3.e0 0x3ed0500) 1332 bytes (queue has 13 ops and 22471 bytes) 2014-06-09 16:22:11.425176 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc240 (1 active) 2014-06-09 16:22:11.425180 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ed80 (2 active) 2014-06-09 16:22:11.425184 7f8690763700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.425185 7f86a5e11700 5 _do_op 0x308ca3c0 seq 3117 osr(0.8c 0x2bd6480)/0x2bd6480 start 2014-06-09 16:22:11.425189 7f86a5e11700 10 _do_transaction on 0xbe8b140 2014-06-09 16:22:11.425193 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.425199 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.425199 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.425205 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.425211 7f8690f64700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.425217 7f8692767700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.416633 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.405458 7f869af78700 5 -- op tracker -- , seq: 1598, time: 2014-06-09 16:22:11.405456, event: waiting_for_osdmap, op: pg_notify(3.4e2(12) epoch 243) 2014-06-09 16:22:11.425283 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.425286 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a28c40 2014-06-09 16:22:11.425294 7f869af78700 5 -- op tracker -- , seq: 1598, time: 2014-06-09 16:22:11.425291, event: started, op: pg_notify(3.4e2(12) epoch 243) 2014-06-09 16:22:11.425302 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3116.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.425308 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.425315 7f869af78700 5 -- op tracker -- , seq: 1598, time: 2014-06-09 16:22:11.425313, event: done, op: pg_notify(3.4e2(12) epoch 243) 2014-06-09 16:22:11.425331 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425333 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425340 7f86a5610700 10 _do_op 0x300336e0 seq 3116 r = 0, finisher 0xbd5be80 0 2014-06-09 16:22:11.425344 7f86a5610700 10 _finish_op 0x300336e0 seq 3116 osr(2.6ec 0x2ed1e10)/0x2ed1e10 2014-06-09 16:22:11.425348 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338360 (1 active) 2014-06-09 16:22:11.425350 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301c20 (2 active) 2014-06-09 16:22:11.425353 7f86a5610700 5 _do_op 0x308ca300 seq 3118 osr(3.6d5 0x3ed1c30)/0x3ed1c30 start 2014-06-09 16:22:11.425354 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425356 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425356 7f86a5610700 10 _do_transaction on 0xbe68d80 2014-06-09 16:22:11.425357 7f869af78700 20 osd.26 243 _dispatch 0x14dcca00 pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.425362 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.425363 7f869af78700 5 -- op tracker -- , seq: 1599, time: 2014-06-09 16:22:11.058555, event: header_read, op: pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425371 7f869af78700 5 -- op tracker -- , seq: 1599, time: 2014-06-09 16:22:11.058555, event: throttled, op: pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425377 7f869af78700 5 -- op tracker -- , seq: 1599, time: 2014-06-09 16:22:11.058569, event: all_read, op: pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425383 7f869af78700 5 -- op tracker -- , seq: 1599, time: 2014-06-09 16:22:11.425353, event: dispatched, op: pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425390 7f869af78700 5 -- op tracker -- , seq: 1599, time: 2014-06-09 16:22:11.425388, event: waiting_for_osdmap, op: pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425394 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.425398 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dcca00 2014-06-09 16:22:11.425400 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.425403 7f869af78700 5 -- op tracker -- , seq: 1599, time: 2014-06-09 16:22:11.425400, event: started, op: pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425417 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3118.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.425417 7f869af78700 5 -- op tracker -- , seq: 1599, time: 2014-06-09 16:22:11.425414, event: done, op: pg_log(0.e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425421 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.425424 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425426 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425438 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425439 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425440 7f869af78700 20 osd.26 243 _dispatch 0x2fe35600 pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.425446 7f869af78700 5 -- op tracker -- , seq: 1600, time: 2014-06-09 16:22:11.021351, event: header_read, op: pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425453 7f869af78700 5 -- op tracker -- , seq: 1600, time: 2014-06-09 16:22:11.021352, event: throttled, op: pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425459 7f869af78700 5 -- op tracker -- , seq: 1600, time: 2014-06-09 16:22:11.021364, event: all_read, op: pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425464 7f86a5610700 10 _do_op 0x308ca300 seq 3118 r = 0, finisher 0xbd652c0 0 2014-06-09 16:22:11.425466 7f869af78700 5 -- op tracker -- , seq: 1600, time: 2014-06-09 16:22:11.425438, event: dispatched, op: pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425468 7f86a5610700 10 _finish_op 0x308ca300 seq 3118 osr(3.6d5 0x3ed1c30)/0x3ed1c30 2014-06-09 16:22:11.425471 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301c20 (1 active) 2014-06-09 16:22:11.425474 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f560 (2 active) 2014-06-09 16:22:11.425473 7f869af78700 5 -- op tracker -- , seq: 1600, time: 2014-06-09 16:22:11.425470, event: waiting_for_osdmap, op: pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425476 7f86a5610700 5 _do_op 0x308ca2a0 seq 3119 osr(3.359 0x3ed10f0)/0x3ed10f0 start 2014-06-09 16:22:11.425477 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.425479 7f86a5610700 10 _do_transaction on 0x390c6c0 2014-06-09 16:22:11.425480 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe35600 2014-06-09 16:22:11.425485 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.425485 7f869af78700 5 -- op tracker -- , seq: 1600, time: 2014-06-09 16:22:11.425483, event: started, op: pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425498 7f869af78700 5 -- op tracker -- , seq: 1600, time: 2014-06-09 16:22:11.425496, event: done, op: pg_log(2.65 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425507 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425509 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425520 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425519 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.425522 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425523 7f869af78700 20 osd.26 243 _dispatch 0x1c039a40 pg_notify(2.19(20) epoch 243) v5 2014-06-09 16:22:11.425527 7f869af78700 5 -- op tracker -- , seq: 1601, time: 2014-06-09 16:22:11.009078, event: header_read, op: pg_notify(2.19(20) epoch 243) 2014-06-09 16:22:11.425533 7f869af78700 5 -- op tracker -- , seq: 1601, time: 2014-06-09 16:22:11.009079, event: throttled, op: pg_notify(2.19(20) epoch 243) 2014-06-09 16:22:11.425535 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3119.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.425540 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.425539 7f869af78700 5 -- op tracker -- , seq: 1601, time: 2014-06-09 16:22:11.009103, event: all_read, op: pg_notify(2.19(20) epoch 243) 2014-06-09 16:22:11.425544 7f869af78700 5 -- op tracker -- , seq: 1601, time: 2014-06-09 16:22:11.425520, event: dispatched, op: pg_notify(2.19(20) epoch 243) 2014-06-09 16:22:11.425550 7f869af78700 5 -- op tracker -- , seq: 1601, time: 2014-06-09 16:22:11.425549, event: waiting_for_osdmap, op: pg_notify(2.19(20) epoch 243) 2014-06-09 16:22:11.425554 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.425556 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c039a40 2014-06-09 16:22:11.425557 7f86a5610700 10 _do_op 0x308ca2a0 seq 3119 r = 0, finisher 0xbd751e0 0 2014-06-09 16:22:11.425561 7f86a5610700 10 _finish_op 0x308ca2a0 seq 3119 osr(3.359 0x3ed10f0)/0x3ed10f0 2014-06-09 16:22:11.425560 7f869af78700 5 -- op tracker -- , seq: 1601, time: 2014-06-09 16:22:11.425558, event: started, op: pg_notify(2.19(20) epoch 243) 2014-06-09 16:22:11.425563 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f560 (1 active) 2014-06-09 16:22:11.425566 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d10e0 (2 active) 2014-06-09 16:22:11.425568 7f86a5610700 5 _do_op 0x308ca240 seq 3120 osr(2.530 0x2ed19b0)/0x2ed19b0 start 2014-06-09 16:22:11.425571 7f86a5610700 10 _do_transaction on 0x3128cc0 2014-06-09 16:22:11.425577 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.425577 7f869af78700 5 -- op tracker -- , seq: 1601, time: 2014-06-09 16:22:11.425575, event: done, op: pg_notify(2.19(20) epoch 243) 2014-06-09 16:22:11.425585 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425586 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425603 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425604 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425605 7f869af78700 20 osd.26 243 _dispatch 0x1607afc0 pg_notify(1.36f(21),0.370(21) epoch 243) v5 2014-06-09 16:22:11.425609 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.425611 7f869af78700 5 -- op tracker -- , seq: 1602, time: 2014-06-09 16:22:11.038852, event: header_read, op: pg_notify(1.36f(21),0.370(21) epoch 243) 2014-06-09 16:22:11.425617 7f869af78700 5 -- op tracker -- , seq: 1602, time: 2014-06-09 16:22:11.038853, event: throttled, op: pg_notify(1.36f(21),0.370(21) epoch 243) 2014-06-09 16:22:11.425624 7f869af78700 5 -- op tracker -- , seq: 1602, time: 2014-06-09 16:22:11.038884, event: all_read, op: pg_notify(1.36f(21),0.370(21) epoch 243) 2014-06-09 16:22:11.425625 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3120.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.425630 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.425630 7f869af78700 5 -- op tracker -- , seq: 1602, time: 2014-06-09 16:22:11.425603, event: dispatched, op: pg_notify(1.36f(21),0.370(21) epoch 243) 2014-06-09 16:22:11.425636 7f869af78700 5 -- op tracker -- , seq: 1602, time: 2014-06-09 16:22:11.425634, event: waiting_for_osdmap, op: pg_notify(1.36f(21),0.370(21) epoch 243) 2014-06-09 16:22:11.425640 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.425642 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1607afc0 2014-06-09 16:22:11.425646 7f869af78700 5 -- op tracker -- , seq: 1602, time: 2014-06-09 16:22:11.425644, event: started, op: pg_notify(1.36f(21),0.370(21) epoch 243) 2014-06-09 16:22:11.425663 7f86a5610700 10 _do_op 0x308ca240 seq 3120 r = 0, finisher 0xbd6c1e0 0 2014-06-09 16:22:11.425666 7f86a5610700 10 _finish_op 0x308ca240 seq 3120 osr(2.530 0x2ed19b0)/0x2ed19b0 2014-06-09 16:22:11.425669 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d10e0 (1 active) 2014-06-09 16:22:11.425672 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d440 (2 active) 2014-06-09 16:22:11.425671 7f869af78700 5 -- op tracker -- , seq: 1602, time: 2014-06-09 16:22:11.425668, event: done, op: pg_notify(1.36f(21),0.370(21) epoch 243) 2014-06-09 16:22:11.425674 7f86a5610700 5 _do_op 0x308ca0c0 seq 3121 osr(1.56 0x384c5f0)/0x384c5f0 start 2014-06-09 16:22:11.425676 7f86a5610700 10 _do_transaction on 0xc3f2cc0 2014-06-09 16:22:11.425678 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425680 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425682 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.425700 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425702 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425703 7f869af78700 20 osd.26 243 _dispatch 0x305e4400 pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.425709 7f869af78700 5 -- op tracker -- , seq: 1603, time: 2014-06-09 16:22:11.086918, event: header_read, op: pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425715 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.425715 7f869af78700 5 -- op tracker -- , seq: 1603, time: 2014-06-09 16:22:11.086919, event: throttled, op: pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425722 7f869af78700 5 -- op tracker -- , seq: 1603, time: 2014-06-09 16:22:11.086930, event: all_read, op: pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425728 7f869af78700 5 -- op tracker -- , seq: 1603, time: 2014-06-09 16:22:11.425700, event: dispatched, op: pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425731 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3121.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.425736 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.425735 7f869af78700 5 -- op tracker -- , seq: 1603, time: 2014-06-09 16:22:11.425732, event: waiting_for_osdmap, op: pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425739 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.425742 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e4400 2014-06-09 16:22:11.425747 7f869af78700 5 -- op tracker -- , seq: 1603, time: 2014-06-09 16:22:11.425745, event: started, op: pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425754 7f86a5610700 10 _do_op 0x308ca0c0 seq 3121 r = 0, finisher 0xbd63c60 0 2014-06-09 16:22:11.425757 7f86a5610700 10 _finish_op 0x308ca0c0 seq 3121 osr(1.56 0x384c5f0)/0x384c5f0 2014-06-09 16:22:11.425760 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d440 (1 active) 2014-06-09 16:22:11.425759 7f869af78700 5 -- op tracker -- , seq: 1603, time: 2014-06-09 16:22:11.425757, event: done, op: pg_log(2.3c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425762 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383ed80 (2 active) 2014-06-09 16:22:11.425764 7f86a5610700 5 _do_op 0x30060960 seq 3122 osr(3.19 0x3ed03a0)/0x3ed03a0 start 2014-06-09 16:22:11.425766 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425767 7f86a5610700 10 _do_transaction on 0x3540f00 2014-06-09 16:22:11.425767 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425773 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.425779 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425782 7f869af78700 20 osd.26 243 _dispatch 0x2d735200 pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.425788 7f869af78700 5 -- op tracker -- , seq: 1604, time: 2014-06-09 16:22:10.931693, event: header_read, op: pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.425795 7f869af78700 5 -- op tracker -- , seq: 1604, time: 2014-06-09 16:22:10.931694, event: throttled, op: pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.425801 7f869af78700 5 -- op tracker -- , seq: 1604, time: 2014-06-09 16:22:10.932516, event: all_read, op: pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.425805 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.425808 7f869af78700 5 -- op tracker -- , seq: 1604, time: 2014-06-09 16:22:11.425779, event: dispatched, op: pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.425814 7f869af78700 5 -- op tracker -- , seq: 1604, time: 2014-06-09 16:22:11.425812, event: waiting_for_osdmap, op: pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.425818 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) v4 from osd.2 2014-06-09 16:22:11.425822 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d735200 2014-06-09 16:22:11.425821 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3122.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.425826 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.425827 7f869af78700 5 -- op tracker -- , seq: 1604, time: 2014-06-09 16:22:11.425824, event: started, op: pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.425838 7f869af78700 5 -- op tracker -- , seq: 1604, time: 2014-06-09 16:22:11.425836, event: done, op: pg_log(3.2b epoch 242 log log((0'0,213'613], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.425841 7f86a5610700 10 _do_op 0x30060960 seq 3122 r = 0, finisher 0xc6e9200 0 2014-06-09 16:22:11.425845 7f86a5610700 10 _finish_op 0x30060960 seq 3122 osr(3.19 0x3ed03a0)/0x3ed03a0 2014-06-09 16:22:11.425847 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383ed80 (1 active) 2014-06-09 16:22:11.425849 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3398c0 (2 active) 2014-06-09 16:22:11.425851 7f86a5610700 5 _do_op 0x30060900 seq 3123 osr(1.a2 0x384c790)/0x384c790 start 2014-06-09 16:22:11.425854 7f86a5610700 10 _do_transaction on 0xc252540 2014-06-09 16:22:11.425854 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425855 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425859 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.425867 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425869 7f869af78700 20 osd.26 243 _dispatch 0x2d095a40 pg_notify(3.6d8(10) epoch 242) v5 2014-06-09 16:22:11.425874 7f869af78700 5 -- op tracker -- , seq: 1605, time: 2014-06-09 16:22:10.860541, event: header_read, op: pg_notify(3.6d8(10) epoch 242) 2014-06-09 16:22:11.425880 7f869af78700 5 -- op tracker -- , seq: 1605, time: 2014-06-09 16:22:10.860542, event: throttled, op: pg_notify(3.6d8(10) epoch 242) 2014-06-09 16:22:11.425885 7f869af78700 5 -- op tracker -- , seq: 1605, time: 2014-06-09 16:22:10.860565, event: all_read, op: pg_notify(3.6d8(10) epoch 242) 2014-06-09 16:22:11.425891 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.425891 7f869af78700 5 -- op tracker -- , seq: 1605, time: 2014-06-09 16:22:11.425866, event: dispatched, op: pg_notify(3.6d8(10) epoch 242) 2014-06-09 16:22:11.425897 7f869af78700 5 -- op tracker -- , seq: 1605, time: 2014-06-09 16:22:11.425895, event: waiting_for_osdmap, op: pg_notify(3.6d8(10) epoch 242) 2014-06-09 16:22:11.425900 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.425902 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d095a40 2014-06-09 16:22:11.425906 7f869af78700 5 -- op tracker -- , seq: 1605, time: 2014-06-09 16:22:11.425905, event: started, op: pg_notify(3.6d8(10) epoch 242) 2014-06-09 16:22:11.425907 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3123.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.425911 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.425921 7f869af78700 5 -- op tracker -- , seq: 1605, time: 2014-06-09 16:22:11.425920, event: done, op: pg_notify(3.6d8(10) epoch 242) 2014-06-09 16:22:11.425928 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425930 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425938 7f86a5610700 10 _do_op 0x30060900 seq 3123 r = 0, finisher 0xc2d63a0 0 2014-06-09 16:22:11.425942 7f86a5610700 10 _finish_op 0x30060900 seq 3123 osr(1.a2 0x384c790)/0x384c790 2014-06-09 16:22:11.425944 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.425945 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.425945 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3398c0 (1 active) 2014-06-09 16:22:11.425947 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc7e0 (2 active) 2014-06-09 16:22:11.425946 7f869af78700 20 osd.26 243 _dispatch 0x30573000 pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.425949 7f86a5610700 5 _do_op 0x309a2cc0 seq 3124 osr(1.560 0x384da90)/0x384da90 start 2014-06-09 16:22:11.425952 7f86a5610700 10 _do_transaction on 0xbf47440 2014-06-09 16:22:11.425952 7f869af78700 5 -- op tracker -- , seq: 1606, time: 2014-06-09 16:22:11.106970, event: header_read, op: pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425957 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.425960 7f869af78700 5 -- op tracker -- , seq: 1606, time: 2014-06-09 16:22:11.106970, event: throttled, op: pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425966 7f869af78700 5 -- op tracker -- , seq: 1606, time: 2014-06-09 16:22:11.106978, event: all_read, op: pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425972 7f869af78700 5 -- op tracker -- , seq: 1606, time: 2014-06-09 16:22:11.425944, event: dispatched, op: pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425979 7f869af78700 5 -- op tracker -- , seq: 1606, time: 2014-06-09 16:22:11.425976, event: waiting_for_osdmap, op: pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.425983 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.425987 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30573000 2014-06-09 16:22:11.425989 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.425991 7f869af78700 5 -- op tracker -- , seq: 1606, time: 2014-06-09 16:22:11.425989, event: started, op: pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426003 7f869af78700 5 -- op tracker -- , seq: 1606, time: 2014-06-09 16:22:11.426000, event: done, op: pg_log(2.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426005 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3124.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.426010 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.426010 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426012 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426024 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426025 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426026 7f869af78700 20 osd.26 243 _dispatch 0x240e8fc0 pg_notify(1.373(20),1.641(20) epoch 243) v5 2014-06-09 16:22:11.426028 7f86a5610700 10 _do_op 0x309a2cc0 seq 3124 r = 0, finisher 0xbe308c0 0 2014-06-09 16:22:11.426032 7f86a5610700 10 _finish_op 0x309a2cc0 seq 3124 osr(1.560 0x384da90)/0x384da90 2014-06-09 16:22:11.426032 7f869af78700 5 -- op tracker -- , seq: 1607, time: 2014-06-09 16:22:11.079862, event: header_read, op: pg_notify(1.373(20),1.641(20) epoch 243) 2014-06-09 16:22:11.426034 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc7e0 (1 active) 2014-06-09 16:22:11.426037 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4d80 (2 active) 2014-06-09 16:22:11.426039 7f86a5610700 5 _do_op 0x309a2c60 seq 3125 osr(2.2b 0x2ed0230)/0x2ed0230 start 2014-06-09 16:22:11.426039 7f869af78700 5 -- op tracker -- , seq: 1607, time: 2014-06-09 16:22:11.079863, event: throttled, op: pg_notify(1.373(20),1.641(20) epoch 243) 2014-06-09 16:22:11.426041 7f86a5610700 10 _do_transaction on 0xbf39080 2014-06-09 16:22:11.426046 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.426045 7f869af78700 5 -- op tracker -- , seq: 1607, time: 2014-06-09 16:22:11.079899, event: all_read, op: pg_notify(1.373(20),1.641(20) epoch 243) 2014-06-09 16:22:11.426051 7f869af78700 5 -- op tracker -- , seq: 1607, time: 2014-06-09 16:22:11.426024, event: dispatched, op: pg_notify(1.373(20),1.641(20) epoch 243) 2014-06-09 16:22:11.426058 7f869af78700 5 -- op tracker -- , seq: 1607, time: 2014-06-09 16:22:11.426055, event: waiting_for_osdmap, op: pg_notify(1.373(20),1.641(20) epoch 243) 2014-06-09 16:22:11.426062 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.426063 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e8fc0 2014-06-09 16:22:11.426068 7f869af78700 5 -- op tracker -- , seq: 1607, time: 2014-06-09 16:22:11.426066, event: started, op: pg_notify(1.373(20),1.641(20) epoch 243) 2014-06-09 16:22:11.426078 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.426089 7f869af78700 5 -- op tracker -- , seq: 1607, time: 2014-06-09 16:22:11.426087, event: done, op: pg_notify(1.373(20),1.641(20) epoch 243) 2014-06-09 16:22:11.426093 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3125.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.426097 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.426098 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426100 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426120 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426121 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426122 7f869af78700 20 osd.26 243 _dispatch 0x305a7200 pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.426125 7f86a5610700 10 _do_op 0x309a2c60 seq 3125 r = 0, finisher 0xcc83620 0 2014-06-09 16:22:11.426128 7f86a5610700 10 _finish_op 0x309a2c60 seq 3125 osr(2.2b 0x2ed0230)/0x2ed0230 2014-06-09 16:22:11.426128 7f869af78700 5 -- op tracker -- , seq: 1608, time: 2014-06-09 16:22:11.085116, event: header_read, op: pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426131 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4d80 (1 active) 2014-06-09 16:22:11.426133 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286240 (2 active) 2014-06-09 16:22:11.426135 7f86a5610700 5 _do_op 0x309a2c00 seq 3126 osr(2.37a 0x2ed1160)/0x2ed1160 start 2014-06-09 16:22:11.426138 7f86a5610700 10 _do_transaction on 0x2c18540 2014-06-09 16:22:11.426136 7f869af78700 5 -- op tracker -- , seq: 1608, time: 2014-06-09 16:22:11.085116, event: throttled, op: pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426143 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.426142 7f869af78700 5 -- op tracker -- , seq: 1608, time: 2014-06-09 16:22:11.085129, event: all_read, op: pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426148 7f869af78700 5 -- op tracker -- , seq: 1608, time: 2014-06-09 16:22:11.426119, event: dispatched, op: pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426155 7f869af78700 5 -- op tracker -- , seq: 1608, time: 2014-06-09 16:22:11.426152, event: waiting_for_osdmap, op: pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426159 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.426162 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a7200 2014-06-09 16:22:11.426167 7f869af78700 5 -- op tracker -- , seq: 1608, time: 2014-06-09 16:22:11.426165, event: started, op: pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426175 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.426179 7f869af78700 5 -- op tracker -- , seq: 1608, time: 2014-06-09 16:22:11.426176, event: done, op: pg_log(0.644 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426185 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426187 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426191 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3126.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.426195 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.426198 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426199 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426200 7f869af78700 20 osd.26 243 _dispatch 0x14dcb800 pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.426206 7f869af78700 5 -- op tracker -- , seq: 1609, time: 2014-06-09 16:22:11.058583, event: header_read, op: pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426212 7f86a5610700 10 _do_op 0x309a2c00 seq 3126 r = 0, finisher 0xbd7b2c0 0 2014-06-09 16:22:11.426213 7f869af78700 5 -- op tracker -- , seq: 1609, time: 2014-06-09 16:22:11.058584, event: throttled, op: pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426215 7f86a5610700 10 _finish_op 0x309a2c00 seq 3126 osr(2.37a 0x2ed1160)/0x2ed1160 2014-06-09 16:22:11.426218 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286240 (1 active) 2014-06-09 16:22:11.426220 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ea20 (2 active) 2014-06-09 16:22:11.426219 7f869af78700 5 -- op tracker -- , seq: 1609, time: 2014-06-09 16:22:11.058598, event: all_read, op: pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426223 7f86a5610700 5 _do_op 0x309a2ba0 seq 3127 osr(0.54c 0x2bd7850)/0x2bd7850 start 2014-06-09 16:22:11.426225 7f86a5610700 10 _do_transaction on 0xbd8e180 2014-06-09 16:22:11.426225 7f869af78700 5 -- op tracker -- , seq: 1609, time: 2014-06-09 16:22:11.426197, event: dispatched, op: pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426231 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.426232 7f869af78700 5 -- op tracker -- , seq: 1609, time: 2014-06-09 16:22:11.426229, event: waiting_for_osdmap, op: pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426236 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.426239 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dcb800 2014-06-09 16:22:11.426244 7f869af78700 5 -- op tracker -- , seq: 1609, time: 2014-06-09 16:22:11.426242, event: started, op: pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426256 7f869af78700 5 -- op tracker -- , seq: 1609, time: 2014-06-09 16:22:11.426253, event: done, op: pg_log(2.67c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426263 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426262 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.426264 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426275 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426276 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426277 7f869af78700 20 osd.26 243 _dispatch 0x2fe35000 pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.426277 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3127.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.426282 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.426283 7f869af78700 5 -- op tracker -- , seq: 1610, time: 2014-06-09 16:22:11.021903, event: header_read, op: pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426290 7f869af78700 5 -- op tracker -- , seq: 1610, time: 2014-06-09 16:22:11.021904, event: throttled, op: pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426296 7f869af78700 5 -- op tracker -- , seq: 1610, time: 2014-06-09 16:22:11.021916, event: all_read, op: pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426302 7f869af78700 5 -- op tracker -- , seq: 1610, time: 2014-06-09 16:22:11.426274, event: dispatched, op: pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426309 7f86a5610700 10 _do_op 0x309a2ba0 seq 3127 r = 0, finisher 0xbd5b960 0 2014-06-09 16:22:11.426309 7f869af78700 5 -- op tracker -- , seq: 1610, time: 2014-06-09 16:22:11.426306, event: waiting_for_osdmap, op: pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426312 7f86a5610700 10 _finish_op 0x309a2ba0 seq 3127 osr(0.54c 0x2bd7850)/0x2bd7850 2014-06-09 16:22:11.426313 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.426315 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ea20 (1 active) 2014-06-09 16:22:11.426317 7f86a5610700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.426317 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe35000 2014-06-09 16:22:11.426321 7f869af78700 5 -- op tracker -- , seq: 1610, time: 2014-06-09 16:22:11.426319, event: started, op: pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426324 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061f3a0,0xbd768a0,0x3061f720,0xbd68980,0xbd746e0,0x2d738e20,0x3050bb00,0xbd78100,0xbd78a20,0xbd77880,0xbfcf880,0x17001820] 2014-06-09 16:22:11.426333 7f869af78700 5 -- op tracker -- , seq: 1610, time: 2014-06-09 16:22:11.426330, event: done, op: pg_log(1.601 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426339 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426341 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426341 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061f3a0,0xbd768a0,0x3061f720,0xbd68980,0xbd746e0,0x2d738e20,0x3050bb00,0xbd78100,0xbd78a20,0xbd77880,0xbfcf880,0x17001820] 2014-06-09 16:22:11.426346 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.426347 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.426351 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426352 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426351 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6b7a0,0xbd5be80,0xbd652c0,0xbd751e0,0xbd6c1e0,0xbd63c60,0xc6e9200,0xc2d63a0,0xbe308c0,0xcc83620,0xbd7b2c0,0xbd5b960] 2014-06-09 16:22:11.426353 7f869af78700 20 osd.26 243 _dispatch 0x19fda700 pg_notify(0.29(20) epoch 243) v5 2014-06-09 16:22:11.426358 7f869af78700 5 -- op tracker -- , seq: 1611, time: 2014-06-09 16:22:11.009146, event: header_read, op: pg_notify(0.29(20) epoch 243) 2014-06-09 16:22:11.426365 7f869af78700 5 -- op tracker -- , seq: 1611, time: 2014-06-09 16:22:11.009146, event: throttled, op: pg_notify(0.29(20) epoch 243) 2014-06-09 16:22:11.426357 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426371 7f869af78700 5 -- op tracker -- , seq: 1611, time: 2014-06-09 16:22:11.009164, event: all_read, op: pg_notify(0.29(20) epoch 243) 2014-06-09 16:22:11.426376 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.426376 7f869af78700 5 -- op tracker -- , seq: 1611, time: 2014-06-09 16:22:11.426351, event: dispatched, op: pg_notify(0.29(20) epoch 243) 2014-06-09 16:22:11.426382 7f869af78700 5 -- op tracker -- , seq: 1611, time: 2014-06-09 16:22:11.426381, event: waiting_for_osdmap, op: pg_notify(0.29(20) epoch 243) 2014-06-09 16:22:11.426386 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.426388 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fda700 2014-06-09 16:22:11.426392 7f869af78700 5 -- op tracker -- , seq: 1611, time: 2014-06-09 16:22:11.426390, event: started, op: pg_notify(0.29(20) epoch 243) 2014-06-09 16:22:11.426396 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.426406 7f869af78700 5 -- op tracker -- , seq: 1611, time: 2014-06-09 16:22:11.426404, event: done, op: pg_notify(0.29(20) epoch 243) 2014-06-09 16:22:11.426416 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426418 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426414 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426429 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.426433 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426435 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426436 7f869af78700 20 osd.26 243 _dispatch 0x14dcd000 pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.426442 7f869af78700 5 -- op tracker -- , seq: 1612, time: 2014-06-09 16:22:11.050825, event: header_read, op: pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426449 7f869af78700 5 -- op tracker -- , seq: 1612, time: 2014-06-09 16:22:11.050825, event: throttled, op: pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426447 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.426455 7f869af78700 5 -- op tracker -- , seq: 1612, time: 2014-06-09 16:22:11.050838, event: all_read, op: pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426461 7f869af78700 5 -- op tracker -- , seq: 1612, time: 2014-06-09 16:22:11.426433, event: dispatched, op: pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426468 7f869af78700 5 -- op tracker -- , seq: 1612, time: 2014-06-09 16:22:11.426466, event: waiting_for_osdmap, op: pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426463 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426472 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.426476 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dcd000 2014-06-09 16:22:11.426479 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.426481 7f869af78700 5 -- op tracker -- , seq: 1612, time: 2014-06-09 16:22:11.426479, event: started, op: pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426492 7f869af78700 5 -- op tracker -- , seq: 1612, time: 2014-06-09 16:22:11.426490, event: done, op: pg_log(2.646 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426499 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426501 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426493 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.426512 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426513 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426514 7f869af78700 20 osd.26 243 _dispatch 0x14dcb200 pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.426511 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426520 7f869af78700 5 -- op tracker -- , seq: 1613, time: 2014-06-09 16:22:11.087103, event: header_read, op: pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426526 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.426527 7f869af78700 5 -- op tracker -- , seq: 1613, time: 2014-06-09 16:22:11.087103, event: throttled, op: pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426533 7f869af78700 5 -- op tracker -- , seq: 1613, time: 2014-06-09 16:22:11.087113, event: all_read, op: pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426539 7f869af78700 5 -- op tracker -- , seq: 1613, time: 2014-06-09 16:22:11.426511, event: dispatched, op: pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426546 7f869af78700 5 -- op tracker -- , seq: 1613, time: 2014-06-09 16:22:11.426543, event: waiting_for_osdmap, op: pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426541 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.426550 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.426553 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dcb200 2014-06-09 16:22:11.426558 7f869af78700 5 -- op tracker -- , seq: 1613, time: 2014-06-09 16:22:11.426556, event: started, op: pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426560 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426570 7f869af78700 5 -- op tracker -- , seq: 1613, time: 2014-06-09 16:22:11.426567, event: done, op: pg_log(2.3d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426576 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426578 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426581 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.426588 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426590 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426591 7f869af78700 20 osd.26 243 _dispatch 0x2d735e00 pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.426597 7f869af78700 5 -- op tracker -- , seq: 1614, time: 2014-06-09 16:22:10.941102, event: header_read, op: pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.426593 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.426604 7f869af78700 5 -- op tracker -- , seq: 1614, time: 2014-06-09 16:22:10.941102, event: throttled, op: pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.426610 7f869af78700 5 -- op tracker -- , seq: 1614, time: 2014-06-09 16:22:10.941980, event: all_read, op: pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.426607 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426617 7f869af78700 5 -- op tracker -- , seq: 1614, time: 2014-06-09 16:22:11.426588, event: dispatched, op: pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.426620 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.426623 7f869af78700 5 -- op tracker -- , seq: 1614, time: 2014-06-09 16:22:11.426621, event: waiting_for_osdmap, op: pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.426628 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) v4 from osd.2 2014-06-09 16:22:11.426631 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d735e00 2014-06-09 16:22:11.426636 7f869af78700 5 -- op tracker -- , seq: 1614, time: 2014-06-09 16:22:11.426633, event: started, op: pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.426633 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.426648 7f869af78700 5 -- op tracker -- , seq: 1614, time: 2014-06-09 16:22:11.426645, event: done, op: pg_log(3.67d epoch 242 log log((0'0,213'678], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.426647 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426659 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.426678 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426679 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426672 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.426692 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426694 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426695 7f869af78700 20 osd.26 243 _dispatch 0x10c674a0 pg_query(3.21e epoch 242) v3 2014-06-09 16:22:11.426692 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426699 7f869af78700 5 -- op tracker -- , seq: 1615, time: 2014-06-09 16:22:10.861874, event: header_read, op: pg_query(3.21e epoch 242) 2014-06-09 16:22:11.426704 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.426706 7f869af78700 5 -- op tracker -- , seq: 1615, time: 2014-06-09 16:22:10.861875, event: throttled, op: pg_query(3.21e epoch 242) 2014-06-09 16:22:11.426711 7f869af78700 5 -- op tracker -- , seq: 1615, time: 2014-06-09 16:22:10.861883, event: all_read, op: pg_query(3.21e epoch 242) 2014-06-09 16:22:11.426716 7f869af78700 5 -- op tracker -- , seq: 1615, time: 2014-06-09 16:22:11.426692, event: dispatched, op: pg_query(3.21e epoch 242) 2014-06-09 16:22:11.426717 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.426722 7f869af78700 5 -- op tracker -- , seq: 1615, time: 2014-06-09 16:22:11.426720, event: waiting_for_osdmap, op: pg_query(3.21e epoch 242) 2014-06-09 16:22:11.426727 7f869af78700 7 osd.26 243 handle_pg_query from osd.4 epoch 242 2014-06-09 16:22:11.426729 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x10c674a0 2014-06-09 16:22:11.426733 7f869af78700 5 -- op tracker -- , seq: 1615, time: 2014-06-09 16:22:11.426731, event: started, op: pg_query(3.21e epoch 242) 2014-06-09 16:22:11.426730 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426741 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.426739 7f869af78700 10 osd.26 pg_epoch: 243 pg[3.21e( v 213'615 (0'0,213'615] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_query query(log 203'0 ec=118 les/c 203/203 242/242/202) from replica 4 2014-06-09 16:22:11.426750 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.426758 7f869af78700 5 -- op tracker -- , seq: 1615, time: 2014-06-09 16:22:11.426757, event: done, op: pg_query(3.21e epoch 242) 2014-06-09 16:22:11.426765 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426767 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426762 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426773 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.426779 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426782 7f869af78700 20 osd.26 243 _dispatch 0x306d3e00 pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.426782 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.426788 7f869af78700 5 -- op tracker -- , seq: 1616, time: 2014-06-09 16:22:11.143250, event: header_read, op: pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426795 7f869af78700 5 -- op tracker -- , seq: 1616, time: 2014-06-09 16:22:11.143251, event: throttled, op: pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426794 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426801 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.426801 7f869af78700 5 -- op tracker -- , seq: 1616, time: 2014-06-09 16:22:11.143261, event: all_read, op: pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426808 7f869af78700 5 -- op tracker -- , seq: 1616, time: 2014-06-09 16:22:11.426779, event: dispatched, op: pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426807 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.426815 7f869af78700 5 -- op tracker -- , seq: 1616, time: 2014-06-09 16:22:11.426812, event: waiting_for_osdmap, op: pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426815 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.426819 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.426822 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.426823 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d3e00 2014-06-09 16:22:11.426828 7f869af78700 5 -- op tracker -- , seq: 1616, time: 2014-06-09 16:22:11.426825, event: started, op: pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426828 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.426835 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6b7a0,0xbd5be80,0xbd652c0,0xbd751e0,0xbd6c1e0,0xbd63c60,0xc6e9200,0xc2d63a0,0xbe308c0,0xcc83620,0xbd7b2c0,0xbd5b960] 2014-06-09 16:22:11.426837 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.426838 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.426840 7f869af78700 5 -- op tracker -- , seq: 1616, time: 2014-06-09 16:22:11.426837, event: done, op: pg_log(1.589 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426847 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426848 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426860 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426861 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426862 7f869af78700 20 osd.26 243 _dispatch 0x3043bc00 pg_notify(3.6e(10),0.7d(20),2.82(20) epoch 243) v5 2014-06-09 16:22:11.426868 7f869af78700 5 -- op tracker -- , seq: 1617, time: 2014-06-09 16:22:11.081311, event: header_read, op: pg_notify(3.6e(10),0.7d(20),2.82(20) epoch 243) 2014-06-09 16:22:11.426875 7f869af78700 5 -- op tracker -- , seq: 1617, time: 2014-06-09 16:22:11.081312, event: throttled, op: pg_notify(3.6e(10),0.7d(20),2.82(20) epoch 243) 2014-06-09 16:22:11.426882 7f869af78700 5 -- op tracker -- , seq: 1617, time: 2014-06-09 16:22:11.081358, event: all_read, op: pg_notify(3.6e(10),0.7d(20),2.82(20) epoch 243) 2014-06-09 16:22:11.426888 7f869af78700 5 -- op tracker -- , seq: 1617, time: 2014-06-09 16:22:11.426859, event: dispatched, op: pg_notify(3.6e(10),0.7d(20),2.82(20) epoch 243) 2014-06-09 16:22:11.426894 7f869af78700 5 -- op tracker -- , seq: 1617, time: 2014-06-09 16:22:11.426892, event: waiting_for_osdmap, op: pg_notify(3.6e(10),0.7d(20),2.82(20) epoch 243) 2014-06-09 16:22:11.426898 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.426900 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043bc00 2014-06-09 16:22:11.426905 7f869af78700 5 -- op tracker -- , seq: 1617, time: 2014-06-09 16:22:11.426902, event: started, op: pg_notify(3.6e(10),0.7d(20),2.82(20) epoch 243) 2014-06-09 16:22:11.426933 7f869af78700 5 -- op tracker -- , seq: 1617, time: 2014-06-09 16:22:11.426931, event: done, op: pg_notify(3.6e(10),0.7d(20),2.82(20) epoch 243) 2014-06-09 16:22:11.426940 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426942 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426963 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.426964 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.426965 7f869af78700 20 osd.26 243 _dispatch 0x305a6c00 pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.426970 7f869af78700 5 -- op tracker -- , seq: 1618, time: 2014-06-09 16:22:11.085137, event: header_read, op: pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426977 7f869af78700 5 -- op tracker -- , seq: 1618, time: 2014-06-09 16:22:11.085138, event: throttled, op: pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426980 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.426984 7f869af78700 5 -- op tracker -- , seq: 1618, time: 2014-06-09 16:22:11.085148, event: all_read, op: pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426990 7f869af78700 5 -- op tracker -- , seq: 1618, time: 2014-06-09 16:22:11.426962, event: dispatched, op: pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.426996 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3117.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.426999 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.426997 7f869af78700 5 -- op tracker -- , seq: 1618, time: 2014-06-09 16:22:11.426994, event: waiting_for_osdmap, op: pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427001 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.427004 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a6c00 2014-06-09 16:22:11.427009 7f869af78700 5 -- op tracker -- , seq: 1618, time: 2014-06-09 16:22:11.427007, event: started, op: pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427012 7f86a5e11700 10 _do_op 0x308ca3c0 seq 3117 r = 0, finisher 0x306281c0 0 2014-06-09 16:22:11.427018 7f86a5e11700 10 _finish_op 0x308ca3c0 seq 3117 osr(0.8c 0x2bd6480)/0x2bd6480 2014-06-09 16:22:11.427021 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd76060] 2014-06-09 16:22:11.427023 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ed80 (0 active) 2014-06-09 16:22:11.427020 7f869af78700 5 -- op tracker -- , seq: 1618, time: 2014-06-09 16:22:11.427018, event: done, op: pg_log(2.69d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427025 7f86a5e11700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.427025 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd76060] 2014-06-09 16:22:11.427027 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.427027 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427028 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.427029 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427039 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427040 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427041 7f869af78700 20 osd.26 243 _dispatch 0x2fef4380 pg_notify(0.6ab(20) epoch 243) v5 2014-06-09 16:22:11.427046 7f869af78700 5 -- op tracker -- , seq: 1619, time: 2014-06-09 16:22:11.058648, event: header_read, op: pg_notify(0.6ab(20) epoch 243) 2014-06-09 16:22:11.427053 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306281c0] 2014-06-09 16:22:11.427052 7f869af78700 5 -- op tracker -- , seq: 1619, time: 2014-06-09 16:22:11.058648, event: throttled, op: pg_notify(0.6ab(20) epoch 243) 2014-06-09 16:22:11.427058 7f869af78700 5 -- op tracker -- , seq: 1619, time: 2014-06-09 16:22:11.058667, event: all_read, op: pg_notify(0.6ab(20) epoch 243) 2014-06-09 16:22:11.427056 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.427064 7f869af78700 5 -- op tracker -- , seq: 1619, time: 2014-06-09 16:22:11.427039, event: dispatched, op: pg_notify(0.6ab(20) epoch 243) 2014-06-09 16:22:11.427067 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.427070 7f869af78700 5 -- op tracker -- , seq: 1619, time: 2014-06-09 16:22:11.427068, event: waiting_for_osdmap, op: pg_notify(0.6ab(20) epoch 243) 2014-06-09 16:22:11.427074 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.427076 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4380 2014-06-09 16:22:11.427080 7f869af78700 5 -- op tracker -- , seq: 1619, time: 2014-06-09 16:22:11.427078, event: started, op: pg_notify(0.6ab(20) epoch 243) 2014-06-09 16:22:11.427083 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.427094 7f869af78700 5 -- op tracker -- , seq: 1619, time: 2014-06-09 16:22:11.427092, event: done, op: pg_notify(0.6ab(20) epoch 243) 2014-06-09 16:22:11.427099 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306281c0] 2014-06-09 16:22:11.427101 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.427101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427102 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.427103 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427108 7f868ef60700 10 op_submit_finish 3128 2014-06-09 16:22:11.427111 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f440 (1 active) 2014-06-09 16:22:11.427114 7f86a5610700 5 _do_op 0x309a2b40 seq 3128 osr(3.e0 0x3ed0500)/0x3ed0500 start 2014-06-09 16:22:11.427117 7f86a5610700 10 _do_transaction on 0xbfe63c0 2014-06-09 16:22:11.427118 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427120 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427112 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:2.61b( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.427121 7f869af78700 20 osd.26 243 _dispatch 0x2fe34a00 pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.427122 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.427127 7f869af78700 5 -- op tracker -- , seq: 1620, time: 2014-06-09 16:22:11.023314, event: header_read, op: pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427126 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.427136 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.427134 7f869af78700 5 -- op tracker -- , seq: 1620, time: 2014-06-09 16:22:11.023314, event: throttled, op: pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427141 7f869af78700 5 -- op tracker -- , seq: 1620, time: 2014-06-09 16:22:11.023328, event: all_read, op: pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427147 7f869af78700 5 -- op tracker -- , seq: 1620, time: 2014-06-09 16:22:11.427118, event: dispatched, op: pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427154 7f869af78700 5 -- op tracker -- , seq: 1620, time: 2014-06-09 16:22:11.427151, event: waiting_for_osdmap, op: pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427155 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.427156 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (13 active) 2014-06-09 16:22:11.427158 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.427161 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe34a00 2014-06-09 16:22:11.427166 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0280 (14 active) 2014-06-09 16:22:11.427166 7f869af78700 5 -- op tracker -- , seq: 1620, time: 2014-06-09 16:22:11.427164, event: started, op: pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427171 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3128.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.427175 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.427170 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.4e2( v 213'643 (0'0,213'643] local-les=191 n=121 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.427178 7f869af78700 5 -- op tracker -- , seq: 1620, time: 2014-06-09 16:22:11.427176, event: done, op: pg_log(1.7a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427185 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427187 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427184 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.427194 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.427197 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427199 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427199 7f86a5610700 10 _do_op 0x309a2b40 seq 3128 r = 0, finisher 0x306ce080 0 2014-06-09 16:22:11.427200 7f869af78700 20 osd.26 243 _dispatch 0x1ae7c000 pg_notify(0.319(20) epoch 243) v5 2014-06-09 16:22:11.427204 7f86a5610700 10 _finish_op 0x309a2b40 seq 3128 osr(3.e0 0x3ed0500)/0x3ed0500 2014-06-09 16:22:11.427198 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.427205 7f869af78700 5 -- op tracker -- , seq: 1621, time: 2014-06-09 16:22:11.027290, event: header_read, op: pg_notify(0.319(20) epoch 243) 2014-06-09 16:22:11.427208 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f440 (0 active) 2014-06-09 16:22:11.427211 7f86a5610700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.427211 7f869af78700 5 -- op tracker -- , seq: 1621, time: 2014-06-09 16:22:11.027291, event: throttled, op: pg_notify(0.319(20) epoch 243) 2014-06-09 16:22:11.427215 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd64900] 2014-06-09 16:22:11.427218 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd64900] 2014-06-09 16:22:11.427219 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.427209 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.427220 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.427218 7f869af78700 5 -- op tracker -- , seq: 1621, time: 2014-06-09 16:22:11.027322, event: all_read, op: pg_notify(0.319(20) epoch 243) 2014-06-09 16:22:11.427221 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.427223 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306ce080] 2014-06-09 16:22:11.427225 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.427224 7f869af78700 5 -- op tracker -- , seq: 1621, time: 2014-06-09 16:22:11.427197, event: dispatched, op: pg_notify(0.319(20) epoch 243) 2014-06-09 16:22:11.427230 7f869af78700 5 -- op tracker -- , seq: 1621, time: 2014-06-09 16:22:11.427228, event: waiting_for_osdmap, op: pg_notify(0.319(20) epoch 243) 2014-06-09 16:22:11.427225 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.427234 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.427229 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.154683 7 0.000192 2014-06-09 16:22:11.427236 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ae7c000 2014-06-09 16:22:11.427239 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.427240 7f869af78700 5 -- op tracker -- , seq: 1621, time: 2014-06-09 16:22:11.427238, event: started, op: pg_notify(0.319(20) epoch 243) 2014-06-09 16:22:11.427239 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.427249 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.427254 7f869af78700 5 -- op tracker -- , seq: 1621, time: 2014-06-09 16:22:11.427252, event: done, op: pg_notify(0.319(20) epoch 243) 2014-06-09 16:22:11.427252 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.427261 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427262 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427259 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.427266 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306ce080] 2014-06-09 16:22:11.427268 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.427269 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.427267 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.427278 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427280 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427276 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.427281 7f869af78700 20 osd.26 243 _dispatch 0x30047600 pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.427287 7f869af78700 5 -- op tracker -- , seq: 1622, time: 2014-06-09 16:22:11.050847, event: header_read, op: pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427294 7f869af78700 5 -- op tracker -- , seq: 1622, time: 2014-06-09 16:22:11.050847, event: throttled, op: pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427300 7f869af78700 5 -- op tracker -- , seq: 1622, time: 2014-06-09 16:22:11.050859, event: all_read, op: pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427306 7f869af78700 5 -- op tracker -- , seq: 1622, time: 2014-06-09 16:22:11.427278, event: dispatched, op: pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427313 7f869af78700 5 -- op tracker -- , seq: 1622, time: 2014-06-09 16:22:11.427310, event: waiting_for_osdmap, op: pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427317 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.427317 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.427321 7f868ef60700 5 queue_transactions existing osr(0.e8 0x2bd6510)/0x2bd6510 2014-06-09 16:22:11.427321 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30047600 2014-06-09 16:22:11.427323 7f868ef60700 10 op_submit_start 3129 2014-06-09 16:22:11.427323 7f868ef60700 5 queue_transactions (trailing journal) 3129 2014-06-09 16:22:11.427324 7f868ef60700 5 queue_op 0x309a2ae0 seq 3129 osr(0.e8 0x2bd6510) 1951 bytes (queue has 1 ops and 1951 bytes) 2014-06-09 16:22:11.427328 7f868ef60700 10 op_submit_finish 3129 2014-06-09 16:22:11.427326 7f869af78700 5 -- op tracker -- , seq: 1622, time: 2014-06-09 16:22:11.427323, event: started, op: pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427330 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1145a0 (1 active) 2014-06-09 16:22:11.427334 7f86a5e11700 5 _do_op 0x309a2ae0 seq 3129 osr(0.e8 0x2bd6510)/0x2bd6510 start 2014-06-09 16:22:11.427337 7f86a5e11700 10 _do_transaction on 0xc05d2c0 2014-06-09 16:22:11.427337 7f869af78700 5 -- op tracker -- , seq: 1622, time: 2014-06-09 16:22:11.427335, event: done, op: pg_log(0.35b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427342 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.427331 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.427347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427349 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427348 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.65( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.427361 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427359 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.427362 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427364 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.427363 7f869af78700 20 osd.26 243 _dispatch 0x305bac00 pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.427369 7f869af78700 5 -- op tracker -- , seq: 1623, time: 2014-06-09 16:22:11.090302, event: header_read, op: pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427367 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.65( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.215227 7 0.000180 2014-06-09 16:22:11.427374 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.427376 7f869af78700 5 -- op tracker -- , seq: 1623, time: 2014-06-09 16:22:11.090303, event: throttled, op: pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427378 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.65( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.427382 7f869af78700 5 -- op tracker -- , seq: 1623, time: 2014-06-09 16:22:11.090317, event: all_read, op: pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427389 7f869af78700 5 -- op tracker -- , seq: 1623, time: 2014-06-09 16:22:11.427360, event: dispatched, op: pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427389 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3129.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.427393 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.427388 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.65( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.427396 7f869af78700 5 -- op tracker -- , seq: 1623, time: 2014-06-09 16:22:11.427393, event: waiting_for_osdmap, op: pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427400 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.427397 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.427403 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305bac00 2014-06-09 16:22:11.427408 7f86a5e11700 10 _do_op 0x309a2ae0 seq 3129 r = 0, finisher 0x18742b00 0 2014-06-09 16:22:11.427409 7f869af78700 5 -- op tracker -- , seq: 1623, time: 2014-06-09 16:22:11.427406, event: started, op: pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427405 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.427414 7f86a5e11700 10 _finish_op 0x309a2ae0 seq 3129 osr(0.e8 0x2bd6510)/0x2bd6510 2014-06-09 16:22:11.427418 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1145a0 (0 active) 2014-06-09 16:22:11.427420 7f86a5e11700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.427413 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.427421 7f869af78700 5 -- op tracker -- , seq: 1623, time: 2014-06-09 16:22:11.427418, event: done, op: pg_log(2.620 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.427424 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189aa9c0] 2014-06-09 16:22:11.427426 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189aa9c0] 2014-06-09 16:22:11.427428 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.427427 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427429 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.427429 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427432 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x18742b00] 2014-06-09 16:22:11.427439 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427441 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427433 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.427442 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a8c0 pg_notify(1.1f0(26) epoch 242) v5 2014-06-09 16:22:11.427446 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.427447 7f869af78700 5 -- op tracker -- , seq: 1624, time: 2014-06-09 16:22:10.941996, event: header_read, op: pg_notify(1.1f0(26) epoch 242) 2014-06-09 16:22:11.427453 7f869af78700 5 -- op tracker -- , seq: 1624, time: 2014-06-09 16:22:10.941997, event: throttled, op: pg_notify(1.1f0(26) epoch 242) 2014-06-09 16:22:11.427459 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.427459 7f869af78700 5 -- op tracker -- , seq: 1624, time: 2014-06-09 16:22:10.942020, event: all_read, op: pg_notify(1.1f0(26) epoch 242) 2014-06-09 16:22:11.427456 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.427463 7f868ef60700 5 queue_transactions existing osr(2.65 0x2ed0570)/0x2ed0570 2014-06-09 16:22:11.427465 7f868ef60700 10 op_submit_start 3130 2014-06-09 16:22:11.427466 7f868ef60700 5 queue_transactions (trailing journal) 3130 2014-06-09 16:22:11.427465 7f869af78700 5 -- op tracker -- , seq: 1624, time: 2014-06-09 16:22:11.427439, event: dispatched, op: pg_notify(1.1f0(26) epoch 242) 2014-06-09 16:22:11.427468 7f868ef60700 5 queue_op 0x309a2a80 seq 3130 osr(2.65 0x2ed0570) 2006 bytes (queue has 1 ops and 2006 bytes) 2014-06-09 16:22:11.427470 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x18742b00] 2014-06-09 16:22:11.427472 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.427472 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.427473 7f868ef60700 10 op_submit_finish 3130 2014-06-09 16:22:11.427471 7f869af78700 5 -- op tracker -- , seq: 1624, time: 2014-06-09 16:22:11.427469, event: waiting_for_osdmap, op: pg_notify(1.1f0(26) epoch 242) 2014-06-09 16:22:11.427475 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.427476 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115200 (1 active) 2014-06-09 16:22:11.427477 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2ea5a8c0 2014-06-09 16:22:11.427478 7f86a5610700 5 _do_op 0x309a2a80 seq 3130 osr(2.65 0x2ed0570)/0x2ed0570 start 2014-06-09 16:22:11.427481 7f86a5610700 10 _do_transaction on 0xc26e9c0 2014-06-09 16:22:11.427481 7f869af78700 5 -- op tracker -- , seq: 1624, time: 2014-06-09 16:22:11.427480, event: started, op: pg_notify(1.1f0(26) epoch 242) 2014-06-09 16:22:11.427475 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.19( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.427485 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.427488 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.427497 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.427495 7f869af78700 5 -- op tracker -- , seq: 1624, time: 2014-06-09 16:22:11.427493, event: done, op: pg_notify(1.1f0(26) epoch 242) 2014-06-09 16:22:11.427502 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427504 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427502 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:1.36f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.427516 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.427513 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.427522 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427522 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.427523 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427524 7f869af78700 20 osd.26 243 _dispatch 0x2d7401c0 pg_notify(0.19a(23),2.2f7(23) epoch 242) v5 2014-06-09 16:22:11.427529 7f869af78700 5 -- op tracker -- , seq: 1625, time: 2014-06-09 16:22:10.863631, event: header_read, op: pg_notify(0.19a(23),2.2f7(23) epoch 242) 2014-06-09 16:22:11.427531 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3130.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.427526 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:0.370( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.427535 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.427536 7f869af78700 5 -- op tracker -- , seq: 1625, time: 2014-06-09 16:22:10.863631, event: throttled, op: pg_notify(0.19a(23),2.2f7(23) epoch 242) 2014-06-09 16:22:11.427537 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.427542 7f869af78700 5 -- op tracker -- , seq: 1625, time: 2014-06-09 16:22:10.863683, event: all_read, op: pg_notify(0.19a(23),2.2f7(23) epoch 242) 2014-06-09 16:22:11.427547 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.427549 7f869af78700 5 -- op tracker -- , seq: 1625, time: 2014-06-09 16:22:11.427521, event: dispatched, op: pg_notify(0.19a(23),2.2f7(23) epoch 242) 2014-06-09 16:22:11.427555 7f869af78700 5 -- op tracker -- , seq: 1625, time: 2014-06-09 16:22:11.427553, event: waiting_for_osdmap, op: pg_notify(0.19a(23),2.2f7(23) epoch 242) 2014-06-09 16:22:11.427550 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.427559 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.427559 7f86a5610700 10 _do_op 0x309a2a80 seq 3130 r = 0, finisher 0xbd67800 0 2014-06-09 16:22:11.427561 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d7401c0 2014-06-09 16:22:11.427565 7f86a5610700 10 _finish_op 0x309a2a80 seq 3130 osr(2.65 0x2ed0570)/0x2ed0570 2014-06-09 16:22:11.427566 7f869af78700 5 -- op tracker -- , seq: 1625, time: 2014-06-09 16:22:11.427563, event: started, op: pg_notify(0.19a(23),2.2f7(23) epoch 242) 2014-06-09 16:22:11.427569 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115200 (0 active) 2014-06-09 16:22:11.427572 7f86a5610700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.427577 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf16a0] 2014-06-09 16:22:11.427579 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf16a0] 2014-06-09 16:22:11.427560 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.3c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.427580 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.427581 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.427581 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.427584 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd67800] 2014-06-09 16:22:11.427585 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.427586 7f869af78700 5 -- op tracker -- , seq: 1625, time: 2014-06-09 16:22:11.427584, event: done, op: pg_notify(0.19a(23),2.2f7(23) epoch 242) 2014-06-09 16:22:11.427585 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.427594 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.427588 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.954473 7 0.000198 2014-06-09 16:22:11.427595 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.427598 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.427598 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.427611 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.427633 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0250 (15 active) 2014-06-09 16:22:11.427639 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.427653 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.427665 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.427669 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.427678 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.3d1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.427688 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.427692 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.427695 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.149729 7 0.000221 2014-06-09 16:22:11.427706 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.427716 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.427724 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.427732 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.427740 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.427783 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.427787 7f8690763700 5 queue_transactions existing osr(2.3d1 0x2ed13b0)/0x2ed13b0 2014-06-09 16:22:11.427789 7f8690763700 10 op_submit_start 3131 2014-06-09 16:22:11.427790 7f8690763700 5 queue_transactions (trailing journal) 3131 2014-06-09 16:22:11.427791 7f8690763700 5 queue_op 0x30607aa0 seq 3131 osr(2.3d1 0x2ed13b0) 1876 bytes (queue has 1 ops and 1876 bytes) 2014-06-09 16:22:11.427797 7f8690763700 10 op_submit_finish 3131 2014-06-09 16:22:11.427799 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.427810 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.530( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.427819 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.427822 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.427831 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.56( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.427839 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.427842 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MLogRec from 2 2014-06-09 16:22:11.427856 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.67d( v 213'678 (0'0,213'678] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) log((0'0,213'678], crt=0'0) missing(0) 2014-06-09 16:22:11.427868 7f8690763700 10 proc_replica_log for osd.2: 3.67d( v 213'678 (0'0,213'678] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) log((0'0,213'678], crt=0'0) missing(0) 2014-06-09 16:22:11.427873 7f8690763700 20 merge_log cut point (usually last shared) is 213'678 (0'0) modify b6bb6e7d/rbd_data.117f6b8b4567.000000000000150d/head//3 by client.4575.0:31981 2014-06-09 10:18:06.643699 2014-06-09 16:22:11.428322 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.67d( v 213'678 (0'0,213'678] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.428337 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.428349 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.493803 6 0.000278 2014-06-09 16:22:11.428361 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.428370 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.129823 0 0.000000 2014-06-09 16:22:11.428383 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.428392 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.428404 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.428413 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.428421 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'684 -> 214'684 2014-06-09 16:22:11.428469 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.67d( v 213'678 (0'0,213'678] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.428491 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'678,214'684], crt=0'0) 2014-06-09 16:22:11.428503 7f8690763700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.428518 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.67d( v 214'684 lc 213'678 (0'0,214'684] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.428532 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:11.428541 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing b6bb6e7d/rbd_data.117f6b8b4567.000000000000150d/head//3 214'684 is on osd.26 2014-06-09 16:22:11.428553 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {b6bb6e7d/rbd_data.117f6b8b4567.000000000000150d/head//3=214'684(213'678)} 2014-06-09 16:22:11.428562 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.428571 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing b6bb6e7d/rbd_data.117f6b8b4567.000000000000150d/head//3 214'684 also missing on osd.2 2014-06-09 16:22:11.428580 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {b6bb6e7d/rbd_data.117f6b8b4567.000000000000150d/head//3=214'684(213'678)} 2014-06-09 16:22:11.428589 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.428597 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.428607 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.428615 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.428625 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.428737 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.428749 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (16 active) 2014-06-09 16:22:11.428758 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.428755 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.319( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.428767 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496c60 (1 active) 2014-06-09 16:22:11.428771 7f86a5e11700 5 _do_op 0x30607aa0 seq 3131 osr(2.3d1 0x2ed13b0)/0x2ed13b0 start 2014-06-09 16:22:11.428774 7f86a5e11700 10 _do_transaction on 0xc233680 2014-06-09 16:22:11.428768 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.428779 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.428780 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.428785 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.428800 7f868e75f700 5 -- op tracker -- , seq: 311, time: 2014-06-09 16:22:11.403651, event: reached_pg, op: osd_op(client.4570.0:33857 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 172032~1032192] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.428813 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.428828 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.428828 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0d6700 2014-06-09 16:22:11.428837 7f868e75f700 10 osd.26 243 dequeue_op 0x1b0d6700 finish 2014-06-09 16:22:11.428844 7f868e75f700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (15 active) 2014-06-09 16:22:11.428847 7f868e75f700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.428846 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3131.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.428851 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.428854 7f8691f66700 10 osd.26 243 dequeue_op 0x1b0d6600 prio 63 cost 516096 latency 1.008584 osd_op(client.4570.0:33858 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1204224~516096] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.428872 7f86a5e11700 10 _do_op 0x30607aa0 seq 3131 r = 0, finisher 0xbd70240 0 2014-06-09 16:22:11.428878 7f86a5e11700 10 _finish_op 0x30607aa0 seq 3131 osr(2.3d1 0x2ed13b0)/0x2ed13b0 2014-06-09 16:22:11.428875 7f8691f66700 5 -- op tracker -- , seq: 314, time: 2014-06-09 16:22:11.428870, event: reached_pg, op: osd_op(client.4570.0:33858 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1204224~516096] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.428881 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496c60 (0 active) 2014-06-09 16:22:11.428883 7f86a5e11700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.428889 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd69cc0] 2014-06-09 16:22:11.428883 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.428894 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd69cc0] 2014-06-09 16:22:11.428895 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.428896 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.428894 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0d6600 2014-06-09 16:22:11.428902 7f8691f66700 10 osd.26 243 dequeue_op 0x1b0d6600 finish 2014-06-09 16:22:11.428907 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.428909 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd67800] 2014-06-09 16:22:11.428910 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.428910 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd70240] 2014-06-09 16:22:11.428912 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.428928 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.428916 7f868df5e700 10 osd.26 243 dequeue_op 0x1b0d6500 prio 63 cost 516096 latency 1.006205 osd_op(client.4570.0:33859 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~516096] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.428937 7f8690763700 5 queue_transactions existing osr(3.67d 0x3ed1d70)/0x3ed1d70 2014-06-09 16:22:11.428940 7f8690763700 10 op_submit_start 3132 2014-06-09 16:22:11.428941 7f8690763700 5 queue_transactions (trailing journal) 3132 2014-06-09 16:22:11.428937 7f868df5e700 5 -- op tracker -- , seq: 315, time: 2014-06-09 16:22:11.428931, event: reached_pg, op: osd_op(client.4570.0:33859 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1720320~516096] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.428942 7f8690763700 5 queue_op 0x306075c0 seq 3132 osr(3.67d 0x3ed1d70) 1053 bytes (queue has 1 ops and 1053 bytes) 2014-06-09 16:22:11.428947 7f8690763700 10 op_submit_finish 3132 2014-06-09 16:22:11.428944 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.428950 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.428955 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0d6500 2014-06-09 16:22:11.428963 7f868df5e700 10 osd.26 243 dequeue_op 0x1b0d6500 finish 2014-06-09 16:22:11.428969 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.428963 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.428971 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.428973 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.428980 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.428982 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.428983 7f869af78700 20 osd.26 243 _dispatch 0x306d3800 pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.428978 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.428990 7f869af78700 5 -- op tracker -- , seq: 1626, time: 2014-06-09 16:22:11.143312, event: header_read, op: pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.428988 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.428997 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.428998 7f869af78700 5 -- op tracker -- , seq: 1626, time: 2014-06-09 16:22:11.143313, event: throttled, op: pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.429004 7f869af78700 5 -- op tracker -- , seq: 1626, time: 2014-06-09 16:22:11.143321, event: all_read, op: pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.429000 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.21e( v 213'615 (0'0,213'615] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MQuery from 4 query_epoch 242 query: query(log 203'0 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.429011 7f869af78700 5 -- op tracker -- , seq: 1626, time: 2014-06-09 16:22:11.428980, event: dispatched, op: pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.429012 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.21e( v 213'615 (0'0,213'615] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] log request from 4 2014-06-09 16:22:11.429018 7f869af78700 5 -- op tracker -- , seq: 1626, time: 2014-06-09 16:22:11.429015, event: waiting_for_osdmap, op: pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.429022 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.429026 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d3800 2014-06-09 16:22:11.429021 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.21e( v 213'615 (0'0,213'615] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] sending info+missing+log since 203'0 2014-06-09 16:22:11.429031 7f869af78700 5 -- op tracker -- , seq: 1626, time: 2014-06-09 16:22:11.429029, event: started, op: pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.429047 7f869af78700 5 -- op tracker -- , seq: 1626, time: 2014-06-09 16:22:11.429044, event: done, op: pg_log(0.2c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.429056 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.429057 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.429070 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.429072 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.429073 7f869af78700 20 osd.26 243 _dispatch 0x2ea5bc00 pg_notify(3.d3(10),3.66f(10) epoch 243) v5 2014-06-09 16:22:11.429078 7f869af78700 5 -- op tracker -- , seq: 1627, time: 2014-06-09 16:22:11.082271, event: header_read, op: pg_notify(3.d3(10),3.66f(10) epoch 243) 2014-06-09 16:22:11.429085 7f869af78700 5 -- op tracker -- , seq: 1627, time: 2014-06-09 16:22:11.082271, event: throttled, op: pg_notify(3.d3(10),3.66f(10) epoch 243) 2014-06-09 16:22:11.429091 7f869af78700 5 -- op tracker -- , seq: 1627, time: 2014-06-09 16:22:11.082291, event: all_read, op: pg_notify(3.d3(10),3.66f(10) epoch 243) 2014-06-09 16:22:11.429097 7f869af78700 5 -- op tracker -- , seq: 1627, time: 2014-06-09 16:22:11.429070, event: dispatched, op: pg_notify(3.d3(10),3.66f(10) epoch 243) 2014-06-09 16:22:11.429104 7f869af78700 5 -- op tracker -- , seq: 1627, time: 2014-06-09 16:22:11.429101, event: waiting_for_osdmap, op: pg_notify(3.d3(10),3.66f(10) epoch 243) 2014-06-09 16:22:11.429107 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.429109 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5bc00 2014-06-09 16:22:11.429114 7f869af78700 5 -- op tracker -- , seq: 1627, time: 2014-06-09 16:22:11.429112, event: started, op: pg_notify(3.d3(10),3.66f(10) epoch 243) 2014-06-09 16:22:11.429109 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.21e( v 213'615 (0'0,213'615] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] sending log((0'0,213'615], crt=0'0) missing(0) 2014-06-09 16:22:11.429121 7f8690763700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.429136 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.429143 7f869af78700 5 -- op tracker -- , seq: 1627, time: 2014-06-09 16:22:11.429140, event: done, op: pg_notify(3.d3(10),3.66f(10) epoch 243) 2014-06-09 16:22:11.429140 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.429151 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.429152 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.429151 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.560( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.429160 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.429164 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.429173 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.429174 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.429175 7f869af78700 20 osd.26 243 _dispatch 0x3046a000 pg_notify(2.3de(21),3.3dc(12) epoch 243) v5 2014-06-09 16:22:11.429174 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.429182 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.429181 7f869af78700 5 -- op tracker -- , seq: 1628, time: 2014-06-09 16:22:11.085206, event: header_read, op: pg_notify(2.3de(21),3.3dc(12) epoch 243) 2014-06-09 16:22:11.429187 7f869af78700 5 -- op tracker -- , seq: 1628, time: 2014-06-09 16:22:11.085206, event: throttled, op: pg_notify(2.3de(21),3.3dc(12) epoch 243) 2014-06-09 16:22:11.429185 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.429193 7f869af78700 5 -- op tracker -- , seq: 1628, time: 2014-06-09 16:22:11.085250, event: all_read, op: pg_notify(2.3de(21),3.3dc(12) epoch 243) 2014-06-09 16:22:11.429195 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.429202 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.429200 7f869af78700 5 -- op tracker -- , seq: 1628, time: 2014-06-09 16:22:11.429172, event: dispatched, op: pg_notify(2.3de(21),3.3dc(12) epoch 243) 2014-06-09 16:22:11.429210 7f869af78700 5 -- op tracker -- , seq: 1628, time: 2014-06-09 16:22:11.429208, event: waiting_for_osdmap, op: pg_notify(2.3de(21),3.3dc(12) epoch 243) 2014-06-09 16:22:11.429205 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.429214 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.429215 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046a000 2014-06-09 16:22:11.429216 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.429220 7f869af78700 5 -- op tracker -- , seq: 1628, time: 2014-06-09 16:22:11.429218, event: started, op: pg_notify(2.3de(21),3.3dc(12) epoch 243) 2014-06-09 16:22:11.429231 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.439275 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.439299 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.439306 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.439318 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.35b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.439331 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.439335 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.439340 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.131417 7 0.000203 2014-06-09 16:22:11.439351 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.439364 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.439374 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.439383 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.439391 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.439440 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.439446 7f868cf5c700 5 queue_transactions existing osr(0.35b 0x2bd6fe0)/0x2bd6fe0 2014-06-09 16:22:11.439450 7f868cf5c700 10 op_submit_start 3133 2014-06-09 16:22:11.439452 7f868cf5c700 5 queue_transactions (trailing journal) 3133 2014-06-09 16:22:11.439453 7f868cf5c700 5 queue_op 0x30033680 seq 3133 osr(0.35b 0x2bd6fe0) 1883 bytes (queue has 2 ops and 2936 bytes) 2014-06-09 16:22:11.429248 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (14 active) 2014-06-09 16:22:11.439467 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.439479 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.2c3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.439491 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.439494 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.439498 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.949356 7 0.000228 2014-06-09 16:22:11.439508 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.439518 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.439527 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.439536 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.439544 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.439589 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.439594 7f868a757700 5 queue_transactions existing osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 2014-06-09 16:22:11.429246 7f869af78700 5 -- op tracker -- , seq: 1628, time: 2014-06-09 16:22:11.429244, event: done, op: pg_notify(2.3de(21),3.3dc(12) epoch 243) 2014-06-09 16:22:11.439642 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.429262 7f868b759700 10 osd.26 243 dequeue_op 0x1b0d6400 prio 63 cost 516096 latency 1.005608 osd_op(client.4570.0:33860 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2236416~516096] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.439657 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.439665 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.439671 7f868e75f700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.439677 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.439680 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.439715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.439719 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.439722 7f869af78700 20 osd.26 243 _dispatch 0x3044d6c0 pg_notify(2.342(20) epoch 243) v5 2014-06-09 16:22:11.439729 7f869af78700 5 -- op tracker -- , seq: 1629, time: 2014-06-09 16:22:11.058714, event: header_read, op: pg_notify(2.342(20) epoch 243) 2014-06-09 16:22:11.439738 7f869af78700 5 -- op tracker -- , seq: 1629, time: 2014-06-09 16:22:11.058714, event: throttled, op: pg_notify(2.342(20) epoch 243) 2014-06-09 16:22:11.439744 7f869af78700 5 -- op tracker -- , seq: 1629, time: 2014-06-09 16:22:11.058735, event: all_read, op: pg_notify(2.342(20) epoch 243) 2014-06-09 16:22:11.439750 7f869af78700 5 -- op tracker -- , seq: 1629, time: 2014-06-09 16:22:11.439714, event: dispatched, op: pg_notify(2.342(20) epoch 243) 2014-06-09 16:22:11.439756 7f869af78700 5 -- op tracker -- , seq: 1629, time: 2014-06-09 16:22:11.439754, event: waiting_for_osdmap, op: pg_notify(2.342(20) epoch 243) 2014-06-09 16:22:11.439760 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.439763 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044d6c0 2014-06-09 16:22:11.439769 7f869af78700 5 -- op tracker -- , seq: 1629, time: 2014-06-09 16:22:11.439767, event: started, op: pg_notify(2.342(20) epoch 243) 2014-06-09 16:22:11.439791 7f869af78700 5 -- op tracker -- , seq: 1629, time: 2014-06-09 16:22:11.439788, event: done, op: pg_notify(2.342(20) epoch 243) 2014-06-09 16:22:11.439799 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.439801 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.439819 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.439821 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.439822 7f869af78700 20 osd.26 243 _dispatch 0x2fe21600 pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.439829 7f869af78700 5 -- op tracker -- , seq: 1630, time: 2014-06-09 16:22:11.035578, event: header_read, op: pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.439837 7f869af78700 5 -- op tracker -- , seq: 1630, time: 2014-06-09 16:22:11.035579, event: throttled, op: pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.439844 7f869af78700 5 -- op tracker -- , seq: 1630, time: 2014-06-09 16:22:11.035594, event: all_read, op: pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.439851 7f869af78700 5 -- op tracker -- , seq: 1630, time: 2014-06-09 16:22:11.439819, event: dispatched, op: pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.439859 7f869af78700 5 -- op tracker -- , seq: 1630, time: 2014-06-09 16:22:11.439856, event: waiting_for_osdmap, op: pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.439864 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.439868 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe21600 2014-06-09 16:22:11.439874 7f869af78700 5 -- op tracker -- , seq: 1630, time: 2014-06-09 16:22:11.439871, event: started, op: pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.439889 7f869af78700 5 -- op tracker -- , seq: 1630, time: 2014-06-09 16:22:11.439886, event: done, op: pg_log(0.33f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.439897 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.439899 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.439910 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.439912 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.439913 7f869af78700 20 osd.26 243 _dispatch 0xbe9cc40 pg_info(1 pgs e243:3.31f(2)) v4 2014-06-09 16:22:11.439918 7f869af78700 5 -- op tracker -- , seq: 1631, time: 2014-06-09 16:22:11.027329, event: header_read, op: pg_info(1 pgs e243:3.31f(2)) 2014-06-09 16:22:11.439925 7f869af78700 5 -- op tracker -- , seq: 1631, time: 2014-06-09 16:22:11.027330, event: throttled, op: pg_info(1 pgs e243:3.31f(2)) 2014-06-09 16:22:11.439931 7f869af78700 5 -- op tracker -- , seq: 1631, time: 2014-06-09 16:22:11.027343, event: all_read, op: pg_info(1 pgs e243:3.31f(2)) 2014-06-09 16:22:11.439937 7f869af78700 5 -- op tracker -- , seq: 1631, time: 2014-06-09 16:22:11.439910, event: dispatched, op: pg_info(1 pgs e243:3.31f(2)) 2014-06-09 16:22:11.439943 7f869af78700 5 -- op tracker -- , seq: 1631, time: 2014-06-09 16:22:11.439942, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.31f(2)) 2014-06-09 16:22:11.439948 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.31f(2)) v4 from osd.6 2014-06-09 16:22:11.439951 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xbe9cc40 2014-06-09 16:22:11.439956 7f869af78700 5 -- op tracker -- , seq: 1631, time: 2014-06-09 16:22:11.439954, event: started, op: pg_info(1 pgs e243:3.31f(2)) 2014-06-09 16:22:11.439975 7f869af78700 5 -- op tracker -- , seq: 1631, time: 2014-06-09 16:22:11.439972, event: done, op: pg_info(1 pgs e243:3.31f(2)) 2014-06-09 16:22:11.439983 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.439985 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.439998 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.440000 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.440001 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a380 pg_notify(0.6e(21) epoch 243) v5 2014-06-09 16:22:11.440006 7f869af78700 5 -- op tracker -- , seq: 1632, time: 2014-06-09 16:22:11.052227, event: header_read, op: pg_notify(0.6e(21) epoch 243) 2014-06-09 16:22:11.440013 7f869af78700 5 -- op tracker -- , seq: 1632, time: 2014-06-09 16:22:11.052228, event: throttled, op: pg_notify(0.6e(21) epoch 243) 2014-06-09 16:22:11.440019 7f869af78700 5 -- op tracker -- , seq: 1632, time: 2014-06-09 16:22:11.052249, event: all_read, op: pg_notify(0.6e(21) epoch 243) 2014-06-09 16:22:11.440026 7f869af78700 5 -- op tracker -- , seq: 1632, time: 2014-06-09 16:22:11.439998, event: dispatched, op: pg_notify(0.6e(21) epoch 243) 2014-06-09 16:22:11.440032 7f869af78700 5 -- op tracker -- , seq: 1632, time: 2014-06-09 16:22:11.440030, event: waiting_for_osdmap, op: pg_notify(0.6e(21) epoch 243) 2014-06-09 16:22:11.440036 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.440038 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5a380 2014-06-09 16:22:11.440043 7f869af78700 5 -- op tracker -- , seq: 1632, time: 2014-06-09 16:22:11.440041, event: started, op: pg_notify(0.6e(21) epoch 243) 2014-06-09 16:22:11.440062 7f869af78700 5 -- op tracker -- , seq: 1632, time: 2014-06-09 16:22:11.440060, event: done, op: pg_notify(0.6e(21) epoch 243) 2014-06-09 16:22:11.440070 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.440072 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.440089 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.440090 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.440091 7f869af78700 20 osd.26 243 _dispatch 0x304dcc00 pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.440097 7f869af78700 5 -- op tracker -- , seq: 1633, time: 2014-06-09 16:22:11.090338, event: header_read, op: pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.440105 7f869af78700 5 -- op tracker -- , seq: 1633, time: 2014-06-09 16:22:11.090339, event: throttled, op: pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.440111 7f869af78700 5 -- op tracker -- , seq: 1633, time: 2014-06-09 16:22:11.090352, event: all_read, op: pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.440118 7f869af78700 5 -- op tracker -- , seq: 1633, time: 2014-06-09 16:22:11.440089, event: dispatched, op: pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.440125 7f869af78700 5 -- op tracker -- , seq: 1633, time: 2014-06-09 16:22:11.440122, event: waiting_for_osdmap, op: pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.440129 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.440133 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304dcc00 2014-06-09 16:22:11.440138 7f869af78700 5 -- op tracker -- , seq: 1633, time: 2014-06-09 16:22:11.440136, event: started, op: pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.440151 7f869af78700 5 -- op tracker -- , seq: 1633, time: 2014-06-09 16:22:11.440149, event: done, op: pg_log(0.5b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.440376 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (15 active) 2014-06-09 16:22:11.440384 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:2.342( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.440400 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.440411 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.440416 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.440426 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.33f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.440437 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.440441 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.440444 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.124131 7 0.000239 2014-06-09 16:22:11.440455 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.440466 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.440475 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.440484 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.440492 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.440548 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.440553 7f8691f66700 5 queue_transactions existing osr(0.33f 0x2bd7060)/0x2bd7060 2014-06-09 16:22:11.440597 7f868e75f700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.440611 7f868b759700 5 -- op tracker -- , seq: 367, time: 2014-06-09 16:22:11.439654, event: reached_pg, op: osd_op(client.4570.0:33860 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2236416~516096] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.440622 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.440637 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0d6400 2014-06-09 16:22:11.440647 7f868b759700 10 osd.26 243 dequeue_op 0x1b0d6400 finish 2014-06-09 16:22:11.440654 7f868b759700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (14 active) 2014-06-09 16:22:11.440657 7f868b759700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.440663 7f868df5e700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.440667 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.440679 7f8689f56700 5 -- op tracker -- , seq: 475, time: 2014-06-09 16:22:11.425156, event: reached_pg, op: osd_op(client.4574.0:31826 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1335296~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.440688 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.440706 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1c54be00 2014-06-09 16:22:11.440717 7f8689f56700 10 osd.26 243 dequeue_op 0x1c54be00 finish 2014-06-09 16:22:11.440724 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (13 active) 2014-06-09 16:22:11.440727 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.440734 7f868bf5a700 10 osd.26 243 dequeue_op 0x13023700 prio 63 cost 557056 latency 1.152071 osd_op(client.4568.0:34999 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 3637248~557056] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.440760 7f868bf5a700 5 -- op tracker -- , seq: 209, time: 2014-06-09 16:22:11.440754, event: reached_pg, op: osd_op(client.4568.0:34999 rbd_data.11786b8b4567.00000000000016ec [set-alloc-hint object_size 4194304 write_size 4194304,write 3637248~557056] 3.d869c3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.440768 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.440782 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x13023700 2014-06-09 16:22:11.440793 7f868bf5a700 10 osd.26 243 dequeue_op 0x13023700 finish 2014-06-09 16:22:11.440799 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (12 active) 2014-06-09 16:22:11.440802 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.440808 7f868af58700 10 osd.26 243 dequeue_op 0xe2ac600 prio 63 cost 516096 latency 0.986363 osd_op(client.4569.0:38603 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1998848~516096] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.440832 7f868af58700 5 -- op tracker -- , seq: 377, time: 2014-06-09 16:22:11.440826, event: reached_pg, op: osd_op(client.4569.0:38603 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1998848~516096] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.440839 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.440853 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0xe2ac600 2014-06-09 16:22:11.440864 7f868af58700 10 osd.26 243 dequeue_op 0xe2ac600 finish 2014-06-09 16:22:11.440869 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (11 active) 2014-06-09 16:22:11.440872 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.440879 7f8689755700 10 osd.26 243 dequeue_op 0x1c54bd00 prio 63 cost 516096 latency 0.913139 osd_op(client.4574.0:31827 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1851392~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.440903 7f8689755700 5 -- op tracker -- , seq: 476, time: 2014-06-09 16:22:11.440897, event: reached_pg, op: osd_op(client.4574.0:31827 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1851392~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.440910 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.440924 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1c54bd00 2014-06-09 16:22:11.440935 7f8689755700 10 osd.26 243 dequeue_op 0x1c54bd00 finish 2014-06-09 16:22:11.440941 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (10 active) 2014-06-09 16:22:11.440944 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.440951 7f868c75b700 10 osd.26 243 dequeue_op 0x1c54bc00 prio 63 cost 1679360 latency 0.985737 osd_op(client.4569.0:38604 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2514944~1679360] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.440974 7f868c75b700 5 -- op tracker -- , seq: 404, time: 2014-06-09 16:22:11.440969, event: reached_pg, op: osd_op(client.4569.0:38604 rbd_data.11796b8b4567.0000000000001aa5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2514944~1679360] 3.bc70b3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.440981 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.440993 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1c54bc00 2014-06-09 16:22:11.441005 7f868c75b700 10 osd.26 243 dequeue_op 0x1c54bc00 finish 2014-06-09 16:22:11.441011 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (9 active) 2014-06-09 16:22:11.441014 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.441023 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441025 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441038 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441039 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441040 7f869af78700 20 osd.26 243 _dispatch 0x1405d500 pg_notify(0.1f1(25) epoch 242) v5 2014-06-09 16:22:11.441045 7f869af78700 5 -- op tracker -- , seq: 1634, time: 2014-06-09 16:22:10.942025, event: header_read, op: pg_notify(0.1f1(25) epoch 242) 2014-06-09 16:22:11.441053 7f869af78700 5 -- op tracker -- , seq: 1634, time: 2014-06-09 16:22:10.942025, event: throttled, op: pg_notify(0.1f1(25) epoch 242) 2014-06-09 16:22:11.441059 7f869af78700 5 -- op tracker -- , seq: 1634, time: 2014-06-09 16:22:10.942047, event: all_read, op: pg_notify(0.1f1(25) epoch 242) 2014-06-09 16:22:11.441065 7f869af78700 5 -- op tracker -- , seq: 1634, time: 2014-06-09 16:22:11.441037, event: dispatched, op: pg_notify(0.1f1(25) epoch 242) 2014-06-09 16:22:11.441071 7f869af78700 5 -- op tracker -- , seq: 1634, time: 2014-06-09 16:22:11.441069, event: waiting_for_osdmap, op: pg_notify(0.1f1(25) epoch 242) 2014-06-09 16:22:11.441076 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.441078 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x1405d500 2014-06-09 16:22:11.441083 7f869af78700 5 -- op tracker -- , seq: 1634, time: 2014-06-09 16:22:11.441081, event: started, op: pg_notify(0.1f1(25) epoch 242) 2014-06-09 16:22:11.441102 7f869af78700 5 -- op tracker -- , seq: 1634, time: 2014-06-09 16:22:11.441100, event: done, op: pg_notify(0.1f1(25) epoch 242) 2014-06-09 16:22:11.441111 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441113 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441133 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441134 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441135 7f869af78700 20 osd.26 243 _dispatch 0x230bb500 pg_notify(3.424(10) epoch 242) v5 2014-06-09 16:22:11.441140 7f869af78700 5 -- op tracker -- , seq: 1635, time: 2014-06-09 16:22:10.886287, event: header_read, op: pg_notify(3.424(10) epoch 242) 2014-06-09 16:22:11.441147 7f869af78700 5 -- op tracker -- , seq: 1635, time: 2014-06-09 16:22:10.886288, event: throttled, op: pg_notify(3.424(10) epoch 242) 2014-06-09 16:22:11.441153 7f869af78700 5 -- op tracker -- , seq: 1635, time: 2014-06-09 16:22:10.886314, event: all_read, op: pg_notify(3.424(10) epoch 242) 2014-06-09 16:22:11.441159 7f869af78700 5 -- op tracker -- , seq: 1635, time: 2014-06-09 16:22:11.441132, event: dispatched, op: pg_notify(3.424(10) epoch 242) 2014-06-09 16:22:11.441165 7f869af78700 5 -- op tracker -- , seq: 1635, time: 2014-06-09 16:22:11.441163, event: waiting_for_osdmap, op: pg_notify(3.424(10) epoch 242) 2014-06-09 16:22:11.441169 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.441171 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x230bb500 2014-06-09 16:22:11.441176 7f869af78700 5 -- op tracker -- , seq: 1635, time: 2014-06-09 16:22:11.441174, event: started, op: pg_notify(3.424(10) epoch 242) 2014-06-09 16:22:11.441195 7f869af78700 5 -- op tracker -- , seq: 1635, time: 2014-06-09 16:22:11.441193, event: done, op: pg_notify(3.424(10) epoch 242) 2014-06-09 16:22:11.441204 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441206 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441221 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441223 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441224 7f869af78700 20 osd.26 243 _dispatch 0x19eb9dc0 pg_notify(3.2c0(10) epoch 243) v5 2014-06-09 16:22:11.441229 7f869af78700 5 -- op tracker -- , seq: 1636, time: 2014-06-09 16:22:11.143375, event: header_read, op: pg_notify(3.2c0(10) epoch 243) 2014-06-09 16:22:11.441236 7f869af78700 5 -- op tracker -- , seq: 1636, time: 2014-06-09 16:22:11.143376, event: throttled, op: pg_notify(3.2c0(10) epoch 243) 2014-06-09 16:22:11.441242 7f869af78700 5 -- op tracker -- , seq: 1636, time: 2014-06-09 16:22:11.143390, event: all_read, op: pg_notify(3.2c0(10) epoch 243) 2014-06-09 16:22:11.441248 7f869af78700 5 -- op tracker -- , seq: 1636, time: 2014-06-09 16:22:11.441221, event: dispatched, op: pg_notify(3.2c0(10) epoch 243) 2014-06-09 16:22:11.441254 7f869af78700 5 -- op tracker -- , seq: 1636, time: 2014-06-09 16:22:11.441252, event: waiting_for_osdmap, op: pg_notify(3.2c0(10) epoch 243) 2014-06-09 16:22:11.441259 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.441261 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb9dc0 2014-06-09 16:22:11.441265 7f869af78700 5 -- op tracker -- , seq: 1636, time: 2014-06-09 16:22:11.441263, event: started, op: pg_notify(3.2c0(10) epoch 243) 2014-06-09 16:22:11.441284 7f869af78700 5 -- op tracker -- , seq: 1636, time: 2014-06-09 16:22:11.441282, event: done, op: pg_notify(3.2c0(10) epoch 243) 2014-06-09 16:22:11.441292 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441294 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441309 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441310 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441311 7f869af78700 20 osd.26 243 _dispatch 0x3044d500 pg_info(1 pgs e243:3.3a9(2)) v4 2014-06-09 16:22:11.441316 7f869af78700 5 -- op tracker -- , seq: 1637, time: 2014-06-09 16:22:11.106354, event: header_read, op: pg_info(1 pgs e243:3.3a9(2)) 2014-06-09 16:22:11.441323 7f869af78700 5 -- op tracker -- , seq: 1637, time: 2014-06-09 16:22:11.106355, event: throttled, op: pg_info(1 pgs e243:3.3a9(2)) 2014-06-09 16:22:11.441329 7f869af78700 5 -- op tracker -- , seq: 1637, time: 2014-06-09 16:22:11.106371, event: all_read, op: pg_info(1 pgs e243:3.3a9(2)) 2014-06-09 16:22:11.441335 7f869af78700 5 -- op tracker -- , seq: 1637, time: 2014-06-09 16:22:11.441308, event: dispatched, op: pg_info(1 pgs e243:3.3a9(2)) 2014-06-09 16:22:11.441342 7f869af78700 5 -- op tracker -- , seq: 1637, time: 2014-06-09 16:22:11.441340, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a9(2)) 2014-06-09 16:22:11.441346 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a9(2)) v4 from osd.3 2014-06-09 16:22:11.441350 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044d500 2014-06-09 16:22:11.441354 7f869af78700 5 -- op tracker -- , seq: 1637, time: 2014-06-09 16:22:11.441352, event: started, op: pg_info(1 pgs e243:3.3a9(2)) 2014-06-09 16:22:11.441373 7f869af78700 5 -- op tracker -- , seq: 1637, time: 2014-06-09 16:22:11.441370, event: done, op: pg_info(1 pgs e243:3.3a9(2)) 2014-06-09 16:22:11.441381 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441383 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441397 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441398 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441399 7f869af78700 20 osd.26 243 _dispatch 0x304de400 pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.441406 7f869af78700 5 -- op tracker -- , seq: 1638, time: 2014-06-09 16:22:11.103022, event: header_read, op: pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441414 7f869af78700 5 -- op tracker -- , seq: 1638, time: 2014-06-09 16:22:11.103023, event: throttled, op: pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441420 7f869af78700 5 -- op tracker -- , seq: 1638, time: 2014-06-09 16:22:11.103035, event: all_read, op: pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441427 7f869af78700 5 -- op tracker -- , seq: 1638, time: 2014-06-09 16:22:11.441396, event: dispatched, op: pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441434 7f869af78700 5 -- op tracker -- , seq: 1638, time: 2014-06-09 16:22:11.441431, event: waiting_for_osdmap, op: pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441438 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.441443 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304de400 2014-06-09 16:22:11.441448 7f869af78700 5 -- op tracker -- , seq: 1638, time: 2014-06-09 16:22:11.441445, event: started, op: pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441464 7f869af78700 5 -- op tracker -- , seq: 1638, time: 2014-06-09 16:22:11.441461, event: done, op: pg_log(0.dd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441472 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441474 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441486 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441487 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441489 7f869af78700 20 osd.26 243 _dispatch 0x3043b880 pg_info(1 pgs e243:3.609(2)) v4 2014-06-09 16:22:11.441494 7f869af78700 5 -- op tracker -- , seq: 1639, time: 2014-06-09 16:22:11.058778, event: header_read, op: pg_info(1 pgs e243:3.609(2)) 2014-06-09 16:22:11.441501 7f869af78700 5 -- op tracker -- , seq: 1639, time: 2014-06-09 16:22:11.058778, event: throttled, op: pg_info(1 pgs e243:3.609(2)) 2014-06-09 16:22:11.441507 7f869af78700 5 -- op tracker -- , seq: 1639, time: 2014-06-09 16:22:11.058788, event: all_read, op: pg_info(1 pgs e243:3.609(2)) 2014-06-09 16:22:11.441513 7f869af78700 5 -- op tracker -- , seq: 1639, time: 2014-06-09 16:22:11.441485, event: dispatched, op: pg_info(1 pgs e243:3.609(2)) 2014-06-09 16:22:11.441511 7f8691765700 10 osd.26 243 dequeue_op 0x1b0d6300 prio 63 cost 1441792 latency 0.996840 osd_op(client.4570.0:33861 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~1441792] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.441534 7f869af78700 5 -- op tracker -- , seq: 1639, time: 2014-06-09 16:22:11.441531, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.609(2)) 2014-06-09 16:22:11.441542 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.609(2)) v4 from osd.9 2014-06-09 16:22:11.441545 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043b880 2014-06-09 16:22:11.441550 7f869af78700 5 -- op tracker -- , seq: 1639, time: 2014-06-09 16:22:11.441548, event: started, op: pg_info(1 pgs e243:3.609(2)) 2014-06-09 16:22:11.441568 7f869af78700 5 -- op tracker -- , seq: 1639, time: 2014-06-09 16:22:11.441566, event: done, op: pg_info(1 pgs e243:3.609(2)) 2014-06-09 16:22:11.441576 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441579 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441598 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441600 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441601 7f869af78700 20 osd.26 243 _dispatch 0x3044cc40 pg_info(1 pgs e243:3.76(2)) v4 2014-06-09 16:22:11.441606 7f869af78700 5 -- op tracker -- , seq: 1640, time: 2014-06-09 16:22:11.035603, event: header_read, op: pg_info(1 pgs e243:3.76(2)) 2014-06-09 16:22:11.441613 7f869af78700 5 -- op tracker -- , seq: 1640, time: 2014-06-09 16:22:11.035604, event: throttled, op: pg_info(1 pgs e243:3.76(2)) 2014-06-09 16:22:11.441619 7f869af78700 5 -- op tracker -- , seq: 1640, time: 2014-06-09 16:22:11.035625, event: all_read, op: pg_info(1 pgs e243:3.76(2)) 2014-06-09 16:22:11.441625 7f869af78700 5 -- op tracker -- , seq: 1640, time: 2014-06-09 16:22:11.441598, event: dispatched, op: pg_info(1 pgs e243:3.76(2)) 2014-06-09 16:22:11.441631 7f869af78700 5 -- op tracker -- , seq: 1640, time: 2014-06-09 16:22:11.441629, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.76(2)) 2014-06-09 16:22:11.441636 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.76(2)) v4 from osd.8 2014-06-09 16:22:11.441639 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044cc40 2014-06-09 16:22:11.441644 7f869af78700 5 -- op tracker -- , seq: 1640, time: 2014-06-09 16:22:11.441642, event: started, op: pg_info(1 pgs e243:3.76(2)) 2014-06-09 16:22:11.441661 7f869af78700 5 -- op tracker -- , seq: 1640, time: 2014-06-09 16:22:11.441659, event: done, op: pg_info(1 pgs e243:3.76(2)) 2014-06-09 16:22:11.441670 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441672 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441685 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441688 7f869af78700 20 osd.26 243 _dispatch 0x3045d600 pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.441701 7f869af78700 5 -- op tracker -- , seq: 1641, time: 2014-06-09 16:22:11.032431, event: header_read, op: pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441710 7f869af78700 5 -- op tracker -- , seq: 1641, time: 2014-06-09 16:22:11.032432, event: throttled, op: pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441716 7f869af78700 5 -- op tracker -- , seq: 1641, time: 2014-06-09 16:22:11.032445, event: all_read, op: pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441723 7f869af78700 5 -- op tracker -- , seq: 1641, time: 2014-06-09 16:22:11.441685, event: dispatched, op: pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441730 7f869af78700 5 -- op tracker -- , seq: 1641, time: 2014-06-09 16:22:11.441728, event: waiting_for_osdmap, op: pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441735 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.441739 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045d600 2014-06-09 16:22:11.441744 7f869af78700 5 -- op tracker -- , seq: 1641, time: 2014-06-09 16:22:11.441741, event: started, op: pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441759 7f869af78700 5 -- op tracker -- , seq: 1641, time: 2014-06-09 16:22:11.441756, event: done, op: pg_log(2.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441766 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441768 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441780 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441782 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441783 7f869af78700 20 osd.26 243 _dispatch 0x2fe71c00 pg_info(1 pgs e243:3.603(4)) v4 2014-06-09 16:22:11.441788 7f869af78700 5 -- op tracker -- , seq: 1642, time: 2014-06-09 16:22:11.052287, event: header_read, op: pg_info(1 pgs e243:3.603(4)) 2014-06-09 16:22:11.441795 7f869af78700 5 -- op tracker -- , seq: 1642, time: 2014-06-09 16:22:11.052287, event: throttled, op: pg_info(1 pgs e243:3.603(4)) 2014-06-09 16:22:11.441802 7f869af78700 5 -- op tracker -- , seq: 1642, time: 2014-06-09 16:22:11.052299, event: all_read, op: pg_info(1 pgs e243:3.603(4)) 2014-06-09 16:22:11.441808 7f869af78700 5 -- op tracker -- , seq: 1642, time: 2014-06-09 16:22:11.441780, event: dispatched, op: pg_info(1 pgs e243:3.603(4)) 2014-06-09 16:22:11.441815 7f869af78700 5 -- op tracker -- , seq: 1642, time: 2014-06-09 16:22:11.441813, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.603(4)) 2014-06-09 16:22:11.441819 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.603(4)) v4 from osd.0 2014-06-09 16:22:11.441822 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe71c00 2014-06-09 16:22:11.441827 7f869af78700 5 -- op tracker -- , seq: 1642, time: 2014-06-09 16:22:11.441825, event: started, op: pg_info(1 pgs e243:3.603(4)) 2014-06-09 16:22:11.441845 7f869af78700 5 -- op tracker -- , seq: 1642, time: 2014-06-09 16:22:11.441843, event: done, op: pg_info(1 pgs e243:3.603(4)) 2014-06-09 16:22:11.441853 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441855 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441873 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441875 7f869af78700 20 osd.26 243 _dispatch 0x2fe1f800 pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.441881 7f869af78700 5 -- op tracker -- , seq: 1643, time: 2014-06-09 16:22:11.115362, event: header_read, op: pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441888 7f869af78700 5 -- op tracker -- , seq: 1643, time: 2014-06-09 16:22:11.115363, event: throttled, op: pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441894 7f869af78700 5 -- op tracker -- , seq: 1643, time: 2014-06-09 16:22:11.115380, event: all_read, op: pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441901 7f869af78700 5 -- op tracker -- , seq: 1643, time: 2014-06-09 16:22:11.441872, event: dispatched, op: pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441908 7f869af78700 5 -- op tracker -- , seq: 1643, time: 2014-06-09 16:22:11.441906, event: waiting_for_osdmap, op: pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441913 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.441917 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe1f800 2014-06-09 16:22:11.441922 7f869af78700 5 -- op tracker -- , seq: 1643, time: 2014-06-09 16:22:11.441920, event: started, op: pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441935 7f869af78700 5 -- op tracker -- , seq: 1643, time: 2014-06-09 16:22:11.441932, event: done, op: pg_log(1.2d0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.441943 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441944 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441956 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.441958 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.441959 7f869af78700 20 osd.26 243 _dispatch 0x2d735800 pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.441965 7f869af78700 5 -- op tracker -- , seq: 1644, time: 2014-06-09 16:22:10.960585, event: header_read, op: pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.441973 7f869af78700 5 -- op tracker -- , seq: 1644, time: 2014-06-09 16:22:10.960586, event: throttled, op: pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.441980 7f869af78700 5 -- op tracker -- , seq: 1644, time: 2014-06-09 16:22:10.961249, event: all_read, op: pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.441987 7f869af78700 5 -- op tracker -- , seq: 1644, time: 2014-06-09 16:22:11.441956, event: dispatched, op: pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.441995 7f869af78700 5 -- op tracker -- , seq: 1644, time: 2014-06-09 16:22:11.441992, event: waiting_for_osdmap, op: pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442000 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) v4 from osd.2 2014-06-09 16:22:11.442004 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d735800 2014-06-09 16:22:11.442009 7f869af78700 5 -- op tracker -- , seq: 1644, time: 2014-06-09 16:22:11.442007, event: started, op: pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442022 7f869af78700 5 -- op tracker -- , seq: 1644, time: 2014-06-09 16:22:11.442019, event: done, op: pg_log(3.74 epoch 242 log log((0'0,213'545], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442030 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442032 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442043 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442046 7f869af78700 20 osd.26 243 _dispatch 0x10d0cfc0 pg_notify(2.1ae(20) epoch 242) v5 2014-06-09 16:22:11.442051 7f869af78700 5 -- op tracker -- , seq: 1645, time: 2014-06-09 16:22:10.898670, event: header_read, op: pg_notify(2.1ae(20) epoch 242) 2014-06-09 16:22:11.442057 7f869af78700 5 -- op tracker -- , seq: 1645, time: 2014-06-09 16:22:10.898671, event: throttled, op: pg_notify(2.1ae(20) epoch 242) 2014-06-09 16:22:11.442063 7f869af78700 5 -- op tracker -- , seq: 1645, time: 2014-06-09 16:22:10.898690, event: all_read, op: pg_notify(2.1ae(20) epoch 242) 2014-06-09 16:22:11.442069 7f869af78700 5 -- op tracker -- , seq: 1645, time: 2014-06-09 16:22:11.442043, event: dispatched, op: pg_notify(2.1ae(20) epoch 242) 2014-06-09 16:22:11.442075 7f869af78700 5 -- op tracker -- , seq: 1645, time: 2014-06-09 16:22:11.442074, event: waiting_for_osdmap, op: pg_notify(2.1ae(20) epoch 242) 2014-06-09 16:22:11.442080 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.442082 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x10d0cfc0 2014-06-09 16:22:11.442087 7f869af78700 5 -- op tracker -- , seq: 1645, time: 2014-06-09 16:22:11.442085, event: started, op: pg_notify(2.1ae(20) epoch 242) 2014-06-09 16:22:11.442103 7f869af78700 5 -- op tracker -- , seq: 1645, time: 2014-06-09 16:22:11.442100, event: done, op: pg_notify(2.1ae(20) epoch 242) 2014-06-09 16:22:11.442111 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442112 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442130 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442132 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442133 7f869af78700 20 osd.26 243 _dispatch 0x16065500 pg_notify(3.21b(10) epoch 243) v5 2014-06-09 16:22:11.442138 7f869af78700 5 -- op tracker -- , seq: 1646, time: 2014-06-09 16:22:11.200571, event: header_read, op: pg_notify(3.21b(10) epoch 243) 2014-06-09 16:22:11.442145 7f869af78700 5 -- op tracker -- , seq: 1646, time: 2014-06-09 16:22:11.200571, event: throttled, op: pg_notify(3.21b(10) epoch 243) 2014-06-09 16:22:11.442150 7f869af78700 5 -- op tracker -- , seq: 1646, time: 2014-06-09 16:22:11.200596, event: all_read, op: pg_notify(3.21b(10) epoch 243) 2014-06-09 16:22:11.442156 7f869af78700 5 -- op tracker -- , seq: 1646, time: 2014-06-09 16:22:11.442129, event: dispatched, op: pg_notify(3.21b(10) epoch 243) 2014-06-09 16:22:11.442163 7f869af78700 5 -- op tracker -- , seq: 1646, time: 2014-06-09 16:22:11.442161, event: waiting_for_osdmap, op: pg_notify(3.21b(10) epoch 243) 2014-06-09 16:22:11.442167 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.442169 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16065500 2014-06-09 16:22:11.442173 7f869af78700 5 -- op tracker -- , seq: 1646, time: 2014-06-09 16:22:11.442172, event: started, op: pg_notify(3.21b(10) epoch 243) 2014-06-09 16:22:11.442189 7f869af78700 5 -- op tracker -- , seq: 1646, time: 2014-06-09 16:22:11.442187, event: done, op: pg_notify(3.21b(10) epoch 243) 2014-06-09 16:22:11.442196 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442198 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442213 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442215 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442216 7f869af78700 20 osd.26 243 _dispatch 0x14dcac00 pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.442223 7f869af78700 5 -- op tracker -- , seq: 1647, time: 2014-06-09 16:22:11.107082, event: header_read, op: pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442230 7f869af78700 5 -- op tracker -- , seq: 1647, time: 2014-06-09 16:22:11.107083, event: throttled, op: pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442237 7f869af78700 5 -- op tracker -- , seq: 1647, time: 2014-06-09 16:22:11.107098, event: all_read, op: pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442244 7f869af78700 5 -- op tracker -- , seq: 1647, time: 2014-06-09 16:22:11.442213, event: dispatched, op: pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442252 7f869af78700 5 -- op tracker -- , seq: 1647, time: 2014-06-09 16:22:11.442249, event: waiting_for_osdmap, op: pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442257 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.442260 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dcac00 2014-06-09 16:22:11.442266 7f869af78700 5 -- op tracker -- , seq: 1647, time: 2014-06-09 16:22:11.442264, event: started, op: pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442279 7f869af78700 5 -- op tracker -- , seq: 1647, time: 2014-06-09 16:22:11.442276, event: done, op: pg_log(0.3b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442286 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442288 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442298 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442300 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442301 7f869af78700 20 osd.26 243 _dispatch 0x2d0948c0 pg_notify(2.3a6(27) epoch 243) v5 2014-06-09 16:22:11.442305 7f869af78700 5 -- op tracker -- , seq: 1648, time: 2014-06-09 16:22:11.103071, event: header_read, op: pg_notify(2.3a6(27) epoch 243) 2014-06-09 16:22:11.442312 7f869af78700 5 -- op tracker -- , seq: 1648, time: 2014-06-09 16:22:11.103071, event: throttled, op: pg_notify(2.3a6(27) epoch 243) 2014-06-09 16:22:11.442318 7f869af78700 5 -- op tracker -- , seq: 1648, time: 2014-06-09 16:22:11.103104, event: all_read, op: pg_notify(2.3a6(27) epoch 243) 2014-06-09 16:22:11.442324 7f869af78700 5 -- op tracker -- , seq: 1648, time: 2014-06-09 16:22:11.442298, event: dispatched, op: pg_notify(2.3a6(27) epoch 243) 2014-06-09 16:22:11.442331 7f869af78700 5 -- op tracker -- , seq: 1648, time: 2014-06-09 16:22:11.442329, event: waiting_for_osdmap, op: pg_notify(2.3a6(27) epoch 243) 2014-06-09 16:22:11.442335 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.442337 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d0948c0 2014-06-09 16:22:11.442342 7f869af78700 5 -- op tracker -- , seq: 1648, time: 2014-06-09 16:22:11.442340, event: started, op: pg_notify(2.3a6(27) epoch 243) 2014-06-09 16:22:11.442358 7f869af78700 5 -- op tracker -- , seq: 1648, time: 2014-06-09 16:22:11.442356, event: done, op: pg_notify(2.3a6(27) epoch 243) 2014-06-09 16:22:11.442365 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442367 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442390 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442391 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442393 7f869af78700 20 osd.26 243 _dispatch 0x304f7e00 pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.442399 7f869af78700 5 -- op tracker -- , seq: 1649, time: 2014-06-09 16:22:11.084633, event: header_read, op: pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442407 7f869af78700 5 -- op tracker -- , seq: 1649, time: 2014-06-09 16:22:11.084634, event: throttled, op: pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442414 7f869af78700 5 -- op tracker -- , seq: 1649, time: 2014-06-09 16:22:11.084643, event: all_read, op: pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442421 7f869af78700 5 -- op tracker -- , seq: 1649, time: 2014-06-09 16:22:11.442389, event: dispatched, op: pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442428 7f869af78700 5 -- op tracker -- , seq: 1649, time: 2014-06-09 16:22:11.442425, event: waiting_for_osdmap, op: pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442433 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.442437 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f7e00 2014-06-09 16:22:11.442449 7f869af78700 5 -- op tracker -- , seq: 1649, time: 2014-06-09 16:22:11.442440, event: started, op: pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442462 7f869af78700 5 -- op tracker -- , seq: 1649, time: 2014-06-09 16:22:11.442460, event: done, op: pg_log(2.d3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442470 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442472 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442484 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442486 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442487 7f869af78700 20 osd.26 243 _dispatch 0x2fe21000 pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.442493 7f869af78700 5 -- op tracker -- , seq: 1650, time: 2014-06-09 16:22:11.035630, event: header_read, op: pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442501 7f869af78700 5 -- op tracker -- , seq: 1650, time: 2014-06-09 16:22:11.035631, event: throttled, op: pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442508 7f869af78700 5 -- op tracker -- , seq: 1650, time: 2014-06-09 16:22:11.035643, event: all_read, op: pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442515 7f869af78700 5 -- op tracker -- , seq: 1650, time: 2014-06-09 16:22:11.442484, event: dispatched, op: pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442522 7f869af78700 5 -- op tracker -- , seq: 1650, time: 2014-06-09 16:22:11.442519, event: waiting_for_osdmap, op: pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442527 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.442531 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe21000 2014-06-09 16:22:11.442536 7f869af78700 5 -- op tracker -- , seq: 1650, time: 2014-06-09 16:22:11.442534, event: started, op: pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442549 7f869af78700 5 -- op tracker -- , seq: 1650, time: 2014-06-09 16:22:11.442546, event: done, op: pg_log(2.350 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442556 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442558 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442569 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442570 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442571 7f869af78700 20 osd.26 243 _dispatch 0x2d740380 pg_notify(0.75(20) epoch 243) v5 2014-06-09 16:22:11.442576 7f869af78700 5 -- op tracker -- , seq: 1651, time: 2014-06-09 16:22:11.032453, event: header_read, op: pg_notify(0.75(20) epoch 243) 2014-06-09 16:22:11.442583 7f869af78700 5 -- op tracker -- , seq: 1651, time: 2014-06-09 16:22:11.032453, event: throttled, op: pg_notify(0.75(20) epoch 243) 2014-06-09 16:22:11.442589 7f869af78700 5 -- op tracker -- , seq: 1651, time: 2014-06-09 16:22:11.032477, event: all_read, op: pg_notify(0.75(20) epoch 243) 2014-06-09 16:22:11.442595 7f869af78700 5 -- op tracker -- , seq: 1651, time: 2014-06-09 16:22:11.442568, event: dispatched, op: pg_notify(0.75(20) epoch 243) 2014-06-09 16:22:11.442602 7f869af78700 5 -- op tracker -- , seq: 1651, time: 2014-06-09 16:22:11.442600, event: waiting_for_osdmap, op: pg_notify(0.75(20) epoch 243) 2014-06-09 16:22:11.442606 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.442608 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d740380 2014-06-09 16:22:11.442613 7f869af78700 5 -- op tracker -- , seq: 1651, time: 2014-06-09 16:22:11.442611, event: started, op: pg_notify(0.75(20) epoch 243) 2014-06-09 16:22:11.442629 7f869af78700 5 -- op tracker -- , seq: 1651, time: 2014-06-09 16:22:11.442627, event: done, op: pg_notify(0.75(20) epoch 243) 2014-06-09 16:22:11.442636 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442656 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442658 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442659 7f869af78700 20 osd.26 243 _dispatch 0x30047000 pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.442666 7f869af78700 5 -- op tracker -- , seq: 1652, time: 2014-06-09 16:22:11.053936, event: header_read, op: pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442673 7f869af78700 5 -- op tracker -- , seq: 1652, time: 2014-06-09 16:22:11.053937, event: throttled, op: pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442680 7f869af78700 5 -- op tracker -- , seq: 1652, time: 2014-06-09 16:22:11.053952, event: all_read, op: pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442687 7f869af78700 5 -- op tracker -- , seq: 1652, time: 2014-06-09 16:22:11.442656, event: dispatched, op: pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442694 7f869af78700 5 -- op tracker -- , seq: 1652, time: 2014-06-09 16:22:11.442691, event: waiting_for_osdmap, op: pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442698 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.442702 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30047000 2014-06-09 16:22:11.442707 7f869af78700 5 -- op tracker -- , seq: 1652, time: 2014-06-09 16:22:11.442705, event: started, op: pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442720 7f869af78700 5 -- op tracker -- , seq: 1652, time: 2014-06-09 16:22:11.442717, event: done, op: pg_log(0.3a9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442728 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442731 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442744 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442745 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442746 7f869af78700 20 osd.26 243 _dispatch 0x240f5000 pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.442753 7f869af78700 5 -- op tracker -- , seq: 1653, time: 2014-06-09 16:22:11.115894, event: header_read, op: pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442761 7f869af78700 5 -- op tracker -- , seq: 1653, time: 2014-06-09 16:22:11.115895, event: throttled, op: pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442767 7f869af78700 5 -- op tracker -- , seq: 1653, time: 2014-06-09 16:22:11.115908, event: all_read, op: pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442774 7f869af78700 5 -- op tracker -- , seq: 1653, time: 2014-06-09 16:22:11.442743, event: dispatched, op: pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442782 7f869af78700 5 -- op tracker -- , seq: 1653, time: 2014-06-09 16:22:11.442779, event: waiting_for_osdmap, op: pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442786 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.442796 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240f5000 2014-06-09 16:22:11.442802 7f869af78700 5 -- op tracker -- , seq: 1653, time: 2014-06-09 16:22:11.442799, event: started, op: pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442815 7f869af78700 5 -- op tracker -- , seq: 1653, time: 2014-06-09 16:22:11.442812, event: done, op: pg_log(2.5c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.442822 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442824 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442836 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442838 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442839 7f869af78700 20 osd.26 243 _dispatch 0x2d737000 pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.442846 7f869af78700 5 -- op tracker -- , seq: 1654, time: 2014-06-09 16:22:10.974967, event: header_read, op: pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442853 7f869af78700 5 -- op tracker -- , seq: 1654, time: 2014-06-09 16:22:10.974968, event: throttled, op: pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442860 7f869af78700 5 -- op tracker -- , seq: 1654, time: 2014-06-09 16:22:10.975742, event: all_read, op: pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442866 7f869af78700 5 -- op tracker -- , seq: 1654, time: 2014-06-09 16:22:11.442836, event: dispatched, op: pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442873 7f869af78700 5 -- op tracker -- , seq: 1654, time: 2014-06-09 16:22:11.442871, event: waiting_for_osdmap, op: pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442878 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) v4 from osd.2 2014-06-09 16:22:11.442882 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d737000 2014-06-09 16:22:11.442887 7f869af78700 5 -- op tracker -- , seq: 1654, time: 2014-06-09 16:22:11.442885, event: started, op: pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442900 7f869af78700 5 -- op tracker -- , seq: 1654, time: 2014-06-09 16:22:11.442897, event: done, op: pg_log(3.562 epoch 242 log log((0'0,213'605], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442923 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442925 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442938 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.442940 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.442941 7f869af78700 20 osd.26 243 _dispatch 0x48fd000 pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.442948 7f869af78700 5 -- op tracker -- , seq: 1655, time: 2014-06-09 16:22:10.911280, event: header_read, op: pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442956 7f869af78700 5 -- op tracker -- , seq: 1655, time: 2014-06-09 16:22:10.911280, event: throttled, op: pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442963 7f869af78700 5 -- op tracker -- , seq: 1655, time: 2014-06-09 16:22:10.911295, event: all_read, op: pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442970 7f869af78700 5 -- op tracker -- , seq: 1655, time: 2014-06-09 16:22:11.442938, event: dispatched, op: pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442978 7f869af78700 5 -- op tracker -- , seq: 1655, time: 2014-06-09 16:22:11.442975, event: waiting_for_osdmap, op: pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.442982 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) v4 from osd.4 2014-06-09 16:22:11.442986 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x48fd000 2014-06-09 16:22:11.442991 7f869af78700 5 -- op tracker -- , seq: 1655, time: 2014-06-09 16:22:11.442989, event: started, op: pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.443004 7f869af78700 5 -- op tracker -- , seq: 1655, time: 2014-06-09 16:22:11.443002, event: done, op: pg_log(3.65d epoch 242 log log((214'533,214'536], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.443012 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443014 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443027 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443028 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443029 7f869af78700 20 osd.26 243 _dispatch 0x306d3200 pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.443036 7f869af78700 5 -- op tracker -- , seq: 1656, time: 2014-06-09 16:22:11.200677, event: header_read, op: pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443044 7f869af78700 5 -- op tracker -- , seq: 1656, time: 2014-06-09 16:22:11.200677, event: throttled, op: pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443051 7f869af78700 5 -- op tracker -- , seq: 1656, time: 2014-06-09 16:22:11.200691, event: all_read, op: pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443057 7f869af78700 5 -- op tracker -- , seq: 1656, time: 2014-06-09 16:22:11.443026, event: dispatched, op: pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443065 7f869af78700 5 -- op tracker -- , seq: 1656, time: 2014-06-09 16:22:11.443062, event: waiting_for_osdmap, op: pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443070 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.443074 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d3200 2014-06-09 16:22:11.443079 7f869af78700 5 -- op tracker -- , seq: 1656, time: 2014-06-09 16:22:11.443076, event: started, op: pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443093 7f869af78700 5 -- op tracker -- , seq: 1656, time: 2014-06-09 16:22:11.443090, event: done, op: pg_log(0.16c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443100 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443102 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443115 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443116 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443117 7f869af78700 20 osd.26 243 _dispatch 0x305b3800 pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.443124 7f869af78700 5 -- op tracker -- , seq: 1657, time: 2014-06-09 16:22:11.109775, event: header_read, op: pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443131 7f869af78700 5 -- op tracker -- , seq: 1657, time: 2014-06-09 16:22:11.109776, event: throttled, op: pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443138 7f869af78700 5 -- op tracker -- , seq: 1657, time: 2014-06-09 16:22:11.109787, event: all_read, op: pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443144 7f869af78700 5 -- op tracker -- , seq: 1657, time: 2014-06-09 16:22:11.443114, event: dispatched, op: pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443151 7f869af78700 5 -- op tracker -- , seq: 1657, time: 2014-06-09 16:22:11.443149, event: waiting_for_osdmap, op: pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443156 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.443160 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b3800 2014-06-09 16:22:11.443165 7f869af78700 5 -- op tracker -- , seq: 1657, time: 2014-06-09 16:22:11.443163, event: started, op: pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443178 7f869af78700 5 -- op tracker -- , seq: 1657, time: 2014-06-09 16:22:11.443175, event: done, op: pg_log(1.697 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443186 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443187 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443199 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443201 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443202 7f869af78700 20 osd.26 243 _dispatch 0x3046a540 pg_info(2 pgs e243:3.39e(4),3.66a(4)) v4 2014-06-09 16:22:11.443208 7f869af78700 5 -- op tracker -- , seq: 1658, time: 2014-06-09 16:22:11.103124, event: header_read, op: pg_info(2 pgs e243:3.39e(4),3.66a(4)) 2014-06-09 16:22:11.443215 7f869af78700 5 -- op tracker -- , seq: 1658, time: 2014-06-09 16:22:11.103125, event: throttled, op: pg_info(2 pgs e243:3.39e(4),3.66a(4)) 2014-06-09 16:22:11.443222 7f869af78700 5 -- op tracker -- , seq: 1658, time: 2014-06-09 16:22:11.103144, event: all_read, op: pg_info(2 pgs e243:3.39e(4),3.66a(4)) 2014-06-09 16:22:11.443229 7f869af78700 5 -- op tracker -- , seq: 1658, time: 2014-06-09 16:22:11.443199, event: dispatched, op: pg_info(2 pgs e243:3.39e(4),3.66a(4)) 2014-06-09 16:22:11.443236 7f869af78700 5 -- op tracker -- , seq: 1658, time: 2014-06-09 16:22:11.443233, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.39e(4),3.66a(4)) 2014-06-09 16:22:11.443240 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.39e(4),3.66a(4)) v4 from osd.1 2014-06-09 16:22:11.443244 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046a540 2014-06-09 16:22:11.443249 7f869af78700 5 -- op tracker -- , seq: 1658, time: 2014-06-09 16:22:11.443246, event: started, op: pg_info(2 pgs e243:3.39e(4),3.66a(4)) 2014-06-09 16:22:11.443274 7f869af78700 5 -- op tracker -- , seq: 1658, time: 2014-06-09 16:22:11.443272, event: done, op: pg_info(2 pgs e243:3.39e(4),3.66a(4)) 2014-06-09 16:22:11.443283 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443284 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443301 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443302 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443303 7f869af78700 20 osd.26 243 _dispatch 0x305e2c00 pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.443310 7f869af78700 5 -- op tracker -- , seq: 1659, time: 2014-06-09 16:22:11.086930, event: header_read, op: pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443317 7f869af78700 5 -- op tracker -- , seq: 1659, time: 2014-06-09 16:22:11.086930, event: throttled, op: pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443324 7f869af78700 5 -- op tracker -- , seq: 1659, time: 2014-06-09 16:22:11.086941, event: all_read, op: pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443330 7f869af78700 5 -- op tracker -- , seq: 1659, time: 2014-06-09 16:22:11.443301, event: dispatched, op: pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443338 7f869af78700 5 -- op tracker -- , seq: 1659, time: 2014-06-09 16:22:11.443335, event: waiting_for_osdmap, op: pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443343 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.443347 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e2c00 2014-06-09 16:22:11.443352 7f869af78700 5 -- op tracker -- , seq: 1659, time: 2014-06-09 16:22:11.443349, event: started, op: pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443365 7f869af78700 5 -- op tracker -- , seq: 1659, time: 2014-06-09 16:22:11.443362, event: done, op: pg_log(2.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443373 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443375 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443387 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443388 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443389 7f869af78700 20 osd.26 243 _dispatch 0x2fe20a00 pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.443396 7f869af78700 5 -- op tracker -- , seq: 1660, time: 2014-06-09 16:22:11.039502, event: header_read, op: pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443404 7f869af78700 5 -- op tracker -- , seq: 1660, time: 2014-06-09 16:22:11.039502, event: throttled, op: pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443411 7f869af78700 5 -- op tracker -- , seq: 1660, time: 2014-06-09 16:22:11.039517, event: all_read, op: pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443418 7f869af78700 5 -- op tracker -- , seq: 1660, time: 2014-06-09 16:22:11.443386, event: dispatched, op: pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443425 7f869af78700 5 -- op tracker -- , seq: 1660, time: 2014-06-09 16:22:11.443423, event: waiting_for_osdmap, op: pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443430 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.443434 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe20a00 2014-06-09 16:22:11.443439 7f869af78700 5 -- op tracker -- , seq: 1660, time: 2014-06-09 16:22:11.443437, event: started, op: pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443452 7f869af78700 5 -- op tracker -- , seq: 1660, time: 2014-06-09 16:22:11.443449, event: done, op: pg_log(1.348 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443459 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443461 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443472 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443473 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443475 7f869af78700 20 osd.26 243 _dispatch 0x10d0ce00 pg_info(1 pgs e243:3.6f(2)) v4 2014-06-09 16:22:11.443480 7f869af78700 5 -- op tracker -- , seq: 1661, time: 2014-06-09 16:22:11.032482, event: header_read, op: pg_info(1 pgs e243:3.6f(2)) 2014-06-09 16:22:11.443487 7f869af78700 5 -- op tracker -- , seq: 1661, time: 2014-06-09 16:22:11.032482, event: throttled, op: pg_info(1 pgs e243:3.6f(2)) 2014-06-09 16:22:11.443493 7f869af78700 5 -- op tracker -- , seq: 1661, time: 2014-06-09 16:22:11.032495, event: all_read, op: pg_info(1 pgs e243:3.6f(2)) 2014-06-09 16:22:11.443499 7f869af78700 5 -- op tracker -- , seq: 1661, time: 2014-06-09 16:22:11.443472, event: dispatched, op: pg_info(1 pgs e243:3.6f(2)) 2014-06-09 16:22:11.443506 7f869af78700 5 -- op tracker -- , seq: 1661, time: 2014-06-09 16:22:11.443504, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6f(2)) 2014-06-09 16:22:11.443511 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6f(2)) v4 from osd.6 2014-06-09 16:22:11.443514 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d0ce00 2014-06-09 16:22:11.443519 7f869af78700 5 -- op tracker -- , seq: 1661, time: 2014-06-09 16:22:11.443517, event: started, op: pg_info(1 pgs e243:3.6f(2)) 2014-06-09 16:22:11.443535 7f869af78700 5 -- op tracker -- , seq: 1661, time: 2014-06-09 16:22:11.443532, event: done, op: pg_info(1 pgs e243:3.6f(2)) 2014-06-09 16:22:11.443543 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443545 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443557 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443559 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443560 7f869af78700 20 osd.26 243 _dispatch 0x30046a00 pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.443567 7f869af78700 5 -- op tracker -- , seq: 1662, time: 2014-06-09 16:22:11.053997, event: header_read, op: pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443574 7f869af78700 5 -- op tracker -- , seq: 1662, time: 2014-06-09 16:22:11.053997, event: throttled, op: pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443581 7f869af78700 5 -- op tracker -- , seq: 1662, time: 2014-06-09 16:22:11.054008, event: all_read, op: pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443588 7f869af78700 5 -- op tracker -- , seq: 1662, time: 2014-06-09 16:22:11.443557, event: dispatched, op: pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443596 7f869af78700 5 -- op tracker -- , seq: 1662, time: 2014-06-09 16:22:11.443593, event: waiting_for_osdmap, op: pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443600 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.443604 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30046a00 2014-06-09 16:22:11.443610 7f869af78700 5 -- op tracker -- , seq: 1662, time: 2014-06-09 16:22:11.443607, event: started, op: pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443629 7f869af78700 5 -- op tracker -- , seq: 1662, time: 2014-06-09 16:22:11.443626, event: done, op: pg_log(0.380 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443636 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443651 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443652 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443653 7f869af78700 20 osd.26 243 _dispatch 0x3045b800 pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.443660 7f869af78700 5 -- op tracker -- , seq: 1663, time: 2014-06-09 16:22:11.129265, event: header_read, op: pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443667 7f869af78700 5 -- op tracker -- , seq: 1663, time: 2014-06-09 16:22:11.129265, event: throttled, op: pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443674 7f869af78700 5 -- op tracker -- , seq: 1663, time: 2014-06-09 16:22:11.129291, event: all_read, op: pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443680 7f869af78700 5 -- op tracker -- , seq: 1663, time: 2014-06-09 16:22:11.443650, event: dispatched, op: pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443688 7f869af78700 5 -- op tracker -- , seq: 1663, time: 2014-06-09 16:22:11.443685, event: waiting_for_osdmap, op: pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443694 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.443698 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045b800 2014-06-09 16:22:11.443703 7f869af78700 5 -- op tracker -- , seq: 1663, time: 2014-06-09 16:22:11.443701, event: started, op: pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443716 7f869af78700 5 -- op tracker -- , seq: 1663, time: 2014-06-09 16:22:11.443713, event: done, op: pg_log(0.151 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.443723 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443725 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443736 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443738 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443739 7f869af78700 20 osd.26 243 _dispatch 0x240ef880 pg_notify(2.b(21) epoch 242) v5 2014-06-09 16:22:11.443744 7f869af78700 5 -- op tracker -- , seq: 1664, time: 2014-06-09 16:22:10.983838, event: header_read, op: pg_notify(2.b(21) epoch 242) 2014-06-09 16:22:11.443752 7f869af78700 5 -- op tracker -- , seq: 1664, time: 2014-06-09 16:22:10.983839, event: throttled, op: pg_notify(2.b(21) epoch 242) 2014-06-09 16:22:11.443758 7f869af78700 5 -- op tracker -- , seq: 1664, time: 2014-06-09 16:22:10.983868, event: all_read, op: pg_notify(2.b(21) epoch 242) 2014-06-09 16:22:11.443764 7f869af78700 5 -- op tracker -- , seq: 1664, time: 2014-06-09 16:22:11.443736, event: dispatched, op: pg_notify(2.b(21) epoch 242) 2014-06-09 16:22:11.443771 7f869af78700 5 -- op tracker -- , seq: 1664, time: 2014-06-09 16:22:11.443769, event: waiting_for_osdmap, op: pg_notify(2.b(21) epoch 242) 2014-06-09 16:22:11.443775 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.443777 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240ef880 2014-06-09 16:22:11.443782 7f869af78700 5 -- op tracker -- , seq: 1664, time: 2014-06-09 16:22:11.443780, event: started, op: pg_notify(2.b(21) epoch 242) 2014-06-09 16:22:11.443798 7f869af78700 5 -- op tracker -- , seq: 1664, time: 2014-06-09 16:22:11.443796, event: done, op: pg_notify(2.b(21) epoch 242) 2014-06-09 16:22:11.443806 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443808 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443825 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443826 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443827 7f869af78700 20 osd.26 243 _dispatch 0x240e9340 pg_notify(2.1b3(20) epoch 242) v5 2014-06-09 16:22:11.443832 7f869af78700 5 -- op tracker -- , seq: 1665, time: 2014-06-09 16:22:10.912044, event: header_read, op: pg_notify(2.1b3(20) epoch 242) 2014-06-09 16:22:11.443839 7f869af78700 5 -- op tracker -- , seq: 1665, time: 2014-06-09 16:22:10.912044, event: throttled, op: pg_notify(2.1b3(20) epoch 242) 2014-06-09 16:22:11.443845 7f869af78700 5 -- op tracker -- , seq: 1665, time: 2014-06-09 16:22:10.912064, event: all_read, op: pg_notify(2.1b3(20) epoch 242) 2014-06-09 16:22:11.443850 7f869af78700 5 -- op tracker -- , seq: 1665, time: 2014-06-09 16:22:11.443824, event: dispatched, op: pg_notify(2.1b3(20) epoch 242) 2014-06-09 16:22:11.443857 7f869af78700 5 -- op tracker -- , seq: 1665, time: 2014-06-09 16:22:11.443855, event: waiting_for_osdmap, op: pg_notify(2.1b3(20) epoch 242) 2014-06-09 16:22:11.443861 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.443863 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x240e9340 2014-06-09 16:22:11.443868 7f869af78700 5 -- op tracker -- , seq: 1665, time: 2014-06-09 16:22:11.443866, event: started, op: pg_notify(2.1b3(20) epoch 242) 2014-06-09 16:22:11.443883 7f869af78700 5 -- op tracker -- , seq: 1665, time: 2014-06-09 16:22:11.443881, event: done, op: pg_notify(2.1b3(20) epoch 242) 2014-06-09 16:22:11.443891 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443893 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443912 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.443915 7f869af78700 20 osd.26 243 _dispatch 0x13be4c40 pg_notify(1.70c(23),2.17c(20) epoch 243) v5 2014-06-09 16:22:11.443921 7f869af78700 5 -- op tracker -- , seq: 1666, time: 2014-06-09 16:22:11.200787, event: header_read, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.443928 7f869af78700 5 -- op tracker -- , seq: 1666, time: 2014-06-09 16:22:11.200788, event: throttled, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.443934 7f869af78700 5 -- op tracker -- , seq: 1666, time: 2014-06-09 16:22:11.200840, event: all_read, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.443940 7f869af78700 5 -- op tracker -- , seq: 1666, time: 2014-06-09 16:22:11.443912, event: dispatched, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.443947 7f869af78700 5 -- op tracker -- , seq: 1666, time: 2014-06-09 16:22:11.443945, event: waiting_for_osdmap, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.443951 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.443953 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x13be4c40 2014-06-09 16:22:11.443958 7f869af78700 5 -- op tracker -- , seq: 1666, time: 2014-06-09 16:22:11.443955, event: started, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.443981 7f869af78700 5 -- op tracker -- , seq: 1666, time: 2014-06-09 16:22:11.443978, event: done, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.443989 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.443991 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444017 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444018 7f869af78700 20 osd.26 243 _dispatch 0x30615600 pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.444025 7f869af78700 5 -- op tracker -- , seq: 1667, time: 2014-06-09 16:22:11.109814, event: header_read, op: pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444033 7f869af78700 5 -- op tracker -- , seq: 1667, time: 2014-06-09 16:22:11.109814, event: throttled, op: pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444040 7f869af78700 5 -- op tracker -- , seq: 1667, time: 2014-06-09 16:22:11.109825, event: all_read, op: pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444047 7f869af78700 5 -- op tracker -- , seq: 1667, time: 2014-06-09 16:22:11.444015, event: dispatched, op: pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444054 7f869af78700 5 -- op tracker -- , seq: 1667, time: 2014-06-09 16:22:11.444051, event: waiting_for_osdmap, op: pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444059 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.444063 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30615600 2014-06-09 16:22:11.444068 7f869af78700 5 -- op tracker -- , seq: 1667, time: 2014-06-09 16:22:11.444066, event: started, op: pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444080 7f869af78700 5 -- op tracker -- , seq: 1667, time: 2014-06-09 16:22:11.444078, event: done, op: pg_log(2.699 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444088 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444089 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444103 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444104 7f869af78700 20 osd.26 243 _dispatch 0x305bd600 pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.444110 7f869af78700 5 -- op tracker -- , seq: 1668, time: 2014-06-09 16:22:11.103152, event: header_read, op: pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444116 7f869af78700 5 -- op tracker -- , seq: 1668, time: 2014-06-09 16:22:11.103152, event: throttled, op: pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444123 7f869af78700 5 -- op tracker -- , seq: 1668, time: 2014-06-09 16:22:11.103159, event: all_read, op: pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444130 7f869af78700 5 -- op tracker -- , seq: 1668, time: 2014-06-09 16:22:11.444101, event: dispatched, op: pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444137 7f869af78700 5 -- op tracker -- , seq: 1668, time: 2014-06-09 16:22:11.444134, event: waiting_for_osdmap, op: pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444142 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.444145 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305bd600 2014-06-09 16:22:11.444151 7f869af78700 5 -- op tracker -- , seq: 1668, time: 2014-06-09 16:22:11.444148, event: started, op: pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444163 7f869af78700 5 -- op tracker -- , seq: 1668, time: 2014-06-09 16:22:11.444160, event: done, op: pg_log(2.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444170 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444172 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444183 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444184 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444186 7f869af78700 20 osd.26 243 _dispatch 0x2fef5dc0 pg_notify(1.301(20) epoch 243) v5 2014-06-09 16:22:11.444191 7f869af78700 5 -- op tracker -- , seq: 1669, time: 2014-06-09 16:22:11.086963, event: header_read, op: pg_notify(1.301(20) epoch 243) 2014-06-09 16:22:11.444197 7f869af78700 5 -- op tracker -- , seq: 1669, time: 2014-06-09 16:22:11.086964, event: throttled, op: pg_notify(1.301(20) epoch 243) 2014-06-09 16:22:11.444204 7f869af78700 5 -- op tracker -- , seq: 1669, time: 2014-06-09 16:22:11.087008, event: all_read, op: pg_notify(1.301(20) epoch 243) 2014-06-09 16:22:11.444210 7f869af78700 5 -- op tracker -- , seq: 1669, time: 2014-06-09 16:22:11.444183, event: dispatched, op: pg_notify(1.301(20) epoch 243) 2014-06-09 16:22:11.444217 7f869af78700 5 -- op tracker -- , seq: 1669, time: 2014-06-09 16:22:11.444215, event: waiting_for_osdmap, op: pg_notify(1.301(20) epoch 243) 2014-06-09 16:22:11.444221 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.444223 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef5dc0 2014-06-09 16:22:11.444228 7f869af78700 5 -- op tracker -- , seq: 1669, time: 2014-06-09 16:22:11.444226, event: started, op: pg_notify(1.301(20) epoch 243) 2014-06-09 16:22:11.444242 7f869af78700 5 -- op tracker -- , seq: 1669, time: 2014-06-09 16:22:11.444240, event: done, op: pg_notify(1.301(20) epoch 243) 2014-06-09 16:22:11.444249 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444251 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444268 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444270 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444271 7f869af78700 20 osd.26 243 _dispatch 0xdafa000 pg_notify(2.349(20) epoch 243) v5 2014-06-09 16:22:11.444275 7f869af78700 5 -- op tracker -- , seq: 1670, time: 2014-06-09 16:22:11.041930, event: header_read, op: pg_notify(2.349(20) epoch 243) 2014-06-09 16:22:11.444282 7f869af78700 5 -- op tracker -- , seq: 1670, time: 2014-06-09 16:22:11.041931, event: throttled, op: pg_notify(2.349(20) epoch 243) 2014-06-09 16:22:11.444288 7f869af78700 5 -- op tracker -- , seq: 1670, time: 2014-06-09 16:22:11.041966, event: all_read, op: pg_notify(2.349(20) epoch 243) 2014-06-09 16:22:11.444294 7f869af78700 5 -- op tracker -- , seq: 1670, time: 2014-06-09 16:22:11.444268, event: dispatched, op: pg_notify(2.349(20) epoch 243) 2014-06-09 16:22:11.444300 7f869af78700 5 -- op tracker -- , seq: 1670, time: 2014-06-09 16:22:11.444298, event: waiting_for_osdmap, op: pg_notify(2.349(20) epoch 243) 2014-06-09 16:22:11.444305 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.444307 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xdafa000 2014-06-09 16:22:11.444311 7f869af78700 5 -- op tracker -- , seq: 1670, time: 2014-06-09 16:22:11.444309, event: started, op: pg_notify(2.349(20) epoch 243) 2014-06-09 16:22:11.444327 7f869af78700 5 -- op tracker -- , seq: 1670, time: 2014-06-09 16:22:11.444324, event: done, op: pg_notify(2.349(20) epoch 243) 2014-06-09 16:22:11.444334 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444336 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444355 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444356 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444358 7f869af78700 20 osd.26 243 _dispatch 0x2fe1e600 pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.444364 7f869af78700 5 -- op tracker -- , seq: 1671, time: 2014-06-09 16:22:11.032499, event: header_read, op: pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444372 7f869af78700 5 -- op tracker -- , seq: 1671, time: 2014-06-09 16:22:11.032500, event: throttled, op: pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444378 7f869af78700 5 -- op tracker -- , seq: 1671, time: 2014-06-09 16:22:11.032510, event: all_read, op: pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444385 7f869af78700 5 -- op tracker -- , seq: 1671, time: 2014-06-09 16:22:11.444354, event: dispatched, op: pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444392 7f869af78700 5 -- op tracker -- , seq: 1671, time: 2014-06-09 16:22:11.444389, event: waiting_for_osdmap, op: pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444396 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.444400 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe1e600 2014-06-09 16:22:11.444405 7f869af78700 5 -- op tracker -- , seq: 1671, time: 2014-06-09 16:22:11.444403, event: started, op: pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444418 7f869af78700 5 -- op tracker -- , seq: 1671, time: 2014-06-09 16:22:11.444415, event: done, op: pg_log(0.631 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444425 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444438 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444440 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444441 7f869af78700 20 osd.26 243 _dispatch 0x30046400 pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.444447 7f869af78700 5 -- op tracker -- , seq: 1672, time: 2014-06-09 16:22:11.054020, event: header_read, op: pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444455 7f869af78700 5 -- op tracker -- , seq: 1672, time: 2014-06-09 16:22:11.054021, event: throttled, op: pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444462 7f869af78700 5 -- op tracker -- , seq: 1672, time: 2014-06-09 16:22:11.054029, event: all_read, op: pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444468 7f869af78700 5 -- op tracker -- , seq: 1672, time: 2014-06-09 16:22:11.444438, event: dispatched, op: pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444476 7f869af78700 5 -- op tracker -- , seq: 1672, time: 2014-06-09 16:22:11.444473, event: waiting_for_osdmap, op: pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444480 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.444484 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30046400 2014-06-09 16:22:11.444490 7f869af78700 5 -- op tracker -- , seq: 1672, time: 2014-06-09 16:22:11.444487, event: started, op: pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444503 7f869af78700 5 -- op tracker -- , seq: 1672, time: 2014-06-09 16:22:11.444500, event: done, op: pg_log(1.382 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444510 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444512 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444523 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444524 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444526 7f869af78700 20 osd.26 243 _dispatch 0x2fef5180 pg_notify(0.67d(20) epoch 243) v5 2014-06-09 16:22:11.444530 7f869af78700 5 -- op tracker -- , seq: 1673, time: 2014-06-09 16:22:11.130307, event: header_read, op: pg_notify(0.67d(20) epoch 243) 2014-06-09 16:22:11.444537 7f869af78700 5 -- op tracker -- , seq: 1673, time: 2014-06-09 16:22:11.130307, event: throttled, op: pg_notify(0.67d(20) epoch 243) 2014-06-09 16:22:11.444542 7f869af78700 5 -- op tracker -- , seq: 1673, time: 2014-06-09 16:22:11.130340, event: all_read, op: pg_notify(0.67d(20) epoch 243) 2014-06-09 16:22:11.444548 7f869af78700 5 -- op tracker -- , seq: 1673, time: 2014-06-09 16:22:11.444523, event: dispatched, op: pg_notify(0.67d(20) epoch 243) 2014-06-09 16:22:11.444554 7f869af78700 5 -- op tracker -- , seq: 1673, time: 2014-06-09 16:22:11.444552, event: waiting_for_osdmap, op: pg_notify(0.67d(20) epoch 243) 2014-06-09 16:22:11.444558 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.444560 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef5180 2014-06-09 16:22:11.444565 7f869af78700 5 -- op tracker -- , seq: 1673, time: 2014-06-09 16:22:11.444563, event: started, op: pg_notify(0.67d(20) epoch 243) 2014-06-09 16:22:11.444581 7f869af78700 5 -- op tracker -- , seq: 1673, time: 2014-06-09 16:22:11.444578, event: done, op: pg_notify(0.67d(20) epoch 243) 2014-06-09 16:22:11.444593 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444595 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444615 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444617 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444618 7f869af78700 20 osd.26 243 _dispatch 0x16064e00 pg_notify(2.755(21),3.1c7(12) epoch 242) v5 2014-06-09 16:22:11.444623 7f869af78700 5 -- op tracker -- , seq: 1674, time: 2014-06-09 16:22:10.992450, event: header_read, op: pg_notify(2.755(21),3.1c7(12) epoch 242) 2014-06-09 16:22:11.444631 7f869af78700 5 -- op tracker -- , seq: 1674, time: 2014-06-09 16:22:10.992451, event: throttled, op: pg_notify(2.755(21),3.1c7(12) epoch 242) 2014-06-09 16:22:11.444638 7f869af78700 5 -- op tracker -- , seq: 1674, time: 2014-06-09 16:22:10.992478, event: all_read, op: pg_notify(2.755(21),3.1c7(12) epoch 242) 2014-06-09 16:22:11.444644 7f869af78700 5 -- op tracker -- , seq: 1674, time: 2014-06-09 16:22:11.444615, event: dispatched, op: pg_notify(2.755(21),3.1c7(12) epoch 242) 2014-06-09 16:22:11.444651 7f869af78700 5 -- op tracker -- , seq: 1674, time: 2014-06-09 16:22:11.444649, event: waiting_for_osdmap, op: pg_notify(2.755(21),3.1c7(12) epoch 242) 2014-06-09 16:22:11.444655 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.444657 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x16064e00 2014-06-09 16:22:11.444662 7f869af78700 5 -- op tracker -- , seq: 1674, time: 2014-06-09 16:22:11.444660, event: started, op: pg_notify(2.755(21),3.1c7(12) epoch 242) 2014-06-09 16:22:11.444689 7f869af78700 5 -- op tracker -- , seq: 1674, time: 2014-06-09 16:22:11.444687, event: done, op: pg_notify(2.755(21),3.1c7(12) epoch 242) 2014-06-09 16:22:11.444697 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444699 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444719 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444720 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444721 7f869af78700 20 osd.26 243 _dispatch 0x2d733a40 pg_notify(0.4b3(23) epoch 242) v5 2014-06-09 16:22:11.444726 7f869af78700 5 -- op tracker -- , seq: 1675, time: 2014-06-09 16:22:10.939459, event: header_read, op: pg_notify(0.4b3(23) epoch 242) 2014-06-09 16:22:11.444733 7f869af78700 5 -- op tracker -- , seq: 1675, time: 2014-06-09 16:22:10.939459, event: throttled, op: pg_notify(0.4b3(23) epoch 242) 2014-06-09 16:22:11.444739 7f869af78700 5 -- op tracker -- , seq: 1675, time: 2014-06-09 16:22:10.939487, event: all_read, op: pg_notify(0.4b3(23) epoch 242) 2014-06-09 16:22:11.444745 7f869af78700 5 -- op tracker -- , seq: 1675, time: 2014-06-09 16:22:11.444718, event: dispatched, op: pg_notify(0.4b3(23) epoch 242) 2014-06-09 16:22:11.444752 7f869af78700 5 -- op tracker -- , seq: 1675, time: 2014-06-09 16:22:11.444750, event: waiting_for_osdmap, op: pg_notify(0.4b3(23) epoch 242) 2014-06-09 16:22:11.444756 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.444759 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x2d733a40 2014-06-09 16:22:11.444763 7f869af78700 5 -- op tracker -- , seq: 1675, time: 2014-06-09 16:22:11.444761, event: started, op: pg_notify(0.4b3(23) epoch 242) 2014-06-09 16:22:11.444779 7f869af78700 5 -- op tracker -- , seq: 1675, time: 2014-06-09 16:22:11.444777, event: done, op: pg_notify(0.4b3(23) epoch 242) 2014-06-09 16:22:11.444787 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444789 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444807 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444808 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444809 7f869af78700 20 osd.26 243 _dispatch 0x306d2c00 pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.444816 7f869af78700 5 -- op tracker -- , seq: 1676, time: 2014-06-09 16:22:11.200857, event: header_read, op: pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444823 7f869af78700 5 -- op tracker -- , seq: 1676, time: 2014-06-09 16:22:11.200858, event: throttled, op: pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444830 7f869af78700 5 -- op tracker -- , seq: 1676, time: 2014-06-09 16:22:11.200871, event: all_read, op: pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444837 7f869af78700 5 -- op tracker -- , seq: 1676, time: 2014-06-09 16:22:11.444806, event: dispatched, op: pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444844 7f869af78700 5 -- op tracker -- , seq: 1676, time: 2014-06-09 16:22:11.444841, event: waiting_for_osdmap, op: pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444848 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.444852 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d2c00 2014-06-09 16:22:11.444858 7f869af78700 5 -- op tracker -- , seq: 1676, time: 2014-06-09 16:22:11.444855, event: started, op: pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444871 7f869af78700 5 -- op tracker -- , seq: 1676, time: 2014-06-09 16:22:11.444868, event: done, op: pg_log(1.45d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444880 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444881 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444893 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444895 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444896 7f869af78700 20 osd.26 243 _dispatch 0x30570c00 pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.444902 7f869af78700 5 -- op tracker -- , seq: 1677, time: 2014-06-09 16:22:11.109859, event: header_read, op: pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444910 7f869af78700 5 -- op tracker -- , seq: 1677, time: 2014-06-09 16:22:11.109860, event: throttled, op: pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444916 7f869af78700 5 -- op tracker -- , seq: 1677, time: 2014-06-09 16:22:11.109875, event: all_read, op: pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444923 7f869af78700 5 -- op tracker -- , seq: 1677, time: 2014-06-09 16:22:11.444893, event: dispatched, op: pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444930 7f869af78700 5 -- op tracker -- , seq: 1677, time: 2014-06-09 16:22:11.444927, event: waiting_for_osdmap, op: pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444934 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.444938 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30570c00 2014-06-09 16:22:11.444943 7f869af78700 5 -- op tracker -- , seq: 1677, time: 2014-06-09 16:22:11.444941, event: started, op: pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444956 7f869af78700 5 -- op tracker -- , seq: 1677, time: 2014-06-09 16:22:11.444953, event: done, op: pg_log(2.69c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.444963 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444965 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444976 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.444978 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.444979 7f869af78700 20 osd.26 243 _dispatch 0x150e1500 pg_info(1 pgs e243:3.7fc(4)) v4 2014-06-09 16:22:11.444984 7f869af78700 5 -- op tracker -- , seq: 1678, time: 2014-06-09 16:22:11.103162, event: header_read, op: pg_info(1 pgs e243:3.7fc(4)) 2014-06-09 16:22:11.444991 7f869af78700 5 -- op tracker -- , seq: 1678, time: 2014-06-09 16:22:11.103163, event: throttled, op: pg_info(1 pgs e243:3.7fc(4)) 2014-06-09 16:22:11.444997 7f869af78700 5 -- op tracker -- , seq: 1678, time: 2014-06-09 16:22:11.103187, event: all_read, op: pg_info(1 pgs e243:3.7fc(4)) 2014-06-09 16:22:11.445003 7f869af78700 5 -- op tracker -- , seq: 1678, time: 2014-06-09 16:22:11.444976, event: dispatched, op: pg_info(1 pgs e243:3.7fc(4)) 2014-06-09 16:22:11.445010 7f869af78700 5 -- op tracker -- , seq: 1678, time: 2014-06-09 16:22:11.445008, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7fc(4)) 2014-06-09 16:22:11.445014 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7fc(4)) v4 from osd.1 2014-06-09 16:22:11.445017 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e1500 2014-06-09 16:22:11.445022 7f869af78700 5 -- op tracker -- , seq: 1678, time: 2014-06-09 16:22:11.445020, event: started, op: pg_info(1 pgs e243:3.7fc(4)) 2014-06-09 16:22:11.445038 7f869af78700 5 -- op tracker -- , seq: 1678, time: 2014-06-09 16:22:11.445036, event: done, op: pg_info(1 pgs e243:3.7fc(4)) 2014-06-09 16:22:11.445046 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445048 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445062 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445063 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445064 7f869af78700 20 osd.26 243 _dispatch 0x305a9000 pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.445071 7f869af78700 5 -- op tracker -- , seq: 1679, time: 2014-06-09 16:22:11.102969, event: header_read, op: pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445078 7f869af78700 5 -- op tracker -- , seq: 1679, time: 2014-06-09 16:22:11.102970, event: throttled, op: pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445085 7f869af78700 5 -- op tracker -- , seq: 1679, time: 2014-06-09 16:22:11.102980, event: all_read, op: pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445092 7f869af78700 5 -- op tracker -- , seq: 1679, time: 2014-06-09 16:22:11.445061, event: dispatched, op: pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445099 7f869af78700 5 -- op tracker -- , seq: 1679, time: 2014-06-09 16:22:11.445096, event: waiting_for_osdmap, op: pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445103 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.445107 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a9000 2014-06-09 16:22:11.445113 7f869af78700 5 -- op tracker -- , seq: 1679, time: 2014-06-09 16:22:11.445110, event: started, op: pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445125 7f869af78700 5 -- op tracker -- , seq: 1679, time: 2014-06-09 16:22:11.445123, event: done, op: pg_log(1.6e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445132 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445134 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445145 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445148 7f869af78700 20 osd.26 243 _dispatch 0x3043a8c0 pg_info(1 pgs e243:3.616(2)) v4 2014-06-09 16:22:11.445153 7f869af78700 5 -- op tracker -- , seq: 1680, time: 2014-06-09 16:22:11.041976, event: header_read, op: pg_info(1 pgs e243:3.616(2)) 2014-06-09 16:22:11.445160 7f869af78700 5 -- op tracker -- , seq: 1680, time: 2014-06-09 16:22:11.041977, event: throttled, op: pg_info(1 pgs e243:3.616(2)) 2014-06-09 16:22:11.445166 7f869af78700 5 -- op tracker -- , seq: 1680, time: 2014-06-09 16:22:11.041992, event: all_read, op: pg_info(1 pgs e243:3.616(2)) 2014-06-09 16:22:11.445173 7f869af78700 5 -- op tracker -- , seq: 1680, time: 2014-06-09 16:22:11.445145, event: dispatched, op: pg_info(1 pgs e243:3.616(2)) 2014-06-09 16:22:11.445179 7f869af78700 5 -- op tracker -- , seq: 1680, time: 2014-06-09 16:22:11.445177, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.616(2)) 2014-06-09 16:22:11.445184 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.616(2)) v4 from osd.8 2014-06-09 16:22:11.445187 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043a8c0 2014-06-09 16:22:11.445192 7f869af78700 5 -- op tracker -- , seq: 1680, time: 2014-06-09 16:22:11.445189, event: started, op: pg_info(1 pgs e243:3.616(2)) 2014-06-09 16:22:11.445208 7f869af78700 5 -- op tracker -- , seq: 1680, time: 2014-06-09 16:22:11.445206, event: done, op: pg_info(1 pgs e243:3.616(2)) 2014-06-09 16:22:11.445216 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445217 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445230 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445231 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445232 7f869af78700 20 osd.26 243 _dispatch 0x2fe1e000 pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.445239 7f869af78700 5 -- op tracker -- , seq: 1681, time: 2014-06-09 16:22:11.059419, event: header_read, op: pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445246 7f869af78700 5 -- op tracker -- , seq: 1681, time: 2014-06-09 16:22:11.059420, event: throttled, op: pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445253 7f869af78700 5 -- op tracker -- , seq: 1681, time: 2014-06-09 16:22:11.059428, event: all_read, op: pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445260 7f869af78700 5 -- op tracker -- , seq: 1681, time: 2014-06-09 16:22:11.445229, event: dispatched, op: pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445267 7f869af78700 5 -- op tracker -- , seq: 1681, time: 2014-06-09 16:22:11.445264, event: waiting_for_osdmap, op: pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445272 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.445276 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe1e000 2014-06-09 16:22:11.445281 7f869af78700 5 -- op tracker -- , seq: 1681, time: 2014-06-09 16:22:11.445278, event: started, op: pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445293 7f869af78700 5 -- op tracker -- , seq: 1681, time: 2014-06-09 16:22:11.445291, event: done, op: pg_log(2.34d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445301 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445303 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445315 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445316 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445318 7f869af78700 20 osd.26 243 _dispatch 0x30045e00 pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.445324 7f869af78700 5 -- op tracker -- , seq: 1682, time: 2014-06-09 16:22:11.054042, event: header_read, op: pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445332 7f869af78700 5 -- op tracker -- , seq: 1682, time: 2014-06-09 16:22:11.054043, event: throttled, op: pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445338 7f869af78700 5 -- op tracker -- , seq: 1682, time: 2014-06-09 16:22:11.054053, event: all_read, op: pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445344 7f869af78700 5 -- op tracker -- , seq: 1682, time: 2014-06-09 16:22:11.445314, event: dispatched, op: pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445351 7f869af78700 5 -- op tracker -- , seq: 1682, time: 2014-06-09 16:22:11.445349, event: waiting_for_osdmap, op: pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445356 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.445359 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30045e00 2014-06-09 16:22:11.445364 7f869af78700 5 -- op tracker -- , seq: 1682, time: 2014-06-09 16:22:11.445362, event: started, op: pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445377 7f869af78700 5 -- op tracker -- , seq: 1682, time: 2014-06-09 16:22:11.445374, event: done, op: pg_log(2.e6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445384 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445386 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445398 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445400 7f869af78700 20 osd.26 243 _dispatch 0x305bc400 pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.445407 7f869af78700 5 -- op tracker -- , seq: 1683, time: 2014-06-09 16:22:11.174205, event: header_read, op: pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445415 7f869af78700 5 -- op tracker -- , seq: 1683, time: 2014-06-09 16:22:11.174206, event: throttled, op: pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445422 7f869af78700 5 -- op tracker -- , seq: 1683, time: 2014-06-09 16:22:11.174217, event: all_read, op: pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445428 7f869af78700 5 -- op tracker -- , seq: 1683, time: 2014-06-09 16:22:11.445397, event: dispatched, op: pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445436 7f869af78700 5 -- op tracker -- , seq: 1683, time: 2014-06-09 16:22:11.445433, event: waiting_for_osdmap, op: pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445441 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.445444 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305bc400 2014-06-09 16:22:11.445449 7f869af78700 5 -- op tracker -- , seq: 1683, time: 2014-06-09 16:22:11.445447, event: started, op: pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445462 7f869af78700 5 -- op tracker -- , seq: 1683, time: 2014-06-09 16:22:11.445460, event: done, op: pg_log(2.fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445470 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445471 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445483 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445485 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445486 7f869af78700 20 osd.26 243 _dispatch 0x30420fc0 osd_map(243..243 src has 1..243) v3 2014-06-09 16:22:11.445493 7f869af78700 3 osd.26 243 handle_osd_map epochs [243,243], i have 243, src has [1,243] 2014-06-09 16:22:11.445496 7f869af78700 10 osd.26 243 no new maps here, dropping 2014-06-09 16:22:11.445498 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445499 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445511 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445512 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445513 7f869af78700 20 osd.26 243 _dispatch 0xbe9cfc0 pg_notify(1.75b(20) epoch 242) v5 2014-06-09 16:22:11.445518 7f869af78700 5 -- op tracker -- , seq: 1684, time: 2014-06-09 16:22:10.945249, event: header_read, op: pg_notify(1.75b(20) epoch 242) 2014-06-09 16:22:11.445525 7f869af78700 5 -- op tracker -- , seq: 1684, time: 2014-06-09 16:22:10.945250, event: throttled, op: pg_notify(1.75b(20) epoch 242) 2014-06-09 16:22:11.445531 7f869af78700 5 -- op tracker -- , seq: 1684, time: 2014-06-09 16:22:10.945284, event: all_read, op: pg_notify(1.75b(20) epoch 242) 2014-06-09 16:22:11.445537 7f869af78700 5 -- op tracker -- , seq: 1684, time: 2014-06-09 16:22:11.445510, event: dispatched, op: pg_notify(1.75b(20) epoch 242) 2014-06-09 16:22:11.445543 7f869af78700 5 -- op tracker -- , seq: 1684, time: 2014-06-09 16:22:11.445541, event: waiting_for_osdmap, op: pg_notify(1.75b(20) epoch 242) 2014-06-09 16:22:11.445548 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.445549 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0xbe9cfc0 2014-06-09 16:22:11.445554 7f869af78700 5 -- op tracker -- , seq: 1684, time: 2014-06-09 16:22:11.445552, event: started, op: pg_notify(1.75b(20) epoch 242) 2014-06-09 16:22:11.445570 7f869af78700 5 -- op tracker -- , seq: 1684, time: 2014-06-09 16:22:11.445568, event: done, op: pg_notify(1.75b(20) epoch 242) 2014-06-09 16:22:11.445578 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445580 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445599 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445600 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445602 7f869af78700 20 osd.26 243 _dispatch 0x306dbe00 pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.445608 7f869af78700 5 -- op tracker -- , seq: 1685, time: 2014-06-09 16:22:11.202360, event: header_read, op: pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445616 7f869af78700 5 -- op tracker -- , seq: 1685, time: 2014-06-09 16:22:11.202361, event: throttled, op: pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445623 7f869af78700 5 -- op tracker -- , seq: 1685, time: 2014-06-09 16:22:11.202373, event: all_read, op: pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445629 7f869af78700 5 -- op tracker -- , seq: 1685, time: 2014-06-09 16:22:11.445598, event: dispatched, op: pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445636 7f869af78700 5 -- op tracker -- , seq: 1685, time: 2014-06-09 16:22:11.445634, event: waiting_for_osdmap, op: pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445641 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.445645 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306dbe00 2014-06-09 16:22:11.445651 7f869af78700 5 -- op tracker -- , seq: 1685, time: 2014-06-09 16:22:11.445648, event: started, op: pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445663 7f869af78700 5 -- op tracker -- , seq: 1685, time: 2014-06-09 16:22:11.445661, event: done, op: pg_log(0.25f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445675 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445676 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445688 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445690 7f869af78700 20 osd.26 243 _dispatch 0xdafbdc0 pg_notify(0.3cf(20) epoch 243) v5 2014-06-09 16:22:11.445695 7f869af78700 5 -- op tracker -- , seq: 1686, time: 2014-06-09 16:22:11.109919, event: header_read, op: pg_notify(0.3cf(20) epoch 243) 2014-06-09 16:22:11.445703 7f869af78700 5 -- op tracker -- , seq: 1686, time: 2014-06-09 16:22:11.109920, event: throttled, op: pg_notify(0.3cf(20) epoch 243) 2014-06-09 16:22:11.445709 7f869af78700 5 -- op tracker -- , seq: 1686, time: 2014-06-09 16:22:11.109948, event: all_read, op: pg_notify(0.3cf(20) epoch 243) 2014-06-09 16:22:11.445715 7f869af78700 5 -- op tracker -- , seq: 1686, time: 2014-06-09 16:22:11.445687, event: dispatched, op: pg_notify(0.3cf(20) epoch 243) 2014-06-09 16:22:11.445721 7f869af78700 5 -- op tracker -- , seq: 1686, time: 2014-06-09 16:22:11.445719, event: waiting_for_osdmap, op: pg_notify(0.3cf(20) epoch 243) 2014-06-09 16:22:11.445725 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.445727 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xdafbdc0 2014-06-09 16:22:11.445732 7f869af78700 5 -- op tracker -- , seq: 1686, time: 2014-06-09 16:22:11.445730, event: started, op: pg_notify(0.3cf(20) epoch 243) 2014-06-09 16:22:11.445747 7f869af78700 5 -- op tracker -- , seq: 1686, time: 2014-06-09 16:22:11.445745, event: done, op: pg_notify(0.3cf(20) epoch 243) 2014-06-09 16:22:11.445755 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445757 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445775 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.445776 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.445777 7f869af78700 20 osd.26 243 _dispatch 0x305a6600 pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.445784 7f869af78700 5 -- op tracker -- , seq: 1687, time: 2014-06-09 16:22:11.103194, event: header_read, op: pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445791 7f869af78700 5 -- op tracker -- , seq: 1687, time: 2014-06-09 16:22:11.103194, event: throttled, op: pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445798 7f869af78700 5 -- op tracker -- , seq: 1687, time: 2014-06-09 16:22:11.103204, event: all_read, op: pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445805 7f869af78700 5 -- op tracker -- , seq: 1687, time: 2014-06-09 16:22:11.445775, event: dispatched, op: pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445812 7f869af78700 5 -- op tracker -- , seq: 1687, time: 2014-06-09 16:22:11.445809, event: waiting_for_osdmap, op: pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445816 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.445820 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a6600 2014-06-09 16:22:11.445826 7f869af78700 5 -- op tracker -- , seq: 1687, time: 2014-06-09 16:22:11.445823, event: started, op: pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.445917 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0290 (10 active) 2014-06-09 16:22:11.445924 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:0.1f1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.445940 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.445951 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.445967 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:3.424( v 214'730 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.445983 7f868e75f700 7 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.4 2014-06-09 16:22:11.445994 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.4 info 3.424( v 214'730 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242), identical to ours 2014-06-09 16:22:11.446015 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.446020 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.2c0( v 214'605 (0'0,214'605] local-les=210 n=122 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.446034 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.446046 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.446050 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=201 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.446092 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02a0 (11 active) 2014-06-09 16:22:11.455428 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MLogRec from 2 2014-06-09 16:22:11.455452 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.562( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) log((0'0,213'605], crt=0'0) missing(0) 2014-06-09 16:22:11.455467 7f868df5e700 10 proc_replica_log for osd.2: 3.562( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) log((0'0,213'605], crt=0'0) missing(0) 2014-06-09 16:22:11.455472 7f868df5e700 20 merge_log cut point (usually last shared) is 213'605 (213'604) modify 5fd44562/rbd_data.11776b8b4567.0000000000001741/head//3 by client.4567.0:34488 2014-06-09 10:18:25.298439 2014-06-09 16:22:11.455974 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.562( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.455992 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.456006 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.490432 6 0.000276 2014-06-09 16:22:11.456020 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.456030 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.958666 0 0.000000 2014-06-09 16:22:11.456046 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.456056 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=191 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.456072 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.456082 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.456090 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'607 -> 214'607 2014-06-09 16:22:11.456129 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.562( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.456150 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'605,214'607], crt=0'0) 2014-06-09 16:22:11.456165 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.456185 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.562( v 214'607 lc 213'605 (0'0,214'607] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.456200 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:11.456209 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing e1b48d62/rbd_data.116c6b8b4567.0000000000001780/head//3 214'607 is on osd.26 2014-06-09 16:22:11.456222 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {e1b48d62/rbd_data.116c6b8b4567.0000000000001780/head//3=214'607} 2014-06-09 16:22:11.456231 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.456241 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing e1b48d62/rbd_data.116c6b8b4567.0000000000001780/head//3 214'607 also missing on osd.2 2014-06-09 16:22:11.456251 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {e1b48d62/rbd_data.116c6b8b4567.0000000000001780/head//3=214'607} 2014-06-09 16:22:11.456261 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.456270 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.456281 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.456290 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.456299 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.456386 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.456392 7f868df5e700 5 queue_transactions existing osr(3.562 0x3ed1920)/0x3ed1920 2014-06-09 16:22:11.446339 7f869af78700 5 -- op tracker -- , seq: 1687, time: 2014-06-09 16:22:11.446335, event: done, op: pg_log(0.3b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456415 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456417 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456434 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456436 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456437 7f869af78700 20 osd.26 243 _dispatch 0x305a9600 pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.456445 7f869af78700 5 -- op tracker -- , seq: 1688, time: 2014-06-09 16:22:11.103022, event: header_read, op: pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456454 7f869af78700 5 -- op tracker -- , seq: 1688, time: 2014-06-09 16:22:11.103022, event: throttled, op: pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456461 7f869af78700 5 -- op tracker -- , seq: 1688, time: 2014-06-09 16:22:11.103035, event: all_read, op: pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456468 7f869af78700 5 -- op tracker -- , seq: 1688, time: 2014-06-09 16:22:11.456433, event: dispatched, op: pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456475 7f869af78700 5 -- op tracker -- , seq: 1688, time: 2014-06-09 16:22:11.456472, event: waiting_for_osdmap, op: pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456480 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.456485 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a9600 2014-06-09 16:22:11.456492 7f869af78700 5 -- op tracker -- , seq: 1688, time: 2014-06-09 16:22:11.456489, event: started, op: pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456509 7f869af78700 5 -- op tracker -- , seq: 1688, time: 2014-06-09 16:22:11.456505, event: done, op: pg_log(0.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456517 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456519 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456533 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456535 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456536 7f869af78700 20 osd.26 243 _dispatch 0x2fe701c0 pg_notify(0.629(20),2.360(23),1.62a(23) epoch 243) v5 2014-06-09 16:22:11.456542 7f869af78700 5 -- op tracker -- , seq: 1689, time: 2014-06-09 16:22:11.047626, event: header_read, op: pg_notify(0.629(20),2.360(23),1.62a(23) epoch 243) 2014-06-09 16:22:11.456551 7f869af78700 5 -- op tracker -- , seq: 1689, time: 2014-06-09 16:22:11.047627, event: throttled, op: pg_notify(0.629(20),2.360(23),1.62a(23) epoch 243) 2014-06-09 16:22:11.456558 7f869af78700 5 -- op tracker -- , seq: 1689, time: 2014-06-09 16:22:11.047701, event: all_read, op: pg_notify(0.629(20),2.360(23),1.62a(23) epoch 243) 2014-06-09 16:22:11.456565 7f869af78700 5 -- op tracker -- , seq: 1689, time: 2014-06-09 16:22:11.456532, event: dispatched, op: pg_notify(0.629(20),2.360(23),1.62a(23) epoch 243) 2014-06-09 16:22:11.456573 7f869af78700 5 -- op tracker -- , seq: 1689, time: 2014-06-09 16:22:11.456571, event: waiting_for_osdmap, op: pg_notify(0.629(20),2.360(23),1.62a(23) epoch 243) 2014-06-09 16:22:11.456578 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.456581 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe701c0 2014-06-09 16:22:11.456587 7f869af78700 5 -- op tracker -- , seq: 1689, time: 2014-06-09 16:22:11.456584, event: started, op: pg_notify(0.629(20),2.360(23),1.62a(23) epoch 243) 2014-06-09 16:22:11.456621 7f869af78700 5 -- op tracker -- , seq: 1689, time: 2014-06-09 16:22:11.456618, event: done, op: pg_notify(0.629(20),2.360(23),1.62a(23) epoch 243) 2014-06-09 16:22:11.456629 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456631 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456661 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456663 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456664 7f869af78700 20 osd.26 243 _dispatch 0x3043aa80 pg_notify(3.7f(10) epoch 243) v5 2014-06-09 16:22:11.456670 7f869af78700 5 -- op tracker -- , seq: 1690, time: 2014-06-09 16:22:11.059487, event: header_read, op: pg_notify(3.7f(10) epoch 243) 2014-06-09 16:22:11.456677 7f869af78700 5 -- op tracker -- , seq: 1690, time: 2014-06-09 16:22:11.059487, event: throttled, op: pg_notify(3.7f(10) epoch 243) 2014-06-09 16:22:11.456684 7f869af78700 5 -- op tracker -- , seq: 1690, time: 2014-06-09 16:22:11.059501, event: all_read, op: pg_notify(3.7f(10) epoch 243) 2014-06-09 16:22:11.456690 7f869af78700 5 -- op tracker -- , seq: 1690, time: 2014-06-09 16:22:11.456661, event: dispatched, op: pg_notify(3.7f(10) epoch 243) 2014-06-09 16:22:11.456696 7f869af78700 5 -- op tracker -- , seq: 1690, time: 2014-06-09 16:22:11.456694, event: waiting_for_osdmap, op: pg_notify(3.7f(10) epoch 243) 2014-06-09 16:22:11.456701 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.456703 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043aa80 2014-06-09 16:22:11.456709 7f869af78700 5 -- op tracker -- , seq: 1690, time: 2014-06-09 16:22:11.456706, event: started, op: pg_notify(3.7f(10) epoch 243) 2014-06-09 16:22:11.456726 7f869af78700 5 -- op tracker -- , seq: 1690, time: 2014-06-09 16:22:11.456724, event: done, op: pg_notify(3.7f(10) epoch 243) 2014-06-09 16:22:11.456734 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456736 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456758 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456760 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456761 7f869af78700 20 osd.26 243 _dispatch 0x304dc000 pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.456768 7f869af78700 5 -- op tracker -- , seq: 1691, time: 2014-06-09 16:22:11.075021, event: header_read, op: pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456776 7f869af78700 5 -- op tracker -- , seq: 1691, time: 2014-06-09 16:22:11.075022, event: throttled, op: pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456783 7f869af78700 5 -- op tracker -- , seq: 1691, time: 2014-06-09 16:22:11.075033, event: all_read, op: pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456790 7f869af78700 5 -- op tracker -- , seq: 1691, time: 2014-06-09 16:22:11.456757, event: dispatched, op: pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456798 7f869af78700 5 -- op tracker -- , seq: 1691, time: 2014-06-09 16:22:11.456795, event: waiting_for_osdmap, op: pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456803 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.456806 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304dc000 2014-06-09 16:22:11.456812 7f869af78700 5 -- op tracker -- , seq: 1691, time: 2014-06-09 16:22:11.456810, event: started, op: pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456826 7f869af78700 5 -- op tracker -- , seq: 1691, time: 2014-06-09 16:22:11.456823, event: done, op: pg_log(0.681 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.456834 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456836 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456850 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456851 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456852 7f869af78700 20 osd.26 243 _dispatch 0x15999c00 pg_notify(3.542(10) epoch 243) v5 2014-06-09 16:22:11.456857 7f869af78700 5 -- op tracker -- , seq: 1692, time: 2014-06-09 16:22:11.189759, event: header_read, op: pg_notify(3.542(10) epoch 243) 2014-06-09 16:22:11.456865 7f869af78700 5 -- op tracker -- , seq: 1692, time: 2014-06-09 16:22:11.189760, event: throttled, op: pg_notify(3.542(10) epoch 243) 2014-06-09 16:22:11.456871 7f869af78700 5 -- op tracker -- , seq: 1692, time: 2014-06-09 16:22:11.189782, event: all_read, op: pg_notify(3.542(10) epoch 243) 2014-06-09 16:22:11.456877 7f869af78700 5 -- op tracker -- , seq: 1692, time: 2014-06-09 16:22:11.456849, event: dispatched, op: pg_notify(3.542(10) epoch 243) 2014-06-09 16:22:11.456884 7f869af78700 5 -- op tracker -- , seq: 1692, time: 2014-06-09 16:22:11.456882, event: waiting_for_osdmap, op: pg_notify(3.542(10) epoch 243) 2014-06-09 16:22:11.456888 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.456890 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15999c00 2014-06-09 16:22:11.456895 7f869af78700 5 -- op tracker -- , seq: 1692, time: 2014-06-09 16:22:11.456893, event: started, op: pg_notify(3.542(10) epoch 243) 2014-06-09 16:22:11.456911 7f869af78700 5 -- op tracker -- , seq: 1692, time: 2014-06-09 16:22:11.456909, event: done, op: pg_notify(3.542(10) epoch 243) 2014-06-09 16:22:11.456919 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456920 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456936 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.456937 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.456938 7f869af78700 20 osd.26 243 _dispatch 0x159988c0 pg_notify(3.7af(12) epoch 243) v5 2014-06-09 16:22:11.456943 7f869af78700 5 -- op tracker -- , seq: 1693, time: 2014-06-09 16:22:11.017459, event: header_read, op: pg_notify(3.7af(12) epoch 243) 2014-06-09 16:22:11.456951 7f869af78700 5 -- op tracker -- , seq: 1693, time: 2014-06-09 16:22:11.017459, event: throttled, op: pg_notify(3.7af(12) epoch 243) 2014-06-09 16:22:11.456957 7f869af78700 5 -- op tracker -- , seq: 1693, time: 2014-06-09 16:22:11.017486, event: all_read, op: pg_notify(3.7af(12) epoch 243) 2014-06-09 16:22:11.456964 7f869af78700 5 -- op tracker -- , seq: 1693, time: 2014-06-09 16:22:11.456935, event: dispatched, op: pg_notify(3.7af(12) epoch 243) 2014-06-09 16:22:11.456970 7f869af78700 5 -- op tracker -- , seq: 1693, time: 2014-06-09 16:22:11.456968, event: waiting_for_osdmap, op: pg_notify(3.7af(12) epoch 243) 2014-06-09 16:22:11.456975 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.456977 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x159988c0 2014-06-09 16:22:11.456982 7f869af78700 5 -- op tracker -- , seq: 1693, time: 2014-06-09 16:22:11.456980, event: started, op: pg_notify(3.7af(12) epoch 243) 2014-06-09 16:22:11.456999 7f869af78700 5 -- op tracker -- , seq: 1693, time: 2014-06-09 16:22:11.456997, event: done, op: pg_notify(3.7af(12) epoch 243) 2014-06-09 16:22:11.457007 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457009 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457024 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457026 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457027 7f869af78700 20 osd.26 243 _dispatch 0x16064fc0 pg_notify(0.759(20),1.1c9(23) epoch 242) v5 2014-06-09 16:22:11.457033 7f869af78700 5 -- op tracker -- , seq: 1694, time: 2014-06-09 16:22:10.992662, event: header_read, op: pg_notify(0.759(20),1.1c9(23) epoch 242) 2014-06-09 16:22:11.457040 7f869af78700 5 -- op tracker -- , seq: 1694, time: 2014-06-09 16:22:10.992662, event: throttled, op: pg_notify(0.759(20),1.1c9(23) epoch 242) 2014-06-09 16:22:11.457046 7f869af78700 5 -- op tracker -- , seq: 1694, time: 2014-06-09 16:22:10.992709, event: all_read, op: pg_notify(0.759(20),1.1c9(23) epoch 242) 2014-06-09 16:22:11.457052 7f869af78700 5 -- op tracker -- , seq: 1694, time: 2014-06-09 16:22:11.457024, event: dispatched, op: pg_notify(0.759(20),1.1c9(23) epoch 242) 2014-06-09 16:22:11.457059 7f869af78700 5 -- op tracker -- , seq: 1694, time: 2014-06-09 16:22:11.457057, event: waiting_for_osdmap, op: pg_notify(0.759(20),1.1c9(23) epoch 242) 2014-06-09 16:22:11.457064 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.457066 7f869af78700 15 osd.26 243 require_same_or_newer_map 242 (i am 243) 0x16064fc0 2014-06-09 16:22:11.457072 7f869af78700 5 -- op tracker -- , seq: 1694, time: 2014-06-09 16:22:11.457069, event: started, op: pg_notify(0.759(20),1.1c9(23) epoch 242) 2014-06-09 16:22:11.457097 7f869af78700 5 -- op tracker -- , seq: 1694, time: 2014-06-09 16:22:11.457095, event: done, op: pg_notify(0.759(20),1.1c9(23) epoch 242) 2014-06-09 16:22:11.457106 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457107 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457134 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457136 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457138 7f869af78700 20 osd.26 243 _dispatch 0x308d3600 pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.457144 7f869af78700 5 -- op tracker -- , seq: 1695, time: 2014-06-09 16:22:11.222668, event: header_read, op: pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457152 7f869af78700 5 -- op tracker -- , seq: 1695, time: 2014-06-09 16:22:11.222669, event: throttled, op: pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457159 7f869af78700 5 -- op tracker -- , seq: 1695, time: 2014-06-09 16:22:11.222682, event: all_read, op: pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457166 7f869af78700 5 -- op tracker -- , seq: 1695, time: 2014-06-09 16:22:11.457134, event: dispatched, op: pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457173 7f869af78700 5 -- op tracker -- , seq: 1695, time: 2014-06-09 16:22:11.457171, event: waiting_for_osdmap, op: pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457178 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.457182 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d3600 2014-06-09 16:22:11.457188 7f869af78700 5 -- op tracker -- , seq: 1695, time: 2014-06-09 16:22:11.457185, event: started, op: pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457201 7f869af78700 5 -- op tracker -- , seq: 1695, time: 2014-06-09 16:22:11.457198, event: done, op: pg_log(2.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457208 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457210 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457222 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457223 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457224 7f869af78700 20 osd.26 243 _dispatch 0x1405ddc0 pg_notify(3.ed(10) epoch 243) v5 2014-06-09 16:22:11.457230 7f869af78700 5 -- op tracker -- , seq: 1696, time: 2014-06-09 16:22:11.110122, event: header_read, op: pg_notify(3.ed(10) epoch 243) 2014-06-09 16:22:11.457237 7f869af78700 5 -- op tracker -- , seq: 1696, time: 2014-06-09 16:22:11.110123, event: throttled, op: pg_notify(3.ed(10) epoch 243) 2014-06-09 16:22:11.457242 7f869af78700 5 -- op tracker -- , seq: 1696, time: 2014-06-09 16:22:11.110142, event: all_read, op: pg_notify(3.ed(10) epoch 243) 2014-06-09 16:22:11.457248 7f869af78700 5 -- op tracker -- , seq: 1696, time: 2014-06-09 16:22:11.457221, event: dispatched, op: pg_notify(3.ed(10) epoch 243) 2014-06-09 16:22:11.457254 7f869af78700 5 -- op tracker -- , seq: 1696, time: 2014-06-09 16:22:11.457252, event: waiting_for_osdmap, op: pg_notify(3.ed(10) epoch 243) 2014-06-09 16:22:11.457259 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.457261 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405ddc0 2014-06-09 16:22:11.457265 7f869af78700 5 -- op tracker -- , seq: 1696, time: 2014-06-09 16:22:11.457263, event: started, op: pg_notify(3.ed(10) epoch 243) 2014-06-09 16:22:11.457281 7f869af78700 5 -- op tracker -- , seq: 1696, time: 2014-06-09 16:22:11.457279, event: done, op: pg_notify(3.ed(10) epoch 243) 2014-06-09 16:22:11.457289 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457290 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457306 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457308 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457309 7f869af78700 20 osd.26 243 _dispatch 0x305a6000 pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.457316 7f869af78700 5 -- op tracker -- , seq: 1697, time: 2014-06-09 16:22:11.103214, event: header_read, op: pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457324 7f869af78700 5 -- op tracker -- , seq: 1697, time: 2014-06-09 16:22:11.103214, event: throttled, op: pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457331 7f869af78700 5 -- op tracker -- , seq: 1697, time: 2014-06-09 16:22:11.103221, event: all_read, op: pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457338 7f869af78700 5 -- op tracker -- , seq: 1697, time: 2014-06-09 16:22:11.457305, event: dispatched, op: pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457345 7f869af78700 5 -- op tracker -- , seq: 1697, time: 2014-06-09 16:22:11.457342, event: waiting_for_osdmap, op: pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457349 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.457353 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a6000 2014-06-09 16:22:11.457358 7f869af78700 5 -- op tracker -- , seq: 1697, time: 2014-06-09 16:22:11.457356, event: started, op: pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457371 7f869af78700 5 -- op tracker -- , seq: 1697, time: 2014-06-09 16:22:11.457368, event: done, op: pg_log(2.f4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457379 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457381 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457394 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457395 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457396 7f869af78700 20 osd.26 243 _dispatch 0x30614a00 pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.457404 7f869af78700 5 -- op tracker -- , seq: 1698, time: 2014-06-09 16:22:11.103074, event: header_read, op: pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457411 7f869af78700 5 -- op tracker -- , seq: 1698, time: 2014-06-09 16:22:11.103075, event: throttled, op: pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457418 7f869af78700 5 -- op tracker -- , seq: 1698, time: 2014-06-09 16:22:11.103087, event: all_read, op: pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457425 7f869af78700 5 -- op tracker -- , seq: 1698, time: 2014-06-09 16:22:11.457394, event: dispatched, op: pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457432 7f869af78700 5 -- op tracker -- , seq: 1698, time: 2014-06-09 16:22:11.457430, event: waiting_for_osdmap, op: pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457437 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.457441 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30614a00 2014-06-09 16:22:11.457446 7f869af78700 5 -- op tracker -- , seq: 1698, time: 2014-06-09 16:22:11.457443, event: started, op: pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457459 7f869af78700 5 -- op tracker -- , seq: 1698, time: 2014-06-09 16:22:11.457456, event: done, op: pg_log(2.501 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457466 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457468 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457482 7f869af78700 20 osd.26 243 _dispatch 0x14dca600 pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.457489 7f869af78700 5 -- op tracker -- , seq: 1699, time: 2014-06-09 16:22:11.092327, event: header_read, op: pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457497 7f869af78700 5 -- op tracker -- , seq: 1699, time: 2014-06-09 16:22:11.092328, event: throttled, op: pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457504 7f869af78700 5 -- op tracker -- , seq: 1699, time: 2014-06-09 16:22:11.092346, event: all_read, op: pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457511 7f869af78700 5 -- op tracker -- , seq: 1699, time: 2014-06-09 16:22:11.457479, event: dispatched, op: pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457518 7f869af78700 5 -- op tracker -- , seq: 1699, time: 2014-06-09 16:22:11.457516, event: waiting_for_osdmap, op: pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457523 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.457527 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dca600 2014-06-09 16:22:11.457534 7f869af78700 5 -- op tracker -- , seq: 1699, time: 2014-06-09 16:22:11.457531, event: started, op: pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457546 7f869af78700 5 -- op tracker -- , seq: 1699, time: 2014-06-09 16:22:11.457543, event: done, op: pg_log(0.36d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457556 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457558 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457570 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457572 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457573 7f869af78700 20 osd.26 243 _dispatch 0x1fc04380 pg_info(1 pgs e243:3.7a(2)) v4 2014-06-09 16:22:11.457578 7f869af78700 5 -- op tracker -- , seq: 1700, time: 2014-06-09 16:22:11.059553, event: header_read, op: pg_info(1 pgs e243:3.7a(2)) 2014-06-09 16:22:11.457585 7f869af78700 5 -- op tracker -- , seq: 1700, time: 2014-06-09 16:22:11.059554, event: throttled, op: pg_info(1 pgs e243:3.7a(2)) 2014-06-09 16:22:11.457592 7f869af78700 5 -- op tracker -- , seq: 1700, time: 2014-06-09 16:22:11.059565, event: all_read, op: pg_info(1 pgs e243:3.7a(2)) 2014-06-09 16:22:11.457598 7f869af78700 5 -- op tracker -- , seq: 1700, time: 2014-06-09 16:22:11.457570, event: dispatched, op: pg_info(1 pgs e243:3.7a(2)) 2014-06-09 16:22:11.457605 7f869af78700 5 -- op tracker -- , seq: 1700, time: 2014-06-09 16:22:11.457603, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7a(2)) 2014-06-09 16:22:11.457610 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7a(2)) v4 from osd.6 2014-06-09 16:22:11.457614 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc04380 2014-06-09 16:22:11.457619 7f869af78700 5 -- op tracker -- , seq: 1700, time: 2014-06-09 16:22:11.457617, event: started, op: pg_info(1 pgs e243:3.7a(2)) 2014-06-09 16:22:11.457635 7f869af78700 5 -- op tracker -- , seq: 1700, time: 2014-06-09 16:22:11.457633, event: done, op: pg_info(1 pgs e243:3.7a(2)) 2014-06-09 16:22:11.457643 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457645 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457661 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457662 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457663 7f869af78700 20 osd.26 243 _dispatch 0x15a29180 pg_notify(2.65c(21) epoch 243) v5 2014-06-09 16:22:11.457669 7f869af78700 5 -- op tracker -- , seq: 1701, time: 2014-06-09 16:22:11.075042, event: header_read, op: pg_notify(2.65c(21) epoch 243) 2014-06-09 16:22:11.457676 7f869af78700 5 -- op tracker -- , seq: 1701, time: 2014-06-09 16:22:11.075042, event: throttled, op: pg_notify(2.65c(21) epoch 243) 2014-06-09 16:22:11.457682 7f869af78700 5 -- op tracker -- , seq: 1701, time: 2014-06-09 16:22:11.075070, event: all_read, op: pg_notify(2.65c(21) epoch 243) 2014-06-09 16:22:11.457689 7f869af78700 5 -- op tracker -- , seq: 1701, time: 2014-06-09 16:22:11.457660, event: dispatched, op: pg_notify(2.65c(21) epoch 243) 2014-06-09 16:22:11.457695 7f869af78700 5 -- op tracker -- , seq: 1701, time: 2014-06-09 16:22:11.457693, event: waiting_for_osdmap, op: pg_notify(2.65c(21) epoch 243) 2014-06-09 16:22:11.457699 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.457701 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a29180 2014-06-09 16:22:11.457706 7f869af78700 5 -- op tracker -- , seq: 1701, time: 2014-06-09 16:22:11.457704, event: started, op: pg_notify(2.65c(21) epoch 243) 2014-06-09 16:22:11.457721 7f869af78700 5 -- op tracker -- , seq: 1701, time: 2014-06-09 16:22:11.457719, event: done, op: pg_notify(2.65c(21) epoch 243) 2014-06-09 16:22:11.457729 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457731 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457747 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457749 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457750 7f869af78700 20 osd.26 243 _dispatch 0x305ba600 pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.457756 7f869af78700 5 -- op tracker -- , seq: 1702, time: 2014-06-09 16:22:11.189825, event: header_read, op: pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457763 7f869af78700 5 -- op tracker -- , seq: 1702, time: 2014-06-09 16:22:11.189825, event: throttled, op: pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457771 7f869af78700 5 -- op tracker -- , seq: 1702, time: 2014-06-09 16:22:11.189834, event: all_read, op: pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457778 7f869af78700 5 -- op tracker -- , seq: 1702, time: 2014-06-09 16:22:11.457747, event: dispatched, op: pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457785 7f869af78700 5 -- op tracker -- , seq: 1702, time: 2014-06-09 16:22:11.457783, event: waiting_for_osdmap, op: pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457790 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.457794 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305ba600 2014-06-09 16:22:11.457801 7f869af78700 5 -- op tracker -- , seq: 1702, time: 2014-06-09 16:22:11.457798, event: started, op: pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457814 7f869af78700 5 -- op tracker -- , seq: 1702, time: 2014-06-09 16:22:11.457811, event: done, op: pg_log(0.186 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457821 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457823 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457835 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457837 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457838 7f869af78700 20 osd.26 243 _dispatch 0x1609ddc0 pg_notify(0.1f(25),3.2b(12) epoch 243) v5 2014-06-09 16:22:11.457844 7f869af78700 5 -- op tracker -- , seq: 1703, time: 2014-06-09 16:22:11.017500, event: header_read, op: pg_notify(0.1f(25),3.2b(12) epoch 243) 2014-06-09 16:22:11.457851 7f869af78700 5 -- op tracker -- , seq: 1703, time: 2014-06-09 16:22:11.017500, event: throttled, op: pg_notify(0.1f(25),3.2b(12) epoch 243) 2014-06-09 16:22:11.457858 7f869af78700 5 -- op tracker -- , seq: 1703, time: 2014-06-09 16:22:11.017528, event: all_read, op: pg_notify(0.1f(25),3.2b(12) epoch 243) 2014-06-09 16:22:11.457864 7f869af78700 5 -- op tracker -- , seq: 1703, time: 2014-06-09 16:22:11.457835, event: dispatched, op: pg_notify(0.1f(25),3.2b(12) epoch 243) 2014-06-09 16:22:11.457871 7f869af78700 5 -- op tracker -- , seq: 1703, time: 2014-06-09 16:22:11.457868, event: waiting_for_osdmap, op: pg_notify(0.1f(25),3.2b(12) epoch 243) 2014-06-09 16:22:11.457875 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.457877 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1609ddc0 2014-06-09 16:22:11.457883 7f869af78700 5 -- op tracker -- , seq: 1703, time: 2014-06-09 16:22:11.457880, event: started, op: pg_notify(0.1f(25),3.2b(12) epoch 243) 2014-06-09 16:22:11.457908 7f869af78700 5 -- op tracker -- , seq: 1703, time: 2014-06-09 16:22:11.457905, event: done, op: pg_notify(0.1f(25),3.2b(12) epoch 243) 2014-06-09 16:22:11.457916 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457918 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457939 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457941 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457942 7f869af78700 20 osd.26 243 _dispatch 0x3002db00 osd_map(243..243 src has 1..243) v3 2014-06-09 16:22:11.457950 7f869af78700 3 osd.26 243 handle_osd_map epochs [243,243], i have 243, src has [1,243] 2014-06-09 16:22:11.457953 7f869af78700 10 osd.26 243 no new maps here, dropping 2014-06-09 16:22:11.457955 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457956 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.457971 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.457972 7f869af78700 20 osd.26 243 _dispatch 0x30718000 pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.457979 7f869af78700 5 -- op tracker -- , seq: 1704, time: 2014-06-09 16:22:11.222697, event: header_read, op: pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457987 7f869af78700 5 -- op tracker -- , seq: 1704, time: 2014-06-09 16:22:11.222698, event: throttled, op: pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.457994 7f869af78700 5 -- op tracker -- , seq: 1704, time: 2014-06-09 16:22:11.222709, event: all_read, op: pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458001 7f869af78700 5 -- op tracker -- , seq: 1704, time: 2014-06-09 16:22:11.457968, event: dispatched, op: pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458009 7f869af78700 5 -- op tracker -- , seq: 1704, time: 2014-06-09 16:22:11.458007, event: waiting_for_osdmap, op: pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458014 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.458018 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30718000 2014-06-09 16:22:11.458023 7f869af78700 5 -- op tracker -- , seq: 1704, time: 2014-06-09 16:22:11.458020, event: started, op: pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458036 7f869af78700 5 -- op tracker -- , seq: 1704, time: 2014-06-09 16:22:11.458033, event: done, op: pg_log(2.5d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458057 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458058 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458059 7f869af78700 20 osd.26 243 _dispatch 0x2fe71a40 pg_notify(3.f6(10) epoch 243) v5 2014-06-09 16:22:11.458064 7f869af78700 5 -- op tracker -- , seq: 1705, time: 2014-06-09 16:22:11.128274, event: header_read, op: pg_notify(3.f6(10) epoch 243) 2014-06-09 16:22:11.458071 7f869af78700 5 -- op tracker -- , seq: 1705, time: 2014-06-09 16:22:11.128275, event: throttled, op: pg_notify(3.f6(10) epoch 243) 2014-06-09 16:22:11.458077 7f869af78700 5 -- op tracker -- , seq: 1705, time: 2014-06-09 16:22:11.128297, event: all_read, op: pg_notify(3.f6(10) epoch 243) 2014-06-09 16:22:11.458084 7f869af78700 5 -- op tracker -- , seq: 1705, time: 2014-06-09 16:22:11.458056, event: dispatched, op: pg_notify(3.f6(10) epoch 243) 2014-06-09 16:22:11.458090 7f869af78700 5 -- op tracker -- , seq: 1705, time: 2014-06-09 16:22:11.458088, event: waiting_for_osdmap, op: pg_notify(3.f6(10) epoch 243) 2014-06-09 16:22:11.458094 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.458096 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe71a40 2014-06-09 16:22:11.458100 7f869af78700 5 -- op tracker -- , seq: 1705, time: 2014-06-09 16:22:11.458098, event: started, op: pg_notify(3.f6(10) epoch 243) 2014-06-09 16:22:11.458116 7f869af78700 5 -- op tracker -- , seq: 1705, time: 2014-06-09 16:22:11.458114, event: done, op: pg_notify(3.f6(10) epoch 243) 2014-06-09 16:22:11.458124 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458126 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458144 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458146 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458147 7f869af78700 20 osd.26 243 _dispatch 0x150e1180 pg_notify(0.686(21),2.3bc(21),2.3bd(21) epoch 243) v5 2014-06-09 16:22:11.458154 7f869af78700 5 -- op tracker -- , seq: 1706, time: 2014-06-09 16:22:11.103249, event: header_read, op: pg_notify(0.686(21),2.3bc(21),2.3bd(21) epoch 243) 2014-06-09 16:22:11.458162 7f869af78700 5 -- op tracker -- , seq: 1706, time: 2014-06-09 16:22:11.103250, event: throttled, op: pg_notify(0.686(21),2.3bc(21),2.3bd(21) epoch 243) 2014-06-09 16:22:11.458168 7f869af78700 5 -- op tracker -- , seq: 1706, time: 2014-06-09 16:22:11.103295, event: all_read, op: pg_notify(0.686(21),2.3bc(21),2.3bd(21) epoch 243) 2014-06-09 16:22:11.458175 7f869af78700 5 -- op tracker -- , seq: 1706, time: 2014-06-09 16:22:11.458144, event: dispatched, op: pg_notify(0.686(21),2.3bc(21),2.3bd(21) epoch 243) 2014-06-09 16:22:11.458183 7f869af78700 5 -- op tracker -- , seq: 1706, time: 2014-06-09 16:22:11.458180, event: waiting_for_osdmap, op: pg_notify(0.686(21),2.3bc(21),2.3bd(21) epoch 243) 2014-06-09 16:22:11.458187 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.458189 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e1180 2014-06-09 16:22:11.458195 7f869af78700 5 -- op tracker -- , seq: 1706, time: 2014-06-09 16:22:11.458192, event: started, op: pg_notify(0.686(21),2.3bc(21),2.3bd(21) epoch 243) 2014-06-09 16:22:11.458226 7f869af78700 5 -- op tracker -- , seq: 1706, time: 2014-06-09 16:22:11.458223, event: done, op: pg_notify(0.686(21),2.3bc(21),2.3bd(21) epoch 243) 2014-06-09 16:22:11.458240 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458242 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458267 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458268 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458269 7f869af78700 20 osd.26 243 _dispatch 0x305bb800 pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.458276 7f869af78700 5 -- op tracker -- , seq: 1707, time: 2014-06-09 16:22:11.136392, event: header_read, op: pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458284 7f869af78700 5 -- op tracker -- , seq: 1707, time: 2014-06-09 16:22:11.136393, event: throttled, op: pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458292 7f869af78700 5 -- op tracker -- , seq: 1707, time: 2014-06-09 16:22:11.136408, event: all_read, op: pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458299 7f869af78700 5 -- op tracker -- , seq: 1707, time: 2014-06-09 16:22:11.458266, event: dispatched, op: pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458307 7f869af78700 5 -- op tracker -- , seq: 1707, time: 2014-06-09 16:22:11.458304, event: waiting_for_osdmap, op: pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458312 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.458315 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305bb800 2014-06-09 16:22:11.458321 7f869af78700 5 -- op tracker -- , seq: 1707, time: 2014-06-09 16:22:11.458318, event: started, op: pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458334 7f869af78700 5 -- op tracker -- , seq: 1707, time: 2014-06-09 16:22:11.458331, event: done, op: pg_log(2.27a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458342 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458344 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458358 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458359 7f869af78700 20 osd.26 243 _dispatch 0x2fe1ec00 pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.458366 7f869af78700 5 -- op tracker -- , seq: 1708, time: 2014-06-09 16:22:11.092396, event: header_read, op: pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458374 7f869af78700 5 -- op tracker -- , seq: 1708, time: 2014-06-09 16:22:11.092396, event: throttled, op: pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458381 7f869af78700 5 -- op tracker -- , seq: 1708, time: 2014-06-09 16:22:11.092408, event: all_read, op: pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458388 7f869af78700 5 -- op tracker -- , seq: 1708, time: 2014-06-09 16:22:11.458356, event: dispatched, op: pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458396 7f869af78700 5 -- op tracker -- , seq: 1708, time: 2014-06-09 16:22:11.458393, event: waiting_for_osdmap, op: pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458401 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.458405 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe1ec00 2014-06-09 16:22:11.458410 7f869af78700 5 -- op tracker -- , seq: 1708, time: 2014-06-09 16:22:11.458408, event: started, op: pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458424 7f869af78700 5 -- op tracker -- , seq: 1708, time: 2014-06-09 16:22:11.458421, event: done, op: pg_log(1.a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458432 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458434 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458445 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458446 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458447 7f869af78700 20 osd.26 243 _dispatch 0x3046ac40 pg_notify(3.7e4(10) epoch 243) v5 2014-06-09 16:22:11.458452 7f869af78700 5 -- op tracker -- , seq: 1709, time: 2014-06-09 16:22:11.062626, event: header_read, op: pg_notify(3.7e4(10) epoch 243) 2014-06-09 16:22:11.458459 7f869af78700 5 -- op tracker -- , seq: 1709, time: 2014-06-09 16:22:11.062626, event: throttled, op: pg_notify(3.7e4(10) epoch 243) 2014-06-09 16:22:11.458466 7f869af78700 5 -- op tracker -- , seq: 1709, time: 2014-06-09 16:22:11.062641, event: all_read, op: pg_notify(3.7e4(10) epoch 243) 2014-06-09 16:22:11.458472 7f869af78700 5 -- op tracker -- , seq: 1709, time: 2014-06-09 16:22:11.458444, event: dispatched, op: pg_notify(3.7e4(10) epoch 243) 2014-06-09 16:22:11.458480 7f869af78700 5 -- op tracker -- , seq: 1709, time: 2014-06-09 16:22:11.458478, event: waiting_for_osdmap, op: pg_notify(3.7e4(10) epoch 243) 2014-06-09 16:22:11.458484 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.458486 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046ac40 2014-06-09 16:22:11.458491 7f869af78700 5 -- op tracker -- , seq: 1709, time: 2014-06-09 16:22:11.458489, event: started, op: pg_notify(3.7e4(10) epoch 243) 2014-06-09 16:22:11.458508 7f869af78700 5 -- op tracker -- , seq: 1709, time: 2014-06-09 16:22:11.458506, event: done, op: pg_notify(3.7e4(10) epoch 243) 2014-06-09 16:22:11.458516 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458517 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458534 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458536 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458537 7f869af78700 20 osd.26 243 _dispatch 0x2d7336c0 pg_info(1 pgs e243:3.d1(4)) v4 2014-06-09 16:22:11.458542 7f869af78700 5 -- op tracker -- , seq: 1710, time: 2014-06-09 16:22:11.075089, event: header_read, op: pg_info(1 pgs e243:3.d1(4)) 2014-06-09 16:22:11.458549 7f869af78700 5 -- op tracker -- , seq: 1710, time: 2014-06-09 16:22:11.075090, event: throttled, op: pg_info(1 pgs e243:3.d1(4)) 2014-06-09 16:22:11.458555 7f869af78700 5 -- op tracker -- , seq: 1710, time: 2014-06-09 16:22:11.075101, event: all_read, op: pg_info(1 pgs e243:3.d1(4)) 2014-06-09 16:22:11.458562 7f869af78700 5 -- op tracker -- , seq: 1710, time: 2014-06-09 16:22:11.458533, event: dispatched, op: pg_info(1 pgs e243:3.d1(4)) 2014-06-09 16:22:11.458568 7f869af78700 5 -- op tracker -- , seq: 1710, time: 2014-06-09 16:22:11.458566, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.d1(4)) 2014-06-09 16:22:11.458573 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.d1(4)) v4 from osd.0 2014-06-09 16:22:11.458576 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d7336c0 2014-06-09 16:22:11.458581 7f869af78700 5 -- op tracker -- , seq: 1710, time: 2014-06-09 16:22:11.458579, event: started, op: pg_info(1 pgs e243:3.d1(4)) 2014-06-09 16:22:11.458598 7f869af78700 5 -- op tracker -- , seq: 1710, time: 2014-06-09 16:22:11.458595, event: done, op: pg_info(1 pgs e243:3.d1(4)) 2014-06-09 16:22:11.458605 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458608 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458622 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458623 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458624 7f869af78700 20 osd.26 243 _dispatch 0x3088b600 pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.458631 7f869af78700 5 -- op tracker -- , seq: 1711, time: 2014-06-09 16:22:11.190828, event: header_read, op: pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458639 7f869af78700 5 -- op tracker -- , seq: 1711, time: 2014-06-09 16:22:11.190828, event: throttled, op: pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458646 7f869af78700 5 -- op tracker -- , seq: 1711, time: 2014-06-09 16:22:11.190836, event: all_read, op: pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458653 7f869af78700 5 -- op tracker -- , seq: 1711, time: 2014-06-09 16:22:11.458621, event: dispatched, op: pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458660 7f869af78700 5 -- op tracker -- , seq: 1711, time: 2014-06-09 16:22:11.458657, event: waiting_for_osdmap, op: pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458665 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.458669 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088b600 2014-06-09 16:22:11.458674 7f869af78700 5 -- op tracker -- , seq: 1711, time: 2014-06-09 16:22:11.458672, event: started, op: pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458688 7f869af78700 5 -- op tracker -- , seq: 1711, time: 2014-06-09 16:22:11.458685, event: done, op: pg_log(2.52e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458696 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458698 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458710 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458712 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458714 7f869af78700 20 osd.26 243 _dispatch 0x1abd56c0 pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) v4 2014-06-09 16:22:11.458720 7f869af78700 5 -- op tracker -- , seq: 1712, time: 2014-06-09 16:22:11.017536, event: header_read, op: pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) 2014-06-09 16:22:11.458727 7f869af78700 5 -- op tracker -- , seq: 1712, time: 2014-06-09 16:22:11.017537, event: throttled, op: pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) 2014-06-09 16:22:11.458733 7f869af78700 5 -- op tracker -- , seq: 1712, time: 2014-06-09 16:22:11.017552, event: all_read, op: pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) 2014-06-09 16:22:11.458740 7f869af78700 5 -- op tracker -- , seq: 1712, time: 2014-06-09 16:22:11.458710, event: dispatched, op: pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) 2014-06-09 16:22:11.458747 7f869af78700 5 -- op tracker -- , seq: 1712, time: 2014-06-09 16:22:11.458745, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) 2014-06-09 16:22:11.458752 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) v4 from osd.2 2014-06-09 16:22:11.458756 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd56c0 2014-06-09 16:22:11.458761 7f869af78700 5 -- op tracker -- , seq: 1712, time: 2014-06-09 16:22:11.458759, event: started, op: pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) 2014-06-09 16:22:11.458786 7f869af78700 5 -- op tracker -- , seq: 1712, time: 2014-06-09 16:22:11.458783, event: done, op: pg_info(2 pgs e243:3.7ae(4),3.7bc(4)) 2014-06-09 16:22:11.458794 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458795 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458812 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458814 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458815 7f869af78700 20 osd.26 243 _dispatch 0x302a0600 pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.458822 7f869af78700 5 -- op tracker -- , seq: 1713, time: 2014-06-09 16:22:11.010741, event: header_read, op: pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458829 7f869af78700 5 -- op tracker -- , seq: 1713, time: 2014-06-09 16:22:11.010741, event: throttled, op: pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458836 7f869af78700 5 -- op tracker -- , seq: 1713, time: 2014-06-09 16:22:11.010757, event: all_read, op: pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458843 7f869af78700 5 -- op tracker -- , seq: 1713, time: 2014-06-09 16:22:11.458812, event: dispatched, op: pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458850 7f869af78700 5 -- op tracker -- , seq: 1713, time: 2014-06-09 16:22:11.458847, event: waiting_for_osdmap, op: pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458855 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.458859 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x302a0600 2014-06-09 16:22:11.458864 7f869af78700 5 -- op tracker -- , seq: 1713, time: 2014-06-09 16:22:11.458862, event: started, op: pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458877 7f869af78700 5 -- op tracker -- , seq: 1713, time: 2014-06-09 16:22:11.458874, event: done, op: pg_log(1.302 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458885 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458887 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458899 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458900 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458901 7f869af78700 20 osd.26 243 _dispatch 0x305b2000 pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.458907 7f869af78700 5 -- op tracker -- , seq: 1714, time: 2014-06-09 16:22:11.222728, event: header_read, op: pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458915 7f869af78700 5 -- op tracker -- , seq: 1714, time: 2014-06-09 16:22:11.222728, event: throttled, op: pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458922 7f869af78700 5 -- op tracker -- , seq: 1714, time: 2014-06-09 16:22:11.222748, event: all_read, op: pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458929 7f869af78700 5 -- op tracker -- , seq: 1714, time: 2014-06-09 16:22:11.458898, event: dispatched, op: pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458937 7f869af78700 5 -- op tracker -- , seq: 1714, time: 2014-06-09 16:22:11.458934, event: waiting_for_osdmap, op: pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458941 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.458946 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b2000 2014-06-09 16:22:11.458952 7f869af78700 5 -- op tracker -- , seq: 1714, time: 2014-06-09 16:22:11.458949, event: started, op: pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458965 7f869af78700 5 -- op tracker -- , seq: 1714, time: 2014-06-09 16:22:11.458962, event: done, op: pg_log(0.129 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.458972 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458974 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458987 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.458988 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.458989 7f869af78700 20 osd.26 243 _dispatch 0x30641200 pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.458996 7f869af78700 5 -- op tracker -- , seq: 1715, time: 2014-06-09 16:22:11.169471, event: header_read, op: pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459004 7f869af78700 5 -- op tracker -- , seq: 1715, time: 2014-06-09 16:22:11.169472, event: throttled, op: pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459011 7f869af78700 5 -- op tracker -- , seq: 1715, time: 2014-06-09 16:22:11.169484, event: all_read, op: pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459018 7f869af78700 5 -- op tracker -- , seq: 1715, time: 2014-06-09 16:22:11.458986, event: dispatched, op: pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459025 7f869af78700 5 -- op tracker -- , seq: 1715, time: 2014-06-09 16:22:11.459023, event: waiting_for_osdmap, op: pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459030 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.459034 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30641200 2014-06-09 16:22:11.459040 7f869af78700 5 -- op tracker -- , seq: 1715, time: 2014-06-09 16:22:11.459037, event: started, op: pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459053 7f869af78700 5 -- op tracker -- , seq: 1715, time: 2014-06-09 16:22:11.459050, event: done, op: pg_log(0.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459060 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459062 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459073 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459075 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459076 7f869af78700 20 osd.26 243 _dispatch 0x220121c0 pg_info(1 pgs e243:3.f7(4)) v4 2014-06-09 16:22:11.459081 7f869af78700 5 -- op tracker -- , seq: 1716, time: 2014-06-09 16:22:11.103302, event: header_read, op: pg_info(1 pgs e243:3.f7(4)) 2014-06-09 16:22:11.459088 7f869af78700 5 -- op tracker -- , seq: 1716, time: 2014-06-09 16:22:11.103303, event: throttled, op: pg_info(1 pgs e243:3.f7(4)) 2014-06-09 16:22:11.459094 7f869af78700 5 -- op tracker -- , seq: 1716, time: 2014-06-09 16:22:11.103312, event: all_read, op: pg_info(1 pgs e243:3.f7(4)) 2014-06-09 16:22:11.459101 7f869af78700 5 -- op tracker -- , seq: 1716, time: 2014-06-09 16:22:11.459073, event: dispatched, op: pg_info(1 pgs e243:3.f7(4)) 2014-06-09 16:22:11.459107 7f869af78700 5 -- op tracker -- , seq: 1716, time: 2014-06-09 16:22:11.459105, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.f7(4)) 2014-06-09 16:22:11.459112 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.f7(4)) v4 from osd.1 2014-06-09 16:22:11.459115 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x220121c0 2014-06-09 16:22:11.459121 7f869af78700 5 -- op tracker -- , seq: 1716, time: 2014-06-09 16:22:11.459118, event: started, op: pg_info(1 pgs e243:3.f7(4)) 2014-06-09 16:22:11.459138 7f869af78700 5 -- op tracker -- , seq: 1716, time: 2014-06-09 16:22:11.459135, event: done, op: pg_info(1 pgs e243:3.f7(4)) 2014-06-09 16:22:11.459150 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459152 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459166 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459168 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459169 7f869af78700 20 osd.26 243 _dispatch 0x305b3e00 pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.459176 7f869af78700 5 -- op tracker -- , seq: 1717, time: 2014-06-09 16:22:11.160989, event: header_read, op: pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459184 7f869af78700 5 -- op tracker -- , seq: 1717, time: 2014-06-09 16:22:11.160989, event: throttled, op: pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459191 7f869af78700 5 -- op tracker -- , seq: 1717, time: 2014-06-09 16:22:11.160999, event: all_read, op: pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459198 7f869af78700 5 -- op tracker -- , seq: 1717, time: 2014-06-09 16:22:11.459166, event: dispatched, op: pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459206 7f869af78700 5 -- op tracker -- , seq: 1717, time: 2014-06-09 16:22:11.459203, event: waiting_for_osdmap, op: pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459210 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.459214 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b3e00 2014-06-09 16:22:11.459220 7f869af78700 5 -- op tracker -- , seq: 1717, time: 2014-06-09 16:22:11.459217, event: started, op: pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459234 7f869af78700 5 -- op tracker -- , seq: 1717, time: 2014-06-09 16:22:11.459231, event: done, op: pg_log(2.55b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459241 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459243 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459255 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459257 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459258 7f869af78700 20 osd.26 243 _dispatch 0x2ea5b500 pg_notify(1.b9(23) epoch 243) v5 2014-06-09 16:22:11.459263 7f869af78700 5 -- op tracker -- , seq: 1718, time: 2014-06-09 16:22:11.096416, event: header_read, op: pg_notify(1.b9(23) epoch 243) 2014-06-09 16:22:11.459270 7f869af78700 5 -- op tracker -- , seq: 1718, time: 2014-06-09 16:22:11.096417, event: throttled, op: pg_notify(1.b9(23) epoch 243) 2014-06-09 16:22:11.459277 7f869af78700 5 -- op tracker -- , seq: 1718, time: 2014-06-09 16:22:11.096440, event: all_read, op: pg_notify(1.b9(23) epoch 243) 2014-06-09 16:22:11.459283 7f869af78700 5 -- op tracker -- , seq: 1718, time: 2014-06-09 16:22:11.459255, event: dispatched, op: pg_notify(1.b9(23) epoch 243) 2014-06-09 16:22:11.459290 7f869af78700 5 -- op tracker -- , seq: 1718, time: 2014-06-09 16:22:11.459288, event: waiting_for_osdmap, op: pg_notify(1.b9(23) epoch 243) 2014-06-09 16:22:11.459295 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.459297 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5b500 2014-06-09 16:22:11.459302 7f869af78700 5 -- op tracker -- , seq: 1718, time: 2014-06-09 16:22:11.459300, event: started, op: pg_notify(1.b9(23) epoch 243) 2014-06-09 16:22:11.459317 7f869af78700 5 -- op tracker -- , seq: 1718, time: 2014-06-09 16:22:11.459315, event: done, op: pg_notify(1.b9(23) epoch 243) 2014-06-09 16:22:11.459325 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459327 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459349 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459350 7f869af78700 20 osd.26 243 _dispatch 0x304f7800 pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.459357 7f869af78700 5 -- op tracker -- , seq: 1719, time: 2014-06-09 16:22:11.064952, event: header_read, op: pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459365 7f869af78700 5 -- op tracker -- , seq: 1719, time: 2014-06-09 16:22:11.064953, event: throttled, op: pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459372 7f869af78700 5 -- op tracker -- , seq: 1719, time: 2014-06-09 16:22:11.064962, event: all_read, op: pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459379 7f869af78700 5 -- op tracker -- , seq: 1719, time: 2014-06-09 16:22:11.459347, event: dispatched, op: pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459387 7f869af78700 5 -- op tracker -- , seq: 1719, time: 2014-06-09 16:22:11.459385, event: waiting_for_osdmap, op: pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459392 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.459396 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f7800 2014-06-09 16:22:11.459402 7f869af78700 5 -- op tracker -- , seq: 1719, time: 2014-06-09 16:22:11.459399, event: started, op: pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459416 7f869af78700 5 -- op tracker -- , seq: 1719, time: 2014-06-09 16:22:11.459413, event: done, op: pg_log(1.384 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459424 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459426 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459437 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459439 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459440 7f869af78700 20 osd.26 243 _dispatch 0x304f6c00 pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.459446 7f869af78700 5 -- op tracker -- , seq: 1720, time: 2014-06-09 16:22:11.075112, event: header_read, op: pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459454 7f869af78700 5 -- op tracker -- , seq: 1720, time: 2014-06-09 16:22:11.075113, event: throttled, op: pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459461 7f869af78700 5 -- op tracker -- , seq: 1720, time: 2014-06-09 16:22:11.075122, event: all_read, op: pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459467 7f869af78700 5 -- op tracker -- , seq: 1720, time: 2014-06-09 16:22:11.459437, event: dispatched, op: pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459475 7f869af78700 5 -- op tracker -- , seq: 1720, time: 2014-06-09 16:22:11.459472, event: waiting_for_osdmap, op: pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459479 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.459484 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f6c00 2014-06-09 16:22:11.459489 7f869af78700 5 -- op tracker -- , seq: 1720, time: 2014-06-09 16:22:11.459487, event: started, op: pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459503 7f869af78700 5 -- op tracker -- , seq: 1720, time: 2014-06-09 16:22:11.459500, event: done, op: pg_log(1.3bc epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459511 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459526 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459527 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459529 7f869af78700 20 osd.26 243 _dispatch 0x3088b000 pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.459535 7f869af78700 5 -- op tracker -- , seq: 1721, time: 2014-06-09 16:22:11.190901, event: header_read, op: pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459543 7f869af78700 5 -- op tracker -- , seq: 1721, time: 2014-06-09 16:22:11.190901, event: throttled, op: pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459550 7f869af78700 5 -- op tracker -- , seq: 1721, time: 2014-06-09 16:22:11.190909, event: all_read, op: pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459557 7f869af78700 5 -- op tracker -- , seq: 1721, time: 2014-06-09 16:22:11.459526, event: dispatched, op: pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459565 7f869af78700 5 -- op tracker -- , seq: 1721, time: 2014-06-09 16:22:11.459562, event: waiting_for_osdmap, op: pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459569 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.459574 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088b000 2014-06-09 16:22:11.459579 7f869af78700 5 -- op tracker -- , seq: 1721, time: 2014-06-09 16:22:11.459577, event: started, op: pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459594 7f869af78700 5 -- op tracker -- , seq: 1721, time: 2014-06-09 16:22:11.459591, event: done, op: pg_log(1.286 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.459601 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.459603 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.459633 7f8691765700 5 -- op tracker -- , seq: 386, time: 2014-06-09 16:22:11.441533, event: reached_pg, op: osd_op(client.4570.0:33861 rbd_data.117a6b8b4567.00000000000016c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2752512~1441792] 3.2b068cb3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.459646 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (12 active) 2014-06-09 16:22:11.459646 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.459655 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.459663 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0d6300 2014-06-09 16:22:11.459674 7f8691765700 10 osd.26 243 dequeue_op 0x1b0d6300 finish 2014-06-09 16:22:11.459669 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.382( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.459679 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (11 active) 2014-06-09 16:22:11.459682 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.459686 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.459696 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0230 (12 active) 2014-06-09 16:22:11.459691 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.382( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.210591 7 0.000205 2014-06-09 16:22:11.459702 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.382( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.459700 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:1.62a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.459713 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.382( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.459714 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.459728 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.459724 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.459734 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.459734 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.7f( v 214'533 (0'0,214'533] local-les=208 n=106 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.459744 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.459747 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.459758 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.459762 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.459772 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.681( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.459784 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.459788 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.459792 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.459798 7f868f761700 5 queue_transactions existing osr(1.382 0x384d5b0)/0x384d5b0 2014-06-09 16:22:11.459792 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.681( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.754104 7 0.000178 2014-06-09 16:22:11.427607 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.459803 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.681( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.459814 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.681( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.459811 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.459824 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.459824 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.459834 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.459835 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.459844 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.459883 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.459888 7f868ef60700 5 queue_transactions existing osr(2.3c0 0x2ed13f0)/0x2ed13f0 2014-06-09 16:22:11.459894 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.459899 7f8691765700 5 queue_transactions existing osr(0.681 0x2bd7e10)/0x2bd7e10 2014-06-09 16:22:11.459905 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02f0 (13 active) 2014-06-09 16:22:11.459913 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:2.3bd( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.459927 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.459937 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.459942 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.459953 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.27a( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.459964 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.459968 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.459971 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 0.985837 7 0.000179 2014-06-09 16:22:11.459982 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.459993 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.460002 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.460012 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.460020 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.460071 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.460076 7f868b759700 5 queue_transactions existing osr(2.27a 0x2ed0d90)/0x2ed0d90 2014-06-09 16:22:11.430404 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.460104 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.460123 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd70240] 2014-06-09 16:22:11.460125 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.460127 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.460133 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.430412 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34970e0 (1 active) 2014-06-09 16:22:11.460145 7f86a5610700 5 _do_op 0x306075c0 seq 3132 osr(3.67d 0x3ed1d70)/0x3ed1d70 start 2014-06-09 16:22:11.460151 7f86a5610700 10 _do_transaction on 0xc2dfe00 2014-06-09 16:22:11.460159 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.460718 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.460742 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3132.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.460748 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.460793 7f86a5610700 10 _do_op 0x306075c0 seq 3132 r = 0, finisher 0xbe67e80 0 2014-06-09 16:22:11.460802 7f86a5610700 10 _finish_op 0x306075c0 seq 3132 osr(3.67d 0x3ed1d70)/0x3ed1d70 2014-06-09 16:22:11.460807 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34970e0 (0 active) 2014-06-09 16:22:11.460810 7f86a5610700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.460860 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.429224 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.460875 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.460889 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 1.589( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.460901 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.460905 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.460908 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.589( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.728768 7 0.000215 2014-06-09 16:22:11.460919 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.589( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.460929 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.589( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.460939 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.460949 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.460958 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.461008 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.461014 7f8690763700 5 queue_transactions existing osr(1.589 0x384da10)/0x384da10 2014-06-09 16:22:11.461024 7f868cf5c700 10 op_submit_finish 3133 2014-06-09 16:22:11.461030 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.461041 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.620( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.461054 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.461058 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.461061 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.620( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.224040 7 0.000197 2014-06-09 16:22:11.461072 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.620( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.461083 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.620( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.461093 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.461101 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.461110 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.461158 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.461163 7f868cf5c700 5 queue_transactions existing osr(2.620 0x2ed1cb0)/0x2ed1cb0 2014-06-09 16:22:11.461165 7f868cf5c700 10 op_submit_start 3134 2014-06-09 16:22:11.461167 7f868cf5c700 5 queue_transactions (trailing journal) 3134 2014-06-09 16:22:11.461169 7f868cf5c700 5 queue_op 0x30033620 seq 3134 osr(2.620 0x2ed1cb0) 2001 bytes (queue has 10 ops and 18689 bytes) 2014-06-09 16:22:11.461174 7f868cf5c700 10 op_submit_finish 3134 2014-06-09 16:22:11.461177 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:1.1f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.461192 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.461202 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.461207 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:0.19a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.461219 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.461228 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.461232 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:2.2f7( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.461244 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.461253 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.461262 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (12 active) 2014-06-09 16:22:11.461265 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461272 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461278 7f868a757700 10 op_submit_start 3135 2014-06-09 16:22:11.461280 7f868a757700 5 queue_transactions (trailing journal) 3135 2014-06-09 16:22:11.461281 7f868a757700 5 queue_op 0x30061a40 seq 3135 osr(0.2c3 0x2bd6ba0) 1903 bytes (queue has 10 ops and 18689 bytes) 2014-06-09 16:22:11.461285 7f868a757700 10 op_submit_finish 3135 2014-06-09 16:22:11.461290 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.d3( v 214'647 (0'0,214'647] local-les=201 n=126 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.461307 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.461318 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.461323 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.66f( v 214'585 (0'0,214'585] local-les=201 n=116 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.461336 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.461346 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.461350 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:2.3de( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.461363 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.461373 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.461377 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.3dc( v 213'619 (0'0,213'619] local-les=243 n=117 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.3dc( v 213'619 (0'0,213'619] local-les=191 n=117 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.461390 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.3dc( v 213'619 (0'0,213'619] local-les=243 n=117 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.461402 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.461411 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (11 active) 2014-06-09 16:22:11.461414 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461420 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461427 7f8691f66700 10 op_submit_start 3136 2014-06-09 16:22:11.461428 7f8691f66700 5 queue_transactions (trailing journal) 3136 2014-06-09 16:22:11.461430 7f8691f66700 5 queue_op 0x308cbb60 seq 3136 osr(0.33f 0x2bd7060) 2001 bytes (queue has 10 ops and 18689 bytes) 2014-06-09 16:22:11.461434 7f8691f66700 10 op_submit_finish 3136 2014-06-09 16:22:11.461440 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.461453 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.461466 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.238840 7 0.000167 2014-06-09 16:22:11.461478 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.461491 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.461501 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.461512 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'583 -> 213'583 2014-06-09 16:22:11.461551 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.461594 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.461599 7f8691f66700 5 queue_transactions existing osr(3.31f 0x3ed0d50)/0x3ed0d50 2014-06-09 16:22:11.461602 7f8691f66700 10 op_submit_start 3137 2014-06-09 16:22:11.461603 7f8691f66700 5 queue_transactions (trailing journal) 3137 2014-06-09 16:22:11.461605 7f8691f66700 5 queue_op 0x308cbb00 seq 3137 osr(3.31f 0x3ed0d50) 1335 bytes (queue has 11 ops and 20024 bytes) 2014-06-09 16:22:11.461607 7f8691f66700 10 op_submit_finish 3137 2014-06-09 16:22:11.461611 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:0.6e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.461625 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.461635 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.461639 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.461650 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.5b0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.461662 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.461667 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.461670 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.761676 7 0.000180 2014-06-09 16:22:11.461681 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.461691 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.461700 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.461709 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.461717 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.461764 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.461768 7f8691f66700 5 queue_transactions existing osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 2014-06-09 16:22:11.461771 7f8691f66700 10 op_submit_start 3138 2014-06-09 16:22:11.461772 7f8691f66700 5 queue_transactions (trailing journal) 3138 2014-06-09 16:22:11.461774 7f8691f66700 5 queue_op 0x308cbaa0 seq 3138 osr(0.5b0 0x2bd7ab0) 1974 bytes (queue has 12 ops and 21998 bytes) 2014-06-09 16:22:11.461777 7f8691f66700 10 op_submit_finish 3138 2014-06-09 16:22:11.461784 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (10 active) 2014-06-09 16:22:11.461787 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461793 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461800 7f8690f64700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461807 7f8692767700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461812 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbe67e80] 2014-06-09 16:22:11.461817 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbe67e80] 2014-06-09 16:22:11.461818 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.461820 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.446457 7f868ff62700 10 osd.26 243 dequeue_op 0x1b0b7100 prio 63 cost 516096 latency 0.917754 osd_op(client.4574.0:31828 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2367488~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.461870 7f868ff62700 5 -- op tracker -- , seq: 477, time: 2014-06-09 16:22:11.461860, event: reached_pg, op: osd_op(client.4574.0:31828 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2367488~516096] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.461881 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.461899 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1b0b7100 2014-06-09 16:22:11.461910 7f868ff62700 10 osd.26 243 dequeue_op 0x1b0b7100 finish 2014-06-09 16:22:11.461917 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (9 active) 2014-06-09 16:22:11.461920 7f868ff62700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.461928 7f8692f68700 10 osd.26 243 dequeue_op 0x1b0b7000 prio 63 cost 1032192 latency 0.932240 osd_op(client.4574.0:31829 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2883584~1032192] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.461954 7f8692f68700 5 -- op tracker -- , seq: 479, time: 2014-06-09 16:22:11.461948, event: reached_pg, op: osd_op(client.4574.0:31829 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2883584~1032192] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.461961 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.461976 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1b0b7000 2014-06-09 16:22:11.461988 7f8692f68700 10 osd.26 243 dequeue_op 0x1b0b7000 finish 2014-06-09 16:22:11.461994 7f8692f68700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (8 active) 2014-06-09 16:22:11.461997 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.462004 7f868d75d700 10 osd.26 243 dequeue_op 0x1b0b6f00 prio 63 cost 278528 latency 0.930478 osd_op(client.4574.0:31830 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3915776~278528] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] 2014-06-09 16:22:11.462028 7f868d75d700 5 -- op tracker -- , seq: 492, time: 2014-06-09 16:22:11.462022, event: reached_pg, op: osd_op(client.4574.0:31830 rbd_data.117e6b8b4567.00000000000014d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3915776~278528] 3.e0fbd3b4 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.462035 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.462049 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] can_handle_while_inactive: 0x1b0b6f00 2014-06-09 16:22:11.462060 7f868d75d700 10 osd.26 243 dequeue_op 0x1b0b6f00 finish 2014-06-09 16:22:11.462064 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (7 active) 2014-06-09 16:22:11.462066 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.446062 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=201 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.462074 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=201 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.999447 7 0.000152 2014-06-09 16:22:11.462089 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=201 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.462101 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=201 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.462113 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=201 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.462123 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'584 -> 213'584 2014-06-09 16:22:11.462161 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.462205 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.462210 7f868e75f700 5 queue_transactions existing osr(3.3a9 0x3ed0f60)/0x3ed0f60 2014-06-09 16:22:11.462213 7f868e75f700 10 op_submit_start 3139 2014-06-09 16:22:11.462214 7f868e75f700 5 queue_transactions (trailing journal) 3139 2014-06-09 16:22:11.462215 7f868e75f700 5 queue_op 0x2fc8cae0 seq 3139 osr(3.3a9 0x3ed0f60) 1335 bytes (queue has 13 ops and 23333 bytes) 2014-06-09 16:22:11.462218 7f868e75f700 10 op_submit_finish 3139 2014-06-09 16:22:11.462221 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.462232 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 0.dd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.462243 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.462247 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.462250 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.183860 7 0.000194 2014-06-09 16:22:11.462261 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.462272 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.462281 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.462290 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.462299 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.462346 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.462352 7f868e75f700 5 queue_transactions existing osr(0.dd 0x2bd6550)/0x2bd6550 2014-06-09 16:22:11.462354 7f868e75f700 10 op_submit_start 3140 2014-06-09 16:22:11.462355 7f868e75f700 5 queue_transactions (trailing journal) 3140 2014-06-09 16:22:11.462356 7f868e75f700 5 queue_op 0x2fc8ca80 seq 3140 osr(0.dd 0x2bd6550) 1880 bytes (queue has 14 ops and 25213 bytes) 2014-06-09 16:22:11.462359 7f868e75f700 10 op_submit_finish 3140 2014-06-09 16:22:11.462362 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.462376 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.9 3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.462389 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.282236 7 0.000182 2014-06-09 16:22:11.462401 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.462412 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.462424 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.462434 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'658 -> 213'658 2014-06-09 16:22:11.462478 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.462518 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.462524 7f868e75f700 5 queue_transactions existing osr(3.609 0x3ed1b00)/0x3ed1b00 2014-06-09 16:22:11.462527 7f868e75f700 10 op_submit_start 3141 2014-06-09 16:22:11.462528 7f868e75f700 5 queue_transactions (trailing journal) 3141 2014-06-09 16:22:11.462529 7f868e75f700 5 queue_op 0x2fc8ca20 seq 3141 osr(3.609 0x3ed1b00) 1335 bytes (queue has 15 ops and 26548 bytes) 2014-06-09 16:22:11.462531 7f868e75f700 10 op_submit_finish 3141 2014-06-09 16:22:11.462534 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=212 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.462548 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=212 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.462561 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=212 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.188983 7 0.000179 2014-06-09 16:22:11.462572 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=212 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.462584 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=212 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.462594 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=212 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.462604 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'536 -> 213'536 2014-06-09 16:22:11.462638 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.462680 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.462684 7f868e75f700 5 queue_transactions existing osr(3.76 0x3ed0240)/0x3ed0240 2014-06-09 16:22:11.462687 7f868e75f700 10 op_submit_start 3142 2014-06-09 16:22:11.462688 7f868e75f700 5 queue_transactions (trailing journal) 3142 2014-06-09 16:22:11.462689 7f868e75f700 5 queue_op 0x2fc8c9c0 seq 3142 osr(3.76 0x3ed0240) 1332 bytes (queue has 16 ops and 27880 bytes) 2014-06-09 16:22:11.462692 7f868e75f700 10 op_submit_finish 3142 2014-06-09 16:22:11.462695 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.462706 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 2.6f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.462718 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.462722 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.462726 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.272528 7 0.000203 2014-06-09 16:22:11.462737 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.462747 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.462757 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.462767 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.462774 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.462821 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.462826 7f868e75f700 5 queue_transactions existing osr(2.6f 0x2ed0550)/0x2ed0550 2014-06-09 16:22:11.462829 7f868e75f700 10 op_submit_start 3143 2014-06-09 16:22:11.462830 7f868e75f700 5 queue_transactions (trailing journal) 3143 2014-06-09 16:22:11.462831 7f868e75f700 5 queue_op 0x2fc8c960 seq 3143 osr(2.6f 0x2ed0550) 2006 bytes (queue has 17 ops and 29886 bytes) 2014-06-09 16:22:11.462833 7f868e75f700 10 op_submit_finish 3143 2014-06-09 16:22:11.462836 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=191 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) 2014-06-09 16:22:11.462849 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=191 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.0 3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) 2014-06-09 16:22:11.462861 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=191 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.272221 7 0.000161 2014-06-09 16:22:11.462872 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=191 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.462884 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=191 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.462895 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=191 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.462906 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'555 -> 213'555 2014-06-09 16:22:11.462943 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.462986 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.462992 7f868e75f700 5 queue_transactions existing osr(3.603 0x3ed1af0)/0x3ed1af0 2014-06-09 16:22:11.462994 7f868e75f700 10 op_submit_start 3144 2014-06-09 16:22:11.462995 7f868e75f700 5 queue_transactions (trailing journal) 3144 2014-06-09 16:22:11.462996 7f868e75f700 5 queue_op 0x2fc8c900 seq 3144 osr(3.603 0x3ed1af0) 1429 bytes (queue has 18 ops and 31315 bytes) 2014-06-09 16:22:11.462999 7f868e75f700 10 op_submit_finish 3144 2014-06-09 16:22:11.463002 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.463014 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 1.2d0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.463026 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.463030 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.463033 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.744283 7 0.000210 2014-06-09 16:22:11.463044 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.463054 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.463064 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.463073 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.463080 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.463129 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.463134 7f868e75f700 5 queue_transactions existing osr(1.2d0 0x384d180)/0x384d180 2014-06-09 16:22:11.463137 7f868e75f700 10 op_submit_start 3145 2014-06-09 16:22:11.463138 7f868e75f700 5 queue_transactions (trailing journal) 3145 2014-06-09 16:22:11.463140 7f868e75f700 5 queue_op 0x2fc8c8a0 seq 3145 osr(1.2d0 0x384d180) 2009 bytes (queue has 19 ops and 33324 bytes) 2014-06-09 16:22:11.463143 7f868e75f700 10 op_submit_finish 3145 2014-06-09 16:22:11.463145 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MLogRec from 2 2014-06-09 16:22:11.463161 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.74( v 213'545 (0'0,213'545] local-les=191 n=109 ec=118 les/c 191/191 242/242/242) log((0'0,213'545], crt=0'0) missing(0) 2014-06-09 16:22:11.463175 7f868e75f700 10 proc_replica_log for osd.2: 3.74( v 213'545 (0'0,213'545] local-les=191 n=109 ec=118 les/c 191/191 242/242/242) log((0'0,213'545], crt=0'0) missing(0) 2014-06-09 16:22:11.463180 7f868e75f700 20 merge_log cut point (usually last shared) is 213'545 (213'544) modify d6e54874/rbd_data.116a6b8b4567.0000000000001626/head//3 by client.4554.0:32764 2014-06-09 10:17:11.220059 2014-06-09 16:22:11.463627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd671e0] 2014-06-09 16:22:11.463633 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.463651 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.463667 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd671e0] 2014-06-09 16:22:11.463668 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.463670 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.463675 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287b00 (1 active) 2014-06-09 16:22:11.463665 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.74( v 213'545 (0'0,213'545] local-les=191 n=109 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.463678 7f86a5e11700 5 _do_op 0x30033680 seq 3133 osr(0.35b 0x2bd6fe0)/0x2bd6fe0 start 2014-06-09 16:22:11.463682 7f86a5e11700 10 _do_transaction on 0xc10a900 2014-06-09 16:22:11.463682 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.463690 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.463694 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.541983 6 0.000288 2014-06-09 16:22:11.463705 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.463713 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.141537 0 0.000000 2014-06-09 16:22:11.463726 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.463735 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=191 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.463746 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.463748 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.463757 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.463766 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3133.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.463769 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.463765 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'550 -> 214'550 2014-06-09 16:22:11.463794 7f86a5e11700 10 _do_op 0x30033680 seq 3133 r = 0, finisher 0xbd66980 0 2014-06-09 16:22:11.463800 7f86a5e11700 10 _finish_op 0x30033680 seq 3133 osr(0.35b 0x2bd6fe0)/0x2bd6fe0 2014-06-09 16:22:11.463805 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287b00 (0 active) 2014-06-09 16:22:11.463807 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f9e0 (1 active) 2014-06-09 16:22:11.463809 7f86a5e11700 5 _do_op 0x30033620 seq 3134 osr(2.620 0x2ed1cb0)/0x2ed1cb0 start 2014-06-09 16:22:11.463812 7f86a5e11700 10 _do_transaction on 0xc2db980 2014-06-09 16:22:11.463805 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.74( v 213'545 (0'0,213'545] local-les=191 n=109 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.463817 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.463828 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'545,214'550], crt=0'0) 2014-06-09 16:22:11.463841 7f868e75f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.463848 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.463864 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3134.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.463855 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.74( v 214'550 lc 213'545 (0'0,214'550] local-les=191 n=109 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.463867 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.463869 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:11.463878 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 833cb874/rbd_data.11766b8b4567.00000000000016b5/head//3 214'550 is on osd.26 2014-06-09 16:22:11.463889 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {833cb874/rbd_data.11766b8b4567.00000000000016b5/head//3=214'550} 2014-06-09 16:22:11.463898 7f86a5e11700 10 _do_op 0x30033620 seq 3134 r = 0, finisher 0xbe302e0 0 2014-06-09 16:22:11.463901 7f86a5e11700 10 _finish_op 0x30033620 seq 3134 osr(2.620 0x2ed1cb0)/0x2ed1cb0 2014-06-09 16:22:11.463904 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f9e0 (0 active) 2014-06-09 16:22:11.463899 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.463906 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6fc0 (1 active) 2014-06-09 16:22:11.463908 7f86a5e11700 5 _do_op 0x30061a40 seq 3135 osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 start 2014-06-09 16:22:11.463910 7f86a5e11700 10 _do_transaction on 0xc1312c0 2014-06-09 16:22:11.463907 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 833cb874/rbd_data.11766b8b4567.00000000000016b5/head//3 214'550 also missing on osd.2 2014-06-09 16:22:11.463916 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.463917 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {833cb874/rbd_data.11766b8b4567.00000000000016b5/head//3=214'550} 2014-06-09 16:22:11.463926 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.463935 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.463947 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.463945 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.463954 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.463962 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3135.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.463966 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.463963 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.463982 7f86a5e11700 10 _do_op 0x30061a40 seq 3135 r = 0, finisher 0xbd72f20 0 2014-06-09 16:22:11.463986 7f86a5e11700 10 _finish_op 0x30061a40 seq 3135 osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 2014-06-09 16:22:11.463988 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6fc0 (0 active) 2014-06-09 16:22:11.463990 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2865a0 (1 active) 2014-06-09 16:22:11.463992 7f86a5e11700 5 _do_op 0x308cbb60 seq 3136 osr(0.33f 0x2bd7060)/0x2bd7060 start 2014-06-09 16:22:11.463994 7f86a5e11700 10 _do_transaction on 0xbf29680 2014-06-09 16:22:11.463999 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.464030 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.464045 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3136.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.464049 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.464054 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.464059 7f868e75f700 5 queue_transactions existing osr(3.74 0x3ed0290)/0x3ed0290 2014-06-09 16:22:11.464062 7f868e75f700 10 op_submit_start 3146 2014-06-09 16:22:11.464062 7f868e75f700 5 queue_transactions (trailing journal) 3146 2014-06-09 16:22:11.464064 7f868e75f700 5 queue_op 0x2fcae240 seq 3146 osr(3.74 0x3ed0290) 1050 bytes (queue has 17 ops and 28587 bytes) 2014-06-09 16:22:11.464066 7f868e75f700 10 op_submit_finish 3146 2014-06-09 16:22:11.464069 7f86a5e11700 10 _do_op 0x308cbb60 seq 3136 r = 0, finisher 0x3050aa80 0 2014-06-09 16:22:11.464071 7f86a5e11700 10 _finish_op 0x308cbb60 seq 3136 osr(0.33f 0x2bd7060)/0x2bd7060 2014-06-09 16:22:11.464074 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2865a0 (0 active) 2014-06-09 16:22:11.464076 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358360 (1 active) 2014-06-09 16:22:11.464077 7f86a5e11700 5 _do_op 0x308cbb00 seq 3137 osr(3.31f 0x3ed0d50)/0x3ed0d50 start 2014-06-09 16:22:11.464070 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:2.1ae( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.464080 7f86a5e11700 10 _do_transaction on 0x32fa6c0 2014-06-09 16:22:11.464085 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.464083 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.464093 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.464097 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.21b( v 214'612 (0'0,214'612] local-les=210 n=123 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.464115 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.464116 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.464131 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3137.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.464135 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.464143 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.464153 7f86a5e11700 10 _do_op 0x308cbb00 seq 3137 r = 0, finisher 0xbfcffa0 0 2014-06-09 16:22:11.464148 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.464156 7f86a5e11700 10 _finish_op 0x308cbb00 seq 3137 osr(3.31f 0x3ed0d50)/0x3ed0d50 2014-06-09 16:22:11.464159 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358360 (0 active) 2014-06-09 16:22:11.464161 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6ddb00 (1 active) 2014-06-09 16:22:11.464163 7f86a5e11700 5 _do_op 0x308cbaa0 seq 3138 osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 start 2014-06-09 16:22:11.464165 7f86a5e11700 10 _do_transaction on 0x3339bc0 2014-06-09 16:22:11.464159 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 0.3b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.464170 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.464170 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.464173 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.464177 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.117107 7 0.009210 2014-06-09 16:22:11.464187 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.464202 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.464197 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.464207 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.464217 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3138.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.464220 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.464216 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.464223 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.464243 7f86a5e11700 10 _do_op 0x308cbaa0 seq 3138 r = 0, finisher 0x189aa340 0 2014-06-09 16:22:11.464246 7f86a5e11700 10 _finish_op 0x308cbaa0 seq 3138 osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 2014-06-09 16:22:11.464248 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6ddb00 (0 active) 2014-06-09 16:22:11.464250 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e360 (1 active) 2014-06-09 16:22:11.464252 7f86a5e11700 5 _do_op 0x2fc8cae0 seq 3139 osr(3.3a9 0x3ed0f60)/0x3ed0f60 start 2014-06-09 16:22:11.464254 7f86a5e11700 10 _do_transaction on 0xc2a09c0 2014-06-09 16:22:11.464259 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.464269 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.464273 7f868e75f700 5 queue_transactions existing osr(0.3b6 0x2bd72d0)/0x2bd72d0 2014-06-09 16:22:11.464276 7f868e75f700 10 op_submit_start 3147 2014-06-09 16:22:11.464277 7f868e75f700 5 queue_transactions (trailing journal) 3147 2014-06-09 16:22:11.464278 7f868e75f700 5 queue_op 0x2fcae0c0 seq 3147 osr(0.3b6 0x2bd72d0) 2009 bytes (queue has 15 ops and 25286 bytes) 2014-06-09 16:22:11.464281 7f868e75f700 10 op_submit_finish 3147 2014-06-09 16:22:11.464289 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.464284 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:2.3a6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.464296 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.464306 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.464304 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3139.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.464308 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.464310 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.464323 7f86a5e11700 10 _do_op 0x2fc8cae0 seq 3139 r = 0, finisher 0x15a3e8a0 0 2014-06-09 16:22:11.464326 7f86a5e11700 10 _finish_op 0x2fc8cae0 seq 3139 osr(3.3a9 0x3ed0f60)/0x3ed0f60 2014-06-09 16:22:11.464319 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.464331 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.472639 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.472650 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.199991 7 0.000180 2014-06-09 16:22:11.472667 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.472681 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.472693 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.472702 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.472711 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.472763 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.472771 7f868e75f700 5 queue_transactions existing osr(2.d3 0x2ed0450)/0x2ed0450 2014-06-09 16:22:11.464334 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x13965d20,0xbd5a1e0,0x30509800,0x3050ae20,0x189aa160,0xbd685a0,0x189abc40] 2014-06-09 16:22:11.472789 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x13965d20,0xbd5a1e0,0x30509800,0x3050ae20,0x189aa160,0xbd685a0,0x189abc40] 2014-06-09 16:22:11.472793 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.472794 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.464340 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.464344 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd66980,0xbe302e0,0xbd72f20,0x3050aa80,0xbfcffa0,0x189aa340,0x15a3e8a0] 2014-06-09 16:22:11.472808 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.472827 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.472847 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.472870 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.472884 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.472899 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.472919 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.472933 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.472949 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.472968 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.472982 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.472997 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.473015 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.473032 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.473047 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.473069 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.473084 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.473096 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.473113 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.473128 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.473144 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.473165 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd66980,0xbe302e0,0xbd72f20,0x3050aa80,0xbfcffa0,0x189aa340,0x15a3e8a0] 2014-06-09 16:22:11.473168 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.473169 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.464394 7f868df5e700 10 op_submit_start 3148 2014-06-09 16:22:11.473174 7f868df5e700 5 queue_transactions (trailing journal) 3148 2014-06-09 16:22:11.473177 7f868df5e700 5 queue_op 0x30060120 seq 3148 osr(3.562 0x3ed1920) 1053 bytes (queue has 15 ops and 25902 bytes) 2014-06-09 16:22:11.473194 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (8 active) 2014-06-09 16:22:11.473199 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.473212 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.473223 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.473229 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.473239 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.620( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.473248 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.473251 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.473261 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.473270 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.473274 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.473284 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.473294 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.473298 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.473310 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.473320 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.473323 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.473333 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.473342 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.473350 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (7 active) 2014-06-09 16:22:11.473354 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.473403 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.473410 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.473436 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.473466 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.473473 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.473502 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.473511 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473516 7f869af78700 20 osd.26 243 _dispatch 0x2d736a00 pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.473527 7f869af78700 5 -- op tracker -- , seq: 1722, time: 2014-06-09 16:22:11.017560, event: header_read, op: pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473546 7f869af78700 5 -- op tracker -- , seq: 1722, time: 2014-06-09 16:22:11.017561, event: throttled, op: pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473554 7f869af78700 5 -- op tracker -- , seq: 1722, time: 2014-06-09 16:22:11.017575, event: all_read, op: pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473561 7f869af78700 5 -- op tracker -- , seq: 1722, time: 2014-06-09 16:22:11.473510, event: dispatched, op: pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473569 7f869af78700 5 -- op tracker -- , seq: 1722, time: 2014-06-09 16:22:11.473566, event: waiting_for_osdmap, op: pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473575 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.473580 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d736a00 2014-06-09 16:22:11.473587 7f869af78700 5 -- op tracker -- , seq: 1722, time: 2014-06-09 16:22:11.473584, event: started, op: pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473606 7f869af78700 5 -- op tracker -- , seq: 1722, time: 2014-06-09 16:22:11.473603, event: done, op: pg_log(1.32f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473618 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473620 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473636 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473639 7f869af78700 20 osd.26 243 _dispatch 0x302a0000 pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.473647 7f869af78700 5 -- op tracker -- , seq: 1723, time: 2014-06-09 16:22:11.011719, event: header_read, op: pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473655 7f869af78700 5 -- op tracker -- , seq: 1723, time: 2014-06-09 16:22:11.011719, event: throttled, op: pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473662 7f869af78700 5 -- op tracker -- , seq: 1723, time: 2014-06-09 16:22:11.011727, event: all_read, op: pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473669 7f869af78700 5 -- op tracker -- , seq: 1723, time: 2014-06-09 16:22:11.473636, event: dispatched, op: pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473677 7f869af78700 5 -- op tracker -- , seq: 1723, time: 2014-06-09 16:22:11.473674, event: waiting_for_osdmap, op: pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473682 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.473686 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x302a0000 2014-06-09 16:22:11.473692 7f869af78700 5 -- op tracker -- , seq: 1723, time: 2014-06-09 16:22:11.473689, event: started, op: pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473707 7f869af78700 5 -- op tracker -- , seq: 1723, time: 2014-06-09 16:22:11.473704, event: done, op: pg_log(2.30b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473718 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473720 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473734 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473736 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473737 7f869af78700 20 osd.26 243 _dispatch 0x3044c700 pg_notify(1.116(20),0.5e4(20) epoch 243) v5 2014-06-09 16:22:11.473743 7f869af78700 5 -- op tracker -- , seq: 1724, time: 2014-06-09 16:22:11.222754, event: header_read, op: pg_notify(1.116(20),0.5e4(20) epoch 243) 2014-06-09 16:22:11.473751 7f869af78700 5 -- op tracker -- , seq: 1724, time: 2014-06-09 16:22:11.222755, event: throttled, op: pg_notify(1.116(20),0.5e4(20) epoch 243) 2014-06-09 16:22:11.473758 7f869af78700 5 -- op tracker -- , seq: 1724, time: 2014-06-09 16:22:11.222819, event: all_read, op: pg_notify(1.116(20),0.5e4(20) epoch 243) 2014-06-09 16:22:11.473765 7f869af78700 5 -- op tracker -- , seq: 1724, time: 2014-06-09 16:22:11.473734, event: dispatched, op: pg_notify(1.116(20),0.5e4(20) epoch 243) 2014-06-09 16:22:11.473774 7f869af78700 5 -- op tracker -- , seq: 1724, time: 2014-06-09 16:22:11.473771, event: waiting_for_osdmap, op: pg_notify(1.116(20),0.5e4(20) epoch 243) 2014-06-09 16:22:11.473779 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.473781 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044c700 2014-06-09 16:22:11.473786 7f869af78700 5 -- op tracker -- , seq: 1724, time: 2014-06-09 16:22:11.473784, event: started, op: pg_notify(1.116(20),0.5e4(20) epoch 243) 2014-06-09 16:22:11.473816 7f869af78700 5 -- op tracker -- , seq: 1724, time: 2014-06-09 16:22:11.473814, event: done, op: pg_notify(1.116(20),0.5e4(20) epoch 243) 2014-06-09 16:22:11.473825 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473827 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473862 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473864 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473865 7f869af78700 20 osd.26 243 _dispatch 0x305bb200 pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.473872 7f869af78700 5 -- op tracker -- , seq: 1725, time: 2014-06-09 16:22:11.169520, event: header_read, op: pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473880 7f869af78700 5 -- op tracker -- , seq: 1725, time: 2014-06-09 16:22:11.169521, event: throttled, op: pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473887 7f869af78700 5 -- op tracker -- , seq: 1725, time: 2014-06-09 16:22:11.169532, event: all_read, op: pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473893 7f869af78700 5 -- op tracker -- , seq: 1725, time: 2014-06-09 16:22:11.473861, event: dispatched, op: pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473901 7f869af78700 5 -- op tracker -- , seq: 1725, time: 2014-06-09 16:22:11.473898, event: waiting_for_osdmap, op: pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473905 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.473910 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305bb200 2014-06-09 16:22:11.473915 7f869af78700 5 -- op tracker -- , seq: 1725, time: 2014-06-09 16:22:11.473913, event: started, op: pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473931 7f869af78700 5 -- op tracker -- , seq: 1725, time: 2014-06-09 16:22:11.473928, event: done, op: pg_log(1.206 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473939 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473941 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473955 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.473957 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.473958 7f869af78700 20 osd.26 243 _dispatch 0x30645800 pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.473965 7f869af78700 5 -- op tracker -- , seq: 1726, time: 2014-06-09 16:22:11.119876, event: header_read, op: pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473973 7f869af78700 5 -- op tracker -- , seq: 1726, time: 2014-06-09 16:22:11.119877, event: throttled, op: pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473980 7f869af78700 5 -- op tracker -- , seq: 1726, time: 2014-06-09 16:22:11.119887, event: all_read, op: pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473986 7f869af78700 5 -- op tracker -- , seq: 1726, time: 2014-06-09 16:22:11.473955, event: dispatched, op: pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473993 7f869af78700 5 -- op tracker -- , seq: 1726, time: 2014-06-09 16:22:11.473991, event: waiting_for_osdmap, op: pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.473998 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.474002 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30645800 2014-06-09 16:22:11.474008 7f869af78700 5 -- op tracker -- , seq: 1726, time: 2014-06-09 16:22:11.474005, event: started, op: pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474024 7f869af78700 5 -- op tracker -- , seq: 1726, time: 2014-06-09 16:22:11.474020, event: done, op: pg_log(1.65e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474032 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474033 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474045 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474047 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474048 7f869af78700 20 osd.26 243 _dispatch 0x19fda380 pg_notify(3.ce(10) epoch 243) v5 2014-06-09 16:22:11.474053 7f869af78700 5 -- op tracker -- , seq: 1727, time: 2014-06-09 16:22:11.161053, event: header_read, op: pg_notify(3.ce(10) epoch 243) 2014-06-09 16:22:11.474060 7f869af78700 5 -- op tracker -- , seq: 1727, time: 2014-06-09 16:22:11.161054, event: throttled, op: pg_notify(3.ce(10) epoch 243) 2014-06-09 16:22:11.474066 7f869af78700 5 -- op tracker -- , seq: 1727, time: 2014-06-09 16:22:11.161074, event: all_read, op: pg_notify(3.ce(10) epoch 243) 2014-06-09 16:22:11.474072 7f869af78700 5 -- op tracker -- , seq: 1727, time: 2014-06-09 16:22:11.474045, event: dispatched, op: pg_notify(3.ce(10) epoch 243) 2014-06-09 16:22:11.474080 7f869af78700 5 -- op tracker -- , seq: 1727, time: 2014-06-09 16:22:11.474078, event: waiting_for_osdmap, op: pg_notify(3.ce(10) epoch 243) 2014-06-09 16:22:11.474084 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.474086 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fda380 2014-06-09 16:22:11.474092 7f869af78700 5 -- op tracker -- , seq: 1727, time: 2014-06-09 16:22:11.474090, event: started, op: pg_notify(3.ce(10) epoch 243) 2014-06-09 16:22:11.474112 7f869af78700 5 -- op tracker -- , seq: 1727, time: 2014-06-09 16:22:11.474109, event: done, op: pg_notify(3.ce(10) epoch 243) 2014-06-09 16:22:11.474120 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474122 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474137 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474139 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474140 7f869af78700 20 osd.26 243 _dispatch 0x2fe70000 pg_info(1 pgs e243:3.381(2)) v4 2014-06-09 16:22:11.474146 7f869af78700 5 -- op tracker -- , seq: 1728, time: 2014-06-09 16:22:11.114991, event: header_read, op: pg_info(1 pgs e243:3.381(2)) 2014-06-09 16:22:11.474153 7f869af78700 5 -- op tracker -- , seq: 1728, time: 2014-06-09 16:22:11.114992, event: throttled, op: pg_info(1 pgs e243:3.381(2)) 2014-06-09 16:22:11.474160 7f869af78700 5 -- op tracker -- , seq: 1728, time: 2014-06-09 16:22:11.115012, event: all_read, op: pg_info(1 pgs e243:3.381(2)) 2014-06-09 16:22:11.474166 7f869af78700 5 -- op tracker -- , seq: 1728, time: 2014-06-09 16:22:11.474137, event: dispatched, op: pg_info(1 pgs e243:3.381(2)) 2014-06-09 16:22:11.474174 7f869af78700 5 -- op tracker -- , seq: 1728, time: 2014-06-09 16:22:11.474171, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.381(2)) 2014-06-09 16:22:11.474179 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.381(2)) v4 from osd.8 2014-06-09 16:22:11.474182 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe70000 2014-06-09 16:22:11.474187 7f869af78700 5 -- op tracker -- , seq: 1728, time: 2014-06-09 16:22:11.474185, event: started, op: pg_info(1 pgs e243:3.381(2)) 2014-06-09 16:22:11.474206 7f869af78700 5 -- op tracker -- , seq: 1728, time: 2014-06-09 16:22:11.474203, event: done, op: pg_info(1 pgs e243:3.381(2)) 2014-06-09 16:22:11.474214 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474216 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474233 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474234 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474236 7f869af78700 20 osd.26 243 _dispatch 0x3004f000 pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.474242 7f869af78700 5 -- op tracker -- , seq: 1729, time: 2014-06-09 16:22:11.075966, event: header_read, op: pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474250 7f869af78700 5 -- op tracker -- , seq: 1729, time: 2014-06-09 16:22:11.075967, event: throttled, op: pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474257 7f869af78700 5 -- op tracker -- , seq: 1729, time: 2014-06-09 16:22:11.075980, event: all_read, op: pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474264 7f869af78700 5 -- op tracker -- , seq: 1729, time: 2014-06-09 16:22:11.474232, event: dispatched, op: pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474272 7f869af78700 5 -- op tracker -- , seq: 1729, time: 2014-06-09 16:22:11.474269, event: waiting_for_osdmap, op: pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474276 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.474280 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004f000 2014-06-09 16:22:11.474286 7f869af78700 5 -- op tracker -- , seq: 1729, time: 2014-06-09 16:22:11.474283, event: started, op: pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474302 7f869af78700 5 -- op tracker -- , seq: 1729, time: 2014-06-09 16:22:11.474299, event: done, op: pg_log(0.66a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474311 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474312 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474325 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474327 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474329 7f869af78700 20 osd.26 243 _dispatch 0x30045800 pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.474335 7f869af78700 5 -- op tracker -- , seq: 1730, time: 2014-06-09 16:22:11.075130, event: header_read, op: pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474343 7f869af78700 5 -- op tracker -- , seq: 1730, time: 2014-06-09 16:22:11.075130, event: throttled, op: pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474350 7f869af78700 5 -- op tracker -- , seq: 1730, time: 2014-06-09 16:22:11.075144, event: all_read, op: pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474357 7f869af78700 5 -- op tracker -- , seq: 1730, time: 2014-06-09 16:22:11.474325, event: dispatched, op: pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474365 7f869af78700 5 -- op tracker -- , seq: 1730, time: 2014-06-09 16:22:11.474362, event: waiting_for_osdmap, op: pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474370 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.474374 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30045800 2014-06-09 16:22:11.474380 7f869af78700 5 -- op tracker -- , seq: 1730, time: 2014-06-09 16:22:11.474377, event: started, op: pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474396 7f869af78700 5 -- op tracker -- , seq: 1730, time: 2014-06-09 16:22:11.474393, event: done, op: pg_log(1.fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474404 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474406 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474419 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474421 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474422 7f869af78700 20 osd.26 243 _dispatch 0x308a7800 pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.474429 7f869af78700 5 -- op tracker -- , seq: 1731, time: 2014-06-09 16:22:11.190967, event: header_read, op: pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474437 7f869af78700 5 -- op tracker -- , seq: 1731, time: 2014-06-09 16:22:11.190968, event: throttled, op: pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474444 7f869af78700 5 -- op tracker -- , seq: 1731, time: 2014-06-09 16:22:11.190980, event: all_read, op: pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474451 7f869af78700 5 -- op tracker -- , seq: 1731, time: 2014-06-09 16:22:11.474419, event: dispatched, op: pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474459 7f869af78700 5 -- op tracker -- , seq: 1731, time: 2014-06-09 16:22:11.474456, event: waiting_for_osdmap, op: pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474463 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.474467 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a7800 2014-06-09 16:22:11.474482 7f869af78700 5 -- op tracker -- , seq: 1731, time: 2014-06-09 16:22:11.474479, event: started, op: pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474498 7f869af78700 5 -- op tracker -- , seq: 1731, time: 2014-06-09 16:22:11.474495, event: done, op: pg_log(0.648 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474506 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474507 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474520 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474522 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474523 7f869af78700 20 osd.26 243 _dispatch 0x2d736400 pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.474529 7f869af78700 5 -- op tracker -- , seq: 1732, time: 2014-06-09 16:22:11.017581, event: header_read, op: pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474537 7f869af78700 5 -- op tracker -- , seq: 1732, time: 2014-06-09 16:22:11.017581, event: throttled, op: pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474545 7f869af78700 5 -- op tracker -- , seq: 1732, time: 2014-06-09 16:22:11.017590, event: all_read, op: pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474558 7f869af78700 5 -- op tracker -- , seq: 1732, time: 2014-06-09 16:22:11.474520, event: dispatched, op: pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474566 7f869af78700 5 -- op tracker -- , seq: 1732, time: 2014-06-09 16:22:11.474564, event: waiting_for_osdmap, op: pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474571 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.474575 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d736400 2014-06-09 16:22:11.474581 7f869af78700 5 -- op tracker -- , seq: 1732, time: 2014-06-09 16:22:11.474578, event: started, op: pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474596 7f869af78700 5 -- op tracker -- , seq: 1732, time: 2014-06-09 16:22:11.474593, event: done, op: pg_log(0.330 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474604 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474606 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474618 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474619 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474620 7f869af78700 20 osd.26 243 _dispatch 0x2fe70700 pg_notify(0.35(23) epoch 243) v5 2014-06-09 16:22:11.474625 7f869af78700 5 -- op tracker -- , seq: 1733, time: 2014-06-09 16:22:11.011835, event: header_read, op: pg_notify(0.35(23) epoch 243) 2014-06-09 16:22:11.474632 7f869af78700 5 -- op tracker -- , seq: 1733, time: 2014-06-09 16:22:11.011836, event: throttled, op: pg_notify(0.35(23) epoch 243) 2014-06-09 16:22:11.474638 7f869af78700 5 -- op tracker -- , seq: 1733, time: 2014-06-09 16:22:11.011858, event: all_read, op: pg_notify(0.35(23) epoch 243) 2014-06-09 16:22:11.474644 7f869af78700 5 -- op tracker -- , seq: 1733, time: 2014-06-09 16:22:11.474618, event: dispatched, op: pg_notify(0.35(23) epoch 243) 2014-06-09 16:22:11.474651 7f869af78700 5 -- op tracker -- , seq: 1733, time: 2014-06-09 16:22:11.474649, event: waiting_for_osdmap, op: pg_notify(0.35(23) epoch 243) 2014-06-09 16:22:11.474656 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.474658 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe70700 2014-06-09 16:22:11.474663 7f869af78700 5 -- op tracker -- , seq: 1733, time: 2014-06-09 16:22:11.474661, event: started, op: pg_notify(0.35(23) epoch 243) 2014-06-09 16:22:11.474682 7f869af78700 5 -- op tracker -- , seq: 1733, time: 2014-06-09 16:22:11.474680, event: done, op: pg_notify(0.35(23) epoch 243) 2014-06-09 16:22:11.474691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474692 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474710 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474711 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474712 7f869af78700 20 osd.26 243 _dispatch 0x308e2700 pg_info(1 pgs e243:3.148(2)) v4 2014-06-09 16:22:11.474718 7f869af78700 5 -- op tracker -- , seq: 1734, time: 2014-06-09 16:22:11.222834, event: header_read, op: pg_info(1 pgs e243:3.148(2)) 2014-06-09 16:22:11.474725 7f869af78700 5 -- op tracker -- , seq: 1734, time: 2014-06-09 16:22:11.222834, event: throttled, op: pg_info(1 pgs e243:3.148(2)) 2014-06-09 16:22:11.474732 7f869af78700 5 -- op tracker -- , seq: 1734, time: 2014-06-09 16:22:11.222848, event: all_read, op: pg_info(1 pgs e243:3.148(2)) 2014-06-09 16:22:11.474738 7f869af78700 5 -- op tracker -- , seq: 1734, time: 2014-06-09 16:22:11.474709, event: dispatched, op: pg_info(1 pgs e243:3.148(2)) 2014-06-09 16:22:11.474745 7f869af78700 5 -- op tracker -- , seq: 1734, time: 2014-06-09 16:22:11.474742, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.148(2)) 2014-06-09 16:22:11.474749 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.148(2)) v4 from osd.7 2014-06-09 16:22:11.474753 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e2700 2014-06-09 16:22:11.474758 7f869af78700 5 -- op tracker -- , seq: 1734, time: 2014-06-09 16:22:11.474756, event: started, op: pg_info(1 pgs e243:3.148(2)) 2014-06-09 16:22:11.474785 7f869af78700 5 -- op tracker -- , seq: 1734, time: 2014-06-09 16:22:11.474772, event: done, op: pg_info(1 pgs e243:3.148(2)) 2014-06-09 16:22:11.474794 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474810 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474811 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474812 7f869af78700 20 osd.26 243 _dispatch 0x2fef4a80 pg_notify(1.6b7(20),3.3f5(10),0.3f9(20) epoch 243) v5 2014-06-09 16:22:11.474819 7f869af78700 5 -- op tracker -- , seq: 1735, time: 2014-06-09 16:22:11.169658, event: header_read, op: pg_notify(1.6b7(20),3.3f5(10),0.3f9(20) epoch 243) 2014-06-09 16:22:11.474827 7f869af78700 5 -- op tracker -- , seq: 1735, time: 2014-06-09 16:22:11.169659, event: throttled, op: pg_notify(1.6b7(20),3.3f5(10),0.3f9(20) epoch 243) 2014-06-09 16:22:11.474834 7f869af78700 5 -- op tracker -- , seq: 1735, time: 2014-06-09 16:22:11.169715, event: all_read, op: pg_notify(1.6b7(20),3.3f5(10),0.3f9(20) epoch 243) 2014-06-09 16:22:11.474841 7f869af78700 5 -- op tracker -- , seq: 1735, time: 2014-06-09 16:22:11.474809, event: dispatched, op: pg_notify(1.6b7(20),3.3f5(10),0.3f9(20) epoch 243) 2014-06-09 16:22:11.474848 7f869af78700 5 -- op tracker -- , seq: 1735, time: 2014-06-09 16:22:11.474845, event: waiting_for_osdmap, op: pg_notify(1.6b7(20),3.3f5(10),0.3f9(20) epoch 243) 2014-06-09 16:22:11.474852 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.474854 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4a80 2014-06-09 16:22:11.474866 7f869af78700 5 -- op tracker -- , seq: 1735, time: 2014-06-09 16:22:11.474857, event: started, op: pg_notify(1.6b7(20),3.3f5(10),0.3f9(20) epoch 243) 2014-06-09 16:22:11.474898 7f869af78700 5 -- op tracker -- , seq: 1735, time: 2014-06-09 16:22:11.474895, event: done, op: pg_notify(1.6b7(20),3.3f5(10),0.3f9(20) epoch 243) 2014-06-09 16:22:11.474907 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474909 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474937 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.474939 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.474940 7f869af78700 20 osd.26 243 _dispatch 0x30643600 pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.474946 7f869af78700 5 -- op tracker -- , seq: 1736, time: 2014-06-09 16:22:11.126978, event: header_read, op: pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474954 7f869af78700 5 -- op tracker -- , seq: 1736, time: 2014-06-09 16:22:11.126979, event: throttled, op: pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474961 7f869af78700 5 -- op tracker -- , seq: 1736, time: 2014-06-09 16:22:11.126995, event: all_read, op: pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474967 7f869af78700 5 -- op tracker -- , seq: 1736, time: 2014-06-09 16:22:11.474936, event: dispatched, op: pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474975 7f869af78700 5 -- op tracker -- , seq: 1736, time: 2014-06-09 16:22:11.474973, event: waiting_for_osdmap, op: pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.474980 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.474984 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30643600 2014-06-09 16:22:11.474990 7f869af78700 5 -- op tracker -- , seq: 1736, time: 2014-06-09 16:22:11.474988, event: started, op: pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475004 7f869af78700 5 -- op tracker -- , seq: 1736, time: 2014-06-09 16:22:11.475001, event: done, op: pg_log(1.10d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475012 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475015 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475027 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475028 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475030 7f869af78700 20 osd.26 243 _dispatch 0x30614400 pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.475036 7f869af78700 5 -- op tracker -- , seq: 1737, time: 2014-06-09 16:22:11.161081, event: header_read, op: pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475044 7f869af78700 5 -- op tracker -- , seq: 1737, time: 2014-06-09 16:22:11.161082, event: throttled, op: pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475050 7f869af78700 5 -- op tracker -- , seq: 1737, time: 2014-06-09 16:22:11.161091, event: all_read, op: pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475057 7f869af78700 5 -- op tracker -- , seq: 1737, time: 2014-06-09 16:22:11.475027, event: dispatched, op: pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475064 7f869af78700 5 -- op tracker -- , seq: 1737, time: 2014-06-09 16:22:11.475062, event: waiting_for_osdmap, op: pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475069 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.475073 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30614400 2014-06-09 16:22:11.475078 7f869af78700 5 -- op tracker -- , seq: 1737, time: 2014-06-09 16:22:11.475076, event: started, op: pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475092 7f869af78700 5 -- op tracker -- , seq: 1737, time: 2014-06-09 16:22:11.475089, event: done, op: pg_log(1.5cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475100 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475102 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475114 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475115 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475116 7f869af78700 20 osd.26 243 _dispatch 0x30612c00 pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.475123 7f869af78700 5 -- op tracker -- , seq: 1738, time: 2014-06-09 16:22:11.116578, event: header_read, op: pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475131 7f869af78700 5 -- op tracker -- , seq: 1738, time: 2014-06-09 16:22:11.116578, event: throttled, op: pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475138 7f869af78700 5 -- op tracker -- , seq: 1738, time: 2014-06-09 16:22:11.116592, event: all_read, op: pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475145 7f869af78700 5 -- op tracker -- , seq: 1738, time: 2014-06-09 16:22:11.475113, event: dispatched, op: pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475153 7f869af78700 5 -- op tracker -- , seq: 1738, time: 2014-06-09 16:22:11.475150, event: waiting_for_osdmap, op: pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475158 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.475161 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30612c00 2014-06-09 16:22:11.475167 7f869af78700 5 -- op tracker -- , seq: 1738, time: 2014-06-09 16:22:11.475164, event: started, op: pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475180 7f869af78700 5 -- op tracker -- , seq: 1738, time: 2014-06-09 16:22:11.475177, event: done, op: pg_log(1.35c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475188 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475190 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475203 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475204 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475206 7f869af78700 20 osd.26 243 _dispatch 0x30571200 pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.475212 7f869af78700 5 -- op tracker -- , seq: 1739, time: 2014-06-09 16:22:11.077385, event: header_read, op: pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475220 7f869af78700 5 -- op tracker -- , seq: 1739, time: 2014-06-09 16:22:11.077385, event: throttled, op: pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475227 7f869af78700 5 -- op tracker -- , seq: 1739, time: 2014-06-09 16:22:11.077398, event: all_read, op: pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475234 7f869af78700 5 -- op tracker -- , seq: 1739, time: 2014-06-09 16:22:11.475202, event: dispatched, op: pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475242 7f869af78700 5 -- op tracker -- , seq: 1739, time: 2014-06-09 16:22:11.475239, event: waiting_for_osdmap, op: pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475247 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.475251 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30571200 2014-06-09 16:22:11.475256 7f869af78700 5 -- op tracker -- , seq: 1739, time: 2014-06-09 16:22:11.475254, event: started, op: pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475269 7f869af78700 5 -- op tracker -- , seq: 1739, time: 2014-06-09 16:22:11.475266, event: done, op: pg_log(1.a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475276 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475278 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475290 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475291 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475293 7f869af78700 20 osd.26 243 _dispatch 0x30572400 pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.475299 7f869af78700 5 -- op tracker -- , seq: 1740, time: 2014-06-09 16:22:11.090988, event: header_read, op: pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475307 7f869af78700 5 -- op tracker -- , seq: 1740, time: 2014-06-09 16:22:11.090989, event: throttled, op: pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475313 7f869af78700 5 -- op tracker -- , seq: 1740, time: 2014-06-09 16:22:11.091002, event: all_read, op: pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475320 7f869af78700 5 -- op tracker -- , seq: 1740, time: 2014-06-09 16:22:11.475290, event: dispatched, op: pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475328 7f869af78700 5 -- op tracker -- , seq: 1740, time: 2014-06-09 16:22:11.475325, event: waiting_for_osdmap, op: pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475332 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.475336 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30572400 2014-06-09 16:22:11.475341 7f869af78700 5 -- op tracker -- , seq: 1740, time: 2014-06-09 16:22:11.475339, event: started, op: pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475354 7f869af78700 5 -- op tracker -- , seq: 1740, time: 2014-06-09 16:22:11.475351, event: done, op: pg_log(1.3f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475362 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475364 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475375 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475377 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475378 7f869af78700 20 osd.26 243 _dispatch 0x31a5c00 pg_notify(2.446(20) epoch 243) v5 2014-06-09 16:22:11.475383 7f869af78700 5 -- op tracker -- , seq: 1741, time: 2014-06-09 16:22:11.196880, event: header_read, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.475390 7f869af78700 5 -- op tracker -- , seq: 1741, time: 2014-06-09 16:22:11.196880, event: throttled, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.475397 7f869af78700 5 -- op tracker -- , seq: 1741, time: 2014-06-09 16:22:11.196900, event: all_read, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.475403 7f869af78700 5 -- op tracker -- , seq: 1741, time: 2014-06-09 16:22:11.475375, event: dispatched, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.475409 7f869af78700 5 -- op tracker -- , seq: 1741, time: 2014-06-09 16:22:11.475407, event: waiting_for_osdmap, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.475414 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.475416 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31a5c00 2014-06-09 16:22:11.475420 7f869af78700 5 -- op tracker -- , seq: 1741, time: 2014-06-09 16:22:11.475418, event: started, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.475436 7f869af78700 5 -- op tracker -- , seq: 1741, time: 2014-06-09 16:22:11.475434, event: done, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.475443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475466 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475468 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475469 7f869af78700 20 osd.26 243 _dispatch 0x1c039c00 pg_notify(1.4a(21),0.4d(21),1.53(21),0.31b(26) epoch 243) v5 2014-06-09 16:22:11.475477 7f869af78700 5 -- op tracker -- , seq: 1742, time: 2014-06-09 16:22:11.017594, event: header_read, op: pg_notify(1.4a(21),0.4d(21),1.53(21),0.31b(26) epoch 243) 2014-06-09 16:22:11.475485 7f869af78700 5 -- op tracker -- , seq: 1742, time: 2014-06-09 16:22:11.017594, event: throttled, op: pg_notify(1.4a(21),0.4d(21),1.53(21),0.31b(26) epoch 243) 2014-06-09 16:22:11.475493 7f869af78700 5 -- op tracker -- , seq: 1742, time: 2014-06-09 16:22:11.017649, event: all_read, op: pg_notify(1.4a(21),0.4d(21),1.53(21),0.31b(26) epoch 243) 2014-06-09 16:22:11.475500 7f869af78700 5 -- op tracker -- , seq: 1742, time: 2014-06-09 16:22:11.475465, event: dispatched, op: pg_notify(1.4a(21),0.4d(21),1.53(21),0.31b(26) epoch 243) 2014-06-09 16:22:11.475507 7f869af78700 5 -- op tracker -- , seq: 1742, time: 2014-06-09 16:22:11.475504, event: waiting_for_osdmap, op: pg_notify(1.4a(21),0.4d(21),1.53(21),0.31b(26) epoch 243) 2014-06-09 16:22:11.475512 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.475514 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c039c00 2014-06-09 16:22:11.475520 7f869af78700 5 -- op tracker -- , seq: 1742, time: 2014-06-09 16:22:11.475516, event: started, op: pg_notify(1.4a(21),0.4d(21),1.53(21),0.31b(26) epoch 243) 2014-06-09 16:22:11.475557 7f869af78700 5 -- op tracker -- , seq: 1742, time: 2014-06-09 16:22:11.475554, event: done, op: pg_notify(1.4a(21),0.4d(21),1.53(21),0.31b(26) epoch 243) 2014-06-09 16:22:11.475568 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475569 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475601 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475603 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475604 7f869af78700 20 osd.26 243 _dispatch 0x1fc04e00 pg_notify(1.30e(23),2.316(20) epoch 243) v5 2014-06-09 16:22:11.475610 7f869af78700 5 -- op tracker -- , seq: 1743, time: 2014-06-09 16:22:11.015727, event: header_read, op: pg_notify(1.30e(23),2.316(20) epoch 243) 2014-06-09 16:22:11.475624 7f869af78700 5 -- op tracker -- , seq: 1743, time: 2014-06-09 16:22:11.015728, event: throttled, op: pg_notify(1.30e(23),2.316(20) epoch 243) 2014-06-09 16:22:11.475631 7f869af78700 5 -- op tracker -- , seq: 1743, time: 2014-06-09 16:22:11.015786, event: all_read, op: pg_notify(1.30e(23),2.316(20) epoch 243) 2014-06-09 16:22:11.475638 7f869af78700 5 -- op tracker -- , seq: 1743, time: 2014-06-09 16:22:11.475601, event: dispatched, op: pg_notify(1.30e(23),2.316(20) epoch 243) 2014-06-09 16:22:11.475644 7f869af78700 5 -- op tracker -- , seq: 1743, time: 2014-06-09 16:22:11.475642, event: waiting_for_osdmap, op: pg_notify(1.30e(23),2.316(20) epoch 243) 2014-06-09 16:22:11.475649 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.475651 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc04e00 2014-06-09 16:22:11.475656 7f869af78700 5 -- op tracker -- , seq: 1743, time: 2014-06-09 16:22:11.475653, event: started, op: pg_notify(1.30e(23),2.316(20) epoch 243) 2014-06-09 16:22:11.475678 7f869af78700 5 -- op tracker -- , seq: 1743, time: 2014-06-09 16:22:11.475676, event: done, op: pg_notify(1.30e(23),2.316(20) epoch 243) 2014-06-09 16:22:11.475686 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475688 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475707 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475709 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475710 7f869af78700 20 osd.26 243 _dispatch 0x308d9500 pg_info(1 pgs e243:2.17c) v4 2014-06-09 16:22:11.475715 7f869af78700 5 -- op tracker -- , seq: 1744, time: 2014-06-09 16:22:11.222984, event: header_read, op: pg_info(1 pgs e243:2.17c) 2014-06-09 16:22:11.475722 7f869af78700 5 -- op tracker -- , seq: 1744, time: 2014-06-09 16:22:11.222984, event: throttled, op: pg_info(1 pgs e243:2.17c) 2014-06-09 16:22:11.475728 7f869af78700 5 -- op tracker -- , seq: 1744, time: 2014-06-09 16:22:11.222999, event: all_read, op: pg_info(1 pgs e243:2.17c) 2014-06-09 16:22:11.475735 7f869af78700 5 -- op tracker -- , seq: 1744, time: 2014-06-09 16:22:11.475706, event: dispatched, op: pg_info(1 pgs e243:2.17c) 2014-06-09 16:22:11.475742 7f869af78700 5 -- op tracker -- , seq: 1744, time: 2014-06-09 16:22:11.475740, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.17c) 2014-06-09 16:22:11.475747 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.17c) v4 from osd.7 2014-06-09 16:22:11.475749 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d9500 2014-06-09 16:22:11.475754 7f869af78700 5 -- op tracker -- , seq: 1744, time: 2014-06-09 16:22:11.475752, event: started, op: pg_info(1 pgs e243:2.17c) 2014-06-09 16:22:11.475769 7f869af78700 5 -- op tracker -- , seq: 1744, time: 2014-06-09 16:22:11.475767, event: done, op: pg_info(1 pgs e243:2.17c) 2014-06-09 16:22:11.475778 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475779 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475796 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475797 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475798 7f869af78700 20 osd.26 243 _dispatch 0x304df600 pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.475805 7f869af78700 5 -- op tracker -- , seq: 1745, time: 2014-06-09 16:22:11.169722, event: header_read, op: pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475813 7f869af78700 5 -- op tracker -- , seq: 1745, time: 2014-06-09 16:22:11.169722, event: throttled, op: pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475821 7f869af78700 5 -- op tracker -- , seq: 1745, time: 2014-06-09 16:22:11.169735, event: all_read, op: pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475828 7f869af78700 5 -- op tracker -- , seq: 1745, time: 2014-06-09 16:22:11.475795, event: dispatched, op: pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475835 7f869af78700 5 -- op tracker -- , seq: 1745, time: 2014-06-09 16:22:11.475833, event: waiting_for_osdmap, op: pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475840 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.475844 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304df600 2014-06-09 16:22:11.475849 7f869af78700 5 -- op tracker -- , seq: 1745, time: 2014-06-09 16:22:11.475847, event: started, op: pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475862 7f869af78700 5 -- op tracker -- , seq: 1745, time: 2014-06-09 16:22:11.475859, event: done, op: pg_log(1.211 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475869 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475871 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475883 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475885 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475886 7f869af78700 20 osd.26 243 _dispatch 0x30643000 pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.475893 7f869af78700 5 -- op tracker -- , seq: 1746, time: 2014-06-09 16:22:11.127013, event: header_read, op: pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475901 7f869af78700 5 -- op tracker -- , seq: 1746, time: 2014-06-09 16:22:11.127014, event: throttled, op: pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475908 7f869af78700 5 -- op tracker -- , seq: 1746, time: 2014-06-09 16:22:11.127025, event: all_read, op: pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475915 7f869af78700 5 -- op tracker -- , seq: 1746, time: 2014-06-09 16:22:11.475883, event: dispatched, op: pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475922 7f869af78700 5 -- op tracker -- , seq: 1746, time: 2014-06-09 16:22:11.475919, event: waiting_for_osdmap, op: pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475926 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.475931 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30643000 2014-06-09 16:22:11.475937 7f869af78700 5 -- op tracker -- , seq: 1746, time: 2014-06-09 16:22:11.475934, event: started, op: pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475951 7f869af78700 5 -- op tracker -- , seq: 1746, time: 2014-06-09 16:22:11.475948, event: done, op: pg_log(2.126 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.475959 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475961 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475973 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.475974 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.475975 7f869af78700 20 osd.26 243 _dispatch 0x1a038540 pg_info(1 pgs e243:3.77(2)) v4 2014-06-09 16:22:11.475980 7f869af78700 5 -- op tracker -- , seq: 1747, time: 2014-06-09 16:22:11.163806, event: header_read, op: pg_info(1 pgs e243:3.77(2)) 2014-06-09 16:22:11.475988 7f869af78700 5 -- op tracker -- , seq: 1747, time: 2014-06-09 16:22:11.163807, event: throttled, op: pg_info(1 pgs e243:3.77(2)) 2014-06-09 16:22:11.475994 7f869af78700 5 -- op tracker -- , seq: 1747, time: 2014-06-09 16:22:11.163831, event: all_read, op: pg_info(1 pgs e243:3.77(2)) 2014-06-09 16:22:11.476000 7f869af78700 5 -- op tracker -- , seq: 1747, time: 2014-06-09 16:22:11.475972, event: dispatched, op: pg_info(1 pgs e243:3.77(2)) 2014-06-09 16:22:11.476007 7f869af78700 5 -- op tracker -- , seq: 1747, time: 2014-06-09 16:22:11.476005, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.77(2)) 2014-06-09 16:22:11.476011 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.77(2)) v4 from osd.9 2014-06-09 16:22:11.476014 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a038540 2014-06-09 16:22:11.476020 7f869af78700 5 -- op tracker -- , seq: 1747, time: 2014-06-09 16:22:11.476017, event: started, op: pg_info(1 pgs e243:3.77(2)) 2014-06-09 16:22:11.476036 7f869af78700 5 -- op tracker -- , seq: 1747, time: 2014-06-09 16:22:11.476034, event: done, op: pg_info(1 pgs e243:3.77(2)) 2014-06-09 16:22:11.476045 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476047 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476060 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476062 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476063 7f869af78700 20 osd.26 243 _dispatch 0x3043b500 pg_notify(0.622(23) epoch 243) v5 2014-06-09 16:22:11.476068 7f869af78700 5 -- op tracker -- , seq: 1748, time: 2014-06-09 16:22:11.116640, event: header_read, op: pg_notify(0.622(23) epoch 243) 2014-06-09 16:22:11.476075 7f869af78700 5 -- op tracker -- , seq: 1748, time: 2014-06-09 16:22:11.116641, event: throttled, op: pg_notify(0.622(23) epoch 243) 2014-06-09 16:22:11.476081 7f869af78700 5 -- op tracker -- , seq: 1748, time: 2014-06-09 16:22:11.116673, event: all_read, op: pg_notify(0.622(23) epoch 243) 2014-06-09 16:22:11.476088 7f869af78700 5 -- op tracker -- , seq: 1748, time: 2014-06-09 16:22:11.476059, event: dispatched, op: pg_notify(0.622(23) epoch 243) 2014-06-09 16:22:11.476095 7f869af78700 5 -- op tracker -- , seq: 1748, time: 2014-06-09 16:22:11.476093, event: waiting_for_osdmap, op: pg_notify(0.622(23) epoch 243) 2014-06-09 16:22:11.476110 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.476113 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043b500 2014-06-09 16:22:11.476118 7f869af78700 5 -- op tracker -- , seq: 1748, time: 2014-06-09 16:22:11.476116, event: started, op: pg_notify(0.622(23) epoch 243) 2014-06-09 16:22:11.476134 7f869af78700 5 -- op tracker -- , seq: 1748, time: 2014-06-09 16:22:11.476131, event: done, op: pg_notify(0.622(23) epoch 243) 2014-06-09 16:22:11.476143 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476144 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476162 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476164 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476165 7f869af78700 20 osd.26 243 _dispatch 0x2fef4fc0 pg_notify(3.661(10) epoch 243) v5 2014-06-09 16:22:11.476170 7f869af78700 5 -- op tracker -- , seq: 1749, time: 2014-06-09 16:22:11.077846, event: header_read, op: pg_notify(3.661(10) epoch 243) 2014-06-09 16:22:11.476177 7f869af78700 5 -- op tracker -- , seq: 1749, time: 2014-06-09 16:22:11.077847, event: throttled, op: pg_notify(3.661(10) epoch 243) 2014-06-09 16:22:11.476184 7f869af78700 5 -- op tracker -- , seq: 1749, time: 2014-06-09 16:22:11.077867, event: all_read, op: pg_notify(3.661(10) epoch 243) 2014-06-09 16:22:11.476190 7f869af78700 5 -- op tracker -- , seq: 1749, time: 2014-06-09 16:22:11.476161, event: dispatched, op: pg_notify(3.661(10) epoch 243) 2014-06-09 16:22:11.476196 7f869af78700 5 -- op tracker -- , seq: 1749, time: 2014-06-09 16:22:11.476194, event: waiting_for_osdmap, op: pg_notify(3.661(10) epoch 243) 2014-06-09 16:22:11.476201 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.476203 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4fc0 2014-06-09 16:22:11.476208 7f869af78700 5 -- op tracker -- , seq: 1749, time: 2014-06-09 16:22:11.476206, event: started, op: pg_notify(3.661(10) epoch 243) 2014-06-09 16:22:11.476225 7f869af78700 5 -- op tracker -- , seq: 1749, time: 2014-06-09 16:22:11.476223, event: done, op: pg_notify(3.661(10) epoch 243) 2014-06-09 16:22:11.476232 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476234 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476251 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476252 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476253 7f869af78700 20 osd.26 243 _dispatch 0x305e2000 pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.476260 7f869af78700 5 -- op tracker -- , seq: 1750, time: 2014-06-09 16:22:11.127038, event: header_read, op: pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476268 7f869af78700 5 -- op tracker -- , seq: 1750, time: 2014-06-09 16:22:11.127039, event: throttled, op: pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476274 7f869af78700 5 -- op tracker -- , seq: 1750, time: 2014-06-09 16:22:11.127051, event: all_read, op: pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476281 7f869af78700 5 -- op tracker -- , seq: 1750, time: 2014-06-09 16:22:11.476250, event: dispatched, op: pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476289 7f869af78700 5 -- op tracker -- , seq: 1750, time: 2014-06-09 16:22:11.476286, event: waiting_for_osdmap, op: pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476294 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.476297 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e2000 2014-06-09 16:22:11.476310 7f869af78700 5 -- op tracker -- , seq: 1750, time: 2014-06-09 16:22:11.476301, event: started, op: pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476323 7f869af78700 5 -- op tracker -- , seq: 1750, time: 2014-06-09 16:22:11.476321, event: done, op: pg_log(0.25b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476332 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476334 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476348 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476350 7f869af78700 20 osd.26 243 _dispatch 0x308a7200 pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.476357 7f869af78700 5 -- op tracker -- , seq: 1751, time: 2014-06-09 16:22:11.197095, event: header_read, op: pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476364 7f869af78700 5 -- op tracker -- , seq: 1751, time: 2014-06-09 16:22:11.197095, event: throttled, op: pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476371 7f869af78700 5 -- op tracker -- , seq: 1751, time: 2014-06-09 16:22:11.197104, event: all_read, op: pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476378 7f869af78700 5 -- op tracker -- , seq: 1751, time: 2014-06-09 16:22:11.476346, event: dispatched, op: pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476385 7f869af78700 5 -- op tracker -- , seq: 1751, time: 2014-06-09 16:22:11.476383, event: waiting_for_osdmap, op: pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476390 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.476394 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a7200 2014-06-09 16:22:11.476400 7f869af78700 5 -- op tracker -- , seq: 1751, time: 2014-06-09 16:22:11.476397, event: started, op: pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476412 7f869af78700 5 -- op tracker -- , seq: 1751, time: 2014-06-09 16:22:11.476410, event: done, op: pg_log(0.652 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476433 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476435 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476436 7f869af78700 20 osd.26 243 _dispatch 0x3045b200 pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.476442 7f869af78700 5 -- op tracker -- , seq: 1752, time: 2014-06-09 16:22:11.026503, event: header_read, op: pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476450 7f869af78700 5 -- op tracker -- , seq: 1752, time: 2014-06-09 16:22:11.026504, event: throttled, op: pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476457 7f869af78700 5 -- op tracker -- , seq: 1752, time: 2014-06-09 16:22:11.026527, event: all_read, op: pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476464 7f869af78700 5 -- op tracker -- , seq: 1752, time: 2014-06-09 16:22:11.476433, event: dispatched, op: pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476471 7f869af78700 5 -- op tracker -- , seq: 1752, time: 2014-06-09 16:22:11.476468, event: waiting_for_osdmap, op: pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476476 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.476479 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045b200 2014-06-09 16:22:11.476485 7f869af78700 5 -- op tracker -- , seq: 1752, time: 2014-06-09 16:22:11.476482, event: started, op: pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476497 7f869af78700 5 -- op tracker -- , seq: 1752, time: 2014-06-09 16:22:11.476494, event: done, op: pg_log(0.616 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476505 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476506 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476519 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476520 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476521 7f869af78700 20 osd.26 243 _dispatch 0x302a2400 pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.476528 7f869af78700 5 -- op tracker -- , seq: 1753, time: 2014-06-09 16:22:11.015814, event: header_read, op: pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476536 7f869af78700 5 -- op tracker -- , seq: 1753, time: 2014-06-09 16:22:11.015814, event: throttled, op: pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476543 7f869af78700 5 -- op tracker -- , seq: 1753, time: 2014-06-09 16:22:11.015826, event: all_read, op: pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476550 7f869af78700 5 -- op tracker -- , seq: 1753, time: 2014-06-09 16:22:11.476518, event: dispatched, op: pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476557 7f869af78700 5 -- op tracker -- , seq: 1753, time: 2014-06-09 16:22:11.476554, event: waiting_for_osdmap, op: pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476561 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.476565 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x302a2400 2014-06-09 16:22:11.476570 7f869af78700 5 -- op tracker -- , seq: 1753, time: 2014-06-09 16:22:11.476567, event: started, op: pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476584 7f869af78700 5 -- op tracker -- , seq: 1753, time: 2014-06-09 16:22:11.476581, event: done, op: pg_log(1.4fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476591 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476593 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476605 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476607 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476608 7f869af78700 20 osd.26 243 _dispatch 0x308d8700 pg_info(1 pgs e243:1.300) v4 2014-06-09 16:22:11.476613 7f869af78700 5 -- op tracker -- , seq: 1754, time: 2014-06-09 16:22:11.239398, event: header_read, op: pg_info(1 pgs e243:1.300) 2014-06-09 16:22:11.476620 7f869af78700 5 -- op tracker -- , seq: 1754, time: 2014-06-09 16:22:11.239399, event: throttled, op: pg_info(1 pgs e243:1.300) 2014-06-09 16:22:11.476626 7f869af78700 5 -- op tracker -- , seq: 1754, time: 2014-06-09 16:22:11.239408, event: all_read, op: pg_info(1 pgs e243:1.300) 2014-06-09 16:22:11.476632 7f869af78700 5 -- op tracker -- , seq: 1754, time: 2014-06-09 16:22:11.476605, event: dispatched, op: pg_info(1 pgs e243:1.300) 2014-06-09 16:22:11.476639 7f869af78700 5 -- op tracker -- , seq: 1754, time: 2014-06-09 16:22:11.476637, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.300) 2014-06-09 16:22:11.476644 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.300) v4 from osd.7 2014-06-09 16:22:11.476647 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d8700 2014-06-09 16:22:11.476651 7f869af78700 5 -- op tracker -- , seq: 1754, time: 2014-06-09 16:22:11.476649, event: started, op: pg_info(1 pgs e243:1.300) 2014-06-09 16:22:11.476670 7f869af78700 5 -- op tracker -- , seq: 1754, time: 2014-06-09 16:22:11.476668, event: done, op: pg_info(1 pgs e243:1.300) 2014-06-09 16:22:11.476677 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476679 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476693 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476695 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476696 7f869af78700 20 osd.26 243 _dispatch 0x30570600 pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.476703 7f869af78700 5 -- op tracker -- , seq: 1755, time: 2014-06-09 16:22:11.169740, event: header_read, op: pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476711 7f869af78700 5 -- op tracker -- , seq: 1755, time: 2014-06-09 16:22:11.169740, event: throttled, op: pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476717 7f869af78700 5 -- op tracker -- , seq: 1755, time: 2014-06-09 16:22:11.169755, event: all_read, op: pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476724 7f869af78700 5 -- op tracker -- , seq: 1755, time: 2014-06-09 16:22:11.476693, event: dispatched, op: pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476732 7f869af78700 5 -- op tracker -- , seq: 1755, time: 2014-06-09 16:22:11.476730, event: waiting_for_osdmap, op: pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476737 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.476741 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30570600 2014-06-09 16:22:11.476747 7f869af78700 5 -- op tracker -- , seq: 1755, time: 2014-06-09 16:22:11.476744, event: started, op: pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476760 7f869af78700 5 -- op tracker -- , seq: 1755, time: 2014-06-09 16:22:11.476757, event: done, op: pg_log(0.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476768 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476770 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476781 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476782 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476784 7f869af78700 20 osd.26 243 _dispatch 0x3043a380 pg_notify(2.2da(21) epoch 243) v5 2014-06-09 16:22:11.476789 7f869af78700 5 -- op tracker -- , seq: 1756, time: 2014-06-09 16:22:11.127042, event: header_read, op: pg_notify(2.2da(21) epoch 243) 2014-06-09 16:22:11.476796 7f869af78700 5 -- op tracker -- , seq: 1756, time: 2014-06-09 16:22:11.127043, event: throttled, op: pg_notify(2.2da(21) epoch 243) 2014-06-09 16:22:11.476802 7f869af78700 5 -- op tracker -- , seq: 1756, time: 2014-06-09 16:22:11.127070, event: all_read, op: pg_notify(2.2da(21) epoch 243) 2014-06-09 16:22:11.476809 7f869af78700 5 -- op tracker -- , seq: 1756, time: 2014-06-09 16:22:11.476780, event: dispatched, op: pg_notify(2.2da(21) epoch 243) 2014-06-09 16:22:11.476815 7f869af78700 5 -- op tracker -- , seq: 1756, time: 2014-06-09 16:22:11.476813, event: waiting_for_osdmap, op: pg_notify(2.2da(21) epoch 243) 2014-06-09 16:22:11.476820 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.476822 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043a380 2014-06-09 16:22:11.476826 7f869af78700 5 -- op tracker -- , seq: 1756, time: 2014-06-09 16:22:11.476824, event: started, op: pg_notify(2.2da(21) epoch 243) 2014-06-09 16:22:11.476843 7f869af78700 5 -- op tracker -- , seq: 1756, time: 2014-06-09 16:22:11.476840, event: done, op: pg_notify(2.2da(21) epoch 243) 2014-06-09 16:22:11.476851 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476852 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476873 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476874 7f869af78700 20 osd.26 243 _dispatch 0x308b7600 pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.476881 7f869af78700 5 -- op tracker -- , seq: 1757, time: 2014-06-09 16:22:11.196458, event: header_read, op: pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476889 7f869af78700 5 -- op tracker -- , seq: 1757, time: 2014-06-09 16:22:11.196459, event: throttled, op: pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476896 7f869af78700 5 -- op tracker -- , seq: 1757, time: 2014-06-09 16:22:11.196469, event: all_read, op: pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476903 7f869af78700 5 -- op tracker -- , seq: 1757, time: 2014-06-09 16:22:11.476871, event: dispatched, op: pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476910 7f869af78700 5 -- op tracker -- , seq: 1757, time: 2014-06-09 16:22:11.476907, event: waiting_for_osdmap, op: pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476915 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.476918 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308b7600 2014-06-09 16:22:11.476924 7f869af78700 5 -- op tracker -- , seq: 1757, time: 2014-06-09 16:22:11.476921, event: started, op: pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476936 7f869af78700 5 -- op tracker -- , seq: 1757, time: 2014-06-09 16:22:11.476934, event: done, op: pg_log(0.6d2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476944 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476945 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476957 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.476958 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.476960 7f869af78700 20 osd.26 243 _dispatch 0x30612000 pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.476967 7f869af78700 5 -- op tracker -- , seq: 1758, time: 2014-06-09 16:22:11.144106, event: header_read, op: pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476975 7f869af78700 5 -- op tracker -- , seq: 1758, time: 2014-06-09 16:22:11.144107, event: throttled, op: pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476981 7f869af78700 5 -- op tracker -- , seq: 1758, time: 2014-06-09 16:22:11.144122, event: all_read, op: pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476988 7f869af78700 5 -- op tracker -- , seq: 1758, time: 2014-06-09 16:22:11.476957, event: dispatched, op: pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.476996 7f869af78700 5 -- op tracker -- , seq: 1758, time: 2014-06-09 16:22:11.476993, event: waiting_for_osdmap, op: pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477001 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.477005 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30612000 2014-06-09 16:22:11.477011 7f869af78700 5 -- op tracker -- , seq: 1758, time: 2014-06-09 16:22:11.477008, event: started, op: pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477023 7f869af78700 5 -- op tracker -- , seq: 1758, time: 2014-06-09 16:22:11.477020, event: done, op: pg_log(1.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477030 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477032 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477047 7f869af78700 20 osd.26 243 _dispatch 0x305bd000 pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.477053 7f869af78700 5 -- op tracker -- , seq: 1759, time: 2014-06-09 16:22:11.145890, event: header_read, op: pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477060 7f869af78700 5 -- op tracker -- , seq: 1759, time: 2014-06-09 16:22:11.145891, event: throttled, op: pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477067 7f869af78700 5 -- op tracker -- , seq: 1759, time: 2014-06-09 16:22:11.145907, event: all_read, op: pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477074 7f869af78700 5 -- op tracker -- , seq: 1759, time: 2014-06-09 16:22:11.477044, event: dispatched, op: pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477081 7f869af78700 5 -- op tracker -- , seq: 1759, time: 2014-06-09 16:22:11.477078, event: waiting_for_osdmap, op: pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477086 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.477089 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305bd000 2014-06-09 16:22:11.477096 7f869af78700 5 -- op tracker -- , seq: 1759, time: 2014-06-09 16:22:11.477093, event: started, op: pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477109 7f869af78700 5 -- op tracker -- , seq: 1759, time: 2014-06-09 16:22:11.477106, event: done, op: pg_log(0.3a6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477117 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477119 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477130 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477132 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477133 7f869af78700 20 osd.26 243 _dispatch 0x3004f600 pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.477139 7f869af78700 5 -- op tracker -- , seq: 1760, time: 2014-06-09 16:22:11.134011, event: header_read, op: pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477147 7f869af78700 5 -- op tracker -- , seq: 1760, time: 2014-06-09 16:22:11.134012, event: throttled, op: pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477154 7f869af78700 5 -- op tracker -- , seq: 1760, time: 2014-06-09 16:22:11.134026, event: all_read, op: pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477161 7f869af78700 5 -- op tracker -- , seq: 1760, time: 2014-06-09 16:22:11.477130, event: dispatched, op: pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477168 7f869af78700 5 -- op tracker -- , seq: 1760, time: 2014-06-09 16:22:11.477165, event: waiting_for_osdmap, op: pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477173 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.477177 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004f600 2014-06-09 16:22:11.477183 7f869af78700 5 -- op tracker -- , seq: 1760, time: 2014-06-09 16:22:11.477180, event: started, op: pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477195 7f869af78700 5 -- op tracker -- , seq: 1760, time: 2014-06-09 16:22:11.477193, event: done, op: pg_log(0.55d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477203 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477205 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477218 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477219 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477220 7f869af78700 20 osd.26 243 _dispatch 0x307d5e00 pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.477227 7f869af78700 5 -- op tracker -- , seq: 1761, time: 2014-06-09 16:22:11.211539, event: header_read, op: pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477235 7f869af78700 5 -- op tracker -- , seq: 1761, time: 2014-06-09 16:22:11.211540, event: throttled, op: pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477241 7f869af78700 5 -- op tracker -- , seq: 1761, time: 2014-06-09 16:22:11.211552, event: all_read, op: pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477248 7f869af78700 5 -- op tracker -- , seq: 1761, time: 2014-06-09 16:22:11.477217, event: dispatched, op: pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477255 7f869af78700 5 -- op tracker -- , seq: 1761, time: 2014-06-09 16:22:11.477253, event: waiting_for_osdmap, op: pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477260 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.477264 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307d5e00 2014-06-09 16:22:11.477269 7f869af78700 5 -- op tracker -- , seq: 1761, time: 2014-06-09 16:22:11.477267, event: started, op: pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477282 7f869af78700 5 -- op tracker -- , seq: 1761, time: 2014-06-09 16:22:11.477279, event: done, op: pg_log(1.3cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477289 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477291 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477302 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477303 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477304 7f869af78700 20 osd.26 243 _dispatch 0x3043bdc0 pg_notify(2.614(23) epoch 243) v5 2014-06-09 16:22:11.477309 7f869af78700 5 -- op tracker -- , seq: 1762, time: 2014-06-09 16:22:11.026544, event: header_read, op: pg_notify(2.614(23) epoch 243) 2014-06-09 16:22:11.477315 7f869af78700 5 -- op tracker -- , seq: 1762, time: 2014-06-09 16:22:11.026545, event: throttled, op: pg_notify(2.614(23) epoch 243) 2014-06-09 16:22:11.477322 7f869af78700 5 -- op tracker -- , seq: 1762, time: 2014-06-09 16:22:11.026575, event: all_read, op: pg_notify(2.614(23) epoch 243) 2014-06-09 16:22:11.477328 7f869af78700 5 -- op tracker -- , seq: 1762, time: 2014-06-09 16:22:11.477301, event: dispatched, op: pg_notify(2.614(23) epoch 243) 2014-06-09 16:22:11.477335 7f869af78700 5 -- op tracker -- , seq: 1762, time: 2014-06-09 16:22:11.477333, event: waiting_for_osdmap, op: pg_notify(2.614(23) epoch 243) 2014-06-09 16:22:11.477340 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.477341 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043bdc0 2014-06-09 16:22:11.477347 7f869af78700 5 -- op tracker -- , seq: 1762, time: 2014-06-09 16:22:11.477345, event: started, op: pg_notify(2.614(23) epoch 243) 2014-06-09 16:22:11.477362 7f869af78700 5 -- op tracker -- , seq: 1762, time: 2014-06-09 16:22:11.477360, event: done, op: pg_notify(2.614(23) epoch 243) 2014-06-09 16:22:11.477370 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477372 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477390 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477392 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477393 7f869af78700 20 osd.26 243 _dispatch 0x11c67c00 pg_notify(1.4ef(20) epoch 243) v5 2014-06-09 16:22:11.477397 7f869af78700 5 -- op tracker -- , seq: 1763, time: 2014-06-09 16:22:11.015836, event: header_read, op: pg_notify(1.4ef(20) epoch 243) 2014-06-09 16:22:11.477405 7f869af78700 5 -- op tracker -- , seq: 1763, time: 2014-06-09 16:22:11.015836, event: throttled, op: pg_notify(1.4ef(20) epoch 243) 2014-06-09 16:22:11.477412 7f869af78700 5 -- op tracker -- , seq: 1763, time: 2014-06-09 16:22:11.015860, event: all_read, op: pg_notify(1.4ef(20) epoch 243) 2014-06-09 16:22:11.477418 7f869af78700 5 -- op tracker -- , seq: 1763, time: 2014-06-09 16:22:11.477390, event: dispatched, op: pg_notify(1.4ef(20) epoch 243) 2014-06-09 16:22:11.477426 7f869af78700 5 -- op tracker -- , seq: 1763, time: 2014-06-09 16:22:11.477423, event: waiting_for_osdmap, op: pg_notify(1.4ef(20) epoch 243) 2014-06-09 16:22:11.477430 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.477432 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x11c67c00 2014-06-09 16:22:11.477437 7f869af78700 5 -- op tracker -- , seq: 1763, time: 2014-06-09 16:22:11.477435, event: started, op: pg_notify(1.4ef(20) epoch 243) 2014-06-09 16:22:11.477453 7f869af78700 5 -- op tracker -- , seq: 1763, time: 2014-06-09 16:22:11.477450, event: done, op: pg_notify(1.4ef(20) epoch 243) 2014-06-09 16:22:11.477464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477466 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477481 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477483 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477484 7f869af78700 20 osd.26 243 _dispatch 0x1a038fc0 pg_info(1 pgs e243:2.4d) v4 2014-06-09 16:22:11.477489 7f869af78700 5 -- op tracker -- , seq: 1764, time: 2014-06-09 16:22:11.239481, event: header_read, op: pg_info(1 pgs e243:2.4d) 2014-06-09 16:22:11.477496 7f869af78700 5 -- op tracker -- , seq: 1764, time: 2014-06-09 16:22:11.239482, event: throttled, op: pg_info(1 pgs e243:2.4d) 2014-06-09 16:22:11.477502 7f869af78700 5 -- op tracker -- , seq: 1764, time: 2014-06-09 16:22:11.239495, event: all_read, op: pg_info(1 pgs e243:2.4d) 2014-06-09 16:22:11.477508 7f869af78700 5 -- op tracker -- , seq: 1764, time: 2014-06-09 16:22:11.477481, event: dispatched, op: pg_info(1 pgs e243:2.4d) 2014-06-09 16:22:11.477515 7f869af78700 5 -- op tracker -- , seq: 1764, time: 2014-06-09 16:22:11.477513, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4d) 2014-06-09 16:22:11.477520 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4d) v4 from osd.7 2014-06-09 16:22:11.477523 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a038fc0 2014-06-09 16:22:11.477528 7f869af78700 5 -- op tracker -- , seq: 1764, time: 2014-06-09 16:22:11.477526, event: started, op: pg_info(1 pgs e243:2.4d) 2014-06-09 16:22:11.477543 7f869af78700 5 -- op tracker -- , seq: 1764, time: 2014-06-09 16:22:11.477541, event: done, op: pg_info(1 pgs e243:2.4d) 2014-06-09 16:22:11.477551 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477554 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477566 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477568 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477569 7f869af78700 20 osd.26 243 _dispatch 0x154588c0 pg_notify(2.5f1(20) epoch 243) v5 2014-06-09 16:22:11.477574 7f869af78700 5 -- op tracker -- , seq: 1765, time: 2014-06-09 16:22:11.173564, event: header_read, op: pg_notify(2.5f1(20) epoch 243) 2014-06-09 16:22:11.477581 7f869af78700 5 -- op tracker -- , seq: 1765, time: 2014-06-09 16:22:11.173564, event: throttled, op: pg_notify(2.5f1(20) epoch 243) 2014-06-09 16:22:11.477587 7f869af78700 5 -- op tracker -- , seq: 1765, time: 2014-06-09 16:22:11.173584, event: all_read, op: pg_notify(2.5f1(20) epoch 243) 2014-06-09 16:22:11.477593 7f869af78700 5 -- op tracker -- , seq: 1765, time: 2014-06-09 16:22:11.477566, event: dispatched, op: pg_notify(2.5f1(20) epoch 243) 2014-06-09 16:22:11.477600 7f869af78700 5 -- op tracker -- , seq: 1765, time: 2014-06-09 16:22:11.477598, event: waiting_for_osdmap, op: pg_notify(2.5f1(20) epoch 243) 2014-06-09 16:22:11.477604 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.477606 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x154588c0 2014-06-09 16:22:11.477611 7f869af78700 5 -- op tracker -- , seq: 1765, time: 2014-06-09 16:22:11.477609, event: started, op: pg_notify(2.5f1(20) epoch 243) 2014-06-09 16:22:11.477627 7f869af78700 5 -- op tracker -- , seq: 1765, time: 2014-06-09 16:22:11.477625, event: done, op: pg_notify(2.5f1(20) epoch 243) 2014-06-09 16:22:11.477635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477637 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477654 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477656 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477658 7f869af78700 20 osd.26 243 _dispatch 0x1a039880 pg_notify(3.709(12) epoch 243) v5 2014-06-09 16:22:11.477663 7f869af78700 5 -- op tracker -- , seq: 1766, time: 2014-06-09 16:22:11.148283, event: header_read, op: pg_notify(3.709(12) epoch 243) 2014-06-09 16:22:11.477670 7f869af78700 5 -- op tracker -- , seq: 1766, time: 2014-06-09 16:22:11.148283, event: throttled, op: pg_notify(3.709(12) epoch 243) 2014-06-09 16:22:11.477676 7f869af78700 5 -- op tracker -- , seq: 1766, time: 2014-06-09 16:22:11.148306, event: all_read, op: pg_notify(3.709(12) epoch 243) 2014-06-09 16:22:11.477682 7f869af78700 5 -- op tracker -- , seq: 1766, time: 2014-06-09 16:22:11.477654, event: dispatched, op: pg_notify(3.709(12) epoch 243) 2014-06-09 16:22:11.477689 7f869af78700 5 -- op tracker -- , seq: 1766, time: 2014-06-09 16:22:11.477687, event: waiting_for_osdmap, op: pg_notify(3.709(12) epoch 243) 2014-06-09 16:22:11.477693 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.477695 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a039880 2014-06-09 16:22:11.477700 7f869af78700 5 -- op tracker -- , seq: 1766, time: 2014-06-09 16:22:11.477698, event: started, op: pg_notify(3.709(12) epoch 243) 2014-06-09 16:22:11.477717 7f869af78700 5 -- op tracker -- , seq: 1766, time: 2014-06-09 16:22:11.477715, event: done, op: pg_notify(3.709(12) epoch 243) 2014-06-09 16:22:11.477725 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477727 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477743 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477745 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477746 7f869af78700 20 osd.26 243 _dispatch 0x308b7000 pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.477753 7f869af78700 5 -- op tracker -- , seq: 1767, time: 2014-06-09 16:22:11.196539, event: header_read, op: pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477760 7f869af78700 5 -- op tracker -- , seq: 1767, time: 2014-06-09 16:22:11.196539, event: throttled, op: pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477767 7f869af78700 5 -- op tracker -- , seq: 1767, time: 2014-06-09 16:22:11.196547, event: all_read, op: pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477774 7f869af78700 5 -- op tracker -- , seq: 1767, time: 2014-06-09 16:22:11.477743, event: dispatched, op: pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477782 7f869af78700 5 -- op tracker -- , seq: 1767, time: 2014-06-09 16:22:11.477779, event: waiting_for_osdmap, op: pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477786 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.477790 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308b7000 2014-06-09 16:22:11.477796 7f869af78700 5 -- op tracker -- , seq: 1767, time: 2014-06-09 16:22:11.477794, event: started, op: pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477809 7f869af78700 5 -- op tracker -- , seq: 1767, time: 2014-06-09 16:22:11.477806, event: done, op: pg_log(0.219 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477817 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477819 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477832 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477833 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477834 7f869af78700 20 osd.26 243 _dispatch 0x3070f600 pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.477841 7f869af78700 5 -- op tracker -- , seq: 1768, time: 2014-06-09 16:22:11.144143, event: header_read, op: pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477849 7f869af78700 5 -- op tracker -- , seq: 1768, time: 2014-06-09 16:22:11.144143, event: throttled, op: pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477856 7f869af78700 5 -- op tracker -- , seq: 1768, time: 2014-06-09 16:22:11.144161, event: all_read, op: pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477863 7f869af78700 5 -- op tracker -- , seq: 1768, time: 2014-06-09 16:22:11.477831, event: dispatched, op: pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477870 7f869af78700 5 -- op tracker -- , seq: 1768, time: 2014-06-09 16:22:11.477868, event: waiting_for_osdmap, op: pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477875 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.477879 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070f600 2014-06-09 16:22:11.477884 7f869af78700 5 -- op tracker -- , seq: 1768, time: 2014-06-09 16:22:11.477882, event: started, op: pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477898 7f869af78700 5 -- op tracker -- , seq: 1768, time: 2014-06-09 16:22:11.477895, event: done, op: pg_log(2.5b7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.477906 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477908 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477921 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.477923 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.477924 7f869af78700 20 osd.26 243 _dispatch 0x19eb9180 pg_notify(2.39f(23),1.668(23),3.672(10) epoch 243) v5 2014-06-09 16:22:11.477931 7f869af78700 5 -- op tracker -- , seq: 1769, time: 2014-06-09 16:22:11.145999, event: header_read, op: pg_notify(2.39f(23),1.668(23),3.672(10) epoch 243) 2014-06-09 16:22:11.477938 7f869af78700 5 -- op tracker -- , seq: 1769, time: 2014-06-09 16:22:11.146000, event: throttled, op: pg_notify(2.39f(23),1.668(23),3.672(10) epoch 243) 2014-06-09 16:22:11.477945 7f869af78700 5 -- op tracker -- , seq: 1769, time: 2014-06-09 16:22:11.146060, event: all_read, op: pg_notify(2.39f(23),1.668(23),3.672(10) epoch 243) 2014-06-09 16:22:11.477952 7f869af78700 5 -- op tracker -- , seq: 1769, time: 2014-06-09 16:22:11.477920, event: dispatched, op: pg_notify(2.39f(23),1.668(23),3.672(10) epoch 243) 2014-06-09 16:22:11.477959 7f869af78700 5 -- op tracker -- , seq: 1769, time: 2014-06-09 16:22:11.477957, event: waiting_for_osdmap, op: pg_notify(2.39f(23),1.668(23),3.672(10) epoch 243) 2014-06-09 16:22:11.477964 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.477967 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb9180 2014-06-09 16:22:11.477972 7f869af78700 5 -- op tracker -- , seq: 1769, time: 2014-06-09 16:22:11.477969, event: started, op: pg_notify(2.39f(23),1.668(23),3.672(10) epoch 243) 2014-06-09 16:22:11.478005 7f869af78700 5 -- op tracker -- , seq: 1769, time: 2014-06-09 16:22:11.478002, event: done, op: pg_notify(2.39f(23),1.668(23),3.672(10) epoch 243) 2014-06-09 16:22:11.478014 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478016 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478042 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478044 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478045 7f869af78700 20 osd.26 243 _dispatch 0x29ec4540 pg_notify(0.29d(21) epoch 243) v5 2014-06-09 16:22:11.478050 7f869af78700 5 -- op tracker -- , seq: 1770, time: 2014-06-09 16:22:11.134075, event: header_read, op: pg_notify(0.29d(21) epoch 243) 2014-06-09 16:22:11.478057 7f869af78700 5 -- op tracker -- , seq: 1770, time: 2014-06-09 16:22:11.134075, event: throttled, op: pg_notify(0.29d(21) epoch 243) 2014-06-09 16:22:11.478064 7f869af78700 5 -- op tracker -- , seq: 1770, time: 2014-06-09 16:22:11.134109, event: all_read, op: pg_notify(0.29d(21) epoch 243) 2014-06-09 16:22:11.478070 7f869af78700 5 -- op tracker -- , seq: 1770, time: 2014-06-09 16:22:11.478042, event: dispatched, op: pg_notify(0.29d(21) epoch 243) 2014-06-09 16:22:11.478077 7f869af78700 5 -- op tracker -- , seq: 1770, time: 2014-06-09 16:22:11.478075, event: waiting_for_osdmap, op: pg_notify(0.29d(21) epoch 243) 2014-06-09 16:22:11.478082 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.478084 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec4540 2014-06-09 16:22:11.478088 7f869af78700 5 -- op tracker -- , seq: 1770, time: 2014-06-09 16:22:11.478086, event: started, op: pg_notify(0.29d(21) epoch 243) 2014-06-09 16:22:11.478103 7f869af78700 5 -- op tracker -- , seq: 1770, time: 2014-06-09 16:22:11.478101, event: done, op: pg_notify(0.29d(21) epoch 243) 2014-06-09 16:22:11.478111 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478113 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478132 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478134 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478135 7f869af78700 20 osd.26 243 _dispatch 0x29ec5500 pg_notify(0.fa(20),3.ff(10) epoch 243) v5 2014-06-09 16:22:11.478142 7f869af78700 5 -- op tracker -- , seq: 1771, time: 2014-06-09 16:22:11.224732, event: header_read, op: pg_notify(0.fa(20),3.ff(10) epoch 243) 2014-06-09 16:22:11.478150 7f869af78700 5 -- op tracker -- , seq: 1771, time: 2014-06-09 16:22:11.224732, event: throttled, op: pg_notify(0.fa(20),3.ff(10) epoch 243) 2014-06-09 16:22:11.478156 7f869af78700 5 -- op tracker -- , seq: 1771, time: 2014-06-09 16:22:11.224773, event: all_read, op: pg_notify(0.fa(20),3.ff(10) epoch 243) 2014-06-09 16:22:11.478162 7f869af78700 5 -- op tracker -- , seq: 1771, time: 2014-06-09 16:22:11.478132, event: dispatched, op: pg_notify(0.fa(20),3.ff(10) epoch 243) 2014-06-09 16:22:11.478169 7f869af78700 5 -- op tracker -- , seq: 1771, time: 2014-06-09 16:22:11.478167, event: waiting_for_osdmap, op: pg_notify(0.fa(20),3.ff(10) epoch 243) 2014-06-09 16:22:11.478174 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.478176 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec5500 2014-06-09 16:22:11.478181 7f869af78700 5 -- op tracker -- , seq: 1771, time: 2014-06-09 16:22:11.478178, event: started, op: pg_notify(0.fa(20),3.ff(10) epoch 243) 2014-06-09 16:22:11.478206 7f869af78700 5 -- op tracker -- , seq: 1771, time: 2014-06-09 16:22:11.478203, event: done, op: pg_notify(0.fa(20),3.ff(10) epoch 243) 2014-06-09 16:22:11.478214 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478216 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478245 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478247 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478248 7f869af78700 20 osd.26 243 _dispatch 0x3045ac00 pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.478254 7f869af78700 5 -- op tracker -- , seq: 1772, time: 2014-06-09 16:22:11.026580, event: header_read, op: pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478262 7f869af78700 5 -- op tracker -- , seq: 1772, time: 2014-06-09 16:22:11.026580, event: throttled, op: pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478269 7f869af78700 5 -- op tracker -- , seq: 1772, time: 2014-06-09 16:22:11.026596, event: all_read, op: pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478276 7f869af78700 5 -- op tracker -- , seq: 1772, time: 2014-06-09 16:22:11.478244, event: dispatched, op: pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478283 7f869af78700 5 -- op tracker -- , seq: 1772, time: 2014-06-09 16:22:11.478281, event: waiting_for_osdmap, op: pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478288 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.478292 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045ac00 2014-06-09 16:22:11.478298 7f869af78700 5 -- op tracker -- , seq: 1772, time: 2014-06-09 16:22:11.478295, event: started, op: pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478312 7f869af78700 5 -- op tracker -- , seq: 1772, time: 2014-06-09 16:22:11.478309, event: done, op: pg_log(1.ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478320 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478321 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478334 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478335 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478337 7f869af78700 20 osd.26 243 _dispatch 0x302a1e00 pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.478344 7f869af78700 5 -- op tracker -- , seq: 1773, time: 2014-06-09 16:22:11.015868, event: header_read, op: pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478351 7f869af78700 5 -- op tracker -- , seq: 1773, time: 2014-06-09 16:22:11.015868, event: throttled, op: pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478359 7f869af78700 5 -- op tracker -- , seq: 1773, time: 2014-06-09 16:22:11.015876, event: all_read, op: pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478366 7f869af78700 5 -- op tracker -- , seq: 1773, time: 2014-06-09 16:22:11.478333, event: dispatched, op: pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478374 7f869af78700 5 -- op tracker -- , seq: 1773, time: 2014-06-09 16:22:11.478371, event: waiting_for_osdmap, op: pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478378 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.478382 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x302a1e00 2014-06-09 16:22:11.478388 7f869af78700 5 -- op tracker -- , seq: 1773, time: 2014-06-09 16:22:11.478385, event: started, op: pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478401 7f869af78700 5 -- op tracker -- , seq: 1773, time: 2014-06-09 16:22:11.478398, event: done, op: pg_log(0.31f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478409 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478411 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478423 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478424 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478425 7f869af78700 20 osd.26 243 _dispatch 0x10d028c0 pg_notify(2.74b(20) epoch 243) v5 2014-06-09 16:22:11.478430 7f869af78700 5 -- op tracker -- , seq: 1774, time: 2014-06-09 16:22:11.241934, event: header_read, op: pg_notify(2.74b(20) epoch 243) 2014-06-09 16:22:11.478437 7f869af78700 5 -- op tracker -- , seq: 1774, time: 2014-06-09 16:22:11.241935, event: throttled, op: pg_notify(2.74b(20) epoch 243) 2014-06-09 16:22:11.478443 7f869af78700 5 -- op tracker -- , seq: 1774, time: 2014-06-09 16:22:11.241962, event: all_read, op: pg_notify(2.74b(20) epoch 243) 2014-06-09 16:22:11.478450 7f869af78700 5 -- op tracker -- , seq: 1774, time: 2014-06-09 16:22:11.478422, event: dispatched, op: pg_notify(2.74b(20) epoch 243) 2014-06-09 16:22:11.478457 7f869af78700 5 -- op tracker -- , seq: 1774, time: 2014-06-09 16:22:11.478454, event: waiting_for_osdmap, op: pg_notify(2.74b(20) epoch 243) 2014-06-09 16:22:11.478461 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.478463 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d028c0 2014-06-09 16:22:11.478468 7f869af78700 5 -- op tracker -- , seq: 1774, time: 2014-06-09 16:22:11.478466, event: started, op: pg_notify(2.74b(20) epoch 243) 2014-06-09 16:22:11.478485 7f869af78700 5 -- op tracker -- , seq: 1774, time: 2014-06-09 16:22:11.478482, event: done, op: pg_notify(2.74b(20) epoch 243) 2014-06-09 16:22:11.478493 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478495 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478513 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478515 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478516 7f869af78700 20 osd.26 243 _dispatch 0x30573600 pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.478523 7f869af78700 5 -- op tracker -- , seq: 1775, time: 2014-06-09 16:22:11.174632, event: header_read, op: pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478530 7f869af78700 5 -- op tracker -- , seq: 1775, time: 2014-06-09 16:22:11.174632, event: throttled, op: pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478537 7f869af78700 5 -- op tracker -- , seq: 1775, time: 2014-06-09 16:22:11.174647, event: all_read, op: pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478544 7f869af78700 5 -- op tracker -- , seq: 1775, time: 2014-06-09 16:22:11.478513, event: dispatched, op: pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478551 7f869af78700 5 -- op tracker -- , seq: 1775, time: 2014-06-09 16:22:11.478548, event: waiting_for_osdmap, op: pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478556 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.478560 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30573600 2014-06-09 16:22:11.478566 7f869af78700 5 -- op tracker -- , seq: 1775, time: 2014-06-09 16:22:11.478563, event: started, op: pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478579 7f869af78700 5 -- op tracker -- , seq: 1775, time: 2014-06-09 16:22:11.478576, event: done, op: pg_log(2.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478588 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478590 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478604 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478606 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478607 7f869af78700 20 osd.26 243 _dispatch 0x1ae7c540 pg_notify(2.401(26),0.403(22) epoch 243) v5 2014-06-09 16:22:11.478613 7f869af78700 5 -- op tracker -- , seq: 1776, time: 2014-06-09 16:22:11.148325, event: header_read, op: pg_notify(2.401(26),0.403(22) epoch 243) 2014-06-09 16:22:11.478620 7f869af78700 5 -- op tracker -- , seq: 1776, time: 2014-06-09 16:22:11.148325, event: throttled, op: pg_notify(2.401(26),0.403(22) epoch 243) 2014-06-09 16:22:11.478627 7f869af78700 5 -- op tracker -- , seq: 1776, time: 2014-06-09 16:22:11.148377, event: all_read, op: pg_notify(2.401(26),0.403(22) epoch 243) 2014-06-09 16:22:11.478634 7f869af78700 5 -- op tracker -- , seq: 1776, time: 2014-06-09 16:22:11.478604, event: dispatched, op: pg_notify(2.401(26),0.403(22) epoch 243) 2014-06-09 16:22:11.478641 7f869af78700 5 -- op tracker -- , seq: 1776, time: 2014-06-09 16:22:11.478638, event: waiting_for_osdmap, op: pg_notify(2.401(26),0.403(22) epoch 243) 2014-06-09 16:22:11.478645 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.478647 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ae7c540 2014-06-09 16:22:11.478652 7f869af78700 5 -- op tracker -- , seq: 1776, time: 2014-06-09 16:22:11.478650, event: started, op: pg_notify(2.401(26),0.403(22) epoch 243) 2014-06-09 16:22:11.478677 7f869af78700 5 -- op tracker -- , seq: 1776, time: 2014-06-09 16:22:11.478674, event: done, op: pg_notify(2.401(26),0.403(22) epoch 243) 2014-06-09 16:22:11.478685 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478688 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478716 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478717 7f869af78700 20 osd.26 243 _dispatch 0x3088da40 pg_notify(2.59f(23) epoch 243) v5 2014-06-09 16:22:11.478722 7f869af78700 5 -- op tracker -- , seq: 1777, time: 2014-06-09 16:22:11.197841, event: header_read, op: pg_notify(2.59f(23) epoch 243) 2014-06-09 16:22:11.478729 7f869af78700 5 -- op tracker -- , seq: 1777, time: 2014-06-09 16:22:11.197842, event: throttled, op: pg_notify(2.59f(23) epoch 243) 2014-06-09 16:22:11.478735 7f869af78700 5 -- op tracker -- , seq: 1777, time: 2014-06-09 16:22:11.197870, event: all_read, op: pg_notify(2.59f(23) epoch 243) 2014-06-09 16:22:11.478742 7f869af78700 5 -- op tracker -- , seq: 1777, time: 2014-06-09 16:22:11.478714, event: dispatched, op: pg_notify(2.59f(23) epoch 243) 2014-06-09 16:22:11.478749 7f869af78700 5 -- op tracker -- , seq: 1777, time: 2014-06-09 16:22:11.478747, event: waiting_for_osdmap, op: pg_notify(2.59f(23) epoch 243) 2014-06-09 16:22:11.478753 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.478755 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088da40 2014-06-09 16:22:11.478760 7f869af78700 5 -- op tracker -- , seq: 1777, time: 2014-06-09 16:22:11.478758, event: started, op: pg_notify(2.59f(23) epoch 243) 2014-06-09 16:22:11.478776 7f869af78700 5 -- op tracker -- , seq: 1777, time: 2014-06-09 16:22:11.478774, event: done, op: pg_notify(2.59f(23) epoch 243) 2014-06-09 16:22:11.478785 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478786 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478805 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478807 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478808 7f869af78700 20 osd.26 243 _dispatch 0x3070f000 pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.478815 7f869af78700 5 -- op tracker -- , seq: 1778, time: 2014-06-09 16:22:11.144224, event: header_read, op: pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478823 7f869af78700 5 -- op tracker -- , seq: 1778, time: 2014-06-09 16:22:11.144225, event: throttled, op: pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478830 7f869af78700 5 -- op tracker -- , seq: 1778, time: 2014-06-09 16:22:11.144245, event: all_read, op: pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478837 7f869af78700 5 -- op tracker -- , seq: 1778, time: 2014-06-09 16:22:11.478805, event: dispatched, op: pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478845 7f869af78700 5 -- op tracker -- , seq: 1778, time: 2014-06-09 16:22:11.478842, event: waiting_for_osdmap, op: pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478849 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.478853 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070f000 2014-06-09 16:22:11.478858 7f869af78700 5 -- op tracker -- , seq: 1778, time: 2014-06-09 16:22:11.478856, event: started, op: pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478871 7f869af78700 5 -- op tracker -- , seq: 1778, time: 2014-06-09 16:22:11.478868, event: done, op: pg_log(1.2ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478878 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478880 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478892 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478893 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478894 7f869af78700 20 osd.26 243 _dispatch 0x30572a00 pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.478900 7f869af78700 5 -- op tracker -- , seq: 1779, time: 2014-06-09 16:22:11.146074, event: header_read, op: pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478908 7f869af78700 5 -- op tracker -- , seq: 1779, time: 2014-06-09 16:22:11.146074, event: throttled, op: pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478915 7f869af78700 5 -- op tracker -- , seq: 1779, time: 2014-06-09 16:22:11.146084, event: all_read, op: pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478922 7f869af78700 5 -- op tracker -- , seq: 1779, time: 2014-06-09 16:22:11.478891, event: dispatched, op: pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478931 7f869af78700 5 -- op tracker -- , seq: 1779, time: 2014-06-09 16:22:11.478927, event: waiting_for_osdmap, op: pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478935 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.478939 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30572a00 2014-06-09 16:22:11.478945 7f869af78700 5 -- op tracker -- , seq: 1779, time: 2014-06-09 16:22:11.478942, event: started, op: pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478958 7f869af78700 5 -- op tracker -- , seq: 1779, time: 2014-06-09 16:22:11.478955, event: done, op: pg_log(2.f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478965 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478967 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478979 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.478980 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.478982 7f869af78700 20 osd.26 243 _dispatch 0x306d4400 pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.478988 7f869af78700 5 -- op tracker -- , seq: 1780, time: 2014-06-09 16:22:11.134114, event: header_read, op: pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.478996 7f869af78700 5 -- op tracker -- , seq: 1780, time: 2014-06-09 16:22:11.134114, event: throttled, op: pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479003 7f869af78700 5 -- op tracker -- , seq: 1780, time: 2014-06-09 16:22:11.134138, event: all_read, op: pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479010 7f869af78700 5 -- op tracker -- , seq: 1780, time: 2014-06-09 16:22:11.478978, event: dispatched, op: pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479018 7f869af78700 5 -- op tracker -- , seq: 1780, time: 2014-06-09 16:22:11.479016, event: waiting_for_osdmap, op: pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479023 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.479027 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d4400 2014-06-09 16:22:11.479032 7f869af78700 5 -- op tracker -- , seq: 1780, time: 2014-06-09 16:22:11.479030, event: started, op: pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479045 7f869af78700 5 -- op tracker -- , seq: 1780, time: 2014-06-09 16:22:11.479042, event: done, op: pg_log(2.57a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479053 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.479055 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.479068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.479070 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.479071 7f869af78700 20 osd.26 243 _dispatch 0x30883000 pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.479077 7f869af78700 5 -- op tracker -- , seq: 1781, time: 2014-06-09 16:22:11.238406, event: header_read, op: pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479085 7f869af78700 5 -- op tracker -- , seq: 1781, time: 2014-06-09 16:22:11.238407, event: throttled, op: pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479092 7f869af78700 5 -- op tracker -- , seq: 1781, time: 2014-06-09 16:22:11.238417, event: all_read, op: pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479099 7f869af78700 5 -- op tracker -- , seq: 1781, time: 2014-06-09 16:22:11.479068, event: dispatched, op: pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479107 7f869af78700 5 -- op tracker -- , seq: 1781, time: 2014-06-09 16:22:11.479104, event: waiting_for_osdmap, op: pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479112 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.479116 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30883000 2014-06-09 16:22:11.479121 7f869af78700 5 -- op tracker -- , seq: 1781, time: 2014-06-09 16:22:11.479118, event: started, op: pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479134 7f869af78700 5 -- op tracker -- , seq: 1781, time: 2014-06-09 16:22:11.479131, event: done, op: pg_log(2.487 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479142 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.479144 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.479155 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.479157 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.479158 7f869af78700 20 osd.26 243 _dispatch 0x10c38380 pg_info(1 pgs e243:3.37a(4)) v4 2014-06-09 16:22:11.479163 7f869af78700 5 -- op tracker -- , seq: 1782, time: 2014-06-09 16:22:11.027632, event: header_read, op: pg_info(1 pgs e243:3.37a(4)) 2014-06-09 16:22:11.479170 7f869af78700 5 -- op tracker -- , seq: 1782, time: 2014-06-09 16:22:11.027633, event: throttled, op: pg_info(1 pgs e243:3.37a(4)) 2014-06-09 16:22:11.479177 7f869af78700 5 -- op tracker -- , seq: 1782, time: 2014-06-09 16:22:11.027655, event: all_read, op: pg_info(1 pgs e243:3.37a(4)) 2014-06-09 16:22:11.479183 7f869af78700 5 -- op tracker -- , seq: 1782, time: 2014-06-09 16:22:11.479155, event: dispatched, op: pg_info(1 pgs e243:3.37a(4)) 2014-06-09 16:22:11.479190 7f869af78700 5 -- op tracker -- , seq: 1782, time: 2014-06-09 16:22:11.479188, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.37a(4)) 2014-06-09 16:22:11.479195 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.37a(4)) v4 from osd.2 2014-06-09 16:22:11.479198 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10c38380 2014-06-09 16:22:11.479203 7f869af78700 5 -- op tracker -- , seq: 1782, time: 2014-06-09 16:22:11.479201, event: started, op: pg_info(1 pgs e243:3.37a(4)) 2014-06-09 16:22:11.479220 7f869af78700 5 -- op tracker -- , seq: 1782, time: 2014-06-09 16:22:11.479217, event: done, op: pg_info(1 pgs e243:3.37a(4)) 2014-06-09 16:22:11.479227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.479229 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.479242 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.479244 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.479245 7f869af78700 20 osd.26 243 _dispatch 0x302a1800 pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.479253 7f869af78700 5 -- op tracker -- , seq: 1783, time: 2014-06-09 16:22:11.015879, event: header_read, op: pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479260 7f869af78700 5 -- op tracker -- , seq: 1783, time: 2014-06-09 16:22:11.015880, event: throttled, op: pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479268 7f869af78700 5 -- op tracker -- , seq: 1783, time: 2014-06-09 16:22:11.015892, event: all_read, op: pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479274 7f869af78700 5 -- op tracker -- , seq: 1783, time: 2014-06-09 16:22:11.479242, event: dispatched, op: pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479282 7f869af78700 5 -- op tracker -- , seq: 1783, time: 2014-06-09 16:22:11.479280, event: waiting_for_osdmap, op: pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.479314 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0240 (8 active) 2014-06-09 16:22:11.487955 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.487976 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.487990 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.487995 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.488000 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.276763 7 0.000236 2014-06-09 16:22:11.488013 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.488027 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.488037 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.488048 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.488057 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.488111 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.488119 7f8690f64700 5 queue_transactions existing osr(1.32f 0x384d350)/0x384d350 2014-06-09 16:22:11.464329 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e360 (0 active) 2014-06-09 16:22:11.488130 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287560 (1 active) 2014-06-09 16:22:11.488136 7f86a5e11700 5 _do_op 0x2fc8ca80 seq 3140 osr(0.dd 0x2bd6550)/0x2bd6550 start 2014-06-09 16:22:11.488141 7f86a5e11700 10 _do_transaction on 0xc33a540 2014-06-09 16:22:11.488150 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.489064 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (9 active) 2014-06-09 16:22:11.489073 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.489087 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 1.a7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.489099 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.489103 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.489106 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 0.990535 7 0.000230 2014-06-09 16:22:11.489117 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.489129 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.489139 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.489149 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.489157 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.489212 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.489218 7f8692767700 5 queue_transactions existing osr(1.a7 0x384c730)/0x384c730 2014-06-09 16:22:11.489224 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcfc0 (2 active) 2014-06-09 16:22:11.489229 7f86a5610700 5 _do_op 0x2fc8ca20 seq 3141 osr(3.609 0x3ed1b00)/0x3ed1b00 start 2014-06-09 16:22:11.489233 7f86a5610700 10 _do_transaction on 0xc145b00 2014-06-09 16:22:11.489241 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.489264 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0260 (10 active) 2014-06-09 16:22:11.489271 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:2.2da( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.489286 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.489298 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.489303 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.489314 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.6d2( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.489325 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.489329 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.489332 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.755323 7 0.000190 2014-06-09 16:22:11.489342 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.489352 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.489362 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.489372 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.489380 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.489431 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.489437 7f868ff62700 5 queue_transactions existing osr(0.6d2 0x384c1f0)/0x384c1f0 2014-06-09 16:22:11.489444 7f868df5e700 10 op_submit_finish 3148 2014-06-09 16:22:11.489451 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'533 (0'0,214'533] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MLogRec from 4 2014-06-09 16:22:11.489467 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'533 (0'0,214'533] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd4 2014-06-09 16:22:11.489479 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'533 (0'0,214'533] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.489489 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'533 (0'0,214'533] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.489497 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'533 (0'0,214'533] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.4: log((214'533,214'536], crt=0'0) missing(0) 2014-06-09 16:22:11.489508 7f868df5e700 10 merge_log log((214'533,214'536], crt=0'0) from osd.4 into log((0'0,214'533], crt=0'0) 2014-06-09 16:22:11.489512 7f868df5e700 10 merge_log extending head to 214'536 2014-06-09 16:22:11.489513 7f868df5e700 20 ? 214'536 (214'535) modify 735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3 by client.4539.0:34414 2014-06-09 10:18:56.040384 2014-06-09 16:22:11.489521 7f868df5e700 20 ? 214'535 (214'534) modify 735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3 by client.4539.0:34413 2014-06-09 10:18:52.385844 2014-06-09 16:22:11.489528 7f868df5e700 20 ? 214'534 (214'533) modify 735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3 by client.4539.0:34412 2014-06-09 10:18:47.634670 2014-06-09 16:22:11.489534 7f868df5e700 20 merge_log 214'534 (214'533) modify 735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3 by client.4539.0:34412 2014-06-09 10:18:47.634670 2014-06-09 16:22:11.489544 7f868df5e700 20 merge_log 214'535 (214'534) modify 735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3 by client.4539.0:34413 2014-06-09 10:18:52.385844 2014-06-09 16:22:11.489555 7f868df5e700 20 merge_log 214'536 (214'535) modify 735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3 by client.4539.0:34414 2014-06-09 10:18:56.040384 2014-06-09 16:22:11.489756 7f868df5e700 10 merge_log result log((0'0,214'536], crt=214'536) missing(1) changed=1 2014-06-09 16:22:11.489761 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 peering m=1] peer osd.4 now 3.65d( v 214'536 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) missing(0) 2014-06-09 16:22:11.489782 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.623545 5 0.000245 2014-06-09 16:22:11.489797 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.489808 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 peering m=1] state: osd.4 has no missing, identical log 2014-06-09 16:22:11.489824 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000027 0 0.000000 2014-06-09 16:22:11.489836 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.489846 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.246927 0 0.000000 2014-06-09 16:22:11.489861 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.489870 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.489882 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.489892 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.489900 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'533 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.490007 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.490021 7f868df5e700 10 osd.26 243 queue_for_recovery queued pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.490035 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 3.65d( v 214'536 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) 2014-06-09 16:22:11.490048 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 is up to date, queueing in pending_activators 2014-06-09 16:22:11.490064 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 3.65d( v 214'536 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242) uptodate 2014-06-09 16:22:11.490078 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.490090 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3 214'536 also missing on osd.26 2014-06-09 16:22:11.490101 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3=214'536(214'533)} 2014-06-09 16:22:11.490113 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 4 as a source 2014-06-09 16:22:11.490123 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3 214'536 is on osd.4 2014-06-09 16:22:11.490136 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {735e6e5d/rbd_data.115b6b8b4567.00000000000018ac/head//3=214'536(214'533)} 2014-06-09 16:22:11.490149 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.490160 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 203 2014-06-09 16:22:11.490172 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 4 2014-06-09 16:22:11.490183 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.490194 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.490232 7f868df5e700 10 write_log with: dirty_to: 0'0, dirty_from: 214'533, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.490278 7f868df5e700 5 queue_transactions existing osr(3.65d 0x3ed1a30)/0x3ed1a30 2014-06-09 16:22:11.490281 7f868df5e700 10 op_submit_start 3149 2014-06-09 16:22:11.490282 7f868df5e700 5 queue_transactions (trailing journal) 3149 2014-06-09 16:22:11.490284 7f868df5e700 5 queue_op 0x241044e0 seq 3149 osr(3.65d 0x3ed1a30) 2160 bytes (queue has 19 ops and 34015 bytes) 2014-06-09 16:22:11.490287 7f868df5e700 10 op_submit_finish 3149 2014-06-09 16:22:11.490290 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.490302 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.16c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.490314 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.490318 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.490322 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] exit Started/Stray 0.631839 7 0.000184 2014-06-09 16:22:11.490332 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.490342 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.490352 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.490361 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.490369 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.490415 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.490420 7f868df5e700 5 queue_transactions existing osr(0.16c 0x2bd67f0)/0x2bd67f0 2014-06-09 16:22:11.490423 7f868df5e700 10 op_submit_start 3150 2014-06-09 16:22:11.490424 7f868df5e700 5 queue_transactions (trailing journal) 3150 2014-06-09 16:22:11.490425 7f868df5e700 5 queue_op 0x30060060 seq 3150 osr(0.16c 0x2bd67f0) 1789 bytes (queue has 20 ops and 35804 bytes) 2014-06-09 16:22:11.490428 7f868df5e700 10 op_submit_finish 3150 2014-06-09 16:22:11.490431 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.490442 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 1.697( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.490454 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.490458 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.490462 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.697( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.218448 7 0.000188 2014-06-09 16:22:11.490472 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.697( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.490483 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.697( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.490493 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.490503 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.490511 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.490557 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.490562 7f868df5e700 5 queue_transactions existing osr(1.697 0x384de10)/0x384de10 2014-06-09 16:22:11.490565 7f868df5e700 10 op_submit_start 3151 2014-06-09 16:22:11.490566 7f868df5e700 5 queue_transactions (trailing journal) 3151 2014-06-09 16:22:11.490567 7f868df5e700 5 queue_op 0x30060000 seq 3151 osr(1.697 0x384de10) 1974 bytes (queue has 21 ops and 37778 bytes) 2014-06-09 16:22:11.490570 7f868df5e700 10 op_submit_finish 3151 2014-06-09 16:22:11.490573 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=191 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.490587 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=191 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.490600 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=191 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.212699 7 0.000191 2014-06-09 16:22:11.490612 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=191 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.490626 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=191 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.490636 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=191 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.490646 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'667 -> 213'667 2014-06-09 16:22:11.490689 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.490733 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.490738 7f868df5e700 5 queue_transactions existing osr(3.39e 0x3ed0fa0)/0x3ed0fa0 2014-06-09 16:22:11.490742 7f868df5e700 10 op_submit_start 3152 2014-06-09 16:22:11.490743 7f868df5e700 5 queue_transactions (trailing journal) 3152 2014-06-09 16:22:11.490745 7f868df5e700 5 queue_op 0x2fcae8a0 seq 3152 osr(3.39e 0x3ed0fa0) 1429 bytes (queue has 22 ops and 39207 bytes) 2014-06-09 16:22:11.490748 7f868df5e700 10 op_submit_finish 3152 2014-06-09 16:22:11.490751 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=191 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.490764 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=191 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.490778 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=191 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.175923 7 0.006485 2014-06-09 16:22:11.490789 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=191 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.490800 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=191 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.490811 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=191 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.490822 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'544 -> 213'544 2014-06-09 16:22:11.490858 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.490899 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.490903 7f868df5e700 5 queue_transactions existing osr(3.66a 0x3ed19d0)/0x3ed19d0 2014-06-09 16:22:11.490906 7f868df5e700 10 op_submit_start 3153 2014-06-09 16:22:11.490907 7f868df5e700 5 queue_transactions (trailing journal) 3153 2014-06-09 16:22:11.490908 7f868df5e700 5 queue_op 0x2fcae840 seq 3153 osr(3.66a 0x3ed19d0) 1429 bytes (queue has 23 ops and 40636 bytes) 2014-06-09 16:22:11.490911 7f868df5e700 10 op_submit_finish 3153 2014-06-09 16:22:11.490914 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.490926 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.c9( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.490938 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.490942 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.490946 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.253297 7 0.000188 2014-06-09 16:22:11.490956 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.490966 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.490975 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.490984 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.490993 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.491041 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.491045 7f868df5e700 5 queue_transactions existing osr(2.c9 0x2ed04a0)/0x2ed04a0 2014-06-09 16:22:11.491048 7f868df5e700 10 op_submit_start 3154 2014-06-09 16:22:11.491048 7f868df5e700 5 queue_transactions (trailing journal) 3154 2014-06-09 16:22:11.491050 7f868df5e700 5 queue_op 0x2fcae7e0 seq 3154 osr(2.c9 0x2ed04a0) 1951 bytes (queue has 24 ops and 42587 bytes) 2014-06-09 16:22:11.491052 7f868df5e700 10 op_submit_finish 3154 2014-06-09 16:22:11.491055 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.491066 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.348( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491077 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491081 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.491084 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.348( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.279793 7 0.000195 2014-06-09 16:22:11.491094 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.348( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.491104 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.348( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.491114 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.491123 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.491131 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.491179 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.491184 7f868df5e700 5 queue_transactions existing osr(1.348 0x384d310)/0x384d310 2014-06-09 16:22:11.491186 7f868df5e700 10 op_submit_start 3155 2014-06-09 16:22:11.491187 7f868df5e700 5 queue_transactions (trailing journal) 3155 2014-06-09 16:22:11.491188 7f868df5e700 5 queue_op 0x2fcae780 seq 3155 osr(1.348 0x384d310) 1903 bytes (queue has 25 ops and 44490 bytes) 2014-06-09 16:22:11.491191 7f868df5e700 10 op_submit_finish 3155 2014-06-09 16:22:11.491194 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=208 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.491208 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=208 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.491221 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=208 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.269112 7 0.000186 2014-06-09 16:22:11.491232 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=208 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.491243 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=208 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.491254 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=208 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.491264 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'510 -> 213'510 2014-06-09 16:22:11.491299 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.491339 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.491343 7f868df5e700 5 queue_transactions existing osr(3.6f 0x3ed0280)/0x3ed0280 2014-06-09 16:22:11.491345 7f868df5e700 10 op_submit_start 3156 2014-06-09 16:22:11.491346 7f868df5e700 5 queue_transactions (trailing journal) 3156 2014-06-09 16:22:11.491347 7f868df5e700 5 queue_op 0x2fcae720 seq 3156 osr(3.6f 0x3ed0280) 1332 bytes (queue has 26 ops and 45822 bytes) 2014-06-09 16:22:11.491350 7f868df5e700 10 op_submit_finish 3156 2014-06-09 16:22:11.491353 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.491364 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.380( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491376 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491379 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.491383 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.380( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.036879 7 0.000221 2014-06-09 16:22:11.491394 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.380( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.491405 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.380( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.491414 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.491423 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.491430 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.491475 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.491479 7f868df5e700 5 queue_transactions existing osr(0.380 0x2bd6f20)/0x2bd6f20 2014-06-09 16:22:11.491482 7f868df5e700 10 op_submit_start 3157 2014-06-09 16:22:11.491483 7f868df5e700 5 queue_transactions (trailing journal) 3157 2014-06-09 16:22:11.491484 7f868df5e700 5 queue_op 0x2fcae6c0 seq 3157 osr(0.380 0x2bd6f20) 1883 bytes (queue has 27 ops and 47705 bytes) 2014-06-09 16:22:11.491487 7f868df5e700 10 op_submit_finish 3157 2014-06-09 16:22:11.491490 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.491502 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.151( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491515 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491519 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.491522 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.151( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.789831 7 0.004215 2014-06-09 16:22:11.491532 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.151( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.491542 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.151( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.491552 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.491561 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.491569 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.491637 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (11 active) 2014-06-09 16:22:11.491644 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.491657 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491668 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491672 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.491676 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.260007 7 0.000197 2014-06-09 16:22:11.491689 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.491699 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.491708 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.491717 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.491726 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.491775 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.491779 7f8692f68700 5 queue_transactions existing osr(0.31f 0x2bd6d50)/0x2bd6d50 2014-06-09 16:22:11.491782 7f8692f68700 10 op_submit_start 3158 2014-06-09 16:22:11.491783 7f8692f68700 5 queue_transactions (trailing journal) 3158 2014-06-09 16:22:11.491784 7f8692f68700 5 queue_op 0x30606060 seq 3158 osr(0.31f 0x2bd6d50) 1903 bytes (queue has 28 ops and 49608 bytes) 2014-06-09 16:22:11.491787 7f8692f68700 10 op_submit_finish 3158 2014-06-09 16:22:11.491790 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:2.74b( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.491803 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.491814 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.491820 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.491831 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.5f8( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491842 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.491846 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.491850 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.748467 7 0.000175 2014-06-09 16:22:11.491859 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.491870 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.491879 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.491887 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.491895 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.491943 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.491948 7f8692f68700 5 queue_transactions existing osr(2.5f8 0x2ed1db0)/0x2ed1db0 2014-06-09 16:22:11.491951 7f8692f68700 10 op_submit_start 3159 2014-06-09 16:22:11.491952 7f8692f68700 5 queue_transactions (trailing journal) 3159 2014-06-09 16:22:11.491953 7f8692f68700 5 queue_op 0x309a3800 seq 3159 osr(2.5f8 0x2ed1db0) 2001 bytes (queue has 29 ops and 51609 bytes) 2014-06-09 16:22:11.491956 7f8692f68700 10 op_submit_finish 3159 2014-06-09 16:22:11.491958 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:2.401( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.491971 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.491982 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.491987 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:0.403( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.491999 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.492009 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.492013 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:2.59f( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.492024 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.492034 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.492038 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.492049 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.2ef( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492060 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492064 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.492067 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.790261 7 0.000201 2014-06-09 16:22:11.492079 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.492089 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.492099 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.492108 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.492116 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.492163 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.492167 7f8692f68700 5 queue_transactions existing osr(1.2ef 0x384d0f0)/0x384d0f0 2014-06-09 16:22:11.492170 7f8692f68700 10 op_submit_start 3160 2014-06-09 16:22:11.492171 7f8692f68700 5 queue_transactions (trailing journal) 3160 2014-06-09 16:22:11.492173 7f8692f68700 5 queue_op 0x309a37a0 seq 3160 osr(1.2ef 0x384d0f0) 2009 bytes (queue has 30 ops and 53618 bytes) 2014-06-09 16:22:11.492176 7f8692f68700 10 op_submit_finish 3160 2014-06-09 16:22:11.492179 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.492190 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 2.f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492203 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492207 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.492210 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.235917 7 0.000188 2014-06-09 16:22:11.492220 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.492230 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.492239 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.492248 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.492256 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.492301 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.492306 7f8692f68700 5 queue_transactions existing osr(2.f2 0x2ed07e0)/0x2ed07e0 2014-06-09 16:22:11.492310 7f8692f68700 10 op_submit_start 3161 2014-06-09 16:22:11.492311 7f8692f68700 5 queue_transactions (trailing journal) 3161 2014-06-09 16:22:11.492312 7f8692f68700 5 queue_op 0x309a3740 seq 3161 osr(2.f2 0x2ed07e0) 1998 bytes (queue has 31 ops and 55616 bytes) 2014-06-09 16:22:11.492315 7f8692f68700 10 op_submit_finish 3161 2014-06-09 16:22:11.492318 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.492330 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.57a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492342 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492345 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.492349 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.043338 7 0.000172 2014-06-09 16:22:11.492359 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.492369 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.492379 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.492388 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.492396 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.492442 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.492447 7f8692f68700 5 queue_transactions existing osr(2.57a 0x2ed1870)/0x2ed1870 2014-06-09 16:22:11.492450 7f8692f68700 10 op_submit_start 3162 2014-06-09 16:22:11.492451 7f8692f68700 5 queue_transactions (trailing journal) 3162 2014-06-09 16:22:11.492452 7f8692f68700 5 queue_op 0x309a36e0 seq 3162 osr(2.57a 0x2ed1870) 1883 bytes (queue has 32 ops and 57499 bytes) 2014-06-09 16:22:11.492455 7f8692f68700 10 op_submit_finish 3162 2014-06-09 16:22:11.492457 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.492469 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.487( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492480 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492484 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.492487 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.487( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.600207 7 0.000206 2014-06-09 16:22:11.492497 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.487( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.492506 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.487( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.492515 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.492524 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.492532 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.492580 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.492585 7f8692f68700 5 queue_transactions existing osr(2.487 0x2ed14b0)/0x2ed14b0 2014-06-09 16:22:11.492588 7f8692f68700 10 op_submit_start 3163 2014-06-09 16:22:11.492589 7f8692f68700 5 queue_transactions (trailing journal) 3163 2014-06-09 16:22:11.492590 7f8692f68700 5 queue_op 0x309a3680 seq 3163 osr(2.487 0x2ed14b0) 2001 bytes (queue has 33 ops and 59500 bytes) 2014-06-09 16:22:11.492593 7f8692f68700 10 op_submit_finish 3163 2014-06-09 16:22:11.492596 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=191 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.492608 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=191 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.492622 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=191 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.249158 7 0.000190 2014-06-09 16:22:11.492633 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=191 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.492645 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=191 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.492655 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=191 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.492665 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'559 -> 213'559 2014-06-09 16:22:11.492702 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.492743 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.492748 7f8692f68700 5 queue_transactions existing osr(3.37a 0x3ed1040)/0x3ed1040 2014-06-09 16:22:11.492751 7f8692f68700 10 op_submit_start 3164 2014-06-09 16:22:11.492752 7f8692f68700 5 queue_transactions (trailing journal) 3164 2014-06-09 16:22:11.492754 7f8692f68700 5 queue_op 0x309a3620 seq 3164 osr(3.37a 0x3ed1040) 1429 bytes (queue has 34 ops and 60929 bytes) 2014-06-09 16:22:11.492756 7f8692f68700 10 op_submit_finish 3164 2014-06-09 16:22:11.492765 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (10 active) 2014-06-09 16:22:11.492769 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.492775 7f868f761700 10 op_submit_start 3165 2014-06-09 16:22:11.492777 7f868f761700 5 queue_transactions (trailing journal) 3165 2014-06-09 16:22:11.492778 7f868f761700 5 queue_op 0x309a3da0 seq 3165 osr(1.382 0x384d5b0) 1883 bytes (queue has 34 ops and 60929 bytes) 2014-06-09 16:22:11.492781 7f868f761700 10 op_submit_finish 3165 2014-06-09 16:22:11.492788 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:0.67d( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.492801 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.492813 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.492818 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:2.755( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.492829 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.492839 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.492843 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:3.1c7( v 213'647 (0'0,213'647] local-les=191 n=131 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.492857 7f868f761700 7 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.492868 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.1c7( v 213'647 (0'0,213'647] local-les=191 n=131 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.492889 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.492893 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:0.4b3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.492905 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.492915 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.492920 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.492930 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 1.45d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492941 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.492944 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.492948 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 0.811922 7 0.003321 2014-06-09 16:22:11.492958 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.492967 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.492977 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.492986 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.492994 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.493045 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.493050 7f868f761700 5 queue_transactions existing osr(1.45d 0x384d670)/0x384d670 2014-06-09 16:22:11.493053 7f868f761700 10 op_submit_start 3166 2014-06-09 16:22:11.493054 7f868f761700 5 queue_transactions (trailing journal) 3166 2014-06-09 16:22:11.493055 7f868f761700 5 queue_op 0x309a3d40 seq 3166 osr(1.45d 0x384d670) 2099 bytes (queue has 35 ops and 63028 bytes) 2014-06-09 16:22:11.493057 7f868f761700 10 op_submit_finish 3166 2014-06-09 16:22:11.493060 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.493071 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.69c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493083 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493086 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.493090 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] exit Started/Stray 1.214624 7 0.000199 2014-06-09 16:22:11.493100 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.493110 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.493120 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.493130 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.493137 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.493181 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.493186 7f868f761700 5 queue_transactions existing osr(2.69c 0x2ed1f20)/0x2ed1f20 2014-06-09 16:22:11.493189 7f868f761700 10 op_submit_start 3167 2014-06-09 16:22:11.493190 7f868f761700 5 queue_transactions (trailing journal) 3167 2014-06-09 16:22:11.493192 7f868f761700 5 queue_op 0x309a3ce0 seq 3167 osr(2.69c 0x2ed1f20) 1789 bytes (queue has 36 ops and 64817 bytes) 2014-06-09 16:22:11.493194 7f868f761700 10 op_submit_finish 3167 2014-06-09 16:22:11.493197 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=191 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.493210 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=191 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.493223 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=191 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.313804 7 0.000175 2014-06-09 16:22:11.493234 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=191 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.493245 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=191 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.493255 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=191 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.493265 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'588 -> 213'588 2014-06-09 16:22:11.493303 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.493348 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.493352 7f868f761700 5 queue_transactions existing osr(3.7fc 0xb7f00e0)/0xb7f00e0 2014-06-09 16:22:11.493355 7f868f761700 10 op_submit_start 3168 2014-06-09 16:22:11.493356 7f868f761700 5 queue_transactions (trailing journal) 3168 2014-06-09 16:22:11.493357 7f868f761700 5 queue_op 0x309a3c80 seq 3168 osr(3.7fc 0xb7f00e0) 1429 bytes (queue has 37 ops and 66246 bytes) 2014-06-09 16:22:11.493360 7f868f761700 10 op_submit_finish 3168 2014-06-09 16:22:11.493363 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.493374 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 1.6e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493385 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493389 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.493392 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] exit Started/Stray 0.778209 7 0.000216 2014-06-09 16:22:11.493403 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.493413 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.493423 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.493433 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.493440 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.493488 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.493492 7f868f761700 5 queue_transactions existing osr(1.6e8 0x2ed0190)/0x2ed0190 2014-06-09 16:22:11.493495 7f868f761700 10 op_submit_start 3169 2014-06-09 16:22:11.493496 7f868f761700 5 queue_transactions (trailing journal) 3169 2014-06-09 16:22:11.493497 7f868f761700 5 queue_op 0x309a3c20 seq 3169 osr(1.6e8 0x2ed0190) 2052 bytes (queue has 38 ops and 68298 bytes) 2014-06-09 16:22:11.493500 7f868f761700 10 op_submit_finish 3169 2014-06-09 16:22:11.493503 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.493516 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.493529 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.272478 7 0.008909 2014-06-09 16:22:11.493540 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.493551 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.493561 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.493572 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'629 -> 213'629 2014-06-09 16:22:11.493614 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.493653 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.493658 7f868f761700 5 queue_transactions existing osr(3.616 0x3ed1a90)/0x3ed1a90 2014-06-09 16:22:11.493660 7f868f761700 10 op_submit_start 3170 2014-06-09 16:22:11.493661 7f868f761700 5 queue_transactions (trailing journal) 3170 2014-06-09 16:22:11.493663 7f868f761700 5 queue_op 0x309a3bc0 seq 3170 osr(3.616 0x3ed1a90) 1335 bytes (queue has 39 ops and 69633 bytes) 2014-06-09 16:22:11.493666 7f868f761700 10 op_submit_finish 3170 2014-06-09 16:22:11.493669 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.493680 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 2.34d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493692 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493696 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.493700 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.281916 7 0.000185 2014-06-09 16:22:11.493710 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.493719 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.493729 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.493738 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.493745 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.493792 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.493797 7f868f761700 5 queue_transactions existing osr(2.34d 0x2ed0e70)/0x2ed0e70 2014-06-09 16:22:11.493800 7f868f761700 10 op_submit_start 3171 2014-06-09 16:22:11.493801 7f868f761700 5 queue_transactions (trailing journal) 3171 2014-06-09 16:22:11.493802 7f868f761700 5 queue_op 0x309a3b60 seq 3171 osr(2.34d 0x2ed0e70) 1974 bytes (queue has 40 ops and 71607 bytes) 2014-06-09 16:22:11.493805 7f868f761700 10 op_submit_finish 3171 2014-06-09 16:22:11.493808 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.493819 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.e6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493830 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493834 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.493837 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.238299 7 0.005122 2014-06-09 16:22:11.493846 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.493856 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.493865 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.493874 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.493883 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.493929 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.493933 7f868f761700 5 queue_transactions existing osr(2.e6 0x2ed07d0)/0x2ed07d0 2014-06-09 16:22:11.493935 7f868f761700 10 op_submit_start 3172 2014-06-09 16:22:11.493936 7f868f761700 5 queue_transactions (trailing journal) 3172 2014-06-09 16:22:11.493938 7f868f761700 5 queue_op 0x309a3b00 seq 3172 osr(2.e6 0x2ed07d0) 1880 bytes (queue has 41 ops and 73487 bytes) 2014-06-09 16:22:11.493940 7f868f761700 10 op_submit_finish 3172 2014-06-09 16:22:11.493943 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.493955 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.fd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493966 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.493970 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.493973 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.228421 7 0.000175 2014-06-09 16:22:11.493983 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.493993 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.494003 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.494012 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.494020 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.494065 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.494070 7f868f761700 5 queue_transactions existing osr(2.fd 0x2ed0770)/0x2ed0770 2014-06-09 16:22:11.494072 7f868f761700 10 op_submit_start 3173 2014-06-09 16:22:11.494073 7f868f761700 5 queue_transactions (trailing journal) 3173 2014-06-09 16:22:11.494074 7f868f761700 5 queue_op 0x309a3aa0 seq 3173 osr(2.fd 0x2ed0770) 1998 bytes (queue has 42 ops and 75485 bytes) 2014-06-09 16:22:11.494077 7f868f761700 10 op_submit_finish 3173 2014-06-09 16:22:11.494080 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:1.75b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.494092 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.494102 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.494107 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.494118 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.25f( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.494129 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.494134 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.494138 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.084829 7 0.008842 2014-06-09 16:22:11.494148 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.494158 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.494166 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.494175 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.494183 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.494231 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.494236 7f868f761700 5 queue_transactions existing osr(0.25f 0x2bd6d00)/0x2bd6d00 2014-06-09 16:22:11.494238 7f868f761700 10 op_submit_start 3174 2014-06-09 16:22:11.494239 7f868f761700 5 queue_transactions (trailing journal) 3174 2014-06-09 16:22:11.494241 7f868f761700 5 queue_op 0x309a3a40 seq 3174 osr(0.25f 0x2bd6d00) 1997 bytes (queue has 43 ops and 77482 bytes) 2014-06-09 16:22:11.494243 7f868f761700 10 op_submit_finish 3174 2014-06-09 16:22:11.494246 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.3cf( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.494259 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.494268 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.494272 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.494282 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 0.3b7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.494294 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.494298 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.494302 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.874827 7 0.000185 2014-06-09 16:22:11.494311 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.494321 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.494332 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.494340 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.494348 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.494394 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.494398 7f868f761700 5 queue_transactions existing osr(0.3b7 0x2bd7290)/0x2bd7290 2014-06-09 16:22:11.494401 7f868f761700 10 op_submit_start 3175 2014-06-09 16:22:11.494402 7f868f761700 5 queue_transactions (trailing journal) 3175 2014-06-09 16:22:11.494403 7f868f761700 5 queue_op 0x309a39e0 seq 3175 osr(0.3b7 0x2bd7290) 1883 bytes (queue has 44 ops and 79365 bytes) 2014-06-09 16:22:11.494406 7f868f761700 10 op_submit_finish 3175 2014-06-09 16:22:11.494409 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.494420 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.5b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.494431 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.494435 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.494438 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 0.778744 7 0.000207 2014-06-09 16:22:11.494448 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.494459 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.494469 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.494478 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.494486 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.494530 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.494534 7f868f761700 5 queue_transactions existing osr(0.5b7 0x2bd7a90)/0x2bd7a90 2014-06-09 16:22:11.494536 7f868f761700 10 op_submit_start 3176 2014-06-09 16:22:11.494537 7f868f761700 5 queue_transactions (trailing journal) 3176 2014-06-09 16:22:11.494539 7f868f761700 5 queue_op 0x309a3980 seq 3176 osr(0.5b7 0x2bd7a90) 1876 bytes (queue has 45 ops and 81241 bytes) 2014-06-09 16:22:11.494541 7f868f761700 10 op_submit_finish 3176 2014-06-09 16:22:11.494544 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.629( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.494557 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.494571 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.494576 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.360( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.494589 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.494598 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.494614 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (9 active) 2014-06-09 16:22:11.494618 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.494626 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0xa41e000 (1 active) 2014-06-09 16:22:11.494629 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.494632 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0xa41e000 (0 active) 2014-06-09 16:22:11.494634 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.494640 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.494646 7f868ef60700 10 op_submit_start 3177 2014-06-09 16:22:11.494648 7f868ef60700 5 queue_transactions (trailing journal) 3177 2014-06-09 16:22:11.494649 7f868ef60700 5 queue_op 0x309a2a20 seq 3177 osr(2.3c0 0x2ed13f0) 1974 bytes (queue has 45 ops and 81241 bytes) 2014-06-09 16:22:11.494652 7f868ef60700 10 op_submit_finish 3177 2014-06-09 16:22:11.494658 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MLogRec from 2 2014-06-09 16:22:11.494674 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.2b( v 213'613 (0'0,213'613] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) log((0'0,213'613], crt=0'0) missing(0) 2014-06-09 16:22:11.494688 7f868ef60700 10 proc_replica_log for osd.2: 3.2b( v 213'613 (0'0,213'613] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) log((0'0,213'613], crt=0'0) missing(0) 2014-06-09 16:22:11.494694 7f868ef60700 20 merge_log cut point (usually last shared) is 213'613 (213'612) modify d874c02b/rbd_data.11626b8b4567.0000000000001860/head//3 by client.4546.0:35449 2014-06-09 10:17:59.324152 2014-06-09 16:22:11.495210 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.2b( v 213'613 (0'0,213'613] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.495229 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.495242 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.567332 6 0.000277 2014-06-09 16:22:11.495255 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.495265 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.334958 0 0.000000 2014-06-09 16:22:11.495278 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.495288 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.495301 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.495311 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.495319 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'623 -> 214'623 2014-06-09 16:22:11.495360 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.2b( v 213'613 (0'0,213'613] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.495388 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'613,214'623], crt=0'0) 2014-06-09 16:22:11.495401 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.495419 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.2b( v 214'623 lc 213'613 (0'0,214'623] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) missing missing(2) 2014-06-09 16:22:11.495434 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 2 missing 2014-06-09 16:22:11.495444 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 77df202b/rbd_data.116b6b8b4567.0000000000001645/head//3 214'623 is on osd.26 2014-06-09 16:22:11.495456 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing bb4282b/rbd_data.11726b8b4567.00000000000018ee/head//3 214'619 is on osd.26 2014-06-09 16:22:11.495467 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {77df202b/rbd_data.116b6b8b4567.0000000000001645/head//3=214'623,bb4282b/rbd_data.11726b8b4567.00000000000018ee/head//3=214'619} 2014-06-09 16:22:11.495478 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.495487 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 77df202b/rbd_data.116b6b8b4567.0000000000001645/head//3 214'623 also missing on osd.2 2014-06-09 16:22:11.495498 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing bb4282b/rbd_data.11726b8b4567.00000000000018ee/head//3 214'619 also missing on osd.2 2014-06-09 16:22:11.495508 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {77df202b/rbd_data.116b6b8b4567.0000000000001645/head//3=214'623,bb4282b/rbd_data.11726b8b4567.00000000000018ee/head//3=214'619} 2014-06-09 16:22:11.495518 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.495527 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.495539 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.495548 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.495558 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.495627 7f8691765700 10 op_submit_start 3178 2014-06-09 16:22:11.495629 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.495631 7f8691765700 5 queue_transactions (trailing journal) 3178 2014-06-09 16:22:11.495634 7f8691765700 5 queue_op 0x2c06e7e0 seq 3178 osr(0.681 0x2bd7e10) 1883 bytes (queue has 45 ops and 81241 bytes) 2014-06-09 16:22:11.495638 7f8691765700 10 op_submit_finish 3178 2014-06-09 16:22:11.495645 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.542( v 214'710 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.495663 7f8691765700 7 osd.26 pg_epoch: 243 pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.5 2014-06-09 16:22:11.495674 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.5 info 3.542( v 214'710 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242), identical to ours 2014-06-09 16:22:11.495695 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.495702 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.495710 7f868b759700 10 op_submit_start 3179 2014-06-09 16:22:11.495700 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7af( v 213'628 (0'0,213'628] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.7af( v 213'628 (0'0,213'628] local-les=191 n=120 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.495712 7f868b759700 5 queue_transactions (trailing journal) 3179 2014-06-09 16:22:11.495714 7f868b759700 5 queue_op 0x308caba0 seq 3179 osr(2.27a 0x2ed0d90) 2099 bytes (queue has 45 ops and 81241 bytes) 2014-06-09 16:22:11.495718 7f868b759700 10 op_submit_finish 3179 2014-06-09 16:22:11.495715 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7af( v 213'628 (0'0,213'628] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.495728 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.495723 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.495732 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:0.759( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.495735 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.a6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.495747 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.495751 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.495745 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.495755 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.495755 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 0.857177 7 0.000194 2014-06-09 16:22:11.495759 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:1.1c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.495766 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.495772 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.495782 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.495778 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.495786 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.495788 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.495800 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.495799 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 2.113( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.495812 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.495808 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.495816 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.495819 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.113( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.726267 7 0.000175 2014-06-09 16:22:11.495831 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.113( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.495842 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.113( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.495854 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.495852 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.495859 7f868b759700 5 queue_transactions existing osr(1.a6 0x384c750)/0x384c750 2014-06-09 16:22:11.495862 7f868b759700 10 op_submit_start 3180 2014-06-09 16:22:11.495864 7f868b759700 5 queue_transactions (trailing journal) 3180 2014-06-09 16:22:11.495865 7f868b759700 5 queue_op 0x308cab40 seq 3180 osr(1.a6 0x384c750) 1873 bytes (queue has 46 ops and 83114 bytes) 2014-06-09 16:22:11.495868 7f868b759700 10 op_submit_finish 3180 2014-06-09 16:22:11.495861 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.495871 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.495871 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.7e4( v 214'548 (0'0,214'548] local-les=208 n=108 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.495887 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.495902 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.495907 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=191 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) 2014-06-09 16:22:11.495920 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.495925 7f8691765700 5 queue_transactions existing osr(2.113 0x2ed06b0)/0x2ed06b0 2014-06-09 16:22:11.495928 7f8691765700 10 op_submit_start 3181 2014-06-09 16:22:11.495929 7f8691765700 5 queue_transactions (trailing journal) 3181 2014-06-09 16:22:11.495920 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=191 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.0 3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) 2014-06-09 16:22:11.495931 7f8691765700 5 queue_op 0x30607a40 seq 3181 osr(2.113 0x2ed06b0) 1954 bytes (queue has 47 ops and 85068 bytes) 2014-06-09 16:22:11.495934 7f8691765700 10 op_submit_finish 3181 2014-06-09 16:22:11.495933 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=191 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.222437 7 0.000166 2014-06-09 16:22:11.495936 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.ed( v 214'568 (0'0,214'568] local-les=201 n=105 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.495946 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=191 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.495952 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.495964 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.495957 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=191 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.495968 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.495969 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=191 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.495980 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'637 -> 213'637 2014-06-09 16:22:11.495980 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.495993 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.495997 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.496000 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.038515 7 0.000197 2014-06-09 16:22:11.496011 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.496022 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.496022 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.496032 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.496041 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.496050 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.496065 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.496070 7f868b759700 5 queue_transactions existing osr(3.d1 0x3ed0560)/0x3ed0560 2014-06-09 16:22:11.496073 7f868b759700 10 op_submit_start 3182 2014-06-09 16:22:11.496074 7f868b759700 5 queue_transactions (trailing journal) 3182 2014-06-09 16:22:11.496075 7f868b759700 5 queue_op 0x308caae0 seq 3182 osr(3.d1 0x3ed0560) 1426 bytes (queue has 48 ops and 86494 bytes) 2014-06-09 16:22:11.496078 7f868b759700 10 op_submit_finish 3182 2014-06-09 16:22:11.496081 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.496096 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.496101 7f8691765700 5 queue_transactions existing osr(2.f4 0x2ed0790)/0x2ed0790 2014-06-09 16:22:11.496092 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.52e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.496103 7f8691765700 10 op_submit_start 3183 2014-06-09 16:22:11.496105 7f8691765700 5 queue_transactions (trailing journal) 3183 2014-06-09 16:22:11.496104 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.496106 7f8691765700 5 queue_op 0x306079e0 seq 3183 osr(2.f4 0x2ed0790) 1880 bytes (queue has 49 ops and 88374 bytes) 2014-06-09 16:22:11.496109 7f8691765700 10 op_submit_finish 3183 2014-06-09 16:22:11.496108 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.496111 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.015454 7 0.000991 2014-06-09 16:22:11.496112 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.496122 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.496124 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.501( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.496136 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.496132 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.496140 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.496142 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.496144 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.501( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] exit Started/Stray 1.048314 7 0.001853 2014-06-09 16:22:11.496151 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.496155 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.501( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.496159 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.496166 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.501( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.496176 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.496185 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.496193 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.496207 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.496212 7f868b759700 5 queue_transactions existing osr(2.52e 0x2ed1600)/0x2ed1600 2014-06-09 16:22:11.496215 7f868b759700 10 op_submit_start 3184 2014-06-09 16:22:11.496216 7f868b759700 5 queue_transactions (trailing journal) 3184 2014-06-09 16:22:11.496217 7f868b759700 5 queue_op 0x308caa80 seq 3184 osr(2.52e 0x2ed1600) 1974 bytes (queue has 50 ops and 90348 bytes) 2014-06-09 16:22:11.496220 7f868b759700 10 op_submit_finish 3184 2014-06-09 16:22:11.496223 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.496241 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.496235 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.496246 7f8691765700 5 queue_transactions existing osr(2.501 0x2ed1690)/0x2ed1690 2014-06-09 16:22:11.496249 7f8691765700 2 waiting 51 > 50 ops || 92400 > 104857600 2014-06-09 16:22:11.496248 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.331951 7 0.000186 2014-06-09 16:22:11.496258 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.496271 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.496281 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.496292 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'629 -> 213'629 2014-06-09 16:22:11.496333 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.496376 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.496381 7f868b759700 5 queue_transactions existing osr(3.7ae 0xb7f01d0)/0xb7f01d0 2014-06-09 16:22:11.496383 7f868b759700 2 waiting 51 > 50 ops || 91777 > 104857600 2014-06-09 16:22:11.496391 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.496398 7f8690763700 10 op_submit_start 3185 2014-06-09 16:22:11.496400 7f8690763700 5 queue_transactions (trailing journal) 3185 2014-06-09 16:22:11.496402 7f8690763700 5 queue_op 0x30607560 seq 3185 osr(1.589 0x384da10) 2009 bytes (queue has 50 ops and 90348 bytes) 2014-06-09 16:22:11.496405 7f8690763700 10 op_submit_finish 3185 2014-06-09 16:22:11.496410 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.6e( v 214'612 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.496425 7f8690763700 7 osd.26 pg_epoch: 243 pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.3 2014-06-09 16:22:11.496435 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.3 info 3.6e( v 214'612 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242), identical to ours 2014-06-09 16:22:11.496455 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.496460 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.7d( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.496471 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.496481 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.496485 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:2.82( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.496496 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.496506 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.496511 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.496522 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.69d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.496533 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.496537 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.496540 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.188997 7 0.000178 2014-06-09 16:22:11.496551 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.496561 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.496570 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.496579 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.496588 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.496636 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.496641 7f8690763700 5 queue_transactions existing osr(2.69d 0x2ed1ef0)/0x2ed1ef0 2014-06-09 16:22:11.496644 7f8690763700 2 waiting 51 > 50 ops || 92231 > 104857600 2014-06-09 16:22:11.496652 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.496658 7f868e75f700 10 op_submit_start 3186 2014-06-09 16:22:11.496660 7f868e75f700 5 queue_transactions (trailing journal) 3186 2014-06-09 16:22:11.496661 7f868e75f700 5 queue_op 0x2fcae060 seq 3186 osr(2.d3 0x2ed0450) 1951 bytes (queue has 50 ops and 90348 bytes) 2014-06-09 16:22:11.496664 7f868e75f700 10 op_submit_finish 3186 2014-06-09 16:22:11.496670 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.496682 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.350( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.496694 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.496698 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.496702 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.350( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.274626 7 0.000191 2014-06-09 16:22:11.496713 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.350( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.496723 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.350( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.496733 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.496742 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.496750 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.496797 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.496802 7f868e75f700 5 queue_transactions existing osr(2.350 0x2ed0e50)/0x2ed0e50 2014-06-09 16:22:11.496804 7f868e75f700 2 waiting 51 > 50 ops || 92357 > 104857600 2014-06-09 16:22:11.496812 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.496820 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.496827 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.496833 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.479287 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.496839 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x302a1800 2014-06-09 16:22:11.496849 7f869af78700 5 -- op tracker -- , seq: 1783, time: 2014-06-09 16:22:11.496844, event: started, op: pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.496872 7f869af78700 5 -- op tracker -- , seq: 1783, time: 2014-06-09 16:22:11.496868, event: done, op: pg_log(0.5e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.496884 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.496886 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.496904 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.496906 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.496907 7f869af78700 20 osd.26 243 _dispatch 0xe046000 pg_notify(1.70c(23),2.17c(20) epoch 243) v5 2014-06-09 16:22:11.496913 7f869af78700 5 -- op tracker -- , seq: 1784, time: 2014-06-09 16:22:11.268198, event: header_read, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.496922 7f869af78700 5 -- op tracker -- , seq: 1784, time: 2014-06-09 16:22:11.268198, event: throttled, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.496929 7f869af78700 5 -- op tracker -- , seq: 1784, time: 2014-06-09 16:22:11.268237, event: all_read, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.496935 7f869af78700 5 -- op tracker -- , seq: 1784, time: 2014-06-09 16:22:11.496903, event: dispatched, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.496943 7f869af78700 5 -- op tracker -- , seq: 1784, time: 2014-06-09 16:22:11.496940, event: waiting_for_osdmap, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.496947 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.496950 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xe046000 2014-06-09 16:22:11.496955 7f869af78700 5 -- op tracker -- , seq: 1784, time: 2014-06-09 16:22:11.496953, event: started, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.496986 7f869af78700 5 -- op tracker -- , seq: 1784, time: 2014-06-09 16:22:11.496984, event: done, op: pg_notify(1.70c(23),2.17c(20) epoch 243) 2014-06-09 16:22:11.496996 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.496998 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497024 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497026 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497027 7f869af78700 20 osd.26 243 _dispatch 0x3005b880 pg_notify(3.206(10),0.214(20) epoch 243) v5 2014-06-09 16:22:11.497033 7f869af78700 5 -- op tracker -- , seq: 1785, time: 2014-06-09 16:22:11.174892, event: header_read, op: pg_notify(3.206(10),0.214(20) epoch 243) 2014-06-09 16:22:11.497041 7f869af78700 5 -- op tracker -- , seq: 1785, time: 2014-06-09 16:22:11.174893, event: throttled, op: pg_notify(3.206(10),0.214(20) epoch 243) 2014-06-09 16:22:11.497048 7f869af78700 5 -- op tracker -- , seq: 1785, time: 2014-06-09 16:22:11.174931, event: all_read, op: pg_notify(3.206(10),0.214(20) epoch 243) 2014-06-09 16:22:11.497054 7f869af78700 5 -- op tracker -- , seq: 1785, time: 2014-06-09 16:22:11.497024, event: dispatched, op: pg_notify(3.206(10),0.214(20) epoch 243) 2014-06-09 16:22:11.497061 7f869af78700 5 -- op tracker -- , seq: 1785, time: 2014-06-09 16:22:11.497059, event: waiting_for_osdmap, op: pg_notify(3.206(10),0.214(20) epoch 243) 2014-06-09 16:22:11.497065 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.497067 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005b880 2014-06-09 16:22:11.497072 7f869af78700 5 -- op tracker -- , seq: 1785, time: 2014-06-09 16:22:11.497070, event: started, op: pg_notify(3.206(10),0.214(20) epoch 243) 2014-06-09 16:22:11.497102 7f869af78700 5 -- op tracker -- , seq: 1785, time: 2014-06-09 16:22:11.497099, event: done, op: pg_notify(3.206(10),0.214(20) epoch 243) 2014-06-09 16:22:11.497114 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497116 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497139 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497141 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497142 7f869af78700 20 osd.26 243 _dispatch 0x2fef5c00 pg_notify(1.446(21) epoch 243) v5 2014-06-09 16:22:11.497147 7f869af78700 5 -- op tracker -- , seq: 1786, time: 2014-06-09 16:22:11.148399, event: header_read, op: pg_notify(1.446(21) epoch 243) 2014-06-09 16:22:11.497155 7f869af78700 5 -- op tracker -- , seq: 1786, time: 2014-06-09 16:22:11.148400, event: throttled, op: pg_notify(1.446(21) epoch 243) 2014-06-09 16:22:11.497161 7f869af78700 5 -- op tracker -- , seq: 1786, time: 2014-06-09 16:22:11.148427, event: all_read, op: pg_notify(1.446(21) epoch 243) 2014-06-09 16:22:11.497167 7f869af78700 5 -- op tracker -- , seq: 1786, time: 2014-06-09 16:22:11.497139, event: dispatched, op: pg_notify(1.446(21) epoch 243) 2014-06-09 16:22:11.497174 7f869af78700 5 -- op tracker -- , seq: 1786, time: 2014-06-09 16:22:11.497172, event: waiting_for_osdmap, op: pg_notify(1.446(21) epoch 243) 2014-06-09 16:22:11.497179 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.497181 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef5c00 2014-06-09 16:22:11.497186 7f869af78700 5 -- op tracker -- , seq: 1786, time: 2014-06-09 16:22:11.497184, event: started, op: pg_notify(1.446(21) epoch 243) 2014-06-09 16:22:11.497205 7f869af78700 5 -- op tracker -- , seq: 1786, time: 2014-06-09 16:22:11.497202, event: done, op: pg_notify(1.446(21) epoch 243) 2014-06-09 16:22:11.497213 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497215 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497236 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497238 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497239 7f869af78700 20 osd.26 243 _dispatch 0x308a9600 pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.497246 7f869af78700 5 -- op tracker -- , seq: 1787, time: 2014-06-09 16:22:11.198707, event: header_read, op: pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497253 7f869af78700 5 -- op tracker -- , seq: 1787, time: 2014-06-09 16:22:11.198707, event: throttled, op: pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497260 7f869af78700 5 -- op tracker -- , seq: 1787, time: 2014-06-09 16:22:11.198720, event: all_read, op: pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497268 7f869af78700 5 -- op tracker -- , seq: 1787, time: 2014-06-09 16:22:11.497235, event: dispatched, op: pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497275 7f869af78700 5 -- op tracker -- , seq: 1787, time: 2014-06-09 16:22:11.497272, event: waiting_for_osdmap, op: pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497280 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.497284 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a9600 2014-06-09 16:22:11.497290 7f869af78700 5 -- op tracker -- , seq: 1787, time: 2014-06-09 16:22:11.497287, event: started, op: pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497306 7f869af78700 5 -- op tracker -- , seq: 1787, time: 2014-06-09 16:22:11.497303, event: done, op: pg_log(0.714 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497313 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497315 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497329 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497331 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497332 7f869af78700 20 osd.26 243 _dispatch 0x1a039a40 pg_notify(2.67f(20),0.67e(23),3.3b4(10),3.3b8(10) epoch 243) v5 2014-06-09 16:22:11.497340 7f869af78700 5 -- op tracker -- , seq: 1788, time: 2014-06-09 16:22:11.159698, event: header_read, op: pg_notify(2.67f(20),0.67e(23),3.3b4(10),3.3b8(10) epoch 243) 2014-06-09 16:22:11.497349 7f869af78700 5 -- op tracker -- , seq: 1788, time: 2014-06-09 16:22:11.159699, event: throttled, op: pg_notify(2.67f(20),0.67e(23),3.3b4(10),3.3b8(10) epoch 243) 2014-06-09 16:22:11.497357 7f869af78700 5 -- op tracker -- , seq: 1788, time: 2014-06-09 16:22:11.159786, event: all_read, op: pg_notify(2.67f(20),0.67e(23),3.3b4(10),3.3b8(10) epoch 243) 2014-06-09 16:22:11.497364 7f869af78700 5 -- op tracker -- , seq: 1788, time: 2014-06-09 16:22:11.497329, event: dispatched, op: pg_notify(2.67f(20),0.67e(23),3.3b4(10),3.3b8(10) epoch 243) 2014-06-09 16:22:11.497372 7f869af78700 5 -- op tracker -- , seq: 1788, time: 2014-06-09 16:22:11.497369, event: waiting_for_osdmap, op: pg_notify(2.67f(20),0.67e(23),3.3b4(10),3.3b8(10) epoch 243) 2014-06-09 16:22:11.497377 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.497379 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a039a40 2014-06-09 16:22:11.497385 7f869af78700 5 -- op tracker -- , seq: 1788, time: 2014-06-09 16:22:11.497382, event: started, op: pg_notify(2.67f(20),0.67e(23),3.3b4(10),3.3b8(10) epoch 243) 2014-06-09 16:22:11.497434 7f869af78700 5 -- op tracker -- , seq: 1788, time: 2014-06-09 16:22:11.497431, event: done, op: pg_notify(2.67f(20),0.67e(23),3.3b4(10),3.3b8(10) epoch 243) 2014-06-09 16:22:11.497443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497491 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497493 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497494 7f869af78700 20 osd.26 243 _dispatch 0x30571800 pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.497501 7f869af78700 5 -- op tracker -- , seq: 1789, time: 2014-06-09 16:22:11.176400, event: header_read, op: pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497510 7f869af78700 5 -- op tracker -- , seq: 1789, time: 2014-06-09 16:22:11.176401, event: throttled, op: pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497517 7f869af78700 5 -- op tracker -- , seq: 1789, time: 2014-06-09 16:22:11.176411, event: all_read, op: pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497524 7f869af78700 5 -- op tracker -- , seq: 1789, time: 2014-06-09 16:22:11.497491, event: dispatched, op: pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497531 7f869af78700 5 -- op tracker -- , seq: 1789, time: 2014-06-09 16:22:11.497529, event: waiting_for_osdmap, op: pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497536 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.497540 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30571800 2014-06-09 16:22:11.497546 7f869af78700 5 -- op tracker -- , seq: 1789, time: 2014-06-09 16:22:11.497543, event: started, op: pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497560 7f869af78700 5 -- op tracker -- , seq: 1789, time: 2014-06-09 16:22:11.497557, event: done, op: pg_log(0.5a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497568 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497570 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497583 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497585 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497586 7f869af78700 20 osd.26 243 _dispatch 0x1ae7ddc0 pg_notify(1.2aa(21) epoch 243) v5 2014-06-09 16:22:11.497592 7f869af78700 5 -- op tracker -- , seq: 1790, time: 2014-06-09 16:22:11.141610, event: header_read, op: pg_notify(1.2aa(21) epoch 243) 2014-06-09 16:22:11.497598 7f869af78700 5 -- op tracker -- , seq: 1790, time: 2014-06-09 16:22:11.141611, event: throttled, op: pg_notify(1.2aa(21) epoch 243) 2014-06-09 16:22:11.497604 7f869af78700 5 -- op tracker -- , seq: 1790, time: 2014-06-09 16:22:11.141633, event: all_read, op: pg_notify(1.2aa(21) epoch 243) 2014-06-09 16:22:11.497612 7f869af78700 5 -- op tracker -- , seq: 1790, time: 2014-06-09 16:22:11.497583, event: dispatched, op: pg_notify(1.2aa(21) epoch 243) 2014-06-09 16:22:11.497619 7f869af78700 5 -- op tracker -- , seq: 1790, time: 2014-06-09 16:22:11.497617, event: waiting_for_osdmap, op: pg_notify(1.2aa(21) epoch 243) 2014-06-09 16:22:11.497624 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.497625 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ae7ddc0 2014-06-09 16:22:11.497630 7f869af78700 5 -- op tracker -- , seq: 1790, time: 2014-06-09 16:22:11.497628, event: started, op: pg_notify(1.2aa(21) epoch 243) 2014-06-09 16:22:11.497647 7f869af78700 5 -- op tracker -- , seq: 1790, time: 2014-06-09 16:22:11.497644, event: done, op: pg_notify(1.2aa(21) epoch 243) 2014-06-09 16:22:11.497657 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497659 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497680 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497683 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497684 7f869af78700 20 osd.26 243 _dispatch 0x2d733dc0 pg_notify(2.746(20),2.1b1(20) epoch 243) v5 2014-06-09 16:22:11.497690 7f869af78700 5 -- op tracker -- , seq: 1791, time: 2014-06-09 16:22:11.238495, event: header_read, op: pg_notify(2.746(20),2.1b1(20) epoch 243) 2014-06-09 16:22:11.497698 7f869af78700 5 -- op tracker -- , seq: 1791, time: 2014-06-09 16:22:11.238496, event: throttled, op: pg_notify(2.746(20),2.1b1(20) epoch 243) 2014-06-09 16:22:11.497705 7f869af78700 5 -- op tracker -- , seq: 1791, time: 2014-06-09 16:22:11.238530, event: all_read, op: pg_notify(2.746(20),2.1b1(20) epoch 243) 2014-06-09 16:22:11.497712 7f869af78700 5 -- op tracker -- , seq: 1791, time: 2014-06-09 16:22:11.497680, event: dispatched, op: pg_notify(2.746(20),2.1b1(20) epoch 243) 2014-06-09 16:22:11.497719 7f869af78700 5 -- op tracker -- , seq: 1791, time: 2014-06-09 16:22:11.497717, event: waiting_for_osdmap, op: pg_notify(2.746(20),2.1b1(20) epoch 243) 2014-06-09 16:22:11.497723 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.497725 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d733dc0 2014-06-09 16:22:11.497731 7f869af78700 5 -- op tracker -- , seq: 1791, time: 2014-06-09 16:22:11.497728, event: started, op: pg_notify(2.746(20),2.1b1(20) epoch 243) 2014-06-09 16:22:11.497753 7f869af78700 5 -- op tracker -- , seq: 1791, time: 2014-06-09 16:22:11.497750, event: done, op: pg_notify(2.746(20),2.1b1(20) epoch 243) 2014-06-09 16:22:11.497761 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497763 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497791 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497792 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497794 7f869af78700 20 osd.26 243 _dispatch 0x3045a600 pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.497801 7f869af78700 5 -- op tracker -- , seq: 1792, time: 2014-06-09 16:22:11.027664, event: header_read, op: pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497808 7f869af78700 5 -- op tracker -- , seq: 1792, time: 2014-06-09 16:22:11.027664, event: throttled, op: pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497816 7f869af78700 5 -- op tracker -- , seq: 1792, time: 2014-06-09 16:22:11.027684, event: all_read, op: pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497823 7f869af78700 5 -- op tracker -- , seq: 1792, time: 2014-06-09 16:22:11.497790, event: dispatched, op: pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497831 7f869af78700 5 -- op tracker -- , seq: 1792, time: 2014-06-09 16:22:11.497828, event: waiting_for_osdmap, op: pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497835 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.497839 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045a600 2014-06-09 16:22:11.497845 7f869af78700 5 -- op tracker -- , seq: 1792, time: 2014-06-09 16:22:11.497842, event: started, op: pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497858 7f869af78700 5 -- op tracker -- , seq: 1792, time: 2014-06-09 16:22:11.497855, event: done, op: pg_log(1.b5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.497866 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497880 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497881 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497882 7f869af78700 20 osd.26 243 _dispatch 0x26000380 pg_notify(3.327(10) epoch 243) v5 2014-06-09 16:22:11.497887 7f869af78700 5 -- op tracker -- , seq: 1793, time: 2014-06-09 16:22:11.015898, event: header_read, op: pg_notify(3.327(10) epoch 243) 2014-06-09 16:22:11.497894 7f869af78700 5 -- op tracker -- , seq: 1793, time: 2014-06-09 16:22:11.015898, event: throttled, op: pg_notify(3.327(10) epoch 243) 2014-06-09 16:22:11.497900 7f869af78700 5 -- op tracker -- , seq: 1793, time: 2014-06-09 16:22:11.015925, event: all_read, op: pg_notify(3.327(10) epoch 243) 2014-06-09 16:22:11.497906 7f869af78700 5 -- op tracker -- , seq: 1793, time: 2014-06-09 16:22:11.497879, event: dispatched, op: pg_notify(3.327(10) epoch 243) 2014-06-09 16:22:11.497913 7f869af78700 5 -- op tracker -- , seq: 1793, time: 2014-06-09 16:22:11.497911, event: waiting_for_osdmap, op: pg_notify(3.327(10) epoch 243) 2014-06-09 16:22:11.497917 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.497920 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26000380 2014-06-09 16:22:11.497925 7f869af78700 5 -- op tracker -- , seq: 1793, time: 2014-06-09 16:22:11.497923, event: started, op: pg_notify(3.327(10) epoch 243) 2014-06-09 16:22:11.497942 7f869af78700 5 -- op tracker -- , seq: 1793, time: 2014-06-09 16:22:11.497940, event: done, op: pg_notify(3.327(10) epoch 243) 2014-06-09 16:22:11.497953 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497954 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.497971 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.497972 7f869af78700 20 osd.26 243 _dispatch 0x307e6000 pg_info(1 pgs e243:0.30b) v4 2014-06-09 16:22:11.497977 7f869af78700 5 -- op tracker -- , seq: 1794, time: 2014-06-09 16:22:11.268255, event: header_read, op: pg_info(1 pgs e243:0.30b) 2014-06-09 16:22:11.497984 7f869af78700 5 -- op tracker -- , seq: 1794, time: 2014-06-09 16:22:11.268256, event: throttled, op: pg_info(1 pgs e243:0.30b) 2014-06-09 16:22:11.497991 7f869af78700 5 -- op tracker -- , seq: 1794, time: 2014-06-09 16:22:11.268270, event: all_read, op: pg_info(1 pgs e243:0.30b) 2014-06-09 16:22:11.497997 7f869af78700 5 -- op tracker -- , seq: 1794, time: 2014-06-09 16:22:11.497969, event: dispatched, op: pg_info(1 pgs e243:0.30b) 2014-06-09 16:22:11.498004 7f869af78700 5 -- op tracker -- , seq: 1794, time: 2014-06-09 16:22:11.498002, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.30b) 2014-06-09 16:22:11.498009 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.30b) v4 from osd.7 2014-06-09 16:22:11.498012 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e6000 2014-06-09 16:22:11.498017 7f869af78700 5 -- op tracker -- , seq: 1794, time: 2014-06-09 16:22:11.498015, event: started, op: pg_info(1 pgs e243:0.30b) 2014-06-09 16:22:11.498033 7f869af78700 5 -- op tracker -- , seq: 1794, time: 2014-06-09 16:22:11.498031, event: done, op: pg_info(1 pgs e243:0.30b) 2014-06-09 16:22:11.498041 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.498044 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.498058 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.498059 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.498060 7f869af78700 20 osd.26 243 _dispatch 0x3043b340 pg_notify(1.282(20),0.28a(23),2.292(23) epoch 243) v5 2014-06-09 16:22:11.498067 7f869af78700 5 -- op tracker -- , seq: 1795, time: 2014-06-09 16:22:11.212955, event: header_read, op: pg_notify(1.282(20),0.28a(23),2.292(23) epoch 243) 2014-06-09 16:22:11.498075 7f869af78700 5 -- op tracker -- , seq: 1795, time: 2014-06-09 16:22:11.212955, event: throttled, op: pg_notify(1.282(20),0.28a(23),2.292(23) epoch 243) 2014-06-09 16:22:11.498082 7f869af78700 5 -- op tracker -- , seq: 1795, time: 2014-06-09 16:22:11.213046, event: all_read, op: pg_notify(1.282(20),0.28a(23),2.292(23) epoch 243) 2014-06-09 16:22:11.498089 7f869af78700 5 -- op tracker -- , seq: 1795, time: 2014-06-09 16:22:11.498057, event: dispatched, op: pg_notify(1.282(20),0.28a(23),2.292(23) epoch 243) 2014-06-09 16:22:11.498096 7f869af78700 5 -- op tracker -- , seq: 1795, time: 2014-06-09 16:22:11.498094, event: waiting_for_osdmap, op: pg_notify(1.282(20),0.28a(23),2.292(23) epoch 243) 2014-06-09 16:22:11.498101 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.498103 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043b340 2014-06-09 16:22:11.498108 7f869af78700 5 -- op tracker -- , seq: 1795, time: 2014-06-09 16:22:11.498106, event: started, op: pg_notify(1.282(20),0.28a(23),2.292(23) epoch 243) 2014-06-09 16:22:11.498139 7f869af78700 5 -- op tracker -- , seq: 1795, time: 2014-06-09 16:22:11.498136, event: done, op: pg_notify(1.282(20),0.28a(23),2.292(23) epoch 243) 2014-06-09 16:22:11.498146 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.498148 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.498184 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.498186 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.498188 7f869af78700 20 osd.26 243 _dispatch 0x30644c00 pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.498195 7f869af78700 5 -- op tracker -- , seq: 1796, time: 2014-06-09 16:22:11.148440, event: header_read, op: pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.498203 7f869af78700 5 -- op tracker -- , seq: 1796, time: 2014-06-09 16:22:11.148440, event: throttled, op: pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.498209 7f869af78700 5 -- op tracker -- , seq: 1796, time: 2014-06-09 16:22:11.148452, event: all_read, op: pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.498216 7f869af78700 5 -- op tracker -- , seq: 1796, time: 2014-06-09 16:22:11.498184, event: dispatched, op: pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.498223 7f869af78700 5 -- op tracker -- , seq: 1796, time: 2014-06-09 16:22:11.498220, event: waiting_for_osdmap, op: pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.498228 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.498231 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30644c00 2014-06-09 16:22:11.498238 7f869af78700 5 -- op tracker -- , seq: 1796, time: 2014-06-09 16:22:11.498235, event: started, op: pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.498251 7f869af78700 5 -- op tracker -- , seq: 1796, time: 2014-06-09 16:22:11.498248, event: done, op: pg_log(2.26a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.498260 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.498262 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.498273 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.498275 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.498276 7f869af78700 20 osd.26 243 _dispatch 0x2fe70c40 pg_notify(3.173(10) epoch 243) v5 2014-06-09 16:22:11.498281 7f869af78700 5 -- op tracker -- , seq: 1797, time: 2014-06-09 16:22:11.232211, event: header_read, op: pg_notify(3.173(10) epoch 243) 2014-06-09 16:22:11.498288 7f869af78700 5 -- op tracker -- , seq: 1797, time: 2014-06-09 16:22:11.232211, event: throttled, op: pg_notify(3.173(10) epoch 243) 2014-06-09 16:22:11.498295 7f869af78700 5 -- op tracker -- , seq: 1797, time: 2014-06-09 16:22:11.232228, event: all_read, op: pg_notify(3.173(10) epoch 243) 2014-06-09 16:22:11.498301 7f869af78700 5 -- op tracker -- , seq: 1797, time: 2014-06-09 16:22:11.498273, event: dispatched, op: pg_notify(3.173(10) epoch 243) 2014-06-09 16:22:11.498307 7f869af78700 5 -- op tracker -- , seq: 1797, time: 2014-06-09 16:22:11.498306, event: waiting_for_osdmap, op: pg_notify(3.173(10) epoch 243) 2014-06-09 16:22:11.498312 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.498314 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe70c40 2014-06-09 16:22:11.498319 7f869af78700 5 -- op tracker -- , seq: 1797, time: 2014-06-09 16:22:11.498317, event: started, op: pg_notify(3.173(10) epoch 243) 2014-06-09 16:22:11.498337 7f869af78700 5 -- op tracker -- , seq: 1797, time: 2014-06-09 16:22:11.498334, event: done, op: pg_notify(3.173(10) epoch 243) 2014-06-09 16:22:11.498345 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.498347 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.498364 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.498366 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.498390 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (10 active) 2014-06-09 16:22:11.506685 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.506705 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.5e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.506718 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.506722 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.506727 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.315029 7 0.000209 2014-06-09 16:22:11.506740 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.506752 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.506763 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.506773 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.506782 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.506837 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.506845 7f8689f56700 5 queue_transactions existing osr(0.5e 0x2bd63b0)/0x2bd63b0 2014-06-09 16:22:11.506848 7f8689f56700 2 waiting 51 > 50 ops || 92346 > 104857600 2014-06-09 16:22:11.498407 7f8690f64700 10 op_submit_start 3187 2014-06-09 16:22:11.506856 7f8690f64700 5 queue_transactions (trailing journal) 3187 2014-06-09 16:22:11.506858 7f8690f64700 5 queue_op 0x300608a0 seq 3187 osr(1.32f 0x384d350) 1883 bytes (queue has 50 ops and 90348 bytes) 2014-06-09 16:22:11.506862 7f8690f64700 10 op_submit_finish 3187 2014-06-09 16:22:11.506868 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.506881 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 2.30b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.506893 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.506897 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.506901 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.303323 7 0.000175 2014-06-09 16:22:11.506913 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.506923 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.506933 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.506941 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.506950 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.507002 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.507007 7f8690f64700 5 queue_transactions existing osr(2.30b 0x2ed0f50)/0x2ed0f50 2014-06-09 16:22:11.507011 7f8690f64700 2 waiting 51 > 50 ops || 92345 > 104857600 2014-06-09 16:22:11.507023 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (11 active) 2014-06-09 16:22:11.507031 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.173( v 213'624 (0'0,213'624] local-les=213 n=119 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.507047 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.507059 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.507066 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (10 active) 2014-06-09 16:22:11.507070 7f868cf5c700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.498503 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.507082 7f868ef60700 5 queue_transactions existing osr(3.2b 0x3ed0350)/0x3ed0350 2014-06-09 16:22:11.507085 7f868ef60700 2 waiting 51 > 50 ops || 91398 > 104857600 2014-06-09 16:22:11.507090 7f8692767700 10 op_submit_start 3188 2014-06-09 16:22:11.507091 7f8692767700 5 queue_transactions (trailing journal) 3188 2014-06-09 16:22:11.507093 7f8692767700 5 queue_op 0x309a2600 seq 3188 osr(1.a7 0x384c730) 2096 bytes (queue has 50 ops and 90348 bytes) 2014-06-09 16:22:11.507096 7f8692767700 10 op_submit_finish 3188 2014-06-09 16:22:11.507101 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.507113 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.507126 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.507130 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.507134 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.981951 7 0.000180 2014-06-09 16:22:11.507149 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.507159 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.507168 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.507177 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.507186 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.507236 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.507242 7f8692767700 5 queue_transactions existing osr(1.3f0 0x384d400)/0x384d400 2014-06-09 16:22:11.507245 7f8692767700 2 waiting 51 > 50 ops || 92231 > 104857600 2014-06-09 16:22:11.507252 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.498518 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.507264 7f868df5e700 5 queue_transactions existing osr(0.151 0x2bd65e0)/0x2bd65e0 2014-06-09 16:22:11.507267 7f868df5e700 2 waiting 51 > 50 ops || 92345 > 104857600 2014-06-09 16:22:11.507271 7f868ff62700 10 op_submit_start 3189 2014-06-09 16:22:11.507272 7f868ff62700 5 queue_transactions (trailing journal) 3189 2014-06-09 16:22:11.507274 7f868ff62700 5 queue_op 0xd4711a0 seq 3189 osr(0.6d2 0x384c1f0) 1974 bytes (queue has 50 ops and 90348 bytes) 2014-06-09 16:22:11.507277 7f868ff62700 10 op_submit_finish 3189 2014-06-09 16:22:11.507281 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.507294 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.c6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.507306 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.507310 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.507313 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.209034 7 0.000211 2014-06-09 16:22:11.507324 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.507335 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.507345 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.507354 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.507364 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.507415 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.507420 7f868ff62700 5 queue_transactions existing osr(1.c6 0x384c610)/0x384c610 2014-06-09 16:22:11.507422 7f868ff62700 2 waiting 51 > 50 ops || 92354 > 104857600 2014-06-09 16:22:11.507429 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.507437 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.507444 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.507451 7f868af58700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.507458 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.507465 7f868c75b700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.507471 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.507478 7f868f761700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.498367 7f869af78700 20 osd.26 243 _dispatch 0x3004de00 pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.507489 7f869af78700 5 -- op tracker -- , seq: 1798, time: 2014-06-09 16:22:11.159799, event: header_read, op: pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507499 7f869af78700 5 -- op tracker -- , seq: 1798, time: 2014-06-09 16:22:11.159800, event: throttled, op: pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507506 7f869af78700 5 -- op tracker -- , seq: 1798, time: 2014-06-09 16:22:11.159815, event: all_read, op: pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507513 7f869af78700 5 -- op tracker -- , seq: 1798, time: 2014-06-09 16:22:11.498364, event: dispatched, op: pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507521 7f869af78700 5 -- op tracker -- , seq: 1798, time: 2014-06-09 16:22:11.507518, event: waiting_for_osdmap, op: pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507526 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.507531 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004de00 2014-06-09 16:22:11.507537 7f869af78700 5 -- op tracker -- , seq: 1798, time: 2014-06-09 16:22:11.507534, event: started, op: pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507554 7f869af78700 5 -- op tracker -- , seq: 1798, time: 2014-06-09 16:22:11.507551, event: done, op: pg_log(2.216 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507566 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.507568 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.507586 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.507588 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.507589 7f869af78700 20 osd.26 243 _dispatch 0x304df000 pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.507596 7f869af78700 5 -- op tracker -- , seq: 1799, time: 2014-06-09 16:22:11.176418, event: header_read, op: pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507604 7f869af78700 5 -- op tracker -- , seq: 1799, time: 2014-06-09 16:22:11.176419, event: throttled, op: pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507640 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (11 active) 2014-06-09 16:22:11.507645 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.507656 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.216( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.507668 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.507671 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.507675 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.216( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.199424 7 0.000191 2014-06-09 16:22:11.507686 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.216( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.507697 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.216( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.507707 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.507716 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.507725 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.507773 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.507777 7f868cf5c700 5 queue_transactions existing osr(2.216 0x2ed0b50)/0x2ed0b50 2014-06-09 16:22:11.507782 7f868cf5c700 2 waiting 51 > 50 ops || 92349 > 104857600 2014-06-09 16:22:11.507611 7f869af78700 5 -- op tracker -- , seq: 1799, time: 2014-06-09 16:22:11.176431, event: all_read, op: pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507807 7f869af78700 5 -- op tracker -- , seq: 1799, time: 2014-06-09 16:22:11.507586, event: dispatched, op: pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507816 7f869af78700 5 -- op tracker -- , seq: 1799, time: 2014-06-09 16:22:11.507813, event: waiting_for_osdmap, op: pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507821 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.507825 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304df000 2014-06-09 16:22:11.507831 7f869af78700 5 -- op tracker -- , seq: 1799, time: 2014-06-09 16:22:11.507828, event: started, op: pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507847 7f869af78700 5 -- op tracker -- , seq: 1799, time: 2014-06-09 16:22:11.507843, event: done, op: pg_log(0.2fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507855 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.507858 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.507870 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.507871 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.507873 7f869af78700 20 osd.26 243 _dispatch 0x2fe71180 pg_notify(3.2d5(12) epoch 243) v5 2014-06-09 16:22:11.507878 7f869af78700 5 -- op tracker -- , seq: 1800, time: 2014-06-09 16:22:11.143850, event: header_read, op: pg_notify(3.2d5(12) epoch 243) 2014-06-09 16:22:11.507885 7f869af78700 5 -- op tracker -- , seq: 1800, time: 2014-06-09 16:22:11.143851, event: throttled, op: pg_notify(3.2d5(12) epoch 243) 2014-06-09 16:22:11.507892 7f869af78700 5 -- op tracker -- , seq: 1800, time: 2014-06-09 16:22:11.143874, event: all_read, op: pg_notify(3.2d5(12) epoch 243) 2014-06-09 16:22:11.507898 7f869af78700 5 -- op tracker -- , seq: 1800, time: 2014-06-09 16:22:11.507870, event: dispatched, op: pg_notify(3.2d5(12) epoch 243) 2014-06-09 16:22:11.507904 7f869af78700 5 -- op tracker -- , seq: 1800, time: 2014-06-09 16:22:11.507902, event: waiting_for_osdmap, op: pg_notify(3.2d5(12) epoch 243) 2014-06-09 16:22:11.507909 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.507911 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe71180 2014-06-09 16:22:11.507916 7f869af78700 5 -- op tracker -- , seq: 1800, time: 2014-06-09 16:22:11.507914, event: started, op: pg_notify(3.2d5(12) epoch 243) 2014-06-09 16:22:11.507937 7f869af78700 5 -- op tracker -- , seq: 1800, time: 2014-06-09 16:22:11.507934, event: done, op: pg_notify(3.2d5(12) epoch 243) 2014-06-09 16:22:11.507945 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.507947 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.507969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.507970 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.507971 7f869af78700 20 osd.26 243 _dispatch 0x30953600 pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.507978 7f869af78700 5 -- op tracker -- , seq: 1801, time: 2014-06-09 16:22:11.238567, event: header_read, op: pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507986 7f869af78700 5 -- op tracker -- , seq: 1801, time: 2014-06-09 16:22:11.238567, event: throttled, op: pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.507994 7f869af78700 5 -- op tracker -- , seq: 1801, time: 2014-06-09 16:22:11.238575, event: all_read, op: pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508001 7f869af78700 5 -- op tracker -- , seq: 1801, time: 2014-06-09 16:22:11.507968, event: dispatched, op: pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508009 7f869af78700 5 -- op tracker -- , seq: 1801, time: 2014-06-09 16:22:11.508006, event: waiting_for_osdmap, op: pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508014 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.508018 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30953600 2014-06-09 16:22:11.508024 7f869af78700 5 -- op tracker -- , seq: 1801, time: 2014-06-09 16:22:11.508021, event: started, op: pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508039 7f869af78700 5 -- op tracker -- , seq: 1801, time: 2014-06-09 16:22:11.508036, event: done, op: pg_log(1.408 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508047 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508062 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508064 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508065 7f869af78700 20 osd.26 243 _dispatch 0x3045a000 pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.508071 7f869af78700 5 -- op tracker -- , seq: 1802, time: 2014-06-09 16:22:11.031203, event: header_read, op: pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508079 7f869af78700 5 -- op tracker -- , seq: 1802, time: 2014-06-09 16:22:11.031203, event: throttled, op: pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508087 7f869af78700 5 -- op tracker -- , seq: 1802, time: 2014-06-09 16:22:11.031219, event: all_read, op: pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508094 7f869af78700 5 -- op tracker -- , seq: 1802, time: 2014-06-09 16:22:11.508062, event: dispatched, op: pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508102 7f869af78700 5 -- op tracker -- , seq: 1802, time: 2014-06-09 16:22:11.508099, event: waiting_for_osdmap, op: pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508107 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.508111 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045a000 2014-06-09 16:22:11.508116 7f869af78700 5 -- op tracker -- , seq: 1802, time: 2014-06-09 16:22:11.508114, event: started, op: pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508132 7f869af78700 5 -- op tracker -- , seq: 1802, time: 2014-06-09 16:22:11.508129, event: done, op: pg_log(2.38a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508140 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508142 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508155 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508157 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508158 7f869af78700 20 osd.26 243 _dispatch 0x30045200 pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.508165 7f869af78700 5 -- op tracker -- , seq: 1803, time: 2014-06-09 16:22:11.047600, event: header_read, op: pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508173 7f869af78700 5 -- op tracker -- , seq: 1803, time: 2014-06-09 16:22:11.047601, event: throttled, op: pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508180 7f869af78700 5 -- op tracker -- , seq: 1803, time: 2014-06-09 16:22:11.047621, event: all_read, op: pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508187 7f869af78700 5 -- op tracker -- , seq: 1803, time: 2014-06-09 16:22:11.508155, event: dispatched, op: pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508195 7f869af78700 5 -- op tracker -- , seq: 1803, time: 2014-06-09 16:22:11.508192, event: waiting_for_osdmap, op: pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508200 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.508204 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30045200 2014-06-09 16:22:11.508210 7f869af78700 5 -- op tracker -- , seq: 1803, time: 2014-06-09 16:22:11.508207, event: started, op: pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508225 7f869af78700 5 -- op tracker -- , seq: 1803, time: 2014-06-09 16:22:11.508222, event: done, op: pg_log(1.67 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508233 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508235 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508247 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508248 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508250 7f869af78700 20 osd.26 243 _dispatch 0x1a038380 pg_info(1 pgs e243:3.60) v4 2014-06-09 16:22:11.508254 7f869af78700 5 -- op tracker -- , seq: 1804, time: 2014-06-09 16:22:11.268287, event: header_read, op: pg_info(1 pgs e243:3.60) 2014-06-09 16:22:11.508262 7f869af78700 5 -- op tracker -- , seq: 1804, time: 2014-06-09 16:22:11.268288, event: throttled, op: pg_info(1 pgs e243:3.60) 2014-06-09 16:22:11.508268 7f869af78700 5 -- op tracker -- , seq: 1804, time: 2014-06-09 16:22:11.268299, event: all_read, op: pg_info(1 pgs e243:3.60) 2014-06-09 16:22:11.508275 7f869af78700 5 -- op tracker -- , seq: 1804, time: 2014-06-09 16:22:11.508246, event: dispatched, op: pg_info(1 pgs e243:3.60) 2014-06-09 16:22:11.508282 7f869af78700 5 -- op tracker -- , seq: 1804, time: 2014-06-09 16:22:11.508280, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.60) 2014-06-09 16:22:11.508286 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.60) v4 from osd.7 2014-06-09 16:22:11.508289 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a038380 2014-06-09 16:22:11.508295 7f869af78700 5 -- op tracker -- , seq: 1804, time: 2014-06-09 16:22:11.508293, event: started, op: pg_info(1 pgs e243:3.60) 2014-06-09 16:22:11.508314 7f869af78700 5 -- op tracker -- , seq: 1804, time: 2014-06-09 16:22:11.508312, event: done, op: pg_info(1 pgs e243:3.60) 2014-06-09 16:22:11.508322 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508324 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508338 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508339 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508341 7f869af78700 20 osd.26 243 _dispatch 0x306d4a00 pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.508347 7f869af78700 5 -- op tracker -- , seq: 1805, time: 2014-06-09 16:22:11.235809, event: header_read, op: pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508354 7f869af78700 5 -- op tracker -- , seq: 1805, time: 2014-06-09 16:22:11.235809, event: throttled, op: pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508361 7f869af78700 5 -- op tracker -- , seq: 1805, time: 2014-06-09 16:22:11.235820, event: all_read, op: pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508369 7f869af78700 5 -- op tracker -- , seq: 1805, time: 2014-06-09 16:22:11.508337, event: dispatched, op: pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508376 7f869af78700 5 -- op tracker -- , seq: 1805, time: 2014-06-09 16:22:11.508373, event: waiting_for_osdmap, op: pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508381 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.508385 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d4a00 2014-06-09 16:22:11.508391 7f869af78700 5 -- op tracker -- , seq: 1805, time: 2014-06-09 16:22:11.508388, event: started, op: pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508407 7f869af78700 5 -- op tracker -- , seq: 1805, time: 2014-06-09 16:22:11.508404, event: done, op: pg_log(1.5c1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508417 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508419 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508433 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508434 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508435 7f869af78700 20 osd.26 243 _dispatch 0x22013dc0 pg_info(1 pgs e243:3.254(4)) v4 2014-06-09 16:22:11.508441 7f869af78700 5 -- op tracker -- , seq: 1806, time: 2014-06-09 16:22:11.148461, event: header_read, op: pg_info(1 pgs e243:3.254(4)) 2014-06-09 16:22:11.508448 7f869af78700 5 -- op tracker -- , seq: 1806, time: 2014-06-09 16:22:11.148461, event: throttled, op: pg_info(1 pgs e243:3.254(4)) 2014-06-09 16:22:11.508454 7f869af78700 5 -- op tracker -- , seq: 1806, time: 2014-06-09 16:22:11.148481, event: all_read, op: pg_info(1 pgs e243:3.254(4)) 2014-06-09 16:22:11.508461 7f869af78700 5 -- op tracker -- , seq: 1806, time: 2014-06-09 16:22:11.508432, event: dispatched, op: pg_info(1 pgs e243:3.254(4)) 2014-06-09 16:22:11.508468 7f869af78700 5 -- op tracker -- , seq: 1806, time: 2014-06-09 16:22:11.508466, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.254(4)) 2014-06-09 16:22:11.508472 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.254(4)) v4 from osd.1 2014-06-09 16:22:11.508475 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22013dc0 2014-06-09 16:22:11.508481 7f869af78700 5 -- op tracker -- , seq: 1806, time: 2014-06-09 16:22:11.508478, event: started, op: pg_info(1 pgs e243:3.254(4)) 2014-06-09 16:22:11.508500 7f869af78700 5 -- op tracker -- , seq: 1806, time: 2014-06-09 16:22:11.508497, event: done, op: pg_info(1 pgs e243:3.254(4)) 2014-06-09 16:22:11.508510 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508512 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508527 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508529 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508530 7f869af78700 20 osd.26 243 _dispatch 0x3070de00 pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.508536 7f869af78700 5 -- op tracker -- , seq: 1807, time: 2014-06-09 16:22:11.232259, event: header_read, op: pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508544 7f869af78700 5 -- op tracker -- , seq: 1807, time: 2014-06-09 16:22:11.232259, event: throttled, op: pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508551 7f869af78700 5 -- op tracker -- , seq: 1807, time: 2014-06-09 16:22:11.232270, event: all_read, op: pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508558 7f869af78700 5 -- op tracker -- , seq: 1807, time: 2014-06-09 16:22:11.508527, event: dispatched, op: pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508566 7f869af78700 5 -- op tracker -- , seq: 1807, time: 2014-06-09 16:22:11.508563, event: waiting_for_osdmap, op: pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508570 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.508574 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070de00 2014-06-09 16:22:11.508580 7f869af78700 5 -- op tracker -- , seq: 1807, time: 2014-06-09 16:22:11.508577, event: started, op: pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508596 7f869af78700 5 -- op tracker -- , seq: 1807, time: 2014-06-09 16:22:11.508593, event: done, op: pg_log(0.458 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508603 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508605 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508617 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508618 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508619 7f869af78700 20 osd.26 243 _dispatch 0x306db800 pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.508625 7f869af78700 5 -- op tracker -- , seq: 1808, time: 2014-06-09 16:22:11.159826, event: header_read, op: pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508633 7f869af78700 5 -- op tracker -- , seq: 1808, time: 2014-06-09 16:22:11.159827, event: throttled, op: pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508640 7f869af78700 5 -- op tracker -- , seq: 1808, time: 2014-06-09 16:22:11.159836, event: all_read, op: pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508647 7f869af78700 5 -- op tracker -- , seq: 1808, time: 2014-06-09 16:22:11.508616, event: dispatched, op: pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508655 7f869af78700 5 -- op tracker -- , seq: 1808, time: 2014-06-09 16:22:11.508652, event: waiting_for_osdmap, op: pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508659 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.508663 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306db800 2014-06-09 16:22:11.508668 7f869af78700 5 -- op tracker -- , seq: 1808, time: 2014-06-09 16:22:11.508666, event: started, op: pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508682 7f869af78700 5 -- op tracker -- , seq: 1808, time: 2014-06-09 16:22:11.508679, event: done, op: pg_log(0.6e7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508690 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508691 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508703 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508705 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508706 7f869af78700 20 osd.26 243 _dispatch 0x304dc600 pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.508712 7f869af78700 5 -- op tracker -- , seq: 1809, time: 2014-06-09 16:22:11.188050, event: header_read, op: pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508720 7f869af78700 5 -- op tracker -- , seq: 1809, time: 2014-06-09 16:22:11.188051, event: throttled, op: pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508727 7f869af78700 5 -- op tracker -- , seq: 1809, time: 2014-06-09 16:22:11.188066, event: all_read, op: pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508733 7f869af78700 5 -- op tracker -- , seq: 1809, time: 2014-06-09 16:22:11.508703, event: dispatched, op: pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508740 7f869af78700 5 -- op tracker -- , seq: 1809, time: 2014-06-09 16:22:11.508738, event: waiting_for_osdmap, op: pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508745 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.508749 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304dc600 2014-06-09 16:22:11.508754 7f869af78700 5 -- op tracker -- , seq: 1809, time: 2014-06-09 16:22:11.508752, event: started, op: pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508768 7f869af78700 5 -- op tracker -- , seq: 1809, time: 2014-06-09 16:22:11.508765, event: done, op: pg_log(1.122 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508776 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508778 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508791 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508792 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508793 7f869af78700 20 osd.26 243 _dispatch 0x3071b000 pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.508800 7f869af78700 5 -- op tracker -- , seq: 1810, time: 2014-06-09 16:22:11.145392, event: header_read, op: pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508807 7f869af78700 5 -- op tracker -- , seq: 1810, time: 2014-06-09 16:22:11.145393, event: throttled, op: pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508814 7f869af78700 5 -- op tracker -- , seq: 1810, time: 2014-06-09 16:22:11.145417, event: all_read, op: pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508821 7f869af78700 5 -- op tracker -- , seq: 1810, time: 2014-06-09 16:22:11.508790, event: dispatched, op: pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508829 7f869af78700 5 -- op tracker -- , seq: 1810, time: 2014-06-09 16:22:11.508826, event: waiting_for_osdmap, op: pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508833 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.508838 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3071b000 2014-06-09 16:22:11.508843 7f869af78700 5 -- op tracker -- , seq: 1810, time: 2014-06-09 16:22:11.508841, event: started, op: pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508857 7f869af78700 5 -- op tracker -- , seq: 1810, time: 2014-06-09 16:22:11.508854, event: done, op: pg_log(2.5be epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508865 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508879 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508881 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508882 7f869af78700 20 osd.26 243 _dispatch 0x308e2a80 pg_info(1 pgs e243:3.139(2)) v4 2014-06-09 16:22:11.508887 7f869af78700 5 -- op tracker -- , seq: 1811, time: 2014-06-09 16:22:11.238586, event: header_read, op: pg_info(1 pgs e243:3.139(2)) 2014-06-09 16:22:11.508894 7f869af78700 5 -- op tracker -- , seq: 1811, time: 2014-06-09 16:22:11.238586, event: throttled, op: pg_info(1 pgs e243:3.139(2)) 2014-06-09 16:22:11.508900 7f869af78700 5 -- op tracker -- , seq: 1811, time: 2014-06-09 16:22:11.238603, event: all_read, op: pg_info(1 pgs e243:3.139(2)) 2014-06-09 16:22:11.508906 7f869af78700 5 -- op tracker -- , seq: 1811, time: 2014-06-09 16:22:11.508879, event: dispatched, op: pg_info(1 pgs e243:3.139(2)) 2014-06-09 16:22:11.508913 7f869af78700 5 -- op tracker -- , seq: 1811, time: 2014-06-09 16:22:11.508911, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.139(2)) 2014-06-09 16:22:11.508917 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.139(2)) v4 from osd.5 2014-06-09 16:22:11.508920 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e2a80 2014-06-09 16:22:11.508925 7f869af78700 5 -- op tracker -- , seq: 1811, time: 2014-06-09 16:22:11.508923, event: started, op: pg_info(1 pgs e243:3.139(2)) 2014-06-09 16:22:11.508943 7f869af78700 5 -- op tracker -- , seq: 1811, time: 2014-06-09 16:22:11.508940, event: done, op: pg_info(1 pgs e243:3.139(2)) 2014-06-09 16:22:11.508951 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508966 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.508967 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.508968 7f869af78700 20 osd.26 243 _dispatch 0x3045d000 pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.508975 7f869af78700 5 -- op tracker -- , seq: 1812, time: 2014-06-09 16:22:11.031243, event: header_read, op: pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508983 7f869af78700 5 -- op tracker -- , seq: 1812, time: 2014-06-09 16:22:11.031244, event: throttled, op: pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508990 7f869af78700 5 -- op tracker -- , seq: 1812, time: 2014-06-09 16:22:11.031255, event: all_read, op: pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.508997 7f869af78700 5 -- op tracker -- , seq: 1812, time: 2014-06-09 16:22:11.508966, event: dispatched, op: pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509005 7f869af78700 5 -- op tracker -- , seq: 1812, time: 2014-06-09 16:22:11.509002, event: waiting_for_osdmap, op: pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509009 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.509013 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045d000 2014-06-09 16:22:11.509019 7f869af78700 5 -- op tracker -- , seq: 1812, time: 2014-06-09 16:22:11.509017, event: started, op: pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509033 7f869af78700 5 -- op tracker -- , seq: 1812, time: 2014-06-09 16:22:11.509030, event: done, op: pg_log(0.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509045 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509047 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509060 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509062 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509063 7f869af78700 20 osd.26 243 _dispatch 0x30044c00 pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.509070 7f869af78700 5 -- op tracker -- , seq: 1813, time: 2014-06-09 16:22:11.047679, event: header_read, op: pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509078 7f869af78700 5 -- op tracker -- , seq: 1813, time: 2014-06-09 16:22:11.047679, event: throttled, op: pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509085 7f869af78700 5 -- op tracker -- , seq: 1813, time: 2014-06-09 16:22:11.047689, event: all_read, op: pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509093 7f869af78700 5 -- op tracker -- , seq: 1813, time: 2014-06-09 16:22:11.509060, event: dispatched, op: pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509100 7f869af78700 5 -- op tracker -- , seq: 1813, time: 2014-06-09 16:22:11.509098, event: waiting_for_osdmap, op: pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509105 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.509109 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30044c00 2014-06-09 16:22:11.509115 7f869af78700 5 -- op tracker -- , seq: 1813, time: 2014-06-09 16:22:11.509112, event: started, op: pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509129 7f869af78700 5 -- op tracker -- , seq: 1813, time: 2014-06-09 16:22:11.509126, event: done, op: pg_log(1.6f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509139 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509141 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509153 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509154 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509155 7f869af78700 20 osd.26 243 _dispatch 0x1405c700 pg_notify(3.4ac(10) epoch 243) v5 2014-06-09 16:22:11.509160 7f869af78700 5 -- op tracker -- , seq: 1814, time: 2014-06-09 16:22:11.268343, event: header_read, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.509167 7f869af78700 5 -- op tracker -- , seq: 1814, time: 2014-06-09 16:22:11.268344, event: throttled, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.509174 7f869af78700 5 -- op tracker -- , seq: 1814, time: 2014-06-09 16:22:11.268364, event: all_read, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.509180 7f869af78700 5 -- op tracker -- , seq: 1814, time: 2014-06-09 16:22:11.509152, event: dispatched, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.509186 7f869af78700 5 -- op tracker -- , seq: 1814, time: 2014-06-09 16:22:11.509185, event: waiting_for_osdmap, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.509191 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.509193 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405c700 2014-06-09 16:22:11.509197 7f869af78700 5 -- op tracker -- , seq: 1814, time: 2014-06-09 16:22:11.509195, event: started, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.509215 7f869af78700 5 -- op tracker -- , seq: 1814, time: 2014-06-09 16:22:11.509212, event: done, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.509223 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509225 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509248 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509250 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509251 7f869af78700 20 osd.26 243 _dispatch 0x16fdba40 pg_notify(0.21b(23),2.6f4(20) epoch 243) v5 2014-06-09 16:22:11.509257 7f869af78700 5 -- op tracker -- , seq: 1815, time: 2014-06-09 16:22:11.240313, event: header_read, op: pg_notify(0.21b(23),2.6f4(20) epoch 243) 2014-06-09 16:22:11.509265 7f869af78700 5 -- op tracker -- , seq: 1815, time: 2014-06-09 16:22:11.240313, event: throttled, op: pg_notify(0.21b(23),2.6f4(20) epoch 243) 2014-06-09 16:22:11.509271 7f869af78700 5 -- op tracker -- , seq: 1815, time: 2014-06-09 16:22:11.240344, event: all_read, op: pg_notify(0.21b(23),2.6f4(20) epoch 243) 2014-06-09 16:22:11.509277 7f869af78700 5 -- op tracker -- , seq: 1815, time: 2014-06-09 16:22:11.509247, event: dispatched, op: pg_notify(0.21b(23),2.6f4(20) epoch 243) 2014-06-09 16:22:11.509284 7f869af78700 5 -- op tracker -- , seq: 1815, time: 2014-06-09 16:22:11.509282, event: waiting_for_osdmap, op: pg_notify(0.21b(23),2.6f4(20) epoch 243) 2014-06-09 16:22:11.509288 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.509290 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16fdba40 2014-06-09 16:22:11.509295 7f869af78700 5 -- op tracker -- , seq: 1815, time: 2014-06-09 16:22:11.509293, event: started, op: pg_notify(0.21b(23),2.6f4(20) epoch 243) 2014-06-09 16:22:11.509319 7f869af78700 5 -- op tracker -- , seq: 1815, time: 2014-06-09 16:22:11.509316, event: done, op: pg_notify(0.21b(23),2.6f4(20) epoch 243) 2014-06-09 16:22:11.509327 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509329 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509357 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509359 7f869af78700 20 osd.26 243 _dispatch 0x30727000 pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.509366 7f869af78700 5 -- op tracker -- , seq: 1816, time: 2014-06-09 16:22:11.161343, event: header_read, op: pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509373 7f869af78700 5 -- op tracker -- , seq: 1816, time: 2014-06-09 16:22:11.161343, event: throttled, op: pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509380 7f869af78700 5 -- op tracker -- , seq: 1816, time: 2014-06-09 16:22:11.161357, event: all_read, op: pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509386 7f869af78700 5 -- op tracker -- , seq: 1816, time: 2014-06-09 16:22:11.509355, event: dispatched, op: pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509394 7f869af78700 5 -- op tracker -- , seq: 1816, time: 2014-06-09 16:22:11.509391, event: waiting_for_osdmap, op: pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509399 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.509403 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30727000 2014-06-09 16:22:11.509409 7f869af78700 5 -- op tracker -- , seq: 1816, time: 2014-06-09 16:22:11.509407, event: started, op: pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509422 7f869af78700 5 -- op tracker -- , seq: 1816, time: 2014-06-09 16:22:11.509419, event: done, op: pg_log(1.27c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509430 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509444 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509446 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509447 7f869af78700 20 osd.26 243 _dispatch 0x12060a80 pg_notify(1.522(20),2.25c(20) epoch 243) v5 2014-06-09 16:22:11.509453 7f869af78700 5 -- op tracker -- , seq: 1817, time: 2014-06-09 16:22:11.235298, event: header_read, op: pg_notify(1.522(20),2.25c(20) epoch 243) 2014-06-09 16:22:11.509461 7f869af78700 5 -- op tracker -- , seq: 1817, time: 2014-06-09 16:22:11.235299, event: throttled, op: pg_notify(1.522(20),2.25c(20) epoch 243) 2014-06-09 16:22:11.509467 7f869af78700 5 -- op tracker -- , seq: 1817, time: 2014-06-09 16:22:11.235348, event: all_read, op: pg_notify(1.522(20),2.25c(20) epoch 243) 2014-06-09 16:22:11.509474 7f869af78700 5 -- op tracker -- , seq: 1817, time: 2014-06-09 16:22:11.509444, event: dispatched, op: pg_notify(1.522(20),2.25c(20) epoch 243) 2014-06-09 16:22:11.509481 7f869af78700 5 -- op tracker -- , seq: 1817, time: 2014-06-09 16:22:11.509479, event: waiting_for_osdmap, op: pg_notify(1.522(20),2.25c(20) epoch 243) 2014-06-09 16:22:11.509485 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.509487 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12060a80 2014-06-09 16:22:11.509493 7f869af78700 5 -- op tracker -- , seq: 1817, time: 2014-06-09 16:22:11.509490, event: started, op: pg_notify(1.522(20),2.25c(20) epoch 243) 2014-06-09 16:22:11.509516 7f869af78700 5 -- op tracker -- , seq: 1817, time: 2014-06-09 16:22:11.509513, event: done, op: pg_notify(1.522(20),2.25c(20) epoch 243) 2014-06-09 16:22:11.509524 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509526 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509549 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509551 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509552 7f869af78700 20 osd.26 243 _dispatch 0x2fe70a80 pg_info(1 pgs e243:3.218(2)) v4 2014-06-09 16:22:11.509557 7f869af78700 5 -- op tracker -- , seq: 1818, time: 2014-06-09 16:22:11.159843, event: header_read, op: pg_info(1 pgs e243:3.218(2)) 2014-06-09 16:22:11.509564 7f869af78700 5 -- op tracker -- , seq: 1818, time: 2014-06-09 16:22:11.159843, event: throttled, op: pg_info(1 pgs e243:3.218(2)) 2014-06-09 16:22:11.509570 7f869af78700 5 -- op tracker -- , seq: 1818, time: 2014-06-09 16:22:11.159855, event: all_read, op: pg_info(1 pgs e243:3.218(2)) 2014-06-09 16:22:11.509576 7f869af78700 5 -- op tracker -- , seq: 1818, time: 2014-06-09 16:22:11.509549, event: dispatched, op: pg_info(1 pgs e243:3.218(2)) 2014-06-09 16:22:11.509584 7f869af78700 5 -- op tracker -- , seq: 1818, time: 2014-06-09 16:22:11.509582, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.218(2)) 2014-06-09 16:22:11.509588 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.218(2)) v4 from osd.8 2014-06-09 16:22:11.509592 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe70a80 2014-06-09 16:22:11.509597 7f869af78700 5 -- op tracker -- , seq: 1818, time: 2014-06-09 16:22:11.509595, event: started, op: pg_info(1 pgs e243:3.218(2)) 2014-06-09 16:22:11.509614 7f869af78700 5 -- op tracker -- , seq: 1818, time: 2014-06-09 16:22:11.509611, event: done, op: pg_info(1 pgs e243:3.218(2)) 2014-06-09 16:22:11.509622 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509623 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509636 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509639 7f869af78700 20 osd.26 243 _dispatch 0x3046b180 pg_notify(0.420(20) epoch 243) v5 2014-06-09 16:22:11.509645 7f869af78700 5 -- op tracker -- , seq: 1819, time: 2014-06-09 16:22:11.188116, event: header_read, op: pg_notify(0.420(20) epoch 243) 2014-06-09 16:22:11.509652 7f869af78700 5 -- op tracker -- , seq: 1819, time: 2014-06-09 16:22:11.188117, event: throttled, op: pg_notify(0.420(20) epoch 243) 2014-06-09 16:22:11.509658 7f869af78700 5 -- op tracker -- , seq: 1819, time: 2014-06-09 16:22:11.188155, event: all_read, op: pg_notify(0.420(20) epoch 243) 2014-06-09 16:22:11.509665 7f869af78700 5 -- op tracker -- , seq: 1819, time: 2014-06-09 16:22:11.509636, event: dispatched, op: pg_notify(0.420(20) epoch 243) 2014-06-09 16:22:11.509672 7f869af78700 5 -- op tracker -- , seq: 1819, time: 2014-06-09 16:22:11.509670, event: waiting_for_osdmap, op: pg_notify(0.420(20) epoch 243) 2014-06-09 16:22:11.509676 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.509678 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046b180 2014-06-09 16:22:11.509684 7f869af78700 5 -- op tracker -- , seq: 1819, time: 2014-06-09 16:22:11.509682, event: started, op: pg_notify(0.420(20) epoch 243) 2014-06-09 16:22:11.509699 7f869af78700 5 -- op tracker -- , seq: 1819, time: 2014-06-09 16:22:11.509697, event: done, op: pg_notify(0.420(20) epoch 243) 2014-06-09 16:22:11.509706 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509708 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509727 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509728 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509730 7f869af78700 20 osd.26 243 _dispatch 0x3071aa00 pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.509736 7f869af78700 5 -- op tracker -- , seq: 1820, time: 2014-06-09 16:22:11.147289, event: header_read, op: pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509743 7f869af78700 5 -- op tracker -- , seq: 1820, time: 2014-06-09 16:22:11.147290, event: throttled, op: pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509750 7f869af78700 5 -- op tracker -- , seq: 1820, time: 2014-06-09 16:22:11.147304, event: all_read, op: pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509757 7f869af78700 5 -- op tracker -- , seq: 1820, time: 2014-06-09 16:22:11.509726, event: dispatched, op: pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509764 7f869af78700 5 -- op tracker -- , seq: 1820, time: 2014-06-09 16:22:11.509761, event: waiting_for_osdmap, op: pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509769 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.509773 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3071aa00 2014-06-09 16:22:11.509779 7f869af78700 5 -- op tracker -- , seq: 1820, time: 2014-06-09 16:22:11.509776, event: started, op: pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509792 7f869af78700 5 -- op tracker -- , seq: 1820, time: 2014-06-09 16:22:11.509789, event: done, op: pg_log(1.113 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509799 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509801 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509814 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509816 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509817 7f869af78700 20 osd.26 243 _dispatch 0x30888600 pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.509824 7f869af78700 5 -- op tracker -- , seq: 1821, time: 2014-06-09 16:22:11.243001, event: header_read, op: pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509832 7f869af78700 5 -- op tracker -- , seq: 1821, time: 2014-06-09 16:22:11.243002, event: throttled, op: pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509838 7f869af78700 5 -- op tracker -- , seq: 1821, time: 2014-06-09 16:22:11.243016, event: all_read, op: pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509845 7f869af78700 5 -- op tracker -- , seq: 1821, time: 2014-06-09 16:22:11.509814, event: dispatched, op: pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509853 7f869af78700 5 -- op tracker -- , seq: 1821, time: 2014-06-09 16:22:11.509850, event: waiting_for_osdmap, op: pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509858 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.509862 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30888600 2014-06-09 16:22:11.509867 7f869af78700 5 -- op tracker -- , seq: 1821, time: 2014-06-09 16:22:11.509865, event: started, op: pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509882 7f869af78700 5 -- op tracker -- , seq: 1821, time: 2014-06-09 16:22:11.509879, event: done, op: pg_log(0.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509889 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509891 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509903 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509905 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509906 7f869af78700 20 osd.26 243 _dispatch 0x3045ca00 pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.509913 7f869af78700 5 -- op tracker -- , seq: 1822, time: 2014-06-09 16:22:11.031275, event: header_read, op: pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509921 7f869af78700 5 -- op tracker -- , seq: 1822, time: 2014-06-09 16:22:11.031276, event: throttled, op: pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509928 7f869af78700 5 -- op tracker -- , seq: 1822, time: 2014-06-09 16:22:11.031287, event: all_read, op: pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509935 7f869af78700 5 -- op tracker -- , seq: 1822, time: 2014-06-09 16:22:11.509903, event: dispatched, op: pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509942 7f869af78700 5 -- op tracker -- , seq: 1822, time: 2014-06-09 16:22:11.509940, event: waiting_for_osdmap, op: pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509947 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.509951 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045ca00 2014-06-09 16:22:11.509957 7f869af78700 5 -- op tracker -- , seq: 1822, time: 2014-06-09 16:22:11.509955, event: started, op: pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509970 7f869af78700 5 -- op tracker -- , seq: 1822, time: 2014-06-09 16:22:11.509967, event: done, op: pg_log(2.c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.509978 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509979 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509991 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.509992 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.509993 7f869af78700 20 osd.26 243 _dispatch 0x3046b500 pg_info(1 pgs e243:3.4fe(2)) v4 2014-06-09 16:22:11.509998 7f869af78700 5 -- op tracker -- , seq: 1823, time: 2014-06-09 16:22:11.050846, event: header_read, op: pg_info(1 pgs e243:3.4fe(2)) 2014-06-09 16:22:11.510005 7f869af78700 5 -- op tracker -- , seq: 1823, time: 2014-06-09 16:22:11.050846, event: throttled, op: pg_info(1 pgs e243:3.4fe(2)) 2014-06-09 16:22:11.510011 7f869af78700 5 -- op tracker -- , seq: 1823, time: 2014-06-09 16:22:11.050861, event: all_read, op: pg_info(1 pgs e243:3.4fe(2)) 2014-06-09 16:22:11.510018 7f869af78700 5 -- op tracker -- , seq: 1823, time: 2014-06-09 16:22:11.509990, event: dispatched, op: pg_info(1 pgs e243:3.4fe(2)) 2014-06-09 16:22:11.510025 7f869af78700 5 -- op tracker -- , seq: 1823, time: 2014-06-09 16:22:11.510022, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4fe(2)) 2014-06-09 16:22:11.510029 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4fe(2)) v4 from osd.4 2014-06-09 16:22:11.510032 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046b500 2014-06-09 16:22:11.510038 7f869af78700 5 -- op tracker -- , seq: 1823, time: 2014-06-09 16:22:11.510036, event: started, op: pg_info(1 pgs e243:3.4fe(2)) 2014-06-09 16:22:11.510055 7f869af78700 5 -- op tracker -- , seq: 1823, time: 2014-06-09 16:22:11.510052, event: done, op: pg_info(1 pgs e243:3.4fe(2)) 2014-06-09 16:22:11.510063 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510065 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510078 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510079 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510080 7f869af78700 20 osd.26 243 _dispatch 0x30889200 pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.510087 7f869af78700 5 -- op tracker -- , seq: 1824, time: 2014-06-09 16:22:11.268428, event: header_read, op: pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510094 7f869af78700 5 -- op tracker -- , seq: 1824, time: 2014-06-09 16:22:11.268429, event: throttled, op: pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510101 7f869af78700 5 -- op tracker -- , seq: 1824, time: 2014-06-09 16:22:11.268441, event: all_read, op: pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510108 7f869af78700 5 -- op tracker -- , seq: 1824, time: 2014-06-09 16:22:11.510077, event: dispatched, op: pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510115 7f869af78700 5 -- op tracker -- , seq: 1824, time: 2014-06-09 16:22:11.510113, event: waiting_for_osdmap, op: pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510120 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.510124 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30889200 2014-06-09 16:22:11.510129 7f869af78700 5 -- op tracker -- , seq: 1824, time: 2014-06-09 16:22:11.510127, event: started, op: pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510144 7f869af78700 5 -- op tracker -- , seq: 1824, time: 2014-06-09 16:22:11.510140, event: done, op: pg_log(0.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510151 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510153 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510166 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510168 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510169 7f869af78700 20 osd.26 243 _dispatch 0x307de000 pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.510175 7f869af78700 5 -- op tracker -- , seq: 1825, time: 2014-06-09 16:22:11.251147, event: header_read, op: pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510183 7f869af78700 5 -- op tracker -- , seq: 1825, time: 2014-06-09 16:22:11.251148, event: throttled, op: pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510189 7f869af78700 5 -- op tracker -- , seq: 1825, time: 2014-06-09 16:22:11.251163, event: all_read, op: pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510196 7f869af78700 5 -- op tracker -- , seq: 1825, time: 2014-06-09 16:22:11.510166, event: dispatched, op: pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510204 7f869af78700 5 -- op tracker -- , seq: 1825, time: 2014-06-09 16:22:11.510201, event: waiting_for_osdmap, op: pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510208 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.510212 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307de000 2014-06-09 16:22:11.510218 7f869af78700 5 -- op tracker -- , seq: 1825, time: 2014-06-09 16:22:11.510215, event: started, op: pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510231 7f869af78700 5 -- op tracker -- , seq: 1825, time: 2014-06-09 16:22:11.510228, event: done, op: pg_log(2.13c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510238 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510241 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510253 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510254 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510255 7f869af78700 20 osd.26 243 _dispatch 0x3005bc00 pg_notify(2.28a(21),3.55d(12) epoch 243) v5 2014-06-09 16:22:11.510261 7f869af78700 5 -- op tracker -- , seq: 1826, time: 2014-06-09 16:22:11.162242, event: header_read, op: pg_notify(2.28a(21),3.55d(12) epoch 243) 2014-06-09 16:22:11.510269 7f869af78700 5 -- op tracker -- , seq: 1826, time: 2014-06-09 16:22:11.162242, event: throttled, op: pg_notify(2.28a(21),3.55d(12) epoch 243) 2014-06-09 16:22:11.510276 7f869af78700 5 -- op tracker -- , seq: 1826, time: 2014-06-09 16:22:11.162286, event: all_read, op: pg_notify(2.28a(21),3.55d(12) epoch 243) 2014-06-09 16:22:11.510282 7f869af78700 5 -- op tracker -- , seq: 1826, time: 2014-06-09 16:22:11.510253, event: dispatched, op: pg_notify(2.28a(21),3.55d(12) epoch 243) 2014-06-09 16:22:11.510289 7f869af78700 5 -- op tracker -- , seq: 1826, time: 2014-06-09 16:22:11.510287, event: waiting_for_osdmap, op: pg_notify(2.28a(21),3.55d(12) epoch 243) 2014-06-09 16:22:11.510293 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.510295 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005bc00 2014-06-09 16:22:11.510300 7f869af78700 5 -- op tracker -- , seq: 1826, time: 2014-06-09 16:22:11.510298, event: started, op: pg_notify(2.28a(21),3.55d(12) epoch 243) 2014-06-09 16:22:11.510324 7f869af78700 5 -- op tracker -- , seq: 1826, time: 2014-06-09 16:22:11.510322, event: done, op: pg_notify(2.28a(21),3.55d(12) epoch 243) 2014-06-09 16:22:11.510332 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510334 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510358 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510359 7f869af78700 20 osd.26 243 _dispatch 0x308d81c0 pg_notify(0.732(23),1.46e(20) epoch 243) v5 2014-06-09 16:22:11.510364 7f869af78700 5 -- op tracker -- , seq: 1827, time: 2014-06-09 16:22:11.243634, event: header_read, op: pg_notify(0.732(23),1.46e(20) epoch 243) 2014-06-09 16:22:11.510371 7f869af78700 5 -- op tracker -- , seq: 1827, time: 2014-06-09 16:22:11.243635, event: throttled, op: pg_notify(0.732(23),1.46e(20) epoch 243) 2014-06-09 16:22:11.510378 7f869af78700 5 -- op tracker -- , seq: 1827, time: 2014-06-09 16:22:11.243689, event: all_read, op: pg_notify(0.732(23),1.46e(20) epoch 243) 2014-06-09 16:22:11.510385 7f869af78700 5 -- op tracker -- , seq: 1827, time: 2014-06-09 16:22:11.510355, event: dispatched, op: pg_notify(0.732(23),1.46e(20) epoch 243) 2014-06-09 16:22:11.510392 7f869af78700 5 -- op tracker -- , seq: 1827, time: 2014-06-09 16:22:11.510390, event: waiting_for_osdmap, op: pg_notify(0.732(23),1.46e(20) epoch 243) 2014-06-09 16:22:11.510397 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.510399 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d81c0 2014-06-09 16:22:11.510404 7f869af78700 5 -- op tracker -- , seq: 1827, time: 2014-06-09 16:22:11.510402, event: started, op: pg_notify(0.732(23),1.46e(20) epoch 243) 2014-06-09 16:22:11.510428 7f869af78700 5 -- op tracker -- , seq: 1827, time: 2014-06-09 16:22:11.510426, event: done, op: pg_notify(0.732(23),1.46e(20) epoch 243) 2014-06-09 16:22:11.510436 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510438 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510461 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510463 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510465 7f869af78700 20 osd.26 243 _dispatch 0x1205e1c0 pg_notify(3.647(10) epoch 243) v5 2014-06-09 16:22:11.510469 7f869af78700 5 -- op tracker -- , seq: 1828, time: 2014-06-09 16:22:11.159864, event: header_read, op: pg_notify(3.647(10) epoch 243) 2014-06-09 16:22:11.510477 7f869af78700 5 -- op tracker -- , seq: 1828, time: 2014-06-09 16:22:11.159865, event: throttled, op: pg_notify(3.647(10) epoch 243) 2014-06-09 16:22:11.510483 7f869af78700 5 -- op tracker -- , seq: 1828, time: 2014-06-09 16:22:11.159884, event: all_read, op: pg_notify(3.647(10) epoch 243) 2014-06-09 16:22:11.510490 7f869af78700 5 -- op tracker -- , seq: 1828, time: 2014-06-09 16:22:11.510461, event: dispatched, op: pg_notify(3.647(10) epoch 243) 2014-06-09 16:22:11.510496 7f869af78700 5 -- op tracker -- , seq: 1828, time: 2014-06-09 16:22:11.510494, event: waiting_for_osdmap, op: pg_notify(3.647(10) epoch 243) 2014-06-09 16:22:11.510500 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.510502 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1205e1c0 2014-06-09 16:22:11.510507 7f869af78700 5 -- op tracker -- , seq: 1828, time: 2014-06-09 16:22:11.510505, event: started, op: pg_notify(3.647(10) epoch 243) 2014-06-09 16:22:11.510525 7f869af78700 5 -- op tracker -- , seq: 1828, time: 2014-06-09 16:22:11.510523, event: done, op: pg_notify(3.647(10) epoch 243) 2014-06-09 16:22:11.510532 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510534 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510549 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510550 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510551 7f869af78700 20 osd.26 243 _dispatch 0x2fef48c0 pg_notify(1.685(23) epoch 243) v5 2014-06-09 16:22:11.510556 7f869af78700 5 -- op tracker -- , seq: 1829, time: 2014-06-09 16:22:11.188182, event: header_read, op: pg_notify(1.685(23) epoch 243) 2014-06-09 16:22:11.510563 7f869af78700 5 -- op tracker -- , seq: 1829, time: 2014-06-09 16:22:11.188183, event: throttled, op: pg_notify(1.685(23) epoch 243) 2014-06-09 16:22:11.510570 7f869af78700 5 -- op tracker -- , seq: 1829, time: 2014-06-09 16:22:11.188212, event: all_read, op: pg_notify(1.685(23) epoch 243) 2014-06-09 16:22:11.510576 7f869af78700 5 -- op tracker -- , seq: 1829, time: 2014-06-09 16:22:11.510548, event: dispatched, op: pg_notify(1.685(23) epoch 243) 2014-06-09 16:22:11.510583 7f869af78700 5 -- op tracker -- , seq: 1829, time: 2014-06-09 16:22:11.510581, event: waiting_for_osdmap, op: pg_notify(1.685(23) epoch 243) 2014-06-09 16:22:11.510587 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.510589 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef48c0 2014-06-09 16:22:11.510594 7f869af78700 5 -- op tracker -- , seq: 1829, time: 2014-06-09 16:22:11.510592, event: started, op: pg_notify(1.685(23) epoch 243) 2014-06-09 16:22:11.510611 7f869af78700 5 -- op tracker -- , seq: 1829, time: 2014-06-09 16:22:11.510609, event: done, op: pg_notify(1.685(23) epoch 243) 2014-06-09 16:22:11.510618 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510620 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510638 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510640 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510641 7f869af78700 20 osd.26 243 _dispatch 0x3071a400 pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.510648 7f869af78700 5 -- op tracker -- , seq: 1830, time: 2014-06-09 16:22:11.147862, event: header_read, op: pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510656 7f869af78700 5 -- op tracker -- , seq: 1830, time: 2014-06-09 16:22:11.147862, event: throttled, op: pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510663 7f869af78700 5 -- op tracker -- , seq: 1830, time: 2014-06-09 16:22:11.147872, event: all_read, op: pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510670 7f869af78700 5 -- op tracker -- , seq: 1830, time: 2014-06-09 16:22:11.510638, event: dispatched, op: pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510677 7f869af78700 5 -- op tracker -- , seq: 1830, time: 2014-06-09 16:22:11.510675, event: waiting_for_osdmap, op: pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510682 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.510686 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3071a400 2014-06-09 16:22:11.510692 7f869af78700 5 -- op tracker -- , seq: 1830, time: 2014-06-09 16:22:11.510689, event: started, op: pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510705 7f869af78700 5 -- op tracker -- , seq: 1830, time: 2014-06-09 16:22:11.510702, event: done, op: pg_log(0.117 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510713 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510715 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510726 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510728 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510729 7f869af78700 20 osd.26 243 _dispatch 0x30726a00 pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.510736 7f869af78700 5 -- op tracker -- , seq: 1831, time: 2014-06-09 16:22:11.243520, event: header_read, op: pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510743 7f869af78700 5 -- op tracker -- , seq: 1831, time: 2014-06-09 16:22:11.243521, event: throttled, op: pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510750 7f869af78700 5 -- op tracker -- , seq: 1831, time: 2014-06-09 16:22:11.243533, event: all_read, op: pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510758 7f869af78700 5 -- op tracker -- , seq: 1831, time: 2014-06-09 16:22:11.510726, event: dispatched, op: pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510765 7f869af78700 5 -- op tracker -- , seq: 1831, time: 2014-06-09 16:22:11.510763, event: waiting_for_osdmap, op: pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510770 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.510774 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30726a00 2014-06-09 16:22:11.510780 7f869af78700 5 -- op tracker -- , seq: 1831, time: 2014-06-09 16:22:11.510777, event: started, op: pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510792 7f869af78700 5 -- op tracker -- , seq: 1831, time: 2014-06-09 16:22:11.510789, event: done, op: pg_log(1.77c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510800 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510802 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510813 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510814 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510815 7f869af78700 20 osd.26 243 _dispatch 0x2fef4c40 pg_notify(1.669(21) epoch 243) v5 2014-06-09 16:22:11.510820 7f869af78700 5 -- op tracker -- , seq: 1832, time: 2014-06-09 16:22:11.043032, event: header_read, op: pg_notify(1.669(21) epoch 243) 2014-06-09 16:22:11.510827 7f869af78700 5 -- op tracker -- , seq: 1832, time: 2014-06-09 16:22:11.043033, event: throttled, op: pg_notify(1.669(21) epoch 243) 2014-06-09 16:22:11.510833 7f869af78700 5 -- op tracker -- , seq: 1832, time: 2014-06-09 16:22:11.043067, event: all_read, op: pg_notify(1.669(21) epoch 243) 2014-06-09 16:22:11.510840 7f869af78700 5 -- op tracker -- , seq: 1832, time: 2014-06-09 16:22:11.510812, event: dispatched, op: pg_notify(1.669(21) epoch 243) 2014-06-09 16:22:11.510847 7f869af78700 5 -- op tracker -- , seq: 1832, time: 2014-06-09 16:22:11.510845, event: waiting_for_osdmap, op: pg_notify(1.669(21) epoch 243) 2014-06-09 16:22:11.510852 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.510854 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4c40 2014-06-09 16:22:11.510859 7f869af78700 5 -- op tracker -- , seq: 1832, time: 2014-06-09 16:22:11.510857, event: started, op: pg_notify(1.669(21) epoch 243) 2014-06-09 16:22:11.510876 7f869af78700 5 -- op tracker -- , seq: 1832, time: 2014-06-09 16:22:11.510874, event: done, op: pg_notify(1.669(21) epoch 243) 2014-06-09 16:22:11.510884 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510886 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510904 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510906 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510907 7f869af78700 20 osd.26 243 _dispatch 0x14dca000 pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.510913 7f869af78700 5 -- op tracker -- , seq: 1833, time: 2014-06-09 16:22:11.050869, event: header_read, op: pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510921 7f869af78700 5 -- op tracker -- , seq: 1833, time: 2014-06-09 16:22:11.050869, event: throttled, op: pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510928 7f869af78700 5 -- op tracker -- , seq: 1833, time: 2014-06-09 16:22:11.050881, event: all_read, op: pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510935 7f869af78700 5 -- op tracker -- , seq: 1833, time: 2014-06-09 16:22:11.510903, event: dispatched, op: pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510942 7f869af78700 5 -- op tracker -- , seq: 1833, time: 2014-06-09 16:22:11.510939, event: waiting_for_osdmap, op: pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510947 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.510951 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dca000 2014-06-09 16:22:11.510957 7f869af78700 5 -- op tracker -- , seq: 1833, time: 2014-06-09 16:22:11.510954, event: started, op: pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510971 7f869af78700 5 -- op tracker -- , seq: 1833, time: 2014-06-09 16:22:11.510968, event: done, op: pg_log(0.33a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.510979 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510980 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510992 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.510993 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.510994 7f869af78700 20 osd.26 243 _dispatch 0x3005a700 pg_notify(3.4ac(10) epoch 243) v5 2014-06-09 16:22:11.510999 7f869af78700 5 -- op tracker -- , seq: 1834, time: 2014-06-09 16:22:11.268479, event: header_read, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.511006 7f869af78700 5 -- op tracker -- , seq: 1834, time: 2014-06-09 16:22:11.268480, event: throttled, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.511013 7f869af78700 5 -- op tracker -- , seq: 1834, time: 2014-06-09 16:22:11.268501, event: all_read, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.511019 7f869af78700 5 -- op tracker -- , seq: 1834, time: 2014-06-09 16:22:11.510991, event: dispatched, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.511026 7f869af78700 5 -- op tracker -- , seq: 1834, time: 2014-06-09 16:22:11.511024, event: waiting_for_osdmap, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.511030 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.511032 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005a700 2014-06-09 16:22:11.511037 7f869af78700 5 -- op tracker -- , seq: 1834, time: 2014-06-09 16:22:11.511035, event: started, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.511054 7f869af78700 5 -- op tracker -- , seq: 1834, time: 2014-06-09 16:22:11.511051, event: done, op: pg_notify(3.4ac(10) epoch 243) 2014-06-09 16:22:11.511062 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511064 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511079 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511080 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511082 7f869af78700 20 osd.26 243 _dispatch 0x307e6540 pg_notify(3.5f9(10),0.13b(20) epoch 243) v5 2014-06-09 16:22:11.511088 7f869af78700 5 -- op tracker -- , seq: 1835, time: 2014-06-09 16:22:11.251247, event: header_read, op: pg_notify(3.5f9(10),0.13b(20) epoch 243) 2014-06-09 16:22:11.511095 7f869af78700 5 -- op tracker -- , seq: 1835, time: 2014-06-09 16:22:11.251248, event: throttled, op: pg_notify(3.5f9(10),0.13b(20) epoch 243) 2014-06-09 16:22:11.511101 7f869af78700 5 -- op tracker -- , seq: 1835, time: 2014-06-09 16:22:11.251298, event: all_read, op: pg_notify(3.5f9(10),0.13b(20) epoch 243) 2014-06-09 16:22:11.511108 7f869af78700 5 -- op tracker -- , seq: 1835, time: 2014-06-09 16:22:11.511079, event: dispatched, op: pg_notify(3.5f9(10),0.13b(20) epoch 243) 2014-06-09 16:22:11.511115 7f869af78700 5 -- op tracker -- , seq: 1835, time: 2014-06-09 16:22:11.511113, event: waiting_for_osdmap, op: pg_notify(3.5f9(10),0.13b(20) epoch 243) 2014-06-09 16:22:11.511119 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.511121 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e6540 2014-06-09 16:22:11.511126 7f869af78700 5 -- op tracker -- , seq: 1835, time: 2014-06-09 16:22:11.511124, event: started, op: pg_notify(3.5f9(10),0.13b(20) epoch 243) 2014-06-09 16:22:11.511150 7f869af78700 5 -- op tracker -- , seq: 1835, time: 2014-06-09 16:22:11.511148, event: done, op: pg_notify(3.5f9(10),0.13b(20) epoch 243) 2014-06-09 16:22:11.511158 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511160 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511183 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511185 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511186 7f869af78700 20 osd.26 243 _dispatch 0x3070c600 pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.511193 7f869af78700 5 -- op tracker -- , seq: 1836, time: 2014-06-09 16:22:11.165258, event: header_read, op: pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511201 7f869af78700 5 -- op tracker -- , seq: 1836, time: 2014-06-09 16:22:11.165259, event: throttled, op: pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511208 7f869af78700 5 -- op tracker -- , seq: 1836, time: 2014-06-09 16:22:11.165273, event: all_read, op: pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511215 7f869af78700 5 -- op tracker -- , seq: 1836, time: 2014-06-09 16:22:11.511183, event: dispatched, op: pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511223 7f869af78700 5 -- op tracker -- , seq: 1836, time: 2014-06-09 16:22:11.511220, event: waiting_for_osdmap, op: pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511228 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.511232 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070c600 2014-06-09 16:22:11.511237 7f869af78700 5 -- op tracker -- , seq: 1836, time: 2014-06-09 16:22:11.511234, event: started, op: pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511250 7f869af78700 5 -- op tracker -- , seq: 1836, time: 2014-06-09 16:22:11.511247, event: done, op: pg_log(1.198 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511257 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511259 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511272 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511273 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511275 7f869af78700 20 osd.26 243 _dispatch 0x308a9000 pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.511281 7f869af78700 5 -- op tracker -- , seq: 1837, time: 2014-06-09 16:22:11.243703, event: header_read, op: pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511289 7f869af78700 5 -- op tracker -- , seq: 1837, time: 2014-06-09 16:22:11.243704, event: throttled, op: pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511296 7f869af78700 5 -- op tracker -- , seq: 1837, time: 2014-06-09 16:22:11.243716, event: all_read, op: pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511303 7f869af78700 5 -- op tracker -- , seq: 1837, time: 2014-06-09 16:22:11.511271, event: dispatched, op: pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511311 7f869af78700 5 -- op tracker -- , seq: 1837, time: 2014-06-09 16:22:11.511308, event: waiting_for_osdmap, op: pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511315 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.511319 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a9000 2014-06-09 16:22:11.511324 7f869af78700 5 -- op tracker -- , seq: 1837, time: 2014-06-09 16:22:11.511322, event: started, op: pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511337 7f869af78700 5 -- op tracker -- , seq: 1837, time: 2014-06-09 16:22:11.511334, event: done, op: pg_log(0.473 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511344 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511346 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511358 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511360 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511361 7f869af78700 20 osd.26 243 _dispatch 0x307e1600 pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.511367 7f869af78700 5 -- op tracker -- , seq: 1838, time: 2014-06-09 16:22:11.181431, event: header_read, op: pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511375 7f869af78700 5 -- op tracker -- , seq: 1838, time: 2014-06-09 16:22:11.181432, event: throttled, op: pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511383 7f869af78700 5 -- op tracker -- , seq: 1838, time: 2014-06-09 16:22:11.181447, event: all_read, op: pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511390 7f869af78700 5 -- op tracker -- , seq: 1838, time: 2014-06-09 16:22:11.511358, event: dispatched, op: pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511397 7f869af78700 5 -- op tracker -- , seq: 1838, time: 2014-06-09 16:22:11.511394, event: waiting_for_osdmap, op: pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511402 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.511406 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e1600 2014-06-09 16:22:11.511413 7f869af78700 5 -- op tracker -- , seq: 1838, time: 2014-06-09 16:22:11.511410, event: started, op: pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511426 7f869af78700 5 -- op tracker -- , seq: 1838, time: 2014-06-09 16:22:11.511423, event: done, op: pg_log(1.3c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.511435 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511437 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511449 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511451 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511452 7f869af78700 20 osd.26 243 _dispatch 0x140508c0 pg_info(2 pgs e243:3.e8(2),3.688(2)) v4 2014-06-09 16:22:11.511458 7f869af78700 5 -- op tracker -- , seq: 1839, time: 2014-06-09 16:22:11.188240, event: header_read, op: pg_info(2 pgs e243:3.e8(2),3.688(2)) 2014-06-09 16:22:11.511466 7f869af78700 5 -- op tracker -- , seq: 1839, time: 2014-06-09 16:22:11.188240, event: throttled, op: pg_info(2 pgs e243:3.e8(2),3.688(2)) 2014-06-09 16:22:11.511473 7f869af78700 5 -- op tracker -- , seq: 1839, time: 2014-06-09 16:22:11.188262, event: all_read, op: pg_info(2 pgs e243:3.e8(2),3.688(2)) 2014-06-09 16:22:11.511480 7f869af78700 5 -- op tracker -- , seq: 1839, time: 2014-06-09 16:22:11.511449, event: dispatched, op: pg_info(2 pgs e243:3.e8(2),3.688(2)) 2014-06-09 16:22:11.511487 7f869af78700 5 -- op tracker -- , seq: 1839, time: 2014-06-09 16:22:11.511484, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.e8(2),3.688(2)) 2014-06-09 16:22:11.511492 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.e8(2),3.688(2)) v4 from osd.6 2014-06-09 16:22:11.511496 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x140508c0 2014-06-09 16:22:11.511501 7f869af78700 5 -- op tracker -- , seq: 1839, time: 2014-06-09 16:22:11.511499, event: started, op: pg_info(2 pgs e243:3.e8(2),3.688(2)) 2014-06-09 16:22:11.511526 7f869af78700 5 -- op tracker -- , seq: 1839, time: 2014-06-09 16:22:11.511523, event: done, op: pg_info(2 pgs e243:3.e8(2),3.688(2)) 2014-06-09 16:22:11.511534 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511536 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511551 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.511552 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.511553 7f869af78700 20 osd.26 243 _dispatch 0x3046aa80 pg_notify(3.362(12) epoch 243) v5 2014-06-09 16:22:11.511559 7f869af78700 5 -- op tracker -- , seq: 1840, time: 2014-06-09 16:22:11.151826, event: header_read, op: pg_notify(3.362(12) epoch 243) 2014-06-09 16:22:11.511565 7f869af78700 5 -- op tracker -- , seq: 1840, time: 2014-06-09 16:22:11.151826, event: throttled, op: pg_notify(3.362(12) epoch 243) 2014-06-09 16:22:11.511572 7f869af78700 5 -- op tracker -- , seq: 1840, time: 2014-06-09 16:22:11.151848, event: all_read, op: pg_notify(3.362(12) epoch 243) 2014-06-09 16:22:11.511578 7f869af78700 5 -- op tracker -- , seq: 1840, time: 2014-06-09 16:22:11.511550, event: dispatched, op: pg_notify(3.362(12) epoch 243) 2014-06-09 16:22:11.511585 7f869af78700 5 -- op tracker -- , seq: 1840, time: 2014-06-09 16:22:11.511582, event: waiting_for_osdmap, op: pg_notify(3.362(12) epoch 243) 2014-06-09 16:22:11.511589 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.511591 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046aa80 2014-06-09 16:22:11.511595 7f869af78700 5 -- op tracker -- , seq: 1840, time: 2014-06-09 16:22:11.511594, event: started, op: pg_notify(3.362(12) epoch 243) 2014-06-09 16:22:11.511639 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (12 active) 2014-06-09 16:22:11.511644 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.511657 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 0.2fd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.511670 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.511674 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.511678 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.008076 7 0.002994 2014-06-09 16:22:11.511690 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.511701 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.511711 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.511720 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.511728 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.511781 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.511787 7f868f761700 5 queue_transactions existing osr(0.2fd 0x2bd6e50)/0x2bd6e50 2014-06-09 16:22:11.511790 7f868f761700 2 waiting 51 > 50 ops || 92251 > 104857600 2014-06-09 16:22:11.511813 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (13 active) 2014-06-09 16:22:11.511818 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:2.25c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.511832 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.511843 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.511848 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=212 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) 2014-06-09 16:22:11.511860 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=212 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) 2014-06-09 16:22:11.511873 7f868af58700 5 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=212 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.021182 7 0.000146 2014-06-09 16:22:11.511885 7f868af58700 5 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=212 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.511896 7f868af58700 5 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=212 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.511906 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=212 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.511916 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'533 -> 213'533 2014-06-09 16:22:11.511950 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.511990 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.511995 7f868af58700 5 queue_transactions existing osr(3.218 0x3ed0cd0)/0x3ed0cd0 2014-06-09 16:22:11.511998 7f868af58700 2 waiting 51 > 50 ops || 91683 > 104857600 2014-06-09 16:22:11.512010 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40a20 (14 active) 2014-06-09 16:22:11.512016 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.13b( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.512030 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.512039 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.512044 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.512053 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.198( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.512064 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.512068 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.512072 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.198( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.795756 7 0.000165 2014-06-09 16:22:11.512083 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.198( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.512093 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.198( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.512104 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.512113 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.512121 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.512166 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.512172 7f868c75b700 5 queue_transactions existing osr(1.198 0x384cb30)/0x384cb30 2014-06-09 16:22:11.512174 7f868c75b700 2 waiting 51 > 50 ops || 92231 > 104857600 2014-06-09 16:22:11.512182 7f8692f68700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.512188 7f868a757700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.512193 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.512199 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.512204 7f8689755700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.512209 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.511612 7f869af78700 5 -- op tracker -- , seq: 1840, time: 2014-06-09 16:22:11.511610, event: done, op: pg_notify(3.362(12) epoch 243) 2014-06-09 16:22:11.512223 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.512225 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.512242 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.512244 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.512245 7f869af78700 20 osd.26 243 _dispatch 0x307d7600 pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.512252 7f869af78700 5 -- op tracker -- , seq: 1841, time: 2014-06-09 16:22:11.243604, event: header_read, op: pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512261 7f869af78700 5 -- op tracker -- , seq: 1841, time: 2014-06-09 16:22:11.243605, event: throttled, op: pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512268 7f869af78700 5 -- op tracker -- , seq: 1841, time: 2014-06-09 16:22:11.243629, event: all_read, op: pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512275 7f869af78700 5 -- op tracker -- , seq: 1841, time: 2014-06-09 16:22:11.512241, event: dispatched, op: pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512283 7f869af78700 5 -- op tracker -- , seq: 1841, time: 2014-06-09 16:22:11.512280, event: waiting_for_osdmap, op: pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512288 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.512292 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307d7600 2014-06-09 16:22:11.512298 7f869af78700 5 -- op tracker -- , seq: 1841, time: 2014-06-09 16:22:11.512295, event: started, op: pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512318 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (15 active) 2014-06-09 16:22:11.512326 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.512339 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 1.48e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.512351 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.512355 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.512359 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.601944 7 0.000196 2014-06-09 16:22:11.512370 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.512381 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.512390 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.512399 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.512407 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.512453 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.512458 7f8692f68700 5 queue_transactions existing osr(1.48e 0x384d9b0)/0x384d9b0 2014-06-09 16:22:11.512461 7f8692f68700 2 waiting 51 > 50 ops || 92357 > 104857600 2014-06-09 16:22:11.512470 7f869af78700 5 -- op tracker -- , seq: 1841, time: 2014-06-09 16:22:11.512467, event: done, op: pg_log(1.48e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512479 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.512481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.512495 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.512497 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.512498 7f869af78700 20 osd.26 243 _dispatch 0x14dcc400 pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.512504 7f869af78700 5 -- op tracker -- , seq: 1842, time: 2014-06-09 16:22:11.043078, event: header_read, op: pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512512 7f869af78700 5 -- op tracker -- , seq: 1842, time: 2014-06-09 16:22:11.043078, event: throttled, op: pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512519 7f869af78700 5 -- op tracker -- , seq: 1842, time: 2014-06-09 16:22:11.043097, event: all_read, op: pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512526 7f869af78700 5 -- op tracker -- , seq: 1842, time: 2014-06-09 16:22:11.512494, event: dispatched, op: pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512533 7f869af78700 5 -- op tracker -- , seq: 1842, time: 2014-06-09 16:22:11.512531, event: waiting_for_osdmap, op: pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512538 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.512542 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dcc400 2014-06-09 16:22:11.512547 7f869af78700 5 -- op tracker -- , seq: 1842, time: 2014-06-09 16:22:11.512545, event: started, op: pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512568 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (16 active) 2014-06-09 16:22:11.512576 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.512589 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.d1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.512602 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.512606 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.512610 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.240361 7 0.000173 2014-06-09 16:22:11.512620 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.512630 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.512641 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.512650 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.512659 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.512707 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.512713 7f868a757700 5 queue_transactions existing osr(1.d1 0x384c9b0)/0x384c9b0 2014-06-09 16:22:11.512715 7f868a757700 2 waiting 51 > 50 ops || 92228 > 104857600 2014-06-09 16:22:11.512724 7f869af78700 5 -- op tracker -- , seq: 1842, time: 2014-06-09 16:22:11.512721, event: done, op: pg_log(1.d1 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.512734 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.512736 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.512750 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.512751 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.512753 7f869af78700 20 osd.26 243 _dispatch 0x11f3c000 pg_notify(3.6d(10) epoch 243) v5 2014-06-09 16:22:11.512758 7f869af78700 5 -- op tracker -- , seq: 1843, time: 2014-06-09 16:22:11.050891, event: header_read, op: pg_notify(3.6d(10) epoch 243) 2014-06-09 16:22:11.512765 7f869af78700 5 -- op tracker -- , seq: 1843, time: 2014-06-09 16:22:11.050891, event: throttled, op: pg_notify(3.6d(10) epoch 243) 2014-06-09 16:22:11.512771 7f869af78700 5 -- op tracker -- , seq: 1843, time: 2014-06-09 16:22:11.050906, event: all_read, op: pg_notify(3.6d(10) epoch 243) 2014-06-09 16:22:11.512777 7f869af78700 5 -- op tracker -- , seq: 1843, time: 2014-06-09 16:22:11.512749, event: dispatched, op: pg_notify(3.6d(10) epoch 243) 2014-06-09 16:22:11.512784 7f869af78700 5 -- op tracker -- , seq: 1843, time: 2014-06-09 16:22:11.512782, event: waiting_for_osdmap, op: pg_notify(3.6d(10) epoch 243) 2014-06-09 16:22:11.512788 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.512790 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x11f3c000 2014-06-09 16:22:11.512795 7f869af78700 5 -- op tracker -- , seq: 1843, time: 2014-06-09 16:22:11.512793, event: started, op: pg_notify(3.6d(10) epoch 243) 2014-06-09 16:22:11.512818 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (17 active) 2014-06-09 16:22:11.512827 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.6d( v 214'604 (0'0,214'604] local-les=243 n=114 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.6d( v 214'604 (0'0,214'604] local-les=203 n=114 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.512844 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.6d( v 214'604 (0'0,214'604] local-les=243 n=114 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.512855 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.512862 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (16 active) 2014-06-09 16:22:11.512865 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.512873 7f869af78700 5 -- op tracker -- , seq: 1843, time: 2014-06-09 16:22:11.512871, event: done, op: pg_notify(3.6d(10) epoch 243) 2014-06-09 16:22:11.512882 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.512885 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.512910 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.512912 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.512914 7f869af78700 20 osd.26 243 _dispatch 0x3088dc00 pg_notify(1.2b0(20) epoch 243) v5 2014-06-09 16:22:11.512919 7f869af78700 5 -- op tracker -- , seq: 1844, time: 2014-06-09 16:22:11.268527, event: header_read, op: pg_notify(1.2b0(20) epoch 243) 2014-06-09 16:22:11.512935 7f869af78700 5 -- op tracker -- , seq: 1844, time: 2014-06-09 16:22:11.268527, event: throttled, op: pg_notify(1.2b0(20) epoch 243) 2014-06-09 16:22:11.512942 7f869af78700 5 -- op tracker -- , seq: 1844, time: 2014-06-09 16:22:11.268562, event: all_read, op: pg_notify(1.2b0(20) epoch 243) 2014-06-09 16:22:11.512948 7f869af78700 5 -- op tracker -- , seq: 1844, time: 2014-06-09 16:22:11.512910, event: dispatched, op: pg_notify(1.2b0(20) epoch 243) 2014-06-09 16:22:11.512955 7f869af78700 5 -- op tracker -- , seq: 1844, time: 2014-06-09 16:22:11.512953, event: waiting_for_osdmap, op: pg_notify(1.2b0(20) epoch 243) 2014-06-09 16:22:11.512959 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.512962 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088dc00 2014-06-09 16:22:11.512971 7f869af78700 5 -- op tracker -- , seq: 1844, time: 2014-06-09 16:22:11.512969, event: started, op: pg_notify(1.2b0(20) epoch 243) 2014-06-09 16:22:11.512995 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (17 active) 2014-06-09 16:22:11.513010 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.2b0( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.513024 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.513034 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.513042 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (16 active) 2014-06-09 16:22:11.513046 7f868bf5a700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.513053 7f869af78700 5 -- op tracker -- , seq: 1844, time: 2014-06-09 16:22:11.513051, event: done, op: pg_notify(1.2b0(20) epoch 243) 2014-06-09 16:22:11.513066 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.513069 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.513086 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.513088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.513089 7f869af78700 20 osd.26 243 _dispatch 0x306d2000 pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.513096 7f869af78700 5 -- op tracker -- , seq: 1845, time: 2014-06-09 16:22:11.251359, event: header_read, op: pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513104 7f869af78700 5 -- op tracker -- , seq: 1845, time: 2014-06-09 16:22:11.251360, event: throttled, op: pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513111 7f869af78700 5 -- op tracker -- , seq: 1845, time: 2014-06-09 16:22:11.251375, event: all_read, op: pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513118 7f869af78700 5 -- op tracker -- , seq: 1845, time: 2014-06-09 16:22:11.513086, event: dispatched, op: pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513127 7f869af78700 5 -- op tracker -- , seq: 1845, time: 2014-06-09 16:22:11.513124, event: waiting_for_osdmap, op: pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513131 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.513135 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d2000 2014-06-09 16:22:11.513141 7f869af78700 5 -- op tracker -- , seq: 1845, time: 2014-06-09 16:22:11.513138, event: started, op: pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513162 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0200 (17 active) 2014-06-09 16:22:11.513171 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.513184 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.463( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.513196 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.513200 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.513204 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.463( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.812177 7 0.000185 2014-06-09 16:22:11.513214 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.463( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.513224 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.463( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.513233 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.513242 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.513250 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.513301 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.513306 7f8689755700 5 queue_transactions existing osr(2.463 0x2ed1510)/0x2ed1510 2014-06-09 16:22:11.513309 7f8689755700 2 waiting 51 > 50 ops || 92345 > 104857600 2014-06-09 16:22:11.513318 7f869af78700 5 -- op tracker -- , seq: 1845, time: 2014-06-09 16:22:11.513315, event: done, op: pg_log(2.463 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513328 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.513330 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.513345 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.513347 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.513348 7f869af78700 20 osd.26 243 _dispatch 0x2fe716c0 pg_notify(3.464(12) epoch 243) v5 2014-06-09 16:22:11.513353 7f869af78700 5 -- op tracker -- , seq: 1846, time: 2014-06-09 16:22:11.165322, event: header_read, op: pg_notify(3.464(12) epoch 243) 2014-06-09 16:22:11.513360 7f869af78700 5 -- op tracker -- , seq: 1846, time: 2014-06-09 16:22:11.165323, event: throttled, op: pg_notify(3.464(12) epoch 243) 2014-06-09 16:22:11.513367 7f869af78700 5 -- op tracker -- , seq: 1846, time: 2014-06-09 16:22:11.165348, event: all_read, op: pg_notify(3.464(12) epoch 243) 2014-06-09 16:22:11.513373 7f869af78700 5 -- op tracker -- , seq: 1846, time: 2014-06-09 16:22:11.513345, event: dispatched, op: pg_notify(3.464(12) epoch 243) 2014-06-09 16:22:11.513380 7f869af78700 5 -- op tracker -- , seq: 1846, time: 2014-06-09 16:22:11.513378, event: waiting_for_osdmap, op: pg_notify(3.464(12) epoch 243) 2014-06-09 16:22:11.513385 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.513387 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe716c0 2014-06-09 16:22:11.513392 7f869af78700 5 -- op tracker -- , seq: 1846, time: 2014-06-09 16:22:11.513390, event: started, op: pg_notify(3.464(12) epoch 243) 2014-06-09 16:22:11.513416 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02b0 (18 active) 2014-06-09 16:22:11.513425 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.464( v 213'621 (0'0,213'621] local-les=191 n=120 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.513451 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.513463 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.513470 7f868d75d700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02b0 (17 active) 2014-06-09 16:22:11.513476 7f868d75d700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.513485 7f869af78700 5 -- op tracker -- , seq: 1846, time: 2014-06-09 16:22:11.513482, event: done, op: pg_notify(3.464(12) epoch 243) 2014-06-09 16:22:11.513494 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.513496 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.513514 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.513515 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.513517 7f869af78700 20 osd.26 243 _dispatch 0x3044ddc0 pg_notify(3.65f(10) epoch 243) v5 2014-06-09 16:22:11.513522 7f869af78700 5 -- op tracker -- , seq: 1847, time: 2014-06-09 16:22:11.248847, event: header_read, op: pg_notify(3.65f(10) epoch 243) 2014-06-09 16:22:11.513529 7f869af78700 5 -- op tracker -- , seq: 1847, time: 2014-06-09 16:22:11.248848, event: throttled, op: pg_notify(3.65f(10) epoch 243) 2014-06-09 16:22:11.513536 7f869af78700 5 -- op tracker -- , seq: 1847, time: 2014-06-09 16:22:11.248861, event: all_read, op: pg_notify(3.65f(10) epoch 243) 2014-06-09 16:22:11.513542 7f869af78700 5 -- op tracker -- , seq: 1847, time: 2014-06-09 16:22:11.513513, event: dispatched, op: pg_notify(3.65f(10) epoch 243) 2014-06-09 16:22:11.513548 7f869af78700 5 -- op tracker -- , seq: 1847, time: 2014-06-09 16:22:11.513546, event: waiting_for_osdmap, op: pg_notify(3.65f(10) epoch 243) 2014-06-09 16:22:11.513553 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.513555 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044ddc0 2014-06-09 16:22:11.513560 7f869af78700 5 -- op tracker -- , seq: 1847, time: 2014-06-09 16:22:11.513558, event: started, op: pg_notify(3.65f(10) epoch 243) 2014-06-09 16:22:11.513674 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (18 active) 2014-06-09 16:22:11.513682 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.65f( v 214'652 (0'0,214'652] local-les=213 n=125 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.513697 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.513708 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.513715 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (17 active) 2014-06-09 16:22:11.513719 7f8691f66700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.513727 7f869af78700 5 -- op tracker -- , seq: 1847, time: 2014-06-09 16:22:11.513724, event: done, op: pg_notify(3.65f(10) epoch 243) 2014-06-09 16:22:11.513736 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.513738 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.513768 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.513770 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.513771 7f869af78700 20 osd.26 243 _dispatch 0x307e1000 pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.513781 7f869af78700 5 -- op tracker -- , seq: 1848, time: 2014-06-09 16:22:11.182774, event: header_read, op: pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513789 7f869af78700 5 -- op tracker -- , seq: 1848, time: 2014-06-09 16:22:11.182775, event: throttled, op: pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513796 7f869af78700 5 -- op tracker -- , seq: 1848, time: 2014-06-09 16:22:11.182785, event: all_read, op: pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513803 7f869af78700 5 -- op tracker -- , seq: 1848, time: 2014-06-09 16:22:11.513768, event: dispatched, op: pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513812 7f869af78700 5 -- op tracker -- , seq: 1848, time: 2014-06-09 16:22:11.513809, event: waiting_for_osdmap, op: pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513817 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.513821 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e1000 2014-06-09 16:22:11.513827 7f869af78700 5 -- op tracker -- , seq: 1848, time: 2014-06-09 16:22:11.513824, event: started, op: pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.513844 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (18 active) 2014-06-09 16:22:11.513853 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.513864 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.3cd( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.513876 7f868bf5a700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.513881 7f868bf5a700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.513884 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.247707 7 0.000177 2014-06-09 16:22:11.513898 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.513909 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.513920 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.513929 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.513938 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.513989 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.513994 7f868bf5a700 5 queue_transactions existing osr(1.3cd 0x384d4b0)/0x384d4b0 2014-06-09 16:22:11.513997 7f868bf5a700 2 waiting 51 > 50 ops || 92302 > 104857600 2014-06-09 16:22:11.514006 7f869af78700 5 -- op tracker -- , seq: 1848, time: 2014-06-09 16:22:11.514003, event: done, op: pg_log(1.3cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.514016 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.514018 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.514031 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.514033 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.514034 7f869af78700 20 osd.26 243 _dispatch 0x19805880 pg_notify(2.244(23) epoch 243) v5 2014-06-09 16:22:11.514039 7f869af78700 5 -- op tracker -- , seq: 1849, time: 2014-06-09 16:22:11.231801, event: header_read, op: pg_notify(2.244(23) epoch 243) 2014-06-09 16:22:11.514046 7f869af78700 5 -- op tracker -- , seq: 1849, time: 2014-06-09 16:22:11.231802, event: throttled, op: pg_notify(2.244(23) epoch 243) 2014-06-09 16:22:11.514052 7f869af78700 5 -- op tracker -- , seq: 1849, time: 2014-06-09 16:22:11.231826, event: all_read, op: pg_notify(2.244(23) epoch 243) 2014-06-09 16:22:11.514926 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.528930 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3140.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.528940 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.514058 7f869af78700 5 -- op tracker -- , seq: 1849, time: 2014-06-09 16:22:11.514031, event: dispatched, op: pg_notify(2.244(23) epoch 243) 2014-06-09 16:22:11.528978 7f869af78700 5 -- op tracker -- , seq: 1849, time: 2014-06-09 16:22:11.528973, event: waiting_for_osdmap, op: pg_notify(2.244(23) epoch 243) 2014-06-09 16:22:11.528986 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.528990 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19805880 2014-06-09 16:22:11.528997 7f869af78700 5 -- op tracker -- , seq: 1849, time: 2014-06-09 16:22:11.528995, event: started, op: pg_notify(2.244(23) epoch 243) 2014-06-09 16:22:11.529022 7f869af78700 5 -- op tracker -- , seq: 1849, time: 2014-06-09 16:22:11.529020, event: done, op: pg_notify(2.244(23) epoch 243) 2014-06-09 16:22:11.529039 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529041 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529070 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529072 7f869af78700 20 osd.26 243 _dispatch 0x30719e00 pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.529079 7f869af78700 5 -- op tracker -- , seq: 1850, time: 2014-06-09 16:22:11.152825, event: header_read, op: pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529088 7f869af78700 5 -- op tracker -- , seq: 1850, time: 2014-06-09 16:22:11.152826, event: throttled, op: pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529095 7f869af78700 5 -- op tracker -- , seq: 1850, time: 2014-06-09 16:22:11.152843, event: all_read, op: pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529102 7f869af78700 5 -- op tracker -- , seq: 1850, time: 2014-06-09 16:22:11.529067, event: dispatched, op: pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529109 7f869af78700 5 -- op tracker -- , seq: 1850, time: 2014-06-09 16:22:11.529107, event: waiting_for_osdmap, op: pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529114 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.529118 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30719e00 2014-06-09 16:22:11.529124 7f869af78700 5 -- op tracker -- , seq: 1850, time: 2014-06-09 16:22:11.529121, event: started, op: pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529143 7f869af78700 5 -- op tracker -- , seq: 1850, time: 2014-06-09 16:22:11.529139, event: done, op: pg_log(1.71e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529152 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529154 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529168 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529170 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529171 7f869af78700 20 osd.26 243 _dispatch 0x308d2a00 pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.529177 7f869af78700 5 -- op tracker -- , seq: 1851, time: 2014-06-09 16:22:11.298223, event: header_read, op: pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529185 7f869af78700 5 -- op tracker -- , seq: 1851, time: 2014-06-09 16:22:11.298223, event: throttled, op: pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529192 7f869af78700 5 -- op tracker -- , seq: 1851, time: 2014-06-09 16:22:11.298233, event: all_read, op: pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529200 7f869af78700 5 -- op tracker -- , seq: 1851, time: 2014-06-09 16:22:11.529168, event: dispatched, op: pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529207 7f869af78700 5 -- op tracker -- , seq: 1851, time: 2014-06-09 16:22:11.529205, event: waiting_for_osdmap, op: pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529212 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.529216 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d2a00 2014-06-09 16:22:11.529222 7f869af78700 5 -- op tracker -- , seq: 1851, time: 2014-06-09 16:22:11.529219, event: started, op: pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529235 7f869af78700 5 -- op tracker -- , seq: 1851, time: 2014-06-09 16:22:11.529232, event: done, op: pg_log(0.530 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529243 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529245 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529259 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529260 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529261 7f869af78700 20 osd.26 243 _dispatch 0x3045c400 pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.529268 7f869af78700 5 -- op tracker -- , seq: 1852, time: 2014-06-09 16:22:11.043102, event: header_read, op: pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529275 7f869af78700 5 -- op tracker -- , seq: 1852, time: 2014-06-09 16:22:11.043102, event: throttled, op: pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529282 7f869af78700 5 -- op tracker -- , seq: 1852, time: 2014-06-09 16:22:11.043113, event: all_read, op: pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529289 7f869af78700 5 -- op tracker -- , seq: 1852, time: 2014-06-09 16:22:11.529258, event: dispatched, op: pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529296 7f869af78700 5 -- op tracker -- , seq: 1852, time: 2014-06-09 16:22:11.529293, event: waiting_for_osdmap, op: pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529300 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.529304 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3045c400 2014-06-09 16:22:11.529310 7f869af78700 5 -- op tracker -- , seq: 1852, time: 2014-06-09 16:22:11.529307, event: started, op: pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529324 7f869af78700 5 -- op tracker -- , seq: 1852, time: 2014-06-09 16:22:11.529321, event: done, op: pg_log(0.3ab epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529333 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529335 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529348 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529350 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529351 7f869af78700 20 osd.26 243 _dispatch 0x2fe1fe00 pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.529358 7f869af78700 5 -- op tracker -- , seq: 1853, time: 2014-06-09 16:22:11.064350, event: header_read, op: pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529365 7f869af78700 5 -- op tracker -- , seq: 1853, time: 2014-06-09 16:22:11.064350, event: throttled, op: pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529372 7f869af78700 5 -- op tracker -- , seq: 1853, time: 2014-06-09 16:22:11.064366, event: all_read, op: pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529379 7f869af78700 5 -- op tracker -- , seq: 1853, time: 2014-06-09 16:22:11.529348, event: dispatched, op: pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529386 7f869af78700 5 -- op tracker -- , seq: 1853, time: 2014-06-09 16:22:11.529384, event: waiting_for_osdmap, op: pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529392 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.529396 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe1fe00 2014-06-09 16:22:11.529402 7f869af78700 5 -- op tracker -- , seq: 1853, time: 2014-06-09 16:22:11.529399, event: started, op: pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529416 7f869af78700 5 -- op tracker -- , seq: 1853, time: 2014-06-09 16:22:11.529413, event: done, op: pg_log(0.9d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529424 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529426 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529440 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529442 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529443 7f869af78700 20 osd.26 243 _dispatch 0x30cf9c00 pg_info(2 pgs e243:3.579(2),3.2b6(2)) v4 2014-06-09 16:22:11.529449 7f869af78700 5 -- op tracker -- , seq: 1854, time: 2014-06-09 16:22:11.268580, event: header_read, op: pg_info(2 pgs e243:3.579(2),3.2b6(2)) 2014-06-09 16:22:11.529456 7f869af78700 5 -- op tracker -- , seq: 1854, time: 2014-06-09 16:22:11.268581, event: throttled, op: pg_info(2 pgs e243:3.579(2),3.2b6(2)) 2014-06-09 16:22:11.529463 7f869af78700 5 -- op tracker -- , seq: 1854, time: 2014-06-09 16:22:11.268606, event: all_read, op: pg_info(2 pgs e243:3.579(2),3.2b6(2)) 2014-06-09 16:22:11.529470 7f869af78700 5 -- op tracker -- , seq: 1854, time: 2014-06-09 16:22:11.529440, event: dispatched, op: pg_info(2 pgs e243:3.579(2),3.2b6(2)) 2014-06-09 16:22:11.529477 7f869af78700 5 -- op tracker -- , seq: 1854, time: 2014-06-09 16:22:11.529474, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.579(2),3.2b6(2)) 2014-06-09 16:22:11.529481 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.579(2),3.2b6(2)) v4 from osd.7 2014-06-09 16:22:11.529485 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf9c00 2014-06-09 16:22:11.529491 7f869af78700 5 -- op tracker -- , seq: 1854, time: 2014-06-09 16:22:11.529488, event: started, op: pg_info(2 pgs e243:3.579(2),3.2b6(2)) 2014-06-09 16:22:11.529516 7f869af78700 5 -- op tracker -- , seq: 1854, time: 2014-06-09 16:22:11.529514, event: done, op: pg_info(2 pgs e243:3.579(2),3.2b6(2)) 2014-06-09 16:22:11.529525 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529527 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529545 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529546 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529548 7f869af78700 20 osd.26 243 _dispatch 0x1c039dc0 pg_notify(0.198(23) epoch 243) v5 2014-06-09 16:22:11.529553 7f869af78700 5 -- op tracker -- , seq: 1855, time: 2014-06-09 16:22:11.251606, event: header_read, op: pg_notify(0.198(23) epoch 243) 2014-06-09 16:22:11.529560 7f869af78700 5 -- op tracker -- , seq: 1855, time: 2014-06-09 16:22:11.251607, event: throttled, op: pg_notify(0.198(23) epoch 243) 2014-06-09 16:22:11.529566 7f869af78700 5 -- op tracker -- , seq: 1855, time: 2014-06-09 16:22:11.251651, event: all_read, op: pg_notify(0.198(23) epoch 243) 2014-06-09 16:22:11.529573 7f869af78700 5 -- op tracker -- , seq: 1855, time: 2014-06-09 16:22:11.529544, event: dispatched, op: pg_notify(0.198(23) epoch 243) 2014-06-09 16:22:11.529579 7f869af78700 5 -- op tracker -- , seq: 1855, time: 2014-06-09 16:22:11.529577, event: waiting_for_osdmap, op: pg_notify(0.198(23) epoch 243) 2014-06-09 16:22:11.529584 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.529586 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c039dc0 2014-06-09 16:22:11.529592 7f869af78700 5 -- op tracker -- , seq: 1855, time: 2014-06-09 16:22:11.529589, event: started, op: pg_notify(0.198(23) epoch 243) 2014-06-09 16:22:11.529608 7f869af78700 5 -- op tracker -- , seq: 1855, time: 2014-06-09 16:22:11.529606, event: done, op: pg_notify(0.198(23) epoch 243) 2014-06-09 16:22:11.529617 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529619 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529639 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529641 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529642 7f869af78700 20 osd.26 243 _dispatch 0x2fef5880 pg_notify(0.105(21) epoch 243) v5 2014-06-09 16:22:11.529647 7f869af78700 5 -- op tracker -- , seq: 1856, time: 2014-06-09 16:22:11.178350, event: header_read, op: pg_notify(0.105(21) epoch 243) 2014-06-09 16:22:11.529654 7f869af78700 5 -- op tracker -- , seq: 1856, time: 2014-06-09 16:22:11.178351, event: throttled, op: pg_notify(0.105(21) epoch 243) 2014-06-09 16:22:11.529660 7f869af78700 5 -- op tracker -- , seq: 1856, time: 2014-06-09 16:22:11.178369, event: all_read, op: pg_notify(0.105(21) epoch 243) 2014-06-09 16:22:11.529667 7f869af78700 5 -- op tracker -- , seq: 1856, time: 2014-06-09 16:22:11.529639, event: dispatched, op: pg_notify(0.105(21) epoch 243) 2014-06-09 16:22:11.529674 7f869af78700 5 -- op tracker -- , seq: 1856, time: 2014-06-09 16:22:11.529672, event: waiting_for_osdmap, op: pg_notify(0.105(21) epoch 243) 2014-06-09 16:22:11.529678 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.529680 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef5880 2014-06-09 16:22:11.529685 7f869af78700 5 -- op tracker -- , seq: 1856, time: 2014-06-09 16:22:11.529683, event: started, op: pg_notify(0.105(21) epoch 243) 2014-06-09 16:22:11.529701 7f869af78700 5 -- op tracker -- , seq: 1856, time: 2014-06-09 16:22:11.529699, event: done, op: pg_notify(0.105(21) epoch 243) 2014-06-09 16:22:11.529709 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529711 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529731 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529733 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529734 7f869af78700 20 osd.26 243 _dispatch 0x30a83600 pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.529741 7f869af78700 5 -- op tracker -- , seq: 1857, time: 2014-06-09 16:22:11.248917, event: header_read, op: pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529748 7f869af78700 5 -- op tracker -- , seq: 1857, time: 2014-06-09 16:22:11.248917, event: throttled, op: pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529755 7f869af78700 5 -- op tracker -- , seq: 1857, time: 2014-06-09 16:22:11.248945, event: all_read, op: pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529761 7f869af78700 5 -- op tracker -- , seq: 1857, time: 2014-06-09 16:22:11.529731, event: dispatched, op: pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529769 7f869af78700 5 -- op tracker -- , seq: 1857, time: 2014-06-09 16:22:11.529766, event: waiting_for_osdmap, op: pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529774 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.529778 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a83600 2014-06-09 16:22:11.529784 7f869af78700 5 -- op tracker -- , seq: 1857, time: 2014-06-09 16:22:11.529782, event: started, op: pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529799 7f869af78700 5 -- op tracker -- , seq: 1857, time: 2014-06-09 16:22:11.529795, event: done, op: pg_log(0.48b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529807 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529809 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529822 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529823 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529824 7f869af78700 20 osd.26 243 _dispatch 0x3070e400 pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.529831 7f869af78700 5 -- op tracker -- , seq: 1858, time: 2014-06-09 16:22:11.184692, event: header_read, op: pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529838 7f869af78700 5 -- op tracker -- , seq: 1858, time: 2014-06-09 16:22:11.184693, event: throttled, op: pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529845 7f869af78700 5 -- op tracker -- , seq: 1858, time: 2014-06-09 16:22:11.184703, event: all_read, op: pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529852 7f869af78700 5 -- op tracker -- , seq: 1858, time: 2014-06-09 16:22:11.529821, event: dispatched, op: pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529859 7f869af78700 5 -- op tracker -- , seq: 1858, time: 2014-06-09 16:22:11.529857, event: waiting_for_osdmap, op: pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529864 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.529868 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070e400 2014-06-09 16:22:11.529873 7f869af78700 5 -- op tracker -- , seq: 1858, time: 2014-06-09 16:22:11.529870, event: started, op: pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529887 7f869af78700 5 -- op tracker -- , seq: 1858, time: 2014-06-09 16:22:11.529884, event: done, op: pg_log(0.529 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.529895 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529897 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529910 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529912 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529913 7f869af78700 20 osd.26 243 _dispatch 0x1ac95180 pg_notify(2.521(20) epoch 243) v5 2014-06-09 16:22:11.529917 7f869af78700 5 -- op tracker -- , seq: 1859, time: 2014-06-09 16:22:11.255649, event: header_read, op: pg_notify(2.521(20) epoch 243) 2014-06-09 16:22:11.529924 7f869af78700 5 -- op tracker -- , seq: 1859, time: 2014-06-09 16:22:11.255650, event: throttled, op: pg_notify(2.521(20) epoch 243) 2014-06-09 16:22:11.529930 7f869af78700 5 -- op tracker -- , seq: 1859, time: 2014-06-09 16:22:11.255677, event: all_read, op: pg_notify(2.521(20) epoch 243) 2014-06-09 16:22:11.529936 7f869af78700 5 -- op tracker -- , seq: 1859, time: 2014-06-09 16:22:11.529910, event: dispatched, op: pg_notify(2.521(20) epoch 243) 2014-06-09 16:22:11.529943 7f869af78700 5 -- op tracker -- , seq: 1859, time: 2014-06-09 16:22:11.529941, event: waiting_for_osdmap, op: pg_notify(2.521(20) epoch 243) 2014-06-09 16:22:11.529948 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.529950 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac95180 2014-06-09 16:22:11.529955 7f869af78700 5 -- op tracker -- , seq: 1859, time: 2014-06-09 16:22:11.529953, event: started, op: pg_notify(2.521(20) epoch 243) 2014-06-09 16:22:11.529971 7f869af78700 5 -- op tracker -- , seq: 1859, time: 2014-06-09 16:22:11.529969, event: done, op: pg_notify(2.521(20) epoch 243) 2014-06-09 16:22:11.529979 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.529981 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.529999 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530001 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530002 7f869af78700 20 osd.26 243 _dispatch 0x2d094000 pg_notify(0.70f(25) epoch 243) v5 2014-06-09 16:22:11.530008 7f869af78700 5 -- op tracker -- , seq: 1860, time: 2014-06-09 16:22:11.154926, event: header_read, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.530015 7f869af78700 5 -- op tracker -- , seq: 1860, time: 2014-06-09 16:22:11.154927, event: throttled, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.530021 7f869af78700 5 -- op tracker -- , seq: 1860, time: 2014-06-09 16:22:11.154947, event: all_read, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.530027 7f869af78700 5 -- op tracker -- , seq: 1860, time: 2014-06-09 16:22:11.529999, event: dispatched, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.530033 7f869af78700 5 -- op tracker -- , seq: 1860, time: 2014-06-09 16:22:11.530031, event: waiting_for_osdmap, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.530037 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.530039 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d094000 2014-06-09 16:22:11.530044 7f869af78700 5 -- op tracker -- , seq: 1860, time: 2014-06-09 16:22:11.530042, event: started, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.530060 7f869af78700 5 -- op tracker -- , seq: 1860, time: 2014-06-09 16:22:11.530058, event: done, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.530072 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530074 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530097 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530099 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530100 7f869af78700 20 osd.26 243 _dispatch 0x3004d200 pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.530107 7f869af78700 5 -- op tracker -- , seq: 1861, time: 2014-06-09 16:22:11.298239, event: header_read, op: pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530115 7f869af78700 5 -- op tracker -- , seq: 1861, time: 2014-06-09 16:22:11.298239, event: throttled, op: pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530122 7f869af78700 5 -- op tracker -- , seq: 1861, time: 2014-06-09 16:22:11.298250, event: all_read, op: pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530129 7f869af78700 5 -- op tracker -- , seq: 1861, time: 2014-06-09 16:22:11.530097, event: dispatched, op: pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530137 7f869af78700 5 -- op tracker -- , seq: 1861, time: 2014-06-09 16:22:11.530134, event: waiting_for_osdmap, op: pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530142 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.530146 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004d200 2014-06-09 16:22:11.530151 7f869af78700 5 -- op tracker -- , seq: 1861, time: 2014-06-09 16:22:11.530148, event: started, op: pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530164 7f869af78700 5 -- op tracker -- , seq: 1861, time: 2014-06-09 16:22:11.530161, event: done, op: pg_log(2.624 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530171 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530172 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530184 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530186 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530187 7f869af78700 20 osd.26 243 _dispatch 0xbe9ce00 pg_notify(3.67d(12) epoch 243) v5 2014-06-09 16:22:11.530193 7f869af78700 5 -- op tracker -- , seq: 1862, time: 2014-06-09 16:22:11.043118, event: header_read, op: pg_notify(3.67d(12) epoch 243) 2014-06-09 16:22:11.530200 7f869af78700 5 -- op tracker -- , seq: 1862, time: 2014-06-09 16:22:11.043118, event: throttled, op: pg_notify(3.67d(12) epoch 243) 2014-06-09 16:22:11.530206 7f869af78700 5 -- op tracker -- , seq: 1862, time: 2014-06-09 16:22:11.043146, event: all_read, op: pg_notify(3.67d(12) epoch 243) 2014-06-09 16:22:11.530213 7f869af78700 5 -- op tracker -- , seq: 1862, time: 2014-06-09 16:22:11.530184, event: dispatched, op: pg_notify(3.67d(12) epoch 243) 2014-06-09 16:22:11.530220 7f869af78700 5 -- op tracker -- , seq: 1862, time: 2014-06-09 16:22:11.530218, event: waiting_for_osdmap, op: pg_notify(3.67d(12) epoch 243) 2014-06-09 16:22:11.530224 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.530226 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xbe9ce00 2014-06-09 16:22:11.530231 7f869af78700 5 -- op tracker -- , seq: 1862, time: 2014-06-09 16:22:11.530229, event: started, op: pg_notify(3.67d(12) epoch 243) 2014-06-09 16:22:11.530248 7f869af78700 5 -- op tracker -- , seq: 1862, time: 2014-06-09 16:22:11.530246, event: done, op: pg_notify(3.67d(12) epoch 243) 2014-06-09 16:22:11.530256 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530258 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530273 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530275 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530276 7f869af78700 20 osd.26 243 _dispatch 0x19eb9880 pg_notify(2.368(23) epoch 243) v5 2014-06-09 16:22:11.530280 7f869af78700 5 -- op tracker -- , seq: 1863, time: 2014-06-09 16:22:11.064375, event: header_read, op: pg_notify(2.368(23) epoch 243) 2014-06-09 16:22:11.530287 7f869af78700 5 -- op tracker -- , seq: 1863, time: 2014-06-09 16:22:11.064376, event: throttled, op: pg_notify(2.368(23) epoch 243) 2014-06-09 16:22:11.530294 7f869af78700 5 -- op tracker -- , seq: 1863, time: 2014-06-09 16:22:11.064406, event: all_read, op: pg_notify(2.368(23) epoch 243) 2014-06-09 16:22:11.530300 7f869af78700 5 -- op tracker -- , seq: 1863, time: 2014-06-09 16:22:11.530273, event: dispatched, op: pg_notify(2.368(23) epoch 243) 2014-06-09 16:22:11.530307 7f869af78700 5 -- op tracker -- , seq: 1863, time: 2014-06-09 16:22:11.530305, event: waiting_for_osdmap, op: pg_notify(2.368(23) epoch 243) 2014-06-09 16:22:11.530311 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.530313 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb9880 2014-06-09 16:22:11.530318 7f869af78700 5 -- op tracker -- , seq: 1863, time: 2014-06-09 16:22:11.530316, event: started, op: pg_notify(2.368(23) epoch 243) 2014-06-09 16:22:11.530334 7f869af78700 5 -- op tracker -- , seq: 1863, time: 2014-06-09 16:22:11.530331, event: done, op: pg_notify(2.368(23) epoch 243) 2014-06-09 16:22:11.530342 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530344 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530365 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530367 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530369 7f869af78700 20 osd.26 243 _dispatch 0x14050e00 pg_notify(2.28b(20),0.290(20) epoch 243) v5 2014-06-09 16:22:11.530374 7f869af78700 5 -- op tracker -- , seq: 1864, time: 2014-06-09 16:22:11.299677, event: header_read, op: pg_notify(2.28b(20),0.290(20) epoch 243) 2014-06-09 16:22:11.530382 7f869af78700 5 -- op tracker -- , seq: 1864, time: 2014-06-09 16:22:11.299677, event: throttled, op: pg_notify(2.28b(20),0.290(20) epoch 243) 2014-06-09 16:22:11.530389 7f869af78700 5 -- op tracker -- , seq: 1864, time: 2014-06-09 16:22:11.299722, event: all_read, op: pg_notify(2.28b(20),0.290(20) epoch 243) 2014-06-09 16:22:11.530396 7f869af78700 5 -- op tracker -- , seq: 1864, time: 2014-06-09 16:22:11.530365, event: dispatched, op: pg_notify(2.28b(20),0.290(20) epoch 243) 2014-06-09 16:22:11.530403 7f869af78700 5 -- op tracker -- , seq: 1864, time: 2014-06-09 16:22:11.530401, event: waiting_for_osdmap, op: pg_notify(2.28b(20),0.290(20) epoch 243) 2014-06-09 16:22:11.530408 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.530410 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14050e00 2014-06-09 16:22:11.530415 7f869af78700 5 -- op tracker -- , seq: 1864, time: 2014-06-09 16:22:11.530413, event: started, op: pg_notify(2.28b(20),0.290(20) epoch 243) 2014-06-09 16:22:11.530439 7f869af78700 5 -- op tracker -- , seq: 1864, time: 2014-06-09 16:22:11.530437, event: done, op: pg_notify(2.28b(20),0.290(20) epoch 243) 2014-06-09 16:22:11.530448 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530450 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530475 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530477 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530478 7f869af78700 20 osd.26 243 _dispatch 0x30952a00 pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.530485 7f869af78700 5 -- op tracker -- , seq: 1865, time: 2014-06-09 16:22:11.253436, event: header_read, op: pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530493 7f869af78700 5 -- op tracker -- , seq: 1865, time: 2014-06-09 16:22:11.253437, event: throttled, op: pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530500 7f869af78700 5 -- op tracker -- , seq: 1865, time: 2014-06-09 16:22:11.253450, event: all_read, op: pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530506 7f869af78700 5 -- op tracker -- , seq: 1865, time: 2014-06-09 16:22:11.530474, event: dispatched, op: pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530514 7f869af78700 5 -- op tracker -- , seq: 1865, time: 2014-06-09 16:22:11.530511, event: waiting_for_osdmap, op: pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530518 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.530522 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30952a00 2014-06-09 16:22:11.530528 7f869af78700 5 -- op tracker -- , seq: 1865, time: 2014-06-09 16:22:11.530525, event: started, op: pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530540 7f869af78700 5 -- op tracker -- , seq: 1865, time: 2014-06-09 16:22:11.530538, event: done, op: pg_log(2.2d9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530548 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530550 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530561 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530563 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530564 7f869af78700 20 osd.26 243 _dispatch 0x3044da40 pg_info(1 pgs e243:3.111(4)) v4 2014-06-09 16:22:11.530570 7f869af78700 5 -- op tracker -- , seq: 1866, time: 2014-06-09 16:22:11.178433, event: header_read, op: pg_info(1 pgs e243:3.111(4)) 2014-06-09 16:22:11.530577 7f869af78700 5 -- op tracker -- , seq: 1866, time: 2014-06-09 16:22:11.178434, event: throttled, op: pg_info(1 pgs e243:3.111(4)) 2014-06-09 16:22:11.530583 7f869af78700 5 -- op tracker -- , seq: 1866, time: 2014-06-09 16:22:11.178445, event: all_read, op: pg_info(1 pgs e243:3.111(4)) 2014-06-09 16:22:11.530590 7f869af78700 5 -- op tracker -- , seq: 1866, time: 2014-06-09 16:22:11.530561, event: dispatched, op: pg_info(1 pgs e243:3.111(4)) 2014-06-09 16:22:11.530597 7f869af78700 5 -- op tracker -- , seq: 1866, time: 2014-06-09 16:22:11.530594, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.111(4)) 2014-06-09 16:22:11.530601 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.111(4)) v4 from osd.1 2014-06-09 16:22:11.530604 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044da40 2014-06-09 16:22:11.530609 7f869af78700 5 -- op tracker -- , seq: 1866, time: 2014-06-09 16:22:11.530607, event: started, op: pg_info(1 pgs e243:3.111(4)) 2014-06-09 16:22:11.530627 7f869af78700 5 -- op tracker -- , seq: 1866, time: 2014-06-09 16:22:11.530625, event: done, op: pg_info(1 pgs e243:3.111(4)) 2014-06-09 16:22:11.530635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530636 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530651 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530653 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530654 7f869af78700 20 osd.26 243 _dispatch 0x30a83000 pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.530660 7f869af78700 5 -- op tracker -- , seq: 1867, time: 2014-06-09 16:22:11.249454, event: header_read, op: pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530668 7f869af78700 5 -- op tracker -- , seq: 1867, time: 2014-06-09 16:22:11.249455, event: throttled, op: pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530675 7f869af78700 5 -- op tracker -- , seq: 1867, time: 2014-06-09 16:22:11.249467, event: all_read, op: pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530681 7f869af78700 5 -- op tracker -- , seq: 1867, time: 2014-06-09 16:22:11.530651, event: dispatched, op: pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530689 7f869af78700 5 -- op tracker -- , seq: 1867, time: 2014-06-09 16:22:11.530686, event: waiting_for_osdmap, op: pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530694 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.530698 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a83000 2014-06-09 16:22:11.530704 7f869af78700 5 -- op tracker -- , seq: 1867, time: 2014-06-09 16:22:11.530701, event: started, op: pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530718 7f869af78700 5 -- op tracker -- , seq: 1867, time: 2014-06-09 16:22:11.530715, event: done, op: pg_log(0.2cb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530725 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530727 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530740 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530742 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530743 7f869af78700 20 osd.26 243 _dispatch 0x3070c000 pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.530750 7f869af78700 5 -- op tracker -- , seq: 1868, time: 2014-06-09 16:22:11.196807, event: header_read, op: pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530757 7f869af78700 5 -- op tracker -- , seq: 1868, time: 2014-06-09 16:22:11.196808, event: throttled, op: pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530764 7f869af78700 5 -- op tracker -- , seq: 1868, time: 2014-06-09 16:22:11.196818, event: all_read, op: pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530770 7f869af78700 5 -- op tracker -- , seq: 1868, time: 2014-06-09 16:22:11.530740, event: dispatched, op: pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530778 7f869af78700 5 -- op tracker -- , seq: 1868, time: 2014-06-09 16:22:11.530776, event: waiting_for_osdmap, op: pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530783 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.530787 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070c000 2014-06-09 16:22:11.530793 7f869af78700 5 -- op tracker -- , seq: 1868, time: 2014-06-09 16:22:11.530790, event: started, op: pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530806 7f869af78700 5 -- op tracker -- , seq: 1868, time: 2014-06-09 16:22:11.530803, event: done, op: pg_log(0.578 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530814 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530816 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530831 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530832 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530834 7f869af78700 20 osd.26 243 _dispatch 0x30881200 pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.530840 7f869af78700 5 -- op tracker -- , seq: 1869, time: 2014-06-09 16:22:11.255690, event: header_read, op: pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530848 7f869af78700 5 -- op tracker -- , seq: 1869, time: 2014-06-09 16:22:11.255690, event: throttled, op: pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530854 7f869af78700 5 -- op tracker -- , seq: 1869, time: 2014-06-09 16:22:11.255704, event: all_read, op: pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530861 7f869af78700 5 -- op tracker -- , seq: 1869, time: 2014-06-09 16:22:11.530830, event: dispatched, op: pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530869 7f869af78700 5 -- op tracker -- , seq: 1869, time: 2014-06-09 16:22:11.530866, event: waiting_for_osdmap, op: pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530873 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.530877 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30881200 2014-06-09 16:22:11.530882 7f869af78700 5 -- op tracker -- , seq: 1869, time: 2014-06-09 16:22:11.530880, event: started, op: pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530896 7f869af78700 5 -- op tracker -- , seq: 1869, time: 2014-06-09 16:22:11.530893, event: done, op: pg_log(0.640 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.530903 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530905 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530916 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530918 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.530919 7f869af78700 20 osd.26 243 _dispatch 0x19805500 pg_notify(3.72c(12) epoch 243) v5 2014-06-09 16:22:11.530924 7f869af78700 5 -- op tracker -- , seq: 1870, time: 2014-06-09 16:22:11.154953, event: header_read, op: pg_notify(3.72c(12) epoch 243) 2014-06-09 16:22:11.530931 7f869af78700 5 -- op tracker -- , seq: 1870, time: 2014-06-09 16:22:11.154953, event: throttled, op: pg_notify(3.72c(12) epoch 243) 2014-06-09 16:22:11.530937 7f869af78700 5 -- op tracker -- , seq: 1870, time: 2014-06-09 16:22:11.154974, event: all_read, op: pg_notify(3.72c(12) epoch 243) 2014-06-09 16:22:11.530945 7f869af78700 5 -- op tracker -- , seq: 1870, time: 2014-06-09 16:22:11.530916, event: dispatched, op: pg_notify(3.72c(12) epoch 243) 2014-06-09 16:22:11.530951 7f869af78700 5 -- op tracker -- , seq: 1870, time: 2014-06-09 16:22:11.530949, event: waiting_for_osdmap, op: pg_notify(3.72c(12) epoch 243) 2014-06-09 16:22:11.530956 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.530958 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19805500 2014-06-09 16:22:11.530963 7f869af78700 5 -- op tracker -- , seq: 1870, time: 2014-06-09 16:22:11.530961, event: started, op: pg_notify(3.72c(12) epoch 243) 2014-06-09 16:22:11.530979 7f869af78700 5 -- op tracker -- , seq: 1870, time: 2014-06-09 16:22:11.530977, event: done, op: pg_notify(3.72c(12) epoch 243) 2014-06-09 16:22:11.530987 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.530989 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531004 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531006 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531008 7f869af78700 20 osd.26 243 _dispatch 0x3004e400 pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.531014 7f869af78700 5 -- op tracker -- , seq: 1871, time: 2014-06-09 16:22:11.298273, event: header_read, op: pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531023 7f869af78700 5 -- op tracker -- , seq: 1871, time: 2014-06-09 16:22:11.298273, event: throttled, op: pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531029 7f869af78700 5 -- op tracker -- , seq: 1871, time: 2014-06-09 16:22:11.298281, event: all_read, op: pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531037 7f869af78700 5 -- op tracker -- , seq: 1871, time: 2014-06-09 16:22:11.531004, event: dispatched, op: pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531044 7f869af78700 5 -- op tracker -- , seq: 1871, time: 2014-06-09 16:22:11.531041, event: waiting_for_osdmap, op: pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531049 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.531052 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004e400 2014-06-09 16:22:11.531058 7f869af78700 5 -- op tracker -- , seq: 1871, time: 2014-06-09 16:22:11.531055, event: started, op: pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531071 7f869af78700 5 -- op tracker -- , seq: 1871, time: 2014-06-09 16:22:11.531069, event: done, op: pg_log(1.66e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531079 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531081 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531093 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531094 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531095 7f869af78700 20 osd.26 243 _dispatch 0x305bbe00 pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.531101 7f869af78700 5 -- op tracker -- , seq: 1872, time: 2014-06-09 16:22:11.099505, event: header_read, op: pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531109 7f869af78700 5 -- op tracker -- , seq: 1872, time: 2014-06-09 16:22:11.099505, event: throttled, op: pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531117 7f869af78700 5 -- op tracker -- , seq: 1872, time: 2014-06-09 16:22:11.099519, event: all_read, op: pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531124 7f869af78700 5 -- op tracker -- , seq: 1872, time: 2014-06-09 16:22:11.531092, event: dispatched, op: pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531132 7f869af78700 5 -- op tracker -- , seq: 1872, time: 2014-06-09 16:22:11.531129, event: waiting_for_osdmap, op: pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531136 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.531140 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305bbe00 2014-06-09 16:22:11.531145 7f869af78700 5 -- op tracker -- , seq: 1872, time: 2014-06-09 16:22:11.531143, event: started, op: pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531160 7f869af78700 5 -- op tracker -- , seq: 1872, time: 2014-06-09 16:22:11.531157, event: done, op: pg_log(1.3d6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531168 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531169 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531182 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531184 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531185 7f869af78700 20 osd.26 243 _dispatch 0x304f6000 pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.531192 7f869af78700 5 -- op tracker -- , seq: 1873, time: 2014-06-09 16:22:11.066682, event: header_read, op: pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531200 7f869af78700 5 -- op tracker -- , seq: 1873, time: 2014-06-09 16:22:11.066683, event: throttled, op: pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531207 7f869af78700 5 -- op tracker -- , seq: 1873, time: 2014-06-09 16:22:11.066695, event: all_read, op: pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531214 7f869af78700 5 -- op tracker -- , seq: 1873, time: 2014-06-09 16:22:11.531182, event: dispatched, op: pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531222 7f869af78700 5 -- op tracker -- , seq: 1873, time: 2014-06-09 16:22:11.531219, event: waiting_for_osdmap, op: pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531227 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.531231 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f6000 2014-06-09 16:22:11.531237 7f869af78700 5 -- op tracker -- , seq: 1873, time: 2014-06-09 16:22:11.531234, event: started, op: pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531250 7f869af78700 5 -- op tracker -- , seq: 1873, time: 2014-06-09 16:22:11.531248, event: done, op: pg_log(1.8c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531258 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531260 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531272 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531275 7f869af78700 20 osd.26 243 _dispatch 0x306dd600 pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.531282 7f869af78700 5 -- op tracker -- , seq: 1874, time: 2014-06-09 16:22:11.299730, event: header_read, op: pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531289 7f869af78700 5 -- op tracker -- , seq: 1874, time: 2014-06-09 16:22:11.299731, event: throttled, op: pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531297 7f869af78700 5 -- op tracker -- , seq: 1874, time: 2014-06-09 16:22:11.299749, event: all_read, op: pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531304 7f869af78700 5 -- op tracker -- , seq: 1874, time: 2014-06-09 16:22:11.531272, event: dispatched, op: pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531311 7f869af78700 5 -- op tracker -- , seq: 1874, time: 2014-06-09 16:22:11.531309, event: waiting_for_osdmap, op: pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531316 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.531320 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306dd600 2014-06-09 16:22:11.531325 7f869af78700 5 -- op tracker -- , seq: 1874, time: 2014-06-09 16:22:11.531323, event: started, op: pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531338 7f869af78700 5 -- op tracker -- , seq: 1874, time: 2014-06-09 16:22:11.531335, event: done, op: pg_log(0.725 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531349 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531351 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531363 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531364 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531365 7f869af78700 20 osd.26 243 _dispatch 0x3005b340 pg_notify(2.423(20),0.158(20) epoch 243) v5 2014-06-09 16:22:11.531371 7f869af78700 5 -- op tracker -- , seq: 1875, time: 2014-06-09 16:22:11.278633, event: header_read, op: pg_notify(2.423(20),0.158(20) epoch 243) 2014-06-09 16:22:11.531378 7f869af78700 5 -- op tracker -- , seq: 1875, time: 2014-06-09 16:22:11.278634, event: throttled, op: pg_notify(2.423(20),0.158(20) epoch 243) 2014-06-09 16:22:11.531385 7f869af78700 5 -- op tracker -- , seq: 1875, time: 2014-06-09 16:22:11.278668, event: all_read, op: pg_notify(2.423(20),0.158(20) epoch 243) 2014-06-09 16:22:11.531392 7f869af78700 5 -- op tracker -- , seq: 1875, time: 2014-06-09 16:22:11.531362, event: dispatched, op: pg_notify(2.423(20),0.158(20) epoch 243) 2014-06-09 16:22:11.531400 7f869af78700 5 -- op tracker -- , seq: 1875, time: 2014-06-09 16:22:11.531397, event: waiting_for_osdmap, op: pg_notify(2.423(20),0.158(20) epoch 243) 2014-06-09 16:22:11.531404 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.531406 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005b340 2014-06-09 16:22:11.531411 7f869af78700 5 -- op tracker -- , seq: 1875, time: 2014-06-09 16:22:11.531409, event: started, op: pg_notify(2.423(20),0.158(20) epoch 243) 2014-06-09 16:22:11.531436 7f869af78700 5 -- op tracker -- , seq: 1875, time: 2014-06-09 16:22:11.531433, event: done, op: pg_notify(2.423(20),0.158(20) epoch 243) 2014-06-09 16:22:11.531444 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531468 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531469 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531470 7f869af78700 20 osd.26 243 _dispatch 0x305f3800 pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.531476 7f869af78700 5 -- op tracker -- , seq: 1876, time: 2014-06-09 16:22:11.178453, event: header_read, op: pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531484 7f869af78700 5 -- op tracker -- , seq: 1876, time: 2014-06-09 16:22:11.178453, event: throttled, op: pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531491 7f869af78700 5 -- op tracker -- , seq: 1876, time: 2014-06-09 16:22:11.178465, event: all_read, op: pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531498 7f869af78700 5 -- op tracker -- , seq: 1876, time: 2014-06-09 16:22:11.531467, event: dispatched, op: pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531506 7f869af78700 5 -- op tracker -- , seq: 1876, time: 2014-06-09 16:22:11.531503, event: waiting_for_osdmap, op: pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531511 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.531515 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f3800 2014-06-09 16:22:11.531521 7f869af78700 5 -- op tracker -- , seq: 1876, time: 2014-06-09 16:22:11.531518, event: started, op: pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531534 7f869af78700 5 -- op tracker -- , seq: 1876, time: 2014-06-09 16:22:11.531532, event: done, op: pg_log(0.5f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531550 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531552 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531565 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.531567 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.531568 7f869af78700 20 osd.26 243 _dispatch 0x30a82a00 pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.531575 7f869af78700 5 -- op tracker -- , seq: 1877, time: 2014-06-09 16:22:11.264435, event: header_read, op: pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531583 7f869af78700 5 -- op tracker -- , seq: 1877, time: 2014-06-09 16:22:11.264436, event: throttled, op: pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531590 7f869af78700 5 -- op tracker -- , seq: 1877, time: 2014-06-09 16:22:11.264450, event: all_read, op: pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531597 7f869af78700 5 -- op tracker -- , seq: 1877, time: 2014-06-09 16:22:11.531565, event: dispatched, op: pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531604 7f869af78700 5 -- op tracker -- , seq: 1877, time: 2014-06-09 16:22:11.531602, event: waiting_for_osdmap, op: pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.531609 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.531653 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02b0 (19 active) 2014-06-09 16:22:11.531659 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.244( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.531675 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.531687 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.531692 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.531703 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.71e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.531714 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.531719 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.531724 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.629318 7 0.000160 2014-06-09 16:22:11.531736 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.531747 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.531758 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.531768 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.531777 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.531827 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.531834 7f868d75d700 5 queue_transactions existing osr(1.71e 0x2ed00b0)/0x2ed00b0 2014-06-09 16:22:11.531838 7f868d75d700 2 waiting 51 > 50 ops || 92231 > 104857600 2014-06-09 16:22:11.531855 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (20 active) 2014-06-09 16:22:11.531860 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.531873 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.578( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.531884 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.531888 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.531892 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.578( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 1.082453 7 0.000262 2014-06-09 16:22:11.531903 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.578( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.531912 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.578( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.531922 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.531931 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.531939 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.531990 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.531995 7f8691f66700 5 queue_transactions existing osr(0.578 0x2bd7800)/0x2bd7800 2014-06-09 16:22:11.531997 7f8691f66700 2 waiting 51 > 50 ops || 92447 > 104857600 2014-06-09 16:22:11.532002 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a82a00 2014-06-09 16:22:11.532009 7f869af78700 5 -- op tracker -- , seq: 1877, time: 2014-06-09 16:22:11.532005, event: started, op: pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532024 7f869af78700 5 -- op tracker -- , seq: 1877, time: 2014-06-09 16:22:11.532021, event: done, op: pg_log(2.220 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532032 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532034 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532046 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532048 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532049 7f869af78700 20 osd.26 243 _dispatch 0x1a039180 pg_notify(2.2b5(20) epoch 243) v5 2014-06-09 16:22:11.532054 7f869af78700 5 -- op tracker -- , seq: 1878, time: 2014-06-09 16:22:11.221705, event: header_read, op: pg_notify(2.2b5(20) epoch 243) 2014-06-09 16:22:11.532061 7f869af78700 5 -- op tracker -- , seq: 1878, time: 2014-06-09 16:22:11.221706, event: throttled, op: pg_notify(2.2b5(20) epoch 243) 2014-06-09 16:22:11.532067 7f869af78700 5 -- op tracker -- , seq: 1878, time: 2014-06-09 16:22:11.221730, event: all_read, op: pg_notify(2.2b5(20) epoch 243) 2014-06-09 16:22:11.532074 7f869af78700 5 -- op tracker -- , seq: 1878, time: 2014-06-09 16:22:11.532046, event: dispatched, op: pg_notify(2.2b5(20) epoch 243) 2014-06-09 16:22:11.532081 7f869af78700 5 -- op tracker -- , seq: 1878, time: 2014-06-09 16:22:11.532078, event: waiting_for_osdmap, op: pg_notify(2.2b5(20) epoch 243) 2014-06-09 16:22:11.532085 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.532087 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a039180 2014-06-09 16:22:11.532092 7f869af78700 5 -- op tracker -- , seq: 1878, time: 2014-06-09 16:22:11.532090, event: started, op: pg_notify(2.2b5(20) epoch 243) 2014-06-09 16:22:11.532109 7f869af78700 5 -- op tracker -- , seq: 1878, time: 2014-06-09 16:22:11.532107, event: done, op: pg_notify(2.2b5(20) epoch 243) 2014-06-09 16:22:11.532117 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532119 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532139 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532142 7f869af78700 20 osd.26 243 _dispatch 0x307df200 pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.532148 7f869af78700 5 -- op tracker -- , seq: 1879, time: 2014-06-09 16:22:11.255710, event: header_read, op: pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532156 7f869af78700 5 -- op tracker -- , seq: 1879, time: 2014-06-09 16:22:11.255710, event: throttled, op: pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532163 7f869af78700 5 -- op tracker -- , seq: 1879, time: 2014-06-09 16:22:11.255721, event: all_read, op: pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532170 7f869af78700 5 -- op tracker -- , seq: 1879, time: 2014-06-09 16:22:11.532138, event: dispatched, op: pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532177 7f869af78700 5 -- op tracker -- , seq: 1879, time: 2014-06-09 16:22:11.532174, event: waiting_for_osdmap, op: pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532182 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.532185 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307df200 2014-06-09 16:22:11.532191 7f869af78700 5 -- op tracker -- , seq: 1879, time: 2014-06-09 16:22:11.532188, event: started, op: pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532204 7f869af78700 5 -- op tracker -- , seq: 1879, time: 2014-06-09 16:22:11.532201, event: done, op: pg_log(1.11d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532211 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532213 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532226 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532227 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532228 7f869af78700 20 osd.26 243 _dispatch 0x307d7000 pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.532235 7f869af78700 5 -- op tracker -- , seq: 1880, time: 2014-06-09 16:22:11.177735, event: header_read, op: pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532243 7f869af78700 5 -- op tracker -- , seq: 1880, time: 2014-06-09 16:22:11.177736, event: throttled, op: pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532250 7f869af78700 5 -- op tracker -- , seq: 1880, time: 2014-06-09 16:22:11.177744, event: all_read, op: pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532257 7f869af78700 5 -- op tracker -- , seq: 1880, time: 2014-06-09 16:22:11.532225, event: dispatched, op: pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532264 7f869af78700 5 -- op tracker -- , seq: 1880, time: 2014-06-09 16:22:11.532261, event: waiting_for_osdmap, op: pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532268 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.532273 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307d7000 2014-06-09 16:22:11.532278 7f869af78700 5 -- op tracker -- , seq: 1880, time: 2014-06-09 16:22:11.532275, event: started, op: pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532291 7f869af78700 5 -- op tracker -- , seq: 1880, time: 2014-06-09 16:22:11.532288, event: done, op: pg_log(2.6b0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532298 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532300 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532311 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532313 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532314 7f869af78700 20 osd.26 243 _dispatch 0x16fdb500 pg_info(1 pgs e243:3.3a5(2)) v4 2014-06-09 16:22:11.532320 7f869af78700 5 -- op tracker -- , seq: 1881, time: 2014-06-09 16:22:11.298285, event: header_read, op: pg_info(1 pgs e243:3.3a5(2)) 2014-06-09 16:22:11.532328 7f869af78700 5 -- op tracker -- , seq: 1881, time: 2014-06-09 16:22:11.298286, event: throttled, op: pg_info(1 pgs e243:3.3a5(2)) 2014-06-09 16:22:11.532334 7f869af78700 5 -- op tracker -- , seq: 1881, time: 2014-06-09 16:22:11.298301, event: all_read, op: pg_info(1 pgs e243:3.3a5(2)) 2014-06-09 16:22:11.532341 7f869af78700 5 -- op tracker -- , seq: 1881, time: 2014-06-09 16:22:11.532311, event: dispatched, op: pg_info(1 pgs e243:3.3a5(2)) 2014-06-09 16:22:11.532348 7f869af78700 5 -- op tracker -- , seq: 1881, time: 2014-06-09 16:22:11.532345, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a5(2)) 2014-06-09 16:22:11.532352 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a5(2)) v4 from osd.5 2014-06-09 16:22:11.532355 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16fdb500 2014-06-09 16:22:11.532360 7f869af78700 5 -- op tracker -- , seq: 1881, time: 2014-06-09 16:22:11.532358, event: started, op: pg_info(1 pgs e243:3.3a5(2)) 2014-06-09 16:22:11.532376 7f869af78700 5 -- op tracker -- , seq: 1881, time: 2014-06-09 16:22:11.532374, event: done, op: pg_info(1 pgs e243:3.3a5(2)) 2014-06-09 16:22:11.532384 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532386 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532403 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532405 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532406 7f869af78700 20 osd.26 243 _dispatch 0x305e2600 pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.532413 7f869af78700 5 -- op tracker -- , seq: 1882, time: 2014-06-09 16:22:11.099528, event: header_read, op: pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532421 7f869af78700 5 -- op tracker -- , seq: 1882, time: 2014-06-09 16:22:11.099528, event: throttled, op: pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532428 7f869af78700 5 -- op tracker -- , seq: 1882, time: 2014-06-09 16:22:11.099539, event: all_read, op: pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532435 7f869af78700 5 -- op tracker -- , seq: 1882, time: 2014-06-09 16:22:11.532403, event: dispatched, op: pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532442 7f869af78700 5 -- op tracker -- , seq: 1882, time: 2014-06-09 16:22:11.532440, event: waiting_for_osdmap, op: pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532447 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.532451 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e2600 2014-06-09 16:22:11.532457 7f869af78700 5 -- op tracker -- , seq: 1882, time: 2014-06-09 16:22:11.532454, event: started, op: pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532470 7f869af78700 5 -- op tracker -- , seq: 1882, time: 2014-06-09 16:22:11.532468, event: done, op: pg_log(0.25d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532478 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532480 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532493 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532495 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532496 7f869af78700 20 osd.26 243 _dispatch 0x2fe1f200 pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.532502 7f869af78700 5 -- op tracker -- , seq: 1883, time: 2014-06-09 16:22:11.066704, event: header_read, op: pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532510 7f869af78700 5 -- op tracker -- , seq: 1883, time: 2014-06-09 16:22:11.066704, event: throttled, op: pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532517 7f869af78700 5 -- op tracker -- , seq: 1883, time: 2014-06-09 16:22:11.066721, event: all_read, op: pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532524 7f869af78700 5 -- op tracker -- , seq: 1883, time: 2014-06-09 16:22:11.532493, event: dispatched, op: pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532531 7f869af78700 5 -- op tracker -- , seq: 1883, time: 2014-06-09 16:22:11.532528, event: waiting_for_osdmap, op: pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532535 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.532539 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe1f200 2014-06-09 16:22:11.532544 7f869af78700 5 -- op tracker -- , seq: 1883, time: 2014-06-09 16:22:11.532542, event: started, op: pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532558 7f869af78700 5 -- op tracker -- , seq: 1883, time: 2014-06-09 16:22:11.532555, event: done, op: pg_log(0.61f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.532566 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532568 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532581 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532583 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532584 7f869af78700 20 osd.26 243 _dispatch 0x3046afc0 pg_notify(1.71b(23),2.45b(23) epoch 243) v5 2014-06-09 16:22:11.532590 7f869af78700 5 -- op tracker -- , seq: 1884, time: 2014-06-09 16:22:11.299763, event: header_read, op: pg_notify(1.71b(23),2.45b(23) epoch 243) 2014-06-09 16:22:11.532597 7f869af78700 5 -- op tracker -- , seq: 1884, time: 2014-06-09 16:22:11.299764, event: throttled, op: pg_notify(1.71b(23),2.45b(23) epoch 243) 2014-06-09 16:22:11.532603 7f869af78700 5 -- op tracker -- , seq: 1884, time: 2014-06-09 16:22:11.299819, event: all_read, op: pg_notify(1.71b(23),2.45b(23) epoch 243) 2014-06-09 16:22:11.532610 7f869af78700 5 -- op tracker -- , seq: 1884, time: 2014-06-09 16:22:11.532580, event: dispatched, op: pg_notify(1.71b(23),2.45b(23) epoch 243) 2014-06-09 16:22:11.532617 7f869af78700 5 -- op tracker -- , seq: 1884, time: 2014-06-09 16:22:11.532614, event: waiting_for_osdmap, op: pg_notify(1.71b(23),2.45b(23) epoch 243) 2014-06-09 16:22:11.532621 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.532623 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046afc0 2014-06-09 16:22:11.532629 7f869af78700 5 -- op tracker -- , seq: 1884, time: 2014-06-09 16:22:11.532626, event: started, op: pg_notify(1.71b(23),2.45b(23) epoch 243) 2014-06-09 16:22:11.532655 7f869af78700 5 -- op tracker -- , seq: 1884, time: 2014-06-09 16:22:11.532652, event: done, op: pg_notify(1.71b(23),2.45b(23) epoch 243) 2014-06-09 16:22:11.532663 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532665 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532693 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532694 7f869af78700 20 osd.26 243 _dispatch 0x3043ba40 pg_notify(3.583(10) epoch 243) v5 2014-06-09 16:22:11.532699 7f869af78700 5 -- op tracker -- , seq: 1885, time: 2014-06-09 16:22:11.278681, event: header_read, op: pg_notify(3.583(10) epoch 243) 2014-06-09 16:22:11.532706 7f869af78700 5 -- op tracker -- , seq: 1885, time: 2014-06-09 16:22:11.278681, event: throttled, op: pg_notify(3.583(10) epoch 243) 2014-06-09 16:22:11.532712 7f869af78700 5 -- op tracker -- , seq: 1885, time: 2014-06-09 16:22:11.278702, event: all_read, op: pg_notify(3.583(10) epoch 243) 2014-06-09 16:22:11.532718 7f869af78700 5 -- op tracker -- , seq: 1885, time: 2014-06-09 16:22:11.532691, event: dispatched, op: pg_notify(3.583(10) epoch 243) 2014-06-09 16:22:11.532724 7f869af78700 5 -- op tracker -- , seq: 1885, time: 2014-06-09 16:22:11.532722, event: waiting_for_osdmap, op: pg_notify(3.583(10) epoch 243) 2014-06-09 16:22:11.532729 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.532731 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043ba40 2014-06-09 16:22:11.532736 7f869af78700 5 -- op tracker -- , seq: 1885, time: 2014-06-09 16:22:11.532734, event: started, op: pg_notify(3.583(10) epoch 243) 2014-06-09 16:22:11.532754 7f869af78700 5 -- op tracker -- , seq: 1885, time: 2014-06-09 16:22:11.532751, event: done, op: pg_notify(3.583(10) epoch 243) 2014-06-09 16:22:11.532762 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532764 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532779 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532780 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532782 7f869af78700 20 osd.26 243 _dispatch 0x15458000 pg_info(1 pgs e243:3.124(4)) v4 2014-06-09 16:22:11.532787 7f869af78700 5 -- op tracker -- , seq: 1886, time: 2014-06-09 16:22:11.178481, event: header_read, op: pg_info(1 pgs e243:3.124(4)) 2014-06-09 16:22:11.532794 7f869af78700 5 -- op tracker -- , seq: 1886, time: 2014-06-09 16:22:11.178481, event: throttled, op: pg_info(1 pgs e243:3.124(4)) 2014-06-09 16:22:11.532800 7f869af78700 5 -- op tracker -- , seq: 1886, time: 2014-06-09 16:22:11.178493, event: all_read, op: pg_info(1 pgs e243:3.124(4)) 2014-06-09 16:22:11.532807 7f869af78700 5 -- op tracker -- , seq: 1886, time: 2014-06-09 16:22:11.532778, event: dispatched, op: pg_info(1 pgs e243:3.124(4)) 2014-06-09 16:22:11.532814 7f869af78700 5 -- op tracker -- , seq: 1886, time: 2014-06-09 16:22:11.532812, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.124(4)) 2014-06-09 16:22:11.532818 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.124(4)) v4 from osd.1 2014-06-09 16:22:11.532821 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15458000 2014-06-09 16:22:11.532827 7f869af78700 5 -- op tracker -- , seq: 1886, time: 2014-06-09 16:22:11.532824, event: started, op: pg_info(1 pgs e243:3.124(4)) 2014-06-09 16:22:11.532842 7f869af78700 5 -- op tracker -- , seq: 1886, time: 2014-06-09 16:22:11.532840, event: done, op: pg_info(1 pgs e243:3.124(4)) 2014-06-09 16:22:11.532850 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532852 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532867 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532869 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532870 7f869af78700 20 osd.26 243 _dispatch 0x1205e8c0 pg_notify(0.1a7(20),1.479(20) epoch 243) v5 2014-06-09 16:22:11.532875 7f869af78700 5 -- op tracker -- , seq: 1887, time: 2014-06-09 16:22:11.264528, event: header_read, op: pg_notify(0.1a7(20),1.479(20) epoch 243) 2014-06-09 16:22:11.532883 7f869af78700 5 -- op tracker -- , seq: 1887, time: 2014-06-09 16:22:11.264529, event: throttled, op: pg_notify(0.1a7(20),1.479(20) epoch 243) 2014-06-09 16:22:11.532890 7f869af78700 5 -- op tracker -- , seq: 1887, time: 2014-06-09 16:22:11.264584, event: all_read, op: pg_notify(0.1a7(20),1.479(20) epoch 243) 2014-06-09 16:22:11.532897 7f869af78700 5 -- op tracker -- , seq: 1887, time: 2014-06-09 16:22:11.532867, event: dispatched, op: pg_notify(0.1a7(20),1.479(20) epoch 243) 2014-06-09 16:22:11.532905 7f869af78700 5 -- op tracker -- , seq: 1887, time: 2014-06-09 16:22:11.532903, event: waiting_for_osdmap, op: pg_notify(0.1a7(20),1.479(20) epoch 243) 2014-06-09 16:22:11.532910 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.532912 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1205e8c0 2014-06-09 16:22:11.532917 7f869af78700 5 -- op tracker -- , seq: 1887, time: 2014-06-09 16:22:11.532915, event: started, op: pg_notify(0.1a7(20),1.479(20) epoch 243) 2014-06-09 16:22:11.532941 7f869af78700 5 -- op tracker -- , seq: 1887, time: 2014-06-09 16:22:11.532939, event: done, op: pg_notify(0.1a7(20),1.479(20) epoch 243) 2014-06-09 16:22:11.532951 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532974 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.532976 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.532977 7f869af78700 20 osd.26 243 _dispatch 0x190b3c00 pg_info(1 pgs e243:3.587(2)) v4 2014-06-09 16:22:11.532982 7f869af78700 5 -- op tracker -- , seq: 1888, time: 2014-06-09 16:22:11.221747, event: header_read, op: pg_info(1 pgs e243:3.587(2)) 2014-06-09 16:22:11.532989 7f869af78700 5 -- op tracker -- , seq: 1888, time: 2014-06-09 16:22:11.221748, event: throttled, op: pg_info(1 pgs e243:3.587(2)) 2014-06-09 16:22:11.532995 7f869af78700 5 -- op tracker -- , seq: 1888, time: 2014-06-09 16:22:11.221766, event: all_read, op: pg_info(1 pgs e243:3.587(2)) 2014-06-09 16:22:11.533002 7f869af78700 5 -- op tracker -- , seq: 1888, time: 2014-06-09 16:22:11.532974, event: dispatched, op: pg_info(1 pgs e243:3.587(2)) 2014-06-09 16:22:11.533009 7f869af78700 5 -- op tracker -- , seq: 1888, time: 2014-06-09 16:22:11.533007, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.587(2)) 2014-06-09 16:22:11.533013 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.587(2)) v4 from osd.8 2014-06-09 16:22:11.533017 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b3c00 2014-06-09 16:22:11.533022 7f869af78700 5 -- op tracker -- , seq: 1888, time: 2014-06-09 16:22:11.533020, event: started, op: pg_info(1 pgs e243:3.587(2)) 2014-06-09 16:22:11.533039 7f869af78700 5 -- op tracker -- , seq: 1888, time: 2014-06-09 16:22:11.533036, event: done, op: pg_info(1 pgs e243:3.587(2)) 2014-06-09 16:22:11.533047 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533061 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533063 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533064 7f869af78700 20 osd.26 243 _dispatch 0x22ac0380 pg_notify(2.5e2(20) epoch 243) v5 2014-06-09 16:22:11.533069 7f869af78700 5 -- op tracker -- , seq: 1889, time: 2014-06-09 16:22:11.282306, event: header_read, op: pg_notify(2.5e2(20) epoch 243) 2014-06-09 16:22:11.533076 7f869af78700 5 -- op tracker -- , seq: 1889, time: 2014-06-09 16:22:11.282306, event: throttled, op: pg_notify(2.5e2(20) epoch 243) 2014-06-09 16:22:11.533083 7f869af78700 5 -- op tracker -- , seq: 1889, time: 2014-06-09 16:22:11.282334, event: all_read, op: pg_notify(2.5e2(20) epoch 243) 2014-06-09 16:22:11.533089 7f869af78700 5 -- op tracker -- , seq: 1889, time: 2014-06-09 16:22:11.533061, event: dispatched, op: pg_notify(2.5e2(20) epoch 243) 2014-06-09 16:22:11.533095 7f869af78700 5 -- op tracker -- , seq: 1889, time: 2014-06-09 16:22:11.533093, event: waiting_for_osdmap, op: pg_notify(2.5e2(20) epoch 243) 2014-06-09 16:22:11.533100 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.533102 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22ac0380 2014-06-09 16:22:11.533106 7f869af78700 5 -- op tracker -- , seq: 1889, time: 2014-06-09 16:22:11.533104, event: started, op: pg_notify(2.5e2(20) epoch 243) 2014-06-09 16:22:11.533123 7f869af78700 5 -- op tracker -- , seq: 1889, time: 2014-06-09 16:22:11.533121, event: done, op: pg_notify(2.5e2(20) epoch 243) 2014-06-09 16:22:11.533131 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533133 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533151 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533153 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533154 7f869af78700 20 osd.26 243 _dispatch 0x2fef5a40 pg_notify(1.3ef(21) epoch 243) v5 2014-06-09 16:22:11.533159 7f869af78700 5 -- op tracker -- , seq: 1890, time: 2014-06-09 16:22:11.177764, event: header_read, op: pg_notify(1.3ef(21) epoch 243) 2014-06-09 16:22:11.533166 7f869af78700 5 -- op tracker -- , seq: 1890, time: 2014-06-09 16:22:11.177764, event: throttled, op: pg_notify(1.3ef(21) epoch 243) 2014-06-09 16:22:11.533172 7f869af78700 5 -- op tracker -- , seq: 1890, time: 2014-06-09 16:22:11.177784, event: all_read, op: pg_notify(1.3ef(21) epoch 243) 2014-06-09 16:22:11.533178 7f869af78700 5 -- op tracker -- , seq: 1890, time: 2014-06-09 16:22:11.533151, event: dispatched, op: pg_notify(1.3ef(21) epoch 243) 2014-06-09 16:22:11.533185 7f869af78700 5 -- op tracker -- , seq: 1890, time: 2014-06-09 16:22:11.533183, event: waiting_for_osdmap, op: pg_notify(1.3ef(21) epoch 243) 2014-06-09 16:22:11.533190 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.533192 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef5a40 2014-06-09 16:22:11.533197 7f869af78700 5 -- op tracker -- , seq: 1890, time: 2014-06-09 16:22:11.533195, event: started, op: pg_notify(1.3ef(21) epoch 243) 2014-06-09 16:22:11.533213 7f869af78700 5 -- op tracker -- , seq: 1890, time: 2014-06-09 16:22:11.533211, event: done, op: pg_notify(1.3ef(21) epoch 243) 2014-06-09 16:22:11.533221 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533223 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533244 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533246 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533247 7f869af78700 20 osd.26 243 _dispatch 0x308a6c00 pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.533254 7f869af78700 5 -- op tracker -- , seq: 1891, time: 2014-06-09 16:22:11.298331, event: header_read, op: pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533262 7f869af78700 5 -- op tracker -- , seq: 1891, time: 2014-06-09 16:22:11.298331, event: throttled, op: pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533269 7f869af78700 5 -- op tracker -- , seq: 1891, time: 2014-06-09 16:22:11.298347, event: all_read, op: pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533276 7f869af78700 5 -- op tracker -- , seq: 1891, time: 2014-06-09 16:22:11.533244, event: dispatched, op: pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533283 7f869af78700 5 -- op tracker -- , seq: 1891, time: 2014-06-09 16:22:11.533280, event: waiting_for_osdmap, op: pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533288 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.533292 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a6c00 2014-06-09 16:22:11.533297 7f869af78700 5 -- op tracker -- , seq: 1891, time: 2014-06-09 16:22:11.533295, event: started, op: pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533311 7f869af78700 5 -- op tracker -- , seq: 1891, time: 2014-06-09 16:22:11.533308, event: done, op: pg_log(3.27 epoch 243 log log((213'666,215'668], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533318 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533320 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533333 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533334 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533336 7f869af78700 20 osd.26 243 _dispatch 0x29ec4fc0 pg_notify(3.562(12),1.29a(23) epoch 243) v5 2014-06-09 16:22:11.533341 7f869af78700 5 -- op tracker -- , seq: 1892, time: 2014-06-09 16:22:11.111469, event: header_read, op: pg_notify(3.562(12),1.29a(23) epoch 243) 2014-06-09 16:22:11.533348 7f869af78700 5 -- op tracker -- , seq: 1892, time: 2014-06-09 16:22:11.111470, event: throttled, op: pg_notify(3.562(12),1.29a(23) epoch 243) 2014-06-09 16:22:11.533355 7f869af78700 5 -- op tracker -- , seq: 1892, time: 2014-06-09 16:22:11.111503, event: all_read, op: pg_notify(3.562(12),1.29a(23) epoch 243) 2014-06-09 16:22:11.533362 7f869af78700 5 -- op tracker -- , seq: 1892, time: 2014-06-09 16:22:11.533333, event: dispatched, op: pg_notify(3.562(12),1.29a(23) epoch 243) 2014-06-09 16:22:11.533369 7f869af78700 5 -- op tracker -- , seq: 1892, time: 2014-06-09 16:22:11.533367, event: waiting_for_osdmap, op: pg_notify(3.562(12),1.29a(23) epoch 243) 2014-06-09 16:22:11.533374 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.533376 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec4fc0 2014-06-09 16:22:11.533381 7f869af78700 5 -- op tracker -- , seq: 1892, time: 2014-06-09 16:22:11.533379, event: started, op: pg_notify(3.562(12),1.29a(23) epoch 243) 2014-06-09 16:22:11.533407 7f869af78700 5 -- op tracker -- , seq: 1892, time: 2014-06-09 16:22:11.533404, event: done, op: pg_notify(3.562(12),1.29a(23) epoch 243) 2014-06-09 16:22:11.533414 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533416 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533444 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533446 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533447 7f869af78700 20 osd.26 243 _dispatch 0x3043b6c0 pg_notify(1.359(23),1.623(20) epoch 243) v5 2014-06-09 16:22:11.533453 7f869af78700 5 -- op tracker -- , seq: 1893, time: 2014-06-09 16:22:11.066727, event: header_read, op: pg_notify(1.359(23),1.623(20) epoch 243) 2014-06-09 16:22:11.533461 7f869af78700 5 -- op tracker -- , seq: 1893, time: 2014-06-09 16:22:11.066728, event: throttled, op: pg_notify(1.359(23),1.623(20) epoch 243) 2014-06-09 16:22:11.533468 7f869af78700 5 -- op tracker -- , seq: 1893, time: 2014-06-09 16:22:11.066778, event: all_read, op: pg_notify(1.359(23),1.623(20) epoch 243) 2014-06-09 16:22:11.533475 7f869af78700 5 -- op tracker -- , seq: 1893, time: 2014-06-09 16:22:11.533443, event: dispatched, op: pg_notify(1.359(23),1.623(20) epoch 243) 2014-06-09 16:22:11.533482 7f869af78700 5 -- op tracker -- , seq: 1893, time: 2014-06-09 16:22:11.533479, event: waiting_for_osdmap, op: pg_notify(1.359(23),1.623(20) epoch 243) 2014-06-09 16:22:11.533486 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.533488 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043b6c0 2014-06-09 16:22:11.533493 7f869af78700 5 -- op tracker -- , seq: 1893, time: 2014-06-09 16:22:11.533491, event: started, op: pg_notify(1.359(23),1.623(20) epoch 243) 2014-06-09 16:22:11.533516 7f869af78700 5 -- op tracker -- , seq: 1893, time: 2014-06-09 16:22:11.533513, event: done, op: pg_notify(1.359(23),1.623(20) epoch 243) 2014-06-09 16:22:11.533529 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533531 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533560 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533562 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533564 7f869af78700 20 osd.26 243 _dispatch 0x306dd000 pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.533570 7f869af78700 5 -- op tracker -- , seq: 1894, time: 2014-06-09 16:22:11.299827, event: header_read, op: pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533579 7f869af78700 5 -- op tracker -- , seq: 1894, time: 2014-06-09 16:22:11.299827, event: throttled, op: pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533586 7f869af78700 5 -- op tracker -- , seq: 1894, time: 2014-06-09 16:22:11.299850, event: all_read, op: pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533593 7f869af78700 5 -- op tracker -- , seq: 1894, time: 2014-06-09 16:22:11.533560, event: dispatched, op: pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533600 7f869af78700 5 -- op tracker -- , seq: 1894, time: 2014-06-09 16:22:11.533598, event: waiting_for_osdmap, op: pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533605 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.533609 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306dd000 2014-06-09 16:22:11.533615 7f869af78700 5 -- op tracker -- , seq: 1894, time: 2014-06-09 16:22:11.533612, event: started, op: pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533628 7f869af78700 5 -- op tracker -- , seq: 1894, time: 2014-06-09 16:22:11.533626, event: done, op: pg_log(0.1a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533636 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533651 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533652 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533653 7f869af78700 20 osd.26 243 _dispatch 0x240ea700 pg_info(1 pgs e243:3.2c9(2)) v4 2014-06-09 16:22:11.533658 7f869af78700 5 -- op tracker -- , seq: 1895, time: 2014-06-09 16:22:11.278733, event: header_read, op: pg_info(1 pgs e243:3.2c9(2)) 2014-06-09 16:22:11.533666 7f869af78700 5 -- op tracker -- , seq: 1895, time: 2014-06-09 16:22:11.278733, event: throttled, op: pg_info(1 pgs e243:3.2c9(2)) 2014-06-09 16:22:11.533673 7f869af78700 5 -- op tracker -- , seq: 1895, time: 2014-06-09 16:22:11.278743, event: all_read, op: pg_info(1 pgs e243:3.2c9(2)) 2014-06-09 16:22:11.533679 7f869af78700 5 -- op tracker -- , seq: 1895, time: 2014-06-09 16:22:11.533650, event: dispatched, op: pg_info(1 pgs e243:3.2c9(2)) 2014-06-09 16:22:11.533686 7f869af78700 5 -- op tracker -- , seq: 1895, time: 2014-06-09 16:22:11.533684, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2c9(2)) 2014-06-09 16:22:11.533690 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2c9(2)) v4 from osd.3 2014-06-09 16:22:11.533693 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ea700 2014-06-09 16:22:11.533698 7f869af78700 5 -- op tracker -- , seq: 1895, time: 2014-06-09 16:22:11.533696, event: started, op: pg_info(1 pgs e243:3.2c9(2)) 2014-06-09 16:22:11.533715 7f869af78700 5 -- op tracker -- , seq: 1895, time: 2014-06-09 16:22:11.533712, event: done, op: pg_info(1 pgs e243:3.2c9(2)) 2014-06-09 16:22:11.533723 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533724 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533737 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533738 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533740 7f869af78700 20 osd.26 243 _dispatch 0x46801c0 pg_notify(0.ca(21) epoch 243) v5 2014-06-09 16:22:11.533745 7f869af78700 5 -- op tracker -- , seq: 1896, time: 2014-06-09 16:22:11.178504, event: header_read, op: pg_notify(0.ca(21) epoch 243) 2014-06-09 16:22:11.533752 7f869af78700 5 -- op tracker -- , seq: 1896, time: 2014-06-09 16:22:11.178505, event: throttled, op: pg_notify(0.ca(21) epoch 243) 2014-06-09 16:22:11.533758 7f869af78700 5 -- op tracker -- , seq: 1896, time: 2014-06-09 16:22:11.178524, event: all_read, op: pg_notify(0.ca(21) epoch 243) 2014-06-09 16:22:11.533763 7f869af78700 5 -- op tracker -- , seq: 1896, time: 2014-06-09 16:22:11.533736, event: dispatched, op: pg_notify(0.ca(21) epoch 243) 2014-06-09 16:22:11.533770 7f869af78700 5 -- op tracker -- , seq: 1896, time: 2014-06-09 16:22:11.533768, event: waiting_for_osdmap, op: pg_notify(0.ca(21) epoch 243) 2014-06-09 16:22:11.533774 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.533776 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x46801c0 2014-06-09 16:22:11.533781 7f869af78700 5 -- op tracker -- , seq: 1896, time: 2014-06-09 16:22:11.533779, event: started, op: pg_notify(0.ca(21) epoch 243) 2014-06-09 16:22:11.533797 7f869af78700 5 -- op tracker -- , seq: 1896, time: 2014-06-09 16:22:11.533795, event: done, op: pg_notify(0.ca(21) epoch 243) 2014-06-09 16:22:11.533804 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533806 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533825 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533827 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533828 7f869af78700 20 osd.26 243 _dispatch 0x30a82400 pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.533835 7f869af78700 5 -- op tracker -- , seq: 1897, time: 2014-06-09 16:22:11.265920, event: header_read, op: pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533843 7f869af78700 5 -- op tracker -- , seq: 1897, time: 2014-06-09 16:22:11.265921, event: throttled, op: pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533851 7f869af78700 5 -- op tracker -- , seq: 1897, time: 2014-06-09 16:22:11.265935, event: all_read, op: pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533857 7f869af78700 5 -- op tracker -- , seq: 1897, time: 2014-06-09 16:22:11.533825, event: dispatched, op: pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533865 7f869af78700 5 -- op tracker -- , seq: 1897, time: 2014-06-09 16:22:11.533862, event: waiting_for_osdmap, op: pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533869 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.533873 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a82400 2014-06-09 16:22:11.533879 7f869af78700 5 -- op tracker -- , seq: 1897, time: 2014-06-09 16:22:11.533876, event: started, op: pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533892 7f869af78700 5 -- op tracker -- , seq: 1897, time: 2014-06-09 16:22:11.533889, event: done, op: pg_log(1.289 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.533899 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533901 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533913 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.533915 7f869af78700 20 osd.26 243 _dispatch 0x19fda1c0 pg_notify(2.5bf(20) epoch 243) v5 2014-06-09 16:22:11.533920 7f869af78700 5 -- op tracker -- , seq: 1898, time: 2014-06-09 16:22:11.223354, event: header_read, op: pg_notify(2.5bf(20) epoch 243) 2014-06-09 16:22:11.533927 7f869af78700 5 -- op tracker -- , seq: 1898, time: 2014-06-09 16:22:11.223354, event: throttled, op: pg_notify(2.5bf(20) epoch 243) 2014-06-09 16:22:11.533934 7f869af78700 5 -- op tracker -- , seq: 1898, time: 2014-06-09 16:22:11.223380, event: all_read, op: pg_notify(2.5bf(20) epoch 243) 2014-06-09 16:22:11.533940 7f869af78700 5 -- op tracker -- , seq: 1898, time: 2014-06-09 16:22:11.533912, event: dispatched, op: pg_notify(2.5bf(20) epoch 243) 2014-06-09 16:22:11.533947 7f869af78700 5 -- op tracker -- , seq: 1898, time: 2014-06-09 16:22:11.533945, event: waiting_for_osdmap, op: pg_notify(2.5bf(20) epoch 243) 2014-06-09 16:22:11.533952 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.533954 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fda1c0 2014-06-09 16:22:11.533959 7f869af78700 5 -- op tracker -- , seq: 1898, time: 2014-06-09 16:22:11.533957, event: started, op: pg_notify(2.5bf(20) epoch 243) 2014-06-09 16:22:11.533974 7f869af78700 5 -- op tracker -- , seq: 1898, time: 2014-06-09 16:22:11.533972, event: done, op: pg_notify(2.5bf(20) epoch 243) 2014-06-09 16:22:11.533982 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.533984 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534001 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534003 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534004 7f869af78700 20 osd.26 243 _dispatch 0x306d5000 pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.534010 7f869af78700 5 -- op tracker -- , seq: 1899, time: 2014-06-09 16:22:11.282344, event: header_read, op: pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534018 7f869af78700 5 -- op tracker -- , seq: 1899, time: 2014-06-09 16:22:11.282344, event: throttled, op: pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534025 7f869af78700 5 -- op tracker -- , seq: 1899, time: 2014-06-09 16:22:11.282362, event: all_read, op: pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534033 7f869af78700 5 -- op tracker -- , seq: 1899, time: 2014-06-09 16:22:11.534000, event: dispatched, op: pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534040 7f869af78700 5 -- op tracker -- , seq: 1899, time: 2014-06-09 16:22:11.534037, event: waiting_for_osdmap, op: pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534045 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.534049 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d5000 2014-06-09 16:22:11.534055 7f869af78700 5 -- op tracker -- , seq: 1899, time: 2014-06-09 16:22:11.534052, event: started, op: pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534068 7f869af78700 5 -- op tracker -- , seq: 1899, time: 2014-06-09 16:22:11.534065, event: done, op: pg_log(1.6f5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534076 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534090 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534091 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534093 7f869af78700 20 osd.26 243 _dispatch 0x305f2600 pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.534099 7f869af78700 5 -- op tracker -- , seq: 1900, time: 2014-06-09 16:22:11.189870, event: header_read, op: pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534107 7f869af78700 5 -- op tracker -- , seq: 1900, time: 2014-06-09 16:22:11.189870, event: throttled, op: pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534113 7f869af78700 5 -- op tracker -- , seq: 1900, time: 2014-06-09 16:22:11.189880, event: all_read, op: pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534120 7f869af78700 5 -- op tracker -- , seq: 1900, time: 2014-06-09 16:22:11.534090, event: dispatched, op: pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534128 7f869af78700 5 -- op tracker -- , seq: 1900, time: 2014-06-09 16:22:11.534125, event: waiting_for_osdmap, op: pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534132 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.534136 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f2600 2014-06-09 16:22:11.534141 7f869af78700 5 -- op tracker -- , seq: 1900, time: 2014-06-09 16:22:11.534139, event: started, op: pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534154 7f869af78700 5 -- op tracker -- , seq: 1900, time: 2014-06-09 16:22:11.534152, event: done, op: pg_log(1.15b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534161 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534163 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534174 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534175 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534177 7f869af78700 20 osd.26 243 _dispatch 0x308a7e00 pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.534183 7f869af78700 5 -- op tracker -- , seq: 1901, time: 2014-06-09 16:22:11.298397, event: header_read, op: pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534190 7f869af78700 5 -- op tracker -- , seq: 1901, time: 2014-06-09 16:22:11.298398, event: throttled, op: pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534197 7f869af78700 5 -- op tracker -- , seq: 1901, time: 2014-06-09 16:22:11.298405, event: all_read, op: pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534203 7f869af78700 5 -- op tracker -- , seq: 1901, time: 2014-06-09 16:22:11.534174, event: dispatched, op: pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534211 7f869af78700 5 -- op tracker -- , seq: 1901, time: 2014-06-09 16:22:11.534208, event: waiting_for_osdmap, op: pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534216 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.534220 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a7e00 2014-06-09 16:22:11.534226 7f869af78700 5 -- op tracker -- , seq: 1901, time: 2014-06-09 16:22:11.534223, event: started, op: pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534240 7f869af78700 5 -- op tracker -- , seq: 1901, time: 2014-06-09 16:22:11.534237, event: done, op: pg_log(0.71b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534248 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534250 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534261 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534263 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534264 7f869af78700 20 osd.26 243 _dispatch 0x305bca00 pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.534270 7f869af78700 5 -- op tracker -- , seq: 1902, time: 2014-06-09 16:22:11.111516, event: header_read, op: pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534277 7f869af78700 5 -- op tracker -- , seq: 1902, time: 2014-06-09 16:22:11.111516, event: throttled, op: pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534284 7f869af78700 5 -- op tracker -- , seq: 1902, time: 2014-06-09 16:22:11.111530, event: all_read, op: pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534292 7f869af78700 5 -- op tracker -- , seq: 1902, time: 2014-06-09 16:22:11.534261, event: dispatched, op: pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534299 7f869af78700 5 -- op tracker -- , seq: 1902, time: 2014-06-09 16:22:11.534296, event: waiting_for_osdmap, op: pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534304 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.534308 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305bca00 2014-06-09 16:22:11.534314 7f869af78700 5 -- op tracker -- , seq: 1902, time: 2014-06-09 16:22:11.534311, event: started, op: pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534326 7f869af78700 5 -- op tracker -- , seq: 1902, time: 2014-06-09 16:22:11.534324, event: done, op: pg_log(2.431 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534334 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534335 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534348 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534349 7f869af78700 20 osd.26 243 _dispatch 0x305e5600 pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.534356 7f869af78700 5 -- op tracker -- , seq: 1903, time: 2014-06-09 16:22:11.103834, event: header_read, op: pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534364 7f869af78700 5 -- op tracker -- , seq: 1903, time: 2014-06-09 16:22:11.103835, event: throttled, op: pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534371 7f869af78700 5 -- op tracker -- , seq: 1903, time: 2014-06-09 16:22:11.103849, event: all_read, op: pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534378 7f869af78700 5 -- op tracker -- , seq: 1903, time: 2014-06-09 16:22:11.534346, event: dispatched, op: pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534385 7f869af78700 5 -- op tracker -- , seq: 1903, time: 2014-06-09 16:22:11.534382, event: waiting_for_osdmap, op: pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534389 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.534393 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e5600 2014-06-09 16:22:11.534399 7f869af78700 5 -- op tracker -- , seq: 1903, time: 2014-06-09 16:22:11.534396, event: started, op: pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534412 7f869af78700 5 -- op tracker -- , seq: 1903, time: 2014-06-09 16:22:11.534409, event: done, op: pg_log(1.375 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534419 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534421 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534433 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534435 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534436 7f869af78700 20 osd.26 243 _dispatch 0x22012540 pg_info(1 pgs e243:3.136(2)) v4 2014-06-09 16:22:11.534442 7f869af78700 5 -- op tracker -- , seq: 1904, time: 2014-06-09 16:22:11.301882, event: header_read, op: pg_info(1 pgs e243:3.136(2)) 2014-06-09 16:22:11.534449 7f869af78700 5 -- op tracker -- , seq: 1904, time: 2014-06-09 16:22:11.301883, event: throttled, op: pg_info(1 pgs e243:3.136(2)) 2014-06-09 16:22:11.534456 7f869af78700 5 -- op tracker -- , seq: 1904, time: 2014-06-09 16:22:11.301899, event: all_read, op: pg_info(1 pgs e243:3.136(2)) 2014-06-09 16:22:11.534462 7f869af78700 5 -- op tracker -- , seq: 1904, time: 2014-06-09 16:22:11.534433, event: dispatched, op: pg_info(1 pgs e243:3.136(2)) 2014-06-09 16:22:11.534469 7f869af78700 5 -- op tracker -- , seq: 1904, time: 2014-06-09 16:22:11.534467, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.136(2)) 2014-06-09 16:22:11.534473 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.136(2)) v4 from osd.7 2014-06-09 16:22:11.534477 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22012540 2014-06-09 16:22:11.534482 7f869af78700 5 -- op tracker -- , seq: 1904, time: 2014-06-09 16:22:11.534480, event: started, op: pg_info(1 pgs e243:3.136(2)) 2014-06-09 16:22:11.534499 7f869af78700 5 -- op tracker -- , seq: 1904, time: 2014-06-09 16:22:11.534496, event: done, op: pg_info(1 pgs e243:3.136(2)) 2014-06-09 16:22:11.534507 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534509 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534523 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534524 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534526 7f869af78700 20 osd.26 243 _dispatch 0x30952400 pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.534532 7f869af78700 5 -- op tracker -- , seq: 1905, time: 2014-06-09 16:22:11.279641, event: header_read, op: pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534541 7f869af78700 5 -- op tracker -- , seq: 1905, time: 2014-06-09 16:22:11.279641, event: throttled, op: pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534548 7f869af78700 5 -- op tracker -- , seq: 1905, time: 2014-06-09 16:22:11.279649, event: all_read, op: pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534555 7f869af78700 5 -- op tracker -- , seq: 1905, time: 2014-06-09 16:22:11.534523, event: dispatched, op: pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534563 7f869af78700 5 -- op tracker -- , seq: 1905, time: 2014-06-09 16:22:11.534560, event: waiting_for_osdmap, op: pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534568 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.534572 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30952400 2014-06-09 16:22:11.534577 7f869af78700 5 -- op tracker -- , seq: 1905, time: 2014-06-09 16:22:11.534575, event: started, op: pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534591 7f869af78700 5 -- op tracker -- , seq: 1905, time: 2014-06-09 16:22:11.534588, event: done, op: pg_log(2.112 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534599 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534601 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534612 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534614 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534616 7f869af78700 20 osd.26 243 _dispatch 0x4680000 pg_notify(0.161(21) epoch 243) v5 2014-06-09 16:22:11.534621 7f869af78700 5 -- op tracker -- , seq: 1906, time: 2014-06-09 16:22:11.178978, event: header_read, op: pg_notify(0.161(21) epoch 243) 2014-06-09 16:22:11.534628 7f869af78700 5 -- op tracker -- , seq: 1906, time: 2014-06-09 16:22:11.178978, event: throttled, op: pg_notify(0.161(21) epoch 243) 2014-06-09 16:22:11.534634 7f869af78700 5 -- op tracker -- , seq: 1906, time: 2014-06-09 16:22:11.178996, event: all_read, op: pg_notify(0.161(21) epoch 243) 2014-06-09 16:22:11.534641 7f869af78700 5 -- op tracker -- , seq: 1906, time: 2014-06-09 16:22:11.534612, event: dispatched, op: pg_notify(0.161(21) epoch 243) 2014-06-09 16:22:11.534647 7f869af78700 5 -- op tracker -- , seq: 1906, time: 2014-06-09 16:22:11.534645, event: waiting_for_osdmap, op: pg_notify(0.161(21) epoch 243) 2014-06-09 16:22:11.534652 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.534654 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x4680000 2014-06-09 16:22:11.534658 7f869af78700 5 -- op tracker -- , seq: 1906, time: 2014-06-09 16:22:11.534657, event: started, op: pg_notify(0.161(21) epoch 243) 2014-06-09 16:22:11.534674 7f869af78700 5 -- op tracker -- , seq: 1906, time: 2014-06-09 16:22:11.534672, event: done, op: pg_notify(0.161(21) epoch 243) 2014-06-09 16:22:11.534682 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534684 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534701 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534703 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534704 7f869af78700 20 osd.26 243 _dispatch 0x3088c1c0 pg_info(1 pgs e243:3.27b(2)) v4 2014-06-09 16:22:11.534709 7f869af78700 5 -- op tracker -- , seq: 1907, time: 2014-06-09 16:22:11.266134, event: header_read, op: pg_info(1 pgs e243:3.27b(2)) 2014-06-09 16:22:11.534717 7f869af78700 5 -- op tracker -- , seq: 1907, time: 2014-06-09 16:22:11.266134, event: throttled, op: pg_info(1 pgs e243:3.27b(2)) 2014-06-09 16:22:11.534724 7f869af78700 5 -- op tracker -- , seq: 1907, time: 2014-06-09 16:22:11.266152, event: all_read, op: pg_info(1 pgs e243:3.27b(2)) 2014-06-09 16:22:11.534730 7f869af78700 5 -- op tracker -- , seq: 1907, time: 2014-06-09 16:22:11.534701, event: dispatched, op: pg_info(1 pgs e243:3.27b(2)) 2014-06-09 16:22:11.534737 7f869af78700 5 -- op tracker -- , seq: 1907, time: 2014-06-09 16:22:11.534735, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.27b(2)) 2014-06-09 16:22:11.534741 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.27b(2)) v4 from osd.9 2014-06-09 16:22:11.534745 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088c1c0 2014-06-09 16:22:11.534750 7f869af78700 5 -- op tracker -- , seq: 1907, time: 2014-06-09 16:22:11.534748, event: started, op: pg_info(1 pgs e243:3.27b(2)) 2014-06-09 16:22:11.534767 7f869af78700 5 -- op tracker -- , seq: 1907, time: 2014-06-09 16:22:11.534765, event: done, op: pg_info(1 pgs e243:3.27b(2)) 2014-06-09 16:22:11.534774 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534776 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534791 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534792 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534794 7f869af78700 20 osd.26 243 _dispatch 0x30718600 pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.534800 7f869af78700 5 -- op tracker -- , seq: 1908, time: 2014-06-09 16:22:11.263887, event: header_read, op: pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534808 7f869af78700 5 -- op tracker -- , seq: 1908, time: 2014-06-09 16:22:11.263887, event: throttled, op: pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534815 7f869af78700 5 -- op tracker -- , seq: 1908, time: 2014-06-09 16:22:11.263905, event: all_read, op: pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534821 7f869af78700 5 -- op tracker -- , seq: 1908, time: 2014-06-09 16:22:11.534790, event: dispatched, op: pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534828 7f869af78700 5 -- op tracker -- , seq: 1908, time: 2014-06-09 16:22:11.534825, event: waiting_for_osdmap, op: pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534833 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.534837 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30718600 2014-06-09 16:22:11.534843 7f869af78700 5 -- op tracker -- , seq: 1908, time: 2014-06-09 16:22:11.534840, event: started, op: pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534857 7f869af78700 5 -- op tracker -- , seq: 1908, time: 2014-06-09 16:22:11.534854, event: done, op: pg_log(2.3aa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.534868 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.534870 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.534953 7f86a5e11700 10 _do_op 0x2fc8ca80 seq 3140 r = 0, finisher 0xbfce5a0 0 2014-06-09 16:22:11.534961 7f86a5e11700 10 _finish_op 0x2fc8ca80 seq 3140 osr(0.dd 0x2bd6550)/0x2bd6550 2014-06-09 16:22:11.534971 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287560 (1 active) 2014-06-09 16:22:11.534974 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2867e0 (2 active) 2014-06-09 16:22:11.534977 7f86a5e11700 5 _do_op 0x2fc8c9c0 seq 3142 osr(3.76 0x3ed0240)/0x3ed0240 start 2014-06-09 16:22:11.534981 7f86a5e11700 10 _do_transaction on 0xc281440 2014-06-09 16:22:11.534988 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.535041 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.535062 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3142.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.535067 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.535121 7f86a5e11700 10 _do_op 0x2fc8c9c0 seq 3142 r = 0, finisher 0xbd66820 0 2014-06-09 16:22:11.535125 7f86a5e11700 10 _finish_op 0x2fc8c9c0 seq 3142 osr(3.76 0x3ed0240)/0x3ed0240 2014-06-09 16:22:11.535128 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2867e0 (1 active) 2014-06-09 16:22:11.535131 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbe60 (2 active) 2014-06-09 16:22:11.535133 7f86a5e11700 5 _do_op 0x2fc8c960 seq 3143 osr(2.6f 0x2ed0550)/0x2ed0550 start 2014-06-09 16:22:11.535136 7f86a5e11700 10 _do_transaction on 0xc2bc600 2014-06-09 16:22:11.535142 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.535175 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.535193 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3143.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.535197 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.535217 7f86a5e11700 10 _do_op 0x2fc8c960 seq 3143 r = 0, finisher 0x3062a0a0 0 2014-06-09 16:22:11.535221 7f86a5e11700 10 _finish_op 0x2fc8c960 seq 3143 osr(2.6f 0x2ed0550)/0x2ed0550 2014-06-09 16:22:11.535224 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbe60 (1 active) 2014-06-09 16:22:11.535226 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbd40 (2 active) 2014-06-09 16:22:11.535228 7f86a5e11700 5 _do_op 0x2fc8c900 seq 3144 osr(3.603 0x3ed1af0)/0x3ed1af0 start 2014-06-09 16:22:11.535231 7f86a5e11700 10 _do_transaction on 0xbf28a80 2014-06-09 16:22:11.535237 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.535269 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.535286 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3144.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.535291 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.535317 7f86a5e11700 10 _do_op 0x2fc8c900 seq 3144 r = 0, finisher 0xcd9e220 0 2014-06-09 16:22:11.535320 7f86a5e11700 10 _finish_op 0x2fc8c900 seq 3144 osr(3.603 0x3ed1af0)/0x3ed1af0 2014-06-09 16:22:11.535323 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbd40 (1 active) 2014-06-09 16:22:11.535325 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f67e0 (2 active) 2014-06-09 16:22:11.535327 7f86a5e11700 5 _do_op 0x2fc8c8a0 seq 3145 osr(1.2d0 0x384d180)/0x384d180 start 2014-06-09 16:22:11.535330 7f86a5e11700 10 _do_transaction on 0xbf88540 2014-06-09 16:22:11.535337 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.535371 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.535389 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3145.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.535393 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.535414 7f86a5e11700 10 _do_op 0x2fc8c8a0 seq 3145 r = 0, finisher 0xbd6bb20 0 2014-06-09 16:22:11.535418 7f86a5e11700 10 _finish_op 0x2fc8c8a0 seq 3145 osr(1.2d0 0x384d180)/0x384d180 2014-06-09 16:22:11.535421 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f67e0 (1 active) 2014-06-09 16:22:11.535422 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339e60 (2 active) 2014-06-09 16:22:11.535425 7f86a5e11700 5 _do_op 0x2fcae240 seq 3146 osr(3.74 0x3ed0290)/0x3ed0290 start 2014-06-09 16:22:11.535427 7f86a5e11700 10 _do_transaction on 0xc14af00 2014-06-09 16:22:11.535433 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.535468 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.535485 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3146.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.535490 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.535507 7f86a5e11700 10 _do_op 0x2fcae240 seq 3146 r = 0, finisher 0x305094e0 0 2014-06-09 16:22:11.535510 7f86a5e11700 10 _finish_op 0x2fcae240 seq 3146 osr(3.74 0x3ed0290)/0x3ed0290 2014-06-09 16:22:11.535512 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339e60 (1 active) 2014-06-09 16:22:11.535514 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339440 (2 active) 2014-06-09 16:22:11.535516 7f86a5e11700 5 _do_op 0x2fcae0c0 seq 3147 osr(0.3b6 0x2bd72d0)/0x2bd72d0 start 2014-06-09 16:22:11.535519 7f86a5e11700 10 _do_transaction on 0x2eaa480 2014-06-09 16:22:11.535525 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.535555 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.535572 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3147.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.535577 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.535599 7f86a5e11700 10 _do_op 0x2fcae0c0 seq 3147 r = 0, finisher 0x3050b780 0 2014-06-09 16:22:11.535603 7f86a5e11700 10 _finish_op 0x2fcae0c0 seq 3147 osr(0.3b6 0x2bd72d0)/0x2bd72d0 2014-06-09 16:22:11.535606 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339440 (1 active) 2014-06-09 16:22:11.535608 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6538c0 (2 active) 2014-06-09 16:22:11.535610 7f86a5e11700 5 _do_op 0x30060120 seq 3148 osr(3.562 0x3ed1920)/0x3ed1920 start 2014-06-09 16:22:11.535618 7f86a5e11700 10 _do_transaction on 0xc20f200 2014-06-09 16:22:11.535624 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.535656 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.535672 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3148.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.535676 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.535695 7f86a5e11700 10 _do_op 0x30060120 seq 3148 r = 0, finisher 0x15a3ed80 0 2014-06-09 16:22:11.535698 7f86a5e11700 10 _finish_op 0x30060120 seq 3148 osr(3.562 0x3ed1920)/0x3ed1920 2014-06-09 16:22:11.535701 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6538c0 (1 active) 2014-06-09 16:22:11.535710 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6d320,0x3061e9c0,0xbd66760,0xbd76f80,0xbd6daa0,0x33a6be0,0xbd6c6e0,0xbd72020] 2014-06-09 16:22:11.544585 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6d320,0x3061e9c0,0xbd66760,0xbd76f80,0xbd6daa0,0x33a6be0,0xbd6c6e0,0xbd72020] 2014-06-09 16:22:11.544593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.544594 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.535716 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfce5a0,0xbd66820,0x3062a0a0,0xcd9e220,0xbd6bb20,0x305094e0,0x3050b780,0x15a3ed80] 2014-06-09 16:22:11.544601 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.544627 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.544648 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.544666 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.544683 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.544698 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.544716 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.544731 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.544747 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.544765 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.544780 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.544796 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.544813 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.544827 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.544841 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.544857 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.544871 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.544887 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.544900 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.544915 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.544930 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.544944 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.544959 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfce5a0,0xbd66820,0x3062a0a0,0xcd9e220,0xbd6bb20,0x305094e0,0x3050b780,0x15a3ed80] 2014-06-09 16:22:11.544963 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.544964 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.535751 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.544996 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3141.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.545002 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.545049 7f86a5610700 10 _do_op 0x2fc8ca20 seq 3141 r = 0, finisher 0x3062a960 0 2014-06-09 16:22:11.535759 7f8691765700 10 op_submit_start 3190 2014-06-09 16:22:11.545152 7f8691765700 5 queue_transactions (trailing journal) 3190 2014-06-09 16:22:11.545157 7f8691765700 5 queue_op 0x30607980 seq 3190 osr(2.501 0x2ed1690) 2052 bytes (queue has 50 ops and 91881 bytes) 2014-06-09 16:22:11.535793 7f868df5e700 2 waiting 51 > 50 ops || 93878 > 104857600 2014-06-09 16:22:11.534882 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545171 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545174 7f869af78700 20 osd.26 243 _dispatch 0x308e2000 pg_notify(0.532(23),2.53d(20) epoch 243) v5 2014-06-09 16:22:11.545184 7f869af78700 5 -- op tracker -- , seq: 1909, time: 2014-06-09 16:22:11.300535, event: header_read, op: pg_notify(0.532(23),2.53d(20) epoch 243) 2014-06-09 16:22:11.545193 7f869af78700 5 -- op tracker -- , seq: 1909, time: 2014-06-09 16:22:11.300536, event: throttled, op: pg_notify(0.532(23),2.53d(20) epoch 243) 2014-06-09 16:22:11.545201 7f869af78700 5 -- op tracker -- , seq: 1909, time: 2014-06-09 16:22:11.300576, event: all_read, op: pg_notify(0.532(23),2.53d(20) epoch 243) 2014-06-09 16:22:11.545208 7f869af78700 5 -- op tracker -- , seq: 1909, time: 2014-06-09 16:22:11.534882, event: dispatched, op: pg_notify(0.532(23),2.53d(20) epoch 243) 2014-06-09 16:22:11.545216 7f869af78700 5 -- op tracker -- , seq: 1909, time: 2014-06-09 16:22:11.545214, event: waiting_for_osdmap, op: pg_notify(0.532(23),2.53d(20) epoch 243) 2014-06-09 16:22:11.545221 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.545224 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e2000 2014-06-09 16:22:11.545231 7f869af78700 5 -- op tracker -- , seq: 1909, time: 2014-06-09 16:22:11.545229, event: started, op: pg_notify(0.532(23),2.53d(20) epoch 243) 2014-06-09 16:22:11.545260 7f869af78700 5 -- op tracker -- , seq: 1909, time: 2014-06-09 16:22:11.545258, event: done, op: pg_notify(0.532(23),2.53d(20) epoch 243) 2014-06-09 16:22:11.545271 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545310 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545312 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545313 7f869af78700 20 osd.26 243 _dispatch 0x160641c0 pg_notify(2.412(26) epoch 243) v5 2014-06-09 16:22:11.545318 7f869af78700 5 -- op tracker -- , seq: 1910, time: 2014-06-09 16:22:11.189914, event: header_read, op: pg_notify(2.412(26) epoch 243) 2014-06-09 16:22:11.545326 7f869af78700 5 -- op tracker -- , seq: 1910, time: 2014-06-09 16:22:11.189914, event: throttled, op: pg_notify(2.412(26) epoch 243) 2014-06-09 16:22:11.545332 7f869af78700 5 -- op tracker -- , seq: 1910, time: 2014-06-09 16:22:11.189937, event: all_read, op: pg_notify(2.412(26) epoch 243) 2014-06-09 16:22:11.545338 7f869af78700 5 -- op tracker -- , seq: 1910, time: 2014-06-09 16:22:11.545309, event: dispatched, op: pg_notify(2.412(26) epoch 243) 2014-06-09 16:22:11.545345 7f869af78700 5 -- op tracker -- , seq: 1910, time: 2014-06-09 16:22:11.545343, event: waiting_for_osdmap, op: pg_notify(2.412(26) epoch 243) 2014-06-09 16:22:11.545349 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.545351 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x160641c0 2014-06-09 16:22:11.545356 7f869af78700 5 -- op tracker -- , seq: 1910, time: 2014-06-09 16:22:11.545354, event: started, op: pg_notify(2.412(26) epoch 243) 2014-06-09 16:22:11.545372 7f869af78700 5 -- op tracker -- , seq: 1910, time: 2014-06-09 16:22:11.545370, event: done, op: pg_notify(2.412(26) epoch 243) 2014-06-09 16:22:11.545381 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545383 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545404 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545407 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545408 7f869af78700 20 osd.26 243 _dispatch 0x19fda540 pg_notify(2.285(23) epoch 243) v5 2014-06-09 16:22:11.545414 7f869af78700 5 -- op tracker -- , seq: 1911, time: 2014-06-09 16:22:11.304981, event: header_read, op: pg_notify(2.285(23) epoch 243) 2014-06-09 16:22:11.545421 7f869af78700 5 -- op tracker -- , seq: 1911, time: 2014-06-09 16:22:11.304982, event: throttled, op: pg_notify(2.285(23) epoch 243) 2014-06-09 16:22:11.545427 7f869af78700 5 -- op tracker -- , seq: 1911, time: 2014-06-09 16:22:11.305016, event: all_read, op: pg_notify(2.285(23) epoch 243) 2014-06-09 16:22:11.545433 7f869af78700 5 -- op tracker -- , seq: 1911, time: 2014-06-09 16:22:11.545404, event: dispatched, op: pg_notify(2.285(23) epoch 243) 2014-06-09 16:22:11.545440 7f869af78700 5 -- op tracker -- , seq: 1911, time: 2014-06-09 16:22:11.545438, event: waiting_for_osdmap, op: pg_notify(2.285(23) epoch 243) 2014-06-09 16:22:11.545445 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.545447 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fda540 2014-06-09 16:22:11.545451 7f869af78700 5 -- op tracker -- , seq: 1911, time: 2014-06-09 16:22:11.545450, event: started, op: pg_notify(2.285(23) epoch 243) 2014-06-09 16:22:11.545467 7f869af78700 5 -- op tracker -- , seq: 1911, time: 2014-06-09 16:22:11.545465, event: done, op: pg_notify(2.285(23) epoch 243) 2014-06-09 16:22:11.545475 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545477 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545503 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545504 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545506 7f869af78700 20 osd.26 243 _dispatch 0x1607b340 pg_notify(0.38c(21),0.c8(21) epoch 243) v5 2014-06-09 16:22:11.545512 7f869af78700 5 -- op tracker -- , seq: 1912, time: 2014-06-09 16:22:11.111538, event: header_read, op: pg_notify(0.38c(21),0.c8(21) epoch 243) 2014-06-09 16:22:11.545519 7f869af78700 5 -- op tracker -- , seq: 1912, time: 2014-06-09 16:22:11.111538, event: throttled, op: pg_notify(0.38c(21),0.c8(21) epoch 243) 2014-06-09 16:22:11.545526 7f869af78700 5 -- op tracker -- , seq: 1912, time: 2014-06-09 16:22:11.111571, event: all_read, op: pg_notify(0.38c(21),0.c8(21) epoch 243) 2014-06-09 16:22:11.545533 7f869af78700 5 -- op tracker -- , seq: 1912, time: 2014-06-09 16:22:11.545502, event: dispatched, op: pg_notify(0.38c(21),0.c8(21) epoch 243) 2014-06-09 16:22:11.545540 7f869af78700 5 -- op tracker -- , seq: 1912, time: 2014-06-09 16:22:11.545538, event: waiting_for_osdmap, op: pg_notify(0.38c(21),0.c8(21) epoch 243) 2014-06-09 16:22:11.545544 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.545546 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1607b340 2014-06-09 16:22:11.545551 7f869af78700 5 -- op tracker -- , seq: 1912, time: 2014-06-09 16:22:11.545549, event: started, op: pg_notify(0.38c(21),0.c8(21) epoch 243) 2014-06-09 16:22:11.545576 7f869af78700 5 -- op tracker -- , seq: 1912, time: 2014-06-09 16:22:11.545573, event: done, op: pg_notify(0.38c(21),0.c8(21) epoch 243) 2014-06-09 16:22:11.545584 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545586 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545615 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545617 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545618 7f869af78700 20 osd.26 243 _dispatch 0x305e5000 pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.545624 7f869af78700 5 -- op tracker -- , seq: 1913, time: 2014-06-09 16:22:11.103864, event: header_read, op: pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545633 7f869af78700 5 -- op tracker -- , seq: 1913, time: 2014-06-09 16:22:11.103865, event: throttled, op: pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545641 7f869af78700 5 -- op tracker -- , seq: 1913, time: 2014-06-09 16:22:11.103876, event: all_read, op: pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545647 7f869af78700 5 -- op tracker -- , seq: 1913, time: 2014-06-09 16:22:11.545615, event: dispatched, op: pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545655 7f869af78700 5 -- op tracker -- , seq: 1913, time: 2014-06-09 16:22:11.545652, event: waiting_for_osdmap, op: pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545660 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.545664 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e5000 2014-06-09 16:22:11.545669 7f869af78700 5 -- op tracker -- , seq: 1913, time: 2014-06-09 16:22:11.545667, event: started, op: pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545683 7f869af78700 5 -- op tracker -- , seq: 1913, time: 2014-06-09 16:22:11.545680, event: done, op: pg_log(0.b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545693 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545707 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545709 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545710 7f869af78700 20 osd.26 243 _dispatch 0x30883600 pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.545717 7f869af78700 5 -- op tracker -- , seq: 1914, time: 2014-06-09 16:22:11.316770, event: header_read, op: pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545724 7f869af78700 5 -- op tracker -- , seq: 1914, time: 2014-06-09 16:22:11.316771, event: throttled, op: pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545731 7f869af78700 5 -- op tracker -- , seq: 1914, time: 2014-06-09 16:22:11.316786, event: all_read, op: pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545738 7f869af78700 5 -- op tracker -- , seq: 1914, time: 2014-06-09 16:22:11.545707, event: dispatched, op: pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545746 7f869af78700 5 -- op tracker -- , seq: 1914, time: 2014-06-09 16:22:11.545743, event: waiting_for_osdmap, op: pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545751 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.545755 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30883600 2014-06-09 16:22:11.545761 7f869af78700 5 -- op tracker -- , seq: 1914, time: 2014-06-09 16:22:11.545758, event: started, op: pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545775 7f869af78700 5 -- op tracker -- , seq: 1914, time: 2014-06-09 16:22:11.545772, event: done, op: pg_log(1.24c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545783 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545785 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545796 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545798 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545799 7f869af78700 20 osd.26 243 _dispatch 0x3043b180 pg_notify(0.5d4(20) epoch 243) v5 2014-06-09 16:22:11.545804 7f869af78700 5 -- op tracker -- , seq: 1915, time: 2014-06-09 16:22:11.279727, event: header_read, op: pg_notify(0.5d4(20) epoch 243) 2014-06-09 16:22:11.545811 7f869af78700 5 -- op tracker -- , seq: 1915, time: 2014-06-09 16:22:11.279728, event: throttled, op: pg_notify(0.5d4(20) epoch 243) 2014-06-09 16:22:11.545817 7f869af78700 5 -- op tracker -- , seq: 1915, time: 2014-06-09 16:22:11.279754, event: all_read, op: pg_notify(0.5d4(20) epoch 243) 2014-06-09 16:22:11.545824 7f869af78700 5 -- op tracker -- , seq: 1915, time: 2014-06-09 16:22:11.545796, event: dispatched, op: pg_notify(0.5d4(20) epoch 243) 2014-06-09 16:22:11.545831 7f869af78700 5 -- op tracker -- , seq: 1915, time: 2014-06-09 16:22:11.545829, event: waiting_for_osdmap, op: pg_notify(0.5d4(20) epoch 243) 2014-06-09 16:22:11.545836 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.545838 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043b180 2014-06-09 16:22:11.545843 7f869af78700 5 -- op tracker -- , seq: 1915, time: 2014-06-09 16:22:11.545841, event: started, op: pg_notify(0.5d4(20) epoch 243) 2014-06-09 16:22:11.545859 7f869af78700 5 -- op tracker -- , seq: 1915, time: 2014-06-09 16:22:11.545857, event: done, op: pg_notify(0.5d4(20) epoch 243) 2014-06-09 16:22:11.545867 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545869 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545887 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545888 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545889 7f869af78700 20 osd.26 243 _dispatch 0x30645200 pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.545896 7f869af78700 5 -- op tracker -- , seq: 1916, time: 2014-06-09 16:22:11.179390, event: header_read, op: pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545904 7f869af78700 5 -- op tracker -- , seq: 1916, time: 2014-06-09 16:22:11.179391, event: throttled, op: pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545911 7f869af78700 5 -- op tracker -- , seq: 1916, time: 2014-06-09 16:22:11.179400, event: all_read, op: pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545918 7f869af78700 5 -- op tracker -- , seq: 1916, time: 2014-06-09 16:22:11.545886, event: dispatched, op: pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545925 7f869af78700 5 -- op tracker -- , seq: 1916, time: 2014-06-09 16:22:11.545922, event: waiting_for_osdmap, op: pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545930 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.545934 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30645200 2014-06-09 16:22:11.545940 7f869af78700 5 -- op tracker -- , seq: 1916, time: 2014-06-09 16:22:11.545937, event: started, op: pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545953 7f869af78700 5 -- op tracker -- , seq: 1916, time: 2014-06-09 16:22:11.545950, event: done, op: pg_log(1.73b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.545961 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545963 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545976 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.545978 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.545979 7f869af78700 20 osd.26 243 _dispatch 0x10d0d880 pg_info(2 pgs e243:3.537(2),3.53d(2)) v4 2014-06-09 16:22:11.545986 7f869af78700 5 -- op tracker -- , seq: 1917, time: 2014-06-09 16:22:11.289051, event: header_read, op: pg_info(2 pgs e243:3.537(2),3.53d(2)) 2014-06-09 16:22:11.545993 7f869af78700 5 -- op tracker -- , seq: 1917, time: 2014-06-09 16:22:11.289052, event: throttled, op: pg_info(2 pgs e243:3.537(2),3.53d(2)) 2014-06-09 16:22:11.546000 7f869af78700 5 -- op tracker -- , seq: 1917, time: 2014-06-09 16:22:11.289074, event: all_read, op: pg_info(2 pgs e243:3.537(2),3.53d(2)) 2014-06-09 16:22:11.546007 7f869af78700 5 -- op tracker -- , seq: 1917, time: 2014-06-09 16:22:11.545976, event: dispatched, op: pg_info(2 pgs e243:3.537(2),3.53d(2)) 2014-06-09 16:22:11.546014 7f869af78700 5 -- op tracker -- , seq: 1917, time: 2014-06-09 16:22:11.546011, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.537(2),3.53d(2)) 2014-06-09 16:22:11.546019 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.537(2),3.53d(2)) v4 from osd.9 2014-06-09 16:22:11.546023 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d0d880 2014-06-09 16:22:11.546029 7f869af78700 5 -- op tracker -- , seq: 1917, time: 2014-06-09 16:22:11.546026, event: started, op: pg_info(2 pgs e243:3.537(2),3.53d(2)) 2014-06-09 16:22:11.546053 7f869af78700 5 -- op tracker -- , seq: 1917, time: 2014-06-09 16:22:11.546050, event: done, op: pg_info(2 pgs e243:3.537(2),3.53d(2)) 2014-06-09 16:22:11.546065 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546067 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546083 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546085 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546086 7f869af78700 20 osd.26 243 _dispatch 0x305f5600 pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.546093 7f869af78700 5 -- op tracker -- , seq: 1918, time: 2014-06-09 16:22:11.263918, event: header_read, op: pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546101 7f869af78700 5 -- op tracker -- , seq: 1918, time: 2014-06-09 16:22:11.263918, event: throttled, op: pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546108 7f869af78700 5 -- op tracker -- , seq: 1918, time: 2014-06-09 16:22:11.263937, event: all_read, op: pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546115 7f869af78700 5 -- op tracker -- , seq: 1918, time: 2014-06-09 16:22:11.546083, event: dispatched, op: pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546123 7f869af78700 5 -- op tracker -- , seq: 1918, time: 2014-06-09 16:22:11.546120, event: waiting_for_osdmap, op: pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546128 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.546132 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f5600 2014-06-09 16:22:11.546137 7f869af78700 5 -- op tracker -- , seq: 1918, time: 2014-06-09 16:22:11.546135, event: started, op: pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546150 7f869af78700 5 -- op tracker -- , seq: 1918, time: 2014-06-09 16:22:11.546147, event: done, op: pg_log(1.517 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546157 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546159 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546172 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546173 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546174 7f869af78700 20 osd.26 243 _dispatch 0x308a6600 pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.546181 7f869af78700 5 -- op tracker -- , seq: 1919, time: 2014-06-09 16:22:11.300583, event: header_read, op: pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546188 7f869af78700 5 -- op tracker -- , seq: 1919, time: 2014-06-09 16:22:11.300584, event: throttled, op: pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546196 7f869af78700 5 -- op tracker -- , seq: 1919, time: 2014-06-09 16:22:11.300600, event: all_read, op: pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546203 7f869af78700 5 -- op tracker -- , seq: 1919, time: 2014-06-09 16:22:11.546171, event: dispatched, op: pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546211 7f869af78700 5 -- op tracker -- , seq: 1919, time: 2014-06-09 16:22:11.546208, event: waiting_for_osdmap, op: pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546216 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.546220 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a6600 2014-06-09 16:22:11.546226 7f869af78700 5 -- op tracker -- , seq: 1919, time: 2014-06-09 16:22:11.546223, event: started, op: pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546239 7f869af78700 5 -- op tracker -- , seq: 1919, time: 2014-06-09 16:22:11.546236, event: done, op: pg_log(0.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546250 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546252 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546265 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546267 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546268 7f869af78700 20 osd.26 243 _dispatch 0x3044dc00 pg_notify(0.ef(21),1.f2(21),2.3c7(27) epoch 243) v5 2014-06-09 16:22:11.546275 7f869af78700 5 -- op tracker -- , seq: 1920, time: 2014-06-09 16:22:11.190006, event: header_read, op: pg_notify(0.ef(21),1.f2(21),2.3c7(27) epoch 243) 2014-06-09 16:22:11.546283 7f869af78700 5 -- op tracker -- , seq: 1920, time: 2014-06-09 16:22:11.190006, event: throttled, op: pg_notify(0.ef(21),1.f2(21),2.3c7(27) epoch 243) 2014-06-09 16:22:11.546289 7f869af78700 5 -- op tracker -- , seq: 1920, time: 2014-06-09 16:22:11.190050, event: all_read, op: pg_notify(0.ef(21),1.f2(21),2.3c7(27) epoch 243) 2014-06-09 16:22:11.546296 7f869af78700 5 -- op tracker -- , seq: 1920, time: 2014-06-09 16:22:11.546265, event: dispatched, op: pg_notify(0.ef(21),1.f2(21),2.3c7(27) epoch 243) 2014-06-09 16:22:11.546304 7f869af78700 5 -- op tracker -- , seq: 1920, time: 2014-06-09 16:22:11.546301, event: waiting_for_osdmap, op: pg_notify(0.ef(21),1.f2(21),2.3c7(27) epoch 243) 2014-06-09 16:22:11.546308 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.546310 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044dc00 2014-06-09 16:22:11.546316 7f869af78700 5 -- op tracker -- , seq: 1920, time: 2014-06-09 16:22:11.546313, event: started, op: pg_notify(0.ef(21),1.f2(21),2.3c7(27) epoch 243) 2014-06-09 16:22:11.546347 7f869af78700 5 -- op tracker -- , seq: 1920, time: 2014-06-09 16:22:11.546344, event: done, op: pg_notify(0.ef(21),1.f2(21),2.3c7(27) epoch 243) 2014-06-09 16:22:11.546359 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546361 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546393 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546395 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546396 7f869af78700 20 osd.26 243 _dispatch 0x306dca00 pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.546402 7f869af78700 5 -- op tracker -- , seq: 1921, time: 2014-06-09 16:22:11.306296, event: header_read, op: pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546410 7f869af78700 5 -- op tracker -- , seq: 1921, time: 2014-06-09 16:22:11.306296, event: throttled, op: pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546418 7f869af78700 5 -- op tracker -- , seq: 1921, time: 2014-06-09 16:22:11.306308, event: all_read, op: pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546424 7f869af78700 5 -- op tracker -- , seq: 1921, time: 2014-06-09 16:22:11.546392, event: dispatched, op: pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546432 7f869af78700 5 -- op tracker -- , seq: 1921, time: 2014-06-09 16:22:11.546429, event: waiting_for_osdmap, op: pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546437 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.546441 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306dca00 2014-06-09 16:22:11.546447 7f869af78700 5 -- op tracker -- , seq: 1921, time: 2014-06-09 16:22:11.546444, event: started, op: pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546460 7f869af78700 5 -- op tracker -- , seq: 1921, time: 2014-06-09 16:22:11.546457, event: done, op: pg_log(2.654 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546468 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546469 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546482 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546483 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546484 7f869af78700 20 osd.26 243 _dispatch 0x305b5600 pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.546491 7f869af78700 5 -- op tracker -- , seq: 1922, time: 2014-06-09 16:22:11.113787, event: header_read, op: pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546498 7f869af78700 5 -- op tracker -- , seq: 1922, time: 2014-06-09 16:22:11.113787, event: throttled, op: pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546505 7f869af78700 5 -- op tracker -- , seq: 1922, time: 2014-06-09 16:22:11.113803, event: all_read, op: pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546511 7f869af78700 5 -- op tracker -- , seq: 1922, time: 2014-06-09 16:22:11.546481, event: dispatched, op: pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546519 7f869af78700 5 -- op tracker -- , seq: 1922, time: 2014-06-09 16:22:11.546516, event: waiting_for_osdmap, op: pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546523 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.546527 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b5600 2014-06-09 16:22:11.546533 7f869af78700 5 -- op tracker -- , seq: 1922, time: 2014-06-09 16:22:11.546530, event: started, op: pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546545 7f869af78700 5 -- op tracker -- , seq: 1922, time: 2014-06-09 16:22:11.546542, event: done, op: pg_log(1.508 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.546552 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546554 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546566 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546568 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546569 7f869af78700 20 osd.26 243 _dispatch 0x2fef5340 pg_notify(2.383(20) epoch 243) v5 2014-06-09 16:22:11.546574 7f869af78700 5 -- op tracker -- , seq: 1923, time: 2014-06-09 16:22:11.103880, event: header_read, op: pg_notify(2.383(20) epoch 243) 2014-06-09 16:22:11.546582 7f869af78700 5 -- op tracker -- , seq: 1923, time: 2014-06-09 16:22:11.103881, event: throttled, op: pg_notify(2.383(20) epoch 243) 2014-06-09 16:22:11.546588 7f869af78700 5 -- op tracker -- , seq: 1923, time: 2014-06-09 16:22:11.103917, event: all_read, op: pg_notify(2.383(20) epoch 243) 2014-06-09 16:22:11.546595 7f869af78700 5 -- op tracker -- , seq: 1923, time: 2014-06-09 16:22:11.546566, event: dispatched, op: pg_notify(2.383(20) epoch 243) 2014-06-09 16:22:11.546602 7f869af78700 5 -- op tracker -- , seq: 1923, time: 2014-06-09 16:22:11.546600, event: waiting_for_osdmap, op: pg_notify(2.383(20) epoch 243) 2014-06-09 16:22:11.546606 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.546608 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef5340 2014-06-09 16:22:11.546613 7f869af78700 5 -- op tracker -- , seq: 1923, time: 2014-06-09 16:22:11.546611, event: started, op: pg_notify(2.383(20) epoch 243) 2014-06-09 16:22:11.546628 7f869af78700 5 -- op tracker -- , seq: 1923, time: 2014-06-09 16:22:11.546625, event: done, op: pg_notify(2.383(20) epoch 243) 2014-06-09 16:22:11.546635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546637 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546656 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546657 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546659 7f869af78700 20 osd.26 243 _dispatch 0x1b0ce8c0 pg_notify(2.744(20) epoch 243) v5 2014-06-09 16:22:11.546664 7f869af78700 5 -- op tracker -- , seq: 1924, time: 2014-06-09 16:22:11.317214, event: header_read, op: pg_notify(2.744(20) epoch 243) 2014-06-09 16:22:11.546671 7f869af78700 5 -- op tracker -- , seq: 1924, time: 2014-06-09 16:22:11.317214, event: throttled, op: pg_notify(2.744(20) epoch 243) 2014-06-09 16:22:11.546677 7f869af78700 5 -- op tracker -- , seq: 1924, time: 2014-06-09 16:22:11.317253, event: all_read, op: pg_notify(2.744(20) epoch 243) 2014-06-09 16:22:11.546683 7f869af78700 5 -- op tracker -- , seq: 1924, time: 2014-06-09 16:22:11.546655, event: dispatched, op: pg_notify(2.744(20) epoch 243) 2014-06-09 16:22:11.546690 7f869af78700 5 -- op tracker -- , seq: 1924, time: 2014-06-09 16:22:11.546688, event: waiting_for_osdmap, op: pg_notify(2.744(20) epoch 243) 2014-06-09 16:22:11.546694 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.546696 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1b0ce8c0 2014-06-09 16:22:11.546712 7f869af78700 5 -- op tracker -- , seq: 1924, time: 2014-06-09 16:22:11.546710, event: started, op: pg_notify(2.744(20) epoch 243) 2014-06-09 16:22:11.546729 7f869af78700 5 -- op tracker -- , seq: 1924, time: 2014-06-09 16:22:11.546727, event: done, op: pg_notify(2.744(20) epoch 243) 2014-06-09 16:22:11.546739 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546740 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546758 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546760 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546761 7f869af78700 20 osd.26 243 _dispatch 0x3005a000 pg_info(1 pgs e243:3.52a(2)) v4 2014-06-09 16:22:11.546766 7f869af78700 5 -- op tracker -- , seq: 1925, time: 2014-06-09 16:22:11.281527, event: header_read, op: pg_info(1 pgs e243:3.52a(2)) 2014-06-09 16:22:11.546773 7f869af78700 5 -- op tracker -- , seq: 1925, time: 2014-06-09 16:22:11.281527, event: throttled, op: pg_info(1 pgs e243:3.52a(2)) 2014-06-09 16:22:11.546779 7f869af78700 5 -- op tracker -- , seq: 1925, time: 2014-06-09 16:22:11.281550, event: all_read, op: pg_info(1 pgs e243:3.52a(2)) 2014-06-09 16:22:11.546786 7f869af78700 5 -- op tracker -- , seq: 1925, time: 2014-06-09 16:22:11.546758, event: dispatched, op: pg_info(1 pgs e243:3.52a(2)) 2014-06-09 16:22:11.546793 7f869af78700 5 -- op tracker -- , seq: 1925, time: 2014-06-09 16:22:11.546791, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.52a(2)) 2014-06-09 16:22:11.546798 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.52a(2)) v4 from osd.3 2014-06-09 16:22:11.546800 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005a000 2014-06-09 16:22:11.546805 7f869af78700 5 -- op tracker -- , seq: 1925, time: 2014-06-09 16:22:11.546803, event: started, op: pg_info(1 pgs e243:3.52a(2)) 2014-06-09 16:22:11.546822 7f869af78700 5 -- op tracker -- , seq: 1925, time: 2014-06-09 16:22:11.546820, event: done, op: pg_info(1 pgs e243:3.52a(2)) 2014-06-09 16:22:11.546830 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546832 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546845 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546847 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546849 7f869af78700 20 osd.26 243 _dispatch 0x1405d6c0 pg_notify(3.1aa(12) epoch 243) v5 2014-06-09 16:22:11.546854 7f869af78700 5 -- op tracker -- , seq: 1926, time: 2014-06-09 16:22:11.191409, event: header_read, op: pg_notify(3.1aa(12) epoch 243) 2014-06-09 16:22:11.546861 7f869af78700 5 -- op tracker -- , seq: 1926, time: 2014-06-09 16:22:11.191409, event: throttled, op: pg_notify(3.1aa(12) epoch 243) 2014-06-09 16:22:11.546867 7f869af78700 5 -- op tracker -- , seq: 1926, time: 2014-06-09 16:22:11.191422, event: all_read, op: pg_notify(3.1aa(12) epoch 243) 2014-06-09 16:22:11.546873 7f869af78700 5 -- op tracker -- , seq: 1926, time: 2014-06-09 16:22:11.546845, event: dispatched, op: pg_notify(3.1aa(12) epoch 243) 2014-06-09 16:22:11.546880 7f869af78700 5 -- op tracker -- , seq: 1926, time: 2014-06-09 16:22:11.546878, event: waiting_for_osdmap, op: pg_notify(3.1aa(12) epoch 243) 2014-06-09 16:22:11.546884 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.546886 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405d6c0 2014-06-09 16:22:11.546891 7f869af78700 5 -- op tracker -- , seq: 1926, time: 2014-06-09 16:22:11.546889, event: started, op: pg_notify(3.1aa(12) epoch 243) 2014-06-09 16:22:11.546908 7f869af78700 5 -- op tracker -- , seq: 1926, time: 2014-06-09 16:22:11.546906, event: done, op: pg_notify(3.1aa(12) epoch 243) 2014-06-09 16:22:11.546916 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546918 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546934 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.546936 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.546937 7f869af78700 20 osd.26 243 _dispatch 0x22ac0000 pg_notify(3.44d(10) epoch 243) v5 2014-06-09 16:22:11.546942 7f869af78700 5 -- op tracker -- , seq: 1927, time: 2014-06-09 16:22:11.290348, event: header_read, op: pg_notify(3.44d(10) epoch 243) 2014-06-09 16:22:11.546949 7f869af78700 5 -- op tracker -- , seq: 1927, time: 2014-06-09 16:22:11.290349, event: throttled, op: pg_notify(3.44d(10) epoch 243) 2014-06-09 16:22:11.546965 7f869af78700 5 -- op tracker -- , seq: 1927, time: 2014-06-09 16:22:11.290374, event: all_read, op: pg_notify(3.44d(10) epoch 243) 2014-06-09 16:22:11.546972 7f869af78700 5 -- op tracker -- , seq: 1927, time: 2014-06-09 16:22:11.546934, event: dispatched, op: pg_notify(3.44d(10) epoch 243) 2014-06-09 16:22:11.546979 7f869af78700 5 -- op tracker -- , seq: 1927, time: 2014-06-09 16:22:11.546977, event: waiting_for_osdmap, op: pg_notify(3.44d(10) epoch 243) 2014-06-09 16:22:11.546983 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.546985 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22ac0000 2014-06-09 16:22:11.546990 7f869af78700 5 -- op tracker -- , seq: 1927, time: 2014-06-09 16:22:11.546988, event: started, op: pg_notify(3.44d(10) epoch 243) 2014-06-09 16:22:11.547006 7f869af78700 5 -- op tracker -- , seq: 1927, time: 2014-06-09 16:22:11.547004, event: done, op: pg_notify(3.44d(10) epoch 243) 2014-06-09 16:22:11.547013 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547015 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547033 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547035 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547036 7f869af78700 20 osd.26 243 _dispatch 0x308d9180 pg_notify(0.dc(20),0.e2(20),0.678(23) epoch 243) v5 2014-06-09 16:22:11.547043 7f869af78700 5 -- op tracker -- , seq: 1928, time: 2014-06-09 16:22:11.263995, event: header_read, op: pg_notify(0.dc(20),0.e2(20),0.678(23) epoch 243) 2014-06-09 16:22:11.547062 7f869af78700 5 -- op tracker -- , seq: 1928, time: 2014-06-09 16:22:11.263996, event: throttled, op: pg_notify(0.dc(20),0.e2(20),0.678(23) epoch 243) 2014-06-09 16:22:11.547069 7f869af78700 5 -- op tracker -- , seq: 1928, time: 2014-06-09 16:22:11.264052, event: all_read, op: pg_notify(0.dc(20),0.e2(20),0.678(23) epoch 243) 2014-06-09 16:22:11.547081 7f869af78700 5 -- op tracker -- , seq: 1928, time: 2014-06-09 16:22:11.547033, event: dispatched, op: pg_notify(0.dc(20),0.e2(20),0.678(23) epoch 243) 2014-06-09 16:22:11.547090 7f869af78700 5 -- op tracker -- , seq: 1928, time: 2014-06-09 16:22:11.547087, event: waiting_for_osdmap, op: pg_notify(0.dc(20),0.e2(20),0.678(23) epoch 243) 2014-06-09 16:22:11.547094 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.547096 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d9180 2014-06-09 16:22:11.547102 7f869af78700 5 -- op tracker -- , seq: 1928, time: 2014-06-09 16:22:11.547099, event: started, op: pg_notify(0.dc(20),0.e2(20),0.678(23) epoch 243) 2014-06-09 16:22:11.547133 7f869af78700 5 -- op tracker -- , seq: 1928, time: 2014-06-09 16:22:11.547129, event: done, op: pg_notify(0.dc(20),0.e2(20),0.678(23) epoch 243) 2014-06-09 16:22:11.547147 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547150 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547178 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547179 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547180 7f869af78700 20 osd.26 243 _dispatch 0x15999500 pg_notify(2.102(20) epoch 243) v5 2014-06-09 16:22:11.547186 7f869af78700 5 -- op tracker -- , seq: 1929, time: 2014-06-09 16:22:11.300668, event: header_read, op: pg_notify(2.102(20) epoch 243) 2014-06-09 16:22:11.547193 7f869af78700 5 -- op tracker -- , seq: 1929, time: 2014-06-09 16:22:11.300669, event: throttled, op: pg_notify(2.102(20) epoch 243) 2014-06-09 16:22:11.547199 7f869af78700 5 -- op tracker -- , seq: 1929, time: 2014-06-09 16:22:11.300691, event: all_read, op: pg_notify(2.102(20) epoch 243) 2014-06-09 16:22:11.547205 7f869af78700 5 -- op tracker -- , seq: 1929, time: 2014-06-09 16:22:11.547177, event: dispatched, op: pg_notify(2.102(20) epoch 243) 2014-06-09 16:22:11.547212 7f869af78700 5 -- op tracker -- , seq: 1929, time: 2014-06-09 16:22:11.547210, event: waiting_for_osdmap, op: pg_notify(2.102(20) epoch 243) 2014-06-09 16:22:11.547217 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.547219 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15999500 2014-06-09 16:22:11.547223 7f869af78700 5 -- op tracker -- , seq: 1929, time: 2014-06-09 16:22:11.547221, event: started, op: pg_notify(2.102(20) epoch 243) 2014-06-09 16:22:11.547246 7f869af78700 5 -- op tracker -- , seq: 1929, time: 2014-06-09 16:22:11.547244, event: done, op: pg_notify(2.102(20) epoch 243) 2014-06-09 16:22:11.547254 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547256 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547276 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547278 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547279 7f869af78700 20 osd.26 243 _dispatch 0x3005ae00 pg_notify(1.54c(21),2.288(23) epoch 243) v5 2014-06-09 16:22:11.547285 7f869af78700 5 -- op tracker -- , seq: 1930, time: 2014-06-09 16:22:11.195257, event: header_read, op: pg_notify(1.54c(21),2.288(23) epoch 243) 2014-06-09 16:22:11.547292 7f869af78700 5 -- op tracker -- , seq: 1930, time: 2014-06-09 16:22:11.195258, event: throttled, op: pg_notify(1.54c(21),2.288(23) epoch 243) 2014-06-09 16:22:11.547299 7f869af78700 5 -- op tracker -- , seq: 1930, time: 2014-06-09 16:22:11.195286, event: all_read, op: pg_notify(1.54c(21),2.288(23) epoch 243) 2014-06-09 16:22:11.547305 7f869af78700 5 -- op tracker -- , seq: 1930, time: 2014-06-09 16:22:11.547276, event: dispatched, op: pg_notify(1.54c(21),2.288(23) epoch 243) 2014-06-09 16:22:11.547312 7f869af78700 5 -- op tracker -- , seq: 1930, time: 2014-06-09 16:22:11.547310, event: waiting_for_osdmap, op: pg_notify(1.54c(21),2.288(23) epoch 243) 2014-06-09 16:22:11.547316 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.547318 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005ae00 2014-06-09 16:22:11.547323 7f869af78700 5 -- op tracker -- , seq: 1930, time: 2014-06-09 16:22:11.547321, event: started, op: pg_notify(1.54c(21),2.288(23) epoch 243) 2014-06-09 16:22:11.547346 7f869af78700 5 -- op tracker -- , seq: 1930, time: 2014-06-09 16:22:11.547344, event: done, op: pg_notify(1.54c(21),2.288(23) epoch 243) 2014-06-09 16:22:11.547354 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547356 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547382 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547384 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547385 7f869af78700 20 osd.26 243 _dispatch 0x210b1500 pg_notify(1.b8(20),3.380(10) epoch 243) v5 2014-06-09 16:22:11.547391 7f869af78700 5 -- op tracker -- , seq: 1931, time: 2014-06-09 16:22:11.306355, event: header_read, op: pg_notify(1.b8(20),3.380(10) epoch 243) 2014-06-09 16:22:11.547399 7f869af78700 5 -- op tracker -- , seq: 1931, time: 2014-06-09 16:22:11.306356, event: throttled, op: pg_notify(1.b8(20),3.380(10) epoch 243) 2014-06-09 16:22:11.547406 7f869af78700 5 -- op tracker -- , seq: 1931, time: 2014-06-09 16:22:11.306386, event: all_read, op: pg_notify(1.b8(20),3.380(10) epoch 243) 2014-06-09 16:22:11.547412 7f869af78700 5 -- op tracker -- , seq: 1931, time: 2014-06-09 16:22:11.547382, event: dispatched, op: pg_notify(1.b8(20),3.380(10) epoch 243) 2014-06-09 16:22:11.547419 7f869af78700 5 -- op tracker -- , seq: 1931, time: 2014-06-09 16:22:11.547417, event: waiting_for_osdmap, op: pg_notify(1.b8(20),3.380(10) epoch 243) 2014-06-09 16:22:11.547423 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.547425 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x210b1500 2014-06-09 16:22:11.547430 7f869af78700 5 -- op tracker -- , seq: 1931, time: 2014-06-09 16:22:11.547428, event: started, op: pg_notify(1.b8(20),3.380(10) epoch 243) 2014-06-09 16:22:11.547454 7f869af78700 5 -- op tracker -- , seq: 1931, time: 2014-06-09 16:22:11.547452, event: done, op: pg_notify(1.b8(20),3.380(10) epoch 243) 2014-06-09 16:22:11.547463 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547465 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547488 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547490 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547491 7f869af78700 20 osd.26 243 _dispatch 0x3043a700 pg_notify(3.443(12),1.449(21) epoch 243) v5 2014-06-09 16:22:11.547497 7f869af78700 5 -- op tracker -- , seq: 1932, time: 2014-06-09 16:22:11.113895, event: header_read, op: pg_notify(3.443(12),1.449(21) epoch 243) 2014-06-09 16:22:11.547504 7f869af78700 5 -- op tracker -- , seq: 1932, time: 2014-06-09 16:22:11.113896, event: throttled, op: pg_notify(3.443(12),1.449(21) epoch 243) 2014-06-09 16:22:11.547510 7f869af78700 5 -- op tracker -- , seq: 1932, time: 2014-06-09 16:22:11.113945, event: all_read, op: pg_notify(3.443(12),1.449(21) epoch 243) 2014-06-09 16:22:11.547517 7f869af78700 5 -- op tracker -- , seq: 1932, time: 2014-06-09 16:22:11.547488, event: dispatched, op: pg_notify(3.443(12),1.449(21) epoch 243) 2014-06-09 16:22:11.547524 7f869af78700 5 -- op tracker -- , seq: 1932, time: 2014-06-09 16:22:11.547521, event: waiting_for_osdmap, op: pg_notify(3.443(12),1.449(21) epoch 243) 2014-06-09 16:22:11.547528 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.547530 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043a700 2014-06-09 16:22:11.547535 7f869af78700 5 -- op tracker -- , seq: 1932, time: 2014-06-09 16:22:11.547533, event: started, op: pg_notify(3.443(12),1.449(21) epoch 243) 2014-06-09 16:22:11.547560 7f869af78700 5 -- op tracker -- , seq: 1932, time: 2014-06-09 16:22:11.547557, event: done, op: pg_notify(3.443(12),1.449(21) epoch 243) 2014-06-09 16:22:11.547568 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547569 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547600 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547602 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547603 7f869af78700 20 osd.26 243 _dispatch 0x305a7e00 pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.547610 7f869af78700 5 -- op tracker -- , seq: 1933, time: 2014-06-09 16:22:11.103923, event: header_read, op: pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547623 7f869af78700 5 -- op tracker -- , seq: 1933, time: 2014-06-09 16:22:11.103923, event: throttled, op: pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547631 7f869af78700 5 -- op tracker -- , seq: 1933, time: 2014-06-09 16:22:11.103934, event: all_read, op: pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547638 7f869af78700 5 -- op tracker -- , seq: 1933, time: 2014-06-09 16:22:11.547600, event: dispatched, op: pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547646 7f869af78700 5 -- op tracker -- , seq: 1933, time: 2014-06-09 16:22:11.547643, event: waiting_for_osdmap, op: pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547650 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.547654 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a7e00 2014-06-09 16:22:11.547661 7f869af78700 5 -- op tracker -- , seq: 1933, time: 2014-06-09 16:22:11.547658, event: started, op: pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547674 7f869af78700 5 -- op tracker -- , seq: 1933, time: 2014-06-09 16:22:11.547671, event: done, op: pg_log(1.c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547682 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547684 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547695 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547697 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547698 7f869af78700 20 osd.26 243 _dispatch 0x22ac1dc0 pg_notify(1.60e(20) epoch 243) v5 2014-06-09 16:22:11.547703 7f869af78700 5 -- op tracker -- , seq: 1934, time: 2014-06-09 16:22:11.317508, event: header_read, op: pg_notify(1.60e(20) epoch 243) 2014-06-09 16:22:11.547711 7f869af78700 5 -- op tracker -- , seq: 1934, time: 2014-06-09 16:22:11.317509, event: throttled, op: pg_notify(1.60e(20) epoch 243) 2014-06-09 16:22:11.547717 7f869af78700 5 -- op tracker -- , seq: 1934, time: 2014-06-09 16:22:11.317538, event: all_read, op: pg_notify(1.60e(20) epoch 243) 2014-06-09 16:22:11.547723 7f869af78700 5 -- op tracker -- , seq: 1934, time: 2014-06-09 16:22:11.547695, event: dispatched, op: pg_notify(1.60e(20) epoch 243) 2014-06-09 16:22:11.547730 7f869af78700 5 -- op tracker -- , seq: 1934, time: 2014-06-09 16:22:11.547728, event: waiting_for_osdmap, op: pg_notify(1.60e(20) epoch 243) 2014-06-09 16:22:11.547734 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.547737 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22ac1dc0 2014-06-09 16:22:11.547741 7f869af78700 5 -- op tracker -- , seq: 1934, time: 2014-06-09 16:22:11.547739, event: started, op: pg_notify(1.60e(20) epoch 243) 2014-06-09 16:22:11.547757 7f869af78700 5 -- op tracker -- , seq: 1934, time: 2014-06-09 16:22:11.547755, event: done, op: pg_notify(1.60e(20) epoch 243) 2014-06-09 16:22:11.547765 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547767 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547785 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547787 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547788 7f869af78700 20 osd.26 243 _dispatch 0x30613200 pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.547795 7f869af78700 5 -- op tracker -- , seq: 1935, time: 2014-06-09 16:22:11.283900, event: header_read, op: pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547803 7f869af78700 5 -- op tracker -- , seq: 1935, time: 2014-06-09 16:22:11.283900, event: throttled, op: pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547809 7f869af78700 5 -- op tracker -- , seq: 1935, time: 2014-06-09 16:22:11.283910, event: all_read, op: pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547816 7f869af78700 5 -- op tracker -- , seq: 1935, time: 2014-06-09 16:22:11.547785, event: dispatched, op: pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547823 7f869af78700 5 -- op tracker -- , seq: 1935, time: 2014-06-09 16:22:11.547820, event: waiting_for_osdmap, op: pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547828 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.547832 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30613200 2014-06-09 16:22:11.547838 7f869af78700 5 -- op tracker -- , seq: 1935, time: 2014-06-09 16:22:11.547835, event: started, op: pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547851 7f869af78700 5 -- op tracker -- , seq: 1935, time: 2014-06-09 16:22:11.547848, event: done, op: pg_log(2.5b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.547859 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547861 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547873 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547875 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547876 7f869af78700 20 osd.26 243 _dispatch 0x1b0ce380 pg_notify(3.718(12) epoch 243) v5 2014-06-09 16:22:11.547881 7f869af78700 5 -- op tracker -- , seq: 1936, time: 2014-06-09 16:22:11.192276, event: header_read, op: pg_notify(3.718(12) epoch 243) 2014-06-09 16:22:11.547888 7f869af78700 5 -- op tracker -- , seq: 1936, time: 2014-06-09 16:22:11.192277, event: throttled, op: pg_notify(3.718(12) epoch 243) 2014-06-09 16:22:11.547894 7f869af78700 5 -- op tracker -- , seq: 1936, time: 2014-06-09 16:22:11.192297, event: all_read, op: pg_notify(3.718(12) epoch 243) 2014-06-09 16:22:11.547900 7f869af78700 5 -- op tracker -- , seq: 1936, time: 2014-06-09 16:22:11.547873, event: dispatched, op: pg_notify(3.718(12) epoch 243) 2014-06-09 16:22:11.547906 7f869af78700 5 -- op tracker -- , seq: 1936, time: 2014-06-09 16:22:11.547904, event: waiting_for_osdmap, op: pg_notify(3.718(12) epoch 243) 2014-06-09 16:22:11.547911 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.547913 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1b0ce380 2014-06-09 16:22:11.547917 7f869af78700 5 -- op tracker -- , seq: 1936, time: 2014-06-09 16:22:11.547915, event: started, op: pg_notify(3.718(12) epoch 243) 2014-06-09 16:22:11.547934 7f869af78700 5 -- op tracker -- , seq: 1936, time: 2014-06-09 16:22:11.547932, event: done, op: pg_notify(3.718(12) epoch 243) 2014-06-09 16:22:11.547942 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547944 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547962 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.547963 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.547964 7f869af78700 20 osd.26 243 _dispatch 0x307e7c00 pg_notify(3.94(10),1.99(23),2.36c(23) epoch 243) v5 2014-06-09 16:22:11.547971 7f869af78700 5 -- op tracker -- , seq: 1937, time: 2014-06-09 16:22:11.294365, event: header_read, op: pg_notify(3.94(10),1.99(23),2.36c(23) epoch 243) 2014-06-09 16:22:11.547979 7f869af78700 5 -- op tracker -- , seq: 1937, time: 2014-06-09 16:22:11.294366, event: throttled, op: pg_notify(3.94(10),1.99(23),2.36c(23) epoch 243) 2014-06-09 16:22:11.547986 7f869af78700 5 -- op tracker -- , seq: 1937, time: 2014-06-09 16:22:11.294416, event: all_read, op: pg_notify(3.94(10),1.99(23),2.36c(23) epoch 243) 2014-06-09 16:22:11.547993 7f869af78700 5 -- op tracker -- , seq: 1937, time: 2014-06-09 16:22:11.547961, event: dispatched, op: pg_notify(3.94(10),1.99(23),2.36c(23) epoch 243) 2014-06-09 16:22:11.548000 7f869af78700 5 -- op tracker -- , seq: 1937, time: 2014-06-09 16:22:11.547997, event: waiting_for_osdmap, op: pg_notify(3.94(10),1.99(23),2.36c(23) epoch 243) 2014-06-09 16:22:11.548004 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.548006 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e7c00 2014-06-09 16:22:11.548012 7f869af78700 5 -- op tracker -- , seq: 1937, time: 2014-06-09 16:22:11.548009, event: started, op: pg_notify(3.94(10),1.99(23),2.36c(23) epoch 243) 2014-06-09 16:22:11.548045 7f869af78700 5 -- op tracker -- , seq: 1937, time: 2014-06-09 16:22:11.548041, event: done, op: pg_notify(3.94(10),1.99(23),2.36c(23) epoch 243) 2014-06-09 16:22:11.548053 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548055 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548081 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548083 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548084 7f869af78700 20 osd.26 243 _dispatch 0x308d8540 pg_info(1 pgs e243:3.3a8(2)) v4 2014-06-09 16:22:11.548090 7f869af78700 5 -- op tracker -- , seq: 1938, time: 2014-06-09 16:22:11.264065, event: header_read, op: pg_info(1 pgs e243:3.3a8(2)) 2014-06-09 16:22:11.548097 7f869af78700 5 -- op tracker -- , seq: 1938, time: 2014-06-09 16:22:11.264065, event: throttled, op: pg_info(1 pgs e243:3.3a8(2)) 2014-06-09 16:22:11.548103 7f869af78700 5 -- op tracker -- , seq: 1938, time: 2014-06-09 16:22:11.264114, event: all_read, op: pg_info(1 pgs e243:3.3a8(2)) 2014-06-09 16:22:11.548109 7f869af78700 5 -- op tracker -- , seq: 1938, time: 2014-06-09 16:22:11.548081, event: dispatched, op: pg_info(1 pgs e243:3.3a8(2)) 2014-06-09 16:22:11.548116 7f869af78700 5 -- op tracker -- , seq: 1938, time: 2014-06-09 16:22:11.548114, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a8(2)) 2014-06-09 16:22:11.548121 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a8(2)) v4 from osd.8 2014-06-09 16:22:11.548124 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d8540 2014-06-09 16:22:11.548129 7f869af78700 5 -- op tracker -- , seq: 1938, time: 2014-06-09 16:22:11.548127, event: started, op: pg_info(1 pgs e243:3.3a8(2)) 2014-06-09 16:22:11.548145 7f869af78700 5 -- op tracker -- , seq: 1938, time: 2014-06-09 16:22:11.548142, event: done, op: pg_info(1 pgs e243:3.3a8(2)) 2014-06-09 16:22:11.548152 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548155 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548168 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548170 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548171 7f869af78700 20 osd.26 243 _dispatch 0x306e8a00 pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.548178 7f869af78700 5 -- op tracker -- , seq: 1939, time: 2014-06-09 16:22:11.304067, event: header_read, op: pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548186 7f869af78700 5 -- op tracker -- , seq: 1939, time: 2014-06-09 16:22:11.304067, event: throttled, op: pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548193 7f869af78700 5 -- op tracker -- , seq: 1939, time: 2014-06-09 16:22:11.304083, event: all_read, op: pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548200 7f869af78700 5 -- op tracker -- , seq: 1939, time: 2014-06-09 16:22:11.548168, event: dispatched, op: pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548209 7f869af78700 5 -- op tracker -- , seq: 1939, time: 2014-06-09 16:22:11.548206, event: waiting_for_osdmap, op: pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548214 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.548218 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306e8a00 2014-06-09 16:22:11.548223 7f869af78700 5 -- op tracker -- , seq: 1939, time: 2014-06-09 16:22:11.548221, event: started, op: pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548236 7f869af78700 5 -- op tracker -- , seq: 1939, time: 2014-06-09 16:22:11.548233, event: done, op: pg_log(2.5f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548243 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548245 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548256 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548258 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548259 7f869af78700 20 osd.26 243 _dispatch 0x15a281c0 pg_notify(0.66b(21) epoch 243) v5 2014-06-09 16:22:11.548264 7f869af78700 5 -- op tracker -- , seq: 1940, time: 2014-06-09 16:22:11.195299, event: header_read, op: pg_notify(0.66b(21) epoch 243) 2014-06-09 16:22:11.548271 7f869af78700 5 -- op tracker -- , seq: 1940, time: 2014-06-09 16:22:11.195299, event: throttled, op: pg_notify(0.66b(21) epoch 243) 2014-06-09 16:22:11.548278 7f869af78700 5 -- op tracker -- , seq: 1940, time: 2014-06-09 16:22:11.195316, event: all_read, op: pg_notify(0.66b(21) epoch 243) 2014-06-09 16:22:11.548284 7f869af78700 5 -- op tracker -- , seq: 1940, time: 2014-06-09 16:22:11.548256, event: dispatched, op: pg_notify(0.66b(21) epoch 243) 2014-06-09 16:22:11.548292 7f869af78700 5 -- op tracker -- , seq: 1940, time: 2014-06-09 16:22:11.548289, event: waiting_for_osdmap, op: pg_notify(0.66b(21) epoch 243) 2014-06-09 16:22:11.548296 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.548298 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a281c0 2014-06-09 16:22:11.548302 7f869af78700 5 -- op tracker -- , seq: 1940, time: 2014-06-09 16:22:11.548301, event: started, op: pg_notify(0.66b(21) epoch 243) 2014-06-09 16:22:11.548317 7f869af78700 5 -- op tracker -- , seq: 1940, time: 2014-06-09 16:22:11.548315, event: done, op: pg_notify(0.66b(21) epoch 243) 2014-06-09 16:22:11.548326 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548328 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548344 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548346 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548347 7f869af78700 20 osd.26 243 _dispatch 0x3070cc00 pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.548354 7f869af78700 5 -- op tracker -- , seq: 1941, time: 2014-06-09 16:22:11.323442, event: header_read, op: pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548362 7f869af78700 5 -- op tracker -- , seq: 1941, time: 2014-06-09 16:22:11.323442, event: throttled, op: pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548368 7f869af78700 5 -- op tracker -- , seq: 1941, time: 2014-06-09 16:22:11.323457, event: all_read, op: pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548376 7f869af78700 5 -- op tracker -- , seq: 1941, time: 2014-06-09 16:22:11.548344, event: dispatched, op: pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548383 7f869af78700 5 -- op tracker -- , seq: 1941, time: 2014-06-09 16:22:11.548380, event: waiting_for_osdmap, op: pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548387 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.548391 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070cc00 2014-06-09 16:22:11.548397 7f869af78700 5 -- op tracker -- , seq: 1941, time: 2014-06-09 16:22:11.548394, event: started, op: pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548410 7f869af78700 5 -- op tracker -- , seq: 1941, time: 2014-06-09 16:22:11.548407, event: done, op: pg_log(2.2f2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548418 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548432 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548434 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548436 7f869af78700 20 osd.26 243 _dispatch 0x30646400 pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.548442 7f869af78700 5 -- op tracker -- , seq: 1942, time: 2014-06-09 16:22:11.122655, event: header_read, op: pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548449 7f869af78700 5 -- op tracker -- , seq: 1942, time: 2014-06-09 16:22:11.122655, event: throttled, op: pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548456 7f869af78700 5 -- op tracker -- , seq: 1942, time: 2014-06-09 16:22:11.122666, event: all_read, op: pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548464 7f869af78700 5 -- op tracker -- , seq: 1942, time: 2014-06-09 16:22:11.548432, event: dispatched, op: pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548471 7f869af78700 5 -- op tracker -- , seq: 1942, time: 2014-06-09 16:22:11.548469, event: waiting_for_osdmap, op: pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548476 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.548480 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30646400 2014-06-09 16:22:11.548486 7f869af78700 5 -- op tracker -- , seq: 1942, time: 2014-06-09 16:22:11.548483, event: started, op: pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548499 7f869af78700 5 -- op tracker -- , seq: 1942, time: 2014-06-09 16:22:11.548496, event: done, op: pg_log(1.2c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.548507 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548509 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548521 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548523 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548524 7f869af78700 20 osd.26 243 _dispatch 0x1609dc00 pg_notify(3.65d(10) epoch 243) v5 2014-06-09 16:22:11.548529 7f869af78700 5 -- op tracker -- , seq: 1943, time: 2014-06-09 16:22:11.104433, event: header_read, op: pg_notify(3.65d(10) epoch 243) 2014-06-09 16:22:11.548537 7f869af78700 5 -- op tracker -- , seq: 1943, time: 2014-06-09 16:22:11.104434, event: throttled, op: pg_notify(3.65d(10) epoch 243) 2014-06-09 16:22:11.548543 7f869af78700 5 -- op tracker -- , seq: 1943, time: 2014-06-09 16:22:11.104457, event: all_read, op: pg_notify(3.65d(10) epoch 243) 2014-06-09 16:22:11.548549 7f869af78700 5 -- op tracker -- , seq: 1943, time: 2014-06-09 16:22:11.548521, event: dispatched, op: pg_notify(3.65d(10) epoch 243) 2014-06-09 16:22:11.548556 7f869af78700 5 -- op tracker -- , seq: 1943, time: 2014-06-09 16:22:11.548554, event: waiting_for_osdmap, op: pg_notify(3.65d(10) epoch 243) 2014-06-09 16:22:11.548561 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.548563 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1609dc00 2014-06-09 16:22:11.548568 7f869af78700 5 -- op tracker -- , seq: 1943, time: 2014-06-09 16:22:11.548566, event: started, op: pg_notify(3.65d(10) epoch 243) 2014-06-09 16:22:11.548584 7f869af78700 5 -- op tracker -- , seq: 1943, time: 2014-06-09 16:22:11.548582, event: done, op: pg_notify(3.65d(10) epoch 243) 2014-06-09 16:22:11.548594 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548596 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548610 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548612 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548613 7f869af78700 20 osd.26 243 _dispatch 0x2fe70e00 pg_notify(1.2fb(20) epoch 243) v5 2014-06-09 16:22:11.548618 7f869af78700 5 -- op tracker -- , seq: 1944, time: 2014-06-09 16:22:11.331821, event: header_read, op: pg_notify(1.2fb(20) epoch 243) 2014-06-09 16:22:11.548625 7f869af78700 5 -- op tracker -- , seq: 1944, time: 2014-06-09 16:22:11.331821, event: throttled, op: pg_notify(1.2fb(20) epoch 243) 2014-06-09 16:22:11.548631 7f869af78700 5 -- op tracker -- , seq: 1944, time: 2014-06-09 16:22:11.331850, event: all_read, op: pg_notify(1.2fb(20) epoch 243) 2014-06-09 16:22:11.548638 7f869af78700 5 -- op tracker -- , seq: 1944, time: 2014-06-09 16:22:11.548610, event: dispatched, op: pg_notify(1.2fb(20) epoch 243) 2014-06-09 16:22:11.548645 7f869af78700 5 -- op tracker -- , seq: 1944, time: 2014-06-09 16:22:11.548643, event: waiting_for_osdmap, op: pg_notify(1.2fb(20) epoch 243) 2014-06-09 16:22:11.548649 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.548651 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe70e00 2014-06-09 16:22:11.548656 7f869af78700 5 -- op tracker -- , seq: 1944, time: 2014-06-09 16:22:11.548654, event: started, op: pg_notify(1.2fb(20) epoch 243) 2014-06-09 16:22:11.548672 7f869af78700 5 -- op tracker -- , seq: 1944, time: 2014-06-09 16:22:11.548670, event: done, op: pg_notify(1.2fb(20) epoch 243) 2014-06-09 16:22:11.548681 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548683 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548700 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548701 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548702 7f869af78700 20 osd.26 243 _dispatch 0x210b0fc0 pg_info(1 pgs e243:3.5c6(2)) v4 2014-06-09 16:22:11.548707 7f869af78700 5 -- op tracker -- , seq: 1945, time: 2014-06-09 16:22:11.291711, event: header_read, op: pg_info(1 pgs e243:3.5c6(2)) 2014-06-09 16:22:11.548715 7f869af78700 5 -- op tracker -- , seq: 1945, time: 2014-06-09 16:22:11.291711, event: throttled, op: pg_info(1 pgs e243:3.5c6(2)) 2014-06-09 16:22:11.548721 7f869af78700 5 -- op tracker -- , seq: 1945, time: 2014-06-09 16:22:11.291722, event: all_read, op: pg_info(1 pgs e243:3.5c6(2)) 2014-06-09 16:22:11.548727 7f869af78700 5 -- op tracker -- , seq: 1945, time: 2014-06-09 16:22:11.548699, event: dispatched, op: pg_info(1 pgs e243:3.5c6(2)) 2014-06-09 16:22:11.548733 7f869af78700 5 -- op tracker -- , seq: 1945, time: 2014-06-09 16:22:11.548731, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5c6(2)) 2014-06-09 16:22:11.548738 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5c6(2)) v4 from osd.3 2014-06-09 16:22:11.548741 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x210b0fc0 2014-06-09 16:22:11.548746 7f869af78700 5 -- op tracker -- , seq: 1945, time: 2014-06-09 16:22:11.548744, event: started, op: pg_info(1 pgs e243:3.5c6(2)) 2014-06-09 16:22:11.548763 7f869af78700 5 -- op tracker -- , seq: 1945, time: 2014-06-09 16:22:11.548760, event: done, op: pg_info(1 pgs e243:3.5c6(2)) 2014-06-09 16:22:11.548771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548773 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548787 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548789 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548790 7f869af78700 20 osd.26 243 _dispatch 0x19eb9a40 pg_notify(1.482(21),1.757(21) epoch 243) v5 2014-06-09 16:22:11.548796 7f869af78700 5 -- op tracker -- , seq: 1946, time: 2014-06-09 16:22:11.208167, event: header_read, op: pg_notify(1.482(21),1.757(21) epoch 243) 2014-06-09 16:22:11.548803 7f869af78700 5 -- op tracker -- , seq: 1946, time: 2014-06-09 16:22:11.208167, event: throttled, op: pg_notify(1.482(21),1.757(21) epoch 243) 2014-06-09 16:22:11.548810 7f869af78700 5 -- op tracker -- , seq: 1946, time: 2014-06-09 16:22:11.208202, event: all_read, op: pg_notify(1.482(21),1.757(21) epoch 243) 2014-06-09 16:22:11.548816 7f869af78700 5 -- op tracker -- , seq: 1946, time: 2014-06-09 16:22:11.548787, event: dispatched, op: pg_notify(1.482(21),1.757(21) epoch 243) 2014-06-09 16:22:11.548824 7f869af78700 5 -- op tracker -- , seq: 1946, time: 2014-06-09 16:22:11.548822, event: waiting_for_osdmap, op: pg_notify(1.482(21),1.757(21) epoch 243) 2014-06-09 16:22:11.548828 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.548830 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb9a40 2014-06-09 16:22:11.548835 7f869af78700 5 -- op tracker -- , seq: 1946, time: 2014-06-09 16:22:11.548833, event: started, op: pg_notify(1.482(21),1.757(21) epoch 243) 2014-06-09 16:22:11.548859 7f869af78700 5 -- op tracker -- , seq: 1946, time: 2014-06-09 16:22:11.548857, event: done, op: pg_notify(1.482(21),1.757(21) epoch 243) 2014-06-09 16:22:11.548867 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548870 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548892 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548894 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548895 7f869af78700 20 osd.26 243 _dispatch 0x19eb8fc0 pg_info(2 pgs e243:3.35c(2),3.98(2)) v4 2014-06-09 16:22:11.548901 7f869af78700 5 -- op tracker -- , seq: 1947, time: 2014-06-09 16:22:11.294428, event: header_read, op: pg_info(2 pgs e243:3.35c(2),3.98(2)) 2014-06-09 16:22:11.548909 7f869af78700 5 -- op tracker -- , seq: 1947, time: 2014-06-09 16:22:11.294428, event: throttled, op: pg_info(2 pgs e243:3.35c(2),3.98(2)) 2014-06-09 16:22:11.548915 7f869af78700 5 -- op tracker -- , seq: 1947, time: 2014-06-09 16:22:11.294442, event: all_read, op: pg_info(2 pgs e243:3.35c(2),3.98(2)) 2014-06-09 16:22:11.548922 7f869af78700 5 -- op tracker -- , seq: 1947, time: 2014-06-09 16:22:11.548892, event: dispatched, op: pg_info(2 pgs e243:3.35c(2),3.98(2)) 2014-06-09 16:22:11.548930 7f869af78700 5 -- op tracker -- , seq: 1947, time: 2014-06-09 16:22:11.548927, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.35c(2),3.98(2)) 2014-06-09 16:22:11.548934 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.35c(2),3.98(2)) v4 from osd.9 2014-06-09 16:22:11.548937 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb8fc0 2014-06-09 16:22:11.548943 7f869af78700 5 -- op tracker -- , seq: 1947, time: 2014-06-09 16:22:11.548940, event: started, op: pg_info(2 pgs e243:3.35c(2),3.98(2)) 2014-06-09 16:22:11.548965 7f869af78700 5 -- op tracker -- , seq: 1947, time: 2014-06-09 16:22:11.548963, event: done, op: pg_info(2 pgs e243:3.35c(2),3.98(2)) 2014-06-09 16:22:11.548973 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548975 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548989 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.548990 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.548992 7f869af78700 20 osd.26 243 _dispatch 0x305f3200 pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.548998 7f869af78700 5 -- op tracker -- , seq: 1948, time: 2014-06-09 16:22:11.264146, event: header_read, op: pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549006 7f869af78700 5 -- op tracker -- , seq: 1948, time: 2014-06-09 16:22:11.264147, event: throttled, op: pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549014 7f869af78700 5 -- op tracker -- , seq: 1948, time: 2014-06-09 16:22:11.264161, event: all_read, op: pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549021 7f869af78700 5 -- op tracker -- , seq: 1948, time: 2014-06-09 16:22:11.548988, event: dispatched, op: pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549028 7f869af78700 5 -- op tracker -- , seq: 1948, time: 2014-06-09 16:22:11.549025, event: waiting_for_osdmap, op: pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549032 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.549036 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f3200 2014-06-09 16:22:11.549042 7f869af78700 5 -- op tracker -- , seq: 1948, time: 2014-06-09 16:22:11.549039, event: started, op: pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549055 7f869af78700 5 -- op tracker -- , seq: 1948, time: 2014-06-09 16:22:11.549052, event: done, op: pg_log(2.24d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549063 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549065 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549076 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549079 7f869af78700 20 osd.26 243 _dispatch 0x3088c700 pg_notify(1.125(20) epoch 243) v5 2014-06-09 16:22:11.549084 7f869af78700 5 -- op tracker -- , seq: 1949, time: 2014-06-09 16:22:11.304150, event: header_read, op: pg_notify(1.125(20) epoch 243) 2014-06-09 16:22:11.549091 7f869af78700 5 -- op tracker -- , seq: 1949, time: 2014-06-09 16:22:11.304151, event: throttled, op: pg_notify(1.125(20) epoch 243) 2014-06-09 16:22:11.549098 7f869af78700 5 -- op tracker -- , seq: 1949, time: 2014-06-09 16:22:11.304180, event: all_read, op: pg_notify(1.125(20) epoch 243) 2014-06-09 16:22:11.549104 7f869af78700 5 -- op tracker -- , seq: 1949, time: 2014-06-09 16:22:11.549076, event: dispatched, op: pg_notify(1.125(20) epoch 243) 2014-06-09 16:22:11.549110 7f869af78700 5 -- op tracker -- , seq: 1949, time: 2014-06-09 16:22:11.549108, event: waiting_for_osdmap, op: pg_notify(1.125(20) epoch 243) 2014-06-09 16:22:11.549115 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.549117 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088c700 2014-06-09 16:22:11.549122 7f869af78700 5 -- op tracker -- , seq: 1949, time: 2014-06-09 16:22:11.549120, event: started, op: pg_notify(1.125(20) epoch 243) 2014-06-09 16:22:11.549138 7f869af78700 5 -- op tracker -- , seq: 1949, time: 2014-06-09 16:22:11.549136, event: done, op: pg_notify(1.125(20) epoch 243) 2014-06-09 16:22:11.549145 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549165 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549167 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549168 7f869af78700 20 osd.26 243 _dispatch 0x159981c0 pg_notify(0.479(21) epoch 243) v5 2014-06-09 16:22:11.549173 7f869af78700 5 -- op tracker -- , seq: 1950, time: 2014-06-09 16:22:11.195353, event: header_read, op: pg_notify(0.479(21) epoch 243) 2014-06-09 16:22:11.549181 7f869af78700 5 -- op tracker -- , seq: 1950, time: 2014-06-09 16:22:11.195353, event: throttled, op: pg_notify(0.479(21) epoch 243) 2014-06-09 16:22:11.549187 7f869af78700 5 -- op tracker -- , seq: 1950, time: 2014-06-09 16:22:11.195370, event: all_read, op: pg_notify(0.479(21) epoch 243) 2014-06-09 16:22:11.549193 7f869af78700 5 -- op tracker -- , seq: 1950, time: 2014-06-09 16:22:11.549165, event: dispatched, op: pg_notify(0.479(21) epoch 243) 2014-06-09 16:22:11.549199 7f869af78700 5 -- op tracker -- , seq: 1950, time: 2014-06-09 16:22:11.549197, event: waiting_for_osdmap, op: pg_notify(0.479(21) epoch 243) 2014-06-09 16:22:11.549204 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.549206 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x159981c0 2014-06-09 16:22:11.549211 7f869af78700 5 -- op tracker -- , seq: 1950, time: 2014-06-09 16:22:11.549209, event: started, op: pg_notify(0.479(21) epoch 243) 2014-06-09 16:22:11.549227 7f869af78700 5 -- op tracker -- , seq: 1950, time: 2014-06-09 16:22:11.549225, event: done, op: pg_notify(0.479(21) epoch 243) 2014-06-09 16:22:11.549235 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549237 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549259 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549260 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549262 7f869af78700 20 osd.26 243 _dispatch 0x1b0cea80 pg_notify(2.40f(20) epoch 243) v5 2014-06-09 16:22:11.549267 7f869af78700 5 -- op tracker -- , seq: 1951, time: 2014-06-09 16:22:11.324578, event: header_read, op: pg_notify(2.40f(20) epoch 243) 2014-06-09 16:22:11.549274 7f869af78700 5 -- op tracker -- , seq: 1951, time: 2014-06-09 16:22:11.324579, event: throttled, op: pg_notify(2.40f(20) epoch 243) 2014-06-09 16:22:11.549280 7f869af78700 5 -- op tracker -- , seq: 1951, time: 2014-06-09 16:22:11.324615, event: all_read, op: pg_notify(2.40f(20) epoch 243) 2014-06-09 16:22:11.549286 7f869af78700 5 -- op tracker -- , seq: 1951, time: 2014-06-09 16:22:11.549258, event: dispatched, op: pg_notify(2.40f(20) epoch 243) 2014-06-09 16:22:11.549293 7f869af78700 5 -- op tracker -- , seq: 1951, time: 2014-06-09 16:22:11.549291, event: waiting_for_osdmap, op: pg_notify(2.40f(20) epoch 243) 2014-06-09 16:22:11.549298 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.549300 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1b0cea80 2014-06-09 16:22:11.549305 7f869af78700 5 -- op tracker -- , seq: 1951, time: 2014-06-09 16:22:11.549303, event: started, op: pg_notify(2.40f(20) epoch 243) 2014-06-09 16:22:11.549320 7f869af78700 5 -- op tracker -- , seq: 1951, time: 2014-06-09 16:22:11.549318, event: done, op: pg_notify(2.40f(20) epoch 243) 2014-06-09 16:22:11.549328 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549330 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549350 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549351 7f869af78700 20 osd.26 243 _dispatch 0x30644000 pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.549358 7f869af78700 5 -- op tracker -- , seq: 1952, time: 2014-06-09 16:22:11.137173, event: header_read, op: pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549366 7f869af78700 5 -- op tracker -- , seq: 1952, time: 2014-06-09 16:22:11.137174, event: throttled, op: pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549373 7f869af78700 5 -- op tracker -- , seq: 1952, time: 2014-06-09 16:22:11.137186, event: all_read, op: pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549380 7f869af78700 5 -- op tracker -- , seq: 1952, time: 2014-06-09 16:22:11.549347, event: dispatched, op: pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549388 7f869af78700 5 -- op tracker -- , seq: 1952, time: 2014-06-09 16:22:11.549385, event: waiting_for_osdmap, op: pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549392 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.549396 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30644000 2014-06-09 16:22:11.549402 7f869af78700 5 -- op tracker -- , seq: 1952, time: 2014-06-09 16:22:11.549399, event: started, op: pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549415 7f869af78700 5 -- op tracker -- , seq: 1952, time: 2014-06-09 16:22:11.549412, event: done, op: pg_log(1.555 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.549422 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549424 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549436 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549437 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549438 7f869af78700 20 osd.26 243 _dispatch 0x19eb9340 pg_notify(3.67f(10),0.f1(20) epoch 243) v5 2014-06-09 16:22:11.549444 7f869af78700 5 -- op tracker -- , seq: 1953, time: 2014-06-09 16:22:11.126218, event: header_read, op: pg_notify(3.67f(10),0.f1(20) epoch 243) 2014-06-09 16:22:11.549452 7f869af78700 5 -- op tracker -- , seq: 1953, time: 2014-06-09 16:22:11.126223, event: throttled, op: pg_notify(3.67f(10),0.f1(20) epoch 243) 2014-06-09 16:22:11.549459 7f869af78700 5 -- op tracker -- , seq: 1953, time: 2014-06-09 16:22:11.126263, event: all_read, op: pg_notify(3.67f(10),0.f1(20) epoch 243) 2014-06-09 16:22:11.549465 7f869af78700 5 -- op tracker -- , seq: 1953, time: 2014-06-09 16:22:11.549435, event: dispatched, op: pg_notify(3.67f(10),0.f1(20) epoch 243) 2014-06-09 16:22:11.549473 7f869af78700 5 -- op tracker -- , seq: 1953, time: 2014-06-09 16:22:11.549470, event: waiting_for_osdmap, op: pg_notify(3.67f(10),0.f1(20) epoch 243) 2014-06-09 16:22:11.549477 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.549480 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb9340 2014-06-09 16:22:11.549486 7f869af78700 5 -- op tracker -- , seq: 1953, time: 2014-06-09 16:22:11.549483, event: started, op: pg_notify(3.67f(10),0.f1(20) epoch 243) 2014-06-09 16:22:11.549510 7f869af78700 5 -- op tracker -- , seq: 1953, time: 2014-06-09 16:22:11.549507, event: done, op: pg_notify(3.67f(10),0.f1(20) epoch 243) 2014-06-09 16:22:11.549518 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549520 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549546 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.549548 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.549549 7f869af78700 20 osd.26 243 _dispatch 0x1a039c00 pg_info(1 pgs e243:3.2fd(2)) v4 2014-06-09 16:22:11.549559 7f868ff62700 2 waiting 51 > 50 ops || 93887 > 104857600 2014-06-09 16:22:11.549730 7f868cf5c700 2 waiting 51 > 50 ops || 93882 > 104857600 2014-06-09 16:22:11.549738 7f868f761700 2 waiting 51 > 50 ops || 93784 > 104857600 2014-06-09 16:22:11.549744 7f868af58700 2 waiting 51 > 50 ops || 93216 > 104857600 2014-06-09 16:22:11.549750 7f868c75b700 2 waiting 51 > 50 ops || 93764 > 104857600 2014-06-09 16:22:11.549756 7f8692f68700 2 waiting 51 > 50 ops || 93890 > 104857600 2014-06-09 16:22:11.549761 7f868a757700 2 waiting 51 > 50 ops || 93761 > 104857600 2014-06-09 16:22:11.549766 7f8689755700 2 waiting 51 > 50 ops || 93878 > 104857600 2014-06-09 16:22:11.549772 7f868bf5a700 2 waiting 51 > 50 ops || 93835 > 104857600 2014-06-09 16:22:11.549776 7f868d75d700 2 waiting 51 > 50 ops || 93764 > 104857600 2014-06-09 16:22:11.549782 7f8691f66700 2 waiting 51 > 50 ops || 93980 > 104857600 2014-06-09 16:22:11.535703 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4fc0 (2 active) 2014-06-09 16:22:11.549791 7f86a5e11700 5 _do_op 0x241044e0 seq 3149 osr(3.65d 0x3ed1a30)/0x3ed1a30 start 2014-06-09 16:22:11.549796 7f86a5e11700 10 _do_transaction on 0xc20fd40 2014-06-09 16:22:11.549806 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.549874 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.549897 7f86a5e11700 15 _omap_rmkeyrange meta/1fe0d6eb/pglog_3.65d/0//-1 [0000000214.00000000000000000533,4294967295.18446744073709551615] 2014-06-09 16:22:11.549900 7f86a5e11700 15 get_omap_iterator meta/1fe0d6eb/pglog_3.65d/0//-1 2014-06-09 16:22:11.549988 7f86a5e11700 15 _omap_rmkeys meta/1fe0d6eb/pglog_3.65d/0//-1 2014-06-09 16:22:11.550030 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe0d6eb/pglog_3.65d/0//-1 2014-06-09 16:22:11.550042 7f86a5e11700 10 _omap_rmkeys meta/1fe0d6eb/pglog_3.65d/0//-1 r = 0 2014-06-09 16:22:11.550048 7f86a5e11700 15 _omap_rmkeys meta/1fe0d6eb/pglog_3.65d/0//-1 2014-06-09 16:22:11.550053 7f86a5e11700 10 _omap_rmkeys meta/1fe0d6eb/pglog_3.65d/0//-1 r = 0 2014-06-09 16:22:11.550065 7f86a5e11700 15 _omap_setkeys meta/1fe0d6eb/pglog_3.65d/0//-1 2014-06-09 16:22:11.550086 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3149.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550093 7f86a5e11700 10 cid: oid: 1fe0d6eb/pglog_3.65d/0//-1 skipping op, spos 3149.0.0 <= header.spos 150402.0.0 2014-06-09 16:22:11.550096 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550134 7f86a5e11700 10 _do_op 0x241044e0 seq 3149 r = 0, finisher 0x33a6e80 0 2014-06-09 16:22:11.550139 7f86a5e11700 10 _finish_op 0x241044e0 seq 3149 osr(3.65d 0x3ed1a30)/0x3ed1a30 2014-06-09 16:22:11.550147 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4fc0 (1 active) 2014-06-09 16:22:11.550150 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0a20 (2 active) 2014-06-09 16:22:11.550153 7f86a5e11700 5 _do_op 0x30060060 seq 3150 osr(0.16c 0x2bd67f0)/0x2bd67f0 start 2014-06-09 16:22:11.550156 7f86a5e11700 10 _do_transaction on 0x37e4b40 2014-06-09 16:22:11.550162 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550197 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550214 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3150.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550219 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550255 7f86a5e11700 10 _do_op 0x30060060 seq 3150 r = 0, finisher 0xbd667e0 0 2014-06-09 16:22:11.550260 7f86a5e11700 10 _finish_op 0x30060060 seq 3150 osr(0.16c 0x2bd67f0)/0x2bd67f0 2014-06-09 16:22:11.550263 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0a20 (1 active) 2014-06-09 16:22:11.550266 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1c20 (2 active) 2014-06-09 16:22:11.550268 7f86a5e11700 5 _do_op 0x30060000 seq 3151 osr(1.697 0x384de10)/0x384de10 start 2014-06-09 16:22:11.550271 7f86a5e11700 10 _do_transaction on 0xc0d5680 2014-06-09 16:22:11.550276 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550311 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550328 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3151.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550332 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550351 7f86a5e11700 10 _do_op 0x30060000 seq 3151 r = 0, finisher 0x3061fea0 0 2014-06-09 16:22:11.550355 7f86a5e11700 10 _finish_op 0x30060000 seq 3151 osr(1.697 0x384de10)/0x384de10 2014-06-09 16:22:11.550357 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1c20 (1 active) 2014-06-09 16:22:11.550359 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb9e0 (2 active) 2014-06-09 16:22:11.550362 7f86a5e11700 5 _do_op 0x2fcae8a0 seq 3152 osr(3.39e 0x3ed0fa0)/0x3ed0fa0 start 2014-06-09 16:22:11.550365 7f86a5e11700 10 _do_transaction on 0xc12a780 2014-06-09 16:22:11.550371 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550405 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550421 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3152.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550425 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550447 7f86a5e11700 10 _do_op 0x2fcae8a0 seq 3152 r = 0, finisher 0x33a7f20 0 2014-06-09 16:22:11.550452 7f86a5e11700 10 _finish_op 0x2fcae8a0 seq 3152 osr(3.39e 0x3ed0fa0)/0x3ed0fa0 2014-06-09 16:22:11.550454 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb9e0 (1 active) 2014-06-09 16:22:11.550456 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1e60 (2 active) 2014-06-09 16:22:11.550458 7f86a5e11700 5 _do_op 0x2fcae840 seq 3153 osr(3.66a 0x3ed19d0)/0x3ed19d0 start 2014-06-09 16:22:11.550461 7f86a5e11700 10 _do_transaction on 0x2fb6900 2014-06-09 16:22:11.550467 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550502 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550518 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3153.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550523 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550543 7f86a5e11700 10 _do_op 0x2fcae840 seq 3153 r = 0, finisher 0xc6e8c40 0 2014-06-09 16:22:11.550546 7f86a5e11700 10 _finish_op 0x2fcae840 seq 3153 osr(3.66a 0x3ed19d0)/0x3ed19d0 2014-06-09 16:22:11.550549 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1e60 (1 active) 2014-06-09 16:22:11.550551 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff50e0 (2 active) 2014-06-09 16:22:11.550553 7f86a5e11700 5 _do_op 0x2fcae7e0 seq 3154 osr(2.c9 0x2ed04a0)/0x2ed04a0 start 2014-06-09 16:22:11.550556 7f86a5e11700 10 _do_transaction on 0x3934b40 2014-06-09 16:22:11.550561 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550592 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550609 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3154.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550613 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550631 7f86a5e11700 10 _do_op 0x2fcae7e0 seq 3154 r = 0, finisher 0x30508940 0 2014-06-09 16:22:11.550634 7f86a5e11700 10 _finish_op 0x2fcae7e0 seq 3154 osr(2.c9 0x2ed04a0)/0x2ed04a0 2014-06-09 16:22:11.550637 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff50e0 (1 active) 2014-06-09 16:22:11.550639 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ade60 (2 active) 2014-06-09 16:22:11.550641 7f86a5e11700 5 _do_op 0x2fcae780 seq 3155 osr(1.348 0x384d310)/0x384d310 start 2014-06-09 16:22:11.550644 7f86a5e11700 10 _do_transaction on 0xbf28300 2014-06-09 16:22:11.550650 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550681 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550698 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3155.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550702 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550727 7f86a5e11700 10 _do_op 0x2fcae780 seq 3155 r = 0, finisher 0xbd77380 0 2014-06-09 16:22:11.550731 7f86a5e11700 10 _finish_op 0x2fcae780 seq 3155 osr(1.348 0x384d310)/0x384d310 2014-06-09 16:22:11.550733 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ade60 (1 active) 2014-06-09 16:22:11.550736 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3005a0 (2 active) 2014-06-09 16:22:11.550737 7f86a5e11700 5 _do_op 0x2fcae720 seq 3156 osr(3.6f 0x3ed0280)/0x3ed0280 start 2014-06-09 16:22:11.550740 7f86a5e11700 10 _do_transaction on 0xc1415c0 2014-06-09 16:22:11.550746 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550776 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550792 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3156.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550797 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550815 7f86a5e11700 10 _do_op 0x2fcae720 seq 3156 r = 0, finisher 0xbd721a0 0 2014-06-09 16:22:11.550819 7f86a5e11700 10 _finish_op 0x2fcae720 seq 3156 osr(3.6f 0x3ed0280)/0x3ed0280 2014-06-09 16:22:11.550822 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3005a0 (1 active) 2014-06-09 16:22:11.550825 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d9e0 (2 active) 2014-06-09 16:22:11.550827 7f86a5e11700 5 _do_op 0x2fcae6c0 seq 3157 osr(0.380 0x2bd6f20)/0x2bd6f20 start 2014-06-09 16:22:11.550830 7f86a5e11700 10 _do_transaction on 0xc127500 2014-06-09 16:22:11.550836 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550869 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550885 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3157.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550890 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.550917 7f86a5e11700 10 _do_op 0x2fcae6c0 seq 3157 r = 0, finisher 0xbe318c0 0 2014-06-09 16:22:11.550921 7f86a5e11700 10 _finish_op 0x2fcae6c0 seq 3157 osr(0.380 0x2bd6f20)/0x2bd6f20 2014-06-09 16:22:11.550923 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d9e0 (1 active) 2014-06-09 16:22:11.550925 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358000 (2 active) 2014-06-09 16:22:11.550927 7f86a5e11700 5 _do_op 0x30606060 seq 3158 osr(0.31f 0x2bd6d50)/0x2bd6d50 start 2014-06-09 16:22:11.550930 7f86a5e11700 10 _do_transaction on 0x2f280c0 2014-06-09 16:22:11.550937 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.550971 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.550987 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3158.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.550992 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551010 7f86a5e11700 10 _do_op 0x30606060 seq 3158 r = 0, finisher 0x2d738ec0 0 2014-06-09 16:22:11.551013 7f86a5e11700 10 _finish_op 0x30606060 seq 3158 osr(0.31f 0x2bd6d50)/0x2bd6d50 2014-06-09 16:22:11.551016 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358000 (1 active) 2014-06-09 16:22:11.551018 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac000 (2 active) 2014-06-09 16:22:11.551020 7f86a5e11700 5 _do_op 0x309a3800 seq 3159 osr(2.5f8 0x2ed1db0)/0x2ed1db0 start 2014-06-09 16:22:11.551022 7f86a5e11700 10 _do_transaction on 0x3541200 2014-06-09 16:22:11.551028 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551059 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551075 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3159.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551080 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551101 7f86a5e11700 10 _do_op 0x309a3800 seq 3159 r = 0, finisher 0x2d739220 0 2014-06-09 16:22:11.551104 7f86a5e11700 10 _finish_op 0x309a3800 seq 3159 osr(2.5f8 0x2ed1db0)/0x2ed1db0 2014-06-09 16:22:11.551108 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac000 (1 active) 2014-06-09 16:22:11.551111 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd680 (2 active) 2014-06-09 16:22:11.551113 7f86a5e11700 5 _do_op 0x309a37a0 seq 3160 osr(1.2ef 0x384d0f0)/0x384d0f0 start 2014-06-09 16:22:11.551116 7f86a5e11700 10 _do_transaction on 0xc0cfc80 2014-06-09 16:22:11.551122 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551153 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551168 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3160.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551173 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551192 7f86a5e11700 10 _do_op 0x309a37a0 seq 3160 r = 0, finisher 0x306cf020 0 2014-06-09 16:22:11.551196 7f86a5e11700 10 _finish_op 0x309a37a0 seq 3160 osr(1.2ef 0x384d0f0)/0x384d0f0 2014-06-09 16:22:11.551199 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd680 (1 active) 2014-06-09 16:22:11.551201 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0900 (2 active) 2014-06-09 16:22:11.551203 7f86a5e11700 5 _do_op 0x309a3740 seq 3161 osr(2.f2 0x2ed07e0)/0x2ed07e0 start 2014-06-09 16:22:11.551205 7f86a5e11700 10 _do_transaction on 0x344c300 2014-06-09 16:22:11.551212 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551245 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551261 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3161.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551265 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551288 7f86a5e11700 10 _do_op 0x309a3740 seq 3161 r = 0, finisher 0x1a002d80 0 2014-06-09 16:22:11.551292 7f86a5e11700 10 _finish_op 0x309a3740 seq 3161 osr(2.f2 0x2ed07e0)/0x2ed07e0 2014-06-09 16:22:11.551294 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0900 (1 active) 2014-06-09 16:22:11.551296 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d06c0 (2 active) 2014-06-09 16:22:11.551299 7f86a5e11700 5 _do_op 0x309a36e0 seq 3162 osr(2.57a 0x2ed1870)/0x2ed1870 start 2014-06-09 16:22:11.551301 7f86a5e11700 10 _do_transaction on 0xc12bec0 2014-06-09 16:22:11.551307 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551341 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551356 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3162.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551361 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551383 7f86a5e11700 10 _do_op 0x309a36e0 seq 3162 r = 0, finisher 0xc411b80 0 2014-06-09 16:22:11.551386 7f86a5e11700 10 _finish_op 0x309a36e0 seq 3162 osr(2.57a 0x2ed1870)/0x2ed1870 2014-06-09 16:22:11.551389 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d06c0 (1 active) 2014-06-09 16:22:11.551391 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7560 (2 active) 2014-06-09 16:22:11.551393 7f86a5e11700 5 _do_op 0x309a3680 seq 3163 osr(2.487 0x2ed14b0)/0x2ed14b0 start 2014-06-09 16:22:11.551395 7f86a5e11700 10 _do_transaction on 0xc22c600 2014-06-09 16:22:11.551401 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551431 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551447 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3163.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551452 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551479 7f86a5e11700 10 _do_op 0x309a3680 seq 3163 r = 0, finisher 0xbd657c0 0 2014-06-09 16:22:11.551483 7f86a5e11700 10 _finish_op 0x309a3680 seq 3163 osr(2.487 0x2ed14b0)/0x2ed14b0 2014-06-09 16:22:11.551485 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7560 (1 active) 2014-06-09 16:22:11.551487 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286120 (2 active) 2014-06-09 16:22:11.551489 7f86a5e11700 5 _do_op 0x309a3620 seq 3164 osr(3.37a 0x3ed1040)/0x3ed1040 start 2014-06-09 16:22:11.551492 7f86a5e11700 10 _do_transaction on 0x97baa80 2014-06-09 16:22:11.551497 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551530 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551547 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3164.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551551 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551567 7f86a5e11700 10 _do_op 0x309a3620 seq 3164 r = 0, finisher 0xbd705e0 0 2014-06-09 16:22:11.551571 7f86a5e11700 10 _finish_op 0x309a3620 seq 3164 osr(3.37a 0x3ed1040)/0x3ed1040 2014-06-09 16:22:11.551573 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286120 (1 active) 2014-06-09 16:22:11.551575 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496120 (2 active) 2014-06-09 16:22:11.551577 7f86a5e11700 5 _do_op 0x309a3da0 seq 3165 osr(1.382 0x384d5b0)/0x384d5b0 start 2014-06-09 16:22:11.551580 7f86a5e11700 10 _do_transaction on 0x336ad80 2014-06-09 16:22:11.551585 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551620 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551637 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3165.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551642 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551671 7f86a5e11700 10 _do_op 0x309a3da0 seq 3165 r = 0, finisher 0x306288a0 0 2014-06-09 16:22:11.551676 7f86a5e11700 10 _finish_op 0x309a3da0 seq 3165 osr(1.382 0x384d5b0)/0x384d5b0 2014-06-09 16:22:11.551679 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496120 (1 active) 2014-06-09 16:22:11.551682 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c19e0 (2 active) 2014-06-09 16:22:11.551684 7f86a5e11700 5 _do_op 0x309a3d40 seq 3166 osr(1.45d 0x384d670)/0x384d670 start 2014-06-09 16:22:11.551686 7f86a5e11700 10 _do_transaction on 0xc0d9c80 2014-06-09 16:22:11.551692 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551725 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551742 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3166.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551746 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551763 7f86a5e11700 10 _do_op 0x309a3d40 seq 3166 r = 0, finisher 0xc40dd00 0 2014-06-09 16:22:11.551767 7f86a5e11700 10 _finish_op 0x309a3d40 seq 3166 osr(1.45d 0x384d670)/0x384d670 2014-06-09 16:22:11.551769 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c19e0 (1 active) 2014-06-09 16:22:11.551771 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496b40 (2 active) 2014-06-09 16:22:11.551773 7f86a5e11700 5 _do_op 0x309a3ce0 seq 3167 osr(2.69c 0x2ed1f20)/0x2ed1f20 start 2014-06-09 16:22:11.551776 7f86a5e11700 10 _do_transaction on 0xc138c00 2014-06-09 16:22:11.551781 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551814 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551831 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3167.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551835 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551856 7f86a5e11700 10 _do_op 0x309a3ce0 seq 3167 r = 0, finisher 0x139651a0 0 2014-06-09 16:22:11.551860 7f86a5e11700 10 _finish_op 0x309a3ce0 seq 3167 osr(2.69c 0x2ed1f20)/0x2ed1f20 2014-06-09 16:22:11.551863 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496b40 (1 active) 2014-06-09 16:22:11.551864 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300480 (2 active) 2014-06-09 16:22:11.551867 7f86a5e11700 5 _do_op 0x309a3c80 seq 3168 osr(3.7fc 0xb7f00e0)/0xb7f00e0 start 2014-06-09 16:22:11.551869 7f86a5e11700 10 _do_transaction on 0x35be000 2014-06-09 16:22:11.551875 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551904 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.551922 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3168.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.551926 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.551944 7f86a5e11700 10 _do_op 0x309a3c80 seq 3168 r = 0, finisher 0xbd74e60 0 2014-06-09 16:22:11.551948 7f86a5e11700 10 _finish_op 0x309a3c80 seq 3168 osr(3.7fc 0xb7f00e0)/0xb7f00e0 2014-06-09 16:22:11.551951 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300480 (1 active) 2014-06-09 16:22:11.551953 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1159e0 (2 active) 2014-06-09 16:22:11.551955 7f86a5e11700 5 _do_op 0x309a3c20 seq 3169 osr(1.6e8 0x2ed0190)/0x2ed0190 start 2014-06-09 16:22:11.551958 7f86a5e11700 10 _do_transaction on 0x2ef9080 2014-06-09 16:22:11.551964 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.551993 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.552009 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3169.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.552013 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.552036 7f86a5e11700 10 _do_op 0x309a3c20 seq 3169 r = 0, finisher 0xbd725c0 0 2014-06-09 16:22:11.552040 7f86a5e11700 10 _finish_op 0x309a3c20 seq 3169 osr(1.6e8 0x2ed0190)/0x2ed0190 2014-06-09 16:22:11.552043 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1159e0 (1 active) 2014-06-09 16:22:11.552044 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4b40 (2 active) 2014-06-09 16:22:11.552047 7f86a5e11700 5 _do_op 0x309a3bc0 seq 3170 osr(3.616 0x3ed1a90)/0x3ed1a90 start 2014-06-09 16:22:11.552049 7f86a5e11700 10 _do_transaction on 0xc1f7e00 2014-06-09 16:22:11.552056 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.552089 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.552105 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3170.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.552110 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.552125 7f86a5e11700 10 _do_op 0x309a3bc0 seq 3170 r = 0, finisher 0xbd7b220 0 2014-06-09 16:22:11.552128 7f86a5e11700 10 _finish_op 0x309a3bc0 seq 3170 osr(3.616 0x3ed1a90)/0x3ed1a90 2014-06-09 16:22:11.552131 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4b40 (1 active) 2014-06-09 16:22:11.552134 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd440 (2 active) 2014-06-09 16:22:11.552136 7f86a5e11700 5 _do_op 0x309a3b60 seq 3171 osr(2.34d 0x2ed0e70)/0x2ed0e70 start 2014-06-09 16:22:11.552138 7f86a5e11700 10 _do_transaction on 0xc2aa900 2014-06-09 16:22:11.552144 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.552177 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.552193 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3171.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.552198 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.552220 7f86a5e11700 10 _do_op 0x309a3b60 seq 3171 r = 0, finisher 0xc40d080 0 2014-06-09 16:22:11.552223 7f86a5e11700 10 _finish_op 0x309a3b60 seq 3171 osr(2.34d 0x2ed0e70)/0x2ed0e70 2014-06-09 16:22:11.552226 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd440 (1 active) 2014-06-09 16:22:11.552228 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f200 (2 active) 2014-06-09 16:22:11.552230 7f86a5e11700 5 _do_op 0x309a3b00 seq 3172 osr(2.e6 0x2ed07d0)/0x2ed07d0 start 2014-06-09 16:22:11.552232 7f86a5e11700 10 _do_transaction on 0xbf83380 2014-06-09 16:22:11.552238 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.552270 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.552287 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3172.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.552292 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.552309 7f86a5e11700 10 _do_op 0x309a3b00 seq 3172 r = 0, finisher 0xbd6a520 0 2014-06-09 16:22:11.552312 7f86a5e11700 10 _finish_op 0x309a3b00 seq 3172 osr(2.e6 0x2ed07d0)/0x2ed07d0 2014-06-09 16:22:11.552315 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f200 (1 active) 2014-06-09 16:22:11.552317 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358b40 (2 active) 2014-06-09 16:22:11.552319 7f86a5e11700 5 _do_op 0x309a3aa0 seq 3173 osr(2.fd 0x2ed0770)/0x2ed0770 start 2014-06-09 16:22:11.552322 7f86a5e11700 10 _do_transaction on 0xc24cd80 2014-06-09 16:22:11.552327 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.552360 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.552376 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3173.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.552381 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.552402 7f86a5e11700 10 _do_op 0x309a3aa0 seq 3173 r = 0, finisher 0x33a70e0 0 2014-06-09 16:22:11.552406 7f86a5e11700 10 _finish_op 0x309a3aa0 seq 3173 osr(2.fd 0x2ed0770)/0x2ed0770 2014-06-09 16:22:11.552408 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358b40 (1 active) 2014-06-09 16:22:11.552410 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287e60 (2 active) 2014-06-09 16:22:11.552412 7f86a5e11700 5 _do_op 0x309a3a40 seq 3174 osr(0.25f 0x2bd6d00)/0x2bd6d00 start 2014-06-09 16:22:11.552414 7f86a5e11700 10 _do_transaction on 0xc140b40 2014-06-09 16:22:11.552420 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.552451 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.552468 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3174.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.552472 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.552492 7f86a5e11700 10 _do_op 0x309a3a40 seq 3174 r = 0, finisher 0xbd73920 0 2014-06-09 16:22:11.552496 7f86a5e11700 10 _finish_op 0x309a3a40 seq 3174 osr(0.25f 0x2bd6d00)/0x2bd6d00 2014-06-09 16:22:11.552566 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061fde0,0x189aaa00,0xc410400,0x3061fee0,0xbd78980,0x3061ee40,0x305089a0,0xbd70fe0,0x13965540,0x3050a340,0xbce73c0,0xbd6d2c0,0x306ce0a0,0x306ce720,0xbd76d60,0xbd62e20,0xbd71640,0x306287a0,0x19940280,0xc5a7d00,0xc40cd40,0x19940bc0,0xbd71980,0xbd7d140,0x33a6000,0xbd6a820,0xc6e9e40] 2014-06-09 16:22:11.561057 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061fde0,0x189aaa00,0xc410400,0x3061fee0,0xbd78980,0x3061ee40,0x305089a0,0xbd70fe0,0x13965540,0x3050a340,0xbce73c0,0xbd6d2c0,0x306ce0a0,0x306ce720,0xbd76d60,0xbd62e20,0xbd71640,0x306287a0,0x19940280,0xc5a7d00,0xc40cd40,0x19940bc0,0xbd71980,0xbd7d140,0x33a6000,0xbd6a820,0xc6e9e40] 2014-06-09 16:22:11.561068 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.561069 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.552579 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a6e80,0xbd667e0,0x3061fea0,0x33a7f20,0xc6e8c40,0x30508940,0xbd77380,0xbd721a0,0xbe318c0,0x2d738ec0,0x2d739220,0x306cf020,0x1a002d80,0xc411b80,0xbd657c0,0xbd705e0,0x306288a0,0xc40dd00,0x139651a0,0xbd74e60,0xbd725c0,0xbd7b220,0xc40d080,0xbd6a520,0x33a70e0,0xbd73920] 2014-06-09 16:22:11.561078 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.561103 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.561121 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.561144 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561164 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.561187 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] flushed 2014-06-09 16:22:11.561205 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561219 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.561234 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.561252 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561269 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.561284 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.561304 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561320 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.561334 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.561350 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561363 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.561377 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.561392 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561405 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.561421 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.561436 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561451 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.561465 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.561482 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561495 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.561511 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.561525 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561538 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.561553 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.561569 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561583 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.561594 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.561610 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561622 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.561633 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.561648 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561660 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.561671 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.561688 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561701 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.561712 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.561728 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561742 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.561756 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.561770 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561785 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.561800 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.561818 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561831 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.561842 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.561857 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561869 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.561883 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.561898 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561911 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.561923 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] flushed 2014-06-09 16:22:11.561938 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.561952 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.552590 7f869af78700 5 -- op tracker -- , seq: 1954, time: 2014-06-09 16:22:11.331877, event: header_read, op: pg_info(1 pgs e243:3.2fd(2)) 2014-06-09 16:22:11.561972 7f869af78700 5 -- op tracker -- , seq: 1954, time: 2014-06-09 16:22:11.331877, event: throttled, op: pg_info(1 pgs e243:3.2fd(2)) 2014-06-09 16:22:11.561982 7f869af78700 5 -- op tracker -- , seq: 1954, time: 2014-06-09 16:22:11.331893, event: all_read, op: pg_info(1 pgs e243:3.2fd(2)) 2014-06-09 16:22:11.561989 7f869af78700 5 -- op tracker -- , seq: 1954, time: 2014-06-09 16:22:11.549546, event: dispatched, op: pg_info(1 pgs e243:3.2fd(2)) 2014-06-09 16:22:11.561996 7f869af78700 5 -- op tracker -- , seq: 1954, time: 2014-06-09 16:22:11.561994, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2fd(2)) 2014-06-09 16:22:11.562002 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2fd(2)) v4 from osd.7 2014-06-09 16:22:11.562006 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a039c00 2014-06-09 16:22:11.562012 7f869af78700 5 -- op tracker -- , seq: 1954, time: 2014-06-09 16:22:11.562010, event: started, op: pg_info(1 pgs e243:3.2fd(2)) 2014-06-09 16:22:11.562033 7f869af78700 5 -- op tracker -- , seq: 1954, time: 2014-06-09 16:22:11.562031, event: done, op: pg_info(1 pgs e243:3.2fd(2)) 2014-06-09 16:22:11.562030 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.562046 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562055 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.562068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562069 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562070 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.562071 7f869af78700 20 osd.26 243 _dispatch 0x307d6a00 pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.562078 7f869af78700 5 -- op tracker -- , seq: 1955, time: 2014-06-09 16:22:11.292488, event: header_read, op: pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562086 7f869af78700 5 -- op tracker -- , seq: 1955, time: 2014-06-09 16:22:11.292488, event: throttled, op: pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562083 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] flushed 2014-06-09 16:22:11.562094 7f869af78700 5 -- op tracker -- , seq: 1955, time: 2014-06-09 16:22:11.292501, event: all_read, op: pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562101 7f869af78700 5 -- op tracker -- , seq: 1955, time: 2014-06-09 16:22:11.562067, event: dispatched, op: pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562099 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.562109 7f869af78700 5 -- op tracker -- , seq: 1955, time: 2014-06-09 16:22:11.562106, event: waiting_for_osdmap, op: pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562113 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.562115 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.562119 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307d6a00 2014-06-09 16:22:11.562127 7f869af78700 5 -- op tracker -- , seq: 1955, time: 2014-06-09 16:22:11.562124, event: started, op: pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562125 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.562141 7f869af78700 5 -- op tracker -- , seq: 1955, time: 2014-06-09 16:22:11.562138, event: done, op: pg_log(2.182 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562149 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562150 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562143 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.562157 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.562163 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562165 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562166 7f869af78700 20 osd.26 243 _dispatch 0x307dec00 pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.562168 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.562173 7f869af78700 5 -- op tracker -- , seq: 1956, time: 2014-06-09 16:22:11.208216, event: header_read, op: pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562181 7f869af78700 5 -- op tracker -- , seq: 1956, time: 2014-06-09 16:22:11.208216, event: throttled, op: pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562188 7f869af78700 5 -- op tracker -- , seq: 1956, time: 2014-06-09 16:22:11.208224, event: all_read, op: pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562183 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.562196 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.562195 7f869af78700 5 -- op tracker -- , seq: 1956, time: 2014-06-09 16:22:11.562163, event: dispatched, op: pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562203 7f869af78700 5 -- op tracker -- , seq: 1956, time: 2014-06-09 16:22:11.562200, event: waiting_for_osdmap, op: pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562208 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.562212 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307dec00 2014-06-09 16:22:11.562210 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.562218 7f869af78700 5 -- op tracker -- , seq: 1956, time: 2014-06-09 16:22:11.562215, event: started, op: pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562225 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.562232 7f869af78700 5 -- op tracker -- , seq: 1956, time: 2014-06-09 16:22:11.562229, event: done, op: pg_log(2.759 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562238 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.562240 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562242 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562255 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562249 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.562256 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562258 7f869af78700 20 osd.26 243 _dispatch 0x22ac0540 pg_info(1 pgs e243:1.522) v4 2014-06-09 16:22:11.562263 7f869af78700 5 -- op tracker -- , seq: 1957, time: 2014-06-09 16:22:11.294493, event: header_read, op: pg_info(1 pgs e243:1.522) 2014-06-09 16:22:11.562265 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.562270 7f869af78700 5 -- op tracker -- , seq: 1957, time: 2014-06-09 16:22:11.294494, event: throttled, op: pg_info(1 pgs e243:1.522) 2014-06-09 16:22:11.562278 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.562277 7f869af78700 5 -- op tracker -- , seq: 1957, time: 2014-06-09 16:22:11.294502, event: all_read, op: pg_info(1 pgs e243:1.522) 2014-06-09 16:22:11.562284 7f869af78700 5 -- op tracker -- , seq: 1957, time: 2014-06-09 16:22:11.562254, event: dispatched, op: pg_info(1 pgs e243:1.522) 2014-06-09 16:22:11.562291 7f869af78700 5 -- op tracker -- , seq: 1957, time: 2014-06-09 16:22:11.562289, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.522) 2014-06-09 16:22:11.562289 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.562296 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.522) v4 from osd.9 2014-06-09 16:22:11.562299 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22ac0540 2014-06-09 16:22:11.562303 7f869af78700 5 -- op tracker -- , seq: 1957, time: 2014-06-09 16:22:11.562302, event: started, op: pg_info(1 pgs e243:1.522) 2014-06-09 16:22:11.562304 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a6e80,0xbd667e0,0x3061fea0,0x33a7f20,0xc6e8c40,0x30508940,0xbd77380,0xbd721a0,0xbe318c0,0x2d738ec0,0x2d739220,0x306cf020,0x1a002d80,0xc411b80,0xbd657c0,0xbd705e0,0x306288a0,0xc40dd00,0x139651a0,0xbd74e60,0xbd725c0,0xbd7b220,0xc40d080,0xbd6a520,0x33a70e0,0xbd73920] 2014-06-09 16:22:11.562310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.562311 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.562320 7f869af78700 5 -- op tracker -- , seq: 1957, time: 2014-06-09 16:22:11.562317, event: done, op: pg_info(1 pgs e243:1.522) 2014-06-09 16:22:11.562327 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562329 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562344 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562346 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562347 7f869af78700 20 osd.26 243 _dispatch 0x30a8d600 pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.562354 7f869af78700 5 -- op tracker -- , seq: 1958, time: 2014-06-09 16:22:11.286387, event: header_read, op: pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562362 7f869af78700 5 -- op tracker -- , seq: 1958, time: 2014-06-09 16:22:11.286388, event: throttled, op: pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562369 7f869af78700 5 -- op tracker -- , seq: 1958, time: 2014-06-09 16:22:11.286400, event: all_read, op: pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562376 7f869af78700 5 -- op tracker -- , seq: 1958, time: 2014-06-09 16:22:11.562343, event: dispatched, op: pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562385 7f869af78700 5 -- op tracker -- , seq: 1958, time: 2014-06-09 16:22:11.562382, event: waiting_for_osdmap, op: pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562390 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.562394 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a8d600 2014-06-09 16:22:11.562400 7f869af78700 5 -- op tracker -- , seq: 1958, time: 2014-06-09 16:22:11.562397, event: started, op: pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562413 7f869af78700 5 -- op tracker -- , seq: 1958, time: 2014-06-09 16:22:11.562411, event: done, op: pg_log(0.6a8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.562420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562435 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562436 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562438 7f869af78700 20 osd.26 243 _dispatch 0x3088d340 pg_notify(3.59f(10) epoch 243) v5 2014-06-09 16:22:11.562443 7f869af78700 5 -- op tracker -- , seq: 1959, time: 2014-06-09 16:22:11.304235, event: header_read, op: pg_notify(3.59f(10) epoch 243) 2014-06-09 16:22:11.562450 7f869af78700 5 -- op tracker -- , seq: 1959, time: 2014-06-09 16:22:11.304236, event: throttled, op: pg_notify(3.59f(10) epoch 243) 2014-06-09 16:22:11.562456 7f869af78700 5 -- op tracker -- , seq: 1959, time: 2014-06-09 16:22:11.304256, event: all_read, op: pg_notify(3.59f(10) epoch 243) 2014-06-09 16:22:11.562462 7f869af78700 5 -- op tracker -- , seq: 1959, time: 2014-06-09 16:22:11.562435, event: dispatched, op: pg_notify(3.59f(10) epoch 243) 2014-06-09 16:22:11.562469 7f869af78700 5 -- op tracker -- , seq: 1959, time: 2014-06-09 16:22:11.562467, event: waiting_for_osdmap, op: pg_notify(3.59f(10) epoch 243) 2014-06-09 16:22:11.562473 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.562476 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088d340 2014-06-09 16:22:11.562481 7f869af78700 5 -- op tracker -- , seq: 1959, time: 2014-06-09 16:22:11.562479, event: started, op: pg_notify(3.59f(10) epoch 243) 2014-06-09 16:22:11.562498 7f869af78700 5 -- op tracker -- , seq: 1959, time: 2014-06-09 16:22:11.562496, event: done, op: pg_notify(3.59f(10) epoch 243) 2014-06-09 16:22:11.562507 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562509 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562527 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562529 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562530 7f869af78700 20 osd.26 243 _dispatch 0x1c039500 pg_notify(1.109(21) epoch 243) v5 2014-06-09 16:22:11.562535 7f869af78700 5 -- op tracker -- , seq: 1960, time: 2014-06-09 16:22:11.195374, event: header_read, op: pg_notify(1.109(21) epoch 243) 2014-06-09 16:22:11.562543 7f869af78700 5 -- op tracker -- , seq: 1960, time: 2014-06-09 16:22:11.195374, event: throttled, op: pg_notify(1.109(21) epoch 243) 2014-06-09 16:22:11.562549 7f869af78700 5 -- op tracker -- , seq: 1960, time: 2014-06-09 16:22:11.195389, event: all_read, op: pg_notify(1.109(21) epoch 243) 2014-06-09 16:22:11.562555 7f869af78700 5 -- op tracker -- , seq: 1960, time: 2014-06-09 16:22:11.562527, event: dispatched, op: pg_notify(1.109(21) epoch 243) 2014-06-09 16:22:11.562562 7f869af78700 5 -- op tracker -- , seq: 1960, time: 2014-06-09 16:22:11.562560, event: waiting_for_osdmap, op: pg_notify(1.109(21) epoch 243) 2014-06-09 16:22:11.562566 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.562568 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c039500 2014-06-09 16:22:11.562572 7f869af78700 5 -- op tracker -- , seq: 1960, time: 2014-06-09 16:22:11.562571, event: started, op: pg_notify(1.109(21) epoch 243) 2014-06-09 16:22:11.562590 7f869af78700 5 -- op tracker -- , seq: 1960, time: 2014-06-09 16:22:11.562587, event: done, op: pg_notify(1.109(21) epoch 243) 2014-06-09 16:22:11.562598 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562600 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562617 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562619 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562620 7f869af78700 20 osd.26 243 _dispatch 0x308e21c0 pg_notify(1.15e(20),3.701(10) epoch 243) v5 2014-06-09 16:22:11.562626 7f869af78700 5 -- op tracker -- , seq: 1961, time: 2014-06-09 16:22:11.324654, event: header_read, op: pg_notify(1.15e(20),3.701(10) epoch 243) 2014-06-09 16:22:11.562633 7f869af78700 5 -- op tracker -- , seq: 1961, time: 2014-06-09 16:22:11.324654, event: throttled, op: pg_notify(1.15e(20),3.701(10) epoch 243) 2014-06-09 16:22:11.562639 7f869af78700 5 -- op tracker -- , seq: 1961, time: 2014-06-09 16:22:11.324683, event: all_read, op: pg_notify(1.15e(20),3.701(10) epoch 243) 2014-06-09 16:22:11.562646 7f869af78700 5 -- op tracker -- , seq: 1961, time: 2014-06-09 16:22:11.562617, event: dispatched, op: pg_notify(1.15e(20),3.701(10) epoch 243) 2014-06-09 16:22:11.562653 7f869af78700 5 -- op tracker -- , seq: 1961, time: 2014-06-09 16:22:11.562650, event: waiting_for_osdmap, op: pg_notify(1.15e(20),3.701(10) epoch 243) 2014-06-09 16:22:11.562658 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.562660 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e21c0 2014-06-09 16:22:11.562665 7f869af78700 5 -- op tracker -- , seq: 1961, time: 2014-06-09 16:22:11.562663, event: started, op: pg_notify(1.15e(20),3.701(10) epoch 243) 2014-06-09 16:22:11.562690 7f869af78700 5 -- op tracker -- , seq: 1961, time: 2014-06-09 16:22:11.562687, event: done, op: pg_notify(1.15e(20),3.701(10) epoch 243) 2014-06-09 16:22:11.562698 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562700 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562723 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562724 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562725 7f869af78700 20 osd.26 243 _dispatch 0x2fef5500 pg_notify(3.74(12),3.60e(12),1.345(22) epoch 243) v5 2014-06-09 16:22:11.562732 7f869af78700 5 -- op tracker -- , seq: 1962, time: 2014-06-09 16:22:11.137252, event: header_read, op: pg_notify(3.74(12),3.60e(12),1.345(22) epoch 243) 2014-06-09 16:22:11.562740 7f869af78700 5 -- op tracker -- , seq: 1962, time: 2014-06-09 16:22:11.137253, event: throttled, op: pg_notify(3.74(12),3.60e(12),1.345(22) epoch 243) 2014-06-09 16:22:11.562747 7f869af78700 5 -- op tracker -- , seq: 1962, time: 2014-06-09 16:22:11.137324, event: all_read, op: pg_notify(3.74(12),3.60e(12),1.345(22) epoch 243) 2014-06-09 16:22:11.562754 7f869af78700 5 -- op tracker -- , seq: 1962, time: 2014-06-09 16:22:11.562722, event: dispatched, op: pg_notify(3.74(12),3.60e(12),1.345(22) epoch 243) 2014-06-09 16:22:11.562761 7f869af78700 5 -- op tracker -- , seq: 1962, time: 2014-06-09 16:22:11.562759, event: waiting_for_osdmap, op: pg_notify(3.74(12),3.60e(12),1.345(22) epoch 243) 2014-06-09 16:22:11.562766 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.562768 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef5500 2014-06-09 16:22:11.562773 7f869af78700 5 -- op tracker -- , seq: 1962, time: 2014-06-09 16:22:11.562770, event: started, op: pg_notify(3.74(12),3.60e(12),1.345(22) epoch 243) 2014-06-09 16:22:11.562805 7f869af78700 5 -- op tracker -- , seq: 1962, time: 2014-06-09 16:22:11.562802, event: done, op: pg_notify(3.74(12),3.60e(12),1.345(22) epoch 243) 2014-06-09 16:22:11.562818 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562820 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562846 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562848 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562849 7f869af78700 20 osd.26 243 _dispatch 0x16064000 pg_info(1 pgs e243:3.f0(2)) v4 2014-06-09 16:22:11.562855 7f869af78700 5 -- op tracker -- , seq: 1963, time: 2014-06-09 16:22:11.126275, event: header_read, op: pg_info(1 pgs e243:3.f0(2)) 2014-06-09 16:22:11.562862 7f869af78700 5 -- op tracker -- , seq: 1963, time: 2014-06-09 16:22:11.126276, event: throttled, op: pg_info(1 pgs e243:3.f0(2)) 2014-06-09 16:22:11.562868 7f869af78700 5 -- op tracker -- , seq: 1963, time: 2014-06-09 16:22:11.126297, event: all_read, op: pg_info(1 pgs e243:3.f0(2)) 2014-06-09 16:22:11.562874 7f869af78700 5 -- op tracker -- , seq: 1963, time: 2014-06-09 16:22:11.562845, event: dispatched, op: pg_info(1 pgs e243:3.f0(2)) 2014-06-09 16:22:11.562881 7f869af78700 5 -- op tracker -- , seq: 1963, time: 2014-06-09 16:22:11.562879, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.f0(2)) 2014-06-09 16:22:11.562885 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.f0(2)) v4 from osd.4 2014-06-09 16:22:11.562888 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16064000 2014-06-09 16:22:11.562893 7f869af78700 5 -- op tracker -- , seq: 1963, time: 2014-06-09 16:22:11.562891, event: started, op: pg_info(1 pgs e243:3.f0(2)) 2014-06-09 16:22:11.562909 7f869af78700 5 -- op tracker -- , seq: 1963, time: 2014-06-09 16:22:11.562907, event: done, op: pg_info(1 pgs e243:3.f0(2)) 2014-06-09 16:22:11.562917 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562919 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562934 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.562936 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.562937 7f869af78700 20 osd.26 243 _dispatch 0x13be4540 pg_notify(1.428(20) epoch 243) v5 2014-06-09 16:22:11.562942 7f869af78700 5 -- op tracker -- , seq: 1964, time: 2014-06-09 16:22:11.331925, event: header_read, op: pg_notify(1.428(20) epoch 243) 2014-06-09 16:22:11.562949 7f869af78700 5 -- op tracker -- , seq: 1964, time: 2014-06-09 16:22:11.331926, event: throttled, op: pg_notify(1.428(20) epoch 243) 2014-06-09 16:22:11.562955 7f869af78700 5 -- op tracker -- , seq: 1964, time: 2014-06-09 16:22:11.331954, event: all_read, op: pg_notify(1.428(20) epoch 243) 2014-06-09 16:22:11.562961 7f869af78700 5 -- op tracker -- , seq: 1964, time: 2014-06-09 16:22:11.562934, event: dispatched, op: pg_notify(1.428(20) epoch 243) 2014-06-09 16:22:11.562967 7f869af78700 5 -- op tracker -- , seq: 1964, time: 2014-06-09 16:22:11.562965, event: waiting_for_osdmap, op: pg_notify(1.428(20) epoch 243) 2014-06-09 16:22:11.562971 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.562974 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x13be4540 2014-06-09 16:22:11.562978 7f869af78700 5 -- op tracker -- , seq: 1964, time: 2014-06-09 16:22:11.562976, event: started, op: pg_notify(1.428(20) epoch 243) 2014-06-09 16:22:11.562993 7f869af78700 5 -- op tracker -- , seq: 1964, time: 2014-06-09 16:22:11.562991, event: done, op: pg_notify(1.428(20) epoch 243) 2014-06-09 16:22:11.563001 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563003 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563020 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563021 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563023 7f869af78700 20 osd.26 243 _dispatch 0x19fdf6c0 pg_notify(1.19f(20) epoch 243) v5 2014-06-09 16:22:11.563028 7f869af78700 5 -- op tracker -- , seq: 1965, time: 2014-06-09 16:22:11.292545, event: header_read, op: pg_notify(1.19f(20) epoch 243) 2014-06-09 16:22:11.563035 7f869af78700 5 -- op tracker -- , seq: 1965, time: 2014-06-09 16:22:11.292545, event: throttled, op: pg_notify(1.19f(20) epoch 243) 2014-06-09 16:22:11.563041 7f869af78700 5 -- op tracker -- , seq: 1965, time: 2014-06-09 16:22:11.292562, event: all_read, op: pg_notify(1.19f(20) epoch 243) 2014-06-09 16:22:11.563047 7f869af78700 5 -- op tracker -- , seq: 1965, time: 2014-06-09 16:22:11.563019, event: dispatched, op: pg_notify(1.19f(20) epoch 243) 2014-06-09 16:22:11.563054 7f869af78700 5 -- op tracker -- , seq: 1965, time: 2014-06-09 16:22:11.563052, event: waiting_for_osdmap, op: pg_notify(1.19f(20) epoch 243) 2014-06-09 16:22:11.563059 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.563061 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdf6c0 2014-06-09 16:22:11.563066 7f869af78700 5 -- op tracker -- , seq: 1965, time: 2014-06-09 16:22:11.563064, event: started, op: pg_notify(1.19f(20) epoch 243) 2014-06-09 16:22:11.563082 7f869af78700 5 -- op tracker -- , seq: 1965, time: 2014-06-09 16:22:11.563079, event: done, op: pg_notify(1.19f(20) epoch 243) 2014-06-09 16:22:11.563090 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563091 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563108 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563110 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563111 7f869af78700 20 osd.26 243 _dispatch 0x19eb96c0 pg_info(1 pgs e243:3.18e(4)) v4 2014-06-09 16:22:11.563116 7f869af78700 5 -- op tracker -- , seq: 1966, time: 2014-06-09 16:22:11.208888, event: header_read, op: pg_info(1 pgs e243:3.18e(4)) 2014-06-09 16:22:11.563123 7f869af78700 5 -- op tracker -- , seq: 1966, time: 2014-06-09 16:22:11.208888, event: throttled, op: pg_info(1 pgs e243:3.18e(4)) 2014-06-09 16:22:11.563130 7f869af78700 5 -- op tracker -- , seq: 1966, time: 2014-06-09 16:22:11.208899, event: all_read, op: pg_info(1 pgs e243:3.18e(4)) 2014-06-09 16:22:11.563136 7f869af78700 5 -- op tracker -- , seq: 1966, time: 2014-06-09 16:22:11.563107, event: dispatched, op: pg_info(1 pgs e243:3.18e(4)) 2014-06-09 16:22:11.563144 7f869af78700 5 -- op tracker -- , seq: 1966, time: 2014-06-09 16:22:11.563141, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.18e(4)) 2014-06-09 16:22:11.563148 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.18e(4)) v4 from osd.1 2014-06-09 16:22:11.563151 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb96c0 2014-06-09 16:22:11.563156 7f869af78700 5 -- op tracker -- , seq: 1966, time: 2014-06-09 16:22:11.563154, event: started, op: pg_info(1 pgs e243:3.18e(4)) 2014-06-09 16:22:11.563173 7f869af78700 5 -- op tracker -- , seq: 1966, time: 2014-06-09 16:22:11.563171, event: done, op: pg_info(1 pgs e243:3.18e(4)) 2014-06-09 16:22:11.563181 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563183 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563197 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563199 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563201 7f869af78700 20 osd.26 243 _dispatch 0x3043a1c0 pg_info(1 pgs e243:0.325) v4 2014-06-09 16:22:11.563205 7f869af78700 5 -- op tracker -- , seq: 1967, time: 2014-06-09 16:22:11.294506, event: header_read, op: pg_info(1 pgs e243:0.325) 2014-06-09 16:22:11.563212 7f869af78700 5 -- op tracker -- , seq: 1967, time: 2014-06-09 16:22:11.294507, event: throttled, op: pg_info(1 pgs e243:0.325) 2014-06-09 16:22:11.563219 7f869af78700 5 -- op tracker -- , seq: 1967, time: 2014-06-09 16:22:11.294513, event: all_read, op: pg_info(1 pgs e243:0.325) 2014-06-09 16:22:11.563225 7f869af78700 5 -- op tracker -- , seq: 1967, time: 2014-06-09 16:22:11.563197, event: dispatched, op: pg_info(1 pgs e243:0.325) 2014-06-09 16:22:11.563231 7f869af78700 5 -- op tracker -- , seq: 1967, time: 2014-06-09 16:22:11.563229, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.325) 2014-06-09 16:22:11.563235 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.325) v4 from osd.9 2014-06-09 16:22:11.563238 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043a1c0 2014-06-09 16:22:11.563243 7f869af78700 5 -- op tracker -- , seq: 1967, time: 2014-06-09 16:22:11.563241, event: started, op: pg_info(1 pgs e243:0.325) 2014-06-09 16:22:11.563257 7f869af78700 5 -- op tracker -- , seq: 1967, time: 2014-06-09 16:22:11.563255, event: done, op: pg_info(1 pgs e243:0.325) 2014-06-09 16:22:11.563265 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563266 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563278 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563280 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563281 7f869af78700 20 osd.26 243 _dispatch 0x13be4a80 pg_notify(1.1da(20) epoch 243) v5 2014-06-09 16:22:11.563286 7f869af78700 5 -- op tracker -- , seq: 1968, time: 2014-06-09 16:22:11.288174, event: header_read, op: pg_notify(1.1da(20) epoch 243) 2014-06-09 16:22:11.563292 7f869af78700 5 -- op tracker -- , seq: 1968, time: 2014-06-09 16:22:11.288174, event: throttled, op: pg_notify(1.1da(20) epoch 243) 2014-06-09 16:22:11.563299 7f869af78700 5 -- op tracker -- , seq: 1968, time: 2014-06-09 16:22:11.288194, event: all_read, op: pg_notify(1.1da(20) epoch 243) 2014-06-09 16:22:11.563305 7f869af78700 5 -- op tracker -- , seq: 1968, time: 2014-06-09 16:22:11.563278, event: dispatched, op: pg_notify(1.1da(20) epoch 243) 2014-06-09 16:22:11.563311 7f869af78700 5 -- op tracker -- , seq: 1968, time: 2014-06-09 16:22:11.563309, event: waiting_for_osdmap, op: pg_notify(1.1da(20) epoch 243) 2014-06-09 16:22:11.563316 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.563318 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x13be4a80 2014-06-09 16:22:11.563323 7f869af78700 5 -- op tracker -- , seq: 1968, time: 2014-06-09 16:22:11.563321, event: started, op: pg_notify(1.1da(20) epoch 243) 2014-06-09 16:22:11.563339 7f869af78700 5 -- op tracker -- , seq: 1968, time: 2014-06-09 16:22:11.563337, event: done, op: pg_notify(1.1da(20) epoch 243) 2014-06-09 16:22:11.563347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563349 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563369 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563371 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563372 7f869af78700 20 osd.26 243 _dispatch 0x3088ddc0 pg_info(1 pgs e243:3.2d6(2)) v4 2014-06-09 16:22:11.563377 7f869af78700 5 -- op tracker -- , seq: 1969, time: 2014-06-09 16:22:11.304264, event: header_read, op: pg_info(1 pgs e243:3.2d6(2)) 2014-06-09 16:22:11.563384 7f869af78700 5 -- op tracker -- , seq: 1969, time: 2014-06-09 16:22:11.304265, event: throttled, op: pg_info(1 pgs e243:3.2d6(2)) 2014-06-09 16:22:11.563390 7f869af78700 5 -- op tracker -- , seq: 1969, time: 2014-06-09 16:22:11.304283, event: all_read, op: pg_info(1 pgs e243:3.2d6(2)) 2014-06-09 16:22:11.563397 7f869af78700 5 -- op tracker -- , seq: 1969, time: 2014-06-09 16:22:11.563369, event: dispatched, op: pg_info(1 pgs e243:3.2d6(2)) 2014-06-09 16:22:11.563404 7f869af78700 5 -- op tracker -- , seq: 1969, time: 2014-06-09 16:22:11.563401, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2d6(2)) 2014-06-09 16:22:11.563408 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2d6(2)) v4 from osd.6 2014-06-09 16:22:11.563411 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088ddc0 2014-06-09 16:22:11.563416 7f869af78700 5 -- op tracker -- , seq: 1969, time: 2014-06-09 16:22:11.563414, event: started, op: pg_info(1 pgs e243:3.2d6(2)) 2014-06-09 16:22:11.552498 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287e60 (1 active) 2014-06-09 16:22:11.563434 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0c60 (2 active) 2014-06-09 16:22:11.563435 7f869af78700 5 -- op tracker -- , seq: 1969, time: 2014-06-09 16:22:11.563432, event: done, op: pg_info(1 pgs e243:3.2d6(2)) 2014-06-09 16:22:11.563443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563444 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563443 7f86a5e11700 5 _do_op 0x309a39e0 seq 3175 osr(0.3b7 0x2bd7290)/0x2bd7290 start 2014-06-09 16:22:11.563449 7f86a5e11700 10 _do_transaction on 0xc058fc0 2014-06-09 16:22:11.563458 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563458 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.563459 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563460 7f869af78700 20 osd.26 243 _dispatch 0x14dcbe00 pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.563467 7f869af78700 5 -- op tracker -- , seq: 1970, time: 2014-06-09 16:22:11.208653, event: header_read, op: pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.563475 7f869af78700 5 -- op tracker -- , seq: 1970, time: 2014-06-09 16:22:11.208653, event: throttled, op: pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.563482 7f869af78700 5 -- op tracker -- , seq: 1970, time: 2014-06-09 16:22:11.208662, event: all_read, op: pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.563489 7f869af78700 5 -- op tracker -- , seq: 1970, time: 2014-06-09 16:22:11.563457, event: dispatched, op: pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.563497 7f869af78700 5 -- op tracker -- , seq: 1970, time: 2014-06-09 16:22:11.563494, event: waiting_for_osdmap, op: pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.563502 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.563506 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14dcbe00 2014-06-09 16:22:11.563512 7f869af78700 5 -- op tracker -- , seq: 1970, time: 2014-06-09 16:22:11.563509, event: started, op: pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.563525 7f869af78700 5 -- op tracker -- , seq: 1970, time: 2014-06-09 16:22:11.563522, event: done, op: pg_log(2.409 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.563527 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.563532 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563535 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563546 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563548 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563549 7f869af78700 20 osd.26 243 _dispatch 0x308e2e00 pg_info(1 pgs e243:3.437(2)) v4 2014-06-09 16:22:11.563551 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3175.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.563556 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.563554 7f869af78700 5 -- op tracker -- , seq: 1971, time: 2014-06-09 16:22:11.324713, event: header_read, op: pg_info(1 pgs e243:3.437(2)) 2014-06-09 16:22:11.563561 7f869af78700 5 -- op tracker -- , seq: 1971, time: 2014-06-09 16:22:11.324714, event: throttled, op: pg_info(1 pgs e243:3.437(2)) 2014-06-09 16:22:11.563567 7f869af78700 5 -- op tracker -- , seq: 1971, time: 2014-06-09 16:22:11.324723, event: all_read, op: pg_info(1 pgs e243:3.437(2)) 2014-06-09 16:22:11.563573 7f869af78700 5 -- op tracker -- , seq: 1971, time: 2014-06-09 16:22:11.563546, event: dispatched, op: pg_info(1 pgs e243:3.437(2)) 2014-06-09 16:22:11.563580 7f869af78700 5 -- op tracker -- , seq: 1971, time: 2014-06-09 16:22:11.563578, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.437(2)) 2014-06-09 16:22:11.563584 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.437(2)) v4 from osd.5 2014-06-09 16:22:11.563588 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e2e00 2014-06-09 16:22:11.563592 7f869af78700 5 -- op tracker -- , seq: 1971, time: 2014-06-09 16:22:11.563590, event: started, op: pg_info(1 pgs e243:3.437(2)) 2014-06-09 16:22:11.563602 7f86a5e11700 10 _do_op 0x309a39e0 seq 3175 r = 0, finisher 0xbd6b5a0 0 2014-06-09 16:22:11.563608 7f869af78700 5 -- op tracker -- , seq: 1971, time: 2014-06-09 16:22:11.563606, event: done, op: pg_info(1 pgs e243:3.437(2)) 2014-06-09 16:22:11.563611 7f86a5e11700 10 _finish_op 0x309a39e0 seq 3175 osr(0.3b7 0x2bd7290)/0x2bd7290 2014-06-09 16:22:11.563621 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563623 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563623 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0c60 (1 active) 2014-06-09 16:22:11.563626 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd560 (2 active) 2014-06-09 16:22:11.563629 7f86a5e11700 5 _do_op 0x309a3980 seq 3176 osr(0.5b7 0x2bd7a90)/0x2bd7a90 start 2014-06-09 16:22:11.563631 7f86a5e11700 10 _do_transaction on 0xbf89500 2014-06-09 16:22:11.563637 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563639 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563638 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.563640 7f869af78700 20 osd.26 243 _dispatch 0x1abd4700 pg_info(1 pgs e243:3.342(4)) v4 2014-06-09 16:22:11.563645 7f869af78700 5 -- op tracker -- , seq: 1972, time: 2014-06-09 16:22:11.137333, event: header_read, op: pg_info(1 pgs e243:3.342(4)) 2014-06-09 16:22:11.563653 7f869af78700 5 -- op tracker -- , seq: 1972, time: 2014-06-09 16:22:11.137334, event: throttled, op: pg_info(1 pgs e243:3.342(4)) 2014-06-09 16:22:11.563659 7f869af78700 5 -- op tracker -- , seq: 1972, time: 2014-06-09 16:22:11.137352, event: all_read, op: pg_info(1 pgs e243:3.342(4)) 2014-06-09 16:22:11.563666 7f869af78700 5 -- op tracker -- , seq: 1972, time: 2014-06-09 16:22:11.563637, event: dispatched, op: pg_info(1 pgs e243:3.342(4)) 2014-06-09 16:22:11.563674 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.563673 7f869af78700 5 -- op tracker -- , seq: 1972, time: 2014-06-09 16:22:11.563671, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.342(4)) 2014-06-09 16:22:11.563678 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.342(4)) v4 from osd.2 2014-06-09 16:22:11.563681 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd4700 2014-06-09 16:22:11.563686 7f869af78700 5 -- op tracker -- , seq: 1972, time: 2014-06-09 16:22:11.563684, event: started, op: pg_info(1 pgs e243:3.342(4)) 2014-06-09 16:22:11.563691 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3176.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.563696 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.563703 7f869af78700 5 -- op tracker -- , seq: 1972, time: 2014-06-09 16:22:11.563701, event: done, op: pg_info(1 pgs e243:3.342(4)) 2014-06-09 16:22:11.563715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563717 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563718 7f86a5e11700 10 _do_op 0x309a3980 seq 3176 r = 0, finisher 0xbd75420 0 2014-06-09 16:22:11.563722 7f86a5e11700 10 _finish_op 0x309a3980 seq 3176 osr(0.5b7 0x2bd7a90)/0x2bd7a90 2014-06-09 16:22:11.563725 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd560 (1 active) 2014-06-09 16:22:11.563727 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407e60 (2 active) 2014-06-09 16:22:11.563729 7f86a5e11700 5 _do_op 0x309a2a20 seq 3177 osr(2.3c0 0x2ed13f0)/0x2ed13f0 start 2014-06-09 16:22:11.563732 7f86a5e11700 10 _do_transaction on 0xc2cb200 2014-06-09 16:22:11.563733 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563735 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563736 7f869af78700 20 osd.26 243 _dispatch 0x11f3c380 pg_notify(0.697(23),2.3d7(20) epoch 243) v5 2014-06-09 16:22:11.563738 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.563742 7f869af78700 5 -- op tracker -- , seq: 1973, time: 2014-06-09 16:22:11.131948, event: header_read, op: pg_notify(0.697(23),2.3d7(20) epoch 243) 2014-06-09 16:22:11.563750 7f869af78700 5 -- op tracker -- , seq: 1973, time: 2014-06-09 16:22:11.131949, event: throttled, op: pg_notify(0.697(23),2.3d7(20) epoch 243) 2014-06-09 16:22:11.563757 7f869af78700 5 -- op tracker -- , seq: 1973, time: 2014-06-09 16:22:11.132014, event: all_read, op: pg_notify(0.697(23),2.3d7(20) epoch 243) 2014-06-09 16:22:11.563763 7f869af78700 5 -- op tracker -- , seq: 1973, time: 2014-06-09 16:22:11.563732, event: dispatched, op: pg_notify(0.697(23),2.3d7(20) epoch 243) 2014-06-09 16:22:11.563771 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.563771 7f869af78700 5 -- op tracker -- , seq: 1973, time: 2014-06-09 16:22:11.563768, event: waiting_for_osdmap, op: pg_notify(0.697(23),2.3d7(20) epoch 243) 2014-06-09 16:22:11.563775 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.563777 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x11f3c380 2014-06-09 16:22:11.563782 7f869af78700 5 -- op tracker -- , seq: 1973, time: 2014-06-09 16:22:11.563780, event: started, op: pg_notify(0.697(23),2.3d7(20) epoch 243) 2014-06-09 16:22:11.563787 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3177.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.563792 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.563815 7f869af78700 5 -- op tracker -- , seq: 1973, time: 2014-06-09 16:22:11.563812, event: done, op: pg_notify(0.697(23),2.3d7(20) epoch 243) 2014-06-09 16:22:11.563823 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563824 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563826 7f86a5e11700 10 _do_op 0x309a2a20 seq 3177 r = 0, finisher 0x189aa140 0 2014-06-09 16:22:11.563830 7f86a5e11700 10 _finish_op 0x309a2a20 seq 3177 osr(2.3c0 0x2ed13f0)/0x2ed13f0 2014-06-09 16:22:11.563833 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407e60 (1 active) 2014-06-09 16:22:11.563835 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0a20 (2 active) 2014-06-09 16:22:11.563838 7f86a5e11700 5 _do_op 0x2c06e7e0 seq 3178 osr(0.681 0x2bd7e10)/0x2bd7e10 start 2014-06-09 16:22:11.563840 7f86a5e11700 10 _do_transaction on 0xc228540 2014-06-09 16:22:11.563846 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563847 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.563848 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563849 7f869af78700 20 osd.26 243 _dispatch 0x3046b6c0 pg_notify(0.537(23) epoch 243) v5 2014-06-09 16:22:11.563855 7f869af78700 5 -- op tracker -- , seq: 1974, time: 2014-06-09 16:22:11.331995, event: header_read, op: pg_notify(0.537(23) epoch 243) 2014-06-09 16:22:11.563862 7f869af78700 5 -- op tracker -- , seq: 1974, time: 2014-06-09 16:22:11.331995, event: throttled, op: pg_notify(0.537(23) epoch 243) 2014-06-09 16:22:11.563868 7f869af78700 5 -- op tracker -- , seq: 1974, time: 2014-06-09 16:22:11.332027, event: all_read, op: pg_notify(0.537(23) epoch 243) 2014-06-09 16:22:11.563874 7f869af78700 5 -- op tracker -- , seq: 1974, time: 2014-06-09 16:22:11.563846, event: dispatched, op: pg_notify(0.537(23) epoch 243) 2014-06-09 16:22:11.563880 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.563882 7f869af78700 5 -- op tracker -- , seq: 1974, time: 2014-06-09 16:22:11.563879, event: waiting_for_osdmap, op: pg_notify(0.537(23) epoch 243) 2014-06-09 16:22:11.563886 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.563888 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046b6c0 2014-06-09 16:22:11.563893 7f869af78700 5 -- op tracker -- , seq: 1974, time: 2014-06-09 16:22:11.563891, event: started, op: pg_notify(0.537(23) epoch 243) 2014-06-09 16:22:11.563897 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3178.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.563901 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.563909 7f869af78700 5 -- op tracker -- , seq: 1974, time: 2014-06-09 16:22:11.563906, event: done, op: pg_notify(0.537(23) epoch 243) 2014-06-09 16:22:11.563916 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563918 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563917 7f86a5e11700 10 _do_op 0x2c06e7e0 seq 3178 r = 0, finisher 0xbcc8340 0 2014-06-09 16:22:11.563921 7f86a5e11700 10 _finish_op 0x2c06e7e0 seq 3178 osr(0.681 0x2bd7e10)/0x2bd7e10 2014-06-09 16:22:11.563924 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0a20 (1 active) 2014-06-09 16:22:11.563926 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300fc0 (2 active) 2014-06-09 16:22:11.563928 7f86a5e11700 5 _do_op 0x308caba0 seq 3179 osr(2.27a 0x2ed0d90)/0x2ed0d90 start 2014-06-09 16:22:11.563931 7f86a5e11700 10 _do_transaction on 0xbe1cd80 2014-06-09 16:22:11.563937 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.563939 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.563941 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.563942 7f869af78700 20 osd.26 243 _dispatch 0x1d038000 pg_info(1 pgs e243:3.1a0(2)) v4 2014-06-09 16:22:11.563948 7f869af78700 5 -- op tracker -- , seq: 1975, time: 2014-06-09 16:22:11.292622, event: header_read, op: pg_info(1 pgs e243:3.1a0(2)) 2014-06-09 16:22:11.563955 7f869af78700 5 -- op tracker -- , seq: 1975, time: 2014-06-09 16:22:11.292623, event: throttled, op: pg_info(1 pgs e243:3.1a0(2)) 2014-06-09 16:22:11.563961 7f869af78700 5 -- op tracker -- , seq: 1975, time: 2014-06-09 16:22:11.292632, event: all_read, op: pg_info(1 pgs e243:3.1a0(2)) 2014-06-09 16:22:11.563969 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.563968 7f869af78700 5 -- op tracker -- , seq: 1975, time: 2014-06-09 16:22:11.563939, event: dispatched, op: pg_info(1 pgs e243:3.1a0(2)) 2014-06-09 16:22:11.563975 7f869af78700 5 -- op tracker -- , seq: 1975, time: 2014-06-09 16:22:11.563973, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1a0(2)) 2014-06-09 16:22:11.563979 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1a0(2)) v4 from osd.3 2014-06-09 16:22:11.563982 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d038000 2014-06-09 16:22:11.563985 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3179.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.563990 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.563988 7f869af78700 5 -- op tracker -- , seq: 1975, time: 2014-06-09 16:22:11.563985, event: started, op: pg_info(1 pgs e243:3.1a0(2)) 2014-06-09 16:22:11.564004 7f869af78700 5 -- op tracker -- , seq: 1975, time: 2014-06-09 16:22:11.564002, event: done, op: pg_info(1 pgs e243:3.1a0(2)) 2014-06-09 16:22:11.564012 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564014 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564018 7f86a5e11700 10 _do_op 0x308caba0 seq 3179 r = 0, finisher 0xbef02a0 0 2014-06-09 16:22:11.564022 7f86a5e11700 10 _finish_op 0x308caba0 seq 3179 osr(2.27a 0x2ed0d90)/0x2ed0d90 2014-06-09 16:22:11.564025 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300fc0 (1 active) 2014-06-09 16:22:11.564027 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3585a0 (2 active) 2014-06-09 16:22:11.564029 7f86a5e11700 5 _do_op 0x308cab40 seq 3180 osr(1.a6 0x384c750)/0x384c750 start 2014-06-09 16:22:11.564030 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564032 7f86a5e11700 10 _do_transaction on 0xc29a900 2014-06-09 16:22:11.564032 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564034 7f869af78700 20 osd.26 243 _dispatch 0x307d6400 pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.564038 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564041 7f869af78700 5 -- op tracker -- , seq: 1976, time: 2014-06-09 16:22:11.211008, event: header_read, op: pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564049 7f869af78700 5 -- op tracker -- , seq: 1976, time: 2014-06-09 16:22:11.211009, event: throttled, op: pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564056 7f869af78700 5 -- op tracker -- , seq: 1976, time: 2014-06-09 16:22:11.211026, event: all_read, op: pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564063 7f869af78700 5 -- op tracker -- , seq: 1976, time: 2014-06-09 16:22:11.564030, event: dispatched, op: pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564070 7f869af78700 5 -- op tracker -- , seq: 1976, time: 2014-06-09 16:22:11.564067, event: waiting_for_osdmap, op: pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564072 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564075 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.564079 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307d6400 2014-06-09 16:22:11.564084 7f869af78700 5 -- op tracker -- , seq: 1976, time: 2014-06-09 16:22:11.564082, event: started, op: pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564089 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3180.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564093 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564097 7f869af78700 5 -- op tracker -- , seq: 1976, time: 2014-06-09 16:22:11.564094, event: done, op: pg_log(1.494 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564104 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564106 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564113 7f86a5e11700 10 _do_op 0x308cab40 seq 3180 r = 0, finisher 0xbd6fb80 0 2014-06-09 16:22:11.564116 7f86a5e11700 10 _finish_op 0x308cab40 seq 3180 osr(1.a6 0x384c750)/0x384c750 2014-06-09 16:22:11.564118 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564120 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564119 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3585a0 (1 active) 2014-06-09 16:22:11.564121 7f869af78700 20 osd.26 243 _dispatch 0x19fda000 pg_info(1 pgs e243:2.18) v4 2014-06-09 16:22:11.564122 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f0e0 (2 active) 2014-06-09 16:22:11.564124 7f86a5e11700 5 _do_op 0x30607a40 seq 3181 osr(2.113 0x2ed06b0)/0x2ed06b0 start 2014-06-09 16:22:11.564127 7f86a5e11700 10 _do_transaction on 0x3472840 2014-06-09 16:22:11.564126 7f869af78700 5 -- op tracker -- , seq: 1977, time: 2014-06-09 16:22:11.294575, event: header_read, op: pg_info(1 pgs e243:2.18) 2014-06-09 16:22:11.564133 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564133 7f869af78700 5 -- op tracker -- , seq: 1977, time: 2014-06-09 16:22:11.294575, event: throttled, op: pg_info(1 pgs e243:2.18) 2014-06-09 16:22:11.564139 7f869af78700 5 -- op tracker -- , seq: 1977, time: 2014-06-09 16:22:11.294587, event: all_read, op: pg_info(1 pgs e243:2.18) 2014-06-09 16:22:11.564146 7f869af78700 5 -- op tracker -- , seq: 1977, time: 2014-06-09 16:22:11.564118, event: dispatched, op: pg_info(1 pgs e243:2.18) 2014-06-09 16:22:11.564153 7f869af78700 5 -- op tracker -- , seq: 1977, time: 2014-06-09 16:22:11.564151, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.18) 2014-06-09 16:22:11.564158 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.18) v4 from osd.9 2014-06-09 16:22:11.564161 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fda000 2014-06-09 16:22:11.564165 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564165 7f869af78700 5 -- op tracker -- , seq: 1977, time: 2014-06-09 16:22:11.564163, event: started, op: pg_info(1 pgs e243:2.18) 2014-06-09 16:22:11.564181 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3181.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564181 7f869af78700 5 -- op tracker -- , seq: 1977, time: 2014-06-09 16:22:11.564179, event: done, op: pg_info(1 pgs e243:2.18) 2014-06-09 16:22:11.564185 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564188 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564191 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564204 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564206 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564207 7f869af78700 20 osd.26 243 _dispatch 0x30719800 pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.564209 7f86a5e11700 10 _do_op 0x30607a40 seq 3181 r = 0, finisher 0xbdf1d00 0 2014-06-09 16:22:11.564212 7f86a5e11700 10 _finish_op 0x30607a40 seq 3181 osr(2.113 0x2ed06b0)/0x2ed06b0 2014-06-09 16:22:11.564215 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f0e0 (1 active) 2014-06-09 16:22:11.564213 7f869af78700 5 -- op tracker -- , seq: 1978, time: 2014-06-09 16:22:11.288250, event: header_read, op: pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564217 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbb00 (2 active) 2014-06-09 16:22:11.564219 7f86a5e11700 5 _do_op 0x308caae0 seq 3182 osr(3.d1 0x3ed0560)/0x3ed0560 start 2014-06-09 16:22:11.564222 7f86a5e11700 10 _do_transaction on 0xc264600 2014-06-09 16:22:11.564221 7f869af78700 5 -- op tracker -- , seq: 1978, time: 2014-06-09 16:22:11.288250, event: throttled, op: pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564228 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564228 7f869af78700 5 -- op tracker -- , seq: 1978, time: 2014-06-09 16:22:11.288258, event: all_read, op: pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564235 7f869af78700 5 -- op tracker -- , seq: 1978, time: 2014-06-09 16:22:11.564204, event: dispatched, op: pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564242 7f869af78700 5 -- op tracker -- , seq: 1978, time: 2014-06-09 16:22:11.564239, event: waiting_for_osdmap, op: pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564247 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.564251 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30719800 2014-06-09 16:22:11.564257 7f869af78700 5 -- op tracker -- , seq: 1978, time: 2014-06-09 16:22:11.564254, event: started, op: pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564260 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564271 7f869af78700 5 -- op tracker -- , seq: 1978, time: 2014-06-09 16:22:11.564268, event: done, op: pg_log(2.26d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564276 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3182.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564279 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564281 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564280 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564292 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564294 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564295 7f869af78700 20 osd.26 243 _dispatch 0x16fdb6c0 pg_notify(0.138(20) epoch 243) v5 2014-06-09 16:22:11.564298 7f86a5e11700 10 _do_op 0x308caae0 seq 3182 r = 0, finisher 0xbdf13e0 0 2014-06-09 16:22:11.564302 7f86a5e11700 10 _finish_op 0x308caae0 seq 3182 osr(3.d1 0x3ed0560)/0x3ed0560 2014-06-09 16:22:11.564300 7f869af78700 5 -- op tracker -- , seq: 1979, time: 2014-06-09 16:22:11.304317, event: header_read, op: pg_notify(0.138(20) epoch 243) 2014-06-09 16:22:11.564304 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbb00 (1 active) 2014-06-09 16:22:11.564307 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa120 (2 active) 2014-06-09 16:22:11.564307 7f869af78700 5 -- op tracker -- , seq: 1979, time: 2014-06-09 16:22:11.304317, event: throttled, op: pg_notify(0.138(20) epoch 243) 2014-06-09 16:22:11.564309 7f86a5e11700 5 _do_op 0x306079e0 seq 3183 osr(2.f4 0x2ed0790)/0x2ed0790 start 2014-06-09 16:22:11.564311 7f86a5e11700 10 _do_transaction on 0xc121e00 2014-06-09 16:22:11.564312 7f869af78700 5 -- op tracker -- , seq: 1979, time: 2014-06-09 16:22:11.304344, event: all_read, op: pg_notify(0.138(20) epoch 243) 2014-06-09 16:22:11.564317 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564318 7f869af78700 5 -- op tracker -- , seq: 1979, time: 2014-06-09 16:22:11.564292, event: dispatched, op: pg_notify(0.138(20) epoch 243) 2014-06-09 16:22:11.564323 7f869af78700 5 -- op tracker -- , seq: 1979, time: 2014-06-09 16:22:11.564322, event: waiting_for_osdmap, op: pg_notify(0.138(20) epoch 243) 2014-06-09 16:22:11.564327 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.564329 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16fdb6c0 2014-06-09 16:22:11.564333 7f869af78700 5 -- op tracker -- , seq: 1979, time: 2014-06-09 16:22:11.564331, event: started, op: pg_notify(0.138(20) epoch 243) 2014-06-09 16:22:11.564348 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564348 7f869af78700 5 -- op tracker -- , seq: 1979, time: 2014-06-09 16:22:11.564346, event: done, op: pg_notify(0.138(20) epoch 243) 2014-06-09 16:22:11.564356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564357 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564363 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3183.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564368 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564373 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564375 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564376 7f869af78700 20 osd.26 243 _dispatch 0x1607ae00 pg_notify(2.5f4(21) epoch 243) v5 2014-06-09 16:22:11.564380 7f869af78700 5 -- op tracker -- , seq: 1980, time: 2014-06-09 16:22:11.225733, event: header_read, op: pg_notify(2.5f4(21) epoch 243) 2014-06-09 16:22:11.564387 7f869af78700 5 -- op tracker -- , seq: 1980, time: 2014-06-09 16:22:11.225734, event: throttled, op: pg_notify(2.5f4(21) epoch 243) 2014-06-09 16:22:11.564392 7f86a5e11700 10 _do_op 0x306079e0 seq 3183 r = 0, finisher 0xcc94240 0 2014-06-09 16:22:11.564393 7f869af78700 5 -- op tracker -- , seq: 1980, time: 2014-06-09 16:22:11.225760, event: all_read, op: pg_notify(2.5f4(21) epoch 243) 2014-06-09 16:22:11.564396 7f86a5e11700 10 _finish_op 0x306079e0 seq 3183 osr(2.f4 0x2ed0790)/0x2ed0790 2014-06-09 16:22:11.564399 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa120 (1 active) 2014-06-09 16:22:11.564399 7f869af78700 5 -- op tracker -- , seq: 1980, time: 2014-06-09 16:22:11.564373, event: dispatched, op: pg_notify(2.5f4(21) epoch 243) 2014-06-09 16:22:11.564401 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fac60 (2 active) 2014-06-09 16:22:11.564404 7f86a5e11700 5 _do_op 0x308caa80 seq 3184 osr(2.52e 0x2ed1600)/0x2ed1600 start 2014-06-09 16:22:11.564407 7f86a5e11700 10 _do_transaction on 0xc05a0c0 2014-06-09 16:22:11.564406 7f869af78700 5 -- op tracker -- , seq: 1980, time: 2014-06-09 16:22:11.564404, event: waiting_for_osdmap, op: pg_notify(2.5f4(21) epoch 243) 2014-06-09 16:22:11.564410 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.564412 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1607ae00 2014-06-09 16:22:11.564413 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564416 7f869af78700 5 -- op tracker -- , seq: 1980, time: 2014-06-09 16:22:11.564415, event: started, op: pg_notify(2.5f4(21) epoch 243) 2014-06-09 16:22:11.564431 7f869af78700 5 -- op tracker -- , seq: 1980, time: 2014-06-09 16:22:11.564429, event: done, op: pg_notify(2.5f4(21) epoch 243) 2014-06-09 16:22:11.564438 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564440 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564445 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564456 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564458 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564459 7f869af78700 20 osd.26 243 _dispatch 0x308e3a40 pg_notify(3.701(10) epoch 243) v5 2014-06-09 16:22:11.564461 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3184.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564466 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564463 7f869af78700 5 -- op tracker -- , seq: 1981, time: 2014-06-09 16:22:11.324775, event: header_read, op: pg_notify(3.701(10) epoch 243) 2014-06-09 16:22:11.564470 7f869af78700 5 -- op tracker -- , seq: 1981, time: 2014-06-09 16:22:11.324775, event: throttled, op: pg_notify(3.701(10) epoch 243) 2014-06-09 16:22:11.564476 7f869af78700 5 -- op tracker -- , seq: 1981, time: 2014-06-09 16:22:11.324797, event: all_read, op: pg_notify(3.701(10) epoch 243) 2014-06-09 16:22:11.564482 7f869af78700 5 -- op tracker -- , seq: 1981, time: 2014-06-09 16:22:11.564456, event: dispatched, op: pg_notify(3.701(10) epoch 243) 2014-06-09 16:22:11.564488 7f86a5e11700 10 _do_op 0x308caa80 seq 3184 r = 0, finisher 0x2d739540 0 2014-06-09 16:22:11.564489 7f869af78700 5 -- op tracker -- , seq: 1981, time: 2014-06-09 16:22:11.564487, event: waiting_for_osdmap, op: pg_notify(3.701(10) epoch 243) 2014-06-09 16:22:11.564491 7f86a5e11700 10 _finish_op 0x308caa80 seq 3184 osr(2.52e 0x2ed1600)/0x2ed1600 2014-06-09 16:22:11.564493 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.564494 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fac60 (1 active) 2014-06-09 16:22:11.564495 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e3a40 2014-06-09 16:22:11.564496 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524c60 (2 active) 2014-06-09 16:22:11.564498 7f86a5e11700 5 _do_op 0x30607560 seq 3185 osr(1.589 0x384da10)/0x384da10 start 2014-06-09 16:22:11.564501 7f86a5e11700 10 _do_transaction on 0xbf2e000 2014-06-09 16:22:11.564500 7f869af78700 5 -- op tracker -- , seq: 1981, time: 2014-06-09 16:22:11.564498, event: started, op: pg_notify(3.701(10) epoch 243) 2014-06-09 16:22:11.564507 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564518 7f869af78700 5 -- op tracker -- , seq: 1981, time: 2014-06-09 16:22:11.564516, event: done, op: pg_notify(3.701(10) epoch 243) 2014-06-09 16:22:11.564527 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564528 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564538 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564545 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564547 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564548 7f869af78700 20 osd.26 243 _dispatch 0x304f7200 pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.564553 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3185.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564558 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564555 7f869af78700 5 -- op tracker -- , seq: 1982, time: 2014-06-09 16:22:11.137357, event: header_read, op: pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564563 7f869af78700 5 -- op tracker -- , seq: 1982, time: 2014-06-09 16:22:11.137357, event: throttled, op: pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564570 7f869af78700 5 -- op tracker -- , seq: 1982, time: 2014-06-09 16:22:11.137374, event: all_read, op: pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564578 7f869af78700 5 -- op tracker -- , seq: 1982, time: 2014-06-09 16:22:11.564544, event: dispatched, op: pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564581 7f86a5e11700 10 _do_op 0x30607560 seq 3185 r = 0, finisher 0x15a3faa0 0 2014-06-09 16:22:11.564585 7f86a5e11700 10 _finish_op 0x30607560 seq 3185 osr(1.589 0x384da10)/0x384da10 2014-06-09 16:22:11.564587 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524c60 (1 active) 2014-06-09 16:22:11.564586 7f869af78700 5 -- op tracker -- , seq: 1982, time: 2014-06-09 16:22:11.564583, event: waiting_for_osdmap, op: pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564589 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3007e0 (2 active) 2014-06-09 16:22:11.564592 7f86a5e11700 5 _do_op 0x2fcae060 seq 3186 osr(2.d3 0x2ed0450)/0x2ed0450 start 2014-06-09 16:22:11.564591 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.564594 7f86a5e11700 10 _do_transaction on 0xc2bed80 2014-06-09 16:22:11.564595 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f7200 2014-06-09 16:22:11.564600 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564600 7f869af78700 5 -- op tracker -- , seq: 1982, time: 2014-06-09 16:22:11.564598, event: started, op: pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564613 7f869af78700 5 -- op tracker -- , seq: 1982, time: 2014-06-09 16:22:11.564610, event: done, op: pg_log(2.101 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564621 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564622 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564634 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564632 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564635 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564636 7f869af78700 20 osd.26 243 _dispatch 0x1c0396c0 pg_notify(1.3e2(20),3.6b4(10) epoch 243) v5 2014-06-09 16:22:11.564642 7f869af78700 5 -- op tracker -- , seq: 1983, time: 2014-06-09 16:22:11.132025, event: header_read, op: pg_notify(1.3e2(20),3.6b4(10) epoch 243) 2014-06-09 16:22:11.564648 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3186.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564650 7f869af78700 5 -- op tracker -- , seq: 1983, time: 2014-06-09 16:22:11.132025, event: throttled, op: pg_notify(1.3e2(20),3.6b4(10) epoch 243) 2014-06-09 16:22:11.564653 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564657 7f869af78700 5 -- op tracker -- , seq: 1983, time: 2014-06-09 16:22:11.132083, event: all_read, op: pg_notify(1.3e2(20),3.6b4(10) epoch 243) 2014-06-09 16:22:11.564663 7f869af78700 5 -- op tracker -- , seq: 1983, time: 2014-06-09 16:22:11.564633, event: dispatched, op: pg_notify(1.3e2(20),3.6b4(10) epoch 243) 2014-06-09 16:22:11.564670 7f869af78700 5 -- op tracker -- , seq: 1983, time: 2014-06-09 16:22:11.564668, event: waiting_for_osdmap, op: pg_notify(1.3e2(20),3.6b4(10) epoch 243) 2014-06-09 16:22:11.564672 7f86a5e11700 10 _do_op 0x2fcae060 seq 3186 r = 0, finisher 0xbd62340 0 2014-06-09 16:22:11.564674 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.564676 7f86a5e11700 10 _finish_op 0x2fcae060 seq 3186 osr(2.d3 0x2ed0450)/0x2ed0450 2014-06-09 16:22:11.564676 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c0396c0 2014-06-09 16:22:11.564678 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3007e0 (1 active) 2014-06-09 16:22:11.564681 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115440 (2 active) 2014-06-09 16:22:11.564683 7f86a5e11700 5 _do_op 0x300608a0 seq 3187 osr(1.32f 0x384d350)/0x384d350 start 2014-06-09 16:22:11.564682 7f869af78700 5 -- op tracker -- , seq: 1983, time: 2014-06-09 16:22:11.564680, event: started, op: pg_notify(1.3e2(20),3.6b4(10) epoch 243) 2014-06-09 16:22:11.564685 7f86a5e11700 10 _do_transaction on 0xbcb1140 2014-06-09 16:22:11.564691 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564707 7f869af78700 5 -- op tracker -- , seq: 1983, time: 2014-06-09 16:22:11.564704, event: done, op: pg_notify(1.3e2(20),3.6b4(10) epoch 243) 2014-06-09 16:22:11.564715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564717 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564723 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564739 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3187.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564741 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564743 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564743 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564744 7f869af78700 20 osd.26 243 _dispatch 0x3046b340 pg_info(1 pgs e243:3.271(2)) v4 2014-06-09 16:22:11.564749 7f869af78700 5 -- op tracker -- , seq: 1984, time: 2014-06-09 16:22:11.332045, event: header_read, op: pg_info(1 pgs e243:3.271(2)) 2014-06-09 16:22:11.564757 7f869af78700 5 -- op tracker -- , seq: 1984, time: 2014-06-09 16:22:11.332045, event: throttled, op: pg_info(1 pgs e243:3.271(2)) 2014-06-09 16:22:11.564764 7f869af78700 5 -- op tracker -- , seq: 1984, time: 2014-06-09 16:22:11.332058, event: all_read, op: pg_info(1 pgs e243:3.271(2)) 2014-06-09 16:22:11.564768 7f86a5e11700 10 _do_op 0x300608a0 seq 3187 r = 0, finisher 0x170018e0 0 2014-06-09 16:22:11.564772 7f86a5e11700 10 _finish_op 0x300608a0 seq 3187 osr(1.32f 0x384d350)/0x384d350 2014-06-09 16:22:11.564770 7f869af78700 5 -- op tracker -- , seq: 1984, time: 2014-06-09 16:22:11.564741, event: dispatched, op: pg_info(1 pgs e243:3.271(2)) 2014-06-09 16:22:11.564774 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115440 (1 active) 2014-06-09 16:22:11.564777 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3587e0 (2 active) 2014-06-09 16:22:11.564779 7f86a5e11700 5 _do_op 0x309a2600 seq 3188 osr(1.a7 0x384c730)/0x384c730 start 2014-06-09 16:22:11.564778 7f869af78700 5 -- op tracker -- , seq: 1984, time: 2014-06-09 16:22:11.564776, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.271(2)) 2014-06-09 16:22:11.564782 7f86a5e11700 10 _do_transaction on 0x4dcc6c0 2014-06-09 16:22:11.564782 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.271(2)) v4 from osd.7 2014-06-09 16:22:11.564786 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046b340 2014-06-09 16:22:11.564788 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564791 7f869af78700 5 -- op tracker -- , seq: 1984, time: 2014-06-09 16:22:11.564789, event: started, op: pg_info(1 pgs e243:3.271(2)) 2014-06-09 16:22:11.564807 7f869af78700 5 -- op tracker -- , seq: 1984, time: 2014-06-09 16:22:11.564805, event: done, op: pg_info(1 pgs e243:3.271(2)) 2014-06-09 16:22:11.564815 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564817 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564819 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564830 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564832 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564833 7f869af78700 20 osd.26 243 _dispatch 0x30613e00 pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.564835 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3188.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564839 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564840 7f869af78700 5 -- op tracker -- , seq: 1985, time: 2014-06-09 16:22:11.293023, event: header_read, op: pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564847 7f869af78700 5 -- op tracker -- , seq: 1985, time: 2014-06-09 16:22:11.293024, event: throttled, op: pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564854 7f869af78700 5 -- op tracker -- , seq: 1985, time: 2014-06-09 16:22:11.293031, event: all_read, op: pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564857 7f86a5e11700 10 _do_op 0x309a2600 seq 3188 r = 0, finisher 0x3061f020 0 2014-06-09 16:22:11.564861 7f86a5e11700 10 _finish_op 0x309a2600 seq 3188 osr(1.a7 0x384c730)/0x384c730 2014-06-09 16:22:11.564861 7f869af78700 5 -- op tracker -- , seq: 1985, time: 2014-06-09 16:22:11.564830, event: dispatched, op: pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564864 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3587e0 (1 active) 2014-06-09 16:22:11.564866 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301d40 (2 active) 2014-06-09 16:22:11.564868 7f86a5e11700 5 _do_op 0xd4711a0 seq 3189 osr(0.6d2 0x384c1f0)/0x384c1f0 start 2014-06-09 16:22:11.564871 7f86a5e11700 10 _do_transaction on 0xbf37a40 2014-06-09 16:22:11.564869 7f869af78700 5 -- op tracker -- , seq: 1985, time: 2014-06-09 16:22:11.564866, event: waiting_for_osdmap, op: pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564874 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.564877 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.564878 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30613e00 2014-06-09 16:22:11.564884 7f869af78700 5 -- op tracker -- , seq: 1985, time: 2014-06-09 16:22:11.564881, event: started, op: pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564897 7f869af78700 5 -- op tracker -- , seq: 1985, time: 2014-06-09 16:22:11.564894, event: done, op: pg_log(1.18a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.564905 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564907 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564908 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.564921 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.564922 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564923 7f869af78700 20 osd.26 243 _dispatch 0x240ea380 pg_notify(3.75f(12),1.764(21) epoch 243) v5 2014-06-09 16:22:11.564925 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3189.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.564929 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.564929 7f869af78700 5 -- op tracker -- , seq: 1986, time: 2014-06-09 16:22:11.211057, event: header_read, op: pg_notify(3.75f(12),1.764(21) epoch 243) 2014-06-09 16:22:11.564937 7f869af78700 5 -- op tracker -- , seq: 1986, time: 2014-06-09 16:22:11.211058, event: throttled, op: pg_notify(3.75f(12),1.764(21) epoch 243) 2014-06-09 16:22:11.564944 7f869af78700 5 -- op tracker -- , seq: 1986, time: 2014-06-09 16:22:11.211102, event: all_read, op: pg_notify(3.75f(12),1.764(21) epoch 243) 2014-06-09 16:22:11.564951 7f869af78700 5 -- op tracker -- , seq: 1986, time: 2014-06-09 16:22:11.564920, event: dispatched, op: pg_notify(3.75f(12),1.764(21) epoch 243) 2014-06-09 16:22:11.564953 7f86a5e11700 10 _do_op 0xd4711a0 seq 3189 r = 0, finisher 0x305099e0 0 2014-06-09 16:22:11.564957 7f86a5e11700 10 _finish_op 0xd4711a0 seq 3189 osr(0.6d2 0x384c1f0)/0x384c1f0 2014-06-09 16:22:11.564958 7f869af78700 5 -- op tracker -- , seq: 1986, time: 2014-06-09 16:22:11.564955, event: waiting_for_osdmap, op: pg_notify(3.75f(12),1.764(21) epoch 243) 2014-06-09 16:22:11.564960 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301d40 (1 active) 2014-06-09 16:22:11.564963 7f86a5e11700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.564963 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.564965 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ea380 2014-06-09 16:22:11.564970 7f869af78700 5 -- op tracker -- , seq: 1986, time: 2014-06-09 16:22:11.564968, event: started, op: pg_notify(3.75f(12),1.764(21) epoch 243) 2014-06-09 16:22:11.564975 7f8691765700 10 op_submit_finish 3190 2014-06-09 16:22:11.564983 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.564997 7f869af78700 5 -- op tracker -- , seq: 1986, time: 2014-06-09 16:22:11.564994, event: done, op: pg_notify(3.75f(12),1.764(21) epoch 243) 2014-06-09 16:22:11.565006 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565008 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.564998 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.36d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.565012 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.565017 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.565028 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565030 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565022 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.173257 7 0.051480 2014-06-09 16:22:11.565032 7f869af78700 20 osd.26 243 _dispatch 0x308e2380 pg_info(1 pgs e243:2.342) v4 2014-06-09 16:22:11.565037 7f869af78700 5 -- op tracker -- , seq: 1987, time: 2014-06-09 16:22:11.294603, event: header_read, op: pg_info(1 pgs e243:2.342) 2014-06-09 16:22:11.565036 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.565044 7f869af78700 5 -- op tracker -- , seq: 1987, time: 2014-06-09 16:22:11.294613, event: throttled, op: pg_info(1 pgs e243:2.342) 2014-06-09 16:22:11.565051 7f869af78700 5 -- op tracker -- , seq: 1987, time: 2014-06-09 16:22:11.294623, event: all_read, op: pg_info(1 pgs e243:2.342) 2014-06-09 16:22:11.565048 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.565058 7f869af78700 5 -- op tracker -- , seq: 1987, time: 2014-06-09 16:22:11.565028, event: dispatched, op: pg_info(1 pgs e243:2.342) 2014-06-09 16:22:11.565060 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.565065 7f869af78700 5 -- op tracker -- , seq: 1987, time: 2014-06-09 16:22:11.565063, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.342) 2014-06-09 16:22:11.565069 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.342) v4 from osd.9 2014-06-09 16:22:11.565072 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e2380 2014-06-09 16:22:11.565070 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.565077 7f869af78700 5 -- op tracker -- , seq: 1987, time: 2014-06-09 16:22:11.565075, event: started, op: pg_info(1 pgs e243:2.342) 2014-06-09 16:22:11.565080 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.565093 7f869af78700 5 -- op tracker -- , seq: 1987, time: 2014-06-09 16:22:11.565091, event: done, op: pg_info(1 pgs e243:2.342) 2014-06-09 16:22:11.565104 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565106 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565119 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565120 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565121 7f869af78700 20 osd.26 243 _dispatch 0xcc74540 pg_info(2 pgs e243:3.5f8(2),3.400(2)) v4 2014-06-09 16:22:11.565127 7f869af78700 5 -- op tracker -- , seq: 1988, time: 2014-06-09 16:22:11.314035, event: header_read, op: pg_info(2 pgs e243:3.5f8(2),3.400(2)) 2014-06-09 16:22:11.565131 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565135 7f869af78700 5 -- op tracker -- , seq: 1988, time: 2014-06-09 16:22:11.314036, event: throttled, op: pg_info(2 pgs e243:3.5f8(2),3.400(2)) 2014-06-09 16:22:11.565138 7f8691765700 5 queue_transactions existing osr(0.36d 0x2bd6fc0)/0x2bd6fc0 2014-06-09 16:22:11.565141 7f8691765700 10 op_submit_start 3191 2014-06-09 16:22:11.565142 7f8691765700 5 queue_transactions (trailing journal) 3191 2014-06-09 16:22:11.565143 7f869af78700 5 -- op tracker -- , seq: 1988, time: 2014-06-09 16:22:11.314057, event: all_read, op: pg_info(2 pgs e243:3.5f8(2),3.400(2)) 2014-06-09 16:22:11.565144 7f8691765700 5 queue_op 0x30607920 seq 3191 osr(0.36d 0x2bd6fc0) 1876 bytes (queue has 22 ops and 40459 bytes) 2014-06-09 16:22:11.565147 7f8691765700 10 op_submit_finish 3191 2014-06-09 16:22:11.565150 7f869af78700 5 -- op tracker -- , seq: 1988, time: 2014-06-09 16:22:11.565118, event: dispatched, op: pg_info(2 pgs e243:3.5f8(2),3.400(2)) 2014-06-09 16:22:11.565158 7f869af78700 5 -- op tracker -- , seq: 1988, time: 2014-06-09 16:22:11.565155, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.5f8(2),3.400(2)) 2014-06-09 16:22:11.565150 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=208 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.565162 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.5f8(2),3.400(2)) v4 from osd.8 2014-06-09 16:22:11.565166 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcc74540 2014-06-09 16:22:11.565172 7f869af78700 5 -- op tracker -- , seq: 1988, time: 2014-06-09 16:22:11.565169, event: started, op: pg_info(2 pgs e243:3.5f8(2),3.400(2)) 2014-06-09 16:22:11.565164 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=208 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.565177 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=208 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.374978 7 0.009255 2014-06-09 16:22:11.565189 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=208 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.565197 7f869af78700 5 -- op tracker -- , seq: 1988, time: 2014-06-09 16:22:11.565194, event: done, op: pg_info(2 pgs e243:3.5f8(2),3.400(2)) 2014-06-09 16:22:11.565205 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565207 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565202 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=208 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.565220 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565213 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=208 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.565222 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565224 7f869af78700 20 osd.26 243 _dispatch 0x150e0c40 pg_info(1 pgs e243:3.13c(2)) v4 2014-06-09 16:22:11.565224 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'617 -> 213'617 2014-06-09 16:22:11.565229 7f869af78700 5 -- op tracker -- , seq: 1989, time: 2014-06-09 16:22:11.304404, event: header_read, op: pg_info(1 pgs e243:3.13c(2)) 2014-06-09 16:22:11.565236 7f869af78700 5 -- op tracker -- , seq: 1989, time: 2014-06-09 16:22:11.304405, event: throttled, op: pg_info(1 pgs e243:3.13c(2)) 2014-06-09 16:22:11.565242 7f869af78700 5 -- op tracker -- , seq: 1989, time: 2014-06-09 16:22:11.304421, event: all_read, op: pg_info(1 pgs e243:3.13c(2)) 2014-06-09 16:22:11.565248 7f869af78700 5 -- op tracker -- , seq: 1989, time: 2014-06-09 16:22:11.565220, event: dispatched, op: pg_info(1 pgs e243:3.13c(2)) 2014-06-09 16:22:11.565256 7f869af78700 5 -- op tracker -- , seq: 1989, time: 2014-06-09 16:22:11.565253, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.13c(2)) 2014-06-09 16:22:11.565259 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.13c(2)) v4 from osd.6 2014-06-09 16:22:11.565263 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e0c40 2014-06-09 16:22:11.565263 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.565269 7f869af78700 5 -- op tracker -- , seq: 1989, time: 2014-06-09 16:22:11.565266, event: started, op: pg_info(1 pgs e243:3.13c(2)) 2014-06-09 16:22:11.565286 7f869af78700 5 -- op tracker -- , seq: 1989, time: 2014-06-09 16:22:11.565283, event: done, op: pg_info(1 pgs e243:3.13c(2)) 2014-06-09 16:22:11.565293 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565295 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565306 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565310 7f8691765700 5 queue_transactions existing osr(3.7a 0x3ed0220)/0x3ed0220 2014-06-09 16:22:11.565311 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565313 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565313 7f8691765700 10 op_submit_start 3192 2014-06-09 16:22:11.565314 7f8691765700 5 queue_transactions (trailing journal) 3192 2014-06-09 16:22:11.565314 7f869af78700 20 osd.26 243 _dispatch 0x3070d200 pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.565315 7f8691765700 5 queue_op 0x306078c0 seq 3192 osr(3.7a 0x3ed0220) 1332 bytes (queue has 23 ops and 41791 bytes) 2014-06-09 16:22:11.565318 7f8691765700 10 op_submit_finish 3192 2014-06-09 16:22:11.565321 7f869af78700 5 -- op tracker -- , seq: 1990, time: 2014-06-09 16:22:11.229742, event: header_read, op: pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.565321 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:2.65c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.565329 7f869af78700 5 -- op tracker -- , seq: 1990, time: 2014-06-09 16:22:11.229742, event: throttled, op: pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.565337 7f869af78700 5 -- op tracker -- , seq: 1990, time: 2014-06-09 16:22:11.229760, event: all_read, op: pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.565335 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.565347 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565343 7f869af78700 5 -- op tracker -- , seq: 1990, time: 2014-06-09 16:22:11.565310, event: dispatched, op: pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.565352 7f869af78700 5 -- op tracker -- , seq: 1990, time: 2014-06-09 16:22:11.565350, event: waiting_for_osdmap, op: pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.565357 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.565353 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.565362 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070d200 2014-06-09 16:22:11.565367 7f869af78700 5 -- op tracker -- , seq: 1990, time: 2014-06-09 16:22:11.565365, event: started, op: pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.565363 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.186( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.565374 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.565377 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.565381 7f869af78700 5 -- op tracker -- , seq: 1990, time: 2014-06-09 16:22:11.565378, event: done, op: pg_log(0.1d4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.565380 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.186( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.886558 7 0.000178 2014-06-09 16:22:11.565389 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565392 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565390 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.186( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.565403 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565405 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565399 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.186( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.565406 7f869af78700 20 osd.26 243 _dispatch 0x3005bdc0 pg_notify(2.446(20) epoch 243) v5 2014-06-09 16:22:11.565408 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.565411 7f869af78700 5 -- op tracker -- , seq: 1991, time: 2014-06-09 16:22:11.326193, event: header_read, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.565419 7f869af78700 5 -- op tracker -- , seq: 1991, time: 2014-06-09 16:22:11.326193, event: throttled, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.565417 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.565426 7f869af78700 5 -- op tracker -- , seq: 1991, time: 2014-06-09 16:22:11.326218, event: all_read, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.565425 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.565432 7f869af78700 5 -- op tracker -- , seq: 1991, time: 2014-06-09 16:22:11.565403, event: dispatched, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.565439 7f869af78700 5 -- op tracker -- , seq: 1991, time: 2014-06-09 16:22:11.565437, event: waiting_for_osdmap, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.565444 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.565446 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005bdc0 2014-06-09 16:22:11.565450 7f869af78700 5 -- op tracker -- , seq: 1991, time: 2014-06-09 16:22:11.565448, event: started, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.565466 7f869af78700 5 -- op tracker -- , seq: 1991, time: 2014-06-09 16:22:11.565463, event: done, op: pg_notify(2.446(20) epoch 243) 2014-06-09 16:22:11.565473 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565473 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565475 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565477 7f8691765700 5 queue_transactions existing osr(0.186 0x2bd67d0)/0x2bd67d0 2014-06-09 16:22:11.565480 7f8691765700 10 op_submit_start 3193 2014-06-09 16:22:11.565481 7f8691765700 5 queue_transactions (trailing journal) 3193 2014-06-09 16:22:11.565482 7f8691765700 5 queue_op 0x30607860 seq 3193 osr(0.186 0x2bd67d0) 1911 bytes (queue has 24 ops and 43702 bytes) 2014-06-09 16:22:11.565485 7f8691765700 10 op_submit_finish 3193 2014-06-09 16:22:11.565494 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565496 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565488 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.1f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.565498 7f869af78700 20 osd.26 243 _dispatch 0x1abd41c0 pg_info(2 pgs e243:3.5ce(4),3.105(4)) v4 2014-06-09 16:22:11.565505 7f869af78700 5 -- op tracker -- , seq: 1992, time: 2014-06-09 16:22:11.137381, event: header_read, op: pg_info(2 pgs e243:3.5ce(4),3.105(4)) 2014-06-09 16:22:11.565501 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.565512 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565512 7f869af78700 5 -- op tracker -- , seq: 1992, time: 2014-06-09 16:22:11.137381, event: throttled, op: pg_info(2 pgs e243:3.5ce(4),3.105(4)) 2014-06-09 16:22:11.565520 7f869af78700 5 -- op tracker -- , seq: 1992, time: 2014-06-09 16:22:11.137403, event: all_read, op: pg_info(2 pgs e243:3.5ce(4),3.105(4)) 2014-06-09 16:22:11.565517 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.565527 7f869af78700 5 -- op tracker -- , seq: 1992, time: 2014-06-09 16:22:11.565494, event: dispatched, op: pg_info(2 pgs e243:3.5ce(4),3.105(4)) 2014-06-09 16:22:11.565528 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 2.5d9( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.565535 7f869af78700 5 -- op tracker -- , seq: 1992, time: 2014-06-09 16:22:11.565532, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.5ce(4),3.105(4)) 2014-06-09 16:22:11.565541 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.565541 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.5ce(4),3.105(4)) v4 from osd.2 2014-06-09 16:22:11.565545 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd41c0 2014-06-09 16:22:11.565545 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.565550 7f869af78700 5 -- op tracker -- , seq: 1992, time: 2014-06-09 16:22:11.565547, event: started, op: pg_info(2 pgs e243:3.5ce(4),3.105(4)) 2014-06-09 16:22:11.565548 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.733419 7 0.000194 2014-06-09 16:22:11.565559 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.565569 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.565574 7f869af78700 5 -- op tracker -- , seq: 1992, time: 2014-06-09 16:22:11.565571, event: done, op: pg_info(2 pgs e243:3.5ce(4),3.105(4)) 2014-06-09 16:22:11.565583 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565585 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565579 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.565588 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.565600 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565602 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565596 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.565604 7f869af78700 20 osd.26 243 _dispatch 0x154581c0 pg_notify(2.3a3(20) epoch 243) v5 2014-06-09 16:22:11.565609 7f869af78700 5 -- op tracker -- , seq: 1993, time: 2014-06-09 16:22:11.132417, event: header_read, op: pg_notify(2.3a3(20) epoch 243) 2014-06-09 16:22:11.565616 7f869af78700 5 -- op tracker -- , seq: 1993, time: 2014-06-09 16:22:11.132418, event: throttled, op: pg_notify(2.3a3(20) epoch 243) 2014-06-09 16:22:11.565622 7f869af78700 5 -- op tracker -- , seq: 1993, time: 2014-06-09 16:22:11.132446, event: all_read, op: pg_notify(2.3a3(20) epoch 243) 2014-06-09 16:22:11.565628 7f869af78700 5 -- op tracker -- , seq: 1993, time: 2014-06-09 16:22:11.565600, event: dispatched, op: pg_notify(2.3a3(20) epoch 243) 2014-06-09 16:22:11.565635 7f869af78700 5 -- op tracker -- , seq: 1993, time: 2014-06-09 16:22:11.565633, event: waiting_for_osdmap, op: pg_notify(2.3a3(20) epoch 243) 2014-06-09 16:22:11.565639 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.565641 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x154581c0 2014-06-09 16:22:11.565644 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565646 7f869af78700 5 -- op tracker -- , seq: 1993, time: 2014-06-09 16:22:11.565644, event: started, op: pg_notify(2.3a3(20) epoch 243) 2014-06-09 16:22:11.565649 7f8691765700 5 queue_transactions existing osr(2.5d9 0x2ed1a70)/0x2ed1a70 2014-06-09 16:22:11.565651 7f8691765700 10 op_submit_start 3194 2014-06-09 16:22:11.565652 7f8691765700 5 queue_transactions (trailing journal) 3194 2014-06-09 16:22:11.565654 7f8691765700 5 queue_op 0x30607800 seq 3194 osr(2.5d9 0x2ed1a70) 2009 bytes (queue has 25 ops and 45711 bytes) 2014-06-09 16:22:11.565657 7f8691765700 10 op_submit_finish 3194 2014-06-09 16:22:11.565662 7f869af78700 5 -- op tracker -- , seq: 1993, time: 2014-06-09 16:22:11.565660, event: done, op: pg_notify(2.3a3(20) epoch 243) 2014-06-09 16:22:11.565670 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565660 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.f6( v 213'683 (0'0,213'683] local-les=201 n=127 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.565672 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565674 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.565686 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565690 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.565692 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.565693 7f869af78700 20 osd.26 243 _dispatch 0x1d039340 pg_info(2 pgs e243:3.461(2),3.468(2)) v4 2014-06-09 16:22:11.565690 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:0.686( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.565699 7f869af78700 5 -- op tracker -- , seq: 1994, time: 2014-06-09 16:22:11.335624, event: header_read, op: pg_info(2 pgs e243:3.461(2),3.468(2)) 2014-06-09 16:22:11.565702 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.565707 7f869af78700 5 -- op tracker -- , seq: 1994, time: 2014-06-09 16:22:11.335625, event: throttled, op: pg_info(2 pgs e243:3.461(2),3.468(2)) 2014-06-09 16:22:11.565712 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565715 7f869af78700 5 -- op tracker -- , seq: 1994, time: 2014-06-09 16:22:11.335650, event: all_read, op: pg_info(2 pgs e243:3.461(2),3.468(2)) 2014-06-09 16:22:11.565722 7f869af78700 5 -- op tracker -- , seq: 1994, time: 2014-06-09 16:22:11.565690, event: dispatched, op: pg_info(2 pgs e243:3.461(2),3.468(2)) 2014-06-09 16:22:11.565717 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:2.3bc( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.565729 7f869af78700 5 -- op tracker -- , seq: 1994, time: 2014-06-09 16:22:11.565727, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.461(2),3.468(2)) 2014-06-09 16:22:11.565733 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.461(2),3.468(2)) v4 from osd.7 2014-06-09 16:22:11.565729 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.565737 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d039340 2014-06-09 16:22:11.565739 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565742 7f869af78700 5 -- op tracker -- , seq: 1994, time: 2014-06-09 16:22:11.565740, event: started, op: pg_info(2 pgs e243:3.461(2),3.468(2)) 2014-06-09 16:22:11.565759 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0230 (19 active) 2014-06-09 16:22:11.565779 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0230 (20 active) 2014-06-09 16:22:11.565786 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.565798 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.220( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.565810 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.565814 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.565817 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.220( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.791630 7 0.000210 2014-06-09 16:22:11.565827 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.220( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.565838 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.220( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.565847 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.565856 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.565867 7f86a5610700 10 _finish_op 0x2fc8ca20 seq 3141 osr(3.609 0x3ed1b00)/0x3ed1b00 2014-06-09 16:22:11.565864 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.565872 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcfc0 (0 active) 2014-06-09 16:22:11.565875 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f7a0 (1 active) 2014-06-09 16:22:11.565880 7f86a5610700 5 _do_op 0x30607980 seq 3190 osr(2.501 0x2ed1690)/0x2ed1690 start 2014-06-09 16:22:11.565884 7f86a5610700 10 _do_transaction on 0xc29e480 2014-06-09 16:22:11.565891 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.565911 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.565918 7f868b759700 10 op_submit_start 3195 2014-06-09 16:22:11.565920 7f868b759700 5 queue_transactions (trailing journal) 3195 2014-06-09 16:22:11.565921 7f868b759700 5 queue_op 0x308caa20 seq 3195 osr(3.7ae 0xb7f01d0) 1429 bytes (queue has 24 ops and 44376 bytes) 2014-06-09 16:22:11.565925 7f868b759700 10 op_submit_finish 3195 2014-06-09 16:22:11.565931 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=191 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.565942 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.565945 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=191 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.565963 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3190.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.565968 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.565959 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=191 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.401846 7 0.000180 2014-06-09 16:22:11.565974 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=191 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.565986 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=191 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.565997 7f86a5610700 10 _do_op 0x30607980 seq 3190 r = 0, finisher 0xbd5aea0 0 2014-06-09 16:22:11.566001 7f86a5610700 10 _finish_op 0x30607980 seq 3190 osr(2.501 0x2ed1690)/0x2ed1690 2014-06-09 16:22:11.565997 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=191 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.566005 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f7a0 (0 active) 2014-06-09 16:22:11.566007 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3397a0 (1 active) 2014-06-09 16:22:11.566009 7f86a5610700 5 _do_op 0x30607920 seq 3191 osr(0.36d 0x2bd6fc0)/0x2bd6fc0 start 2014-06-09 16:22:11.566012 7f86a5610700 10 _do_transaction on 0xc33ab40 2014-06-09 16:22:11.566008 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'570 -> 213'570 2014-06-09 16:22:11.566017 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.566049 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.566045 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.566067 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3191.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.566079 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.566089 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.566109 7f86a5610700 10 _do_op 0x30607920 seq 3191 r = 0, finisher 0xbd78880 0 2014-06-09 16:22:11.566117 7f86a5610700 10 _finish_op 0x30607920 seq 3191 osr(0.36d 0x2bd6fc0)/0x2bd6fc0 2014-06-09 16:22:11.566120 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3397a0 (0 active) 2014-06-09 16:22:11.566122 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcd80 (1 active) 2014-06-09 16:22:11.566124 7f86a5610700 5 _do_op 0x306078c0 seq 3192 osr(3.7a 0x3ed0220)/0x3ed0220 start 2014-06-09 16:22:11.566126 7f86a5610700 10 _do_transaction on 0xc12f200 2014-06-09 16:22:11.566139 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.566157 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc40dea0,0xbd75640,0xc40c500,0xbd5ba80,0x33a6620,0x19941fe0,0xc2d73e0,0x189aa280,0xc410940,0xc5a7840,0x4eb90c0,0x3062b0a0,0x292823c0,0xbd5a2e0,0x20051da0,0xbcc8280,0xc2d6b20] 2014-06-09 16:22:11.566169 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6b5a0,0xbd75420,0x189aa140,0xbcc8340,0xbef02a0,0xbd6fb80,0xbdf1d00,0xbdf13e0,0xcc94240,0x2d739540,0x15a3faa0,0xbd62340,0x170018e0,0x3061f020,0x305099e0,0x3062a960,0xbd5aea0,0xbd78880] 2014-06-09 16:22:11.571810 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.571821 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc40dea0,0xbd75640,0xc40c500,0xbd5ba80,0x33a6620,0x19941fe0,0xc2d73e0,0x189aa280,0xc410940,0xc5a7840,0x4eb90c0,0x3062b0a0,0x292823c0,0xbd5a2e0,0x20051da0,0xbcc8280,0xc2d6b20] 2014-06-09 16:22:11.571827 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.571828 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.571832 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.571848 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.571859 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.571867 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.571876 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.571894 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.571907 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.571917 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.571931 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.571943 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.571953 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.571962 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.571969 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.571978 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.571991 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572003 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.572014 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.572023 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572030 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.572040 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.572054 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572067 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.572077 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.572088 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572096 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.572103 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.572112 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572119 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.572127 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.572142 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572154 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.572163 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.572178 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572189 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.572198 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.572212 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572224 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.572234 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.572248 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572260 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.572270 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.572278 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572285 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.572293 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.572306 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572319 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.572329 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.572345 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572357 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.572367 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] flushed 2014-06-09 16:22:11.572380 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.572390 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.572401 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.572413 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6b5a0,0xbd75420,0x189aa140,0xbcc8340,0xbef02a0,0xbd6fb80,0xbdf1d00,0xbdf13e0,0xcc94240,0x2d739540,0x15a3faa0,0xbd62340,0x170018e0,0x3061f020,0x305099e0,0x3062a960,0xbd5aea0,0xbd78880] 2014-06-09 16:22:11.572415 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.572416 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.566176 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e240 (2 active) 2014-06-09 16:22:11.566175 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.572422 7f86a5e11700 5 _do_op 0x30607860 seq 3193 osr(0.186 0x2bd67d0)/0x2bd67d0 start 2014-06-09 16:22:11.572426 7f86a5e11700 10 _do_transaction on 0xc2a0540 2014-06-09 16:22:11.572433 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.566182 7f8690763700 10 op_submit_start 3196 2014-06-09 16:22:11.572445 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3192.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.572450 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.572449 7f8690763700 5 queue_transactions (trailing journal) 3196 2014-06-09 16:22:11.572453 7f8690763700 5 queue_op 0x2fcaf140 seq 3196 osr(2.69d 0x2ed1ef0) 1883 bytes (queue has 22 ops and 40448 bytes) 2014-06-09 16:22:11.572457 7f8690763700 10 op_submit_finish 3196 2014-06-09 16:22:11.572464 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:0.6ab( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.566190 7f8691765700 5 queue_transactions existing osr(2.220 0x2ed0b10)/0x2ed0b10 2014-06-09 16:22:11.572479 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.572490 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.572491 7f8691765700 10 op_submit_start 3197 2014-06-09 16:22:11.572490 7f86a5610700 10 _do_op 0x306078c0 seq 3192 r = 0, finisher 0xbd69c00 0 2014-06-09 16:22:11.572492 7f8691765700 5 queue_transactions (trailing journal) 3197 2014-06-09 16:22:11.572494 7f8691765700 5 queue_op 0x30607740 seq 3197 osr(2.220 0x2ed0b10) 1911 bytes (queue has 23 ops and 42359 bytes) 2014-06-09 16:22:11.572496 7f86a5610700 10 _finish_op 0x306078c0 seq 3192 osr(3.7a 0x3ed0220)/0x3ed0220 2014-06-09 16:22:11.566196 7f868b759700 5 queue_transactions existing osr(3.7bc 0xb7f0190)/0xb7f0190 2014-06-09 16:22:11.572501 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcd80 (1 active) 2014-06-09 16:22:11.572503 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd69c00] 2014-06-09 16:22:11.572504 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653560 (2 active) 2014-06-09 16:22:11.572495 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.572507 7f86a5610700 5 _do_op 0x30607800 seq 3194 osr(2.5d9 0x2ed1a70)/0x2ed1a70 start 2014-06-09 16:22:11.572508 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd69c00] 2014-06-09 16:22:11.572509 7f86a5610700 10 _do_transaction on 0x2f29c80 2014-06-09 16:22:11.572510 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.572511 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.572514 7f8691765700 10 op_submit_finish 3197 2014-06-09 16:22:11.572516 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.572509 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.572521 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.572519 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.2b5( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.572525 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.572533 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.572543 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.572536 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.7a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.572549 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.572550 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.572553 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.572548 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.572557 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.381594 7 0.000210 2014-06-09 16:22:11.572565 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3194.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.572569 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.572559 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 1.11d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.572570 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.572574 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.572568 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.572577 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.856100 7 0.000216 2014-06-09 16:22:11.572580 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.572587 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.572590 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.572602 7f86a5610700 10 _do_op 0x30607800 seq 3194 r = 0, finisher 0xbce6460 0 2014-06-09 16:22:11.572597 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.572605 7f86a5610700 10 _finish_op 0x30607800 seq 3194 osr(2.5d9 0x2ed1a70)/0x2ed1a70 2014-06-09 16:22:11.572599 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.572610 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653560 (1 active) 2014-06-09 16:22:11.572606 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.572612 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f8c0 (2 active) 2014-06-09 16:22:11.572608 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.572615 7f86a5610700 5 _do_op 0x308caa20 seq 3195 osr(3.7ae 0xb7f01d0)/0xb7f01d0 start 2014-06-09 16:22:11.572617 7f86a5610700 10 _do_transaction on 0xbebb140 2014-06-09 16:22:11.572615 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.572623 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.572622 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.572656 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.572661 7f8690763700 5 queue_transactions existing osr(1.7a 0x384c510)/0x384c510 2014-06-09 16:22:11.572664 7f8690763700 10 op_submit_start 3198 2014-06-09 16:22:11.572665 7f8690763700 5 queue_transactions (trailing journal) 3198 2014-06-09 16:22:11.572664 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.572666 7f8690763700 5 queue_op 0x2fcaf0e0 seq 3198 osr(1.7a 0x384c510) 1971 bytes (queue has 23 ops and 42418 bytes) 2014-06-09 16:22:11.572669 7f8690763700 10 op_submit_finish 3198 2014-06-09 16:22:11.572670 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.572675 7f8691765700 5 queue_transactions existing osr(1.11d 0x384c8b0)/0x384c8b0 2014-06-09 16:22:11.572678 7f8691765700 10 op_submit_start 3199 2014-06-09 16:22:11.572679 7f8691765700 5 queue_transactions (trailing journal) 3199 2014-06-09 16:22:11.572680 7f8691765700 5 queue_op 0x306076e0 seq 3199 osr(1.11d 0x384c8b0) 1954 bytes (queue has 24 ops and 44372 bytes) 2014-06-09 16:22:11.572680 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3195.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.572683 7f8691765700 10 op_submit_finish 3199 2014-06-09 16:22:11.572684 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.572689 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (19 active) 2014-06-09 16:22:11.572685 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.572699 7f86a5610700 10 _do_op 0x308caa20 seq 3195 r = 0, finisher 0x170019a0 0 2014-06-09 16:22:11.572702 7f86a5610700 10 _finish_op 0x308caa20 seq 3195 osr(3.7ae 0xb7f01d0)/0xb7f01d0 2014-06-09 16:22:11.572704 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f8c0 (1 active) 2014-06-09 16:22:11.572696 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.6b0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.572706 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338b40 (2 active) 2014-06-09 16:22:11.572708 7f86a5610700 5 _do_op 0x2fcaf140 seq 3196 osr(2.69d 0x2ed1ef0)/0x2ed1ef0 start 2014-06-09 16:22:11.572707 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.572710 7f86a5610700 10 _do_transaction on 0xbebd080 2014-06-09 16:22:11.572709 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0250 (20 active) 2014-06-09 16:22:11.572711 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.572715 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.572714 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.234593 7 0.000174 2014-06-09 16:22:11.572713 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.572724 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.572725 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.572735 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.572734 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.572746 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.572744 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.572740 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.572752 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.572761 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3196.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.572764 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.572755 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x10c54f00,0x11e63200,0x11e63100,0x11e63000,0x11e63600,0x11e63500,0x11e63400,0x257c4900,0x257c4800,0x257c4700,0x257c4600 2014-06-09 16:22:11.572760 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.572781 7f86a5610700 10 _do_op 0x2fcaf140 seq 3196 r = 0, finisher 0x33a65c0 0 2014-06-09 16:22:11.572784 7f86a5610700 10 _finish_op 0x2fcaf140 seq 3196 osr(2.69d 0x2ed1ef0)/0x2ed1ef0 2014-06-09 16:22:11.572786 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338b40 (1 active) 2014-06-09 16:22:11.572788 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497d40 (2 active) 2014-06-09 16:22:11.572790 7f86a5610700 5 _do_op 0x30607740 seq 3197 osr(2.220 0x2ed0b10)/0x2ed0b10 start 2014-06-09 16:22:11.572792 7f86a5610700 10 _do_transaction on 0xc22bec0 2014-06-09 16:22:11.572797 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.572797 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.572802 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.572806 7f8691765700 5 queue_transactions existing osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 2014-06-09 16:22:11.572809 7f8691765700 10 op_submit_start 3200 2014-06-09 16:22:11.572810 7f8691765700 5 queue_transactions (trailing journal) 3200 2014-06-09 16:22:11.572802 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.67d( v 213'678 (0'0,213'678] local-les=191 n=131 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.572811 7f8691765700 5 queue_op 0x30607680 seq 3200 osr(2.6b0 0x2ed1ed0) 1883 bytes (queue has 23 ops and 42943 bytes) 2014-06-09 16:22:11.572814 7f8691765700 10 op_submit_finish 3200 2014-06-09 16:22:11.572814 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.572826 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.572817 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=206 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.572828 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.572830 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.572830 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=206 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.572843 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3197.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.572846 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.572840 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.1a4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.572843 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=206 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.309334 7 0.000174 2014-06-09 16:22:11.572851 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.572856 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.572854 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=206 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.572859 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 0.881783 7 0.000178 2014-06-09 16:22:11.572864 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=206 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.572872 7f86a5610700 10 _do_op 0x30607740 seq 3197 r = 0, finisher 0x189abdc0 0 2014-06-09 16:22:11.572876 7f86a5610700 10 _finish_op 0x30607740 seq 3197 osr(2.220 0x2ed0b10)/0x2ed0b10 2014-06-09 16:22:11.572870 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.572878 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497d40 (1 active) 2014-06-09 16:22:11.572880 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359d40 (2 active) 2014-06-09 16:22:11.572874 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=206 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.572881 7f86a5610700 5 _do_op 0x2fcaf0e0 seq 3198 osr(1.7a 0x384c510)/0x384c510 start 2014-06-09 16:22:11.572884 7f86a5610700 10 _do_transaction on 0x3a06240 2014-06-09 16:22:11.572880 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.572888 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.572883 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'523 -> 213'523 2014-06-09 16:22:11.572890 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.572898 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.572906 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.572918 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.572918 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.572933 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3198.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.572936 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.572951 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.572952 7f86a5610700 10 _do_op 0x2fcaf0e0 seq 3198 r = 0, finisher 0xbd7c360 0 2014-06-09 16:22:11.572955 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.572955 7f86a5610700 10 _finish_op 0x2fcaf0e0 seq 3198 osr(1.7a 0x384c510)/0x384c510 2014-06-09 16:22:11.572955 7f8690763700 5 queue_transactions existing osr(0.1a4 0x2bd6710)/0x2bd6710 2014-06-09 16:22:11.572958 7f8690763700 10 op_submit_start 3201 2014-06-09 16:22:11.572958 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359d40 (1 active) 2014-06-09 16:22:11.572959 7f8690763700 5 queue_transactions (trailing journal) 3201 2014-06-09 16:22:11.572959 7f8691765700 5 queue_transactions existing osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 2014-06-09 16:22:11.572960 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7807e0 (2 active) 2014-06-09 16:22:11.572961 7f86a5610700 5 _do_op 0x306076e0 seq 3199 osr(1.11d 0x384c8b0)/0x384c8b0 start 2014-06-09 16:22:11.572961 7f8690763700 5 queue_op 0x2fcaf080 seq 3201 osr(0.1a4 0x2bd6710) 1876 bytes (queue has 22 ops and 40937 bytes) 2014-06-09 16:22:11.572963 7f8690763700 10 op_submit_finish 3201 2014-06-09 16:22:11.572963 7f86a5610700 10 _do_transaction on 0xbf42300 2014-06-09 16:22:11.572965 7f8689f56700 10 op_submit_start 3202 2014-06-09 16:22:11.572966 7f8689f56700 5 queue_transactions (trailing journal) 3202 2014-06-09 16:22:11.572968 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.572967 7f8689f56700 5 queue_op 0x30606900 seq 3202 osr(0.5e 0x2bd63b0) 1998 bytes (queue has 23 ops and 42272 bytes) 2014-06-09 16:22:11.572970 7f8689f56700 10 op_submit_finish 3202 2014-06-09 16:22:11.572969 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.572976 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.206( v 214'596 (0'0,214'596] local-les=201 n=113 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.572983 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.572998 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.572992 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.573004 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.572996 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.952412 7 0.005447 2014-06-09 16:22:11.573007 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.573008 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.214( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.573021 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.573030 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.573018 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.573034 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:1.446( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.573036 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.573044 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.573053 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.573047 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'596 -> 213'596 2014-06-09 16:22:11.573057 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.573067 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3199.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.573071 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.573066 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.714( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.573077 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.573081 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.573084 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.714( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.899877 7 0.000187 2014-06-09 16:22:11.573095 7f86a5610700 10 _do_op 0x306076e0 seq 3199 r = 0, finisher 0x306cf3e0 0 2014-06-09 16:22:11.573088 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.573098 7f86a5610700 10 _finish_op 0x306076e0 seq 3199 osr(1.11d 0x384c8b0)/0x384c8b0 2014-06-09 16:22:11.573094 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.714( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.573100 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7807e0 (1 active) 2014-06-09 16:22:11.573102 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5258c0 (2 active) 2014-06-09 16:22:11.573104 7f86a5610700 5 _do_op 0x30607680 seq 3200 osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 start 2014-06-09 16:22:11.573106 7f86a5610700 10 _do_transaction on 0xc295500 2014-06-09 16:22:11.573104 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.714( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.573111 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.573113 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.573127 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.573122 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.573132 7f8690763700 5 queue_transactions existing osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 2014-06-09 16:22:11.573135 7f8690763700 10 op_submit_start 3203 2014-06-09 16:22:11.573136 7f8690763700 5 queue_transactions (trailing journal) 3203 2014-06-09 16:22:11.573130 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.573137 7f8690763700 5 queue_op 0x2fcaf020 seq 3203 osr(3.2c9 0x3ed0ea0) 1335 bytes (queue has 23 ops and 41653 bytes) 2014-06-09 16:22:11.573140 7f8690763700 10 op_submit_finish 3203 2014-06-09 16:22:11.573141 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.573142 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:0.ca( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.573155 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3200.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.573159 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.573155 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.573165 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.573175 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.573175 7f86a5610700 10 _do_op 0x30607680 seq 3200 r = 0, finisher 0xcd9ea60 0 2014-06-09 16:22:11.573177 7f86a5610700 10 _finish_op 0x30607680 seq 3200 osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 2014-06-09 16:22:11.573171 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.573179 7f8689f56700 5 queue_transactions existing osr(0.714 0x384c0d0)/0x384c0d0 2014-06-09 16:22:11.573180 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5258c0 (1 active) 2014-06-09 16:22:11.573182 7f8689f56700 10 op_submit_start 3204 2014-06-09 16:22:11.573181 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9237a0 (2 active) 2014-06-09 16:22:11.573183 7f8689f56700 5 queue_transactions (trailing journal) 3204 2014-06-09 16:22:11.573183 7f86a5610700 5 _do_op 0x2fcaf080 seq 3201 osr(0.1a4 0x2bd6710)/0x2bd6710 start 2014-06-09 16:22:11.573184 7f8689f56700 5 queue_op 0x306068a0 seq 3204 osr(0.714 0x384c0d0) 1954 bytes (queue has 23 ops and 41724 bytes) 2014-06-09 16:22:11.573185 7f86a5610700 10 _do_transaction on 0xc28b800 2014-06-09 16:22:11.573186 7f8689f56700 10 op_submit_finish 3204 2014-06-09 16:22:11.573189 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.573181 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 1.289( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.573193 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.573189 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.67f( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.573196 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.573200 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.289( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.251004 7 0.000199 2014-06-09 16:22:11.573200 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.573211 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.573210 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.289( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.573219 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.573215 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.67e( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.573220 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.289( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.573227 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.573233 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3201.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.573229 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.573236 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.573236 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.573238 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.573246 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.573240 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.3b4( v 214'677 (0'0,214'677] local-les=212 n=133 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.573256 7f86a5610700 10 _do_op 0x2fcaf080 seq 3201 r = 0, finisher 0xc5a76a0 0 2014-06-09 16:22:11.573260 7f86a5610700 10 _finish_op 0x2fcaf080 seq 3201 osr(0.1a4 0x2bd6710)/0x2bd6710 2014-06-09 16:22:11.573262 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9237a0 (1 active) 2014-06-09 16:22:11.573264 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cea0 (2 active) 2014-06-09 16:22:11.573265 7f86a5610700 5 _do_op 0x30606900 seq 3202 osr(0.5e 0x2bd63b0)/0x2bd63b0 start 2014-06-09 16:22:11.573268 7f86a5610700 10 _do_transaction on 0xc05fe00 2014-06-09 16:22:11.573258 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.573272 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.573273 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.573276 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.3b8( v 214'622 (0'0,214'622] local-les=212 n=120 ec=118 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.573292 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.573288 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.573296 7f8690763700 5 queue_transactions existing osr(1.289 0x384ceb0)/0x384ceb0 2014-06-09 16:22:11.573298 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.573299 7f8690763700 10 op_submit_start 3205 2014-06-09 16:22:11.573300 7f8690763700 5 queue_transactions (trailing journal) 3205 2014-06-09 16:22:11.573301 7f8690763700 5 queue_op 0x2fcaefc0 seq 3205 osr(1.289 0x384ceb0) 2009 bytes (queue has 23 ops and 41857 bytes) 2014-06-09 16:22:11.573303 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.573303 7f8690763700 10 op_submit_finish 3205 2014-06-09 16:22:11.573302 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.573307 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.5bf( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.573317 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3202.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.573320 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.573313 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 0.5a2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.573324 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.573320 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.573327 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.573330 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.573330 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] exit Started/Stray 0.806553 7 0.000168 2014-06-09 16:22:11.573338 7f86a5610700 10 _do_op 0x30606900 seq 3202 r = 0, finisher 0x30509300 0 2014-06-09 16:22:11.573341 7f86a5610700 10 _finish_op 0x30606900 seq 3202 osr(0.5e 0x2bd63b0)/0x2bd63b0 2014-06-09 16:22:11.573335 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.573343 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cea0 (1 active) 2014-06-09 16:22:11.573345 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6c60 (2 active) 2014-06-09 16:22:11.573340 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.573347 7f86a5610700 5 _do_op 0x2fcaf020 seq 3203 osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 start 2014-06-09 16:22:11.573349 7f86a5610700 10 _do_transaction on 0xc244840 2014-06-09 16:22:11.573353 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.573345 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 1.6f5( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.573350 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.573357 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.573361 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.573359 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.573364 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.740473 7 0.000215 2014-06-09 16:22:11.573368 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.573374 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.573376 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.573383 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.573384 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.573393 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.573397 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3203.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.573401 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.573401 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.573409 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.573415 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.573420 7f8689f56700 5 queue_transactions existing osr(0.5a2 0x2bd7af0)/0x2bd7af0 2014-06-09 16:22:11.573422 7f8689f56700 10 op_submit_start 3206 2014-06-09 16:22:11.573423 7f8689f56700 5 queue_transactions (trailing journal) 3206 2014-06-09 16:22:11.573422 7f86a5610700 10 _do_op 0x2fcaf020 seq 3203 r = 0, finisher 0xcc946a0 0 2014-06-09 16:22:11.573424 7f8689f56700 5 queue_op 0x30606840 seq 3206 osr(0.5a2 0x2bd7af0) 1621 bytes (queue has 23 ops and 41480 bytes) 2014-06-09 16:22:11.573425 7f86a5610700 10 _finish_op 0x2fcaf020 seq 3203 osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 2014-06-09 16:22:11.573427 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6c60 (1 active) 2014-06-09 16:22:11.573429 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e5a0 (2 active) 2014-06-09 16:22:11.566301 7f869af78700 5 -- op tracker -- , seq: 1994, time: 2014-06-09 16:22:11.566298, event: done, op: pg_info(2 pgs e243:3.461(2),3.468(2)) 2014-06-09 16:22:11.573433 7f86a5610700 5 _do_op 0x306068a0 seq 3204 osr(0.714 0x384c0d0)/0x384c0d0 start 2014-06-09 16:22:11.573436 7f86a5610700 10 _do_transaction on 0xbfec840 2014-06-09 16:22:11.573440 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.573444 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573446 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573454 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.573458 7f8690763700 5 queue_transactions existing osr(1.6f5 0x2ed0170)/0x2ed0170 2014-06-09 16:22:11.573464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573465 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573467 7f869af78700 20 osd.26 243 _dispatch 0x3088a400 pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.573471 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.573474 7f869af78700 5 -- op tracker -- , seq: 1995, time: 2014-06-09 16:22:11.293212, event: header_read, op: pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573483 7f869af78700 5 -- op tracker -- , seq: 1995, time: 2014-06-09 16:22:11.293212, event: throttled, op: pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573485 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3204.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.573489 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.573490 7f869af78700 5 -- op tracker -- , seq: 1995, time: 2014-06-09 16:22:11.293219, event: all_read, op: pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573497 7f869af78700 5 -- op tracker -- , seq: 1995, time: 2014-06-09 16:22:11.573463, event: dispatched, op: pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573504 7f869af78700 5 -- op tracker -- , seq: 1995, time: 2014-06-09 16:22:11.573501, event: waiting_for_osdmap, op: pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573508 7f86a5610700 10 _do_op 0x306068a0 seq 3204 r = 0, finisher 0xcc82300 0 2014-06-09 16:22:11.573508 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.573511 7f86a5610700 10 _finish_op 0x306068a0 seq 3204 osr(0.714 0x384c0d0)/0x384c0d0 2014-06-09 16:22:11.573512 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088a400 2014-06-09 16:22:11.573513 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e5a0 (1 active) 2014-06-09 16:22:11.573515 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652480 (2 active) 2014-06-09 16:22:11.573516 7f86a5610700 5 _do_op 0x2fcaefc0 seq 3205 osr(1.289 0x384ceb0)/0x384ceb0 start 2014-06-09 16:22:11.573518 7f86a5610700 10 _do_transaction on 0xbf4f080 2014-06-09 16:22:11.573518 7f869af78700 5 -- op tracker -- , seq: 1995, time: 2014-06-09 16:22:11.573516, event: started, op: pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573523 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.573533 7f869af78700 5 -- op tracker -- , seq: 1995, time: 2014-06-09 16:22:11.573530, event: done, op: pg_log(0.456 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573541 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573543 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573553 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.573555 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573556 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573557 7f869af78700 20 osd.26 243 _dispatch 0x3043afc0 pg_info(1 pgs e243:3.761(4)) v4 2014-06-09 16:22:11.573563 7f869af78700 5 -- op tracker -- , seq: 1996, time: 2014-06-09 16:22:11.211108, event: header_read, op: pg_info(1 pgs e243:3.761(4)) 2014-06-09 16:22:11.573567 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3205.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.573571 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.573570 7f869af78700 5 -- op tracker -- , seq: 1996, time: 2014-06-09 16:22:11.211108, event: throttled, op: pg_info(1 pgs e243:3.761(4)) 2014-06-09 16:22:11.573577 7f869af78700 5 -- op tracker -- , seq: 1996, time: 2014-06-09 16:22:11.211122, event: all_read, op: pg_info(1 pgs e243:3.761(4)) 2014-06-09 16:22:11.573583 7f869af78700 5 -- op tracker -- , seq: 1996, time: 2014-06-09 16:22:11.573554, event: dispatched, op: pg_info(1 pgs e243:3.761(4)) 2014-06-09 16:22:11.573589 7f869af78700 5 -- op tracker -- , seq: 1996, time: 2014-06-09 16:22:11.573587, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.761(4)) 2014-06-09 16:22:11.573593 7f86a5610700 10 _do_op 0x2fcaefc0 seq 3205 r = 0, finisher 0x19940b40 0 2014-06-09 16:22:11.573593 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.761(4)) v4 from osd.1 2014-06-09 16:22:11.573596 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043afc0 2014-06-09 16:22:11.573596 7f86a5610700 10 _finish_op 0x2fcaefc0 seq 3205 osr(1.289 0x384ceb0)/0x384ceb0 2014-06-09 16:22:11.573598 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652480 (1 active) 2014-06-09 16:22:11.573600 7f86a5610700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.573601 7f869af78700 5 -- op tracker -- , seq: 1996, time: 2014-06-09 16:22:11.573599, event: started, op: pg_info(1 pgs e243:3.761(4)) 2014-06-09 16:22:11.573607 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x139654a0,0x199419e0,0xc410840,0xc40c6a0,0xc5a6a40,0xbd6d860,0xbd7c5a0,0xbdf0040,0x3050bba0,0x230c1be0,0xc5a6080,0x3050a780,0xbd5b0a0] 2014-06-09 16:22:11.573617 7f869af78700 5 -- op tracker -- , seq: 1996, time: 2014-06-09 16:22:11.573615, event: done, op: pg_info(1 pgs e243:3.761(4)) 2014-06-09 16:22:11.573614 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.573624 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573626 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573630 7f86a460e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.573639 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573641 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573642 7f869af78700 20 osd.26 243 _dispatch 0x308e36c0 pg_info(1 pgs e243:1.99) v4 2014-06-09 16:22:11.573647 7f869af78700 5 -- op tracker -- , seq: 1997, time: 2014-06-09 16:22:11.294654, event: header_read, op: pg_info(1 pgs e243:1.99) 2014-06-09 16:22:11.573644 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.573653 7f869af78700 5 -- op tracker -- , seq: 1997, time: 2014-06-09 16:22:11.294655, event: throttled, op: pg_info(1 pgs e243:1.99) 2014-06-09 16:22:11.573659 7f869af78700 5 -- op tracker -- , seq: 1997, time: 2014-06-09 16:22:11.294666, event: all_read, op: pg_info(1 pgs e243:1.99) 2014-06-09 16:22:11.573663 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x139654a0,0x199419e0,0xc410840,0xc40c6a0,0xc5a6a40,0xbd6d860,0xbd7c5a0,0xbdf0040,0x3050bba0,0x230c1be0,0xc5a6080,0x3050a780,0xbd5b0a0] 2014-06-09 16:22:11.573666 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.573664 7f869af78700 5 -- op tracker -- , seq: 1997, time: 2014-06-09 16:22:11.573639, event: dispatched, op: pg_info(1 pgs e243:1.99) 2014-06-09 16:22:11.573668 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.573671 7f869af78700 5 -- op tracker -- , seq: 1997, time: 2014-06-09 16:22:11.573669, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.99) 2014-06-09 16:22:11.573675 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.99) v4 from osd.9 2014-06-09 16:22:11.573678 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e36c0 2014-06-09 16:22:11.573683 7f869af78700 5 -- op tracker -- , seq: 1997, time: 2014-06-09 16:22:11.573681, event: started, op: pg_info(1 pgs e243:1.99) 2014-06-09 16:22:11.573699 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.573698 7f869af78700 5 -- op tracker -- , seq: 1997, time: 2014-06-09 16:22:11.573696, event: done, op: pg_info(1 pgs e243:1.99) 2014-06-09 16:22:11.573706 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573707 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573715 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3193.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.573719 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.573721 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573722 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573723 7f869af78700 20 osd.26 243 _dispatch 0x2fe20400 pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.573729 7f869af78700 5 -- op tracker -- , seq: 1998, time: 2014-06-09 16:22:11.314575, event: header_read, op: pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573737 7f86a5e11700 10 _do_op 0x30607860 seq 3193 r = 0, finisher 0xbd686e0 0 2014-06-09 16:22:11.573736 7f869af78700 5 -- op tracker -- , seq: 1998, time: 2014-06-09 16:22:11.314576, event: throttled, op: pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573743 7f86a5e11700 10 _finish_op 0x30607860 seq 3193 osr(0.186 0x2bd67d0)/0x2bd67d0 2014-06-09 16:22:11.573743 7f869af78700 5 -- op tracker -- , seq: 1998, time: 2014-06-09 16:22:11.314585, event: all_read, op: pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573746 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e240 (0 active) 2014-06-09 16:22:11.573747 7f86a5e11700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.573750 7f869af78700 5 -- op tracker -- , seq: 1998, time: 2014-06-09 16:22:11.573720, event: dispatched, op: pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573757 7f869af78700 5 -- op tracker -- , seq: 1998, time: 2014-06-09 16:22:11.573754, event: waiting_for_osdmap, op: pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573762 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.573765 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe20400 2014-06-09 16:22:11.573771 7f869af78700 5 -- op tracker -- , seq: 1998, time: 2014-06-09 16:22:11.573769, event: started, op: pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5b0e0] 2014-06-09 16:22:11.573781 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5b0e0] 2014-06-09 16:22:11.573782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.573783 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.573784 7f869af78700 5 -- op tracker -- , seq: 1998, time: 2014-06-09 16:22:11.573782, event: done, op: pg_log(0.58a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.573787 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbce6460,0x170019a0,0x33a65c0,0x189abdc0,0xbd7c360,0x306cf3e0,0xcd9ea60,0xc5a76a0,0x30509300,0xcc946a0,0xcc82300,0x19940b40,0xbd686e0] 2014-06-09 16:22:11.573792 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573793 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573791 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.573804 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.573806 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573808 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573809 7f869af78700 20 osd.26 243 _dispatch 0x1405cfc0 pg_notify(1.b3(23),1.37d(20),2.648(20) epoch 243) v5 2014-06-09 16:22:11.573815 7f869af78700 5 -- op tracker -- , seq: 1999, time: 2014-06-09 16:22:11.305284, event: header_read, op: pg_notify(1.b3(23),1.37d(20),2.648(20) epoch 243) 2014-06-09 16:22:11.573823 7f869af78700 5 -- op tracker -- , seq: 1999, time: 2014-06-09 16:22:11.305285, event: throttled, op: pg_notify(1.b3(23),1.37d(20),2.648(20) epoch 243) 2014-06-09 16:22:11.573818 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.573830 7f869af78700 5 -- op tracker -- , seq: 1999, time: 2014-06-09 16:22:11.305354, event: all_read, op: pg_notify(1.b3(23),1.37d(20),2.648(20) epoch 243) 2014-06-09 16:22:11.573837 7f869af78700 5 -- op tracker -- , seq: 1999, time: 2014-06-09 16:22:11.573806, event: dispatched, op: pg_notify(1.b3(23),1.37d(20),2.648(20) epoch 243) 2014-06-09 16:22:11.573844 7f869af78700 5 -- op tracker -- , seq: 1999, time: 2014-06-09 16:22:11.573841, event: waiting_for_osdmap, op: pg_notify(1.b3(23),1.37d(20),2.648(20) epoch 243) 2014-06-09 16:22:11.573848 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.573840 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.573849 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405cfc0 2014-06-09 16:22:11.573854 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.573855 7f869af78700 5 -- op tracker -- , seq: 1999, time: 2014-06-09 16:22:11.573852, event: started, op: pg_notify(1.b3(23),1.37d(20),2.648(20) epoch 243) 2014-06-09 16:22:11.573867 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.573886 7f869af78700 5 -- op tracker -- , seq: 1999, time: 2014-06-09 16:22:11.573884, event: done, op: pg_notify(1.b3(23),1.37d(20),2.648(20) epoch 243) 2014-06-09 16:22:11.573886 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.573895 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573897 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573898 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.573912 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.573927 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.573929 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.573930 7f869af78700 20 osd.26 243 _dispatch 0x3005a540 pg_info(2 pgs e243:3.778(4),3.1fc(4)) v4 2014-06-09 16:22:11.573935 7f869af78700 5 -- op tracker -- , seq: 2000, time: 2014-06-09 16:22:11.229771, event: header_read, op: pg_info(2 pgs e243:3.778(4),3.1fc(4)) 2014-06-09 16:22:11.573934 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.573943 7f869af78700 5 -- op tracker -- , seq: 2000, time: 2014-06-09 16:22:11.229771, event: throttled, op: pg_info(2 pgs e243:3.778(4),3.1fc(4)) 2014-06-09 16:22:11.573946 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.573949 7f869af78700 5 -- op tracker -- , seq: 2000, time: 2014-06-09 16:22:11.229796, event: all_read, op: pg_info(2 pgs e243:3.778(4),3.1fc(4)) 2014-06-09 16:22:11.573956 7f869af78700 5 -- op tracker -- , seq: 2000, time: 2014-06-09 16:22:11.573927, event: dispatched, op: pg_info(2 pgs e243:3.778(4),3.1fc(4)) 2014-06-09 16:22:11.573963 7f869af78700 5 -- op tracker -- , seq: 2000, time: 2014-06-09 16:22:11.573960, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.778(4),3.1fc(4)) 2014-06-09 16:22:11.573967 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.778(4),3.1fc(4)) v4 from osd.0 2014-06-09 16:22:11.573971 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005a540 2014-06-09 16:22:11.573967 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.573976 7f869af78700 5 -- op tracker -- , seq: 2000, time: 2014-06-09 16:22:11.573973, event: started, op: pg_info(2 pgs e243:3.778(4),3.1fc(4)) 2014-06-09 16:22:11.573980 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.573991 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.573999 7f869af78700 5 -- op tracker -- , seq: 2000, time: 2014-06-09 16:22:11.573997, event: done, op: pg_info(2 pgs e243:3.778(4),3.1fc(4)) 2014-06-09 16:22:11.574007 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574009 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574004 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.574023 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.574031 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574032 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574034 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.574033 7f869af78700 20 osd.26 243 _dispatch 0x14050540 pg_notify(3.730(10),2.73a(20) epoch 243) v5 2014-06-09 16:22:11.574039 7f869af78700 5 -- op tracker -- , seq: 2001, time: 2014-06-09 16:22:11.339431, event: header_read, op: pg_notify(3.730(10),2.73a(20) epoch 243) 2014-06-09 16:22:11.574046 7f869af78700 5 -- op tracker -- , seq: 2001, time: 2014-06-09 16:22:11.339432, event: throttled, op: pg_notify(3.730(10),2.73a(20) epoch 243) 2014-06-09 16:22:11.574046 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.574053 7f869af78700 5 -- op tracker -- , seq: 2001, time: 2014-06-09 16:22:11.339479, event: all_read, op: pg_notify(3.730(10),2.73a(20) epoch 243) 2014-06-09 16:22:11.574059 7f869af78700 5 -- op tracker -- , seq: 2001, time: 2014-06-09 16:22:11.574030, event: dispatched, op: pg_notify(3.730(10),2.73a(20) epoch 243) 2014-06-09 16:22:11.574067 7f869af78700 5 -- op tracker -- , seq: 2001, time: 2014-06-09 16:22:11.574064, event: waiting_for_osdmap, op: pg_notify(3.730(10),2.73a(20) epoch 243) 2014-06-09 16:22:11.574071 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.574065 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.574072 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14050540 2014-06-09 16:22:11.574077 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.574078 7f869af78700 5 -- op tracker -- , seq: 2001, time: 2014-06-09 16:22:11.574075, event: started, op: pg_notify(3.730(10),2.73a(20) epoch 243) 2014-06-09 16:22:11.574087 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.574100 7f869af78700 5 -- op tracker -- , seq: 2001, time: 2014-06-09 16:22:11.574098, event: done, op: pg_notify(3.730(10),2.73a(20) epoch 243) 2014-06-09 16:22:11.574100 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.574108 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574110 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574111 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.574120 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.574131 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574133 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574134 7f869af78700 20 osd.26 243 _dispatch 0xe06f180 pg_notify(3.3ae(12) epoch 243) v5 2014-06-09 16:22:11.574133 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.574139 7f869af78700 5 -- op tracker -- , seq: 2002, time: 2014-06-09 16:22:11.137412, event: header_read, op: pg_notify(3.3ae(12) epoch 243) 2014-06-09 16:22:11.574144 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.574145 7f869af78700 5 -- op tracker -- , seq: 2002, time: 2014-06-09 16:22:11.137412, event: throttled, op: pg_notify(3.3ae(12) epoch 243) 2014-06-09 16:22:11.574151 7f869af78700 5 -- op tracker -- , seq: 2002, time: 2014-06-09 16:22:11.137434, event: all_read, op: pg_notify(3.3ae(12) epoch 243) 2014-06-09 16:22:11.574157 7f869af78700 5 -- op tracker -- , seq: 2002, time: 2014-06-09 16:22:11.574131, event: dispatched, op: pg_notify(3.3ae(12) epoch 243) 2014-06-09 16:22:11.574158 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.574164 7f869af78700 5 -- op tracker -- , seq: 2002, time: 2014-06-09 16:22:11.574162, event: waiting_for_osdmap, op: pg_notify(3.3ae(12) epoch 243) 2014-06-09 16:22:11.574168 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.574170 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xe06f180 2014-06-09 16:22:11.574174 7f869af78700 5 -- op tracker -- , seq: 2002, time: 2014-06-09 16:22:11.574172, event: started, op: pg_notify(3.3ae(12) epoch 243) 2014-06-09 16:22:11.574171 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.574184 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.574190 7f869af78700 5 -- op tracker -- , seq: 2002, time: 2014-06-09 16:22:11.574187, event: done, op: pg_notify(3.3ae(12) epoch 243) 2014-06-09 16:22:11.574197 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574199 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574196 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.574213 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574215 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574210 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.574216 7f869af78700 20 osd.26 243 _dispatch 0x305e3e00 pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.574220 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.574222 7f869af78700 5 -- op tracker -- , seq: 2003, time: 2014-06-09 16:22:11.168120, event: header_read, op: pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574229 7f869af78700 5 -- op tracker -- , seq: 2003, time: 2014-06-09 16:22:11.168121, event: throttled, op: pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574230 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.574236 7f869af78700 5 -- op tracker -- , seq: 2003, time: 2014-06-09 16:22:11.168138, event: all_read, op: pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574243 7f869af78700 5 -- op tracker -- , seq: 2003, time: 2014-06-09 16:22:11.574213, event: dispatched, op: pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574242 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.574252 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.574250 7f869af78700 5 -- op tracker -- , seq: 2003, time: 2014-06-09 16:22:11.574247, event: waiting_for_osdmap, op: pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574254 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.574258 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e3e00 2014-06-09 16:22:11.574263 7f869af78700 5 -- op tracker -- , seq: 2003, time: 2014-06-09 16:22:11.574260, event: started, op: pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574261 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.574275 7f869af78700 5 -- op tracker -- , seq: 2003, time: 2014-06-09 16:22:11.574272, event: done, op: pg_log(2.236 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574273 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.574281 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574283 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574284 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.574293 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574295 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574296 7f869af78700 20 osd.26 243 _dispatch 0x10d0da40 pg_notify(3.2e7(10) epoch 243) v5 2014-06-09 16:22:11.574296 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.574300 7f869af78700 5 -- op tracker -- , seq: 2004, time: 2014-06-09 16:22:11.339632, event: header_read, op: pg_notify(3.2e7(10) epoch 243) 2014-06-09 16:22:11.574307 7f869af78700 5 -- op tracker -- , seq: 2004, time: 2014-06-09 16:22:11.339634, event: throttled, op: pg_notify(3.2e7(10) epoch 243) 2014-06-09 16:22:11.574309 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbce6460,0x170019a0,0x33a65c0,0x189abdc0,0xbd7c360,0x306cf3e0,0xcd9ea60,0xc5a76a0,0x30509300,0xcc946a0,0xcc82300,0x19940b40,0xbd686e0] 2014-06-09 16:22:11.574312 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.574313 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.574312 7f869af78700 5 -- op tracker -- , seq: 2004, time: 2014-06-09 16:22:11.339660, event: all_read, op: pg_notify(3.2e7(10) epoch 243) 2014-06-09 16:22:11.574318 7f869af78700 5 -- op tracker -- , seq: 2004, time: 2014-06-09 16:22:11.574293, event: dispatched, op: pg_notify(3.2e7(10) epoch 243) 2014-06-09 16:22:11.574325 7f869af78700 5 -- op tracker -- , seq: 2004, time: 2014-06-09 16:22:11.574323, event: waiting_for_osdmap, op: pg_notify(3.2e7(10) epoch 243) 2014-06-09 16:22:11.574329 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.574331 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d0da40 2014-06-09 16:22:11.574336 7f869af78700 5 -- op tracker -- , seq: 2004, time: 2014-06-09 16:22:11.574334, event: started, op: pg_notify(3.2e7(10) epoch 243) 2014-06-09 16:22:11.574352 7f869af78700 5 -- op tracker -- , seq: 2004, time: 2014-06-09 16:22:11.574350, event: done, op: pg_notify(3.2e7(10) epoch 243) 2014-06-09 16:22:11.574364 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574367 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574385 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574387 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574388 7f869af78700 20 osd.26 243 _dispatch 0x16065340 pg_notify(0.44f(20),0.718(20) epoch 243) v5 2014-06-09 16:22:11.574394 7f869af78700 5 -- op tracker -- , seq: 2005, time: 2014-06-09 16:22:11.293785, event: header_read, op: pg_notify(0.44f(20),0.718(20) epoch 243) 2014-06-09 16:22:11.574402 7f869af78700 5 -- op tracker -- , seq: 2005, time: 2014-06-09 16:22:11.293786, event: throttled, op: pg_notify(0.44f(20),0.718(20) epoch 243) 2014-06-09 16:22:11.574408 7f869af78700 5 -- op tracker -- , seq: 2005, time: 2014-06-09 16:22:11.293814, event: all_read, op: pg_notify(0.44f(20),0.718(20) epoch 243) 2014-06-09 16:22:11.574415 7f869af78700 5 -- op tracker -- , seq: 2005, time: 2014-06-09 16:22:11.574385, event: dispatched, op: pg_notify(0.44f(20),0.718(20) epoch 243) 2014-06-09 16:22:11.574423 7f869af78700 5 -- op tracker -- , seq: 2005, time: 2014-06-09 16:22:11.574420, event: waiting_for_osdmap, op: pg_notify(0.44f(20),0.718(20) epoch 243) 2014-06-09 16:22:11.574427 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.574429 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16065340 2014-06-09 16:22:11.574434 7f869af78700 5 -- op tracker -- , seq: 2005, time: 2014-06-09 16:22:11.574432, event: started, op: pg_notify(0.44f(20),0.718(20) epoch 243) 2014-06-09 16:22:11.574458 7f869af78700 5 -- op tracker -- , seq: 2005, time: 2014-06-09 16:22:11.574455, event: done, op: pg_notify(0.44f(20),0.718(20) epoch 243) 2014-06-09 16:22:11.574466 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574468 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574496 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574498 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574499 7f869af78700 20 osd.26 243 _dispatch 0x30612600 pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.574505 7f869af78700 5 -- op tracker -- , seq: 2006, time: 2014-06-09 16:22:11.227354, event: header_read, op: pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574514 7f869af78700 5 -- op tracker -- , seq: 2006, time: 2014-06-09 16:22:11.227354, event: throttled, op: pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574521 7f869af78700 5 -- op tracker -- , seq: 2006, time: 2014-06-09 16:22:11.227368, event: all_read, op: pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574528 7f869af78700 5 -- op tracker -- , seq: 2006, time: 2014-06-09 16:22:11.574495, event: dispatched, op: pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574536 7f869af78700 5 -- op tracker -- , seq: 2006, time: 2014-06-09 16:22:11.574533, event: waiting_for_osdmap, op: pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574540 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.574544 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30612600 2014-06-09 16:22:11.574550 7f869af78700 5 -- op tracker -- , seq: 2006, time: 2014-06-09 16:22:11.574547, event: started, op: pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574563 7f869af78700 5 -- op tracker -- , seq: 2006, time: 2014-06-09 16:22:11.574560, event: done, op: pg_log(2.48d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574570 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574572 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574583 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574585 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574587 7f869af78700 20 osd.26 243 _dispatch 0x15a28380 pg_info(1 pgs e243:3.1e) v4 2014-06-09 16:22:11.574592 7f869af78700 5 -- op tracker -- , seq: 2007, time: 2014-06-09 16:22:11.294672, event: header_read, op: pg_info(1 pgs e243:3.1e) 2014-06-09 16:22:11.574598 7f869af78700 5 -- op tracker -- , seq: 2007, time: 2014-06-09 16:22:11.294672, event: throttled, op: pg_info(1 pgs e243:3.1e) 2014-06-09 16:22:11.574605 7f869af78700 5 -- op tracker -- , seq: 2007, time: 2014-06-09 16:22:11.294679, event: all_read, op: pg_info(1 pgs e243:3.1e) 2014-06-09 16:22:11.574610 7f869af78700 5 -- op tracker -- , seq: 2007, time: 2014-06-09 16:22:11.574583, event: dispatched, op: pg_info(1 pgs e243:3.1e) 2014-06-09 16:22:11.574614 7f869af78700 5 -- op tracker -- , seq: 2007, time: 2014-06-09 16:22:11.574613, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1e) 2014-06-09 16:22:11.574616 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1e) v4 from osd.9 2014-06-09 16:22:11.574618 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a28380 2014-06-09 16:22:11.574621 7f869af78700 5 -- op tracker -- , seq: 2007, time: 2014-06-09 16:22:11.574620, event: started, op: pg_info(1 pgs e243:3.1e) 2014-06-09 16:22:11.574631 7f869af78700 5 -- op tracker -- , seq: 2007, time: 2014-06-09 16:22:11.574629, event: done, op: pg_info(1 pgs e243:3.1e) 2014-06-09 16:22:11.574635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574636 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574645 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574647 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574648 7f869af78700 20 osd.26 243 _dispatch 0x305f2c00 pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.574652 7f869af78700 5 -- op tracker -- , seq: 2008, time: 2014-06-09 16:22:11.333396, event: header_read, op: pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574656 7f869af78700 5 -- op tracker -- , seq: 2008, time: 2014-06-09 16:22:11.333397, event: throttled, op: pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574660 7f869af78700 5 -- op tracker -- , seq: 2008, time: 2014-06-09 16:22:11.333411, event: all_read, op: pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574664 7f869af78700 5 -- op tracker -- , seq: 2008, time: 2014-06-09 16:22:11.574645, event: dispatched, op: pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574668 7f869af78700 5 -- op tracker -- , seq: 2008, time: 2014-06-09 16:22:11.574666, event: waiting_for_osdmap, op: pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574670 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.574672 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f2c00 2014-06-09 16:22:11.574675 7f869af78700 5 -- op tracker -- , seq: 2008, time: 2014-06-09 16:22:11.574674, event: started, op: pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574683 7f869af78700 5 -- op tracker -- , seq: 2008, time: 2014-06-09 16:22:11.574681, event: done, op: pg_log(2.194 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574688 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574695 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574696 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574697 7f869af78700 20 osd.26 243 _dispatch 0x30cf9dc0 pg_info(1 pgs e243:3.577(2)) v4 2014-06-09 16:22:11.574700 7f869af78700 5 -- op tracker -- , seq: 2009, time: 2014-06-09 16:22:11.325216, event: header_read, op: pg_info(1 pgs e243:3.577(2)) 2014-06-09 16:22:11.574704 7f869af78700 5 -- op tracker -- , seq: 2009, time: 2014-06-09 16:22:11.325217, event: throttled, op: pg_info(1 pgs e243:3.577(2)) 2014-06-09 16:22:11.574707 7f869af78700 5 -- op tracker -- , seq: 2009, time: 2014-06-09 16:22:11.325231, event: all_read, op: pg_info(1 pgs e243:3.577(2)) 2014-06-09 16:22:11.574710 7f869af78700 5 -- op tracker -- , seq: 2009, time: 2014-06-09 16:22:11.574695, event: dispatched, op: pg_info(1 pgs e243:3.577(2)) 2014-06-09 16:22:11.574714 7f869af78700 5 -- op tracker -- , seq: 2009, time: 2014-06-09 16:22:11.574713, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.577(2)) 2014-06-09 16:22:11.574716 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.577(2)) v4 from osd.6 2014-06-09 16:22:11.574718 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf9dc0 2014-06-09 16:22:11.574720 7f869af78700 5 -- op tracker -- , seq: 2009, time: 2014-06-09 16:22:11.574719, event: started, op: pg_info(1 pgs e243:3.577(2)) 2014-06-09 16:22:11.574730 7f869af78700 5 -- op tracker -- , seq: 2009, time: 2014-06-09 16:22:11.574728, event: done, op: pg_info(1 pgs e243:3.577(2)) 2014-06-09 16:22:11.574733 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574734 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574742 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574742 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574743 7f869af78700 20 osd.26 243 _dispatch 0x30953000 pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.574747 7f869af78700 5 -- op tracker -- , seq: 2010, time: 2014-06-09 16:22:11.241548, event: header_read, op: pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574751 7f869af78700 5 -- op tracker -- , seq: 2010, time: 2014-06-09 16:22:11.241549, event: throttled, op: pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574754 7f869af78700 5 -- op tracker -- , seq: 2010, time: 2014-06-09 16:22:11.241558, event: all_read, op: pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574758 7f869af78700 5 -- op tracker -- , seq: 2010, time: 2014-06-09 16:22:11.574741, event: dispatched, op: pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574761 7f869af78700 5 -- op tracker -- , seq: 2010, time: 2014-06-09 16:22:11.574760, event: waiting_for_osdmap, op: pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574764 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.574766 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30953000 2014-06-09 16:22:11.574769 7f869af78700 5 -- op tracker -- , seq: 2010, time: 2014-06-09 16:22:11.574767, event: started, op: pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574775 7f869af78700 5 -- op tracker -- , seq: 2010, time: 2014-06-09 16:22:11.574774, event: done, op: pg_log(0.47d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574779 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574780 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574787 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574787 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574788 7f869af78700 20 osd.26 243 _dispatch 0x2d733180 pg_info(1 pgs e243:3.46f(2)) v4 2014-06-09 16:22:11.574791 7f869af78700 5 -- op tracker -- , seq: 2011, time: 2014-06-09 16:22:11.339492, event: header_read, op: pg_info(1 pgs e243:3.46f(2)) 2014-06-09 16:22:11.574794 7f869af78700 5 -- op tracker -- , seq: 2011, time: 2014-06-09 16:22:11.339493, event: throttled, op: pg_info(1 pgs e243:3.46f(2)) 2014-06-09 16:22:11.574798 7f869af78700 5 -- op tracker -- , seq: 2011, time: 2014-06-09 16:22:11.339507, event: all_read, op: pg_info(1 pgs e243:3.46f(2)) 2014-06-09 16:22:11.574801 7f869af78700 5 -- op tracker -- , seq: 2011, time: 2014-06-09 16:22:11.574787, event: dispatched, op: pg_info(1 pgs e243:3.46f(2)) 2014-06-09 16:22:11.574805 7f869af78700 5 -- op tracker -- , seq: 2011, time: 2014-06-09 16:22:11.574804, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.46f(2)) 2014-06-09 16:22:11.574807 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.46f(2)) v4 from osd.5 2014-06-09 16:22:11.574809 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d733180 2014-06-09 16:22:11.574811 7f869af78700 5 -- op tracker -- , seq: 2011, time: 2014-06-09 16:22:11.574810, event: started, op: pg_info(1 pgs e243:3.46f(2)) 2014-06-09 16:22:11.574820 7f869af78700 5 -- op tracker -- , seq: 2011, time: 2014-06-09 16:22:11.574819, event: done, op: pg_info(1 pgs e243:3.46f(2)) 2014-06-09 16:22:11.574824 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574825 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574834 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574834 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574835 7f869af78700 20 osd.26 243 _dispatch 0xe06ec40 pg_notify(0.69f(21) epoch 243) v5 2014-06-09 16:22:11.574838 7f869af78700 5 -- op tracker -- , seq: 2012, time: 2014-06-09 16:22:11.137442, event: header_read, op: pg_notify(0.69f(21) epoch 243) 2014-06-09 16:22:11.574841 7f869af78700 5 -- op tracker -- , seq: 2012, time: 2014-06-09 16:22:11.137443, event: throttled, op: pg_notify(0.69f(21) epoch 243) 2014-06-09 16:22:11.574845 7f869af78700 5 -- op tracker -- , seq: 2012, time: 2014-06-09 16:22:11.137471, event: all_read, op: pg_notify(0.69f(21) epoch 243) 2014-06-09 16:22:11.574848 7f869af78700 5 -- op tracker -- , seq: 2012, time: 2014-06-09 16:22:11.574834, event: dispatched, op: pg_notify(0.69f(21) epoch 243) 2014-06-09 16:22:11.574851 7f869af78700 5 -- op tracker -- , seq: 2012, time: 2014-06-09 16:22:11.574850, event: waiting_for_osdmap, op: pg_notify(0.69f(21) epoch 243) 2014-06-09 16:22:11.574853 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.574854 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xe06ec40 2014-06-09 16:22:11.574857 7f869af78700 5 -- op tracker -- , seq: 2012, time: 2014-06-09 16:22:11.574856, event: started, op: pg_notify(0.69f(21) epoch 243) 2014-06-09 16:22:11.574866 7f869af78700 5 -- op tracker -- , seq: 2012, time: 2014-06-09 16:22:11.574865, event: done, op: pg_notify(0.69f(21) epoch 243) 2014-06-09 16:22:11.574870 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574871 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574881 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574882 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574883 7f869af78700 20 osd.26 243 _dispatch 0x305b4400 pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.574887 7f869af78700 5 -- op tracker -- , seq: 2013, time: 2014-06-09 16:22:11.168200, event: header_read, op: pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574891 7f869af78700 5 -- op tracker -- , seq: 2013, time: 2014-06-09 16:22:11.168200, event: throttled, op: pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574894 7f869af78700 5 -- op tracker -- , seq: 2013, time: 2014-06-09 16:22:11.168217, event: all_read, op: pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574898 7f869af78700 5 -- op tracker -- , seq: 2013, time: 2014-06-09 16:22:11.574881, event: dispatched, op: pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574901 7f869af78700 5 -- op tracker -- , seq: 2013, time: 2014-06-09 16:22:11.574900, event: waiting_for_osdmap, op: pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574904 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.574906 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b4400 2014-06-09 16:22:11.574909 7f869af78700 5 -- op tracker -- , seq: 2013, time: 2014-06-09 16:22:11.574907, event: started, op: pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574916 7f869af78700 5 -- op tracker -- , seq: 2013, time: 2014-06-09 16:22:11.574914, event: done, op: pg_log(1.504 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.574920 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574920 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574927 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574929 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.574930 7f869af78700 20 osd.26 243 _dispatch 0x10d0dc00 pg_info(2 pgs e243:3.2df(2),3.5b3(2)) v4 2014-06-09 16:22:11.574935 7f869af78700 5 -- op tracker -- , seq: 2014, time: 2014-06-09 16:22:11.339669, event: header_read, op: pg_info(2 pgs e243:3.2df(2),3.5b3(2)) 2014-06-09 16:22:11.574942 7f869af78700 5 -- op tracker -- , seq: 2014, time: 2014-06-09 16:22:11.339670, event: throttled, op: pg_info(2 pgs e243:3.2df(2),3.5b3(2)) 2014-06-09 16:22:11.574948 7f869af78700 5 -- op tracker -- , seq: 2014, time: 2014-06-09 16:22:11.339692, event: all_read, op: pg_info(2 pgs e243:3.2df(2),3.5b3(2)) 2014-06-09 16:22:11.574954 7f869af78700 5 -- op tracker -- , seq: 2014, time: 2014-06-09 16:22:11.574927, event: dispatched, op: pg_info(2 pgs e243:3.2df(2),3.5b3(2)) 2014-06-09 16:22:11.574961 7f869af78700 5 -- op tracker -- , seq: 2014, time: 2014-06-09 16:22:11.574958, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.2df(2),3.5b3(2)) 2014-06-09 16:22:11.574964 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.2df(2),3.5b3(2)) v4 from osd.7 2014-06-09 16:22:11.574967 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d0dc00 2014-06-09 16:22:11.574972 7f869af78700 5 -- op tracker -- , seq: 2014, time: 2014-06-09 16:22:11.574970, event: started, op: pg_info(2 pgs e243:3.2df(2),3.5b3(2)) 2014-06-09 16:22:11.574992 7f869af78700 5 -- op tracker -- , seq: 2014, time: 2014-06-09 16:22:11.574990, event: done, op: pg_info(2 pgs e243:3.2df(2),3.5b3(2)) 2014-06-09 16:22:11.574997 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.574998 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575007 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575007 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575008 7f869af78700 20 osd.26 243 _dispatch 0x10d0cc40 pg_notify(0.74a(23),1.1bc(20) epoch 243) v5 2014-06-09 16:22:11.575011 7f869af78700 5 -- op tracker -- , seq: 2015, time: 2014-06-09 16:22:11.328433, event: header_read, op: pg_notify(0.74a(23),1.1bc(20) epoch 243) 2014-06-09 16:22:11.575015 7f869af78700 5 -- op tracker -- , seq: 2015, time: 2014-06-09 16:22:11.328434, event: throttled, op: pg_notify(0.74a(23),1.1bc(20) epoch 243) 2014-06-09 16:22:11.575019 7f869af78700 5 -- op tracker -- , seq: 2015, time: 2014-06-09 16:22:11.328474, event: all_read, op: pg_notify(0.74a(23),1.1bc(20) epoch 243) 2014-06-09 16:22:11.575022 7f869af78700 5 -- op tracker -- , seq: 2015, time: 2014-06-09 16:22:11.575006, event: dispatched, op: pg_notify(0.74a(23),1.1bc(20) epoch 243) 2014-06-09 16:22:11.575026 7f869af78700 5 -- op tracker -- , seq: 2015, time: 2014-06-09 16:22:11.575024, event: waiting_for_osdmap, op: pg_notify(0.74a(23),1.1bc(20) epoch 243) 2014-06-09 16:22:11.575028 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.575029 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d0cc40 2014-06-09 16:22:11.575032 7f869af78700 5 -- op tracker -- , seq: 2015, time: 2014-06-09 16:22:11.575031, event: started, op: pg_notify(0.74a(23),1.1bc(20) epoch 243) 2014-06-09 16:22:11.575051 7f869af78700 5 -- op tracker -- , seq: 2015, time: 2014-06-09 16:22:11.575048, event: done, op: pg_notify(0.74a(23),1.1bc(20) epoch 243) 2014-06-09 16:22:11.575058 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575061 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575086 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575089 7f869af78700 20 osd.26 243 _dispatch 0x307de600 pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.575095 7f869af78700 5 -- op tracker -- , seq: 2016, time: 2014-06-09 16:22:11.227380, event: header_read, op: pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575104 7f869af78700 5 -- op tracker -- , seq: 2016, time: 2014-06-09 16:22:11.227380, event: throttled, op: pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575111 7f869af78700 5 -- op tracker -- , seq: 2016, time: 2014-06-09 16:22:11.227394, event: all_read, op: pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575118 7f869af78700 5 -- op tracker -- , seq: 2016, time: 2014-06-09 16:22:11.575085, event: dispatched, op: pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575126 7f869af78700 5 -- op tracker -- , seq: 2016, time: 2014-06-09 16:22:11.575123, event: waiting_for_osdmap, op: pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575129 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.575132 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307de600 2014-06-09 16:22:11.575135 7f869af78700 5 -- op tracker -- , seq: 2016, time: 2014-06-09 16:22:11.575134, event: started, op: pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575143 7f869af78700 5 -- op tracker -- , seq: 2016, time: 2014-06-09 16:22:11.575141, event: done, op: pg_log(1.1e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575151 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575152 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575159 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575159 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575160 7f869af78700 20 osd.26 243 _dispatch 0x190b3880 pg_info(1 pgs e243:2.36c) v4 2014-06-09 16:22:11.575163 7f869af78700 5 -- op tracker -- , seq: 2017, time: 2014-06-09 16:22:11.294736, event: header_read, op: pg_info(1 pgs e243:2.36c) 2014-06-09 16:22:11.575167 7f869af78700 5 -- op tracker -- , seq: 2017, time: 2014-06-09 16:22:11.294736, event: throttled, op: pg_info(1 pgs e243:2.36c) 2014-06-09 16:22:11.575170 7f869af78700 5 -- op tracker -- , seq: 2017, time: 2014-06-09 16:22:11.294753, event: all_read, op: pg_info(1 pgs e243:2.36c) 2014-06-09 16:22:11.575173 7f869af78700 5 -- op tracker -- , seq: 2017, time: 2014-06-09 16:22:11.575158, event: dispatched, op: pg_info(1 pgs e243:2.36c) 2014-06-09 16:22:11.575177 7f869af78700 5 -- op tracker -- , seq: 2017, time: 2014-06-09 16:22:11.575176, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.36c) 2014-06-09 16:22:11.575179 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.36c) v4 from osd.9 2014-06-09 16:22:11.575180 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b3880 2014-06-09 16:22:11.575183 7f869af78700 5 -- op tracker -- , seq: 2017, time: 2014-06-09 16:22:11.575182, event: started, op: pg_info(1 pgs e243:2.36c) 2014-06-09 16:22:11.575191 7f869af78700 5 -- op tracker -- , seq: 2017, time: 2014-06-09 16:22:11.575190, event: done, op: pg_info(1 pgs e243:2.36c) 2014-06-09 16:22:11.575195 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575196 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575203 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575204 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575204 7f869af78700 20 osd.26 243 _dispatch 0x1003a380 pg_info(1 pgs e243:3.300) v4 2014-06-09 16:22:11.575207 7f869af78700 5 -- op tracker -- , seq: 2018, time: 2014-06-09 16:22:11.333745, event: header_read, op: pg_info(1 pgs e243:3.300) 2014-06-09 16:22:11.575211 7f869af78700 5 -- op tracker -- , seq: 2018, time: 2014-06-09 16:22:11.333745, event: throttled, op: pg_info(1 pgs e243:3.300) 2014-06-09 16:22:11.575214 7f869af78700 5 -- op tracker -- , seq: 2018, time: 2014-06-09 16:22:11.333761, event: all_read, op: pg_info(1 pgs e243:3.300) 2014-06-09 16:22:11.575217 7f869af78700 5 -- op tracker -- , seq: 2018, time: 2014-06-09 16:22:11.575203, event: dispatched, op: pg_info(1 pgs e243:3.300) 2014-06-09 16:22:11.575220 7f869af78700 5 -- op tracker -- , seq: 2018, time: 2014-06-09 16:22:11.575219, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.300) 2014-06-09 16:22:11.575222 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.300) v4 from osd.8 2014-06-09 16:22:11.575224 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1003a380 2014-06-09 16:22:11.575226 7f869af78700 5 -- op tracker -- , seq: 2018, time: 2014-06-09 16:22:11.575225, event: started, op: pg_info(1 pgs e243:3.300) 2014-06-09 16:22:11.575235 7f869af78700 5 -- op tracker -- , seq: 2018, time: 2014-06-09 16:22:11.575234, event: done, op: pg_info(1 pgs e243:3.300) 2014-06-09 16:22:11.575239 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575240 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575247 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575248 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575248 7f869af78700 20 osd.26 243 _dispatch 0x19fdf500 pg_notify(3.48a(10) epoch 243) v5 2014-06-09 16:22:11.575251 7f869af78700 5 -- op tracker -- , seq: 2019, time: 2014-06-09 16:22:11.347570, event: header_read, op: pg_notify(3.48a(10) epoch 243) 2014-06-09 16:22:11.575255 7f869af78700 5 -- op tracker -- , seq: 2019, time: 2014-06-09 16:22:11.347571, event: throttled, op: pg_notify(3.48a(10) epoch 243) 2014-06-09 16:22:11.575258 7f869af78700 5 -- op tracker -- , seq: 2019, time: 2014-06-09 16:22:11.347602, event: all_read, op: pg_notify(3.48a(10) epoch 243) 2014-06-09 16:22:11.575261 7f869af78700 5 -- op tracker -- , seq: 2019, time: 2014-06-09 16:22:11.575247, event: dispatched, op: pg_notify(3.48a(10) epoch 243) 2014-06-09 16:22:11.575265 7f869af78700 5 -- op tracker -- , seq: 2019, time: 2014-06-09 16:22:11.575264, event: waiting_for_osdmap, op: pg_notify(3.48a(10) epoch 243) 2014-06-09 16:22:11.575267 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.575268 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdf500 2014-06-09 16:22:11.575270 7f869af78700 5 -- op tracker -- , seq: 2019, time: 2014-06-09 16:22:11.575269, event: started, op: pg_notify(3.48a(10) epoch 243) 2014-06-09 16:22:11.575279 7f869af78700 5 -- op tracker -- , seq: 2019, time: 2014-06-09 16:22:11.575278, event: done, op: pg_notify(3.48a(10) epoch 243) 2014-06-09 16:22:11.575283 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575284 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575293 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575294 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575294 7f869af78700 20 osd.26 243 _dispatch 0x16fdb880 pg_notify(3.74f(12),2.754(21) epoch 243) v5 2014-06-09 16:22:11.575297 7f869af78700 5 -- op tracker -- , seq: 2020, time: 2014-06-09 16:22:11.243633, event: header_read, op: pg_notify(3.74f(12),2.754(21) epoch 243) 2014-06-09 16:22:11.575302 7f869af78700 5 -- op tracker -- , seq: 2020, time: 2014-06-09 16:22:11.243634, event: throttled, op: pg_notify(3.74f(12),2.754(21) epoch 243) 2014-06-09 16:22:11.575305 7f869af78700 5 -- op tracker -- , seq: 2020, time: 2014-06-09 16:22:11.243676, event: all_read, op: pg_notify(3.74f(12),2.754(21) epoch 243) 2014-06-09 16:22:11.575308 7f869af78700 5 -- op tracker -- , seq: 2020, time: 2014-06-09 16:22:11.575293, event: dispatched, op: pg_notify(3.74f(12),2.754(21) epoch 243) 2014-06-09 16:22:11.575312 7f869af78700 5 -- op tracker -- , seq: 2020, time: 2014-06-09 16:22:11.575311, event: waiting_for_osdmap, op: pg_notify(3.74f(12),2.754(21) epoch 243) 2014-06-09 16:22:11.575314 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.575315 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16fdb880 2014-06-09 16:22:11.575318 7f869af78700 5 -- op tracker -- , seq: 2020, time: 2014-06-09 16:22:11.575316, event: started, op: pg_notify(3.74f(12),2.754(21) epoch 243) 2014-06-09 16:22:11.575331 7f869af78700 5 -- op tracker -- , seq: 2020, time: 2014-06-09 16:22:11.575330, event: done, op: pg_notify(3.74f(12),2.754(21) epoch 243) 2014-06-09 16:22:11.575335 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575336 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575351 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575352 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575353 7f869af78700 20 osd.26 243 _dispatch 0x3070ea00 pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.575356 7f869af78700 5 -- op tracker -- , seq: 2021, time: 2014-06-09 16:22:11.339512, event: header_read, op: pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.575360 7f869af78700 5 -- op tracker -- , seq: 2021, time: 2014-06-09 16:22:11.339512, event: throttled, op: pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.575364 7f869af78700 5 -- op tracker -- , seq: 2021, time: 2014-06-09 16:22:11.339535, event: all_read, op: pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.575368 7f869af78700 5 -- op tracker -- , seq: 2021, time: 2014-06-09 16:22:11.575351, event: dispatched, op: pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.575372 7f869af78700 5 -- op tracker -- , seq: 2021, time: 2014-06-09 16:22:11.575371, event: waiting_for_osdmap, op: pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.575374 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) v4 from osd.5 2014-06-09 16:22:11.575376 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070ea00 2014-06-09 16:22:11.575379 7f869af78700 5 -- op tracker -- , seq: 2021, time: 2014-06-09 16:22:11.575378, event: started, op: pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.575386 7f869af78700 5 -- op tracker -- , seq: 2021, time: 2014-06-09 16:22:11.575384, event: done, op: pg_log(3.730 epoch 243 log log((214'647,214'648], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.575390 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575390 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575397 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575398 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575399 7f869af78700 20 osd.26 243 _dispatch 0x306e7e00 pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.575402 7f869af78700 5 -- op tracker -- , seq: 2022, time: 2014-06-09 16:22:11.164435, event: header_read, op: pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575406 7f869af78700 5 -- op tracker -- , seq: 2022, time: 2014-06-09 16:22:11.164436, event: throttled, op: pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575410 7f869af78700 5 -- op tracker -- , seq: 2022, time: 2014-06-09 16:22:11.164445, event: all_read, op: pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575413 7f869af78700 5 -- op tracker -- , seq: 2022, time: 2014-06-09 16:22:11.575397, event: dispatched, op: pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575417 7f869af78700 5 -- op tracker -- , seq: 2022, time: 2014-06-09 16:22:11.575416, event: waiting_for_osdmap, op: pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575419 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.575421 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306e7e00 2014-06-09 16:22:11.575424 7f869af78700 5 -- op tracker -- , seq: 2022, time: 2014-06-09 16:22:11.575423, event: started, op: pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575431 7f869af78700 5 -- op tracker -- , seq: 2022, time: 2014-06-09 16:22:11.575430, event: done, op: pg_log(2.3e8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575435 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575436 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575442 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575443 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575444 7f869af78700 20 osd.26 243 _dispatch 0x2ea5ba40 pg_notify(1.238(20) epoch 243) v5 2014-06-09 16:22:11.575446 7f869af78700 5 -- op tracker -- , seq: 2023, time: 2014-06-09 16:22:11.168288, event: header_read, op: pg_notify(1.238(20) epoch 243) 2014-06-09 16:22:11.575450 7f869af78700 5 -- op tracker -- , seq: 2023, time: 2014-06-09 16:22:11.168288, event: throttled, op: pg_notify(1.238(20) epoch 243) 2014-06-09 16:22:11.575453 7f869af78700 5 -- op tracker -- , seq: 2023, time: 2014-06-09 16:22:11.168316, event: all_read, op: pg_notify(1.238(20) epoch 243) 2014-06-09 16:22:11.575456 7f869af78700 5 -- op tracker -- , seq: 2023, time: 2014-06-09 16:22:11.575442, event: dispatched, op: pg_notify(1.238(20) epoch 243) 2014-06-09 16:22:11.575459 7f869af78700 5 -- op tracker -- , seq: 2023, time: 2014-06-09 16:22:11.575458, event: waiting_for_osdmap, op: pg_notify(1.238(20) epoch 243) 2014-06-09 16:22:11.575461 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.575462 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5ba40 2014-06-09 16:22:11.575465 7f869af78700 5 -- op tracker -- , seq: 2023, time: 2014-06-09 16:22:11.575464, event: started, op: pg_notify(1.238(20) epoch 243) 2014-06-09 16:22:11.575473 7f869af78700 5 -- op tracker -- , seq: 2023, time: 2014-06-09 16:22:11.575472, event: done, op: pg_notify(1.238(20) epoch 243) 2014-06-09 16:22:11.575477 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575478 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575488 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575489 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575489 7f869af78700 20 osd.26 243 _dispatch 0x1fc09dc0 pg_info(1 pgs e243:2.e4) v4 2014-06-09 16:22:11.575492 7f869af78700 5 -- op tracker -- , seq: 2024, time: 2014-06-09 16:22:11.339697, event: header_read, op: pg_info(1 pgs e243:2.e4) 2014-06-09 16:22:11.575496 7f869af78700 5 -- op tracker -- , seq: 2024, time: 2014-06-09 16:22:11.339697, event: throttled, op: pg_info(1 pgs e243:2.e4) 2014-06-09 16:22:11.575499 7f869af78700 5 -- op tracker -- , seq: 2024, time: 2014-06-09 16:22:11.339708, event: all_read, op: pg_info(1 pgs e243:2.e4) 2014-06-09 16:22:11.575502 7f869af78700 5 -- op tracker -- , seq: 2024, time: 2014-06-09 16:22:11.575488, event: dispatched, op: pg_info(1 pgs e243:2.e4) 2014-06-09 16:22:11.575505 7f869af78700 5 -- op tracker -- , seq: 2024, time: 2014-06-09 16:22:11.575504, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.e4) 2014-06-09 16:22:11.575508 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.e4) v4 from osd.7 2014-06-09 16:22:11.575509 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc09dc0 2014-06-09 16:22:11.575512 7f869af78700 5 -- op tracker -- , seq: 2024, time: 2014-06-09 16:22:11.575511, event: started, op: pg_info(1 pgs e243:2.e4) 2014-06-09 16:22:11.575520 7f869af78700 5 -- op tracker -- , seq: 2024, time: 2014-06-09 16:22:11.575519, event: done, op: pg_info(1 pgs e243:2.e4) 2014-06-09 16:22:11.575523 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575524 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575531 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575532 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575533 7f869af78700 20 osd.26 243 _dispatch 0x305b2c00 pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.575536 7f869af78700 5 -- op tracker -- , seq: 2025, time: 2014-06-09 16:22:11.328485, event: header_read, op: pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575540 7f869af78700 5 -- op tracker -- , seq: 2025, time: 2014-06-09 16:22:11.328485, event: throttled, op: pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575544 7f869af78700 5 -- op tracker -- , seq: 2025, time: 2014-06-09 16:22:11.328495, event: all_read, op: pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575547 7f869af78700 5 -- op tracker -- , seq: 2025, time: 2014-06-09 16:22:11.575531, event: dispatched, op: pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575551 7f869af78700 5 -- op tracker -- , seq: 2025, time: 2014-06-09 16:22:11.575550, event: waiting_for_osdmap, op: pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575554 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.575556 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b2c00 2014-06-09 16:22:11.575559 7f869af78700 5 -- op tracker -- , seq: 2025, time: 2014-06-09 16:22:11.575557, event: started, op: pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575566 7f869af78700 5 -- op tracker -- , seq: 2025, time: 2014-06-09 16:22:11.575564, event: done, op: pg_log(0.765 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575569 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575570 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575577 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575578 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575579 7f869af78700 20 osd.26 243 _dispatch 0x306d5600 pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.575582 7f869af78700 5 -- op tracker -- , seq: 2026, time: 2014-06-09 16:22:11.230669, event: header_read, op: pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575586 7f869af78700 5 -- op tracker -- , seq: 2026, time: 2014-06-09 16:22:11.230670, event: throttled, op: pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575590 7f869af78700 5 -- op tracker -- , seq: 2026, time: 2014-06-09 16:22:11.230680, event: all_read, op: pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575593 7f869af78700 5 -- op tracker -- , seq: 2026, time: 2014-06-09 16:22:11.575577, event: dispatched, op: pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575597 7f869af78700 5 -- op tracker -- , seq: 2026, time: 2014-06-09 16:22:11.575596, event: waiting_for_osdmap, op: pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575599 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.575601 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d5600 2014-06-09 16:22:11.575604 7f869af78700 5 -- op tracker -- , seq: 2026, time: 2014-06-09 16:22:11.575603, event: started, op: pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575611 7f869af78700 5 -- op tracker -- , seq: 2026, time: 2014-06-09 16:22:11.575610, event: done, op: pg_log(0.4c8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575622 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575624 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575636 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.575636 7f8689f56700 10 op_submit_finish 3206 2014-06-09 16:22:11.575637 7f869af78700 20 osd.26 243 _dispatch 0x29ec5340 pg_info(1 pgs e243:2.59f) v4 2014-06-09 16:22:11.575641 7f869af78700 5 -- op tracker -- , seq: 2027, time: 2014-06-09 16:22:11.294761, event: header_read, op: pg_info(1 pgs e243:2.59f) 2014-06-09 16:22:11.575648 7f869af78700 5 -- op tracker -- , seq: 2027, time: 2014-06-09 16:22:11.294761, event: throttled, op: pg_info(1 pgs e243:2.59f) 2014-06-09 16:22:11.575642 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:1.2aa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.575654 7f869af78700 5 -- op tracker -- , seq: 2027, time: 2014-06-09 16:22:11.294769, event: all_read, op: pg_info(1 pgs e243:2.59f) 2014-06-09 16:22:11.575660 7f869af78700 5 -- op tracker -- , seq: 2027, time: 2014-06-09 16:22:11.575634, event: dispatched, op: pg_info(1 pgs e243:2.59f) 2014-06-09 16:22:11.575655 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.575665 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.575666 7f869af78700 5 -- op tracker -- , seq: 2027, time: 2014-06-09 16:22:11.575664, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.59f) 2014-06-09 16:22:11.575670 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.59f) v4 from osd.9 2014-06-09 16:22:11.575673 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec5340 2014-06-09 16:22:11.575670 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.746( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.575677 7f869af78700 5 -- op tracker -- , seq: 2027, time: 2014-06-09 16:22:11.575675, event: started, op: pg_info(1 pgs e243:2.59f) 2014-06-09 16:22:11.575681 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.575692 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.575691 7f869af78700 5 -- op tracker -- , seq: 2027, time: 2014-06-09 16:22:11.575689, event: done, op: pg_info(1 pgs e243:2.59f) 2014-06-09 16:22:11.575698 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.575696 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.1b1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.580944 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.580950 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.580962 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.580963 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.580965 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.580967 7f869af78700 20 osd.26 243 _dispatch 0x1ae7c700 pg_info(1 pgs e243:0.622) v4 2014-06-09 16:22:11.580967 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.580974 7f869af78700 5 -- op tracker -- , seq: 2028, time: 2014-06-09 16:22:11.333775, event: header_read, op: pg_info(1 pgs e243:0.622) 2014-06-09 16:22:11.580975 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.b5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.580983 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.580985 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.580983 7f869af78700 5 -- op tracker -- , seq: 2028, time: 2014-06-09 16:22:11.333775, event: throttled, op: pg_info(1 pgs e243:0.622) 2014-06-09 16:22:11.580988 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.296903 7 0.000217 2014-06-09 16:22:11.580990 7f869af78700 5 -- op tracker -- , seq: 2028, time: 2014-06-09 16:22:11.333786, event: all_read, op: pg_info(1 pgs e243:0.622) 2014-06-09 16:22:11.580995 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.580996 7f869af78700 5 -- op tracker -- , seq: 2028, time: 2014-06-09 16:22:11.580963, event: dispatched, op: pg_info(1 pgs e243:0.622) 2014-06-09 16:22:11.581002 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581005 7f869af78700 5 -- op tracker -- , seq: 2028, time: 2014-06-09 16:22:11.581002, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.622) 2014-06-09 16:22:11.581009 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.622) v4 from osd.8 2014-06-09 16:22:11.581009 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581013 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ae7c700 2014-06-09 16:22:11.581015 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.581019 7f869af78700 5 -- op tracker -- , seq: 2028, time: 2014-06-09 16:22:11.581016, event: started, op: pg_info(1 pgs e243:0.622) 2014-06-09 16:22:11.581020 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581036 7f869af78700 5 -- op tracker -- , seq: 2028, time: 2014-06-09 16:22:11.581034, event: done, op: pg_info(1 pgs e243:0.622) 2014-06-09 16:22:11.581045 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581048 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581053 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.581056 7f8689f56700 5 queue_transactions existing osr(1.b5 0x384c690)/0x384c690 2014-06-09 16:22:11.581060 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581061 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581063 7f869af78700 20 osd.26 243 _dispatch 0x1d039880 pg_info(1 pgs e243:3.753(2)) v4 2014-06-09 16:22:11.581068 7f869af78700 5 -- op tracker -- , seq: 2029, time: 2014-06-09 16:22:11.347612, event: header_read, op: pg_info(1 pgs e243:3.753(2)) 2014-06-09 16:22:11.581075 7f869af78700 5 -- op tracker -- , seq: 2029, time: 2014-06-09 16:22:11.347617, event: throttled, op: pg_info(1 pgs e243:3.753(2)) 2014-06-09 16:22:11.575708 7f8692767700 10 op_submit_start 3207 2014-06-09 16:22:11.581083 7f869af78700 5 -- op tracker -- , seq: 2029, time: 2014-06-09 16:22:11.347632, event: all_read, op: pg_info(1 pgs e243:3.753(2)) 2014-06-09 16:22:11.581097 7f8692767700 5 queue_transactions (trailing journal) 3207 2014-06-09 16:22:11.581100 7f8692767700 5 queue_op 0x309a25a0 seq 3207 osr(1.3f0 0x384d400) 1883 bytes (queue has 21 ops and 38125 bytes) 2014-06-09 16:22:11.581101 7f869af78700 5 -- op tracker -- , seq: 2029, time: 2014-06-09 16:22:11.581059, event: dispatched, op: pg_info(1 pgs e243:3.753(2)) 2014-06-09 16:22:11.581108 7f869af78700 5 -- op tracker -- , seq: 2029, time: 2014-06-09 16:22:11.581105, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.753(2)) 2014-06-09 16:22:11.581112 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.753(2)) v4 from osd.6 2014-06-09 16:22:11.581116 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d039880 2014-06-09 16:22:11.581121 7f869af78700 5 -- op tracker -- , seq: 2029, time: 2014-06-09 16:22:11.581119, event: started, op: pg_info(1 pgs e243:3.753(2)) 2014-06-09 16:22:11.581138 7f869af78700 5 -- op tracker -- , seq: 2029, time: 2014-06-09 16:22:11.581135, event: done, op: pg_info(1 pgs e243:3.753(2)) 2014-06-09 16:22:11.581145 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581162 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581163 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581164 7f869af78700 20 osd.26 243 _dispatch 0x30725800 pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.581172 7f869af78700 5 -- op tracker -- , seq: 2030, time: 2014-06-09 16:22:11.243690, event: header_read, op: pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581179 7f869af78700 5 -- op tracker -- , seq: 2030, time: 2014-06-09 16:22:11.243690, event: throttled, op: pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581186 7f869af78700 5 -- op tracker -- , seq: 2030, time: 2014-06-09 16:22:11.243703, event: all_read, op: pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581194 7f869af78700 5 -- op tracker -- , seq: 2030, time: 2014-06-09 16:22:11.581161, event: dispatched, op: pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.575713 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52dc20 (1 active) 2014-06-09 16:22:11.581208 7f86a5610700 5 _do_op 0x30606840 seq 3206 osr(0.5a2 0x2bd7af0)/0x2bd7af0 start 2014-06-09 16:22:11.581212 7f86a5610700 10 _do_transaction on 0xc039080 2014-06-09 16:22:11.581212 7f8692767700 10 op_submit_finish 3207 2014-06-09 16:22:11.581201 7f869af78700 5 -- op tracker -- , seq: 2030, time: 2014-06-09 16:22:11.581198, event: waiting_for_osdmap, op: pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581215 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.581220 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581219 7f868df5e700 10 op_submit_start 3208 2014-06-09 16:22:11.581220 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30725800 2014-06-09 16:22:11.581222 7f868df5e700 5 queue_transactions (trailing journal) 3208 2014-06-09 16:22:11.581224 7f868df5e700 5 queue_op 0x2fcae660 seq 3208 osr(0.151 0x2bd65e0) 1997 bytes (queue has 21 ops and 38125 bytes) 2014-06-09 16:22:11.581228 7f868df5e700 10 op_submit_finish 3208 2014-06-09 16:22:11.581218 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.446( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.581228 7f869af78700 5 -- op tracker -- , seq: 2030, time: 2014-06-09 16:22:11.581224, event: started, op: pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581232 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.581243 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581235 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:242, info:2.b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.581243 7f869af78700 5 -- op tracker -- , seq: 2030, time: 2014-06-09 16:22:11.581240, event: done, op: pg_log(0.448 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581252 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581254 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581248 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.4a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.581250 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.581261 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.581259 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.581269 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581270 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581272 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581271 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.581273 7f869af78700 20 osd.26 243 _dispatch 0x30615000 pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.581266 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:2.1b3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.581277 7f868ff62700 10 op_submit_start 3209 2014-06-09 16:22:11.581279 7f868ff62700 5 queue_transactions (trailing journal) 3209 2014-06-09 16:22:11.581273 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.4d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.4d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.581279 7f869af78700 5 -- op tracker -- , seq: 2031, time: 2014-06-09 16:22:11.339541, event: header_read, op: pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581281 7f868ff62700 5 queue_op 0xd470960 seq 3209 osr(1.c6 0x384c610) 2006 bytes (queue has 21 ops and 38125 bytes) 2014-06-09 16:22:11.581285 7f868ff62700 10 op_submit_finish 3209 2014-06-09 16:22:11.581278 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.581289 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.581288 7f869af78700 5 -- op tracker -- , seq: 2031, time: 2014-06-09 16:22:11.339541, event: throttled, op: pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581290 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3206.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.581285 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.4d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.581294 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.581294 7f869af78700 5 -- op tracker -- , seq: 2031, time: 2014-06-09 16:22:11.339552, event: all_read, op: pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581300 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581291 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.581293 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.70c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.581301 7f869af78700 5 -- op tracker -- , seq: 2031, time: 2014-06-09 16:22:11.581270, event: dispatched, op: pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581309 7f869af78700 5 -- op tracker -- , seq: 2031, time: 2014-06-09 16:22:11.581305, event: waiting_for_osdmap, op: pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581306 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.581303 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 0.3a6( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581313 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.581316 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30615000 2014-06-09 16:22:11.581304 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.53( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.53( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.581321 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.581320 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581321 7f869af78700 5 -- op tracker -- , seq: 2031, time: 2014-06-09 16:22:11.581319, event: started, op: pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581324 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.581323 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.53( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.581334 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581333 7f86a5610700 10 _do_op 0x30606840 seq 3206 r = 0, finisher 0xbdf0200 0 2014-06-09 16:22:11.581327 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.302490 7 0.000203 2014-06-09 16:22:11.581326 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:2.17c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.581334 7f869af78700 5 -- op tracker -- , seq: 2031, time: 2014-06-09 16:22:11.581331, event: done, op: pg_log(2.5a4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581340 7f86a5610700 10 _finish_op 0x30606840 seq 3206 osr(0.5a2 0x2bd7af0)/0x2bd7af0 2014-06-09 16:22:11.581341 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581343 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581344 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52dc20 (0 active) 2014-06-09 16:22:11.581339 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.581338 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.31b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.31b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.581347 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301440 (1 active) 2014-06-09 16:22:11.581341 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.581349 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.581349 7f86a5610700 5 _do_op 0x309a25a0 seq 3207 osr(1.3f0 0x384d400)/0x384d400 start 2014-06-09 16:22:11.581352 7f86a5610700 10 _do_transaction on 0xc33d5c0 2014-06-09 16:22:11.581356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581357 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581350 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.31b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.581352 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581358 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581360 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581358 7f869af78700 20 osd.26 243 _dispatch 0x305f3e00 pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.581354 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.581361 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581365 7f869af78700 5 -- op tracker -- , seq: 2032, time: 2014-06-09 16:22:11.164480, event: header_read, op: pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581363 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.30e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.581372 7f869af78700 5 -- op tracker -- , seq: 2032, time: 2014-06-09 16:22:11.164481, event: throttled, op: pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581365 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.699( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581370 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.581377 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581379 7f869af78700 5 -- op tracker -- , seq: 2032, time: 2014-06-09 16:22:11.164490, event: all_read, op: pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581381 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.581375 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.581379 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581386 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581386 7f869af78700 5 -- op tracker -- , seq: 2032, time: 2014-06-09 16:22:11.581355, event: dispatched, op: pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581391 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.581385 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.699( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] exit Started/Stray 1.307887 7 0.000195 2014-06-09 16:22:11.581394 7f869af78700 5 -- op tracker -- , seq: 2032, time: 2014-06-09 16:22:11.581391, event: waiting_for_osdmap, op: pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581390 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.316( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.581399 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.581402 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f3e00 2014-06-09 16:22:11.581397 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.699( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.581407 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3207.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.581402 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.581410 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.581408 7f869af78700 5 -- op tracker -- , seq: 2032, time: 2014-06-09 16:22:11.581405, event: started, op: pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581412 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581408 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.699( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581420 7f869af78700 5 -- op tracker -- , seq: 2032, time: 2014-06-09 16:22:11.581418, event: done, op: pg_log(2.40a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581416 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.581418 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581426 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.581430 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581430 7f86a5610700 10 _do_op 0x309a25a0 seq 3207 r = 0, finisher 0xbd626c0 0 2014-06-09 16:22:11.581431 7f868ff62700 5 queue_transactions existing osr(0.3a6 0x2bd7330)/0x2bd7330 2014-06-09 16:22:11.581432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581434 7f868ff62700 10 op_submit_start 3210 2014-06-09 16:22:11.581427 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.581435 7f868ff62700 5 queue_transactions (trailing journal) 3210 2014-06-09 16:22:11.581435 7f86a5610700 10 _finish_op 0x309a25a0 seq 3207 osr(1.3f0 0x384d400)/0x384d400 2014-06-09 16:22:11.581426 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 1.211( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581437 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301440 (0 active) 2014-06-09 16:22:11.581437 7f868ff62700 5 queue_op 0xd471f80 seq 3210 osr(0.3a6 0x2bd7330) 1997 bytes (queue has 20 ops and 36618 bytes) 2014-06-09 16:22:11.581438 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581439 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fdb00 (1 active) 2014-06-09 16:22:11.581441 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.581437 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581444 7f86a5610700 5 _do_op 0x2fcae660 seq 3208 osr(0.151 0x2bd65e0)/0x2bd65e0 start 2014-06-09 16:22:11.581445 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581446 7f86a5610700 10 _do_transaction on 0x398ad80 2014-06-09 16:22:11.581447 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581448 7f869af78700 20 osd.26 243 _dispatch 0x305a8a00 pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.581452 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581445 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.211( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 0.788097 7 0.000216 2014-06-09 16:22:11.581455 7f869af78700 5 -- op tracker -- , seq: 2033, time: 2014-06-09 16:22:11.168331, event: header_read, op: pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581455 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.211( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.581464 7f869af78700 5 -- op tracker -- , seq: 2033, time: 2014-06-09 16:22:11.168332, event: throttled, op: pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581466 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.211( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581472 7f869af78700 5 -- op tracker -- , seq: 2033, time: 2014-06-09 16:22:11.168344, event: all_read, op: pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581475 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581480 7f869af78700 5 -- op tracker -- , seq: 2033, time: 2014-06-09 16:22:11.581444, event: dispatched, op: pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581484 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.581484 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.581489 7f868df5e700 5 queue_transactions existing osr(2.699 0x2ed1f10)/0x2ed1f10 2014-06-09 16:22:11.581484 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.581488 7f869af78700 5 -- op tracker -- , seq: 2033, time: 2014-06-09 16:22:11.581484, event: waiting_for_osdmap, op: pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581493 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.581497 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a8a00 2014-06-09 16:22:11.581492 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581498 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3208.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.581502 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.581504 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3f880,0x189ab340] 2014-06-09 16:22:11.581503 7f869af78700 5 -- op tracker -- , seq: 2033, time: 2014-06-09 16:22:11.581500, event: started, op: pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581509 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3f880,0x189ab340] 2014-06-09 16:22:11.581510 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.581510 7f868ff62700 10 op_submit_finish 3210 2014-06-09 16:22:11.581512 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.581515 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdf0200,0xbd626c0] 2014-06-09 16:22:11.581516 7f869af78700 5 -- op tracker -- , seq: 2033, time: 2014-06-09 16:22:11.581513, event: done, op: pg_log(0.512 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.581521 7f86a5610700 10 _do_op 0x2fcae660 seq 3208 r = 0, finisher 0xbd6d5a0 0 2014-06-09 16:22:11.581523 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581525 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581516 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.581518 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.581527 7f86a5610700 10 _finish_op 0x2fcae660 seq 3208 osr(0.151 0x2bd65e0)/0x2bd65e0 2014-06-09 16:22:11.581530 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fdb00 (0 active) 2014-06-09 16:22:11.581532 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62fe60 (1 active) 2014-06-09 16:22:11.581533 7f86a5610700 5 _do_op 0xd470960 seq 3209 osr(1.c6 0x384c610)/0x384c610 start 2014-06-09 16:22:11.581534 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.581535 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581536 7f86a5610700 10 _do_transaction on 0xbebd440 2014-06-09 16:22:11.581537 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581528 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.55d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581538 7f869af78700 20 osd.26 243 _dispatch 0x2a070c40 pg_info(1 pgs e243:2.28b) v4 2014-06-09 16:22:11.581539 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581541 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581541 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581542 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.581542 7f869af78700 5 -- op tracker -- , seq: 2034, time: 2014-06-09 16:22:11.339713, event: header_read, op: pg_info(1 pgs e243:2.28b) 2014-06-09 16:22:11.581545 7f8692767700 5 queue_transactions existing osr(1.211 0x384cc90)/0x384cc90 2014-06-09 16:22:11.581548 7f8692767700 10 op_submit_start 3211 2014-06-09 16:22:11.581549 7f8692767700 5 queue_transactions (trailing journal) 3211 2014-06-09 16:22:11.581549 7f869af78700 5 -- op tracker -- , seq: 2034, time: 2014-06-09 16:22:11.339713, event: throttled, op: pg_info(1 pgs e243:2.28b) 2014-06-09 16:22:11.581551 7f8692767700 5 queue_op 0x309a2540 seq 3211 osr(1.211 0x384cc90) 2099 bytes (queue has 21 ops and 38466 bytes) 2014-06-09 16:22:11.581546 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.117995 7 0.007127 2014-06-09 16:22:11.581553 7f8692767700 10 op_submit_finish 3211 2014-06-09 16:22:11.581555 7f869af78700 5 -- op tracker -- , seq: 2034, time: 2014-06-09 16:22:11.339724, event: all_read, op: pg_info(1 pgs e243:2.28b) 2014-06-09 16:22:11.581551 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] flushed 2014-06-09 16:22:11.581556 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.581561 7f869af78700 5 -- op tracker -- , seq: 2034, time: 2014-06-09 16:22:11.581535, event: dispatched, op: pg_info(1 pgs e243:2.28b) 2014-06-09 16:22:11.581556 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.581567 7f869af78700 5 -- op tracker -- , seq: 2034, time: 2014-06-09 16:22:11.581565, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.28b) 2014-06-09 16:22:11.581565 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581571 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.581571 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.28b) v4 from osd.7 2014-06-09 16:22:11.581574 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a070c40 2014-06-09 16:22:11.581568 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.581568 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.126( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581574 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581579 7f869af78700 5 -- op tracker -- , seq: 2034, time: 2014-06-09 16:22:11.581577, event: started, op: pg_info(1 pgs e243:2.28b) 2014-06-09 16:22:11.581580 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581583 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.581584 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.581586 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3209.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.581589 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.581583 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.581587 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.126( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.812851 7 0.000226 2014-06-09 16:22:11.581593 7f869af78700 5 -- op tracker -- , seq: 2034, time: 2014-06-09 16:22:11.581591, event: done, op: pg_info(1 pgs e243:2.28b) 2014-06-09 16:22:11.581592 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581600 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581602 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581598 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.581601 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.126( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.581607 7f86a5610700 10 _do_op 0xd470960 seq 3209 r = 0, finisher 0xbd6b480 0 2014-06-09 16:22:11.581610 7f86a5610700 10 _finish_op 0xd470960 seq 3209 osr(1.c6 0x384c610)/0x384c610 2014-06-09 16:22:11.581613 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdf0200,0xbd626c0] 2014-06-09 16:22:11.581614 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581614 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62fe60 (0 active) 2014-06-09 16:22:11.581615 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581615 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6d5a0,0xbd6b480] 2014-06-09 16:22:11.581616 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd9e0 (1 active) 2014-06-09 16:22:11.581611 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.126( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581617 7f869af78700 20 osd.26 243 _dispatch 0x3088d500 pg_info(1 pgs e243:3.1d5(2)) v4 2014-06-09 16:22:11.581618 7f86a5610700 5 _do_op 0xd471f80 seq 3210 osr(0.3a6 0x2bd7330)/0x2bd7330 start 2014-06-09 16:22:11.581620 7f86a5610700 10 _do_transaction on 0x3129500 2014-06-09 16:22:11.581622 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6d5a0,0xbd6b480] 2014-06-09 16:22:11.581624 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.581625 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581625 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.581622 7f869af78700 5 -- op tracker -- , seq: 2035, time: 2014-06-09 16:22:11.328526, event: header_read, op: pg_info(1 pgs e243:3.1d5(2)) 2014-06-09 16:22:11.581620 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581629 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7ccc0,0xcc82a60] 2014-06-09 16:22:11.581629 7f869af78700 5 -- op tracker -- , seq: 2035, time: 2014-06-09 16:22:11.328527, event: throttled, op: pg_info(1 pgs e243:3.1d5(2)) 2014-06-09 16:22:11.581635 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.581629 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.581635 7f869af78700 5 -- op tracker -- , seq: 2035, time: 2014-06-09 16:22:11.328538, event: all_read, op: pg_info(1 pgs e243:3.1d5(2)) 2014-06-09 16:22:11.581639 7f868ff62700 5 queue_transactions existing osr(0.55d 0x2bd77f0)/0x2bd77f0 2014-06-09 16:22:11.581632 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.581642 7f868ff62700 10 op_submit_start 3212 2014-06-09 16:22:11.581643 7f868ff62700 5 queue_transactions (trailing journal) 3212 2014-06-09 16:22:11.581637 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581641 7f869af78700 5 -- op tracker -- , seq: 2035, time: 2014-06-09 16:22:11.581614, event: dispatched, op: pg_info(1 pgs e243:3.1d5(2)) 2014-06-09 16:22:11.581644 7f868ff62700 5 queue_op 0xd471da0 seq 3212 osr(0.55d 0x2bd77f0) 1883 bytes (queue has 21 ops and 38343 bytes) 2014-06-09 16:22:11.581646 7f868ff62700 10 op_submit_finish 3212 2014-06-09 16:22:11.581646 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.581647 7f869af78700 5 -- op tracker -- , seq: 2035, time: 2014-06-09 16:22:11.581645, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1d5(2)) 2014-06-09 16:22:11.581652 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1d5(2)) v4 from osd.3 2014-06-09 16:22:11.581654 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088d500 2014-06-09 16:22:11.581654 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.581649 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.581658 7f869af78700 5 -- op tracker -- , seq: 2035, time: 2014-06-09 16:22:11.581657, event: started, op: pg_info(1 pgs e243:3.1d5(2)) 2014-06-09 16:22:11.581661 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.581659 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 1.3cb( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581668 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3210.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.581672 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.581670 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581674 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.581673 7f869af78700 5 -- op tracker -- , seq: 2035, time: 2014-06-09 16:22:11.581671, event: done, op: pg_info(1 pgs e243:3.1d5(2)) 2014-06-09 16:22:11.581680 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581673 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.581682 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581683 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581677 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.315826 7 0.000152 2014-06-09 16:22:11.581686 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.581688 7f8692767700 5 queue_transactions existing osr(2.126 0x2ed0670)/0x2ed0670 2014-06-09 16:22:11.581691 7f8692767700 10 op_submit_start 3213 2014-06-09 16:22:11.581691 7f8692767700 5 queue_transactions (trailing journal) 3213 2014-06-09 16:22:11.581693 7f8692767700 5 queue_op 0x309a24e0 seq 3213 osr(2.126 0x2ed0670) 1883 bytes (queue has 22 ops and 40226 bytes) 2014-06-09 16:22:11.581688 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.581694 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581695 7f8692767700 10 op_submit_finish 3213 2014-06-09 16:22:11.581696 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581697 7f869af78700 20 osd.26 243 _dispatch 0x120608c0 pg_info(1 pgs e243:3.76f(4)) v4 2014-06-09 16:22:11.581698 7f86a5610700 10 _do_op 0xd471f80 seq 3210 r = 0, finisher 0x3061e420 0 2014-06-09 16:22:11.581701 7f86a5610700 10 _finish_op 0xd471f80 seq 3210 osr(0.3a6 0x2bd7330)/0x2bd7330 2014-06-09 16:22:11.581698 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581702 7f869af78700 5 -- op tracker -- , seq: 2036, time: 2014-06-09 16:22:11.230686, event: header_read, op: pg_info(1 pgs e243:3.76f(4)) 2014-06-09 16:22:11.581700 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.581698 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=213 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.581710 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd9e0 (0 active) 2014-06-09 16:22:11.581709 7f869af78700 5 -- op tracker -- , seq: 2036, time: 2014-06-09 16:22:11.230686, event: throttled, op: pg_info(1 pgs e243:3.76f(4)) 2014-06-09 16:22:11.581712 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb200 (1 active) 2014-06-09 16:22:11.581707 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581713 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7ccc0,0xcc82a60] 2014-06-09 16:22:11.581714 7f86a5610700 5 _do_op 0x309a2540 seq 3211 osr(1.211 0x384cc90)/0x384cc90 start 2014-06-09 16:22:11.581716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd65720] 2014-06-09 16:22:11.581716 7f86a5610700 10 _do_transaction on 0x2fb7980 2014-06-09 16:22:11.581715 7f869af78700 5 -- op tracker -- , seq: 2036, time: 2014-06-09 16:22:11.230717, event: all_read, op: pg_info(1 pgs e243:3.76f(4)) 2014-06-09 16:22:11.581718 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd65720] 2014-06-09 16:22:11.581720 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.581721 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.581721 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581716 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.581712 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=213 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.9 3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.581721 7f869af78700 5 -- op tracker -- , seq: 2036, time: 2014-06-09 16:22:11.581694, event: dispatched, op: pg_info(1 pgs e243:3.76f(4)) 2014-06-09 16:22:11.581724 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581728 7f869af78700 5 -- op tracker -- , seq: 2036, time: 2014-06-09 16:22:11.581726, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.76f(4)) 2014-06-09 16:22:11.581732 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.76f(4)) v4 from osd.1 2014-06-09 16:22:11.581725 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=213 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.308592 7 0.000163 2014-06-09 16:22:11.581735 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x120608c0 2014-06-09 16:22:11.581739 7f869af78700 5 -- op tracker -- , seq: 2036, time: 2014-06-09 16:22:11.581737, event: started, op: pg_info(1 pgs e243:3.76f(4)) 2014-06-09 16:22:11.581737 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=213 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.581748 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=213 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581756 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.581755 7f869af78700 5 -- op tracker -- , seq: 2036, time: 2014-06-09 16:22:11.581752, event: done, op: pg_info(1 pgs e243:3.76f(4)) 2014-06-09 16:22:11.581764 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581758 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=213 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581766 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581769 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.581771 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3211.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.581773 7f868ff62700 5 queue_transactions existing osr(1.3cb 0x384d4f0)/0x384d4f0 2014-06-09 16:22:11.581767 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'477 -> 213'477 2014-06-09 16:22:11.581776 7f868ff62700 10 op_submit_start 3214 2014-06-09 16:22:11.581775 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.581777 7f868ff62700 5 queue_transactions (trailing journal) 3214 2014-06-09 16:22:11.581777 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581779 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581778 7f868ff62700 5 queue_op 0xd471800 seq 3214 osr(1.3cb 0x384d4f0) 2009 bytes (queue has 22 ops and 40238 bytes) 2014-06-09 16:22:11.581781 7f868ff62700 10 op_submit_finish 3214 2014-06-09 16:22:11.581780 7f869af78700 20 osd.26 243 _dispatch 0x3088d180 pg_info(1 pgs e243:2.25c) v4 2014-06-09 16:22:11.581784 7f869af78700 5 -- op tracker -- , seq: 2037, time: 2014-06-09 16:22:11.295192, event: header_read, op: pg_info(1 pgs e243:2.25c) 2014-06-09 16:22:11.581783 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:2.614( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.581791 7f869af78700 5 -- op tracker -- , seq: 2037, time: 2014-06-09 16:22:11.295193, event: throttled, op: pg_info(1 pgs e243:2.25c) 2014-06-09 16:22:11.581793 7f86a5610700 10 _do_op 0x309a2540 seq 3211 r = 0, finisher 0xbd71a00 0 2014-06-09 16:22:11.581796 7f869af78700 5 -- op tracker -- , seq: 2037, time: 2014-06-09 16:22:11.295209, event: all_read, op: pg_info(1 pgs e243:2.25c) 2014-06-09 16:22:11.581799 7f86a5610700 10 _finish_op 0x309a2540 seq 3211 osr(1.211 0x384cc90)/0x384cc90 2014-06-09 16:22:11.581801 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb200 (0 active) 2014-06-09 16:22:11.581796 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.581803 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653b00 (1 active) 2014-06-09 16:22:11.581802 7f869af78700 5 -- op tracker -- , seq: 2037, time: 2014-06-09 16:22:11.581777, event: dispatched, op: pg_info(1 pgs e243:2.25c) 2014-06-09 16:22:11.581805 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.581805 7f86a5610700 5 _do_op 0xd471da0 seq 3212 osr(0.55d 0x2bd77f0)/0x2bd77f0 start 2014-06-09 16:22:11.581807 7f86a5610700 10 _do_transaction on 0xc05e540 2014-06-09 16:22:11.581809 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd777e0] 2014-06-09 16:22:11.581808 7f869af78700 5 -- op tracker -- , seq: 2037, time: 2014-06-09 16:22:11.581806, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.25c) 2014-06-09 16:22:11.581803 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581812 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581813 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd777e0] 2014-06-09 16:22:11.581815 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.581813 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.25c) v4 from osd.9 2014-06-09 16:22:11.581816 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.581816 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088d180 2014-06-09 16:22:11.581809 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.4ef( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.581821 7f869af78700 5 -- op tracker -- , seq: 2037, time: 2014-06-09 16:22:11.581819, event: started, op: pg_info(1 pgs e243:2.25c) 2014-06-09 16:22:11.581821 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.581830 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.581834 7f869af78700 5 -- op tracker -- , seq: 2037, time: 2014-06-09 16:22:11.581832, event: done, op: pg_info(1 pgs e243:2.25c) 2014-06-09 16:22:11.581833 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.581842 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581842 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581842 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.581844 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581847 7f8692767700 5 queue_transactions existing osr(3.77 0x3ed0250)/0x3ed0250 2014-06-09 16:22:11.581850 7f8692767700 10 op_submit_start 3215 2014-06-09 16:22:11.581851 7f8692767700 5 queue_transactions (trailing journal) 3215 2014-06-09 16:22:11.581844 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.581852 7f8692767700 5 queue_op 0x309a2480 seq 3215 osr(3.77 0x3ed0250) 1332 bytes (queue has 22 ops and 39471 bytes) 2014-06-09 16:22:11.581855 7f8692767700 10 op_submit_finish 3215 2014-06-09 16:22:11.581856 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581857 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581856 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3212.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.581852 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.581858 7f869af78700 20 osd.26 243 _dispatch 0x308e28c0 pg_info(1 pgs e243:3.647) v4 2014-06-09 16:22:11.581860 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.581863 7f869af78700 5 -- op tracker -- , seq: 2038, time: 2014-06-09 16:22:11.333819, event: header_read, op: pg_info(1 pgs e243:3.647) 2014-06-09 16:22:11.581857 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.622( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.581863 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.581869 7f869af78700 5 -- op tracker -- , seq: 2038, time: 2014-06-09 16:22:11.333819, event: throttled, op: pg_info(1 pgs e243:3.647) 2014-06-09 16:22:11.581876 7f868ff62700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.581870 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.581875 7f869af78700 5 -- op tracker -- , seq: 2038, time: 2014-06-09 16:22:11.333831, event: all_read, op: pg_info(1 pgs e243:3.647) 2014-06-09 16:22:11.581880 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581880 7f86a5610700 10 _do_op 0xd471da0 seq 3212 r = 0, finisher 0xbd6c8e0 0 2014-06-09 16:22:11.581881 7f869af78700 5 -- op tracker -- , seq: 2038, time: 2014-06-09 16:22:11.581855, event: dispatched, op: pg_info(1 pgs e243:3.647) 2014-06-09 16:22:11.581885 7f86a5610700 10 _finish_op 0xd471da0 seq 3212 osr(0.55d 0x2bd77f0)/0x2bd77f0 2014-06-09 16:22:11.581888 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653b00 (0 active) 2014-06-09 16:22:11.581888 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x4eb80e0] 2014-06-09 16:22:11.581890 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7805a0 (1 active) 2014-06-09 16:22:11.581888 7f869af78700 5 -- op tracker -- , seq: 2038, time: 2014-06-09 16:22:11.581886, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.647) 2014-06-09 16:22:11.581891 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x4eb80e0] 2014-06-09 16:22:11.581892 7f86a5610700 5 _do_op 0x309a24e0 seq 3213 osr(2.126 0x2ed0670)/0x2ed0670 start 2014-06-09 16:22:11.581893 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.581892 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.647) v4 from osd.8 2014-06-09 16:22:11.581884 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.661( v 213'549 (0'0,213'549] local-les=208 n=103 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.581894 7f86a5610700 10 _do_transaction on 0xc2aed80 2014-06-09 16:22:11.581894 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.581895 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e28c0 2014-06-09 16:22:11.581899 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581892 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:178 2014-06-09 16:22:11.581900 7f869af78700 5 -- op tracker -- , seq: 2038, time: 2014-06-09 16:22:11.581898, event: started, op: pg_info(1 pgs e243:3.647) 2014-06-09 16:22:11.581897 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.581906 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.581908 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.581915 7f869af78700 5 -- op tracker -- , seq: 2038, time: 2014-06-09 16:22:11.581913, event: done, op: pg_info(1 pgs e243:3.647) 2014-06-09 16:22:11.581910 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:2.5f1( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.581912 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.581924 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581925 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581922 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.581929 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.581931 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.581923 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.25b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581934 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.581938 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.581938 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.581940 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.581941 7f869af78700 20 osd.26 243 _dispatch 0x22ac1c00 pg_notify(0.6b4(20),0.6bb(20),1.3f9(20),2.6c7(20) epoch 243) v5 2014-06-09 16:22:11.581942 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3213.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.581935 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.709( v 213'575 (0'0,213'575] local-les=191 n=111 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.581946 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.581941 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.259541 7 0.000175 2014-06-09 16:22:11.581948 7f869af78700 5 -- op tracker -- , seq: 2039, time: 2014-06-09 16:22:11.347637, event: header_read, op: pg_notify(0.6b4(20),0.6bb(20),1.3f9(20),2.6c7(20) epoch 243) 2014-06-09 16:22:11.581950 7f868ff62700 7 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.1 2014-06-09 16:22:11.581952 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.581956 7f869af78700 5 -- op tracker -- , seq: 2039, time: 2014-06-09 16:22:11.347638, event: throttled, op: pg_notify(0.6b4(20),0.6bb(20),1.3f9(20),2.6c7(20) epoch 243) 2014-06-09 16:22:11.581962 7f86a5610700 10 _do_op 0x309a24e0 seq 3213 r = 0, finisher 0xbccc520 0 2014-06-09 16:22:11.581963 7f869af78700 5 -- op tracker -- , seq: 2039, time: 2014-06-09 16:22:11.347736, event: all_read, op: pg_notify(0.6b4(20),0.6bb(20),1.3f9(20),2.6c7(20) epoch 243) 2014-06-09 16:22:11.581967 7f86a5610700 10 _finish_op 0x309a24e0 seq 3213 osr(2.126 0x2ed0670)/0x2ed0670 2014-06-09 16:22:11.581962 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.581970 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7805a0 (0 active) 2014-06-09 16:22:11.581970 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbef1f80] 2014-06-09 16:22:11.581972 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1e60 (1 active) 2014-06-09 16:22:11.581971 7f869af78700 5 -- op tracker -- , seq: 2039, time: 2014-06-09 16:22:11.581938, event: dispatched, op: pg_notify(0.6b4(20),0.6bb(20),1.3f9(20),2.6c7(20) epoch 243) 2014-06-09 16:22:11.581973 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbef1f80] 2014-06-09 16:22:11.581974 7f86a5610700 5 _do_op 0xd471800 seq 3214 osr(1.3cb 0x384d4f0)/0x384d4f0 start 2014-06-09 16:22:11.581975 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.581976 7f86a5610700 10 _do_transaction on 0xc2dea80 2014-06-09 16:22:11.581976 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.581960 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.1 info 3.709( v 213'575 (0'0,213'575] local-les=191 n=111 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.581972 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.581979 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.581981 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.581979 7f869af78700 5 -- op tracker -- , seq: 2039, time: 2014-06-09 16:22:11.581975, event: waiting_for_osdmap, op: pg_notify(0.6b4(20),0.6bb(20),1.3f9(20),2.6c7(20) epoch 243) 2014-06-09 16:22:11.581984 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.581986 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22ac1c00 2014-06-09 16:22:11.581980 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.581983 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.581992 7f869af78700 5 -- op tracker -- , seq: 2039, time: 2014-06-09 16:22:11.581988, event: started, op: pg_notify(0.6b4(20),0.6bb(20),1.3f9(20),2.6c7(20) epoch 243) 2014-06-09 16:22:11.581988 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.581993 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.219( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582012 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582011 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582016 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.582019 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.219( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.117172 7 0.000204 2014-06-09 16:22:11.582025 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3214.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582029 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582027 7f869af78700 5 -- op tracker -- , seq: 2039, time: 2014-06-09 16:22:11.582024, event: done, op: pg_notify(0.6b4(20),0.6bb(20),1.3f9(20),2.6c7(20) epoch 243) 2014-06-09 16:22:11.582033 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582028 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.219( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.582035 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582036 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582038 7f8692767700 5 queue_transactions existing osr(0.25b 0x2bd69f0)/0x2bd69f0 2014-06-09 16:22:11.582041 7f8692767700 10 op_submit_start 3216 2014-06-09 16:22:11.582041 7f8692767700 5 queue_transactions (trailing journal) 3216 2014-06-09 16:22:11.582042 7f8692767700 5 queue_op 0x309a2420 seq 3216 osr(0.25b 0x2bd69f0) 1883 bytes (queue has 21 ops and 37588 bytes) 2014-06-09 16:22:11.582045 7f8692767700 10 op_submit_finish 3216 2014-06-09 16:22:11.582038 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.219( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.582051 7f86a5610700 10 _do_op 0xd471800 seq 3214 r = 0, finisher 0xcd9e120 0 2014-06-09 16:22:11.582047 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.582047 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.582057 7f86a5610700 10 _finish_op 0xd471800 seq 3214 osr(1.3cb 0x384d4f0)/0x384d4f0 2014-06-09 16:22:11.582059 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1e60 (0 active) 2014-06-09 16:22:11.582059 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbebe3c0] 2014-06-09 16:22:11.582061 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff58c0 (1 active) 2014-06-09 16:22:11.582055 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.582063 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbebe3c0] 2014-06-09 16:22:11.582063 7f86a5610700 5 _do_op 0x309a2480 seq 3215 osr(3.77 0x3ed0250)/0x3ed0250 start 2014-06-09 16:22:11.582064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582065 7f86a5610700 10 _do_transaction on 0x336b8c0 2014-06-09 16:22:11.582066 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582063 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.582069 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582058 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.652( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582070 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582071 7f869af78700 20 osd.26 243 _dispatch 0x30a21600 pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.582071 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582074 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.582077 7f869af78700 5 -- op tracker -- , seq: 2040, time: 2014-06-09 16:22:11.243708, event: header_read, op: pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582078 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.652( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.324819 7 0.000708 2014-06-09 16:22:11.582084 7f869af78700 5 -- op tracker -- , seq: 2040, time: 2014-06-09 16:22:11.243709, event: throttled, op: pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582091 7f869af78700 5 -- op tracker -- , seq: 2040, time: 2014-06-09 16:22:11.245194, event: all_read, op: pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582088 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.652( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.582100 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582099 7f869af78700 5 -- op tracker -- , seq: 2040, time: 2014-06-09 16:22:11.582068, event: dispatched, op: pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582098 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.652( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.582106 7f869af78700 5 -- op tracker -- , seq: 2040, time: 2014-06-09 16:22:11.582103, event: waiting_for_osdmap, op: pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582110 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.582107 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.582114 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a21600 2014-06-09 16:22:11.582116 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582115 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3215.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582118 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582120 7f868ff62700 5 queue_transactions existing osr(0.219 0x2bd6ad0)/0x2bd6ad0 2014-06-09 16:22:11.582119 7f869af78700 5 -- op tracker -- , seq: 2040, time: 2014-06-09 16:22:11.582116, event: started, op: pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582116 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.582123 7f868ff62700 10 op_submit_start 3217 2014-06-09 16:22:11.582123 7f868ff62700 5 queue_transactions (trailing journal) 3217 2014-06-09 16:22:11.582125 7f868ff62700 5 queue_op 0x15996b40 seq 3217 osr(0.219 0x2bd6ad0) 1876 bytes (queue has 21 ops and 37455 bytes) 2014-06-09 16:22:11.582127 7f868ff62700 10 op_submit_finish 3217 2014-06-09 16:22:11.582124 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.582132 7f869af78700 5 -- op tracker -- , seq: 2040, time: 2014-06-09 16:22:11.582129, event: done, op: pg_log(2.4a5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582130 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.582138 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582142 7f86a5610700 10 _do_op 0x309a2480 seq 3215 r = 0, finisher 0x1a002160 0 2014-06-09 16:22:11.582148 7f86a5610700 10 _finish_op 0x309a2480 seq 3215 osr(3.77 0x3ed0250)/0x3ed0250 2014-06-09 16:22:11.582140 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.5b7( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582150 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff58c0 (0 active) 2014-06-09 16:22:11.582150 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd626c0] 2014-06-09 16:22:11.582152 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582152 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525200 (1 active) 2014-06-09 16:22:11.582151 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582153 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582154 7f86a5610700 5 _do_op 0x309a2420 seq 3216 osr(0.25b 0x2bd69f0)/0x2bd69f0 start 2014-06-09 16:22:11.582154 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd626c0] 2014-06-09 16:22:11.582155 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.582156 7f86a5610700 10 _do_transaction on 0xc038d80 2014-06-09 16:22:11.582156 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582154 7f869af78700 20 osd.26 243 _dispatch 0x1607aa80 pg_notify(2.5a7(20),0.5a8(20),1.2e0(23) epoch 243) v5 2014-06-09 16:22:11.582157 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582160 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582162 7f869af78700 5 -- op tracker -- , seq: 2041, time: 2014-06-09 16:22:11.342672, event: header_read, op: pg_notify(2.5a7(20),0.5a8(20),1.2e0(23) epoch 243) 2014-06-09 16:22:11.582158 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.881385 7 0.000182 2014-06-09 16:22:11.582167 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582171 7f8692767700 5 queue_transactions existing osr(0.652 0x2bd7b90)/0x2bd7b90 2014-06-09 16:22:11.582169 7f869af78700 5 -- op tracker -- , seq: 2041, time: 2014-06-09 16:22:11.342674, event: throttled, op: pg_notify(2.5a7(20),0.5a8(20),1.2e0(23) epoch 243) 2014-06-09 16:22:11.582173 7f8692767700 10 op_submit_start 3218 2014-06-09 16:22:11.582174 7f8692767700 5 queue_transactions (trailing journal) 3218 2014-06-09 16:22:11.582170 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.582176 7f8692767700 5 queue_op 0x309a23c0 seq 3218 osr(0.652 0x2bd7b90) 2009 bytes (queue has 21 ops and 38132 bytes) 2014-06-09 16:22:11.582178 7f8692767700 10 op_submit_finish 3218 2014-06-09 16:22:11.582176 7f869af78700 5 -- op tracker -- , seq: 2041, time: 2014-06-09 16:22:11.342748, event: all_read, op: pg_notify(2.5a7(20),0.5a8(20),1.2e0(23) epoch 243) 2014-06-09 16:22:11.582183 7f869af78700 5 -- op tracker -- , seq: 2041, time: 2014-06-09 16:22:11.582152, event: dispatched, op: pg_notify(2.5a7(20),0.5a8(20),1.2e0(23) epoch 243) 2014-06-09 16:22:11.582180 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.582181 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.582190 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582190 7f869af78700 5 -- op tracker -- , seq: 2041, time: 2014-06-09 16:22:11.582187, event: waiting_for_osdmap, op: pg_notify(2.5a7(20),0.5a8(20),1.2e0(23) epoch 243) 2014-06-09 16:22:11.582194 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.582188 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.582196 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1607aa80 2014-06-09 16:22:11.582192 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.616( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582197 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.582201 7f869af78700 5 -- op tracker -- , seq: 2041, time: 2014-06-09 16:22:11.582198, event: started, op: pg_notify(2.5a7(20),0.5a8(20),1.2e0(23) epoch 243) 2014-06-09 16:22:11.582203 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582205 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3216.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582207 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.582208 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582204 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.582210 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.616( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.350879 7 0.000196 2014-06-09 16:22:11.582220 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.616( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.582232 7f86a5610700 10 _do_op 0x309a2420 seq 3216 r = 0, finisher 0xc6e90e0 0 2014-06-09 16:22:11.582229 7f869af78700 5 -- op tracker -- , seq: 2041, time: 2014-06-09 16:22:11.582227, event: done, op: pg_notify(2.5a7(20),0.5a8(20),1.2e0(23) epoch 243) 2014-06-09 16:22:11.582229 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.616( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.582237 7f86a5610700 10 _finish_op 0x309a2420 seq 3216 osr(0.25b 0x2bd69f0)/0x2bd69f0 2014-06-09 16:22:11.582238 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582240 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582239 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd70aa0] 2014-06-09 16:22:11.582239 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525200 (0 active) 2014-06-09 16:22:11.582242 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd70aa0] 2014-06-09 16:22:11.582242 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4079e0 (1 active) 2014-06-09 16:22:11.582244 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582238 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.582244 7f86a5610700 5 _do_op 0x15996b40 seq 3217 osr(0.219 0x2bd6ad0)/0x2bd6ad0 start 2014-06-09 16:22:11.582245 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582246 7f86a5610700 10 _do_transaction on 0xbe69200 2014-06-09 16:22:11.582249 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582251 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582247 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.582253 7f868ff62700 5 queue_transactions existing osr(2.5b7 0x2ed1b70)/0x2ed1b70 2014-06-09 16:22:11.582256 7f868ff62700 10 op_submit_start 3219 2014-06-09 16:22:11.582257 7f868ff62700 5 queue_transactions (trailing journal) 3219 2014-06-09 16:22:11.582258 7f868ff62700 5 queue_op 0x300321e0 seq 3219 osr(2.5b7 0x2ed1b70) 2009 bytes (queue has 21 ops and 38258 bytes) 2014-06-09 16:22:11.582254 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.582261 7f868ff62700 10 op_submit_finish 3219 2014-06-09 16:22:11.582268 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582269 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582270 7f869af78700 20 osd.26 243 _dispatch 0x2fe70540 pg_notify(1.5e1(26) epoch 243) v5 2014-06-09 16:22:11.582264 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.39f( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.582274 7f869af78700 5 -- op tracker -- , seq: 2042, time: 2014-06-09 16:22:11.169594, event: header_read, op: pg_notify(1.5e1(26) epoch 243) 2014-06-09 16:22:11.582281 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582276 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.582281 7f869af78700 5 -- op tracker -- , seq: 2042, time: 2014-06-09 16:22:11.169594, event: throttled, op: pg_notify(1.5e1(26) epoch 243) 2014-06-09 16:22:11.582285 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582287 7f869af78700 5 -- op tracker -- , seq: 2042, time: 2014-06-09 16:22:11.169625, event: all_read, op: pg_notify(1.5e1(26) epoch 243) 2014-06-09 16:22:11.582292 7f869af78700 5 -- op tracker -- , seq: 2042, time: 2014-06-09 16:22:11.582267, event: dispatched, op: pg_notify(1.5e1(26) epoch 243) 2014-06-09 16:22:11.582295 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3217.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582289 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.668( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.582298 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582299 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582298 7f869af78700 5 -- op tracker -- , seq: 2042, time: 2014-06-09 16:22:11.582297, event: waiting_for_osdmap, op: pg_notify(1.5e1(26) epoch 243) 2014-06-09 16:22:11.582302 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.582302 7f8692767700 5 queue_transactions existing osr(0.616 0x2bd7950)/0x2bd7950 2014-06-09 16:22:11.582304 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe70540 2014-06-09 16:22:11.582305 7f8692767700 10 op_submit_start 3220 2014-06-09 16:22:11.582306 7f8692767700 5 queue_transactions (trailing journal) 3220 2014-06-09 16:22:11.582301 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.582307 7f8692767700 5 queue_op 0x309a2360 seq 3220 osr(0.616 0x2bd7950) 1883 bytes (queue has 22 ops and 40141 bytes) 2014-06-09 16:22:11.582310 7f8692767700 10 op_submit_finish 3220 2014-06-09 16:22:11.582309 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582309 7f869af78700 5 -- op tracker -- , seq: 2042, time: 2014-06-09 16:22:11.582307, event: started, op: pg_notify(1.5e1(26) epoch 243) 2014-06-09 16:22:11.582314 7f86a5610700 10 _do_op 0x15996b40 seq 3217 r = 0, finisher 0x139652a0 0 2014-06-09 16:22:11.582319 7f86a5610700 10 _finish_op 0x15996b40 seq 3217 osr(0.219 0x2bd6ad0)/0x2bd6ad0 2014-06-09 16:22:11.582312 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.582321 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd6e0] 2014-06-09 16:22:11.582321 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4079e0 (0 active) 2014-06-09 16:22:11.582313 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.672( v 214'615 (0'0,214'615] local-les=208 n=121 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.582324 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd6e0] 2014-06-09 16:22:11.582324 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3019e0 (1 active) 2014-06-09 16:22:11.582325 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582326 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582326 7f86a5610700 5 _do_op 0x309a23c0 seq 3218 osr(0.652 0x2bd7b90)/0x2bd7b90 start 2014-06-09 16:22:11.582323 7f869af78700 5 -- op tracker -- , seq: 2042, time: 2014-06-09 16:22:11.582321, event: done, op: pg_notify(1.5e1(26) epoch 243) 2014-06-09 16:22:11.582328 7f86a5610700 10 _do_transaction on 0xc2dae40 2014-06-09 16:22:11.582331 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582333 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582333 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582323 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.4fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582326 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.582336 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582335 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582339 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.582342 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.412084 7 0.001658 2014-06-09 16:22:11.582352 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582354 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582340 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:0.29d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.582355 7f869af78700 20 osd.26 243 _dispatch 0x30044600 pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.582352 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.582361 7f869af78700 5 -- op tracker -- , seq: 2043, time: 2014-06-09 16:22:11.168370, event: header_read, op: pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582357 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.582367 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582363 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.582368 7f869af78700 5 -- op tracker -- , seq: 2043, time: 2014-06-09 16:22:11.168371, event: throttled, op: pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582375 7f869af78700 5 -- op tracker -- , seq: 2043, time: 2014-06-09 16:22:11.168382, event: all_read, op: pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582372 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.582371 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:0.fa( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.582381 7f869af78700 5 -- op tracker -- , seq: 2043, time: 2014-06-09 16:22:11.582352, event: dispatched, op: pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582381 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.582382 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.582392 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582389 7f869af78700 5 -- op tracker -- , seq: 2043, time: 2014-06-09 16:22:11.582386, event: waiting_for_osdmap, op: pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582389 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.582393 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.582397 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30044600 2014-06-09 16:22:11.582401 7f869af78700 5 -- op tracker -- , seq: 2043, time: 2014-06-09 16:22:11.582399, event: started, op: pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582395 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.ff( v 213'612 (0'0,213'612] local-les=206 n=117 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.582407 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.582413 7f869af78700 5 -- op tracker -- , seq: 2043, time: 2014-06-09 16:22:11.582410, event: done, op: pg_log(1.26c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582418 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582417 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582422 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.582431 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582433 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582434 7f869af78700 20 osd.26 243 _dispatch 0x2a070e00 pg_info(1 pgs e243:0.290) v4 2014-06-09 16:22:11.582432 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3218.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582437 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582437 7f8692767700 5 queue_transactions existing osr(1.4fb 0x384d7f0)/0x384d7f0 2014-06-09 16:22:11.582440 7f8692767700 10 op_submit_start 3221 2014-06-09 16:22:11.582441 7f8692767700 5 queue_transactions (trailing journal) 3221 2014-06-09 16:22:11.582432 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582439 7f869af78700 5 -- op tracker -- , seq: 2044, time: 2014-06-09 16:22:11.339729, event: header_read, op: pg_info(1 pgs e243:0.290) 2014-06-09 16:22:11.582442 7f8692767700 5 queue_op 0x309a2300 seq 3221 osr(1.4fb 0x384d7f0) 1974 bytes (queue has 22 ops and 40239 bytes) 2014-06-09 16:22:11.582445 7f8692767700 10 op_submit_finish 3221 2014-06-09 16:22:11.582443 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582445 7f869af78700 5 -- op tracker -- , seq: 2044, time: 2014-06-09 16:22:11.339729, event: throttled, op: pg_info(1 pgs e243:0.290) 2014-06-09 16:22:11.582447 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.582451 7f869af78700 5 -- op tracker -- , seq: 2044, time: 2014-06-09 16:22:11.339744, event: all_read, op: pg_info(1 pgs e243:0.290) 2014-06-09 16:22:11.582448 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.582450 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.881788 7 0.000166 2014-06-09 16:22:11.582457 7f869af78700 5 -- op tracker -- , seq: 2044, time: 2014-06-09 16:22:11.582431, event: dispatched, op: pg_info(1 pgs e243:0.290) 2014-06-09 16:22:11.582459 7f86a5610700 10 _do_op 0x309a23c0 seq 3218 r = 0, finisher 0x2d739480 0 2014-06-09 16:22:11.582465 7f86a5610700 10 _finish_op 0x309a23c0 seq 3218 osr(0.652 0x2bd7b90)/0x2bd7b90 2014-06-09 16:22:11.582463 7f869af78700 5 -- op tracker -- , seq: 2044, time: 2014-06-09 16:22:11.582461, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.290) 2014-06-09 16:22:11.582460 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.582460 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.582468 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3019e0 (0 active) 2014-06-09 16:22:11.582467 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.290) v4 from osd.7 2014-06-09 16:22:11.582467 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc40cf20] 2014-06-09 16:22:11.582470 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd8c0 (1 active) 2014-06-09 16:22:11.582470 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a070e00 2014-06-09 16:22:11.582471 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc40cf20] 2014-06-09 16:22:11.582471 7f86a5610700 5 _do_op 0x300321e0 seq 3219 osr(2.5b7 0x2ed1b70)/0x2ed1b70 start 2014-06-09 16:22:11.582474 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582474 7f86a5610700 10 _do_transaction on 0xc130e40 2014-06-09 16:22:11.582475 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582470 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.582470 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.582475 7f869af78700 5 -- op tracker -- , seq: 2044, time: 2014-06-09 16:22:11.582473, event: started, op: pg_info(1 pgs e243:0.290) 2014-06-09 16:22:11.582479 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582478 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.582480 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.582489 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.582490 7f869af78700 5 -- op tracker -- , seq: 2044, time: 2014-06-09 16:22:11.582488, event: done, op: pg_info(1 pgs e243:0.290) 2014-06-09 16:22:11.582488 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.582496 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582498 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582495 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.582510 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582511 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582511 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582506 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.582512 7f869af78700 20 osd.26 243 _dispatch 0x30888c00 pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.582520 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582519 7f869af78700 5 -- op tracker -- , seq: 2045, time: 2014-06-09 16:22:11.328550, event: header_read, op: pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582525 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3219.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582529 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582526 7f869af78700 5 -- op tracker -- , seq: 2045, time: 2014-06-09 16:22:11.328551, event: throttled, op: pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582524 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.582533 7f869af78700 5 -- op tracker -- , seq: 2045, time: 2014-06-09 16:22:11.328561, event: all_read, op: pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582537 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582539 7f869af78700 5 -- op tracker -- , seq: 2045, time: 2014-06-09 16:22:11.582510, event: dispatched, op: pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582541 7f868ff62700 5 queue_transactions existing osr(1.ab 0x384c6f0)/0x384c6f0 2014-06-09 16:22:11.582535 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 0.220( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582544 7f868ff62700 10 op_submit_start 3222 2014-06-09 16:22:11.582545 7f868ff62700 5 queue_transactions (trailing journal) 3222 2014-06-09 16:22:11.582545 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582546 7f868ff62700 5 queue_op 0x2ea529c0 seq 3222 osr(1.ab 0x384c6f0) 1880 bytes (queue has 22 ops and 40110 bytes) 2014-06-09 16:22:11.582548 7f868ff62700 10 op_submit_finish 3222 2014-06-09 16:22:11.582546 7f869af78700 5 -- op tracker -- , seq: 2045, time: 2014-06-09 16:22:11.582543, event: waiting_for_osdmap, op: pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582549 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.582549 7f86a5610700 10 _do_op 0x300321e0 seq 3219 r = 0, finisher 0xbcc92c0 0 2014-06-09 16:22:11.582550 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.582554 7f86a5610700 10 _finish_op 0x300321e0 seq 3219 osr(2.5b7 0x2ed1b70)/0x2ed1b70 2014-06-09 16:22:11.582556 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd8c0 (0 active) 2014-06-09 16:22:11.582556 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7b720] 2014-06-09 16:22:11.582558 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300c60 (1 active) 2014-06-09 16:22:11.582552 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.220( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.883415 7 0.007875 2014-06-09 16:22:11.582559 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7b720] 2014-06-09 16:22:11.582560 7f86a5610700 5 _do_op 0x309a2360 seq 3220 osr(0.616 0x2bd7950)/0x2bd7950 start 2014-06-09 16:22:11.582554 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30888c00 2014-06-09 16:22:11.582561 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582562 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582562 7f86a5610700 10 _do_transaction on 0xc2ae180 2014-06-09 16:22:11.582566 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0260 (19 active) 2014-06-09 16:22:11.582567 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582561 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.220( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.582566 7f869af78700 5 -- op tracker -- , seq: 2045, time: 2014-06-09 16:22:11.582563, event: started, op: pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582573 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.582572 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.220( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.582579 7f869af78700 5 -- op tracker -- , seq: 2045, time: 2014-06-09 16:22:11.582576, event: done, op: pg_log(1.1ef epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582586 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582587 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582582 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.582578 7f868ff62700 10 osd.26 243 dequeue_op 0x257c4900 prio 63 cost 217088 latency 0.798823 osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.582590 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.582599 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582599 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582600 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582601 7f869af78700 20 osd.26 243 _dispatch 0x307e68c0 pg_notify(1.757(21) epoch 243) v5 2014-06-09 16:22:11.582598 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.582602 7f868ff62700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:11.582597, event: reached_pg, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.582613 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3220.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582617 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582615 7f869af78700 5 -- op tracker -- , seq: 2046, time: 2014-06-09 16:22:11.230822, event: header_read, op: pg_notify(1.757(21) epoch 243) 2014-06-09 16:22:11.582613 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.582622 7f869af78700 5 -- op tracker -- , seq: 2046, time: 2014-06-09 16:22:11.230823, event: throttled, op: pg_notify(1.757(21) epoch 243) 2014-06-09 16:22:11.582628 7f869af78700 5 -- op tracker -- , seq: 2046, time: 2014-06-09 16:22:11.230853, event: all_read, op: pg_notify(1.757(21) epoch 243) 2014-06-09 16:22:11.582626 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x257c4900 2014-06-09 16:22:11.582633 7f869af78700 5 -- op tracker -- , seq: 2046, time: 2014-06-09 16:22:11.582598, event: dispatched, op: pg_notify(1.757(21) epoch 243) 2014-06-09 16:22:11.582635 7f868ff62700 10 osd.26 243 dequeue_op 0x257c4900 finish 2014-06-09 16:22:11.582639 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.582640 7f869af78700 5 -- op tracker -- , seq: 2046, time: 2014-06-09 16:22:11.582638, event: waiting_for_osdmap, op: pg_notify(1.757(21) epoch 243) 2014-06-09 16:22:11.582641 7f86a5610700 10 _do_op 0x309a2360 seq 3220 r = 0, finisher 0xbccd640 0 2014-06-09 16:22:11.582643 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582644 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.582645 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e68c0 2014-06-09 16:22:11.582647 7f86a5610700 10 _finish_op 0x309a2360 seq 3220 osr(0.616 0x2bd7950)/0x2bd7950 2014-06-09 16:22:11.582648 7f8692767700 5 queue_transactions existing osr(0.220 0x2bd6a70)/0x2bd6a70 2014-06-09 16:22:11.582650 7f8692767700 10 op_submit_start 3223 2014-06-09 16:22:11.582651 7f8692767700 5 queue_transactions (trailing journal) 3223 2014-06-09 16:22:11.582650 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300c60 (0 active) 2014-06-09 16:22:11.582652 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5680 (1 active) 2014-06-09 16:22:11.582650 7f869af78700 5 -- op tracker -- , seq: 2046, time: 2014-06-09 16:22:11.582648, event: started, op: pg_notify(1.757(21) epoch 243) 2014-06-09 16:22:11.582652 7f8692767700 5 queue_op 0x309a2240 seq 3223 osr(0.220 0x2bd6a70) 2009 bytes (queue has 21 ops and 38227 bytes) 2014-06-09 16:22:11.582654 7f8692767700 10 op_submit_finish 3223 2014-06-09 16:22:11.582654 7f86a5610700 5 _do_op 0x309a2300 seq 3221 osr(1.4fb 0x384d7f0)/0x384d7f0 start 2014-06-09 16:22:11.582656 7f86a5610700 10 _do_transaction on 0xc2afbc0 2014-06-09 16:22:11.582657 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0260 (20 active) 2014-06-09 16:22:11.582660 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582660 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:2.412( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.582668 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (19 active) 2014-06-09 16:22:11.582671 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a6e20] 2014-06-09 16:22:11.582671 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.582674 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.582677 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582676 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a6e20] 2014-06-09 16:22:11.582678 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582679 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582684 7f869af78700 5 -- op tracker -- , seq: 2046, time: 2014-06-09 16:22:11.582682, event: done, op: pg_notify(1.757(21) epoch 243) 2014-06-09 16:22:11.582680 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.285( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.582691 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582693 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582681 7f8692767700 10 osd.26 243 dequeue_op 0x10c54f00 prio 63 cost 925696 latency 1.487909 osd_op(client.4575.0:31981 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.582691 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.582703 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582705 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3221.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582703 7f8692767700 5 -- op tracker -- , seq: 87, time: 2014-06-09 16:22:11.582698, event: reached_pg, op: osd_op(client.4575.0:31981 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~925696] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.582709 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582711 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582712 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582713 7f869af78700 20 osd.26 243 _dispatch 0x3088c8c0 pg_notify(3.19b(10) epoch 243) v5 2014-06-09 16:22:11.582707 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.38c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.582711 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.582718 7f869af78700 5 -- op tracker -- , seq: 2047, time: 2014-06-09 16:22:11.314155, event: header_read, op: pg_notify(3.19b(10) epoch 243) 2014-06-09 16:22:11.582718 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.582725 7f86a5610700 10 _do_op 0x309a2300 seq 3221 r = 0, finisher 0xbd76900 0 2014-06-09 16:22:11.582725 7f869af78700 5 -- op tracker -- , seq: 2047, time: 2014-06-09 16:22:11.314156, event: throttled, op: pg_notify(3.19b(10) epoch 243) 2014-06-09 16:22:11.582730 7f86a5610700 10 _finish_op 0x309a2300 seq 3221 osr(1.4fb 0x384d7f0)/0x384d7f0 2014-06-09 16:22:11.582723 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x10c54f00 2014-06-09 16:22:11.582733 7f8692767700 10 osd.26 243 dequeue_op 0x10c54f00 finish 2014-06-09 16:22:11.582733 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5680 (0 active) 2014-06-09 16:22:11.582733 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbebf4c0] 2014-06-09 16:22:11.582731 7f869af78700 5 -- op tracker -- , seq: 2047, time: 2014-06-09 16:22:11.314172, event: all_read, op: pg_notify(3.19b(10) epoch 243) 2014-06-09 16:22:11.582735 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e360 (1 active) 2014-06-09 16:22:11.582736 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582736 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.582736 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbebf4c0] 2014-06-09 16:22:11.582737 7f86a5610700 5 _do_op 0x2ea529c0 seq 3222 osr(1.ab 0x384c6f0)/0x384c6f0 start 2014-06-09 16:22:11.582738 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582739 7f86a5610700 10 _do_transaction on 0xc26e3c0 2014-06-09 16:22:11.582739 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582738 7f869af78700 5 -- op tracker -- , seq: 2047, time: 2014-06-09 16:22:11.582710, event: dispatched, op: pg_notify(3.19b(10) epoch 243) 2014-06-09 16:22:11.582743 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582745 7f869af78700 5 -- op tracker -- , seq: 2047, time: 2014-06-09 16:22:11.582743, event: waiting_for_osdmap, op: pg_notify(3.19b(10) epoch 243) 2014-06-09 16:22:11.582741 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.c8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.582749 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.582751 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088c8c0 2014-06-09 16:22:11.582751 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (20 active) 2014-06-09 16:22:11.582755 7f869af78700 5 -- op tracker -- , seq: 2047, time: 2014-06-09 16:22:11.582753, event: started, op: pg_notify(3.19b(10) epoch 243) 2014-06-09 16:22:11.582751 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.582761 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582755 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.1aa( v 213'493 (0'0,213'493] local-les=191 n=97 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.582770 7f869af78700 5 -- op tracker -- , seq: 2047, time: 2014-06-09 16:22:11.582769, event: done, op: pg_notify(3.19b(10) epoch 243) 2014-06-09 16:22:11.582773 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582764 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.582769 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.1 2014-06-09 16:22:11.582778 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582780 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582779 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.b9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582788 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3222.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582791 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582790 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.582794 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582794 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.582795 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582780 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.1 info 3.1aa( v 213'493 (0'0,213'493] local-les=191 n=97 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.582796 7f869af78700 20 osd.26 243 _dispatch 0x308e3880 pg_info(1 pgs e243:2.349) v4 2014-06-09 16:22:11.582800 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582797 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.339545 7 0.000243 2014-06-09 16:22:11.582801 7f869af78700 5 -- op tracker -- , seq: 2048, time: 2014-06-09 16:22:11.333858, event: header_read, op: pg_info(1 pgs e243:2.349) 2014-06-09 16:22:11.582808 7f869af78700 5 -- op tracker -- , seq: 2048, time: 2014-06-09 16:22:11.333859, event: throttled, op: pg_info(1 pgs e243:2.349) 2014-06-09 16:22:11.582808 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.582805 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.44d( v 214'635 (0'0,214'635] local-les=213 n=122 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.582814 7f86a5610700 10 _do_op 0x2ea529c0 seq 3222 r = 0, finisher 0xc5a7b60 0 2014-06-09 16:22:11.582815 7f869af78700 5 -- op tracker -- , seq: 2048, time: 2014-06-09 16:22:11.333871, event: all_read, op: pg_info(1 pgs e243:2.349) 2014-06-09 16:22:11.582819 7f86a5610700 10 _finish_op 0x2ea529c0 seq 3222 osr(1.ab 0x384c6f0)/0x384c6f0 2014-06-09 16:22:11.582822 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e360 (0 active) 2014-06-09 16:22:11.582821 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1b0c00e0] 2014-06-09 16:22:11.582821 7f869af78700 5 -- op tracker -- , seq: 2048, time: 2014-06-09 16:22:11.582794, event: dispatched, op: pg_info(1 pgs e243:2.349) 2014-06-09 16:22:11.582824 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115b00 (1 active) 2014-06-09 16:22:11.582818 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.582825 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1b0c00e0] 2014-06-09 16:22:11.582826 7f86a5610700 5 _do_op 0x309a2240 seq 3223 osr(0.220 0x2bd6a70)/0x2bd6a70 start 2014-06-09 16:22:11.582827 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.582818 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.582828 7f86a5610700 10 _do_transaction on 0xc2cb500 2014-06-09 16:22:11.582828 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.582830 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582828 7f869af78700 5 -- op tracker -- , seq: 2048, time: 2014-06-09 16:22:11.582825, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.349) 2014-06-09 16:22:11.582832 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.349) v4 from osd.8 2014-06-09 16:22:11.582833 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.582828 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.582834 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e3880 2014-06-09 16:22:11.582839 7f869af78700 5 -- op tracker -- , seq: 2048, time: 2014-06-09 16:22:11.582837, event: started, op: pg_info(1 pgs e243:2.349) 2014-06-09 16:22:11.582833 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.dc( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.582836 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.582844 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.582845 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.582855 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582853 7f869af78700 5 -- op tracker -- , seq: 2048, time: 2014-06-09 16:22:11.582851, event: done, op: pg_info(1 pgs e243:2.349) 2014-06-09 16:22:11.582861 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582862 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582863 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.582859 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.e2( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.582874 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.582875 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.582870 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.582877 7f869af78700 20 osd.26 243 _dispatch 0x22013880 pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) v4 2014-06-09 16:22:11.582877 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3223.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.582880 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582880 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.582882 7f869af78700 5 -- op tracker -- , seq: 2049, time: 2014-06-09 16:22:11.347742, event: header_read, op: pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) 2014-06-09 16:22:11.582891 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.582889 7f869af78700 5 -- op tracker -- , seq: 2049, time: 2014-06-09 16:22:11.347743, event: throttled, op: pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) 2014-06-09 16:22:11.582884 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.678( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.582895 7f868ff62700 5 queue_transactions existing osr(0.b9 0x2bd6400)/0x2bd6400 2014-06-09 16:22:11.582895 7f86a5610700 10 _do_op 0x309a2240 seq 3223 r = 0, finisher 0x3050ab40 0 2014-06-09 16:22:11.582897 7f868ff62700 10 op_submit_start 3224 2014-06-09 16:22:11.582898 7f868ff62700 5 queue_transactions (trailing journal) 3224 2014-06-09 16:22:11.582895 7f869af78700 5 -- op tracker -- , seq: 2049, time: 2014-06-09 16:22:11.347763, event: all_read, op: pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) 2014-06-09 16:22:11.582900 7f86a5610700 10 _finish_op 0x309a2240 seq 3223 osr(0.220 0x2bd6a70)/0x2bd6a70 2014-06-09 16:22:11.582895 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.582904 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582902 7f869af78700 5 -- op tracker -- , seq: 2049, time: 2014-06-09 16:22:11.582874, event: dispatched, op: pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) 2014-06-09 16:22:11.582904 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115b00 (0 active) 2014-06-09 16:22:11.582906 7f86a5610700 20 KeyValueStore::op_tp worker waiting 2014-06-09 16:22:11.582899 7f868ff62700 5 queue_op 0x2ea53440 seq 3224 osr(0.b9 0x2bd6400) 2006 bytes (queue has 20 ops and 36379 bytes) 2014-06-09 16:22:11.582916 7f868ff62700 10 op_submit_finish 3224 2014-06-09 16:22:11.582908 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.102( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.582921 7f869af78700 5 -- op tracker -- , seq: 2049, time: 2014-06-09 16:22:11.582906, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) 2014-06-09 16:22:11.582925 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) v4 from osd.6 2014-06-09 16:22:11.582919 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.582920 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.582929 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22013880 2014-06-09 16:22:11.582931 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582934 7f869af78700 5 -- op tracker -- , seq: 2049, time: 2014-06-09 16:22:11.582931, event: started, op: pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) 2014-06-09 16:22:11.587737 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:1.54c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.587751 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.587761 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.587757 7f869af78700 5 -- op tracker -- , seq: 2049, time: 2014-06-09 16:22:11.587755, event: done, op: pg_info(2 pgs e243:3.6ba(2),3.6cb(2)) 2014-06-09 16:22:11.587769 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.587771 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.587766 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:2.288( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.587777 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.587785 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.587786 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.587787 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.587788 7f869af78700 20 osd.26 243 _dispatch 0x30a21000 pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.587789 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:1.b8( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.587795 7f869af78700 5 -- op tracker -- , seq: 2050, time: 2014-06-09 16:22:11.245201, event: header_read, op: pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.587804 7f869af78700 5 -- op tracker -- , seq: 2050, time: 2014-06-09 16:22:11.245202, event: throttled, op: pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.587800 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.587810 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.587810 7f869af78700 5 -- op tracker -- , seq: 2050, time: 2014-06-09 16:22:11.246197, event: all_read, op: pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.587819 7f869af78700 5 -- op tracker -- , seq: 2050, time: 2014-06-09 16:22:11.587785, event: dispatched, op: pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.587813 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.380( v 213'672 (0'0,213'672] local-les=206 n=131 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.587826 7f869af78700 5 -- op tracker -- , seq: 2050, time: 2014-06-09 16:22:11.587823, event: waiting_for_osdmap, op: pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.587830 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) v4 from osd.0 2014-06-09 16:22:11.587826 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.587834 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a21000 2014-06-09 16:22:11.587836 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.587840 7f869af78700 5 -- op tracker -- , seq: 2050, time: 2014-06-09 16:22:11.587837, event: started, op: pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.587839 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.443( v 213'673 (0'0,213'673] local-les=191 n=125 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.587852 7f869af78700 5 -- op tracker -- , seq: 2050, time: 2014-06-09 16:22:11.587849, event: done, op: pg_log(3.72c epoch 243 log log((0'0,213'555], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.587851 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.587859 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.587860 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.587861 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.587864 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.449( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.587872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.587873 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.587874 7f869af78700 20 osd.26 243 _dispatch 0x30613800 pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.587874 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.587885 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.587881 7f869af78700 5 -- op tracker -- , seq: 2051, time: 2014-06-09 16:22:11.342892, event: header_read, op: pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587890 7f869af78700 5 -- op tracker -- , seq: 2051, time: 2014-06-09 16:22:11.342893, event: throttled, op: pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587889 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.587896 7f869af78700 5 -- op tracker -- , seq: 2051, time: 2014-06-09 16:22:11.342907, event: all_read, op: pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587903 7f869af78700 5 -- op tracker -- , seq: 2051, time: 2014-06-09 16:22:11.587872, event: dispatched, op: pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587899 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.587911 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.587914 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.587912 7f869af78700 5 -- op tracker -- , seq: 2051, time: 2014-06-09 16:22:11.587910, event: waiting_for_osdmap, op: pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587918 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.587921 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30613800 2014-06-09 16:22:11.587918 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.222204 7 0.028545 2014-06-09 16:22:11.587926 7f869af78700 5 -- op tracker -- , seq: 2051, time: 2014-06-09 16:22:11.587924, event: started, op: pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587928 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.587939 7f869af78700 5 -- op tracker -- , seq: 2051, time: 2014-06-09 16:22:11.587936, event: done, op: pg_log(0.4ba epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587938 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.587948 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.587949 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.587947 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.587961 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.587962 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.587956 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.587963 7f869af78700 20 osd.26 243 _dispatch 0x305e3200 pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.587964 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.587969 7f869af78700 5 -- op tracker -- , seq: 2052, time: 2014-06-09 16:22:11.169636, event: header_read, op: pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587979 7f869af78700 5 -- op tracker -- , seq: 2052, time: 2014-06-09 16:22:11.169636, event: throttled, op: pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587987 7f869af78700 5 -- op tracker -- , seq: 2052, time: 2014-06-09 16:22:11.169648, event: all_read, op: pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.587995 7f869af78700 5 -- op tracker -- , seq: 2052, time: 2014-06-09 16:22:11.587961, event: dispatched, op: pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.582930 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 1.24c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588009 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588012 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.582941 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x13964b60] 2014-06-09 16:22:11.588013 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588004 7f869af78700 5 -- op tracker -- , seq: 2052, time: 2014-06-09 16:22:11.588001, event: waiting_for_osdmap, op: pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.588016 7f8692767700 5 queue_transactions existing osr(1.c9 0x384c9f0)/0x384c9f0 2014-06-09 16:22:11.588017 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x13964b60] 2014-06-09 16:22:11.588020 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.588022 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.588018 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.588016 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] exit Started/Stray 0.914647 7 0.000184 2014-06-09 16:22:11.582944 7f868f761700 10 op_submit_start 3225 2014-06-09 16:22:11.588024 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305e3200 2014-06-09 16:22:11.582956 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114360 (1 active) 2014-06-09 16:22:11.588030 7f868f761700 5 queue_transactions (trailing journal) 3225 2014-06-09 16:22:11.588029 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588032 7f868f761700 5 queue_op 0x30d97260 seq 3225 osr(0.2fd 0x2bd6e50) 1903 bytes (queue has 20 ops and 36368 bytes) 2014-06-09 16:22:11.588034 7f86a5e11700 5 _do_op 0x2ea53440 seq 3224 osr(0.b9 0x2bd6400)/0x2bd6400 start 2014-06-09 16:22:11.588036 7f868f761700 10 op_submit_finish 3225 2014-06-09 16:22:11.588038 7f86a5e11700 10 _do_transaction on 0xc2c6240 2014-06-09 16:22:11.588035 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588041 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588045 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.588046 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588051 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588042 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:3.2d5( v 213'531 (0'0,213'531] local-les=191 n=105 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.588058 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.588070 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.588082 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588075 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.588085 7f868ff62700 5 queue_transactions existing osr(1.24c 0x384cfd0)/0x384cfd0 2014-06-09 16:22:11.588087 7f868ff62700 10 op_submit_start 3226 2014-06-09 16:22:11.588087 7f868ff62700 5 queue_transactions (trailing journal) 3226 2014-06-09 16:22:11.588088 7f868ff62700 5 queue_op 0x2ea52cc0 seq 3226 osr(1.24c 0x384cfd0) 2052 bytes (queue has 21 ops and 38420 bytes) 2014-06-09 16:22:11.588090 7f868ff62700 10 op_submit_finish 3226 2014-06-09 16:22:11.588094 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.588086 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 1.408( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588092 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.5d4( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.588098 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588102 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588100 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.588106 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588108 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.588105 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.408( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.075706 7 0.000227 2014-06-09 16:22:11.588114 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.73b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588121 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588116 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.408( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588123 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588125 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.813091 7 0.000184 2014-06-09 16:22:11.588127 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.408( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588132 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588138 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588137 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588143 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588148 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588146 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588152 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588155 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588170 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3224.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.588176 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.588181 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588183 7f868ff62700 5 queue_transactions existing osr(1.73b 0x2ed00c0)/0x2ed00c0 2014-06-09 16:22:11.588185 7f868ff62700 10 op_submit_start 3227 2014-06-09 16:22:11.588185 7f868ff62700 5 queue_transactions (trailing journal) 3227 2014-06-09 16:22:11.588186 7f868ff62700 5 queue_op 0x2ea531a0 seq 3227 osr(1.73b 0x2ed00c0) 1883 bytes (queue has 22 ops and 40303 bytes) 2014-06-09 16:22:11.588188 7f868ff62700 10 op_submit_finish 3227 2014-06-09 16:22:11.588189 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.588197 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.9 3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.588204 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.588204 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.289638 7 0.000186 2014-06-09 16:22:11.588206 7f86a5e11700 10 _do_op 0x2ea53440 seq 3224 r = 0, finisher 0x306ce140 0 2014-06-09 16:22:11.588210 7f868f761700 5 queue_transactions existing osr(1.408 0x384d770)/0x384d770 2014-06-09 16:22:11.588213 7f868f761700 10 op_submit_start 3228 2014-06-09 16:22:11.588210 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588214 7f868f761700 5 queue_transactions (trailing journal) 3228 2014-06-09 16:22:11.588214 7f86a5e11700 10 _finish_op 0x2ea53440 seq 3224 osr(0.b9 0x2bd6400)/0x2bd6400 2014-06-09 16:22:11.588215 7f868f761700 5 queue_op 0x30d97200 seq 3228 osr(1.408 0x384d770) 2001 bytes (queue has 23 ops and 42304 bytes) 2014-06-09 16:22:11.588216 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588219 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114360 (0 active) 2014-06-09 16:22:11.588222 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6526c0 (1 active) 2014-06-09 16:22:11.588224 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d70a0] 2014-06-09 16:22:11.588222 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588227 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d70a0] 2014-06-09 16:22:11.588227 7f86a5e11700 5 _do_op 0x30d97260 seq 3225 osr(0.2fd 0x2bd6e50)/0x2bd6e50 start 2014-06-09 16:22:11.588229 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.588230 7f86a5e11700 10 _do_transaction on 0x36b3ec0 2014-06-09 16:22:11.588230 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.588228 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'606 -> 213'606 2014-06-09 16:22:11.588234 7f868f761700 10 op_submit_finish 3228 2014-06-09 16:22:11.588236 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.588239 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.588251 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.38a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588259 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588263 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588267 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588275 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.588270 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.331735 7 0.000428 2014-06-09 16:22:11.588282 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588285 7f868ff62700 5 queue_transactions existing osr(3.537 0x3ed1620)/0x3ed1620 2014-06-09 16:22:11.588286 7f868ff62700 10 op_submit_start 3229 2014-06-09 16:22:11.588287 7f868ff62700 5 queue_transactions (trailing journal) 3229 2014-06-09 16:22:11.588280 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588287 7f868ff62700 5 queue_op 0x14dd60c0 seq 3229 osr(3.537 0x3ed1620) 1335 bytes (queue has 23 ops and 41633 bytes) 2014-06-09 16:22:11.588289 7f868ff62700 10 op_submit_finish 3229 2014-06-09 16:22:11.588293 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3225.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.588290 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=213 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.588291 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588298 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.588299 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=213 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.9 3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.588300 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588306 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=213 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.251075 7 0.000162 2014-06-09 16:22:11.588309 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588312 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=213 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588316 7f86a5e11700 10 _do_op 0x30d97260 seq 3225 r = 0, finisher 0xc5a6ec0 0 2014-06-09 16:22:11.588318 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=213 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588317 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588322 7f86a5e11700 10 _finish_op 0x30d97260 seq 3225 osr(0.2fd 0x2bd6e50)/0x2bd6e50 2014-06-09 16:22:11.588325 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6526c0 (0 active) 2014-06-09 16:22:11.588324 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=213 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588328 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300000 (1 active) 2014-06-09 16:22:11.588330 7f86a5e11700 5 _do_op 0x2ea52cc0 seq 3226 osr(1.24c 0x384cfd0)/0x384cfd0 start 2014-06-09 16:22:11.588333 7f86a5e11700 10 _do_transaction on 0x336b2c0 2014-06-09 16:22:11.588330 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'594 -> 213'594 2014-06-09 16:22:11.588338 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.588361 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588364 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.588369 7f868f761700 5 queue_transactions existing osr(2.38a 0x2ed10f0)/0x2ed10f0 2014-06-09 16:22:11.588372 7f868f761700 10 op_submit_start 3230 2014-06-09 16:22:11.588373 7f868f761700 5 queue_transactions (trailing journal) 3230 2014-06-09 16:22:11.588371 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.588374 7f868f761700 5 queue_op 0x30d971a0 seq 3230 osr(2.38a 0x2ed10f0) 1883 bytes (queue has 23 ops and 41613 bytes) 2014-06-09 16:22:11.588377 7f868f761700 10 op_submit_finish 3230 2014-06-09 16:22:11.588382 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588385 7f868ff62700 5 queue_transactions existing osr(3.53d 0x3ed1600)/0x3ed1600 2014-06-09 16:22:11.588386 7f868ff62700 10 op_submit_start 3231 2014-06-09 16:22:11.588387 7f868ff62700 5 queue_transactions (trailing journal) 3231 2014-06-09 16:22:11.588387 7f868ff62700 5 queue_op 0x14dd6060 seq 3231 osr(3.53d 0x3ed1600) 1335 bytes (queue has 24 ops and 42948 bytes) 2014-06-09 16:22:11.588380 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.588389 7f868ff62700 10 op_submit_finish 3231 2014-06-09 16:22:11.588387 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3226.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.588392 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.588391 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.588391 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.67( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588398 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.517( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588402 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588404 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588406 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588406 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588408 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.517( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.880751 7 0.003908 2014-06-09 16:22:11.588409 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.67( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.376694 7 0.000238 2014-06-09 16:22:11.588414 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.517( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588415 7f86a5e11700 10 _do_op 0x2ea52cc0 seq 3226 r = 0, finisher 0xc40dee0 0 2014-06-09 16:22:11.588419 7f86a5e11700 10 _finish_op 0x2ea52cc0 seq 3226 osr(1.24c 0x384cfd0)/0x384cfd0 2014-06-09 16:22:11.588419 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.517( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588422 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300000 (0 active) 2014-06-09 16:22:11.588425 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f65a0 (1 active) 2014-06-09 16:22:11.588427 7f86a5e11700 5 _do_op 0x2ea531a0 seq 3227 osr(1.73b 0x2ed00c0)/0x2ed00c0 start 2014-06-09 16:22:11.588425 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588423 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.67( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588430 7f86a5e11700 10 _do_transaction on 0xc0dad80 2014-06-09 16:22:11.588430 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588435 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.588434 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588433 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.67( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588443 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588451 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588462 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588464 7f868ff62700 5 queue_transactions existing osr(1.517 0x384db70)/0x384db70 2014-06-09 16:22:11.588459 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588466 7f868ff62700 10 op_submit_start 3232 2014-06-09 16:22:11.588467 7f868ff62700 5 queue_transactions (trailing journal) 3232 2014-06-09 16:22:11.588467 7f868ff62700 5 queue_op 0x11fcf740 seq 3232 osr(1.517 0x384db70) 1954 bytes (queue has 24 ops and 42850 bytes) 2014-06-09 16:22:11.588469 7f868ff62700 10 op_submit_finish 3232 2014-06-09 16:22:11.588468 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.588470 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.588477 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 0.4c9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588483 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588485 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588484 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3227.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.588488 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.588487 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.722694 7 0.000311 2014-06-09 16:22:11.588492 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588498 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588503 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588507 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.588507 7f86a5e11700 10 _do_op 0x2ea531a0 seq 3227 r = 0, finisher 0x187425e0 0 2014-06-09 16:22:11.588507 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588510 7f86a5e11700 10 _finish_op 0x2ea531a0 seq 3227 osr(1.73b 0x2ed00c0)/0x2ed00c0 2014-06-09 16:22:11.588512 7f868f761700 5 queue_transactions existing osr(1.67 0x384c5b0)/0x384c5b0 2014-06-09 16:22:11.588513 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f65a0 (0 active) 2014-06-09 16:22:11.588515 7f868f761700 10 op_submit_start 3233 2014-06-09 16:22:11.588512 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588516 7f868f761700 5 queue_transactions (trailing journal) 3233 2014-06-09 16:22:11.588516 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4acea0 (1 active) 2014-06-09 16:22:11.588517 7f868f761700 5 queue_op 0x30d97140 seq 3233 osr(1.67 0x384c5b0) 2006 bytes (queue has 24 ops and 42973 bytes) 2014-06-09 16:22:11.588518 7f86a5e11700 5 _do_op 0x30d97200 seq 3228 osr(1.408 0x384d770)/0x384d770 start 2014-06-09 16:22:11.588520 7f868f761700 10 op_submit_finish 3233 2014-06-09 16:22:11.588520 7f86a5e11700 10 _do_transaction on 0xc1023c0 2014-06-09 16:22:11.588525 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.588523 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.588540 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588542 7f868ff62700 5 queue_transactions existing osr(0.4c9 0x2bd7650)/0x2bd7650 2014-06-09 16:22:11.588544 7f868ff62700 10 op_submit_start 3234 2014-06-09 16:22:11.588544 7f868ff62700 5 queue_transactions (trailing journal) 3234 2014-06-09 16:22:11.588537 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.588545 7f868ff62700 5 queue_op 0x11fce3c0 seq 3234 osr(0.4c9 0x2bd7650) 2009 bytes (queue has 25 ops and 44982 bytes) 2014-06-09 16:22:11.588547 7f868ff62700 10 op_submit_finish 3234 2014-06-09 16:22:11.588548 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:0.ef( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.588547 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.588557 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.588556 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.588562 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588555 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.588565 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:1.f2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.588569 7f868f761700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.588571 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.588573 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3228.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.588577 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588577 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.588579 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:2.3c7( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.588585 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.588590 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588587 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1015 2014-06-09 16:22:11.588592 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.588602 7f86a5e11700 10 _do_op 0x30d97200 seq 3228 r = 0, finisher 0xbd7cdc0 0 2014-06-09 16:22:11.588599 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.654( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588603 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.588606 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588606 7f86a5e11700 10 _finish_op 0x30d97200 seq 3228 osr(1.408 0x384d770)/0x384d770 2014-06-09 16:22:11.588608 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588609 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4acea0 (0 active) 2014-06-09 16:22:11.588612 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0ea0 (1 active) 2014-06-09 16:22:11.588609 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.654( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.316431 7 0.000184 2014-06-09 16:22:11.588614 7f86a5e11700 5 _do_op 0x14dd60c0 seq 3229 osr(3.537 0x3ed1620)/0x3ed1620 start 2014-06-09 16:22:11.588617 7f86a5e11700 10 _do_transaction on 0xc2c46c0 2014-06-09 16:22:11.588609 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.588615 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.654( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588622 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.588621 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.654( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588626 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588620 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 1.5c1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588631 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588631 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588635 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588635 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588638 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.744202 7 0.000243 2014-06-09 16:22:11.588648 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588654 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.588662 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588658 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588665 7f868ff62700 5 queue_transactions existing osr(2.654 0x2ed1fd0)/0x2ed1fd0 2014-06-09 16:22:11.588666 7f868ff62700 10 op_submit_start 3235 2014-06-09 16:22:11.588667 7f868ff62700 5 queue_transactions (trailing journal) 3235 2014-06-09 16:22:11.588667 7f868ff62700 5 queue_op 0x100381e0 seq 3235 osr(2.654 0x2ed1fd0) 1974 bytes (queue has 25 ops and 44955 bytes) 2014-06-09 16:22:11.588669 7f868ff62700 10 op_submit_finish 3235 2014-06-09 16:22:11.588671 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3229.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.588667 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588671 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.588676 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.588677 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.508( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588677 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588684 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588686 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588688 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.508( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.098446 7 0.000169 2014-06-09 16:22:11.588685 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588693 7f86a5e11700 10 _do_op 0x14dd60c0 seq 3229 r = 0, finisher 0x30509720 0 2014-06-09 16:22:11.588693 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.508( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588697 7f86a5e11700 10 _finish_op 0x14dd60c0 seq 3229 osr(3.537 0x3ed1620)/0x3ed1620 2014-06-09 16:22:11.588700 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0ea0 (0 active) 2014-06-09 16:22:11.588699 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.508( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588702 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301b00 (1 active) 2014-06-09 16:22:11.588705 7f86a5e11700 5 _do_op 0x30d971a0 seq 3230 osr(2.38a 0x2ed10f0)/0x2ed10f0 start 2014-06-09 16:22:11.588707 7f86a5e11700 10 _do_transaction on 0xc0ca9c0 2014-06-09 16:22:11.588704 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588709 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588712 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.588713 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588733 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.588738 7f868f761700 5 queue_transactions existing osr(1.5c1 0x384dd90)/0x384dd90 2014-06-09 16:22:11.588739 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.588740 7f868f761700 10 op_submit_start 3236 2014-06-09 16:22:11.588741 7f868f761700 5 queue_transactions (trailing journal) 3236 2014-06-09 16:22:11.588742 7f868ff62700 5 queue_transactions existing osr(1.508 0x384db90)/0x384db90 2014-06-09 16:22:11.588743 7f868f761700 5 queue_op 0x30d97080 seq 3236 osr(1.5c1 0x384dd90) 1997 bytes (queue has 26 ops and 47500 bytes) 2014-06-09 16:22:11.588745 7f868f761700 10 op_submit_finish 3236 2014-06-09 16:22:11.588744 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.588751 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.588758 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3230.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.588763 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.588763 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.588775 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.293000 7 0.000178 2014-06-09 16:22:11.588787 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588797 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588807 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588817 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'590 -> 213'590 2014-06-09 16:22:11.588855 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.588897 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.588902 7f868f761700 5 queue_transactions existing osr(3.254 0x3ed0c30)/0x3ed0c30 2014-06-09 16:22:11.588905 7f868f761700 10 op_submit_start 3237 2014-06-09 16:22:11.588906 7f868f761700 5 queue_transactions (trailing journal) 3237 2014-06-09 16:22:11.588907 7f868f761700 5 queue_op 0x30d97020 seq 3237 osr(3.254 0x3ed0c30) 1429 bytes (queue has 27 ops and 48929 bytes) 2014-06-09 16:22:11.588910 7f868f761700 10 op_submit_finish 3237 2014-06-09 16:22:11.588913 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.588923 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.458( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588934 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.588938 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.588941 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.458( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] exit Started/Stray 0.685529 7 0.000184 2014-06-09 16:22:11.588951 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.458( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.588961 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.458( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.588970 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.588978 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.588986 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589033 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589037 7f868f761700 5 queue_transactions existing osr(0.458 0x2bd7510)/0x2bd7510 2014-06-09 16:22:11.589040 7f868f761700 10 op_submit_start 3238 2014-06-09 16:22:11.589041 7f868f761700 5 queue_transactions (trailing journal) 3238 2014-06-09 16:22:11.589043 7f868f761700 5 queue_op 0x30d96fc0 seq 3238 osr(0.458 0x2bd7510) 2052 bytes (queue has 28 ops and 50981 bytes) 2014-06-09 16:22:11.589045 7f868f761700 10 op_submit_finish 3238 2014-06-09 16:22:11.589048 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.589059 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.6e7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589070 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589073 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589077 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 0.952581 7 0.000192 2014-06-09 16:22:11.589088 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589098 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589107 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589116 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589123 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589168 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589172 7f868f761700 5 queue_transactions existing osr(0.6e7 0x384c1b0)/0x384c1b0 2014-06-09 16:22:11.589175 7f868f761700 10 op_submit_start 3239 2014-06-09 16:22:11.589176 7f868f761700 5 queue_transactions (trailing journal) 3239 2014-06-09 16:22:11.589177 7f868f761700 5 queue_op 0x30d96f60 seq 3239 osr(0.6e7 0x384c1b0) 1876 bytes (queue has 29 ops and 52857 bytes) 2014-06-09 16:22:11.589179 7f868f761700 10 op_submit_finish 3239 2014-06-09 16:22:11.589182 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.589193 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 1.122( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589204 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589207 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589210 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.122( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.845171 7 0.000225 2014-06-09 16:22:11.589218 7f86a5e11700 10 _do_op 0x30d971a0 seq 3230 r = 0, finisher 0x4eb9f60 0 2014-06-09 16:22:11.589223 7f86a5e11700 10 _finish_op 0x30d971a0 seq 3230 osr(2.38a 0x2ed10f0)/0x2ed10f0 2014-06-09 16:22:11.589220 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.122( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589225 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301b00 (0 active) 2014-06-09 16:22:11.589228 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ca20 (1 active) 2014-06-09 16:22:11.589230 7f86a5e11700 5 _do_op 0x14dd6060 seq 3231 osr(3.53d 0x3ed1600)/0x3ed1600 start 2014-06-09 16:22:11.589232 7f86a5e11700 10 _do_transaction on 0xc33a180 2014-06-09 16:22:11.589230 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.122( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589238 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.589239 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589247 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589255 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589273 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.589289 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3231.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.589293 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.589302 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589307 7f868f761700 5 queue_transactions existing osr(1.122 0x384c890)/0x384c890 2014-06-09 16:22:11.589309 7f868f761700 10 op_submit_start 3240 2014-06-09 16:22:11.589310 7f868f761700 5 queue_transactions (trailing journal) 3240 2014-06-09 16:22:11.589309 7f86a5e11700 10 _do_op 0x14dd6060 seq 3231 r = 0, finisher 0x33a7fa0 0 2014-06-09 16:22:11.589312 7f868f761700 5 queue_op 0x30d96f00 seq 3240 osr(1.122 0x384c890) 2001 bytes (queue has 29 ops and 52975 bytes) 2014-06-09 16:22:11.589313 7f86a5e11700 10 _finish_op 0x14dd6060 seq 3231 osr(3.53d 0x3ed1600)/0x3ed1600 2014-06-09 16:22:11.589315 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ca20 (0 active) 2014-06-09 16:22:11.589317 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3387e0 (1 active) 2014-06-09 16:22:11.589322 7f86a5e11700 5 _do_op 0x11fcf740 seq 3232 osr(1.517 0x384db70)/0x384db70 start 2014-06-09 16:22:11.589325 7f86a5e11700 10 _do_transaction on 0xc103500 2014-06-09 16:22:11.589325 7f868f761700 10 op_submit_finish 3240 2014-06-09 16:22:11.589330 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.589331 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.589340 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.5be( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589347 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589349 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589351 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.813262 7 0.000189 2014-06-09 16:22:11.589351 7f868a757700 10 op_submit_start 3241 2014-06-09 16:22:11.589358 7f868a757700 5 queue_transactions (trailing journal) 3241 2014-06-09 16:22:11.589359 7f868a757700 5 queue_op 0x300619e0 seq 3241 osr(1.d1 0x384c9b0) 1880 bytes (queue has 28 ops and 51640 bytes) 2014-06-09 16:22:11.589363 7f868a757700 10 op_submit_finish 3241 2014-06-09 16:22:11.589362 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.589359 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589372 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (19 active) 2014-06-09 16:22:11.589371 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589380 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.589378 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3232.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.589383 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.589380 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589387 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589392 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589385 7f868a757700 10 osd.26 243 dequeue_op 0x257c4800 prio 63 cost 516096 latency 0.804051 osd_op(client.4576.0:36702 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~516096] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.589403 7f86a5e11700 10 _do_op 0x11fcf740 seq 3232 r = 0, finisher 0x20051d80 0 2014-06-09 16:22:11.589407 7f86a5e11700 10 _finish_op 0x11fcf740 seq 3232 osr(1.517 0x384db70)/0x384db70 2014-06-09 16:22:11.589410 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3387e0 (0 active) 2014-06-09 16:22:11.589412 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115320 (1 active) 2014-06-09 16:22:11.589414 7f86a5e11700 5 _do_op 0x30d97140 seq 3233 osr(1.67 0x384c5b0)/0x384c5b0 start 2014-06-09 16:22:11.589417 7f86a5e11700 10 _do_transaction on 0xc20b680 2014-06-09 16:22:11.589410 7f868a757700 5 -- op tracker -- , seq: 720, time: 2014-06-09 16:22:11.589405, event: reached_pg, op: osd_op(client.4576.0:36702 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~516096] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.589421 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.589421 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.589435 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589439 7f868f761700 5 queue_transactions existing osr(2.5be 0x2ed1b30)/0x2ed1b30 2014-06-09 16:22:11.589441 7f868f761700 10 op_submit_start 3242 2014-06-09 16:22:11.589441 7f868f761700 5 queue_transactions (trailing journal) 3242 2014-06-09 16:22:11.589442 7f868f761700 5 queue_op 0x30d96ea0 seq 3242 osr(2.5be 0x2ed1b30) 1883 bytes (queue has 28 ops and 51569 bytes) 2014-06-09 16:22:11.589444 7f868f761700 10 op_submit_finish 3242 2014-06-09 16:22:11.589435 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x257c4800 2014-06-09 16:22:11.589445 7f868a757700 10 osd.26 243 dequeue_op 0x257c4800 finish 2014-06-09 16:22:11.589449 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.589446 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=206 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.589454 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.589454 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=206 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.589462 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=206 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.755202 7 0.000169 2014-06-09 16:22:11.589468 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=206 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589470 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3233.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.589474 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.589475 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (20 active) 2014-06-09 16:22:11.589475 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=206 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589480 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=206 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589479 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.589487 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'571 -> 213'571 2014-06-09 16:22:11.589492 7f86a5e11700 10 _do_op 0x30d97140 seq 3233 r = 0, finisher 0xbd78f40 0 2014-06-09 16:22:11.589496 7f86a5e11700 10 _finish_op 0x30d97140 seq 3233 osr(1.67 0x384c5b0)/0x384c5b0 2014-06-09 16:22:11.589499 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115320 (0 active) 2014-06-09 16:22:11.589491 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 2.5f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589501 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcc60 (1 active) 2014-06-09 16:22:11.589503 7f86a5e11700 5 _do_op 0x11fce3c0 seq 3234 osr(0.4c9 0x2bd7650)/0x2bd7650 start 2014-06-09 16:22:11.589502 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589505 7f86a5e11700 10 _do_transaction on 0xc2e0cc0 2014-06-09 16:22:11.589506 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589510 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.589510 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.115323 7 0.000214 2014-06-09 16:22:11.589519 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589520 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589531 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589542 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.589540 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589552 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589549 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589557 7f868f761700 5 queue_transactions existing osr(3.139 0x3ed0780)/0x3ed0780 2014-06-09 16:22:11.589559 7f868f761700 10 op_submit_start 3243 2014-06-09 16:22:11.589560 7f868f761700 5 queue_transactions (trailing journal) 3243 2014-06-09 16:22:11.589557 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3234.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.589561 7f868f761700 5 queue_op 0x30d96e40 seq 3243 osr(3.139 0x3ed0780) 1335 bytes (queue has 28 ops and 50898 bytes) 2014-06-09 16:22:11.589562 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.589563 7f868f761700 10 op_submit_finish 3243 2014-06-09 16:22:11.589557 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589565 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.589572 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.654( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589579 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589581 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589582 7f86a5e11700 10 _do_op 0x11fce3c0 seq 3234 r = 0, finisher 0xc411000 0 2014-06-09 16:22:11.589586 7f86a5e11700 10 _finish_op 0x11fce3c0 seq 3234 osr(0.4c9 0x2bd7650)/0x2bd7650 2014-06-09 16:22:11.589583 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.654( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.324390 7 0.000220 2014-06-09 16:22:11.589588 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcc60 (0 active) 2014-06-09 16:22:11.589590 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359440 (1 active) 2014-06-09 16:22:11.589593 7f86a5e11700 5 _do_op 0x100381e0 seq 3235 osr(2.654 0x2ed1fd0)/0x2ed1fd0 start 2014-06-09 16:22:11.589590 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.654( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589595 7f86a5e11700 10 _do_transaction on 0x34883c0 2014-06-09 16:22:11.589596 7f868f761700 5 osd.26 pg_epoch: 243 pg[0.654( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589600 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.589605 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.589602 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589610 7f868a757700 5 queue_transactions existing osr(2.5f2 0x2ed1df0)/0x2ed1df0 2014-06-09 16:22:11.589613 7f868a757700 10 op_submit_start 3244 2014-06-09 16:22:11.589614 7f868a757700 5 queue_transactions (trailing journal) 3244 2014-06-09 16:22:11.589615 7f868a757700 5 queue_op 0x30061980 seq 3244 osr(2.5f2 0x2ed1df0) 2009 bytes (queue has 28 ops and 50898 bytes) 2014-06-09 16:22:11.589618 7f868a757700 10 op_submit_finish 3244 2014-06-09 16:22:11.589610 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589620 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589621 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:0.66b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.589632 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.589633 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.589643 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.589648 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3235.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.589653 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.589647 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.589657 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.2f2( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589667 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589667 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589671 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589672 7f868f761700 5 queue_transactions existing osr(0.654 0x2bd7b50)/0x2bd7b50 2014-06-09 16:22:11.589672 7f86a5e11700 10 _do_op 0x100381e0 seq 3235 r = 0, finisher 0x29282ca0 0 2014-06-09 16:22:11.589675 7f868f761700 10 op_submit_start 3245 2014-06-09 16:22:11.589676 7f868f761700 5 queue_transactions (trailing journal) 3245 2014-06-09 16:22:11.589675 7f86a5e11700 10 _finish_op 0x100381e0 seq 3235 osr(2.654 0x2ed1fd0)/0x2ed1fd0 2014-06-09 16:22:11.589678 7f868f761700 5 queue_op 0x30d96de0 seq 3245 osr(0.654 0x2bd7b50) 1883 bytes (queue has 28 ops and 50807 bytes) 2014-06-09 16:22:11.589678 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359440 (0 active) 2014-06-09 16:22:11.589681 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652d80 (1 active) 2014-06-09 16:22:11.589674 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.855665 7 0.000211 2014-06-09 16:22:11.589684 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589696 7f86a5e11700 5 _do_op 0x30d97080 seq 3236 osr(1.5c1 0x384dd90)/0x384dd90 start 2014-06-09 16:22:11.589699 7f86a5e11700 10 _do_transaction on 0x331f380 2014-06-09 16:22:11.589698 7f868f761700 10 op_submit_finish 3245 2014-06-09 16:22:11.589694 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589703 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.589703 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc480 (2 active) 2014-06-09 16:22:11.589707 7f86a5610700 5 _do_op 0x30d97020 seq 3237 osr(3.254 0x3ed0c30)/0x3ed0c30 start 2014-06-09 16:22:11.589709 7f86a5610700 10 _do_transaction on 0xc0e3800 2014-06-09 16:22:11.589704 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589703 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.582983 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061e420,0xbd71a00,0xbd6c8e0,0xbccc520,0xcd9e120,0x1a002160,0xc6e90e0,0x139652a0,0x2d739480,0xbcc92c0,0xbccd640,0xbd76900,0xc5a7b60,0x3050ab40] 2014-06-09 16:22:11.589716 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.589713 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589715 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.6f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589715 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.589721 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589727 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589731 7f8689755700 10 op_submit_start 3246 2014-06-09 16:22:11.589730 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589732 7f8689755700 5 queue_transactions (trailing journal) 3246 2014-06-09 16:22:11.589732 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.589734 7f8689755700 5 queue_op 0x30032060 seq 3246 osr(2.463 0x2ed1510) 1997 bytes (queue has 28 ops and 50807 bytes) 2014-06-09 16:22:11.589736 7f8689755700 10 op_submit_finish 3246 2014-06-09 16:22:11.589734 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.367475 7 0.000190 2014-06-09 16:22:11.589744 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0200 (19 active) 2014-06-09 16:22:11.589744 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589750 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.589750 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.589753 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589762 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589754 7f8689755700 10 osd.26 243 dequeue_op 0x11e63200 prio 63 cost 516096 latency 1.461174 osd_op(client.4575.0:31982 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.589768 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.589773 7f868a757700 5 queue_transactions existing osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 2014-06-09 16:22:11.589767 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.589776 7f868a757700 10 op_submit_start 3247 2014-06-09 16:22:11.589770 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589777 7f868a757700 5 queue_transactions (trailing journal) 3247 2014-06-09 16:22:11.589779 7f868a757700 5 queue_op 0x30061920 seq 3247 osr(2.2f2 0x2ed0fb0) 2001 bytes (queue has 29 ops and 52808 bytes) 2014-06-09 16:22:11.589782 7f868a757700 10 op_submit_finish 3247 2014-06-09 16:22:11.589781 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.589775 7f8689755700 5 -- op tracker -- , seq: 89, time: 2014-06-09 16:22:11.589770, event: reached_pg, op: osd_op(client.4575.0:31982 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 925696~516096] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.589778 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589784 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.589786 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.589797 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x11e63200 2014-06-09 16:22:11.589797 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.589796 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.2c0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589806 7f8689755700 10 osd.26 243 dequeue_op 0x11e63200 finish 2014-06-09 16:22:11.589808 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589810 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.589811 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589815 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.845437 7 0.000185 2014-06-09 16:22:11.589824 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589829 7f868f761700 5 queue_transactions existing osr(1.6f 0x384c570)/0x384c570 2014-06-09 16:22:11.589830 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0200 (20 active) 2014-06-09 16:22:11.589832 7f868f761700 10 op_submit_start 3248 2014-06-09 16:22:11.589833 7f868f761700 5 queue_transactions (trailing journal) 3248 2014-06-09 16:22:11.589827 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589834 7f868f761700 5 queue_op 0x30d96d80 seq 3248 osr(1.6f 0x384c570) 2006 bytes (queue has 30 ops and 54814 bytes) 2014-06-09 16:22:11.589837 7f868f761700 10 op_submit_finish 3248 2014-06-09 16:22:11.589837 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589836 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.589840 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.4ac( v 213'565 (0'0,213'565] local-les=210 n=109 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.589846 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589845 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.589849 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.589859 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.589853 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.589860 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.589855 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589864 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589863 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589863 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.589868 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.21b( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.589874 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.589883 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.589876 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.589879 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.589888 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589887 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.589892 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:2.6f4( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.589894 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.589897 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.487( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.589905 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.589902 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.589907 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.589911 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.589910 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.589915 7f868a757700 5 queue_transactions existing osr(1.2c0 0x384d170)/0x384d170 2014-06-09 16:22:11.589918 7f868a757700 10 op_submit_start 3249 2014-06-09 16:22:11.589909 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.589919 7f868a757700 5 queue_transactions (trailing journal) 3249 2014-06-09 16:22:11.589914 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.589920 7f868a757700 5 queue_op 0x300618c0 seq 3249 osr(1.2c0 0x384d170) 1883 bytes (queue has 31 ops and 56697 bytes) 2014-06-09 16:22:11.589923 7f868a757700 10 op_submit_finish 3249 2014-06-09 16:22:11.589921 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.589923 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.589932 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.589924 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.27c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589935 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.589926 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.2fb( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.589938 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.589935 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.589940 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.589941 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.093209 7 0.000208 2014-06-09 16:22:11.589939 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.589951 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.589946 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.382( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.589954 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.589955 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.589952 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.589956 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=201 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.589959 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.589961 7f868f761700 5 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.589970 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.589969 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.589969 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.589979 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.589969 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=201 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.589979 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.589982 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.589983 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=201 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.101272 7 0.000188 2014-06-09 16:22:11.589986 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.589986 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.589993 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.590002 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.589994 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=201 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590001 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.590006 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=201 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590006 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.590016 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=201 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590017 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.590029 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590033 7f868f761700 5 queue_transactions existing osr(1.27c 0x384cf30)/0x384cf30 2014-06-09 16:22:11.590027 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'693 -> 213'693 2014-06-09 16:22:11.590035 7f868f761700 10 op_submit_start 3250 2014-06-09 16:22:11.590036 7f868f761700 5 queue_transactions (trailing journal) 3250 2014-06-09 16:22:11.590030 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.052328 7 0.000162 2014-06-09 16:22:11.590037 7f868f761700 5 queue_op 0x30d96d20 seq 3250 osr(1.27c 0x384cf30) 1883 bytes (queue has 32 ops and 58580 bytes) 2014-06-09 16:22:11.590040 7f868f761700 10 op_submit_finish 3250 2014-06-09 16:22:11.590040 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590042 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.522( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.590050 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590054 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.590062 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590058 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=210 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590068 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'618 -> 213'618 2014-06-09 16:22:11.590076 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (19 active) 2014-06-09 16:22:11.590081 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.590076 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590018 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.590085 7f868f761700 10 osd.26 243 dequeue_op 0x257c4700 prio 63 cost 2580480 latency 0.803889 osd_op(client.4576.0:36703 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 733184~2580480] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.590097 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590106 7f868f761700 5 -- op tracker -- , seq: 724, time: 2014-06-09 16:22:11.590101, event: reached_pg, op: osd_op(client.4576.0:36703 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 733184~2580480] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.590111 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.590114 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.590110 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590119 7f868a757700 5 queue_transactions existing osr(3.5c6 0x3ed17f0)/0x3ed17f0 2014-06-09 16:22:11.590122 7f868a757700 10 op_submit_start 3251 2014-06-09 16:22:11.590113 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.590123 7f868a757700 5 queue_transactions (trailing journal) 3251 2014-06-09 16:22:11.590124 7f868a757700 5 queue_op 0x30061860 seq 3251 osr(3.5c6 0x3ed17f0) 1335 bytes (queue has 33 ops and 59915 bytes) 2014-06-09 16:22:11.590127 7f868a757700 10 op_submit_finish 3251 2014-06-09 16:22:11.590124 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.590124 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x257c4700 2014-06-09 16:22:11.590133 7f868f761700 10 osd.26 243 dequeue_op 0x257c4700 finish 2014-06-09 16:22:11.590137 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.590130 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:1.482( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.590140 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590149 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590143 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.590153 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (20 active) 2014-06-09 16:22:11.590153 7f8689755700 5 queue_transactions existing osr(3.2fd 0x3ed0de0)/0x3ed0de0 2014-06-09 16:22:11.590153 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.590155 7f8689755700 10 op_submit_start 3252 2014-06-09 16:22:11.590156 7f8689755700 5 queue_transactions (trailing journal) 3252 2014-06-09 16:22:11.590157 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.590158 7f8689755700 5 queue_op 0x30032000 seq 3252 osr(3.2fd 0x3ed0de0) 1335 bytes (queue has 34 ops and 61250 bytes) 2014-06-09 16:22:11.590160 7f8689755700 10 op_submit_finish 3252 2014-06-09 16:22:11.590156 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 7 notify: (query_epoch:242, epoch_sent:243, info:3.4ac( v 213'565 (0'0,213'565] local-les=210 n=109 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.590162 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:1.757( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.590164 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.590162 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.590167 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.590177 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590175 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.590175 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.590185 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.590186 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590180 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.590181 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590194 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.590190 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.590191 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.590191 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=213 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.590199 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.590201 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.590209 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590206 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.590205 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.590204 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=213 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.9 3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.590218 7f868f761700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.590213 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.590217 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=213 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.312195 7 0.000174 2014-06-09 16:22:11.590222 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.182( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590233 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590230 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.590237 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.590230 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=213 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590242 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590220 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590240 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.182( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] exit Started/Stray 0.917452 7 0.000174 2014-06-09 16:22:11.590242 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=213 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590250 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.590246 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:1.15e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.590249 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.182( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590252 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=213 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590258 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.182( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590257 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.590266 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590263 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'649 -> 213'649 2014-06-09 16:22:11.590266 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590266 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.590270 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.701( v 213'541 (0'0,213'541] local-les=206 n=108 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.590275 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.590281 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590282 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.590282 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590293 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590295 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.590296 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.60e( v 213'574 (0'0,213'574] local-les=191 n=109 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.590304 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590308 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.590310 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.590318 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590321 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590324 7f8689755700 5 queue_transactions existing osr(2.182 0x2ed0990)/0x2ed0990 2014-06-09 16:22:11.590327 7f8689755700 10 op_submit_start 3253 2014-06-09 16:22:11.590328 7f8689755700 5 queue_transactions (trailing journal) 3253 2014-06-09 16:22:11.590322 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.345( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.590329 7f8689755700 5 queue_op 0x14dd7140 seq 3253 osr(2.182 0x2ed0990) 1746 bytes (queue has 35 ops and 62996 bytes) 2014-06-09 16:22:11.590332 7f8689755700 10 op_submit_finish 3253 2014-06-09 16:22:11.590325 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590332 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.590339 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.590341 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590335 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.590346 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.590352 7f868bf5a700 10 op_submit_start 3254 2014-06-09 16:22:11.590347 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.590354 7f868bf5a700 5 queue_transactions (trailing journal) 3254 2014-06-09 16:22:11.590345 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=203 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.590357 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590356 7f868bf5a700 5 queue_op 0x308cb320 seq 3254 osr(1.3cd 0x384d4b0) 1954 bytes (queue has 35 ops and 62996 bytes) 2014-06-09 16:22:11.590359 7f868bf5a700 10 op_submit_finish 3254 2014-06-09 16:22:11.590356 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.590356 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=203 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.590360 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.590368 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (19 active) 2014-06-09 16:22:11.590374 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.590370 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.590369 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=203 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.312104 7 0.000178 2014-06-09 16:22:11.590380 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590378 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=203 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590383 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.590388 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=203 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590380 7f868bf5a700 10 osd.26 243 dequeue_op 0x11e63100 prio 63 cost 516096 latency 1.460945 osd_op(client.4575.0:31983 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.590393 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.590402 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590397 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=203 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590403 7f868bf5a700 5 -- op tracker -- , seq: 92, time: 2014-06-09 16:22:11.590397, event: reached_pg, op: osd_op(client.4575.0:31983 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 1441792~516096] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.590405 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.590407 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'549 -> 213'549 2014-06-09 16:22:11.590415 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.759( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590414 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.590426 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590429 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.590427 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x11e63100 2014-06-09 16:22:11.590436 7f868bf5a700 10 osd.26 243 dequeue_op 0x11e63100 finish 2014-06-09 16:22:11.590432 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.759( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.858688 7 0.000179 2014-06-09 16:22:11.590440 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.590442 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.759( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590446 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590451 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.759( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590459 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02e0 (20 active) 2014-06-09 16:22:11.590459 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590463 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.590467 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.590474 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590481 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590474 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.494( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590484 7f868f761700 5 queue_transactions existing osr(3.f0 0x3ed04a0)/0x3ed04a0 2014-06-09 16:22:11.590487 7f868f761700 10 op_submit_start 3255 2014-06-09 16:22:11.590488 7f868f761700 5 queue_transactions (trailing journal) 3255 2014-06-09 16:22:11.590486 7f868bf5a700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590489 7f868f761700 5 queue_op 0x30d96c60 seq 3255 osr(3.f0 0x3ed04a0) 1332 bytes (queue has 36 ops and 64328 bytes) 2014-06-09 16:22:11.590490 7f868bf5a700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.590491 7f868f761700 10 op_submit_finish 3255 2014-06-09 16:22:11.590493 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.494( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.790481 7 0.000216 2014-06-09 16:22:11.590494 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.428( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.590503 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.494( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590505 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.590514 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590515 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590519 7f8689755700 5 queue_transactions existing osr(2.759 0x3ed00b0)/0x3ed00b0 2014-06-09 16:22:11.590513 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.494( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590521 7f8689755700 10 op_submit_start 3256 2014-06-09 16:22:11.590522 7f8689755700 5 queue_transactions (trailing journal) 3256 2014-06-09 16:22:11.590523 7f8689755700 5 queue_op 0x14dd6de0 seq 3256 osr(2.759 0x3ed00b0) 1883 bytes (queue has 37 ops and 66211 bytes) 2014-06-09 16:22:11.590525 7f8689755700 10 op_submit_finish 3256 2014-06-09 16:22:11.590518 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.19f( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.590518 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590522 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590534 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.590527 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.590531 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.590529 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.590539 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.590540 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590546 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590548 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590550 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.590551 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.590551 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.590562 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.590571 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590563 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.590565 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590579 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.590574 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.590574 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.724107 7 0.000183 2014-06-09 16:22:11.590587 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.590584 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590583 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.6a8( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590593 7f868d75d700 10 op_submit_start 3257 2014-06-09 16:22:11.590593 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590595 7f868d75d700 5 queue_transactions (trailing journal) 3257 2014-06-09 16:22:11.590597 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.590597 7f868d75d700 5 queue_op 0x308ca000 seq 3257 osr(1.71e 0x2ed00b0) 1883 bytes (queue has 37 ops and 66211 bytes) 2014-06-09 16:22:11.590593 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590601 7f868d75d700 10 op_submit_finish 3257 2014-06-09 16:22:11.590596 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.590600 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.118207 7 0.009433 2014-06-09 16:22:11.590603 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=191 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590609 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061e420,0xbd71a00,0xbd6c8e0,0xbccc520,0xcd9e120,0x1a002160,0xc6e90e0,0x139652a0,0x2d739480,0xbcc92c0,0xbccd640,0xbd76900,0xc5a7b60,0x3050ab40] 2014-06-09 16:22:11.590606 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.590612 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'606 -> 213'606 2014-06-09 16:22:11.590617 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306ce140,0xc5a6ec0,0xc40dee0,0x187425e0,0xbd7cdc0,0x30509720,0x4eb9f60,0x33a7fa0,0x20051d80,0xbd78f40,0xc411000,0x29282ca0] 2014-06-09 16:22:11.590618 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590618 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.530( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590621 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.590631 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590627 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590635 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.590635 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.590636 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.590638 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.530( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.084900 7 0.000170 2014-06-09 16:22:11.590644 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590646 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.590649 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.530( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590653 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590660 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.530( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590670 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590675 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306ce140,0xc5a6ec0,0xc40dee0,0x187425e0,0xbd7cdc0,0x30509720,0x4eb9f60,0x33a7fa0,0x20051d80,0xbd78f40,0xc411000,0x29282ca0] 2014-06-09 16:22:11.590682 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.590683 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.590678 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.590686 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590687 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.590689 7f8689755700 5 queue_transactions existing osr(0.6a8 0x2bd7d90)/0x2bd7d90 2014-06-09 16:22:11.590690 7f868f761700 5 queue_transactions existing osr(3.18e 0x3ed0af0)/0x3ed0af0 2014-06-09 16:22:11.590692 7f8689755700 10 op_submit_start 3258 2014-06-09 16:22:11.590693 7f8689755700 5 queue_transactions (trailing journal) 3258 2014-06-09 16:22:11.590686 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590694 7f8689755700 5 queue_op 0x14dd6cc0 seq 3258 osr(0.6a8 0x2bd7d90) 1954 bytes (queue has 39 ops and 69594 bytes) 2014-06-09 16:22:11.590697 7f8689755700 10 op_submit_finish 3258 2014-06-09 16:22:11.590704 7f868b759700 10 op_submit_start 3259 2014-06-09 16:22:11.590706 7f868b759700 5 queue_transactions (trailing journal) 3259 2014-06-09 16:22:11.590708 7f868b759700 5 queue_op 0x308ca9c0 seq 3259 osr(3.7bc 0xb7f0190) 1429 bytes (queue has 39 ops and 69594 bytes) 2014-06-09 16:22:11.590709 7f868bf5a700 5 queue_transactions existing osr(1.494 0x384d990)/0x384d990 2014-06-09 16:22:11.590712 7f868b759700 10 op_submit_finish 3259 2014-06-09 16:22:11.590702 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.59f( v 213'624 (0'0,213'624] local-les=208 n=117 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.590714 7f868bf5a700 10 op_submit_start 3260 2014-06-09 16:22:11.590715 7f868bf5a700 5 queue_transactions (trailing journal) 3260 2014-06-09 16:22:11.590716 7f868bf5a700 5 queue_op 0x308cb2c0 seq 3260 osr(1.494 0x384d990) 1883 bytes (queue has 40 ops and 71477 bytes) 2014-06-09 16:22:11.590718 7f868bf5a700 10 op_submit_finish 3260 2014-06-09 16:22:11.590716 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.590727 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590718 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.590721 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.590733 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.590738 7f868d75d700 5 queue_transactions existing osr(0.530 0x2bd78f0)/0x2bd78f0 2014-06-09 16:22:11.590733 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.590730 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590742 7f868d75d700 10 op_submit_start 3261 2014-06-09 16:22:11.590743 7f868d75d700 5 queue_transactions (trailing journal) 3261 2014-06-09 16:22:11.590742 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590744 7f868d75d700 5 queue_op 0x308ca240 seq 3261 osr(0.530 0x2bd78f0) 1954 bytes (queue has 41 ops and 73431 bytes) 2014-06-09 16:22:11.590747 7f868d75d700 10 op_submit_finish 3261 2014-06-09 16:22:11.590746 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.590741 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.590732 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:1.109( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.590749 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.590750 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.302( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.389023 7 0.000223 2014-06-09 16:22:11.590750 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.590760 7f868bf5a700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.590756 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.590760 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.302( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590768 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.590762 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.3ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590774 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.590770 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.302( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590772 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.590778 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.590781 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0200 (19 active) 2014-06-09 16:22:11.590780 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.590788 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.590781 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.111033 7 0.000185 2014-06-09 16:22:11.590788 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.590796 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.590792 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590796 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.590800 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.590802 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.590813 7f8691f66700 10 op_submit_start 3262 2014-06-09 16:22:11.590795 7f8689755700 10 osd.26 243 dequeue_op 0x257c4600 prio 63 cost 880640 latency 0.797666 osd_op(client.4576.0:36704 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~880640] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.594415 7f8691f66700 5 queue_transactions (trailing journal) 3262 2014-06-09 16:22:11.590810 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.26d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594419 7f8691f66700 5 queue_op 0x308cba40 seq 3262 osr(0.578 0x2bd7800) 2099 bytes (queue has 41 ops and 73431 bytes) 2014-06-09 16:22:11.594424 7f8691f66700 10 op_submit_finish 3262 2014-06-09 16:22:11.594416 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594425 7f868bf5a700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594430 7f868ef60700 10 op_submit_start 3263 2014-06-09 16:22:11.594432 7f868ef60700 5 queue_transactions (trailing journal) 3263 2014-06-09 16:22:11.594431 7f868bf5a700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594434 7f868ef60700 5 queue_op 0x2fcaff80 seq 3263 osr(3.2b 0x3ed0350) 1050 bytes (queue has 41 ops and 73431 bytes) 2014-06-09 16:22:11.594437 7f868ef60700 10 op_submit_finish 3263 2014-06-09 16:22:11.594430 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.594430 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594435 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.258092 7 0.013220 2014-06-09 16:22:11.594439 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594440 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 0.640( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594448 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594451 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594444 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 242 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:242, info:3.6d8( v 213'642 (0'0,213'642] local-les=203 n=123 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.594450 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594453 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.640( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.357310 7 0.000186 2014-06-09 16:22:11.594460 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594458 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.594470 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.594464 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.640( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594469 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594476 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.640( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594477 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594476 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.594485 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594490 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.594484 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594485 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 2.298( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594491 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594496 7f868d75d700 5 queue_transactions existing osr(0.3ab 0x2bd72f0)/0x2bd72f0 2014-06-09 16:22:11.594496 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594499 7f868d75d700 10 op_submit_start 3264 2014-06-09 16:22:11.594496 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594501 7f868d75d700 5 queue_transactions (trailing journal) 3264 2014-06-09 16:22:11.594500 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594502 7f868d75d700 5 queue_op 0x308ca2a0 seq 3264 osr(0.3ab 0x2bd72f0) 1883 bytes (queue has 42 ops and 75314 bytes) 2014-06-09 16:22:11.594505 7f868d75d700 10 op_submit_finish 3264 2014-06-09 16:22:11.594503 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.298( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive NOTIFY] exit Started/Stray 1.193098 7 0.000156 2014-06-09 16:22:11.594509 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.594513 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.298( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594523 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.298( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594530 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.594521 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.9d( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594533 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.594533 7f8691f66700 5 queue_transactions existing osr(0.640 0x2bd7c10)/0x2bd7c10 2014-06-09 16:22:11.594535 7f8691f66700 10 op_submit_start 3265 2014-06-09 16:22:11.594533 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594536 7f8691f66700 5 queue_transactions (trailing journal) 3265 2014-06-09 16:22:11.594531 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594537 7f8691f66700 5 queue_op 0x308cb9e0 seq 3265 osr(0.640 0x2bd7c10) 2009 bytes (queue has 43 ops and 77323 bytes) 2014-06-09 16:22:11.594538 7f868bf5a700 5 queue_transactions existing osr(2.26d 0x2ed0de0)/0x2ed0de0 2014-06-09 16:22:11.594540 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594541 7f8691f66700 10 op_submit_finish 3265 2014-06-09 16:22:11.590789 7f869af78700 5 -- op tracker -- , seq: 2052, time: 2014-06-09 16:22:11.590786, event: started, op: pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.590898 7f868a757700 5 queue_transactions existing osr(3.35c 0x3ed10c0)/0x3ed10c0 2014-06-09 16:22:11.594547 7f868a757700 10 op_submit_start 3266 2014-06-09 16:22:11.594540 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594549 7f868a757700 5 queue_transactions (trailing journal) 3266 2014-06-09 16:22:11.594550 7f868a757700 5 queue_op 0x30061800 seq 3266 osr(3.35c 0x3ed10c0) 1335 bytes (queue has 45 ops and 80569 bytes) 2014-06-09 16:22:11.594544 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.352032 7 0.000180 2014-06-09 16:22:11.594553 7f868a757700 10 op_submit_finish 3266 2014-06-09 16:22:11.594549 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594546 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:3.72c( v 213'555 (0'0,213'555] local-les=191 n=112 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.594555 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594558 7f869af78700 5 -- op tracker -- , seq: 2052, time: 2014-06-09 16:22:11.594555, event: done, op: pg_log(2.52d epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.594557 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=213 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.594561 7f8691f66700 7 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.0 2014-06-09 16:22:11.594571 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594566 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594573 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594569 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=213 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.9 3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.594576 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594571 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.0 info 3.72c( v 213'555 (0'0,213'555] local-les=191 n=112 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.594589 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.594588 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594582 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=213 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.236209 7 0.000189 2014-06-09 16:22:11.594590 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.594590 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594585 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594592 7f869af78700 20 osd.26 243 _dispatch 0x1d038fc0 pg_info(2 pgs e243:3.527(2),3.52b(2)) v4 2014-06-09 16:22:11.594593 7f868ef60700 5 queue_transactions existing osr(2.298 0x2ed0c70)/0x2ed0c70 2014-06-09 16:22:11.594595 7f868ef60700 10 op_submit_start 3267 2014-06-09 16:22:11.594596 7f868ef60700 5 queue_transactions (trailing journal) 3267 2014-06-09 16:22:11.594598 7f868ef60700 5 queue_op 0x2fcaff20 seq 3267 osr(2.298 0x2ed0c70) 1621 bytes (queue has 46 ops and 82190 bytes) 2014-06-09 16:22:11.594592 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=213 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594593 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594600 7f868ef60700 10 op_submit_finish 3267 2014-06-09 16:22:11.594598 7f869af78700 5 -- op tracker -- , seq: 2053, time: 2014-06-09 16:22:11.168455, event: header_read, op: pg_info(2 pgs e243:3.527(2),3.52b(2)) 2014-06-09 16:22:11.594595 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.594607 7f869af78700 5 -- op tracker -- , seq: 2053, time: 2014-06-09 16:22:11.168455, event: throttled, op: pg_info(2 pgs e243:3.527(2),3.52b(2)) 2014-06-09 16:22:11.594603 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=213 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594603 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.373( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.594605 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 1.66e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594613 7f869af78700 5 -- op tracker -- , seq: 2053, time: 2014-06-09 16:22:11.168476, event: all_read, op: pg_info(2 pgs e243:3.527(2),3.52b(2)) 2014-06-09 16:22:11.594615 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594619 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594614 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=213 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594620 7f869af78700 5 -- op tracker -- , seq: 2053, time: 2014-06-09 16:22:11.594588, event: dispatched, op: pg_info(2 pgs e243:3.527(2),3.52b(2)) 2014-06-09 16:22:11.594617 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.594627 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.594622 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] exit Started/Stray 1.310821 7 0.000181 2014-06-09 16:22:11.594628 7f869af78700 5 -- op tracker -- , seq: 2053, time: 2014-06-09 16:22:11.594625, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.527(2),3.52b(2)) 2014-06-09 16:22:11.594624 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'611 -> 213'611 2014-06-09 16:22:11.594632 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.527(2),3.52b(2)) v4 from osd.4 2014-06-09 16:22:11.594636 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d038fc0 2014-06-09 16:22:11.594632 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594631 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.641( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.594643 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.594642 7f869af78700 5 -- op tracker -- , seq: 2053, time: 2014-06-09 16:22:11.594639, event: started, op: pg_info(2 pgs e243:3.527(2),3.52b(2)) 2014-06-09 16:22:11.594642 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594648 7f868d75d700 5 queue_transactions existing osr(0.9d 0x2bd6490)/0x2bd6490 2014-06-09 16:22:11.594643 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.594651 7f868d75d700 10 op_submit_start 3268 2014-06-09 16:22:11.594652 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.594653 7f868d75d700 5 queue_transactions (trailing journal) 3268 2014-06-09 16:22:11.594655 7f868d75d700 5 queue_op 0x308ca300 seq 3268 osr(0.9d 0x2bd6490) 2006 bytes (queue has 47 ops and 84196 bytes) 2014-06-09 16:22:11.594657 7f868d75d700 10 op_submit_finish 3268 2014-06-09 16:22:11.594651 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594656 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.594660 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594666 7f869af78700 5 -- op tracker -- , seq: 2053, time: 2014-06-09 16:22:11.594664, event: done, op: pg_info(2 pgs e243:3.527(2),3.52b(2)) 2014-06-09 16:22:11.594661 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=210 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.594669 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594667 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594674 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594676 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594666 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 0.644( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594678 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594681 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594675 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=210 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.594691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594685 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.644( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.337296 7 0.000178 2014-06-09 16:22:11.594693 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594694 7f869af78700 20 osd.26 243 _dispatch 0x2a070000 pg_notify(3.4c0(10),3.0(10),3.6(10) epoch 243) v5 2014-06-09 16:22:11.594688 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=210 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.144180 7 0.000168 2014-06-09 16:22:11.594694 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.644( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594700 7f869af78700 5 -- op tracker -- , seq: 2054, time: 2014-06-09 16:22:11.339748, event: header_read, op: pg_notify(3.4c0(10),3.0(10),3.6(10) epoch 243) 2014-06-09 16:22:11.594704 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.594699 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=210 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594707 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.594709 7f868a757700 5 queue_transactions existing osr(3.98 0x3ed0620)/0x3ed0620 2014-06-09 16:22:11.594704 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.644( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594708 7f869af78700 5 -- op tracker -- , seq: 2054, time: 2014-06-09 16:22:11.339748, event: throttled, op: pg_notify(3.4c0(10),3.0(10),3.6(10) epoch 243) 2014-06-09 16:22:11.594711 7f8691f66700 5 queue_transactions existing osr(1.66e 0x384ded0)/0x384ded0 2014-06-09 16:22:11.594711 7f868a757700 10 op_submit_start 3269 2014-06-09 16:22:11.594713 7f868a757700 5 queue_transactions (trailing journal) 3269 2014-06-09 16:22:11.594714 7f868a757700 5 queue_op 0x300617a0 seq 3269 osr(3.98 0x3ed0620) 1332 bytes (queue has 49 ops and 87149 bytes) 2014-06-09 16:22:11.594714 7f869af78700 5 -- op tracker -- , seq: 2054, time: 2014-06-09 16:22:11.339794, event: all_read, op: pg_notify(3.4c0(10),3.0(10),3.6(10) epoch 243) 2014-06-09 16:22:11.594717 7f868a757700 10 op_submit_finish 3269 2014-06-09 16:22:11.594713 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594711 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=210 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594721 7f869af78700 5 -- op tracker -- , seq: 2054, time: 2014-06-09 16:22:11.594690, event: dispatched, op: pg_notify(3.4c0(10),3.0(10),3.6(10) epoch 243) 2014-06-09 16:22:11.594726 7f8690f64700 10 op_submit_start 3270 2014-06-09 16:22:11.594727 7f8690f64700 5 queue_transactions (trailing journal) 3270 2014-06-09 16:22:11.594722 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594729 7f8690f64700 5 queue_op 0x2fcaf9e0 seq 3270 osr(2.30b 0x2ed0f50) 1997 bytes (queue has 49 ops and 87149 bytes) 2014-06-09 16:22:11.594732 7f8690f64700 10 op_submit_finish 3270 2014-06-09 16:22:11.594729 7f869af78700 5 -- op tracker -- , seq: 2054, time: 2014-06-09 16:22:11.594726, event: waiting_for_osdmap, op: pg_notify(3.4c0(10),3.0(10),3.6(10) epoch 243) 2014-06-09 16:22:11.594722 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=210 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594722 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.594733 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.594735 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a070000 2014-06-09 16:22:11.594729 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594734 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'720 -> 213'720 2014-06-09 16:22:11.594741 7f869af78700 5 -- op tracker -- , seq: 2054, time: 2014-06-09 16:22:11.594738, event: started, op: pg_notify(3.4c0(10),3.0(10),3.6(10) epoch 243) 2014-06-09 16:22:11.594735 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.24d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594736 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.116( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.594747 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594750 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594748 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.594758 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.594753 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.869400 7 0.000216 2014-06-09 16:22:11.594762 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:0.5e4( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.594765 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594772 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.594773 7f869af78700 5 -- op tracker -- , seq: 2054, time: 2014-06-09 16:22:11.594770, event: done, op: pg_notify(3.4c0(10),3.0(10),3.6(10) epoch 243) 2014-06-09 16:22:11.594776 7f868ef60700 5 queue_transactions existing osr(0.644 0x2bd7bf0)/0x2bd7bf0 2014-06-09 16:22:11.594779 7f868ef60700 10 op_submit_start 3271 2014-06-09 16:22:11.594780 7f868ef60700 5 queue_transactions (trailing journal) 3271 2014-06-09 16:22:11.594773 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.594774 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594781 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594782 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.594781 7f868ef60700 5 queue_op 0x2fcafec0 seq 3271 osr(0.644 0x2bd7bf0) 1883 bytes (queue has 50 ops and 89032 bytes) 2014-06-09 16:22:11.594782 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594784 7f868ef60700 10 op_submit_finish 3271 2014-06-09 16:22:11.594782 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594781 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594786 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.594787 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.594791 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594802 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594804 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594798 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594796 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 1.206( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594805 7f869af78700 20 osd.26 243 _dispatch 0x30718c00 pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.594798 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.67c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594807 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594809 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.594811 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594813 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.594812 7f869af78700 5 -- op tracker -- , seq: 2055, time: 2014-06-09 16:22:11.331770, event: header_read, op: pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.594820 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.594813 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.206( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.879032 7 0.000184 2014-06-09 16:22:11.594819 7f869af78700 5 -- op tracker -- , seq: 2055, time: 2014-06-09 16:22:11.331771, event: throttled, op: pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.594816 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] exit Started/Stray 1.287294 7 0.000235 2014-06-09 16:22:11.594825 7f868d75d700 5 queue_transactions existing osr(3.579 0x3ed18d0)/0x3ed18d0 2014-06-09 16:22:11.594827 7f868d75d700 2 waiting 51 > 50 ops || 90367 > 104857600 2014-06-09 16:22:11.594826 7f869af78700 5 -- op tracker -- , seq: 2055, time: 2014-06-09 16:22:11.331783, event: all_read, op: pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.594824 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.206( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.594826 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.590889 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a6020,0x2d739820,0xbd75400,0xbd7b720,0xbef10e0,0xbd70140,0xbd7c5a0,0xcd9ea80,0xbd631a0,0xbd6be00,0xbd63460] 2014-06-09 16:22:11.594833 7f869af78700 5 -- op tracker -- , seq: 2055, time: 2014-06-09 16:22:11.594802, event: dispatched, op: pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.594833 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.206( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594840 7f869af78700 5 -- op tracker -- , seq: 2055, time: 2014-06-09 16:22:11.594837, event: waiting_for_osdmap, op: pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.594836 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.594843 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.594835 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.594844 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.594847 7f868a757700 5 queue_transactions existing osr(2.24d 0x2ed0a50)/0x2ed0a50 2014-06-09 16:22:11.594842 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594848 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30718c00 2014-06-09 16:22:11.594845 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.594852 7f868a757700 2 waiting 51 > 50 ops || 90986 > 104857600 2014-06-09 16:22:11.594852 7f86a460e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.594854 7f869af78700 5 -- op tracker -- , seq: 2055, time: 2014-06-09 16:22:11.594851, event: started, op: pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.594850 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594854 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.594859 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594862 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.594867 7f869af78700 5 -- op tracker -- , seq: 2055, time: 2014-06-09 16:22:11.594864, event: done, op: pg_log(1.4bb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.594874 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594875 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594869 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.594888 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594889 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594884 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.594890 7f869af78700 20 osd.26 243 _dispatch 0x22ac1880 pg_notify(3.78a(12),2.1fa(21),1.4c2(21) epoch 243) v5 2014-06-09 16:22:11.594897 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.594897 7f869af78700 5 -- op tracker -- , seq: 2056, time: 2014-06-09 16:22:11.232760, event: header_read, op: pg_notify(3.78a(12),2.1fa(21),1.4c2(21) epoch 243) 2014-06-09 16:22:11.594904 7f869af78700 5 -- op tracker -- , seq: 2056, time: 2014-06-09 16:22:11.232761, event: throttled, op: pg_notify(3.78a(12),2.1fa(21),1.4c2(21) epoch 243) 2014-06-09 16:22:11.594909 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.594911 7f869af78700 5 -- op tracker -- , seq: 2056, time: 2014-06-09 16:22:11.232833, event: all_read, op: pg_notify(3.78a(12),2.1fa(21),1.4c2(21) epoch 243) 2014-06-09 16:22:11.594914 7f8690f64700 5 queue_transactions existing osr(1.206 0x384cd10)/0x384cd10 2014-06-09 16:22:11.594916 7f8690f64700 2 waiting 51 > 50 ops || 91033 > 104857600 2014-06-09 16:22:11.594911 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] flushed 2014-06-09 16:22:11.594919 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.594918 7f869af78700 5 -- op tracker -- , seq: 2056, time: 2014-06-09 16:22:11.594888, event: dispatched, op: pg_notify(3.78a(12),2.1fa(21),1.4c2(21) epoch 243) 2014-06-09 16:22:11.594923 7f868ef60700 5 queue_transactions existing osr(2.67c 0x2ed1f90)/0x2ed1f90 2014-06-09 16:22:11.594925 7f868ef60700 2 waiting 51 > 50 ops || 91080 > 104857600 2014-06-09 16:22:11.594925 7f869af78700 5 -- op tracker -- , seq: 2056, time: 2014-06-09 16:22:11.594923, event: waiting_for_osdmap, op: pg_notify(3.78a(12),2.1fa(21),1.4c2(21) epoch 243) 2014-06-09 16:22:11.590843 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.594929 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.594931 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22ac1880 2014-06-09 16:22:11.594924 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.594933 7f868b759700 5 queue_transactions existing osr(1.302 0x384d010)/0x384d010 2014-06-09 16:22:11.594935 7f868b759700 2 waiting 51 > 50 ops || 90935 > 104857600 2014-06-09 16:22:11.594938 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.594936 7f869af78700 5 -- op tracker -- , seq: 2056, time: 2014-06-09 16:22:11.594934, event: started, op: pg_notify(3.78a(12),2.1fa(21),1.4c2(21) epoch 243) 2014-06-09 16:22:11.594945 7f8689f56700 10 op_submit_start 3272 2014-06-09 16:22:11.594950 7f8689f56700 5 queue_transactions (trailing journal) 3272 2014-06-09 16:22:11.594952 7f8689f56700 5 queue_op 0x306067e0 seq 3272 osr(1.b5 0x384c690) 1880 bytes (queue has 50 ops and 89032 bytes) 2014-06-09 16:22:11.594955 7f8689f56700 10 op_submit_finish 3272 2014-06-09 16:22:11.594953 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.594970 7f869af78700 5 -- op tracker -- , seq: 2056, time: 2014-06-09 16:22:11.594967, event: done, op: pg_notify(3.78a(12),2.1fa(21),1.4c2(21) epoch 243) 2014-06-09 16:22:11.594966 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.594962 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.327( v 214'680 (0'0,214'680] local-les=203 n=132 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.594978 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.594979 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.594981 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.594978 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.594991 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.594993 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.594996 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.595005 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.595015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.595008 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.595016 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.595018 7f869af78700 20 osd.26 243 _dispatch 0x3088c380 pg_notify(1.37e(23),0.b7(20) epoch 243) v5 2014-06-09 16:22:11.595020 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.595018 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.595025 7f869af78700 5 -- op tracker -- , seq: 2057, time: 2014-06-09 16:22:11.314301, event: header_read, op: pg_notify(1.37e(23),0.b7(20) epoch 243) 2014-06-09 16:22:11.595026 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.595033 7f869af78700 5 -- op tracker -- , seq: 2057, time: 2014-06-09 16:22:11.314301, event: throttled, op: pg_notify(1.37e(23),0.b7(20) epoch 243) 2014-06-09 16:22:11.595039 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.595039 7f869af78700 5 -- op tracker -- , seq: 2057, time: 2014-06-09 16:22:11.314339, event: all_read, op: pg_notify(1.37e(23),0.b7(20) epoch 243) 2014-06-09 16:22:11.595034 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.595046 7f869af78700 5 -- op tracker -- , seq: 2057, time: 2014-06-09 16:22:11.595014, event: dispatched, op: pg_notify(1.37e(23),0.b7(20) epoch 243) 2014-06-09 16:22:11.595049 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.595052 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.595057 7f869af78700 5 -- op tracker -- , seq: 2057, time: 2014-06-09 16:22:11.595054, event: waiting_for_osdmap, op: pg_notify(1.37e(23),0.b7(20) epoch 243) 2014-06-09 16:22:11.595062 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.595062 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.595065 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.595064 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088c380 2014-06-09 16:22:11.595068 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.282( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.591891 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595076 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.595076 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.595085 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.595089 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.595090 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.28a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.595097 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3236.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.595102 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.595103 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.595102 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.595110 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.595117 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.595121 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:2.292( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.595132 7f868df5e700 10 op_submit_start 3273 2014-06-09 16:22:11.595134 7f868df5e700 5 queue_transactions (trailing journal) 3273 2014-06-09 16:22:11.595131 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.595138 7f868df5e700 5 queue_op 0x2fcae600 seq 3273 osr(2.699 0x2ed1f10) 1746 bytes (queue has 50 ops and 89032 bytes) 2014-06-09 16:22:11.595141 7f868df5e700 10 op_submit_finish 3273 2014-06-09 16:22:11.595135 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.595145 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.595144 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.595146 7f86a5e11700 10 _do_op 0x30d97080 seq 3236 r = 0, finisher 0x3050a1c0 0 2014-06-09 16:22:11.595151 7f86a5e11700 10 _finish_op 0x30d97080 seq 3236 osr(1.5c1 0x384dd90)/0x384dd90 2014-06-09 16:22:11.595148 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.595147 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.595159 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652d80 (1 active) 2014-06-09 16:22:11.595162 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c1e60 (2 active) 2014-06-09 16:22:11.595165 7f86a5e11700 5 _do_op 0x30d96fc0 seq 3238 osr(0.458 0x2bd7510)/0x2bd7510 start 2014-06-09 16:22:11.595159 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.595158 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.33a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595158 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.26a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595169 7f86a5e11700 10 _do_transaction on 0x97bbd40 2014-06-09 16:22:11.595169 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595169 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595173 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.595173 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.595176 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.595172 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.595176 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.298008 7 0.000175 2014-06-09 16:22:11.595176 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.364315 7 0.000192 2014-06-09 16:22:11.595185 7f86a460e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.595187 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.595187 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.595197 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.595197 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.595198 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.595207 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.595207 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.595215 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595211 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.595217 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.595216 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.595224 7f86a460e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.595224 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.595224 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.595233 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3238.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.595238 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.595237 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.595250 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.595256 7f86a5e11700 10 _do_op 0x30d96fc0 seq 3238 r = 0, finisher 0xbdbb320 0 2014-06-09 16:22:11.595260 7f86a5e11700 10 _finish_op 0x30d96fc0 seq 3238 osr(0.458 0x2bd7510)/0x2bd7510 2014-06-09 16:22:11.595261 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.595263 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c1e60 (1 active) 2014-06-09 16:22:11.595266 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c6c0 (2 active) 2014-06-09 16:22:11.595266 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595267 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.595268 7f86a5e11700 5 _do_op 0x30d96f60 seq 3239 osr(0.6e7 0x384c1b0)/0x384c1b0 start 2014-06-09 16:22:11.595270 7f86a5e11700 10 _do_transaction on 0xc327800 2014-06-09 16:22:11.595274 7f8689f56700 5 queue_transactions existing osr(2.26a 0x2ed0dd0)/0x2ed0dd0 2014-06-09 16:22:11.595276 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.595279 7f8689f56700 10 op_submit_start 3274 2014-06-09 16:22:11.595280 7f8689f56700 5 queue_transactions (trailing journal) 3274 2014-06-09 16:22:11.595281 7f8689f56700 5 queue_op 0x30d975c0 seq 3274 osr(2.26a 0x2ed0dd0) 1883 bytes (queue has 49 ops and 86866 bytes) 2014-06-09 16:22:11.595276 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.595284 7f8689f56700 10 op_submit_finish 3274 2014-06-09 16:22:11.595287 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a6020,0x2d739820,0xbd75400,0xbd7b720,0xbef10e0,0xbd70140,0xbd7c5a0,0xcd9ea80,0xbd631a0,0xbd6be00,0xbd63460] 2014-06-09 16:22:11.595291 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x19941fe0,0xc5a7b40] 2014-06-09 16:22:11.595294 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x19941fe0,0xc5a7b40] 2014-06-09 16:22:11.595295 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.595296 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.595301 7f868cf5c700 10 op_submit_start 3275 2014-06-09 16:22:11.595302 7f868cf5c700 5 queue_transactions (trailing journal) 3275 2014-06-09 16:22:11.595302 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (19 active) 2014-06-09 16:22:11.595304 7f868cf5c700 5 queue_op 0x30d969c0 seq 3275 osr(2.216 0x2ed0b50) 2001 bytes (queue has 49 ops and 86866 bytes) 2014-06-09 16:22:11.595307 7f868cf5c700 10 op_submit_finish 3275 2014-06-09 16:22:11.595310 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595323 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (20 active) 2014-06-09 16:22:11.595327 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3239.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.595331 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.595329 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.3a3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.595342 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.595353 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.595356 7f86a5e11700 10 _do_op 0x30d96f60 seq 3239 r = 0, finisher 0xbcc9160 0 2014-06-09 16:22:11.595364 7f86a5e11700 10 _finish_op 0x30d96f60 seq 3239 osr(0.6e7 0x384c1b0)/0x384c1b0 2014-06-09 16:22:11.595364 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (19 active) 2014-06-09 16:22:11.595365 7f868df5e700 5 queue_transactions existing osr(2.33a 0x2ed0ed0)/0x2ed0ed0 2014-06-09 16:22:11.595357 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=210 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.595369 7f868df5e700 10 op_submit_start 3276 2014-06-09 16:22:11.595368 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c6c0 (1 active) 2014-06-09 16:22:11.595369 7f868df5e700 5 queue_transactions (trailing journal) 3276 2014-06-09 16:22:11.595370 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7806c0 (2 active) 2014-06-09 16:22:11.595371 7f868df5e700 5 queue_op 0x2fcae5a0 seq 3276 osr(2.33a 0x2ed0ed0) 1883 bytes (queue has 49 ops and 86873 bytes) 2014-06-09 16:22:11.595373 7f868df5e700 10 op_submit_finish 3276 2014-06-09 16:22:11.595373 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.595373 7f86a5e11700 5 _do_op 0x30d96f00 seq 3240 osr(1.122 0x384c890)/0x384c890 start 2014-06-09 16:22:11.595376 7f86a5e11700 10 _do_transaction on 0xc3fc180 2014-06-09 16:22:11.595379 7f8692767700 10 op_submit_start 3277 2014-06-09 16:22:11.595369 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=210 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.595382 7f8692767700 5 queue_transactions (trailing journal) 3277 2014-06-09 16:22:11.595383 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.595383 7f8692767700 5 queue_op 0x309a21e0 seq 3277 osr(1.c9 0x384c9f0) 1998 bytes (queue has 49 ops and 86873 bytes) 2014-06-09 16:22:11.595387 7f8692767700 10 op_submit_finish 3277 2014-06-09 16:22:11.595383 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=210 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.889398 7 0.005410 2014-06-09 16:22:11.595376 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.301( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.595394 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=210 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.595395 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.595404 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595393 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.60e( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.595404 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=210 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.595408 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.349( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.595414 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=210 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.595414 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.595419 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.595426 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.595428 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595427 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595423 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'587 -> 213'587 2014-06-09 16:22:11.595431 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.595431 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.595444 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3240.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.595449 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.595441 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 0.631( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595442 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.5b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595452 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595453 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595456 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.595457 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.595459 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.631( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.352070 7 0.000168 2014-06-09 16:22:11.595466 7f86a5e11700 10 _do_op 0x30d96f00 seq 3240 r = 0, finisher 0xbfcecc0 0 2014-06-09 16:22:11.595460 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.895082 7 0.000155 2014-06-09 16:22:11.595464 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.595471 7f86a5e11700 10 _finish_op 0x30d96f00 seq 3240 osr(1.122 0x384c890)/0x384c890 2014-06-09 16:22:11.595469 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.631( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.595476 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7806c0 (1 active) 2014-06-09 16:22:11.595471 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.595479 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300900 (2 active) 2014-06-09 16:22:11.595481 7f86a5e11700 5 _do_op 0x300619e0 seq 3241 osr(1.d1 0x384c9b0)/0x384c9b0 start 2014-06-09 16:22:11.595484 7f86a5e11700 10 _do_transaction on 0xc130300 2014-06-09 16:22:11.595478 7f868df5e700 5 osd.26 pg_epoch: 243 pg[0.631( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.595482 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.595490 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.595487 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.595492 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.595495 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.595502 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.595506 7f8689f56700 5 queue_transactions existing osr(3.461 0x3ed1120)/0x3ed1120 2014-06-09 16:22:11.595501 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.595509 7f8689f56700 10 op_submit_start 3278 2014-06-09 16:22:11.595503 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.595510 7f8689f56700 5 queue_transactions (trailing journal) 3278 2014-06-09 16:22:11.595511 7f8689f56700 5 queue_op 0x30d97560 seq 3278 osr(3.461 0x3ed1120) 1335 bytes (queue has 49 ops and 86207 bytes) 2014-06-09 16:22:11.595513 7f8689f56700 10 op_submit_finish 3278 2014-06-09 16:22:11.595510 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.595516 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=210 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.595524 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595527 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=210 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.595543 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3241.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.595539 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=210 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.414506 7 0.009311 2014-06-09 16:22:11.595547 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595547 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.595550 7f868df5e700 5 queue_transactions existing osr(0.631 0x2bd7c70)/0x2bd7c70 2014-06-09 16:22:11.595553 7f868df5e700 10 op_submit_start 3279 2014-06-09 16:22:11.595554 7f868df5e700 5 queue_transactions (trailing journal) 3279 2014-06-09 16:22:11.595549 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=210 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.595555 7f868df5e700 5 queue_op 0x2fcae4e0 seq 3279 osr(0.631 0x2bd7c70) 2009 bytes (queue has 50 ops and 88216 bytes) 2014-06-09 16:22:11.595557 7f868df5e700 10 op_submit_finish 3279 2014-06-09 16:22:11.595559 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.595559 7f868df5e700 20 osd.26 243 Sending info 3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) to shard 4 2014-06-09 16:22:11.595564 7f8692767700 5 queue_transactions existing osr(2.5b6 0x2ed1b90)/0x2ed1b90 2014-06-09 16:22:11.595566 7f868df5e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.595560 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=210 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.595567 7f8692767700 2 waiting 51 > 50 ops || 90213 > 104857600 2014-06-09 16:22:11.595572 7f868af58700 10 op_submit_start 3280 2014-06-09 16:22:11.595574 7f868af58700 5 queue_transactions (trailing journal) 3280 2014-06-09 16:22:11.595575 7f86a5e11700 10 _do_op 0x300619e0 seq 3241 r = 0, finisher 0x189ab4c0 0 2014-06-09 16:22:11.595570 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=210 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.595575 7f868af58700 5 queue_op 0x2fc8c0c0 seq 3280 osr(3.218 0x3ed0cd0) 1335 bytes (queue has 50 ops and 88216 bytes) 2014-06-09 16:22:11.595579 7f86a5e11700 10 _finish_op 0x300619e0 seq 3241 osr(1.d1 0x384c9b0)/0x384c9b0 2014-06-09 16:22:11.595581 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc4108e0,0xbd6d040,0xc5a7d80] 2014-06-09 16:22:11.595585 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300900 (1 active) 2014-06-09 16:22:11.595586 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc4108e0,0xbd6d040,0xc5a7d80] 2014-06-09 16:22:11.595579 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'658 -> 213'658 2014-06-09 16:22:11.595587 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.595587 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd320 (2 active) 2014-06-09 16:22:11.595589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.595591 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02a0 (19 active) 2014-06-09 16:22:11.595592 7f86a5e11700 5 _do_op 0x30d96ea0 seq 3242 osr(2.5be 0x2ed1b30)/0x2ed1b30 start 2014-06-09 16:22:11.595595 7f86a5e11700 10 _do_transaction on 0xc015980 2014-06-09 16:22:11.595596 7f868af58700 10 op_submit_finish 3280 2014-06-09 16:22:11.595601 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.595603 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02a0 (20 active) 2014-06-09 16:22:11.595602 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.420( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.595606 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.595631 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.595631 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.595640 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595635 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.595643 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.595640 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.595647 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.595648 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.595658 7f868df5e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.595656 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3242.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.595661 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.595658 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.113( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595670 7f868af58700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595672 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.595674 7f868af58700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.595676 7f8689f56700 5 queue_transactions existing osr(3.468 0x3ed1590)/0x3ed1590 2014-06-09 16:22:11.595679 7f8689f56700 2 waiting 51 > 50 ops || 89668 > 104857600 2014-06-09 16:22:11.595672 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.595677 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.113( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.122000 7 0.005067 2014-06-09 16:22:11.595684 7f86a5e11700 10 _do_op 0x30d96ea0 seq 3242 r = 0, finisher 0xbd7c4e0 0 2014-06-09 16:22:11.595686 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595691 7f86a5e11700 10 _finish_op 0x30d96ea0 seq 3242 osr(2.5be 0x2ed1b30)/0x2ed1b30 2014-06-09 16:22:11.595696 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd320 (1 active) 2014-06-09 16:22:11.595690 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.113( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.595698 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f440 (2 active) 2014-06-09 16:22:11.595690 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 7 notify: (query_epoch:242, epoch_sent:243, info:1.70c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.595700 7f86a5e11700 5 _do_op 0x30d96e40 seq 3243 osr(3.139 0x3ed0780)/0x3ed0780 start 2014-06-09 16:22:11.595703 7f86a5e11700 10 _do_transaction on 0xc33af00 2014-06-09 16:22:11.595701 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.113( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.595708 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.595703 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.595710 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.595723 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595719 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.595726 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.595728 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 7 notify: (query_epoch:242, epoch_sent:243, info:2.17c( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.595741 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595739 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.595749 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595757 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3243.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.595761 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.595752 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.562( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.595774 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.595778 7f868af58700 5 queue_transactions existing osr(1.113 0x384c910)/0x384c910 2014-06-09 16:22:11.595781 7f868af58700 10 op_submit_start 3281 2014-06-09 16:22:11.595782 7f868af58700 5 queue_transactions (trailing journal) 3281 2014-06-09 16:22:11.595773 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.595784 7f868af58700 5 queue_op 0x2fc8c060 seq 3281 osr(1.113 0x384c910) 1883 bytes (queue has 50 ops and 88333 bytes) 2014-06-09 16:22:11.595785 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595786 7f868af58700 10 op_submit_finish 3281 2014-06-09 16:22:11.595789 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.595789 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.595802 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x1b0b6700,0x1b0b6600,0x1b0b6500,0x1b0b6400,0x1b0b6300,0x1b0b6200 2014-06-09 16:22:11.595809 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.2d9( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595820 7f868af58700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.595824 7f868af58700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.595834 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595827 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.829815 7 0.000250 2014-06-09 16:22:11.595840 7f86a5e11700 10 _do_op 0x30d96e40 seq 3243 r = 0, finisher 0xbdf17a0 0 2014-06-09 16:22:11.595845 7f86a5e11700 10 _finish_op 0x30d96e40 seq 3243 osr(3.139 0x3ed0780)/0x3ed0780 2014-06-09 16:22:11.595848 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f440 (1 active) 2014-06-09 16:22:11.595844 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.595850 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780120 (2 active) 2014-06-09 16:22:11.595838 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.65d( v 214'536 (0'0,214'536] local-les=203 n=106 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.595852 7f86a5e11700 5 _do_op 0x30061980 seq 3244 osr(2.5f2 0x2ed1df0)/0x2ed1df0 start 2014-06-09 16:22:11.595855 7f86a5e11700 10 _do_transaction on 0xc0ec900 2014-06-09 16:22:11.595855 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.595861 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.595856 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.595870 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595864 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.595872 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.595874 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.595881 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.595889 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595894 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595893 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.595910 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3244.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.595915 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.595907 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x270be800,0x270be700,0x270be600,0x270be500,0x270be400 2014-06-09 16:22:11.595926 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.595931 7f868af58700 5 queue_transactions existing osr(0.2d9 0x2bd6e90)/0x2bd6e90 2014-06-09 16:22:11.595932 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595934 7f868af58700 10 op_submit_start 3282 2014-06-09 16:22:11.595935 7f868af58700 5 queue_transactions (trailing journal) 3282 2014-06-09 16:22:11.595933 7f86a5e11700 10 _do_op 0x30061980 seq 3244 r = 0, finisher 0xbd5ade0 0 2014-06-09 16:22:11.595936 7f868af58700 5 queue_op 0x2fc8c000 seq 3282 osr(0.2d9 0x2bd6e90) 1974 bytes (queue has 50 ops and 88972 bytes) 2014-06-09 16:22:11.595937 7f86a5e11700 10 _finish_op 0x30061980 seq 3244 osr(2.5f2 0x2ed1df0)/0x2ed1df0 2014-06-09 16:22:11.595939 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780120 (1 active) 2014-06-09 16:22:11.595942 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3017a0 (2 active) 2014-06-09 16:22:11.595936 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.595946 7f86a5e11700 5 _do_op 0x30d96de0 seq 3245 osr(0.654 0x2bd7b50)/0x2bd7b50 start 2014-06-09 16:22:11.595949 7f86a5e11700 10 _do_transaction on 0xc2f4c00 2014-06-09 16:22:11.595955 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.595949 7f868df5e700 15 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.595959 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595963 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.595974 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.697( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.595983 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.595987 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.595987 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.595998 7f868af58700 10 op_submit_finish 3282 2014-06-09 16:22:11.596002 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3245.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596005 7f8689f56700 10 op_submit_start 3283 2014-06-09 16:22:11.596007 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596007 7f8689f56700 5 queue_transactions (trailing journal) 3283 2014-06-09 16:22:11.596000 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.596009 7f8689f56700 5 queue_op 0x30d97500 seq 3283 osr(3.468 0x3ed1590) 1335 bytes (queue has 50 ops and 88298 bytes) 2014-06-09 16:22:11.596011 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596012 7f8689f56700 10 op_submit_finish 3283 2014-06-09 16:22:11.596003 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.596014 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596014 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596025 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596025 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.596031 7f86a5e11700 10 _do_op 0x30d96de0 seq 3245 r = 0, finisher 0xbd70a40 0 2014-06-09 16:22:11.596023 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596035 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596034 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596034 7f86a5e11700 10 _finish_op 0x30d96de0 seq 3245 osr(0.654 0x2bd7b50)/0x2bd7b50 2014-06-09 16:22:11.596036 7f868af58700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596038 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3017a0 (1 active) 2014-06-09 16:22:11.596040 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e7e0 (2 active) 2014-06-09 16:22:11.596040 7f868af58700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596042 7f86a5e11700 5 _do_op 0x30032060 seq 3246 osr(2.463 0x2ed1510)/0x2ed1510 start 2014-06-09 16:22:11.596045 7f86a5e11700 10 _do_transaction on 0xbfa4b40 2014-06-09 16:22:11.596039 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596039 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596050 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596044 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.323466 7 0.000206 2014-06-09 16:22:11.596050 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596050 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596059 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596059 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596055 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596063 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596062 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596065 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596072 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.681( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596072 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596080 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596080 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596074 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596082 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596084 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.596087 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596092 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596098 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3246.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596103 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596099 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.113( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596109 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596113 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596112 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596124 7f86a5e11700 10 _do_op 0x30032060 seq 3246 r = 0, finisher 0xbd74f00 0 2014-06-09 16:22:11.596128 7f86a5e11700 10 _finish_op 0x30032060 seq 3246 osr(2.463 0x2ed1510)/0x2ed1510 2014-06-09 16:22:11.596123 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596124 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.596132 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596131 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e7e0 (1 active) 2014-06-09 16:22:11.596133 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596133 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd440 (2 active) 2014-06-09 16:22:11.596135 7f86a5e11700 5 _do_op 0x30061920 seq 3247 osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 start 2014-06-09 16:22:11.596138 7f86a5e11700 10 _do_transaction on 0x331ec00 2014-06-09 16:22:11.596140 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596135 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596136 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596144 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596145 7f868af58700 5 queue_transactions existing osr(2.c6 0x2ed0490)/0x2ed0490 2014-06-09 16:22:11.596149 7f868af58700 10 op_submit_start 3284 2014-06-09 16:22:11.596150 7f868af58700 5 queue_transactions (trailing journal) 3284 2014-06-09 16:22:11.596145 7f868df5e700 15 osd.26 pg_epoch: 243 pg[0.380( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596145 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.589( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596151 7f868af58700 5 queue_op 0x30d97e60 seq 3284 osr(2.c6 0x2ed0490) 1880 bytes (queue has 49 ops and 86298 bytes) 2014-06-09 16:22:11.596154 7f868af58700 10 op_submit_finish 3284 2014-06-09 16:22:11.596154 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596154 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596158 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596157 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=203 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.596158 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.648( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.596169 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596178 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596176 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596169 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=203 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.596175 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.596185 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596182 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596182 7f868af58700 5 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=203 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.425342 7 0.000187 2014-06-09 16:22:11.596193 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3247.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596188 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596192 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596197 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596200 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596193 7f868af58700 5 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=203 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596198 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596207 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596204 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596205 7f868af58700 5 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=203 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596210 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=191 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) 2014-06-09 16:22:11.596218 7f86a5e11700 10 _do_op 0x30061920 seq 3247 r = 0, finisher 0xc410920 0 2014-06-09 16:22:11.596214 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.501( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596222 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596221 7f86a5e11700 10 _finish_op 0x30061920 seq 3247 osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 2014-06-09 16:22:11.596215 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=203 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596224 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd440 (1 active) 2014-06-09 16:22:11.596227 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0360 (2 active) 2014-06-09 16:22:11.596229 7f86a5e11700 5 _do_op 0x30d96d80 seq 3248 osr(1.6f 0x384c570)/0x384c570 start 2014-06-09 16:22:11.596231 7f86a5e11700 10 _do_transaction on 0xc258cc0 2014-06-09 16:22:11.596222 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=191 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.0 3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) 2014-06-09 16:22:11.596226 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596226 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'605 -> 213'605 2014-06-09 16:22:11.596237 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596234 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=191 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.794129 7 0.000278 2014-06-09 16:22:11.596237 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596245 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596244 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=191 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596248 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.596254 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=191 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596258 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 0.456( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596268 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596263 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=191 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596269 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596272 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596265 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596273 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'697 -> 213'697 2014-06-09 16:22:11.596275 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.456( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.698972 7 0.000215 2014-06-09 16:22:11.596286 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3248.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596284 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.456( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596290 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596293 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.456( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596302 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596309 7f86a5e11700 10 _do_op 0x30d96d80 seq 3248 r = 0, finisher 0x139645c0 0 2014-06-09 16:22:11.596311 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596312 7f86a5e11700 10 _finish_op 0x30d96d80 seq 3248 osr(1.6f 0x384c570)/0x384c570 2014-06-09 16:22:11.596309 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.596315 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0360 (1 active) 2014-06-09 16:22:11.596316 7f868af58700 5 queue_transactions existing osr(3.4fe 0x3ed1730)/0x3ed1730 2014-06-09 16:22:11.596317 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524b40 (2 active) 2014-06-09 16:22:11.596318 7f868af58700 10 op_submit_start 3285 2014-06-09 16:22:11.596319 7f868af58700 5 queue_transactions (trailing journal) 3285 2014-06-09 16:22:11.596320 7f86a5e11700 5 _do_op 0x300618c0 seq 3249 osr(1.2c0 0x384d170)/0x384d170 start 2014-06-09 16:22:11.596321 7f868af58700 5 queue_op 0x30d97e00 seq 3285 osr(3.4fe 0x3ed1730) 1335 bytes (queue has 48 ops and 83626 bytes) 2014-06-09 16:22:11.596322 7f86a5e11700 10 _do_transaction on 0xc326780 2014-06-09 16:22:11.596317 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596323 7f868af58700 10 op_submit_finish 3285 2014-06-09 16:22:11.596319 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596328 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596326 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.596354 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596345 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.4a7( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596357 7f868af58700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596358 7f868df5e700 5 queue_transactions existing osr(3.778 0x3ed1df0)/0x3ed1df0 2014-06-09 16:22:11.596360 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.596361 7f868df5e700 10 op_submit_start 3286 2014-06-09 16:22:11.596360 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596362 7f868df5e700 5 queue_transactions (trailing journal) 3286 2014-06-09 16:22:11.596361 7f868af58700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596364 7f8689f56700 5 queue_transactions existing osr(0.456 0x2bd7530)/0x2bd7530 2014-06-09 16:22:11.596363 7f868df5e700 5 queue_op 0x2fcae420 seq 3286 osr(3.778 0x3ed1df0) 1429 bytes (queue has 49 ops and 85055 bytes) 2014-06-09 16:22:11.596366 7f868df5e700 10 op_submit_finish 3286 2014-06-09 16:22:11.596369 7f8690f64700 2 waiting 51 > 50 ops || 89057 > 104857600 2014-06-09 16:22:11.596364 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.704919 7 0.000158 2014-06-09 16:22:11.596376 7f868ef60700 2 waiting 51 > 50 ops || 89104 > 104857600 2014-06-09 16:22:11.596375 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3249.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596380 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596371 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=191 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) 2014-06-09 16:22:11.596383 7f868b759700 2 waiting 51 > 50 ops || 88959 > 104857600 2014-06-09 16:22:11.596374 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596384 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=191 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.0 3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) 2014-06-09 16:22:11.596390 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596404 7f86a5e11700 10 _do_op 0x300618c0 seq 3249 r = 0, finisher 0xbd72720 0 2014-06-09 16:22:11.596400 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596398 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=191 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.751787 7 0.000177 2014-06-09 16:22:11.596408 7f86a5e11700 10 _finish_op 0x300618c0 seq 3249 osr(1.2c0 0x384d170)/0x384d170 2014-06-09 16:22:11.596414 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524b40 (1 active) 2014-06-09 16:22:11.596410 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.596417 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fde60 (2 active) 2014-06-09 16:22:11.596419 7f86a5e11700 5 _do_op 0x30d96d20 seq 3250 osr(1.27c 0x384cf30)/0x384cf30 start 2014-06-09 16:22:11.596422 7f86a5e11700 10 _do_transaction on 0xc20e540 2014-06-09 16:22:11.596413 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=191 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596419 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596427 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596425 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=191 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596436 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=191 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596446 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'579 -> 213'579 2014-06-09 16:22:11.596459 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596473 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596475 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3250.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596479 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596483 7f868af58700 5 queue_transactions existing osr(0.4a7 0x2bd7370)/0x2bd7370 2014-06-09 16:22:11.596486 7f868af58700 10 op_submit_start 3287 2014-06-09 16:22:11.596487 7f868af58700 5 queue_transactions (trailing journal) 3287 2014-06-09 16:22:11.596489 7f868af58700 5 queue_op 0x30d97da0 seq 3287 osr(0.4a7 0x2bd7370) 2009 bytes (queue has 50 ops and 87182 bytes) 2014-06-09 16:22:11.596491 7f868af58700 10 op_submit_finish 3287 2014-06-09 16:22:11.596485 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596495 7f86a5e11700 10 _do_op 0x30d96d20 seq 3250 r = 0, finisher 0xc6e84c0 0 2014-06-09 16:22:11.596499 7f86a5e11700 10 _finish_op 0x30d96d20 seq 3250 osr(1.27c 0x384cf30)/0x384cf30 2014-06-09 16:22:11.596501 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fde60 (1 active) 2014-06-09 16:22:11.596494 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.596504 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6527e0 (2 active) 2014-06-09 16:22:11.596506 7f86a5e11700 5 _do_op 0x30061860 seq 3251 osr(3.5c6 0x3ed17f0)/0x3ed17f0 start 2014-06-09 16:22:11.596508 7f86a5e11700 10 _do_transaction on 0xc0d9980 2014-06-09 16:22:11.596506 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.13c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596517 7f868af58700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596521 7f868af58700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596522 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596524 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 1.107307 7 0.000201 2014-06-09 16:22:11.596536 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.596534 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596541 7f868df5e700 5 queue_transactions existing osr(3.1fc 0x3ed0930)/0x3ed0930 2014-06-09 16:22:11.596544 7f868df5e700 10 op_submit_start 3288 2014-06-09 16:22:11.596545 7f868df5e700 5 queue_transactions (trailing journal) 3288 2014-06-09 16:22:11.596547 7f868df5e700 5 queue_op 0x2fcae3c0 seq 3288 osr(3.1fc 0x3ed0930) 1429 bytes (queue has 50 ops and 86728 bytes) 2014-06-09 16:22:11.596549 7f868df5e700 10 op_submit_finish 3288 2014-06-09 16:22:11.596544 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596553 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596559 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02a0 (19 active) 2014-06-09 16:22:11.596555 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596565 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.596562 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.596571 7f868c75b700 10 op_submit_start 3289 2014-06-09 16:22:11.596573 7f868c75b700 5 queue_transactions (trailing journal) 3289 2014-06-09 16:22:11.596570 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596574 7f868c75b700 5 queue_op 0x30032e40 seq 3289 osr(1.198 0x384cb30) 1883 bytes (queue has 50 ops and 86728 bytes) 2014-06-09 16:22:11.596578 7f868c75b700 10 op_submit_finish 3289 2014-06-09 16:22:11.596578 7f8690f64700 2 waiting 51 > 50 ops || 88729 > 104857600 2014-06-09 16:22:11.596578 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3251.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596583 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596585 7f868ef60700 2 waiting 51 > 50 ops || 88776 > 104857600 2014-06-09 16:22:11.596591 7f868b759700 2 waiting 51 > 50 ops || 88631 > 104857600 2014-06-09 16:22:11.596583 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.596596 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.473( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596607 7f86a5e11700 10 _do_op 0x30061860 seq 3251 r = 0, finisher 0xbd6c780 0 2014-06-09 16:22:11.596608 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596610 7f86a5e11700 10 _finish_op 0x30061860 seq 3251 osr(3.5c6 0x3ed17f0)/0x3ed17f0 2014-06-09 16:22:11.596612 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596616 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596616 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6527e0 (1 active) 2014-06-09 16:22:11.596619 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc7e0 (2 active) 2014-06-09 16:22:11.596621 7f868af58700 5 queue_transactions existing osr(2.13c 0x2ed0610)/0x2ed0610 2014-06-09 16:22:11.596621 7f86a5e11700 5 _do_op 0x30032000 seq 3252 osr(3.2fd 0x3ed0de0)/0x3ed0de0 start 2014-06-09 16:22:11.596623 7f868af58700 10 op_submit_start 3290 2014-06-09 16:22:11.596624 7f86a5e11700 10 _do_transaction on 0x35629c0 2014-06-09 16:22:11.596615 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.473( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.809604 7 0.000202 2014-06-09 16:22:11.596624 7f868af58700 5 queue_transactions (trailing journal) 3290 2014-06-09 16:22:11.596627 7f868af58700 5 queue_op 0x30d97d40 seq 3290 osr(2.13c 0x2ed0610) 2099 bytes (queue has 50 ops and 87492 bytes) 2014-06-09 16:22:11.596630 7f868af58700 10 op_submit_finish 3290 2014-06-09 16:22:11.596630 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596628 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.473( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596633 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:2.28a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.596639 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.473( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596646 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.596654 7f8690f64700 2 waiting 51 > 50 ops || 89493 > 104857600 2014-06-09 16:22:11.596649 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596656 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596660 7f868ef60700 2 waiting 51 > 50 ops || 89540 > 104857600 2014-06-09 16:22:11.596662 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596658 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.596667 7f868b759700 2 waiting 51 > 50 ops || 89395 > 104857600 2014-06-09 16:22:11.596666 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596660 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.55d( v 213'582 (0'0,213'582] local-les=191 n=112 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.596677 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3252.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596682 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596674 7f868af58700 7 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.1 2014-06-09 16:22:11.596698 7f86a5e11700 10 _do_op 0x30032000 seq 3252 r = 0, finisher 0xbd7b400 0 2014-06-09 16:22:11.596702 7f86a5e11700 10 _finish_op 0x30032000 seq 3252 osr(3.2fd 0x3ed0de0)/0x3ed0de0 2014-06-09 16:22:11.596684 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.1 info 3.55d( v 213'582 (0'0,213'582] local-les=191 n=112 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.596704 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.596705 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596707 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc7e0 (1 active) 2014-06-09 16:22:11.596709 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e6c0 (2 active) 2014-06-09 16:22:11.596711 7f8690f64700 10 op_submit_start 3291 2014-06-09 16:22:11.596712 7f868ef60700 2 waiting 51 > 50 ops || 90206 > 104857600 2014-06-09 16:22:11.596712 7f86a5e11700 5 _do_op 0x14dd7140 seq 3253 osr(2.182 0x2ed0990)/0x2ed0990 start 2014-06-09 16:22:11.596713 7f8690f64700 5 queue_transactions (trailing journal) 3291 2014-06-09 16:22:11.596715 7f86a5e11700 10 _do_transaction on 0x34f0240 2014-06-09 16:22:11.596709 7f868c75b700 5 queue_transactions existing osr(0.473 0x2bd74d0)/0x2bd74d0 2014-06-09 16:22:11.596714 7f8690f64700 5 queue_op 0x2fcaf980 seq 3291 osr(1.206 0x384cd10) 2001 bytes (queue has 50 ops and 88158 bytes) 2014-06-09 16:22:11.596717 7f8690f64700 10 op_submit_finish 3291 2014-06-09 16:22:11.596717 7f868b759700 2 waiting 51 > 50 ops || 90061 > 104857600 2014-06-09 16:22:11.596710 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:0.732( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.596720 7f8692f68700 10 op_submit_start 3292 2014-06-09 16:22:11.596721 7f8692f68700 5 queue_transactions (trailing journal) 3292 2014-06-09 16:22:11.596720 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596722 7f868c75b700 2 waiting 51 > 50 ops || 90112 > 104857600 2014-06-09 16:22:11.596722 7f8692f68700 5 queue_op 0x309a35c0 seq 3292 osr(1.48e 0x384d9b0) 2009 bytes (queue has 50 ops and 88158 bytes) 2014-06-09 16:22:11.596725 7f8692f68700 10 op_submit_finish 3292 2014-06-09 16:22:11.596721 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.596721 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.596731 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596733 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (19 active) 2014-06-09 16:22:11.596733 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.65e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596735 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.46e( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.596744 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596747 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596751 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (20 active) 2014-06-09 16:22:11.596747 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.596753 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596756 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596750 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.347392 7 0.000236 2014-06-09 16:22:11.596754 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596760 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596766 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596770 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3253.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.596774 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.596775 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.596771 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596760 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.647( v 214'591 (0'0,214'591] local-les=212 n=121 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.596779 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596779 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.730( v 214'648 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.596782 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.596787 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.596794 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596796 7f86a5e11700 10 _do_op 0x14dd7140 seq 3253 r = 0, finisher 0xbd6a060 0 2014-06-09 16:22:11.596795 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596793 7f8692f68700 7 osd.26 pg_epoch: 243 pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.5 2014-06-09 16:22:11.596799 7f86a5e11700 10 _finish_op 0x14dd7140 seq 3253 osr(2.182 0x2ed0990)/0x2ed0990 2014-06-09 16:22:11.596807 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e6c0 (1 active) 2014-06-09 16:22:11.596809 7f868ef60700 10 op_submit_start 3293 2014-06-09 16:22:11.596810 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1d40 (2 active) 2014-06-09 16:22:11.596811 7f868ef60700 5 queue_transactions (trailing journal) 3293 2014-06-09 16:22:11.596811 7f86a5e11700 5 _do_op 0x308cb320 seq 3254 osr(1.3cd 0x384d4b0)/0x384d4b0 start 2014-06-09 16:22:11.596798 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.685( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.596812 7f868ef60700 5 queue_op 0x2fcafe60 seq 3293 osr(2.67c 0x2ed1f90) 2048 bytes (queue has 50 ops and 88460 bytes) 2014-06-09 16:22:11.596814 7f86a5e11700 10 _do_transaction on 0xc0006c0 2014-06-09 16:22:11.596815 7f868ef60700 10 op_submit_finish 3293 2014-06-09 16:22:11.596803 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.5 info 3.730( v 214'648 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242), identical to ours 2014-06-09 16:22:11.596820 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.596822 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.596816 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.596826 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.596818 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.601( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.596826 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.73a( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.596837 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.596831 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.596830 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.601( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.601( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596841 7f8690f64700 5 queue_transactions existing osr(1.65e 0x384df30)/0x384df30 2014-06-09 16:22:11.596843 7f8690f64700 2 waiting 51 > 50 ops || 90343 > 104857600 2014-06-09 16:22:11.596837 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.596842 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596846 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596848 7f868f761700 10 op_submit_start 3294 2014-06-09 16:22:11.596850 7f868f761700 5 queue_transactions (trailing journal) 3294 2014-06-09 16:22:11.596850 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.596850 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.596841 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.117( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596851 7f868f761700 5 queue_op 0x30d96c00 seq 3294 osr(3.18e 0x3ed0af0) 1429 bytes (queue has 50 ops and 88460 bytes) 2014-06-09 16:22:11.596854 7f868f761700 10 op_submit_finish 3294 2014-06-09 16:22:11.596853 7f868af58700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596850 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.601( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.366455 7 0.000204 2014-06-09 16:22:11.596858 7f868af58700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596860 7f8691765700 10 op_submit_start 3295 2014-06-09 16:22:11.596862 7f8691765700 5 queue_transactions (trailing journal) 3295 2014-06-09 16:22:11.596854 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.596864 7f8691765700 5 queue_op 0x30607620 seq 3295 osr(3.3a5 0x3ed0fb0) 1335 bytes (queue has 50 ops and 88460 bytes) 2014-06-09 16:22:11.596867 7f8691765700 10 op_submit_finish 3295 2014-06-09 16:22:11.596861 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.601( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596860 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.596862 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.117( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.083264 7 0.000177 2014-06-09 16:22:11.596865 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.596874 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.596871 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.601( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596872 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.596873 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.117( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.596873 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.596881 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.601( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596878 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.3ae( v 213'576 (0'0,213'576] local-les=191 n=111 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.596882 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.596884 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.117( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.596885 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.25d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596889 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.601( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.596890 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.596897 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.596891 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.596902 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.596894 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.596898 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.601( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.596903 7f868f761700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.596906 7f868bf5a700 10 op_submit_start 3296 2014-06-09 16:22:11.601870 7f868bf5a700 5 queue_transactions (trailing journal) 3296 2014-06-09 16:22:11.601874 7f868bf5a700 5 queue_op 0x308cb200 seq 3296 osr(2.26d 0x2ed0de0) 1911 bytes (queue has 50 ops and 88460 bytes) 2014-06-09 16:22:11.601878 7f868bf5a700 10 op_submit_finish 3296 2014-06-09 16:22:11.601885 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.601883 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.138( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.596905 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.601903 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.601899 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.601909 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.601913 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.601908 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:1.1da( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.601905 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.601918 7f868ef60700 5 queue_transactions existing osr(1.601 0x384dcd0)/0x384dcd0 2014-06-09 16:22:11.601914 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:2.5f4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.596906 7f868b759700 2 waiting 51 > 50 ops || 90363 > 104857600 2014-06-09 16:22:11.601919 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.601928 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.601929 7f868c75b700 2 waiting 51 > 50 ops || 90414 > 104857600 2014-06-09 16:22:11.601924 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.601933 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.601934 7f868d75d700 2 waiting 51 > 50 ops || 89795 > 104857600 2014-06-09 16:22:11.596978 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a1c0,0xbdbb320,0xbcc9160,0xbfcecc0,0x189ab4c0,0xbd7c4e0,0xbdf17a0,0xbd5ade0,0xbd70a40,0xbd74f00,0xc410920,0x139645c0,0xbd72720,0xc6e84c0,0xbd6c780,0xbd7b400,0xbd6a060] 2014-06-09 16:22:11.601932 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.601937 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.601942 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.601944 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.601946 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.601957 7f868bf5a700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.601959 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.601960 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.601960 7f868bf5a700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.601956 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.837513 7 0.000178 2014-06-09 16:22:11.601964 7f868af58700 5 queue_transactions existing osr(0.117 0x2bd66a0)/0x2bd66a0 2014-06-09 16:22:11.601967 7f868af58700 2 waiting 51 > 50 ops || 90343 > 104857600 2014-06-09 16:22:11.601964 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.101( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.966031 7 0.000175 2014-06-09 16:22:11.601973 7f868a757700 2 waiting 51 > 50 ops || 90414 > 104857600 2014-06-09 16:22:11.601967 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.601973 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.101( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.601979 7f868ef60700 2 waiting 51 > 50 ops || 90414 > 104857600 2014-06-09 16:22:11.601976 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.601979 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.601982 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.101( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.601990 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.601989 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=208 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.601994 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.601998 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.601998 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'618 -> 213'618 2014-06-09 16:22:11.602007 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.602006 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602018 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] flushed 2014-06-09 16:22:11.602032 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602043 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.602042 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602052 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.602057 7f868bf5a700 5 queue_transactions existing osr(2.101 0x2ed0730)/0x2ed0730 2014-06-09 16:22:11.602059 7f868bf5a700 2 waiting 51 > 50 ops || 90343 > 104857600 2014-06-09 16:22:11.602056 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.602078 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.602072 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602082 7f868f761700 5 queue_transactions existing osr(3.2d6 0x3ed0e60)/0x3ed0e60 2014-06-09 16:22:11.602084 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.602085 7f868f761700 2 waiting 51 > 50 ops || 89795 > 104857600 2014-06-09 16:22:11.596905 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.602096 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.602095 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.714( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.602109 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.602113 7f8691f66700 10 op_submit_start 3297 2014-06-09 16:22:11.602115 7f8691f66700 5 queue_transactions (trailing journal) 3297 2014-06-09 16:22:11.602116 7f8691f66700 5 queue_op 0x308cb980 seq 3297 osr(1.66e 0x384ded0) 1621 bytes (queue has 50 ops and 88460 bytes) 2014-06-09 16:22:11.602119 7f8691f66700 10 op_submit_finish 3297 2014-06-09 16:22:11.602112 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602125 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.602137 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.3d6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602150 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602154 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.596996 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3254.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.602158 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.236258 7 0.021581 2014-06-09 16:22:11.602169 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.602172 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602183 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602194 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602203 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.602216 7f86a5e11700 10 _do_op 0x308cb320 seq 3254 r = 0, finisher 0x30628ba0 0 2014-06-09 16:22:11.602212 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602221 7f86a5e11700 10 _finish_op 0x308cb320 seq 3254 osr(1.3cd 0x384d4b0)/0x384d4b0 2014-06-09 16:22:11.602227 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1d40 (1 active) 2014-06-09 16:22:11.602230 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52dd40 (2 active) 2014-06-09 16:22:11.602232 7f86a5e11700 5 _do_op 0x30d96c60 seq 3255 osr(3.f0 0x3ed04a0)/0x3ed04a0 start 2014-06-09 16:22:11.602235 7f86a5e11700 10 _do_transaction on 0x3859200 2014-06-09 16:22:11.602241 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.602263 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.602269 7f868e75f700 10 op_submit_start 3298 2014-06-09 16:22:11.602271 7f868e75f700 5 queue_transactions (trailing journal) 3298 2014-06-09 16:22:11.602273 7f868e75f700 5 queue_op 0x2fcae000 seq 3298 osr(2.350 0x2ed0e50) 2009 bytes (queue has 49 ops and 86506 bytes) 2014-06-09 16:22:11.602277 7f868e75f700 10 op_submit_finish 3298 2014-06-09 16:22:11.602284 7f8692767700 10 op_submit_start 3299 2014-06-09 16:22:11.602287 7f8692767700 5 queue_transactions (trailing journal) 3299 2014-06-09 16:22:11.602286 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.602289 7f8692767700 5 queue_op 0x309a2180 seq 3299 osr(2.5b6 0x2ed1b90) 1997 bytes (queue has 49 ops and 86506 bytes) 2014-06-09 16:22:11.602292 7f8692767700 10 op_submit_finish 3299 2014-06-09 16:22:11.602282 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.75( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.602296 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.602304 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3255.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.602307 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.602308 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.602299 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.718( v 213'620 (0'0,213'620] local-les=191 n=121 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.602316 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.602317 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.1 2014-06-09 16:22:11.602328 7f86a5e11700 10 _do_op 0x30d96c60 seq 3255 r = 0, finisher 0x1322dfa0 0 2014-06-09 16:22:11.602332 7f86a5e11700 10 _finish_op 0x30d96c60 seq 3255 osr(3.f0 0x3ed04a0)/0x3ed04a0 2014-06-09 16:22:11.602334 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52dd40 (1 active) 2014-06-09 16:22:11.602336 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd7a0 (2 active) 2014-06-09 16:22:11.602328 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 2.236( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602339 7f86a5e11700 5 _do_op 0x14dd6de0 seq 3256 osr(2.759 0x3ed00b0)/0x3ed00b0 start 2014-06-09 16:22:11.602339 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602341 7f86a5e11700 10 _do_transaction on 0xc144b40 2014-06-09 16:22:11.602342 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.602328 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.1 info 3.718( v 213'620 (0'0,213'620] local-les=191 n=121 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.602347 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.602344 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.236( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.287346 7 0.000175 2014-06-09 16:22:11.602349 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.602351 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.236( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602357 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.236( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602363 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602355 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.94( v 214'571 (0'0,214'571] local-les=213 n=112 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.602368 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.602373 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602379 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.602372 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.602386 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.602395 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3256.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.602400 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.602391 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.99( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.602403 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.602412 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.602413 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.602416 7f8692f68700 5 queue_transactions existing osr(2.236 0x2ed0ad0)/0x2ed0ad0 2014-06-09 16:22:11.602419 7f8692f68700 10 op_submit_start 3300 2014-06-09 16:22:11.602420 7f8692f68700 5 queue_transactions (trailing journal) 3300 2014-06-09 16:22:11.602421 7f8692f68700 5 queue_op 0x309a3560 seq 3300 osr(2.236 0x2ed0ad0) 2009 bytes (queue has 49 ops and 87183 bytes) 2014-06-09 16:22:11.602424 7f8692f68700 10 op_submit_finish 3300 2014-06-09 16:22:11.602425 7f86a5e11700 10 _do_op 0x14dd6de0 seq 3256 r = 0, finisher 0xbccd760 0 2014-06-09 16:22:11.602418 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:2.36c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.602428 7f86a5e11700 10 _finish_op 0x14dd6de0 seq 3256 osr(2.759 0x3ed00b0)/0x3ed00b0 2014-06-09 16:22:11.602431 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd7a0 (1 active) 2014-06-09 16:22:11.602433 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fca20 (2 active) 2014-06-09 16:22:11.602426 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.602435 7f86a5e11700 5 _do_op 0x308ca000 seq 3257 osr(1.71e 0x2ed00b0)/0x2ed00b0 start 2014-06-09 16:22:11.602430 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.602440 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.596901 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.602441 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.186( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.602449 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.602443 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.602444 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.246091 7 0.000184 2014-06-09 16:22:11.602452 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.2e7( v 214'619 (0'0,214'619] local-les=210 n=127 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.602458 7f8690f64700 10 op_submit_start 3301 2014-06-09 16:22:11.602461 7f8690f64700 5 queue_transactions (trailing journal) 3301 2014-06-09 16:22:11.602462 7f8690f64700 5 queue_op 0x2fcaf920 seq 3301 osr(1.65e 0x384df30) 1883 bytes (queue has 49 ops and 87183 bytes) 2014-06-09 16:22:11.602465 7f8690f64700 10 op_submit_finish 3301 2014-06-09 16:22:11.602461 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.602468 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.602461 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602457 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.602471 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.44f( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.602473 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602469 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.ce( v 214'547 (0'0,214'547] local-les=213 n=109 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.602471 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.139356 7 0.000167 2014-06-09 16:22:11.602478 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.602485 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.602481 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602483 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.602484 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602494 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.602491 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.602489 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.718( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.602496 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602499 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602498 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=212 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.602502 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.602513 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.602507 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=212 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602509 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=212 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.602518 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'701 -> 213'701 2014-06-09 16:22:11.602518 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.602521 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=212 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.359697 7 0.000177 2014-06-09 16:22:11.602532 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=212 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602529 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.48d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602541 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602545 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.602545 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.602542 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=212 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602549 7f8691765700 5 queue_transactions existing osr(0.25d 0x2bd6a00)/0x2bd6a00 2014-06-09 16:22:11.602552 7f8691765700 10 op_submit_start 3302 2014-06-09 16:22:11.602553 7f8691765700 5 queue_transactions (trailing journal) 3302 2014-06-09 16:22:11.602554 7f8691765700 5 queue_op 0x30060360 seq 3302 osr(0.25d 0x2bd6a00) 1883 bytes (queue has 50 ops and 89066 bytes) 2014-06-09 16:22:11.602557 7f8691765700 10 op_submit_finish 3302 2014-06-09 16:22:11.602548 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.692604 7 0.000190 2014-06-09 16:22:11.602552 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=212 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602560 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602560 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.602561 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'647 -> 213'647 2014-06-09 16:22:11.602567 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602563 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602573 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602570 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.61f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602578 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.602582 7f8691765700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602582 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602586 7f8691765700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.602589 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.371399 7 0.000176 2014-06-09 16:22:11.602599 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602605 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.602610 7f8692767700 5 queue_transactions existing osr(3.3a8 0x3ed0f80)/0x3ed0f80 2014-06-09 16:22:11.602605 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602613 7f8692767700 2 waiting 51 > 50 ops || 90401 > 104857600 2014-06-09 16:22:11.602610 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602617 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.602618 7f8689f56700 10 op_submit_start 3303 2014-06-09 16:22:11.602619 7f8689f56700 5 queue_transactions (trailing journal) 3303 2014-06-09 16:22:11.602621 7f8689f56700 5 queue_op 0x30d974a0 seq 3303 osr(0.456 0x2bd7530) 2001 bytes (queue has 50 ops and 89066 bytes) 2014-06-09 16:22:11.602621 7f8692f68700 5 queue_transactions existing osr(2.48d 0x2ed1470)/0x2ed1470 2014-06-09 16:22:11.602624 7f8689f56700 10 op_submit_finish 3303 2014-06-09 16:22:11.602619 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602625 7f8692f68700 2 waiting 51 > 50 ops || 90949 > 104857600 2014-06-09 16:22:11.602630 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.602627 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.602635 7f8691f66700 5 queue_transactions existing osr(1.3d6 0x384d470)/0x384d470 2014-06-09 16:22:11.602639 7f8691f66700 2 waiting 51 > 50 ops || 90949 > 104857600 2014-06-09 16:22:11.602634 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602641 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.602630 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=191 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.602647 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.602645 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=191 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.602646 7f8690f64700 5 queue_transactions existing osr(3.381 0x3ed0fe0)/0x3ed0fe0 2014-06-09 16:22:11.602660 7f8690f64700 2 waiting 51 > 50 ops || 90401 > 104857600 2014-06-09 16:22:11.597111 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd66060,0xc169640,0x189ab680,0xbd70ec0,0xbfcf600,0xc410680,0x275cfe00,0xbd73b00,0xbd71d80,0xbd70420,0x305094c0,0x187420a0] 2014-06-09 16:22:11.602658 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=191 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.711716 7 0.000358 2014-06-09 16:22:11.602666 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602670 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=191 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602668 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.602679 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.602681 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.602683 7f8691765700 5 queue_transactions existing osr(0.61f 0x2bd7d30)/0x2bd7d30 2014-06-09 16:22:11.602686 7f8691765700 2 waiting 51 > 50 ops || 91040 > 104857600 2014-06-09 16:22:11.602682 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=191 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602690 7f868b759700 2 waiting 51 > 50 ops || 90969 > 104857600 2014-06-09 16:22:11.602682 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.3a9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602692 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a1c0,0xbdbb320,0xbcc9160,0xbfcecc0,0x189ab4c0,0xbd7c4e0,0xbdf17a0,0xbd5ade0,0xbd70a40,0xbd74f00,0xc410920,0x139645c0,0xbd72720,0xc6e84c0,0xbd6c780,0xbd7b400,0xbd6a060] 2014-06-09 16:22:11.602695 7f868c75b700 2 waiting 51 > 50 ops || 91020 > 104857600 2014-06-09 16:22:11.602694 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602696 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30628ba0,0x1322dfa0,0xbccd760] 2014-06-09 16:22:11.602698 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.602699 7f868d75d700 2 waiting 51 > 50 ops || 90401 > 104857600 2014-06-09 16:22:11.602693 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=191 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602704 7f86a5e11700 10 _do_transaction on 0xc326fc0 2014-06-09 16:22:11.602704 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30628ba0,0x1322dfa0,0xbccd760] 2014-06-09 16:22:11.602706 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.602698 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.602707 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.602701 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.140577 7 0.000173 2014-06-09 16:22:11.602710 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.602703 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'670 -> 213'670 2014-06-09 16:22:11.602712 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602712 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602727 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.602722 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602731 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602740 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.602750 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.602741 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.602748 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602748 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602759 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602770 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3237.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.602772 7f86a460e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.602775 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.602791 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.602793 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.602788 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.602796 7f8689f56700 5 queue_transactions existing osr(3.761 0x3ed1e10)/0x3ed1e10 2014-06-09 16:22:11.602797 7f868e75f700 5 queue_transactions existing osr(0.3a9 0x2bd7310)/0x2bd7310 2014-06-09 16:22:11.602798 7f8689f56700 2 waiting 51 > 50 ops || 90495 > 104857600 2014-06-09 16:22:11.602798 7f86a5610700 10 _do_op 0x30d97020 seq 3237 r = 0, finisher 0xbd78100 0 2014-06-09 16:22:11.602800 7f868e75f700 2 waiting 51 > 50 ops || 90949 > 104857600 2014-06-09 16:22:11.602801 7f86a5610700 10 _finish_op 0x30d97020 seq 3237 osr(3.254 0x3ed0c30)/0x3ed0c30 2014-06-09 16:22:11.602804 7f8690763700 10 op_submit_start 3304 2014-06-09 16:22:11.602805 7f8690763700 5 queue_transactions (trailing journal) 3304 2014-06-09 16:22:11.602807 7f8690763700 5 queue_op 0x2fcaef60 seq 3304 osr(1.6f5 0x2ed0170) 1974 bytes (queue has 49 ops and 87637 bytes) 2014-06-09 16:22:11.602801 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602808 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc480 (1 active) 2014-06-09 16:22:11.602811 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406240 (2 active) 2014-06-09 16:22:11.602814 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.602815 7f86a5610700 5 _do_op 0x14dd6cc0 seq 3258 osr(0.6a8 0x2bd7d90)/0x2bd7d90 start 2014-06-09 16:22:11.602817 7f8690763700 10 op_submit_finish 3304 2014-06-09 16:22:11.602818 7f86a5610700 10 _do_transaction on 0xc0ed980 2014-06-09 16:22:11.602825 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.602822 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.602830 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.602839 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.602834 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.15b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602847 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602843 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602851 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.602852 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd78100] 2014-06-09 16:22:11.602856 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.602855 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3257.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.602859 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.602860 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd78100] 2014-06-09 16:22:11.602862 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.602855 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.065714 7 0.000209 2014-06-09 16:22:11.602863 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.602866 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.602870 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.602877 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.602882 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602886 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.602895 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.602895 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.602904 7f86a5e11700 10 _do_op 0x308ca000 seq 3257 r = 0, finisher 0x2d738bc0 0 2014-06-09 16:22:11.602909 7f86a5e11700 10 _finish_op 0x308ca000 seq 3257 osr(1.71e 0x2ed00b0)/0x2ed00b0 2014-06-09 16:22:11.602903 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.602913 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fca20 (1 active) 2014-06-09 16:22:11.602909 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.602915 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383eb40 (2 active) 2014-06-09 16:22:11.602917 7f86a5e11700 5 _do_op 0x308ca9c0 seq 3259 osr(3.7bc 0xb7f0190)/0xb7f0190 start 2014-06-09 16:22:11.602920 7f86a5e11700 10 _do_transaction on 0xc294540 2014-06-09 16:22:11.602926 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.602923 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602934 7f86a460e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.595070 7f869af78700 5 -- op tracker -- , seq: 2057, time: 2014-06-09 16:22:11.595067, event: started, op: pg_notify(1.37e(23),0.b7(20) epoch 243) 2014-06-09 16:22:11.602950 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.602955 7f8690763700 5 queue_transactions existing osr(1.15b 0x384cbf0)/0x384cbf0 2014-06-09 16:22:11.602958 7f8690763700 10 op_submit_start 3305 2014-06-09 16:22:11.602959 7f8690763700 5 queue_transactions (trailing journal) 3305 2014-06-09 16:22:11.602960 7f8690763700 5 queue_op 0x2fcaef00 seq 3305 osr(1.15b 0x384cbf0) 1883 bytes (queue has 50 ops and 88972 bytes) 2014-06-09 16:22:11.602963 7f8690763700 10 op_submit_finish 3305 2014-06-09 16:22:11.602961 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.602948 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.602966 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.602970 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.602973 7f869af78700 5 -- op tracker -- , seq: 2057, time: 2014-06-09 16:22:11.602969, event: done, op: pg_notify(1.37e(23),0.b7(20) epoch 243) 2014-06-09 16:22:11.602978 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3259.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.602981 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.602983 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.602977 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.71b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602989 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.602991 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.602989 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.602993 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.603002 7f86a5e11700 10 _do_op 0x308ca9c0 seq 3259 r = 0, finisher 0xc40c860 0 2014-06-09 16:22:11.602993 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.603004 7f86a5e11700 10 _finish_op 0x308ca9c0 seq 3259 osr(3.7bc 0xb7f0190)/0xb7f0190 2014-06-09 16:22:11.603008 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383eb40 (1 active) 2014-06-09 16:22:11.603009 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd200 (2 active) 2014-06-09 16:22:11.603004 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.603012 7f86a5e11700 5 _do_op 0x308cb2c0 seq 3260 osr(1.494 0x384d990)/0x384d990 start 2014-06-09 16:22:11.603014 7f86a5e11700 10 _do_transaction on 0xbf26240 2014-06-09 16:22:11.603015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603016 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603007 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] exit Started/Stray 0.826963 7 0.000178 2014-06-09 16:22:11.603016 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.603018 7f869af78700 20 osd.26 243 _dispatch 0x307e7880 pg_info(1 pgs e243:0.67e) v4 2014-06-09 16:22:11.603020 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603019 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603024 7f869af78700 5 -- op tracker -- , seq: 2058, time: 2014-06-09 16:22:11.333902, event: header_read, op: pg_info(1 pgs e243:0.67e) 2014-06-09 16:22:11.603030 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.603031 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.603051 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603044 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.603039 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.603058 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.603058 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.603066 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3260.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603070 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.603066 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.603069 7f869af78700 5 -- op tracker -- , seq: 2058, time: 2014-06-09 16:22:11.333903, event: throttled, op: pg_info(1 pgs e243:0.67e) 2014-06-09 16:22:11.603072 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.603080 7f869af78700 5 -- op tracker -- , seq: 2058, time: 2014-06-09 16:22:11.333915, event: all_read, op: pg_info(1 pgs e243:0.67e) 2014-06-09 16:22:11.603087 7f869af78700 5 -- op tracker -- , seq: 2058, time: 2014-06-09 16:22:11.603014, event: dispatched, op: pg_info(1 pgs e243:0.67e) 2014-06-09 16:22:11.603091 7f86a5e11700 10 _do_op 0x308cb2c0 seq 3260 r = 0, finisher 0xbd7d0a0 0 2014-06-09 16:22:11.603085 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.603094 7f86a5e11700 10 _finish_op 0x308cb2c0 seq 3260 osr(1.494 0x384d990)/0x384d990 2014-06-09 16:22:11.603098 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.603100 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd200 (1 active) 2014-06-09 16:22:11.603102 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa900 (2 active) 2014-06-09 16:22:11.603104 7f86a5e11700 5 _do_op 0x308ca240 seq 3261 osr(0.530 0x2bd78f0)/0x2bd78f0 start 2014-06-09 16:22:11.603107 7f86a5e11700 10 _do_transaction on 0x3473800 2014-06-09 16:22:11.603107 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.603104 7f8689755700 5 -- op tracker -- , seq: 759, time: 2014-06-09 16:22:11.594421, event: reached_pg, op: osd_op(client.4576.0:36704 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~880640] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.603112 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603118 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d738bc0,0xc40c860,0xbd7d0a0] 2014-06-09 16:22:11.603126 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603127 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d738bc0,0xc40c860,0xbd7d0a0] 2014-06-09 16:22:11.603129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.603130 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.603118 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.603122 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.603134 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x257c4600 2014-06-09 16:22:11.603136 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.603144 7f8689755700 10 osd.26 243 dequeue_op 0x257c4600 finish 2014-06-09 16:22:11.603143 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3258.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603147 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.603150 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.603149 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.603157 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.603161 7f8689755700 10 osd.26 243 dequeue_op 0x1b0b6700 prio 63 cost 245760 latency 1.204192 osd_op(client.4572.0:30404 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.603175 7f869af78700 5 -- op tracker -- , seq: 2058, time: 2014-06-09 16:22:11.603092, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.67e) 2014-06-09 16:22:11.603171 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] flushed 2014-06-09 16:22:11.603184 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.67e) v4 from osd.8 2014-06-09 16:22:11.603188 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e7880 2014-06-09 16:22:11.603186 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd66060,0xc169640,0x189ab680,0xbd70ec0,0xbfcf600,0xc410680,0x275cfe00,0xbd73b00,0xbd71d80,0xbd70420,0x305094c0,0x187420a0] 2014-06-09 16:22:11.603190 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x17001780,0xbd70f00,0xbd73b60,0xc2d7ce0,0x2d7391c0,0xccf8920,0x3062a160] 2014-06-09 16:22:11.603194 7f869af78700 5 -- op tracker -- , seq: 2058, time: 2014-06-09 16:22:11.603192, event: started, op: pg_info(1 pgs e243:0.67e) 2014-06-09 16:22:11.603182 7f868cf5c700 10 osd.26 243 dequeue_op 0x11e63000 prio 63 cost 1032192 latency 1.472900 osd_op(client.4575.0:31984 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~1032192] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.603194 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.603207 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.603207 7f868cf5c700 5 -- op tracker -- , seq: 103, time: 2014-06-09 16:22:11.603201, event: reached_pg, op: osd_op(client.4575.0:31984 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 1957888~1032192] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.603217 7f86a5610700 10 _do_op 0x14dd6cc0 seq 3258 r = 0, finisher 0xc6e8180 0 2014-06-09 16:22:11.603221 7f86a5610700 10 _finish_op 0x14dd6cc0 seq 3258 osr(0.6a8 0x2bd7d90)/0x2bd7d90 2014-06-09 16:22:11.603226 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406240 (1 active) 2014-06-09 16:22:11.603218 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.603222 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.603229 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d05a0 (2 active) 2014-06-09 16:22:11.603231 7f86a5610700 5 _do_op 0x308cba40 seq 3262 osr(0.578 0x2bd7800)/0x2bd7800 start 2014-06-09 16:22:11.603234 7f86a5610700 10 _do_transaction on 0xc227440 2014-06-09 16:22:11.603238 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603231 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x11e63000 2014-06-09 16:22:11.603240 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8180] 2014-06-09 16:22:11.603241 7f868cf5c700 10 osd.26 243 dequeue_op 0x11e63000 finish 2014-06-09 16:22:11.603235 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.603247 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8180] 2014-06-09 16:22:11.603247 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.603249 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.603248 7f86a460e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.603250 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.603245 7f869af78700 5 -- op tracker -- , seq: 2058, time: 2014-06-09 16:22:11.603209, event: done, op: pg_info(1 pgs e243:0.67e) 2014-06-09 16:22:11.603257 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603260 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603261 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.603271 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603272 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603273 7f869af78700 20 osd.26 243 _dispatch 0x19eb9c00 pg_notify(2.18e(20) epoch 243) v5 2014-06-09 16:22:11.603273 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (20 active) 2014-06-09 16:22:11.603276 7f869af78700 5 -- op tracker -- , seq: 2059, time: 2014-06-09 16:22:11.347767, event: header_read, op: pg_notify(2.18e(20) epoch 243) 2014-06-09 16:22:11.603283 7f869af78700 5 -- op tracker -- , seq: 2059, time: 2014-06-09 16:22:11.347768, event: throttled, op: pg_notify(2.18e(20) epoch 243) 2014-06-09 16:22:11.603280 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=210 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.603290 7f869af78700 5 -- op tracker -- , seq: 2059, time: 2014-06-09 16:22:11.347799, event: all_read, op: pg_notify(2.18e(20) epoch 243) 2014-06-09 16:22:11.603297 7f869af78700 5 -- op tracker -- , seq: 2059, time: 2014-06-09 16:22:11.603271, event: dispatched, op: pg_notify(2.18e(20) epoch 243) 2014-06-09 16:22:11.603293 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=210 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.603304 7f869af78700 5 -- op tracker -- , seq: 2059, time: 2014-06-09 16:22:11.603302, event: waiting_for_osdmap, op: pg_notify(2.18e(20) epoch 243) 2014-06-09 16:22:11.603307 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.603308 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb9c00 2014-06-09 16:22:11.603311 7f869af78700 5 -- op tracker -- , seq: 2059, time: 2014-06-09 16:22:11.603310, event: started, op: pg_notify(2.18e(20) epoch 243) 2014-06-09 16:22:11.603306 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=210 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.967293 7 0.000155 2014-06-09 16:22:11.603321 7f869af78700 5 -- op tracker -- , seq: 2059, time: 2014-06-09 16:22:11.603320, event: done, op: pg_notify(2.18e(20) epoch 243) 2014-06-09 16:22:11.603317 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=210 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603326 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603327 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603326 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603329 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=210 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.603341 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603342 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603342 7f869af78700 20 osd.26 243 _dispatch 0x3044c540 pg_notify(0.70f(25) epoch 243) v5 2014-06-09 16:22:11.603345 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3261.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603340 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=210 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.603345 7f869af78700 5 -- op tracker -- , seq: 2060, time: 2014-06-09 16:22:11.247632, event: header_read, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.603349 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.603353 7f869af78700 5 -- op tracker -- , seq: 2060, time: 2014-06-09 16:22:11.247632, event: throttled, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.603350 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'676 -> 213'676 2014-06-09 16:22:11.603359 7f869af78700 5 -- op tracker -- , seq: 2060, time: 2014-06-09 16:22:11.247656, event: all_read, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.603368 7f86a5e11700 10 _do_op 0x308ca240 seq 3261 r = 0, finisher 0xbd653a0 0 2014-06-09 16:22:11.603366 7f869af78700 5 -- op tracker -- , seq: 2060, time: 2014-06-09 16:22:11.603341, event: dispatched, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.603372 7f86a5e11700 10 _finish_op 0x308ca240 seq 3261 osr(0.530 0x2bd78f0)/0x2bd78f0 2014-06-09 16:22:11.603374 7f869af78700 5 -- op tracker -- , seq: 2060, time: 2014-06-09 16:22:11.603372, event: waiting_for_osdmap, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.603377 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa900 (1 active) 2014-06-09 16:22:11.603379 7f869af78700 7 osd.26 243 handle_pg_notify from osd.0 2014-06-09 16:22:11.603380 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383e900 (2 active) 2014-06-09 16:22:11.603381 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044c540 2014-06-09 16:22:11.603382 7f86a5e11700 5 _do_op 0x2fcaff80 seq 3263 osr(3.2b 0x3ed0350)/0x3ed0350 start 2014-06-09 16:22:11.603384 7f86a5e11700 10 _do_transaction on 0xc059740 2014-06-09 16:22:11.603386 7f869af78700 5 -- op tracker -- , seq: 2060, time: 2014-06-09 16:22:11.603384, event: started, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.603390 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603397 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.603402 7f869af78700 5 -- op tracker -- , seq: 2060, time: 2014-06-09 16:22:11.603400, event: done, op: pg_notify(0.70f(25) epoch 243) 2014-06-09 16:22:11.603409 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603411 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603424 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603432 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603433 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603434 7f869af78700 20 osd.26 243 _dispatch 0x1c03f880 pg_notify(2.523(20),1.25a(23),2.266(20) epoch 243) v5 2014-06-09 16:22:11.603437 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.603438 7f869af78700 5 -- op tracker -- , seq: 2061, time: 2014-06-09 16:22:11.343049, event: header_read, op: pg_notify(2.523(20),1.25a(23),2.266(20) epoch 243) 2014-06-09 16:22:11.603442 7f868cf5c700 5 queue_transactions existing osr(3.5b3 0x3ed17e0)/0x3ed17e0 2014-06-09 16:22:11.603442 7f869af78700 5 -- op tracker -- , seq: 2061, time: 2014-06-09 16:22:11.343050, event: throttled, op: pg_notify(2.523(20),1.25a(23),2.266(20) epoch 243) 2014-06-09 16:22:11.603442 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3263.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603444 7f868cf5c700 10 op_submit_start 3306 2014-06-09 16:22:11.603446 7f868cf5c700 5 queue_transactions (trailing journal) 3306 2014-06-09 16:22:11.603446 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.603446 7f869af78700 5 -- op tracker -- , seq: 2061, time: 2014-06-09 16:22:11.343125, event: all_read, op: pg_notify(2.523(20),1.25a(23),2.266(20) epoch 243) 2014-06-09 16:22:11.603447 7f868cf5c700 5 queue_op 0x30d96960 seq 3306 osr(3.5b3 0x3ed17e0) 1335 bytes (queue has 47 ops and 83087 bytes) 2014-06-09 16:22:11.603449 7f868cf5c700 10 op_submit_finish 3306 2014-06-09 16:22:11.603450 7f869af78700 5 -- op tracker -- , seq: 2061, time: 2014-06-09 16:22:11.603432, event: dispatched, op: pg_notify(2.523(20),1.25a(23),2.266(20) epoch 243) 2014-06-09 16:22:11.603454 7f869af78700 5 -- op tracker -- , seq: 2061, time: 2014-06-09 16:22:11.603453, event: waiting_for_osdmap, op: pg_notify(2.523(20),1.25a(23),2.266(20) epoch 243) 2014-06-09 16:22:11.603457 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.603458 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03f880 2014-06-09 16:22:11.603461 7f869af78700 5 -- op tracker -- , seq: 2061, time: 2014-06-09 16:22:11.603459, event: started, op: pg_notify(2.523(20),1.25a(23),2.266(20) epoch 243) 2014-06-09 16:22:11.603452 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.74a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.603468 7f86a5e11700 10 _do_op 0x2fcaff80 seq 3263 r = 0, finisher 0xbd6ce60 0 2014-06-09 16:22:11.603471 7f86a5e11700 10 _finish_op 0x2fcaff80 seq 3263 osr(3.2b 0x3ed0350)/0x3ed0350 2014-06-09 16:22:11.603466 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.603474 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383e900 (1 active) 2014-06-09 16:22:11.603476 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.603479 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e000 (2 active) 2014-06-09 16:22:11.603481 7f86a5e11700 5 _do_op 0x308ca2a0 seq 3264 osr(0.3ab 0x2bd72f0)/0x2bd72f0 start 2014-06-09 16:22:11.603483 7f86a5e11700 10 _do_transaction on 0xc0faf00 2014-06-09 16:22:11.603480 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.1bc( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.603489 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603490 7f869af78700 5 -- op tracker -- , seq: 2061, time: 2014-06-09 16:22:11.603487, event: done, op: pg_notify(2.523(20),1.25a(23),2.266(20) epoch 243) 2014-06-09 16:22:11.603496 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603498 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603492 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.603502 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.603505 7f868ff62700 10 op_submit_start 3307 2014-06-09 16:22:11.603508 7f868ff62700 5 queue_transactions (trailing journal) 3307 2014-06-09 16:22:11.603510 7f868ff62700 5 queue_op 0x15996ba0 seq 3307 osr(1.508 0x384db90) 1883 bytes (queue has 46 ops and 82037 bytes) 2014-06-09 16:22:11.603513 7f868ff62700 10 op_submit_finish 3307 2014-06-09 16:22:11.603505 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.603521 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603516 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.1e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603519 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.383( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.603527 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603530 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603531 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603532 7f869af78700 20 osd.26 243 _dispatch 0x1a038700 pg_notify(1.25c(21) epoch 243) v5 2014-06-09 16:22:11.603531 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.603535 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3264.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603535 7f869af78700 5 -- op tracker -- , seq: 2062, time: 2014-06-09 16:22:11.171636, event: header_read, op: pg_notify(1.25c(21) epoch 243) 2014-06-09 16:22:11.603539 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.603532 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.603540 7f869af78700 5 -- op tracker -- , seq: 2062, time: 2014-06-09 16:22:11.171637, event: throttled, op: pg_notify(1.25c(21) epoch 243) 2014-06-09 16:22:11.603543 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.603535 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.795544 7 0.000901 2014-06-09 16:22:11.603544 7f869af78700 5 -- op tracker -- , seq: 2062, time: 2014-06-09 16:22:11.171662, event: all_read, op: pg_notify(1.25c(21) epoch 243) 2014-06-09 16:22:11.603547 7f869af78700 5 -- op tracker -- , seq: 2062, time: 2014-06-09 16:22:11.603530, event: dispatched, op: pg_notify(1.25c(21) epoch 243) 2014-06-09 16:22:11.603551 7f869af78700 5 -- op tracker -- , seq: 2062, time: 2014-06-09 16:22:11.603549, event: waiting_for_osdmap, op: pg_notify(1.25c(21) epoch 243) 2014-06-09 16:22:11.603553 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.603555 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a038700 2014-06-09 16:22:11.603548 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603547 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:2.744( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.603557 7f86a5e11700 10 _do_op 0x308ca2a0 seq 3264 r = 0, finisher 0xbd7bce0 0 2014-06-09 16:22:11.603557 7f869af78700 5 -- op tracker -- , seq: 2062, time: 2014-06-09 16:22:11.603556, event: started, op: pg_notify(1.25c(21) epoch 243) 2014-06-09 16:22:11.603560 7f86a5e11700 10 _finish_op 0x308ca2a0 seq 3264 osr(0.3ab 0x2bd72f0)/0x2bd72f0 2014-06-09 16:22:11.603563 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e000 (1 active) 2014-06-09 16:22:11.603565 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5c20 (2 active) 2014-06-09 16:22:11.603559 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.603559 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.603566 7f86a5e11700 5 _do_op 0x308cb9e0 seq 3265 osr(0.640 0x2bd7c10)/0x2bd7c10 start 2014-06-09 16:22:11.603568 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.603569 7f86a5e11700 10 _do_transaction on 0xc0d7e00 2014-06-09 16:22:11.603574 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603569 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.603573 7f869af78700 5 -- op tracker -- , seq: 2062, time: 2014-06-09 16:22:11.603571, event: done, op: pg_notify(1.25c(21) epoch 243) 2014-06-09 16:22:11.603572 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.603581 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603583 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603578 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.603587 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.603584 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.603602 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.603604 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.603603 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603596 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.138488 7 0.000169 2014-06-09 16:22:11.603605 7f869af78700 20 osd.26 243 _dispatch 0x26000540 pg_notify(3.55b(10) epoch 243) v5 2014-06-09 16:22:11.603628 7f8692767700 10 op_submit_start 3308 2014-06-09 16:22:11.603627 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd653a0,0xbd6ce60,0xbd7bce0] 2014-06-09 16:22:11.603630 7f8692767700 5 queue_transactions (trailing journal) 3308 2014-06-09 16:22:11.603631 7f8692767700 5 queue_op 0x309a2120 seq 3308 osr(3.3a8 0x3ed0f80) 1335 bytes (queue has 45 ops and 80154 bytes) 2014-06-09 16:22:11.603634 7f8692767700 10 op_submit_finish 3308 2014-06-09 16:22:11.603637 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd653a0,0xbd6ce60,0xbd7bce0] 2014-06-09 16:22:11.603639 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.603640 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.603645 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.603650 7f868cf5c700 5 queue_transactions existing osr(1.1e8 0x384cdf0)/0x384cdf0 2014-06-09 16:22:11.603649 7f8690763700 5 queue_transactions existing osr(0.71b 0x384c050)/0x384c050 2014-06-09 16:22:11.603652 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (19 active) 2014-06-09 16:22:11.603653 7f868cf5c700 10 op_submit_start 3309 2014-06-09 16:22:11.603654 7f868cf5c700 5 queue_transactions (trailing journal) 3309 2014-06-09 16:22:11.603655 7f868cf5c700 5 queue_op 0x30d96900 seq 3309 osr(1.1e8 0x384cdf0) 1883 bytes (queue has 47 ops and 83658 bytes) 2014-06-09 16:22:11.603658 7f868cf5c700 10 op_submit_finish 3309 2014-06-09 16:22:11.603658 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.603649 7f868df5e700 10 osd.26 243 dequeue_op 0x11e63600 prio 63 cost 516096 latency 1.464919 osd_op(client.4575.0:31985 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.603665 7f8690763700 10 op_submit_start 3310 2014-06-09 16:22:11.603667 7f8690763700 5 queue_transactions (trailing journal) 3310 2014-06-09 16:22:11.603611 7f869af78700 5 -- op tracker -- , seq: 2063, time: 2014-06-09 16:22:11.190273, event: header_read, op: pg_notify(3.55b(10) epoch 243) 2014-06-09 16:22:11.603669 7f8690763700 5 queue_op 0x2fcaeea0 seq 3310 osr(0.71b 0x384c050) 1621 bytes (queue has 47 ops and 83658 bytes) 2014-06-09 16:22:11.603672 7f8690763700 10 op_submit_finish 3310 2014-06-09 16:22:11.603663 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) 2014-06-09 16:22:11.603674 7f869af78700 5 -- op tracker -- , seq: 2063, time: 2014-06-09 16:22:11.190274, event: throttled, op: pg_notify(3.55b(10) epoch 243) 2014-06-09 16:22:11.603662 7f8692767700 10 osd.26 243 dequeue_op 0x270be800 prio 63 cost 212992 latency 0.836117 osd_op(client.4573.0:34086 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~212992] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.603680 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3265.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603681 7f869af78700 5 -- op tracker -- , seq: 2063, time: 2014-06-09 16:22:11.190288, event: all_read, op: pg_notify(3.55b(10) epoch 243) 2014-06-09 16:22:11.603683 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.603676 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.603678 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.603608 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603688 7f869af78700 5 -- op tracker -- , seq: 2063, time: 2014-06-09 16:22:11.603602, event: dispatched, op: pg_notify(3.55b(10) epoch 243) 2014-06-09 16:22:11.603689 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.603693 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.603690 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.431( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603703 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603699 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.603702 7f868df5e700 5 -- op tracker -- , seq: 111, time: 2014-06-09 16:22:11.603664, event: reached_pg, op: osd_op(client.4575.0:31985 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 2990080~516096] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.603707 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.603710 7f868a757700 10 op_submit_start 3311 2014-06-09 16:22:11.603704 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=201 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.603712 7f868a757700 5 queue_transactions (trailing journal) 3311 2014-06-09 16:22:11.603713 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.603713 7f86a5e11700 10 _do_op 0x308cb9e0 seq 3265 r = 0, finisher 0x1a003a00 0 2014-06-09 16:22:11.603715 7f868a757700 5 queue_op 0x309a36e0 seq 3311 osr(2.24d 0x2ed0a50) 1954 bytes (queue has 48 ops and 85612 bytes) 2014-06-09 16:22:11.603717 7f86a5e11700 10 _finish_op 0x308cb9e0 seq 3265 osr(0.640 0x2bd7c10)/0x2bd7c10 2014-06-09 16:22:11.603711 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.431( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.829113 7 0.000186 2014-06-09 16:22:11.603721 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5c20 (1 active) 2014-06-09 16:22:11.603723 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb440 (2 active) 2014-06-09 16:22:11.603711 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.603715 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'585 -> 213'585 2014-06-09 16:22:11.603727 7f86a5e11700 5 _do_op 0x30061800 seq 3266 osr(3.35c 0x3ed10c0)/0x3ed10c0 start 2014-06-09 16:22:11.603729 7f8692f68700 2 waiting 51 > 50 ops || 90587 > 104857600 2014-06-09 16:22:11.603729 7f86a5e11700 10 _do_transaction on 0xc23e000 2014-06-09 16:22:11.603723 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.431( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603733 7f8691f66700 2 waiting 51 > 50 ops || 90587 > 104857600 2014-06-09 16:22:11.603725 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x11e63600 2014-06-09 16:22:11.603735 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603735 7f868df5e700 10 osd.26 243 dequeue_op 0x11e63600 finish 2014-06-09 16:22:11.603738 7f868a757700 10 op_submit_finish 3311 2014-06-09 16:22:11.603729 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:971 2014-06-09 16:22:11.603740 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.603735 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.431( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.603746 7f868ef60700 2 waiting 51 > 50 ops || 90658 > 104857600 2014-06-09 16:22:11.603751 7f8691765700 2 waiting 51 > 50 ops || 90678 > 104857600 2014-06-09 16:22:11.603743 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.125( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.603746 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.603756 7f868b759700 2 waiting 51 > 50 ops || 90607 > 104857600 2014-06-09 16:22:11.603759 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02a0 (20 active) 2014-06-09 16:22:11.603761 7f868c75b700 2 waiting 51 > 50 ops || 90658 > 104857600 2014-06-09 16:22:11.603755 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.603755 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.603767 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.603767 7f868d75d700 2 waiting 51 > 50 ops || 90039 > 104857600 2014-06-09 16:22:11.603764 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.603765 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.603773 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1a003a00] 2014-06-09 16:22:11.603756 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.603778 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1a003a00] 2014-06-09 16:22:11.603771 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:0.479( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.603780 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.603774 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.603782 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.603783 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.603783 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.603789 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603791 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.603791 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.603790 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.603801 7f868df5e700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.603795 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.40f( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.603804 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3266.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603808 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.603797 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.48a( v 214'648 (0'0,214'648] local-les=208 n=120 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.603809 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.603807 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.603816 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.603816 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.603816 7f868ff62700 5 queue_transactions existing osr(3.52a 0x3ed1670)/0x3ed1670 2014-06-09 16:22:11.603819 7f868ff62700 2 waiting 51 > 50 ops || 90039 > 104857600 2014-06-09 16:22:11.603812 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.603815 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:189 2014-06-09 16:22:11.603822 7f8690763700 5 queue_transactions existing osr(2.431 0x2ed1590)/0x2ed1590 2014-06-09 16:22:11.603823 7f86a5e11700 10 _do_op 0x30061800 seq 3266 r = 0, finisher 0x1b0c10c0 0 2014-06-09 16:22:11.603825 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.603824 7f8690763700 2 waiting 51 > 50 ops || 90587 > 104857600 2014-06-09 16:22:11.603827 7f86a5e11700 10 _finish_op 0x30061800 seq 3266 osr(3.35c 0x3ed10c0)/0x3ed10c0 2014-06-09 16:22:11.603820 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.603830 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.603834 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb440 (1 active) 2014-06-09 16:22:11.603836 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc6c0 (2 active) 2014-06-09 16:22:11.603838 7f86a5e11700 5 _do_op 0x2fcaff20 seq 3267 osr(2.298 0x2ed0c70)/0x2ed0c70 start 2014-06-09 16:22:11.603830 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.555( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603830 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:3.74f( v 213'530 (0'0,213'530] local-les=191 n=106 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.603833 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.603841 7f86a5e11700 10 _do_transaction on 0xc2afc80 2014-06-09 16:22:11.603841 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603845 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.603847 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603843 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.603843 7f868cf5c700 7 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.0 2014-06-09 16:22:11.603848 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.555( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.154796 7 0.000185 2014-06-09 16:22:11.603851 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.603859 7f8690f64700 10 op_submit_start 3312 2014-06-09 16:22:11.603862 7f8690f64700 5 queue_transactions (trailing journal) 3312 2014-06-09 16:22:11.603858 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.603858 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.555( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603863 7f8690f64700 5 queue_op 0x2fcaf8c0 seq 3312 osr(3.381 0x3ed0fe0) 1335 bytes (queue has 50 ops and 88704 bytes) 2014-06-09 16:22:11.603866 7f8690f64700 10 op_submit_finish 3312 2014-06-09 16:22:11.603869 7f868df5e700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.603853 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.0 info 3.74f( v 213'530 (0'0,213'530] local-les=191 n=106 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.603868 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.555( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.603878 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.603869 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.603880 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603877 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.603879 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:191 2014-06-09 16:22:11.603882 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:2.754( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.603881 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 0.66a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603886 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.603893 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.603893 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603895 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3267.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603897 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.603899 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.603894 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.603894 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.603904 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.603897 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.603900 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.361256 7 0.000217 2014-06-09 16:22:11.603908 7f868df5e700 15 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.603908 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.603911 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603917 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.603923 7f86a5e11700 10 _do_op 0x2fcaff20 seq 3267 r = 0, finisher 0xc6e94a0 0 2014-06-09 16:22:11.603926 7f86a5e11700 10 _finish_op 0x2fcaff20 seq 3267 osr(2.298 0x2ed0c70)/0x2ed0c70 2014-06-09 16:22:11.603922 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.603919 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.3e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603922 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.603929 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc6c0 (1 active) 2014-06-09 16:22:11.603932 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114120 (2 active) 2014-06-09 16:22:11.603931 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.603933 7f86a5e11700 5 _do_op 0x308ca300 seq 3268 osr(0.9d 0x2bd6490)/0x2bd6490 start 2014-06-09 16:22:11.603936 7f86a5e11700 10 _do_transaction on 0xc161440 2014-06-09 16:22:11.603935 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.603931 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.603933 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.603938 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.603941 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.603941 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.603940 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.603938 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.245163 7 0.000159 2014-06-09 16:22:11.603943 7f868a757700 5 queue_transactions existing osr(1.555 0x384dab0)/0x384dab0 2014-06-09 16:22:11.603947 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.603944 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=201 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.603955 7f868a757700 10 op_submit_start 3313 2014-06-09 16:22:11.603956 7f868a757700 5 queue_transactions (trailing journal) 3313 2014-06-09 16:22:11.603950 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603958 7f868a757700 5 queue_op 0x30607c20 seq 3313 osr(1.555 0x384dab0) 1883 bytes (queue has 50 ops and 88966 bytes) 2014-06-09 16:22:11.603960 7f868a757700 10 op_submit_finish 3313 2014-06-09 16:22:11.603960 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.603957 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=201 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.603971 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.603963 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.67f( v 214'639 (0'0,214'639] local-les=203 n=124 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.603969 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.603970 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=201 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.688559 7 0.000145 2014-06-09 16:22:11.603977 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.603979 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.603987 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.603985 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3268.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.603981 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=201 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.603989 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.603987 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.603994 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.603999 7f8690f64700 5 queue_transactions existing osr(0.66a 0x2bd7ef0)/0x2bd7ef0 2014-06-09 16:22:11.603992 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=201 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.604001 7f8690f64700 2 waiting 51 > 50 ops || 90975 > 104857600 2014-06-09 16:22:11.604005 7f868e75f700 2 waiting 51 > 50 ops || 90849 > 104857600 2014-06-09 16:22:11.604005 7f86a5e11700 10 _do_op 0x308ca300 seq 3268 r = 0, finisher 0x17001f00 0 2014-06-09 16:22:11.603998 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:0.f1( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.604002 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=201 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.604008 7f86a5e11700 10 _finish_op 0x308ca300 seq 3268 osr(0.9d 0x2bd6490)/0x2bd6490 2014-06-09 16:22:11.604010 7f8691f66700 2 waiting 51 > 50 ops || 90849 > 104857600 2014-06-09 16:22:11.604014 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1b0c10c0,0xc6e94a0] 2014-06-09 16:22:11.604010 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.604012 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'531 -> 213'531 2014-06-09 16:22:11.604019 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.604022 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1b0c10c0,0xc6e94a0] 2014-06-09 16:22:11.604023 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.604024 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.604021 7f8692767700 5 -- op tracker -- , seq: 698, time: 2014-06-09 16:22:11.603683, event: reached_pg, op: osd_op(client.4573.0:34086 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~212992] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.604023 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.604032 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.604036 7f868cf5c700 5 queue_transactions existing osr(2.3e8 0x2ed1310)/0x2ed1310 2014-06-09 16:22:11.604039 7f868cf5c700 2 waiting 51 > 50 ops || 90849 > 104857600 2014-06-09 16:22:11.604036 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.604043 7f868ef60700 2 waiting 51 > 50 ops || 90920 > 104857600 2014-06-09 16:22:11.604045 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.604032 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.604048 7f8691765700 2 waiting 51 > 50 ops || 90940 > 104857600 2014-06-09 16:22:11.604052 7f868b759700 2 waiting 51 > 50 ops || 90869 > 104857600 2014-06-09 16:22:11.604048 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.604055 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.604049 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.604056 7f868c75b700 2 waiting 51 > 50 ops || 90920 > 104857600 2014-06-09 16:22:11.604047 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x270be800 2014-06-09 16:22:11.604060 7f8692767700 10 osd.26 243 dequeue_op 0x270be800 finish 2014-06-09 16:22:11.604063 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.604060 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.604068 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.604069 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.604071 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3262.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.604065 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604075 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.604071 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.604079 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.604085 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.604072 7f8692767700 10 osd.26 243 dequeue_op 0x11e63500 prio 63 cost 516096 latency 1.459482 osd_op(client.4575.0:31986 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.604082 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.604089 7f868df5e700 5 queue_transactions existing osr(3.1d5 0x3ed09f0)/0x3ed09f0 2014-06-09 16:22:11.604091 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.604092 7f868df5e700 2 waiting 51 > 50 ops || 90301 > 104857600 2014-06-09 16:22:11.604093 7f8692767700 5 -- op tracker -- , seq: 125, time: 2014-06-09 16:22:11.604088, event: reached_pg, op: osd_op(client.4575.0:31986 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 3506176~516096] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.604097 7f86a5610700 10 _do_op 0x308cba40 seq 3262 r = 0, finisher 0xc168ce0 0 2014-06-09 16:22:11.604094 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.604106 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (19 active) 2014-06-09 16:22:11.604100 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.604110 7f868d75d700 2 waiting 51 > 50 ops || 90301 > 104857600 2014-06-09 16:22:11.604111 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x11e63500 2014-06-09 16:22:11.604120 7f8692767700 10 osd.26 243 dequeue_op 0x11e63500 finish 2014-06-09 16:22:11.604123 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (20 active) 2014-06-09 16:22:11.604127 7f8689f56700 2 waiting 51 > 50 ops || 90395 > 104857600 2014-06-09 16:22:11.604131 7f8689755700 5 -- op tracker -- , seq: 308, time: 2014-06-09 16:22:11.603178, event: reached_pg, op: osd_op(client.4572.0:30404 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.604129 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.604137 7f8690763700 2 waiting 51 > 50 ops || 90849 > 104857600 2014-06-09 16:22:11.604140 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604141 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 1.1ef( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.604141 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.604152 7f868e75f700 10 op_submit_start 3314 2014-06-09 16:22:11.604154 7f868e75f700 5 queue_transactions (trailing journal) 3314 2014-06-09 16:22:11.604155 7f868e75f700 5 queue_op 0x30d97f80 seq 3314 osr(0.3a9 0x2bd7310) 1883 bytes (queue has 50 ops and 88843 bytes) 2014-06-09 16:22:11.604156 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.604161 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114120 (1 active) 2014-06-09 16:22:11.604154 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.604163 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525c20 (2 active) 2014-06-09 16:22:11.604165 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.604167 7f86a5e11700 5 _do_op 0x300617a0 seq 3269 osr(3.98 0x3ed0620)/0x3ed0620 start 2014-06-09 16:22:11.604154 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0b6700 2014-06-09 16:22:11.604170 7f86a5e11700 10 _do_transaction on 0x3e63d40 2014-06-09 16:22:11.604171 7f8689755700 10 osd.26 243 dequeue_op 0x1b0b6700 finish 2014-06-09 16:22:11.604174 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.604176 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.604168 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.713313 7 0.000184 2014-06-09 16:22:11.604170 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.604180 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.604183 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.604185 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604183 7f8689755700 10 osd.26 243 dequeue_op 0x1b0b6600 prio 63 cost 516096 latency 1.185832 osd_op(client.4572.0:30405 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 245760~516096] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.604197 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.604195 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.604203 7f8689755700 5 -- op tracker -- , seq: 309, time: 2014-06-09 16:22:11.604198, event: reached_pg, op: osd_op(client.4572.0:30405 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 245760~516096] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.604208 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.604205 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.604212 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.604210 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.604215 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.604223 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3269.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.604227 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.604223 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0b6600 2014-06-09 16:22:11.604224 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.604232 7f8689755700 10 osd.26 243 dequeue_op 0x1b0b6600 finish 2014-06-09 16:22:11.604225 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604235 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.604239 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.604244 7f86a5e11700 10 _do_op 0x300617a0 seq 3269 r = 0, finisher 0x2d739f40 0 2014-06-09 16:22:11.604247 7f86a5e11700 10 _finish_op 0x300617a0 seq 3269 osr(3.98 0x3ed0620)/0x3ed0620 2014-06-09 16:22:11.604249 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525c20 (1 active) 2014-06-09 16:22:11.604251 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcb40 (2 active) 2014-06-09 16:22:11.604253 7f86a5e11700 5 _do_op 0x2fcaf9e0 seq 3270 osr(2.30b 0x2ed0f50)/0x2ed0f50 start 2014-06-09 16:22:11.604255 7f86a5e11700 10 _do_transaction on 0xc262600 2014-06-09 16:22:11.604255 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0200 (20 active) 2014-06-09 16:22:11.604260 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.604253 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.604258 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.604274 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.604270 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.604268 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604280 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.604279 7f868a757700 5 queue_transactions existing osr(1.1ef 0x384cdb0)/0x384cdb0 2014-06-09 16:22:11.604282 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.604286 7f8692f68700 2 waiting 51 > 50 ops || 91395 > 104857600 2014-06-09 16:22:11.604290 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.604284 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.604292 7f868cf5c700 2 waiting 51 > 50 ops || 91395 > 104857600 2014-06-09 16:22:11.604293 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.604296 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.220( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.604305 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.604304 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x17001780,0xbd70f00,0xbd73b60,0xc2d7ce0,0x2d7391c0,0xccf8920,0x3062a160] 2014-06-09 16:22:11.604304 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3270.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.604308 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.604307 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x4eb8900,0xc169de0,0xbfcfe20,0xbef1a20,0xbebe1c0,0xbebf3a0,0xbd75e60,0xbd64340,0x33a7860,0xbd77480] 2014-06-09 16:22:11.604309 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.604311 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604323 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.604319 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.52d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.604330 7f86a5e11700 10 _do_op 0x2fcaf9e0 seq 3270 r = 0, finisher 0xbccd020 0 2014-06-09 16:22:11.604330 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.604333 7f86a5e11700 10 _finish_op 0x2fcaf9e0 seq 3270 osr(2.30b 0x2ed0f50)/0x2ed0f50 2014-06-09 16:22:11.604334 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.604338 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcb40 (1 active) 2014-06-09 16:22:11.604340 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa5a0 (2 active) 2014-06-09 16:22:11.604342 7f86a5e11700 5 _do_op 0x2fcafec0 seq 3271 osr(0.644 0x2bd7bf0)/0x2bd7bf0 start 2014-06-09 16:22:11.604336 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.604337 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.103935 7 0.000223 2014-06-09 16:22:11.604345 7f86a5e11700 10 _do_transaction on 0xc058f00 2014-06-09 16:22:11.604344 7f8691765700 2 waiting 51 > 50 ops || 91372 > 104857600 2014-06-09 16:22:11.604350 7f868b759700 2 waiting 51 > 50 ops || 91301 > 104857600 2014-06-09 16:22:11.604351 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.604347 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.604355 7f868c75b700 2 waiting 51 > 50 ops || 91352 > 104857600 2014-06-09 16:22:11.604349 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604362 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.604357 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.604365 7f868df5e700 2 waiting 51 > 50 ops || 90733 > 104857600 2014-06-09 16:22:11.604370 7f868d75d700 2 waiting 51 > 50 ops || 90733 > 104857600 2014-06-09 16:22:11.604367 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.604382 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.604376 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.604376 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.604384 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.604396 7f8689f56700 2 waiting 51 > 50 ops || 90827 > 104857600 2014-06-09 16:22:11.604396 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3271.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.604400 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.604389 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.604405 7f8690763700 2 waiting 51 > 50 ops || 91281 > 104857600 2014-06-09 16:22:11.604402 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.604412 7f868cf5c700 2 waiting 51 > 50 ops || 91281 > 104857600 2014-06-09 16:22:11.604416 7f86a5e11700 10 _do_op 0x2fcafec0 seq 3271 r = 0, finisher 0xbd6d860 0 2014-06-09 16:22:11.604419 7f86a5e11700 10 _finish_op 0x2fcafec0 seq 3271 osr(0.644 0x2bd7bf0)/0x2bd7bf0 2014-06-09 16:22:11.604415 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604425 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa5a0 (1 active) 2014-06-09 16:22:11.604427 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407560 (2 active) 2014-06-09 16:22:11.604428 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.604428 7f86a5e11700 5 _do_op 0x306067e0 seq 3272 osr(1.b5 0x384c690)/0x384c690 start 2014-06-09 16:22:11.604430 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.604429 7f868b759700 2 waiting 51 > 50 ops || 91392 > 104857600 2014-06-09 16:22:11.604431 7f86a5e11700 10 _do_transaction on 0xc0d9500 2014-06-09 16:22:11.604435 7f8689755700 5 queue_transactions existing osr(2.52d 0x2ed19f0)/0x2ed19f0 2014-06-09 16:22:11.604436 7f868c75b700 2 waiting 51 > 50 ops || 91443 > 104857600 2014-06-09 16:22:11.604437 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.604440 7f8691f66700 2 waiting 51 > 50 ops || 91372 > 104857600 2014-06-09 16:22:11.604439 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.604446 7f868d75d700 2 waiting 51 > 50 ops || 90824 > 104857600 2014-06-09 16:22:11.604451 7f869af78700 5 -- op tracker -- , seq: 2063, time: 2014-06-09 16:22:11.603700, event: waiting_for_osdmap, op: pg_notify(3.55b(10) epoch 243) 2014-06-09 16:22:11.604452 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.604458 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.604461 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26000540 2014-06-09 16:22:11.604463 7f8689f56700 2 waiting 51 > 50 ops || 90918 > 104857600 2014-06-09 16:22:11.604463 7f86a460e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.604466 7f869af78700 5 -- op tracker -- , seq: 2063, time: 2014-06-09 16:22:11.604464, event: started, op: pg_notify(3.55b(10) epoch 243) 2014-06-09 16:22:11.604468 7f8690763700 2 waiting 51 > 50 ops || 91372 > 104857600 2014-06-09 16:22:11.604470 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.604475 7f868cf5c700 2 waiting 51 > 50 ops || 91372 > 104857600 2014-06-09 16:22:11.604480 7f8689755700 2 waiting 51 > 50 ops || 91372 > 104857600 2014-06-09 16:22:11.604478 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.604486 7f868df5e700 2 waiting 51 > 50 ops || 90824 > 104857600 2014-06-09 16:22:11.604494 7f869af78700 5 -- op tracker -- , seq: 2063, time: 2014-06-09 16:22:11.604491, event: done, op: pg_notify(3.55b(10) epoch 243) 2014-06-09 16:22:11.609803 7f8690f64700 2 waiting 51 > 50 ops || 91498 > 104857600 2014-06-09 16:22:11.609806 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.609810 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.604491 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.609811 7f868ff62700 2 waiting 51 > 50 ops || 90824 > 104857600 2014-06-09 16:22:11.604501 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17001f00,0x2d739f40,0xbccd020,0xbd6d860] 2014-06-09 16:22:11.609818 7f868ef60700 2 waiting 51 > 50 ops || 91443 > 104857600 2014-06-09 16:22:11.609820 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.609825 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.604525 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3272.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.604509 7f86a5610700 10 _finish_op 0x308cba40 seq 3262 osr(0.578 0x2bd7800)/0x2bd7800 2014-06-09 16:22:11.609827 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.609827 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17001f00,0x2d739f40,0xbccd020,0xbd6d860] 2014-06-09 16:22:11.609831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.609831 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.609830 7f869af78700 20 osd.26 243 _dispatch 0x305f4400 pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.609832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.609836 7f869af78700 5 -- op tracker -- , seq: 2064, time: 2014-06-09 16:22:11.339799, event: header_read, op: pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.609842 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc168ce0] 2014-06-09 16:22:11.609841 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d05a0 (1 active) 2014-06-09 16:22:11.609845 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496d80 (2 active) 2014-06-09 16:22:11.609847 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc168ce0] 2014-06-09 16:22:11.609846 7f869af78700 5 -- op tracker -- , seq: 2064, time: 2014-06-09 16:22:11.339799, event: throttled, op: pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.609849 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.609849 7f86a5610700 5 _do_op 0x2fcae600 seq 3273 osr(2.699 0x2ed1f10)/0x2ed1f10 start 2014-06-09 16:22:11.609850 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.609838 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.609854 7f86a5610700 10 _do_transaction on 0x399ce40 2014-06-09 16:22:11.609852 7f869af78700 5 -- op tracker -- , seq: 2064, time: 2014-06-09 16:22:11.339812, event: all_read, op: pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.609861 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.604548 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.609873 7f868c75b700 2 waiting 51 > 50 ops || 91247 > 104857600 2014-06-09 16:22:11.609865 7f869af78700 5 -- op tracker -- , seq: 2064, time: 2014-06-09 16:22:11.609825, event: dispatched, op: pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.609879 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.609885 7f869af78700 5 -- op tracker -- , seq: 2064, time: 2014-06-09 16:22:11.609882, event: waiting_for_osdmap, op: pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.609890 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.609893 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f4400 2014-06-09 16:22:11.609891 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.609899 7f869af78700 5 -- op tracker -- , seq: 2064, time: 2014-06-09 16:22:11.609897, event: started, op: pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.609886 7f8692767700 10 osd.26 243 dequeue_op 0x270be700 prio 63 cost 516096 latency 0.841903 osd_op(client.4573.0:34087 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 212992~516096] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.609908 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.609910 7f868e75f700 10 op_submit_finish 3314 2014-06-09 16:22:11.609916 7f868bf5a700 10 op_submit_start 3315 2014-06-09 16:22:11.609918 7f868bf5a700 5 queue_transactions (trailing journal) 3315 2014-06-09 16:22:11.609919 7f86a5e11700 10 _do_op 0x306067e0 seq 3272 r = 0, finisher 0x2d738b00 0 2014-06-09 16:22:11.609920 7f868bf5a700 5 queue_op 0x308cb1a0 seq 3315 osr(2.101 0x2ed0730) 1883 bytes (queue has 50 ops and 89293 bytes) 2014-06-09 16:22:11.609922 7f86a5e11700 10 _finish_op 0x306067e0 seq 3272 osr(1.b5 0x384c690)/0x384c690 2014-06-09 16:22:11.609917 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.609928 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407560 (1 active) 2014-06-09 16:22:11.609930 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d738b00] 2014-06-09 16:22:11.609932 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1200 (2 active) 2014-06-09 16:22:11.609924 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 active] flushed 2014-06-09 16:22:11.609935 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d738b00] 2014-06-09 16:22:11.609935 7f86a5e11700 5 _do_op 0x30d975c0 seq 3274 osr(2.26a 0x2ed0dd0)/0x2ed0dd0 start 2014-06-09 16:22:11.609936 7f868bf5a700 10 op_submit_finish 3315 2014-06-09 16:22:11.609937 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.609938 7f86a5e11700 10 _do_transaction on 0xc1f2540 2014-06-09 16:22:11.609938 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.609931 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.5c6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.609944 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.609944 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.609939 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.609949 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.609941 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3e2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.3e2( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.609952 7f86a460e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.609953 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.137418 7 0.009788 2014-06-09 16:22:11.609954 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3e2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.609965 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.609965 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.609967 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.609971 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.6b4( v 213'576 (0'0,213'576] local-les=203 n=115 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.609981 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.609977 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.609980 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.609984 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.609994 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.609988 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.609995 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.609998 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3274.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610003 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.609997 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.609998 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=210 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.610007 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.610010 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=210 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.610012 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.610021 7f86a5e11700 10 _do_op 0x30d975c0 seq 3274 r = 0, finisher 0xbd74600 0 2014-06-09 16:22:11.610024 7f86a5e11700 10 _finish_op 0x30d975c0 seq 3274 osr(2.26a 0x2ed0dd0)/0x2ed0dd0 2014-06-09 16:22:11.610030 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1200 (1 active) 2014-06-09 16:22:11.610022 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=210 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.273241 7 0.000166 2014-06-09 16:22:11.610032 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3faea0 (2 active) 2014-06-09 16:22:11.610031 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd74600] 2014-06-09 16:22:11.610034 7f86a5e11700 5 _do_op 0x30d969c0 seq 3275 osr(2.216 0x2ed0b50)/0x2ed0b50 start 2014-06-09 16:22:11.610037 7f86a5e11700 10 _do_transaction on 0xc149980 2014-06-09 16:22:11.610027 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610036 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd74600] 2014-06-09 16:22:11.610038 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610039 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610041 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.610043 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610037 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=210 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.610048 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=210 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.610058 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610063 7f868e75f700 5 queue_transactions existing osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 2014-06-09 16:22:11.610056 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.610058 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=210 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.610067 7f868e75f700 10 op_submit_start 3316 2014-06-09 16:22:11.610068 7f868e75f700 5 queue_transactions (trailing journal) 3316 2014-06-09 16:22:11.610069 7f868e75f700 5 queue_op 0x30d97f20 seq 3316 osr(2.5c6 0x2ed1ad0) 2009 bytes (queue has 50 ops and 89493 bytes) 2014-06-09 16:22:11.610070 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x4eb8900,0xc169de0,0xbfcfe20,0xbef1a20,0xbebe1c0,0xbebf3a0,0xbd75e60,0xbd64340,0x33a7860,0xbd77480] 2014-06-09 16:22:11.610072 7f868e75f700 10 op_submit_finish 3316 2014-06-09 16:22:11.610073 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6aaa0,0x2d739b80,0xbebe5a0,0x1a003640] 2014-06-09 16:22:11.610068 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'580 -> 213'580 2014-06-09 16:22:11.610075 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610076 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610089 7f86a460e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.610091 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0290 (19 active) 2014-06-09 16:22:11.610091 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3275.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610095 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.610100 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.610111 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0290 (20 active) 2014-06-09 16:22:11.610107 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.610117 7f86a5e11700 10 _do_op 0x30d969c0 seq 3275 r = 0, finisher 0x230c18e0 0 2014-06-09 16:22:11.610121 7f86a5e11700 10 _finish_op 0x30d969c0 seq 3275 osr(2.216 0x2ed0b50)/0x2ed0b50 2014-06-09 16:22:11.610126 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3faea0 (1 active) 2014-06-09 16:22:11.610128 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0120 (2 active) 2014-06-09 16:22:11.610128 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x230c18e0] 2014-06-09 16:22:11.610117 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610130 7f86a5e11700 5 _do_op 0x2fcae5a0 seq 3276 osr(2.33a 0x2ed0ed0)/0x2ed0ed0 start 2014-06-09 16:22:11.610132 7f86a5e11700 10 _do_transaction on 0x3338b40 2014-06-09 16:22:11.610132 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x230c18e0] 2014-06-09 16:22:11.610134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610135 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610138 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610134 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.610146 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610148 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.610153 7f868bf5a700 5 queue_transactions existing osr(3.271 0x3ed0bc0)/0x3ed0bc0 2014-06-09 16:22:11.610156 7f868bf5a700 10 op_submit_start 3317 2014-06-09 16:22:11.610157 7f868bf5a700 5 queue_transactions (trailing journal) 3317 2014-06-09 16:22:11.610159 7f868bf5a700 5 queue_op 0x308cb140 seq 3317 osr(3.271 0x3ed0bc0) 1335 bytes (queue has 50 ops and 88827 bytes) 2014-06-09 16:22:11.610161 7f868bf5a700 10 op_submit_finish 3317 2014-06-09 16:22:11.610152 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610159 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610170 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610165 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610164 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.610175 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610174 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.610175 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 1.18a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.610185 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3276.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610188 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.610187 7f868bf5a700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.610179 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610190 7f868bf5a700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.610189 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.610194 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] exit Started/Stray 0.764626 7 0.000172 2014-06-09 16:22:11.610193 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.610205 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610204 7f86a5e11700 10 _do_op 0x2fcae5a0 seq 3276 r = 0, finisher 0xcd9e940 0 2014-06-09 16:22:11.610207 7f86a5e11700 10 _finish_op 0x2fcae5a0 seq 3276 osr(2.33a 0x2ed0ed0)/0x2ed0ed0 2014-06-09 16:22:11.610203 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.610204 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610212 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0120 (1 active) 2014-06-09 16:22:11.610214 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62fc20 (2 active) 2014-06-09 16:22:11.610214 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd9e940] 2014-06-09 16:22:11.610209 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610216 7f86a5e11700 5 _do_op 0x309a21e0 seq 3277 osr(1.c9 0x384c9f0)/0x384c9f0 start 2014-06-09 16:22:11.610219 7f86a5e11700 10 _do_transaction on 0xc2cb440 2014-06-09 16:22:11.610219 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd9e940] 2014-06-09 16:22:11.610214 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.610221 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610219 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.610222 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610224 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610220 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610229 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610223 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.610232 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.610237 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.610240 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.610239 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610257 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610250 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610254 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.610265 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610265 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.610272 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3277.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610276 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.610269 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610282 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.610277 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.610280 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610286 7f868bf5a700 5 queue_transactions existing osr(1.18a 0x384cb50)/0x384cb50 2014-06-09 16:22:11.610289 7f868bf5a700 10 op_submit_start 3318 2014-06-09 16:22:11.610289 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610290 7f868bf5a700 5 queue_transactions (trailing journal) 3318 2014-06-09 16:22:11.610289 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6aaa0,0x2d739b80,0xbebe5a0,0x1a003640] 2014-06-09 16:22:11.610291 7f868bf5a700 5 queue_op 0x308cb0e0 seq 3318 osr(1.18a 0x384cb50) 1746 bytes (queue has 50 ops and 88690 bytes) 2014-06-09 16:22:11.610292 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189abfc0,0x17001980] 2014-06-09 16:22:11.610294 7f868bf5a700 10 op_submit_finish 3318 2014-06-09 16:22:11.610299 7f86a5e11700 10 _do_op 0x309a21e0 seq 3277 r = 0, finisher 0xbd76740 0 2014-06-09 16:22:11.610295 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610302 7f86a5e11700 10 _finish_op 0x309a21e0 seq 3277 osr(1.c9 0x384c9f0)/0x384c9f0 2014-06-09 16:22:11.610296 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.75f( v 213'591 (0'0,213'591] local-les=191 n=113 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.610309 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62fc20 (1 active) 2014-06-09 16:22:11.610309 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.610310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76740] 2014-06-09 16:22:11.610311 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b7a0 (2 active) 2014-06-09 16:22:11.610293 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610314 7f86a5e11700 5 _do_op 0x30d97560 seq 3278 osr(3.461 0x3ed1120)/0x3ed1120 start 2014-06-09 16:22:11.610315 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76740] 2014-06-09 16:22:11.610316 7f86a5e11700 10 _do_transaction on 0xbf0fbc0 2014-06-09 16:22:11.610317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610318 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610311 7f868bf5a700 7 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.1 2014-06-09 16:22:11.610322 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610315 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x11e63900,0x11e63800,0x14053f00,0x14053e00,0x14053d00,0x14053c00,0x14053b00 2014-06-09 16:22:11.610321 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.610321 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.1 info 3.75f( v 213'591 (0'0,213'591] local-les=191 n=113 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.610335 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610343 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.610348 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610349 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.610354 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610348 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:1.764( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.610352 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610360 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.610360 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.610371 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.610369 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3278.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610365 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610373 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.610374 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189abfc0,0x17001980] 2014-06-09 16:22:11.610376 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610376 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1b0c1520] 2014-06-09 16:22:11.610375 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.610379 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610380 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.74( v 213'545 (0'0,213'545] local-les=191 n=109 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.610391 7f86a5e11700 10 _do_op 0x30d97560 seq 3278 r = 0, finisher 0xbce7580 0 2014-06-09 16:22:11.610386 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.610394 7f86a5e11700 10 _finish_op 0x30d97560 seq 3278 osr(3.461 0x3ed1120)/0x3ed1120 2014-06-09 16:22:11.610400 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b7a0 (1 active) 2014-06-09 16:22:11.610395 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.610400 7f86a460e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.610392 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.610402 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34977a0 (2 active) 2014-06-09 16:22:11.610402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbce7580] 2014-06-09 16:22:11.610404 7f86a5e11700 5 _do_op 0x2fcae4e0 seq 3279 osr(0.631 0x2bd7c70)/0x2bd7c70 start 2014-06-09 16:22:11.610406 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610406 7f86a5e11700 10 _do_transaction on 0xc2c78c0 2014-06-09 16:22:11.610407 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbce7580] 2014-06-09 16:22:11.610402 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.610408 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610410 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610412 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610414 7f868bf5a700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.610410 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610413 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.610426 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1b0c1520] 2014-06-09 16:22:11.610428 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x4eb9b40] 2014-06-09 16:22:11.610422 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610432 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610429 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.610430 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610443 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.610444 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610444 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.610436 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610448 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=212 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) 2014-06-09 16:22:11.610451 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.610459 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3279.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610462 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610463 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.610459 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=212 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) 2014-06-09 16:22:11.610460 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.610466 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610475 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x4eb9b40] 2014-06-09 16:22:11.610477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.610479 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.610471 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=212 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.833922 7 0.000188 2014-06-09 16:22:11.610478 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610484 7f86a5e11700 10 _do_op 0x2fcae4e0 seq 3279 r = 0, finisher 0xbd7afe0 0 2014-06-09 16:22:11.610488 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610490 7f86a5e11700 10 _finish_op 0x2fcae4e0 seq 3279 osr(0.631 0x2bd7c70)/0x2bd7c70 2014-06-09 16:22:11.610483 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=212 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.610495 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34977a0 (1 active) 2014-06-09 16:22:11.610497 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4078c0 (2 active) 2014-06-09 16:22:11.610496 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7afe0] 2014-06-09 16:22:11.610499 7f86a5e11700 5 _do_op 0x2fc8c0c0 seq 3280 osr(3.218 0x3ed0cd0)/0x3ed0cd0 start 2014-06-09 16:22:11.610492 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610501 7f86a5e11700 10 _do_transaction on 0xc30ce40 2014-06-09 16:22:11.610501 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7afe0] 2014-06-09 16:22:11.610495 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=212 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.610502 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610503 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610506 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610503 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.122( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610512 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610505 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=212 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.610516 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610520 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'577 -> 213'577 2014-06-09 16:22:11.610528 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610537 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610537 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610541 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610552 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3280.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610556 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.610552 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610561 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610559 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.610565 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610581 7f86a5e11700 10 _do_op 0x2fc8c0c0 seq 3280 r = 0, finisher 0xbd69400 0 2014-06-09 16:22:11.610585 7f86a5e11700 10 _finish_op 0x2fc8c0c0 seq 3280 osr(3.218 0x3ed0cd0)/0x3ed0cd0 2014-06-09 16:22:11.610578 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.610590 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610591 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4078c0 (1 active) 2014-06-09 16:22:11.610592 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd69400] 2014-06-09 16:22:11.610593 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653440 (2 active) 2014-06-09 16:22:11.610595 7f86a5e11700 5 _do_op 0x2fc8c060 seq 3281 osr(1.113 0x384c910)/0x384c910 start 2014-06-09 16:22:11.610597 7f86a5e11700 10 _do_transaction on 0x39da900 2014-06-09 16:22:11.610598 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd69400] 2014-06-09 16:22:11.610599 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610600 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610593 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610603 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610603 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610610 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.610612 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610615 7f868bf5a700 5 queue_transactions existing osr(3.5f8 0x3ed1b30)/0x3ed1b30 2014-06-09 16:22:11.610618 7f868bf5a700 10 op_submit_start 3319 2014-06-09 16:22:11.610619 7f868bf5a700 5 queue_transactions (trailing journal) 3319 2014-06-09 16:22:11.610620 7f868bf5a700 5 queue_op 0x308cb020 seq 3319 osr(3.5f8 0x3ed1b30) 1335 bytes (queue has 47 ops and 83348 bytes) 2014-06-09 16:22:11.610623 7f868bf5a700 10 op_submit_finish 3319 2014-06-09 16:22:11.610616 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610627 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610634 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610633 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062b1e0,0xc40ca20] 2014-06-09 16:22:11.610637 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610625 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.610638 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610641 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610649 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3281.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610653 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.610644 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.610653 7f86a460e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.610653 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.463( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610663 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.610657 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.777663 7 0.000184 2014-06-09 16:22:11.610667 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.610674 7f8691f66700 10 op_submit_start 3320 2014-06-09 16:22:11.610673 7f86a5e11700 10 _do_op 0x2fc8c060 seq 3281 r = 0, finisher 0x3050bde0 0 2014-06-09 16:22:11.610675 7f8691f66700 5 queue_transactions (trailing journal) 3320 2014-06-09 16:22:11.610676 7f86a5e11700 10 _finish_op 0x2fc8c060 seq 3281 osr(1.113 0x384c910)/0x384c910 2014-06-09 16:22:11.610671 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.610677 7f8691f66700 5 queue_op 0x308cb920 seq 3320 osr(1.3d6 0x384d470) 1883 bytes (queue has 48 ops and 85231 bytes) 2014-06-09 16:22:11.610681 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653440 (1 active) 2014-06-09 16:22:11.610683 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b680 (2 active) 2014-06-09 16:22:11.610683 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050bde0] 2014-06-09 16:22:11.610677 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.610686 7f86a5e11700 5 _do_op 0x2fc8c000 seq 3282 osr(0.2d9 0x2bd6e90)/0x2bd6e90 start 2014-06-09 16:22:11.610688 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050bde0] 2014-06-09 16:22:11.610689 7f86a5e11700 10 _do_transaction on 0xbe8b740 2014-06-09 16:22:11.610689 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610694 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610687 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=212 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.610687 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610697 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'625 -> 213'625 2014-06-09 16:22:11.610716 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610734 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.610735 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3273.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610739 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.609911 7f8692767700 5 -- op tracker -- , seq: 707, time: 2014-06-09 16:22:11.609906, event: reached_pg, op: osd_op(client.4573.0:34087 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 212992~516096] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.610740 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.610750 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.610751 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.610766 7f86a5610700 10 _do_op 0x2fcae600 seq 3273 r = 0, finisher 0xbd72160 0 2014-06-09 16:22:11.610767 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062b1e0,0xc40ca20] 2014-06-09 16:22:11.610770 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30508740,0xbdf1940] 2014-06-09 16:22:11.610770 7f86a5610700 10 _finish_op 0x2fcae600 seq 3273 osr(2.699 0x2ed1f10)/0x2ed1f10 2014-06-09 16:22:11.610776 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496d80 (1 active) 2014-06-09 16:22:11.610778 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.610769 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x270be700 2014-06-09 16:22:11.610778 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd72160] 2014-06-09 16:22:11.610779 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc923b00 (2 active) 2014-06-09 16:22:11.610780 7f8692767700 10 osd.26 243 dequeue_op 0x270be700 finish 2014-06-09 16:22:11.610772 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610781 7f86a5610700 5 _do_op 0x30d97500 seq 3283 osr(3.468 0x3ed1590)/0x3ed1590 start 2014-06-09 16:22:11.610783 7f86a5610700 10 _do_transaction on 0x3562c00 2014-06-09 16:22:11.610783 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd72160] 2014-06-09 16:22:11.610783 7f868bf5a700 5 queue_transactions existing osr(3.400 0x3ed1260)/0x3ed1260 2014-06-09 16:22:11.610784 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.610785 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610786 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610786 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.610789 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610790 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.610801 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.610794 7f8692767700 10 osd.26 243 dequeue_op 0x11e63400 prio 63 cost 172032 latency 1.456936 osd_op(client.4575.0:31987 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.610817 7f8692767700 5 -- op tracker -- , seq: 143, time: 2014-06-09 16:22:11.610811, event: reached_pg, op: osd_op(client.4575.0:31987 rbd_data.117f6b8b4567.000000000000150d [set-alloc-hint object_size 4194304 write_size 4194304,write 4022272~172032] 3.b6bb6e7d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.610814 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610823 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610827 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.610825 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.610840 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3283.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610844 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.610837 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x11e63400 2014-06-09 16:22:11.610847 7f8692767700 10 osd.26 243 dequeue_op 0x11e63400 finish 2014-06-09 16:22:11.610840 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] flushed 2014-06-09 16:22:11.610850 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.610849 7f869af78700 5 -- op tracker -- , seq: 2064, time: 2014-06-09 16:22:11.609911, event: done, op: pg_log(1.1fa epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.610859 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.610861 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.610860 7f86a5610700 10 _do_op 0x30d97500 seq 3283 r = 0, finisher 0xc6e9b80 0 2014-06-09 16:22:11.610864 7f86a5610700 10 _finish_op 0x30d97500 seq 3283 osr(3.468 0x3ed1590)/0x3ed1590 2014-06-09 16:22:11.610870 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc923b00 (1 active) 2014-06-09 16:22:11.610872 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e9b80] 2014-06-09 16:22:11.610872 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359320 (2 active) 2014-06-09 16:22:11.610872 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (20 active) 2014-06-09 16:22:11.610874 7f86a5610700 5 _do_op 0x30d97e60 seq 3284 osr(2.c6 0x2ed0490)/0x2ed0490 start 2014-06-09 16:22:11.610877 7f86a5610700 10 _do_transaction on 0xc2c4cc0 2014-06-09 16:22:11.610877 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e9b80] 2014-06-09 16:22:11.610878 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610879 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610881 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.610883 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.610882 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610884 7f869af78700 20 osd.26 243 _dispatch 0x307e6c40 pg_notify(3.5(10),3.4d1(10) epoch 243) v5 2014-06-09 16:22:11.610879 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.37e( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.610890 7f869af78700 5 -- op tracker -- , seq: 2065, time: 2014-06-09 16:22:11.331827, event: header_read, op: pg_notify(3.5(10),3.4d1(10) epoch 243) 2014-06-09 16:22:11.610891 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.610901 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.610899 7f869af78700 5 -- op tracker -- , seq: 2065, time: 2014-06-09 16:22:11.331828, event: throttled, op: pg_notify(3.5(10),3.4d1(10) epoch 243) 2014-06-09 16:22:11.610905 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:0.b7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.610909 7f869af78700 5 -- op tracker -- , seq: 2065, time: 2014-06-09 16:22:11.331859, event: all_read, op: pg_notify(3.5(10),3.4d1(10) epoch 243) 2014-06-09 16:22:11.610915 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.610917 7f869af78700 5 -- op tracker -- , seq: 2065, time: 2014-06-09 16:22:11.610881, event: dispatched, op: pg_notify(3.5(10),3.4d1(10) epoch 243) 2014-06-09 16:22:11.610916 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.610925 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.610924 7f869af78700 5 -- op tracker -- , seq: 2065, time: 2014-06-09 16:22:11.610922, event: waiting_for_osdmap, op: pg_notify(3.5(10),3.4d1(10) epoch 243) 2014-06-09 16:22:11.610929 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.610931 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e6c40 2014-06-09 16:22:11.610933 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3284.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.610929 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610937 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.610935 7f869af78700 5 -- op tracker -- , seq: 2065, time: 2014-06-09 16:22:11.610933, event: started, op: pg_notify(3.5(10),3.4d1(10) epoch 243) 2014-06-09 16:22:11.610940 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610950 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.610948 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30508740,0xbdf1940] 2014-06-09 16:22:11.610952 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050a320] 2014-06-09 16:22:11.610953 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610960 7f869af78700 5 -- op tracker -- , seq: 2065, time: 2014-06-09 16:22:11.610958, event: done, op: pg_notify(3.5(10),3.4d1(10) epoch 243) 2014-06-09 16:22:11.610961 7f86a5610700 10 _do_op 0x30d97e60 seq 3284 r = 0, finisher 0xbd5a100 0 2014-06-09 16:22:11.610954 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.610965 7f86a5610700 10 _finish_op 0x30d97e60 seq 3284 osr(2.c6 0x2ed0490)/0x2ed0490 2014-06-09 16:22:11.610968 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.610969 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.610963 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610970 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.610972 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.610971 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359320 (1 active) 2014-06-09 16:22:11.610973 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5a100] 2014-06-09 16:22:11.610974 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5560 (2 active) 2014-06-09 16:22:11.610975 7f86a5610700 5 _do_op 0x30d97e00 seq 3285 osr(3.4fe 0x3ed1730)/0x3ed1730 start 2014-06-09 16:22:11.610978 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5a100] 2014-06-09 16:22:11.610978 7f86a5610700 10 _do_transaction on 0xbf27a40 2014-06-09 16:22:11.610979 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.610980 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.610975 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.610984 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.610986 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.610988 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.610989 7f869af78700 20 osd.26 243 _dispatch 0x306d2600 pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.610986 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.610994 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.610984 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.610995 7f869af78700 5 -- op tracker -- , seq: 2066, time: 2014-06-09 16:22:11.244906, event: header_read, op: pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.611000 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050a320] 2014-06-09 16:22:11.611002 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7dd20] 2014-06-09 16:22:11.610998 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611003 7f869af78700 5 -- op tracker -- , seq: 2066, time: 2014-06-09 16:22:11.244906, event: throttled, op: pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.611004 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.611016 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.611016 7f869af78700 5 -- op tracker -- , seq: 2066, time: 2014-06-09 16:22:11.244918, event: all_read, op: pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.611012 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.611018 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611023 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611022 7f869af78700 5 -- op tracker -- , seq: 2066, time: 2014-06-09 16:22:11.610986, event: dispatched, op: pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.611032 7f869af78700 5 -- op tracker -- , seq: 2066, time: 2014-06-09 16:22:11.611029, event: waiting_for_osdmap, op: pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.611026 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611030 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.611036 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.611036 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3285.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611040 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306d2600 2014-06-09 16:22:11.611041 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.611044 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7dd20] 2014-06-09 16:22:11.611038 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.611046 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.611048 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.611049 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611045 7f869af78700 5 -- op tracker -- , seq: 2066, time: 2014-06-09 16:22:11.611043, event: started, op: pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.611053 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611059 7f86a5610700 10 _do_op 0x30d97e00 seq 3285 r = 0, finisher 0xc40cb00 0 2014-06-09 16:22:11.611065 7f86a5610700 10 _finish_op 0x30d97e00 seq 3285 osr(3.4fe 0x3ed1730)/0x3ed1730 2014-06-09 16:22:11.611063 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.182( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.611062 7f869af78700 5 -- op tracker -- , seq: 2066, time: 2014-06-09 16:22:11.611060, event: done, op: pg_log(2.4 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.611070 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5560 (1 active) 2014-06-09 16:22:11.611072 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611072 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc40cb00] 2014-06-09 16:22:11.611072 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a7e0 (2 active) 2014-06-09 16:22:11.611074 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611074 7f86a5610700 5 _do_op 0x2fcae420 seq 3286 osr(3.778 0x3ed1df0)/0x3ed1df0 start 2014-06-09 16:22:11.611076 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611077 7f86a5610700 10 _do_transaction on 0xc2ee900 2014-06-09 16:22:11.611077 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc40cb00] 2014-06-09 16:22:11.611078 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611080 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611083 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611076 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.611087 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611089 7f869af78700 20 osd.26 243 _dispatch 0x3088cfc0 pg_info(1 pgs e243:3.375(2)) v4 2014-06-09 16:22:11.611089 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a7240] 2014-06-09 16:22:11.611088 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.611094 7f869af78700 5 -- op tracker -- , seq: 2067, time: 2014-06-09 16:22:11.314351, event: header_read, op: pg_info(1 pgs e243:3.375(2)) 2014-06-09 16:22:11.611096 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.611101 7f869af78700 5 -- op tracker -- , seq: 2067, time: 2014-06-09 16:22:11.314351, event: throttled, op: pg_info(1 pgs e243:3.375(2)) 2014-06-09 16:22:11.611094 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.611103 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.611107 7f869af78700 5 -- op tracker -- , seq: 2067, time: 2014-06-09 16:22:11.314361, event: all_read, op: pg_info(1 pgs e243:3.375(2)) 2014-06-09 16:22:11.611115 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.611113 7f869af78700 5 -- op tracker -- , seq: 2067, time: 2014-06-09 16:22:11.611086, event: dispatched, op: pg_info(1 pgs e243:3.375(2)) 2014-06-09 16:22:11.611118 7f86a460e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.611120 7f869af78700 5 -- op tracker -- , seq: 2067, time: 2014-06-09 16:22:11.611118, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.375(2)) 2014-06-09 16:22:11.611127 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:179 2014-06-09 16:22:11.611140 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611135 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.611151 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a7240] 2014-06-09 16:22:11.611130 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.375(2)) v4 from osd.9 2014-06-09 16:22:11.611144 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611153 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.611154 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.611154 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088cfc0 2014-06-09 16:22:11.611156 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.611165 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611162 7f869af78700 5 -- op tracker -- , seq: 2067, time: 2014-06-09 16:22:11.611159, event: started, op: pg_info(1 pgs e243:3.375(2)) 2014-06-09 16:22:11.611168 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.18e( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.611175 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611183 7f869af78700 5 -- op tracker -- , seq: 2067, time: 2014-06-09 16:22:11.611180, event: done, op: pg_info(1 pgs e243:3.375(2)) 2014-06-09 16:22:11.611180 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.611190 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611190 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611192 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611192 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3286.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611197 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.611193 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.523( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.611205 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.611213 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611213 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611214 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611215 7f869af78700 20 osd.26 243 _dispatch 0x307e7500 pg_notify(0.164(23) epoch 243) v5 2014-06-09 16:22:11.611221 7f86a5610700 10 _do_op 0x2fcae420 seq 3286 r = 0, finisher 0x139657c0 0 2014-06-09 16:22:11.611220 7f869af78700 5 -- op tracker -- , seq: 2068, time: 2014-06-09 16:22:11.349039, event: header_read, op: pg_notify(0.164(23) epoch 243) 2014-06-09 16:22:11.611217 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:1.25a( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.611228 7f86a5610700 10 _finish_op 0x2fcae420 seq 3286 osr(3.778 0x3ed1df0)/0x3ed1df0 2014-06-09 16:22:11.611227 7f869af78700 5 -- op tracker -- , seq: 2068, time: 2014-06-09 16:22:11.349040, event: throttled, op: pg_notify(0.164(23) epoch 243) 2014-06-09 16:22:11.611227 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.611233 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a7e0 (1 active) 2014-06-09 16:22:11.611232 7f869af78700 5 -- op tracker -- , seq: 2068, time: 2014-06-09 16:22:11.349073, event: all_read, op: pg_notify(0.164(23) epoch 243) 2014-06-09 16:22:11.611235 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9227e0 (2 active) 2014-06-09 16:22:11.611235 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x139657c0] 2014-06-09 16:22:11.611236 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611237 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd693e0] 2014-06-09 16:22:11.611237 7f86a5610700 5 _do_op 0x30d97da0 seq 3287 osr(0.4a7 0x2bd7370)/0x2bd7370 start 2014-06-09 16:22:11.611240 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x139657c0] 2014-06-09 16:22:11.611241 7f86a5610700 10 _do_transaction on 0xbf47440 2014-06-09 16:22:11.611238 7f869af78700 5 -- op tracker -- , seq: 2068, time: 2014-06-09 16:22:11.611212, event: dispatched, op: pg_notify(0.164(23) epoch 243) 2014-06-09 16:22:11.611242 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611243 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611245 7f869af78700 5 -- op tracker -- , seq: 2068, time: 2014-06-09 16:22:11.611243, event: waiting_for_osdmap, op: pg_notify(0.164(23) epoch 243) 2014-06-09 16:22:11.611247 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611239 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.266( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.611249 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.611251 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e7500 2014-06-09 16:22:11.611241 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.611251 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.611256 7f869af78700 5 -- op tracker -- , seq: 2068, time: 2014-06-09 16:22:11.611254, event: started, op: pg_notify(0.164(23) epoch 243) 2014-06-09 16:22:11.611258 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.611260 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611263 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.25c( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.611270 7f869af78700 5 -- op tracker -- , seq: 2068, time: 2014-06-09 16:22:11.611268, event: done, op: pg_notify(0.164(23) epoch 243) 2014-06-09 16:22:11.611280 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611273 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.611281 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611280 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611282 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611273 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.611288 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd693e0] 2014-06-09 16:22:11.611290 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.611291 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.611286 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611298 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611299 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611297 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3287.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611300 7f869af78700 20 osd.26 243 _dispatch 0x1b0ce700 pg_notify(2.4ba(20),3.1f4(10) epoch 243) v5 2014-06-09 16:22:11.611301 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.611296 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.759( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.611305 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611305 7f869af78700 5 -- op tracker -- , seq: 2069, time: 2014-06-09 16:22:11.347808, event: header_read, op: pg_notify(2.4ba(20),3.1f4(10) epoch 243) 2014-06-09 16:22:11.611313 7f869af78700 5 -- op tracker -- , seq: 2069, time: 2014-06-09 16:22:11.347808, event: throttled, op: pg_notify(2.4ba(20),3.1f4(10) epoch 243) 2014-06-09 16:22:11.611308 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611319 7f869af78700 5 -- op tracker -- , seq: 2069, time: 2014-06-09 16:22:11.347850, event: all_read, op: pg_notify(2.4ba(20),3.1f4(10) epoch 243) 2014-06-09 16:22:11.611323 7f86a5610700 10 _do_op 0x30d97da0 seq 3287 r = 0, finisher 0xc5a6ea0 0 2014-06-09 16:22:11.611320 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.611325 7f869af78700 5 -- op tracker -- , seq: 2069, time: 2014-06-09 16:22:11.611297, event: dispatched, op: pg_notify(2.4ba(20),3.1f4(10) epoch 243) 2014-06-09 16:22:11.611330 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611330 7f86a5610700 10 _finish_op 0x30d97da0 seq 3287 osr(0.4a7 0x2bd7370)/0x2bd7370 2014-06-09 16:22:11.611332 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6f280] 2014-06-09 16:22:11.611332 7f869af78700 5 -- op tracker -- , seq: 2069, time: 2014-06-09 16:22:11.611330, event: waiting_for_osdmap, op: pg_notify(2.4ba(20),3.1f4(10) epoch 243) 2014-06-09 16:22:11.611334 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9227e0 (1 active) 2014-06-09 16:22:11.611336 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6f280] 2014-06-09 16:22:11.611336 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.611336 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a6ea0] 2014-06-09 16:22:11.611337 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd0e0 (2 active) 2014-06-09 16:22:11.611337 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.611339 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.611338 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1b0ce700 2014-06-09 16:22:11.611339 7f86a5610700 5 _do_op 0x2fcae3c0 seq 3288 osr(3.1fc 0x3ed0930)/0x3ed0930 start 2014-06-09 16:22:11.611342 7f86a5610700 10 _do_transaction on 0x3014900 2014-06-09 16:22:11.611344 7f869af78700 5 -- op tracker -- , seq: 2069, time: 2014-06-09 16:22:11.611341, event: started, op: pg_notify(2.4ba(20),3.1f4(10) epoch 243) 2014-06-09 16:22:11.611333 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611347 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611340 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.611350 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.611359 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611360 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.611363 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.55b( v 214'716 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.611374 7f869af78700 5 -- op tracker -- , seq: 2069, time: 2014-06-09 16:22:11.611365, event: done, op: pg_notify(2.4ba(20),3.1f4(10) epoch 243) 2014-06-09 16:22:11.611379 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611382 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611384 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611376 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.4 2014-06-09 16:22:11.611395 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3288.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611390 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.611399 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.611386 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.4 info 3.55b( v 214'716 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242), identical to ours 2014-06-09 16:22:11.611403 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611404 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611406 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611405 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a6ea0] 2014-06-09 16:22:11.611408 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611408 7f869af78700 20 osd.26 243 _dispatch 0x308e3340 pg_info(1 pgs e243:2.4e3) v4 2014-06-09 16:22:11.611409 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611407 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611412 7f869af78700 5 -- op tracker -- , seq: 2070, time: 2014-06-09 16:22:11.259448, event: header_read, op: pg_info(1 pgs e243:2.4e3) 2014-06-09 16:22:11.611416 7f86a5610700 10 _do_op 0x2fcae3c0 seq 3288 r = 0, finisher 0xbccdb80 0 2014-06-09 16:22:11.611419 7f869af78700 5 -- op tracker -- , seq: 2070, time: 2014-06-09 16:22:11.259449, event: throttled, op: pg_info(1 pgs e243:2.4e3) 2014-06-09 16:22:11.611424 7f86a5610700 10 _finish_op 0x2fcae3c0 seq 3288 osr(3.1fc 0x3ed0930)/0x3ed0930 2014-06-09 16:22:11.611419 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.611425 7f869af78700 5 -- op tracker -- , seq: 2070, time: 2014-06-09 16:22:11.259464, event: all_read, op: pg_info(1 pgs e243:2.4e3) 2014-06-09 16:22:11.611429 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.611425 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbef1580] 2014-06-09 16:22:11.611429 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd0e0 (1 active) 2014-06-09 16:22:11.611430 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbccdb80] 2014-06-09 16:22:11.611431 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b680 (2 active) 2014-06-09 16:22:11.611432 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbef1580] 2014-06-09 16:22:11.611433 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.611433 7f86a5610700 5 _do_op 0x30032e40 seq 3289 osr(1.198 0x384cb30)/0x384cb30 start 2014-06-09 16:22:11.611435 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.611436 7f86a5610700 10 _do_transaction on 0xc20fec0 2014-06-09 16:22:11.611440 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (19 active) 2014-06-09 16:22:11.611441 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611433 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.611446 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.611447 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.611450 7f8692767700 10 osd.26 243 dequeue_op 0x11e63900 prio 63 cost 671744 latency 1.431817 osd_op(client.4566.0:33819 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~671744] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.611462 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.611472 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611478 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbccdb80] 2014-06-09 16:22:11.611479 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611480 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611487 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3289.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611491 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.611431 7f869af78700 5 -- op tracker -- , seq: 2070, time: 2014-06-09 16:22:11.611404, event: dispatched, op: pg_info(1 pgs e243:2.4e3) 2014-06-09 16:22:11.611500 7f869af78700 5 -- op tracker -- , seq: 2070, time: 2014-06-09 16:22:11.611496, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4e3) 2014-06-09 16:22:11.611505 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4e3) v4 from osd.0 2014-06-09 16:22:11.611508 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e3340 2014-06-09 16:22:11.611513 7f869af78700 5 -- op tracker -- , seq: 2070, time: 2014-06-09 16:22:11.611511, event: started, op: pg_info(1 pgs e243:2.4e3) 2014-06-09 16:22:11.611516 7f86a5610700 10 _do_op 0x30032e40 seq 3289 r = 0, finisher 0xbd7a980 0 2014-06-09 16:22:11.611523 7f86a5610700 10 _finish_op 0x30032e40 seq 3289 osr(1.198 0x384cb30)/0x384cb30 2014-06-09 16:22:11.611524 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbef0060] 2014-06-09 16:22:11.611528 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbef0060] 2014-06-09 16:22:11.611529 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.611528 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b680 (1 active) 2014-06-09 16:22:11.611530 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.611530 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7a980] 2014-06-09 16:22:11.611530 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca49b00 (2 active) 2014-06-09 16:22:11.611528 7f869af78700 5 -- op tracker -- , seq: 2070, time: 2014-06-09 16:22:11.611526, event: done, op: pg_info(1 pgs e243:2.4e3) 2014-06-09 16:22:11.611532 7f86a5610700 5 _do_op 0x30d97d40 seq 3290 osr(2.13c 0x2ed0610)/0x2ed0610 start 2014-06-09 16:22:11.611535 7f86a5610700 10 _do_transaction on 0x390c6c0 2014-06-09 16:22:11.611537 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.611540 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611533 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.611545 7f8691f66700 10 op_submit_finish 3320 2014-06-09 16:22:11.611546 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.611546 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.611551 7f868a757700 10 op_submit_start 3321 2014-06-09 16:22:11.611553 7f868a757700 5 queue_transactions (trailing journal) 3321 2014-06-09 16:22:11.611554 7f868a757700 5 queue_op 0x306066c0 seq 3321 osr(1.1ef 0x384cdb0) 2001 bytes (queue has 43 ops and 76738 bytes) 2014-06-09 16:22:11.611558 7f868a757700 10 op_submit_finish 3321 2014-06-09 16:22:11.611557 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611552 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.611560 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.611572 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7a980] 2014-06-09 16:22:11.611563 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.8c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.611562 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 1 notify: (query_epoch:242, epoch_sent:243, info:1.757( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.611573 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611574 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611574 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.611574 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3282.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611578 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.611578 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.611576 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.611586 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.611582 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.380828 7 0.000171 2014-06-09 16:22:11.611591 7f868af58700 10 op_submit_start 3322 2014-06-09 16:22:11.611593 7f868af58700 5 queue_transactions (trailing journal) 3322 2014-06-09 16:22:11.611593 7f86a5e11700 10 _do_op 0x2fc8c000 seq 3282 r = 0, finisher 0xbfcfbe0 0 2014-06-09 16:22:11.611595 7f868af58700 5 queue_op 0x30d97ce0 seq 3322 osr(0.117 0x2bd66a0) 1883 bytes (queue has 43 ops and 76738 bytes) 2014-06-09 16:22:11.611597 7f868af58700 10 op_submit_finish 3322 2014-06-09 16:22:11.611598 7f86a5e11700 10 _finish_op 0x2fc8c000 seq 3282 osr(0.2d9 0x2bd6e90)/0x2bd6e90 2014-06-09 16:22:11.611593 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.611602 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b680 (1 active) 2014-06-09 16:22:11.611604 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301e60 (2 active) 2014-06-09 16:22:11.611605 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfcfbe0] 2014-06-09 16:22:11.611606 7f86a5e11700 5 _do_op 0x2fcaf980 seq 3291 osr(1.206 0x384cd10)/0x384cd10 start 2014-06-09 16:22:11.611609 7f86a5e11700 10 _do_transaction on 0xbe68c00 2014-06-09 16:22:11.611610 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfcfbe0] 2014-06-09 16:22:11.611602 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.611605 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.611611 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611613 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611620 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611624 7f8692f68700 10 op_submit_start 3323 2014-06-09 16:22:11.611625 7f8692f68700 5 queue_transactions (trailing journal) 3323 2014-06-09 16:22:11.611619 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.611627 7f8692f68700 5 queue_op 0x309a3500 seq 3323 osr(2.48d 0x2ed1470) 1883 bytes (queue has 42 ops and 74764 bytes) 2014-06-09 16:22:11.611630 7f8692f68700 10 op_submit_finish 3323 2014-06-09 16:22:11.611629 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.611638 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.611635 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.611652 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611655 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.611668 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3291.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611672 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.611665 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.611674 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.611689 7f8691765700 10 op_submit_start 3324 2014-06-09 16:22:11.611690 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.611690 7f8691765700 5 queue_transactions (trailing journal) 3324 2014-06-09 16:22:11.611692 7f8691765700 5 queue_op 0x30060300 seq 3324 osr(0.61f 0x2bd7d30) 1974 bytes (queue has 42 ops and 74764 bytes) 2014-06-09 16:22:11.611695 7f8691765700 10 op_submit_finish 3324 2014-06-09 16:22:11.611699 7f86a5e11700 10 _do_op 0x2fcaf980 seq 3291 r = 0, finisher 0xbd6b3e0 0 2014-06-09 16:22:11.611704 7f86a5e11700 10 _finish_op 0x2fcaf980 seq 3291 osr(1.206 0x384cd10)/0x384cd10 2014-06-09 16:22:11.611704 7f8689f56700 10 op_submit_start 3325 2014-06-09 16:22:11.611707 7f8689f56700 5 queue_transactions (trailing journal) 3325 2014-06-09 16:22:11.611709 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301e60 (1 active) 2014-06-09 16:22:11.611708 7f8689f56700 5 queue_op 0x30d97440 seq 3325 osr(3.761 0x3ed1e10) 1429 bytes (queue has 43 ops and 76075 bytes) 2014-06-09 16:22:11.611711 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c05a0 (2 active) 2014-06-09 16:22:11.611710 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6b3e0] 2014-06-09 16:22:11.611715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6b3e0] 2014-06-09 16:22:11.611714 7f86a5e11700 5 _do_op 0x309a35c0 seq 3292 osr(1.48e 0x384d9b0)/0x384d9b0 start 2014-06-09 16:22:11.611706 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.71b( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.611716 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611717 7f86a5e11700 10 _do_transaction on 0x399de00 2014-06-09 16:22:11.611717 7f8689f56700 10 op_submit_finish 3325 2014-06-09 16:22:11.611718 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611723 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611719 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.611729 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.611722 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.611734 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.611738 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.611746 7f868af58700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.611750 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.611750 7f868af58700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.611755 7f868c75b700 10 op_submit_start 3326 2014-06-09 16:22:11.611755 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611756 7f868c75b700 5 queue_transactions (trailing journal) 3326 2014-06-09 16:22:11.611758 7f868c75b700 5 queue_op 0x2fc8de00 seq 3326 osr(0.473 0x2bd74d0) 1954 bytes (queue has 44 ops and 78084 bytes) 2014-06-09 16:22:11.611761 7f868c75b700 10 op_submit_finish 3326 2014-06-09 16:22:11.611754 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.745842 7 0.000183 2014-06-09 16:22:11.611770 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3292.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611774 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.611766 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.611773 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.611778 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.3c3( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.611784 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.611791 7f86a5e11700 10 _do_op 0x309a35c0 seq 3292 r = 0, finisher 0xc6e8440 0 2014-06-09 16:22:11.611792 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.611794 7f86a5e11700 10 _finish_op 0x309a35c0 seq 3292 osr(1.48e 0x384d9b0)/0x384d9b0 2014-06-09 16:22:11.611796 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.611793 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.611800 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c05a0 (1 active) 2014-06-09 16:22:11.611802 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339680 (2 active) 2014-06-09 16:22:11.611802 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8440] 2014-06-09 16:22:11.611804 7f86a5e11700 5 _do_op 0x2fcafe60 seq 3293 osr(2.67c 0x2ed1f90)/0x2ed1f90 start 2014-06-09 16:22:11.611806 7f86a5e11700 10 _do_transaction on 0x4800fc0 2014-06-09 16:22:11.611799 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.106938 7 0.000185 2014-06-09 16:22:11.611802 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.611809 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8440] 2014-06-09 16:22:11.611811 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611811 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611813 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611811 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.611810 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.611823 7f868c75b700 5 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.611833 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.611843 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611842 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.611850 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.611857 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3293.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611861 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.611865 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.611883 7f86a5e11700 10 _do_op 0x2fcafe60 seq 3293 r = 0, finisher 0xcc82320 0 2014-06-09 16:22:11.611887 7f86a5e11700 10 _finish_op 0x2fcafe60 seq 3293 osr(2.67c 0x2ed1f90)/0x2ed1f90 2014-06-09 16:22:11.611893 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339680 (1 active) 2014-06-09 16:22:11.611895 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcd80 (2 active) 2014-06-09 16:22:11.611897 7f86a5e11700 5 _do_op 0x30d96c00 seq 3294 osr(3.18e 0x3ed0af0)/0x3ed0af0 start 2014-06-09 16:22:11.611898 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.611900 7f86a5e11700 10 _do_transaction on 0x4734180 2014-06-09 16:22:11.611903 7f868d75d700 10 op_submit_start 3327 2014-06-09 16:22:11.611903 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc82320] 2014-06-09 16:22:11.611905 7f868d75d700 5 queue_transactions (trailing journal) 3327 2014-06-09 16:22:11.611905 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.611907 7f868d75d700 5 queue_op 0x308ca660 seq 3327 osr(3.579 0x3ed18d0) 1335 bytes (queue has 42 ops and 74027 bytes) 2014-06-09 16:22:11.611908 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc82320] 2014-06-09 16:22:11.611910 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611910 7f868d75d700 10 op_submit_finish 3327 2014-06-09 16:22:11.611911 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611925 7f8690763700 10 op_submit_start 3328 2014-06-09 16:22:11.611916 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=210 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.611928 7f8690763700 5 queue_transactions (trailing journal) 3328 2014-06-09 16:22:11.611930 7f8690763700 5 queue_op 0x2fcaee40 seq 3328 osr(2.431 0x2ed1590) 1883 bytes (queue has 42 ops and 74027 bytes) 2014-06-09 16:22:11.611935 7f8690763700 10 op_submit_finish 3328 2014-06-09 16:22:11.611939 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.611930 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=210 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.611941 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.611943 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=210 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.158532 7 0.000164 2014-06-09 16:22:11.611956 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3294.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.611953 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.375( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.611961 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.611961 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.611955 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=210 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.611963 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.611966 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.375( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.910896 7 0.000195 2014-06-09 16:22:11.611973 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.375( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.611968 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=210 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.611979 7f86a5e11700 10 _do_op 0x30d96c00 seq 3294 r = 0, finisher 0x13965260 0 2014-06-09 16:22:11.611980 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.375( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.611983 7f86a5e11700 10 _finish_op 0x30d96c00 seq 3294 osr(3.18e 0x3ed0af0)/0x3ed0af0 2014-06-09 16:22:11.611980 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=210 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.611987 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcd80 (1 active) 2014-06-09 16:22:11.611990 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2877a0 (2 active) 2014-06-09 16:22:11.611990 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13965260] 2014-06-09 16:22:11.611986 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.611992 7f86a5e11700 5 _do_op 0x30607620 seq 3295 osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 start 2014-06-09 16:22:11.611995 7f86a5e11700 10 _do_transaction on 0xc0df440 2014-06-09 16:22:11.611995 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13965260] 2014-06-09 16:22:11.611997 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.611998 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.611991 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'590 -> 213'590 2014-06-09 16:22:11.611993 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.612001 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.612002 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.612034 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.612030 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.612046 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.612050 7f8690763700 5 queue_transactions existing osr(1.375 0x384d230)/0x384d230 2014-06-09 16:22:11.612052 7f8690763700 10 op_submit_start 3329 2014-06-09 16:22:11.612053 7f8690763700 5 queue_transactions (trailing journal) 3329 2014-06-09 16:22:11.612051 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3295.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.612055 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.612054 7f8690763700 5 queue_op 0x2fcaede0 seq 3329 osr(1.375 0x384d230) 2009 bytes (queue has 42 ops and 74607 bytes) 2014-06-09 16:22:11.612057 7f8690763700 10 op_submit_finish 3329 2014-06-09 16:22:11.612047 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.19b( v 214'526 (0'0,214'526] local-les=213 n=101 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.612062 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.612059 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=210 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.612073 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.612074 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.612076 7f86a5e11700 10 _do_op 0x30607620 seq 3295 r = 0, finisher 0xbd687a0 0 2014-06-09 16:22:11.612079 7f86a5e11700 10 _finish_op 0x30607620 seq 3295 osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 2014-06-09 16:22:11.612081 7f868d75d700 5 queue_transactions existing osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 2014-06-09 16:22:11.612073 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=210 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.612084 7f868d75d700 10 op_submit_start 3330 2014-06-09 16:22:11.612084 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2877a0 (1 active) 2014-06-09 16:22:11.612085 7f868d75d700 5 queue_transactions (trailing journal) 3330 2014-06-09 16:22:11.612086 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cc60 (2 active) 2014-06-09 16:22:11.612086 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd687a0] 2014-06-09 16:22:11.612087 7f868d75d700 5 queue_op 0x308ca3c0 seq 3330 osr(3.2b6 0x3ed0ee0) 1335 bytes (queue has 42 ops and 74607 bytes) 2014-06-09 16:22:11.612088 7f86a5e11700 5 _do_op 0x308cb200 seq 3296 osr(2.26d 0x2ed0de0)/0x2ed0de0 start 2014-06-09 16:22:11.612090 7f868d75d700 10 op_submit_finish 3330 2014-06-09 16:22:11.612090 7f86a5e11700 10 _do_transaction on 0x3144300 2014-06-09 16:22:11.612082 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.612092 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd687a0] 2014-06-09 16:22:11.612093 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.612085 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=210 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.810205 7 0.000191 2014-06-09 16:22:11.612094 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.612095 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.612095 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.612096 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=210 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.612093 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.198( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.612105 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=210 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.612103 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.612106 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.612111 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=210 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.612111 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.612117 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.612118 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'630 -> 213'630 2014-06-09 16:22:11.612126 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.612128 7f868a757700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.612122 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:0.105( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.612134 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.612141 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3296.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.612144 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.612146 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.612143 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.612157 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.612162 7f86a5e11700 10 _do_op 0x308cb200 seq 3296 r = 0, finisher 0x3062a800 0 2014-06-09 16:22:11.612154 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.612165 7f86a5e11700 10 _finish_op 0x308cb200 seq 3296 osr(2.26d 0x2ed0de0)/0x2ed0de0 2014-06-09 16:22:11.612148 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.612160 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=208 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.612170 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cc60 (1 active) 2014-06-09 16:22:11.612172 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd8c0 (2 active) 2014-06-09 16:22:11.612174 7f86a5e11700 5 _do_op 0x308cb980 seq 3297 osr(1.66e 0x384ded0)/0x384ded0 start 2014-06-09 16:22:11.612176 7f86a5e11700 10 _do_transaction on 0x471d080 2014-06-09 16:22:11.612169 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.48b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.612182 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.612173 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=208 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.612182 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.612186 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.612185 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=208 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.098618 7 0.000171 2014-06-09 16:22:11.612196 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.612190 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.843889 7 0.000182 2014-06-09 16:22:11.612201 7f8690763700 5 queue_transactions existing osr(3.136 0x3ed07a0)/0x3ed07a0 2014-06-09 16:22:11.612196 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=208 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.612204 7f8690763700 10 op_submit_start 3331 2014-06-09 16:22:11.612206 7f8690763700 5 queue_transactions (trailing journal) 3331 2014-06-09 16:22:11.612201 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.612207 7f8690763700 5 queue_op 0x2fcaed80 seq 3331 osr(3.136 0x3ed07a0) 1335 bytes (queue has 42 ops and 74031 bytes) 2014-06-09 16:22:11.612210 7f8690763700 10 op_submit_finish 3331 2014-06-09 16:22:11.612210 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062a800] 2014-06-09 16:22:11.612212 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.612208 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=208 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.612216 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062a800] 2014-06-09 16:22:11.612217 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.612218 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.612212 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.612212 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.612218 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=208 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.612227 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3297.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.612222 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.612231 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.612224 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.112( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.612232 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.612236 7f868cf5c700 10 op_submit_start 3332 2014-06-09 16:22:11.612229 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'670 -> 213'670 2014-06-09 16:22:11.616175 7f868cf5c700 5 queue_transactions (trailing journal) 3332 2014-06-09 16:22:11.616174 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.616178 7f868cf5c700 5 queue_op 0x30d96840 seq 3332 osr(2.3e8 0x2ed1310) 1883 bytes (queue has 45 ops and 78655 bytes) 2014-06-09 16:22:11.616181 7f868cf5c700 10 op_submit_finish 3332 2014-06-09 16:22:11.616181 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.112( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.847030 7 0.000193 2014-06-09 16:22:11.616186 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.238( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.616196 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.616195 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.112( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616204 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.616206 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.112( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616210 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.616217 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616212 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616222 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.616227 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616233 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.616238 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.616236 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616247 7f868cf5c700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.616260 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.616265 7f868a757700 5 queue_transactions existing osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 2014-06-09 16:22:11.616259 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.616268 7f868a757700 10 op_submit_start 3333 2014-06-09 16:22:11.616270 7f868a757700 5 queue_transactions (trailing journal) 3333 2014-06-09 16:22:11.616271 7f868a757700 5 queue_op 0x30606840 seq 3333 osr(3.6ba 0x3ed1ce0) 1335 bytes (queue has 46 ops and 79990 bytes) 2014-06-09 16:22:11.616274 7f868a757700 10 op_submit_finish 3333 2014-06-09 16:22:11.616274 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.616286 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.616277 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=208 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.616281 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.616291 7f8690763700 5 queue_transactions existing osr(2.112 0x2ed06d0)/0x2ed06d0 2014-06-09 16:22:11.616294 7f8690763700 10 op_submit_start 3334 2014-06-09 16:22:11.616295 7f8690763700 5 queue_transactions (trailing journal) 3334 2014-06-09 16:22:11.616297 7f8690763700 5 queue_op 0x2fcaed20 seq 3334 osr(2.112 0x2ed06d0) 2009 bytes (queue has 47 ops and 81999 bytes) 2014-06-09 16:22:11.616300 7f8690763700 10 op_submit_finish 3334 2014-06-09 16:22:11.616291 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=208 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.616292 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 0.765( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616305 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616309 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.616305 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=208 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.102579 7 0.000177 2014-06-09 16:22:11.616303 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:0.161( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.616313 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.765( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.719631 7 0.000179 2014-06-09 16:22:11.616319 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.616319 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=208 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616331 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.616326 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.765( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616332 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=208 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616337 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.765( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616336 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=213 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.616347 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616340 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=208 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616349 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=213 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.9 3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.612242 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.616356 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616358 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'564 -> 213'564 2014-06-09 16:22:11.616364 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616362 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=213 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.142723 7 0.000191 2014-06-09 16:22:11.616374 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=213 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616378 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:962 2014-06-09 16:22:11.616386 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=213 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616396 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.616397 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616397 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=213 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616400 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.194( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.616411 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.616416 7f868cf5c700 5 queue_transactions existing osr(0.765 0x384c360)/0x384c360 2014-06-09 16:22:11.616408 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'605 -> 213'605 2014-06-09 16:22:11.616418 7f868cf5c700 10 op_submit_start 3335 2014-06-09 16:22:11.616419 7f868cf5c700 5 queue_transactions (trailing journal) 3335 2014-06-09 16:22:11.616410 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.194( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.194( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616420 7f868cf5c700 5 queue_op 0x30d96780 seq 3335 osr(0.765 0x384c360) 2001 bytes (queue has 48 ops and 84000 bytes) 2014-06-09 16:22:11.616423 7f868cf5c700 10 op_submit_finish 3335 2014-06-09 16:22:11.616421 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616425 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.616425 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.616428 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.194( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.916489 7 0.000197 2014-06-09 16:22:11.616438 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.616436 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.194( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616443 7f868a757700 5 queue_transactions existing osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 2014-06-09 16:22:11.616435 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 0.4c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616446 7f868a757700 10 op_submit_start 3336 2014-06-09 16:22:11.616447 7f868a757700 5 queue_transactions (trailing journal) 3336 2014-06-09 16:22:11.616446 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616448 7f868a757700 5 queue_op 0x306068a0 seq 3336 osr(3.6cb 0x3ed1cb0) 1335 bytes (queue has 49 ops and 85335 bytes) 2014-06-09 16:22:11.616449 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.616451 7f868a757700 10 op_submit_finish 3336 2014-06-09 16:22:11.616446 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.194( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616450 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616452 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.771376 7 0.000191 2014-06-09 16:22:11.616455 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.194( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616454 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.616462 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616463 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.194( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616471 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616471 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.194( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616466 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616479 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616483 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.616479 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616491 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.616487 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616487 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.699929 7 0.000170 2014-06-09 16:22:11.616495 7f8690763700 5 queue_transactions existing osr(3.27b 0x3ed0bd0)/0x3ed0bd0 2014-06-09 16:22:11.616498 7f8690763700 10 op_submit_start 3337 2014-06-09 16:22:11.616500 7f8690763700 5 queue_transactions (trailing journal) 3337 2014-06-09 16:22:11.616497 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616495 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616501 7f8690763700 5 queue_op 0x2fcaecc0 seq 3337 osr(3.27b 0x3ed0bd0) 1335 bytes (queue has 50 ops and 86670 bytes) 2014-06-09 16:22:11.616504 7f8690763700 10 op_submit_finish 3337 2014-06-09 16:22:11.616505 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616515 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.616507 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.616519 7f8692f68700 5 queue_transactions existing osr(2.194 0x2ed0910)/0x2ed0910 2014-06-09 16:22:11.616515 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616522 7f8692f68700 2 waiting 51 > 50 ops || 88624 > 104857600 2014-06-09 16:22:11.616519 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616523 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616531 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616535 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.616531 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616539 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.612248 7f8691f66700 5 queue_transactions existing osr(1.8c 0x384c470)/0x384c470 2014-06-09 16:22:11.616544 7f868cf5c700 5 queue_transactions existing osr(0.4c8 0x2bd7670)/0x2bd7670 2014-06-09 16:22:11.616539 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.268606 7 0.005147 2014-06-09 16:22:11.616546 7f868cf5c700 2 waiting 51 > 50 ops || 88553 > 104857600 2014-06-09 16:22:11.616556 7f8691f66700 2 waiting 51 > 50 ops || 88668 > 104857600 2014-06-09 16:22:11.612286 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.616553 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616564 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616563 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.58a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616579 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616581 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.616583 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.616585 7f868a757700 5 queue_transactions existing osr(0.4ba 0x2bd7710)/0x2bd7710 2014-06-09 16:22:11.616588 7f868a757700 2 waiting 51 > 50 ops || 88667 > 104857600 2014-06-09 16:22:11.616582 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.612272 7f868c75b700 5 queue_transactions existing osr(1.3c3 0x384d530)/0x384d530 2014-06-09 16:22:11.616587 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.839681 7 0.000190 2014-06-09 16:22:11.612305 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.616596 7f868c75b700 2 waiting 51 > 50 ops || 88624 > 104857600 2014-06-09 16:22:11.612250 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.616592 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616598 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616601 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616604 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.616609 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616616 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.616618 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3290.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.616621 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.616623 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.616624 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.616618 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616625 7f869af78700 20 osd.26 243 _dispatch 0x30882a00 pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.616627 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616635 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0290 (19 active) 2014-06-09 16:22:11.612321 7f8692767700 5 -- op tracker -- , seq: 170, time: 2014-06-09 16:22:11.611467, event: reached_pg, op: osd_op(client.4566.0:33819 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~671744] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.616635 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616649 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.616654 7f8690763700 5 queue_transactions existing osr(2.3aa 0x2ed1070)/0x2ed1070 2014-06-09 16:22:11.616655 7f86a5610700 10 _do_op 0x30d97d40 seq 3290 r = 0, finisher 0xbfcef20 0 2014-06-09 16:22:11.616656 7f8690763700 2 waiting 51 > 50 ops || 88624 > 104857600 2014-06-09 16:22:11.616646 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.616659 7f86a5610700 10 _finish_op 0x30d97d40 seq 3290 osr(2.13c 0x2ed0610)/0x2ed0610 2014-06-09 16:22:11.616659 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0290 (20 active) 2014-06-09 16:22:11.612278 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:2.45b( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.616667 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49b00 (1 active) 2014-06-09 16:22:11.616660 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x11e63900 2014-06-09 16:22:11.616669 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52efc0 (2 active) 2014-06-09 16:22:11.616669 7f8692767700 10 osd.26 243 dequeue_op 0x11e63900 finish 2014-06-09 16:22:11.616671 7f86a5610700 5 _do_op 0x2fcae000 seq 3298 osr(2.350 0x2ed0e50)/0x2ed0e50 start 2014-06-09 16:22:11.616663 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.616673 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.616674 7f86a5610700 10 _do_transaction on 0x3084000 2014-06-09 16:22:11.616664 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.616678 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.616679 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.616679 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.616680 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.616675 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 1.1fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616684 7f8689f56700 5 queue_transactions existing osr(0.58a 0x2bd7770)/0x2bd7770 2014-06-09 16:22:11.616686 7f8689f56700 10 op_submit_start 3338 2014-06-09 16:22:11.616687 7f8689f56700 5 queue_transactions (trailing journal) 3338 2014-06-09 16:22:11.616686 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.616688 7f8689f56700 5 queue_op 0x30d973e0 seq 3338 osr(0.58a 0x2bd7770) 1954 bytes (queue has 50 ops and 86525 bytes) 2014-06-09 16:22:11.616689 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.616691 7f8689f56700 10 op_submit_finish 3338 2014-06-09 16:22:11.616683 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.583( v 213'614 (0'0,213'614] local-les=201 n=113 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.616683 7f8692767700 10 osd.26 243 dequeue_op 0x1b0b6500 prio 63 cost 516096 latency 1.196247 osd_op(client.4572.0:30406 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 761856~516096] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.616693 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.680851 7 0.000159 2014-06-09 16:22:11.616694 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.616696 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.616709 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.616703 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616705 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.616705 7f8692767700 5 -- op tracker -- , seq: 378, time: 2014-06-09 16:22:11.616701, event: reached_pg, op: osd_op(client.4572.0:30406 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 761856~516096] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.616715 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.616713 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616713 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=191 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.616716 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.616719 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.b3( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.616722 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616729 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0b6500 2014-06-09 16:22:11.616730 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.616737 7f8692767700 10 osd.26 243 dequeue_op 0x1b0b6500 finish 2014-06-09 16:22:11.616731 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616727 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=191 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.616740 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.616741 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.616739 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616741 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=191 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.816213 7 0.000364 2014-06-09 16:22:11.616744 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.37d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.616754 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.616756 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.616755 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=191 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616765 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.616758 7f8692767700 10 osd.26 243 dequeue_op 0x270be600 prio 63 cost 1032192 latency 0.841011 osd_op(client.4573.0:34088 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 729088~1032192] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.616768 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=191 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616780 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (19 active) 2014-06-09 16:22:11.616781 7f8692767700 5 -- op tracker -- , seq: 719, time: 2014-06-09 16:22:11.616776, event: reached_pg, op: osd_op(client.4573.0:34088 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 729088~1032192] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.616778 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=191 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616787 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.616788 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (20 active) 2014-06-09 16:22:11.616792 7f868e75f700 5 queue_transactions existing osr(1.1fa 0x384cd50)/0x384cd50 2014-06-09 16:22:11.616794 7f868e75f700 2 waiting 51 > 50 ops || 88534 > 104857600 2014-06-09 16:22:11.612265 7f868af58700 5 queue_transactions existing osr(1.77c 0x2ed0330)/0x2ed0330 2014-06-09 16:22:11.616789 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'648 -> 213'648 2014-06-09 16:22:11.616788 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.616791 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.616800 7f868af58700 2 waiting 51 > 50 ops || 88534 > 104857600 2014-06-09 16:22:11.616804 7f868bf5a700 10 op_submit_start 3339 2014-06-09 16:22:11.616805 7f868bf5a700 5 queue_transactions (trailing journal) 3339 2014-06-09 16:22:11.616807 7f868bf5a700 5 queue_op 0x308cafc0 seq 3339 osr(3.400 0x3ed1260) 1335 bytes (queue has 50 ops and 86525 bytes) 2014-06-09 16:22:11.616810 7f868bf5a700 10 op_submit_finish 3339 2014-06-09 16:22:11.616801 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x270be600 2014-06-09 16:22:11.616804 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.616811 7f8692767700 10 osd.26 243 dequeue_op 0x270be600 finish 2014-06-09 16:22:11.616814 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.616814 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.616818 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.616815 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.13c( v 213'621 (0'0,213'621] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.616817 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.616822 7f8692767700 10 osd.26 243 dequeue_op 0x11e63800 prio 63 cost 516096 latency 1.395221 osd_op(client.4566.0:33820 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 671744~516096] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.616830 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.616828 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.13c( v 213'621 (0'0,213'621] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.616840 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.616835 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616845 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (19 active) 2014-06-09 16:22:11.616840 7f8692767700 5 -- op tracker -- , seq: 171, time: 2014-06-09 16:22:11.616836, event: reached_pg, op: osd_op(client.4566.0:33820 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 671744~516096] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.616841 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.112587 7 0.000162 2014-06-09 16:22:11.616856 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.616849 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.616854 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.616860 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x11e63800 2014-06-09 16:22:11.616869 7f8692767700 10 osd.26 243 dequeue_op 0x11e63800 finish 2014-06-09 16:22:11.616859 7f8689f56700 10 osd.26 243 dequeue_op 0x1b0b6400 prio 63 cost 516096 latency 1.161535 osd_op(client.4572.0:30407 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 1277952~516096] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.616872 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.616866 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.616881 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.616878 7f8689f56700 5 -- op tracker -- , seq: 379, time: 2014-06-09 16:22:11.616873, event: reached_pg, op: osd_op(client.4572.0:30407 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 1277952~516096] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.616876 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.616886 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'621 -> 213'621 2014-06-09 16:22:11.616885 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.616884 7f8692767700 10 osd.26 243 dequeue_op 0x270be500 prio 63 cost 1032192 latency 0.830784 osd_op(client.4573.0:34089 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 1761280~1032192] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.616896 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0b6400 2014-06-09 16:22:11.616904 7f8689f56700 10 osd.26 243 dequeue_op 0x1b0b6400 finish 2014-06-09 16:22:11.616908 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.616905 7f8692767700 5 -- op tracker -- , seq: 726, time: 2014-06-09 16:22:11.616900, event: reached_pg, op: osd_op(client.4573.0:34089 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 1761280~1032192] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.616916 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.616911 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.616920 7f8689f56700 10 osd.26 243 dequeue_op 0x14053f00 prio 63 cost 516096 latency 1.394459 osd_op(client.4566.0:33821 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1187840~516096] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.616923 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x270be500 2014-06-09 16:22:11.616933 7f8692767700 10 osd.26 243 dequeue_op 0x270be500 finish 2014-06-09 16:22:11.616936 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.616938 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.616932 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616938 7f8689f56700 5 -- op tracker -- , seq: 182, time: 2014-06-09 16:22:11.616933, event: reached_pg, op: osd_op(client.4566.0:33821 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1187840~516096] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.616943 7f8691765700 5 queue_transactions existing osr(3.124 0x3ed07c0)/0x3ed07c0 2014-06-09 16:22:11.616944 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.616946 7f8691765700 2 waiting 51 > 50 ops || 87954 > 104857600 2014-06-09 16:22:11.612232 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.616944 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.616948 7f8692767700 10 osd.26 243 dequeue_op 0x1b0b6300 prio 63 cost 516096 latency 1.160684 osd_op(client.4572.0:30408 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~516096] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.616952 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.616955 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053f00 2014-06-09 16:22:11.616964 7f8689f56700 10 osd.26 243 dequeue_op 0x14053f00 finish 2014-06-09 16:22:11.616967 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.616969 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.616966 7f8692767700 5 -- op tracker -- , seq: 393, time: 2014-06-09 16:22:11.616961, event: reached_pg, op: osd_op(client.4572.0:30408 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~516096] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.616973 7f868bf5a700 5 queue_transactions existing osr(3.13c 0x3ed0760)/0x3ed0760 2014-06-09 16:22:11.616975 7f868bf5a700 2 waiting 51 > 50 ops || 87860 > 104857600 2014-06-09 16:22:11.616975 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.616980 7f868f761700 10 op_submit_start 3340 2014-06-09 16:22:11.616981 7f868f761700 5 queue_transactions (trailing journal) 3340 2014-06-09 16:22:11.616983 7f868f761700 5 queue_op 0x30d96b40 seq 3340 osr(3.2d6 0x3ed0e60) 1335 bytes (queue has 50 ops and 86525 bytes) 2014-06-09 16:22:11.616975 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.616986 7f868f761700 10 op_submit_finish 3340 2014-06-09 16:22:11.616979 7f8689f56700 10 osd.26 243 dequeue_op 0x270be400 prio 63 cost 1400832 latency 0.823289 osd_op(client.4573.0:34090 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 2793472~1400832] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.616987 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0b6300 2014-06-09 16:22:11.616995 7f8692767700 10 osd.26 243 dequeue_op 0x1b0b6300 finish 2014-06-09 16:22:11.616998 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.616991 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.617000 7f8689f56700 5 -- op tracker -- , seq: 761, time: 2014-06-09 16:22:11.616995, event: reached_pg, op: osd_op(client.4573.0:34090 rbd_data.117d6b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 2793472~1400832] 3.22748e5d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.617003 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.617006 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.617008 7f868d75d700 5 queue_transactions existing osr(0.48b 0x2bd7450)/0x2bd7450 2014-06-09 16:22:11.617011 7f868d75d700 2 waiting 51 > 50 ops || 88479 > 104857600 2014-06-09 16:22:11.617003 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617015 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617006 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.617018 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.617010 7f8692767700 10 osd.26 243 dequeue_op 0x14053e00 prio 63 cost 516096 latency 1.367346 osd_op(client.4566.0:33822 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~516096] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.612247 7f86a5e11700 10 _do_op 0x308cb980 seq 3297 r = 0, finisher 0xc40c4e0 0 2014-06-09 16:22:11.617018 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x270be400 2014-06-09 16:22:11.617029 7f8689f56700 10 osd.26 243 dequeue_op 0x270be400 finish 2014-06-09 16:22:11.617021 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.409( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.077583 7 0.004230 2014-06-09 16:22:11.617028 7f86a5e11700 10 _finish_op 0x308cb980 seq 3297 osr(1.66e 0x384ded0)/0x384ded0 2014-06-09 16:22:11.617032 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.617028 7f8692767700 5 -- op tracker -- , seq: 185, time: 2014-06-09 16:22:11.617024, event: reached_pg, op: osd_op(client.4566.0:33822 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~516096] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.617035 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd8c0 (1 active) 2014-06-09 16:22:11.617032 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.409( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.617038 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd9e0 (2 active) 2014-06-09 16:22:11.617040 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.617041 7f86a5e11700 5 _do_op 0x309a2180 seq 3299 osr(2.5b6 0x2ed1b90)/0x2ed1b90 start 2014-06-09 16:22:11.617044 7f86a5e11700 10 _do_transaction on 0xc1612c0 2014-06-09 16:22:11.617034 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.617041 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.409( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.617051 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.617046 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053e00 2014-06-09 16:22:11.617055 7f8692767700 10 osd.26 243 dequeue_op 0x14053e00 finish 2014-06-09 16:22:11.617044 7f8689f56700 10 osd.26 243 dequeue_op 0x1b0b6200 prio 63 cost 1884160 latency 1.136403 osd_op(client.4572.0:30409 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 2310144~1884160] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.617051 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.617057 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.617064 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.617059 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.617062 7f8689f56700 5 -- op tracker -- , seq: 422, time: 2014-06-09 16:22:11.617058, event: reached_pg, op: osd_op(client.4572.0:30409 rbd_data.117c6b8b4567.0000000000001452 [set-alloc-hint object_size 4194304 write_size 4194304,write 2310144~1884160] 3.75f5d562 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.617068 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfcef20,0xc40c4e0] 2014-06-09 16:22:11.617067 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.617068 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.617067 7f8692767700 10 osd.26 243 dequeue_op 0x14053d00 prio 63 cost 516096 latency 1.361508 osd_op(client.4566.0:33823 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2220032~516096] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.617079 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfcef20,0xc40c4e0] 2014-06-09 16:22:11.617081 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.617083 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.617079 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x1b0b6200 2014-06-09 16:22:11.617088 7f8692f68700 10 op_submit_start 3341 2014-06-09 16:22:11.617088 7f8689f56700 10 osd.26 243 dequeue_op 0x1b0b6200 finish 2014-06-09 16:22:11.617086 7f8692767700 5 -- op tracker -- , seq: 187, time: 2014-06-09 16:22:11.617081, event: reached_pg, op: osd_op(client.4566.0:33823 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2220032~516096] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.617090 7f8692f68700 5 queue_transactions (trailing journal) 3341 2014-06-09 16:22:11.617091 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.617091 7f8692f68700 5 queue_op 0x309a3440 seq 3341 osr(2.194 0x2ed0910) 1954 bytes (queue has 50 ops and 86858 bytes) 2014-06-09 16:22:11.617095 7f8692f68700 10 op_submit_finish 3341 2014-06-09 16:22:11.617099 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.617091 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.617105 7f868cf5c700 2 waiting 51 > 50 ops || 88741 > 104857600 2014-06-09 16:22:11.617098 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=208 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.617110 7f8691f66700 2 waiting 51 > 50 ops || 88856 > 104857600 2014-06-09 16:22:11.617103 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053d00 2014-06-09 16:22:11.617111 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.617112 7f8692767700 10 osd.26 243 dequeue_op 0x14053d00 finish 2014-06-09 16:22:11.617114 7f868a757700 2 waiting 51 > 50 ops || 88855 > 104857600 2014-06-09 16:22:11.617115 7f868f761700 5 queue_transactions existing osr(2.409 0x2ed1270)/0x2ed1270 2014-06-09 16:22:11.617117 7f868f761700 2 waiting 51 > 50 ops || 88741 > 104857600 2014-06-09 16:22:11.617117 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.617123 7f868e75f700 2 waiting 51 > 50 ops || 88867 > 104857600 2014-06-09 16:22:11.617112 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=208 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.617126 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.617127 7f868af58700 2 waiting 51 > 50 ops || 88867 > 104857600 2014-06-09 16:22:11.612328 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd73ba0,0xc40d860,0x18742a20,0x189aadc0,0xcc95d00,0xbccc5c0,0x3062a840] 2014-06-09 16:22:11.617132 7f8689755700 10 op_submit_start 3342 2014-06-09 16:22:11.617134 7f8689755700 5 queue_transactions (trailing journal) 3342 2014-06-09 16:22:11.617121 7f8689f56700 10 osd.26 243 dequeue_op 0x14053c00 prio 63 cost 1032192 latency 1.360601 osd_op(client.4566.0:33824 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2736128~1032192] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.617127 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=208 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.302131 7 0.000165 2014-06-09 16:22:11.617135 7f8689755700 5 queue_op 0x100396e0 seq 3342 osr(2.52d 0x2ed19f0) 1883 bytes (queue has 50 ops and 86858 bytes) 2014-06-09 16:22:11.617138 7f8689755700 10 op_submit_finish 3342 2014-06-09 16:22:11.617133 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617141 7f8689f56700 5 -- op tracker -- , seq: 192, time: 2014-06-09 16:22:11.617137, event: reached_pg, op: osd_op(client.4566.0:33824 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2736128~1032192] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.617139 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=208 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.617149 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.617143 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.617147 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.617152 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=208 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.617159 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053c00 2014-06-09 16:22:11.617157 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.617168 7f8689f56700 10 osd.26 243 dequeue_op 0x14053c00 finish 2014-06-09 16:22:11.617164 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.617163 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=208 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.617173 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.617169 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.302931 7 0.000179 2014-06-09 16:22:11.617179 7f8689f56700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.617174 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'564 -> 213'564 2014-06-09 16:22:11.617180 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.617182 7f869af78700 5 -- op tracker -- , seq: 2071, time: 2014-06-09 16:22:11.343134, event: header_read, op: pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617193 7f869af78700 5 -- op tracker -- , seq: 2071, time: 2014-06-09 16:22:11.343135, event: throttled, op: pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617191 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.617187 7f8692767700 10 osd.26 243 dequeue_op 0x14053b00 prio 63 cost 425984 latency 1.353797 osd_op(client.4566.0:33825 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3768320~425984] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.617200 7f869af78700 5 -- op tracker -- , seq: 2071, time: 2014-06-09 16:22:11.343147, event: all_read, op: pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617200 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.617209 7f869af78700 5 -- op tracker -- , seq: 2071, time: 2014-06-09 16:22:11.616620, event: dispatched, op: pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617210 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'671 -> 213'671 2014-06-09 16:22:11.617210 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617217 7f869af78700 5 -- op tracker -- , seq: 2071, time: 2014-06-09 16:22:11.617214, event: waiting_for_osdmap, op: pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617213 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.617224 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.617223 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.617227 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30882a00 2014-06-09 16:22:11.617234 7f869af78700 5 -- op tracker -- , seq: 2071, time: 2014-06-09 16:22:11.617231, event: started, op: pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617238 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.617248 7f869af78700 5 -- op tracker -- , seq: 2071, time: 2014-06-09 16:22:11.617245, event: done, op: pg_log(0.6c0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617256 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.617257 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617250 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617259 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617261 7f8692f68700 5 queue_transactions existing osr(3.577 0x3ed18c0)/0x3ed18c0 2014-06-09 16:22:11.617255 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.617262 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.617264 7f8692f68700 2 waiting 51 > 50 ops || 88193 > 104857600 2014-06-09 16:22:11.617268 7f8691765700 2 waiting 51 > 50 ops || 88287 > 104857600 2014-06-09 16:22:11.617273 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617274 7f8690763700 2 waiting 51 > 50 ops || 88812 > 104857600 2014-06-09 16:22:11.617275 7f869af78700 20 osd.26 243 _dispatch 0x3046a8c0 pg_notify(3.15c(12),1.16d(22) epoch 243) v5 2014-06-09 16:22:11.617272 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.617278 7f868d75d700 2 waiting 51 > 50 ops || 88812 > 104857600 2014-06-09 16:22:11.617283 7f868c75b700 2 waiting 51 > 50 ops || 88812 > 104857600 2014-06-09 16:22:11.617281 7f869af78700 5 -- op tracker -- , seq: 2072, time: 2014-06-09 16:22:11.171675, event: header_read, op: pg_notify(3.15c(12),1.16d(22) epoch 243) 2014-06-09 16:22:11.617289 7f869af78700 5 -- op tracker -- , seq: 2072, time: 2014-06-09 16:22:11.171675, event: throttled, op: pg_notify(3.15c(12),1.16d(22) epoch 243) 2014-06-09 16:22:11.617284 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617293 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.617296 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.617297 7f8689755700 5 queue_transactions existing osr(3.527 0x3ed1660)/0x3ed1660 2014-06-09 16:22:11.617295 7f869af78700 5 -- op tracker -- , seq: 2072, time: 2014-06-09 16:22:11.171715, event: all_read, op: pg_notify(3.15c(12),1.16d(22) epoch 243) 2014-06-09 16:22:11.617299 7f8689755700 2 waiting 51 > 50 ops || 88193 > 104857600 2014-06-09 16:22:11.617303 7f8690f64700 10 op_submit_start 3343 2014-06-09 16:22:11.617302 7f869af78700 5 -- op tracker -- , seq: 2072, time: 2014-06-09 16:22:11.617272, event: dispatched, op: pg_notify(3.15c(12),1.16d(22) epoch 243) 2014-06-09 16:22:11.617305 7f8690f64700 5 queue_transactions (trailing journal) 3343 2014-06-09 16:22:11.617306 7f8690f64700 5 queue_op 0x2fcaf860 seq 3343 osr(0.66a 0x2bd7ef0) 2009 bytes (queue has 50 ops and 86858 bytes) 2014-06-09 16:22:11.617309 7f8690f64700 10 op_submit_finish 3343 2014-06-09 16:22:11.617309 7f869af78700 5 -- op tracker -- , seq: 2072, time: 2014-06-09 16:22:11.617306, event: waiting_for_osdmap, op: pg_notify(3.15c(12),1.16d(22) epoch 243) 2014-06-09 16:22:11.617313 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.617315 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046a8c0 2014-06-09 16:22:11.617310 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] flushed 2014-06-09 16:22:11.617320 7f869af78700 5 -- op tracker -- , seq: 2072, time: 2014-06-09 16:22:11.617318, event: started, op: pg_notify(3.15c(12),1.16d(22) epoch 243) 2014-06-09 16:22:11.617314 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.617322 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617335 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.617326 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617338 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617341 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.617344 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.977760 7 0.003490 2014-06-09 16:22:11.617348 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.617354 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.617364 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.617362 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617344 7f869af78700 5 -- op tracker -- , seq: 2072, time: 2014-06-09 16:22:11.617342, event: done, op: pg_notify(3.15c(12),1.16d(22) epoch 243) 2014-06-09 16:22:11.617377 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.617373 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.617381 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617383 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617382 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.617390 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.617391 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.617404 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617406 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617407 7f869af78700 20 osd.26 243 _dispatch 0x305f4a00 pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.617405 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617413 7f869af78700 5 -- op tracker -- , seq: 2073, time: 2014-06-09 16:22:11.190325, event: header_read, op: pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617422 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.617421 7f869af78700 5 -- op tracker -- , seq: 2073, time: 2014-06-09 16:22:11.190325, event: throttled, op: pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617428 7f869af78700 5 -- op tracker -- , seq: 2073, time: 2014-06-09 16:22:11.190333, event: all_read, op: pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617433 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.617434 7f869af78700 5 -- op tracker -- , seq: 2073, time: 2014-06-09 16:22:11.617404, event: dispatched, op: pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617437 7f8690f64700 5 queue_transactions existing osr(1.fa 0x384c970)/0x384c970 2014-06-09 16:22:11.617440 7f8690f64700 2 waiting 51 > 50 ops || 88738 > 104857600 2014-06-09 16:22:11.617436 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.617441 7f869af78700 5 -- op tracker -- , seq: 2073, time: 2014-06-09 16:22:11.617439, event: waiting_for_osdmap, op: pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617444 7f868b759700 10 op_submit_start 3344 2014-06-09 16:22:11.617446 7f868b759700 5 queue_transactions (trailing journal) 3344 2014-06-09 16:22:11.617447 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd73ba0,0xc40d860,0x18742a20,0x189aadc0,0xcc95d00,0xbccc5c0,0x3062a840] 2014-06-09 16:22:11.617446 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.617447 7f868b759700 5 queue_op 0x308ca960 seq 3344 osr(1.302 0x384d010) 1903 bytes (queue has 50 ops and 86858 bytes) 2014-06-09 16:22:11.617449 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a7680,0xc40c1e0] 2014-06-09 16:22:11.617449 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f4a00 2014-06-09 16:22:11.617450 7f868b759700 10 op_submit_finish 3344 2014-06-09 16:22:11.617452 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617455 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.617464 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.617467 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.129( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617479 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617475 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.617483 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.617486 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.129( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 1.121811 7 0.003102 2014-06-09 16:22:11.617488 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.617500 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.617500 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.129( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.617455 7f869af78700 5 -- op tracker -- , seq: 2073, time: 2014-06-09 16:22:11.617452, event: started, op: pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617511 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] flushed 2014-06-09 16:22:11.617512 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.129( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.617523 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a7680,0xc40c1e0] 2014-06-09 16:22:11.617525 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.617526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.617522 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.617527 7f869af78700 5 -- op tracker -- , seq: 2073, time: 2014-06-09 16:22:11.617523, event: done, op: pg_log(0.2a2 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617532 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.617539 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (20 active) 2014-06-09 16:22:11.617541 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617542 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617541 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.617544 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.617555 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617557 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617558 7f869af78700 20 osd.26 243 _dispatch 0x10c39c00 pg_info(1 pgs e243:1.60e) v4 2014-06-09 16:22:11.617555 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.6c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617562 7f869af78700 5 -- op tracker -- , seq: 2074, time: 2014-06-09 16:22:11.345017, event: header_read, op: pg_info(1 pgs e243:1.60e) 2014-06-09 16:22:11.617566 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617569 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.617569 7f869af78700 5 -- op tracker -- , seq: 2074, time: 2014-06-09 16:22:11.345017, event: throttled, op: pg_info(1 pgs e243:1.60e) 2014-06-09 16:22:11.617573 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.944626 7 0.000179 2014-06-09 16:22:11.617577 7f869af78700 5 -- op tracker -- , seq: 2074, time: 2014-06-09 16:22:11.345032, event: all_read, op: pg_info(1 pgs e243:1.60e) 2014-06-09 16:22:11.617583 7f869af78700 5 -- op tracker -- , seq: 2074, time: 2014-06-09 16:22:11.617555, event: dispatched, op: pg_info(1 pgs e243:1.60e) 2014-06-09 16:22:11.617582 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.617592 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.617592 7f869af78700 5 -- op tracker -- , seq: 2074, time: 2014-06-09 16:22:11.617590, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.60e) 2014-06-09 16:22:11.617597 7f868b759700 5 queue_transactions existing osr(0.129 0x2bd66b0)/0x2bd66b0 2014-06-09 16:22:11.617596 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.60e) v4 from osd.7 2014-06-09 16:22:11.617592 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.617599 7f868b759700 2 waiting 51 > 50 ops || 88957 > 104857600 2014-06-09 16:22:11.617600 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10c39c00 2014-06-09 16:22:11.617602 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.617605 7f869af78700 5 -- op tracker -- , seq: 2074, time: 2014-06-09 16:22:11.617603, event: started, op: pg_info(1 pgs e243:1.60e) 2014-06-09 16:22:11.617610 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.617620 7f869af78700 5 -- op tracker -- , seq: 2074, time: 2014-06-09 16:22:11.617618, event: done, op: pg_info(1 pgs e243:1.60e) 2014-06-09 16:22:11.617618 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.617627 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617629 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617632 7f868bf5a700 2 waiting 51 > 50 ops || 88193 > 104857600 2014-06-09 16:22:11.617641 7f868df5e700 10 op_submit_start 3345 2014-06-09 16:22:11.617642 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617643 7f868df5e700 5 queue_transactions (trailing journal) 3345 2014-06-09 16:22:11.617644 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617645 7f869af78700 20 osd.26 243 _dispatch 0x15998540 pg_notify(2.777(20),1.1e4(20) epoch 243) v5 2014-06-09 16:22:11.617645 7f868df5e700 5 queue_op 0x2fcae2a0 seq 3345 osr(3.1d5 0x3ed09f0) 1335 bytes (queue has 50 ops and 86858 bytes) 2014-06-09 16:22:11.617648 7f868df5e700 10 op_submit_finish 3345 2014-06-09 16:22:11.617650 7f869af78700 5 -- op tracker -- , seq: 2075, time: 2014-06-09 16:22:11.332916, event: header_read, op: pg_notify(2.777(20),1.1e4(20) epoch 243) 2014-06-09 16:22:11.617657 7f869af78700 5 -- op tracker -- , seq: 2075, time: 2014-06-09 16:22:11.332916, event: throttled, op: pg_notify(2.777(20),1.1e4(20) epoch 243) 2014-06-09 16:22:11.617663 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.617654 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.617663 7f869af78700 5 -- op tracker -- , seq: 2075, time: 2014-06-09 16:22:11.333337, event: all_read, op: pg_notify(2.777(20),1.1e4(20) epoch 243) 2014-06-09 16:22:11.617667 7f8689f56700 5 queue_transactions existing osr(0.6c0 0x2bd7f50)/0x2bd7f50 2014-06-09 16:22:11.617670 7f8689f56700 2 waiting 51 > 50 ops || 88859 > 104857600 2014-06-09 16:22:11.617668 7f868df5e700 15 osd.26 pg_epoch: 243 pg[0.151( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.617679 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.617669 7f869af78700 5 -- op tracker -- , seq: 2075, time: 2014-06-09 16:22:11.617642, event: dispatched, op: pg_notify(2.777(20),1.1e4(20) epoch 243) 2014-06-09 16:22:11.617684 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.617695 7f8692767700 5 -- op tracker -- , seq: 195, time: 2014-06-09 16:22:11.617202, event: reached_pg, op: osd_op(client.4566.0:33825 rbd_data.11766b8b4567.00000000000016b5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3768320~425984] 3.833cb874 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.617696 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.617705 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.617707 7f869af78700 5 -- op tracker -- , seq: 2075, time: 2014-06-09 16:22:11.617688, event: waiting_for_osdmap, op: pg_notify(2.777(20),1.1e4(20) epoch 243) 2014-06-09 16:22:11.617703 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.617714 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.617716 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15998540 2014-06-09 16:22:11.617709 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=191 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.617716 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x14053b00 2014-06-09 16:22:11.617725 7f8692767700 10 osd.26 243 dequeue_op 0x14053b00 finish 2014-06-09 16:22:11.617729 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.617721 7f869af78700 5 -- op tracker -- , seq: 2075, time: 2014-06-09 16:22:11.617719, event: started, op: pg_notify(2.777(20),1.1e4(20) epoch 243) 2014-06-09 16:22:11.617723 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=191 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.617737 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (20 active) 2014-06-09 16:22:11.617740 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.617751 7f869af78700 5 -- op tracker -- , seq: 2075, time: 2014-06-09 16:22:11.617749, event: done, op: pg_notify(2.777(20),1.1e4(20) epoch 243) 2014-06-09 16:22:11.617736 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=191 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.694653 7 0.000181 2014-06-09 16:22:11.617759 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617753 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.617761 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617761 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.617761 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=191 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.617769 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.617780 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.617774 7f868df5e700 5 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=191 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.617785 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=191 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.617794 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.617797 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'666 -> 213'666 2014-06-09 16:22:11.617809 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.617791 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.617816 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (19 active) 2014-06-09 16:22:11.617815 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.617819 7f869af78700 20 osd.26 243 _dispatch 0x30724000 pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.617824 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (20 active) 2014-06-09 16:22:11.617827 7f868ef60700 10 op_submit_start 3346 2014-06-09 16:22:11.617826 7f869af78700 5 -- op tracker -- , seq: 2076, time: 2014-06-09 16:22:11.244944, event: header_read, op: pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617831 7f868ef60700 5 queue_transactions (trailing journal) 3346 2014-06-09 16:22:11.617826 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:2.777( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.617833 7f868ef60700 5 queue_op 0x2fcafe00 seq 3346 osr(1.601 0x384dcd0) 1954 bytes (queue has 50 ops and 86858 bytes) 2014-06-09 16:22:11.617837 7f868ef60700 10 op_submit_finish 3346 2014-06-09 16:22:11.617835 7f869af78700 5 -- op tracker -- , seq: 2076, time: 2014-06-09 16:22:11.244945, event: throttled, op: pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617842 7f869af78700 5 -- op tracker -- , seq: 2076, time: 2014-06-09 16:22:11.244956, event: all_read, op: pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617839 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.617848 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.617840 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.617844 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.29( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.617848 7f869af78700 5 -- op tracker -- , seq: 2076, time: 2014-06-09 16:22:11.617791, event: dispatched, op: pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617852 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.1e4( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.617864 7f869af78700 5 -- op tracker -- , seq: 2076, time: 2014-06-09 16:22:11.617861, event: waiting_for_osdmap, op: pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.617859 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.617869 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.617864 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.617872 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.617872 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30724000 2014-06-09 16:22:11.617873 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.617877 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.617885 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.617886 7f868ff62700 10 op_submit_start 3347 2014-06-09 16:22:11.617877 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.617887 7f868ff62700 5 queue_transactions (trailing journal) 3347 2014-06-09 16:22:11.617890 7f868df5e700 5 queue_transactions existing osr(3.76f 0x3ed1e20)/0x3ed1e20 2014-06-09 16:22:11.617890 7f868ff62700 5 queue_op 0x15997c80 seq 3347 osr(3.52a 0x3ed1670) 1335 bytes (queue has 50 ops and 86858 bytes) 2014-06-09 16:22:11.617893 7f868ff62700 10 op_submit_finish 3347 2014-06-09 16:22:11.617893 7f868df5e700 2 waiting 51 > 50 ops || 88287 > 104857600 2014-06-09 16:22:11.617887 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.617891 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.521( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.617895 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.617903 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.617908 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0260 (19 active) 2014-06-09 16:22:11.617906 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.617907 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.617919 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0260 (20 active) 2014-06-09 16:22:11.617914 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.617919 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.617923 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.617929 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.617923 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.617934 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.617936 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.617944 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.617944 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.617941 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.617948 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.617951 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.617959 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.617959 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.617965 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (19 active) 2014-06-09 16:22:11.617954 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.646( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617968 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.617968 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.617972 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0210 (20 active) 2014-06-09 16:22:11.617972 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.617972 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.617975 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.617976 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.646( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.361502 7 0.000184 2014-06-09 16:22:11.617983 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.617992 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.617985 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.472602 6 0.000284 2014-06-09 16:22:11.617987 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.646( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.617996 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.617996 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.617998 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.646( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.618007 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.618008 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.618009 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.618018 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.618017 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:11.618018 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.618022 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.618027 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.618027 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.618034 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.618043 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.618038 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.618045 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.618046 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.618056 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.517( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.618064 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.618057 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.618066 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.618067 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.618075 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.618078 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.618084 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.618085 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.618085 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.618089 7f868ef60700 5 queue_transactions existing osr(2.646 0x2ed1c30)/0x2ed1c30 2014-06-09 16:22:11.618091 7f868ef60700 2 waiting 51 > 50 ops || 88741 > 104857600 2014-06-09 16:22:11.618090 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.618099 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:11.618102 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.654( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.618110 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.618117 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0260 (19 active) 2014-06-09 16:22:11.618123 7f868ff62700 20 OSD::op_tp worker waiting 2014-06-09 16:22:11.618129 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.618134 7f8692767700 5 queue_transactions existing osr(1.60e 0x384dc30)/0x384dc30 2014-06-09 16:22:11.618136 7f8692767700 2 waiting 51 > 50 ops || 87719 > 104857600 2014-06-09 16:22:11.617878 7f869af78700 5 -- op tracker -- , seq: 2076, time: 2014-06-09 16:22:11.617875, event: started, op: pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618202 7f869af78700 5 -- op tracker -- , seq: 2076, time: 2014-06-09 16:22:11.618199, event: done, op: pg_log(1.a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618206 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0260 (20 active) 2014-06-09 16:22:11.618210 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618212 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618210 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.618225 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618226 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618227 7f869af78700 20 osd.26 243 _dispatch 0x2fe708c0 pg_info(1 pgs e243:1.37e) v4 2014-06-09 16:22:11.618222 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.618232 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.618232 7f869af78700 5 -- op tracker -- , seq: 2077, time: 2014-06-09 16:22:11.314373, event: header_read, op: pg_info(1 pgs e243:1.37e) 2014-06-09 16:22:11.618235 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.618238 7f869af78700 5 -- op tracker -- , seq: 2077, time: 2014-06-09 16:22:11.314373, event: throttled, op: pg_info(1 pgs e243:1.37e) 2014-06-09 16:22:11.618239 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.676624 7 0.000201 2014-06-09 16:22:11.618244 7f869af78700 5 -- op tracker -- , seq: 2077, time: 2014-06-09 16:22:11.314382, event: all_read, op: pg_info(1 pgs e243:1.37e) 2014-06-09 16:22:11.618248 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.618250 7f869af78700 5 -- op tracker -- , seq: 2077, time: 2014-06-09 16:22:11.618224, event: dispatched, op: pg_info(1 pgs e243:1.37e) 2014-06-09 16:22:11.618259 7f869af78700 5 -- op tracker -- , seq: 2077, time: 2014-06-09 16:22:11.618257, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.37e) 2014-06-09 16:22:11.618258 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.618264 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.37e) v4 from osd.9 2014-06-09 16:22:11.618266 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe708c0 2014-06-09 16:22:11.618266 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.618271 7f869af78700 5 -- op tracker -- , seq: 2077, time: 2014-06-09 16:22:11.618269, event: started, op: pg_info(1 pgs e243:1.37e) 2014-06-09 16:22:11.618275 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.618282 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.618285 7f869af78700 5 -- op tracker -- , seq: 2077, time: 2014-06-09 16:22:11.618283, event: done, op: pg_info(1 pgs e243:1.37e) 2014-06-09 16:22:11.618292 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618293 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618307 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618308 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618309 7f869af78700 20 osd.26 243 _dispatch 0x3046b880 pg_notify(0.451(20) epoch 243) v5 2014-06-09 16:22:11.618314 7f869af78700 5 -- op tracker -- , seq: 2078, time: 2014-06-09 16:22:11.349232, event: header_read, op: pg_notify(0.451(20) epoch 243) 2014-06-09 16:22:11.618320 7f869af78700 5 -- op tracker -- , seq: 2078, time: 2014-06-09 16:22:11.349233, event: throttled, op: pg_notify(0.451(20) epoch 243) 2014-06-09 16:22:11.618326 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.618325 7f869af78700 5 -- op tracker -- , seq: 2078, time: 2014-06-09 16:22:11.349260, event: all_read, op: pg_notify(0.451(20) epoch 243) 2014-06-09 16:22:11.618330 7f868ff62700 5 queue_transactions existing osr(1.a 0x384c270)/0x384c270 2014-06-09 16:22:11.618332 7f868ff62700 2 waiting 51 > 50 ops || 88735 > 104857600 2014-06-09 16:22:11.618331 7f869af78700 5 -- op tracker -- , seq: 2078, time: 2014-06-09 16:22:11.618307, event: dispatched, op: pg_notify(0.451(20) epoch 243) 2014-06-09 16:22:11.618337 7f869af78700 5 -- op tracker -- , seq: 2078, time: 2014-06-09 16:22:11.618336, event: waiting_for_osdmap, op: pg_notify(0.451(20) epoch 243) 2014-06-09 16:22:11.618341 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.618342 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046b880 2014-06-09 16:22:11.618345 7f869af78700 5 -- op tracker -- , seq: 2078, time: 2014-06-09 16:22:11.618344, event: started, op: pg_notify(0.451(20) epoch 243) 2014-06-09 16:22:11.618355 7f869af78700 5 -- op tracker -- , seq: 2078, time: 2014-06-09 16:22:11.618354, event: done, op: pg_notify(0.451(20) epoch 243) 2014-06-09 16:22:11.618360 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618361 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618372 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618373 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618374 7f869af78700 20 osd.26 243 _dispatch 0x308d9880 pg_info(2 pgs e243:3.786(2),3.1f0(2)) v4 2014-06-09 16:22:11.618377 7f869af78700 5 -- op tracker -- , seq: 2079, time: 2014-06-09 16:22:11.347856, event: header_read, op: pg_info(2 pgs e243:3.786(2),3.1f0(2)) 2014-06-09 16:22:11.618381 7f869af78700 5 -- op tracker -- , seq: 2079, time: 2014-06-09 16:22:11.347856, event: throttled, op: pg_info(2 pgs e243:3.786(2),3.1f0(2)) 2014-06-09 16:22:11.618385 7f869af78700 5 -- op tracker -- , seq: 2079, time: 2014-06-09 16:22:11.347877, event: all_read, op: pg_info(2 pgs e243:3.786(2),3.1f0(2)) 2014-06-09 16:22:11.618388 7f869af78700 5 -- op tracker -- , seq: 2079, time: 2014-06-09 16:22:11.618372, event: dispatched, op: pg_info(2 pgs e243:3.786(2),3.1f0(2)) 2014-06-09 16:22:11.618392 7f869af78700 5 -- op tracker -- , seq: 2079, time: 2014-06-09 16:22:11.618391, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.786(2),3.1f0(2)) 2014-06-09 16:22:11.618395 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.786(2),3.1f0(2)) v4 from osd.6 2014-06-09 16:22:11.618397 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d9880 2014-06-09 16:22:11.618400 7f869af78700 5 -- op tracker -- , seq: 2079, time: 2014-06-09 16:22:11.618399, event: started, op: pg_info(2 pgs e243:3.786(2),3.1f0(2)) 2014-06-09 16:22:11.618413 7f869af78700 5 -- op tracker -- , seq: 2079, time: 2014-06-09 16:22:11.618412, event: done, op: pg_info(2 pgs e243:3.786(2),3.1f0(2)) 2014-06-09 16:22:11.618418 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618419 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618427 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618428 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618429 7f869af78700 20 osd.26 243 _dispatch 0x1a039340 pg_info(1 pgs e243:0.3e) v4 2014-06-09 16:22:11.618431 7f869af78700 5 -- op tracker -- , seq: 2080, time: 2014-06-09 16:22:11.259503, event: header_read, op: pg_info(1 pgs e243:0.3e) 2014-06-09 16:22:11.618435 7f869af78700 5 -- op tracker -- , seq: 2080, time: 2014-06-09 16:22:11.259504, event: throttled, op: pg_info(1 pgs e243:0.3e) 2014-06-09 16:22:11.618438 7f869af78700 5 -- op tracker -- , seq: 2080, time: 2014-06-09 16:22:11.259521, event: all_read, op: pg_info(1 pgs e243:0.3e) 2014-06-09 16:22:11.618441 7f869af78700 5 -- op tracker -- , seq: 2080, time: 2014-06-09 16:22:11.618427, event: dispatched, op: pg_info(1 pgs e243:0.3e) 2014-06-09 16:22:11.618445 7f869af78700 5 -- op tracker -- , seq: 2080, time: 2014-06-09 16:22:11.618444, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3e) 2014-06-09 16:22:11.618447 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3e) v4 from osd.0 2014-06-09 16:22:11.618449 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a039340 2014-06-09 16:22:11.618451 7f869af78700 5 -- op tracker -- , seq: 2080, time: 2014-06-09 16:22:11.618450, event: started, op: pg_info(1 pgs e243:0.3e) 2014-06-09 16:22:11.618459 7f869af78700 5 -- op tracker -- , seq: 2080, time: 2014-06-09 16:22:11.618458, event: done, op: pg_info(1 pgs e243:0.3e) 2014-06-09 16:22:11.618471 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618472 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618481 7f869af78700 20 osd.26 243 _dispatch 0x230ba380 pg_notify(2.3fa(20) epoch 243) v5 2014-06-09 16:22:11.618484 7f869af78700 5 -- op tracker -- , seq: 2081, time: 2014-06-09 16:22:11.343152, event: header_read, op: pg_notify(2.3fa(20) epoch 243) 2014-06-09 16:22:11.618488 7f869af78700 5 -- op tracker -- , seq: 2081, time: 2014-06-09 16:22:11.343152, event: throttled, op: pg_notify(2.3fa(20) epoch 243) 2014-06-09 16:22:11.618491 7f869af78700 5 -- op tracker -- , seq: 2081, time: 2014-06-09 16:22:11.343182, event: all_read, op: pg_notify(2.3fa(20) epoch 243) 2014-06-09 16:22:11.618495 7f869af78700 5 -- op tracker -- , seq: 2081, time: 2014-06-09 16:22:11.618480, event: dispatched, op: pg_notify(2.3fa(20) epoch 243) 2014-06-09 16:22:11.618498 7f869af78700 5 -- op tracker -- , seq: 2081, time: 2014-06-09 16:22:11.618497, event: waiting_for_osdmap, op: pg_notify(2.3fa(20) epoch 243) 2014-06-09 16:22:11.618500 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.618501 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230ba380 2014-06-09 16:22:11.618504 7f869af78700 5 -- op tracker -- , seq: 2081, time: 2014-06-09 16:22:11.618503, event: started, op: pg_notify(2.3fa(20) epoch 243) 2014-06-09 16:22:11.618518 7f869af78700 5 -- op tracker -- , seq: 2081, time: 2014-06-09 16:22:11.618517, event: done, op: pg_notify(2.3fa(20) epoch 243) 2014-06-09 16:22:11.618523 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618524 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618534 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618535 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618536 7f869af78700 20 osd.26 243 _dispatch 0x13be4700 pg_notify(0.553(25) epoch 243) v5 2014-06-09 16:22:11.618538 7f869af78700 5 -- op tracker -- , seq: 2082, time: 2014-06-09 16:22:11.185426, event: header_read, op: pg_notify(0.553(25) epoch 243) 2014-06-09 16:22:11.618542 7f869af78700 5 -- op tracker -- , seq: 2082, time: 2014-06-09 16:22:11.185427, event: throttled, op: pg_notify(0.553(25) epoch 243) 2014-06-09 16:22:11.618546 7f869af78700 5 -- op tracker -- , seq: 2082, time: 2014-06-09 16:22:11.185449, event: all_read, op: pg_notify(0.553(25) epoch 243) 2014-06-09 16:22:11.618549 7f869af78700 5 -- op tracker -- , seq: 2082, time: 2014-06-09 16:22:11.618534, event: dispatched, op: pg_notify(0.553(25) epoch 243) 2014-06-09 16:22:11.618552 7f869af78700 5 -- op tracker -- , seq: 2082, time: 2014-06-09 16:22:11.618551, event: waiting_for_osdmap, op: pg_notify(0.553(25) epoch 243) 2014-06-09 16:22:11.618554 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.618555 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x13be4700 2014-06-09 16:22:11.618558 7f869af78700 5 -- op tracker -- , seq: 2082, time: 2014-06-09 16:22:11.618557, event: started, op: pg_notify(0.553(25) epoch 243) 2014-06-09 16:22:11.618566 7f869af78700 5 -- op tracker -- , seq: 2082, time: 2014-06-09 16:22:11.618565, event: done, op: pg_notify(0.553(25) epoch 243) 2014-06-09 16:22:11.618570 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618571 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618583 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618584 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618585 7f869af78700 20 osd.26 243 _dispatch 0x306db200 pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.618589 7f869af78700 5 -- op tracker -- , seq: 2083, time: 2014-06-09 16:22:11.190381, event: header_read, op: pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618593 7f869af78700 5 -- op tracker -- , seq: 2083, time: 2014-06-09 16:22:11.190381, event: throttled, op: pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618599 7f869af78700 5 -- op tracker -- , seq: 2083, time: 2014-06-09 16:22:11.190389, event: all_read, op: pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618603 7f869af78700 5 -- op tracker -- , seq: 2083, time: 2014-06-09 16:22:11.618583, event: dispatched, op: pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618607 7f869af78700 5 -- op tracker -- , seq: 2083, time: 2014-06-09 16:22:11.618605, event: waiting_for_osdmap, op: pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618609 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.618611 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306db200 2014-06-09 16:22:11.618614 7f869af78700 5 -- op tracker -- , seq: 2083, time: 2014-06-09 16:22:11.618613, event: started, op: pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618625 7f869af78700 5 -- op tracker -- , seq: 2083, time: 2014-06-09 16:22:11.618624, event: done, op: pg_log(1.2a3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618632 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618633 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618641 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618642 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618642 7f869af78700 20 osd.26 243 _dispatch 0x10c38700 pg_notify(3.77c(10),1.4bc(20) epoch 243) v5 2014-06-09 16:22:11.618646 7f869af78700 5 -- op tracker -- , seq: 2084, time: 2014-06-09 16:22:11.355625, event: header_read, op: pg_notify(3.77c(10),1.4bc(20) epoch 243) 2014-06-09 16:22:11.618650 7f869af78700 5 -- op tracker -- , seq: 2084, time: 2014-06-09 16:22:11.355626, event: throttled, op: pg_notify(3.77c(10),1.4bc(20) epoch 243) 2014-06-09 16:22:11.618653 7f869af78700 5 -- op tracker -- , seq: 2084, time: 2014-06-09 16:22:11.355676, event: all_read, op: pg_notify(3.77c(10),1.4bc(20) epoch 243) 2014-06-09 16:22:11.618656 7f869af78700 5 -- op tracker -- , seq: 2084, time: 2014-06-09 16:22:11.618641, event: dispatched, op: pg_notify(3.77c(10),1.4bc(20) epoch 243) 2014-06-09 16:22:11.618660 7f869af78700 5 -- op tracker -- , seq: 2084, time: 2014-06-09 16:22:11.618659, event: waiting_for_osdmap, op: pg_notify(3.77c(10),1.4bc(20) epoch 243) 2014-06-09 16:22:11.618663 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.618664 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10c38700 2014-06-09 16:22:11.618667 7f869af78700 5 -- op tracker -- , seq: 2084, time: 2014-06-09 16:22:11.618665, event: started, op: pg_notify(3.77c(10),1.4bc(20) epoch 243) 2014-06-09 16:22:11.618680 7f869af78700 5 -- op tracker -- , seq: 2084, time: 2014-06-09 16:22:11.618679, event: done, op: pg_notify(3.77c(10),1.4bc(20) epoch 243) 2014-06-09 16:22:11.618684 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618685 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618707 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618707 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618708 7f869af78700 20 osd.26 243 _dispatch 0x30640c00 pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.618712 7f869af78700 5 -- op tracker -- , seq: 2085, time: 2014-06-09 16:22:11.399828, event: header_read, op: pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618716 7f869af78700 5 -- op tracker -- , seq: 2085, time: 2014-06-09 16:22:11.399829, event: throttled, op: pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618720 7f869af78700 5 -- op tracker -- , seq: 2085, time: 2014-06-09 16:22:11.399840, event: all_read, op: pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618723 7f869af78700 5 -- op tracker -- , seq: 2085, time: 2014-06-09 16:22:11.618707, event: dispatched, op: pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618727 7f869af78700 5 -- op tracker -- , seq: 2085, time: 2014-06-09 16:22:11.618725, event: waiting_for_osdmap, op: pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618729 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.618731 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30640c00 2014-06-09 16:22:11.618734 7f869af78700 5 -- op tracker -- , seq: 2085, time: 2014-06-09 16:22:11.618733, event: started, op: pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618743 7f869af78700 5 -- op tracker -- , seq: 2085, time: 2014-06-09 16:22:11.618742, event: done, op: pg_log(0.1cd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.618748 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618749 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618756 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618757 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618758 7f869af78700 20 osd.26 243 _dispatch 0x260008c0 pg_notify(0.1fb(21) epoch 243) v5 2014-06-09 16:22:11.618761 7f869af78700 5 -- op tracker -- , seq: 2086, time: 2014-06-09 16:22:11.244986, event: header_read, op: pg_notify(0.1fb(21) epoch 243) 2014-06-09 16:22:11.618764 7f869af78700 5 -- op tracker -- , seq: 2086, time: 2014-06-09 16:22:11.244987, event: throttled, op: pg_notify(0.1fb(21) epoch 243) 2014-06-09 16:22:11.618768 7f869af78700 5 -- op tracker -- , seq: 2086, time: 2014-06-09 16:22:11.245014, event: all_read, op: pg_notify(0.1fb(21) epoch 243) 2014-06-09 16:22:11.618771 7f869af78700 5 -- op tracker -- , seq: 2086, time: 2014-06-09 16:22:11.618756, event: dispatched, op: pg_notify(0.1fb(21) epoch 243) 2014-06-09 16:22:11.618774 7f869af78700 5 -- op tracker -- , seq: 2086, time: 2014-06-09 16:22:11.618773, event: waiting_for_osdmap, op: pg_notify(0.1fb(21) epoch 243) 2014-06-09 16:22:11.618776 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.618777 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x260008c0 2014-06-09 16:22:11.618780 7f869af78700 5 -- op tracker -- , seq: 2086, time: 2014-06-09 16:22:11.618779, event: started, op: pg_notify(0.1fb(21) epoch 243) 2014-06-09 16:22:11.618789 7f869af78700 5 -- op tracker -- , seq: 2086, time: 2014-06-09 16:22:11.618788, event: done, op: pg_notify(0.1fb(21) epoch 243) 2014-06-09 16:22:11.618793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618794 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618804 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618804 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618805 7f869af78700 20 osd.26 243 _dispatch 0x308d96c0 pg_info(1 pgs e243:0.b7) v4 2014-06-09 16:22:11.618808 7f869af78700 5 -- op tracker -- , seq: 2087, time: 2014-06-09 16:22:11.314467, event: header_read, op: pg_info(1 pgs e243:0.b7) 2014-06-09 16:22:11.618812 7f869af78700 5 -- op tracker -- , seq: 2087, time: 2014-06-09 16:22:11.314467, event: throttled, op: pg_info(1 pgs e243:0.b7) 2014-06-09 16:22:11.618815 7f869af78700 5 -- op tracker -- , seq: 2087, time: 2014-06-09 16:22:11.314474, event: all_read, op: pg_info(1 pgs e243:0.b7) 2014-06-09 16:22:11.618823 7f869af78700 5 -- op tracker -- , seq: 2087, time: 2014-06-09 16:22:11.618804, event: dispatched, op: pg_info(1 pgs e243:0.b7) 2014-06-09 16:22:11.618827 7f869af78700 5 -- op tracker -- , seq: 2087, time: 2014-06-09 16:22:11.618825, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.b7) 2014-06-09 16:22:11.618829 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.b7) v4 from osd.9 2014-06-09 16:22:11.618831 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d96c0 2014-06-09 16:22:11.618833 7f869af78700 5 -- op tracker -- , seq: 2087, time: 2014-06-09 16:22:11.618832, event: started, op: pg_info(1 pgs e243:0.b7) 2014-06-09 16:22:11.618847 7f869af78700 5 -- op tracker -- , seq: 2087, time: 2014-06-09 16:22:11.618846, event: done, op: pg_info(1 pgs e243:0.b7) 2014-06-09 16:22:11.618852 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618853 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618860 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618861 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618862 7f869af78700 20 osd.26 243 _dispatch 0x240eaa80 pg_notify(1.3e1(20) epoch 243) v5 2014-06-09 16:22:11.618864 7f869af78700 5 -- op tracker -- , seq: 2088, time: 2014-06-09 16:22:11.351090, event: header_read, op: pg_notify(1.3e1(20) epoch 243) 2014-06-09 16:22:11.618868 7f869af78700 5 -- op tracker -- , seq: 2088, time: 2014-06-09 16:22:11.351091, event: throttled, op: pg_notify(1.3e1(20) epoch 243) 2014-06-09 16:22:11.618871 7f869af78700 5 -- op tracker -- , seq: 2088, time: 2014-06-09 16:22:11.351122, event: all_read, op: pg_notify(1.3e1(20) epoch 243) 2014-06-09 16:22:11.618876 7f869af78700 5 -- op tracker -- , seq: 2088, time: 2014-06-09 16:22:11.618860, event: dispatched, op: pg_notify(1.3e1(20) epoch 243) 2014-06-09 16:22:11.618880 7f869af78700 5 -- op tracker -- , seq: 2088, time: 2014-06-09 16:22:11.618879, event: waiting_for_osdmap, op: pg_notify(1.3e1(20) epoch 243) 2014-06-09 16:22:11.618882 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.618883 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eaa80 2014-06-09 16:22:11.618886 7f869af78700 5 -- op tracker -- , seq: 2088, time: 2014-06-09 16:22:11.618885, event: started, op: pg_notify(1.3e1(20) epoch 243) 2014-06-09 16:22:11.618896 7f869af78700 5 -- op tracker -- , seq: 2088, time: 2014-06-09 16:22:11.618895, event: done, op: pg_notify(1.3e1(20) epoch 243) 2014-06-09 16:22:11.618900 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618901 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618911 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618912 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618912 7f869af78700 20 osd.26 243 _dispatch 0x308d88c0 pg_notify(3.1f4(10) epoch 243) v5 2014-06-09 16:22:11.618915 7f869af78700 5 -- op tracker -- , seq: 2089, time: 2014-06-09 16:22:11.347881, event: header_read, op: pg_notify(3.1f4(10) epoch 243) 2014-06-09 16:22:11.618921 7f869af78700 5 -- op tracker -- , seq: 2089, time: 2014-06-09 16:22:11.347882, event: throttled, op: pg_notify(3.1f4(10) epoch 243) 2014-06-09 16:22:11.618928 7f869af78700 5 -- op tracker -- , seq: 2089, time: 2014-06-09 16:22:11.347901, event: all_read, op: pg_notify(3.1f4(10) epoch 243) 2014-06-09 16:22:11.618932 7f869af78700 5 -- op tracker -- , seq: 2089, time: 2014-06-09 16:22:11.618911, event: dispatched, op: pg_notify(3.1f4(10) epoch 243) 2014-06-09 16:22:11.618940 7f869af78700 5 -- op tracker -- , seq: 2089, time: 2014-06-09 16:22:11.618939, event: waiting_for_osdmap, op: pg_notify(3.1f4(10) epoch 243) 2014-06-09 16:22:11.618942 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.618943 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d88c0 2014-06-09 16:22:11.618946 7f869af78700 5 -- op tracker -- , seq: 2089, time: 2014-06-09 16:22:11.618945, event: started, op: pg_notify(3.1f4(10) epoch 243) 2014-06-09 16:22:11.618956 7f869af78700 5 -- op tracker -- , seq: 2089, time: 2014-06-09 16:22:11.618955, event: done, op: pg_notify(3.1f4(10) epoch 243) 2014-06-09 16:22:11.618961 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618962 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618976 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.618977 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.618977 7f869af78700 20 osd.26 243 _dispatch 0x2d7328c0 pg_info(1 pgs e243:0.6e) v4 2014-06-09 16:22:11.618980 7f869af78700 5 -- op tracker -- , seq: 2090, time: 2014-06-09 16:22:11.259538, event: header_read, op: pg_info(1 pgs e243:0.6e) 2014-06-09 16:22:11.618984 7f869af78700 5 -- op tracker -- , seq: 2090, time: 2014-06-09 16:22:11.259538, event: throttled, op: pg_info(1 pgs e243:0.6e) 2014-06-09 16:22:11.618988 7f869af78700 5 -- op tracker -- , seq: 2090, time: 2014-06-09 16:22:11.259547, event: all_read, op: pg_info(1 pgs e243:0.6e) 2014-06-09 16:22:11.618991 7f869af78700 5 -- op tracker -- , seq: 2090, time: 2014-06-09 16:22:11.618975, event: dispatched, op: pg_info(1 pgs e243:0.6e) 2014-06-09 16:22:11.618997 7f869af78700 5 -- op tracker -- , seq: 2090, time: 2014-06-09 16:22:11.618996, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6e) 2014-06-09 16:22:11.618999 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6e) v4 from osd.0 2014-06-09 16:22:11.619001 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d7328c0 2014-06-09 16:22:11.619005 7f869af78700 5 -- op tracker -- , seq: 2090, time: 2014-06-09 16:22:11.619003, event: started, op: pg_info(1 pgs e243:0.6e) 2014-06-09 16:22:11.619021 7f869af78700 5 -- op tracker -- , seq: 2090, time: 2014-06-09 16:22:11.619020, event: done, op: pg_info(1 pgs e243:0.6e) 2014-06-09 16:22:11.619026 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619027 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619034 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619035 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619035 7f869af78700 20 osd.26 243 _dispatch 0x230bb6c0 pg_info(1 pgs e243:3.6bb(2)) v4 2014-06-09 16:22:11.619038 7f869af78700 5 -- op tracker -- , seq: 2091, time: 2014-06-09 16:22:11.343188, event: header_read, op: pg_info(1 pgs e243:3.6bb(2)) 2014-06-09 16:22:11.619044 7f869af78700 5 -- op tracker -- , seq: 2091, time: 2014-06-09 16:22:11.343188, event: throttled, op: pg_info(1 pgs e243:3.6bb(2)) 2014-06-09 16:22:11.619047 7f869af78700 5 -- op tracker -- , seq: 2091, time: 2014-06-09 16:22:11.343209, event: all_read, op: pg_info(1 pgs e243:3.6bb(2)) 2014-06-09 16:22:11.619052 7f869af78700 5 -- op tracker -- , seq: 2091, time: 2014-06-09 16:22:11.619034, event: dispatched, op: pg_info(1 pgs e243:3.6bb(2)) 2014-06-09 16:22:11.619055 7f869af78700 5 -- op tracker -- , seq: 2091, time: 2014-06-09 16:22:11.619054, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6bb(2)) 2014-06-09 16:22:11.619061 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6bb(2)) v4 from osd.5 2014-06-09 16:22:11.619063 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230bb6c0 2014-06-09 16:22:11.619066 7f869af78700 5 -- op tracker -- , seq: 2091, time: 2014-06-09 16:22:11.619064, event: started, op: pg_info(1 pgs e243:3.6bb(2)) 2014-06-09 16:22:11.619076 7f869af78700 5 -- op tracker -- , seq: 2091, time: 2014-06-09 16:22:11.619075, event: done, op: pg_info(1 pgs e243:3.6bb(2)) 2014-06-09 16:22:11.619080 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619081 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619089 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619090 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619090 7f869af78700 20 osd.26 243 _dispatch 0x30646a00 pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.619094 7f869af78700 5 -- op tracker -- , seq: 2092, time: 2014-06-09 16:22:11.185470, event: header_read, op: pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619098 7f869af78700 5 -- op tracker -- , seq: 2092, time: 2014-06-09 16:22:11.185470, event: throttled, op: pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619102 7f869af78700 5 -- op tracker -- , seq: 2092, time: 2014-06-09 16:22:11.185479, event: all_read, op: pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619105 7f869af78700 5 -- op tracker -- , seq: 2092, time: 2014-06-09 16:22:11.619089, event: dispatched, op: pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619109 7f869af78700 5 -- op tracker -- , seq: 2092, time: 2014-06-09 16:22:11.619108, event: waiting_for_osdmap, op: pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619113 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.619115 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30646a00 2014-06-09 16:22:11.619118 7f869af78700 5 -- op tracker -- , seq: 2092, time: 2014-06-09 16:22:11.619117, event: started, op: pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619125 7f869af78700 5 -- op tracker -- , seq: 2092, time: 2014-06-09 16:22:11.619123, event: done, op: pg_log(0.2c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619129 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619130 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619136 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619137 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619138 7f869af78700 20 osd.26 243 _dispatch 0x305a8400 pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.619141 7f869af78700 5 -- op tracker -- , seq: 2093, time: 2014-06-09 16:22:11.194415, event: header_read, op: pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619146 7f869af78700 5 -- op tracker -- , seq: 2093, time: 2014-06-09 16:22:11.194416, event: throttled, op: pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619150 7f869af78700 5 -- op tracker -- , seq: 2093, time: 2014-06-09 16:22:11.194425, event: all_read, op: pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619154 7f869af78700 5 -- op tracker -- , seq: 2093, time: 2014-06-09 16:22:11.619136, event: dispatched, op: pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619158 7f869af78700 5 -- op tracker -- , seq: 2093, time: 2014-06-09 16:22:11.619156, event: waiting_for_osdmap, op: pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619160 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.619163 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305a8400 2014-06-09 16:22:11.619165 7f869af78700 5 -- op tracker -- , seq: 2093, time: 2014-06-09 16:22:11.619164, event: started, op: pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619176 7f869af78700 5 -- op tracker -- , seq: 2093, time: 2014-06-09 16:22:11.619174, event: done, op: pg_log(1.582 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.619181 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619182 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619189 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619190 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619190 7f869af78700 20 osd.26 243 _dispatch 0x12060c40 pg_notify(1.4bc(20) epoch 243) v5 2014-06-09 16:22:11.619193 7f869af78700 5 -- op tracker -- , seq: 2094, time: 2014-06-09 16:22:11.355698, event: header_read, op: pg_notify(1.4bc(20) epoch 243) 2014-06-09 16:22:11.619196 7f869af78700 5 -- op tracker -- , seq: 2094, time: 2014-06-09 16:22:11.355699, event: throttled, op: pg_notify(1.4bc(20) epoch 243) 2014-06-09 16:22:11.619200 7f869af78700 5 -- op tracker -- , seq: 2094, time: 2014-06-09 16:22:11.355732, event: all_read, op: pg_notify(1.4bc(20) epoch 243) 2014-06-09 16:22:11.619204 7f869af78700 5 -- op tracker -- , seq: 2094, time: 2014-06-09 16:22:11.619189, event: dispatched, op: pg_notify(1.4bc(20) epoch 243) 2014-06-09 16:22:11.619209 7f869af78700 5 -- op tracker -- , seq: 2094, time: 2014-06-09 16:22:11.619208, event: waiting_for_osdmap, op: pg_notify(1.4bc(20) epoch 243) 2014-06-09 16:22:11.619211 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.619212 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12060c40 2014-06-09 16:22:11.619215 7f869af78700 5 -- op tracker -- , seq: 2094, time: 2014-06-09 16:22:11.619214, event: started, op: pg_notify(1.4bc(20) epoch 243) 2014-06-09 16:22:11.619223 7f869af78700 5 -- op tracker -- , seq: 2094, time: 2014-06-09 16:22:11.619222, event: done, op: pg_notify(1.4bc(20) epoch 243) 2014-06-09 16:22:11.619229 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619230 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.619241 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.619242 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622177 7f869af78700 20 osd.26 243 _dispatch 0x305b3200 pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.622185 7f869af78700 5 -- op tracker -- , seq: 2095, time: 2014-06-09 16:22:11.399851, event: header_read, op: pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622210 7f869af78700 5 -- op tracker -- , seq: 2095, time: 2014-06-09 16:22:11.399852, event: throttled, op: pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622216 7f869af78700 5 -- op tracker -- , seq: 2095, time: 2014-06-09 16:22:11.399869, event: all_read, op: pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622220 7f869af78700 5 -- op tracker -- , seq: 2095, time: 2014-06-09 16:22:11.619241, event: dispatched, op: pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622227 7f869af78700 5 -- op tracker -- , seq: 2095, time: 2014-06-09 16:22:11.622226, event: waiting_for_osdmap, op: pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622230 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.622232 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b3200 2014-06-09 16:22:11.622236 7f869af78700 5 -- op tracker -- , seq: 2095, time: 2014-06-09 16:22:11.622234, event: started, op: pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622244 7f869af78700 5 -- op tracker -- , seq: 2095, time: 2014-06-09 16:22:11.622243, event: done, op: pg_log(1.1cf epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622249 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622250 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622259 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622260 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622262 7f869af78700 20 osd.26 243 _dispatch 0x30724600 pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.622268 7f869af78700 5 -- op tracker -- , seq: 2096, time: 2014-06-09 16:22:11.254069, event: header_read, op: pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622275 7f869af78700 5 -- op tracker -- , seq: 2096, time: 2014-06-09 16:22:11.254070, event: throttled, op: pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622284 7f869af78700 5 -- op tracker -- , seq: 2096, time: 2014-06-09 16:22:11.254084, event: all_read, op: pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622291 7f869af78700 5 -- op tracker -- , seq: 2096, time: 2014-06-09 16:22:11.622259, event: dispatched, op: pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622301 7f869af78700 5 -- op tracker -- , seq: 2096, time: 2014-06-09 16:22:11.622298, event: waiting_for_osdmap, op: pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622311 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.622315 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30724600 2014-06-09 16:22:11.622321 7f869af78700 5 -- op tracker -- , seq: 2096, time: 2014-06-09 16:22:11.622318, event: started, op: pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622340 7f869af78700 5 -- op tracker -- , seq: 2096, time: 2014-06-09 16:22:11.622337, event: done, op: pg_log(0.70e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.622347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622348 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622357 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622357 7f869af78700 20 osd.26 243 _dispatch 0x308d9a40 pg_notify(2.573(20),3.2b3(10) epoch 243) v5 2014-06-09 16:22:11.622361 7f869af78700 5 -- op tracker -- , seq: 2097, time: 2014-06-09 16:22:11.318246, event: header_read, op: pg_notify(2.573(20),3.2b3(10) epoch 243) 2014-06-09 16:22:11.622365 7f869af78700 5 -- op tracker -- , seq: 2097, time: 2014-06-09 16:22:11.318246, event: throttled, op: pg_notify(2.573(20),3.2b3(10) epoch 243) 2014-06-09 16:22:11.622368 7f869af78700 5 -- op tracker -- , seq: 2097, time: 2014-06-09 16:22:11.318277, event: all_read, op: pg_notify(2.573(20),3.2b3(10) epoch 243) 2014-06-09 16:22:11.622374 7f869af78700 5 -- op tracker -- , seq: 2097, time: 2014-06-09 16:22:11.622356, event: dispatched, op: pg_notify(2.573(20),3.2b3(10) epoch 243) 2014-06-09 16:22:11.622377 7f869af78700 5 -- op tracker -- , seq: 2097, time: 2014-06-09 16:22:11.622376, event: waiting_for_osdmap, op: pg_notify(2.573(20),3.2b3(10) epoch 243) 2014-06-09 16:22:11.622385 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.622387 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d9a40 2014-06-09 16:22:11.622392 7f869af78700 5 -- op tracker -- , seq: 2097, time: 2014-06-09 16:22:11.622390, event: started, op: pg_notify(2.573(20),3.2b3(10) epoch 243) 2014-06-09 16:22:11.622416 7f869af78700 5 -- op tracker -- , seq: 2097, time: 2014-06-09 16:22:11.622415, event: done, op: pg_notify(2.573(20),3.2b3(10) epoch 243) 2014-06-09 16:22:11.622421 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622437 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622438 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622439 7f869af78700 20 osd.26 243 _dispatch 0x240eb500 pg_notify(0.221(20),1.223(20),3.22d(10) epoch 243) v5 2014-06-09 16:22:11.622443 7f869af78700 5 -- op tracker -- , seq: 2098, time: 2014-06-09 16:22:11.351204, event: header_read, op: pg_notify(0.221(20),1.223(20),3.22d(10) epoch 243) 2014-06-09 16:22:11.622447 7f869af78700 5 -- op tracker -- , seq: 2098, time: 2014-06-09 16:22:11.351205, event: throttled, op: pg_notify(0.221(20),1.223(20),3.22d(10) epoch 243) 2014-06-09 16:22:11.622451 7f869af78700 5 -- op tracker -- , seq: 2098, time: 2014-06-09 16:22:11.351266, event: all_read, op: pg_notify(0.221(20),1.223(20),3.22d(10) epoch 243) 2014-06-09 16:22:11.622457 7f869af78700 5 -- op tracker -- , seq: 2098, time: 2014-06-09 16:22:11.622437, event: dispatched, op: pg_notify(0.221(20),1.223(20),3.22d(10) epoch 243) 2014-06-09 16:22:11.622461 7f869af78700 5 -- op tracker -- , seq: 2098, time: 2014-06-09 16:22:11.622459, event: waiting_for_osdmap, op: pg_notify(0.221(20),1.223(20),3.22d(10) epoch 243) 2014-06-09 16:22:11.622467 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.622468 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eb500 2014-06-09 16:22:11.622471 7f869af78700 5 -- op tracker -- , seq: 2098, time: 2014-06-09 16:22:11.622470, event: started, op: pg_notify(0.221(20),1.223(20),3.22d(10) epoch 243) 2014-06-09 16:22:11.622492 7f869af78700 5 -- op tracker -- , seq: 2098, time: 2014-06-09 16:22:11.622489, event: done, op: pg_notify(0.221(20),1.223(20),3.22d(10) epoch 243) 2014-06-09 16:22:11.622502 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622504 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622531 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622533 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622534 7f869af78700 20 osd.26 243 _dispatch 0x1ac94fc0 pg_notify(1.1b1(20) epoch 243) v5 2014-06-09 16:22:11.622539 7f869af78700 5 -- op tracker -- , seq: 2099, time: 2014-06-09 16:22:11.354327, event: header_read, op: pg_notify(1.1b1(20) epoch 243) 2014-06-09 16:22:11.622547 7f869af78700 5 -- op tracker -- , seq: 2099, time: 2014-06-09 16:22:11.354327, event: throttled, op: pg_notify(1.1b1(20) epoch 243) 2014-06-09 16:22:11.622551 7f869af78700 5 -- op tracker -- , seq: 2099, time: 2014-06-09 16:22:11.354358, event: all_read, op: pg_notify(1.1b1(20) epoch 243) 2014-06-09 16:22:11.622554 7f869af78700 5 -- op tracker -- , seq: 2099, time: 2014-06-09 16:22:11.622530, event: dispatched, op: pg_notify(1.1b1(20) epoch 243) 2014-06-09 16:22:11.622558 7f869af78700 5 -- op tracker -- , seq: 2099, time: 2014-06-09 16:22:11.622557, event: waiting_for_osdmap, op: pg_notify(1.1b1(20) epoch 243) 2014-06-09 16:22:11.622560 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.622561 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac94fc0 2014-06-09 16:22:11.622564 7f869af78700 5 -- op tracker -- , seq: 2099, time: 2014-06-09 16:22:11.622563, event: started, op: pg_notify(1.1b1(20) epoch 243) 2014-06-09 16:22:11.622578 7f869af78700 5 -- op tracker -- , seq: 2099, time: 2014-06-09 16:22:11.622577, event: done, op: pg_notify(1.1b1(20) epoch 243) 2014-06-09 16:22:11.622582 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622583 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622598 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622599 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622600 7f869af78700 20 osd.26 243 _dispatch 0x19fdf880 pg_info(1 pgs e243:0.66b) v4 2014-06-09 16:22:11.622603 7f869af78700 5 -- op tracker -- , seq: 2100, time: 2014-06-09 16:22:11.259555, event: header_read, op: pg_info(1 pgs e243:0.66b) 2014-06-09 16:22:11.622606 7f869af78700 5 -- op tracker -- , seq: 2100, time: 2014-06-09 16:22:11.259556, event: throttled, op: pg_info(1 pgs e243:0.66b) 2014-06-09 16:22:11.622610 7f869af78700 5 -- op tracker -- , seq: 2100, time: 2014-06-09 16:22:11.259564, event: all_read, op: pg_info(1 pgs e243:0.66b) 2014-06-09 16:22:11.622613 7f869af78700 5 -- op tracker -- , seq: 2100, time: 2014-06-09 16:22:11.622598, event: dispatched, op: pg_info(1 pgs e243:0.66b) 2014-06-09 16:22:11.622618 7f869af78700 5 -- op tracker -- , seq: 2100, time: 2014-06-09 16:22:11.622616, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.66b) 2014-06-09 16:22:11.622622 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.66b) v4 from osd.0 2014-06-09 16:22:11.622625 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdf880 2014-06-09 16:22:11.622630 7f869af78700 5 -- op tracker -- , seq: 2100, time: 2014-06-09 16:22:11.622628, event: started, op: pg_info(1 pgs e243:0.66b) 2014-06-09 16:22:11.622645 7f869af78700 5 -- op tracker -- , seq: 2100, time: 2014-06-09 16:22:11.622642, event: done, op: pg_info(1 pgs e243:0.66b) 2014-06-09 16:22:11.622651 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622652 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622663 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622665 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622666 7f869af78700 20 osd.26 243 _dispatch 0x1fc05880 pg_info(1 pgs e243:0.67d) v4 2014-06-09 16:22:11.622671 7f869af78700 5 -- op tracker -- , seq: 2101, time: 2014-06-09 16:22:11.381791, event: header_read, op: pg_info(1 pgs e243:0.67d) 2014-06-09 16:22:11.622677 7f869af78700 5 -- op tracker -- , seq: 2101, time: 2014-06-09 16:22:11.381792, event: throttled, op: pg_info(1 pgs e243:0.67d) 2014-06-09 16:22:11.622685 7f869af78700 5 -- op tracker -- , seq: 2101, time: 2014-06-09 16:22:11.381807, event: all_read, op: pg_info(1 pgs e243:0.67d) 2014-06-09 16:22:11.622689 7f869af78700 5 -- op tracker -- , seq: 2101, time: 2014-06-09 16:22:11.622662, event: dispatched, op: pg_info(1 pgs e243:0.67d) 2014-06-09 16:22:11.622696 7f869af78700 5 -- op tracker -- , seq: 2101, time: 2014-06-09 16:22:11.622695, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.67d) 2014-06-09 16:22:11.622699 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.67d) v4 from osd.5 2014-06-09 16:22:11.622701 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc05880 2014-06-09 16:22:11.622703 7f869af78700 5 -- op tracker -- , seq: 2101, time: 2014-06-09 16:22:11.622702, event: started, op: pg_info(1 pgs e243:0.67d) 2014-06-09 16:22:11.622713 7f869af78700 5 -- op tracker -- , seq: 2101, time: 2014-06-09 16:22:11.622712, event: done, op: pg_info(1 pgs e243:0.67d) 2014-06-09 16:22:11.622717 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622718 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622727 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622728 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622729 7f869af78700 20 osd.26 243 _dispatch 0x1c03ee00 pg_notify(3.582(12),0.2c2(21),1.591(26) epoch 243) v5 2014-06-09 16:22:11.622736 7f869af78700 5 -- op tracker -- , seq: 2102, time: 2014-06-09 16:22:11.185484, event: header_read, op: pg_notify(3.582(12),0.2c2(21),1.591(26) epoch 243) 2014-06-09 16:22:11.622746 7f869af78700 5 -- op tracker -- , seq: 2102, time: 2014-06-09 16:22:11.185484, event: throttled, op: pg_notify(3.582(12),0.2c2(21),1.591(26) epoch 243) 2014-06-09 16:22:11.622755 7f869af78700 5 -- op tracker -- , seq: 2102, time: 2014-06-09 16:22:11.185531, event: all_read, op: pg_notify(3.582(12),0.2c2(21),1.591(26) epoch 243) 2014-06-09 16:22:11.622765 7f869af78700 5 -- op tracker -- , seq: 2102, time: 2014-06-09 16:22:11.622726, event: dispatched, op: pg_notify(3.582(12),0.2c2(21),1.591(26) epoch 243) 2014-06-09 16:22:11.622774 7f869af78700 5 -- op tracker -- , seq: 2102, time: 2014-06-09 16:22:11.622771, event: waiting_for_osdmap, op: pg_notify(3.582(12),0.2c2(21),1.591(26) epoch 243) 2014-06-09 16:22:11.622784 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.622786 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03ee00 2014-06-09 16:22:11.622792 7f869af78700 5 -- op tracker -- , seq: 2102, time: 2014-06-09 16:22:11.622789, event: started, op: pg_notify(3.582(12),0.2c2(21),1.591(26) epoch 243) 2014-06-09 16:22:11.622821 7f869af78700 5 -- op tracker -- , seq: 2102, time: 2014-06-09 16:22:11.622819, event: done, op: pg_notify(3.582(12),0.2c2(21),1.591(26) epoch 243) 2014-06-09 16:22:11.622829 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622831 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622850 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622851 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622853 7f869af78700 20 osd.26 243 _dispatch 0x1fc04540 pg_notify(2.587(20),0.594(20) epoch 243) v5 2014-06-09 16:22:11.622858 7f869af78700 5 -- op tracker -- , seq: 2103, time: 2014-06-09 16:22:11.194506, event: header_read, op: pg_notify(2.587(20),0.594(20) epoch 243) 2014-06-09 16:22:11.622866 7f869af78700 5 -- op tracker -- , seq: 2103, time: 2014-06-09 16:22:11.194507, event: throttled, op: pg_notify(2.587(20),0.594(20) epoch 243) 2014-06-09 16:22:11.622873 7f869af78700 5 -- op tracker -- , seq: 2103, time: 2014-06-09 16:22:11.194544, event: all_read, op: pg_notify(2.587(20),0.594(20) epoch 243) 2014-06-09 16:22:11.622880 7f869af78700 5 -- op tracker -- , seq: 2103, time: 2014-06-09 16:22:11.622849, event: dispatched, op: pg_notify(2.587(20),0.594(20) epoch 243) 2014-06-09 16:22:11.622884 7f869af78700 5 -- op tracker -- , seq: 2103, time: 2014-06-09 16:22:11.622883, event: waiting_for_osdmap, op: pg_notify(2.587(20),0.594(20) epoch 243) 2014-06-09 16:22:11.622887 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.622888 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc04540 2014-06-09 16:22:11.622891 7f869af78700 5 -- op tracker -- , seq: 2103, time: 2014-06-09 16:22:11.622890, event: started, op: pg_notify(2.587(20),0.594(20) epoch 243) 2014-06-09 16:22:11.622904 7f869af78700 5 -- op tracker -- , seq: 2103, time: 2014-06-09 16:22:11.622903, event: done, op: pg_notify(2.587(20),0.594(20) epoch 243) 2014-06-09 16:22:11.622909 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622910 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622930 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.622931 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.622932 7f869af78700 20 osd.26 243 _dispatch 0x12060e00 pg_info(1 pgs e243:1.2fb) v4 2014-06-09 16:22:11.622937 7f869af78700 5 -- op tracker -- , seq: 2104, time: 2014-06-09 16:22:11.356772, event: header_read, op: pg_info(1 pgs e243:1.2fb) 2014-06-09 16:22:11.622944 7f869af78700 5 -- op tracker -- , seq: 2104, time: 2014-06-09 16:22:11.356773, event: throttled, op: pg_info(1 pgs e243:1.2fb) 2014-06-09 16:22:11.622953 7f869af78700 5 -- op tracker -- , seq: 2104, time: 2014-06-09 16:22:11.356796, event: all_read, op: pg_info(1 pgs e243:1.2fb) 2014-06-09 16:22:11.622959 7f869af78700 5 -- op tracker -- , seq: 2104, time: 2014-06-09 16:22:11.622929, event: dispatched, op: pg_info(1 pgs e243:1.2fb) 2014-06-09 16:22:11.622968 7f869af78700 5 -- op tracker -- , seq: 2104, time: 2014-06-09 16:22:11.622966, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2fb) 2014-06-09 16:22:11.622974 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2fb) v4 from osd.7 2014-06-09 16:22:11.622977 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12060e00 2014-06-09 16:22:11.622982 7f869af78700 5 -- op tracker -- , seq: 2104, time: 2014-06-09 16:22:11.622980, event: started, op: pg_info(1 pgs e243:1.2fb) 2014-06-09 16:22:11.622997 7f869af78700 5 -- op tracker -- , seq: 2104, time: 2014-06-09 16:22:11.622996, event: done, op: pg_info(1 pgs e243:1.2fb) 2014-06-09 16:22:11.623003 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623004 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623012 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623013 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623013 7f869af78700 20 osd.26 243 _dispatch 0x30d25340 pg_notify(2.75d(20),0.498(20) epoch 243) v5 2014-06-09 16:22:11.623017 7f869af78700 5 -- op tracker -- , seq: 2105, time: 2014-06-09 16:22:11.399874, event: header_read, op: pg_notify(2.75d(20),0.498(20) epoch 243) 2014-06-09 16:22:11.623021 7f869af78700 5 -- op tracker -- , seq: 2105, time: 2014-06-09 16:22:11.399875, event: throttled, op: pg_notify(2.75d(20),0.498(20) epoch 243) 2014-06-09 16:22:11.623024 7f869af78700 5 -- op tracker -- , seq: 2105, time: 2014-06-09 16:22:11.399923, event: all_read, op: pg_notify(2.75d(20),0.498(20) epoch 243) 2014-06-09 16:22:11.623028 7f869af78700 5 -- op tracker -- , seq: 2105, time: 2014-06-09 16:22:11.623012, event: dispatched, op: pg_notify(2.75d(20),0.498(20) epoch 243) 2014-06-09 16:22:11.623033 7f869af78700 5 -- op tracker -- , seq: 2105, time: 2014-06-09 16:22:11.623032, event: waiting_for_osdmap, op: pg_notify(2.75d(20),0.498(20) epoch 243) 2014-06-09 16:22:11.623035 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.623037 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d25340 2014-06-09 16:22:11.623039 7f869af78700 5 -- op tracker -- , seq: 2105, time: 2014-06-09 16:22:11.623038, event: started, op: pg_notify(2.75d(20),0.498(20) epoch 243) 2014-06-09 16:22:11.623052 7f869af78700 5 -- op tracker -- , seq: 2105, time: 2014-06-09 16:22:11.623051, event: done, op: pg_notify(2.75d(20),0.498(20) epoch 243) 2014-06-09 16:22:11.623057 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623057 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623079 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623080 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623082 7f869af78700 20 osd.26 243 _dispatch 0x307d5200 pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.623088 7f869af78700 5 -- op tracker -- , seq: 2106, time: 2014-06-09 16:22:11.255192, event: header_read, op: pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.623093 7f869af78700 5 -- op tracker -- , seq: 2106, time: 2014-06-09 16:22:11.255194, event: throttled, op: pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.623097 7f869af78700 5 -- op tracker -- , seq: 2106, time: 2014-06-09 16:22:11.256375, event: all_read, op: pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.623101 7f869af78700 5 -- op tracker -- , seq: 2106, time: 2014-06-09 16:22:11.623078, event: dispatched, op: pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.623112 7f869af78700 5 -- op tracker -- , seq: 2106, time: 2014-06-09 16:22:11.623109, event: waiting_for_osdmap, op: pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.623118 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) v4 from osd.1 2014-06-09 16:22:11.623122 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307d5200 2014-06-09 16:22:11.623127 7f869af78700 5 -- op tracker -- , seq: 2106, time: 2014-06-09 16:22:11.623125, event: started, op: pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.623138 7f869af78700 5 -- op tracker -- , seq: 2106, time: 2014-06-09 16:22:11.623136, event: done, op: pg_log(3.55d epoch 243 log log((0'0,213'582], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.623142 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623143 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623150 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623151 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623151 7f869af78700 20 osd.26 243 _dispatch 0x308d8a80 pg_info(1 pgs e243:2.573) v4 2014-06-09 16:22:11.623154 7f869af78700 5 -- op tracker -- , seq: 2107, time: 2014-06-09 16:22:11.319073, event: header_read, op: pg_info(1 pgs e243:2.573) 2014-06-09 16:22:11.623158 7f869af78700 5 -- op tracker -- , seq: 2107, time: 2014-06-09 16:22:11.319073, event: throttled, op: pg_info(1 pgs e243:2.573) 2014-06-09 16:22:11.623161 7f869af78700 5 -- op tracker -- , seq: 2107, time: 2014-06-09 16:22:11.319082, event: all_read, op: pg_info(1 pgs e243:2.573) 2014-06-09 16:22:11.623164 7f869af78700 5 -- op tracker -- , seq: 2107, time: 2014-06-09 16:22:11.623150, event: dispatched, op: pg_info(1 pgs e243:2.573) 2014-06-09 16:22:11.623172 7f869af78700 5 -- op tracker -- , seq: 2107, time: 2014-06-09 16:22:11.623170, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.573) 2014-06-09 16:22:11.623176 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.573) v4 from osd.9 2014-06-09 16:22:11.623179 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d8a80 2014-06-09 16:22:11.623184 7f869af78700 5 -- op tracker -- , seq: 2107, time: 2014-06-09 16:22:11.623182, event: started, op: pg_info(1 pgs e243:2.573) 2014-06-09 16:22:11.623200 7f869af78700 5 -- op tracker -- , seq: 2107, time: 2014-06-09 16:22:11.623198, event: done, op: pg_info(1 pgs e243:2.573) 2014-06-09 16:22:11.623208 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623210 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623222 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623224 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623225 7f869af78700 20 osd.26 243 _dispatch 0x308d1800 pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.623232 7f869af78700 5 -- op tracker -- , seq: 2108, time: 2014-06-09 16:22:11.355134, event: header_read, op: pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623239 7f869af78700 5 -- op tracker -- , seq: 2108, time: 2014-06-09 16:22:11.355135, event: throttled, op: pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623246 7f869af78700 5 -- op tracker -- , seq: 2108, time: 2014-06-09 16:22:11.355150, event: all_read, op: pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623255 7f869af78700 5 -- op tracker -- , seq: 2108, time: 2014-06-09 16:22:11.623222, event: dispatched, op: pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623260 7f869af78700 5 -- op tracker -- , seq: 2108, time: 2014-06-09 16:22:11.623259, event: waiting_for_osdmap, op: pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623263 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.623265 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d1800 2014-06-09 16:22:11.623268 7f869af78700 5 -- op tracker -- , seq: 2108, time: 2014-06-09 16:22:11.623267, event: started, op: pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623278 7f869af78700 5 -- op tracker -- , seq: 2108, time: 2014-06-09 16:22:11.623276, event: done, op: pg_log(2.48f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623282 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623283 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623291 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623292 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623293 7f869af78700 20 osd.26 243 _dispatch 0x1ac95500 pg_info(1 pgs e243:3.470(2)) v4 2014-06-09 16:22:11.623298 7f869af78700 5 -- op tracker -- , seq: 2109, time: 2014-06-09 16:22:11.354366, event: header_read, op: pg_info(1 pgs e243:3.470(2)) 2014-06-09 16:22:11.623305 7f869af78700 5 -- op tracker -- , seq: 2109, time: 2014-06-09 16:22:11.354367, event: throttled, op: pg_info(1 pgs e243:3.470(2)) 2014-06-09 16:22:11.623312 7f869af78700 5 -- op tracker -- , seq: 2109, time: 2014-06-09 16:22:11.354387, event: all_read, op: pg_info(1 pgs e243:3.470(2)) 2014-06-09 16:22:11.623318 7f869af78700 5 -- op tracker -- , seq: 2109, time: 2014-06-09 16:22:11.623290, event: dispatched, op: pg_info(1 pgs e243:3.470(2)) 2014-06-09 16:22:11.623322 7f869af78700 5 -- op tracker -- , seq: 2109, time: 2014-06-09 16:22:11.623321, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.470(2)) 2014-06-09 16:22:11.623325 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.470(2)) v4 from osd.6 2014-06-09 16:22:11.623327 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac95500 2014-06-09 16:22:11.623329 7f869af78700 5 -- op tracker -- , seq: 2109, time: 2014-06-09 16:22:11.623328, event: started, op: pg_info(1 pgs e243:3.470(2)) 2014-06-09 16:22:11.623343 7f869af78700 5 -- op tracker -- , seq: 2109, time: 2014-06-09 16:22:11.623342, event: done, op: pg_info(1 pgs e243:3.470(2)) 2014-06-09 16:22:11.623347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623348 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623356 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623357 7f869af78700 20 osd.26 243 _dispatch 0x22ac16c0 pg_info(1 pgs e243:1.f2) v4 2014-06-09 16:22:11.623360 7f869af78700 5 -- op tracker -- , seq: 2110, time: 2014-06-09 16:22:11.259581, event: header_read, op: pg_info(1 pgs e243:1.f2) 2014-06-09 16:22:11.623365 7f869af78700 5 -- op tracker -- , seq: 2110, time: 2014-06-09 16:22:11.259582, event: throttled, op: pg_info(1 pgs e243:1.f2) 2014-06-09 16:22:11.623373 7f869af78700 5 -- op tracker -- , seq: 2110, time: 2014-06-09 16:22:11.259590, event: all_read, op: pg_info(1 pgs e243:1.f2) 2014-06-09 16:22:11.623380 7f869af78700 5 -- op tracker -- , seq: 2110, time: 2014-06-09 16:22:11.623355, event: dispatched, op: pg_info(1 pgs e243:1.f2) 2014-06-09 16:22:11.623388 7f869af78700 5 -- op tracker -- , seq: 2110, time: 2014-06-09 16:22:11.623386, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.f2) 2014-06-09 16:22:11.623391 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.f2) v4 from osd.0 2014-06-09 16:22:11.623394 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22ac16c0 2014-06-09 16:22:11.623399 7f869af78700 5 -- op tracker -- , seq: 2110, time: 2014-06-09 16:22:11.623397, event: started, op: pg_info(1 pgs e243:1.f2) 2014-06-09 16:22:11.623417 7f869af78700 5 -- op tracker -- , seq: 2110, time: 2014-06-09 16:22:11.623414, event: done, op: pg_info(1 pgs e243:1.f2) 2014-06-09 16:22:11.623424 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623425 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623432 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623433 7f869af78700 20 osd.26 243 _dispatch 0x1fc05a40 pg_info(1 pgs e243:3.310) v4 2014-06-09 16:22:11.623436 7f869af78700 5 -- op tracker -- , seq: 2111, time: 2014-06-09 16:22:11.381873, event: header_read, op: pg_info(1 pgs e243:3.310) 2014-06-09 16:22:11.623440 7f869af78700 5 -- op tracker -- , seq: 2111, time: 2014-06-09 16:22:11.381874, event: throttled, op: pg_info(1 pgs e243:3.310) 2014-06-09 16:22:11.623443 7f869af78700 5 -- op tracker -- , seq: 2111, time: 2014-06-09 16:22:11.381887, event: all_read, op: pg_info(1 pgs e243:3.310) 2014-06-09 16:22:11.623447 7f869af78700 5 -- op tracker -- , seq: 2111, time: 2014-06-09 16:22:11.623431, event: dispatched, op: pg_info(1 pgs e243:3.310) 2014-06-09 16:22:11.623451 7f869af78700 5 -- op tracker -- , seq: 2111, time: 2014-06-09 16:22:11.623450, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.310) 2014-06-09 16:22:11.623454 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.310) v4 from osd.5 2014-06-09 16:22:11.623456 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc05a40 2014-06-09 16:22:11.623458 7f869af78700 5 -- op tracker -- , seq: 2111, time: 2014-06-09 16:22:11.623457, event: started, op: pg_info(1 pgs e243:3.310) 2014-06-09 16:22:11.623469 7f869af78700 5 -- op tracker -- , seq: 2111, time: 2014-06-09 16:22:11.623468, event: done, op: pg_info(1 pgs e243:3.310) 2014-06-09 16:22:11.623473 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623474 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623482 7f869af78700 20 osd.26 243 _dispatch 0x1a0388c0 pg_info(1 pgs e243:3.58f(4)) v4 2014-06-09 16:22:11.623484 7f869af78700 5 -- op tracker -- , seq: 2112, time: 2014-06-09 16:22:11.185536, event: header_read, op: pg_info(1 pgs e243:3.58f(4)) 2014-06-09 16:22:11.623489 7f869af78700 5 -- op tracker -- , seq: 2112, time: 2014-06-09 16:22:11.185536, event: throttled, op: pg_info(1 pgs e243:3.58f(4)) 2014-06-09 16:22:11.623493 7f869af78700 5 -- op tracker -- , seq: 2112, time: 2014-06-09 16:22:11.185546, event: all_read, op: pg_info(1 pgs e243:3.58f(4)) 2014-06-09 16:22:11.623497 7f869af78700 5 -- op tracker -- , seq: 2112, time: 2014-06-09 16:22:11.623480, event: dispatched, op: pg_info(1 pgs e243:3.58f(4)) 2014-06-09 16:22:11.623501 7f869af78700 5 -- op tracker -- , seq: 2112, time: 2014-06-09 16:22:11.623500, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.58f(4)) 2014-06-09 16:22:11.623507 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.58f(4)) v4 from osd.2 2014-06-09 16:22:11.623509 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a0388c0 2014-06-09 16:22:11.623514 7f869af78700 5 -- op tracker -- , seq: 2112, time: 2014-06-09 16:22:11.623512, event: started, op: pg_info(1 pgs e243:3.58f(4)) 2014-06-09 16:22:11.623530 7f869af78700 5 -- op tracker -- , seq: 2112, time: 2014-06-09 16:22:11.623528, event: done, op: pg_info(1 pgs e243:3.58f(4)) 2014-06-09 16:22:11.623539 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623540 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623549 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623550 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623550 7f869af78700 20 osd.26 243 _dispatch 0x306e9600 pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.623554 7f869af78700 5 -- op tracker -- , seq: 2113, time: 2014-06-09 16:22:11.194556, event: header_read, op: pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623566 7f869af78700 5 -- op tracker -- , seq: 2113, time: 2014-06-09 16:22:11.194557, event: throttled, op: pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623577 7f869af78700 5 -- op tracker -- , seq: 2113, time: 2014-06-09 16:22:11.194571, event: all_read, op: pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623583 7f869af78700 5 -- op tracker -- , seq: 2113, time: 2014-06-09 16:22:11.623549, event: dispatched, op: pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623587 7f869af78700 5 -- op tracker -- , seq: 2113, time: 2014-06-09 16:22:11.623585, event: waiting_for_osdmap, op: pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623589 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.623592 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306e9600 2014-06-09 16:22:11.623595 7f869af78700 5 -- op tracker -- , seq: 2113, time: 2014-06-09 16:22:11.623593, event: started, op: pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623604 7f869af78700 5 -- op tracker -- , seq: 2113, time: 2014-06-09 16:22:11.623602, event: done, op: pg_log(1.2f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623609 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623610 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623634 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623635 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623636 7f869af78700 20 osd.26 243 _dispatch 0x12060fc0 pg_info(1 pgs e243:3.de) v4 2014-06-09 16:22:11.623640 7f869af78700 5 -- op tracker -- , seq: 2114, time: 2014-06-09 16:22:11.359014, event: header_read, op: pg_info(1 pgs e243:3.de) 2014-06-09 16:22:11.623644 7f869af78700 5 -- op tracker -- , seq: 2114, time: 2014-06-09 16:22:11.359015, event: throttled, op: pg_info(1 pgs e243:3.de) 2014-06-09 16:22:11.623648 7f869af78700 5 -- op tracker -- , seq: 2114, time: 2014-06-09 16:22:11.359032, event: all_read, op: pg_info(1 pgs e243:3.de) 2014-06-09 16:22:11.623652 7f869af78700 5 -- op tracker -- , seq: 2114, time: 2014-06-09 16:22:11.623633, event: dispatched, op: pg_info(1 pgs e243:3.de) 2014-06-09 16:22:11.623655 7f869af78700 5 -- op tracker -- , seq: 2114, time: 2014-06-09 16:22:11.623654, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.de) 2014-06-09 16:22:11.623658 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.de) v4 from osd.7 2014-06-09 16:22:11.623659 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12060fc0 2014-06-09 16:22:11.623662 7f869af78700 5 -- op tracker -- , seq: 2114, time: 2014-06-09 16:22:11.623661, event: started, op: pg_info(1 pgs e243:3.de) 2014-06-09 16:22:11.623672 7f869af78700 5 -- op tracker -- , seq: 2114, time: 2014-06-09 16:22:11.623671, event: done, op: pg_info(1 pgs e243:3.de) 2014-06-09 16:22:11.623676 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623677 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623686 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623687 7f869af78700 20 osd.26 243 _dispatch 0x304dd800 pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.623691 7f869af78700 5 -- op tracker -- , seq: 2115, time: 2014-06-09 16:22:11.399932, event: header_read, op: pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623695 7f869af78700 5 -- op tracker -- , seq: 2115, time: 2014-06-09 16:22:11.399932, event: throttled, op: pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623699 7f869af78700 5 -- op tracker -- , seq: 2115, time: 2014-06-09 16:22:11.399943, event: all_read, op: pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623703 7f869af78700 5 -- op tracker -- , seq: 2115, time: 2014-06-09 16:22:11.623686, event: dispatched, op: pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623706 7f869af78700 5 -- op tracker -- , seq: 2115, time: 2014-06-09 16:22:11.623705, event: waiting_for_osdmap, op: pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623709 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.623711 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304dd800 2014-06-09 16:22:11.623714 7f869af78700 5 -- op tracker -- , seq: 2115, time: 2014-06-09 16:22:11.623713, event: started, op: pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623725 7f869af78700 5 -- op tracker -- , seq: 2115, time: 2014-06-09 16:22:11.623724, event: done, op: pg_log(2.1e3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623732 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623733 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623740 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623741 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623742 7f869af78700 20 osd.26 243 _dispatch 0x200401c0 pg_notify(1.446(21),3.709(12) epoch 243) v5 2014-06-09 16:22:11.623745 7f869af78700 5 -- op tracker -- , seq: 2116, time: 2014-06-09 16:22:11.256407, event: header_read, op: pg_notify(1.446(21),3.709(12) epoch 243) 2014-06-09 16:22:11.623753 7f869af78700 5 -- op tracker -- , seq: 2116, time: 2014-06-09 16:22:11.256408, event: throttled, op: pg_notify(1.446(21),3.709(12) epoch 243) 2014-06-09 16:22:11.623758 7f869af78700 5 -- op tracker -- , seq: 2116, time: 2014-06-09 16:22:11.256453, event: all_read, op: pg_notify(1.446(21),3.709(12) epoch 243) 2014-06-09 16:22:11.623762 7f869af78700 5 -- op tracker -- , seq: 2116, time: 2014-06-09 16:22:11.623740, event: dispatched, op: pg_notify(1.446(21),3.709(12) epoch 243) 2014-06-09 16:22:11.623769 7f869af78700 5 -- op tracker -- , seq: 2116, time: 2014-06-09 16:22:11.623767, event: waiting_for_osdmap, op: pg_notify(1.446(21),3.709(12) epoch 243) 2014-06-09 16:22:11.623771 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.623772 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x200401c0 2014-06-09 16:22:11.623775 7f869af78700 5 -- op tracker -- , seq: 2116, time: 2014-06-09 16:22:11.623774, event: started, op: pg_notify(1.446(21),3.709(12) epoch 243) 2014-06-09 16:22:11.623789 7f869af78700 5 -- op tracker -- , seq: 2116, time: 2014-06-09 16:22:11.623787, event: done, op: pg_notify(1.446(21),3.709(12) epoch 243) 2014-06-09 16:22:11.623793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623794 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623807 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623808 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623808 7f869af78700 20 osd.26 243 _dispatch 0xeffb340 pg_info(1 pgs e243:3.2b3) v4 2014-06-09 16:22:11.623811 7f869af78700 5 -- op tracker -- , seq: 2117, time: 2014-06-09 16:22:11.319137, event: header_read, op: pg_info(1 pgs e243:3.2b3) 2014-06-09 16:22:11.623815 7f869af78700 5 -- op tracker -- , seq: 2117, time: 2014-06-09 16:22:11.319137, event: throttled, op: pg_info(1 pgs e243:3.2b3) 2014-06-09 16:22:11.623818 7f869af78700 5 -- op tracker -- , seq: 2117, time: 2014-06-09 16:22:11.319144, event: all_read, op: pg_info(1 pgs e243:3.2b3) 2014-06-09 16:22:11.623822 7f869af78700 5 -- op tracker -- , seq: 2117, time: 2014-06-09 16:22:11.623807, event: dispatched, op: pg_info(1 pgs e243:3.2b3) 2014-06-09 16:22:11.623825 7f869af78700 5 -- op tracker -- , seq: 2117, time: 2014-06-09 16:22:11.623824, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2b3) 2014-06-09 16:22:11.623828 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2b3) v4 from osd.9 2014-06-09 16:22:11.623829 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xeffb340 2014-06-09 16:22:11.623832 7f869af78700 5 -- op tracker -- , seq: 2117, time: 2014-06-09 16:22:11.623831, event: started, op: pg_info(1 pgs e243:3.2b3) 2014-06-09 16:22:11.623840 7f869af78700 5 -- op tracker -- , seq: 2117, time: 2014-06-09 16:22:11.623839, event: done, op: pg_info(1 pgs e243:3.2b3) 2014-06-09 16:22:11.623844 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623845 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623851 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623852 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623853 7f869af78700 20 osd.26 243 _dispatch 0x1c03f340 pg_notify(3.5d1(10) epoch 243) v5 2014-06-09 16:22:11.623856 7f869af78700 5 -- op tracker -- , seq: 2118, time: 2014-06-09 16:22:11.355193, event: header_read, op: pg_notify(3.5d1(10) epoch 243) 2014-06-09 16:22:11.623859 7f869af78700 5 -- op tracker -- , seq: 2118, time: 2014-06-09 16:22:11.355193, event: throttled, op: pg_notify(3.5d1(10) epoch 243) 2014-06-09 16:22:11.623863 7f869af78700 5 -- op tracker -- , seq: 2118, time: 2014-06-09 16:22:11.355218, event: all_read, op: pg_notify(3.5d1(10) epoch 243) 2014-06-09 16:22:11.623866 7f869af78700 5 -- op tracker -- , seq: 2118, time: 2014-06-09 16:22:11.623851, event: dispatched, op: pg_notify(3.5d1(10) epoch 243) 2014-06-09 16:22:11.623871 7f869af78700 5 -- op tracker -- , seq: 2118, time: 2014-06-09 16:22:11.623870, event: waiting_for_osdmap, op: pg_notify(3.5d1(10) epoch 243) 2014-06-09 16:22:11.623873 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.623874 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03f340 2014-06-09 16:22:11.623877 7f869af78700 5 -- op tracker -- , seq: 2118, time: 2014-06-09 16:22:11.623876, event: started, op: pg_notify(3.5d1(10) epoch 243) 2014-06-09 16:22:11.623886 7f869af78700 5 -- op tracker -- , seq: 2118, time: 2014-06-09 16:22:11.623885, event: done, op: pg_notify(3.5d1(10) epoch 243) 2014-06-09 16:22:11.623891 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623892 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623902 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623902 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623903 7f869af78700 20 osd.26 243 _dispatch 0x30888000 pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.623907 7f869af78700 5 -- op tracker -- , seq: 2119, time: 2014-06-09 16:22:11.356209, event: header_read, op: pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623911 7f869af78700 5 -- op tracker -- , seq: 2119, time: 2014-06-09 16:22:11.356210, event: throttled, op: pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623914 7f869af78700 5 -- op tracker -- , seq: 2119, time: 2014-06-09 16:22:11.356224, event: all_read, op: pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623918 7f869af78700 5 -- op tracker -- , seq: 2119, time: 2014-06-09 16:22:11.623901, event: dispatched, op: pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623922 7f869af78700 5 -- op tracker -- , seq: 2119, time: 2014-06-09 16:22:11.623921, event: waiting_for_osdmap, op: pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623928 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.623931 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30888000 2014-06-09 16:22:11.623934 7f869af78700 5 -- op tracker -- , seq: 2119, time: 2014-06-09 16:22:11.623932, event: started, op: pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623941 7f869af78700 5 -- op tracker -- , seq: 2119, time: 2014-06-09 16:22:11.623939, event: done, op: pg_log(2.213 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.623945 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623946 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623953 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.623954 7f869af78700 20 osd.26 243 _dispatch 0x308e3dc0 pg_info(1 pgs e243:0.ef) v4 2014-06-09 16:22:11.623957 7f869af78700 5 -- op tracker -- , seq: 2120, time: 2014-06-09 16:22:11.259596, event: header_read, op: pg_info(1 pgs e243:0.ef) 2014-06-09 16:22:11.623961 7f869af78700 5 -- op tracker -- , seq: 2120, time: 2014-06-09 16:22:11.259596, event: throttled, op: pg_info(1 pgs e243:0.ef) 2014-06-09 16:22:11.623965 7f869af78700 5 -- op tracker -- , seq: 2120, time: 2014-06-09 16:22:11.259610, event: all_read, op: pg_info(1 pgs e243:0.ef) 2014-06-09 16:22:11.623968 7f869af78700 5 -- op tracker -- , seq: 2120, time: 2014-06-09 16:22:11.623953, event: dispatched, op: pg_info(1 pgs e243:0.ef) 2014-06-09 16:22:11.623975 7f869af78700 5 -- op tracker -- , seq: 2120, time: 2014-06-09 16:22:11.623974, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.ef) 2014-06-09 16:22:11.623978 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.ef) v4 from osd.0 2014-06-09 16:22:11.623979 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e3dc0 2014-06-09 16:22:11.623982 7f869af78700 5 -- op tracker -- , seq: 2120, time: 2014-06-09 16:22:11.623981, event: started, op: pg_info(1 pgs e243:0.ef) 2014-06-09 16:22:11.623990 7f869af78700 5 -- op tracker -- , seq: 2120, time: 2014-06-09 16:22:11.623989, event: done, op: pg_info(1 pgs e243:0.ef) 2014-06-09 16:22:11.623994 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.623995 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624002 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624003 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624003 7f869af78700 20 osd.26 243 _dispatch 0x1fc05c00 pg_info(1 pgs e243:3.4d) v4 2014-06-09 16:22:11.624006 7f869af78700 5 -- op tracker -- , seq: 2121, time: 2014-06-09 16:22:11.381894, event: header_read, op: pg_info(1 pgs e243:3.4d) 2014-06-09 16:22:11.624010 7f869af78700 5 -- op tracker -- , seq: 2121, time: 2014-06-09 16:22:11.381895, event: throttled, op: pg_info(1 pgs e243:3.4d) 2014-06-09 16:22:11.624014 7f869af78700 5 -- op tracker -- , seq: 2121, time: 2014-06-09 16:22:11.381906, event: all_read, op: pg_info(1 pgs e243:3.4d) 2014-06-09 16:22:11.624017 7f869af78700 5 -- op tracker -- , seq: 2121, time: 2014-06-09 16:22:11.624002, event: dispatched, op: pg_info(1 pgs e243:3.4d) 2014-06-09 16:22:11.624024 7f869af78700 5 -- op tracker -- , seq: 2121, time: 2014-06-09 16:22:11.624023, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4d) 2014-06-09 16:22:11.624029 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4d) v4 from osd.5 2014-06-09 16:22:11.624031 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc05c00 2014-06-09 16:22:11.624034 7f869af78700 5 -- op tracker -- , seq: 2121, time: 2014-06-09 16:22:11.624033, event: started, op: pg_info(1 pgs e243:3.4d) 2014-06-09 16:22:11.624043 7f869af78700 5 -- op tracker -- , seq: 2121, time: 2014-06-09 16:22:11.624041, event: done, op: pg_info(1 pgs e243:3.4d) 2014-06-09 16:22:11.624047 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624047 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624054 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624055 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624056 7f869af78700 20 osd.26 243 _dispatch 0x2fe70fc0 pg_notify(2.5ad(22) epoch 243) v5 2014-06-09 16:22:11.624058 7f869af78700 5 -- op tracker -- , seq: 2122, time: 2014-06-09 16:22:11.185550, event: header_read, op: pg_notify(2.5ad(22) epoch 243) 2014-06-09 16:22:11.624062 7f869af78700 5 -- op tracker -- , seq: 2122, time: 2014-06-09 16:22:11.185550, event: throttled, op: pg_notify(2.5ad(22) epoch 243) 2014-06-09 16:22:11.624069 7f869af78700 5 -- op tracker -- , seq: 2122, time: 2014-06-09 16:22:11.185567, event: all_read, op: pg_notify(2.5ad(22) epoch 243) 2014-06-09 16:22:11.624072 7f869af78700 5 -- op tracker -- , seq: 2122, time: 2014-06-09 16:22:11.624054, event: dispatched, op: pg_notify(2.5ad(22) epoch 243) 2014-06-09 16:22:11.624077 7f869af78700 5 -- op tracker -- , seq: 2122, time: 2014-06-09 16:22:11.624076, event: waiting_for_osdmap, op: pg_notify(2.5ad(22) epoch 243) 2014-06-09 16:22:11.624080 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.624081 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe70fc0 2014-06-09 16:22:11.624083 7f869af78700 5 -- op tracker -- , seq: 2122, time: 2014-06-09 16:22:11.624082, event: started, op: pg_notify(2.5ad(22) epoch 243) 2014-06-09 16:22:11.624092 7f869af78700 5 -- op tracker -- , seq: 2122, time: 2014-06-09 16:22:11.624091, event: done, op: pg_notify(2.5ad(22) epoch 243) 2014-06-09 16:22:11.624096 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624097 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624108 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624109 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624109 7f869af78700 20 osd.26 243 _dispatch 0x3004d800 pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.624113 7f869af78700 5 -- op tracker -- , seq: 2123, time: 2014-06-09 16:22:11.194590, event: header_read, op: pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624117 7f869af78700 5 -- op tracker -- , seq: 2123, time: 2014-06-09 16:22:11.194590, event: throttled, op: pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624121 7f869af78700 5 -- op tracker -- , seq: 2123, time: 2014-06-09 16:22:11.194598, event: all_read, op: pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624124 7f869af78700 5 -- op tracker -- , seq: 2123, time: 2014-06-09 16:22:11.624108, event: dispatched, op: pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624130 7f869af78700 5 -- op tracker -- , seq: 2123, time: 2014-06-09 16:22:11.624128, event: waiting_for_osdmap, op: pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624132 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.624134 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3004d800 2014-06-09 16:22:11.624137 7f869af78700 5 -- op tracker -- , seq: 2123, time: 2014-06-09 16:22:11.624136, event: started, op: pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624145 7f869af78700 5 -- op tracker -- , seq: 2123, time: 2014-06-09 16:22:11.624144, event: done, op: pg_log(0.5c3 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624153 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624154 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624160 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624161 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624161 7f869af78700 20 osd.26 243 _dispatch 0x1d4c4700 pg_info(1 pgs e243:1.2b0) v4 2014-06-09 16:22:11.624164 7f869af78700 5 -- op tracker -- , seq: 2124, time: 2014-06-09 16:22:11.359041, event: header_read, op: pg_info(1 pgs e243:1.2b0) 2014-06-09 16:22:11.624168 7f869af78700 5 -- op tracker -- , seq: 2124, time: 2014-06-09 16:22:11.359041, event: throttled, op: pg_info(1 pgs e243:1.2b0) 2014-06-09 16:22:11.624171 7f869af78700 5 -- op tracker -- , seq: 2124, time: 2014-06-09 16:22:11.359054, event: all_read, op: pg_info(1 pgs e243:1.2b0) 2014-06-09 16:22:11.624175 7f869af78700 5 -- op tracker -- , seq: 2124, time: 2014-06-09 16:22:11.624160, event: dispatched, op: pg_info(1 pgs e243:1.2b0) 2014-06-09 16:22:11.624179 7f869af78700 5 -- op tracker -- , seq: 2124, time: 2014-06-09 16:22:11.624178, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2b0) 2014-06-09 16:22:11.624182 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2b0) v4 from osd.7 2014-06-09 16:22:11.624183 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d4c4700 2014-06-09 16:22:11.624186 7f869af78700 5 -- op tracker -- , seq: 2124, time: 2014-06-09 16:22:11.624185, event: started, op: pg_info(1 pgs e243:1.2b0) 2014-06-09 16:22:11.624194 7f869af78700 5 -- op tracker -- , seq: 2124, time: 2014-06-09 16:22:11.624193, event: done, op: pg_info(1 pgs e243:1.2b0) 2014-06-09 16:22:11.624198 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624199 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624206 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624206 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624207 7f869af78700 20 osd.26 243 _dispatch 0x14050700 pg_notify(1.1e4(20),2.777(20) epoch 243) v5 2014-06-09 16:22:11.624210 7f869af78700 5 -- op tracker -- , seq: 2125, time: 2014-06-09 16:22:11.399947, event: header_read, op: pg_notify(1.1e4(20),2.777(20) epoch 243) 2014-06-09 16:22:11.624214 7f869af78700 5 -- op tracker -- , seq: 2125, time: 2014-06-09 16:22:11.399948, event: throttled, op: pg_notify(1.1e4(20),2.777(20) epoch 243) 2014-06-09 16:22:11.624217 7f869af78700 5 -- op tracker -- , seq: 2125, time: 2014-06-09 16:22:11.399995, event: all_read, op: pg_notify(1.1e4(20),2.777(20) epoch 243) 2014-06-09 16:22:11.624221 7f869af78700 5 -- op tracker -- , seq: 2125, time: 2014-06-09 16:22:11.624205, event: dispatched, op: pg_notify(1.1e4(20),2.777(20) epoch 243) 2014-06-09 16:22:11.624225 7f869af78700 5 -- op tracker -- , seq: 2125, time: 2014-06-09 16:22:11.624223, event: waiting_for_osdmap, op: pg_notify(1.1e4(20),2.777(20) epoch 243) 2014-06-09 16:22:11.624227 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.624228 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x14050700 2014-06-09 16:22:11.624231 7f869af78700 5 -- op tracker -- , seq: 2125, time: 2014-06-09 16:22:11.624229, event: started, op: pg_notify(1.1e4(20),2.777(20) epoch 243) 2014-06-09 16:22:11.624244 7f869af78700 5 -- op tracker -- , seq: 2125, time: 2014-06-09 16:22:11.624243, event: done, op: pg_notify(1.1e4(20),2.777(20) epoch 243) 2014-06-09 16:22:11.624248 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624249 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624263 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624263 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624264 7f869af78700 20 osd.26 243 _dispatch 0x240eb340 pg_info(1 pgs e243:3.172(4)) v4 2014-06-09 16:22:11.624267 7f869af78700 5 -- op tracker -- , seq: 2126, time: 2014-06-09 16:22:11.256468, event: header_read, op: pg_info(1 pgs e243:3.172(4)) 2014-06-09 16:22:11.624271 7f869af78700 5 -- op tracker -- , seq: 2126, time: 2014-06-09 16:22:11.256469, event: throttled, op: pg_info(1 pgs e243:3.172(4)) 2014-06-09 16:22:11.624274 7f869af78700 5 -- op tracker -- , seq: 2126, time: 2014-06-09 16:22:11.256485, event: all_read, op: pg_info(1 pgs e243:3.172(4)) 2014-06-09 16:22:11.624281 7f869af78700 5 -- op tracker -- , seq: 2126, time: 2014-06-09 16:22:11.624262, event: dispatched, op: pg_info(1 pgs e243:3.172(4)) 2014-06-09 16:22:11.624285 7f869af78700 5 -- op tracker -- , seq: 2126, time: 2014-06-09 16:22:11.624284, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.172(4)) 2014-06-09 16:22:11.624290 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.172(4)) v4 from osd.1 2014-06-09 16:22:11.624292 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eb340 2014-06-09 16:22:11.624295 7f869af78700 5 -- op tracker -- , seq: 2126, time: 2014-06-09 16:22:11.624293, event: started, op: pg_info(1 pgs e243:3.172(4)) 2014-06-09 16:22:11.624304 7f869af78700 5 -- op tracker -- , seq: 2126, time: 2014-06-09 16:22:11.624303, event: done, op: pg_info(1 pgs e243:3.172(4)) 2014-06-09 16:22:11.624308 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624309 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624317 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624318 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624318 7f869af78700 20 osd.26 243 _dispatch 0x3005a8c0 pg_notify(2.2ef(20) epoch 243) v5 2014-06-09 16:22:11.624321 7f869af78700 5 -- op tracker -- , seq: 2127, time: 2014-06-09 16:22:11.319431, event: header_read, op: pg_notify(2.2ef(20) epoch 243) 2014-06-09 16:22:11.624325 7f869af78700 5 -- op tracker -- , seq: 2127, time: 2014-06-09 16:22:11.319431, event: throttled, op: pg_notify(2.2ef(20) epoch 243) 2014-06-09 16:22:11.624331 7f869af78700 5 -- op tracker -- , seq: 2127, time: 2014-06-09 16:22:11.319449, event: all_read, op: pg_notify(2.2ef(20) epoch 243) 2014-06-09 16:22:11.624334 7f869af78700 5 -- op tracker -- , seq: 2127, time: 2014-06-09 16:22:11.624317, event: dispatched, op: pg_notify(2.2ef(20) epoch 243) 2014-06-09 16:22:11.624338 7f869af78700 5 -- op tracker -- , seq: 2127, time: 2014-06-09 16:22:11.624337, event: waiting_for_osdmap, op: pg_notify(2.2ef(20) epoch 243) 2014-06-09 16:22:11.624340 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.624341 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005a8c0 2014-06-09 16:22:11.624344 7f869af78700 5 -- op tracker -- , seq: 2127, time: 2014-06-09 16:22:11.624343, event: started, op: pg_notify(2.2ef(20) epoch 243) 2014-06-09 16:22:11.624356 7f869af78700 5 -- op tracker -- , seq: 2127, time: 2014-06-09 16:22:11.624355, event: done, op: pg_notify(2.2ef(20) epoch 243) 2014-06-09 16:22:11.624360 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624361 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624371 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624371 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624372 7f869af78700 20 osd.26 243 _dispatch 0x1c03f500 pg_info(1 pgs e243:1.62a) v4 2014-06-09 16:22:11.624375 7f869af78700 5 -- op tracker -- , seq: 2128, time: 2014-06-09 16:22:11.361827, event: header_read, op: pg_info(1 pgs e243:1.62a) 2014-06-09 16:22:11.624378 7f869af78700 5 -- op tracker -- , seq: 2128, time: 2014-06-09 16:22:11.361828, event: throttled, op: pg_info(1 pgs e243:1.62a) 2014-06-09 16:22:11.624381 7f869af78700 5 -- op tracker -- , seq: 2128, time: 2014-06-09 16:22:11.361844, event: all_read, op: pg_info(1 pgs e243:1.62a) 2014-06-09 16:22:11.624385 7f869af78700 5 -- op tracker -- , seq: 2128, time: 2014-06-09 16:22:11.624370, event: dispatched, op: pg_info(1 pgs e243:1.62a) 2014-06-09 16:22:11.624388 7f869af78700 5 -- op tracker -- , seq: 2128, time: 2014-06-09 16:22:11.624387, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.62a) 2014-06-09 16:22:11.624390 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.62a) v4 from osd.8 2014-06-09 16:22:11.624392 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03f500 2014-06-09 16:22:11.624394 7f869af78700 5 -- op tracker -- , seq: 2128, time: 2014-06-09 16:22:11.624393, event: started, op: pg_info(1 pgs e243:1.62a) 2014-06-09 16:22:11.624403 7f869af78700 5 -- op tracker -- , seq: 2128, time: 2014-06-09 16:22:11.624402, event: done, op: pg_info(1 pgs e243:1.62a) 2014-06-09 16:22:11.624409 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624410 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624416 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624417 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624418 7f869af78700 20 osd.26 243 _dispatch 0x1609d500 pg_notify(1.207(20) epoch 243) v5 2014-06-09 16:22:11.624420 7f869af78700 5 -- op tracker -- , seq: 2129, time: 2014-06-09 16:22:11.356272, event: header_read, op: pg_notify(1.207(20) epoch 243) 2014-06-09 16:22:11.624424 7f869af78700 5 -- op tracker -- , seq: 2129, time: 2014-06-09 16:22:11.356273, event: throttled, op: pg_notify(1.207(20) epoch 243) 2014-06-09 16:22:11.624427 7f869af78700 5 -- op tracker -- , seq: 2129, time: 2014-06-09 16:22:11.356305, event: all_read, op: pg_notify(1.207(20) epoch 243) 2014-06-09 16:22:11.624432 7f869af78700 5 -- op tracker -- , seq: 2129, time: 2014-06-09 16:22:11.624416, event: dispatched, op: pg_notify(1.207(20) epoch 243) 2014-06-09 16:22:11.624436 7f869af78700 5 -- op tracker -- , seq: 2129, time: 2014-06-09 16:22:11.624435, event: waiting_for_osdmap, op: pg_notify(1.207(20) epoch 243) 2014-06-09 16:22:11.624438 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.624439 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1609d500 2014-06-09 16:22:11.624442 7f869af78700 5 -- op tracker -- , seq: 2129, time: 2014-06-09 16:22:11.624441, event: started, op: pg_notify(1.207(20) epoch 243) 2014-06-09 16:22:11.624450 7f869af78700 5 -- op tracker -- , seq: 2129, time: 2014-06-09 16:22:11.624449, event: done, op: pg_notify(1.207(20) epoch 243) 2014-06-09 16:22:11.624454 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624455 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624465 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624465 7f869af78700 20 osd.26 243 _dispatch 0x308e3c00 pg_info(1 pgs e243:2.288) v4 2014-06-09 16:22:11.624468 7f869af78700 5 -- op tracker -- , seq: 2130, time: 2014-06-09 16:22:11.259635, event: header_read, op: pg_info(1 pgs e243:2.288) 2014-06-09 16:22:11.624472 7f869af78700 5 -- op tracker -- , seq: 2130, time: 2014-06-09 16:22:11.259635, event: throttled, op: pg_info(1 pgs e243:2.288) 2014-06-09 16:22:11.624475 7f869af78700 5 -- op tracker -- , seq: 2130, time: 2014-06-09 16:22:11.259643, event: all_read, op: pg_info(1 pgs e243:2.288) 2014-06-09 16:22:11.624479 7f869af78700 5 -- op tracker -- , seq: 2130, time: 2014-06-09 16:22:11.624464, event: dispatched, op: pg_info(1 pgs e243:2.288) 2014-06-09 16:22:11.624482 7f869af78700 5 -- op tracker -- , seq: 2130, time: 2014-06-09 16:22:11.624481, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.288) 2014-06-09 16:22:11.624485 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.288) v4 from osd.0 2014-06-09 16:22:11.624486 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e3c00 2014-06-09 16:22:11.624489 7f869af78700 5 -- op tracker -- , seq: 2130, time: 2014-06-09 16:22:11.624488, event: started, op: pg_info(1 pgs e243:2.288) 2014-06-09 16:22:11.624498 7f869af78700 5 -- op tracker -- , seq: 2130, time: 2014-06-09 16:22:11.624497, event: done, op: pg_info(1 pgs e243:2.288) 2014-06-09 16:22:11.624504 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624505 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624511 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624512 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624512 7f869af78700 20 osd.26 243 _dispatch 0x1fc04fc0 pg_info(1 pgs e243:3.319) v4 2014-06-09 16:22:11.624515 7f869af78700 5 -- op tracker -- , seq: 2131, time: 2014-06-09 16:22:11.381954, event: header_read, op: pg_info(1 pgs e243:3.319) 2014-06-09 16:22:11.624519 7f869af78700 5 -- op tracker -- , seq: 2131, time: 2014-06-09 16:22:11.381955, event: throttled, op: pg_info(1 pgs e243:3.319) 2014-06-09 16:22:11.624522 7f869af78700 5 -- op tracker -- , seq: 2131, time: 2014-06-09 16:22:11.381967, event: all_read, op: pg_info(1 pgs e243:3.319) 2014-06-09 16:22:11.624526 7f869af78700 5 -- op tracker -- , seq: 2131, time: 2014-06-09 16:22:11.624511, event: dispatched, op: pg_info(1 pgs e243:3.319) 2014-06-09 16:22:11.624529 7f869af78700 5 -- op tracker -- , seq: 2131, time: 2014-06-09 16:22:11.624528, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.319) 2014-06-09 16:22:11.624535 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.319) v4 from osd.5 2014-06-09 16:22:11.624537 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc04fc0 2014-06-09 16:22:11.624539 7f869af78700 5 -- op tracker -- , seq: 2131, time: 2014-06-09 16:22:11.624538, event: started, op: pg_info(1 pgs e243:3.319) 2014-06-09 16:22:11.624548 7f869af78700 5 -- op tracker -- , seq: 2131, time: 2014-06-09 16:22:11.624547, event: done, op: pg_info(1 pgs e243:3.319) 2014-06-09 16:22:11.624553 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624553 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624560 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624561 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624562 7f869af78700 20 osd.26 243 _dispatch 0xe06efc0 pg_info(1 pgs e243:3.2d3(4)) v4 2014-06-09 16:22:11.624564 7f869af78700 5 -- op tracker -- , seq: 2132, time: 2014-06-09 16:22:11.185571, event: header_read, op: pg_info(1 pgs e243:3.2d3(4)) 2014-06-09 16:22:11.624568 7f869af78700 5 -- op tracker -- , seq: 2132, time: 2014-06-09 16:22:11.185571, event: throttled, op: pg_info(1 pgs e243:3.2d3(4)) 2014-06-09 16:22:11.624575 7f869af78700 5 -- op tracker -- , seq: 2132, time: 2014-06-09 16:22:11.185589, event: all_read, op: pg_info(1 pgs e243:3.2d3(4)) 2014-06-09 16:22:11.624580 7f869af78700 5 -- op tracker -- , seq: 2132, time: 2014-06-09 16:22:11.624560, event: dispatched, op: pg_info(1 pgs e243:3.2d3(4)) 2014-06-09 16:22:11.624584 7f869af78700 5 -- op tracker -- , seq: 2132, time: 2014-06-09 16:22:11.624583, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2d3(4)) 2014-06-09 16:22:11.624589 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2d3(4)) v4 from osd.2 2014-06-09 16:22:11.624591 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xe06efc0 2014-06-09 16:22:11.624593 7f869af78700 5 -- op tracker -- , seq: 2132, time: 2014-06-09 16:22:11.624592, event: started, op: pg_info(1 pgs e243:3.2d3(4)) 2014-06-09 16:22:11.624602 7f869af78700 5 -- op tracker -- , seq: 2132, time: 2014-06-09 16:22:11.624601, event: done, op: pg_info(1 pgs e243:3.2d3(4)) 2014-06-09 16:22:11.624608 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624609 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624617 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624618 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624619 7f869af78700 20 osd.26 243 _dispatch 0x190b3dc0 pg_notify(2.2f7(23) epoch 243) v5 2014-06-09 16:22:11.624621 7f869af78700 5 -- op tracker -- , seq: 2133, time: 2014-06-09 16:22:11.194679, event: header_read, op: pg_notify(2.2f7(23) epoch 243) 2014-06-09 16:22:11.624625 7f869af78700 5 -- op tracker -- , seq: 2133, time: 2014-06-09 16:22:11.194679, event: throttled, op: pg_notify(2.2f7(23) epoch 243) 2014-06-09 16:22:11.624628 7f869af78700 5 -- op tracker -- , seq: 2133, time: 2014-06-09 16:22:11.194698, event: all_read, op: pg_notify(2.2f7(23) epoch 243) 2014-06-09 16:22:11.624632 7f869af78700 5 -- op tracker -- , seq: 2133, time: 2014-06-09 16:22:11.624617, event: dispatched, op: pg_notify(2.2f7(23) epoch 243) 2014-06-09 16:22:11.624637 7f869af78700 5 -- op tracker -- , seq: 2133, time: 2014-06-09 16:22:11.624636, event: waiting_for_osdmap, op: pg_notify(2.2f7(23) epoch 243) 2014-06-09 16:22:11.624640 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.624641 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b3dc0 2014-06-09 16:22:11.624643 7f869af78700 5 -- op tracker -- , seq: 2133, time: 2014-06-09 16:22:11.624642, event: started, op: pg_notify(2.2f7(23) epoch 243) 2014-06-09 16:22:11.624652 7f869af78700 5 -- op tracker -- , seq: 2133, time: 2014-06-09 16:22:11.624651, event: done, op: pg_notify(2.2f7(23) epoch 243) 2014-06-09 16:22:11.624656 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624657 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624668 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624669 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624669 7f869af78700 20 osd.26 243 _dispatch 0x2d094380 pg_info(1 pgs e243:1.70c) v4 2014-06-09 16:22:11.624672 7f869af78700 5 -- op tracker -- , seq: 2134, time: 2014-06-09 16:22:11.359058, event: header_read, op: pg_info(1 pgs e243:1.70c) 2014-06-09 16:22:11.624676 7f869af78700 5 -- op tracker -- , seq: 2134, time: 2014-06-09 16:22:11.359059, event: throttled, op: pg_info(1 pgs e243:1.70c) 2014-06-09 16:22:11.624679 7f869af78700 5 -- op tracker -- , seq: 2134, time: 2014-06-09 16:22:11.359070, event: all_read, op: pg_info(1 pgs e243:1.70c) 2014-06-09 16:22:11.624686 7f869af78700 5 -- op tracker -- , seq: 2134, time: 2014-06-09 16:22:11.624668, event: dispatched, op: pg_info(1 pgs e243:1.70c) 2014-06-09 16:22:11.624692 7f869af78700 5 -- op tracker -- , seq: 2134, time: 2014-06-09 16:22:11.624691, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.70c) 2014-06-09 16:22:11.624697 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.70c) v4 from osd.7 2014-06-09 16:22:11.624699 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d094380 2014-06-09 16:22:11.624701 7f869af78700 5 -- op tracker -- , seq: 2134, time: 2014-06-09 16:22:11.624700, event: started, op: pg_info(1 pgs e243:1.70c) 2014-06-09 16:22:11.624713 7f869af78700 5 -- op tracker -- , seq: 2134, time: 2014-06-09 16:22:11.624712, event: done, op: pg_info(1 pgs e243:1.70c) 2014-06-09 16:22:11.624720 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624721 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624729 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624730 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624731 7f869af78700 20 osd.26 243 _dispatch 0x30a8ac00 pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.624734 7f869af78700 5 -- op tracker -- , seq: 2135, time: 2014-06-09 16:22:11.400000, event: header_read, op: pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.624739 7f869af78700 5 -- op tracker -- , seq: 2135, time: 2014-06-09 16:22:11.400001, event: throttled, op: pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.624743 7f869af78700 5 -- op tracker -- , seq: 2135, time: 2014-06-09 16:22:11.400016, event: all_read, op: pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.624746 7f869af78700 5 -- op tracker -- , seq: 2135, time: 2014-06-09 16:22:11.624729, event: dispatched, op: pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.624752 7f869af78700 5 -- op tracker -- , seq: 2135, time: 2014-06-09 16:22:11.624750, event: waiting_for_osdmap, op: pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.624754 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) v4 from osd.3 2014-06-09 16:22:11.624756 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a8ac00 2014-06-09 16:22:11.624759 7f869af78700 5 -- op tracker -- , seq: 2135, time: 2014-06-09 16:22:11.624758, event: started, op: pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.624766 7f869af78700 5 -- op tracker -- , seq: 2135, time: 2014-06-09 16:22:11.624765, event: done, op: pg_log(3.5f9 epoch 243 log log((213'610,214'611], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.624773 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624775 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624782 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624782 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624783 7f869af78700 20 osd.26 243 _dispatch 0x30645e00 pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.624787 7f869af78700 5 -- op tracker -- , seq: 2136, time: 2014-06-09 16:22:11.259472, event: header_read, op: pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624792 7f869af78700 5 -- op tracker -- , seq: 2136, time: 2014-06-09 16:22:11.259473, event: throttled, op: pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624796 7f869af78700 5 -- op tracker -- , seq: 2136, time: 2014-06-09 16:22:11.259488, event: all_read, op: pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624800 7f869af78700 5 -- op tracker -- , seq: 2136, time: 2014-06-09 16:22:11.624781, event: dispatched, op: pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624805 7f869af78700 5 -- op tracker -- , seq: 2136, time: 2014-06-09 16:22:11.624803, event: waiting_for_osdmap, op: pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624807 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.624809 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30645e00 2014-06-09 16:22:11.624812 7f869af78700 5 -- op tracker -- , seq: 2136, time: 2014-06-09 16:22:11.624811, event: started, op: pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624820 7f869af78700 5 -- op tracker -- , seq: 2136, time: 2014-06-09 16:22:11.624819, event: done, op: pg_log(0.771 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624827 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624828 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624834 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624835 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624836 7f869af78700 20 osd.26 243 _dispatch 0x3005aa80 pg_notify(3.505(10) epoch 243) v5 2014-06-09 16:22:11.624838 7f869af78700 5 -- op tracker -- , seq: 2137, time: 2014-06-09 16:22:11.347626, event: header_read, op: pg_notify(3.505(10) epoch 243) 2014-06-09 16:22:11.624842 7f869af78700 5 -- op tracker -- , seq: 2137, time: 2014-06-09 16:22:11.347627, event: throttled, op: pg_notify(3.505(10) epoch 243) 2014-06-09 16:22:11.624846 7f869af78700 5 -- op tracker -- , seq: 2137, time: 2014-06-09 16:22:11.347656, event: all_read, op: pg_notify(3.505(10) epoch 243) 2014-06-09 16:22:11.624852 7f869af78700 5 -- op tracker -- , seq: 2137, time: 2014-06-09 16:22:11.624834, event: dispatched, op: pg_notify(3.505(10) epoch 243) 2014-06-09 16:22:11.624856 7f869af78700 5 -- op tracker -- , seq: 2137, time: 2014-06-09 16:22:11.624855, event: waiting_for_osdmap, op: pg_notify(3.505(10) epoch 243) 2014-06-09 16:22:11.624859 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.624860 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005aa80 2014-06-09 16:22:11.624863 7f869af78700 5 -- op tracker -- , seq: 2137, time: 2014-06-09 16:22:11.624862, event: started, op: pg_notify(3.505(10) epoch 243) 2014-06-09 16:22:11.624874 7f869af78700 5 -- op tracker -- , seq: 2137, time: 2014-06-09 16:22:11.624873, event: done, op: pg_notify(3.505(10) epoch 243) 2014-06-09 16:22:11.624878 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624879 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624887 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624888 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624888 7f869af78700 20 osd.26 243 _dispatch 0x1c03fdc0 pg_info(1 pgs e243:2.2b5) v4 2014-06-09 16:22:11.624891 7f869af78700 5 -- op tracker -- , seq: 2138, time: 2014-06-09 16:22:11.361892, event: header_read, op: pg_info(1 pgs e243:2.2b5) 2014-06-09 16:22:11.624898 7f869af78700 5 -- op tracker -- , seq: 2138, time: 2014-06-09 16:22:11.361893, event: throttled, op: pg_info(1 pgs e243:2.2b5) 2014-06-09 16:22:11.624904 7f869af78700 5 -- op tracker -- , seq: 2138, time: 2014-06-09 16:22:11.361909, event: all_read, op: pg_info(1 pgs e243:2.2b5) 2014-06-09 16:22:11.624907 7f869af78700 5 -- op tracker -- , seq: 2138, time: 2014-06-09 16:22:11.624887, event: dispatched, op: pg_info(1 pgs e243:2.2b5) 2014-06-09 16:22:11.624911 7f869af78700 5 -- op tracker -- , seq: 2138, time: 2014-06-09 16:22:11.624910, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2b5) 2014-06-09 16:22:11.624915 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2b5) v4 from osd.8 2014-06-09 16:22:11.624916 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03fdc0 2014-06-09 16:22:11.624919 7f869af78700 5 -- op tracker -- , seq: 2138, time: 2014-06-09 16:22:11.624918, event: started, op: pg_info(1 pgs e243:2.2b5) 2014-06-09 16:22:11.624928 7f869af78700 5 -- op tracker -- , seq: 2138, time: 2014-06-09 16:22:11.624927, event: done, op: pg_info(1 pgs e243:2.2b5) 2014-06-09 16:22:11.624932 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624933 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624940 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624941 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624941 7f869af78700 20 osd.26 243 _dispatch 0x30889800 pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.624945 7f869af78700 5 -- op tracker -- , seq: 2139, time: 2014-06-09 16:22:11.356335, event: header_read, op: pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624949 7f869af78700 5 -- op tracker -- , seq: 2139, time: 2014-06-09 16:22:11.356336, event: throttled, op: pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624952 7f869af78700 5 -- op tracker -- , seq: 2139, time: 2014-06-09 16:22:11.356351, event: all_read, op: pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624956 7f869af78700 5 -- op tracker -- , seq: 2139, time: 2014-06-09 16:22:11.624940, event: dispatched, op: pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624963 7f869af78700 5 -- op tracker -- , seq: 2139, time: 2014-06-09 16:22:11.624962, event: waiting_for_osdmap, op: pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624966 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.624968 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30889800 2014-06-09 16:22:11.624971 7f869af78700 5 -- op tracker -- , seq: 2139, time: 2014-06-09 16:22:11.624969, event: started, op: pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624980 7f869af78700 5 -- op tracker -- , seq: 2139, time: 2014-06-09 16:22:11.624978, event: done, op: pg_log(0.493 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.624984 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624985 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624991 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.624992 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.624992 7f869af78700 20 osd.26 243 _dispatch 0x3005b6c0 pg_info(1 pgs e243:3.362) v4 2014-06-09 16:22:11.624995 7f869af78700 5 -- op tracker -- , seq: 2140, time: 2014-06-09 16:22:11.259648, event: header_read, op: pg_info(1 pgs e243:3.362) 2014-06-09 16:22:11.624998 7f869af78700 5 -- op tracker -- , seq: 2140, time: 2014-06-09 16:22:11.259649, event: throttled, op: pg_info(1 pgs e243:3.362) 2014-06-09 16:22:11.625002 7f869af78700 5 -- op tracker -- , seq: 2140, time: 2014-06-09 16:22:11.259662, event: all_read, op: pg_info(1 pgs e243:3.362) 2014-06-09 16:22:11.625005 7f869af78700 5 -- op tracker -- , seq: 2140, time: 2014-06-09 16:22:11.624991, event: dispatched, op: pg_info(1 pgs e243:3.362) 2014-06-09 16:22:11.625008 7f869af78700 5 -- op tracker -- , seq: 2140, time: 2014-06-09 16:22:11.625007, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.362) 2014-06-09 16:22:11.625010 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.362) v4 from osd.0 2014-06-09 16:22:11.625012 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005b6c0 2014-06-09 16:22:11.625014 7f869af78700 5 -- op tracker -- , seq: 2140, time: 2014-06-09 16:22:11.625013, event: started, op: pg_info(1 pgs e243:3.362) 2014-06-09 16:22:11.625023 7f869af78700 5 -- op tracker -- , seq: 2140, time: 2014-06-09 16:22:11.625022, event: done, op: pg_info(1 pgs e243:3.362) 2014-06-09 16:22:11.625027 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625028 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625034 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625035 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625036 7f869af78700 20 osd.26 243 _dispatch 0x1fc04a80 pg_info(1 pgs e243:2.5e) v4 2014-06-09 16:22:11.625038 7f869af78700 5 -- op tracker -- , seq: 2141, time: 2014-06-09 16:22:11.381974, event: header_read, op: pg_info(1 pgs e243:2.5e) 2014-06-09 16:22:11.625042 7f869af78700 5 -- op tracker -- , seq: 2141, time: 2014-06-09 16:22:11.381974, event: throttled, op: pg_info(1 pgs e243:2.5e) 2014-06-09 16:22:11.625045 7f869af78700 5 -- op tracker -- , seq: 2141, time: 2014-06-09 16:22:11.381987, event: all_read, op: pg_info(1 pgs e243:2.5e) 2014-06-09 16:22:11.625048 7f869af78700 5 -- op tracker -- , seq: 2141, time: 2014-06-09 16:22:11.625034, event: dispatched, op: pg_info(1 pgs e243:2.5e) 2014-06-09 16:22:11.625055 7f869af78700 5 -- op tracker -- , seq: 2141, time: 2014-06-09 16:22:11.625054, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5e) 2014-06-09 16:22:11.625060 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5e) v4 from osd.5 2014-06-09 16:22:11.625062 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc04a80 2014-06-09 16:22:11.625065 7f869af78700 5 -- op tracker -- , seq: 2141, time: 2014-06-09 16:22:11.625064, event: started, op: pg_info(1 pgs e243:2.5e) 2014-06-09 16:22:11.625073 7f869af78700 5 -- op tracker -- , seq: 2141, time: 2014-06-09 16:22:11.625072, event: done, op: pg_info(1 pgs e243:2.5e) 2014-06-09 16:22:11.625077 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625085 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625086 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625086 7f869af78700 20 osd.26 243 _dispatch 0x30719200 pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.625090 7f869af78700 5 -- op tracker -- , seq: 2142, time: 2014-06-09 16:22:11.204913, event: header_read, op: pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625095 7f869af78700 5 -- op tracker -- , seq: 2142, time: 2014-06-09 16:22:11.204914, event: throttled, op: pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625099 7f869af78700 5 -- op tracker -- , seq: 2142, time: 2014-06-09 16:22:11.204926, event: all_read, op: pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625103 7f869af78700 5 -- op tracker -- , seq: 2142, time: 2014-06-09 16:22:11.625085, event: dispatched, op: pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625110 7f869af78700 5 -- op tracker -- , seq: 2142, time: 2014-06-09 16:22:11.625108, event: waiting_for_osdmap, op: pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625112 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.625115 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30719200 2014-06-09 16:22:11.625118 7f869af78700 5 -- op tracker -- , seq: 2142, time: 2014-06-09 16:22:11.625116, event: started, op: pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625126 7f869af78700 5 -- op tracker -- , seq: 2142, time: 2014-06-09 16:22:11.625125, event: done, op: pg_log(0.3fe epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625133 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625134 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625141 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625141 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625142 7f869af78700 20 osd.26 243 _dispatch 0x30044000 pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.625145 7f869af78700 5 -- op tracker -- , seq: 2143, time: 2014-06-09 16:22:11.245110, event: header_read, op: pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625150 7f869af78700 5 -- op tracker -- , seq: 2143, time: 2014-06-09 16:22:11.245111, event: throttled, op: pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625153 7f869af78700 5 -- op tracker -- , seq: 2143, time: 2014-06-09 16:22:11.245124, event: all_read, op: pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625157 7f869af78700 5 -- op tracker -- , seq: 2143, time: 2014-06-09 16:22:11.625140, event: dispatched, op: pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625164 7f869af78700 5 -- op tracker -- , seq: 2143, time: 2014-06-09 16:22:11.625162, event: waiting_for_osdmap, op: pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625166 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.625169 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30044000 2014-06-09 16:22:11.625172 7f869af78700 5 -- op tracker -- , seq: 2143, time: 2014-06-09 16:22:11.625170, event: started, op: pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625180 7f869af78700 5 -- op tracker -- , seq: 2143, time: 2014-06-09 16:22:11.625179, event: done, op: pg_log(2.40c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625184 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625185 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625193 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625193 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625194 7f869af78700 20 osd.26 243 _dispatch 0x305b2600 pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.625197 7f869af78700 5 -- op tracker -- , seq: 2144, time: 2014-06-09 16:22:11.375633, event: header_read, op: pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625202 7f869af78700 5 -- op tracker -- , seq: 2144, time: 2014-06-09 16:22:11.375634, event: throttled, op: pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625205 7f869af78700 5 -- op tracker -- , seq: 2144, time: 2014-06-09 16:22:11.375651, event: all_read, op: pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625209 7f869af78700 5 -- op tracker -- , seq: 2144, time: 2014-06-09 16:22:11.625192, event: dispatched, op: pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625212 7f869af78700 5 -- op tracker -- , seq: 2144, time: 2014-06-09 16:22:11.625211, event: waiting_for_osdmap, op: pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625215 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.7 2014-06-09 16:22:11.625217 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b2600 2014-06-09 16:22:11.625220 7f869af78700 5 -- op tracker -- , seq: 2144, time: 2014-06-09 16:22:11.625218, event: started, op: pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625226 7f869af78700 5 -- op tracker -- , seq: 2144, time: 2014-06-09 16:22:11.625225, event: done, op: pg_log(0.1fd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625234 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625235 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625242 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625243 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625243 7f869af78700 20 osd.26 243 _dispatch 0x30a8a600 pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.625247 7f869af78700 5 -- op tracker -- , seq: 2145, time: 2014-06-09 16:22:11.400021, event: header_read, op: pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625252 7f869af78700 5 -- op tracker -- , seq: 2145, time: 2014-06-09 16:22:11.400021, event: throttled, op: pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625256 7f869af78700 5 -- op tracker -- , seq: 2145, time: 2014-06-09 16:22:11.400044, event: all_read, op: pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625260 7f869af78700 5 -- op tracker -- , seq: 2145, time: 2014-06-09 16:22:11.625242, event: dispatched, op: pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625265 7f869af78700 5 -- op tracker -- , seq: 2145, time: 2014-06-09 16:22:11.625264, event: waiting_for_osdmap, op: pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625268 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) v4 from osd.3 2014-06-09 16:22:11.625270 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a8a600 2014-06-09 16:22:11.625273 7f869af78700 5 -- op tracker -- , seq: 2145, time: 2014-06-09 16:22:11.625271, event: started, op: pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625283 7f869af78700 5 -- op tracker -- , seq: 2145, time: 2014-06-09 16:22:11.625282, event: done, op: pg_log(3.6e epoch 243 log log((214'610,214'612], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625288 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625288 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625295 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625296 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625296 7f869af78700 20 osd.26 243 _dispatch 0x30642a00 pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.625300 7f869af78700 5 -- op tracker -- , seq: 2146, time: 2014-06-09 16:22:11.260682, event: header_read, op: pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625304 7f869af78700 5 -- op tracker -- , seq: 2146, time: 2014-06-09 16:22:11.260683, event: throttled, op: pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625308 7f869af78700 5 -- op tracker -- , seq: 2146, time: 2014-06-09 16:22:11.261532, event: all_read, op: pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625311 7f869af78700 5 -- op tracker -- , seq: 2146, time: 2014-06-09 16:22:11.625295, event: dispatched, op: pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625315 7f869af78700 5 -- op tracker -- , seq: 2146, time: 2014-06-09 16:22:11.625314, event: waiting_for_osdmap, op: pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625317 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) v4 from osd.1 2014-06-09 16:22:11.625319 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30642a00 2014-06-09 16:22:11.625322 7f869af78700 5 -- op tracker -- , seq: 2146, time: 2014-06-09 16:22:11.625321, event: started, op: pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625333 7f869af78700 5 -- op tracker -- , seq: 2146, time: 2014-06-09 16:22:11.625332, event: done, op: pg_log(3.718 epoch 243 log log((0'0,213'620], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625341 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625342 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625348 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625349 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625350 7f869af78700 20 osd.26 243 _dispatch 0x30a81e00 pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.625353 7f869af78700 5 -- op tracker -- , seq: 2147, time: 2014-06-09 16:22:11.347666, event: header_read, op: pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625359 7f869af78700 5 -- op tracker -- , seq: 2147, time: 2014-06-09 16:22:11.347666, event: throttled, op: pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625362 7f869af78700 5 -- op tracker -- , seq: 2147, time: 2014-06-09 16:22:11.347679, event: all_read, op: pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625366 7f869af78700 5 -- op tracker -- , seq: 2147, time: 2014-06-09 16:22:11.625348, event: dispatched, op: pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625371 7f869af78700 5 -- op tracker -- , seq: 2147, time: 2014-06-09 16:22:11.625370, event: waiting_for_osdmap, op: pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625374 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.625376 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a81e00 2014-06-09 16:22:11.625379 7f869af78700 5 -- op tracker -- , seq: 2147, time: 2014-06-09 16:22:11.625377, event: started, op: pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625386 7f869af78700 5 -- op tracker -- , seq: 2147, time: 2014-06-09 16:22:11.625384, event: done, op: pg_log(0.6f0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625390 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625391 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625397 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625397 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625398 7f869af78700 20 osd.26 243 _dispatch 0x22013c00 pg_info(1 pgs e243:1.b9) v4 2014-06-09 16:22:11.625401 7f869af78700 5 -- op tracker -- , seq: 2148, time: 2014-06-09 16:22:11.361955, event: header_read, op: pg_info(1 pgs e243:1.b9) 2014-06-09 16:22:11.625404 7f869af78700 5 -- op tracker -- , seq: 2148, time: 2014-06-09 16:22:11.361956, event: throttled, op: pg_info(1 pgs e243:1.b9) 2014-06-09 16:22:11.625408 7f869af78700 5 -- op tracker -- , seq: 2148, time: 2014-06-09 16:22:11.361971, event: all_read, op: pg_info(1 pgs e243:1.b9) 2014-06-09 16:22:11.625412 7f869af78700 5 -- op tracker -- , seq: 2148, time: 2014-06-09 16:22:11.625397, event: dispatched, op: pg_info(1 pgs e243:1.b9) 2014-06-09 16:22:11.625417 7f869af78700 5 -- op tracker -- , seq: 2148, time: 2014-06-09 16:22:11.625416, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.b9) 2014-06-09 16:22:11.625419 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.b9) v4 from osd.8 2014-06-09 16:22:11.625421 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22013c00 2014-06-09 16:22:11.625423 7f869af78700 5 -- op tracker -- , seq: 2148, time: 2014-06-09 16:22:11.625422, event: started, op: pg_info(1 pgs e243:1.b9) 2014-06-09 16:22:11.625432 7f869af78700 5 -- op tracker -- , seq: 2148, time: 2014-06-09 16:22:11.625431, event: done, op: pg_info(1 pgs e243:1.b9) 2014-06-09 16:22:11.625436 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625437 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625444 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625445 7f869af78700 20 osd.26 243 _dispatch 0x1609da40 pg_notify(0.1c8(20),3.491(10),3.1db(10) epoch 243) v5 2014-06-09 16:22:11.625449 7f869af78700 5 -- op tracker -- , seq: 2149, time: 2014-06-09 16:22:11.357067, event: header_read, op: pg_notify(0.1c8(20),3.491(10),3.1db(10) epoch 243) 2014-06-09 16:22:11.625453 7f869af78700 5 -- op tracker -- , seq: 2149, time: 2014-06-09 16:22:11.357067, event: throttled, op: pg_notify(0.1c8(20),3.491(10),3.1db(10) epoch 243) 2014-06-09 16:22:11.625457 7f869af78700 5 -- op tracker -- , seq: 2149, time: 2014-06-09 16:22:11.357126, event: all_read, op: pg_notify(0.1c8(20),3.491(10),3.1db(10) epoch 243) 2014-06-09 16:22:11.625464 7f869af78700 5 -- op tracker -- , seq: 2149, time: 2014-06-09 16:22:11.625444, event: dispatched, op: pg_notify(0.1c8(20),3.491(10),3.1db(10) epoch 243) 2014-06-09 16:22:11.625468 7f869af78700 5 -- op tracker -- , seq: 2149, time: 2014-06-09 16:22:11.625467, event: waiting_for_osdmap, op: pg_notify(0.1c8(20),3.491(10),3.1db(10) epoch 243) 2014-06-09 16:22:11.625473 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.625475 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1609da40 2014-06-09 16:22:11.625478 7f869af78700 5 -- op tracker -- , seq: 2149, time: 2014-06-09 16:22:11.625476, event: started, op: pg_notify(0.1c8(20),3.491(10),3.1db(10) epoch 243) 2014-06-09 16:22:11.625496 7f869af78700 5 -- op tracker -- , seq: 2149, time: 2014-06-09 16:22:11.625494, event: done, op: pg_notify(0.1c8(20),3.491(10),3.1db(10) epoch 243) 2014-06-09 16:22:11.625500 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625501 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625515 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625516 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625516 7f869af78700 20 osd.26 243 _dispatch 0x3005ba40 pg_info(1 pgs e243:0.29d) v4 2014-06-09 16:22:11.625519 7f869af78700 5 -- op tracker -- , seq: 2150, time: 2014-06-09 16:22:11.259673, event: header_read, op: pg_info(1 pgs e243:0.29d) 2014-06-09 16:22:11.625523 7f869af78700 5 -- op tracker -- , seq: 2150, time: 2014-06-09 16:22:11.259674, event: throttled, op: pg_info(1 pgs e243:0.29d) 2014-06-09 16:22:11.625526 7f869af78700 5 -- op tracker -- , seq: 2150, time: 2014-06-09 16:22:11.259681, event: all_read, op: pg_info(1 pgs e243:0.29d) 2014-06-09 16:22:11.625533 7f869af78700 5 -- op tracker -- , seq: 2150, time: 2014-06-09 16:22:11.625515, event: dispatched, op: pg_info(1 pgs e243:0.29d) 2014-06-09 16:22:11.625537 7f869af78700 5 -- op tracker -- , seq: 2150, time: 2014-06-09 16:22:11.625536, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.29d) 2014-06-09 16:22:11.625540 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.29d) v4 from osd.0 2014-06-09 16:22:11.625541 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005ba40 2014-06-09 16:22:11.625544 7f869af78700 5 -- op tracker -- , seq: 2150, time: 2014-06-09 16:22:11.625543, event: started, op: pg_info(1 pgs e243:0.29d) 2014-06-09 16:22:11.625552 7f869af78700 5 -- op tracker -- , seq: 2150, time: 2014-06-09 16:22:11.625551, event: done, op: pg_info(1 pgs e243:0.29d) 2014-06-09 16:22:11.625556 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625557 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625564 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625565 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625566 7f869af78700 20 osd.26 243 _dispatch 0x240e8a80 pg_notify(2.359(23),0.627(20) epoch 243) v5 2014-06-09 16:22:11.625569 7f869af78700 5 -- op tracker -- , seq: 2151, time: 2014-06-09 16:22:11.382038, event: header_read, op: pg_notify(2.359(23),0.627(20) epoch 243) 2014-06-09 16:22:11.625573 7f869af78700 5 -- op tracker -- , seq: 2151, time: 2014-06-09 16:22:11.382039, event: throttled, op: pg_notify(2.359(23),0.627(20) epoch 243) 2014-06-09 16:22:11.625576 7f869af78700 5 -- op tracker -- , seq: 2151, time: 2014-06-09 16:22:11.382090, event: all_read, op: pg_notify(2.359(23),0.627(20) epoch 243) 2014-06-09 16:22:11.625580 7f869af78700 5 -- op tracker -- , seq: 2151, time: 2014-06-09 16:22:11.625564, event: dispatched, op: pg_notify(2.359(23),0.627(20) epoch 243) 2014-06-09 16:22:11.625583 7f869af78700 5 -- op tracker -- , seq: 2151, time: 2014-06-09 16:22:11.625582, event: waiting_for_osdmap, op: pg_notify(2.359(23),0.627(20) epoch 243) 2014-06-09 16:22:11.625585 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.625586 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e8a80 2014-06-09 16:22:11.625589 7f869af78700 5 -- op tracker -- , seq: 2151, time: 2014-06-09 16:22:11.625588, event: started, op: pg_notify(2.359(23),0.627(20) epoch 243) 2014-06-09 16:22:11.625602 7f869af78700 5 -- op tracker -- , seq: 2151, time: 2014-06-09 16:22:11.625600, event: done, op: pg_notify(2.359(23),0.627(20) epoch 243) 2014-06-09 16:22:11.625606 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625607 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625623 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625623 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625624 7f869af78700 20 osd.26 243 _dispatch 0x3043ae00 pg_notify(3.18b(12),3.71f(12),0.722(27),3.458(12) epoch 243) v5 2014-06-09 16:22:11.625628 7f869af78700 5 -- op tracker -- , seq: 2152, time: 2014-06-09 16:22:11.207309, event: header_read, op: pg_notify(3.18b(12),3.71f(12),0.722(27),3.458(12) epoch 243) 2014-06-09 16:22:11.625633 7f869af78700 5 -- op tracker -- , seq: 2152, time: 2014-06-09 16:22:11.207310, event: throttled, op: pg_notify(3.18b(12),3.71f(12),0.722(27),3.458(12) epoch 243) 2014-06-09 16:22:11.625637 7f869af78700 5 -- op tracker -- , seq: 2152, time: 2014-06-09 16:22:11.207358, event: all_read, op: pg_notify(3.18b(12),3.71f(12),0.722(27),3.458(12) epoch 243) 2014-06-09 16:22:11.625640 7f869af78700 5 -- op tracker -- , seq: 2152, time: 2014-06-09 16:22:11.625622, event: dispatched, op: pg_notify(3.18b(12),3.71f(12),0.722(27),3.458(12) epoch 243) 2014-06-09 16:22:11.625644 7f869af78700 5 -- op tracker -- , seq: 2152, time: 2014-06-09 16:22:11.625643, event: waiting_for_osdmap, op: pg_notify(3.18b(12),3.71f(12),0.722(27),3.458(12) epoch 243) 2014-06-09 16:22:11.625647 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.625648 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043ae00 2014-06-09 16:22:11.625651 7f869af78700 5 -- op tracker -- , seq: 2152, time: 2014-06-09 16:22:11.625649, event: started, op: pg_notify(3.18b(12),3.71f(12),0.722(27),3.458(12) epoch 243) 2014-06-09 16:22:11.625677 7f869af78700 5 -- op tracker -- , seq: 2152, time: 2014-06-09 16:22:11.625675, event: done, op: pg_notify(3.18b(12),3.71f(12),0.722(27),3.458(12) epoch 243) 2014-06-09 16:22:11.625682 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625683 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625701 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625702 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625702 7f869af78700 20 osd.26 243 _dispatch 0x30a20400 pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.625706 7f869af78700 5 -- op tracker -- , seq: 2153, time: 2014-06-09 16:22:11.245149, event: header_read, op: pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625710 7f869af78700 5 -- op tracker -- , seq: 2153, time: 2014-06-09 16:22:11.245149, event: throttled, op: pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625714 7f869af78700 5 -- op tracker -- , seq: 2153, time: 2014-06-09 16:22:11.245170, event: all_read, op: pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625721 7f869af78700 5 -- op tracker -- , seq: 2153, time: 2014-06-09 16:22:11.625701, event: dispatched, op: pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625725 7f869af78700 5 -- op tracker -- , seq: 2153, time: 2014-06-09 16:22:11.625724, event: waiting_for_osdmap, op: pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625728 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.625730 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a20400 2014-06-09 16:22:11.625733 7f869af78700 5 -- op tracker -- , seq: 2153, time: 2014-06-09 16:22:11.625732, event: started, op: pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625740 7f869af78700 5 -- op tracker -- , seq: 2153, time: 2014-06-09 16:22:11.625739, event: done, op: pg_log(1.14c epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625744 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625745 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625751 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625752 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625753 7f869af78700 20 osd.26 243 _dispatch 0x2d095c00 pg_info(1 pgs e243:0.537) v4 2014-06-09 16:22:11.625755 7f869af78700 5 -- op tracker -- , seq: 2154, time: 2014-06-09 16:22:11.375660, event: header_read, op: pg_info(1 pgs e243:0.537) 2014-06-09 16:22:11.625759 7f869af78700 5 -- op tracker -- , seq: 2154, time: 2014-06-09 16:22:11.375660, event: throttled, op: pg_info(1 pgs e243:0.537) 2014-06-09 16:22:11.625762 7f869af78700 5 -- op tracker -- , seq: 2154, time: 2014-06-09 16:22:11.375673, event: all_read, op: pg_info(1 pgs e243:0.537) 2014-06-09 16:22:11.625769 7f869af78700 5 -- op tracker -- , seq: 2154, time: 2014-06-09 16:22:11.625751, event: dispatched, op: pg_info(1 pgs e243:0.537) 2014-06-09 16:22:11.625772 7f869af78700 5 -- op tracker -- , seq: 2154, time: 2014-06-09 16:22:11.625771, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.537) 2014-06-09 16:22:11.625776 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.537) v4 from osd.7 2014-06-09 16:22:11.625778 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d095c00 2014-06-09 16:22:11.625780 7f869af78700 5 -- op tracker -- , seq: 2154, time: 2014-06-09 16:22:11.625779, event: started, op: pg_info(1 pgs e243:0.537) 2014-06-09 16:22:11.625789 7f869af78700 5 -- op tracker -- , seq: 2154, time: 2014-06-09 16:22:11.625788, event: done, op: pg_info(1 pgs e243:0.537) 2014-06-09 16:22:11.625793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625794 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625801 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625802 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625803 7f869af78700 20 osd.26 243 _dispatch 0x15999dc0 pg_notify(1.1bc(20) epoch 243) v5 2014-06-09 16:22:11.625805 7f869af78700 5 -- op tracker -- , seq: 2155, time: 2014-06-09 16:22:11.408990, event: header_read, op: pg_notify(1.1bc(20) epoch 243) 2014-06-09 16:22:11.625809 7f869af78700 5 -- op tracker -- , seq: 2155, time: 2014-06-09 16:22:11.408991, event: throttled, op: pg_notify(1.1bc(20) epoch 243) 2014-06-09 16:22:11.625812 7f869af78700 5 -- op tracker -- , seq: 2155, time: 2014-06-09 16:22:11.409020, event: all_read, op: pg_notify(1.1bc(20) epoch 243) 2014-06-09 16:22:11.625815 7f869af78700 5 -- op tracker -- , seq: 2155, time: 2014-06-09 16:22:11.625801, event: dispatched, op: pg_notify(1.1bc(20) epoch 243) 2014-06-09 16:22:11.625819 7f869af78700 5 -- op tracker -- , seq: 2155, time: 2014-06-09 16:22:11.625818, event: waiting_for_osdmap, op: pg_notify(1.1bc(20) epoch 243) 2014-06-09 16:22:11.625821 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.625822 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15999dc0 2014-06-09 16:22:11.625825 7f869af78700 5 -- op tracker -- , seq: 2155, time: 2014-06-09 16:22:11.625824, event: started, op: pg_notify(1.1bc(20) epoch 243) 2014-06-09 16:22:11.625833 7f869af78700 5 -- op tracker -- , seq: 2155, time: 2014-06-09 16:22:11.625832, event: done, op: pg_notify(1.1bc(20) epoch 243) 2014-06-09 16:22:11.625837 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625838 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625848 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625849 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625849 7f869af78700 20 osd.26 243 _dispatch 0x30642400 pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.625853 7f869af78700 5 -- op tracker -- , seq: 2156, time: 2014-06-09 16:22:11.264735, event: header_read, op: pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625859 7f869af78700 5 -- op tracker -- , seq: 2156, time: 2014-06-09 16:22:11.264736, event: throttled, op: pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625863 7f869af78700 5 -- op tracker -- , seq: 2156, time: 2014-06-09 16:22:11.265990, event: all_read, op: pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625866 7f869af78700 5 -- op tracker -- , seq: 2156, time: 2014-06-09 16:22:11.625848, event: dispatched, op: pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625871 7f869af78700 5 -- op tracker -- , seq: 2156, time: 2014-06-09 16:22:11.625870, event: waiting_for_osdmap, op: pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625874 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) v4 from osd.1 2014-06-09 16:22:11.625876 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30642400 2014-06-09 16:22:11.625879 7f869af78700 5 -- op tracker -- , seq: 2156, time: 2014-06-09 16:22:11.625877, event: started, op: pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625889 7f869af78700 5 -- op tracker -- , seq: 2156, time: 2014-06-09 16:22:11.625887, event: done, op: pg_log(3.709 epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.625895 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625896 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625903 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625903 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625904 7f869af78700 20 osd.26 243 _dispatch 0x30a81800 pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.625908 7f869af78700 5 -- op tracker -- , seq: 2157, time: 2014-06-09 16:22:11.347686, event: header_read, op: pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625912 7f869af78700 5 -- op tracker -- , seq: 2157, time: 2014-06-09 16:22:11.347686, event: throttled, op: pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625916 7f869af78700 5 -- op tracker -- , seq: 2157, time: 2014-06-09 16:22:11.347712, event: all_read, op: pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625921 7f869af78700 5 -- op tracker -- , seq: 2157, time: 2014-06-09 16:22:11.625902, event: dispatched, op: pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625925 7f869af78700 5 -- op tracker -- , seq: 2157, time: 2014-06-09 16:22:11.625923, event: waiting_for_osdmap, op: pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625927 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.625929 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a81800 2014-06-09 16:22:11.625932 7f869af78700 5 -- op tracker -- , seq: 2157, time: 2014-06-09 16:22:11.625931, event: started, op: pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625940 7f869af78700 5 -- op tracker -- , seq: 2157, time: 2014-06-09 16:22:11.625938, event: done, op: pg_log(2.1c6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.625943 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625944 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625950 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625951 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.625952 7f869af78700 20 osd.26 243 _dispatch 0x220136c0 pg_info(1 pgs e243:1.3e1) v4 2014-06-09 16:22:11.625954 7f869af78700 5 -- op tracker -- , seq: 2158, time: 2014-06-09 16:22:11.361983, event: header_read, op: pg_info(1 pgs e243:1.3e1) 2014-06-09 16:22:11.625958 7f869af78700 5 -- op tracker -- , seq: 2158, time: 2014-06-09 16:22:11.361984, event: throttled, op: pg_info(1 pgs e243:1.3e1) 2014-06-09 16:22:11.625961 7f869af78700 5 -- op tracker -- , seq: 2158, time: 2014-06-09 16:22:11.362000, event: all_read, op: pg_info(1 pgs e243:1.3e1) 2014-06-09 16:22:11.625968 7f869af78700 5 -- op tracker -- , seq: 2158, time: 2014-06-09 16:22:11.625950, event: dispatched, op: pg_info(1 pgs e243:1.3e1) 2014-06-09 16:22:11.625974 7f869af78700 5 -- op tracker -- , seq: 2158, time: 2014-06-09 16:22:11.625973, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3e1) 2014-06-09 16:22:11.625977 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3e1) v4 from osd.8 2014-06-09 16:22:11.625978 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x220136c0 2014-06-09 16:22:11.625981 7f869af78700 5 -- op tracker -- , seq: 2158, time: 2014-06-09 16:22:11.625980, event: started, op: pg_info(1 pgs e243:1.3e1) 2014-06-09 16:22:11.625989 7f869af78700 5 -- op tracker -- , seq: 2158, time: 2014-06-09 16:22:11.625988, event: done, op: pg_info(1 pgs e243:1.3e1) 2014-06-09 16:22:11.625993 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.625994 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.626001 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.626001 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.626002 7f869af78700 20 osd.26 243 _dispatch 0x1205e380 pg_info(1 pgs e243:0.29) v4 2014-06-09 16:22:11.626005 7f869af78700 5 -- op tracker -- , seq: 2159, time: 2014-06-09 16:22:11.360603, event: header_read, op: pg_info(1 pgs e243:0.29) 2014-06-09 16:22:11.626009 7f869af78700 5 -- op tracker -- , seq: 2159, time: 2014-06-09 16:22:11.360604, event: throttled, op: pg_info(1 pgs e243:0.29) 2014-06-09 16:22:11.626012 7f869af78700 5 -- op tracker -- , seq: 2159, time: 2014-06-09 16:22:11.360618, event: all_read, op: pg_info(1 pgs e243:0.29) 2014-06-09 16:22:11.626015 7f869af78700 5 -- op tracker -- , seq: 2159, time: 2014-06-09 16:22:11.626001, event: dispatched, op: pg_info(1 pgs e243:0.29) 2014-06-09 16:22:11.626019 7f869af78700 5 -- op tracker -- , seq: 2159, time: 2014-06-09 16:22:11.626018, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.29) 2014-06-09 16:22:11.626021 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.29) v4 from osd.6 2014-06-09 16:22:11.626022 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1205e380 2014-06-09 16:22:11.626025 7f869af78700 5 -- op tracker -- , seq: 2159, time: 2014-06-09 16:22:11.626024, event: started, op: pg_info(1 pgs e243:0.29) 2014-06-09 16:22:11.626033 7f869af78700 5 -- op tracker -- , seq: 2159, time: 2014-06-09 16:22:11.626032, event: done, op: pg_info(1 pgs e243:0.29) 2014-06-09 16:22:11.626037 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.626038 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.626044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.626044 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.626045 7f869af78700 20 osd.26 243 _dispatch 0x3005afc0 pg_info(1 pgs e243:1.3ef) v4 2014-06-09 16:22:11.626048 7f869af78700 5 -- op tracker -- , seq: 2160, time: 2014-06-09 16:22:11.259691, event: header_read, op: pg_info(1 pgs e243:1.3ef) 2014-06-09 16:22:11.626051 7f869af78700 5 -- op tracker -- , seq: 2160, time: 2014-06-09 16:22:11.259691, event: throttled, op: pg_info(1 pgs e243:1.3ef) 2014-06-09 16:22:11.626058 7f869af78700 5 -- op tracker -- , seq: 2160, time: 2014-06-09 16:22:11.259698, event: all_read, op: pg_info(1 pgs e243:1.3ef) 2014-06-09 16:22:11.626061 7f869af78700 5 -- op tracker -- , seq: 2160, time: 2014-06-09 16:22:11.626044, event: dispatched, op: pg_info(1 pgs e243:1.3ef) 2014-06-09 16:22:11.626065 7f869af78700 5 -- op tracker -- , seq: 2160, time: 2014-06-09 16:22:11.626064, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3ef) 2014-06-09 16:22:11.626067 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3ef) v4 from osd.0 2014-06-09 16:22:11.626069 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005afc0 2014-06-09 16:22:11.626071 7f869af78700 5 -- op tracker -- , seq: 2160, time: 2014-06-09 16:22:11.626070, event: started, op: pg_info(1 pgs e243:1.3ef) 2014-06-09 16:22:11.626083 7f869af78700 5 -- op tracker -- , seq: 2160, time: 2014-06-09 16:22:11.626082, event: done, op: pg_info(1 pgs e243:1.3ef) 2014-06-09 16:22:11.626087 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.626088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.626095 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.626096 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.626097 7f869af78700 20 osd.26 243 _dispatch 0x240e96c0 pg_info(2 pgs e243:3.9f(2),3.ad(2)) v4 2014-06-09 16:22:11.626100 7f869af78700 5 -- op tracker -- , seq: 2161, time: 2014-06-09 16:22:11.382124, event: header_read, op: pg_info(2 pgs e243:3.9f(2),3.ad(2)) 2014-06-09 16:22:11.626104 7f869af78700 5 -- op tracker -- , seq: 2161, time: 2014-06-09 16:22:11.382124, event: throttled, op: pg_info(2 pgs e243:3.9f(2),3.ad(2)) 2014-06-09 16:22:11.626108 7f869af78700 5 -- op tracker -- , seq: 2161, time: 2014-06-09 16:22:11.382147, event: all_read, op: pg_info(2 pgs e243:3.9f(2),3.ad(2)) 2014-06-09 16:22:11.626112 7f869af78700 5 -- op tracker -- , seq: 2161, time: 2014-06-09 16:22:11.626095, event: dispatched, op: pg_info(2 pgs e243:3.9f(2),3.ad(2)) 2014-06-09 16:22:11.626115 7f869af78700 5 -- op tracker -- , seq: 2161, time: 2014-06-09 16:22:11.626114, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.9f(2),3.ad(2)) 2014-06-09 16:22:11.626117 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.9f(2),3.ad(2)) v4 from osd.5 2014-06-09 16:22:11.626119 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e96c0 2014-06-09 16:22:11.626122 7f869af78700 5 -- op tracker -- , seq: 2161, time: 2014-06-09 16:22:11.626121, event: started, op: pg_info(2 pgs e243:3.9f(2),3.ad(2)) 2014-06-09 16:22:11.626138 7f869af78700 5 -- op tracker -- , seq: 2161, time: 2014-06-09 16:22:11.626137, event: done, op: pg_info(2 pgs e243:3.9f(2),3.ad(2)) 2014-06-09 16:22:11.626143 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627375 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627385 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627386 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627386 7f869af78700 20 osd.26 243 _dispatch 0x307df800 pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.627390 7f869af78700 5 -- op tracker -- , seq: 2162, time: 2014-06-09 16:22:11.207367, event: header_read, op: pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627395 7f869af78700 5 -- op tracker -- , seq: 2162, time: 2014-06-09 16:22:11.207367, event: throttled, op: pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627398 7f869af78700 5 -- op tracker -- , seq: 2162, time: 2014-06-09 16:22:11.207377, event: all_read, op: pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627402 7f869af78700 5 -- op tracker -- , seq: 2162, time: 2014-06-09 16:22:11.627385, event: dispatched, op: pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627408 7f869af78700 5 -- op tracker -- , seq: 2162, time: 2014-06-09 16:22:11.627406, event: waiting_for_osdmap, op: pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627410 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.627412 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307df800 2014-06-09 16:22:11.627415 7f869af78700 5 -- op tracker -- , seq: 2162, time: 2014-06-09 16:22:11.627414, event: started, op: pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627424 7f869af78700 5 -- op tracker -- , seq: 2162, time: 2014-06-09 16:22:11.627422, event: done, op: pg_log(1.465 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627433 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627435 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627442 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627443 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627443 7f869af78700 20 osd.26 243 _dispatch 0x2d740700 pg_notify(1.43b(23) epoch 243) v5 2014-06-09 16:22:11.627446 7f869af78700 5 -- op tracker -- , seq: 2163, time: 2014-06-09 16:22:11.282483, event: header_read, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.627450 7f869af78700 5 -- op tracker -- , seq: 2163, time: 2014-06-09 16:22:11.282483, event: throttled, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.627453 7f869af78700 5 -- op tracker -- , seq: 2163, time: 2014-06-09 16:22:11.282511, event: all_read, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.627456 7f869af78700 5 -- op tracker -- , seq: 2163, time: 2014-06-09 16:22:11.627442, event: dispatched, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.627460 7f869af78700 5 -- op tracker -- , seq: 2163, time: 2014-06-09 16:22:11.627459, event: waiting_for_osdmap, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.627462 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.627463 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d740700 2014-06-09 16:22:11.627466 7f869af78700 5 -- op tracker -- , seq: 2163, time: 2014-06-09 16:22:11.627465, event: started, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.627474 7f869af78700 5 -- op tracker -- , seq: 2163, time: 2014-06-09 16:22:11.627473, event: done, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.627478 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627479 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627490 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627490 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627491 7f869af78700 20 osd.26 243 _dispatch 0x2d733880 pg_info(1 pgs e243:2.45b) v4 2014-06-09 16:22:11.627493 7f869af78700 5 -- op tracker -- , seq: 2164, time: 2014-06-09 16:22:11.375678, event: header_read, op: pg_info(1 pgs e243:2.45b) 2014-06-09 16:22:11.627497 7f869af78700 5 -- op tracker -- , seq: 2164, time: 2014-06-09 16:22:11.375678, event: throttled, op: pg_info(1 pgs e243:2.45b) 2014-06-09 16:22:11.627500 7f869af78700 5 -- op tracker -- , seq: 2164, time: 2014-06-09 16:22:11.375690, event: all_read, op: pg_info(1 pgs e243:2.45b) 2014-06-09 16:22:11.627504 7f869af78700 5 -- op tracker -- , seq: 2164, time: 2014-06-09 16:22:11.627489, event: dispatched, op: pg_info(1 pgs e243:2.45b) 2014-06-09 16:22:11.627509 7f869af78700 5 -- op tracker -- , seq: 2164, time: 2014-06-09 16:22:11.627508, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.45b) 2014-06-09 16:22:11.627511 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.45b) v4 from osd.7 2014-06-09 16:22:11.627513 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d733880 2014-06-09 16:22:11.627515 7f869af78700 5 -- op tracker -- , seq: 2164, time: 2014-06-09 16:22:11.627514, event: started, op: pg_info(1 pgs e243:2.45b) 2014-06-09 16:22:11.627524 7f869af78700 5 -- op tracker -- , seq: 2164, time: 2014-06-09 16:22:11.627523, event: done, op: pg_info(1 pgs e243:2.45b) 2014-06-09 16:22:11.627529 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627530 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627538 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627539 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627539 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a1c0 pg_notify(1.4c3(23) epoch 243) v5 2014-06-09 16:22:11.627542 7f869af78700 5 -- op tracker -- , seq: 2165, time: 2014-06-09 16:22:11.415586, event: header_read, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.627546 7f869af78700 5 -- op tracker -- , seq: 2165, time: 2014-06-09 16:22:11.415586, event: throttled, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.627549 7f869af78700 5 -- op tracker -- , seq: 2165, time: 2014-06-09 16:22:11.415622, event: all_read, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.627552 7f869af78700 5 -- op tracker -- , seq: 2165, time: 2014-06-09 16:22:11.627538, event: dispatched, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.627557 7f869af78700 5 -- op tracker -- , seq: 2165, time: 2014-06-09 16:22:11.627556, event: waiting_for_osdmap, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.627559 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.627560 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5a1c0 2014-06-09 16:22:11.627562 7f869af78700 5 -- op tracker -- , seq: 2165, time: 2014-06-09 16:22:11.627561, event: started, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.627571 7f869af78700 5 -- op tracker -- , seq: 2165, time: 2014-06-09 16:22:11.627570, event: done, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.627575 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627576 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627588 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627588 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627589 7f869af78700 20 osd.26 243 _dispatch 0x30641e00 pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.627593 7f869af78700 5 -- op tracker -- , seq: 2166, time: 2014-06-09 16:22:11.271364, event: header_read, op: pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627597 7f869af78700 5 -- op tracker -- , seq: 2166, time: 2014-06-09 16:22:11.271365, event: throttled, op: pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627600 7f869af78700 5 -- op tracker -- , seq: 2166, time: 2014-06-09 16:22:11.272610, event: all_read, op: pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627604 7f869af78700 5 -- op tracker -- , seq: 2166, time: 2014-06-09 16:22:11.627587, event: dispatched, op: pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627625 7f869af78700 5 -- op tracker -- , seq: 2166, time: 2014-06-09 16:22:11.627612, event: waiting_for_osdmap, op: pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627630 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) v4 from osd.1 2014-06-09 16:22:11.627634 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30641e00 2014-06-09 16:22:11.627640 7f869af78700 5 -- op tracker -- , seq: 2166, time: 2014-06-09 16:22:11.627637, event: started, op: pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627653 7f869af78700 5 -- op tracker -- , seq: 2166, time: 2014-06-09 16:22:11.627650, event: done, op: pg_log(3.75f epoch 243 log log((0'0,213'591], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627669 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627670 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627678 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627679 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627680 7f869af78700 20 osd.26 243 _dispatch 0x30a81200 pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.627684 7f869af78700 5 -- op tracker -- , seq: 2167, time: 2014-06-09 16:22:11.350046, event: header_read, op: pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627689 7f869af78700 5 -- op tracker -- , seq: 2167, time: 2014-06-09 16:22:11.350046, event: throttled, op: pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627693 7f869af78700 5 -- op tracker -- , seq: 2167, time: 2014-06-09 16:22:11.350067, event: all_read, op: pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627696 7f869af78700 5 -- op tracker -- , seq: 2167, time: 2014-06-09 16:22:11.627678, event: dispatched, op: pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627702 7f869af78700 5 -- op tracker -- , seq: 2167, time: 2014-06-09 16:22:11.627700, event: waiting_for_osdmap, op: pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627704 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) v4 from osd.9 2014-06-09 16:22:11.627706 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a81200 2014-06-09 16:22:11.627709 7f869af78700 5 -- op tracker -- , seq: 2167, time: 2014-06-09 16:22:11.627708, event: started, op: pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627718 7f869af78700 5 -- op tracker -- , seq: 2167, time: 2014-06-09 16:22:11.627717, event: done, op: pg_log(3.799 epoch 243 log log((213'662,214'665], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.627728 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627729 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627736 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627737 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627738 7f869af78700 20 osd.26 243 _dispatch 0x22012000 pg_info(1 pgs e243:0.221) v4 2014-06-09 16:22:11.627740 7f869af78700 5 -- op tracker -- , seq: 2168, time: 2014-06-09 16:22:11.362015, event: header_read, op: pg_info(1 pgs e243:0.221) 2014-06-09 16:22:11.627749 7f869af78700 5 -- op tracker -- , seq: 2168, time: 2014-06-09 16:22:11.362016, event: throttled, op: pg_info(1 pgs e243:0.221) 2014-06-09 16:22:11.627752 7f869af78700 5 -- op tracker -- , seq: 2168, time: 2014-06-09 16:22:11.362030, event: all_read, op: pg_info(1 pgs e243:0.221) 2014-06-09 16:22:11.627760 7f869af78700 5 -- op tracker -- , seq: 2168, time: 2014-06-09 16:22:11.627736, event: dispatched, op: pg_info(1 pgs e243:0.221) 2014-06-09 16:22:11.627764 7f869af78700 5 -- op tracker -- , seq: 2168, time: 2014-06-09 16:22:11.627763, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.221) 2014-06-09 16:22:11.627766 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.221) v4 from osd.8 2014-06-09 16:22:11.627768 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22012000 2014-06-09 16:22:11.627770 7f869af78700 5 -- op tracker -- , seq: 2168, time: 2014-06-09 16:22:11.627769, event: started, op: pg_info(1 pgs e243:0.221) 2014-06-09 16:22:11.627779 7f869af78700 5 -- op tracker -- , seq: 2168, time: 2014-06-09 16:22:11.627778, event: done, op: pg_info(1 pgs e243:0.221) 2014-06-09 16:22:11.627783 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627783 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627791 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627791 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627792 7f869af78700 20 osd.26 243 _dispatch 0x3046a380 pg_info(1 pgs e243:0.319) v4 2014-06-09 16:22:11.627795 7f869af78700 5 -- op tracker -- , seq: 2169, time: 2014-06-09 16:22:11.360670, event: header_read, op: pg_info(1 pgs e243:0.319) 2014-06-09 16:22:11.627800 7f869af78700 5 -- op tracker -- , seq: 2169, time: 2014-06-09 16:22:11.360671, event: throttled, op: pg_info(1 pgs e243:0.319) 2014-06-09 16:22:11.627803 7f869af78700 5 -- op tracker -- , seq: 2169, time: 2014-06-09 16:22:11.360691, event: all_read, op: pg_info(1 pgs e243:0.319) 2014-06-09 16:22:11.627807 7f869af78700 5 -- op tracker -- , seq: 2169, time: 2014-06-09 16:22:11.627790, event: dispatched, op: pg_info(1 pgs e243:0.319) 2014-06-09 16:22:11.627810 7f869af78700 5 -- op tracker -- , seq: 2169, time: 2014-06-09 16:22:11.627809, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.319) 2014-06-09 16:22:11.627812 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.319) v4 from osd.6 2014-06-09 16:22:11.627814 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046a380 2014-06-09 16:22:11.627816 7f869af78700 5 -- op tracker -- , seq: 2169, time: 2014-06-09 16:22:11.627815, event: started, op: pg_info(1 pgs e243:0.319) 2014-06-09 16:22:11.627824 7f869af78700 5 -- op tracker -- , seq: 2169, time: 2014-06-09 16:22:11.627823, event: done, op: pg_info(1 pgs e243:0.319) 2014-06-09 16:22:11.627828 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627829 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627836 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627836 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627837 7f869af78700 20 osd.26 243 _dispatch 0x29ec5880 pg_info(1 pgs e243:1.86) v4 2014-06-09 16:22:11.627840 7f869af78700 5 -- op tracker -- , seq: 2170, time: 2014-06-09 16:22:11.259705, event: header_read, op: pg_info(1 pgs e243:1.86) 2014-06-09 16:22:11.627845 7f869af78700 5 -- op tracker -- , seq: 2170, time: 2014-06-09 16:22:11.259705, event: throttled, op: pg_info(1 pgs e243:1.86) 2014-06-09 16:22:11.627848 7f869af78700 5 -- op tracker -- , seq: 2170, time: 2014-06-09 16:22:11.259721, event: all_read, op: pg_info(1 pgs e243:1.86) 2014-06-09 16:22:11.627853 7f869af78700 5 -- op tracker -- , seq: 2170, time: 2014-06-09 16:22:11.627836, event: dispatched, op: pg_info(1 pgs e243:1.86) 2014-06-09 16:22:11.627856 7f869af78700 5 -- op tracker -- , seq: 2170, time: 2014-06-09 16:22:11.627855, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.86) 2014-06-09 16:22:11.627859 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.86) v4 from osd.0 2014-06-09 16:22:11.627860 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec5880 2014-06-09 16:22:11.627862 7f869af78700 5 -- op tracker -- , seq: 2170, time: 2014-06-09 16:22:11.627861, event: started, op: pg_info(1 pgs e243:1.86) 2014-06-09 16:22:11.627871 7f869af78700 5 -- op tracker -- , seq: 2170, time: 2014-06-09 16:22:11.627870, event: done, op: pg_info(1 pgs e243:1.86) 2014-06-09 16:22:11.627878 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627879 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627886 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627886 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627887 7f869af78700 20 osd.26 243 _dispatch 0x240e81c0 pg_info(1 pgs e243:2.5a7) v4 2014-06-09 16:22:11.627889 7f869af78700 5 -- op tracker -- , seq: 2171, time: 2014-06-09 16:22:11.382153, event: header_read, op: pg_info(1 pgs e243:2.5a7) 2014-06-09 16:22:11.627894 7f869af78700 5 -- op tracker -- , seq: 2171, time: 2014-06-09 16:22:11.382154, event: throttled, op: pg_info(1 pgs e243:2.5a7) 2014-06-09 16:22:11.627902 7f869af78700 5 -- op tracker -- , seq: 2171, time: 2014-06-09 16:22:11.382164, event: all_read, op: pg_info(1 pgs e243:2.5a7) 2014-06-09 16:22:11.627906 7f869af78700 5 -- op tracker -- , seq: 2171, time: 2014-06-09 16:22:11.627885, event: dispatched, op: pg_info(1 pgs e243:2.5a7) 2014-06-09 16:22:11.627912 7f869af78700 5 -- op tracker -- , seq: 2171, time: 2014-06-09 16:22:11.627911, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5a7) 2014-06-09 16:22:11.627915 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5a7) v4 from osd.5 2014-06-09 16:22:11.627916 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e81c0 2014-06-09 16:22:11.627919 7f869af78700 5 -- op tracker -- , seq: 2171, time: 2014-06-09 16:22:11.627918, event: started, op: pg_info(1 pgs e243:2.5a7) 2014-06-09 16:22:11.627927 7f869af78700 5 -- op tracker -- , seq: 2171, time: 2014-06-09 16:22:11.627926, event: done, op: pg_info(1 pgs e243:2.5a7) 2014-06-09 16:22:11.627931 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627932 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627939 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627940 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627941 7f869af78700 20 osd.26 243 _dispatch 0x30880c00 pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.627944 7f869af78700 5 -- op tracker -- , seq: 2172, time: 2014-06-09 16:22:11.211344, event: header_read, op: pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627949 7f869af78700 5 -- op tracker -- , seq: 2172, time: 2014-06-09 16:22:11.211344, event: throttled, op: pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627952 7f869af78700 5 -- op tracker -- , seq: 2172, time: 2014-06-09 16:22:11.211358, event: all_read, op: pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627956 7f869af78700 5 -- op tracker -- , seq: 2172, time: 2014-06-09 16:22:11.627939, event: dispatched, op: pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627960 7f869af78700 5 -- op tracker -- , seq: 2172, time: 2014-06-09 16:22:11.627959, event: waiting_for_osdmap, op: pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627962 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.627965 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30880c00 2014-06-09 16:22:11.627967 7f869af78700 5 -- op tracker -- , seq: 2172, time: 2014-06-09 16:22:11.627966, event: started, op: pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627975 7f869af78700 5 -- op tracker -- , seq: 2172, time: 2014-06-09 16:22:11.627973, event: done, op: pg_log(2.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627978 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627979 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627986 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.627987 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.627987 7f869af78700 20 osd.26 243 _dispatch 0x308a6000 pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.627991 7f869af78700 5 -- op tracker -- , seq: 2173, time: 2014-06-09 16:22:11.284918, event: header_read, op: pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627995 7f869af78700 5 -- op tracker -- , seq: 2173, time: 2014-06-09 16:22:11.284919, event: throttled, op: pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.627998 7f869af78700 5 -- op tracker -- , seq: 2173, time: 2014-06-09 16:22:11.284935, event: all_read, op: pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.628002 7f869af78700 5 -- op tracker -- , seq: 2173, time: 2014-06-09 16:22:11.627986, event: dispatched, op: pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.628006 7f869af78700 5 -- op tracker -- , seq: 2173, time: 2014-06-09 16:22:11.628004, event: waiting_for_osdmap, op: pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.628009 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.628011 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a6000 2014-06-09 16:22:11.628013 7f869af78700 5 -- op tracker -- , seq: 2173, time: 2014-06-09 16:22:11.628012, event: started, op: pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.628020 7f869af78700 5 -- op tracker -- , seq: 2173, time: 2014-06-09 16:22:11.628019, event: done, op: pg_log(1.6b6 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.628024 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628025 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628032 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628032 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628033 7f869af78700 20 osd.26 243 _dispatch 0x30725e00 pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.628037 7f869af78700 5 -- op tracker -- , seq: 2174, time: 2014-06-09 16:22:11.375694, event: header_read, op: pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.628045 7f869af78700 5 -- op tracker -- , seq: 2174, time: 2014-06-09 16:22:11.375695, event: throttled, op: pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.628048 7f869af78700 5 -- op tracker -- , seq: 2174, time: 2014-06-09 16:22:11.375712, event: all_read, op: pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.628052 7f869af78700 5 -- op tracker -- , seq: 2174, time: 2014-06-09 16:22:11.628032, event: dispatched, op: pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.628060 7f869af78700 5 -- op tracker -- , seq: 2174, time: 2014-06-09 16:22:11.628058, event: waiting_for_osdmap, op: pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.628062 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) v4 from osd.7 2014-06-09 16:22:11.628064 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30725e00 2014-06-09 16:22:11.628067 7f869af78700 5 -- op tracker -- , seq: 2174, time: 2014-06-09 16:22:11.628066, event: started, op: pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.628075 7f869af78700 5 -- op tracker -- , seq: 2174, time: 2014-06-09 16:22:11.628073, event: done, op: pg_log(3.4c0 epoch 243 log log((214'583,214'586], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.628079 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628080 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628087 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628088 7f869af78700 20 osd.26 243 _dispatch 0x310e8000 pg_info(1 pgs e243:0.3f9) v4 2014-06-09 16:22:11.628091 7f869af78700 5 -- op tracker -- , seq: 2175, time: 2014-06-09 16:22:11.433849, event: header_read, op: pg_info(1 pgs e243:0.3f9) 2014-06-09 16:22:11.628095 7f869af78700 5 -- op tracker -- , seq: 2175, time: 2014-06-09 16:22:11.433850, event: throttled, op: pg_info(1 pgs e243:0.3f9) 2014-06-09 16:22:11.628098 7f869af78700 5 -- op tracker -- , seq: 2175, time: 2014-06-09 16:22:11.433862, event: all_read, op: pg_info(1 pgs e243:0.3f9) 2014-06-09 16:22:11.628102 7f869af78700 5 -- op tracker -- , seq: 2175, time: 2014-06-09 16:22:11.628087, event: dispatched, op: pg_info(1 pgs e243:0.3f9) 2014-06-09 16:22:11.628106 7f869af78700 5 -- op tracker -- , seq: 2175, time: 2014-06-09 16:22:11.628104, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3f9) 2014-06-09 16:22:11.628108 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3f9) v4 from osd.3 2014-06-09 16:22:11.628109 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e8000 2014-06-09 16:22:11.628112 7f869af78700 5 -- op tracker -- , seq: 2175, time: 2014-06-09 16:22:11.628111, event: started, op: pg_info(1 pgs e243:0.3f9) 2014-06-09 16:22:11.628122 7f869af78700 5 -- op tracker -- , seq: 2175, time: 2014-06-09 16:22:11.628121, event: done, op: pg_info(1 pgs e243:0.3f9) 2014-06-09 16:22:11.628126 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628127 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628134 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628135 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628135 7f869af78700 20 osd.26 243 _dispatch 0x308e3180 pg_notify(0.1fb(21),1.4c2(21),2.1fa(21),3.78a(12) epoch 243) v5 2014-06-09 16:22:11.628140 7f869af78700 5 -- op tracker -- , seq: 2176, time: 2014-06-09 16:22:11.272636, event: header_read, op: pg_notify(0.1fb(21),1.4c2(21),2.1fa(21),3.78a(12) epoch 243) 2014-06-09 16:22:11.628144 7f869af78700 5 -- op tracker -- , seq: 2176, time: 2014-06-09 16:22:11.272637, event: throttled, op: pg_notify(0.1fb(21),1.4c2(21),2.1fa(21),3.78a(12) epoch 243) 2014-06-09 16:22:11.628149 7f869af78700 5 -- op tracker -- , seq: 2176, time: 2014-06-09 16:22:11.272720, event: all_read, op: pg_notify(0.1fb(21),1.4c2(21),2.1fa(21),3.78a(12) epoch 243) 2014-06-09 16:22:11.628156 7f869af78700 5 -- op tracker -- , seq: 2176, time: 2014-06-09 16:22:11.628134, event: dispatched, op: pg_notify(0.1fb(21),1.4c2(21),2.1fa(21),3.78a(12) epoch 243) 2014-06-09 16:22:11.628160 7f869af78700 5 -- op tracker -- , seq: 2176, time: 2014-06-09 16:22:11.628159, event: waiting_for_osdmap, op: pg_notify(0.1fb(21),1.4c2(21),2.1fa(21),3.78a(12) epoch 243) 2014-06-09 16:22:11.628163 7f869af78700 7 osd.26 243 handle_pg_notify from osd.1 2014-06-09 16:22:11.628164 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e3180 2014-06-09 16:22:11.628167 7f869af78700 5 -- op tracker -- , seq: 2176, time: 2014-06-09 16:22:11.628165, event: started, op: pg_notify(0.1fb(21),1.4c2(21),2.1fa(21),3.78a(12) epoch 243) 2014-06-09 16:22:11.628188 7f869af78700 5 -- op tracker -- , seq: 2176, time: 2014-06-09 16:22:11.628187, event: done, op: pg_notify(0.1fb(21),1.4c2(21),2.1fa(21),3.78a(12) epoch 243) 2014-06-09 16:22:11.628193 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628194 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628212 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628213 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628214 7f869af78700 20 osd.26 243 _dispatch 0x19fdfc00 pg_notify(3.799(10),1.4d3(20),3.173(10) epoch 243) v5 2014-06-09 16:22:11.628217 7f869af78700 5 -- op tracker -- , seq: 2177, time: 2014-06-09 16:22:11.350236, event: header_read, op: pg_notify(3.799(10),1.4d3(20),3.173(10) epoch 243) 2014-06-09 16:22:11.628222 7f869af78700 5 -- op tracker -- , seq: 2177, time: 2014-06-09 16:22:11.350237, event: throttled, op: pg_notify(3.799(10),1.4d3(20),3.173(10) epoch 243) 2014-06-09 16:22:11.628225 7f869af78700 5 -- op tracker -- , seq: 2177, time: 2014-06-09 16:22:11.350294, event: all_read, op: pg_notify(3.799(10),1.4d3(20),3.173(10) epoch 243) 2014-06-09 16:22:11.628229 7f869af78700 5 -- op tracker -- , seq: 2177, time: 2014-06-09 16:22:11.628212, event: dispatched, op: pg_notify(3.799(10),1.4d3(20),3.173(10) epoch 243) 2014-06-09 16:22:11.628233 7f869af78700 5 -- op tracker -- , seq: 2177, time: 2014-06-09 16:22:11.628231, event: waiting_for_osdmap, op: pg_notify(3.799(10),1.4d3(20),3.173(10) epoch 243) 2014-06-09 16:22:11.628235 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.628236 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdfc00 2014-06-09 16:22:11.628239 7f869af78700 5 -- op tracker -- , seq: 2177, time: 2014-06-09 16:22:11.628238, event: started, op: pg_notify(3.799(10),1.4d3(20),3.173(10) epoch 243) 2014-06-09 16:22:11.628257 7f869af78700 5 -- op tracker -- , seq: 2177, time: 2014-06-09 16:22:11.628255, event: done, op: pg_notify(3.799(10),1.4d3(20),3.173(10) epoch 243) 2014-06-09 16:22:11.628261 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628262 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628274 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628275 7f869af78700 20 osd.26 243 _dispatch 0x22012380 pg_info(1 pgs e243:1.223) v4 2014-06-09 16:22:11.628278 7f869af78700 5 -- op tracker -- , seq: 2178, time: 2014-06-09 16:22:11.362039, event: header_read, op: pg_info(1 pgs e243:1.223) 2014-06-09 16:22:11.628281 7f869af78700 5 -- op tracker -- , seq: 2178, time: 2014-06-09 16:22:11.362040, event: throttled, op: pg_info(1 pgs e243:1.223) 2014-06-09 16:22:11.628288 7f869af78700 5 -- op tracker -- , seq: 2178, time: 2014-06-09 16:22:11.362051, event: all_read, op: pg_info(1 pgs e243:1.223) 2014-06-09 16:22:11.628292 7f869af78700 5 -- op tracker -- , seq: 2178, time: 2014-06-09 16:22:11.628273, event: dispatched, op: pg_info(1 pgs e243:1.223) 2014-06-09 16:22:11.628299 7f869af78700 5 -- op tracker -- , seq: 2178, time: 2014-06-09 16:22:11.628297, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.223) 2014-06-09 16:22:11.628301 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.223) v4 from osd.8 2014-06-09 16:22:11.628303 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22012380 2014-06-09 16:22:11.628305 7f869af78700 5 -- op tracker -- , seq: 2178, time: 2014-06-09 16:22:11.628304, event: started, op: pg_info(1 pgs e243:1.223) 2014-06-09 16:22:11.628313 7f869af78700 5 -- op tracker -- , seq: 2178, time: 2014-06-09 16:22:11.628312, event: done, op: pg_info(1 pgs e243:1.223) 2014-06-09 16:22:11.628320 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628321 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628328 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628328 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628329 7f869af78700 20 osd.26 243 _dispatch 0x1c03f180 pg_info(1 pgs e243:2.19) v4 2014-06-09 16:22:11.628332 7f869af78700 5 -- op tracker -- , seq: 2179, time: 2014-06-09 16:22:11.360735, event: header_read, op: pg_info(1 pgs e243:2.19) 2014-06-09 16:22:11.628335 7f869af78700 5 -- op tracker -- , seq: 2179, time: 2014-06-09 16:22:11.360736, event: throttled, op: pg_info(1 pgs e243:2.19) 2014-06-09 16:22:11.628339 7f869af78700 5 -- op tracker -- , seq: 2179, time: 2014-06-09 16:22:11.360750, event: all_read, op: pg_info(1 pgs e243:2.19) 2014-06-09 16:22:11.628342 7f869af78700 5 -- op tracker -- , seq: 2179, time: 2014-06-09 16:22:11.628328, event: dispatched, op: pg_info(1 pgs e243:2.19) 2014-06-09 16:22:11.628346 7f869af78700 5 -- op tracker -- , seq: 2179, time: 2014-06-09 16:22:11.628345, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.19) 2014-06-09 16:22:11.628348 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.19) v4 from osd.6 2014-06-09 16:22:11.628349 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03f180 2014-06-09 16:22:11.628352 7f869af78700 5 -- op tracker -- , seq: 2179, time: 2014-06-09 16:22:11.628351, event: started, op: pg_info(1 pgs e243:2.19) 2014-06-09 16:22:11.628361 7f869af78700 5 -- op tracker -- , seq: 2179, time: 2014-06-09 16:22:11.628360, event: done, op: pg_info(1 pgs e243:2.19) 2014-06-09 16:22:11.628365 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628365 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628372 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628373 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628373 7f869af78700 20 osd.26 243 _dispatch 0x308d8380 pg_info(1 pgs e243:1.2aa) v4 2014-06-09 16:22:11.628376 7f869af78700 5 -- op tracker -- , seq: 2180, time: 2014-06-09 16:22:11.259731, event: header_read, op: pg_info(1 pgs e243:1.2aa) 2014-06-09 16:22:11.628380 7f869af78700 5 -- op tracker -- , seq: 2180, time: 2014-06-09 16:22:11.259732, event: throttled, op: pg_info(1 pgs e243:1.2aa) 2014-06-09 16:22:11.628383 7f869af78700 5 -- op tracker -- , seq: 2180, time: 2014-06-09 16:22:11.259740, event: all_read, op: pg_info(1 pgs e243:1.2aa) 2014-06-09 16:22:11.628386 7f869af78700 5 -- op tracker -- , seq: 2180, time: 2014-06-09 16:22:11.628372, event: dispatched, op: pg_info(1 pgs e243:1.2aa) 2014-06-09 16:22:11.628390 7f869af78700 5 -- op tracker -- , seq: 2180, time: 2014-06-09 16:22:11.628389, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2aa) 2014-06-09 16:22:11.628396 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2aa) v4 from osd.0 2014-06-09 16:22:11.628397 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d8380 2014-06-09 16:22:11.628400 7f869af78700 5 -- op tracker -- , seq: 2180, time: 2014-06-09 16:22:11.628399, event: started, op: pg_info(1 pgs e243:1.2aa) 2014-06-09 16:22:11.628408 7f869af78700 5 -- op tracker -- , seq: 2180, time: 2014-06-09 16:22:11.628407, event: done, op: pg_info(1 pgs e243:1.2aa) 2014-06-09 16:22:11.628412 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628413 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628421 7f869af78700 20 osd.26 243 _dispatch 0x240e8540 pg_info(1 pgs e243:0.5a8) v4 2014-06-09 16:22:11.628424 7f869af78700 5 -- op tracker -- , seq: 2181, time: 2014-06-09 16:22:11.382207, event: header_read, op: pg_info(1 pgs e243:0.5a8) 2014-06-09 16:22:11.628427 7f869af78700 5 -- op tracker -- , seq: 2181, time: 2014-06-09 16:22:11.382208, event: throttled, op: pg_info(1 pgs e243:0.5a8) 2014-06-09 16:22:11.628430 7f869af78700 5 -- op tracker -- , seq: 2181, time: 2014-06-09 16:22:11.382220, event: all_read, op: pg_info(1 pgs e243:0.5a8) 2014-06-09 16:22:11.628434 7f869af78700 5 -- op tracker -- , seq: 2181, time: 2014-06-09 16:22:11.628420, event: dispatched, op: pg_info(1 pgs e243:0.5a8) 2014-06-09 16:22:11.628437 7f869af78700 5 -- op tracker -- , seq: 2181, time: 2014-06-09 16:22:11.628436, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5a8) 2014-06-09 16:22:11.628439 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5a8) v4 from osd.5 2014-06-09 16:22:11.628441 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e8540 2014-06-09 16:22:11.628443 7f869af78700 5 -- op tracker -- , seq: 2181, time: 2014-06-09 16:22:11.628442, event: started, op: pg_info(1 pgs e243:0.5a8) 2014-06-09 16:22:11.628451 7f869af78700 5 -- op tracker -- , seq: 2181, time: 2014-06-09 16:22:11.628450, event: done, op: pg_info(1 pgs e243:0.5a8) 2014-06-09 16:22:11.628457 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628457 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628465 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628465 7f869af78700 20 osd.26 243 _dispatch 0x3088d6c0 pg_notify(0.411(21) epoch 243) v5 2014-06-09 16:22:11.628468 7f869af78700 5 -- op tracker -- , seq: 2182, time: 2014-06-09 16:22:11.216487, event: header_read, op: pg_notify(0.411(21) epoch 243) 2014-06-09 16:22:11.628472 7f869af78700 5 -- op tracker -- , seq: 2182, time: 2014-06-09 16:22:11.216488, event: throttled, op: pg_notify(0.411(21) epoch 243) 2014-06-09 16:22:11.628475 7f869af78700 5 -- op tracker -- , seq: 2182, time: 2014-06-09 16:22:11.216516, event: all_read, op: pg_notify(0.411(21) epoch 243) 2014-06-09 16:22:11.628478 7f869af78700 5 -- op tracker -- , seq: 2182, time: 2014-06-09 16:22:11.628464, event: dispatched, op: pg_notify(0.411(21) epoch 243) 2014-06-09 16:22:11.628483 7f869af78700 5 -- op tracker -- , seq: 2182, time: 2014-06-09 16:22:11.628482, event: waiting_for_osdmap, op: pg_notify(0.411(21) epoch 243) 2014-06-09 16:22:11.628486 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.628487 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088d6c0 2014-06-09 16:22:11.628489 7f869af78700 5 -- op tracker -- , seq: 2182, time: 2014-06-09 16:22:11.628488, event: started, op: pg_notify(0.411(21) epoch 243) 2014-06-09 16:22:11.628498 7f869af78700 5 -- op tracker -- , seq: 2182, time: 2014-06-09 16:22:11.628497, event: done, op: pg_notify(0.411(21) epoch 243) 2014-06-09 16:22:11.628502 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628502 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628514 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628515 7f869af78700 20 osd.26 243 _dispatch 0x19805340 pg_notify(3.6cd(10) epoch 243) v5 2014-06-09 16:22:11.628518 7f869af78700 5 -- op tracker -- , seq: 2183, time: 2014-06-09 16:22:11.285655, event: header_read, op: pg_notify(3.6cd(10) epoch 243) 2014-06-09 16:22:11.628523 7f869af78700 5 -- op tracker -- , seq: 2183, time: 2014-06-09 16:22:11.285655, event: throttled, op: pg_notify(3.6cd(10) epoch 243) 2014-06-09 16:22:11.628526 7f869af78700 5 -- op tracker -- , seq: 2183, time: 2014-06-09 16:22:11.285676, event: all_read, op: pg_notify(3.6cd(10) epoch 243) 2014-06-09 16:22:11.628531 7f869af78700 5 -- op tracker -- , seq: 2183, time: 2014-06-09 16:22:11.628513, event: dispatched, op: pg_notify(3.6cd(10) epoch 243) 2014-06-09 16:22:11.628534 7f869af78700 5 -- op tracker -- , seq: 2183, time: 2014-06-09 16:22:11.628533, event: waiting_for_osdmap, op: pg_notify(3.6cd(10) epoch 243) 2014-06-09 16:22:11.628536 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.628537 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19805340 2014-06-09 16:22:11.628540 7f869af78700 5 -- op tracker -- , seq: 2183, time: 2014-06-09 16:22:11.628539, event: started, op: pg_notify(3.6cd(10) epoch 243) 2014-06-09 16:22:11.628549 7f869af78700 5 -- op tracker -- , seq: 2183, time: 2014-06-09 16:22:11.628548, event: done, op: pg_notify(3.6cd(10) epoch 243) 2014-06-09 16:22:11.628553 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628554 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628563 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628564 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628564 7f869af78700 20 osd.26 243 _dispatch 0x2d732380 pg_notify(3.4c0(10) epoch 243) v5 2014-06-09 16:22:11.628567 7f869af78700 5 -- op tracker -- , seq: 2184, time: 2014-06-09 16:22:11.375717, event: header_read, op: pg_notify(3.4c0(10) epoch 243) 2014-06-09 16:22:11.628571 7f869af78700 5 -- op tracker -- , seq: 2184, time: 2014-06-09 16:22:11.375718, event: throttled, op: pg_notify(3.4c0(10) epoch 243) 2014-06-09 16:22:11.628575 7f869af78700 5 -- op tracker -- , seq: 2184, time: 2014-06-09 16:22:11.375737, event: all_read, op: pg_notify(3.4c0(10) epoch 243) 2014-06-09 16:22:11.628579 7f869af78700 5 -- op tracker -- , seq: 2184, time: 2014-06-09 16:22:11.628563, event: dispatched, op: pg_notify(3.4c0(10) epoch 243) 2014-06-09 16:22:11.628582 7f869af78700 5 -- op tracker -- , seq: 2184, time: 2014-06-09 16:22:11.628581, event: waiting_for_osdmap, op: pg_notify(3.4c0(10) epoch 243) 2014-06-09 16:22:11.628585 7f869af78700 7 osd.26 243 handle_pg_notify from osd.7 2014-06-09 16:22:11.628586 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d732380 2014-06-09 16:22:11.628588 7f869af78700 5 -- op tracker -- , seq: 2184, time: 2014-06-09 16:22:11.628587, event: started, op: pg_notify(3.4c0(10) epoch 243) 2014-06-09 16:22:11.628597 7f869af78700 5 -- op tracker -- , seq: 2184, time: 2014-06-09 16:22:11.628596, event: done, op: pg_notify(3.4c0(10) epoch 243) 2014-06-09 16:22:11.628601 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628602 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628611 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628611 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628612 7f869af78700 20 osd.26 243 _dispatch 0x2ea5bdc0 pg_info(1 pgs e243:0.21b) v4 2014-06-09 16:22:11.628614 7f869af78700 5 -- op tracker -- , seq: 2185, time: 2014-06-09 16:22:11.433946, event: header_read, op: pg_info(1 pgs e243:0.21b) 2014-06-09 16:22:11.628618 7f869af78700 5 -- op tracker -- , seq: 2185, time: 2014-06-09 16:22:11.433947, event: throttled, op: pg_info(1 pgs e243:0.21b) 2014-06-09 16:22:11.628626 7f869af78700 5 -- op tracker -- , seq: 2185, time: 2014-06-09 16:22:11.433963, event: all_read, op: pg_info(1 pgs e243:0.21b) 2014-06-09 16:22:11.628629 7f869af78700 5 -- op tracker -- , seq: 2185, time: 2014-06-09 16:22:11.628610, event: dispatched, op: pg_info(1 pgs e243:0.21b) 2014-06-09 16:22:11.628636 7f869af78700 5 -- op tracker -- , seq: 2185, time: 2014-06-09 16:22:11.628635, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.21b) 2014-06-09 16:22:11.628638 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.21b) v4 from osd.3 2014-06-09 16:22:11.628640 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5bdc0 2014-06-09 16:22:11.628643 7f869af78700 5 -- op tracker -- , seq: 2185, time: 2014-06-09 16:22:11.628642, event: started, op: pg_info(1 pgs e243:0.21b) 2014-06-09 16:22:11.628651 7f869af78700 5 -- op tracker -- , seq: 2185, time: 2014-06-09 16:22:11.628650, event: done, op: pg_info(1 pgs e243:0.21b) 2014-06-09 16:22:11.628655 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628656 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628663 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628663 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628664 7f869af78700 20 osd.26 243 _dispatch 0x15458a80 pg_info(1 pgs e243:1.764) v4 2014-06-09 16:22:11.628667 7f869af78700 5 -- op tracker -- , seq: 2186, time: 2014-06-09 16:22:11.284610, event: header_read, op: pg_info(1 pgs e243:1.764) 2014-06-09 16:22:11.628670 7f869af78700 5 -- op tracker -- , seq: 2186, time: 2014-06-09 16:22:11.284611, event: throttled, op: pg_info(1 pgs e243:1.764) 2014-06-09 16:22:11.628674 7f869af78700 5 -- op tracker -- , seq: 2186, time: 2014-06-09 16:22:11.284623, event: all_read, op: pg_info(1 pgs e243:1.764) 2014-06-09 16:22:11.628678 7f869af78700 5 -- op tracker -- , seq: 2186, time: 2014-06-09 16:22:11.628663, event: dispatched, op: pg_info(1 pgs e243:1.764) 2014-06-09 16:22:11.628681 7f869af78700 5 -- op tracker -- , seq: 2186, time: 2014-06-09 16:22:11.628680, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.764) 2014-06-09 16:22:11.628683 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.764) v4 from osd.1 2014-06-09 16:22:11.628685 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15458a80 2014-06-09 16:22:11.628687 7f869af78700 5 -- op tracker -- , seq: 2186, time: 2014-06-09 16:22:11.628686, event: started, op: pg_info(1 pgs e243:1.764) 2014-06-09 16:22:11.628695 7f869af78700 5 -- op tracker -- , seq: 2186, time: 2014-06-09 16:22:11.628694, event: done, op: pg_info(1 pgs e243:1.764) 2014-06-09 16:22:11.628699 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628700 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628706 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628707 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628707 7f869af78700 20 osd.26 243 _dispatch 0x19fdf180 pg_notify(3.6d3(10) epoch 243) v5 2014-06-09 16:22:11.628710 7f869af78700 5 -- op tracker -- , seq: 2187, time: 2014-06-09 16:22:11.351304, event: header_read, op: pg_notify(3.6d3(10) epoch 243) 2014-06-09 16:22:11.628714 7f869af78700 5 -- op tracker -- , seq: 2187, time: 2014-06-09 16:22:11.351305, event: throttled, op: pg_notify(3.6d3(10) epoch 243) 2014-06-09 16:22:11.628717 7f869af78700 5 -- op tracker -- , seq: 2187, time: 2014-06-09 16:22:11.351327, event: all_read, op: pg_notify(3.6d3(10) epoch 243) 2014-06-09 16:22:11.628721 7f869af78700 5 -- op tracker -- , seq: 2187, time: 2014-06-09 16:22:11.628706, event: dispatched, op: pg_notify(3.6d3(10) epoch 243) 2014-06-09 16:22:11.628724 7f869af78700 5 -- op tracker -- , seq: 2187, time: 2014-06-09 16:22:11.628723, event: waiting_for_osdmap, op: pg_notify(3.6d3(10) epoch 243) 2014-06-09 16:22:11.628727 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.628728 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdf180 2014-06-09 16:22:11.628730 7f869af78700 5 -- op tracker -- , seq: 2187, time: 2014-06-09 16:22:11.628729, event: started, op: pg_notify(3.6d3(10) epoch 243) 2014-06-09 16:22:11.628739 7f869af78700 5 -- op tracker -- , seq: 2187, time: 2014-06-09 16:22:11.628738, event: done, op: pg_notify(3.6d3(10) epoch 243) 2014-06-09 16:22:11.628743 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628744 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628751 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628752 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628753 7f869af78700 20 osd.26 243 _dispatch 0x230bafc0 pg_info(1 pgs e243:0.dc) v4 2014-06-09 16:22:11.628755 7f869af78700 5 -- op tracker -- , seq: 2188, time: 2014-06-09 16:22:11.362080, event: header_read, op: pg_info(1 pgs e243:0.dc) 2014-06-09 16:22:11.628760 7f869af78700 5 -- op tracker -- , seq: 2188, time: 2014-06-09 16:22:11.362080, event: throttled, op: pg_info(1 pgs e243:0.dc) 2014-06-09 16:22:11.628764 7f869af78700 5 -- op tracker -- , seq: 2188, time: 2014-06-09 16:22:11.362094, event: all_read, op: pg_info(1 pgs e243:0.dc) 2014-06-09 16:22:11.628768 7f869af78700 5 -- op tracker -- , seq: 2188, time: 2014-06-09 16:22:11.628751, event: dispatched, op: pg_info(1 pgs e243:0.dc) 2014-06-09 16:22:11.628772 7f869af78700 5 -- op tracker -- , seq: 2188, time: 2014-06-09 16:22:11.628771, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.dc) 2014-06-09 16:22:11.628774 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.dc) v4 from osd.8 2014-06-09 16:22:11.628775 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230bafc0 2014-06-09 16:22:11.628778 7f869af78700 5 -- op tracker -- , seq: 2188, time: 2014-06-09 16:22:11.628777, event: started, op: pg_info(1 pgs e243:0.dc) 2014-06-09 16:22:11.628786 7f869af78700 5 -- op tracker -- , seq: 2188, time: 2014-06-09 16:22:11.628785, event: done, op: pg_info(1 pgs e243:0.dc) 2014-06-09 16:22:11.628790 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628791 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628797 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628798 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628798 7f869af78700 20 osd.26 243 _dispatch 0x31041a40 pg_info(1 pgs e243:1.b3) v4 2014-06-09 16:22:11.628801 7f869af78700 5 -- op tracker -- , seq: 2189, time: 2014-06-09 16:22:11.360763, event: header_read, op: pg_info(1 pgs e243:1.b3) 2014-06-09 16:22:11.628805 7f869af78700 5 -- op tracker -- , seq: 2189, time: 2014-06-09 16:22:11.360764, event: throttled, op: pg_info(1 pgs e243:1.b3) 2014-06-09 16:22:11.628808 7f869af78700 5 -- op tracker -- , seq: 2189, time: 2014-06-09 16:22:11.360776, event: all_read, op: pg_info(1 pgs e243:1.b3) 2014-06-09 16:22:11.628811 7f869af78700 5 -- op tracker -- , seq: 2189, time: 2014-06-09 16:22:11.628797, event: dispatched, op: pg_info(1 pgs e243:1.b3) 2014-06-09 16:22:11.628816 7f869af78700 5 -- op tracker -- , seq: 2189, time: 2014-06-09 16:22:11.628815, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.b3) 2014-06-09 16:22:11.628818 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.b3) v4 from osd.6 2014-06-09 16:22:11.628820 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31041a40 2014-06-09 16:22:11.628822 7f869af78700 5 -- op tracker -- , seq: 2189, time: 2014-06-09 16:22:11.628821, event: started, op: pg_info(1 pgs e243:1.b3) 2014-06-09 16:22:11.628832 7f869af78700 5 -- op tracker -- , seq: 2189, time: 2014-06-09 16:22:11.628831, event: done, op: pg_info(1 pgs e243:1.b3) 2014-06-09 16:22:11.628836 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628837 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628843 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628844 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628844 7f869af78700 20 osd.26 243 _dispatch 0x30b8d500 pg_info(1 pgs e243:2.754) v4 2014-06-09 16:22:11.628847 7f869af78700 5 -- op tracker -- , seq: 2190, time: 2014-06-09 16:22:11.260963, event: header_read, op: pg_info(1 pgs e243:2.754) 2014-06-09 16:22:11.628852 7f869af78700 5 -- op tracker -- , seq: 2190, time: 2014-06-09 16:22:11.260964, event: throttled, op: pg_info(1 pgs e243:2.754) 2014-06-09 16:22:11.628856 7f869af78700 5 -- op tracker -- , seq: 2190, time: 2014-06-09 16:22:11.260982, event: all_read, op: pg_info(1 pgs e243:2.754) 2014-06-09 16:22:11.628860 7f869af78700 5 -- op tracker -- , seq: 2190, time: 2014-06-09 16:22:11.628843, event: dispatched, op: pg_info(1 pgs e243:2.754) 2014-06-09 16:22:11.628864 7f869af78700 5 -- op tracker -- , seq: 2190, time: 2014-06-09 16:22:11.628863, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.754) 2014-06-09 16:22:11.628867 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.754) v4 from osd.0 2014-06-09 16:22:11.628869 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8d500 2014-06-09 16:22:11.628871 7f869af78700 5 -- op tracker -- , seq: 2190, time: 2014-06-09 16:22:11.628870, event: started, op: pg_info(1 pgs e243:2.754) 2014-06-09 16:22:11.628883 7f869af78700 5 -- op tracker -- , seq: 2190, time: 2014-06-09 16:22:11.628882, event: done, op: pg_info(1 pgs e243:2.754) 2014-06-09 16:22:11.628890 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628891 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628898 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628899 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628899 7f869af78700 20 osd.26 243 _dispatch 0x240e9180 pg_info(1 pgs e243:1.b8) v4 2014-06-09 16:22:11.628902 7f869af78700 5 -- op tracker -- , seq: 2191, time: 2014-06-09 16:22:11.382227, event: header_read, op: pg_info(1 pgs e243:1.b8) 2014-06-09 16:22:11.628906 7f869af78700 5 -- op tracker -- , seq: 2191, time: 2014-06-09 16:22:11.382228, event: throttled, op: pg_info(1 pgs e243:1.b8) 2014-06-09 16:22:11.628909 7f869af78700 5 -- op tracker -- , seq: 2191, time: 2014-06-09 16:22:11.382238, event: all_read, op: pg_info(1 pgs e243:1.b8) 2014-06-09 16:22:11.628913 7f869af78700 5 -- op tracker -- , seq: 2191, time: 2014-06-09 16:22:11.628898, event: dispatched, op: pg_info(1 pgs e243:1.b8) 2014-06-09 16:22:11.628918 7f869af78700 5 -- op tracker -- , seq: 2191, time: 2014-06-09 16:22:11.628917, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.b8) 2014-06-09 16:22:11.628920 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.b8) v4 from osd.5 2014-06-09 16:22:11.628922 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e9180 2014-06-09 16:22:11.628924 7f869af78700 5 -- op tracker -- , seq: 2191, time: 2014-06-09 16:22:11.628923, event: started, op: pg_info(1 pgs e243:1.b8) 2014-06-09 16:22:11.628933 7f869af78700 5 -- op tracker -- , seq: 2191, time: 2014-06-09 16:22:11.628932, event: done, op: pg_info(1 pgs e243:1.b8) 2014-06-09 16:22:11.628937 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628938 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628944 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628945 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.628946 7f869af78700 20 osd.26 243 _dispatch 0x3044c1c0 pg_info(1 pgs e243:3.408(4)) v4 2014-06-09 16:22:11.628949 7f869af78700 5 -- op tracker -- , seq: 2192, time: 2014-06-09 16:22:11.216533, event: header_read, op: pg_info(1 pgs e243:3.408(4)) 2014-06-09 16:22:11.628956 7f869af78700 5 -- op tracker -- , seq: 2192, time: 2014-06-09 16:22:11.216534, event: throttled, op: pg_info(1 pgs e243:3.408(4)) 2014-06-09 16:22:11.628960 7f869af78700 5 -- op tracker -- , seq: 2192, time: 2014-06-09 16:22:11.216552, event: all_read, op: pg_info(1 pgs e243:3.408(4)) 2014-06-09 16:22:11.628966 7f869af78700 5 -- op tracker -- , seq: 2192, time: 2014-06-09 16:22:11.628944, event: dispatched, op: pg_info(1 pgs e243:3.408(4)) 2014-06-09 16:22:11.628970 7f869af78700 5 -- op tracker -- , seq: 2192, time: 2014-06-09 16:22:11.628969, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.408(4)) 2014-06-09 16:22:11.628975 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.408(4)) v4 from osd.2 2014-06-09 16:22:11.628977 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044c1c0 2014-06-09 16:22:11.628979 7f869af78700 5 -- op tracker -- , seq: 2192, time: 2014-06-09 16:22:11.628978, event: started, op: pg_info(1 pgs e243:3.408(4)) 2014-06-09 16:22:11.628989 7f869af78700 5 -- op tracker -- , seq: 2192, time: 2014-06-09 16:22:11.628987, event: done, op: pg_info(1 pgs e243:3.408(4)) 2014-06-09 16:22:11.628993 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.628994 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629001 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629001 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629002 7f869af78700 20 osd.26 243 _dispatch 0x16065180 pg_notify(2.5d3(20) epoch 243) v5 2014-06-09 16:22:11.629005 7f869af78700 5 -- op tracker -- , seq: 2193, time: 2014-06-09 16:22:11.285702, event: header_read, op: pg_notify(2.5d3(20) epoch 243) 2014-06-09 16:22:11.629008 7f869af78700 5 -- op tracker -- , seq: 2193, time: 2014-06-09 16:22:11.285703, event: throttled, op: pg_notify(2.5d3(20) epoch 243) 2014-06-09 16:22:11.629011 7f869af78700 5 -- op tracker -- , seq: 2193, time: 2014-06-09 16:22:11.285730, event: all_read, op: pg_notify(2.5d3(20) epoch 243) 2014-06-09 16:22:11.629015 7f869af78700 5 -- op tracker -- , seq: 2193, time: 2014-06-09 16:22:11.629000, event: dispatched, op: pg_notify(2.5d3(20) epoch 243) 2014-06-09 16:22:11.629018 7f869af78700 5 -- op tracker -- , seq: 2193, time: 2014-06-09 16:22:11.629017, event: waiting_for_osdmap, op: pg_notify(2.5d3(20) epoch 243) 2014-06-09 16:22:11.629024 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.629025 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16065180 2014-06-09 16:22:11.629027 7f869af78700 5 -- op tracker -- , seq: 2193, time: 2014-06-09 16:22:11.629026, event: started, op: pg_notify(2.5d3(20) epoch 243) 2014-06-09 16:22:11.629036 7f869af78700 5 -- op tracker -- , seq: 2193, time: 2014-06-09 16:22:11.629035, event: done, op: pg_notify(2.5d3(20) epoch 243) 2014-06-09 16:22:11.629040 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629041 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629051 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629052 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629052 7f869af78700 20 osd.26 243 _dispatch 0x2d732540 pg_info(1 pgs e243:3.4c5(2)) v4 2014-06-09 16:22:11.629055 7f869af78700 5 -- op tracker -- , seq: 2194, time: 2014-06-09 16:22:11.375743, event: header_read, op: pg_info(1 pgs e243:3.4c5(2)) 2014-06-09 16:22:11.629060 7f869af78700 5 -- op tracker -- , seq: 2194, time: 2014-06-09 16:22:11.375744, event: throttled, op: pg_info(1 pgs e243:3.4c5(2)) 2014-06-09 16:22:11.629064 7f869af78700 5 -- op tracker -- , seq: 2194, time: 2014-06-09 16:22:11.375756, event: all_read, op: pg_info(1 pgs e243:3.4c5(2)) 2014-06-09 16:22:11.629071 7f869af78700 5 -- op tracker -- , seq: 2194, time: 2014-06-09 16:22:11.629051, event: dispatched, op: pg_info(1 pgs e243:3.4c5(2)) 2014-06-09 16:22:11.629074 7f869af78700 5 -- op tracker -- , seq: 2194, time: 2014-06-09 16:22:11.629073, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c5(2)) 2014-06-09 16:22:11.629077 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c5(2)) v4 from osd.7 2014-06-09 16:22:11.629078 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d732540 2014-06-09 16:22:11.629081 7f869af78700 5 -- op tracker -- , seq: 2194, time: 2014-06-09 16:22:11.629080, event: started, op: pg_info(1 pgs e243:3.4c5(2)) 2014-06-09 16:22:11.629090 7f869af78700 5 -- op tracker -- , seq: 2194, time: 2014-06-09 16:22:11.629088, event: done, op: pg_info(1 pgs e243:3.4c5(2)) 2014-06-09 16:22:11.629094 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629095 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629102 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629103 7f869af78700 20 osd.26 243 _dispatch 0x2d740a80 pg_info(1 pgs e243:1.6b7) v4 2014-06-09 16:22:11.629105 7f869af78700 5 -- op tracker -- , seq: 2195, time: 2014-06-09 16:22:11.434059, event: header_read, op: pg_info(1 pgs e243:1.6b7) 2014-06-09 16:22:11.629109 7f869af78700 5 -- op tracker -- , seq: 2195, time: 2014-06-09 16:22:11.434060, event: throttled, op: pg_info(1 pgs e243:1.6b7) 2014-06-09 16:22:11.629112 7f869af78700 5 -- op tracker -- , seq: 2195, time: 2014-06-09 16:22:11.434072, event: all_read, op: pg_info(1 pgs e243:1.6b7) 2014-06-09 16:22:11.629115 7f869af78700 5 -- op tracker -- , seq: 2195, time: 2014-06-09 16:22:11.629101, event: dispatched, op: pg_info(1 pgs e243:1.6b7) 2014-06-09 16:22:11.629121 7f869af78700 5 -- op tracker -- , seq: 2195, time: 2014-06-09 16:22:11.629119, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6b7) 2014-06-09 16:22:11.629123 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6b7) v4 from osd.3 2014-06-09 16:22:11.629124 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d740a80 2014-06-09 16:22:11.629127 7f869af78700 5 -- op tracker -- , seq: 2195, time: 2014-06-09 16:22:11.629126, event: started, op: pg_info(1 pgs e243:1.6b7) 2014-06-09 16:22:11.629135 7f869af78700 5 -- op tracker -- , seq: 2195, time: 2014-06-09 16:22:11.629134, event: done, op: pg_info(1 pgs e243:1.6b7) 2014-06-09 16:22:11.629139 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629146 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629147 7f869af78700 20 osd.26 243 _dispatch 0x240eb6c0 pg_info(1 pgs e243:1.327) v4 2014-06-09 16:22:11.629150 7f869af78700 5 -- op tracker -- , seq: 2196, time: 2014-06-09 16:22:11.284683, event: header_read, op: pg_info(1 pgs e243:1.327) 2014-06-09 16:22:11.629154 7f869af78700 5 -- op tracker -- , seq: 2196, time: 2014-06-09 16:22:11.284683, event: throttled, op: pg_info(1 pgs e243:1.327) 2014-06-09 16:22:11.629157 7f869af78700 5 -- op tracker -- , seq: 2196, time: 2014-06-09 16:22:11.284697, event: all_read, op: pg_info(1 pgs e243:1.327) 2014-06-09 16:22:11.629160 7f869af78700 5 -- op tracker -- , seq: 2196, time: 2014-06-09 16:22:11.629146, event: dispatched, op: pg_info(1 pgs e243:1.327) 2014-06-09 16:22:11.629164 7f869af78700 5 -- op tracker -- , seq: 2196, time: 2014-06-09 16:22:11.629163, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.327) 2014-06-09 16:22:11.629166 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.327) v4 from osd.1 2014-06-09 16:22:11.629167 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eb6c0 2014-06-09 16:22:11.629170 7f869af78700 5 -- op tracker -- , seq: 2196, time: 2014-06-09 16:22:11.629169, event: started, op: pg_info(1 pgs e243:1.327) 2014-06-09 16:22:11.629178 7f869af78700 5 -- op tracker -- , seq: 2196, time: 2014-06-09 16:22:11.629177, event: done, op: pg_info(1 pgs e243:1.327) 2014-06-09 16:22:11.629187 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629188 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629195 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629196 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629196 7f869af78700 20 osd.26 243 _dispatch 0x19fdec40 pg_notify(0.34e(20),0.89(23) epoch 243) v5 2014-06-09 16:22:11.629199 7f869af78700 5 -- op tracker -- , seq: 2197, time: 2014-06-09 16:22:11.352411, event: header_read, op: pg_notify(0.34e(20),0.89(23) epoch 243) 2014-06-09 16:22:11.629203 7f869af78700 5 -- op tracker -- , seq: 2197, time: 2014-06-09 16:22:11.352412, event: throttled, op: pg_notify(0.34e(20),0.89(23) epoch 243) 2014-06-09 16:22:11.629207 7f869af78700 5 -- op tracker -- , seq: 2197, time: 2014-06-09 16:22:11.352462, event: all_read, op: pg_notify(0.34e(20),0.89(23) epoch 243) 2014-06-09 16:22:11.629212 7f869af78700 5 -- op tracker -- , seq: 2197, time: 2014-06-09 16:22:11.629195, event: dispatched, op: pg_notify(0.34e(20),0.89(23) epoch 243) 2014-06-09 16:22:11.629216 7f869af78700 5 -- op tracker -- , seq: 2197, time: 2014-06-09 16:22:11.629214, event: waiting_for_osdmap, op: pg_notify(0.34e(20),0.89(23) epoch 243) 2014-06-09 16:22:11.629221 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.629222 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdec40 2014-06-09 16:22:11.629225 7f869af78700 5 -- op tracker -- , seq: 2197, time: 2014-06-09 16:22:11.629224, event: started, op: pg_notify(0.34e(20),0.89(23) epoch 243) 2014-06-09 16:22:11.629238 7f869af78700 5 -- op tracker -- , seq: 2197, time: 2014-06-09 16:22:11.629237, event: done, op: pg_notify(0.34e(20),0.89(23) epoch 243) 2014-06-09 16:22:11.629244 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629245 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629257 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629258 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629258 7f869af78700 20 osd.26 243 _dispatch 0x230bba40 pg_info(1 pgs e243:3.22d) v4 2014-06-09 16:22:11.629261 7f869af78700 5 -- op tracker -- , seq: 2198, time: 2014-06-09 16:22:11.362148, event: header_read, op: pg_info(1 pgs e243:3.22d) 2014-06-09 16:22:11.629268 7f869af78700 5 -- op tracker -- , seq: 2198, time: 2014-06-09 16:22:11.362149, event: throttled, op: pg_info(1 pgs e243:3.22d) 2014-06-09 16:22:11.629272 7f869af78700 5 -- op tracker -- , seq: 2198, time: 2014-06-09 16:22:11.362161, event: all_read, op: pg_info(1 pgs e243:3.22d) 2014-06-09 16:22:11.629275 7f869af78700 5 -- op tracker -- , seq: 2198, time: 2014-06-09 16:22:11.629257, event: dispatched, op: pg_info(1 pgs e243:3.22d) 2014-06-09 16:22:11.629279 7f869af78700 5 -- op tracker -- , seq: 2198, time: 2014-06-09 16:22:11.629278, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.22d) 2014-06-09 16:22:11.629281 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.22d) v4 from osd.8 2014-06-09 16:22:11.629282 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230bba40 2014-06-09 16:22:11.629285 7f869af78700 5 -- op tracker -- , seq: 2198, time: 2014-06-09 16:22:11.629284, event: started, op: pg_info(1 pgs e243:3.22d) 2014-06-09 16:22:11.629294 7f869af78700 5 -- op tracker -- , seq: 2198, time: 2014-06-09 16:22:11.629293, event: done, op: pg_info(1 pgs e243:3.22d) 2014-06-09 16:22:11.629298 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629299 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629305 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629306 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629307 7f869af78700 20 osd.26 243 _dispatch 0x31041880 pg_info(1 pgs e243:0.75) v4 2014-06-09 16:22:11.629309 7f869af78700 5 -- op tracker -- , seq: 2199, time: 2014-06-09 16:22:11.360799, event: header_read, op: pg_info(1 pgs e243:0.75) 2014-06-09 16:22:11.629313 7f869af78700 5 -- op tracker -- , seq: 2199, time: 2014-06-09 16:22:11.360800, event: throttled, op: pg_info(1 pgs e243:0.75) 2014-06-09 16:22:11.629316 7f869af78700 5 -- op tracker -- , seq: 2199, time: 2014-06-09 16:22:11.360813, event: all_read, op: pg_info(1 pgs e243:0.75) 2014-06-09 16:22:11.629319 7f869af78700 5 -- op tracker -- , seq: 2199, time: 2014-06-09 16:22:11.629305, event: dispatched, op: pg_info(1 pgs e243:0.75) 2014-06-09 16:22:11.629326 7f869af78700 5 -- op tracker -- , seq: 2199, time: 2014-06-09 16:22:11.629325, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.75) 2014-06-09 16:22:11.629329 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.75) v4 from osd.6 2014-06-09 16:22:11.629330 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31041880 2014-06-09 16:22:11.629333 7f869af78700 5 -- op tracker -- , seq: 2199, time: 2014-06-09 16:22:11.629332, event: started, op: pg_info(1 pgs e243:0.75) 2014-06-09 16:22:11.629341 7f869af78700 5 -- op tracker -- , seq: 2199, time: 2014-06-09 16:22:11.629340, event: done, op: pg_info(1 pgs e243:0.75) 2014-06-09 16:22:11.629345 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629346 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629353 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629354 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629354 7f869af78700 20 osd.26 243 _dispatch 0x30b8d340 pg_info(1 pgs e243:1.109) v4 2014-06-09 16:22:11.629357 7f869af78700 5 -- op tracker -- , seq: 2200, time: 2014-06-09 16:22:11.261053, event: header_read, op: pg_info(1 pgs e243:1.109) 2014-06-09 16:22:11.629360 7f869af78700 5 -- op tracker -- , seq: 2200, time: 2014-06-09 16:22:11.261054, event: throttled, op: pg_info(1 pgs e243:1.109) 2014-06-09 16:22:11.629364 7f869af78700 5 -- op tracker -- , seq: 2200, time: 2014-06-09 16:22:11.261066, event: all_read, op: pg_info(1 pgs e243:1.109) 2014-06-09 16:22:11.629367 7f869af78700 5 -- op tracker -- , seq: 2200, time: 2014-06-09 16:22:11.629353, event: dispatched, op: pg_info(1 pgs e243:1.109) 2014-06-09 16:22:11.629372 7f869af78700 5 -- op tracker -- , seq: 2200, time: 2014-06-09 16:22:11.629371, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.109) 2014-06-09 16:22:11.629374 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.109) v4 from osd.0 2014-06-09 16:22:11.629376 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8d340 2014-06-09 16:22:11.629378 7f869af78700 5 -- op tracker -- , seq: 2200, time: 2014-06-09 16:22:11.629377, event: started, op: pg_info(1 pgs e243:1.109) 2014-06-09 16:22:11.629387 7f869af78700 5 -- op tracker -- , seq: 2200, time: 2014-06-09 16:22:11.629386, event: done, op: pg_info(1 pgs e243:1.109) 2014-06-09 16:22:11.629391 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629392 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629399 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629400 7f869af78700 20 osd.26 243 _dispatch 0x1d039c00 pg_info(1 pgs e243:1.2e0) v4 2014-06-09 16:22:11.629402 7f869af78700 5 -- op tracker -- , seq: 2201, time: 2014-06-09 16:22:11.382294, event: header_read, op: pg_info(1 pgs e243:1.2e0) 2014-06-09 16:22:11.629406 7f869af78700 5 -- op tracker -- , seq: 2201, time: 2014-06-09 16:22:11.382295, event: throttled, op: pg_info(1 pgs e243:1.2e0) 2014-06-09 16:22:11.629409 7f869af78700 5 -- op tracker -- , seq: 2201, time: 2014-06-09 16:22:11.382307, event: all_read, op: pg_info(1 pgs e243:1.2e0) 2014-06-09 16:22:11.629413 7f869af78700 5 -- op tracker -- , seq: 2201, time: 2014-06-09 16:22:11.629398, event: dispatched, op: pg_info(1 pgs e243:1.2e0) 2014-06-09 16:22:11.629417 7f869af78700 5 -- op tracker -- , seq: 2201, time: 2014-06-09 16:22:11.629416, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2e0) 2014-06-09 16:22:11.629419 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2e0) v4 from osd.5 2014-06-09 16:22:11.629420 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d039c00 2014-06-09 16:22:11.629423 7f869af78700 5 -- op tracker -- , seq: 2201, time: 2014-06-09 16:22:11.629422, event: started, op: pg_info(1 pgs e243:1.2e0) 2014-06-09 16:22:11.629435 7f869af78700 5 -- op tracker -- , seq: 2201, time: 2014-06-09 16:22:11.629434, event: done, op: pg_info(1 pgs e243:1.2e0) 2014-06-09 16:22:11.629439 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629440 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629448 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629448 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629449 7f869af78700 20 osd.26 243 _dispatch 0x308a8a00 pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.629453 7f869af78700 5 -- op tracker -- , seq: 2202, time: 2014-06-09 16:22:11.222507, event: header_read, op: pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629457 7f869af78700 5 -- op tracker -- , seq: 2202, time: 2014-06-09 16:22:11.222507, event: throttled, op: pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629464 7f869af78700 5 -- op tracker -- , seq: 2202, time: 2014-06-09 16:22:11.222526, event: all_read, op: pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629468 7f869af78700 5 -- op tracker -- , seq: 2202, time: 2014-06-09 16:22:11.629447, event: dispatched, op: pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629472 7f869af78700 5 -- op tracker -- , seq: 2202, time: 2014-06-09 16:22:11.629471, event: waiting_for_osdmap, op: pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629476 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.629478 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308a8a00 2014-06-09 16:22:11.629481 7f869af78700 5 -- op tracker -- , seq: 2202, time: 2014-06-09 16:22:11.629480, event: started, op: pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629489 7f869af78700 5 -- op tracker -- , seq: 2202, time: 2014-06-09 16:22:11.629487, event: done, op: pg_log(2.4b9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629493 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629494 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629500 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629501 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629501 7f869af78700 20 osd.26 243 _dispatch 0x308d8fc0 pg_notify(3.6d8(10) epoch 243) v5 2014-06-09 16:22:11.629504 7f869af78700 5 -- op tracker -- , seq: 2203, time: 2014-06-09 16:22:11.287338, event: header_read, op: pg_notify(3.6d8(10) epoch 243) 2014-06-09 16:22:11.629508 7f869af78700 5 -- op tracker -- , seq: 2203, time: 2014-06-09 16:22:11.287339, event: throttled, op: pg_notify(3.6d8(10) epoch 243) 2014-06-09 16:22:11.629511 7f869af78700 5 -- op tracker -- , seq: 2203, time: 2014-06-09 16:22:11.287353, event: all_read, op: pg_notify(3.6d8(10) epoch 243) 2014-06-09 16:22:11.629514 7f869af78700 5 -- op tracker -- , seq: 2203, time: 2014-06-09 16:22:11.629500, event: dispatched, op: pg_notify(3.6d8(10) epoch 243) 2014-06-09 16:22:11.629521 7f869af78700 5 -- op tracker -- , seq: 2203, time: 2014-06-09 16:22:11.629520, event: waiting_for_osdmap, op: pg_notify(3.6d8(10) epoch 243) 2014-06-09 16:22:11.629524 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.629525 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d8fc0 2014-06-09 16:22:11.629527 7f869af78700 5 -- op tracker -- , seq: 2203, time: 2014-06-09 16:22:11.629526, event: started, op: pg_notify(3.6d8(10) epoch 243) 2014-06-09 16:22:11.629536 7f869af78700 5 -- op tracker -- , seq: 2203, time: 2014-06-09 16:22:11.629535, event: done, op: pg_notify(3.6d8(10) epoch 243) 2014-06-09 16:22:11.629540 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629541 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629550 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629550 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629551 7f869af78700 20 osd.26 243 _dispatch 0x2d733c00 pg_info(1 pgs e243:2.3fb) v4 2014-06-09 16:22:11.629554 7f869af78700 5 -- op tracker -- , seq: 2204, time: 2014-06-09 16:22:11.380294, event: header_read, op: pg_info(1 pgs e243:2.3fb) 2014-06-09 16:22:11.629557 7f869af78700 5 -- op tracker -- , seq: 2204, time: 2014-06-09 16:22:11.380295, event: throttled, op: pg_info(1 pgs e243:2.3fb) 2014-06-09 16:22:11.629561 7f869af78700 5 -- op tracker -- , seq: 2204, time: 2014-06-09 16:22:11.380308, event: all_read, op: pg_info(1 pgs e243:2.3fb) 2014-06-09 16:22:11.629564 7f869af78700 5 -- op tracker -- , seq: 2204, time: 2014-06-09 16:22:11.629550, event: dispatched, op: pg_info(1 pgs e243:2.3fb) 2014-06-09 16:22:11.629567 7f869af78700 5 -- op tracker -- , seq: 2204, time: 2014-06-09 16:22:11.629566, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3fb) 2014-06-09 16:22:11.629569 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3fb) v4 from osd.7 2014-06-09 16:22:11.629571 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d733c00 2014-06-09 16:22:11.629573 7f869af78700 5 -- op tracker -- , seq: 2204, time: 2014-06-09 16:22:11.629572, event: started, op: pg_info(1 pgs e243:2.3fb) 2014-06-09 16:22:11.629583 7f869af78700 5 -- op tracker -- , seq: 2204, time: 2014-06-09 16:22:11.629582, event: done, op: pg_info(1 pgs e243:2.3fb) 2014-06-09 16:22:11.629587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629588 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629594 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629595 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629595 7f869af78700 20 osd.26 243 _dispatch 0x3046bdc0 pg_info(1 pgs e243:2.61b) v4 2014-06-09 16:22:11.629598 7f869af78700 5 -- op tracker -- , seq: 2205, time: 2014-06-09 16:22:11.434140, event: header_read, op: pg_info(1 pgs e243:2.61b) 2014-06-09 16:22:11.629602 7f869af78700 5 -- op tracker -- , seq: 2205, time: 2014-06-09 16:22:11.434141, event: throttled, op: pg_info(1 pgs e243:2.61b) 2014-06-09 16:22:11.629608 7f869af78700 5 -- op tracker -- , seq: 2205, time: 2014-06-09 16:22:11.434155, event: all_read, op: pg_info(1 pgs e243:2.61b) 2014-06-09 16:22:11.629612 7f869af78700 5 -- op tracker -- , seq: 2205, time: 2014-06-09 16:22:11.629594, event: dispatched, op: pg_info(1 pgs e243:2.61b) 2014-06-09 16:22:11.629615 7f869af78700 5 -- op tracker -- , seq: 2205, time: 2014-06-09 16:22:11.629614, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.61b) 2014-06-09 16:22:11.629618 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.61b) v4 from osd.3 2014-06-09 16:22:11.629619 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046bdc0 2014-06-09 16:22:11.629622 7f869af78700 5 -- op tracker -- , seq: 2205, time: 2014-06-09 16:22:11.629620, event: started, op: pg_info(1 pgs e243:2.61b) 2014-06-09 16:22:11.629630 7f869af78700 5 -- op tracker -- , seq: 2205, time: 2014-06-09 16:22:11.629629, event: done, op: pg_info(1 pgs e243:2.61b) 2014-06-09 16:22:11.629635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629636 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629643 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629643 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629644 7f869af78700 20 osd.26 243 _dispatch 0x1fc05dc0 pg_info(1 pgs e243:1.446) v4 2014-06-09 16:22:11.629647 7f869af78700 5 -- op tracker -- , seq: 2206, time: 2014-06-09 16:22:11.284702, event: header_read, op: pg_info(1 pgs e243:1.446) 2014-06-09 16:22:11.629650 7f869af78700 5 -- op tracker -- , seq: 2206, time: 2014-06-09 16:22:11.284702, event: throttled, op: pg_info(1 pgs e243:1.446) 2014-06-09 16:22:11.629654 7f869af78700 5 -- op tracker -- , seq: 2206, time: 2014-06-09 16:22:11.284713, event: all_read, op: pg_info(1 pgs e243:1.446) 2014-06-09 16:22:11.629659 7f869af78700 5 -- op tracker -- , seq: 2206, time: 2014-06-09 16:22:11.629642, event: dispatched, op: pg_info(1 pgs e243:1.446) 2014-06-09 16:22:11.629663 7f869af78700 5 -- op tracker -- , seq: 2206, time: 2014-06-09 16:22:11.629662, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.446) 2014-06-09 16:22:11.629666 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.446) v4 from osd.1 2014-06-09 16:22:11.629667 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc05dc0 2014-06-09 16:22:11.629670 7f869af78700 5 -- op tracker -- , seq: 2206, time: 2014-06-09 16:22:11.629669, event: started, op: pg_info(1 pgs e243:1.446) 2014-06-09 16:22:11.629683 7f869af78700 5 -- op tracker -- , seq: 2206, time: 2014-06-09 16:22:11.629682, event: done, op: pg_info(1 pgs e243:1.446) 2014-06-09 16:22:11.629687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629688 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629695 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629696 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629697 7f869af78700 20 osd.26 243 _dispatch 0x19fdee00 pg_notify(1.299(20),1.56c(23),3.2a5(10) epoch 243) v5 2014-06-09 16:22:11.629700 7f869af78700 5 -- op tracker -- , seq: 2207, time: 2014-06-09 16:22:11.355717, event: header_read, op: pg_notify(1.299(20),1.56c(23),3.2a5(10) epoch 243) 2014-06-09 16:22:11.629704 7f869af78700 5 -- op tracker -- , seq: 2207, time: 2014-06-09 16:22:11.355718, event: throttled, op: pg_notify(1.299(20),1.56c(23),3.2a5(10) epoch 243) 2014-06-09 16:22:11.629708 7f869af78700 5 -- op tracker -- , seq: 2207, time: 2014-06-09 16:22:11.355788, event: all_read, op: pg_notify(1.299(20),1.56c(23),3.2a5(10) epoch 243) 2014-06-09 16:22:11.629712 7f869af78700 5 -- op tracker -- , seq: 2207, time: 2014-06-09 16:22:11.629695, event: dispatched, op: pg_notify(1.299(20),1.56c(23),3.2a5(10) epoch 243) 2014-06-09 16:22:11.629720 7f869af78700 5 -- op tracker -- , seq: 2207, time: 2014-06-09 16:22:11.629718, event: waiting_for_osdmap, op: pg_notify(1.299(20),1.56c(23),3.2a5(10) epoch 243) 2014-06-09 16:22:11.629722 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.629723 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdee00 2014-06-09 16:22:11.629726 7f869af78700 5 -- op tracker -- , seq: 2207, time: 2014-06-09 16:22:11.629725, event: started, op: pg_notify(1.299(20),1.56c(23),3.2a5(10) epoch 243) 2014-06-09 16:22:11.629743 7f869af78700 5 -- op tracker -- , seq: 2207, time: 2014-06-09 16:22:11.629742, event: done, op: pg_notify(1.299(20),1.56c(23),3.2a5(10) epoch 243) 2014-06-09 16:22:11.629748 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629749 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629765 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629766 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629767 7f869af78700 20 osd.26 243 _dispatch 0x230ba8c0 pg_info(1 pgs e243:0.678) v4 2014-06-09 16:22:11.629769 7f869af78700 5 -- op tracker -- , seq: 2208, time: 2014-06-09 16:22:11.362174, event: header_read, op: pg_info(1 pgs e243:0.678) 2014-06-09 16:22:11.629773 7f869af78700 5 -- op tracker -- , seq: 2208, time: 2014-06-09 16:22:11.362175, event: throttled, op: pg_info(1 pgs e243:0.678) 2014-06-09 16:22:11.629777 7f869af78700 5 -- op tracker -- , seq: 2208, time: 2014-06-09 16:22:11.362187, event: all_read, op: pg_info(1 pgs e243:0.678) 2014-06-09 16:22:11.629780 7f869af78700 5 -- op tracker -- , seq: 2208, time: 2014-06-09 16:22:11.629765, event: dispatched, op: pg_info(1 pgs e243:0.678) 2014-06-09 16:22:11.629783 7f869af78700 5 -- op tracker -- , seq: 2208, time: 2014-06-09 16:22:11.629782, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.678) 2014-06-09 16:22:11.629786 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.678) v4 from osd.8 2014-06-09 16:22:11.629787 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230ba8c0 2014-06-09 16:22:11.629790 7f869af78700 5 -- op tracker -- , seq: 2208, time: 2014-06-09 16:22:11.629789, event: started, op: pg_info(1 pgs e243:0.678) 2014-06-09 16:22:11.629798 7f869af78700 5 -- op tracker -- , seq: 2208, time: 2014-06-09 16:22:11.629797, event: done, op: pg_info(1 pgs e243:0.678) 2014-06-09 16:22:11.629802 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629803 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629809 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629810 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629810 7f869af78700 20 osd.26 243 _dispatch 0x310416c0 pg_info(1 pgs e243:1.37d) v4 2014-06-09 16:22:11.629813 7f869af78700 5 -- op tracker -- , seq: 2209, time: 2014-06-09 16:22:11.360825, event: header_read, op: pg_info(1 pgs e243:1.37d) 2014-06-09 16:22:11.629818 7f869af78700 5 -- op tracker -- , seq: 2209, time: 2014-06-09 16:22:11.360826, event: throttled, op: pg_info(1 pgs e243:1.37d) 2014-06-09 16:22:11.629822 7f869af78700 5 -- op tracker -- , seq: 2209, time: 2014-06-09 16:22:11.360840, event: all_read, op: pg_info(1 pgs e243:1.37d) 2014-06-09 16:22:11.629825 7f869af78700 5 -- op tracker -- , seq: 2209, time: 2014-06-09 16:22:11.629809, event: dispatched, op: pg_info(1 pgs e243:1.37d) 2014-06-09 16:22:11.629828 7f869af78700 5 -- op tracker -- , seq: 2209, time: 2014-06-09 16:22:11.629827, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.37d) 2014-06-09 16:22:11.629830 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.37d) v4 from osd.6 2014-06-09 16:22:11.629832 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310416c0 2014-06-09 16:22:11.629834 7f869af78700 5 -- op tracker -- , seq: 2209, time: 2014-06-09 16:22:11.629833, event: started, op: pg_info(1 pgs e243:1.37d) 2014-06-09 16:22:11.629842 7f869af78700 5 -- op tracker -- , seq: 2209, time: 2014-06-09 16:22:11.629841, event: done, op: pg_info(1 pgs e243:1.37d) 2014-06-09 16:22:11.629847 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629848 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629854 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629855 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629856 7f869af78700 20 osd.26 243 _dispatch 0x30b8d180 pg_info(1 pgs e243:2.3c7) v4 2014-06-09 16:22:11.629858 7f869af78700 5 -- op tracker -- , seq: 2210, time: 2014-06-09 16:22:11.261077, event: header_read, op: pg_info(1 pgs e243:2.3c7) 2014-06-09 16:22:11.629863 7f869af78700 5 -- op tracker -- , seq: 2210, time: 2014-06-09 16:22:11.261077, event: throttled, op: pg_info(1 pgs e243:2.3c7) 2014-06-09 16:22:11.629867 7f869af78700 5 -- op tracker -- , seq: 2210, time: 2014-06-09 16:22:11.261088, event: all_read, op: pg_info(1 pgs e243:2.3c7) 2014-06-09 16:22:11.629871 7f869af78700 5 -- op tracker -- , seq: 2210, time: 2014-06-09 16:22:11.629854, event: dispatched, op: pg_info(1 pgs e243:2.3c7) 2014-06-09 16:22:11.629874 7f869af78700 5 -- op tracker -- , seq: 2210, time: 2014-06-09 16:22:11.629873, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3c7) 2014-06-09 16:22:11.629877 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3c7) v4 from osd.0 2014-06-09 16:22:11.629878 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8d180 2014-06-09 16:22:11.629881 7f869af78700 5 -- op tracker -- , seq: 2210, time: 2014-06-09 16:22:11.629880, event: started, op: pg_info(1 pgs e243:2.3c7) 2014-06-09 16:22:11.629889 7f869af78700 5 -- op tracker -- , seq: 2210, time: 2014-06-09 16:22:11.629888, event: done, op: pg_info(1 pgs e243:2.3c7) 2014-06-09 16:22:11.629893 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629894 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629899 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629900 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629901 7f869af78700 20 osd.26 243 _dispatch 0x1d039a40 pg_info(1 pgs e243:1.65d) v4 2014-06-09 16:22:11.629903 7f869af78700 5 -- op tracker -- , seq: 2211, time: 2014-06-09 16:22:11.382314, event: header_read, op: pg_info(1 pgs e243:1.65d) 2014-06-09 16:22:11.629907 7f869af78700 5 -- op tracker -- , seq: 2211, time: 2014-06-09 16:22:11.382315, event: throttled, op: pg_info(1 pgs e243:1.65d) 2014-06-09 16:22:11.629910 7f869af78700 5 -- op tracker -- , seq: 2211, time: 2014-06-09 16:22:11.382326, event: all_read, op: pg_info(1 pgs e243:1.65d) 2014-06-09 16:22:11.629915 7f869af78700 5 -- op tracker -- , seq: 2211, time: 2014-06-09 16:22:11.629899, event: dispatched, op: pg_info(1 pgs e243:1.65d) 2014-06-09 16:22:11.629918 7f869af78700 5 -- op tracker -- , seq: 2211, time: 2014-06-09 16:22:11.629917, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.65d) 2014-06-09 16:22:11.629921 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.65d) v4 from osd.5 2014-06-09 16:22:11.629922 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d039a40 2014-06-09 16:22:11.629924 7f869af78700 5 -- op tracker -- , seq: 2211, time: 2014-06-09 16:22:11.629923, event: started, op: pg_info(1 pgs e243:1.65d) 2014-06-09 16:22:11.629933 7f869af78700 5 -- op tracker -- , seq: 2211, time: 2014-06-09 16:22:11.629932, event: done, op: pg_info(1 pgs e243:1.65d) 2014-06-09 16:22:11.629936 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629937 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629945 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629945 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629946 7f869af78700 20 osd.26 243 _dispatch 0x30724c00 pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.629949 7f869af78700 5 -- op tracker -- , seq: 2212, time: 2014-06-09 16:22:11.222788, event: header_read, op: pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629954 7f869af78700 5 -- op tracker -- , seq: 2212, time: 2014-06-09 16:22:11.222788, event: throttled, op: pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629957 7f869af78700 5 -- op tracker -- , seq: 2212, time: 2014-06-09 16:22:11.222809, event: all_read, op: pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629961 7f869af78700 5 -- op tracker -- , seq: 2212, time: 2014-06-09 16:22:11.629944, event: dispatched, op: pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629966 7f869af78700 5 -- op tracker -- , seq: 2212, time: 2014-06-09 16:22:11.629964, event: waiting_for_osdmap, op: pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629968 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.629971 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30724c00 2014-06-09 16:22:11.629973 7f869af78700 5 -- op tracker -- , seq: 2212, time: 2014-06-09 16:22:11.629972, event: started, op: pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629984 7f869af78700 5 -- op tracker -- , seq: 2212, time: 2014-06-09 16:22:11.629983, event: done, op: pg_log(0.4d5 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.629989 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629990 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629996 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.629997 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.629998 7f869af78700 20 osd.26 243 _dispatch 0x308e2fc0 pg_notify(3.424(10) epoch 243) v5 2014-06-09 16:22:11.630000 7f869af78700 5 -- op tracker -- , seq: 2213, time: 2014-06-09 16:22:11.287398, event: header_read, op: pg_notify(3.424(10) epoch 243) 2014-06-09 16:22:11.630004 7f869af78700 5 -- op tracker -- , seq: 2213, time: 2014-06-09 16:22:11.287398, event: throttled, op: pg_notify(3.424(10) epoch 243) 2014-06-09 16:22:11.630007 7f869af78700 5 -- op tracker -- , seq: 2213, time: 2014-06-09 16:22:11.287410, event: all_read, op: pg_notify(3.424(10) epoch 243) 2014-06-09 16:22:11.630014 7f869af78700 5 -- op tracker -- , seq: 2213, time: 2014-06-09 16:22:11.629996, event: dispatched, op: pg_notify(3.424(10) epoch 243) 2014-06-09 16:22:11.630018 7f869af78700 5 -- op tracker -- , seq: 2213, time: 2014-06-09 16:22:11.630016, event: waiting_for_osdmap, op: pg_notify(3.424(10) epoch 243) 2014-06-09 16:22:11.630021 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.630022 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e2fc0 2014-06-09 16:22:11.630025 7f869af78700 5 -- op tracker -- , seq: 2213, time: 2014-06-09 16:22:11.630024, event: started, op: pg_notify(3.424(10) epoch 243) 2014-06-09 16:22:11.630034 7f869af78700 5 -- op tracker -- , seq: 2213, time: 2014-06-09 16:22:11.630033, event: done, op: pg_notify(3.424(10) epoch 243) 2014-06-09 16:22:11.630038 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630039 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630048 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630050 7f869af78700 20 osd.26 243 _dispatch 0x30d25dc0 pg_info(1 pgs e243:1.71b) v4 2014-06-09 16:22:11.630052 7f869af78700 5 -- op tracker -- , seq: 2214, time: 2014-06-09 16:22:11.380316, event: header_read, op: pg_info(1 pgs e243:1.71b) 2014-06-09 16:22:11.630056 7f869af78700 5 -- op tracker -- , seq: 2214, time: 2014-06-09 16:22:11.380316, event: throttled, op: pg_info(1 pgs e243:1.71b) 2014-06-09 16:22:11.630061 7f869af78700 5 -- op tracker -- , seq: 2214, time: 2014-06-09 16:22:11.380327, event: all_read, op: pg_info(1 pgs e243:1.71b) 2014-06-09 16:22:11.630064 7f869af78700 5 -- op tracker -- , seq: 2214, time: 2014-06-09 16:22:11.630048, event: dispatched, op: pg_info(1 pgs e243:1.71b) 2014-06-09 16:22:11.630068 7f869af78700 5 -- op tracker -- , seq: 2214, time: 2014-06-09 16:22:11.630067, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.71b) 2014-06-09 16:22:11.630070 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.71b) v4 from osd.7 2014-06-09 16:22:11.630072 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d25dc0 2014-06-09 16:22:11.630074 7f869af78700 5 -- op tracker -- , seq: 2214, time: 2014-06-09 16:22:11.630073, event: started, op: pg_info(1 pgs e243:1.71b) 2014-06-09 16:22:11.630084 7f869af78700 5 -- op tracker -- , seq: 2214, time: 2014-06-09 16:22:11.630083, event: done, op: pg_info(1 pgs e243:1.71b) 2014-06-09 16:22:11.630088 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630089 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630096 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630096 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630097 7f869af78700 20 osd.26 243 _dispatch 0xbeee700 pg_info(1 pgs e243:0.214) v4 2014-06-09 16:22:11.630099 7f869af78700 5 -- op tracker -- , seq: 2215, time: 2014-06-09 16:22:11.434644, event: header_read, op: pg_info(1 pgs e243:0.214) 2014-06-09 16:22:11.630103 7f869af78700 5 -- op tracker -- , seq: 2215, time: 2014-06-09 16:22:11.434644, event: throttled, op: pg_info(1 pgs e243:0.214) 2014-06-09 16:22:11.630106 7f869af78700 5 -- op tracker -- , seq: 2215, time: 2014-06-09 16:22:11.434655, event: all_read, op: pg_info(1 pgs e243:0.214) 2014-06-09 16:22:11.630110 7f869af78700 5 -- op tracker -- , seq: 2215, time: 2014-06-09 16:22:11.630095, event: dispatched, op: pg_info(1 pgs e243:0.214) 2014-06-09 16:22:11.630114 7f869af78700 5 -- op tracker -- , seq: 2215, time: 2014-06-09 16:22:11.630113, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.214) 2014-06-09 16:22:11.630117 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.214) v4 from osd.3 2014-06-09 16:22:11.630118 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xbeee700 2014-06-09 16:22:11.630121 7f869af78700 5 -- op tracker -- , seq: 2215, time: 2014-06-09 16:22:11.630120, event: started, op: pg_info(1 pgs e243:0.214) 2014-06-09 16:22:11.630129 7f869af78700 5 -- op tracker -- , seq: 2215, time: 2014-06-09 16:22:11.630128, event: done, op: pg_info(1 pgs e243:0.214) 2014-06-09 16:22:11.630133 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630133 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630139 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630141 7f869af78700 20 osd.26 243 _dispatch 0x1fc04000 pg_info(1 pgs e243:2.3a6) v4 2014-06-09 16:22:11.630143 7f869af78700 5 -- op tracker -- , seq: 2216, time: 2014-06-09 16:22:11.284718, event: header_read, op: pg_info(1 pgs e243:2.3a6) 2014-06-09 16:22:11.630147 7f869af78700 5 -- op tracker -- , seq: 2216, time: 2014-06-09 16:22:11.284718, event: throttled, op: pg_info(1 pgs e243:2.3a6) 2014-06-09 16:22:11.630150 7f869af78700 5 -- op tracker -- , seq: 2216, time: 2014-06-09 16:22:11.284736, event: all_read, op: pg_info(1 pgs e243:2.3a6) 2014-06-09 16:22:11.630153 7f869af78700 5 -- op tracker -- , seq: 2216, time: 2014-06-09 16:22:11.630139, event: dispatched, op: pg_info(1 pgs e243:2.3a6) 2014-06-09 16:22:11.630157 7f869af78700 5 -- op tracker -- , seq: 2216, time: 2014-06-09 16:22:11.630156, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3a6) 2014-06-09 16:22:11.630159 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3a6) v4 from osd.1 2014-06-09 16:22:11.630160 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc04000 2014-06-09 16:22:11.630163 7f869af78700 5 -- op tracker -- , seq: 2216, time: 2014-06-09 16:22:11.630162, event: started, op: pg_info(1 pgs e243:2.3a6) 2014-06-09 16:22:11.630171 7f869af78700 5 -- op tracker -- , seq: 2216, time: 2014-06-09 16:22:11.630170, event: done, op: pg_info(1 pgs e243:2.3a6) 2014-06-09 16:22:11.630174 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630175 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630181 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630182 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630183 7f869af78700 20 osd.26 243 _dispatch 0x307e61c0 pg_notify(2.154(20) epoch 243) v5 2014-06-09 16:22:11.630185 7f869af78700 5 -- op tracker -- , seq: 2217, time: 2014-06-09 16:22:11.357403, event: header_read, op: pg_notify(2.154(20) epoch 243) 2014-06-09 16:22:11.630189 7f869af78700 5 -- op tracker -- , seq: 2217, time: 2014-06-09 16:22:11.357403, event: throttled, op: pg_notify(2.154(20) epoch 243) 2014-06-09 16:22:11.630192 7f869af78700 5 -- op tracker -- , seq: 2217, time: 2014-06-09 16:22:11.357433, event: all_read, op: pg_notify(2.154(20) epoch 243) 2014-06-09 16:22:11.630195 7f869af78700 5 -- op tracker -- , seq: 2217, time: 2014-06-09 16:22:11.630181, event: dispatched, op: pg_notify(2.154(20) epoch 243) 2014-06-09 16:22:11.630199 7f869af78700 5 -- op tracker -- , seq: 2217, time: 2014-06-09 16:22:11.630198, event: waiting_for_osdmap, op: pg_notify(2.154(20) epoch 243) 2014-06-09 16:22:11.630201 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.630202 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e61c0 2014-06-09 16:22:11.630204 7f869af78700 5 -- op tracker -- , seq: 2217, time: 2014-06-09 16:22:11.630203, event: started, op: pg_notify(2.154(20) epoch 243) 2014-06-09 16:22:11.630213 7f869af78700 5 -- op tracker -- , seq: 2217, time: 2014-06-09 16:22:11.630212, event: done, op: pg_notify(2.154(20) epoch 243) 2014-06-09 16:22:11.630216 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630217 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630228 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630228 7f869af78700 20 osd.26 243 _dispatch 0x230bbc00 pg_info(1 pgs e243:2.67f) v4 2014-06-09 16:22:11.630231 7f869af78700 5 -- op tracker -- , seq: 2218, time: 2014-06-09 16:22:11.362214, event: header_read, op: pg_info(1 pgs e243:2.67f) 2014-06-09 16:22:11.630236 7f869af78700 5 -- op tracker -- , seq: 2218, time: 2014-06-09 16:22:11.362215, event: throttled, op: pg_info(1 pgs e243:2.67f) 2014-06-09 16:22:11.630239 7f869af78700 5 -- op tracker -- , seq: 2218, time: 2014-06-09 16:22:11.362226, event: all_read, op: pg_info(1 pgs e243:2.67f) 2014-06-09 16:22:11.630243 7f869af78700 5 -- op tracker -- , seq: 2218, time: 2014-06-09 16:22:11.630227, event: dispatched, op: pg_info(1 pgs e243:2.67f) 2014-06-09 16:22:11.630246 7f869af78700 5 -- op tracker -- , seq: 2218, time: 2014-06-09 16:22:11.630245, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.67f) 2014-06-09 16:22:11.630249 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.67f) v4 from osd.8 2014-06-09 16:22:11.630251 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230bbc00 2014-06-09 16:22:11.630253 7f869af78700 5 -- op tracker -- , seq: 2218, time: 2014-06-09 16:22:11.630252, event: started, op: pg_info(1 pgs e243:2.67f) 2014-06-09 16:22:11.630261 7f869af78700 5 -- op tracker -- , seq: 2218, time: 2014-06-09 16:22:11.630260, event: done, op: pg_info(1 pgs e243:2.67f) 2014-06-09 16:22:11.630265 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630266 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630272 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630273 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630274 7f869af78700 20 osd.26 243 _dispatch 0x31041500 pg_info(1 pgs e243:3.48a) v4 2014-06-09 16:22:11.630276 7f869af78700 5 -- op tracker -- , seq: 2219, time: 2014-06-09 16:22:11.360864, event: header_read, op: pg_info(1 pgs e243:3.48a) 2014-06-09 16:22:11.630280 7f869af78700 5 -- op tracker -- , seq: 2219, time: 2014-06-09 16:22:11.360864, event: throttled, op: pg_info(1 pgs e243:3.48a) 2014-06-09 16:22:11.630283 7f869af78700 5 -- op tracker -- , seq: 2219, time: 2014-06-09 16:22:11.360878, event: all_read, op: pg_info(1 pgs e243:3.48a) 2014-06-09 16:22:11.630286 7f869af78700 5 -- op tracker -- , seq: 2219, time: 2014-06-09 16:22:11.630272, event: dispatched, op: pg_info(1 pgs e243:3.48a) 2014-06-09 16:22:11.630290 7f869af78700 5 -- op tracker -- , seq: 2219, time: 2014-06-09 16:22:11.630289, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.48a) 2014-06-09 16:22:11.630292 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.48a) v4 from osd.6 2014-06-09 16:22:11.630293 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31041500 2014-06-09 16:22:11.630296 7f869af78700 5 -- op tracker -- , seq: 2219, time: 2014-06-09 16:22:11.630295, event: started, op: pg_info(1 pgs e243:3.48a) 2014-06-09 16:22:11.630304 7f869af78700 5 -- op tracker -- , seq: 2219, time: 2014-06-09 16:22:11.630303, event: done, op: pg_info(1 pgs e243:3.48a) 2014-06-09 16:22:11.630308 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630309 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630315 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630316 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630316 7f869af78700 20 osd.26 243 _dispatch 0x30b8cfc0 pg_info(1 pgs e243:1.36f) v4 2014-06-09 16:22:11.630319 7f869af78700 5 -- op tracker -- , seq: 2220, time: 2014-06-09 16:22:11.261098, event: header_read, op: pg_info(1 pgs e243:1.36f) 2014-06-09 16:22:11.630323 7f869af78700 5 -- op tracker -- , seq: 2220, time: 2014-06-09 16:22:11.261098, event: throttled, op: pg_info(1 pgs e243:1.36f) 2014-06-09 16:22:11.630326 7f869af78700 5 -- op tracker -- , seq: 2220, time: 2014-06-09 16:22:11.261116, event: all_read, op: pg_info(1 pgs e243:1.36f) 2014-06-09 16:22:11.630329 7f869af78700 5 -- op tracker -- , seq: 2220, time: 2014-06-09 16:22:11.630315, event: dispatched, op: pg_info(1 pgs e243:1.36f) 2014-06-09 16:22:11.630335 7f869af78700 5 -- op tracker -- , seq: 2220, time: 2014-06-09 16:22:11.630333, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.36f) 2014-06-09 16:22:11.630337 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.36f) v4 from osd.0 2014-06-09 16:22:11.630338 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8cfc0 2014-06-09 16:22:11.630341 7f869af78700 5 -- op tracker -- , seq: 2220, time: 2014-06-09 16:22:11.630340, event: started, op: pg_info(1 pgs e243:1.36f) 2014-06-09 16:22:11.630349 7f869af78700 5 -- op tracker -- , seq: 2220, time: 2014-06-09 16:22:11.630348, event: done, op: pg_info(1 pgs e243:1.36f) 2014-06-09 16:22:11.630353 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630354 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630360 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630361 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630361 7f869af78700 20 osd.26 243 _dispatch 0x310cb880 pg_info(1 pgs e243:2.40f) v4 2014-06-09 16:22:11.630364 7f869af78700 5 -- op tracker -- , seq: 2221, time: 2014-06-09 16:22:11.382380, event: header_read, op: pg_info(1 pgs e243:2.40f) 2014-06-09 16:22:11.630368 7f869af78700 5 -- op tracker -- , seq: 2221, time: 2014-06-09 16:22:11.382380, event: throttled, op: pg_info(1 pgs e243:2.40f) 2014-06-09 16:22:11.630371 7f869af78700 5 -- op tracker -- , seq: 2221, time: 2014-06-09 16:22:11.382398, event: all_read, op: pg_info(1 pgs e243:2.40f) 2014-06-09 16:22:11.630374 7f869af78700 5 -- op tracker -- , seq: 2221, time: 2014-06-09 16:22:11.630360, event: dispatched, op: pg_info(1 pgs e243:2.40f) 2014-06-09 16:22:11.630378 7f869af78700 5 -- op tracker -- , seq: 2221, time: 2014-06-09 16:22:11.630377, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.40f) 2014-06-09 16:22:11.630380 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.40f) v4 from osd.5 2014-06-09 16:22:11.630382 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cb880 2014-06-09 16:22:11.630384 7f869af78700 5 -- op tracker -- , seq: 2221, time: 2014-06-09 16:22:11.630383, event: started, op: pg_info(1 pgs e243:2.40f) 2014-06-09 16:22:11.630392 7f869af78700 5 -- op tracker -- , seq: 2221, time: 2014-06-09 16:22:11.630391, event: done, op: pg_info(1 pgs e243:2.40f) 2014-06-09 16:22:11.630398 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630405 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630406 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630406 7f869af78700 20 osd.26 243 _dispatch 0x307e6700 pg_notify(2.12d(27) epoch 243) v5 2014-06-09 16:22:11.630409 7f869af78700 5 -- op tracker -- , seq: 2222, time: 2014-06-09 16:22:11.234772, event: header_read, op: pg_notify(2.12d(27) epoch 243) 2014-06-09 16:22:11.630413 7f869af78700 5 -- op tracker -- , seq: 2222, time: 2014-06-09 16:22:11.234773, event: throttled, op: pg_notify(2.12d(27) epoch 243) 2014-06-09 16:22:11.630416 7f869af78700 5 -- op tracker -- , seq: 2222, time: 2014-06-09 16:22:11.234808, event: all_read, op: pg_notify(2.12d(27) epoch 243) 2014-06-09 16:22:11.630423 7f869af78700 5 -- op tracker -- , seq: 2222, time: 2014-06-09 16:22:11.630405, event: dispatched, op: pg_notify(2.12d(27) epoch 243) 2014-06-09 16:22:11.630427 7f869af78700 5 -- op tracker -- , seq: 2222, time: 2014-06-09 16:22:11.630426, event: waiting_for_osdmap, op: pg_notify(2.12d(27) epoch 243) 2014-06-09 16:22:11.630433 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.630434 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e6700 2014-06-09 16:22:11.630437 7f869af78700 5 -- op tracker -- , seq: 2222, time: 2014-06-09 16:22:11.630436, event: started, op: pg_notify(2.12d(27) epoch 243) 2014-06-09 16:22:11.630445 7f869af78700 5 -- op tracker -- , seq: 2222, time: 2014-06-09 16:22:11.630444, event: done, op: pg_notify(2.12d(27) epoch 243) 2014-06-09 16:22:11.630451 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630452 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630464 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630465 7f869af78700 20 osd.26 243 _dispatch 0x305ba000 pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.630469 7f869af78700 5 -- op tracker -- , seq: 2223, time: 2014-06-09 16:22:11.306123, event: header_read, op: pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.630473 7f869af78700 5 -- op tracker -- , seq: 2223, time: 2014-06-09 16:22:11.306124, event: throttled, op: pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.630477 7f869af78700 5 -- op tracker -- , seq: 2223, time: 2014-06-09 16:22:11.306140, event: all_read, op: pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.630480 7f869af78700 5 -- op tracker -- , seq: 2223, time: 2014-06-09 16:22:11.630463, event: dispatched, op: pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.630484 7f869af78700 5 -- op tracker -- , seq: 2223, time: 2014-06-09 16:22:11.630483, event: waiting_for_osdmap, op: pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.630490 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.630492 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305ba000 2014-06-09 16:22:11.630496 7f869af78700 5 -- op tracker -- , seq: 2223, time: 2014-06-09 16:22:11.630494, event: started, op: pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.630506 7f869af78700 5 -- op tracker -- , seq: 2223, time: 2014-06-09 16:22:11.630505, event: done, op: pg_log(0.57b epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.630513 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630521 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630522 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630522 7f869af78700 20 osd.26 243 _dispatch 0x30d25c00 pg_info(1 pgs e243:1.3fa) v4 2014-06-09 16:22:11.630525 7f869af78700 5 -- op tracker -- , seq: 2224, time: 2014-06-09 16:22:11.380331, event: header_read, op: pg_info(1 pgs e243:1.3fa) 2014-06-09 16:22:11.630529 7f869af78700 5 -- op tracker -- , seq: 2224, time: 2014-06-09 16:22:11.380331, event: throttled, op: pg_info(1 pgs e243:1.3fa) 2014-06-09 16:22:11.630532 7f869af78700 5 -- op tracker -- , seq: 2224, time: 2014-06-09 16:22:11.380341, event: all_read, op: pg_info(1 pgs e243:1.3fa) 2014-06-09 16:22:11.630535 7f869af78700 5 -- op tracker -- , seq: 2224, time: 2014-06-09 16:22:11.630521, event: dispatched, op: pg_info(1 pgs e243:1.3fa) 2014-06-09 16:22:11.630539 7f869af78700 5 -- op tracker -- , seq: 2224, time: 2014-06-09 16:22:11.630538, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3fa) 2014-06-09 16:22:11.630541 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3fa) v4 from osd.7 2014-06-09 16:22:11.630543 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d25c00 2014-06-09 16:22:11.630545 7f869af78700 5 -- op tracker -- , seq: 2224, time: 2014-06-09 16:22:11.630544, event: started, op: pg_info(1 pgs e243:1.3fa) 2014-06-09 16:22:11.630555 7f869af78700 5 -- op tracker -- , seq: 2224, time: 2014-06-09 16:22:11.630554, event: done, op: pg_info(1 pgs e243:1.3fa) 2014-06-09 16:22:11.630559 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630560 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630566 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630567 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630568 7f869af78700 20 osd.26 243 _dispatch 0x4680380 pg_info(1 pgs e243:1.641) v4 2014-06-09 16:22:11.630570 7f869af78700 5 -- op tracker -- , seq: 2225, time: 2014-06-09 16:22:11.436491, event: header_read, op: pg_info(1 pgs e243:1.641) 2014-06-09 16:22:11.630578 7f869af78700 5 -- op tracker -- , seq: 2225, time: 2014-06-09 16:22:11.436492, event: throttled, op: pg_info(1 pgs e243:1.641) 2014-06-09 16:22:11.630581 7f869af78700 5 -- op tracker -- , seq: 2225, time: 2014-06-09 16:22:11.436505, event: all_read, op: pg_info(1 pgs e243:1.641) 2014-06-09 16:22:11.630587 7f869af78700 5 -- op tracker -- , seq: 2225, time: 2014-06-09 16:22:11.630566, event: dispatched, op: pg_info(1 pgs e243:1.641) 2014-06-09 16:22:11.630591 7f869af78700 5 -- op tracker -- , seq: 2225, time: 2014-06-09 16:22:11.630590, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.641) 2014-06-09 16:22:11.630593 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.641) v4 from osd.3 2014-06-09 16:22:11.630595 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x4680380 2014-06-09 16:22:11.630597 7f869af78700 5 -- op tracker -- , seq: 2225, time: 2014-06-09 16:22:11.630596, event: started, op: pg_info(1 pgs e243:1.641) 2014-06-09 16:22:11.630607 7f869af78700 5 -- op tracker -- , seq: 2225, time: 2014-06-09 16:22:11.630606, event: done, op: pg_info(1 pgs e243:1.641) 2014-06-09 16:22:11.630611 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630612 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630619 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630619 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630620 7f869af78700 20 osd.26 243 _dispatch 0x1405cc40 pg_info(1 pgs e243:1.482) v4 2014-06-09 16:22:11.630623 7f869af78700 5 -- op tracker -- , seq: 2226, time: 2014-06-09 16:22:11.284786, event: header_read, op: pg_info(1 pgs e243:1.482) 2014-06-09 16:22:11.630626 7f869af78700 5 -- op tracker -- , seq: 2226, time: 2014-06-09 16:22:11.284786, event: throttled, op: pg_info(1 pgs e243:1.482) 2014-06-09 16:22:11.630630 7f869af78700 5 -- op tracker -- , seq: 2226, time: 2014-06-09 16:22:11.284797, event: all_read, op: pg_info(1 pgs e243:1.482) 2014-06-09 16:22:11.630633 7f869af78700 5 -- op tracker -- , seq: 2226, time: 2014-06-09 16:22:11.630619, event: dispatched, op: pg_info(1 pgs e243:1.482) 2014-06-09 16:22:11.630637 7f869af78700 5 -- op tracker -- , seq: 2226, time: 2014-06-09 16:22:11.630636, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.482) 2014-06-09 16:22:11.630639 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.482) v4 from osd.1 2014-06-09 16:22:11.630641 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405cc40 2014-06-09 16:22:11.630643 7f869af78700 5 -- op tracker -- , seq: 2226, time: 2014-06-09 16:22:11.630642, event: started, op: pg_info(1 pgs e243:1.482) 2014-06-09 16:22:11.630656 7f869af78700 5 -- op tracker -- , seq: 2226, time: 2014-06-09 16:22:11.630655, event: done, op: pg_info(1 pgs e243:1.482) 2014-06-09 16:22:11.630660 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630661 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630668 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630668 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630669 7f869af78700 20 osd.26 243 _dispatch 0x307e6e00 pg_info(1 pgs e243:2.2ef) v4 2014-06-09 16:22:11.630671 7f869af78700 5 -- op tracker -- , seq: 2227, time: 2014-06-09 16:22:11.359305, event: header_read, op: pg_info(1 pgs e243:2.2ef) 2014-06-09 16:22:11.630678 7f869af78700 5 -- op tracker -- , seq: 2227, time: 2014-06-09 16:22:11.359306, event: throttled, op: pg_info(1 pgs e243:2.2ef) 2014-06-09 16:22:11.630682 7f869af78700 5 -- op tracker -- , seq: 2227, time: 2014-06-09 16:22:11.359323, event: all_read, op: pg_info(1 pgs e243:2.2ef) 2014-06-09 16:22:11.630685 7f869af78700 5 -- op tracker -- , seq: 2227, time: 2014-06-09 16:22:11.630667, event: dispatched, op: pg_info(1 pgs e243:2.2ef) 2014-06-09 16:22:11.630689 7f869af78700 5 -- op tracker -- , seq: 2227, time: 2014-06-09 16:22:11.630688, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2ef) 2014-06-09 16:22:11.630691 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2ef) v4 from osd.9 2014-06-09 16:22:11.630693 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e6e00 2014-06-09 16:22:11.630695 7f869af78700 5 -- op tracker -- , seq: 2227, time: 2014-06-09 16:22:11.630694, event: started, op: pg_info(1 pgs e243:2.2ef) 2014-06-09 16:22:11.630703 7f869af78700 5 -- op tracker -- , seq: 2227, time: 2014-06-09 16:22:11.630702, event: done, op: pg_info(1 pgs e243:2.2ef) 2014-06-09 16:22:11.630707 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630708 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.630716 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.630717 7f869af78700 20 osd.26 243 _dispatch 0x230bb340 pg_notify(3.4d7(10),2.d(20),1.1da(20) epoch 243) v5 2014-06-09 16:22:11.630720 7f869af78700 5 -- op tracker -- , seq: 2228, time: 2014-06-09 16:22:11.373611, event: header_read, op: pg_notify(3.4d7(10),2.d(20),1.1da(20) epoch 243) 2014-06-09 16:22:11.630725 7f869af78700 5 -- op tracker -- , seq: 2228, time: 2014-06-09 16:22:11.373612, event: throttled, op: pg_notify(3.4d7(10),2.d(20),1.1da(20) epoch 243) 2014-06-09 16:22:11.630728 7f869af78700 5 -- op tracker -- , seq: 2228, time: 2014-06-09 16:22:11.373688, event: all_read, op: pg_notify(3.4d7(10),2.d(20),1.1da(20) epoch 243) 2014-06-09 16:22:11.630732 7f869af78700 5 -- op tracker -- , seq: 2228, time: 2014-06-09 16:22:11.630715, event: dispatched, op: pg_notify(3.4d7(10),2.d(20),1.1da(20) epoch 243) 2014-06-09 16:22:11.630738 7f869af78700 5 -- op tracker -- , seq: 2228, time: 2014-06-09 16:22:11.630736, event: waiting_for_osdmap, op: pg_notify(3.4d7(10),2.d(20),1.1da(20) epoch 243) 2014-06-09 16:22:11.630740 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.632278 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230bb340 2014-06-09 16:22:11.632284 7f869af78700 5 -- op tracker -- , seq: 2228, time: 2014-06-09 16:22:11.632282, event: started, op: pg_notify(3.4d7(10),2.d(20),1.1da(20) epoch 243) 2014-06-09 16:22:11.632303 7f869af78700 5 -- op tracker -- , seq: 2228, time: 2014-06-09 16:22:11.632301, event: done, op: pg_notify(3.4d7(10),2.d(20),1.1da(20) epoch 243) 2014-06-09 16:22:11.632310 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632311 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632328 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632328 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632329 7f869af78700 20 osd.26 243 _dispatch 0x31041340 pg_info(1 pgs e243:2.18e) v4 2014-06-09 16:22:11.632332 7f869af78700 5 -- op tracker -- , seq: 2229, time: 2014-06-09 16:22:11.360934, event: header_read, op: pg_info(1 pgs e243:2.18e) 2014-06-09 16:22:11.632338 7f869af78700 5 -- op tracker -- , seq: 2229, time: 2014-06-09 16:22:11.360935, event: throttled, op: pg_info(1 pgs e243:2.18e) 2014-06-09 16:22:11.632341 7f869af78700 5 -- op tracker -- , seq: 2229, time: 2014-06-09 16:22:11.360947, event: all_read, op: pg_info(1 pgs e243:2.18e) 2014-06-09 16:22:11.632345 7f869af78700 5 -- op tracker -- , seq: 2229, time: 2014-06-09 16:22:11.632327, event: dispatched, op: pg_info(1 pgs e243:2.18e) 2014-06-09 16:22:11.632348 7f869af78700 5 -- op tracker -- , seq: 2229, time: 2014-06-09 16:22:11.632347, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.18e) 2014-06-09 16:22:11.632351 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.18e) v4 from osd.6 2014-06-09 16:22:11.632353 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31041340 2014-06-09 16:22:11.632355 7f869af78700 5 -- op tracker -- , seq: 2229, time: 2014-06-09 16:22:11.632354, event: started, op: pg_info(1 pgs e243:2.18e) 2014-06-09 16:22:11.632364 7f869af78700 5 -- op tracker -- , seq: 2229, time: 2014-06-09 16:22:11.632363, event: done, op: pg_info(1 pgs e243:2.18e) 2014-06-09 16:22:11.632368 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632369 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632375 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632376 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632376 7f869af78700 20 osd.26 243 _dispatch 0x30b8ce00 pg_info(1 pgs e243:2.412) v4 2014-06-09 16:22:11.632379 7f869af78700 5 -- op tracker -- , seq: 2230, time: 2014-06-09 16:22:11.261126, event: header_read, op: pg_info(1 pgs e243:2.412) 2014-06-09 16:22:11.632383 7f869af78700 5 -- op tracker -- , seq: 2230, time: 2014-06-09 16:22:11.261126, event: throttled, op: pg_info(1 pgs e243:2.412) 2014-06-09 16:22:11.632386 7f869af78700 5 -- op tracker -- , seq: 2230, time: 2014-06-09 16:22:11.261137, event: all_read, op: pg_info(1 pgs e243:2.412) 2014-06-09 16:22:11.632393 7f869af78700 5 -- op tracker -- , seq: 2230, time: 2014-06-09 16:22:11.632375, event: dispatched, op: pg_info(1 pgs e243:2.412) 2014-06-09 16:22:11.632397 7f869af78700 5 -- op tracker -- , seq: 2230, time: 2014-06-09 16:22:11.632396, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.412) 2014-06-09 16:22:11.632399 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.412) v4 from osd.0 2014-06-09 16:22:11.632400 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8ce00 2014-06-09 16:22:11.632403 7f869af78700 5 -- op tracker -- , seq: 2230, time: 2014-06-09 16:22:11.632402, event: started, op: pg_info(1 pgs e243:2.412) 2014-06-09 16:22:11.632411 7f869af78700 5 -- op tracker -- , seq: 2230, time: 2014-06-09 16:22:11.632410, event: done, op: pg_info(1 pgs e243:2.412) 2014-06-09 16:22:11.632416 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632417 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632424 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632424 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632425 7f869af78700 20 osd.26 243 _dispatch 0x310cb6c0 pg_info(1 pgs e243:2.285) v4 2014-06-09 16:22:11.632428 7f869af78700 5 -- op tracker -- , seq: 2231, time: 2014-06-09 16:22:11.382406, event: header_read, op: pg_info(1 pgs e243:2.285) 2014-06-09 16:22:11.632431 7f869af78700 5 -- op tracker -- , seq: 2231, time: 2014-06-09 16:22:11.382406, event: throttled, op: pg_info(1 pgs e243:2.285) 2014-06-09 16:22:11.632435 7f869af78700 5 -- op tracker -- , seq: 2231, time: 2014-06-09 16:22:11.382421, event: all_read, op: pg_info(1 pgs e243:2.285) 2014-06-09 16:22:11.632438 7f869af78700 5 -- op tracker -- , seq: 2231, time: 2014-06-09 16:22:11.632423, event: dispatched, op: pg_info(1 pgs e243:2.285) 2014-06-09 16:22:11.632445 7f869af78700 5 -- op tracker -- , seq: 2231, time: 2014-06-09 16:22:11.632444, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.285) 2014-06-09 16:22:11.632447 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.285) v4 from osd.5 2014-06-09 16:22:11.632449 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cb6c0 2014-06-09 16:22:11.632451 7f869af78700 5 -- op tracker -- , seq: 2231, time: 2014-06-09 16:22:11.632450, event: started, op: pg_info(1 pgs e243:2.285) 2014-06-09 16:22:11.632460 7f869af78700 5 -- op tracker -- , seq: 2231, time: 2014-06-09 16:22:11.632459, event: done, op: pg_info(1 pgs e243:2.285) 2014-06-09 16:22:11.632464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632465 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632472 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632472 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632473 7f869af78700 20 osd.26 243 _dispatch 0x30882400 pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.632477 7f869af78700 5 -- op tracker -- , seq: 2232, time: 2014-06-09 16:22:11.241671, event: header_read, op: pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.632481 7f869af78700 5 -- op tracker -- , seq: 2232, time: 2014-06-09 16:22:11.241672, event: throttled, op: pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.632484 7f869af78700 5 -- op tracker -- , seq: 2232, time: 2014-06-09 16:22:11.241685, event: all_read, op: pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.632488 7f869af78700 5 -- op tracker -- , seq: 2232, time: 2014-06-09 16:22:11.632471, event: dispatched, op: pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.632493 7f869af78700 5 -- op tracker -- , seq: 2232, time: 2014-06-09 16:22:11.632491, event: waiting_for_osdmap, op: pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.632498 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.632501 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30882400 2014-06-09 16:22:11.632504 7f869af78700 5 -- op tracker -- , seq: 2232, time: 2014-06-09 16:22:11.632502, event: started, op: pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.632512 7f869af78700 5 -- op tracker -- , seq: 2232, time: 2014-06-09 16:22:11.632511, event: done, op: pg_log(0.179 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.632516 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632517 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632524 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632525 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632525 7f869af78700 20 osd.26 243 _dispatch 0x3005b180 pg_notify(3.573(10),3.2ac(10) epoch 243) v5 2014-06-09 16:22:11.632529 7f869af78700 5 -- op tracker -- , seq: 2233, time: 2014-06-09 16:22:11.306206, event: header_read, op: pg_notify(3.573(10),3.2ac(10) epoch 243) 2014-06-09 16:22:11.632533 7f869af78700 5 -- op tracker -- , seq: 2233, time: 2014-06-09 16:22:11.306207, event: throttled, op: pg_notify(3.573(10),3.2ac(10) epoch 243) 2014-06-09 16:22:11.632536 7f869af78700 5 -- op tracker -- , seq: 2233, time: 2014-06-09 16:22:11.306242, event: all_read, op: pg_notify(3.573(10),3.2ac(10) epoch 243) 2014-06-09 16:22:11.632540 7f869af78700 5 -- op tracker -- , seq: 2233, time: 2014-06-09 16:22:11.632524, event: dispatched, op: pg_notify(3.573(10),3.2ac(10) epoch 243) 2014-06-09 16:22:11.632547 7f869af78700 5 -- op tracker -- , seq: 2233, time: 2014-06-09 16:22:11.632545, event: waiting_for_osdmap, op: pg_notify(3.573(10),3.2ac(10) epoch 243) 2014-06-09 16:22:11.632549 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.632550 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005b180 2014-06-09 16:22:11.632553 7f869af78700 5 -- op tracker -- , seq: 2233, time: 2014-06-09 16:22:11.632552, event: started, op: pg_notify(3.573(10),3.2ac(10) epoch 243) 2014-06-09 16:22:11.632567 7f869af78700 5 -- op tracker -- , seq: 2233, time: 2014-06-09 16:22:11.632566, event: done, op: pg_notify(3.573(10),3.2ac(10) epoch 243) 2014-06-09 16:22:11.632571 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632572 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632583 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632583 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632584 7f869af78700 20 osd.26 243 _dispatch 0x30d25a40 pg_info(1 pgs e243:0.5e4) v4 2014-06-09 16:22:11.632587 7f869af78700 5 -- op tracker -- , seq: 2234, time: 2014-06-09 16:22:11.380345, event: header_read, op: pg_info(1 pgs e243:0.5e4) 2014-06-09 16:22:11.632590 7f869af78700 5 -- op tracker -- , seq: 2234, time: 2014-06-09 16:22:11.380346, event: throttled, op: pg_info(1 pgs e243:0.5e4) 2014-06-09 16:22:11.632594 7f869af78700 5 -- op tracker -- , seq: 2234, time: 2014-06-09 16:22:11.380355, event: all_read, op: pg_info(1 pgs e243:0.5e4) 2014-06-09 16:22:11.632597 7f869af78700 5 -- op tracker -- , seq: 2234, time: 2014-06-09 16:22:11.632582, event: dispatched, op: pg_info(1 pgs e243:0.5e4) 2014-06-09 16:22:11.632602 7f869af78700 5 -- op tracker -- , seq: 2234, time: 2014-06-09 16:22:11.632601, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5e4) 2014-06-09 16:22:11.632604 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5e4) v4 from osd.7 2014-06-09 16:22:11.632606 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d25a40 2014-06-09 16:22:11.632608 7f869af78700 5 -- op tracker -- , seq: 2234, time: 2014-06-09 16:22:11.632607, event: started, op: pg_info(1 pgs e243:0.5e4) 2014-06-09 16:22:11.632617 7f869af78700 5 -- op tracker -- , seq: 2234, time: 2014-06-09 16:22:11.632616, event: done, op: pg_info(1 pgs e243:0.5e4) 2014-06-09 16:22:11.632621 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632622 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632629 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632629 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632630 7f869af78700 20 osd.26 243 _dispatch 0x4680540 pg_info(1 pgs e243:3.d3) v4 2014-06-09 16:22:11.632632 7f869af78700 5 -- op tracker -- , seq: 2235, time: 2014-06-09 16:22:11.436562, event: header_read, op: pg_info(1 pgs e243:3.d3) 2014-06-09 16:22:11.632636 7f869af78700 5 -- op tracker -- , seq: 2235, time: 2014-06-09 16:22:11.436563, event: throttled, op: pg_info(1 pgs e243:3.d3) 2014-06-09 16:22:11.632639 7f869af78700 5 -- op tracker -- , seq: 2235, time: 2014-06-09 16:22:11.436576, event: all_read, op: pg_info(1 pgs e243:3.d3) 2014-06-09 16:22:11.632644 7f869af78700 5 -- op tracker -- , seq: 2235, time: 2014-06-09 16:22:11.632628, event: dispatched, op: pg_info(1 pgs e243:3.d3) 2014-06-09 16:22:11.632647 7f869af78700 5 -- op tracker -- , seq: 2235, time: 2014-06-09 16:22:11.632646, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.d3) 2014-06-09 16:22:11.632653 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.d3) v4 from osd.3 2014-06-09 16:22:11.632655 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x4680540 2014-06-09 16:22:11.632657 7f869af78700 5 -- op tracker -- , seq: 2235, time: 2014-06-09 16:22:11.632656, event: started, op: pg_info(1 pgs e243:3.d3) 2014-06-09 16:22:11.632667 7f869af78700 5 -- op tracker -- , seq: 2235, time: 2014-06-09 16:22:11.632666, event: done, op: pg_info(1 pgs e243:3.d3) 2014-06-09 16:22:11.632671 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632672 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632678 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632679 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632680 7f869af78700 20 osd.26 243 _dispatch 0x307e6380 pg_info(1 pgs e243:0.633) v4 2014-06-09 16:22:11.632682 7f869af78700 5 -- op tracker -- , seq: 2236, time: 2014-06-09 16:22:11.288857, event: header_read, op: pg_info(1 pgs e243:0.633) 2014-06-09 16:22:11.632687 7f869af78700 5 -- op tracker -- , seq: 2236, time: 2014-06-09 16:22:11.288857, event: throttled, op: pg_info(1 pgs e243:0.633) 2014-06-09 16:22:11.632691 7f869af78700 5 -- op tracker -- , seq: 2236, time: 2014-06-09 16:22:11.288869, event: all_read, op: pg_info(1 pgs e243:0.633) 2014-06-09 16:22:11.632695 7f869af78700 5 -- op tracker -- , seq: 2236, time: 2014-06-09 16:22:11.632678, event: dispatched, op: pg_info(1 pgs e243:0.633) 2014-06-09 16:22:11.632699 7f869af78700 5 -- op tracker -- , seq: 2236, time: 2014-06-09 16:22:11.632698, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.633) 2014-06-09 16:22:11.632704 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.633) v4 from osd.1 2014-06-09 16:22:11.632706 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e6380 2014-06-09 16:22:11.632709 7f869af78700 5 -- op tracker -- , seq: 2236, time: 2014-06-09 16:22:11.632708, event: started, op: pg_info(1 pgs e243:0.633) 2014-06-09 16:22:11.632717 7f869af78700 5 -- op tracker -- , seq: 2236, time: 2014-06-09 16:22:11.632716, event: done, op: pg_info(1 pgs e243:0.633) 2014-06-09 16:22:11.632721 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632722 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632729 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632730 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632731 7f869af78700 20 osd.26 243 _dispatch 0x1c038fc0 pg_info(1 pgs e243:3.505) v4 2014-06-09 16:22:11.632733 7f869af78700 5 -- op tracker -- , seq: 2237, time: 2014-06-09 16:22:11.363636, event: header_read, op: pg_info(1 pgs e243:3.505) 2014-06-09 16:22:11.632740 7f869af78700 5 -- op tracker -- , seq: 2237, time: 2014-06-09 16:22:11.363637, event: throttled, op: pg_info(1 pgs e243:3.505) 2014-06-09 16:22:11.632744 7f869af78700 5 -- op tracker -- , seq: 2237, time: 2014-06-09 16:22:11.363654, event: all_read, op: pg_info(1 pgs e243:3.505) 2014-06-09 16:22:11.632750 7f869af78700 5 -- op tracker -- , seq: 2237, time: 2014-06-09 16:22:11.632729, event: dispatched, op: pg_info(1 pgs e243:3.505) 2014-06-09 16:22:11.632754 7f869af78700 5 -- op tracker -- , seq: 2237, time: 2014-06-09 16:22:11.632753, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.505) 2014-06-09 16:22:11.632759 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.505) v4 from osd.9 2014-06-09 16:22:11.632761 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c038fc0 2014-06-09 16:22:11.632763 7f869af78700 5 -- op tracker -- , seq: 2237, time: 2014-06-09 16:22:11.632762, event: started, op: pg_info(1 pgs e243:3.505) 2014-06-09 16:22:11.632772 7f869af78700 5 -- op tracker -- , seq: 2237, time: 2014-06-09 16:22:11.632771, event: done, op: pg_info(1 pgs e243:3.505) 2014-06-09 16:22:11.632776 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632777 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632783 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632784 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632785 7f869af78700 20 osd.26 243 _dispatch 0x230bb180 pg_info(1 pgs e243:3.4c8(2)) v4 2014-06-09 16:22:11.632788 7f869af78700 5 -- op tracker -- , seq: 2238, time: 2014-06-09 16:22:11.373698, event: header_read, op: pg_info(1 pgs e243:3.4c8(2)) 2014-06-09 16:22:11.632792 7f869af78700 5 -- op tracker -- , seq: 2238, time: 2014-06-09 16:22:11.373699, event: throttled, op: pg_info(1 pgs e243:3.4c8(2)) 2014-06-09 16:22:11.632795 7f869af78700 5 -- op tracker -- , seq: 2238, time: 2014-06-09 16:22:11.373713, event: all_read, op: pg_info(1 pgs e243:3.4c8(2)) 2014-06-09 16:22:11.632799 7f869af78700 5 -- op tracker -- , seq: 2238, time: 2014-06-09 16:22:11.632783, event: dispatched, op: pg_info(1 pgs e243:3.4c8(2)) 2014-06-09 16:22:11.632803 7f869af78700 5 -- op tracker -- , seq: 2238, time: 2014-06-09 16:22:11.632802, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c8(2)) 2014-06-09 16:22:11.632805 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c8(2)) v4 from osd.8 2014-06-09 16:22:11.632807 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230bb180 2014-06-09 16:22:11.632809 7f869af78700 5 -- op tracker -- , seq: 2238, time: 2014-06-09 16:22:11.632808, event: started, op: pg_info(1 pgs e243:3.4c8(2)) 2014-06-09 16:22:11.632819 7f869af78700 5 -- op tracker -- , seq: 2238, time: 2014-06-09 16:22:11.632818, event: done, op: pg_info(1 pgs e243:3.4c8(2)) 2014-06-09 16:22:11.632823 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632824 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632830 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632831 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632831 7f869af78700 20 osd.26 243 _dispatch 0x31041180 pg_info(1 pgs e243:2.39f) v4 2014-06-09 16:22:11.632834 7f869af78700 5 -- op tracker -- , seq: 2239, time: 2014-06-09 16:22:11.360960, event: header_read, op: pg_info(1 pgs e243:2.39f) 2014-06-09 16:22:11.632838 7f869af78700 5 -- op tracker -- , seq: 2239, time: 2014-06-09 16:22:11.360960, event: throttled, op: pg_info(1 pgs e243:2.39f) 2014-06-09 16:22:11.632841 7f869af78700 5 -- op tracker -- , seq: 2239, time: 2014-06-09 16:22:11.360972, event: all_read, op: pg_info(1 pgs e243:2.39f) 2014-06-09 16:22:11.632845 7f869af78700 5 -- op tracker -- , seq: 2239, time: 2014-06-09 16:22:11.632830, event: dispatched, op: pg_info(1 pgs e243:2.39f) 2014-06-09 16:22:11.632848 7f869af78700 5 -- op tracker -- , seq: 2239, time: 2014-06-09 16:22:11.632847, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.39f) 2014-06-09 16:22:11.632851 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.39f) v4 from osd.6 2014-06-09 16:22:11.632852 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31041180 2014-06-09 16:22:11.632855 7f869af78700 5 -- op tracker -- , seq: 2239, time: 2014-06-09 16:22:11.632854, event: started, op: pg_info(1 pgs e243:2.39f) 2014-06-09 16:22:11.632865 7f869af78700 5 -- op tracker -- , seq: 2239, time: 2014-06-09 16:22:11.632864, event: done, op: pg_info(1 pgs e243:2.39f) 2014-06-09 16:22:11.632869 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632870 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632876 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632876 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632877 7f869af78700 20 osd.26 243 _dispatch 0x30b8cc40 pg_info(1 pgs e243:0.70f) v4 2014-06-09 16:22:11.632880 7f869af78700 5 -- op tracker -- , seq: 2240, time: 2014-06-09 16:22:11.261146, event: header_read, op: pg_info(1 pgs e243:0.70f) 2014-06-09 16:22:11.632883 7f869af78700 5 -- op tracker -- , seq: 2240, time: 2014-06-09 16:22:11.261147, event: throttled, op: pg_info(1 pgs e243:0.70f) 2014-06-09 16:22:11.632886 7f869af78700 5 -- op tracker -- , seq: 2240, time: 2014-06-09 16:22:11.261158, event: all_read, op: pg_info(1 pgs e243:0.70f) 2014-06-09 16:22:11.632890 7f869af78700 5 -- op tracker -- , seq: 2240, time: 2014-06-09 16:22:11.632876, event: dispatched, op: pg_info(1 pgs e243:0.70f) 2014-06-09 16:22:11.632895 7f869af78700 5 -- op tracker -- , seq: 2240, time: 2014-06-09 16:22:11.632894, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.70f) 2014-06-09 16:22:11.632897 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.70f) v4 from osd.0 2014-06-09 16:22:11.632898 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8cc40 2014-06-09 16:22:11.632901 7f869af78700 5 -- op tracker -- , seq: 2240, time: 2014-06-09 16:22:11.632900, event: started, op: pg_info(1 pgs e243:0.70f) 2014-06-09 16:22:11.632909 7f869af78700 5 -- op tracker -- , seq: 2240, time: 2014-06-09 16:22:11.632908, event: done, op: pg_info(1 pgs e243:0.70f) 2014-06-09 16:22:11.632913 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632920 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632921 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632922 7f869af78700 20 osd.26 243 _dispatch 0x310cb500 pg_info(1 pgs e243:2.73a) v4 2014-06-09 16:22:11.632924 7f869af78700 5 -- op tracker -- , seq: 2241, time: 2014-06-09 16:22:11.382426, event: header_read, op: pg_info(1 pgs e243:2.73a) 2014-06-09 16:22:11.632928 7f869af78700 5 -- op tracker -- , seq: 2241, time: 2014-06-09 16:22:11.382426, event: throttled, op: pg_info(1 pgs e243:2.73a) 2014-06-09 16:22:11.632931 7f869af78700 5 -- op tracker -- , seq: 2241, time: 2014-06-09 16:22:11.382436, event: all_read, op: pg_info(1 pgs e243:2.73a) 2014-06-09 16:22:11.632934 7f869af78700 5 -- op tracker -- , seq: 2241, time: 2014-06-09 16:22:11.632920, event: dispatched, op: pg_info(1 pgs e243:2.73a) 2014-06-09 16:22:11.632940 7f869af78700 5 -- op tracker -- , seq: 2241, time: 2014-06-09 16:22:11.632938, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.73a) 2014-06-09 16:22:11.632942 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.73a) v4 from osd.5 2014-06-09 16:22:11.632943 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cb500 2014-06-09 16:22:11.632946 7f869af78700 5 -- op tracker -- , seq: 2241, time: 2014-06-09 16:22:11.632945, event: started, op: pg_info(1 pgs e243:2.73a) 2014-06-09 16:22:11.632954 7f869af78700 5 -- op tracker -- , seq: 2241, time: 2014-06-09 16:22:11.632953, event: done, op: pg_info(1 pgs e243:2.73a) 2014-06-09 16:22:11.632960 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632961 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.632970 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.632970 7f869af78700 20 osd.26 243 _dispatch 0x308d8000 pg_notify(1.246(21),3.51e(12),3.25a(12) epoch 243) v5 2014-06-09 16:22:11.632974 7f869af78700 5 -- op tracker -- , seq: 2242, time: 2014-06-09 16:22:11.241734, event: header_read, op: pg_notify(1.246(21),3.51e(12),3.25a(12) epoch 243) 2014-06-09 16:22:11.632978 7f869af78700 5 -- op tracker -- , seq: 2242, time: 2014-06-09 16:22:11.241734, event: throttled, op: pg_notify(1.246(21),3.51e(12),3.25a(12) epoch 243) 2014-06-09 16:22:11.632982 7f869af78700 5 -- op tracker -- , seq: 2242, time: 2014-06-09 16:22:11.241799, event: all_read, op: pg_notify(1.246(21),3.51e(12),3.25a(12) epoch 243) 2014-06-09 16:22:11.632985 7f869af78700 5 -- op tracker -- , seq: 2242, time: 2014-06-09 16:22:11.632969, event: dispatched, op: pg_notify(1.246(21),3.51e(12),3.25a(12) epoch 243) 2014-06-09 16:22:11.632991 7f869af78700 5 -- op tracker -- , seq: 2242, time: 2014-06-09 16:22:11.632989, event: waiting_for_osdmap, op: pg_notify(1.246(21),3.51e(12),3.25a(12) epoch 243) 2014-06-09 16:22:11.632993 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.632994 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d8000 2014-06-09 16:22:11.632997 7f869af78700 5 -- op tracker -- , seq: 2242, time: 2014-06-09 16:22:11.632995, event: started, op: pg_notify(1.246(21),3.51e(12),3.25a(12) epoch 243) 2014-06-09 16:22:11.633014 7f869af78700 5 -- op tracker -- , seq: 2242, time: 2014-06-09 16:22:11.633013, event: done, op: pg_notify(1.246(21),3.51e(12),3.25a(12) epoch 243) 2014-06-09 16:22:11.633019 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633020 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633034 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633034 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633035 7f869af78700 20 osd.26 243 _dispatch 0x307e76c0 pg_info(1 pgs e243:3.2a9(2)) v4 2014-06-09 16:22:11.633038 7f869af78700 5 -- op tracker -- , seq: 2243, time: 2014-06-09 16:22:11.306270, event: header_read, op: pg_info(1 pgs e243:3.2a9(2)) 2014-06-09 16:22:11.633042 7f869af78700 5 -- op tracker -- , seq: 2243, time: 2014-06-09 16:22:11.306270, event: throttled, op: pg_info(1 pgs e243:3.2a9(2)) 2014-06-09 16:22:11.633045 7f869af78700 5 -- op tracker -- , seq: 2243, time: 2014-06-09 16:22:11.306285, event: all_read, op: pg_info(1 pgs e243:3.2a9(2)) 2014-06-09 16:22:11.633049 7f869af78700 5 -- op tracker -- , seq: 2243, time: 2014-06-09 16:22:11.633034, event: dispatched, op: pg_info(1 pgs e243:3.2a9(2)) 2014-06-09 16:22:11.633054 7f869af78700 5 -- op tracker -- , seq: 2243, time: 2014-06-09 16:22:11.633053, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2a9(2)) 2014-06-09 16:22:11.633057 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2a9(2)) v4 from osd.4 2014-06-09 16:22:11.633058 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e76c0 2014-06-09 16:22:11.633061 7f869af78700 5 -- op tracker -- , seq: 2243, time: 2014-06-09 16:22:11.633060, event: started, op: pg_info(1 pgs e243:3.2a9(2)) 2014-06-09 16:22:11.633070 7f869af78700 5 -- op tracker -- , seq: 2243, time: 2014-06-09 16:22:11.633068, event: done, op: pg_info(1 pgs e243:3.2a9(2)) 2014-06-09 16:22:11.633074 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633075 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633082 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633083 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633083 7f869af78700 20 osd.26 243 _dispatch 0x30d25880 pg_info(1 pgs e243:3.2c0) v4 2014-06-09 16:22:11.633086 7f869af78700 5 -- op tracker -- , seq: 2244, time: 2014-06-09 16:22:11.380360, event: header_read, op: pg_info(1 pgs e243:3.2c0) 2014-06-09 16:22:11.633090 7f869af78700 5 -- op tracker -- , seq: 2244, time: 2014-06-09 16:22:11.380360, event: throttled, op: pg_info(1 pgs e243:3.2c0) 2014-06-09 16:22:11.633093 7f869af78700 5 -- op tracker -- , seq: 2244, time: 2014-06-09 16:22:11.380370, event: all_read, op: pg_info(1 pgs e243:3.2c0) 2014-06-09 16:22:11.633096 7f869af78700 5 -- op tracker -- , seq: 2244, time: 2014-06-09 16:22:11.633082, event: dispatched, op: pg_info(1 pgs e243:3.2c0) 2014-06-09 16:22:11.633099 7f869af78700 5 -- op tracker -- , seq: 2244, time: 2014-06-09 16:22:11.633098, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2c0) 2014-06-09 16:22:11.633101 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2c0) v4 from osd.7 2014-06-09 16:22:11.633103 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d25880 2014-06-09 16:22:11.633105 7f869af78700 5 -- op tracker -- , seq: 2244, time: 2014-06-09 16:22:11.633104, event: started, op: pg_info(1 pgs e243:3.2c0) 2014-06-09 16:22:11.633118 7f869af78700 5 -- op tracker -- , seq: 2244, time: 2014-06-09 16:22:11.633117, event: done, op: pg_info(1 pgs e243:3.2c0) 2014-06-09 16:22:11.633124 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633125 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633131 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633132 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633133 7f869af78700 20 osd.26 243 _dispatch 0x13be4380 pg_info(1 pgs e243:1.282) v4 2014-06-09 16:22:11.633135 7f869af78700 5 -- op tracker -- , seq: 2245, time: 2014-06-09 16:22:11.436583, event: header_read, op: pg_info(1 pgs e243:1.282) 2014-06-09 16:22:11.633139 7f869af78700 5 -- op tracker -- , seq: 2245, time: 2014-06-09 16:22:11.436584, event: throttled, op: pg_info(1 pgs e243:1.282) 2014-06-09 16:22:11.633142 7f869af78700 5 -- op tracker -- , seq: 2245, time: 2014-06-09 16:22:11.436598, event: all_read, op: pg_info(1 pgs e243:1.282) 2014-06-09 16:22:11.633146 7f869af78700 5 -- op tracker -- , seq: 2245, time: 2014-06-09 16:22:11.633131, event: dispatched, op: pg_info(1 pgs e243:1.282) 2014-06-09 16:22:11.633149 7f869af78700 5 -- op tracker -- , seq: 2245, time: 2014-06-09 16:22:11.633148, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.282) 2014-06-09 16:22:11.633151 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.282) v4 from osd.3 2014-06-09 16:22:11.633153 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x13be4380 2014-06-09 16:22:11.633155 7f869af78700 5 -- op tracker -- , seq: 2245, time: 2014-06-09 16:22:11.633154, event: started, op: pg_info(1 pgs e243:1.282) 2014-06-09 16:22:11.633163 7f869af78700 5 -- op tracker -- , seq: 2245, time: 2014-06-09 16:22:11.633162, event: done, op: pg_info(1 pgs e243:1.282) 2014-06-09 16:22:11.633167 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633168 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633175 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633176 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633176 7f869af78700 20 osd.26 243 _dispatch 0x19804fc0 pg_info(1 pgs e243:0.686) v4 2014-06-09 16:22:11.633179 7f869af78700 5 -- op tracker -- , seq: 2246, time: 2014-06-09 16:22:11.288933, event: header_read, op: pg_info(1 pgs e243:0.686) 2014-06-09 16:22:11.633183 7f869af78700 5 -- op tracker -- , seq: 2246, time: 2014-06-09 16:22:11.288933, event: throttled, op: pg_info(1 pgs e243:0.686) 2014-06-09 16:22:11.633186 7f869af78700 5 -- op tracker -- , seq: 2246, time: 2014-06-09 16:22:11.288945, event: all_read, op: pg_info(1 pgs e243:0.686) 2014-06-09 16:22:11.633189 7f869af78700 5 -- op tracker -- , seq: 2246, time: 2014-06-09 16:22:11.633175, event: dispatched, op: pg_info(1 pgs e243:0.686) 2014-06-09 16:22:11.633194 7f869af78700 5 -- op tracker -- , seq: 2246, time: 2014-06-09 16:22:11.633193, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.686) 2014-06-09 16:22:11.633197 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.686) v4 from osd.1 2014-06-09 16:22:11.633198 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19804fc0 2014-06-09 16:22:11.633201 7f869af78700 5 -- op tracker -- , seq: 2246, time: 2014-06-09 16:22:11.633200, event: started, op: pg_info(1 pgs e243:0.686) 2014-06-09 16:22:11.633209 7f869af78700 5 -- op tracker -- , seq: 2246, time: 2014-06-09 16:22:11.633208, event: done, op: pg_info(1 pgs e243:0.686) 2014-06-09 16:22:11.633213 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633214 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633220 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633221 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633221 7f869af78700 20 osd.26 243 _dispatch 0x1abd4380 pg_info(1 pgs e243:1.301) v4 2014-06-09 16:22:11.633224 7f869af78700 5 -- op tracker -- , seq: 2247, time: 2014-06-09 16:22:11.363664, event: header_read, op: pg_info(1 pgs e243:1.301) 2014-06-09 16:22:11.633231 7f869af78700 5 -- op tracker -- , seq: 2247, time: 2014-06-09 16:22:11.363664, event: throttled, op: pg_info(1 pgs e243:1.301) 2014-06-09 16:22:11.633234 7f869af78700 5 -- op tracker -- , seq: 2247, time: 2014-06-09 16:22:11.363675, event: all_read, op: pg_info(1 pgs e243:1.301) 2014-06-09 16:22:11.633238 7f869af78700 5 -- op tracker -- , seq: 2247, time: 2014-06-09 16:22:11.633220, event: dispatched, op: pg_info(1 pgs e243:1.301) 2014-06-09 16:22:11.633241 7f869af78700 5 -- op tracker -- , seq: 2247, time: 2014-06-09 16:22:11.633240, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.301) 2014-06-09 16:22:11.633245 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.301) v4 from osd.9 2014-06-09 16:22:11.633246 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd4380 2014-06-09 16:22:11.633249 7f869af78700 5 -- op tracker -- , seq: 2247, time: 2014-06-09 16:22:11.633248, event: started, op: pg_info(1 pgs e243:1.301) 2014-06-09 16:22:11.633258 7f869af78700 5 -- op tracker -- , seq: 2247, time: 2014-06-09 16:22:11.633257, event: done, op: pg_info(1 pgs e243:1.301) 2014-06-09 16:22:11.633262 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633263 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633269 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633270 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633271 7f869af78700 20 osd.26 243 _dispatch 0x230ba000 pg_info(1 pgs e243:0.629) v4 2014-06-09 16:22:11.633273 7f869af78700 5 -- op tracker -- , seq: 2248, time: 2014-06-09 16:22:11.374483, event: header_read, op: pg_info(1 pgs e243:0.629) 2014-06-09 16:22:11.633277 7f869af78700 5 -- op tracker -- , seq: 2248, time: 2014-06-09 16:22:11.374484, event: throttled, op: pg_info(1 pgs e243:0.629) 2014-06-09 16:22:11.633280 7f869af78700 5 -- op tracker -- , seq: 2248, time: 2014-06-09 16:22:11.374497, event: all_read, op: pg_info(1 pgs e243:0.629) 2014-06-09 16:22:11.633285 7f869af78700 5 -- op tracker -- , seq: 2248, time: 2014-06-09 16:22:11.633269, event: dispatched, op: pg_info(1 pgs e243:0.629) 2014-06-09 16:22:11.633288 7f869af78700 5 -- op tracker -- , seq: 2248, time: 2014-06-09 16:22:11.633287, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.629) 2014-06-09 16:22:11.633294 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.629) v4 from osd.8 2014-06-09 16:22:11.633296 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230ba000 2014-06-09 16:22:11.633298 7f869af78700 5 -- op tracker -- , seq: 2248, time: 2014-06-09 16:22:11.633297, event: started, op: pg_info(1 pgs e243:0.629) 2014-06-09 16:22:11.633308 7f869af78700 5 -- op tracker -- , seq: 2248, time: 2014-06-09 16:22:11.633307, event: done, op: pg_info(1 pgs e243:0.629) 2014-06-09 16:22:11.633313 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633314 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633320 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633321 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633322 7f869af78700 20 osd.26 243 _dispatch 0x31040fc0 pg_info(1 pgs e243:1.668) v4 2014-06-09 16:22:11.633324 7f869af78700 5 -- op tracker -- , seq: 2249, time: 2014-06-09 16:22:11.361001, event: header_read, op: pg_info(1 pgs e243:1.668) 2014-06-09 16:22:11.633331 7f869af78700 5 -- op tracker -- , seq: 2249, time: 2014-06-09 16:22:11.361002, event: throttled, op: pg_info(1 pgs e243:1.668) 2014-06-09 16:22:11.633335 7f869af78700 5 -- op tracker -- , seq: 2249, time: 2014-06-09 16:22:11.361014, event: all_read, op: pg_info(1 pgs e243:1.668) 2014-06-09 16:22:11.633340 7f869af78700 5 -- op tracker -- , seq: 2249, time: 2014-06-09 16:22:11.633320, event: dispatched, op: pg_info(1 pgs e243:1.668) 2014-06-09 16:22:11.633343 7f869af78700 5 -- op tracker -- , seq: 2249, time: 2014-06-09 16:22:11.633342, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.668) 2014-06-09 16:22:11.633346 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.668) v4 from osd.6 2014-06-09 16:22:11.633347 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040fc0 2014-06-09 16:22:11.633350 7f869af78700 5 -- op tracker -- , seq: 2249, time: 2014-06-09 16:22:11.633349, event: started, op: pg_info(1 pgs e243:1.668) 2014-06-09 16:22:11.633358 7f869af78700 5 -- op tracker -- , seq: 2249, time: 2014-06-09 16:22:11.633357, event: done, op: pg_info(1 pgs e243:1.668) 2014-06-09 16:22:11.633362 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633363 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633369 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633370 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633370 7f869af78700 20 osd.26 243 _dispatch 0x30b8ca80 pg_info(1 pgs e243:0.370) v4 2014-06-09 16:22:11.633373 7f869af78700 5 -- op tracker -- , seq: 2250, time: 2014-06-09 16:22:11.261167, event: header_read, op: pg_info(1 pgs e243:0.370) 2014-06-09 16:22:11.633377 7f869af78700 5 -- op tracker -- , seq: 2250, time: 2014-06-09 16:22:11.261168, event: throttled, op: pg_info(1 pgs e243:0.370) 2014-06-09 16:22:11.633380 7f869af78700 5 -- op tracker -- , seq: 2250, time: 2014-06-09 16:22:11.261182, event: all_read, op: pg_info(1 pgs e243:0.370) 2014-06-09 16:22:11.633384 7f869af78700 5 -- op tracker -- , seq: 2250, time: 2014-06-09 16:22:11.633369, event: dispatched, op: pg_info(1 pgs e243:0.370) 2014-06-09 16:22:11.633387 7f869af78700 5 -- op tracker -- , seq: 2250, time: 2014-06-09 16:22:11.633386, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.370) 2014-06-09 16:22:11.633389 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.370) v4 from osd.0 2014-06-09 16:22:11.633391 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8ca80 2014-06-09 16:22:11.633393 7f869af78700 5 -- op tracker -- , seq: 2250, time: 2014-06-09 16:22:11.633392, event: started, op: pg_info(1 pgs e243:0.370) 2014-06-09 16:22:11.633402 7f869af78700 5 -- op tracker -- , seq: 2250, time: 2014-06-09 16:22:11.633400, event: done, op: pg_info(1 pgs e243:0.370) 2014-06-09 16:22:11.633409 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633410 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633416 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633417 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633417 7f869af78700 20 osd.26 243 _dispatch 0x310cb340 pg_info(1 pgs e243:3.ff) v4 2014-06-09 16:22:11.633420 7f869af78700 5 -- op tracker -- , seq: 2251, time: 2014-06-09 16:22:11.382459, event: header_read, op: pg_info(1 pgs e243:3.ff) 2014-06-09 16:22:11.633424 7f869af78700 5 -- op tracker -- , seq: 2251, time: 2014-06-09 16:22:11.382459, event: throttled, op: pg_info(1 pgs e243:3.ff) 2014-06-09 16:22:11.633427 7f869af78700 5 -- op tracker -- , seq: 2251, time: 2014-06-09 16:22:11.382470, event: all_read, op: pg_info(1 pgs e243:3.ff) 2014-06-09 16:22:11.633430 7f869af78700 5 -- op tracker -- , seq: 2251, time: 2014-06-09 16:22:11.633416, event: dispatched, op: pg_info(1 pgs e243:3.ff) 2014-06-09 16:22:11.633437 7f869af78700 5 -- op tracker -- , seq: 2251, time: 2014-06-09 16:22:11.633436, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.ff) 2014-06-09 16:22:11.633440 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.ff) v4 from osd.5 2014-06-09 16:22:11.633441 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cb340 2014-06-09 16:22:11.633444 7f869af78700 5 -- op tracker -- , seq: 2251, time: 2014-06-09 16:22:11.633443, event: started, op: pg_info(1 pgs e243:3.ff) 2014-06-09 16:22:11.633452 7f869af78700 5 -- op tracker -- , seq: 2251, time: 2014-06-09 16:22:11.633451, event: done, op: pg_info(1 pgs e243:3.ff) 2014-06-09 16:22:11.633456 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633457 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633464 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633465 7f869af78700 20 osd.26 243 _dispatch 0x29ec56c0 pg_info(1 pgs e243:3.251(4)) v4 2014-06-09 16:22:11.633468 7f869af78700 5 -- op tracker -- , seq: 2252, time: 2014-06-09 16:22:11.241806, event: header_read, op: pg_info(1 pgs e243:3.251(4)) 2014-06-09 16:22:11.633471 7f869af78700 5 -- op tracker -- , seq: 2252, time: 2014-06-09 16:22:11.241807, event: throttled, op: pg_info(1 pgs e243:3.251(4)) 2014-06-09 16:22:11.633475 7f869af78700 5 -- op tracker -- , seq: 2252, time: 2014-06-09 16:22:11.241821, event: all_read, op: pg_info(1 pgs e243:3.251(4)) 2014-06-09 16:22:11.633478 7f869af78700 5 -- op tracker -- , seq: 2252, time: 2014-06-09 16:22:11.633463, event: dispatched, op: pg_info(1 pgs e243:3.251(4)) 2014-06-09 16:22:11.633485 7f869af78700 5 -- op tracker -- , seq: 2252, time: 2014-06-09 16:22:11.633484, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.251(4)) 2014-06-09 16:22:11.633488 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.251(4)) v4 from osd.2 2014-06-09 16:22:11.633489 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec56c0 2014-06-09 16:22:11.633492 7f869af78700 5 -- op tracker -- , seq: 2252, time: 2014-06-09 16:22:11.633491, event: started, op: pg_info(1 pgs e243:3.251(4)) 2014-06-09 16:22:11.633500 7f869af78700 5 -- op tracker -- , seq: 2252, time: 2014-06-09 16:22:11.633499, event: done, op: pg_info(1 pgs e243:3.251(4)) 2014-06-09 16:22:11.633504 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633505 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633513 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633514 7f869af78700 20 osd.26 243 _dispatch 0x30a99200 pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.633518 7f869af78700 5 -- op tracker -- , seq: 2253, time: 2014-06-09 16:22:11.310413, event: header_read, op: pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.633522 7f869af78700 5 -- op tracker -- , seq: 2253, time: 2014-06-09 16:22:11.310413, event: throttled, op: pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.633529 7f869af78700 5 -- op tracker -- , seq: 2253, time: 2014-06-09 16:22:11.310422, event: all_read, op: pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.633533 7f869af78700 5 -- op tracker -- , seq: 2253, time: 2014-06-09 16:22:11.633513, event: dispatched, op: pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.633540 7f869af78700 5 -- op tracker -- , seq: 2253, time: 2014-06-09 16:22:11.633538, event: waiting_for_osdmap, op: pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.633542 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.633544 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a99200 2014-06-09 16:22:11.633547 7f869af78700 5 -- op tracker -- , seq: 2253, time: 2014-06-09 16:22:11.633546, event: started, op: pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.633557 7f869af78700 5 -- op tracker -- , seq: 2253, time: 2014-06-09 16:22:11.633556, event: done, op: pg_log(2.719 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.633562 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633563 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633570 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633570 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633571 7f869af78700 20 osd.26 243 _dispatch 0x31053dc0 pg_info(1 pgs e243:1.116) v4 2014-06-09 16:22:11.633573 7f869af78700 5 -- op tracker -- , seq: 2254, time: 2014-06-09 16:22:11.380376, event: header_read, op: pg_info(1 pgs e243:1.116) 2014-06-09 16:22:11.633577 7f869af78700 5 -- op tracker -- , seq: 2254, time: 2014-06-09 16:22:11.380376, event: throttled, op: pg_info(1 pgs e243:1.116) 2014-06-09 16:22:11.633584 7f869af78700 5 -- op tracker -- , seq: 2254, time: 2014-06-09 16:22:11.380387, event: all_read, op: pg_info(1 pgs e243:1.116) 2014-06-09 16:22:11.633587 7f869af78700 5 -- op tracker -- , seq: 2254, time: 2014-06-09 16:22:11.633569, event: dispatched, op: pg_info(1 pgs e243:1.116) 2014-06-09 16:22:11.633591 7f869af78700 5 -- op tracker -- , seq: 2254, time: 2014-06-09 16:22:11.633590, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.116) 2014-06-09 16:22:11.633593 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.116) v4 from osd.7 2014-06-09 16:22:11.633595 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31053dc0 2014-06-09 16:22:11.633597 7f869af78700 5 -- op tracker -- , seq: 2254, time: 2014-06-09 16:22:11.633596, event: started, op: pg_info(1 pgs e243:1.116) 2014-06-09 16:22:11.633607 7f869af78700 5 -- op tracker -- , seq: 2254, time: 2014-06-09 16:22:11.633606, event: done, op: pg_info(1 pgs e243:1.116) 2014-06-09 16:22:11.633611 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633612 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633618 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633619 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633620 7f869af78700 20 osd.26 243 _dispatch 0x1405ca80 pg_info(1 pgs e243:2.5f1) v4 2014-06-09 16:22:11.633622 7f869af78700 5 -- op tracker -- , seq: 2255, time: 2014-06-09 16:22:11.436628, event: header_read, op: pg_info(1 pgs e243:2.5f1) 2014-06-09 16:22:11.633626 7f869af78700 5 -- op tracker -- , seq: 2255, time: 2014-06-09 16:22:11.436629, event: throttled, op: pg_info(1 pgs e243:2.5f1) 2014-06-09 16:22:11.633629 7f869af78700 5 -- op tracker -- , seq: 2255, time: 2014-06-09 16:22:11.436640, event: all_read, op: pg_info(1 pgs e243:2.5f1) 2014-06-09 16:22:11.633634 7f869af78700 5 -- op tracker -- , seq: 2255, time: 2014-06-09 16:22:11.633618, event: dispatched, op: pg_info(1 pgs e243:2.5f1) 2014-06-09 16:22:11.633638 7f869af78700 5 -- op tracker -- , seq: 2255, time: 2014-06-09 16:22:11.633637, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5f1) 2014-06-09 16:22:11.633640 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5f1) v4 from osd.3 2014-06-09 16:22:11.633641 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405ca80 2014-06-09 16:22:11.633644 7f869af78700 5 -- op tracker -- , seq: 2255, time: 2014-06-09 16:22:11.633643, event: started, op: pg_info(1 pgs e243:2.5f1) 2014-06-09 16:22:11.633652 7f869af78700 5 -- op tracker -- , seq: 2255, time: 2014-06-09 16:22:11.633651, event: done, op: pg_info(1 pgs e243:2.5f1) 2014-06-09 16:22:11.633655 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633656 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633663 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633663 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633664 7f869af78700 20 osd.26 243 _dispatch 0x1405d180 pg_info(1 pgs e243:3.78a) v4 2014-06-09 16:22:11.633667 7f869af78700 5 -- op tracker -- , seq: 2256, time: 2014-06-09 16:22:11.288958, event: header_read, op: pg_info(1 pgs e243:3.78a) 2014-06-09 16:22:11.633670 7f869af78700 5 -- op tracker -- , seq: 2256, time: 2014-06-09 16:22:11.288958, event: throttled, op: pg_info(1 pgs e243:3.78a) 2014-06-09 16:22:11.633674 7f869af78700 5 -- op tracker -- , seq: 2256, time: 2014-06-09 16:22:11.288977, event: all_read, op: pg_info(1 pgs e243:3.78a) 2014-06-09 16:22:11.633680 7f869af78700 5 -- op tracker -- , seq: 2256, time: 2014-06-09 16:22:11.633663, event: dispatched, op: pg_info(1 pgs e243:3.78a) 2014-06-09 16:22:11.633684 7f869af78700 5 -- op tracker -- , seq: 2256, time: 2014-06-09 16:22:11.633683, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.78a) 2014-06-09 16:22:11.633689 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.78a) v4 from osd.1 2014-06-09 16:22:11.633691 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405d180 2014-06-09 16:22:11.633693 7f869af78700 5 -- op tracker -- , seq: 2256, time: 2014-06-09 16:22:11.633692, event: started, op: pg_info(1 pgs e243:3.78a) 2014-06-09 16:22:11.633702 7f869af78700 5 -- op tracker -- , seq: 2256, time: 2014-06-09 16:22:11.633701, event: done, op: pg_info(1 pgs e243:3.78a) 2014-06-09 16:22:11.633706 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633707 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633714 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633714 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633715 7f869af78700 20 osd.26 243 _dispatch 0x1abd5a40 pg_info(1 pgs e243:0.34e) v4 2014-06-09 16:22:11.633718 7f869af78700 5 -- op tracker -- , seq: 2257, time: 2014-06-09 16:22:11.363680, event: header_read, op: pg_info(1 pgs e243:0.34e) 2014-06-09 16:22:11.633722 7f869af78700 5 -- op tracker -- , seq: 2257, time: 2014-06-09 16:22:11.363681, event: throttled, op: pg_info(1 pgs e243:0.34e) 2014-06-09 16:22:11.633729 7f869af78700 5 -- op tracker -- , seq: 2257, time: 2014-06-09 16:22:11.363691, event: all_read, op: pg_info(1 pgs e243:0.34e) 2014-06-09 16:22:11.633733 7f869af78700 5 -- op tracker -- , seq: 2257, time: 2014-06-09 16:22:11.633714, event: dispatched, op: pg_info(1 pgs e243:0.34e) 2014-06-09 16:22:11.633737 7f869af78700 5 -- op tracker -- , seq: 2257, time: 2014-06-09 16:22:11.633736, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.34e) 2014-06-09 16:22:11.633742 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.34e) v4 from osd.9 2014-06-09 16:22:11.633744 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd5a40 2014-06-09 16:22:11.633746 7f869af78700 5 -- op tracker -- , seq: 2257, time: 2014-06-09 16:22:11.633745, event: started, op: pg_info(1 pgs e243:0.34e) 2014-06-09 16:22:11.633757 7f869af78700 5 -- op tracker -- , seq: 2257, time: 2014-06-09 16:22:11.633756, event: done, op: pg_info(1 pgs e243:0.34e) 2014-06-09 16:22:11.633764 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633765 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633772 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633773 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633773 7f869af78700 20 osd.26 243 _dispatch 0x3104bc00 pg_info(1 pgs e243:2.360) v4 2014-06-09 16:22:11.633776 7f869af78700 5 -- op tracker -- , seq: 2258, time: 2014-06-09 16:22:11.374522, event: header_read, op: pg_info(1 pgs e243:2.360) 2014-06-09 16:22:11.633781 7f869af78700 5 -- op tracker -- , seq: 2258, time: 2014-06-09 16:22:11.374523, event: throttled, op: pg_info(1 pgs e243:2.360) 2014-06-09 16:22:11.633784 7f869af78700 5 -- op tracker -- , seq: 2258, time: 2014-06-09 16:22:11.374537, event: all_read, op: pg_info(1 pgs e243:2.360) 2014-06-09 16:22:11.633788 7f869af78700 5 -- op tracker -- , seq: 2258, time: 2014-06-09 16:22:11.633772, event: dispatched, op: pg_info(1 pgs e243:2.360) 2014-06-09 16:22:11.633793 7f869af78700 5 -- op tracker -- , seq: 2258, time: 2014-06-09 16:22:11.633792, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.360) 2014-06-09 16:22:11.633796 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.360) v4 from osd.8 2014-06-09 16:22:11.633797 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104bc00 2014-06-09 16:22:11.633799 7f869af78700 5 -- op tracker -- , seq: 2258, time: 2014-06-09 16:22:11.633798, event: started, op: pg_info(1 pgs e243:2.360) 2014-06-09 16:22:11.633812 7f869af78700 5 -- op tracker -- , seq: 2258, time: 2014-06-09 16:22:11.633811, event: done, op: pg_info(1 pgs e243:2.360) 2014-06-09 16:22:11.633816 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633817 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633823 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633824 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633824 7f869af78700 20 osd.26 243 _dispatch 0x31040e00 pg_info(1 pgs e243:3.672) v4 2014-06-09 16:22:11.633827 7f869af78700 5 -- op tracker -- , seq: 2259, time: 2014-06-09 16:22:11.380210, event: header_read, op: pg_info(1 pgs e243:3.672) 2014-06-09 16:22:11.633830 7f869af78700 5 -- op tracker -- , seq: 2259, time: 2014-06-09 16:22:11.380219, event: throttled, op: pg_info(1 pgs e243:3.672) 2014-06-09 16:22:11.633834 7f869af78700 5 -- op tracker -- , seq: 2259, time: 2014-06-09 16:22:11.380233, event: all_read, op: pg_info(1 pgs e243:3.672) 2014-06-09 16:22:11.633837 7f869af78700 5 -- op tracker -- , seq: 2259, time: 2014-06-09 16:22:11.633823, event: dispatched, op: pg_info(1 pgs e243:3.672) 2014-06-09 16:22:11.633842 7f869af78700 5 -- op tracker -- , seq: 2259, time: 2014-06-09 16:22:11.633841, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.672) 2014-06-09 16:22:11.633845 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.672) v4 from osd.6 2014-06-09 16:22:11.633846 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040e00 2014-06-09 16:22:11.633849 7f869af78700 5 -- op tracker -- , seq: 2259, time: 2014-06-09 16:22:11.633848, event: started, op: pg_info(1 pgs e243:3.672) 2014-06-09 16:22:11.633857 7f869af78700 5 -- op tracker -- , seq: 2259, time: 2014-06-09 16:22:11.633856, event: done, op: pg_info(1 pgs e243:3.672) 2014-06-09 16:22:11.633861 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633862 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633868 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633869 7f869af78700 20 osd.26 243 _dispatch 0x30b8c8c0 pg_info(1 pgs e243:2.5f4) v4 2014-06-09 16:22:11.633872 7f869af78700 5 -- op tracker -- , seq: 2260, time: 2014-06-09 16:22:11.261598, event: header_read, op: pg_info(1 pgs e243:2.5f4) 2014-06-09 16:22:11.633875 7f869af78700 5 -- op tracker -- , seq: 2260, time: 2014-06-09 16:22:11.261599, event: throttled, op: pg_info(1 pgs e243:2.5f4) 2014-06-09 16:22:11.633879 7f869af78700 5 -- op tracker -- , seq: 2260, time: 2014-06-09 16:22:11.261611, event: all_read, op: pg_info(1 pgs e243:2.5f4) 2014-06-09 16:22:11.633882 7f869af78700 5 -- op tracker -- , seq: 2260, time: 2014-06-09 16:22:11.633868, event: dispatched, op: pg_info(1 pgs e243:2.5f4) 2014-06-09 16:22:11.633887 7f869af78700 5 -- op tracker -- , seq: 2260, time: 2014-06-09 16:22:11.633886, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5f4) 2014-06-09 16:22:11.633889 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5f4) v4 from osd.0 2014-06-09 16:22:11.633891 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8c8c0 2014-06-09 16:22:11.633893 7f869af78700 5 -- op tracker -- , seq: 2260, time: 2014-06-09 16:22:11.633892, event: started, op: pg_info(1 pgs e243:2.5f4) 2014-06-09 16:22:11.633901 7f869af78700 5 -- op tracker -- , seq: 2260, time: 2014-06-09 16:22:11.633900, event: done, op: pg_info(1 pgs e243:2.5f4) 2014-06-09 16:22:11.633906 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633907 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633913 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633914 7f869af78700 20 osd.26 243 _dispatch 0x310cb180 pg_info(1 pgs e243:2.523) v4 2014-06-09 16:22:11.633917 7f869af78700 5 -- op tracker -- , seq: 2261, time: 2014-06-09 16:22:11.382476, event: header_read, op: pg_info(1 pgs e243:2.523) 2014-06-09 16:22:11.633921 7f869af78700 5 -- op tracker -- , seq: 2261, time: 2014-06-09 16:22:11.382477, event: throttled, op: pg_info(1 pgs e243:2.523) 2014-06-09 16:22:11.633926 7f869af78700 5 -- op tracker -- , seq: 2261, time: 2014-06-09 16:22:11.382487, event: all_read, op: pg_info(1 pgs e243:2.523) 2014-06-09 16:22:11.633929 7f869af78700 5 -- op tracker -- , seq: 2261, time: 2014-06-09 16:22:11.633913, event: dispatched, op: pg_info(1 pgs e243:2.523) 2014-06-09 16:22:11.633933 7f869af78700 5 -- op tracker -- , seq: 2261, time: 2014-06-09 16:22:11.633932, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.523) 2014-06-09 16:22:11.633935 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.523) v4 from osd.5 2014-06-09 16:22:11.633936 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cb180 2014-06-09 16:22:11.633939 7f869af78700 5 -- op tracker -- , seq: 2261, time: 2014-06-09 16:22:11.633938, event: started, op: pg_info(1 pgs e243:2.523) 2014-06-09 16:22:11.633949 7f869af78700 5 -- op tracker -- , seq: 2261, time: 2014-06-09 16:22:11.633947, event: done, op: pg_info(1 pgs e243:2.523) 2014-06-09 16:22:11.633952 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633961 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.633961 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.633962 7f869af78700 20 osd.26 243 _dispatch 0x304dea00 pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.633965 7f869af78700 5 -- op tracker -- , seq: 2262, time: 2014-06-09 16:22:11.242858, event: header_read, op: pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.633970 7f869af78700 5 -- op tracker -- , seq: 2262, time: 2014-06-09 16:22:11.242859, event: throttled, op: pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.633978 7f869af78700 5 -- op tracker -- , seq: 2262, time: 2014-06-09 16:22:11.243945, event: all_read, op: pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.633981 7f869af78700 5 -- op tracker -- , seq: 2262, time: 2014-06-09 16:22:11.633960, event: dispatched, op: pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.633989 7f869af78700 5 -- op tracker -- , seq: 2262, time: 2014-06-09 16:22:11.633987, event: waiting_for_osdmap, op: pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.633991 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) v4 from osd.2 2014-06-09 16:22:11.633993 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304dea00 2014-06-09 16:22:11.633996 7f869af78700 5 -- op tracker -- , seq: 2262, time: 2014-06-09 16:22:11.633995, event: started, op: pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634006 7f869af78700 5 -- op tracker -- , seq: 2262, time: 2014-06-09 16:22:11.634004, event: done, op: pg_log(3.25a epoch 243 log log((0'0,213'576], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634011 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634012 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634020 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634020 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634021 7f869af78700 20 osd.26 243 _dispatch 0x30a8be00 pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.634024 7f869af78700 5 -- op tracker -- , seq: 2263, time: 2014-06-09 16:22:11.310484, event: header_read, op: pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634029 7f869af78700 5 -- op tracker -- , seq: 2263, time: 2014-06-09 16:22:11.310484, event: throttled, op: pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634032 7f869af78700 5 -- op tracker -- , seq: 2263, time: 2014-06-09 16:22:11.310496, event: all_read, op: pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634037 7f869af78700 5 -- op tracker -- , seq: 2263, time: 2014-06-09 16:22:11.634019, event: dispatched, op: pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634044 7f869af78700 5 -- op tracker -- , seq: 2263, time: 2014-06-09 16:22:11.634043, event: waiting_for_osdmap, op: pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634047 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.634049 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a8be00 2014-06-09 16:22:11.634052 7f869af78700 5 -- op tracker -- , seq: 2263, time: 2014-06-09 16:22:11.634050, event: started, op: pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634059 7f869af78700 5 -- op tracker -- , seq: 2263, time: 2014-06-09 16:22:11.634057, event: done, op: pg_log(2.482 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634063 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634064 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634070 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634070 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634071 7f869af78700 20 osd.26 243 _dispatch 0x31053c00 pg_info(1 pgs e243:3.3e6) v4 2014-06-09 16:22:11.634073 7f869af78700 5 -- op tracker -- , seq: 2264, time: 2014-06-09 16:22:11.380392, event: header_read, op: pg_info(1 pgs e243:3.3e6) 2014-06-09 16:22:11.634077 7f869af78700 5 -- op tracker -- , seq: 2264, time: 2014-06-09 16:22:11.380392, event: throttled, op: pg_info(1 pgs e243:3.3e6) 2014-06-09 16:22:11.634080 7f869af78700 5 -- op tracker -- , seq: 2264, time: 2014-06-09 16:22:11.380408, event: all_read, op: pg_info(1 pgs e243:3.3e6) 2014-06-09 16:22:11.634084 7f869af78700 5 -- op tracker -- , seq: 2264, time: 2014-06-09 16:22:11.634069, event: dispatched, op: pg_info(1 pgs e243:3.3e6) 2014-06-09 16:22:11.634087 7f869af78700 5 -- op tracker -- , seq: 2264, time: 2014-06-09 16:22:11.634086, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3e6) 2014-06-09 16:22:11.634090 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3e6) v4 from osd.7 2014-06-09 16:22:11.634091 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31053c00 2014-06-09 16:22:11.634094 7f869af78700 5 -- op tracker -- , seq: 2264, time: 2014-06-09 16:22:11.634093, event: started, op: pg_info(1 pgs e243:3.3e6) 2014-06-09 16:22:11.634102 7f869af78700 5 -- op tracker -- , seq: 2264, time: 2014-06-09 16:22:11.634101, event: done, op: pg_info(1 pgs e243:3.3e6) 2014-06-09 16:22:11.634106 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634107 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634114 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634114 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634115 7f869af78700 20 osd.26 243 _dispatch 0x22ac01c0 pg_info(1 pgs e243:3.f6) v4 2014-06-09 16:22:11.634118 7f869af78700 5 -- op tracker -- , seq: 2265, time: 2014-06-09 16:22:11.436650, event: header_read, op: pg_info(1 pgs e243:3.f6) 2014-06-09 16:22:11.634122 7f869af78700 5 -- op tracker -- , seq: 2265, time: 2014-06-09 16:22:11.436650, event: throttled, op: pg_info(1 pgs e243:3.f6) 2014-06-09 16:22:11.634125 7f869af78700 5 -- op tracker -- , seq: 2265, time: 2014-06-09 16:22:11.436662, event: all_read, op: pg_info(1 pgs e243:3.f6) 2014-06-09 16:22:11.634128 7f869af78700 5 -- op tracker -- , seq: 2265, time: 2014-06-09 16:22:11.634114, event: dispatched, op: pg_info(1 pgs e243:3.f6) 2014-06-09 16:22:11.634132 7f869af78700 5 -- op tracker -- , seq: 2265, time: 2014-06-09 16:22:11.634131, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.f6) 2014-06-09 16:22:11.634134 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.f6) v4 from osd.3 2014-06-09 16:22:11.634135 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x22ac01c0 2014-06-09 16:22:11.634138 7f869af78700 5 -- op tracker -- , seq: 2265, time: 2014-06-09 16:22:11.634137, event: started, op: pg_info(1 pgs e243:3.f6) 2014-06-09 16:22:11.634146 7f869af78700 5 -- op tracker -- , seq: 2265, time: 2014-06-09 16:22:11.634145, event: done, op: pg_info(1 pgs e243:3.f6) 2014-06-09 16:22:11.634150 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634151 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634157 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634158 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634158 7f869af78700 20 osd.26 243 _dispatch 0x1b0ce000 pg_info(1 pgs e243:0.ca) v4 2014-06-09 16:22:11.634161 7f869af78700 5 -- op tracker -- , seq: 2266, time: 2014-06-09 16:22:11.310994, event: header_read, op: pg_info(1 pgs e243:0.ca) 2014-06-09 16:22:11.634165 7f869af78700 5 -- op tracker -- , seq: 2266, time: 2014-06-09 16:22:11.310995, event: throttled, op: pg_info(1 pgs e243:0.ca) 2014-06-09 16:22:11.634168 7f869af78700 5 -- op tracker -- , seq: 2266, time: 2014-06-09 16:22:11.311004, event: all_read, op: pg_info(1 pgs e243:0.ca) 2014-06-09 16:22:11.634171 7f869af78700 5 -- op tracker -- , seq: 2266, time: 2014-06-09 16:22:11.634157, event: dispatched, op: pg_info(1 pgs e243:0.ca) 2014-06-09 16:22:11.634175 7f869af78700 5 -- op tracker -- , seq: 2266, time: 2014-06-09 16:22:11.634174, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.ca) 2014-06-09 16:22:11.634181 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.ca) v4 from osd.1 2014-06-09 16:22:11.634183 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1b0ce000 2014-06-09 16:22:11.634185 7f869af78700 5 -- op tracker -- , seq: 2266, time: 2014-06-09 16:22:11.634184, event: started, op: pg_info(1 pgs e243:0.ca) 2014-06-09 16:22:11.634193 7f869af78700 5 -- op tracker -- , seq: 2266, time: 2014-06-09 16:22:11.634192, event: done, op: pg_info(1 pgs e243:0.ca) 2014-06-09 16:22:11.634197 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634198 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634205 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634206 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634206 7f869af78700 20 osd.26 243 _dispatch 0x1ac94c40 pg_info(1 pgs e243:0.89) v4 2014-06-09 16:22:11.634209 7f869af78700 5 -- op tracker -- , seq: 2267, time: 2014-06-09 16:22:11.363696, event: header_read, op: pg_info(1 pgs e243:0.89) 2014-06-09 16:22:11.634213 7f869af78700 5 -- op tracker -- , seq: 2267, time: 2014-06-09 16:22:11.363696, event: throttled, op: pg_info(1 pgs e243:0.89) 2014-06-09 16:22:11.634216 7f869af78700 5 -- op tracker -- , seq: 2267, time: 2014-06-09 16:22:11.363708, event: all_read, op: pg_info(1 pgs e243:0.89) 2014-06-09 16:22:11.634220 7f869af78700 5 -- op tracker -- , seq: 2267, time: 2014-06-09 16:22:11.634205, event: dispatched, op: pg_info(1 pgs e243:0.89) 2014-06-09 16:22:11.634225 7f869af78700 5 -- op tracker -- , seq: 2267, time: 2014-06-09 16:22:11.634224, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.89) 2014-06-09 16:22:11.634227 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.89) v4 from osd.9 2014-06-09 16:22:11.634229 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac94c40 2014-06-09 16:22:11.634231 7f869af78700 5 -- op tracker -- , seq: 2267, time: 2014-06-09 16:22:11.634230, event: started, op: pg_info(1 pgs e243:0.89) 2014-06-09 16:22:11.634239 7f869af78700 5 -- op tracker -- , seq: 2267, time: 2014-06-09 16:22:11.634238, event: done, op: pg_info(1 pgs e243:0.89) 2014-06-09 16:22:11.634243 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634244 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634251 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634251 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634252 7f869af78700 20 osd.26 243 _dispatch 0x3104ba40 pg_info(1 pgs e243:0.e2) v4 2014-06-09 16:22:11.634255 7f869af78700 5 -- op tracker -- , seq: 2268, time: 2014-06-09 16:22:11.374563, event: header_read, op: pg_info(1 pgs e243:0.e2) 2014-06-09 16:22:11.634260 7f869af78700 5 -- op tracker -- , seq: 2268, time: 2014-06-09 16:22:11.374564, event: throttled, op: pg_info(1 pgs e243:0.e2) 2014-06-09 16:22:11.634263 7f869af78700 5 -- op tracker -- , seq: 2268, time: 2014-06-09 16:22:11.374576, event: all_read, op: pg_info(1 pgs e243:0.e2) 2014-06-09 16:22:11.634270 7f869af78700 5 -- op tracker -- , seq: 2268, time: 2014-06-09 16:22:11.634251, event: dispatched, op: pg_info(1 pgs e243:0.e2) 2014-06-09 16:22:11.634273 7f869af78700 5 -- op tracker -- , seq: 2268, time: 2014-06-09 16:22:11.634272, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.e2) 2014-06-09 16:22:11.634277 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.e2) v4 from osd.8 2014-06-09 16:22:11.634278 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104ba40 2014-06-09 16:22:11.634281 7f869af78700 5 -- op tracker -- , seq: 2268, time: 2014-06-09 16:22:11.634280, event: started, op: pg_info(1 pgs e243:0.e2) 2014-06-09 16:22:11.634291 7f869af78700 5 -- op tracker -- , seq: 2268, time: 2014-06-09 16:22:11.634290, event: done, op: pg_info(1 pgs e243:0.e2) 2014-06-09 16:22:11.634296 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634297 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634303 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634304 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634304 7f869af78700 20 osd.26 243 _dispatch 0x31040c40 pg_info(1 pgs e243:1.207) v4 2014-06-09 16:22:11.634307 7f869af78700 5 -- op tracker -- , seq: 2269, time: 2014-06-09 16:22:11.380241, event: header_read, op: pg_info(1 pgs e243:1.207) 2014-06-09 16:22:11.634311 7f869af78700 5 -- op tracker -- , seq: 2269, time: 2014-06-09 16:22:11.380241, event: throttled, op: pg_info(1 pgs e243:1.207) 2014-06-09 16:22:11.634314 7f869af78700 5 -- op tracker -- , seq: 2269, time: 2014-06-09 16:22:11.380251, event: all_read, op: pg_info(1 pgs e243:1.207) 2014-06-09 16:22:11.634317 7f869af78700 5 -- op tracker -- , seq: 2269, time: 2014-06-09 16:22:11.634303, event: dispatched, op: pg_info(1 pgs e243:1.207) 2014-06-09 16:22:11.634321 7f869af78700 5 -- op tracker -- , seq: 2269, time: 2014-06-09 16:22:11.634320, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.207) 2014-06-09 16:22:11.634323 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.207) v4 from osd.6 2014-06-09 16:22:11.634325 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040c40 2014-06-09 16:22:11.634327 7f869af78700 5 -- op tracker -- , seq: 2269, time: 2014-06-09 16:22:11.634326, event: started, op: pg_info(1 pgs e243:1.207) 2014-06-09 16:22:11.634338 7f869af78700 5 -- op tracker -- , seq: 2269, time: 2014-06-09 16:22:11.634337, event: done, op: pg_info(1 pgs e243:1.207) 2014-06-09 16:22:11.634341 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634342 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634349 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634349 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634350 7f869af78700 20 osd.26 243 _dispatch 0x30b8c700 pg_info(1 pgs e243:1.54c) v4 2014-06-09 16:22:11.634352 7f869af78700 5 -- op tracker -- , seq: 2270, time: 2014-06-09 16:22:11.261841, event: header_read, op: pg_info(1 pgs e243:1.54c) 2014-06-09 16:22:11.634356 7f869af78700 5 -- op tracker -- , seq: 2270, time: 2014-06-09 16:22:11.261842, event: throttled, op: pg_info(1 pgs e243:1.54c) 2014-06-09 16:22:11.634360 7f869af78700 5 -- op tracker -- , seq: 2270, time: 2014-06-09 16:22:11.261854, event: all_read, op: pg_info(1 pgs e243:1.54c) 2014-06-09 16:22:11.634363 7f869af78700 5 -- op tracker -- , seq: 2270, time: 2014-06-09 16:22:11.634348, event: dispatched, op: pg_info(1 pgs e243:1.54c) 2014-06-09 16:22:11.634366 7f869af78700 5 -- op tracker -- , seq: 2270, time: 2014-06-09 16:22:11.634365, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.54c) 2014-06-09 16:22:11.634369 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.54c) v4 from osd.0 2014-06-09 16:22:11.634370 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8c700 2014-06-09 16:22:11.634373 7f869af78700 5 -- op tracker -- , seq: 2270, time: 2014-06-09 16:22:11.634372, event: started, op: pg_info(1 pgs e243:1.54c) 2014-06-09 16:22:11.634380 7f869af78700 5 -- op tracker -- , seq: 2270, time: 2014-06-09 16:22:11.634379, event: done, op: pg_info(1 pgs e243:1.54c) 2014-06-09 16:22:11.634384 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634385 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634391 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634392 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634392 7f869af78700 20 osd.26 243 _dispatch 0x310cafc0 pg_info(1 pgs e243:1.25a) v4 2014-06-09 16:22:11.634395 7f869af78700 5 -- op tracker -- , seq: 2271, time: 2014-06-09 16:22:11.382547, event: header_read, op: pg_info(1 pgs e243:1.25a) 2014-06-09 16:22:11.634399 7f869af78700 5 -- op tracker -- , seq: 2271, time: 2014-06-09 16:22:11.382548, event: throttled, op: pg_info(1 pgs e243:1.25a) 2014-06-09 16:22:11.634406 7f869af78700 5 -- op tracker -- , seq: 2271, time: 2014-06-09 16:22:11.382559, event: all_read, op: pg_info(1 pgs e243:1.25a) 2014-06-09 16:22:11.634409 7f869af78700 5 -- op tracker -- , seq: 2271, time: 2014-06-09 16:22:11.634391, event: dispatched, op: pg_info(1 pgs e243:1.25a) 2014-06-09 16:22:11.634413 7f869af78700 5 -- op tracker -- , seq: 2271, time: 2014-06-09 16:22:11.634412, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.25a) 2014-06-09 16:22:11.634415 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.25a) v4 from osd.5 2014-06-09 16:22:11.634416 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cafc0 2014-06-09 16:22:11.634419 7f869af78700 5 -- op tracker -- , seq: 2271, time: 2014-06-09 16:22:11.634418, event: started, op: pg_info(1 pgs e243:1.25a) 2014-06-09 16:22:11.634429 7f869af78700 5 -- op tracker -- , seq: 2271, time: 2014-06-09 16:22:11.634428, event: done, op: pg_info(1 pgs e243:1.25a) 2014-06-09 16:22:11.634433 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634434 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634442 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634442 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634443 7f869af78700 20 osd.26 243 _dispatch 0x30725200 pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.634447 7f869af78700 5 -- op tracker -- , seq: 2272, time: 2014-06-09 16:22:11.244245, event: header_read, op: pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634451 7f869af78700 5 -- op tracker -- , seq: 2272, time: 2014-06-09 16:22:11.244246, event: throttled, op: pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634458 7f869af78700 5 -- op tracker -- , seq: 2272, time: 2014-06-09 16:22:11.245248, event: all_read, op: pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634462 7f869af78700 5 -- op tracker -- , seq: 2272, time: 2014-06-09 16:22:11.634441, event: dispatched, op: pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634466 7f869af78700 5 -- op tracker -- , seq: 2272, time: 2014-06-09 16:22:11.634465, event: waiting_for_osdmap, op: pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634468 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) v4 from osd.2 2014-06-09 16:22:11.634471 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30725200 2014-06-09 16:22:11.634474 7f869af78700 5 -- op tracker -- , seq: 2272, time: 2014-06-09 16:22:11.634472, event: started, op: pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634480 7f869af78700 5 -- op tracker -- , seq: 2272, time: 2014-06-09 16:22:11.634479, event: done, op: pg_log(3.51e epoch 243 log log((0'0,213'674], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.634484 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634485 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634492 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634493 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634493 7f869af78700 20 osd.26 243 _dispatch 0x3043a000 pg_notify(2.1ae(20),2.1b3(20) epoch 243) v5 2014-06-09 16:22:11.634496 7f869af78700 5 -- op tracker -- , seq: 2273, time: 2014-06-09 16:22:11.310553, event: header_read, op: pg_notify(2.1ae(20),2.1b3(20) epoch 243) 2014-06-09 16:22:11.634500 7f869af78700 5 -- op tracker -- , seq: 2273, time: 2014-06-09 16:22:11.310553, event: throttled, op: pg_notify(2.1ae(20),2.1b3(20) epoch 243) 2014-06-09 16:22:11.634508 7f869af78700 5 -- op tracker -- , seq: 2273, time: 2014-06-09 16:22:11.310585, event: all_read, op: pg_notify(2.1ae(20),2.1b3(20) epoch 243) 2014-06-09 16:22:11.634511 7f869af78700 5 -- op tracker -- , seq: 2273, time: 2014-06-09 16:22:11.634492, event: dispatched, op: pg_notify(2.1ae(20),2.1b3(20) epoch 243) 2014-06-09 16:22:11.634517 7f869af78700 5 -- op tracker -- , seq: 2273, time: 2014-06-09 16:22:11.634515, event: waiting_for_osdmap, op: pg_notify(2.1ae(20),2.1b3(20) epoch 243) 2014-06-09 16:22:11.634519 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.634520 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043a000 2014-06-09 16:22:11.634523 7f869af78700 5 -- op tracker -- , seq: 2273, time: 2014-06-09 16:22:11.634521, event: started, op: pg_notify(2.1ae(20),2.1b3(20) epoch 243) 2014-06-09 16:22:11.634535 7f869af78700 5 -- op tracker -- , seq: 2273, time: 2014-06-09 16:22:11.634534, event: done, op: pg_notify(2.1ae(20),2.1b3(20) epoch 243) 2014-06-09 16:22:11.634539 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634540 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634553 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634554 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634555 7f869af78700 20 osd.26 243 _dispatch 0x31053a40 pg_info(1 pgs e243:1.428) v4 2014-06-09 16:22:11.634557 7f869af78700 5 -- op tracker -- , seq: 2274, time: 2014-06-09 16:22:11.380415, event: header_read, op: pg_info(1 pgs e243:1.428) 2014-06-09 16:22:11.634562 7f869af78700 5 -- op tracker -- , seq: 2274, time: 2014-06-09 16:22:11.380416, event: throttled, op: pg_info(1 pgs e243:1.428) 2014-06-09 16:22:11.634567 7f869af78700 5 -- op tracker -- , seq: 2274, time: 2014-06-09 16:22:11.380427, event: all_read, op: pg_info(1 pgs e243:1.428) 2014-06-09 16:22:11.634570 7f869af78700 5 -- op tracker -- , seq: 2274, time: 2014-06-09 16:22:11.634553, event: dispatched, op: pg_info(1 pgs e243:1.428) 2014-06-09 16:22:11.634577 7f869af78700 5 -- op tracker -- , seq: 2274, time: 2014-06-09 16:22:11.634576, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.428) 2014-06-09 16:22:11.634579 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.428) v4 from osd.7 2014-06-09 16:22:11.634581 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31053a40 2014-06-09 16:22:11.634583 7f869af78700 5 -- op tracker -- , seq: 2274, time: 2014-06-09 16:22:11.634582, event: started, op: pg_info(1 pgs e243:1.428) 2014-06-09 16:22:11.634592 7f869af78700 5 -- op tracker -- , seq: 2274, time: 2014-06-09 16:22:11.634591, event: done, op: pg_info(1 pgs e243:1.428) 2014-06-09 16:22:11.634596 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634597 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634604 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634604 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634605 7f869af78700 20 osd.26 243 _dispatch 0x29ec48c0 pg_info(1 pgs e243:0.5d4) v4 2014-06-09 16:22:11.634608 7f869af78700 5 -- op tracker -- , seq: 2275, time: 2014-06-09 16:22:11.436694, event: header_read, op: pg_info(1 pgs e243:0.5d4) 2014-06-09 16:22:11.634611 7f869af78700 5 -- op tracker -- , seq: 2275, time: 2014-06-09 16:22:11.436694, event: throttled, op: pg_info(1 pgs e243:0.5d4) 2014-06-09 16:22:11.634618 7f869af78700 5 -- op tracker -- , seq: 2275, time: 2014-06-09 16:22:11.436706, event: all_read, op: pg_info(1 pgs e243:0.5d4) 2014-06-09 16:22:11.634621 7f869af78700 5 -- op tracker -- , seq: 2275, time: 2014-06-09 16:22:11.634604, event: dispatched, op: pg_info(1 pgs e243:0.5d4) 2014-06-09 16:22:11.634625 7f869af78700 5 -- op tracker -- , seq: 2275, time: 2014-06-09 16:22:11.634624, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5d4) 2014-06-09 16:22:11.634627 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5d4) v4 from osd.3 2014-06-09 16:22:11.634629 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec48c0 2014-06-09 16:22:11.634631 7f869af78700 5 -- op tracker -- , seq: 2275, time: 2014-06-09 16:22:11.634630, event: started, op: pg_info(1 pgs e243:0.5d4) 2014-06-09 16:22:11.634639 7f869af78700 5 -- op tracker -- , seq: 2275, time: 2014-06-09 16:22:11.634638, event: done, op: pg_info(1 pgs e243:0.5d4) 2014-06-09 16:22:11.634644 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634645 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634651 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634652 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634653 7f869af78700 20 osd.26 243 _dispatch 0x10d02540 pg_info(1 pgs e243:2.28a) v4 2014-06-09 16:22:11.634655 7f869af78700 5 -- op tracker -- , seq: 2276, time: 2014-06-09 16:22:11.311082, event: header_read, op: pg_info(1 pgs e243:2.28a) 2014-06-09 16:22:11.634659 7f869af78700 5 -- op tracker -- , seq: 2276, time: 2014-06-09 16:22:11.311083, event: throttled, op: pg_info(1 pgs e243:2.28a) 2014-06-09 16:22:11.634665 7f869af78700 5 -- op tracker -- , seq: 2276, time: 2014-06-09 16:22:11.311092, event: all_read, op: pg_info(1 pgs e243:2.28a) 2014-06-09 16:22:11.634669 7f869af78700 5 -- op tracker -- , seq: 2276, time: 2014-06-09 16:22:11.634651, event: dispatched, op: pg_info(1 pgs e243:2.28a) 2014-06-09 16:22:11.634672 7f869af78700 5 -- op tracker -- , seq: 2276, time: 2014-06-09 16:22:11.634671, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.28a) 2014-06-09 16:22:11.634675 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.28a) v4 from osd.1 2014-06-09 16:22:11.634676 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d02540 2014-06-09 16:22:11.634679 7f869af78700 5 -- op tracker -- , seq: 2276, time: 2014-06-09 16:22:11.634678, event: started, op: pg_info(1 pgs e243:2.28a) 2014-06-09 16:22:11.634688 7f869af78700 5 -- op tracker -- , seq: 2276, time: 2014-06-09 16:22:11.634687, event: done, op: pg_info(1 pgs e243:2.28a) 2014-06-09 16:22:11.634693 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634694 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634701 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634701 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634702 7f869af78700 20 osd.26 243 _dispatch 0x1ac94540 pg_info(1 pgs e243:2.154) v4 2014-06-09 16:22:11.634705 7f869af78700 5 -- op tracker -- , seq: 2277, time: 2014-06-09 16:22:11.363713, event: header_read, op: pg_info(1 pgs e243:2.154) 2014-06-09 16:22:11.634709 7f869af78700 5 -- op tracker -- , seq: 2277, time: 2014-06-09 16:22:11.363713, event: throttled, op: pg_info(1 pgs e243:2.154) 2014-06-09 16:22:11.634712 7f869af78700 5 -- op tracker -- , seq: 2277, time: 2014-06-09 16:22:11.363724, event: all_read, op: pg_info(1 pgs e243:2.154) 2014-06-09 16:22:11.634715 7f869af78700 5 -- op tracker -- , seq: 2277, time: 2014-06-09 16:22:11.634700, event: dispatched, op: pg_info(1 pgs e243:2.154) 2014-06-09 16:22:11.634720 7f869af78700 5 -- op tracker -- , seq: 2277, time: 2014-06-09 16:22:11.634719, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.154) 2014-06-09 16:22:11.634722 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.154) v4 from osd.9 2014-06-09 16:22:11.634723 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac94540 2014-06-09 16:22:11.634726 7f869af78700 5 -- op tracker -- , seq: 2277, time: 2014-06-09 16:22:11.634725, event: started, op: pg_info(1 pgs e243:2.154) 2014-06-09 16:22:11.634734 7f869af78700 5 -- op tracker -- , seq: 2277, time: 2014-06-09 16:22:11.634732, event: done, op: pg_info(1 pgs e243:2.154) 2014-06-09 16:22:11.634737 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634738 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634745 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634746 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634746 7f869af78700 20 osd.26 243 _dispatch 0x3104b880 pg_info(1 pgs e243:3.5d1) v4 2014-06-09 16:22:11.634749 7f869af78700 5 -- op tracker -- , seq: 2278, time: 2014-06-09 16:22:11.374609, event: header_read, op: pg_info(1 pgs e243:3.5d1) 2014-06-09 16:22:11.634752 7f869af78700 5 -- op tracker -- , seq: 2278, time: 2014-06-09 16:22:11.374610, event: throttled, op: pg_info(1 pgs e243:3.5d1) 2014-06-09 16:22:11.634756 7f869af78700 5 -- op tracker -- , seq: 2278, time: 2014-06-09 16:22:11.374623, event: all_read, op: pg_info(1 pgs e243:3.5d1) 2014-06-09 16:22:11.634759 7f869af78700 5 -- op tracker -- , seq: 2278, time: 2014-06-09 16:22:11.634745, event: dispatched, op: pg_info(1 pgs e243:3.5d1) 2014-06-09 16:22:11.634762 7f869af78700 5 -- op tracker -- , seq: 2278, time: 2014-06-09 16:22:11.634761, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5d1) 2014-06-09 16:22:11.634764 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5d1) v4 from osd.8 2014-06-09 16:22:11.634766 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104b880 2014-06-09 16:22:11.634768 7f869af78700 5 -- op tracker -- , seq: 2278, time: 2014-06-09 16:22:11.634767, event: started, op: pg_info(1 pgs e243:3.5d1) 2014-06-09 16:22:11.634779 7f869af78700 5 -- op tracker -- , seq: 2278, time: 2014-06-09 16:22:11.634777, event: done, op: pg_info(1 pgs e243:3.5d1) 2014-06-09 16:22:11.634786 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634787 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634796 7f869af78700 20 osd.26 243 _dispatch 0x308d3000 pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.634800 7f869af78700 5 -- op tracker -- , seq: 2279, time: 2014-06-09 16:22:11.380256, event: header_read, op: pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634806 7f869af78700 5 -- op tracker -- , seq: 2279, time: 2014-06-09 16:22:11.380257, event: throttled, op: pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634810 7f869af78700 5 -- op tracker -- , seq: 2279, time: 2014-06-09 16:22:11.380267, event: all_read, op: pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634814 7f869af78700 5 -- op tracker -- , seq: 2279, time: 2014-06-09 16:22:11.634795, event: dispatched, op: pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634822 7f869af78700 5 -- op tracker -- , seq: 2279, time: 2014-06-09 16:22:11.634820, event: waiting_for_osdmap, op: pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634824 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.634826 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d3000 2014-06-09 16:22:11.634829 7f869af78700 5 -- op tracker -- , seq: 2279, time: 2014-06-09 16:22:11.634828, event: started, op: pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634836 7f869af78700 5 -- op tracker -- , seq: 2279, time: 2014-06-09 16:22:11.634835, event: done, op: pg_log(1.233 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.634840 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634841 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634847 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634848 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634849 7f869af78700 20 osd.26 243 _dispatch 0x30b8c540 pg_info(1 pgs e243:2.65c) v4 2014-06-09 16:22:11.634851 7f869af78700 5 -- op tracker -- , seq: 2280, time: 2014-06-09 16:22:11.261943, event: header_read, op: pg_info(1 pgs e243:2.65c) 2014-06-09 16:22:11.634855 7f869af78700 5 -- op tracker -- , seq: 2280, time: 2014-06-09 16:22:11.261943, event: throttled, op: pg_info(1 pgs e243:2.65c) 2014-06-09 16:22:11.634860 7f869af78700 5 -- op tracker -- , seq: 2280, time: 2014-06-09 16:22:11.261955, event: all_read, op: pg_info(1 pgs e243:2.65c) 2014-06-09 16:22:11.634863 7f869af78700 5 -- op tracker -- , seq: 2280, time: 2014-06-09 16:22:11.634847, event: dispatched, op: pg_info(1 pgs e243:2.65c) 2014-06-09 16:22:11.634870 7f869af78700 5 -- op tracker -- , seq: 2280, time: 2014-06-09 16:22:11.634869, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.65c) 2014-06-09 16:22:11.634872 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.65c) v4 from osd.0 2014-06-09 16:22:11.634874 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8c540 2014-06-09 16:22:11.634876 7f869af78700 5 -- op tracker -- , seq: 2280, time: 2014-06-09 16:22:11.634875, event: started, op: pg_info(1 pgs e243:2.65c) 2014-06-09 16:22:11.634885 7f869af78700 5 -- op tracker -- , seq: 2280, time: 2014-06-09 16:22:11.634884, event: done, op: pg_info(1 pgs e243:2.65c) 2014-06-09 16:22:11.634889 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634890 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634896 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634897 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634897 7f869af78700 20 osd.26 243 _dispatch 0x310cae00 pg_info(1 pgs e243:0.fa) v4 2014-06-09 16:22:11.634900 7f869af78700 5 -- op tracker -- , seq: 2281, time: 2014-06-09 16:22:11.382566, event: header_read, op: pg_info(1 pgs e243:0.fa) 2014-06-09 16:22:11.634904 7f869af78700 5 -- op tracker -- , seq: 2281, time: 2014-06-09 16:22:11.382567, event: throttled, op: pg_info(1 pgs e243:0.fa) 2014-06-09 16:22:11.634907 7f869af78700 5 -- op tracker -- , seq: 2281, time: 2014-06-09 16:22:11.382577, event: all_read, op: pg_info(1 pgs e243:0.fa) 2014-06-09 16:22:11.634913 7f869af78700 5 -- op tracker -- , seq: 2281, time: 2014-06-09 16:22:11.634896, event: dispatched, op: pg_info(1 pgs e243:0.fa) 2014-06-09 16:22:11.634917 7f869af78700 5 -- op tracker -- , seq: 2281, time: 2014-06-09 16:22:11.634916, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.fa) 2014-06-09 16:22:11.634921 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.fa) v4 from osd.5 2014-06-09 16:22:11.634923 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cae00 2014-06-09 16:22:11.634925 7f869af78700 5 -- op tracker -- , seq: 2281, time: 2014-06-09 16:22:11.634924, event: started, op: pg_info(1 pgs e243:0.fa) 2014-06-09 16:22:11.634934 7f869af78700 5 -- op tracker -- , seq: 2281, time: 2014-06-09 16:22:11.634933, event: done, op: pg_info(1 pgs e243:0.fa) 2014-06-09 16:22:11.634938 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634939 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634946 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634947 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634947 7f869af78700 20 osd.26 243 _dispatch 0x190b3a40 pg_notify(2.621(21) epoch 243) v5 2014-06-09 16:22:11.634950 7f869af78700 5 -- op tracker -- , seq: 2282, time: 2014-06-09 16:22:11.245263, event: header_read, op: pg_notify(2.621(21) epoch 243) 2014-06-09 16:22:11.634954 7f869af78700 5 -- op tracker -- , seq: 2282, time: 2014-06-09 16:22:11.245264, event: throttled, op: pg_notify(2.621(21) epoch 243) 2014-06-09 16:22:11.634957 7f869af78700 5 -- op tracker -- , seq: 2282, time: 2014-06-09 16:22:11.245292, event: all_read, op: pg_notify(2.621(21) epoch 243) 2014-06-09 16:22:11.634960 7f869af78700 5 -- op tracker -- , seq: 2282, time: 2014-06-09 16:22:11.634946, event: dispatched, op: pg_notify(2.621(21) epoch 243) 2014-06-09 16:22:11.634964 7f869af78700 5 -- op tracker -- , seq: 2282, time: 2014-06-09 16:22:11.634963, event: waiting_for_osdmap, op: pg_notify(2.621(21) epoch 243) 2014-06-09 16:22:11.634967 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.634968 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b3a40 2014-06-09 16:22:11.634971 7f869af78700 5 -- op tracker -- , seq: 2282, time: 2014-06-09 16:22:11.634970, event: started, op: pg_notify(2.621(21) epoch 243) 2014-06-09 16:22:11.634979 7f869af78700 5 -- op tracker -- , seq: 2282, time: 2014-06-09 16:22:11.634978, event: done, op: pg_notify(2.621(21) epoch 243) 2014-06-09 16:22:11.634988 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.634989 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.634999 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635000 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635001 7f869af78700 20 osd.26 243 _dispatch 0x2fef4540 pg_info(1 pgs e243:3.1b1(2)) v4 2014-06-09 16:22:11.635004 7f869af78700 5 -- op tracker -- , seq: 2283, time: 2014-06-09 16:22:11.310620, event: header_read, op: pg_info(1 pgs e243:3.1b1(2)) 2014-06-09 16:22:11.635008 7f869af78700 5 -- op tracker -- , seq: 2283, time: 2014-06-09 16:22:11.310621, event: throttled, op: pg_info(1 pgs e243:3.1b1(2)) 2014-06-09 16:22:11.635011 7f869af78700 5 -- op tracker -- , seq: 2283, time: 2014-06-09 16:22:11.310630, event: all_read, op: pg_info(1 pgs e243:3.1b1(2)) 2014-06-09 16:22:11.635016 7f869af78700 5 -- op tracker -- , seq: 2283, time: 2014-06-09 16:22:11.634999, event: dispatched, op: pg_info(1 pgs e243:3.1b1(2)) 2014-06-09 16:22:11.635019 7f869af78700 5 -- op tracker -- , seq: 2283, time: 2014-06-09 16:22:11.635018, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1b1(2)) 2014-06-09 16:22:11.635022 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1b1(2)) v4 from osd.4 2014-06-09 16:22:11.635024 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4540 2014-06-09 16:22:11.635027 7f869af78700 5 -- op tracker -- , seq: 2283, time: 2014-06-09 16:22:11.635025, event: started, op: pg_info(1 pgs e243:3.1b1(2)) 2014-06-09 16:22:11.635035 7f869af78700 5 -- op tracker -- , seq: 2283, time: 2014-06-09 16:22:11.635034, event: done, op: pg_info(1 pgs e243:3.1b1(2)) 2014-06-09 16:22:11.635039 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635040 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635048 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635048 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635049 7f869af78700 20 osd.26 243 _dispatch 0x31053880 pg_info(1 pgs e243:1.4bc) v4 2014-06-09 16:22:11.635051 7f869af78700 5 -- op tracker -- , seq: 2284, time: 2014-06-09 16:22:11.380432, event: header_read, op: pg_info(1 pgs e243:1.4bc) 2014-06-09 16:22:11.635056 7f869af78700 5 -- op tracker -- , seq: 2284, time: 2014-06-09 16:22:11.380432, event: throttled, op: pg_info(1 pgs e243:1.4bc) 2014-06-09 16:22:11.635060 7f869af78700 5 -- op tracker -- , seq: 2284, time: 2014-06-09 16:22:11.380441, event: all_read, op: pg_info(1 pgs e243:1.4bc) 2014-06-09 16:22:11.635067 7f869af78700 5 -- op tracker -- , seq: 2284, time: 2014-06-09 16:22:11.635047, event: dispatched, op: pg_info(1 pgs e243:1.4bc) 2014-06-09 16:22:11.635070 7f869af78700 5 -- op tracker -- , seq: 2284, time: 2014-06-09 16:22:11.635069, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4bc) 2014-06-09 16:22:11.635076 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4bc) v4 from osd.7 2014-06-09 16:22:11.635077 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31053880 2014-06-09 16:22:11.635080 7f869af78700 5 -- op tracker -- , seq: 2284, time: 2014-06-09 16:22:11.635079, event: started, op: pg_info(1 pgs e243:1.4bc) 2014-06-09 16:22:11.635089 7f869af78700 5 -- op tracker -- , seq: 2284, time: 2014-06-09 16:22:11.635087, event: done, op: pg_info(1 pgs e243:1.4bc) 2014-06-09 16:22:11.635093 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635094 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635100 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635101 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635101 7f869af78700 20 osd.26 243 _dispatch 0xe0461c0 pg_info(1 pgs e243:2.6f4) v4 2014-06-09 16:22:11.635104 7f869af78700 5 -- op tracker -- , seq: 2285, time: 2014-06-09 16:22:11.436772, event: header_read, op: pg_info(1 pgs e243:2.6f4) 2014-06-09 16:22:11.635108 7f869af78700 5 -- op tracker -- , seq: 2285, time: 2014-06-09 16:22:11.436772, event: throttled, op: pg_info(1 pgs e243:2.6f4) 2014-06-09 16:22:11.635111 7f869af78700 5 -- op tracker -- , seq: 2285, time: 2014-06-09 16:22:11.436784, event: all_read, op: pg_info(1 pgs e243:2.6f4) 2014-06-09 16:22:11.635114 7f869af78700 5 -- op tracker -- , seq: 2285, time: 2014-06-09 16:22:11.635100, event: dispatched, op: pg_info(1 pgs e243:2.6f4) 2014-06-09 16:22:11.635117 7f869af78700 5 -- op tracker -- , seq: 2285, time: 2014-06-09 16:22:11.635116, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6f4) 2014-06-09 16:22:11.635123 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6f4) v4 from osd.3 2014-06-09 16:22:11.635125 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xe0461c0 2014-06-09 16:22:11.635127 7f869af78700 5 -- op tracker -- , seq: 2285, time: 2014-06-09 16:22:11.635126, event: started, op: pg_info(1 pgs e243:2.6f4) 2014-06-09 16:22:11.635135 7f869af78700 5 -- op tracker -- , seq: 2285, time: 2014-06-09 16:22:11.635134, event: done, op: pg_info(1 pgs e243:2.6f4) 2014-06-09 16:22:11.635139 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635147 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635148 7f869af78700 20 osd.26 243 _dispatch 0x3044cfc0 pg_info(1 pgs e243:2.401) v4 2014-06-09 16:22:11.635150 7f869af78700 5 -- op tracker -- , seq: 2286, time: 2014-06-09 16:22:11.311109, event: header_read, op: pg_info(1 pgs e243:2.401) 2014-06-09 16:22:11.635154 7f869af78700 5 -- op tracker -- , seq: 2286, time: 2014-06-09 16:22:11.311109, event: throttled, op: pg_info(1 pgs e243:2.401) 2014-06-09 16:22:11.635158 7f869af78700 5 -- op tracker -- , seq: 2286, time: 2014-06-09 16:22:11.311119, event: all_read, op: pg_info(1 pgs e243:2.401) 2014-06-09 16:22:11.635161 7f869af78700 5 -- op tracker -- , seq: 2286, time: 2014-06-09 16:22:11.635146, event: dispatched, op: pg_info(1 pgs e243:2.401) 2014-06-09 16:22:11.635164 7f869af78700 5 -- op tracker -- , seq: 2286, time: 2014-06-09 16:22:11.635163, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.401) 2014-06-09 16:22:11.635167 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.401) v4 from osd.1 2014-06-09 16:22:11.635168 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044cfc0 2014-06-09 16:22:11.635171 7f869af78700 5 -- op tracker -- , seq: 2286, time: 2014-06-09 16:22:11.635170, event: started, op: pg_info(1 pgs e243:2.401) 2014-06-09 16:22:11.635180 7f869af78700 5 -- op tracker -- , seq: 2286, time: 2014-06-09 16:22:11.635178, event: done, op: pg_info(1 pgs e243:2.401) 2014-06-09 16:22:11.635183 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635184 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635190 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635191 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635192 7f869af78700 20 osd.26 243 _dispatch 0x1ac94700 pg_info(1 pgs e243:3.65f) v4 2014-06-09 16:22:11.635194 7f869af78700 5 -- op tracker -- , seq: 2287, time: 2014-06-09 16:22:11.363728, event: header_read, op: pg_info(1 pgs e243:3.65f) 2014-06-09 16:22:11.635198 7f869af78700 5 -- op tracker -- , seq: 2287, time: 2014-06-09 16:22:11.363729, event: throttled, op: pg_info(1 pgs e243:3.65f) 2014-06-09 16:22:11.635201 7f869af78700 5 -- op tracker -- , seq: 2287, time: 2014-06-09 16:22:11.363739, event: all_read, op: pg_info(1 pgs e243:3.65f) 2014-06-09 16:22:11.635205 7f869af78700 5 -- op tracker -- , seq: 2287, time: 2014-06-09 16:22:11.635190, event: dispatched, op: pg_info(1 pgs e243:3.65f) 2014-06-09 16:22:11.635208 7f869af78700 5 -- op tracker -- , seq: 2287, time: 2014-06-09 16:22:11.635207, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.65f) 2014-06-09 16:22:11.635210 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.65f) v4 from osd.9 2014-06-09 16:22:11.635212 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac94700 2014-06-09 16:22:11.635214 7f869af78700 5 -- op tracker -- , seq: 2287, time: 2014-06-09 16:22:11.635213, event: started, op: pg_info(1 pgs e243:3.65f) 2014-06-09 16:22:11.635222 7f869af78700 5 -- op tracker -- , seq: 2287, time: 2014-06-09 16:22:11.635221, event: done, op: pg_info(1 pgs e243:3.65f) 2014-06-09 16:22:11.635226 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635227 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635234 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635234 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635235 7f869af78700 20 osd.26 243 _dispatch 0x3104b6c0 pg_info(1 pgs e243:2.5bf) v4 2014-06-09 16:22:11.635237 7f869af78700 5 -- op tracker -- , seq: 2288, time: 2014-06-09 16:22:11.374649, event: header_read, op: pg_info(1 pgs e243:2.5bf) 2014-06-09 16:22:11.635243 7f869af78700 5 -- op tracker -- , seq: 2288, time: 2014-06-09 16:22:11.374649, event: throttled, op: pg_info(1 pgs e243:2.5bf) 2014-06-09 16:22:11.635246 7f869af78700 5 -- op tracker -- , seq: 2288, time: 2014-06-09 16:22:11.374660, event: all_read, op: pg_info(1 pgs e243:2.5bf) 2014-06-09 16:22:11.635250 7f869af78700 5 -- op tracker -- , seq: 2288, time: 2014-06-09 16:22:11.635233, event: dispatched, op: pg_info(1 pgs e243:2.5bf) 2014-06-09 16:22:11.635253 7f869af78700 5 -- op tracker -- , seq: 2288, time: 2014-06-09 16:22:11.635252, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5bf) 2014-06-09 16:22:11.635256 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5bf) v4 from osd.8 2014-06-09 16:22:11.635257 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104b6c0 2014-06-09 16:22:11.635260 7f869af78700 5 -- op tracker -- , seq: 2288, time: 2014-06-09 16:22:11.635259, event: started, op: pg_info(1 pgs e243:2.5bf) 2014-06-09 16:22:11.635269 7f869af78700 5 -- op tracker -- , seq: 2288, time: 2014-06-09 16:22:11.635268, event: done, op: pg_info(1 pgs e243:2.5bf) 2014-06-09 16:22:11.635273 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635281 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635281 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635282 7f869af78700 20 osd.26 243 _dispatch 0x31040a80 pg_notify(2.6dd(20),3.224(10),0.230(20) epoch 243) v5 2014-06-09 16:22:11.635286 7f869af78700 5 -- op tracker -- , seq: 2289, time: 2014-06-09 16:22:11.380272, event: header_read, op: pg_notify(2.6dd(20),3.224(10),0.230(20) epoch 243) 2014-06-09 16:22:11.635290 7f869af78700 5 -- op tracker -- , seq: 2289, time: 2014-06-09 16:22:11.380272, event: throttled, op: pg_notify(2.6dd(20),3.224(10),0.230(20) epoch 243) 2014-06-09 16:22:11.635294 7f869af78700 5 -- op tracker -- , seq: 2289, time: 2014-06-09 16:22:11.380342, event: all_read, op: pg_notify(2.6dd(20),3.224(10),0.230(20) epoch 243) 2014-06-09 16:22:11.635297 7f869af78700 5 -- op tracker -- , seq: 2289, time: 2014-06-09 16:22:11.635281, event: dispatched, op: pg_notify(2.6dd(20),3.224(10),0.230(20) epoch 243) 2014-06-09 16:22:11.635302 7f869af78700 5 -- op tracker -- , seq: 2289, time: 2014-06-09 16:22:11.635301, event: waiting_for_osdmap, op: pg_notify(2.6dd(20),3.224(10),0.230(20) epoch 243) 2014-06-09 16:22:11.635305 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.635306 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040a80 2014-06-09 16:22:11.635309 7f869af78700 5 -- op tracker -- , seq: 2289, time: 2014-06-09 16:22:11.635307, event: started, op: pg_notify(2.6dd(20),3.224(10),0.230(20) epoch 243) 2014-06-09 16:22:11.635328 7f869af78700 5 -- op tracker -- , seq: 2289, time: 2014-06-09 16:22:11.635326, event: done, op: pg_notify(2.6dd(20),3.224(10),0.230(20) epoch 243) 2014-06-09 16:22:11.635332 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635333 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635349 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635349 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635350 7f869af78700 20 osd.26 243 _dispatch 0x19eb8e00 pg_info(1 pgs e243:3.2d5) v4 2014-06-09 16:22:11.635353 7f869af78700 5 -- op tracker -- , seq: 2290, time: 2014-06-09 16:22:11.283742, event: header_read, op: pg_info(1 pgs e243:3.2d5) 2014-06-09 16:22:11.635356 7f869af78700 5 -- op tracker -- , seq: 2290, time: 2014-06-09 16:22:11.283742, event: throttled, op: pg_info(1 pgs e243:3.2d5) 2014-06-09 16:22:11.635360 7f869af78700 5 -- op tracker -- , seq: 2290, time: 2014-06-09 16:22:11.283756, event: all_read, op: pg_info(1 pgs e243:3.2d5) 2014-06-09 16:22:11.635364 7f869af78700 5 -- op tracker -- , seq: 2290, time: 2014-06-09 16:22:11.635348, event: dispatched, op: pg_info(1 pgs e243:3.2d5) 2014-06-09 16:22:11.635368 7f869af78700 5 -- op tracker -- , seq: 2290, time: 2014-06-09 16:22:11.635367, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2d5) 2014-06-09 16:22:11.635373 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2d5) v4 from osd.0 2014-06-09 16:22:11.635375 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb8e00 2014-06-09 16:22:11.635378 7f869af78700 5 -- op tracker -- , seq: 2290, time: 2014-06-09 16:22:11.635377, event: started, op: pg_info(1 pgs e243:3.2d5) 2014-06-09 16:22:11.635387 7f869af78700 5 -- op tracker -- , seq: 2290, time: 2014-06-09 16:22:11.635386, event: done, op: pg_info(1 pgs e243:3.2d5) 2014-06-09 16:22:11.635391 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635392 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635399 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635400 7f869af78700 20 osd.26 243 _dispatch 0x310cac40 pg_info(1 pgs e243:3.701) v4 2014-06-09 16:22:11.635402 7f869af78700 5 -- op tracker -- , seq: 2291, time: 2014-06-09 16:22:11.382629, event: header_read, op: pg_info(1 pgs e243:3.701) 2014-06-09 16:22:11.635406 7f869af78700 5 -- op tracker -- , seq: 2291, time: 2014-06-09 16:22:11.382629, event: throttled, op: pg_info(1 pgs e243:3.701) 2014-06-09 16:22:11.635413 7f869af78700 5 -- op tracker -- , seq: 2291, time: 2014-06-09 16:22:11.382641, event: all_read, op: pg_info(1 pgs e243:3.701) 2014-06-09 16:22:11.635416 7f869af78700 5 -- op tracker -- , seq: 2291, time: 2014-06-09 16:22:11.635398, event: dispatched, op: pg_info(1 pgs e243:3.701) 2014-06-09 16:22:11.635421 7f869af78700 5 -- op tracker -- , seq: 2291, time: 2014-06-09 16:22:11.635420, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.701) 2014-06-09 16:22:11.635423 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.701) v4 from osd.5 2014-06-09 16:22:11.635425 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cac40 2014-06-09 16:22:11.635427 7f869af78700 5 -- op tracker -- , seq: 2291, time: 2014-06-09 16:22:11.635426, event: started, op: pg_info(1 pgs e243:3.701) 2014-06-09 16:22:11.635436 7f869af78700 5 -- op tracker -- , seq: 2291, time: 2014-06-09 16:22:11.635435, event: done, op: pg_info(1 pgs e243:3.701) 2014-06-09 16:22:11.635440 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635441 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635447 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635448 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635448 7f869af78700 20 osd.26 243 _dispatch 0x3005a1c0 pg_info(1 pgs e243:3.366(4)) v4 2014-06-09 16:22:11.635451 7f869af78700 5 -- op tracker -- , seq: 2292, time: 2014-06-09 16:22:11.245303, event: header_read, op: pg_info(1 pgs e243:3.366(4)) 2014-06-09 16:22:11.635455 7f869af78700 5 -- op tracker -- , seq: 2292, time: 2014-06-09 16:22:11.245304, event: throttled, op: pg_info(1 pgs e243:3.366(4)) 2014-06-09 16:22:11.635458 7f869af78700 5 -- op tracker -- , seq: 2292, time: 2014-06-09 16:22:11.245318, event: all_read, op: pg_info(1 pgs e243:3.366(4)) 2014-06-09 16:22:11.635461 7f869af78700 5 -- op tracker -- , seq: 2292, time: 2014-06-09 16:22:11.635447, event: dispatched, op: pg_info(1 pgs e243:3.366(4)) 2014-06-09 16:22:11.635465 7f869af78700 5 -- op tracker -- , seq: 2292, time: 2014-06-09 16:22:11.635464, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.366(4)) 2014-06-09 16:22:11.635467 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.366(4)) v4 from osd.2 2014-06-09 16:22:11.635469 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005a1c0 2014-06-09 16:22:11.635471 7f869af78700 5 -- op tracker -- , seq: 2292, time: 2014-06-09 16:22:11.635470, event: started, op: pg_info(1 pgs e243:3.366(4)) 2014-06-09 16:22:11.635483 7f869af78700 5 -- op tracker -- , seq: 2292, time: 2014-06-09 16:22:11.635482, event: done, op: pg_info(1 pgs e243:3.366(4)) 2014-06-09 16:22:11.635489 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635490 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635498 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635499 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635499 7f869af78700 20 osd.26 243 _dispatch 0x1c03efc0 pg_notify(1.779(20) epoch 243) v5 2014-06-09 16:22:11.635502 7f869af78700 5 -- op tracker -- , seq: 2293, time: 2014-06-09 16:22:11.310694, event: header_read, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.635506 7f869af78700 5 -- op tracker -- , seq: 2293, time: 2014-06-09 16:22:11.310694, event: throttled, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.635509 7f869af78700 5 -- op tracker -- , seq: 2293, time: 2014-06-09 16:22:11.310712, event: all_read, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.635516 7f869af78700 5 -- op tracker -- , seq: 2293, time: 2014-06-09 16:22:11.635498, event: dispatched, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.635520 7f869af78700 5 -- op tracker -- , seq: 2293, time: 2014-06-09 16:22:11.635518, event: waiting_for_osdmap, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.635523 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.635524 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03efc0 2014-06-09 16:22:11.635527 7f869af78700 5 -- op tracker -- , seq: 2293, time: 2014-06-09 16:22:11.635526, event: started, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.635537 7f869af78700 5 -- op tracker -- , seq: 2293, time: 2014-06-09 16:22:11.635536, event: done, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.635541 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635542 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635552 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635553 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635554 7f869af78700 20 osd.26 243 _dispatch 0x310536c0 pg_info(1 pgs e243:0.6b3) v4 2014-06-09 16:22:11.635556 7f869af78700 5 -- op tracker -- , seq: 2294, time: 2014-06-09 16:22:11.380446, event: header_read, op: pg_info(1 pgs e243:0.6b3) 2014-06-09 16:22:11.635560 7f869af78700 5 -- op tracker -- , seq: 2294, time: 2014-06-09 16:22:11.380447, event: throttled, op: pg_info(1 pgs e243:0.6b3) 2014-06-09 16:22:11.635563 7f869af78700 5 -- op tracker -- , seq: 2294, time: 2014-06-09 16:22:11.380456, event: all_read, op: pg_info(1 pgs e243:0.6b3) 2014-06-09 16:22:11.635567 7f869af78700 5 -- op tracker -- , seq: 2294, time: 2014-06-09 16:22:11.635552, event: dispatched, op: pg_info(1 pgs e243:0.6b3) 2014-06-09 16:22:11.635571 7f869af78700 5 -- op tracker -- , seq: 2294, time: 2014-06-09 16:22:11.635570, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6b3) 2014-06-09 16:22:11.635574 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6b3) v4 from osd.7 2014-06-09 16:22:11.635575 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310536c0 2014-06-09 16:22:11.635578 7f869af78700 5 -- op tracker -- , seq: 2294, time: 2014-06-09 16:22:11.635577, event: started, op: pg_info(1 pgs e243:0.6b3) 2014-06-09 16:22:11.635586 7f869af78700 5 -- op tracker -- , seq: 2294, time: 2014-06-09 16:22:11.635585, event: done, op: pg_info(1 pgs e243:0.6b3) 2014-06-09 16:22:11.635590 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635591 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635597 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.635598 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.635599 7f869af78700 20 osd.26 243 _dispatch 0x2eef7880 pg_info(1 pgs e243:0.44f) v4 2014-06-09 16:22:11.635601 7f869af78700 5 -- op tracker -- , seq: 2295, time: 2014-06-09 16:22:11.436792, event: header_read, op: pg_info(1 pgs e243:0.44f) 2014-06-09 16:22:11.635606 7f869af78700 5 -- op tracker -- , seq: 2295, time: 2014-06-09 16:22:11.436792, event: throttled, op: pg_info(1 pgs e243:0.44f) 2014-06-09 16:22:11.637117 7f869af78700 5 -- op tracker -- , seq: 2295, time: 2014-06-09 16:22:11.436804, event: all_read, op: pg_info(1 pgs e243:0.44f) 2014-06-09 16:22:11.637121 7f869af78700 5 -- op tracker -- , seq: 2295, time: 2014-06-09 16:22:11.635597, event: dispatched, op: pg_info(1 pgs e243:0.44f) 2014-06-09 16:22:11.637125 7f869af78700 5 -- op tracker -- , seq: 2295, time: 2014-06-09 16:22:11.637123, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.44f) 2014-06-09 16:22:11.637127 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.44f) v4 from osd.3 2014-06-09 16:22:11.637129 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef7880 2014-06-09 16:22:11.637131 7f869af78700 5 -- op tracker -- , seq: 2295, time: 2014-06-09 16:22:11.637130, event: started, op: pg_info(1 pgs e243:0.44f) 2014-06-09 16:22:11.637140 7f869af78700 5 -- op tracker -- , seq: 2295, time: 2014-06-09 16:22:11.637139, event: done, op: pg_info(1 pgs e243:0.44f) 2014-06-09 16:22:11.637144 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637145 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637152 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637153 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637153 7f869af78700 20 osd.26 243 _dispatch 0x230ba700 pg_info(1 pgs e243:2.3bd) v4 2014-06-09 16:22:11.637156 7f869af78700 5 -- op tracker -- , seq: 2296, time: 2014-06-09 16:22:11.311146, event: header_read, op: pg_info(1 pgs e243:2.3bd) 2014-06-09 16:22:11.637162 7f869af78700 5 -- op tracker -- , seq: 2296, time: 2014-06-09 16:22:11.311146, event: throttled, op: pg_info(1 pgs e243:2.3bd) 2014-06-09 16:22:11.637165 7f869af78700 5 -- op tracker -- , seq: 2296, time: 2014-06-09 16:22:11.311155, event: all_read, op: pg_info(1 pgs e243:2.3bd) 2014-06-09 16:22:11.637170 7f869af78700 5 -- op tracker -- , seq: 2296, time: 2014-06-09 16:22:11.637152, event: dispatched, op: pg_info(1 pgs e243:2.3bd) 2014-06-09 16:22:11.637173 7f869af78700 5 -- op tracker -- , seq: 2296, time: 2014-06-09 16:22:11.637172, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3bd) 2014-06-09 16:22:11.637176 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3bd) v4 from osd.1 2014-06-09 16:22:11.637177 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230ba700 2014-06-09 16:22:11.637180 7f869af78700 5 -- op tracker -- , seq: 2296, time: 2014-06-09 16:22:11.637179, event: started, op: pg_info(1 pgs e243:2.3bd) 2014-06-09 16:22:11.637188 7f869af78700 5 -- op tracker -- , seq: 2296, time: 2014-06-09 16:22:11.637187, event: done, op: pg_info(1 pgs e243:2.3bd) 2014-06-09 16:22:11.637192 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637193 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637199 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637200 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637200 7f869af78700 20 osd.26 243 _dispatch 0x1ac948c0 pg_info(1 pgs e243:3.ce) v4 2014-06-09 16:22:11.637203 7f869af78700 5 -- op tracker -- , seq: 2297, time: 2014-06-09 16:22:11.363744, event: header_read, op: pg_info(1 pgs e243:3.ce) 2014-06-09 16:22:11.637208 7f869af78700 5 -- op tracker -- , seq: 2297, time: 2014-06-09 16:22:11.363744, event: throttled, op: pg_info(1 pgs e243:3.ce) 2014-06-09 16:22:11.637211 7f869af78700 5 -- op tracker -- , seq: 2297, time: 2014-06-09 16:22:11.363758, event: all_read, op: pg_info(1 pgs e243:3.ce) 2014-06-09 16:22:11.637220 7f869af78700 5 -- op tracker -- , seq: 2297, time: 2014-06-09 16:22:11.637199, event: dispatched, op: pg_info(1 pgs e243:3.ce) 2014-06-09 16:22:11.637224 7f869af78700 5 -- op tracker -- , seq: 2297, time: 2014-06-09 16:22:11.637223, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.ce) 2014-06-09 16:22:11.637231 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.ce) v4 from osd.9 2014-06-09 16:22:11.637232 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac948c0 2014-06-09 16:22:11.637235 7f869af78700 5 -- op tracker -- , seq: 2297, time: 2014-06-09 16:22:11.637234, event: started, op: pg_info(1 pgs e243:3.ce) 2014-06-09 16:22:11.637247 7f869af78700 5 -- op tracker -- , seq: 2297, time: 2014-06-09 16:22:11.637246, event: done, op: pg_info(1 pgs e243:3.ce) 2014-06-09 16:22:11.637251 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637252 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637258 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637259 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637260 7f869af78700 20 osd.26 243 _dispatch 0x3104b500 pg_info(1 pgs e243:0.164) v4 2014-06-09 16:22:11.637262 7f869af78700 5 -- op tracker -- , seq: 2298, time: 2014-06-09 16:22:11.374698, event: header_read, op: pg_info(1 pgs e243:0.164) 2014-06-09 16:22:11.637267 7f869af78700 5 -- op tracker -- , seq: 2298, time: 2014-06-09 16:22:11.374699, event: throttled, op: pg_info(1 pgs e243:0.164) 2014-06-09 16:22:11.637271 7f869af78700 5 -- op tracker -- , seq: 2298, time: 2014-06-09 16:22:11.374711, event: all_read, op: pg_info(1 pgs e243:0.164) 2014-06-09 16:22:11.637275 7f869af78700 5 -- op tracker -- , seq: 2298, time: 2014-06-09 16:22:11.637258, event: dispatched, op: pg_info(1 pgs e243:0.164) 2014-06-09 16:22:11.637278 7f869af78700 5 -- op tracker -- , seq: 2298, time: 2014-06-09 16:22:11.637277, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.164) 2014-06-09 16:22:11.637280 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.164) v4 from osd.8 2014-06-09 16:22:11.637282 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104b500 2014-06-09 16:22:11.637284 7f869af78700 5 -- op tracker -- , seq: 2298, time: 2014-06-09 16:22:11.637283, event: started, op: pg_info(1 pgs e243:0.164) 2014-06-09 16:22:11.637294 7f869af78700 5 -- op tracker -- , seq: 2298, time: 2014-06-09 16:22:11.637292, event: done, op: pg_info(1 pgs e243:0.164) 2014-06-09 16:22:11.637298 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637299 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637305 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637306 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637307 7f869af78700 20 osd.26 243 _dispatch 0x310408c0 pg_notify(3.428(10) epoch 243) v5 2014-06-09 16:22:11.637309 7f869af78700 5 -- op tracker -- , seq: 2299, time: 2014-06-09 16:22:11.384770, event: header_read, op: pg_notify(3.428(10) epoch 243) 2014-06-09 16:22:11.637314 7f869af78700 5 -- op tracker -- , seq: 2299, time: 2014-06-09 16:22:11.384771, event: throttled, op: pg_notify(3.428(10) epoch 243) 2014-06-09 16:22:11.637322 7f869af78700 5 -- op tracker -- , seq: 2299, time: 2014-06-09 16:22:11.384794, event: all_read, op: pg_notify(3.428(10) epoch 243) 2014-06-09 16:22:11.637326 7f869af78700 5 -- op tracker -- , seq: 2299, time: 2014-06-09 16:22:11.637305, event: dispatched, op: pg_notify(3.428(10) epoch 243) 2014-06-09 16:22:11.637329 7f869af78700 5 -- op tracker -- , seq: 2299, time: 2014-06-09 16:22:11.637328, event: waiting_for_osdmap, op: pg_notify(3.428(10) epoch 243) 2014-06-09 16:22:11.637331 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.637333 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310408c0 2014-06-09 16:22:11.637335 7f869af78700 5 -- op tracker -- , seq: 2299, time: 2014-06-09 16:22:11.637334, event: started, op: pg_notify(3.428(10) epoch 243) 2014-06-09 16:22:11.637346 7f869af78700 5 -- op tracker -- , seq: 2299, time: 2014-06-09 16:22:11.637345, event: done, op: pg_notify(3.428(10) epoch 243) 2014-06-09 16:22:11.637350 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637351 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637361 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637361 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637362 7f869af78700 20 osd.26 243 _dispatch 0x307e7340 pg_info(1 pgs e243:0.479) v4 2014-06-09 16:22:11.637365 7f869af78700 5 -- op tracker -- , seq: 2300, time: 2014-06-09 16:22:11.290707, event: header_read, op: pg_info(1 pgs e243:0.479) 2014-06-09 16:22:11.637369 7f869af78700 5 -- op tracker -- , seq: 2300, time: 2014-06-09 16:22:11.290707, event: throttled, op: pg_info(1 pgs e243:0.479) 2014-06-09 16:22:11.637377 7f869af78700 5 -- op tracker -- , seq: 2300, time: 2014-06-09 16:22:11.290721, event: all_read, op: pg_info(1 pgs e243:0.479) 2014-06-09 16:22:11.637381 7f869af78700 5 -- op tracker -- , seq: 2300, time: 2014-06-09 16:22:11.637360, event: dispatched, op: pg_info(1 pgs e243:0.479) 2014-06-09 16:22:11.637386 7f869af78700 5 -- op tracker -- , seq: 2300, time: 2014-06-09 16:22:11.637385, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.479) 2014-06-09 16:22:11.637388 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.479) v4 from osd.0 2014-06-09 16:22:11.637390 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e7340 2014-06-09 16:22:11.637392 7f869af78700 5 -- op tracker -- , seq: 2300, time: 2014-06-09 16:22:11.637391, event: started, op: pg_info(1 pgs e243:0.479) 2014-06-09 16:22:11.637400 7f869af78700 5 -- op tracker -- , seq: 2300, time: 2014-06-09 16:22:11.637399, event: done, op: pg_info(1 pgs e243:0.479) 2014-06-09 16:22:11.637405 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637406 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637412 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637413 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637414 7f869af78700 20 osd.26 243 _dispatch 0x310caa80 pg_info(1 pgs e243:2.3fa) v4 2014-06-09 16:22:11.637416 7f869af78700 5 -- op tracker -- , seq: 2301, time: 2014-06-09 16:22:11.382649, event: header_read, op: pg_info(1 pgs e243:2.3fa) 2014-06-09 16:22:11.637421 7f869af78700 5 -- op tracker -- , seq: 2301, time: 2014-06-09 16:22:11.382649, event: throttled, op: pg_info(1 pgs e243:2.3fa) 2014-06-09 16:22:11.637425 7f869af78700 5 -- op tracker -- , seq: 2301, time: 2014-06-09 16:22:11.382660, event: all_read, op: pg_info(1 pgs e243:2.3fa) 2014-06-09 16:22:11.637428 7f869af78700 5 -- op tracker -- , seq: 2301, time: 2014-06-09 16:22:11.637412, event: dispatched, op: pg_info(1 pgs e243:2.3fa) 2014-06-09 16:22:11.637432 7f869af78700 5 -- op tracker -- , seq: 2301, time: 2014-06-09 16:22:11.637431, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3fa) 2014-06-09 16:22:11.637434 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3fa) v4 from osd.5 2014-06-09 16:22:11.637436 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310caa80 2014-06-09 16:22:11.637438 7f869af78700 5 -- op tracker -- , seq: 2301, time: 2014-06-09 16:22:11.637437, event: started, op: pg_info(1 pgs e243:2.3fa) 2014-06-09 16:22:11.637446 7f869af78700 5 -- op tracker -- , seq: 2301, time: 2014-06-09 16:22:11.637445, event: done, op: pg_info(1 pgs e243:2.3fa) 2014-06-09 16:22:11.637450 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637451 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637458 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637458 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637459 7f869af78700 20 osd.26 243 _dispatch 0x30640000 pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.637463 7f869af78700 5 -- op tracker -- , seq: 2302, time: 2014-06-09 16:22:11.245846, event: header_read, op: pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637467 7f869af78700 5 -- op tracker -- , seq: 2302, time: 2014-06-09 16:22:11.245847, event: throttled, op: pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637470 7f869af78700 5 -- op tracker -- , seq: 2302, time: 2014-06-09 16:22:11.245855, event: all_read, op: pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637474 7f869af78700 5 -- op tracker -- , seq: 2302, time: 2014-06-09 16:22:11.637458, event: dispatched, op: pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637483 7f869af78700 5 -- op tracker -- , seq: 2302, time: 2014-06-09 16:22:11.637482, event: waiting_for_osdmap, op: pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637486 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.637488 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30640000 2014-06-09 16:22:11.637491 7f869af78700 5 -- op tracker -- , seq: 2302, time: 2014-06-09 16:22:11.637490, event: started, op: pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637499 7f869af78700 5 -- op tracker -- , seq: 2302, time: 2014-06-09 16:22:11.637498, event: done, op: pg_log(0.763 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637508 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637509 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637516 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637516 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637517 7f869af78700 20 osd.26 243 _dispatch 0x30880600 pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.637520 7f869af78700 5 -- op tracker -- , seq: 2303, time: 2014-06-09 16:22:11.310723, event: header_read, op: pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637525 7f869af78700 5 -- op tracker -- , seq: 2303, time: 2014-06-09 16:22:11.310723, event: throttled, op: pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637528 7f869af78700 5 -- op tracker -- , seq: 2303, time: 2014-06-09 16:22:11.310732, event: all_read, op: pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637532 7f869af78700 5 -- op tracker -- , seq: 2303, time: 2014-06-09 16:22:11.637515, event: dispatched, op: pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637536 7f869af78700 5 -- op tracker -- , seq: 2303, time: 2014-06-09 16:22:11.637535, event: waiting_for_osdmap, op: pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637542 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.637544 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30880600 2014-06-09 16:22:11.637548 7f869af78700 5 -- op tracker -- , seq: 2303, time: 2014-06-09 16:22:11.637546, event: started, op: pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637555 7f869af78700 5 -- op tracker -- , seq: 2303, time: 2014-06-09 16:22:11.637553, event: done, op: pg_log(1.4c9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637559 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637560 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637566 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637567 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637567 7f869af78700 20 osd.26 243 _dispatch 0x31053500 pg_info(1 pgs e243:3.2e7) v4 2014-06-09 16:22:11.637570 7f869af78700 5 -- op tracker -- , seq: 2304, time: 2014-06-09 16:22:11.380460, event: header_read, op: pg_info(1 pgs e243:3.2e7) 2014-06-09 16:22:11.637577 7f869af78700 5 -- op tracker -- , seq: 2304, time: 2014-06-09 16:22:11.380460, event: throttled, op: pg_info(1 pgs e243:3.2e7) 2014-06-09 16:22:11.637581 7f869af78700 5 -- op tracker -- , seq: 2304, time: 2014-06-09 16:22:11.380470, event: all_read, op: pg_info(1 pgs e243:3.2e7) 2014-06-09 16:22:11.637585 7f869af78700 5 -- op tracker -- , seq: 2304, time: 2014-06-09 16:22:11.637566, event: dispatched, op: pg_info(1 pgs e243:3.2e7) 2014-06-09 16:22:11.637589 7f869af78700 5 -- op tracker -- , seq: 2304, time: 2014-06-09 16:22:11.637588, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2e7) 2014-06-09 16:22:11.637592 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2e7) v4 from osd.7 2014-06-09 16:22:11.637594 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31053500 2014-06-09 16:22:11.637596 7f869af78700 5 -- op tracker -- , seq: 2304, time: 2014-06-09 16:22:11.637595, event: started, op: pg_info(1 pgs e243:3.2e7) 2014-06-09 16:22:11.637606 7f869af78700 5 -- op tracker -- , seq: 2304, time: 2014-06-09 16:22:11.637605, event: done, op: pg_info(1 pgs e243:3.2e7) 2014-06-09 16:22:11.637611 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637611 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637618 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637619 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637620 7f869af78700 20 osd.26 243 _dispatch 0x2eef76c0 pg_info(1 pgs e243:3.206) v4 2014-06-09 16:22:11.637622 7f869af78700 5 -- op tracker -- , seq: 2305, time: 2014-06-09 16:22:11.436842, event: header_read, op: pg_info(1 pgs e243:3.206) 2014-06-09 16:22:11.637630 7f869af78700 5 -- op tracker -- , seq: 2305, time: 2014-06-09 16:22:11.436843, event: throttled, op: pg_info(1 pgs e243:3.206) 2014-06-09 16:22:11.637633 7f869af78700 5 -- op tracker -- , seq: 2305, time: 2014-06-09 16:22:11.436854, event: all_read, op: pg_info(1 pgs e243:3.206) 2014-06-09 16:22:11.637636 7f869af78700 5 -- op tracker -- , seq: 2305, time: 2014-06-09 16:22:11.637618, event: dispatched, op: pg_info(1 pgs e243:3.206) 2014-06-09 16:22:11.637640 7f869af78700 5 -- op tracker -- , seq: 2305, time: 2014-06-09 16:22:11.637639, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.206) 2014-06-09 16:22:11.637642 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.206) v4 from osd.3 2014-06-09 16:22:11.637644 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef76c0 2014-06-09 16:22:11.637646 7f869af78700 5 -- op tracker -- , seq: 2305, time: 2014-06-09 16:22:11.637645, event: started, op: pg_info(1 pgs e243:3.206) 2014-06-09 16:22:11.637654 7f869af78700 5 -- op tracker -- , seq: 2305, time: 2014-06-09 16:22:11.637653, event: done, op: pg_info(1 pgs e243:3.206) 2014-06-09 16:22:11.637658 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637659 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637666 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637667 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637667 7f869af78700 20 osd.26 243 _dispatch 0x307e7180 pg_info(1 pgs e243:2.2da) v4 2014-06-09 16:22:11.637670 7f869af78700 5 -- op tracker -- , seq: 2306, time: 2014-06-09 16:22:11.311161, event: header_read, op: pg_info(1 pgs e243:2.2da) 2014-06-09 16:22:11.637675 7f869af78700 5 -- op tracker -- , seq: 2306, time: 2014-06-09 16:22:11.311161, event: throttled, op: pg_info(1 pgs e243:2.2da) 2014-06-09 16:22:11.637678 7f869af78700 5 -- op tracker -- , seq: 2306, time: 2014-06-09 16:22:11.311171, event: all_read, op: pg_info(1 pgs e243:2.2da) 2014-06-09 16:22:11.637683 7f869af78700 5 -- op tracker -- , seq: 2306, time: 2014-06-09 16:22:11.637666, event: dispatched, op: pg_info(1 pgs e243:2.2da) 2014-06-09 16:22:11.637686 7f869af78700 5 -- op tracker -- , seq: 2306, time: 2014-06-09 16:22:11.637685, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2da) 2014-06-09 16:22:11.637692 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2da) v4 from osd.1 2014-06-09 16:22:11.637693 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e7180 2014-06-09 16:22:11.637696 7f869af78700 5 -- op tracker -- , seq: 2306, time: 2014-06-09 16:22:11.637695, event: started, op: pg_info(1 pgs e243:2.2da) 2014-06-09 16:22:11.637704 7f869af78700 5 -- op tracker -- , seq: 2306, time: 2014-06-09 16:22:11.637703, event: done, op: pg_info(1 pgs e243:2.2da) 2014-06-09 16:22:11.637711 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637712 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637719 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637720 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637720 7f869af78700 20 osd.26 243 _dispatch 0x150e1dc0 pg_info(1 pgs e243:3.44d) v4 2014-06-09 16:22:11.637723 7f869af78700 5 -- op tracker -- , seq: 2307, time: 2014-06-09 16:22:11.363763, event: header_read, op: pg_info(1 pgs e243:3.44d) 2014-06-09 16:22:11.637727 7f869af78700 5 -- op tracker -- , seq: 2307, time: 2014-06-09 16:22:11.363763, event: throttled, op: pg_info(1 pgs e243:3.44d) 2014-06-09 16:22:11.637730 7f869af78700 5 -- op tracker -- , seq: 2307, time: 2014-06-09 16:22:11.363773, event: all_read, op: pg_info(1 pgs e243:3.44d) 2014-06-09 16:22:11.637733 7f869af78700 5 -- op tracker -- , seq: 2307, time: 2014-06-09 16:22:11.637719, event: dispatched, op: pg_info(1 pgs e243:3.44d) 2014-06-09 16:22:11.637740 7f869af78700 5 -- op tracker -- , seq: 2307, time: 2014-06-09 16:22:11.637739, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.44d) 2014-06-09 16:22:11.637743 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.44d) v4 from osd.9 2014-06-09 16:22:11.637745 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e1dc0 2014-06-09 16:22:11.637747 7f869af78700 5 -- op tracker -- , seq: 2307, time: 2014-06-09 16:22:11.637746, event: started, op: pg_info(1 pgs e243:3.44d) 2014-06-09 16:22:11.637756 7f869af78700 5 -- op tracker -- , seq: 2307, time: 2014-06-09 16:22:11.637755, event: done, op: pg_info(1 pgs e243:3.44d) 2014-06-09 16:22:11.637760 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637761 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637768 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637769 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637769 7f869af78700 20 osd.26 243 _dispatch 0x3104b340 pg_info(1 pgs e243:3.3b8) v4 2014-06-09 16:22:11.637772 7f869af78700 5 -- op tracker -- , seq: 2308, time: 2014-06-09 16:22:11.374719, event: header_read, op: pg_info(1 pgs e243:3.3b8) 2014-06-09 16:22:11.637779 7f869af78700 5 -- op tracker -- , seq: 2308, time: 2014-06-09 16:22:11.374719, event: throttled, op: pg_info(1 pgs e243:3.3b8) 2014-06-09 16:22:11.637782 7f869af78700 5 -- op tracker -- , seq: 2308, time: 2014-06-09 16:22:11.374737, event: all_read, op: pg_info(1 pgs e243:3.3b8) 2014-06-09 16:22:11.637785 7f869af78700 5 -- op tracker -- , seq: 2308, time: 2014-06-09 16:22:11.637768, event: dispatched, op: pg_info(1 pgs e243:3.3b8) 2014-06-09 16:22:11.637789 7f869af78700 5 -- op tracker -- , seq: 2308, time: 2014-06-09 16:22:11.637788, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3b8) 2014-06-09 16:22:11.637791 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3b8) v4 from osd.8 2014-06-09 16:22:11.637792 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104b340 2014-06-09 16:22:11.637795 7f869af78700 5 -- op tracker -- , seq: 2308, time: 2014-06-09 16:22:11.637794, event: started, op: pg_info(1 pgs e243:3.3b8) 2014-06-09 16:22:11.637805 7f869af78700 5 -- op tracker -- , seq: 2308, time: 2014-06-09 16:22:11.637804, event: done, op: pg_info(1 pgs e243:3.3b8) 2014-06-09 16:22:11.637809 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637810 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637818 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637819 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637819 7f869af78700 20 osd.26 243 _dispatch 0x31040700 pg_notify(1.700(23) epoch 243) v5 2014-06-09 16:22:11.637822 7f869af78700 5 -- op tracker -- , seq: 2309, time: 2014-06-09 16:22:11.388317, event: header_read, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.637826 7f869af78700 5 -- op tracker -- , seq: 2309, time: 2014-06-09 16:22:11.388318, event: throttled, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.637829 7f869af78700 5 -- op tracker -- , seq: 2309, time: 2014-06-09 16:22:11.388357, event: all_read, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.637835 7f869af78700 5 -- op tracker -- , seq: 2309, time: 2014-06-09 16:22:11.637818, event: dispatched, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.637839 7f869af78700 5 -- op tracker -- , seq: 2309, time: 2014-06-09 16:22:11.637838, event: waiting_for_osdmap, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.637843 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.637844 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040700 2014-06-09 16:22:11.637847 7f869af78700 5 -- op tracker -- , seq: 2309, time: 2014-06-09 16:22:11.637846, event: started, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.637855 7f869af78700 5 -- op tracker -- , seq: 2309, time: 2014-06-09 16:22:11.637854, event: done, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.637859 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637860 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637871 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637872 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637872 7f869af78700 20 osd.26 243 _dispatch 0x30880000 pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.637876 7f869af78700 5 -- op tracker -- , seq: 2310, time: 2014-06-09 16:22:11.304593, event: header_read, op: pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637880 7f869af78700 5 -- op tracker -- , seq: 2310, time: 2014-06-09 16:22:11.304594, event: throttled, op: pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637884 7f869af78700 5 -- op tracker -- , seq: 2310, time: 2014-06-09 16:22:11.305482, event: all_read, op: pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637888 7f869af78700 5 -- op tracker -- , seq: 2310, time: 2014-06-09 16:22:11.637871, event: dispatched, op: pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637892 7f869af78700 5 -- op tracker -- , seq: 2310, time: 2014-06-09 16:22:11.637890, event: waiting_for_osdmap, op: pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637894 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) v4 from osd.0 2014-06-09 16:22:11.637896 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30880000 2014-06-09 16:22:11.637899 7f869af78700 5 -- op tracker -- , seq: 2310, time: 2014-06-09 16:22:11.637897, event: started, op: pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637906 7f869af78700 5 -- op tracker -- , seq: 2310, time: 2014-06-09 16:22:11.637905, event: done, op: pg_log(3.74f epoch 243 log log((0'0,213'530], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.637917 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637918 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637925 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637926 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637926 7f869af78700 20 osd.26 243 _dispatch 0x310ca8c0 pg_notify(0.18b(23),3.18d(10) epoch 243) v5 2014-06-09 16:22:11.637929 7f869af78700 5 -- op tracker -- , seq: 2311, time: 2014-06-09 16:22:11.387639, event: header_read, op: pg_notify(0.18b(23),3.18d(10) epoch 243) 2014-06-09 16:22:11.637933 7f869af78700 5 -- op tracker -- , seq: 2311, time: 2014-06-09 16:22:11.387640, event: throttled, op: pg_notify(0.18b(23),3.18d(10) epoch 243) 2014-06-09 16:22:11.637937 7f869af78700 5 -- op tracker -- , seq: 2311, time: 2014-06-09 16:22:11.387690, event: all_read, op: pg_notify(0.18b(23),3.18d(10) epoch 243) 2014-06-09 16:22:11.637940 7f869af78700 5 -- op tracker -- , seq: 2311, time: 2014-06-09 16:22:11.637925, event: dispatched, op: pg_notify(0.18b(23),3.18d(10) epoch 243) 2014-06-09 16:22:11.637944 7f869af78700 5 -- op tracker -- , seq: 2311, time: 2014-06-09 16:22:11.637943, event: waiting_for_osdmap, op: pg_notify(0.18b(23),3.18d(10) epoch 243) 2014-06-09 16:22:11.637946 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.637947 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310ca8c0 2014-06-09 16:22:11.637950 7f869af78700 5 -- op tracker -- , seq: 2311, time: 2014-06-09 16:22:11.637949, event: started, op: pg_notify(0.18b(23),3.18d(10) epoch 243) 2014-06-09 16:22:11.637964 7f869af78700 5 -- op tracker -- , seq: 2311, time: 2014-06-09 16:22:11.637962, event: done, op: pg_notify(0.18b(23),3.18d(10) epoch 243) 2014-06-09 16:22:11.637968 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637969 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637981 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.637982 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.637982 7f869af78700 20 osd.26 243 _dispatch 0x3005b500 pg_notify(2.740(22) epoch 243) v5 2014-06-09 16:22:11.637985 7f869af78700 5 -- op tracker -- , seq: 2312, time: 2014-06-09 16:22:11.251873, event: header_read, op: pg_notify(2.740(22) epoch 243) 2014-06-09 16:22:11.637989 7f869af78700 5 -- op tracker -- , seq: 2312, time: 2014-06-09 16:22:11.251873, event: throttled, op: pg_notify(2.740(22) epoch 243) 2014-06-09 16:22:11.637992 7f869af78700 5 -- op tracker -- , seq: 2312, time: 2014-06-09 16:22:11.251897, event: all_read, op: pg_notify(2.740(22) epoch 243) 2014-06-09 16:22:11.637995 7f869af78700 5 -- op tracker -- , seq: 2312, time: 2014-06-09 16:22:11.637981, event: dispatched, op: pg_notify(2.740(22) epoch 243) 2014-06-09 16:22:11.638003 7f869af78700 5 -- op tracker -- , seq: 2312, time: 2014-06-09 16:22:11.638002, event: waiting_for_osdmap, op: pg_notify(2.740(22) epoch 243) 2014-06-09 16:22:11.638006 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.638007 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005b500 2014-06-09 16:22:11.638009 7f869af78700 5 -- op tracker -- , seq: 2312, time: 2014-06-09 16:22:11.638008, event: started, op: pg_notify(2.740(22) epoch 243) 2014-06-09 16:22:11.638018 7f869af78700 5 -- op tracker -- , seq: 2312, time: 2014-06-09 16:22:11.638017, event: done, op: pg_notify(2.740(22) epoch 243) 2014-06-09 16:22:11.638023 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638024 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638035 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638036 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638036 7f869af78700 20 osd.26 243 _dispatch 0x30951200 pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.638040 7f869af78700 5 -- op tracker -- , seq: 2313, time: 2014-06-09 16:22:11.310760, event: header_read, op: pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.638044 7f869af78700 5 -- op tracker -- , seq: 2313, time: 2014-06-09 16:22:11.310760, event: throttled, op: pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.638048 7f869af78700 5 -- op tracker -- , seq: 2313, time: 2014-06-09 16:22:11.310767, event: all_read, op: pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.638051 7f869af78700 5 -- op tracker -- , seq: 2313, time: 2014-06-09 16:22:11.638035, event: dispatched, op: pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.638059 7f869af78700 5 -- op tracker -- , seq: 2313, time: 2014-06-09 16:22:11.638057, event: waiting_for_osdmap, op: pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.638061 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.638063 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30951200 2014-06-09 16:22:11.638066 7f869af78700 5 -- op tracker -- , seq: 2313, time: 2014-06-09 16:22:11.638065, event: started, op: pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.638073 7f869af78700 5 -- op tracker -- , seq: 2313, time: 2014-06-09 16:22:11.638071, event: done, op: pg_log(1.54a epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.638077 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638084 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638085 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638085 7f869af78700 20 osd.26 243 _dispatch 0x31053340 pg_info(1 pgs e243:3.0) v4 2014-06-09 16:22:11.638088 7f869af78700 5 -- op tracker -- , seq: 2314, time: 2014-06-09 16:22:11.380474, event: header_read, op: pg_info(1 pgs e243:3.0) 2014-06-09 16:22:11.638092 7f869af78700 5 -- op tracker -- , seq: 2314, time: 2014-06-09 16:22:11.380474, event: throttled, op: pg_info(1 pgs e243:3.0) 2014-06-09 16:22:11.638098 7f869af78700 5 -- op tracker -- , seq: 2314, time: 2014-06-09 16:22:11.380485, event: all_read, op: pg_info(1 pgs e243:3.0) 2014-06-09 16:22:11.638105 7f869af78700 5 -- op tracker -- , seq: 2314, time: 2014-06-09 16:22:11.638084, event: dispatched, op: pg_info(1 pgs e243:3.0) 2014-06-09 16:22:11.638108 7f869af78700 5 -- op tracker -- , seq: 2314, time: 2014-06-09 16:22:11.638107, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.0) 2014-06-09 16:22:11.638112 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.0) v4 from osd.7 2014-06-09 16:22:11.638114 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31053340 2014-06-09 16:22:11.638116 7f869af78700 5 -- op tracker -- , seq: 2314, time: 2014-06-09 16:22:11.638115, event: started, op: pg_info(1 pgs e243:3.0) 2014-06-09 16:22:11.638126 7f869af78700 5 -- op tracker -- , seq: 2314, time: 2014-06-09 16:22:11.638125, event: done, op: pg_info(1 pgs e243:3.0) 2014-06-09 16:22:11.638130 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638131 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638138 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638138 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638139 7f869af78700 20 osd.26 243 _dispatch 0x2eef7500 pg_info(1 pgs e243:0.718) v4 2014-06-09 16:22:11.638142 7f869af78700 5 -- op tracker -- , seq: 2315, time: 2014-06-09 16:22:11.436861, event: header_read, op: pg_info(1 pgs e243:0.718) 2014-06-09 16:22:11.638145 7f869af78700 5 -- op tracker -- , seq: 2315, time: 2014-06-09 16:22:11.436862, event: throttled, op: pg_info(1 pgs e243:0.718) 2014-06-09 16:22:11.638149 7f869af78700 5 -- op tracker -- , seq: 2315, time: 2014-06-09 16:22:11.436872, event: all_read, op: pg_info(1 pgs e243:0.718) 2014-06-09 16:22:11.638152 7f869af78700 5 -- op tracker -- , seq: 2315, time: 2014-06-09 16:22:11.638137, event: dispatched, op: pg_info(1 pgs e243:0.718) 2014-06-09 16:22:11.638155 7f869af78700 5 -- op tracker -- , seq: 2315, time: 2014-06-09 16:22:11.638154, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.718) 2014-06-09 16:22:11.638157 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.718) v4 from osd.3 2014-06-09 16:22:11.638159 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef7500 2014-06-09 16:22:11.638161 7f869af78700 5 -- op tracker -- , seq: 2315, time: 2014-06-09 16:22:11.638160, event: started, op: pg_info(1 pgs e243:0.718) 2014-06-09 16:22:11.638171 7f869af78700 5 -- op tracker -- , seq: 2315, time: 2014-06-09 16:22:11.638170, event: done, op: pg_info(1 pgs e243:0.718) 2014-06-09 16:22:11.638176 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638177 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638183 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638184 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638185 7f869af78700 20 osd.26 243 _dispatch 0x3088ca80 pg_info(1 pgs e243:2.3bc) v4 2014-06-09 16:22:11.638187 7f869af78700 5 -- op tracker -- , seq: 2316, time: 2014-06-09 16:22:11.311183, event: header_read, op: pg_info(1 pgs e243:2.3bc) 2014-06-09 16:22:11.638194 7f869af78700 5 -- op tracker -- , seq: 2316, time: 2014-06-09 16:22:11.311183, event: throttled, op: pg_info(1 pgs e243:2.3bc) 2014-06-09 16:22:11.638198 7f869af78700 5 -- op tracker -- , seq: 2316, time: 2014-06-09 16:22:11.311191, event: all_read, op: pg_info(1 pgs e243:2.3bc) 2014-06-09 16:22:11.638202 7f869af78700 5 -- op tracker -- , seq: 2316, time: 2014-06-09 16:22:11.638183, event: dispatched, op: pg_info(1 pgs e243:2.3bc) 2014-06-09 16:22:11.638206 7f869af78700 5 -- op tracker -- , seq: 2316, time: 2014-06-09 16:22:11.638205, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3bc) 2014-06-09 16:22:11.638209 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3bc) v4 from osd.1 2014-06-09 16:22:11.638210 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088ca80 2014-06-09 16:22:11.638213 7f869af78700 5 -- op tracker -- , seq: 2316, time: 2014-06-09 16:22:11.638212, event: started, op: pg_info(1 pgs e243:2.3bc) 2014-06-09 16:22:11.638221 7f869af78700 5 -- op tracker -- , seq: 2316, time: 2014-06-09 16:22:11.638220, event: done, op: pg_info(1 pgs e243:2.3bc) 2014-06-09 16:22:11.638225 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638226 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638232 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638233 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638234 7f869af78700 20 osd.26 243 _dispatch 0x150e1c00 pg_info(1 pgs e243:0.732) v4 2014-06-09 16:22:11.638236 7f869af78700 5 -- op tracker -- , seq: 2317, time: 2014-06-09 16:22:11.363778, event: header_read, op: pg_info(1 pgs e243:0.732) 2014-06-09 16:22:11.638243 7f869af78700 5 -- op tracker -- , seq: 2317, time: 2014-06-09 16:22:11.363779, event: throttled, op: pg_info(1 pgs e243:0.732) 2014-06-09 16:22:11.638246 7f869af78700 5 -- op tracker -- , seq: 2317, time: 2014-06-09 16:22:11.363789, event: all_read, op: pg_info(1 pgs e243:0.732) 2014-06-09 16:22:11.638251 7f869af78700 5 -- op tracker -- , seq: 2317, time: 2014-06-09 16:22:11.638232, event: dispatched, op: pg_info(1 pgs e243:0.732) 2014-06-09 16:22:11.638255 7f869af78700 5 -- op tracker -- , seq: 2317, time: 2014-06-09 16:22:11.638254, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.732) 2014-06-09 16:22:11.638260 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.732) v4 from osd.9 2014-06-09 16:22:11.638262 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e1c00 2014-06-09 16:22:11.638265 7f869af78700 5 -- op tracker -- , seq: 2317, time: 2014-06-09 16:22:11.638264, event: started, op: pg_info(1 pgs e243:0.732) 2014-06-09 16:22:11.638275 7f869af78700 5 -- op tracker -- , seq: 2317, time: 2014-06-09 16:22:11.638274, event: done, op: pg_info(1 pgs e243:0.732) 2014-06-09 16:22:11.638282 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638283 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638289 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638290 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638291 7f869af78700 20 osd.26 243 _dispatch 0x3104b180 pg_info(1 pgs e243:1.1da) v4 2014-06-09 16:22:11.638293 7f869af78700 5 -- op tracker -- , seq: 2318, time: 2014-06-09 16:22:11.374742, event: header_read, op: pg_info(1 pgs e243:1.1da) 2014-06-09 16:22:11.638297 7f869af78700 5 -- op tracker -- , seq: 2318, time: 2014-06-09 16:22:11.374742, event: throttled, op: pg_info(1 pgs e243:1.1da) 2014-06-09 16:22:11.638300 7f869af78700 5 -- op tracker -- , seq: 2318, time: 2014-06-09 16:22:11.374754, event: all_read, op: pg_info(1 pgs e243:1.1da) 2014-06-09 16:22:11.638303 7f869af78700 5 -- op tracker -- , seq: 2318, time: 2014-06-09 16:22:11.638289, event: dispatched, op: pg_info(1 pgs e243:1.1da) 2014-06-09 16:22:11.638307 7f869af78700 5 -- op tracker -- , seq: 2318, time: 2014-06-09 16:22:11.638306, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1da) 2014-06-09 16:22:11.638309 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1da) v4 from osd.8 2014-06-09 16:22:11.638310 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104b180 2014-06-09 16:22:11.638313 7f869af78700 5 -- op tracker -- , seq: 2318, time: 2014-06-09 16:22:11.638312, event: started, op: pg_info(1 pgs e243:1.1da) 2014-06-09 16:22:11.638321 7f869af78700 5 -- op tracker -- , seq: 2318, time: 2014-06-09 16:22:11.638320, event: done, op: pg_info(1 pgs e243:1.1da) 2014-06-09 16:22:11.638325 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638326 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638333 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638334 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638334 7f869af78700 20 osd.26 243 _dispatch 0x31040540 pg_notify(1.700(23) epoch 243) v5 2014-06-09 16:22:11.638337 7f869af78700 5 -- op tracker -- , seq: 2319, time: 2014-06-09 16:22:11.388384, event: header_read, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.638341 7f869af78700 5 -- op tracker -- , seq: 2319, time: 2014-06-09 16:22:11.388384, event: throttled, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.638347 7f869af78700 5 -- op tracker -- , seq: 2319, time: 2014-06-09 16:22:11.388418, event: all_read, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.638351 7f869af78700 5 -- op tracker -- , seq: 2319, time: 2014-06-09 16:22:11.638333, event: dispatched, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.638354 7f869af78700 5 -- op tracker -- , seq: 2319, time: 2014-06-09 16:22:11.638353, event: waiting_for_osdmap, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.638356 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.638357 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040540 2014-06-09 16:22:11.638360 7f869af78700 5 -- op tracker -- , seq: 2319, time: 2014-06-09 16:22:11.638359, event: started, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.638368 7f869af78700 5 -- op tracker -- , seq: 2319, time: 2014-06-09 16:22:11.638367, event: done, op: pg_notify(1.700(23) epoch 243) 2014-06-09 16:22:11.638372 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638373 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638384 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638385 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638385 7f869af78700 20 osd.26 243 _dispatch 0x30bd8c40 pg_info(1 pgs e243:3.17) v4 2014-06-09 16:22:11.638388 7f869af78700 5 -- op tracker -- , seq: 2320, time: 2014-06-09 16:22:11.370974, event: header_read, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.638392 7f869af78700 5 -- op tracker -- , seq: 2320, time: 2014-06-09 16:22:11.370975, event: throttled, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.638398 7f869af78700 5 -- op tracker -- , seq: 2320, time: 2014-06-09 16:22:11.370996, event: all_read, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.638402 7f869af78700 5 -- op tracker -- , seq: 2320, time: 2014-06-09 16:22:11.638384, event: dispatched, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.638405 7f869af78700 5 -- op tracker -- , seq: 2320, time: 2014-06-09 16:22:11.638404, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.638408 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.17) v4 from osd.0 2014-06-09 16:22:11.638409 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd8c40 2014-06-09 16:22:11.638411 7f869af78700 5 -- op tracker -- , seq: 2320, time: 2014-06-09 16:22:11.638410, event: started, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.638424 7f869af78700 5 -- op tracker -- , seq: 2320, time: 2014-06-09 16:22:11.638423, event: done, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.638428 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638429 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638435 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638436 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638436 7f869af78700 20 osd.26 243 _dispatch 0x310ca700 pg_notify(1.b6(20) epoch 243) v5 2014-06-09 16:22:11.638439 7f869af78700 5 -- op tracker -- , seq: 2321, time: 2014-06-09 16:22:11.387699, event: header_read, op: pg_notify(1.b6(20) epoch 243) 2014-06-09 16:22:11.638444 7f869af78700 5 -- op tracker -- , seq: 2321, time: 2014-06-09 16:22:11.387700, event: throttled, op: pg_notify(1.b6(20) epoch 243) 2014-06-09 16:22:11.638448 7f869af78700 5 -- op tracker -- , seq: 2321, time: 2014-06-09 16:22:11.387727, event: all_read, op: pg_notify(1.b6(20) epoch 243) 2014-06-09 16:22:11.638451 7f869af78700 5 -- op tracker -- , seq: 2321, time: 2014-06-09 16:22:11.638435, event: dispatched, op: pg_notify(1.b6(20) epoch 243) 2014-06-09 16:22:11.638455 7f869af78700 5 -- op tracker -- , seq: 2321, time: 2014-06-09 16:22:11.638454, event: waiting_for_osdmap, op: pg_notify(1.b6(20) epoch 243) 2014-06-09 16:22:11.638457 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.638458 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310ca700 2014-06-09 16:22:11.638460 7f869af78700 5 -- op tracker -- , seq: 2321, time: 2014-06-09 16:22:11.638459, event: started, op: pg_notify(1.b6(20) epoch 243) 2014-06-09 16:22:11.638469 7f869af78700 5 -- op tracker -- , seq: 2321, time: 2014-06-09 16:22:11.638468, event: done, op: pg_notify(1.b6(20) epoch 243) 2014-06-09 16:22:11.638473 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638474 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638484 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638484 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638485 7f869af78700 20 osd.26 243 _dispatch 0x3088cc40 pg_info(1 pgs e243:3.1b5(4)) v4 2014-06-09 16:22:11.638488 7f869af78700 5 -- op tracker -- , seq: 2322, time: 2014-06-09 16:22:11.251906, event: header_read, op: pg_info(1 pgs e243:3.1b5(4)) 2014-06-09 16:22:11.638491 7f869af78700 5 -- op tracker -- , seq: 2322, time: 2014-06-09 16:22:11.251907, event: throttled, op: pg_info(1 pgs e243:3.1b5(4)) 2014-06-09 16:22:11.638495 7f869af78700 5 -- op tracker -- , seq: 2322, time: 2014-06-09 16:22:11.251928, event: all_read, op: pg_info(1 pgs e243:3.1b5(4)) 2014-06-09 16:22:11.638498 7f869af78700 5 -- op tracker -- , seq: 2322, time: 2014-06-09 16:22:11.638483, event: dispatched, op: pg_info(1 pgs e243:3.1b5(4)) 2014-06-09 16:22:11.638501 7f869af78700 5 -- op tracker -- , seq: 2322, time: 2014-06-09 16:22:11.638500, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1b5(4)) 2014-06-09 16:22:11.638504 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1b5(4)) v4 from osd.2 2014-06-09 16:22:11.638505 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088cc40 2014-06-09 16:22:11.638508 7f869af78700 5 -- op tracker -- , seq: 2322, time: 2014-06-09 16:22:11.638507, event: started, op: pg_info(1 pgs e243:3.1b5(4)) 2014-06-09 16:22:11.638516 7f869af78700 5 -- op tracker -- , seq: 2322, time: 2014-06-09 16:22:11.638515, event: done, op: pg_info(1 pgs e243:3.1b5(4)) 2014-06-09 16:22:11.638516 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.638520 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638521 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638529 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638530 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638530 7f869af78700 20 osd.26 243 _dispatch 0x2d740000 pg_notify(0.19a(23) epoch 243) v5 2014-06-09 16:22:11.638533 7f869af78700 5 -- op tracker -- , seq: 2323, time: 2014-06-09 16:22:11.326114, event: header_read, op: pg_notify(0.19a(23) epoch 243) 2014-06-09 16:22:11.638537 7f869af78700 5 -- op tracker -- , seq: 2323, time: 2014-06-09 16:22:11.326115, event: throttled, op: pg_notify(0.19a(23) epoch 243) 2014-06-09 16:22:11.638540 7f869af78700 5 -- op tracker -- , seq: 2323, time: 2014-06-09 16:22:11.326141, event: all_read, op: pg_notify(0.19a(23) epoch 243) 2014-06-09 16:22:11.638543 7f869af78700 5 -- op tracker -- , seq: 2323, time: 2014-06-09 16:22:11.638529, event: dispatched, op: pg_notify(0.19a(23) epoch 243) 2014-06-09 16:22:11.638544 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3298.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.638546 7f869af78700 5 -- op tracker -- , seq: 2323, time: 2014-06-09 16:22:11.638545, event: waiting_for_osdmap, op: pg_notify(0.19a(23) epoch 243) 2014-06-09 16:22:11.638549 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.638549 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.638550 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d740000 2014-06-09 16:22:11.638552 7f869af78700 5 -- op tracker -- , seq: 2323, time: 2014-06-09 16:22:11.638551, event: started, op: pg_notify(0.19a(23) epoch 243) 2014-06-09 16:22:11.638561 7f869af78700 5 -- op tracker -- , seq: 2323, time: 2014-06-09 16:22:11.638560, event: done, op: pg_notify(0.19a(23) epoch 243) 2014-06-09 16:22:11.638568 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638568 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638578 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638579 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638580 7f869af78700 20 osd.26 243 _dispatch 0x31053180 pg_info(1 pgs e243:2.74b) v4 2014-06-09 16:22:11.638582 7f869af78700 5 -- op tracker -- , seq: 2324, time: 2014-06-09 16:22:11.380489, event: header_read, op: pg_info(1 pgs e243:2.74b) 2014-06-09 16:22:11.638586 7f869af78700 5 -- op tracker -- , seq: 2324, time: 2014-06-09 16:22:11.380489, event: throttled, op: pg_info(1 pgs e243:2.74b) 2014-06-09 16:22:11.638589 7f869af78700 5 -- op tracker -- , seq: 2324, time: 2014-06-09 16:22:11.380499, event: all_read, op: pg_info(1 pgs e243:2.74b) 2014-06-09 16:22:11.638593 7f869af78700 5 -- op tracker -- , seq: 2324, time: 2014-06-09 16:22:11.638578, event: dispatched, op: pg_info(1 pgs e243:2.74b) 2014-06-09 16:22:11.638596 7f869af78700 5 -- op tracker -- , seq: 2324, time: 2014-06-09 16:22:11.638595, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.74b) 2014-06-09 16:22:11.638596 7f86a5610700 10 _do_op 0x2fcae000 seq 3298 r = 0, finisher 0xc2d6e00 0 2014-06-09 16:22:11.638598 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.74b) v4 from osd.7 2014-06-09 16:22:11.638601 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31053180 2014-06-09 16:22:11.638605 7f86a5610700 10 _finish_op 0x2fcae000 seq 3298 osr(2.350 0x2ed0e50)/0x2ed0e50 2014-06-09 16:22:11.638603 7f869af78700 5 -- op tracker -- , seq: 2324, time: 2014-06-09 16:22:11.638602, event: started, op: pg_info(1 pgs e243:2.74b) 2014-06-09 16:22:11.638610 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a69e0] 2014-06-09 16:22:11.638615 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a69e0] 2014-06-09 16:22:11.638617 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.638618 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.638619 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52efc0 (1 active) 2014-06-09 16:22:11.638623 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1320 (2 active) 2014-06-09 16:22:11.638621 7f869af78700 5 -- op tracker -- , seq: 2324, time: 2014-06-09 16:22:11.638619, event: done, op: pg_info(1 pgs e243:2.74b) 2014-06-09 16:22:11.638625 7f86a5610700 5 _do_op 0x309a3560 seq 3300 osr(2.236 0x2ed0ad0)/0x2ed0ad0 start 2014-06-09 16:22:11.638626 7f8691f66700 10 op_submit_start 3348 2014-06-09 16:22:11.638628 7f8691f66700 5 queue_transactions (trailing journal) 3348 2014-06-09 16:22:11.638628 7f86a5610700 10 _do_transaction on 0xbfe5b00 2014-06-09 16:22:11.638630 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638628 7f868a757700 2 waiting 51 > 50 ops || 88844 > 104857600 2014-06-09 16:22:11.638632 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638630 7f8691f66700 5 queue_op 0x308cb8c0 seq 3348 osr(1.8c 0x384c470) 1998 bytes (queue has 50 ops and 86847 bytes) 2014-06-09 16:22:11.638634 7f8691f66700 10 op_submit_finish 3348 2014-06-09 16:22:11.638636 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.638637 7f868f761700 2 waiting 51 > 50 ops || 88730 > 104857600 2014-06-09 16:22:11.638642 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638644 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638643 7f868e75f700 2 waiting 51 > 50 ops || 88856 > 104857600 2014-06-09 16:22:11.638645 7f869af78700 20 osd.26 243 _dispatch 0x2eef7340 pg_info(1 pgs e243:3.3f5) v4 2014-06-09 16:22:11.638639 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.638650 7f868af58700 2 waiting 51 > 50 ops || 88856 > 104857600 2014-06-09 16:22:11.638649 7f869af78700 5 -- op tracker -- , seq: 2325, time: 2014-06-09 16:22:11.436914, event: header_read, op: pg_info(1 pgs e243:3.3f5) 2014-06-09 16:22:11.638655 7f869af78700 5 -- op tracker -- , seq: 2325, time: 2014-06-09 16:22:11.436915, event: throttled, op: pg_info(1 pgs e243:3.3f5) 2014-06-09 16:22:11.638653 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.638665 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.638669 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.638665 7f869af78700 5 -- op tracker -- , seq: 2325, time: 2014-06-09 16:22:11.436929, event: all_read, op: pg_info(1 pgs e243:3.3f5) 2014-06-09 16:22:11.638675 7f869af78700 5 -- op tracker -- , seq: 2325, time: 2014-06-09 16:22:11.638642, event: dispatched, op: pg_info(1 pgs e243:3.3f5) 2014-06-09 16:22:11.638675 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.725( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 0.956414 7 0.001118 2014-06-09 16:22:11.638684 7f869af78700 5 -- op tracker -- , seq: 2325, time: 2014-06-09 16:22:11.638682, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3f5) 2014-06-09 16:22:11.638687 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.638688 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3f5) v4 from osd.3 2014-06-09 16:22:11.638691 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef7340 2014-06-09 16:22:11.638689 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.725( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.638695 7f869af78700 5 -- op tracker -- , seq: 2325, time: 2014-06-09 16:22:11.638693, event: started, op: pg_info(1 pgs e243:3.3f5) 2014-06-09 16:22:11.638703 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3300.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.638701 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.725( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.638708 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.638710 7f869af78700 5 -- op tracker -- , seq: 2325, time: 2014-06-09 16:22:11.638708, event: done, op: pg_info(1 pgs e243:3.3f5) 2014-06-09 16:22:11.638717 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638718 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638712 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.638721 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.638729 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638730 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638729 7f86a5610700 10 _do_op 0x309a3560 seq 3300 r = 0, finisher 0x4eb8940 0 2014-06-09 16:22:11.638731 7f869af78700 20 osd.26 243 _dispatch 0x3088d880 pg_info(1 pgs e243:1.757) v4 2014-06-09 16:22:11.638729 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.638736 7f86a5610700 10 _finish_op 0x309a3560 seq 3300 osr(2.236 0x2ed0ad0)/0x2ed0ad0 2014-06-09 16:22:11.638736 7f869af78700 5 -- op tracker -- , seq: 2326, time: 2014-06-09 16:22:11.311203, event: header_read, op: pg_info(1 pgs e243:1.757) 2014-06-09 16:22:11.638741 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1320 (1 active) 2014-06-09 16:22:11.638743 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115680 (2 active) 2014-06-09 16:22:11.638742 7f869af78700 5 -- op tracker -- , seq: 2326, time: 2014-06-09 16:22:11.311203, event: throttled, op: pg_info(1 pgs e243:1.757) 2014-06-09 16:22:11.638745 7f868a757700 10 op_submit_start 3349 2014-06-09 16:22:11.638745 7f86a5610700 5 _do_op 0x2fcaf920 seq 3301 osr(1.65e 0x384df30)/0x384df30 start 2014-06-09 16:22:11.638747 7f868a757700 5 queue_transactions (trailing journal) 3349 2014-06-09 16:22:11.638747 7f8691765700 2 waiting 51 > 50 ops || 88264 > 104857600 2014-06-09 16:22:11.638749 7f86a5610700 10 _do_transaction on 0xc144300 2014-06-09 16:22:11.638749 7f868a757700 5 queue_op 0x30606900 seq 3349 osr(0.4ba 0x2bd7710) 1997 bytes (queue has 50 ops and 86835 bytes) 2014-06-09 16:22:11.638753 7f868a757700 10 op_submit_finish 3349 2014-06-09 16:22:11.638754 7f8690763700 2 waiting 51 > 50 ops || 88789 > 104857600 2014-06-09 16:22:11.638749 7f869af78700 5 -- op tracker -- , seq: 2326, time: 2014-06-09 16:22:11.311211, event: all_read, op: pg_info(1 pgs e243:1.757) 2014-06-09 16:22:11.638755 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.638759 7f868d75d700 2 waiting 51 > 50 ops || 88789 > 104857600 2014-06-09 16:22:11.638758 7f869af78700 5 -- op tracker -- , seq: 2326, time: 2014-06-09 16:22:11.638728, event: dispatched, op: pg_info(1 pgs e243:1.757) 2014-06-09 16:22:11.638764 7f868c75b700 2 waiting 51 > 50 ops || 88789 > 104857600 2014-06-09 16:22:11.638758 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.638764 7f869af78700 5 -- op tracker -- , seq: 2326, time: 2014-06-09 16:22:11.638762, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.757) 2014-06-09 16:22:11.638773 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.757) v4 from osd.1 2014-06-09 16:22:11.638776 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088d880 2014-06-09 16:22:11.638781 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.638780 7f869af78700 5 -- op tracker -- , seq: 2326, time: 2014-06-09 16:22:11.638779, event: started, op: pg_info(1 pgs e243:1.757) 2014-06-09 16:22:11.638775 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.638789 7f8691f66700 5 queue_transactions existing osr(0.725 0x384c3d0)/0x384c3d0 2014-06-09 16:22:11.638789 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.638792 7f8691f66700 2 waiting 51 > 50 ops || 88675 > 104857600 2014-06-09 16:22:11.638797 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7a700] 2014-06-09 16:22:11.638787 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1b0b7200,0x257c5700,0x257c5600,0x257c5500,0x257c5400,0x257c5300,0x230da200,0x230da100,0x230da000,0x270bff00 2014-06-09 16:22:11.638795 7f869af78700 5 -- op tracker -- , seq: 2326, time: 2014-06-09 16:22:11.638793, event: done, op: pg_info(1 pgs e243:1.757) 2014-06-09 16:22:11.638800 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7a700] 2014-06-09 16:22:11.638801 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.638802 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.638803 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638805 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638805 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3301.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.638810 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.638815 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638817 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638818 7f8689755700 2 waiting 51 > 50 ops || 88170 > 104857600 2014-06-09 16:22:11.638818 7f869af78700 20 osd.26 243 _dispatch 0x240eba40 pg_info(1 pgs e243:1.479) v4 2014-06-09 16:22:11.638824 7f8690f64700 2 waiting 51 > 50 ops || 88715 > 104857600 2014-06-09 16:22:11.638830 7f868b759700 2 waiting 51 > 50 ops || 88934 > 104857600 2014-06-09 16:22:11.638823 7f869af78700 5 -- op tracker -- , seq: 2327, time: 2014-06-09 16:22:11.363794, event: header_read, op: pg_info(1 pgs e243:1.479) 2014-06-09 16:22:11.638836 7f868bf5a700 2 waiting 51 > 50 ops || 88170 > 104857600 2014-06-09 16:22:11.638827 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.638835 7f869af78700 5 -- op tracker -- , seq: 2327, time: 2014-06-09 16:22:11.363794, event: throttled, op: pg_info(1 pgs e243:1.479) 2014-06-09 16:22:11.638836 7f86a5610700 10 _do_op 0x2fcaf920 seq 3301 r = 0, finisher 0xbd73500 0 2014-06-09 16:22:11.638842 7f8689f56700 2 waiting 51 > 50 ops || 88836 > 104857600 2014-06-09 16:22:11.638843 7f86a5610700 10 _finish_op 0x2fcaf920 seq 3301 osr(1.65e 0x384df30)/0x384df30 2014-06-09 16:22:11.638842 7f869af78700 5 -- op tracker -- , seq: 2327, time: 2014-06-09 16:22:11.363806, event: all_read, op: pg_info(1 pgs e243:1.479) 2014-06-09 16:22:11.638839 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.638849 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd74660] 2014-06-09 16:22:11.638851 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115680 (1 active) 2014-06-09 16:22:11.638852 7f8691765700 10 op_submit_start 3350 2014-06-09 16:22:11.638853 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd74660] 2014-06-09 16:22:11.638854 7f8691765700 5 queue_transactions (trailing journal) 3350 2014-06-09 16:22:11.638853 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524fc0 (2 active) 2014-06-09 16:22:11.638854 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.638848 7f869af78700 5 -- op tracker -- , seq: 2327, time: 2014-06-09 16:22:11.638815, event: dispatched, op: pg_info(1 pgs e243:1.479) 2014-06-09 16:22:11.638856 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.638849 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.638856 7f86a5610700 5 _do_op 0x30060360 seq 3302 osr(0.25d 0x2bd6a00)/0x2bd6a00 start 2014-06-09 16:22:11.638856 7f8691765700 5 queue_op 0x300602a0 seq 3350 osr(3.124 0x3ed07c0) 1429 bytes (queue has 50 ops and 86381 bytes) 2014-06-09 16:22:11.638859 7f8691765700 10 op_submit_finish 3350 2014-06-09 16:22:11.638859 7f86a5610700 10 _do_transaction on 0x3859440 2014-06-09 16:22:11.638859 7f8692767700 2 waiting 51 > 50 ops || 87242 > 104857600 2014-06-09 16:22:11.638859 7f869af78700 5 -- op tracker -- , seq: 2327, time: 2014-06-09 16:22:11.638857, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.479) 2014-06-09 16:22:11.638864 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.479) v4 from osd.9 2014-06-09 16:22:11.638866 7f868ff62700 2 waiting 51 > 50 ops || 88258 > 104857600 2014-06-09 16:22:11.638866 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.638867 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eba40 2014-06-09 16:22:11.638870 7f868cf5c700 2 waiting 51 > 50 ops || 88264 > 104857600 2014-06-09 16:22:11.638865 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1016 2014-06-09 16:22:11.638863 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:0.1a7( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.638875 7f868f761700 2 waiting 51 > 50 ops || 88264 > 104857600 2014-06-09 16:22:11.638879 7f868e75f700 2 waiting 51 > 50 ops || 88390 > 104857600 2014-06-09 16:22:11.638872 7f869af78700 5 -- op tracker -- , seq: 2327, time: 2014-06-09 16:22:11.638870, event: started, op: pg_info(1 pgs e243:1.479) 2014-06-09 16:22:11.638875 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.638876 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.638883 7f868af58700 2 waiting 51 > 50 ops || 88390 > 104857600 2014-06-09 16:22:11.638887 7f8690763700 2 waiting 51 > 50 ops || 88335 > 104857600 2014-06-09 16:22:11.638890 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.638884 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.638891 7f868d75d700 2 waiting 51 > 50 ops || 88335 > 104857600 2014-06-09 16:22:11.638893 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.638895 7f868c75b700 2 waiting 51 > 50 ops || 88335 > 104857600 2014-06-09 16:22:11.638899 7f8691f66700 2 waiting 51 > 50 ops || 88221 > 104857600 2014-06-09 16:22:11.638899 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.638896 7f869af78700 5 -- op tracker -- , seq: 2327, time: 2014-06-09 16:22:11.638893, event: done, op: pg_info(1 pgs e243:1.479) 2014-06-09 16:22:11.638903 7f8689755700 2 waiting 51 > 50 ops || 87716 > 104857600 2014-06-09 16:22:11.638895 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.479( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.638897 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.638906 7f8690f64700 2 waiting 51 > 50 ops || 88261 > 104857600 2014-06-09 16:22:11.638906 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638909 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638911 7f868b759700 2 waiting 51 > 50 ops || 88480 > 104857600 2014-06-09 16:22:11.638906 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.638908 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.638914 7f868bf5a700 2 waiting 51 > 50 ops || 87716 > 104857600 2014-06-09 16:22:11.638915 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.638918 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.638918 7f8689f56700 2 waiting 51 > 50 ops || 88382 > 104857600 2014-06-09 16:22:11.638917 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3302.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.638921 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.638922 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.638922 7f868df5e700 2 waiting 51 > 50 ops || 87810 > 104857600 2014-06-09 16:22:11.638923 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.638925 7f869af78700 20 osd.26 243 _dispatch 0x3104afc0 pg_info(1 pgs e243:2.d) v4 2014-06-09 16:22:11.638919 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=212 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.638921 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.638931 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.211( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.638930 7f869af78700 5 -- op tracker -- , seq: 2328, time: 2014-06-09 16:22:11.374787, event: header_read, op: pg_info(1 pgs e243:2.d) 2014-06-09 16:22:11.638939 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.638931 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=212 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.638943 7f86a5610700 10 _do_op 0x30060360 seq 3302 r = 0, finisher 0xbd6baa0 0 2014-06-09 16:22:11.638942 7f869af78700 5 -- op tracker -- , seq: 2328, time: 2014-06-09 16:22:11.374788, event: throttled, op: pg_info(1 pgs e243:2.d) 2014-06-09 16:22:11.638943 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.638949 7f86a5610700 10 _finish_op 0x30060360 seq 3302 osr(0.25d 0x2bd6a00)/0x2bd6a00 2014-06-09 16:22:11.638943 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=212 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.925071 7 0.000183 2014-06-09 16:22:11.638948 7f869af78700 5 -- op tracker -- , seq: 2328, time: 2014-06-09 16:22:11.374801, event: all_read, op: pg_info(1 pgs e243:2.d) 2014-06-09 16:22:11.638951 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d71e0] 2014-06-09 16:22:11.638955 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d71e0] 2014-06-09 16:22:11.638956 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.638956 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524fc0 (1 active) 2014-06-09 16:22:11.638957 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.638958 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcc60 (2 active) 2014-06-09 16:22:11.638953 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.638953 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=212 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.638954 7f869af78700 5 -- op tracker -- , seq: 2328, time: 2014-06-09 16:22:11.638921, event: dispatched, op: pg_info(1 pgs e243:2.d) 2014-06-09 16:22:11.638961 7f86a5610700 5 _do_op 0x30d974a0 seq 3303 osr(0.456 0x2bd7530)/0x2bd7530 start 2014-06-09 16:22:11.638962 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.638963 7f8692767700 10 op_submit_start 3351 2014-06-09 16:22:11.638963 7f86a5610700 10 _do_transaction on 0x2fb63c0 2014-06-09 16:22:11.638964 7f8692767700 5 queue_transactions (trailing journal) 3351 2014-06-09 16:22:11.638966 7f8692767700 5 queue_op 0x309a2000 seq 3351 osr(1.60e 0x384dc30) 861 bytes (queue has 50 ops and 85359 bytes) 2014-06-09 16:22:11.638965 7f869af78700 5 -- op tracker -- , seq: 2328, time: 2014-06-09 16:22:11.638963, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.d) 2014-06-09 16:22:11.638969 7f8692767700 10 op_submit_finish 3351 2014-06-09 16:22:11.638970 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.638969 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.d) v4 from osd.8 2014-06-09 16:22:11.638964 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=212 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.638972 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104afc0 2014-06-09 16:22:11.638966 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.638976 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0210 (19 active) 2014-06-09 16:22:11.638975 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.126( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.638975 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=212 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.638984 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.638983 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.638977 7f869af78700 5 -- op tracker -- , seq: 2328, time: 2014-06-09 16:22:11.638975, event: started, op: pg_info(1 pgs e243:2.d) 2014-06-09 16:22:11.638984 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'571 -> 213'571 2014-06-09 16:22:11.638988 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.638998 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.639003 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.638988 7f8692767700 10 osd.26 243 dequeue_op 0x230da200 prio 63 cost 962560 latency 0.877607 osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.639007 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639002 7f869af78700 5 -- op tracker -- , seq: 2328, time: 2014-06-09 16:22:11.639000, event: done, op: pg_info(1 pgs e243:2.d) 2014-06-09 16:22:11.639010 7f868ff62700 2 waiting 51 > 50 ops || 87236 > 104857600 2014-06-09 16:22:11.639015 7f868cf5c700 2 waiting 51 > 50 ops || 87242 > 104857600 2014-06-09 16:22:11.639010 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.639020 7f868f761700 2 waiting 51 > 50 ops || 87242 > 104857600 2014-06-09 16:22:11.639021 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.639020 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3303.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.639024 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.639024 7f868e75f700 2 waiting 51 > 50 ops || 87368 > 104857600 2014-06-09 16:22:11.639024 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.639027 7f868af58700 2 waiting 51 > 50 ops || 87368 > 104857600 2014-06-09 16:22:11.639023 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.639025 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.639031 7f8690763700 2 waiting 51 > 50 ops || 87313 > 104857600 2014-06-09 16:22:11.639032 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639035 7f868d75d700 2 waiting 51 > 50 ops || 87313 > 104857600 2014-06-09 16:22:11.639037 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.639039 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.639039 7f868c75b700 2 waiting 51 > 50 ops || 87313 > 104857600 2014-06-09 16:22:11.639040 7f869af78700 20 osd.26 243 _dispatch 0x31040380 pg_notify(1.293(20) epoch 243) v5 2014-06-09 16:22:11.639043 7f8691f66700 2 waiting 51 > 50 ops || 87199 > 104857600 2014-06-09 16:22:11.639036 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.639047 7f8689755700 2 waiting 51 > 50 ops || 86694 > 104857600 2014-06-09 16:22:11.639045 7f869af78700 5 -- op tracker -- , seq: 2329, time: 2014-06-09 16:22:11.388455, event: header_read, op: pg_notify(1.293(20) epoch 243) 2014-06-09 16:22:11.639049 7f86a5610700 10 _do_op 0x30d974a0 seq 3303 r = 0, finisher 0xbef1c80 0 2014-06-09 16:22:11.639051 7f8690f64700 2 waiting 51 > 50 ops || 87239 > 104857600 2014-06-09 16:22:11.639046 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.639054 7f868b759700 2 waiting 51 > 50 ops || 87458 > 104857600 2014-06-09 16:22:11.639055 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639053 7f869af78700 5 -- op tracker -- , seq: 2329, time: 2014-06-09 16:22:11.388456, event: throttled, op: pg_notify(1.293(20) epoch 243) 2014-06-09 16:22:11.639056 7f86a5610700 10 _finish_op 0x30d974a0 seq 3303 osr(0.456 0x2bd7530)/0x2bd7530 2014-06-09 16:22:11.639060 7f869af78700 5 -- op tracker -- , seq: 2329, time: 2014-06-09 16:22:11.388484, event: all_read, op: pg_notify(1.293(20) epoch 243) 2014-06-09 16:22:11.639064 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.639066 7f8689f56700 10 op_submit_start 3352 2014-06-09 16:22:11.639065 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcc60 (1 active) 2014-06-09 16:22:11.639059 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.639067 7f8689f56700 5 queue_transactions (trailing journal) 3352 2014-06-09 16:22:11.639068 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338000 (2 active) 2014-06-09 16:22:11.639068 7f8691765700 5 queue_transactions existing osr(3.587 0x3ed1840)/0x3ed1840 2014-06-09 16:22:11.639068 7f8689f56700 5 queue_op 0x30d97380 seq 3352 osr(0.6c0 0x2bd7f50) 2001 bytes (queue has 50 ops and 85359 bytes) 2014-06-09 16:22:11.639070 7f8691765700 2 waiting 51 > 50 ops || 86694 > 104857600 2014-06-09 16:22:11.639070 7f86a5610700 5 _do_op 0x2fcaef60 seq 3304 osr(1.6f5 0x2ed0170)/0x2ed0170 start 2014-06-09 16:22:11.639071 7f8689f56700 10 op_submit_finish 3352 2014-06-09 16:22:11.639073 7f86a5610700 10 _do_transaction on 0x336acc0 2014-06-09 16:22:11.639069 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.219( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.639078 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639079 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.639073 7f8692767700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.639007, event: reached_pg, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.639075 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.639082 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.639088 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.639087 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.639098 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.639094 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.652( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.639104 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639100 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x230da200 2014-06-09 16:22:11.639103 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.15c( v 213'624 (0'0,213'624] local-les=191 n=123 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.639113 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.639108 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.639117 7f8689f56700 7 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.639120 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.639114 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.639130 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639129 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3304.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.639133 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.639126 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.15c( v 213'624 (0'0,213'624] local-les=191 n=123 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.639134 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.639144 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.639145 7f8692767700 15 getattr 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 '_' 2014-06-09 16:22:11.639147 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.616( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.639156 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639149 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.16d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.639156 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d6e00,0x4eb8940,0xbd73500,0xbd6baa0,0xbef1c80] 2014-06-09 16:22:11.639160 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.639160 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.639171 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.639161 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.639173 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.639180 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.639183 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639175 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.639186 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.2a2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.639197 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.639201 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.639203 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (19 active) 2014-06-09 16:22:11.639200 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.639204 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.158912 7 0.000196 2014-06-09 16:22:11.639216 7f868ef60700 2 waiting 51 > 50 ops || 87242 > 104857600 2014-06-09 16:22:11.639214 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.639224 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.639233 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (20 active) 2014-06-09 16:22:11.639232 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.639240 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.639240 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.639249 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.639254 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.639263 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.639267 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.639273 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.639282 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.639281 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.639289 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.639297 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.639302 7f8689f56700 5 queue_transactions existing osr(0.2a2 0x2bd6c20)/0x2bd6c20 2014-06-09 16:22:11.639304 7f8689f56700 2 waiting 51 > 50 ops || 87368 > 104857600 2014-06-09 16:22:11.639298 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.639309 7f868cf5c700 2 waiting 51 > 50 ops || 87242 > 104857600 2014-06-09 16:22:11.639302 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.639314 7f868df5e700 2 waiting 51 > 50 ops || 86788 > 104857600 2014-06-09 16:22:11.639318 7f868e75f700 2 waiting 51 > 50 ops || 87368 > 104857600 2014-06-09 16:22:11.639312 7f868a757700 20 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.639322 7f868af58700 2 waiting 51 > 50 ops || 87368 > 104857600 2014-06-09 16:22:11.639314 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.639326 7f8690763700 2 waiting 51 > 50 ops || 87313 > 104857600 2014-06-09 16:22:11.639321 7f868a757700 20 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.639327 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.639329 7f868d75d700 2 waiting 51 > 50 ops || 87313 > 104857600 2014-06-09 16:22:11.639330 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639333 7f868c75b700 2 waiting 51 > 50 ops || 87313 > 104857600 2014-06-09 16:22:11.639336 7f8691f66700 2 waiting 51 > 50 ops || 87199 > 104857600 2014-06-09 16:22:11.639340 7f8689755700 2 waiting 51 > 50 ops || 86694 > 104857600 2014-06-09 16:22:11.639335 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.639344 7f8690f64700 2 waiting 51 > 50 ops || 87239 > 104857600 2014-06-09 16:22:11.639347 7f868b759700 2 waiting 51 > 50 ops || 87458 > 104857600 2014-06-09 16:22:11.639342 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.639351 7f868bf5a700 2 waiting 51 > 50 ops || 86694 > 104857600 2014-06-09 16:22:11.639346 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.639354 7f868ff62700 2 waiting 51 > 50 ops || 87236 > 104857600 2014-06-09 16:22:11.639357 7f868f761700 2 waiting 51 > 50 ops || 87242 > 104857600 2014-06-09 16:22:11.639360 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x305097c0] 2014-06-09 16:22:11.639356 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.639364 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x305097c0] 2014-06-09 16:22:11.639366 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.639357 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.639367 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.639364 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.639371 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.639377 7f868a757700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.639386 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.639393 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:188 2014-06-09 16:22:11.639408 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639401 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.639414 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.639412 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.451( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.639424 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.639435 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639430 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.639444 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d6e00,0x4eb8940,0xbd73500,0xbd6baa0,0xbef1c80] 2014-06-09 16:22:11.639447 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.639448 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.639439 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.639452 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.639466 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.705336 7 0.000153 2014-06-09 16:22:11.639477 7f8692f68700 2 waiting 51 > 50 ops || 86694 > 104857600 2014-06-09 16:22:11.639479 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.639489 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.639496 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.639507 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'522 -> 213'522 2014-06-09 16:22:11.639543 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.639580 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.639585 7f868a757700 5 queue_transactions existing osr(3.786 0x3ed1d90)/0x3ed1d90 2014-06-09 16:22:11.639587 7f868a757700 2 waiting 51 > 50 ops || 86694 > 104857600 2014-06-09 16:22:11.639635 7f869af78700 5 -- op tracker -- , seq: 2329, time: 2014-06-09 16:22:11.639037, event: dispatched, op: pg_notify(1.293(20) epoch 243) 2014-06-09 16:22:11.639645 7f869af78700 5 -- op tracker -- , seq: 2329, time: 2014-06-09 16:22:11.639643, event: waiting_for_osdmap, op: pg_notify(1.293(20) epoch 243) 2014-06-09 16:22:11.639649 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.639652 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040380 2014-06-09 16:22:11.639656 7f869af78700 5 -- op tracker -- , seq: 2329, time: 2014-06-09 16:22:11.639654, event: started, op: pg_notify(1.293(20) epoch 243) 2014-06-09 16:22:11.639672 7f869af78700 5 -- op tracker -- , seq: 2329, time: 2014-06-09 16:22:11.639670, event: done, op: pg_notify(1.293(20) epoch 243) 2014-06-09 16:22:11.639682 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.639684 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.639701 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.639702 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.639703 7f869af78700 20 osd.26 243 _dispatch 0x30bd8540 pg_info(1 pgs e243:2.4f4) v4 2014-06-09 16:22:11.639708 7f869af78700 5 -- op tracker -- , seq: 2330, time: 2014-06-09 16:22:11.371008, event: header_read, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.639715 7f869af78700 5 -- op tracker -- , seq: 2330, time: 2014-06-09 16:22:11.371009, event: throttled, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.639721 7f869af78700 5 -- op tracker -- , seq: 2330, time: 2014-06-09 16:22:11.371022, event: all_read, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.639727 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:11.639727 7f869af78700 5 -- op tracker -- , seq: 2330, time: 2014-06-09 16:22:11.639700, event: dispatched, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.639733 7f86a5610700 10 _do_op 0x2fcaef60 seq 3304 r = 0, finisher 0xbdbaba0 0 2014-06-09 16:22:11.639736 7f869af78700 5 -- op tracker -- , seq: 2330, time: 2014-06-09 16:22:11.639734, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.639730 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 but can_create 2014-06-09 16:22:11.639741 7f86a5610700 10 _finish_op 0x2fcaef60 seq 3304 osr(1.6f5 0x2ed0170)/0x2ed0170 2014-06-09 16:22:11.639743 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7dfe0] 2014-06-09 16:22:11.639746 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7dfe0] 2014-06-09 16:22:11.639747 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.639748 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.639747 7f8692767700 15 getattr 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 'snapset' 2014-06-09 16:22:11.639748 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338000 (1 active) 2014-06-09 16:22:11.639751 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922ea0 (2 active) 2014-06-09 16:22:11.639740 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4f4) v4 from osd.0 2014-06-09 16:22:11.639754 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd8540 2014-06-09 16:22:11.639753 7f86a5610700 5 _do_op 0x2fcaef00 seq 3305 osr(1.15b 0x384cbf0)/0x384cbf0 start 2014-06-09 16:22:11.639757 7f86a5610700 10 _do_transaction on 0xc22c9c0 2014-06-09 16:22:11.639759 7f869af78700 5 -- op tracker -- , seq: 2330, time: 2014-06-09 16:22:11.639757, event: started, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.639763 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.639769 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdbaba0] 2014-06-09 16:22:11.639778 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:11.639775 7f869af78700 5 -- op tracker -- , seq: 2330, time: 2014-06-09 16:22:11.639773, event: done, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.639772 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.639780 7f8692767700 15 getattr 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/snapdir//3 'snapset' 2014-06-09 16:22:11.639784 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.639786 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.639788 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.639800 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.639802 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.639803 7f869af78700 20 osd.26 243 _dispatch 0x310ca540 pg_info(1 pgs e243:2.359) v4 2014-06-09 16:22:11.639806 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:11.639803 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.639808 7f869af78700 5 -- op tracker -- , seq: 2331, time: 2014-06-09 16:22:11.387732, event: header_read, op: pg_info(1 pgs e243:2.359) 2014-06-09 16:22:11.639815 7f869af78700 5 -- op tracker -- , seq: 2331, time: 2014-06-09 16:22:11.387733, event: throttled, op: pg_info(1 pgs e243:2.359) 2014-06-09 16:22:11.639819 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdbaba0] 2014-06-09 16:22:11.639821 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.639822 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.639820 7f869af78700 5 -- op tracker -- , seq: 2331, time: 2014-06-09 16:22:11.387745, event: all_read, op: pg_info(1 pgs e243:2.359) 2014-06-09 16:22:11.639827 7f8691765700 10 op_submit_start 3353 2014-06-09 16:22:11.639829 7f8691765700 5 queue_transactions (trailing journal) 3353 2014-06-09 16:22:11.639826 7f869af78700 5 -- op tracker -- , seq: 2331, time: 2014-06-09 16:22:11.639800, event: dispatched, op: pg_info(1 pgs e243:2.359) 2014-06-09 16:22:11.639820 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30d99080 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.639830 7f8691765700 5 queue_op 0x30060240 seq 3353 osr(3.587 0x3ed1840) 1335 bytes (queue has 50 ops and 84720 bytes) 2014-06-09 16:22:11.639834 7f8691765700 10 op_submit_finish 3353 2014-06-09 16:22:11.639833 7f869af78700 5 -- op tracker -- , seq: 2331, time: 2014-06-09 16:22:11.639831, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.359) 2014-06-09 16:22:11.639833 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.639843 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.359) v4 from osd.5 2014-06-09 16:22:11.639844 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.639854 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310ca540 2014-06-09 16:22:11.639859 7f869af78700 5 -- op tracker -- , seq: 2331, time: 2014-06-09 16:22:11.639857, event: started, op: pg_info(1 pgs e243:2.359) 2014-06-09 16:22:11.639855 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30d99080 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 rwstate(none n=0 w=0) oi: 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd4c40 snapset: 0=[]:[] 2014-06-09 16:22:11.639871 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 @head oi=886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.639874 7f869af78700 5 -- op tracker -- , seq: 2331, time: 2014-06-09 16:22:11.639872, event: done, op: pg_info(1 pgs e243:2.359) 2014-06-09 16:22:11.639889 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.639891 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.639895 7f8692767700 15 getattr 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/snapdir//3 '_' 2014-06-09 16:22:11.639904 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.639893 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.5e2( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.639906 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.639907 7f869af78700 20 osd.26 243 _dispatch 0x1abd5dc0 pg_notify(3.1c7(12),2.755(21) epoch 243) v5 2014-06-09 16:22:11.639908 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.639913 7f869af78700 5 -- op tracker -- , seq: 2332, time: 2014-06-09 16:22:11.259722, event: header_read, op: pg_notify(3.1c7(12),2.755(21) epoch 243) 2014-06-09 16:22:11.639920 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.639923 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:11.639922 7f869af78700 5 -- op tracker -- , seq: 2332, time: 2014-06-09 16:22:11.259722, event: throttled, op: pg_notify(3.1c7(12),2.755(21) epoch 243) 2014-06-09 16:22:11.639929 7f869af78700 5 -- op tracker -- , seq: 2332, time: 2014-06-09 16:22:11.259755, event: all_read, op: pg_notify(3.1c7(12),2.755(21) epoch 243) 2014-06-09 16:22:11.639926 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:1.3ef( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.639925 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/snapdir//3 and !can_create 2014-06-09 16:22:11.639935 7f869af78700 5 -- op tracker -- , seq: 2332, time: 2014-06-09 16:22:11.639904, event: dispatched, op: pg_notify(3.1c7(12),2.755(21) epoch 243) 2014-06-09 16:22:11.639938 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.639945 7f8692767700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.639939, event: started, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.639959 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.639964 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.27( v 213'666 (0'0,213'666] local-les=206 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.639964 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30640600 2014-06-09 16:22:11.639982 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.639977 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.27( v 213'666 (0'0,213'666] local-les=206 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info+log from osd.5 3.27( v 215'668 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) log((213'666,215'668], crt=0'0) 2014-06-09 16:22:11.639986 7f8692767700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.639980, event: started, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.639991 7f8691765700 10 merge_log log((213'666,215'668], crt=0'0) from osd.5 into log((0'0,213'666], crt=0'0) 2014-06-09 16:22:11.639995 7f8691765700 10 merge_log extending head to 215'668 2014-06-09 16:22:11.639999 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3305.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.640003 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.639997 7f8691765700 20 ? 215'668 (215'667) modify 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 by client.4565.0:33630 2014-06-09 10:20:52.446948 2014-06-09 16:22:11.639993 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] ov 0'0 av 243'637 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.640005 7f8691765700 20 ? 215'667 (213'666) modify 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 by client.4565.0:33629 2014-06-09 10:20:51.128696 2014-06-09 16:22:11.640012 7f8691765700 20 merge_log 215'667 (213'666) modify 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 by client.4565.0:33629 2014-06-09 10:20:51.128696 2014-06-09 16:22:11.640008 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 2014-06-09 16:22:11.640019 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.640029 7f868ef60700 2 waiting 51 > 50 ops || 86603 > 104857600 2014-06-09 16:22:11.640022 7f8691765700 20 merge_log 215'668 (215'667) modify 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 by client.4565.0:33630 2014-06-09 10:20:52.446948 2014-06-09 16:22:11.640036 7f86a5610700 10 _do_op 0x2fcaef00 seq 3305 r = 0, finisher 0x17001320 0 2014-06-09 16:22:11.640042 7f86a5610700 10 _finish_op 0x2fcaef00 seq 3305 osr(1.15b 0x384cbf0)/0x384cbf0 2014-06-09 16:22:11.640036 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~962560 2014-06-09 16:22:11.640048 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922ea0 (1 active) 2014-06-09 16:22:11.640051 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6ddc20 (2 active) 2014-06-09 16:22:11.640053 7f86a5610700 5 _do_op 0x30d96960 seq 3306 osr(3.5b3 0x3ed17e0)/0x3ed17e0 start 2014-06-09 16:22:11.640055 7f86a5610700 10 _do_transaction on 0xbf2e0c0 2014-06-09 16:22:11.640061 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.640064 7f868ef60700 10 op_submit_start 3354 2014-06-09 16:22:11.640066 7f868ef60700 5 queue_transactions (trailing journal) 3354 2014-06-09 16:22:11.640067 7f868ef60700 5 queue_op 0x2fcafda0 seq 3354 osr(2.646 0x2ed1c30) 1883 bytes (queue has 50 ops and 84720 bytes) 2014-06-09 16:22:11.640070 7f868ef60700 10 op_submit_finish 3354 2014-06-09 16:22:11.640061 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228a00 2014-06-09 16:22:11.640074 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 snapset=0x30dd4c78 snapc=0=[] 2014-06-09 16:22:11.640073 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.640085 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.640094 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.640088 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.640098 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.640096 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.640110 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3306.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.640113 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.640116 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0280 (19 active) 2014-06-09 16:22:11.640107 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 0x30640600 op modify 2014-06-09 16:22:11.640122 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.640121 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:04.581819 2014-06-09 16:22:11.640131 7f86a5610700 10 _do_op 0x30d96960 seq 3306 r = 0, finisher 0x33a7340 0 2014-06-09 16:22:11.640133 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17001320] 2014-06-09 16:22:11.640134 7f86a5610700 10 _finish_op 0x30d96960 seq 3306 osr(3.5b3 0x3ed17e0)/0x3ed17e0 2014-06-09 16:22:11.640137 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6ddc20 (1 active) 2014-06-09 16:22:11.640138 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17001320] 2014-06-09 16:22:11.640139 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f440 (2 active) 2014-06-09 16:22:11.640139 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a7340] 2014-06-09 16:22:11.640141 7f86a5610700 5 _do_op 0x15996ba0 seq 3307 osr(1.508 0x384db90)/0x384db90 start 2014-06-09 16:22:11.640143 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a7340] 2014-06-09 16:22:11.640144 7f86a5610700 10 _do_transaction on 0xbf8b2c0 2014-06-09 16:22:11.640144 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.640145 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.640149 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.640155 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd740a0,0xbccc260] 2014-06-09 16:22:11.640144 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.640157 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.640171 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.640167 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.640186 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.640179 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 1 on osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.640187 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.640197 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 1 o 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.640199 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.640206 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3299.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.640210 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.640212 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.640221 7f8692767700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.640216, event: waiting for subops from 7, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.640232 7f86a5e11700 10 _do_op 0x309a2180 seq 3299 r = 0, finisher 0x4eb8700 0 2014-06-09 16:22:11.640226 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.640235 7f86a5e11700 10 _finish_op 0x309a2180 seq 3299 osr(2.5b6 0x2ed1b90)/0x2ed1b90 2014-06-09 16:22:11.640239 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd740a0,0xbccc260] 2014-06-09 16:22:11.640239 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.640241 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061f8c0] 2014-06-09 16:22:11.640240 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd9e0 (1 active) 2014-06-09 16:22:11.640243 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e240 (2 active) 2014-06-09 16:22:11.640243 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061f8c0] 2014-06-09 16:22:11.640244 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.640245 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.640244 7f86a5e11700 5 _do_op 0x309a2120 seq 3308 osr(3.3a8 0x3ed0f80)/0x3ed0f80 start 2014-06-09 16:22:11.640247 7f86a5e11700 10 _do_transaction on 0xc160240 2014-06-09 16:22:11.642951 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.642958 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 214'636 (0'0,214'636] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'636], crt=0'0) [243'637 (0'0) modify 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 by client.4570.0:34069 2014-06-09 10:20:04.581819] 2014-06-09 16:22:11.640250 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x4eb8700] 2014-06-09 16:22:11.640268 7f8691765700 10 merge_log result log((0'0,215'668], crt=215'668) missing(1) changed=1 2014-06-09 16:22:11.642989 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'637 (0'0) modify 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 by client.4570.0:34069 2014-06-09 10:20:04.581819 2014-06-09 16:22:11.642995 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.642999 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 213'666 (0'0,215'668] local-les=206 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=215'668 lcod 0'0 inactive NOTIFY m=1] exit Started/Stray 1.482727 8 0.000342 2014-06-09 16:22:11.643011 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.643015 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.643016 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 213'666 (0'0,215'668] local-les=206 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=215'668 lcod 0'0 inactive NOTIFY m=1] enter Started/ReplicaActive 2014-06-09 16:22:11.643026 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.643028 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 213'666 (0'0,215'668] local-les=206 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=215'668 lcod 0'0 inactive NOTIFY m=1] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.643039 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x4eb8700] 2014-06-09 16:22:11.643041 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.643042 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.643043 7f8692767700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'637, trimmed: 2014-06-09 16:22:11.643038 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 213'666 (0'0,215'668] local-les=206 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=215'668 lcod 0'0 inactive NOTIFY m=1] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.643049 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 213'666 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=215'668 lcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.643065 7f8692767700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.640281 7f868cf5c700 10 op_submit_start 3355 2014-06-09 16:22:11.643074 7f868cf5c700 5 queue_transactions (trailing journal) 3355 2014-06-09 16:22:11.643076 7f868cf5c700 5 queue_op 0x30d96720 seq 3355 osr(0.4c8 0x2bd7670) 1883 bytes (queue has 50 ops and 84700 bytes) 2014-06-09 16:22:11.643080 7f868cf5c700 10 op_submit_finish 3355 2014-06-09 16:22:11.643085 7f868df5e700 10 op_submit_start 3356 2014-06-09 16:22:11.643087 7f868df5e700 5 queue_transactions (trailing journal) 3356 2014-06-09 16:22:11.643089 7f868df5e700 5 queue_op 0x2fcae1e0 seq 3356 osr(3.76f 0x3ed1e20) 1429 bytes (queue has 50 ops and 84700 bytes) 2014-06-09 16:22:11.643092 7f868df5e700 10 op_submit_finish 3356 2014-06-09 16:22:11.643086 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.643095 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.643100 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.643108 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.643107 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.643115 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.643117 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.643128 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.643125 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.643128 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=215'668 lcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.643134 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.643144 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.643156 7f8691765700 10 write_log with: dirty_to: 0'0, dirty_from: 213'666, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.643158 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.643163 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.643141 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.643180 7f8691765700 5 queue_transactions existing osr(3.27 0x3ed0340)/0x3ed0340 2014-06-09 16:22:11.643175 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.643184 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.643191 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.643205 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.643211 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:179 2014-06-09 16:22:11.643219 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:181 2014-06-09 16:22:11.643222 7f868df5e700 20 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.643232 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.643229 7f868df5e700 20 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.643237 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.643236 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=208 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.753( v 213'594 (0'0,213'594] local-les=243 n=110 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.643248 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=208 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.753( v 213'594 (0'0,213'594] local-les=243 n=110 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.643256 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=208 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.911146 7 0.000178 2014-06-09 16:22:11.643265 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=208 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.643272 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=208 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.643279 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=208 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.643285 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=243 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'594 -> 213'594 2014-06-09 16:22:11.643318 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=243 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.643354 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.640289 7f868e75f700 2 waiting 51 > 50 ops || 86709 > 104857600 2014-06-09 16:22:11.640308 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.643385 7f868af58700 2 waiting 51 > 50 ops || 86709 > 104857600 2014-06-09 16:22:11.643392 7f8690763700 2 waiting 51 > 50 ops || 86654 > 104857600 2014-06-09 16:22:11.643397 7f868d75d700 2 waiting 51 > 50 ops || 86654 > 104857600 2014-06-09 16:22:11.643401 7f868c75b700 2 waiting 51 > 50 ops || 86654 > 104857600 2014-06-09 16:22:11.643400 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3307.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.643405 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.643405 7f8691f66700 2 waiting 51 > 50 ops || 86540 > 104857600 2014-06-09 16:22:11.643409 7f8689755700 2 waiting 51 > 50 ops || 86035 > 104857600 2014-06-09 16:22:11.643414 7f8690f64700 2 waiting 51 > 50 ops || 86580 > 104857600 2014-06-09 16:22:11.643418 7f868b759700 2 waiting 51 > 50 ops || 86799 > 104857600 2014-06-09 16:22:11.643422 7f868bf5a700 2 waiting 51 > 50 ops || 86035 > 104857600 2014-06-09 16:22:11.643427 7f868ff62700 2 waiting 51 > 50 ops || 86577 > 104857600 2014-06-09 16:22:11.643432 7f868f761700 2 waiting 51 > 50 ops || 86583 > 104857600 2014-06-09 16:22:11.640296 7f869af78700 5 -- op tracker -- , seq: 2332, time: 2014-06-09 16:22:11.639939, event: waiting_for_osdmap, op: pg_notify(3.1c7(12),2.755(21) epoch 243) 2014-06-09 16:22:11.643439 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.643439 7f8692f68700 2 waiting 51 > 50 ops || 86035 > 104857600 2014-06-09 16:22:11.643442 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd5dc0 2014-06-09 16:22:11.643449 7f869af78700 5 -- op tracker -- , seq: 2332, time: 2014-06-09 16:22:11.643446, event: started, op: pg_notify(3.1c7(12),2.755(21) epoch 243) 2014-06-09 16:22:11.643481 7f869af78700 5 -- op tracker -- , seq: 2332, time: 2014-06-09 16:22:11.643478, event: done, op: pg_notify(3.1c7(12),2.755(21) epoch 243) 2014-06-09 16:22:11.643492 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.643494 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.643503 7f868cf5c700 5 queue_transactions existing osr(3.753 0x3ed1e50)/0x3ed1e50 2014-06-09 16:22:11.643506 7f868cf5c700 2 waiting 51 > 50 ops || 86035 > 104857600 2014-06-09 16:22:11.643501 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.6b4( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.643520 7f86a5610700 10 _do_op 0x15996ba0 seq 3307 r = 0, finisher 0x18742140 0 2014-06-09 16:22:11.643526 7f86a5610700 10 _finish_op 0x15996ba0 seq 3307 osr(1.508 0x384db90)/0x384db90 2014-06-09 16:22:11.643521 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.643533 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.643534 7f868e75f700 10 op_submit_start 3357 2014-06-09 16:22:11.643535 7f868e75f700 5 queue_transactions (trailing journal) 3357 2014-06-09 16:22:11.643536 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x18742140] 2014-06-09 16:22:11.643536 7f868e75f700 5 queue_op 0x30d97ec0 seq 3357 osr(1.1fa 0x384cd50) 2009 bytes (queue has 50 ops and 84826 bytes) 2014-06-09 16:22:11.643542 7f8689f56700 2 waiting 51 > 50 ops || 86835 > 104857600 2014-06-09 16:22:11.643543 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x18742140] 2014-06-09 16:22:11.643545 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.643546 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.643547 7f8692767700 2 waiting 51 > 50 ops || 1049774 > 104857600 2014-06-09 16:22:11.643538 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.6bb( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.6bb( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.643529 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.643554 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.643553 7f8691765700 2 waiting 51 > 50 ops || 87135 > 104857600 2014-06-09 16:22:11.643552 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.6bb( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.643560 7f868af58700 2 waiting 51 > 50 ops || 86835 > 104857600 2014-06-09 16:22:11.643565 7f8690763700 2 waiting 51 > 50 ops || 86780 > 104857600 2014-06-09 16:22:11.643569 7f868d75d700 2 waiting 51 > 50 ops || 86780 > 104857600 2014-06-09 16:22:11.643569 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f440 (1 active) 2014-06-09 16:22:11.643573 7f868c75b700 2 waiting 51 > 50 ops || 86780 > 104857600 2014-06-09 16:22:11.643573 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781b00 (2 active) 2014-06-09 16:22:11.643576 7f8691f66700 2 waiting 51 > 50 ops || 86666 > 104857600 2014-06-09 16:22:11.643577 7f86a5610700 5 _do_op 0x30d96900 seq 3309 osr(1.1e8 0x384cdf0)/0x384cdf0 start 2014-06-09 16:22:11.643580 7f86a5610700 10 _do_transaction on 0xc1f7500 2014-06-09 16:22:11.643581 7f868e75f700 10 op_submit_finish 3357 2014-06-09 16:22:11.643581 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.643581 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.643587 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.643584 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.643587 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.3f9( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.643600 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3308.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.643605 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.643599 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.643601 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.643611 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643617 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.643622 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.6c7( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.643629 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.643634 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.643647 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.643647 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189aa280] 2014-06-09 16:22:11.643643 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.643653 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643651 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.643657 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.643667 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.643651 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.643670 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.643679 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643672 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 2.4a5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.643684 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.643688 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.643683 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.643683 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.643697 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189aa280] 2014-06-09 16:22:11.643699 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.643691 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.777039 7 0.000164 2014-06-09 16:22:11.643700 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.643696 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.643703 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643703 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.643706 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.643713 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.643718 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643715 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.643721 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.643725 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.643727 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.643732 7f86a5e11700 10 _do_op 0x309a2120 seq 3308 r = 0, finisher 0xbd6f7a0 0 2014-06-09 16:22:11.643736 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643734 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.643739 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.643742 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.643751 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.631( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.643760 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643760 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.643790 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.643764 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.643794 7f868df5e700 5 queue_transactions existing osr(2.4a5 0x2ed17f0)/0x2ed17f0 2014-06-09 16:22:11.643794 7f86a5e11700 10 _finish_op 0x309a2120 seq 3308 osr(3.3a8 0x3ed0f80)/0x3ed0f80 2014-06-09 16:22:11.643800 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6cea0] 2014-06-09 16:22:11.643795 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.699( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.643807 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6cea0] 2014-06-09 16:22:11.643809 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.643810 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.643811 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.5( v 214'624 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.643806 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3309.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.643827 7f868e75f700 7 osd.26 pg_epoch: 243 pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.3 2014-06-09 16:22:11.643837 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.643838 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.3 info 3.5( v 214'624 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242), identical to ours 2014-06-09 16:22:11.643857 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643776 7f8689755700 2 waiting 51 > 50 ops || 86161 > 104857600 2014-06-09 16:22:11.643866 7f8690f64700 2 waiting 51 > 50 ops || 86706 > 104857600 2014-06-09 16:22:11.643871 7f868b759700 2 waiting 51 > 50 ops || 86925 > 104857600 2014-06-09 16:22:11.643871 7f86a5610700 10 _do_op 0x30d96900 seq 3309 r = 0, finisher 0x189aa640 0 2014-06-09 16:22:11.643879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3e100] 2014-06-09 16:22:11.643883 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3e100] 2014-06-09 16:22:11.643885 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.643886 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.643877 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.4d1( v 213'606 (0'0,213'606] local-les=201 n=120 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.643892 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.643903 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.643909 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.643920 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 2.4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.643932 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.643937 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.643940 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.452883 7 0.000185 2014-06-09 16:22:11.643953 7f868a757700 2 waiting 51 > 50 ops || 86161 > 104857600 2014-06-09 16:22:11.643951 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.643961 7f868ff62700 2 waiting 51 > 50 ops || 86703 > 104857600 2014-06-09 16:22:11.643966 7f868f761700 2 waiting 51 > 50 ops || 86709 > 104857600 2014-06-09 16:22:11.643962 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.643972 7f8692f68700 2 waiting 51 > 50 ops || 86161 > 104857600 2014-06-09 16:22:11.643971 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.643979 7f868cf5c700 2 waiting 51 > 50 ops || 86161 > 104857600 2014-06-09 16:22:11.643977 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.643983 7f868df5e700 2 waiting 51 > 50 ops || 86709 > 104857600 2014-06-09 16:22:11.643982 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.643997 7f8692767700 10 op_submit_start 3358 2014-06-09 16:22:11.643997 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e240 (1 active) 2014-06-09 16:22:11.643999 7f8692767700 5 queue_transactions (trailing journal) 3358 2014-06-09 16:22:11.643999 7f8691765700 2 waiting 51 > 50 ops || 1050748 > 104857600 2014-06-09 16:22:11.644000 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4adb00 (2 active) 2014-06-09 16:22:11.644005 7f86a5e11700 5 _do_op 0x2fcaeea0 seq 3310 osr(0.71b 0x384c050)/0x384c050 start 2014-06-09 16:22:11.644008 7f86a5e11700 10 _do_transaction on 0xc1f6000 2014-06-09 16:22:11.644001 7f8692767700 5 queue_op 0x30d96ea0 seq 3358 osr(3.60 0x3ed02e0) 964948 bytes (queue has 50 ops and 1048439 bytes) 2014-06-09 16:22:11.644012 7f8692767700 10 op_submit_finish 3358 2014-06-09 16:22:11.644015 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.644018 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.644023 7f868e75f700 5 queue_transactions existing osr(2.4 0x2ed0310)/0x2ed0310 2014-06-09 16:22:11.644026 7f868e75f700 2 waiting 51 > 50 ops || 1050316 > 104857600 2014-06-09 16:22:11.644030 7f86a5610700 10 _finish_op 0x30d96900 seq 3309 osr(1.1e8 0x384cdf0)/0x384cdf0 2014-06-09 16:22:11.644032 7f868bf5a700 2 waiting 51 > 50 ops || 1049774 > 104857600 2014-06-09 16:22:11.644016 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc295140 243'637 rep_tid=1 committed?=0 applied?=0 lock=0 op=osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.644038 7f8692767700 10 osd.26 243 dequeue_op 0x230da200 finish 2014-06-09 16:22:11.644039 7f868d75d700 10 op_submit_start 3359 2014-06-09 16:22:11.644039 7f8691765700 2 waiting 51 > 50 ops || 1050819 > 104857600 2014-06-09 16:22:11.644040 7f868d75d700 5 queue_transactions (trailing journal) 3359 2014-06-09 16:22:11.644039 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781b00 (1 active) 2014-06-09 16:22:11.644042 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3386c0 (2 active) 2014-06-09 16:22:11.644042 7f868d75d700 5 queue_op 0x2fc8ca20 seq 3359 osr(0.48b 0x2bd7450) 1954 bytes (queue has 50 ops and 1048510 bytes) 2014-06-09 16:22:11.644045 7f868d75d700 10 op_submit_finish 3359 2014-06-09 16:22:11.644044 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.644044 7f86a5610700 5 _do_op 0x309a36e0 seq 3311 osr(2.24d 0x2ed0a50)/0x2ed0a50 start 2014-06-09 16:22:11.644045 7f8689755700 2 waiting 51 > 50 ops || 1049845 > 104857600 2014-06-09 16:22:11.644047 7f86a5610700 10 _do_transaction on 0xc0ddb00 2014-06-09 16:22:11.644050 7f8690f64700 2 waiting 51 > 50 ops || 1050390 > 104857600 2014-06-09 16:22:11.644050 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.644053 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.644054 7f868b759700 2 waiting 51 > 50 ops || 1050609 > 104857600 2014-06-09 16:22:11.644048 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.644060 7f868a757700 2 waiting 51 > 50 ops || 1049845 > 104857600 2014-06-09 16:22:11.644059 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.529( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.644067 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3310.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.644072 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.644071 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.644074 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.644084 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f23c0 (20 active) 2014-06-09 16:22:11.644078 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.529( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.164731 7 0.000190 2014-06-09 16:22:11.644095 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6f7a0,0x189aa640] 2014-06-09 16:22:11.644089 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.529( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.644089 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644102 7f868c75b700 2 waiting 51 > 50 ops || 1050464 > 104857600 2014-06-09 16:22:11.644101 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.529( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.644108 7f868f761700 2 waiting 51 > 50 ops || 1050393 > 104857600 2014-06-09 16:22:11.644099 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.644103 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644114 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644110 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.644116 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.644119 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.644119 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644129 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.644131 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.644142 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.65( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644153 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644149 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.644161 7f86a5e11700 10 _do_op 0x2fcaeea0 seq 3310 r = 0, finisher 0xbfcf240 0 2014-06-09 16:22:11.644162 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.644166 7f86a5e11700 10 _finish_op 0x2fcaeea0 seq 3310 osr(0.71b 0x384c050)/0x384c050 2014-06-09 16:22:11.644158 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.2b( v 213'613 (0'0,213'613] local-les=191 n=121 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.644170 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4adb00 (1 active) 2014-06-09 16:22:11.644172 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d560 (2 active) 2014-06-09 16:22:11.644174 7f8691765700 10 op_submit_start 3360 2014-06-09 16:22:11.644175 7f86a5e11700 5 _do_op 0x2fcaf8c0 seq 3312 osr(3.381 0x3ed0fe0)/0x3ed0fe0 start 2014-06-09 16:22:11.644176 7f8691765700 5 queue_transactions (trailing journal) 3360 2014-06-09 16:22:11.644177 7f86a5e11700 10 _do_transaction on 0x394eb40 2014-06-09 16:22:11.644178 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.644177 7f8691765700 5 queue_op 0x308cb2c0 seq 3360 osr(3.27 0x3ed0340) 2309 bytes (queue has 50 ops and 1049198 bytes) 2014-06-09 16:22:11.644180 7f8691765700 10 op_submit_finish 3360 2014-06-09 16:22:11.644171 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.644175 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.644183 7f868d75d700 5 queue_transactions existing osr(0.529 0x2bd7930)/0x2bd7930 2014-06-09 16:22:11.644183 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.644184 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644185 7f868d75d700 2 waiting 51 > 50 ops || 1051152 > 104857600 2014-06-09 16:22:11.644191 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6f7a0,0x189aa640] 2014-06-09 16:22:11.644183 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.29a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.644193 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfcf240] 2014-06-09 16:22:11.644196 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfcf240] 2014-06-09 16:22:11.644197 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.644189 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644198 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.644196 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.644206 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.644209 7f8691f66700 2 waiting 51 > 50 ops || 1051038 > 104857600 2014-06-09 16:22:11.644218 7f8690763700 2 waiting 51 > 50 ops || 1051152 > 104857600 2014-06-09 16:22:11.644211 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.359( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.644214 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644223 7f8689f56700 2 waiting 51 > 50 ops || 1051207 > 104857600 2014-06-09 16:22:11.643556 7f869af78700 20 osd.26 243 _dispatch 0x308e3500 pg_info(2 pgs e243:3.50d(2),3.516(2)) v4 2014-06-09 16:22:11.644226 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644228 7f868e75f700 2 waiting 51 > 50 ops || 1051075 > 104857600 2014-06-09 16:22:11.644223 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.644234 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.644234 7f868bf5a700 2 waiting 51 > 50 ops || 1050533 > 104857600 2014-06-09 16:22:11.644239 7f8689755700 2 waiting 51 > 50 ops || 1050533 > 104857600 2014-06-09 16:22:11.644231 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644243 7f8690f64700 2 waiting 51 > 50 ops || 1051078 > 104857600 2014-06-09 16:22:11.644239 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.623( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.644244 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.644247 7f868b759700 2 waiting 51 > 50 ops || 1051297 > 104857600 2014-06-09 16:22:11.644247 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.644251 7f868a757700 2 waiting 51 > 50 ops || 1050533 > 104857600 2014-06-09 16:22:11.644253 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.644243 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x230dbf00,0x230dbe00,0x230dbd00,0x230dbc00,0x230dbb00,0x270be900 2014-06-09 16:22:11.644261 7f869af78700 5 -- op tracker -- , seq: 2333, time: 2014-06-09 16:22:11.327827, event: header_read, op: pg_info(2 pgs e243:3.50d(2),3.516(2)) 2014-06-09 16:22:11.644266 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0230 (19 active) 2014-06-09 16:22:11.644271 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.644271 7f869af78700 5 -- op tracker -- , seq: 2333, time: 2014-06-09 16:22:11.327827, event: throttled, op: pg_info(2 pgs e243:3.50d(2),3.516(2)) 2014-06-09 16:22:11.644277 7f868cf5c700 2 waiting 51 > 50 ops || 1050533 > 104857600 2014-06-09 16:22:11.644279 7f869af78700 5 -- op tracker -- , seq: 2333, time: 2014-06-09 16:22:11.327841, event: all_read, op: pg_info(2 pgs e243:3.50d(2),3.516(2)) 2014-06-09 16:22:11.644282 7f868f761700 2 waiting 51 > 50 ops || 1051081 > 104857600 2014-06-09 16:22:11.644288 7f868af58700 2 waiting 51 > 50 ops || 1051207 > 104857600 2014-06-09 16:22:11.644289 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3312.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.644293 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.644294 7f8692f68700 2 waiting 51 > 50 ops || 1050533 > 104857600 2014-06-09 16:22:11.644266 7f868ef60700 10 osd.26 243 dequeue_op 0x1b0b7200 prio 63 cost 217088 latency 0.911317 osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.644286 7f869af78700 5 -- op tracker -- , seq: 2333, time: 2014-06-09 16:22:11.643529, event: dispatched, op: pg_info(2 pgs e243:3.50d(2),3.516(2)) 2014-06-09 16:22:11.644307 7f868df5e700 2 waiting 51 > 50 ops || 1051081 > 104857600 2014-06-09 16:22:11.644311 7f868ff62700 2 waiting 51 > 50 ops || 1051075 > 104857600 2014-06-09 16:22:11.644303 7f868ef60700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.644298, event: reached_pg, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.644314 7f86a5e11700 10 _do_op 0x2fcaf8c0 seq 3312 r = 0, finisher 0x306ce720 0 2014-06-09 16:22:11.644318 7f86a5e11700 10 _finish_op 0x2fcaf8c0 seq 3312 osr(3.381 0x3ed0fe0)/0x3ed0fe0 2014-06-09 16:22:11.644326 7f868d75d700 10 op_submit_start 3361 2014-06-09 16:22:11.644327 7f868d75d700 5 queue_transactions (trailing journal) 3361 2014-06-09 16:22:11.644329 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644328 7f868d75d700 5 queue_op 0x2fc8cd20 seq 3361 osr(0.529 0x2bd7930) 1954 bytes (queue has 50 ops and 1049817 bytes) 2014-06-09 16:22:11.644333 7f8691f66700 2 waiting 51 > 50 ops || 1051657 > 104857600 2014-06-09 16:22:11.644338 7f8690763700 2 waiting 51 > 50 ops || 1051771 > 104857600 2014-06-09 16:22:11.644326 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.644334 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644344 7f869af78700 5 -- op tracker -- , seq: 2333, time: 2014-06-09 16:22:11.644298, event: waiting_for_osdmap, op: pg_info(2 pgs e243:3.50d(2),3.516(2)) 2014-06-09 16:22:11.644351 7f869af78700 7 osd.26 243 handle_pg_info pg_info(2 pgs e243:3.50d(2),3.516(2)) v4 from osd.4 2014-06-09 16:22:11.644355 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e3500 2014-06-09 16:22:11.644346 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.298( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644342 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1b0b7200 2014-06-09 16:22:11.644370 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644362 7f869af78700 5 -- op tracker -- , seq: 2333, time: 2014-06-09 16:22:11.644359, event: started, op: pg_info(2 pgs e243:3.50d(2),3.516(2)) 2014-06-09 16:22:11.644374 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644370 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.644386 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.644( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644394 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644396 7f868ef60700 15 getattr 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 '_' 2014-06-09 16:22:11.644398 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644400 7f869af78700 5 -- op tracker -- , seq: 2333, time: 2014-06-09 16:22:11.644397, event: done, op: pg_info(2 pgs e243:3.50d(2),3.516(2)) 2014-06-09 16:22:11.644410 7f868c75b700 2 waiting 51 > 50 ops || 1051771 > 104857600 2014-06-09 16:22:11.644413 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.644408 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644417 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644416 7f868e75f700 2 waiting 51 > 50 ops || 1051694 > 104857600 2014-06-09 16:22:11.644420 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d560 (1 active) 2014-06-09 16:22:11.644421 7f868bf5a700 2 waiting 51 > 50 ops || 1051152 > 104857600 2014-06-09 16:22:11.644422 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dde60 (2 active) 2014-06-09 16:22:11.644425 7f8689755700 2 waiting 51 > 50 ops || 1051152 > 104857600 2014-06-09 16:22:11.644426 7f86a5e11700 5 _do_op 0x30607c20 seq 3313 osr(1.555 0x384dab0)/0x384dab0 start 2014-06-09 16:22:11.644428 7f8690f64700 2 waiting 51 > 50 ops || 1051697 > 104857600 2014-06-09 16:22:11.644429 7f86a5e11700 10 _do_transaction on 0xc228780 2014-06-09 16:22:11.644432 7f868b759700 2 waiting 51 > 50 ops || 1051916 > 104857600 2014-06-09 16:22:11.644437 7f868d75d700 10 op_submit_finish 3361 2014-06-09 16:22:11.644436 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.644421 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.644442 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd71b40,0x306ce740] 2014-06-09 16:22:11.644440 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.521( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.644443 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 1.1cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.644456 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.644459 7f868ef60700 10 getattr got.size() is 0 2014-06-09 16:22:11.644460 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.644453 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.644464 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.644463 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.758873 7 0.000190 2014-06-09 16:22:11.644462 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 but can_create 2014-06-09 16:22:11.644476 7f868ef60700 15 getattr 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 'snapset' 2014-06-09 16:22:11.644469 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:0.70f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.644445 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.644475 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.644486 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.644481 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.644491 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.644486 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.644495 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.644497 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.644505 7f868ef60700 10 getattr got.size() is 0 2014-06-09 16:22:11.644502 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] flushed 2014-06-09 16:22:11.644508 7f868ef60700 15 getattr 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/snapdir//3 'snapset' 2014-06-09 16:22:11.644505 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.624( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.644514 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd71b40,0x306ce740] 2014-06-09 16:22:11.644516 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.644517 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.644516 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.644519 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.644522 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.624( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.330177 7 0.000174 2014-06-09 16:22:11.644507 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.644534 7f868ef60700 10 getattr got.size() is 0 2014-06-09 16:22:11.644533 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.624( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.644533 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.644545 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.624( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.644542 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x305c6b00 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.644555 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.644555 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.644565 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.644566 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.644574 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.644583 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.644588 7f8692767700 5 queue_transactions existing osr(1.1cf 0x384ca70)/0x384ca70 2014-06-09 16:22:11.644577 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x305c6b00 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 rwstate(none n=0 w=0) oi: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2e8775e0 snapset: 0=[]:[] 2014-06-09 16:22:11.644593 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 @head oi=5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.644609 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.644612 7f868d75d700 5 queue_transactions existing osr(2.624 0x2ed1c50)/0x2ed1c50 2014-06-09 16:22:11.644612 7f868ef60700 15 getattr 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/snapdir//3 '_' 2014-06-09 16:22:11.644617 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.644632 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.644633 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.644635 7f869af78700 20 osd.26 243 _dispatch 0x31052fc0 pg_info(1 pgs e243:3.21b) v4 2014-06-09 16:22:11.644640 7f868ef60700 10 getattr got.size() is 0 2014-06-09 16:22:11.644639 7f869af78700 5 -- op tracker -- , seq: 2334, time: 2014-06-09 16:22:11.380503, event: header_read, op: pg_info(1 pgs e243:3.21b) 2014-06-09 16:22:11.644643 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 5a3860/rbd_data.117c6b8b4567.0000000000001472/snapdir//3 and !can_create 2014-06-09 16:22:11.644591 7f8692767700 2 waiting 51 > 50 ops || 1051791 > 104857600 2014-06-09 16:22:11.644652 7f869af78700 5 -- op tracker -- , seq: 2334, time: 2014-06-09 16:22:11.380503, event: throttled, op: pg_info(1 pgs e243:3.21b) 2014-06-09 16:22:11.644659 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306ce720] 2014-06-09 16:22:11.644659 7f868cf5c700 2 waiting 51 > 50 ops || 1051152 > 104857600 2014-06-09 16:22:11.644663 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.644671 7f868ef60700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.644656, event: started, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.644678 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.644680 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x14dcd000 2014-06-09 16:22:11.644694 7f868ef60700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.644691, event: started, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.644694 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.644712 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306ce720] 2014-06-09 16:22:11.644699 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] ov 0'0 av 243'638 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.644714 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.644715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.644715 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 2014-06-09 16:22:11.644725 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.644735 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~217088 2014-06-09 16:22:11.644745 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229400 2014-06-09 16:22:11.644751 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 snapset=0x2e877618 snapc=0=[] 2014-06-09 16:22:11.644757 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.644762 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.644768 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 0x14dcd000 op modify 2014-06-09 16:22:11.644780 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.644776 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:18.299725 2014-06-09 16:22:11.644799 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3313.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.644804 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.644797 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=125 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.644817 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.644830 7f86a5e11700 10 _do_op 0x30607c20 seq 3313 r = 0, finisher 0xc6e8180 0 2014-06-09 16:22:11.644837 7f86a5e11700 10 _finish_op 0x30607c20 seq 3313 osr(1.555 0x384dab0)/0x384dab0 2014-06-09 16:22:11.644828 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 2 on osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.644847 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dde60 (1 active) 2014-06-09 16:22:11.644849 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e480 (2 active) 2014-06-09 16:22:11.644852 7f86a5e11700 5 _do_op 0x30d97f80 seq 3314 osr(0.3a9 0x2bd7310)/0x2bd7310 start 2014-06-09 16:22:11.644843 7f868ef60700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 2 o 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.644855 7f86a5e11700 10 _do_transaction on 0xbfb29c0 2014-06-09 16:22:11.644861 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.644863 7f868ef60700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.644860, event: waiting for subops from 7, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.644867 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8180] 2014-06-09 16:22:11.644869 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.644873 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8180] 2014-06-09 16:22:11.644874 7f868ef60700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.644875 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.644876 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.644882 7f8689f56700 10 op_submit_start 3362 2014-06-09 16:22:11.644883 7f8689f56700 5 queue_transactions (trailing journal) 3362 2014-06-09 16:22:11.644884 7f868af58700 2 waiting 51 > 50 ops || 1051952 > 104857600 2014-06-09 16:22:11.644885 7f8689f56700 5 queue_op 0x30d97320 seq 3362 osr(0.2a2 0x2bd6c20) 2009 bytes (queue has 50 ops and 1049943 bytes) 2014-06-09 16:22:11.644888 7f8689f56700 10 op_submit_finish 3362 2014-06-09 16:22:11.644891 7f8692f68700 2 waiting 51 > 50 ops || 1051278 > 104857600 2014-06-09 16:22:11.644889 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3311.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.644894 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.644896 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (19 active) 2014-06-09 16:22:11.644888 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'637 (0'0,243'637] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'637], crt=0'0) [243'638 (0'0) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30632 2014-06-09 10:20:18.299725] 2014-06-09 16:22:11.644917 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (20 active) 2014-06-09 16:22:11.644895 7f868df5e700 2 waiting 51 > 50 ops || 1051826 > 104857600 2014-06-09 16:22:11.644910 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'638 (0'0) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30632 2014-06-09 10:20:18.299725 2014-06-09 16:22:11.644925 7f868ff62700 2 waiting 51 > 50 ops || 1051820 > 104857600 2014-06-09 16:22:11.644924 7f86a5610700 10 _do_op 0x309a36e0 seq 3311 r = 0, finisher 0x139649a0 0 2014-06-09 16:22:11.644930 7f868d75d700 2 waiting 51 > 50 ops || 1051846 > 104857600 2014-06-09 16:22:11.644921 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644934 7f868a757700 2 waiting 51 > 50 ops || 1051278 > 104857600 2014-06-09 16:22:11.644938 7f8690763700 2 waiting 51 > 50 ops || 1051897 > 104857600 2014-06-09 16:22:11.644932 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.644941 7f868c75b700 2 waiting 51 > 50 ops || 1051897 > 104857600 2014-06-09 16:22:11.644935 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644928 7f86a5610700 10 _finish_op 0x309a36e0 seq 3311 osr(2.24d 0x2ed0a50)/0x2ed0a50 2014-06-09 16:22:11.644945 7f868e75f700 2 waiting 51 > 50 ops || 1051820 > 104857600 2014-06-09 16:22:11.644946 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.644949 7f868bf5a700 2 waiting 51 > 50 ops || 1051278 > 104857600 2014-06-09 16:22:11.644953 7f8689755700 2 waiting 51 > 50 ops || 1051278 > 104857600 2014-06-09 16:22:11.644957 7f8690f64700 2 waiting 51 > 50 ops || 1051823 > 104857600 2014-06-09 16:22:11.644951 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644961 7f868b759700 2 waiting 51 > 50 ops || 1052042 > 104857600 2014-06-09 16:22:11.644961 7f868ef60700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'638, trimmed: 2014-06-09 16:22:11.644965 7f8692767700 2 waiting 51 > 50 ops || 1051917 > 104857600 2014-06-09 16:22:11.644963 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644970 7f868cf5c700 2 waiting 51 > 50 ops || 1051278 > 104857600 2014-06-09 16:22:11.644972 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.644980 7f868ef60700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.644983 7f868ef60700 2 waiting 51 > 50 ops || 1269419 > 104857600 2014-06-09 16:22:11.644976 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.644988 7f8691f66700 2 waiting 51 > 50 ops || 1051783 > 104857600 2014-06-09 16:22:11.644988 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.644998 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.644995 7f869af78700 5 -- op tracker -- , seq: 2334, time: 2014-06-09 16:22:11.380512, event: all_read, op: pg_info(1 pgs e243:3.21b) 2014-06-09 16:22:11.645002 7f869af78700 5 -- op tracker -- , seq: 2334, time: 2014-06-09 16:22:11.644631, event: dispatched, op: pg_info(1 pgs e243:3.21b) 2014-06-09 16:22:11.645010 7f869af78700 5 -- op tracker -- , seq: 2334, time: 2014-06-09 16:22:11.645008, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.21b) 2014-06-09 16:22:11.645016 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.21b) v4 from osd.7 2014-06-09 16:22:11.645019 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31052fc0 2014-06-09 16:22:11.645002 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.645025 7f869af78700 5 -- op tracker -- , seq: 2334, time: 2014-06-09 16:22:11.645023, event: started, op: pg_info(1 pgs e243:3.21b) 2014-06-09 16:22:11.645027 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.645038 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645042 7f869af78700 5 -- op tracker -- , seq: 2334, time: 2014-06-09 16:22:11.645040, event: done, op: pg_info(1 pgs e243:3.21b) 2014-06-09 16:22:11.645048 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645042 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.591( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.645058 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645059 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645060 7f869af78700 20 osd.26 243 _dispatch 0x2eef7180 pg_info(1 pgs e243:1.303) v4 2014-06-09 16:22:11.645054 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.645063 7f869af78700 5 -- op tracker -- , seq: 2335, time: 2014-06-09 16:22:11.436936, event: header_read, op: pg_info(1 pgs e243:1.303) 2014-06-09 16:22:11.645065 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645067 7f869af78700 5 -- op tracker -- , seq: 2335, time: 2014-06-09 16:22:11.436937, event: throttled, op: pg_info(1 pgs e243:1.303) 2014-06-09 16:22:11.645070 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.587( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.645077 7f869af78700 5 -- op tracker -- , seq: 2335, time: 2014-06-09 16:22:11.436955, event: all_read, op: pg_info(1 pgs e243:1.303) 2014-06-09 16:22:11.645081 7f869af78700 5 -- op tracker -- , seq: 2335, time: 2014-06-09 16:22:11.645058, event: dispatched, op: pg_info(1 pgs e243:1.303) 2014-06-09 16:22:11.645085 7f869af78700 5 -- op tracker -- , seq: 2335, time: 2014-06-09 16:22:11.645084, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.303) 2014-06-09 16:22:11.645087 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.303) v4 from osd.3 2014-06-09 16:22:11.645089 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef7180 2014-06-09 16:22:11.645081 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.645091 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645092 7f869af78700 5 -- op tracker -- , seq: 2335, time: 2014-06-09 16:22:11.645091, event: started, op: pg_info(1 pgs e243:1.303) 2014-06-09 16:22:11.645096 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:0.594( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.645101 7f869af78700 5 -- op tracker -- , seq: 2335, time: 2014-06-09 16:22:11.645100, event: done, op: pg_info(1 pgs e243:1.303) 2014-06-09 16:22:11.645108 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645109 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645108 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.645116 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645117 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645118 7f869af78700 20 osd.26 243 _dispatch 0x19fdf340 pg_info(1 pgs e243:0.403) v4 2014-06-09 16:22:11.645118 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645120 7f869af78700 5 -- op tracker -- , seq: 2336, time: 2014-06-09 16:22:11.311217, event: header_read, op: pg_info(1 pgs e243:0.403) 2014-06-09 16:22:11.645125 7f869af78700 5 -- op tracker -- , seq: 2336, time: 2014-06-09 16:22:11.311217, event: throttled, op: pg_info(1 pgs e243:0.403) 2014-06-09 16:22:11.645122 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.645130 7f869af78700 5 -- op tracker -- , seq: 2336, time: 2014-06-09 16:22:11.311231, event: all_read, op: pg_info(1 pgs e243:0.403) 2014-06-09 16:22:11.645139 7f869af78700 5 -- op tracker -- , seq: 2336, time: 2014-06-09 16:22:11.645116, event: dispatched, op: pg_info(1 pgs e243:0.403) 2014-06-09 16:22:11.645135 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.645147 7f869af78700 5 -- op tracker -- , seq: 2336, time: 2014-06-09 16:22:11.645144, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.403) 2014-06-09 16:22:11.645145 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.645151 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.403) v4 from osd.1 2014-06-09 16:22:11.645154 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdf340 2014-06-09 16:22:11.645157 7f869af78700 5 -- op tracker -- , seq: 2336, time: 2014-06-09 16:22:11.645156, event: started, op: pg_info(1 pgs e243:0.403) 2014-06-09 16:22:11.645153 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.645167 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.645169 7f869af78700 5 -- op tracker -- , seq: 2336, time: 2014-06-09 16:22:11.645167, event: done, op: pg_info(1 pgs e243:0.403) 2014-06-09 16:22:11.645178 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645181 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645180 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.645195 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645195 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645197 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645198 7f869af78700 20 osd.26 243 _dispatch 0x240f3800 pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.645200 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:2.75d( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.645205 7f869af78700 5 -- op tracker -- , seq: 2337, time: 2014-06-09 16:22:11.363811, event: header_read, op: pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.645217 7f869af78700 5 -- op tracker -- , seq: 2337, time: 2014-06-09 16:22:11.363811, event: throttled, op: pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.645213 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.645223 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645224 7f869af78700 5 -- op tracker -- , seq: 2337, time: 2014-06-09 16:22:11.363831, event: all_read, op: pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.645230 7f869af78700 5 -- op tracker -- , seq: 2337, time: 2014-06-09 16:22:11.645195, event: dispatched, op: pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.645234 7f869af78700 5 -- op tracker -- , seq: 2337, time: 2014-06-09 16:22:11.645233, event: waiting_for_osdmap, op: pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.645228 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.498( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.645237 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) v4 from osd.9 2014-06-09 16:22:11.645240 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240f3800 2014-06-09 16:22:11.645240 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.645244 7f869af78700 5 -- op tracker -- , seq: 2337, time: 2014-06-09 16:22:11.645242, event: started, op: pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.645241 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.645251 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645258 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3314.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.645263 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.645255 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.645259 7f869af78700 5 -- op tracker -- , seq: 2337, time: 2014-06-09 16:22:11.645255, event: done, op: pg_log(3.2a5 epoch 243 log log((214'632,214'635], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.645269 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645272 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645266 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.48f( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.645285 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645278 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.645287 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645288 7f869af78700 20 osd.26 243 _dispatch 0x3104ae00 pg_notify(2.442(20),2.185(20) epoch 243) v5 2014-06-09 16:22:11.645288 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.645294 7f86a5e11700 10 _do_op 0x30d97f80 seq 3314 r = 0, finisher 0xbebfd80 0 2014-06-09 16:22:11.645294 7f869af78700 5 -- op tracker -- , seq: 2338, time: 2014-06-09 16:22:11.376540, event: header_read, op: pg_notify(2.442(20),2.185(20) epoch 243) 2014-06-09 16:22:11.645292 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.812887 7 0.000209 2014-06-09 16:22:11.645304 7f869af78700 5 -- op tracker -- , seq: 2338, time: 2014-06-09 16:22:11.376541, event: throttled, op: pg_notify(2.442(20),2.185(20) epoch 243) 2014-06-09 16:22:11.645304 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.645313 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3386c0 (1 active) 2014-06-09 16:22:11.645312 7f869af78700 5 -- op tracker -- , seq: 2338, time: 2014-06-09 16:22:11.376592, event: all_read, op: pg_notify(2.442(20),2.185(20) epoch 243) 2014-06-09 16:22:11.645316 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc480 (2 active) 2014-06-09 16:22:11.645314 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.645320 7f86a5610700 5 _do_op 0x308cb1a0 seq 3315 osr(2.101 0x2ed0730)/0x2ed0730 start 2014-06-09 16:22:11.645323 7f86a5610700 10 _do_transaction on 0xbfcba40 2014-06-09 16:22:11.645319 7f869af78700 5 -- op tracker -- , seq: 2338, time: 2014-06-09 16:22:11.645285, event: dispatched, op: pg_notify(2.442(20),2.185(20) epoch 243) 2014-06-09 16:22:11.645330 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.645329 7f869af78700 5 -- op tracker -- , seq: 2338, time: 2014-06-09 16:22:11.645326, event: waiting_for_osdmap, op: pg_notify(2.442(20),2.185(20) epoch 243) 2014-06-09 16:22:11.645333 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.645335 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104ae00 2014-06-09 16:22:11.645324 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.645338 7f869af78700 5 -- op tracker -- , seq: 2338, time: 2014-06-09 16:22:11.645337, event: started, op: pg_notify(2.442(20),2.185(20) epoch 243) 2014-06-09 16:22:11.645340 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.645353 7f869af78700 5 -- op tracker -- , seq: 2338, time: 2014-06-09 16:22:11.645351, event: done, op: pg_notify(2.442(20),2.185(20) epoch 243) 2014-06-09 16:22:11.645349 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.645357 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645358 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645380 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645379 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.645382 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645383 7f869af78700 20 osd.26 243 _dispatch 0x310401c0 pg_info(1 pgs e243:3.544(2)) v4 2014-06-09 16:22:11.645389 7f869af78700 5 -- op tracker -- , seq: 2339, time: 2014-06-09 16:22:11.388529, event: header_read, op: pg_info(1 pgs e243:3.544(2)) 2014-06-09 16:22:11.645398 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645397 7f869af78700 5 -- op tracker -- , seq: 2339, time: 2014-06-09 16:22:11.388530, event: throttled, op: pg_info(1 pgs e243:3.544(2)) 2014-06-09 16:22:11.645397 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3315.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.645402 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.645402 7f869af78700 5 -- op tracker -- , seq: 2339, time: 2014-06-09 16:22:11.388544, event: all_read, op: pg_info(1 pgs e243:3.544(2)) 2014-06-09 16:22:11.645403 7f8689f56700 5 queue_transactions existing osr(2.48f 0x2ed1450)/0x2ed1450 2014-06-09 16:22:11.645406 7f8689f56700 10 op_submit_start 3363 2014-06-09 16:22:11.645405 7f869af78700 5 -- op tracker -- , seq: 2339, time: 2014-06-09 16:22:11.645380, event: dispatched, op: pg_info(1 pgs e243:3.544(2)) 2014-06-09 16:22:11.645407 7f8689f56700 5 queue_transactions (trailing journal) 3363 2014-06-09 16:22:11.645408 7f8689f56700 5 queue_op 0xd471f80 seq 3363 osr(2.48f 0x2ed1450) 1954 bytes (queue has 50 ops and 1049943 bytes) 2014-06-09 16:22:11.645409 7f869af78700 5 -- op tracker -- , seq: 2339, time: 2014-06-09 16:22:11.645408, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.544(2)) 2014-06-09 16:22:11.645411 7f8689f56700 10 op_submit_finish 3363 2014-06-09 16:22:11.645412 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.544(2)) v4 from osd.6 2014-06-09 16:22:11.645414 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310401c0 2014-06-09 16:22:11.645417 7f869af78700 5 -- op tracker -- , seq: 2339, time: 2014-06-09 16:22:11.645416, event: started, op: pg_info(1 pgs e243:3.544(2)) 2014-06-09 16:22:11.645414 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.645427 7f869af78700 5 -- op tracker -- , seq: 2339, time: 2014-06-09 16:22:11.645425, event: done, op: pg_info(1 pgs e243:3.544(2)) 2014-06-09 16:22:11.645427 7f86a5610700 10 _do_op 0x308cb1a0 seq 3315 r = 0, finisher 0x3061ff20 0 2014-06-09 16:22:11.645431 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645431 7f86a5610700 10 _finish_op 0x308cb1a0 seq 3315 osr(2.101 0x2ed0730)/0x2ed0730 2014-06-09 16:22:11.645435 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc480 (1 active) 2014-06-09 16:22:11.645437 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652900 (2 active) 2014-06-09 16:22:11.645428 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.645439 7f86a5610700 5 _do_op 0x30d97f20 seq 3316 osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 start 2014-06-09 16:22:11.645443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645443 7f86a5610700 10 _do_transaction on 0x97ba9c0 2014-06-09 16:22:11.645445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645446 7f869af78700 20 osd.26 243 _dispatch 0x30bd8a80 pg_info(1 pgs e243:3.17) v4 2014-06-09 16:22:11.645448 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.645442 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.858890 7 0.000177 2014-06-09 16:22:11.645451 7f869af78700 5 -- op tracker -- , seq: 2340, time: 2014-06-09 16:22:11.371027, event: header_read, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.645454 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.645460 7f869af78700 5 -- op tracker -- , seq: 2340, time: 2014-06-09 16:22:11.371027, event: throttled, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.645466 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.645466 7f869af78700 5 -- op tracker -- , seq: 2340, time: 2014-06-09 16:22:11.371037, event: all_read, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.645478 7f869af78700 5 -- op tracker -- , seq: 2340, time: 2014-06-09 16:22:11.645442, event: dispatched, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.645476 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.645489 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.645485 7f869af78700 5 -- op tracker -- , seq: 2340, time: 2014-06-09 16:22:11.645483, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.645487 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'652 -> 213'652 2014-06-09 16:22:11.645494 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.17) v4 from osd.0 2014-06-09 16:22:11.645498 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd8a80 2014-06-09 16:22:11.645502 7f869af78700 5 -- op tracker -- , seq: 2340, time: 2014-06-09 16:22:11.645501, event: started, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.645506 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3316.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.645511 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.645513 7f869af78700 5 -- op tracker -- , seq: 2340, time: 2014-06-09 16:22:11.645512, event: done, op: pg_info(1 pgs e243:3.17) 2014-06-09 16:22:11.645518 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645519 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645526 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645527 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645528 7f869af78700 20 osd.26 243 _dispatch 0x310ca380 pg_info(1 pgs e243:0.627) v4 2014-06-09 16:22:11.645530 7f869af78700 5 -- op tracker -- , seq: 2341, time: 2014-06-09 16:22:11.387750, event: header_read, op: pg_info(1 pgs e243:0.627) 2014-06-09 16:22:11.645534 7f869af78700 5 -- op tracker -- , seq: 2341, time: 2014-06-09 16:22:11.387750, event: throttled, op: pg_info(1 pgs e243:0.627) 2014-06-09 16:22:11.645534 7f86a5610700 10 _do_op 0x30d97f20 seq 3316 r = 0, finisher 0x13965a60 0 2014-06-09 16:22:11.645538 7f86a5610700 10 _finish_op 0x30d97f20 seq 3316 osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 2014-06-09 16:22:11.645538 7f869af78700 5 -- op tracker -- , seq: 2341, time: 2014-06-09 16:22:11.387760, event: all_read, op: pg_info(1 pgs e243:0.627) 2014-06-09 16:22:11.645541 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652900 (1 active) 2014-06-09 16:22:11.645544 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cb40 (2 active) 2014-06-09 16:22:11.645542 7f869af78700 5 -- op tracker -- , seq: 2341, time: 2014-06-09 16:22:11.645526, event: dispatched, op: pg_info(1 pgs e243:0.627) 2014-06-09 16:22:11.645536 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.645546 7f86a5610700 5 _do_op 0x308cb140 seq 3317 osr(3.271 0x3ed0bc0)/0x3ed0bc0 start 2014-06-09 16:22:11.645549 7f86a5610700 10 _do_transaction on 0xc2aa240 2014-06-09 16:22:11.645548 7f869af78700 5 -- op tracker -- , seq: 2341, time: 2014-06-09 16:22:11.645547, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.627) 2014-06-09 16:22:11.645550 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.627) v4 from osd.5 2014-06-09 16:22:11.645552 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310ca380 2014-06-09 16:22:11.645554 7f869af78700 5 -- op tracker -- , seq: 2341, time: 2014-06-09 16:22:11.645553, event: started, op: pg_info(1 pgs e243:0.627) 2014-06-09 16:22:11.645564 7f869af78700 5 -- op tracker -- , seq: 2341, time: 2014-06-09 16:22:11.645563, event: done, op: pg_info(1 pgs e243:0.627) 2014-06-09 16:22:11.645568 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645567 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.645569 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645580 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645580 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645582 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645583 7f869af78700 20 osd.26 243 _dispatch 0x31a5a40 pg_notify(3.72b(12) epoch 243) v5 2014-06-09 16:22:11.645584 7f8689f56700 5 queue_transactions existing osr(3.470 0x3ed1540)/0x3ed1540 2014-06-09 16:22:11.645587 7f8689f56700 10 op_submit_start 3364 2014-06-09 16:22:11.645588 7f8689f56700 5 queue_transactions (trailing journal) 3364 2014-06-09 16:22:11.645589 7f8689f56700 5 queue_op 0xd470960 seq 3364 osr(3.470 0x3ed1540) 1335 bytes (queue has 49 ops and 1047386 bytes) 2014-06-09 16:22:11.645592 7f8689f56700 10 op_submit_finish 3364 2014-06-09 16:22:11.645589 7f869af78700 5 -- op tracker -- , seq: 2342, time: 2014-06-09 16:22:11.263609, event: header_read, op: pg_notify(3.72b(12) epoch 243) 2014-06-09 16:22:11.645598 7f869af78700 5 -- op tracker -- , seq: 2342, time: 2014-06-09 16:22:11.263610, event: throttled, op: pg_notify(3.72b(12) epoch 243) 2014-06-09 16:22:11.645601 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.645603 7f869af78700 5 -- op tracker -- , seq: 2342, time: 2014-06-09 16:22:11.263644, event: all_read, op: pg_notify(3.72b(12) epoch 243) 2014-06-09 16:22:11.645595 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.645606 7f869af78700 5 -- op tracker -- , seq: 2342, time: 2014-06-09 16:22:11.645579, event: dispatched, op: pg_notify(3.72b(12) epoch 243) 2014-06-09 16:22:11.645610 7f869af78700 5 -- op tracker -- , seq: 2342, time: 2014-06-09 16:22:11.645609, event: waiting_for_osdmap, op: pg_notify(3.72b(12) epoch 243) 2014-06-09 16:22:11.645613 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.645615 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31a5a40 2014-06-09 16:22:11.645608 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.645618 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3317.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.645623 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.645620 7f869af78700 5 -- op tracker -- , seq: 2342, time: 2014-06-09 16:22:11.645618, event: started, op: pg_notify(3.72b(12) epoch 243) 2014-06-09 16:22:11.645618 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.645627 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.645638 7f8689f56700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.645638 7f869af78700 5 -- op tracker -- , seq: 2342, time: 2014-06-09 16:22:11.645637, event: done, op: pg_notify(3.72b(12) epoch 243) 2014-06-09 16:22:11.645644 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645645 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645644 7f86a5610700 10 _do_op 0x308cb140 seq 3317 r = 0, finisher 0xbd65780 0 2014-06-09 16:22:11.645648 7f86a5610700 10 _finish_op 0x308cb140 seq 3317 osr(3.271 0x3ed0bc0)/0x3ed0bc0 2014-06-09 16:22:11.645656 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645657 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645658 7f869af78700 20 osd.26 243 _dispatch 0x240eee00 pg_info(1 pgs e243:3.465(2)) v4 2014-06-09 16:22:11.645651 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cb40 (1 active) 2014-06-09 16:22:11.645661 7f869af78700 5 -- op tracker -- , seq: 2343, time: 2014-06-09 16:22:11.327941, event: header_read, op: pg_info(1 pgs e243:3.465(2)) 2014-06-09 16:22:11.645664 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcfc0 (2 active) 2014-06-09 16:22:11.645667 7f86a5610700 5 _do_op 0x308cb0e0 seq 3318 osr(1.18a 0x384cb50)/0x384cb50 start 2014-06-09 16:22:11.645667 7f869af78700 5 -- op tracker -- , seq: 2343, time: 2014-06-09 16:22:11.327941, event: throttled, op: pg_info(1 pgs e243:3.465(2)) 2014-06-09 16:22:11.645670 7f86a5610700 10 _do_transaction on 0xc147380 2014-06-09 16:22:11.645671 7f869af78700 5 -- op tracker -- , seq: 2343, time: 2014-06-09 16:22:11.327951, event: all_read, op: pg_info(1 pgs e243:3.465(2)) 2014-06-09 16:22:11.645675 7f869af78700 5 -- op tracker -- , seq: 2343, time: 2014-06-09 16:22:11.645656, event: dispatched, op: pg_info(1 pgs e243:3.465(2)) 2014-06-09 16:22:11.645676 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.645680 7f869af78700 5 -- op tracker -- , seq: 2343, time: 2014-06-09 16:22:11.645678, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.465(2)) 2014-06-09 16:22:11.645682 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.465(2)) v4 from osd.4 2014-06-09 16:22:11.645684 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eee00 2014-06-09 16:22:11.645688 7f869af78700 5 -- op tracker -- , seq: 2343, time: 2014-06-09 16:22:11.645686, event: started, op: pg_info(1 pgs e243:3.465(2)) 2014-06-09 16:22:11.645704 7f869af78700 5 -- op tracker -- , seq: 2343, time: 2014-06-09 16:22:11.645701, event: done, op: pg_info(1 pgs e243:3.465(2)) 2014-06-09 16:22:11.645711 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645710 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.645713 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645711 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:189 2014-06-09 16:22:11.645727 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645727 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645729 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645728 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3318.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.645730 7f869af78700 20 osd.26 243 _dispatch 0x31052e00 pg_info(1 pgs e243:3.4ac) v4 2014-06-09 16:22:11.645732 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.645735 7f869af78700 5 -- op tracker -- , seq: 2344, time: 2014-06-09 16:22:11.380516, event: header_read, op: pg_info(1 pgs e243:3.4ac) 2014-06-09 16:22:11.645731 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:11.645745 7f869af78700 5 -- op tracker -- , seq: 2344, time: 2014-06-09 16:22:11.380516, event: throttled, op: pg_info(1 pgs e243:3.4ac) 2014-06-09 16:22:11.645745 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.645752 7f869af78700 5 -- op tracker -- , seq: 2344, time: 2014-06-09 16:22:11.380526, event: all_read, op: pg_info(1 pgs e243:3.4ac) 2014-06-09 16:22:11.645756 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.645759 7f869af78700 5 -- op tracker -- , seq: 2344, time: 2014-06-09 16:22:11.645726, event: dispatched, op: pg_info(1 pgs e243:3.4ac) 2014-06-09 16:22:11.645765 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.645773 7f86a5610700 10 _do_op 0x308cb0e0 seq 3318 r = 0, finisher 0x30508ea0 0 2014-06-09 16:22:11.645771 7f869af78700 5 -- op tracker -- , seq: 2344, time: 2014-06-09 16:22:11.645770, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4ac) 2014-06-09 16:22:11.645777 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4ac) v4 from osd.7 2014-06-09 16:22:11.645777 7f86a5610700 10 _finish_op 0x308cb0e0 seq 3318 osr(1.18a 0x384cb50)/0x384cb50 2014-06-09 16:22:11.645779 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31052e00 2014-06-09 16:22:11.645778 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.645781 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcfc0 (1 active) 2014-06-09 16:22:11.645783 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f9e0 (2 active) 2014-06-09 16:22:11.645782 7f869af78700 5 -- op tracker -- , seq: 2344, time: 2014-06-09 16:22:11.645781, event: started, op: pg_info(1 pgs e243:3.4ac) 2014-06-09 16:22:11.645785 7f86a5610700 5 _do_op 0x308cb020 seq 3319 osr(3.5f8 0x3ed1b30)/0x3ed1b30 start 2014-06-09 16:22:11.645788 7f86a5610700 10 _do_transaction on 0xc0ccb40 2014-06-09 16:22:11.645792 7f869af78700 5 -- op tracker -- , seq: 2344, time: 2014-06-09 16:22:11.645791, event: done, op: pg_info(1 pgs e243:3.4ac) 2014-06-09 16:22:11.645794 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.645797 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645798 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645793 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1074 2014-06-09 16:22:11.645806 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645807 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645808 7f869af78700 20 osd.26 243 _dispatch 0x2eef6fc0 pg_info(1 pgs e243:0.4e3) v4 2014-06-09 16:22:11.645809 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645813 7f869af78700 5 -- op tracker -- , seq: 2345, time: 2014-06-09 16:22:11.436989, event: header_read, op: pg_info(1 pgs e243:0.4e3) 2014-06-09 16:22:11.645813 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=191 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.645823 7f869af78700 5 -- op tracker -- , seq: 2345, time: 2014-06-09 16:22:11.436989, event: throttled, op: pg_info(1 pgs e243:0.4e3) 2014-06-09 16:22:11.645827 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.645830 7f869af78700 5 -- op tracker -- , seq: 2345, time: 2014-06-09 16:22:11.437001, event: all_read, op: pg_info(1 pgs e243:0.4e3) 2014-06-09 16:22:11.645826 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=191 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.645836 7f869af78700 5 -- op tracker -- , seq: 2345, time: 2014-06-09 16:22:11.645806, event: dispatched, op: pg_info(1 pgs e243:0.4e3) 2014-06-09 16:22:11.645842 7f869af78700 5 -- op tracker -- , seq: 2345, time: 2014-06-09 16:22:11.645840, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4e3) 2014-06-09 16:22:11.645844 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4e3) v4 from osd.3 2014-06-09 16:22:11.645846 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef6fc0 2014-06-09 16:22:11.645844 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3319.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.645849 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.645840 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=191 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.222668 7 0.000191 2014-06-09 16:22:11.645849 7f869af78700 5 -- op tracker -- , seq: 2345, time: 2014-06-09 16:22:11.645848, event: started, op: pg_info(1 pgs e243:0.4e3) 2014-06-09 16:22:11.645852 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=191 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.645858 7f869af78700 5 -- op tracker -- , seq: 2345, time: 2014-06-09 16:22:11.645857, event: done, op: pg_info(1 pgs e243:0.4e3) 2014-06-09 16:22:11.645863 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645864 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645869 7f86a5610700 10 _do_op 0x308cb020 seq 3319 r = 0, finisher 0x3061fda0 0 2014-06-09 16:22:11.645872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645864 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=191 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.645873 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645872 7f86a5610700 10 _finish_op 0x308cb020 seq 3319 osr(3.5f8 0x3ed1b30)/0x3ed1b30 2014-06-09 16:22:11.645874 7f869af78700 20 osd.26 243 _dispatch 0x1a0396c0 pg_info(1 pgs e243:3.4e2) v4 2014-06-09 16:22:11.645875 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f9e0 (1 active) 2014-06-09 16:22:11.645877 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338900 (2 active) 2014-06-09 16:22:11.645876 7f869af78700 5 -- op tracker -- , seq: 2346, time: 2014-06-09 16:22:11.311236, event: header_read, op: pg_info(1 pgs e243:3.4e2) 2014-06-09 16:22:11.645879 7f86a5610700 5 _do_op 0x308cb920 seq 3320 osr(1.3d6 0x384d470)/0x384d470 start 2014-06-09 16:22:11.645882 7f86a5610700 10 _do_transaction on 0xbe88e40 2014-06-09 16:22:11.645875 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=191 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.645882 7f869af78700 5 -- op tracker -- , seq: 2346, time: 2014-06-09 16:22:11.311237, event: throttled, op: pg_info(1 pgs e243:3.4e2) 2014-06-09 16:22:11.645888 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.645888 7f869af78700 5 -- op tracker -- , seq: 2346, time: 2014-06-09 16:22:11.311245, event: all_read, op: pg_info(1 pgs e243:3.4e2) 2014-06-09 16:22:11.645887 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'596 -> 213'596 2014-06-09 16:22:11.645895 7f869af78700 5 -- op tracker -- , seq: 2346, time: 2014-06-09 16:22:11.645872, event: dispatched, op: pg_info(1 pgs e243:3.4e2) 2014-06-09 16:22:11.645902 7f869af78700 5 -- op tracker -- , seq: 2346, time: 2014-06-09 16:22:11.645900, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4e2) 2014-06-09 16:22:11.645906 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4e2) v4 from osd.1 2014-06-09 16:22:11.645908 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a0396c0 2014-06-09 16:22:11.645912 7f869af78700 5 -- op tracker -- , seq: 2346, time: 2014-06-09 16:22:11.645910, event: started, op: pg_info(1 pgs e243:3.4e2) 2014-06-09 16:22:11.645921 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.645929 7f869af78700 5 -- op tracker -- , seq: 2346, time: 2014-06-09 16:22:11.645926, event: done, op: pg_info(1 pgs e243:3.4e2) 2014-06-09 16:22:11.645927 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.645939 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645937 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3320.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.645942 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645942 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.645951 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.645952 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645952 7f869af78700 20 osd.26 243 _dispatch 0x240eb180 pg_notify(3.4c7(10) epoch 243) v5 2014-06-09 16:22:11.645956 7f869af78700 5 -- op tracker -- , seq: 2347, time: 2014-06-09 16:22:11.395946, event: header_read, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.645960 7f869af78700 5 -- op tracker -- , seq: 2347, time: 2014-06-09 16:22:11.395947, event: throttled, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.645965 7f869af78700 5 -- op tracker -- , seq: 2347, time: 2014-06-09 16:22:11.395970, event: all_read, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.645966 7f86a5610700 10 _do_op 0x308cb920 seq 3320 r = 0, finisher 0x2d7395e0 0 2014-06-09 16:22:11.645969 7f869af78700 5 -- op tracker -- , seq: 2347, time: 2014-06-09 16:22:11.645951, event: dispatched, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.645970 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.645971 7f86a5610700 10 _finish_op 0x308cb920 seq 3320 osr(1.3d6 0x384d470)/0x384d470 2014-06-09 16:22:11.645973 7f869af78700 5 -- op tracker -- , seq: 2347, time: 2014-06-09 16:22:11.645972, event: waiting_for_osdmap, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.645973 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338900 (1 active) 2014-06-09 16:22:11.645975 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.645975 7f8689f56700 5 queue_transactions existing osr(3.58f 0x3ed1850)/0x3ed1850 2014-06-09 16:22:11.645976 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eb180 2014-06-09 16:22:11.645976 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781440 (2 active) 2014-06-09 16:22:11.645978 7f8689f56700 10 op_submit_start 3365 2014-06-09 16:22:11.645979 7f8689f56700 5 queue_transactions (trailing journal) 3365 2014-06-09 16:22:11.645978 7f86a5610700 5 _do_op 0x306066c0 seq 3321 osr(1.1ef 0x384cdb0)/0x384cdb0 start 2014-06-09 16:22:11.645981 7f86a5610700 10 _do_transaction on 0xc1f7980 2014-06-09 16:22:11.645981 7f8689f56700 5 queue_op 0x309a2660 seq 3365 osr(3.58f 0x3ed1850) 1429 bytes (queue has 46 ops and 1042516 bytes) 2014-06-09 16:22:11.645983 7f8689f56700 10 op_submit_finish 3365 2014-06-09 16:22:11.645979 7f869af78700 5 -- op tracker -- , seq: 2347, time: 2014-06-09 16:22:11.645978, event: started, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.645987 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.645986 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.645996 7f869af78700 5 -- op tracker -- , seq: 2347, time: 2014-06-09 16:22:11.645995, event: done, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.646001 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.646001 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.645998 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.2f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.646012 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.646013 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.646011 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.646013 7f869af78700 20 osd.26 243 _dispatch 0x3104ac40 pg_notify(2.442(20) epoch 243) v5 2014-06-09 16:22:11.646015 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.646017 7f869af78700 5 -- op tracker -- , seq: 2348, time: 2014-06-09 16:22:11.399636, event: header_read, op: pg_notify(2.442(20) epoch 243) 2014-06-09 16:22:11.646020 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.646018 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 0.801094 7 0.000186 2014-06-09 16:22:11.646024 7f869af78700 5 -- op tracker -- , seq: 2348, time: 2014-06-09 16:22:11.399637, event: throttled, op: pg_notify(2.442(20) epoch 243) 2014-06-09 16:22:11.646031 7f869af78700 5 -- op tracker -- , seq: 2348, time: 2014-06-09 16:22:11.399672, event: all_read, op: pg_notify(2.442(20) epoch 243) 2014-06-09 16:22:11.646029 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.646036 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3321.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.646040 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.646037 7f869af78700 5 -- op tracker -- , seq: 2348, time: 2014-06-09 16:22:11.646012, event: dispatched, op: pg_notify(2.442(20) epoch 243) 2014-06-09 16:22:11.646044 7f869af78700 5 -- op tracker -- , seq: 2348, time: 2014-06-09 16:22:11.646043, event: waiting_for_osdmap, op: pg_notify(2.442(20) epoch 243) 2014-06-09 16:22:11.646040 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.646047 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.646048 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104ac40 2014-06-09 16:22:11.646051 7f869af78700 5 -- op tracker -- , seq: 2348, time: 2014-06-09 16:22:11.646050, event: started, op: pg_notify(2.442(20) epoch 243) 2014-06-09 16:22:11.646049 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.646058 7f86a5610700 10 _do_op 0x306066c0 seq 3321 r = 0, finisher 0xc5a7c20 0 2014-06-09 16:22:11.646062 7f86a5610700 10 _finish_op 0x306066c0 seq 3321 osr(1.1ef 0x384cdb0)/0x384cdb0 2014-06-09 16:22:11.646064 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781440 (1 active) 2014-06-09 16:22:11.646059 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.646066 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653200 (2 active) 2014-06-09 16:22:11.646066 7f869af78700 5 -- op tracker -- , seq: 2348, time: 2014-06-09 16:22:11.646063, event: done, op: pg_notify(2.442(20) epoch 243) 2014-06-09 16:22:11.646069 7f86a5610700 5 _do_op 0x30d97ce0 seq 3322 osr(0.117 0x2bd66a0)/0x2bd66a0 start 2014-06-09 16:22:11.646071 7f86a5610700 10 _do_transaction on 0xc3372c0 2014-06-09 16:22:11.646073 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.646068 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.646075 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.646077 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.646089 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.646089 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.646090 7f869af78700 20 osd.26 243 _dispatch 0x31040000 pg_notify(0.1dc(20),0.77a(23) epoch 243) v5 2014-06-09 16:22:11.646093 7f869af78700 5 -- op tracker -- , seq: 2349, time: 2014-06-09 16:22:11.388612, event: header_read, op: pg_notify(0.1dc(20),0.77a(23) epoch 243) 2014-06-09 16:22:11.646098 7f869af78700 5 -- op tracker -- , seq: 2349, time: 2014-06-09 16:22:11.388612, event: throttled, op: pg_notify(0.1dc(20),0.77a(23) epoch 243) 2014-06-09 16:22:11.646101 7f869af78700 5 -- op tracker -- , seq: 2349, time: 2014-06-09 16:22:11.388665, event: all_read, op: pg_notify(0.1dc(20),0.77a(23) epoch 243) 2014-06-09 16:22:11.646105 7f869af78700 5 -- op tracker -- , seq: 2349, time: 2014-06-09 16:22:11.646088, event: dispatched, op: pg_notify(0.1dc(20),0.77a(23) epoch 243) 2014-06-09 16:22:11.646110 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.646109 7f869af78700 5 -- op tracker -- , seq: 2349, time: 2014-06-09 16:22:11.646107, event: waiting_for_osdmap, op: pg_notify(0.1dc(20),0.77a(23) epoch 243) 2014-06-09 16:22:11.646113 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.646114 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040000 2014-06-09 16:22:11.646117 7f869af78700 5 -- op tracker -- , seq: 2349, time: 2014-06-09 16:22:11.646116, event: started, op: pg_notify(0.1dc(20),0.77a(23) epoch 243) 2014-06-09 16:22:11.646118 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.646123 7f8689f56700 5 queue_transactions existing osr(1.2f8 0x384d0d0)/0x384d0d0 2014-06-09 16:22:11.646125 7f8689f56700 10 op_submit_start 3366 2014-06-09 16:22:11.646127 7f8689f56700 5 queue_transactions (trailing journal) 3366 2014-06-09 16:22:11.646126 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3322.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.646128 7f8689f56700 5 queue_op 0x309a25a0 seq 3366 osr(1.2f8 0x384d0d0) 2099 bytes (queue has 46 ops and 1042614 bytes) 2014-06-09 16:22:11.646131 7f8689f56700 10 op_submit_finish 3366 2014-06-09 16:22:11.646131 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.646139 7f869af78700 5 -- op tracker -- , seq: 2349, time: 2014-06-09 16:22:11.646136, event: done, op: pg_notify(0.1dc(20),0.77a(23) epoch 243) 2014-06-09 16:22:11.646133 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.646147 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.646149 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.646152 7f86a5610700 10 _do_op 0x30d97ce0 seq 3322 r = 0, finisher 0xbfce540 0 2014-06-09 16:22:11.646148 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.646156 7f86a5610700 10 _finish_op 0x30d97ce0 seq 3322 osr(0.117 0x2bd66a0)/0x2bd66a0 2014-06-09 16:22:11.646159 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653200 (1 active) 2014-06-09 16:22:11.646162 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c06c0 (2 active) 2014-06-09 16:22:11.646164 7f86a5610700 5 _do_op 0x309a3500 seq 3323 osr(2.48d 0x2ed1470)/0x2ed1470 start 2014-06-09 16:22:11.646158 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.646167 7f86a5610700 10 _do_transaction on 0x2bf5b00 2014-06-09 16:22:11.646170 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.646172 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.646173 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.646173 7f869af78700 20 osd.26 243 _dispatch 0x30bd8000 pg_info(1 pgs e243:2.4f4) v4 2014-06-09 16:22:11.646167 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.646179 7f869af78700 5 -- op tracker -- , seq: 2350, time: 2014-06-09 16:22:11.371042, event: header_read, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.646181 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.646185 7f869af78700 5 -- op tracker -- , seq: 2350, time: 2014-06-09 16:22:11.371042, event: throttled, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.646192 7f869af78700 5 -- op tracker -- , seq: 2350, time: 2014-06-09 16:22:11.371055, event: all_read, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.646198 7f869af78700 5 -- op tracker -- , seq: 2350, time: 2014-06-09 16:22:11.646170, event: dispatched, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.646196 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:973 2014-06-09 16:22:11.646206 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.646208 7f869af78700 5 -- op tracker -- , seq: 2350, time: 2014-06-09 16:22:11.646206, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.646211 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4f4) v4 from osd.0 2014-06-09 16:22:11.646213 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.646213 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd8000 2014-06-09 16:22:11.646216 7f869af78700 5 -- op tracker -- , seq: 2350, time: 2014-06-09 16:22:11.646215, event: started, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.646222 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3323.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.646217 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.646226 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.646225 7f869af78700 5 -- op tracker -- , seq: 2350, time: 2014-06-09 16:22:11.646224, event: done, op: pg_info(1 pgs e243:2.4f4) 2014-06-09 16:22:11.646230 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.646231 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.646239 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.646228 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.1e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.646239 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.646240 7f869af78700 20 osd.26 243 _dispatch 0x310ca1c0 pg_info(1 pgs e243:1.b6) v4 2014-06-09 16:22:11.646240 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.646244 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.646243 7f869af78700 5 -- op tracker -- , seq: 2351, time: 2014-06-09 16:22:11.387764, event: header_read, op: pg_info(1 pgs e243:1.b6) 2014-06-09 16:22:11.646248 7f86a5610700 10 _do_op 0x309a3500 seq 3323 r = 0, finisher 0x13965480 0 2014-06-09 16:22:11.646252 7f86a5610700 10 _finish_op 0x309a3500 seq 3323 osr(2.48d 0x2ed1470)/0x2ed1470 2014-06-09 16:22:11.646250 7f869af78700 5 -- op tracker -- , seq: 2351, time: 2014-06-09 16:22:11.387764, event: throttled, op: pg_info(1 pgs e243:1.b6) 2014-06-09 16:22:11.646248 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.736889 7 0.000180 2014-06-09 16:22:11.646255 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c06c0 (1 active) 2014-06-09 16:22:11.646258 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52eb40 (2 active) 2014-06-09 16:22:11.646257 7f869af78700 5 -- op tracker -- , seq: 2351, time: 2014-06-09 16:22:11.387781, event: all_read, op: pg_info(1 pgs e243:1.b6) 2014-06-09 16:22:11.646260 7f86a5610700 5 _do_op 0x30060300 seq 3324 osr(0.61f 0x2bd7d30)/0x2bd7d30 start 2014-06-09 16:22:11.646263 7f86a5610700 10 _do_transaction on 0x2eaafc0 2014-06-09 16:22:11.646259 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.646264 7f869af78700 5 -- op tracker -- , seq: 2351, time: 2014-06-09 16:22:11.646238, event: dispatched, op: pg_info(1 pgs e243:1.b6) 2014-06-09 16:22:11.646269 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.646271 7f869af78700 5 -- op tracker -- , seq: 2351, time: 2014-06-09 16:22:11.646269, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.b6) 2014-06-09 16:22:11.646274 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.b6) v4 from osd.5 2014-06-09 16:22:11.646270 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.646281 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x139649a0,0x3061ff20,0x13965a60,0xbd65780,0x30508ea0,0x3061fda0,0x2d7395e0,0xc5a7c20,0xbfce540,0x13965480] 2014-06-09 16:22:11.646298 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.646338 7f8692f68700 10 op_submit_start 3367 2014-06-09 16:22:11.649368 7f8692f68700 5 queue_transactions (trailing journal) 3367 2014-06-09 16:22:11.649360 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.649372 7f8692f68700 5 queue_op 0x309a33e0 seq 3367 osr(3.577 0x3ed18c0) 1335 bytes (queue has 50 ops and 1049190 bytes) 2014-06-09 16:22:11.649374 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.649381 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3324.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.649383 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x139649a0,0x3061ff20,0x13965a60,0xbd65780,0x30508ea0,0x3061fda0,0x2d7395e0,0xc5a7c20,0xbfce540,0x13965480] 2014-06-09 16:22:11.649387 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.649388 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.649390 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.649383 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.646347 7f86a5e11700 10 _finish_op 0x30d97f80 seq 3314 osr(0.3a9 0x2bd7310)/0x2bd7310 2014-06-09 16:22:11.646350 7f868c75b700 2 waiting 51 > 50 ops || 1051144 > 104857600 2014-06-09 16:22:11.649405 7f868e75f700 2 waiting 51 > 50 ops || 1051067 > 104857600 2014-06-09 16:22:11.649431 7f86a5610700 10 _do_op 0x30060300 seq 3324 r = 0, finisher 0xbd75e40 0 2014-06-09 16:22:11.649434 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.649437 7f868bf5a700 2 waiting 51 > 50 ops || 1050525 > 104857600 2014-06-09 16:22:11.649440 7f8689f56700 5 queue_transactions existing osr(2.1e3 0x2ed0810)/0x2ed0810 2014-06-09 16:22:11.649442 7f8689755700 2 waiting 51 > 50 ops || 1050525 > 104857600 2014-06-09 16:22:11.646404 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd79160,0xbd72900,0xbd74000,0x20051ce0,0xc2d7720,0xc40c520,0x30508580,0xc169b60,0xc40cd00,0xbd71ae0,0xbd79b00,0xbd6adc0] 2014-06-09 16:22:11.649449 7f8690f64700 2 waiting 51 > 50 ops || 1051070 > 104857600 2014-06-09 16:22:11.649454 7f868b759700 2 waiting 51 > 50 ops || 1051289 > 104857600 2014-06-09 16:22:11.649447 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649460 7f8692767700 2 waiting 51 > 50 ops || 1051164 > 104857600 2014-06-09 16:22:11.649466 7f868cf5c700 2 waiting 51 > 50 ops || 1050525 > 104857600 2014-06-09 16:22:11.649466 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.649473 7f868ef60700 2 waiting 51 > 50 ops || 1268666 > 104857600 2014-06-09 16:22:11.649478 7f8691f66700 2 waiting 51 > 50 ops || 1051030 > 104857600 2014-06-09 16:22:11.649484 7f8690763700 2 waiting 51 > 50 ops || 1051144 > 104857600 2014-06-09 16:22:11.649490 7f868f761700 2 waiting 51 > 50 ops || 1051073 > 104857600 2014-06-09 16:22:11.649501 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e480 (1 active) 2014-06-09 16:22:11.649502 7f868e75f700 2 waiting 51 > 50 ops || 1051138 > 104857600 2014-06-09 16:22:11.649505 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b9e0 (2 active) 2014-06-09 16:22:11.649508 7f868bf5a700 2 waiting 51 > 50 ops || 1050596 > 104857600 2014-06-09 16:22:11.649509 7f86a5e11700 5 _do_op 0x30d97440 seq 3325 osr(3.761 0x3ed1e10)/0x3ed1e10 start 2014-06-09 16:22:11.649512 7f86a5e11700 10 _do_transaction on 0xc2563c0 2014-06-09 16:22:11.649513 7f8689755700 2 waiting 51 > 50 ops || 1050596 > 104857600 2014-06-09 16:22:11.649518 7f8690f64700 2 waiting 51 > 50 ops || 1051141 > 104857600 2014-06-09 16:22:11.649518 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.649523 7f868b759700 2 waiting 51 > 50 ops || 1051360 > 104857600 2014-06-09 16:22:11.649527 7f8692767700 2 waiting 51 > 50 ops || 1051235 > 104857600 2014-06-09 16:22:11.649533 7f868cf5c700 2 waiting 51 > 50 ops || 1050596 > 104857600 2014-06-09 16:22:11.649539 7f868ef60700 2 waiting 51 > 50 ops || 1268737 > 104857600 2014-06-09 16:22:11.649543 7f8691f66700 2 waiting 51 > 50 ops || 1051101 > 104857600 2014-06-09 16:22:11.649483 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.649557 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.649553 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649563 7f8690763700 2 waiting 51 > 50 ops || 1051215 > 104857600 2014-06-09 16:22:11.649568 7f868f761700 2 waiting 51 > 50 ops || 1051144 > 104857600 2014-06-09 16:22:11.649569 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.649575 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3325.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.649580 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.649584 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.649603 7f86a5e11700 10 _do_op 0x30d97440 seq 3325 r = 0, finisher 0xbd694c0 0 2014-06-09 16:22:11.649599 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649608 7f86a5e11700 10 _finish_op 0x30d97440 seq 3325 osr(3.761 0x3ed1e10)/0x3ed1e10 2014-06-09 16:22:11.649612 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.649613 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b9e0 (1 active) 2014-06-09 16:22:11.649616 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6240 (2 active) 2014-06-09 16:22:11.649616 7f868bf5a700 2 waiting 51 > 50 ops || 1051044 > 104857600 2014-06-09 16:22:11.649618 7f86a5e11700 5 _do_op 0x2fc8de00 seq 3326 osr(0.473 0x2bd74d0)/0x2bd74d0 start 2014-06-09 16:22:11.649621 7f86a5e11700 10 _do_transaction on 0xbf28180 2014-06-09 16:22:11.649622 7f8689755700 2 waiting 51 > 50 ops || 1051044 > 104857600 2014-06-09 16:22:11.649627 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.649627 7f8690f64700 2 waiting 51 > 50 ops || 1051589 > 104857600 2014-06-09 16:22:11.649624 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.649631 7f868b759700 2 waiting 51 > 50 ops || 1051808 > 104857600 2014-06-09 16:22:11.649636 7f8692767700 2 waiting 51 > 50 ops || 1051683 > 104857600 2014-06-09 16:22:11.649641 7f868cf5c700 2 waiting 51 > 50 ops || 1051044 > 104857600 2014-06-09 16:22:11.649636 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649647 7f868ef60700 2 waiting 51 > 50 ops || 1269185 > 104857600 2014-06-09 16:22:11.649649 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.649651 7f8691f66700 2 waiting 51 > 50 ops || 1051549 > 104857600 2014-06-09 16:22:11.649655 7f8690763700 2 waiting 51 > 50 ops || 1051663 > 104857600 2014-06-09 16:22:11.649658 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.649660 7f868f761700 2 waiting 51 > 50 ops || 1051592 > 104857600 2014-06-09 16:22:11.649663 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.649675 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3326.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.649680 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.649676 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649689 7f86a460e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.649704 7f86a5e11700 10 _do_op 0x2fc8de00 seq 3326 r = 0, finisher 0xc5a7b40 0 2014-06-09 16:22:11.649708 7f86a5e11700 10 _finish_op 0x2fc8de00 seq 3326 osr(0.473 0x2bd74d0)/0x2bd74d0 2014-06-09 16:22:11.649702 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.649714 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6240 (1 active) 2014-06-09 16:22:11.649717 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0360 (2 active) 2014-06-09 16:22:11.649717 7f8689755700 2 waiting 51 > 50 ops || 1050425 > 104857600 2014-06-09 16:22:11.649718 7f86a5e11700 5 _do_op 0x308ca660 seq 3327 osr(3.579 0x3ed18d0)/0x3ed18d0 start 2014-06-09 16:22:11.649721 7f86a5e11700 10 _do_transaction on 0xc20be00 2014-06-09 16:22:11.649722 7f8690f64700 2 waiting 51 > 50 ops || 1050970 > 104857600 2014-06-09 16:22:11.649717 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649727 7f868b759700 2 waiting 51 > 50 ops || 1051189 > 104857600 2014-06-09 16:22:11.649727 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.649730 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.649731 7f8692767700 2 waiting 51 > 50 ops || 1051064 > 104857600 2014-06-09 16:22:11.649738 7f868cf5c700 2 waiting 51 > 50 ops || 1050425 > 104857600 2014-06-09 16:22:11.649743 7f868ef60700 2 waiting 51 > 50 ops || 1268566 > 104857600 2014-06-09 16:22:11.649744 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] flushed 2014-06-09 16:22:11.649757 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.649766 7f8691f66700 2 waiting 51 > 50 ops || 1050930 > 104857600 2014-06-09 16:22:11.649759 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649772 7f8690763700 2 waiting 51 > 50 ops || 1051044 > 104857600 2014-06-09 16:22:11.649773 7f86a460e700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.649773 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3327.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.649778 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.649778 7f868f761700 2 waiting 51 > 50 ops || 1050973 > 104857600 2014-06-09 16:22:11.649784 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.649793 7f86a5e11700 10 _do_op 0x308ca660 seq 3327 r = 0, finisher 0x292823a0 0 2014-06-09 16:22:11.649797 7f86a5e11700 10 _finish_op 0x308ca660 seq 3327 osr(3.579 0x3ed18d0)/0x3ed18d0 2014-06-09 16:22:11.649803 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0360 (1 active) 2014-06-09 16:22:11.649806 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0c60 (2 active) 2014-06-09 16:22:11.649799 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649808 7f86a5e11700 5 _do_op 0x2fcaee40 seq 3328 osr(2.431 0x2ed1590)/0x2ed1590 start 2014-06-09 16:22:11.649811 7f86a5e11700 10 _do_transaction on 0xbebb2c0 2014-06-09 16:22:11.649811 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.649817 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.649817 7f8690f64700 2 waiting 51 > 50 ops || 1050970 > 104857600 2014-06-09 16:22:11.649823 7f868b759700 2 waiting 51 > 50 ops || 1051189 > 104857600 2014-06-09 16:22:11.649828 7f8692767700 2 waiting 51 > 50 ops || 1051064 > 104857600 2014-06-09 16:22:11.649822 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.649834 7f868cf5c700 2 waiting 51 > 50 ops || 1050425 > 104857600 2014-06-09 16:22:11.649839 7f868ef60700 2 waiting 51 > 50 ops || 1268566 > 104857600 2014-06-09 16:22:11.649835 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649844 7f8691f66700 2 waiting 51 > 50 ops || 1050930 > 104857600 2014-06-09 16:22:11.649846 7f86a460e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.649849 7f8690763700 2 waiting 51 > 50 ops || 1051044 > 104857600 2014-06-09 16:22:11.649849 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.649854 7f868f761700 2 waiting 51 > 50 ops || 1050973 > 104857600 2014-06-09 16:22:11.649865 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3328.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.649862 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.649870 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.649875 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649888 7f86a460e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.649893 7f86a5e11700 10 _do_op 0x2fcaee40 seq 3328 r = 0, finisher 0xbd6de80 0 2014-06-09 16:22:11.649898 7f86a5e11700 10 _finish_op 0x2fcaee40 seq 3328 osr(2.431 0x2ed1590)/0x2ed1590 2014-06-09 16:22:11.649904 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0c60 (1 active) 2014-06-09 16:22:11.649907 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e240 (2 active) 2014-06-09 16:22:11.649901 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.649909 7f86a5e11700 5 _do_op 0x2fcaede0 seq 3329 osr(1.375 0x384d230)/0x384d230 start 2014-06-09 16:22:11.649912 7f86a5e11700 10 _do_transaction on 0xc3f29c0 2014-06-09 16:22:11.649916 7f868b759700 2 waiting 51 > 50 ops || 1051186 > 104857600 2014-06-09 16:22:11.649918 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.649915 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.649922 7f8692767700 2 waiting 51 > 50 ops || 1051061 > 104857600 2014-06-09 16:22:11.649927 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.649928 7f868cf5c700 2 waiting 51 > 50 ops || 1050422 > 104857600 2014-06-09 16:22:11.649933 7f868ef60700 2 waiting 51 > 50 ops || 1268563 > 104857600 2014-06-09 16:22:11.649950 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.649949 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.649960 7f8691f66700 2 waiting 51 > 50 ops || 1050927 > 104857600 2014-06-09 16:22:11.649962 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd79160,0xbd72900,0xbd74000,0x20051ce0,0xc2d7720,0xc40c520,0x30508580,0xc169b60,0xc40cd00,0xbd71ae0,0xbd79b00,0xbd6adc0] 2014-06-09 16:22:11.649967 7f8690763700 2 waiting 51 > 50 ops || 1051041 > 104857600 2014-06-09 16:22:11.649967 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe300e0,0xbd73cc0,0xbd7b940,0xc169ea0,0x275cfc00] 2014-06-09 16:22:11.649966 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3329.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.649971 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.649972 7f868f761700 2 waiting 51 > 50 ops || 1050970 > 104857600 2014-06-09 16:22:11.649973 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe300e0,0xbd73cc0,0xbd7b940,0xc169ea0,0x275cfc00] 2014-06-09 16:22:11.649976 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.649977 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.649987 7f86a5e11700 10 _do_op 0x2fcaede0 seq 3329 r = 0, finisher 0xbdba360 0 2014-06-09 16:22:11.649994 7f86a5e11700 10 _finish_op 0x2fcaede0 seq 3329 osr(1.375 0x384d230)/0x384d230 2014-06-09 16:22:11.649999 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e240 (1 active) 2014-06-09 16:22:11.650002 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0000 (2 active) 2014-06-09 16:22:11.649995 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf1460] 2014-06-09 16:22:11.650003 7f8692767700 2 waiting 51 > 50 ops || 1051151 > 104857600 2014-06-09 16:22:11.650004 7f86a5e11700 5 _do_op 0x308ca3c0 seq 3330 osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 start 2014-06-09 16:22:11.650005 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf1460] 2014-06-09 16:22:11.650007 7f86a5e11700 10 _do_transaction on 0xc12e3c0 2014-06-09 16:22:11.650007 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650008 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650009 7f868cf5c700 2 waiting 51 > 50 ops || 1050512 > 104857600 2014-06-09 16:22:11.650013 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650016 7f868ef60700 2 waiting 51 > 50 ops || 1268653 > 104857600 2014-06-09 16:22:11.650022 7f8691f66700 2 waiting 51 > 50 ops || 1051017 > 104857600 2014-06-09 16:22:11.650027 7f8690763700 2 waiting 51 > 50 ops || 1051131 > 104857600 2014-06-09 16:22:11.650031 7f868f761700 2 waiting 51 > 50 ops || 1051060 > 104857600 2014-06-09 16:22:11.650042 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650067 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3330.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650073 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650079 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310ca1c0 2014-06-09 16:22:11.650089 7f86a5e11700 10 _do_op 0x308ca3c0 seq 3330 r = 0, finisher 0xbdba300 0 2014-06-09 16:22:11.650087 7f869af78700 5 -- op tracker -- , seq: 2351, time: 2014-06-09 16:22:11.650084, event: started, op: pg_info(1 pgs e243:1.b6) 2014-06-09 16:22:11.650094 7f86a5e11700 10 _finish_op 0x308ca3c0 seq 3330 osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 2014-06-09 16:22:11.650096 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7bca0] 2014-06-09 16:22:11.650098 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7bca0] 2014-06-09 16:22:11.650100 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650099 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0000 (1 active) 2014-06-09 16:22:11.650101 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650102 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f7a0 (2 active) 2014-06-09 16:22:11.650104 7f86a5e11700 5 _do_op 0x2fcaed80 seq 3331 osr(3.136 0x3ed07a0)/0x3ed07a0 start 2014-06-09 16:22:11.650104 7f868cf5c700 2 waiting 51 > 50 ops || 1051151 > 104857600 2014-06-09 16:22:11.650107 7f86a5e11700 10 _do_transaction on 0xbebdb00 2014-06-09 16:22:11.650110 7f868ef60700 2 waiting 51 > 50 ops || 1269292 > 104857600 2014-06-09 16:22:11.650113 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650114 7f8691f66700 2 waiting 51 > 50 ops || 1051656 > 104857600 2014-06-09 16:22:11.650119 7f8690763700 2 waiting 51 > 50 ops || 1051770 > 104857600 2014-06-09 16:22:11.650105 7f869af78700 5 -- op tracker -- , seq: 2351, time: 2014-06-09 16:22:11.650103, event: done, op: pg_info(1 pgs e243:1.b6) 2014-06-09 16:22:11.650124 7f868f761700 2 waiting 51 > 50 ops || 1051699 > 104857600 2014-06-09 16:22:11.650126 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650129 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650139 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650143 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650145 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650146 7f869af78700 20 osd.26 243 _dispatch 0x26000a80 pg_notify(3.4b3(12) epoch 243) v5 2014-06-09 16:22:11.650152 7f869af78700 5 -- op tracker -- , seq: 2352, time: 2014-06-09 16:22:11.263669, event: header_read, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.650155 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3331.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650160 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650161 7f869af78700 5 -- op tracker -- , seq: 2352, time: 2014-06-09 16:22:11.263669, event: throttled, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.650168 7f869af78700 5 -- op tracker -- , seq: 2352, time: 2014-06-09 16:22:11.263691, event: all_read, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.650181 7f86a5e11700 10 _do_op 0x2fcaed80 seq 3331 r = 0, finisher 0xcc94640 0 2014-06-09 16:22:11.650175 7f869af78700 5 -- op tracker -- , seq: 2352, time: 2014-06-09 16:22:11.650142, event: dispatched, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.650188 7f86a5e11700 10 _finish_op 0x2fcaed80 seq 3331 osr(3.136 0x3ed07a0)/0x3ed07a0 2014-06-09 16:22:11.650190 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1322d980] 2014-06-09 16:22:11.650189 7f869af78700 5 -- op tracker -- , seq: 2352, time: 2014-06-09 16:22:11.650187, event: waiting_for_osdmap, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.650194 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1322d980] 2014-06-09 16:22:11.650193 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f7a0 (1 active) 2014-06-09 16:22:11.650196 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650196 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5257a0 (2 active) 2014-06-09 16:22:11.650197 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650198 7f86a5e11700 5 _do_op 0x30d96840 seq 3332 osr(2.3e8 0x2ed1310)/0x2ed1310 start 2014-06-09 16:22:11.650200 7f868ef60700 2 waiting 51 > 50 ops || 1269292 > 104857600 2014-06-09 16:22:11.650194 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.650201 7f86a5e11700 10 _do_transaction on 0x2c80cc0 2014-06-09 16:22:11.650203 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26000a80 2014-06-09 16:22:11.650205 7f8691f66700 2 waiting 51 > 50 ops || 1051656 > 104857600 2014-06-09 16:22:11.650209 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650210 7f8690763700 2 waiting 51 > 50 ops || 1051770 > 104857600 2014-06-09 16:22:11.650209 7f869af78700 5 -- op tracker -- , seq: 2352, time: 2014-06-09 16:22:11.650207, event: started, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.650215 7f868f761700 2 waiting 51 > 50 ops || 1051699 > 104857600 2014-06-09 16:22:11.650230 7f869af78700 5 -- op tracker -- , seq: 2352, time: 2014-06-09 16:22:11.650227, event: done, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.650238 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650240 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650241 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650257 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650259 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650257 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3332.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650262 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650260 7f869af78700 20 osd.26 243 _dispatch 0x306dc400 pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.650267 7f869af78700 5 -- op tracker -- , seq: 2353, time: 2014-06-09 16:22:11.328493, event: header_read, op: pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.650275 7f869af78700 5 -- op tracker -- , seq: 2353, time: 2014-06-09 16:22:11.328493, event: throttled, op: pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.650278 7f86a5e11700 10 _do_op 0x30d96840 seq 3332 r = 0, finisher 0x1b0c10c0 0 2014-06-09 16:22:11.650284 7f86a5e11700 10 _finish_op 0x30d96840 seq 3332 osr(2.3e8 0x2ed1310)/0x2ed1310 2014-06-09 16:22:11.650286 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062a000] 2014-06-09 16:22:11.650289 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062a000] 2014-06-09 16:22:11.650289 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5257a0 (1 active) 2014-06-09 16:22:11.650291 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650292 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650282 7f869af78700 5 -- op tracker -- , seq: 2353, time: 2014-06-09 16:22:11.328506, event: all_read, op: pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.650292 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301560 (2 active) 2014-06-09 16:22:11.650294 7f86a5e11700 5 _do_op 0x30606840 seq 3333 osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 start 2014-06-09 16:22:11.650297 7f86a5e11700 10 _do_transaction on 0xc2aee40 2014-06-09 16:22:11.650298 7f8691f66700 2 waiting 51 > 50 ops || 1269249 > 104857600 2014-06-09 16:22:11.650297 7f869af78700 5 -- op tracker -- , seq: 2353, time: 2014-06-09 16:22:11.650257, event: dispatched, op: pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.650303 7f8690763700 2 waiting 51 > 50 ops || 1269363 > 104857600 2014-06-09 16:22:11.650303 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650304 7f869af78700 5 -- op tracker -- , seq: 2353, time: 2014-06-09 16:22:11.650302, event: waiting_for_osdmap, op: pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.650308 7f868f761700 2 waiting 51 > 50 ops || 1269292 > 104857600 2014-06-09 16:22:11.650309 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.650314 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306dc400 2014-06-09 16:22:11.650320 7f869af78700 5 -- op tracker -- , seq: 2353, time: 2014-06-09 16:22:11.650317, event: started, op: pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.650333 7f869af78700 5 -- op tracker -- , seq: 2353, time: 2014-06-09 16:22:11.650330, event: done, op: pg_log(1.438 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.650336 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650341 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650343 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650355 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650353 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3333.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650357 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650357 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650358 7f869af78700 20 osd.26 243 _dispatch 0x31052c40 pg_info(1 pgs e243:3.77c) v4 2014-06-09 16:22:11.650363 7f869af78700 5 -- op tracker -- , seq: 2354, time: 2014-06-09 16:22:11.380531, event: header_read, op: pg_info(1 pgs e243:3.77c) 2014-06-09 16:22:11.650370 7f869af78700 5 -- op tracker -- , seq: 2354, time: 2014-06-09 16:22:11.380531, event: throttled, op: pg_info(1 pgs e243:3.77c) 2014-06-09 16:22:11.650376 7f869af78700 5 -- op tracker -- , seq: 2354, time: 2014-06-09 16:22:11.380544, event: all_read, op: pg_info(1 pgs e243:3.77c) 2014-06-09 16:22:11.650380 7f86a5e11700 10 _do_op 0x30606840 seq 3333 r = 0, finisher 0xbd68a20 0 2014-06-09 16:22:11.650386 7f86a5e11700 10 _finish_op 0x30606840 seq 3333 osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 2014-06-09 16:22:11.650383 7f869af78700 5 -- op tracker -- , seq: 2354, time: 2014-06-09 16:22:11.650354, event: dispatched, op: pg_info(1 pgs e243:3.77c) 2014-06-09 16:22:11.650388 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd65200] 2014-06-09 16:22:11.650391 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd65200] 2014-06-09 16:22:11.650392 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650391 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301560 (1 active) 2014-06-09 16:22:11.650394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650394 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f200 (2 active) 2014-06-09 16:22:11.650392 7f869af78700 5 -- op tracker -- , seq: 2354, time: 2014-06-09 16:22:11.650390, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.77c) 2014-06-09 16:22:11.650396 7f86a5e11700 5 _do_op 0x2fcaed20 seq 3334 osr(2.112 0x2ed06d0)/0x2ed06d0 start 2014-06-09 16:22:11.650399 7f86a5e11700 10 _do_transaction on 0xc27f440 2014-06-09 16:22:11.650397 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.77c) v4 from osd.7 2014-06-09 16:22:11.650400 7f8690763700 2 waiting 51 > 50 ops || 1269868 > 104857600 2014-06-09 16:22:11.650400 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31052c40 2014-06-09 16:22:11.650404 7f868f761700 2 waiting 51 > 50 ops || 1269797 > 104857600 2014-06-09 16:22:11.650404 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650405 7f869af78700 5 -- op tracker -- , seq: 2354, time: 2014-06-09 16:22:11.650403, event: started, op: pg_info(1 pgs e243:3.77c) 2014-06-09 16:22:11.650421 7f869af78700 5 -- op tracker -- , seq: 2354, time: 2014-06-09 16:22:11.650419, event: done, op: pg_info(1 pgs e243:3.77c) 2014-06-09 16:22:11.650429 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650431 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650438 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650444 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650446 7f869af78700 20 osd.26 243 _dispatch 0x2eef6e00 pg_info(1 pgs e243:2.82) v4 2014-06-09 16:22:11.650452 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3334.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650451 7f869af78700 5 -- op tracker -- , seq: 2355, time: 2014-06-09 16:22:11.437071, event: header_read, op: pg_info(1 pgs e243:2.82) 2014-06-09 16:22:11.650456 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650458 7f869af78700 5 -- op tracker -- , seq: 2355, time: 2014-06-09 16:22:11.437071, event: throttled, op: pg_info(1 pgs e243:2.82) 2014-06-09 16:22:11.650465 7f869af78700 5 -- op tracker -- , seq: 2355, time: 2014-06-09 16:22:11.437083, event: all_read, op: pg_info(1 pgs e243:2.82) 2014-06-09 16:22:11.650475 7f86a5e11700 10 _do_op 0x2fcaed20 seq 3334 r = 0, finisher 0xbd7dec0 0 2014-06-09 16:22:11.650471 7f869af78700 5 -- op tracker -- , seq: 2355, time: 2014-06-09 16:22:11.650443, event: dispatched, op: pg_info(1 pgs e243:2.82) 2014-06-09 16:22:11.650481 7f86a5e11700 10 _finish_op 0x2fcaed20 seq 3334 osr(2.112 0x2ed06d0)/0x2ed06d0 2014-06-09 16:22:11.650482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6e8a0] 2014-06-09 16:22:11.650485 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6e8a0] 2014-06-09 16:22:11.650486 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650486 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f200 (1 active) 2014-06-09 16:22:11.650487 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650488 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b7a0 (2 active) 2014-06-09 16:22:11.650490 7f868f761700 2 waiting 51 > 50 ops || 1269742 > 104857600 2014-06-09 16:22:11.650490 7f86a5e11700 5 _do_op 0x30d96780 seq 3335 osr(0.765 0x384c360)/0x384c360 start 2014-06-09 16:22:11.650493 7f86a5e11700 10 _do_transaction on 0xc0d7980 2014-06-09 16:22:11.650486 7f869af78700 5 -- op tracker -- , seq: 2355, time: 2014-06-09 16:22:11.650483, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.82) 2014-06-09 16:22:11.650496 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.82) v4 from osd.3 2014-06-09 16:22:11.650499 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650499 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef6e00 2014-06-09 16:22:11.650504 7f869af78700 5 -- op tracker -- , seq: 2355, time: 2014-06-09 16:22:11.650502, event: started, op: pg_info(1 pgs e243:2.82) 2014-06-09 16:22:11.650519 7f869af78700 5 -- op tracker -- , seq: 2355, time: 2014-06-09 16:22:11.650517, event: done, op: pg_info(1 pgs e243:2.82) 2014-06-09 16:22:11.650530 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650533 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650535 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650546 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3335.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650548 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650549 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650550 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650551 7f869af78700 20 osd.26 243 _dispatch 0x1a038000 pg_info(1 pgs e243:2.3de) v4 2014-06-09 16:22:11.650556 7f869af78700 5 -- op tracker -- , seq: 2356, time: 2014-06-09 16:22:11.311256, event: header_read, op: pg_info(1 pgs e243:2.3de) 2014-06-09 16:22:11.650565 7f86a5e11700 10 _do_op 0x30d96780 seq 3335 r = 0, finisher 0xcc951e0 0 2014-06-09 16:22:11.650564 7f869af78700 5 -- op tracker -- , seq: 2356, time: 2014-06-09 16:22:11.311256, event: throttled, op: pg_info(1 pgs e243:2.3de) 2014-06-09 16:22:11.650569 7f86a5e11700 10 _finish_op 0x30d96780 seq 3335 osr(0.765 0x384c360)/0x384c360 2014-06-09 16:22:11.650571 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x4eb98c0] 2014-06-09 16:22:11.650572 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b7a0 (1 active) 2014-06-09 16:22:11.650574 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524000 (2 active) 2014-06-09 16:22:11.650574 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x4eb98c0] 2014-06-09 16:22:11.650575 7f86a5e11700 5 _do_op 0x306068a0 seq 3336 osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 start 2014-06-09 16:22:11.650576 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650573 7f869af78700 5 -- op tracker -- , seq: 2356, time: 2014-06-09 16:22:11.311267, event: all_read, op: pg_info(1 pgs e243:2.3de) 2014-06-09 16:22:11.650577 7f86a5e11700 10 _do_transaction on 0xc2c4b40 2014-06-09 16:22:11.650577 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650581 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650580 7f869af78700 5 -- op tracker -- , seq: 2356, time: 2014-06-09 16:22:11.650548, event: dispatched, op: pg_info(1 pgs e243:2.3de) 2014-06-09 16:22:11.650587 7f869af78700 5 -- op tracker -- , seq: 2356, time: 2014-06-09 16:22:11.650585, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3de) 2014-06-09 16:22:11.650596 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3de) v4 from osd.1 2014-06-09 16:22:11.650599 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a038000 2014-06-09 16:22:11.650601 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650604 7f869af78700 5 -- op tracker -- , seq: 2356, time: 2014-06-09 16:22:11.650602, event: started, op: pg_info(1 pgs e243:2.3de) 2014-06-09 16:22:11.650612 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3336.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650614 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650621 7f869af78700 5 -- op tracker -- , seq: 2356, time: 2014-06-09 16:22:11.650619, event: done, op: pg_info(1 pgs e243:2.3de) 2014-06-09 16:22:11.650633 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650633 7f86a5e11700 10 _do_op 0x306068a0 seq 3336 r = 0, finisher 0xc411a00 0 2014-06-09 16:22:11.650636 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650638 7f86a5e11700 10 _finish_op 0x306068a0 seq 3336 osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 2014-06-09 16:22:11.650640 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524000 (1 active) 2014-06-09 16:22:11.650641 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3010e0 (2 active) 2014-06-09 16:22:11.650640 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7c920] 2014-06-09 16:22:11.650642 7f86a5e11700 5 _do_op 0x2fcaecc0 seq 3337 osr(3.27b 0x3ed0bd0)/0x3ed0bd0 start 2014-06-09 16:22:11.650644 7f86a5e11700 10 _do_transaction on 0xbf50fc0 2014-06-09 16:22:11.650645 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7c920] 2014-06-09 16:22:11.650647 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650648 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650648 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650649 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650651 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650652 7f869af78700 20 osd.26 243 _dispatch 0x1d0396c0 pg_info(1 pgs e243:1.56c) v4 2014-06-09 16:22:11.650657 7f869af78700 5 -- op tracker -- , seq: 2357, time: 2014-06-09 16:22:11.399766, event: header_read, op: pg_info(1 pgs e243:1.56c) 2014-06-09 16:22:11.650663 7f869af78700 5 -- op tracker -- , seq: 2357, time: 2014-06-09 16:22:11.399767, event: throttled, op: pg_info(1 pgs e243:1.56c) 2014-06-09 16:22:11.650673 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650675 7f869af78700 5 -- op tracker -- , seq: 2357, time: 2014-06-09 16:22:11.399779, event: all_read, op: pg_info(1 pgs e243:1.56c) 2014-06-09 16:22:11.650684 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3337.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650686 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650681 7f869af78700 5 -- op tracker -- , seq: 2357, time: 2014-06-09 16:22:11.650648, event: dispatched, op: pg_info(1 pgs e243:1.56c) 2014-06-09 16:22:11.650691 7f869af78700 5 -- op tracker -- , seq: 2357, time: 2014-06-09 16:22:11.650689, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.56c) 2014-06-09 16:22:11.650696 7f86a5e11700 10 _do_op 0x2fcaecc0 seq 3337 r = 0, finisher 0x3061e700 0 2014-06-09 16:22:11.650696 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.56c) v4 from osd.9 2014-06-09 16:22:11.650699 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d0396c0 2014-06-09 16:22:11.650701 7f86a5e11700 10 _finish_op 0x2fcaecc0 seq 3337 osr(3.27b 0x3ed0bd0)/0x3ed0bd0 2014-06-09 16:22:11.650703 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3010e0 (1 active) 2014-06-09 16:22:11.650704 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524a20 (2 active) 2014-06-09 16:22:11.650703 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x18742300] 2014-06-09 16:22:11.650705 7f86a5e11700 5 _do_op 0x30d973e0 seq 3338 osr(0.58a 0x2bd7770)/0x2bd7770 start 2014-06-09 16:22:11.650707 7f86a5e11700 10 _do_transaction on 0xbf52600 2014-06-09 16:22:11.650704 7f869af78700 5 -- op tracker -- , seq: 2357, time: 2014-06-09 16:22:11.650702, event: started, op: pg_info(1 pgs e243:1.56c) 2014-06-09 16:22:11.650707 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x18742300] 2014-06-09 16:22:11.650709 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650710 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650710 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650720 7f869af78700 5 -- op tracker -- , seq: 2357, time: 2014-06-09 16:22:11.650717, event: done, op: pg_info(1 pgs e243:1.56c) 2014-06-09 16:22:11.650727 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650729 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650737 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650742 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650744 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650745 7f869af78700 20 osd.26 243 _dispatch 0x3104aa80 pg_info(1 pgs e243:3.70e(2)) v4 2014-06-09 16:22:11.650748 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3338.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650750 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650758 7f869af78700 5 -- op tracker -- , seq: 2358, time: 2014-06-09 16:22:11.399683, event: header_read, op: pg_info(1 pgs e243:3.70e(2)) 2014-06-09 16:22:11.650767 7f86a5e11700 10 _do_op 0x30d973e0 seq 3338 r = 0, finisher 0xbd68ae0 0 2014-06-09 16:22:11.650770 7f869af78700 5 -- op tracker -- , seq: 2358, time: 2014-06-09 16:22:11.399683, event: throttled, op: pg_info(1 pgs e243:3.70e(2)) 2014-06-09 16:22:11.650776 7f869af78700 5 -- op tracker -- , seq: 2358, time: 2014-06-09 16:22:11.399697, event: all_read, op: pg_info(1 pgs e243:3.70e(2)) 2014-06-09 16:22:11.650782 7f86a5e11700 10 _finish_op 0x30d973e0 seq 3338 osr(0.58a 0x2bd7770)/0x2bd7770 2014-06-09 16:22:11.650785 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524a20 (1 active) 2014-06-09 16:22:11.650786 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f560 (2 active) 2014-06-09 16:22:11.650787 7f86a5e11700 5 _do_op 0x308cafc0 seq 3339 osr(3.400 0x3ed1260)/0x3ed1260 start 2014-06-09 16:22:11.650785 7f869af78700 5 -- op tracker -- , seq: 2358, time: 2014-06-09 16:22:11.650741, event: dispatched, op: pg_info(1 pgs e243:3.70e(2)) 2014-06-09 16:22:11.650789 7f86a5e11700 10 _do_transaction on 0x2fb6c00 2014-06-09 16:22:11.650793 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650792 7f869af78700 5 -- op tracker -- , seq: 2358, time: 2014-06-09 16:22:11.650790, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.70e(2)) 2014-06-09 16:22:11.650797 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.70e(2)) v4 from osd.8 2014-06-09 16:22:11.650809 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104aa80 2014-06-09 16:22:11.650813 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650815 7f869af78700 5 -- op tracker -- , seq: 2358, time: 2014-06-09 16:22:11.650812, event: started, op: pg_info(1 pgs e243:3.70e(2)) 2014-06-09 16:22:11.650837 7f869af78700 5 -- op tracker -- , seq: 2358, time: 2014-06-09 16:22:11.650835, event: done, op: pg_info(1 pgs e243:3.70e(2)) 2014-06-09 16:22:11.650838 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3339.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650843 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650845 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650847 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650847 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94720] 2014-06-09 16:22:11.650851 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94720] 2014-06-09 16:22:11.650853 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650854 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650859 7f86a5e11700 10 _do_op 0x308cafc0 seq 3339 r = 0, finisher 0xbdf19a0 0 2014-06-09 16:22:11.650866 7f86a5e11700 10 _finish_op 0x308cafc0 seq 3339 osr(3.400 0x3ed1260)/0x3ed1260 2014-06-09 16:22:11.650866 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061e4e0] 2014-06-09 16:22:11.650869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061e4e0] 2014-06-09 16:22:11.650870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650869 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f560 (1 active) 2014-06-09 16:22:11.650871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650870 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650871 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b9e0 (2 active) 2014-06-09 16:22:11.650871 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650873 7f86a5e11700 5 _do_op 0x30d96b40 seq 3340 osr(3.2d6 0x3ed0e60)/0x3ed0e60 start 2014-06-09 16:22:11.650873 7f869af78700 20 osd.26 243 _dispatch 0x310cbdc0 pg_notify(0.1dc(20) epoch 243) v5 2014-06-09 16:22:11.650876 7f86a5e11700 10 _do_transaction on 0xbf28900 2014-06-09 16:22:11.650879 7f869af78700 5 -- op tracker -- , seq: 2359, time: 2014-06-09 16:22:11.388676, event: header_read, op: pg_notify(0.1dc(20) epoch 243) 2014-06-09 16:22:11.650882 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650886 7f869af78700 5 -- op tracker -- , seq: 2359, time: 2014-06-09 16:22:11.388677, event: throttled, op: pg_notify(0.1dc(20) epoch 243) 2014-06-09 16:22:11.650892 7f869af78700 5 -- op tracker -- , seq: 2359, time: 2014-06-09 16:22:11.388706, event: all_read, op: pg_notify(0.1dc(20) epoch 243) 2014-06-09 16:22:11.650900 7f869af78700 5 -- op tracker -- , seq: 2359, time: 2014-06-09 16:22:11.650869, event: dispatched, op: pg_notify(0.1dc(20) epoch 243) 2014-06-09 16:22:11.650908 7f869af78700 5 -- op tracker -- , seq: 2359, time: 2014-06-09 16:22:11.650906, event: waiting_for_osdmap, op: pg_notify(0.1dc(20) epoch 243) 2014-06-09 16:22:11.650912 7f869af78700 7 osd.26 243 handle_pg_notify from osd.6 2014-06-09 16:22:11.650914 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cbdc0 2014-06-09 16:22:11.650917 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650919 7f869af78700 5 -- op tracker -- , seq: 2359, time: 2014-06-09 16:22:11.650917, event: started, op: pg_notify(0.1dc(20) epoch 243) 2014-06-09 16:22:11.650930 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3340.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650933 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650935 7f869af78700 5 -- op tracker -- , seq: 2359, time: 2014-06-09 16:22:11.650933, event: done, op: pg_notify(0.1dc(20) epoch 243) 2014-06-09 16:22:11.650943 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650945 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650948 7f86a5e11700 10 _do_op 0x30d96b40 seq 3340 r = 0, finisher 0xbfce9a0 0 2014-06-09 16:22:11.650953 7f86a5e11700 10 _finish_op 0x30d96b40 seq 3340 osr(3.2d6 0x3ed0e60)/0x3ed0e60 2014-06-09 16:22:11.650954 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6bd00] 2014-06-09 16:22:11.650956 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b9e0 (1 active) 2014-06-09 16:22:11.650957 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62ea20 (2 active) 2014-06-09 16:22:11.650958 7f86a5e11700 5 _do_op 0x309a3440 seq 3341 osr(2.194 0x2ed0910)/0x2ed0910 start 2014-06-09 16:22:11.650960 7f86a5e11700 10 _do_transaction on 0xc29a480 2014-06-09 16:22:11.650959 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6bd00] 2014-06-09 16:22:11.650960 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.650962 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.650963 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.650963 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.650965 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.650966 7f869af78700 20 osd.26 243 _dispatch 0x30bd8380 pg_info(1 pgs e243:3.7ce) v4 2014-06-09 16:22:11.650972 7f869af78700 5 -- op tracker -- , seq: 2360, time: 2014-06-09 16:22:11.371060, event: header_read, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.650979 7f869af78700 5 -- op tracker -- , seq: 2360, time: 2014-06-09 16:22:11.371060, event: throttled, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.650983 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.650985 7f869af78700 5 -- op tracker -- , seq: 2360, time: 2014-06-09 16:22:11.371070, event: all_read, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.650994 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3341.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.650998 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.650992 7f869af78700 5 -- op tracker -- , seq: 2360, time: 2014-06-09 16:22:11.650963, event: dispatched, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.651004 7f869af78700 5 -- op tracker -- , seq: 2360, time: 2014-06-09 16:22:11.651002, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.651009 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7ce) v4 from osd.0 2014-06-09 16:22:11.651012 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd8380 2014-06-09 16:22:11.651017 7f86a5e11700 10 _do_op 0x309a3440 seq 3341 r = 0, finisher 0xc6e8400 0 2014-06-09 16:22:11.651017 7f869af78700 5 -- op tracker -- , seq: 2360, time: 2014-06-09 16:22:11.651015, event: started, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.651022 7f86a5e11700 10 _finish_op 0x309a3440 seq 3341 osr(2.194 0x2ed0910)/0x2ed0910 2014-06-09 16:22:11.651023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6f0a0] 2014-06-09 16:22:11.651024 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62ea20 (1 active) 2014-06-09 16:22:11.651026 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3faa20 (2 active) 2014-06-09 16:22:11.651026 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6f0a0] 2014-06-09 16:22:11.651027 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.651028 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.651027 7f86a5e11700 5 _do_op 0x100396e0 seq 3342 osr(2.52d 0x2ed19f0)/0x2ed19f0 start 2014-06-09 16:22:11.651029 7f86a5e11700 10 _do_transaction on 0xc27e480 2014-06-09 16:22:11.651033 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.651034 7f869af78700 5 -- op tracker -- , seq: 2360, time: 2014-06-09 16:22:11.651032, event: done, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.651044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651046 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651053 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.651060 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651062 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651063 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3342.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.651066 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.651063 7f869af78700 20 osd.26 243 _dispatch 0x308b5e00 pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.651070 7f869af78700 5 -- op tracker -- , seq: 2361, time: 2014-06-09 16:22:11.397409, event: header_read, op: pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.651080 7f869af78700 5 -- op tracker -- , seq: 2361, time: 2014-06-09 16:22:11.397410, event: throttled, op: pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.651087 7f869af78700 5 -- op tracker -- , seq: 2361, time: 2014-06-09 16:22:11.397427, event: all_read, op: pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.651090 7f86a5e11700 10 _do_op 0x100396e0 seq 3342 r = 0, finisher 0x139655c0 0 2014-06-09 16:22:11.651096 7f86a5e11700 10 _finish_op 0x100396e0 seq 3342 osr(2.52d 0x2ed19f0)/0x2ed19f0 2014-06-09 16:22:11.651095 7f869af78700 5 -- op tracker -- , seq: 2361, time: 2014-06-09 16:22:11.651060, event: dispatched, op: pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.651098 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3faa20 (1 active) 2014-06-09 16:22:11.651098 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x13964cc0] 2014-06-09 16:22:11.651100 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1b00 (2 active) 2014-06-09 16:22:11.651101 7f86a5e11700 5 _do_op 0x2fcaf860 seq 3343 osr(0.66a 0x2bd7ef0)/0x2bd7ef0 start 2014-06-09 16:22:11.651101 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x13964cc0] 2014-06-09 16:22:11.651103 7f86a5e11700 10 _do_transaction on 0xc2c6900 2014-06-09 16:22:11.651103 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.651103 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.651108 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.651141 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.651153 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3343.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.651156 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.651158 7f869af78700 5 -- op tracker -- , seq: 2361, time: 2014-06-09 16:22:11.651147, event: waiting_for_osdmap, op: pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.651166 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.651168 7f86a5e11700 10 _do_op 0x2fcaf860 seq 3343 r = 0, finisher 0xcd9efc0 0 2014-06-09 16:22:11.651170 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308b5e00 2014-06-09 16:22:11.651172 7f86a5e11700 10 _finish_op 0x2fcaf860 seq 3343 osr(0.66a 0x2bd7ef0)/0x2bd7ef0 2014-06-09 16:22:11.651174 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1b00 (1 active) 2014-06-09 16:22:11.651175 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287200 (2 active) 2014-06-09 16:22:11.651175 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2d739d60] 2014-06-09 16:22:11.651177 7f86a5e11700 5 _do_op 0x308ca960 seq 3344 osr(1.302 0x384d010)/0x384d010 start 2014-06-09 16:22:11.651178 7f86a5e11700 10 _do_transaction on 0xc28c0c0 2014-06-09 16:22:11.651179 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2d739d60] 2014-06-09 16:22:11.651180 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.651181 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.651181 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.651176 7f869af78700 5 -- op tracker -- , seq: 2361, time: 2014-06-09 16:22:11.651173, event: started, op: pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.651201 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.651202 7f869af78700 5 -- op tracker -- , seq: 2361, time: 2014-06-09 16:22:11.651198, event: done, op: pg_log(2.4a7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.651223 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3344.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.651228 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.651247 7f86a5e11700 10 _do_op 0x308ca960 seq 3344 r = 0, finisher 0xc168da0 0 2014-06-09 16:22:11.651247 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651250 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651252 7f86a5e11700 10 _finish_op 0x308ca960 seq 3344 osr(1.302 0x384d010)/0x384d010 2014-06-09 16:22:11.651255 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287200 (1 active) 2014-06-09 16:22:11.651257 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a6c0 (2 active) 2014-06-09 16:22:11.651259 7f86a5e11700 5 _do_op 0x2fcae2a0 seq 3345 osr(3.1d5 0x3ed09f0)/0x3ed09f0 start 2014-06-09 16:22:11.651262 7f86a5e11700 10 _do_transaction on 0xc2d8600 2014-06-09 16:22:11.651275 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.651291 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061f4c0] 2014-06-09 16:22:11.651295 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061f4c0] 2014-06-09 16:22:11.651296 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.651297 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.651298 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.651307 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651309 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651309 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3345.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.651312 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.651310 7f869af78700 20 osd.26 243 _dispatch 0x1c03fa40 pg_info(1 pgs e243:3.1eb(4)) v4 2014-06-09 16:22:11.651319 7f869af78700 5 -- op tracker -- , seq: 2362, time: 2014-06-09 16:22:11.263708, event: header_read, op: pg_info(1 pgs e243:3.1eb(4)) 2014-06-09 16:22:11.651325 7f86a5e11700 10 _do_op 0x2fcae2a0 seq 3345 r = 0, finisher 0xcc838a0 0 2014-06-09 16:22:11.651327 7f869af78700 5 -- op tracker -- , seq: 2362, time: 2014-06-09 16:22:11.263709, event: throttled, op: pg_info(1 pgs e243:3.1eb(4)) 2014-06-09 16:22:11.651332 7f86a5e11700 10 _finish_op 0x2fcae2a0 seq 3345 osr(3.1d5 0x3ed09f0)/0x3ed09f0 2014-06-09 16:22:11.651335 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a6c0 (1 active) 2014-06-09 16:22:11.651334 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30628700] 2014-06-09 16:22:11.651334 7f869af78700 5 -- op tracker -- , seq: 2362, time: 2014-06-09 16:22:11.263723, event: all_read, op: pg_info(1 pgs e243:3.1eb(4)) 2014-06-09 16:22:11.651337 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0240 (2 active) 2014-06-09 16:22:11.651339 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30628700] 2014-06-09 16:22:11.651339 7f86a5e11700 5 _do_op 0x2fcafe00 seq 3346 osr(1.601 0x384dcd0)/0x384dcd0 start 2014-06-09 16:22:11.651341 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.651342 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.651342 7f86a5e11700 10 _do_transaction on 0xc3375c0 2014-06-09 16:22:11.651340 7f869af78700 5 -- op tracker -- , seq: 2362, time: 2014-06-09 16:22:11.651306, event: dispatched, op: pg_info(1 pgs e243:3.1eb(4)) 2014-06-09 16:22:11.651349 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.651353 7f869af78700 5 -- op tracker -- , seq: 2362, time: 2014-06-09 16:22:11.651350, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1eb(4)) 2014-06-09 16:22:11.651357 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1eb(4)) v4 from osd.2 2014-06-09 16:22:11.651361 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03fa40 2014-06-09 16:22:11.651366 7f869af78700 5 -- op tracker -- , seq: 2362, time: 2014-06-09 16:22:11.651364, event: started, op: pg_info(1 pgs e243:3.1eb(4)) 2014-06-09 16:22:11.651381 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.651384 7f869af78700 5 -- op tracker -- , seq: 2362, time: 2014-06-09 16:22:11.651381, event: done, op: pg_info(1 pgs e243:3.1eb(4)) 2014-06-09 16:22:11.651397 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651397 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3346.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.651401 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.651426 7f86a5e11700 10 _do_op 0x2fcafe00 seq 3346 r = 0, finisher 0x33a7620 0 2014-06-09 16:22:11.651433 7f86a5e11700 10 _finish_op 0x2fcafe00 seq 3346 osr(1.601 0x384dcd0)/0x384dcd0 2014-06-09 16:22:11.651434 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30629140] 2014-06-09 16:22:11.651436 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0240 (1 active) 2014-06-09 16:22:11.651438 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30629140] 2014-06-09 16:22:11.651438 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287d40 (2 active) 2014-06-09 16:22:11.651439 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.651440 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.651441 7f86a5e11700 5 _do_op 0x15997c80 seq 3347 osr(3.52a 0x3ed1670)/0x3ed1670 start 2014-06-09 16:22:11.651443 7f86a5e11700 10 _do_transaction on 0xc0e9d40 2014-06-09 16:22:11.651449 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.651471 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651473 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651474 7f869af78700 20 osd.26 243 _dispatch 0x2d740c40 pg_notify(1.43b(23) epoch 243) v5 2014-06-09 16:22:11.651480 7f869af78700 5 -- op tracker -- , seq: 2363, time: 2014-06-09 16:22:11.330116, event: header_read, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.651487 7f869af78700 5 -- op tracker -- , seq: 2363, time: 2014-06-09 16:22:11.330117, event: throttled, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.651489 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.651493 7f869af78700 5 -- op tracker -- , seq: 2363, time: 2014-06-09 16:22:11.330156, event: all_read, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.651505 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3347.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.651511 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.651512 7f869af78700 5 -- op tracker -- , seq: 2363, time: 2014-06-09 16:22:11.651471, event: dispatched, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.651520 7f869af78700 5 -- op tracker -- , seq: 2363, time: 2014-06-09 16:22:11.651517, event: waiting_for_osdmap, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.651525 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.651527 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d740c40 2014-06-09 16:22:11.651528 7f86a5e11700 10 _do_op 0x15997c80 seq 3347 r = 0, finisher 0x18742520 0 2014-06-09 16:22:11.651533 7f86a5e11700 10 _finish_op 0x15997c80 seq 3347 osr(3.52a 0x3ed1670)/0x3ed1670 2014-06-09 16:22:11.651534 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd766e0] 2014-06-09 16:22:11.651532 7f869af78700 5 -- op tracker -- , seq: 2363, time: 2014-06-09 16:22:11.651530, event: started, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.651536 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287d40 (1 active) 2014-06-09 16:22:11.651537 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd766e0] 2014-06-09 16:22:11.651538 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ec60 (2 active) 2014-06-09 16:22:11.651539 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.651541 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.651541 7f86a5e11700 5 _do_op 0x308cb8c0 seq 3348 osr(1.8c 0x384c470)/0x384c470 start 2014-06-09 16:22:11.651544 7f86a5e11700 10 _do_transaction on 0x35409c0 2014-06-09 16:22:11.651550 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.651549 7f869af78700 5 -- op tracker -- , seq: 2363, time: 2014-06-09 16:22:11.651547, event: done, op: pg_notify(1.43b(23) epoch 243) 2014-06-09 16:22:11.651558 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651560 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651579 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651581 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651582 7f869af78700 20 osd.26 243 _dispatch 0x30d24c40 pg_info(1 pgs e243:1.4db) v4 2014-06-09 16:22:11.651583 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.651587 7f869af78700 5 -- op tracker -- , seq: 2364, time: 2014-06-09 16:22:11.390451, event: header_read, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.651595 7f869af78700 5 -- op tracker -- , seq: 2364, time: 2014-06-09 16:22:11.390452, event: throttled, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.651599 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3348.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.651604 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.651601 7f869af78700 5 -- op tracker -- , seq: 2364, time: 2014-06-09 16:22:11.390466, event: all_read, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.651608 7f869af78700 5 -- op tracker -- , seq: 2364, time: 2014-06-09 16:22:11.651578, event: dispatched, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.651626 7f869af78700 5 -- op tracker -- , seq: 2364, time: 2014-06-09 16:22:11.651613, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.651631 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4db) v4 from osd.7 2014-06-09 16:22:11.651635 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d24c40 2014-06-09 16:22:11.651640 7f869af78700 5 -- op tracker -- , seq: 2364, time: 2014-06-09 16:22:11.651638, event: started, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.651649 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbebfd80,0xbd694c0,0xc5a7b40,0x292823a0,0xbd6de80,0xbdba360,0xbdba300,0xcc94640,0x1b0c10c0,0xbd68a20,0xbd7dec0,0xcc951e0,0xc411a00,0x3061e700,0xbd68ae0,0xbdf19a0,0xbfce9a0,0xc6e8400,0x139655c0,0xcd9efc0,0xc168da0,0xcc838a0,0x33a7620,0x18742520] 2014-06-09 16:22:11.651655 7f869af78700 5 -- op tracker -- , seq: 2364, time: 2014-06-09 16:22:11.651653, event: done, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.651663 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651665 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651658 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.651673 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.651677 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651678 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651680 7f869af78700 20 osd.26 243 _dispatch 0x2eef6c40 pg_info(1 pgs e243:3.ed) v4 2014-06-09 16:22:11.651685 7f869af78700 5 -- op tracker -- , seq: 2365, time: 2014-06-09 16:22:11.437090, event: header_read, op: pg_info(1 pgs e243:3.ed) 2014-06-09 16:22:11.651687 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.651693 7f869af78700 5 -- op tracker -- , seq: 2365, time: 2014-06-09 16:22:11.437091, event: throttled, op: pg_info(1 pgs e243:3.ed) 2014-06-09 16:22:11.651699 7f869af78700 5 -- op tracker -- , seq: 2365, time: 2014-06-09 16:22:11.437101, event: all_read, op: pg_info(1 pgs e243:3.ed) 2014-06-09 16:22:11.651706 7f869af78700 5 -- op tracker -- , seq: 2365, time: 2014-06-09 16:22:11.651676, event: dispatched, op: pg_info(1 pgs e243:3.ed) 2014-06-09 16:22:11.651703 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.651713 7f869af78700 5 -- op tracker -- , seq: 2365, time: 2014-06-09 16:22:11.651711, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.ed) 2014-06-09 16:22:11.651718 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.651718 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.ed) v4 from osd.3 2014-06-09 16:22:11.651721 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef6c40 2014-06-09 16:22:11.651726 7f869af78700 5 -- op tracker -- , seq: 2365, time: 2014-06-09 16:22:11.651724, event: started, op: pg_info(1 pgs e243:3.ed) 2014-06-09 16:22:11.651733 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.651749 7f869af78700 5 -- op tracker -- , seq: 2365, time: 2014-06-09 16:22:11.651746, event: done, op: pg_info(1 pgs e243:3.ed) 2014-06-09 16:22:11.651750 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.651760 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651762 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651763 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.651774 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651776 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651778 7f869af78700 20 osd.26 243 _dispatch 0x1a038a80 pg_info(1 pgs e243:0.105) v4 2014-06-09 16:22:11.651777 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.651783 7f869af78700 5 -- op tracker -- , seq: 2366, time: 2014-06-09 16:22:11.311276, event: header_read, op: pg_info(1 pgs e243:0.105) 2014-06-09 16:22:11.651790 7f869af78700 5 -- op tracker -- , seq: 2366, time: 2014-06-09 16:22:11.311277, event: throttled, op: pg_info(1 pgs e243:0.105) 2014-06-09 16:22:11.651796 7f869af78700 5 -- op tracker -- , seq: 2366, time: 2014-06-09 16:22:11.311284, event: all_read, op: pg_info(1 pgs e243:0.105) 2014-06-09 16:22:11.651793 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.651803 7f869af78700 5 -- op tracker -- , seq: 2366, time: 2014-06-09 16:22:11.651774, event: dispatched, op: pg_info(1 pgs e243:0.105) 2014-06-09 16:22:11.651809 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.651812 7f869af78700 5 -- op tracker -- , seq: 2366, time: 2014-06-09 16:22:11.651810, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.105) 2014-06-09 16:22:11.651817 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.105) v4 from osd.1 2014-06-09 16:22:11.651820 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a038a80 2014-06-09 16:22:11.651822 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.651825 7f869af78700 5 -- op tracker -- , seq: 2366, time: 2014-06-09 16:22:11.651823, event: started, op: pg_info(1 pgs e243:0.105) 2014-06-09 16:22:11.651839 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.651845 7f869af78700 5 -- op tracker -- , seq: 2366, time: 2014-06-09 16:22:11.651843, event: done, op: pg_info(1 pgs e243:0.105) 2014-06-09 16:22:11.651853 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651852 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.651855 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651868 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.651883 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.651896 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.651896 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.651899 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.651900 7f869af78700 20 osd.26 243 _dispatch 0x3046a1c0 pg_notify(3.4c7(10) epoch 243) v5 2014-06-09 16:22:11.651912 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.651906 7f869af78700 5 -- op tracker -- , seq: 2367, time: 2014-06-09 16:22:11.399788, event: header_read, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.651926 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.651936 7f869af78700 5 -- op tracker -- , seq: 2367, time: 2014-06-09 16:22:11.399789, event: throttled, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.651941 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.651944 7f869af78700 5 -- op tracker -- , seq: 2367, time: 2014-06-09 16:22:11.399808, event: all_read, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.651955 7f869af78700 5 -- op tracker -- , seq: 2367, time: 2014-06-09 16:22:11.651896, event: dispatched, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.651954 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.651963 7f869af78700 5 -- op tracker -- , seq: 2367, time: 2014-06-09 16:22:11.651961, event: waiting_for_osdmap, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.651968 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.651970 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046a1c0 2014-06-09 16:22:11.651975 7f869af78700 5 -- op tracker -- , seq: 2367, time: 2014-06-09 16:22:11.651973, event: started, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.651972 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.651998 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.651998 7f869af78700 5 -- op tracker -- , seq: 2367, time: 2014-06-09 16:22:11.651995, event: done, op: pg_notify(3.4c7(10) epoch 243) 2014-06-09 16:22:11.652020 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652012 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.652022 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652029 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652039 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652039 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.652040 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652042 7f869af78700 20 osd.26 243 _dispatch 0x305f5000 pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.652052 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.652067 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652049 7f869af78700 5 -- op tracker -- , seq: 2368, time: 2014-06-09 16:22:11.399701, event: header_read, op: pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.652080 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.652084 7f869af78700 5 -- op tracker -- , seq: 2368, time: 2014-06-09 16:22:11.399702, event: throttled, op: pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.652096 7f869af78700 5 -- op tracker -- , seq: 2368, time: 2014-06-09 16:22:11.399713, event: all_read, op: pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.652103 7f869af78700 5 -- op tracker -- , seq: 2368, time: 2014-06-09 16:22:11.652038, event: dispatched, op: pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.652093 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.652115 7f869af78700 5 -- op tracker -- , seq: 2368, time: 2014-06-09 16:22:11.652111, event: waiting_for_osdmap, op: pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.652123 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.652127 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f5000 2014-06-09 16:22:11.652120 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652133 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.652133 7f869af78700 5 -- op tracker -- , seq: 2368, time: 2014-06-09 16:22:11.652130, event: started, op: pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.652148 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.652164 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652177 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.652175 7f869af78700 5 -- op tracker -- , seq: 2368, time: 2014-06-09 16:22:11.652169, event: done, op: pg_log(0.298 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.652184 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652186 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652198 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652192 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.652200 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652201 7f869af78700 20 osd.26 243 _dispatch 0x310cbc00 pg_info(1 pgs e243:0.230) v4 2014-06-09 16:22:11.652206 7f869af78700 5 -- op tracker -- , seq: 2369, time: 2014-06-09 16:22:11.400056, event: header_read, op: pg_info(1 pgs e243:0.230) 2014-06-09 16:22:11.652206 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652213 7f869af78700 5 -- op tracker -- , seq: 2369, time: 2014-06-09 16:22:11.400057, event: throttled, op: pg_info(1 pgs e243:0.230) 2014-06-09 16:22:11.652221 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.652222 7f869af78700 5 -- op tracker -- , seq: 2369, time: 2014-06-09 16:22:11.400071, event: all_read, op: pg_info(1 pgs e243:0.230) 2014-06-09 16:22:11.652229 7f869af78700 5 -- op tracker -- , seq: 2369, time: 2014-06-09 16:22:11.652198, event: dispatched, op: pg_info(1 pgs e243:0.230) 2014-06-09 16:22:11.652236 7f869af78700 5 -- op tracker -- , seq: 2369, time: 2014-06-09 16:22:11.652234, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.230) 2014-06-09 16:22:11.652236 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.652245 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.230) v4 from osd.6 2014-06-09 16:22:11.652248 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cbc00 2014-06-09 16:22:11.652254 7f869af78700 5 -- op tracker -- , seq: 2369, time: 2014-06-09 16:22:11.652251, event: started, op: pg_info(1 pgs e243:0.230) 2014-06-09 16:22:11.652258 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652272 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.652269 7f869af78700 5 -- op tracker -- , seq: 2369, time: 2014-06-09 16:22:11.652267, event: done, op: pg_info(1 pgs e243:0.230) 2014-06-09 16:22:11.652279 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652282 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652294 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652286 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.652295 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652297 7f869af78700 20 osd.26 243 _dispatch 0x31041dc0 pg_info(1 pgs e243:1.30) v4 2014-06-09 16:22:11.652302 7f869af78700 5 -- op tracker -- , seq: 2370, time: 2014-06-09 16:22:11.371074, event: header_read, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.652302 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652309 7f869af78700 5 -- op tracker -- , seq: 2370, time: 2014-06-09 16:22:11.371074, event: throttled, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.652316 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.652317 7f869af78700 5 -- op tracker -- , seq: 2370, time: 2014-06-09 16:22:11.371084, event: all_read, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.652332 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.652346 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652361 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.652355 7f869af78700 5 -- op tracker -- , seq: 2370, time: 2014-06-09 16:22:11.652293, event: dispatched, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.652368 7f869af78700 5 -- op tracker -- , seq: 2370, time: 2014-06-09 16:22:11.652366, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.652375 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.652388 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652395 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.30) v4 from osd.0 2014-06-09 16:22:11.652400 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31041dc0 2014-06-09 16:22:11.652402 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.652405 7f869af78700 5 -- op tracker -- , seq: 2370, time: 2014-06-09 16:22:11.652403, event: started, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.652416 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.652422 7f869af78700 5 -- op tracker -- , seq: 2370, time: 2014-06-09 16:22:11.652420, event: done, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.652430 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652428 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.194( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652441 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.652456 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.194( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.652470 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652478 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652482 7f869af78700 20 osd.26 243 _dispatch 0x310ca000 pg_notify(0.1ca(23) epoch 243) v5 2014-06-09 16:22:11.652483 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.652490 7f869af78700 5 -- op tracker -- , seq: 2371, time: 2014-06-09 16:22:11.399651, event: header_read, op: pg_notify(0.1ca(23) epoch 243) 2014-06-09 16:22:11.652498 7f869af78700 5 -- op tracker -- , seq: 2371, time: 2014-06-09 16:22:11.399652, event: throttled, op: pg_notify(0.1ca(23) epoch 243) 2014-06-09 16:22:11.652496 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.652505 7f869af78700 5 -- op tracker -- , seq: 2371, time: 2014-06-09 16:22:11.399689, event: all_read, op: pg_notify(0.1ca(23) epoch 243) 2014-06-09 16:22:11.652511 7f869af78700 5 -- op tracker -- , seq: 2371, time: 2014-06-09 16:22:11.652453, event: dispatched, op: pg_notify(0.1ca(23) epoch 243) 2014-06-09 16:22:11.652512 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652521 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.652520 7f869af78700 5 -- op tracker -- , seq: 2371, time: 2014-06-09 16:22:11.652518, event: waiting_for_osdmap, op: pg_notify(0.1ca(23) epoch 243) 2014-06-09 16:22:11.652525 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.652527 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310ca000 2014-06-09 16:22:11.652533 7f869af78700 5 -- op tracker -- , seq: 2371, time: 2014-06-09 16:22:11.652530, event: started, op: pg_notify(0.1ca(23) epoch 243) 2014-06-09 16:22:11.652534 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.652548 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652560 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.652560 7f869af78700 5 -- op tracker -- , seq: 2371, time: 2014-06-09 16:22:11.652558, event: done, op: pg_notify(0.1ca(23) epoch 243) 2014-06-09 16:22:11.652576 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652578 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652574 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.652600 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652601 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652593 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652603 7f869af78700 20 osd.26 243 _dispatch 0x240efdc0 pg_notify(0.1f1(25),1.1f0(26) epoch 243) v5 2014-06-09 16:22:11.652608 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.652609 7f869af78700 5 -- op tracker -- , seq: 2372, time: 2014-06-09 16:22:11.263747, event: header_read, op: pg_notify(0.1f1(25),1.1f0(26) epoch 243) 2014-06-09 16:22:11.652617 7f869af78700 5 -- op tracker -- , seq: 2372, time: 2014-06-09 16:22:11.263747, event: throttled, op: pg_notify(0.1f1(25),1.1f0(26) epoch 243) 2014-06-09 16:22:11.652624 7f869af78700 5 -- op tracker -- , seq: 2372, time: 2014-06-09 16:22:11.263810, event: all_read, op: pg_notify(0.1f1(25),1.1f0(26) epoch 243) 2014-06-09 16:22:11.652631 7f869af78700 5 -- op tracker -- , seq: 2372, time: 2014-06-09 16:22:11.652599, event: dispatched, op: pg_notify(0.1f1(25),1.1f0(26) epoch 243) 2014-06-09 16:22:11.652618 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.652638 7f869af78700 5 -- op tracker -- , seq: 2372, time: 2014-06-09 16:22:11.652636, event: waiting_for_osdmap, op: pg_notify(0.1f1(25),1.1f0(26) epoch 243) 2014-06-09 16:22:11.652643 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.652642 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.601( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652657 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.652645 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240efdc0 2014-06-09 16:22:11.652664 7f869af78700 5 -- op tracker -- , seq: 2372, time: 2014-06-09 16:22:11.652661, event: started, op: pg_notify(0.1f1(25),1.1f0(26) epoch 243) 2014-06-09 16:22:11.652672 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.601( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.652687 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652703 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.652717 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.652723 7f869af78700 5 -- op tracker -- , seq: 2372, time: 2014-06-09 16:22:11.652720, event: done, op: pg_notify(0.1f1(25),1.1f0(26) epoch 243) 2014-06-09 16:22:11.652731 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652733 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652733 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbebfd80,0xbd694c0,0xc5a7b40,0x292823a0,0xbd6de80,0xbdba360,0xbdba300,0xcc94640,0x1b0c10c0,0xbd68a20,0xbd7dec0,0xcc951e0,0xc411a00,0x3061e700,0xbd68ae0,0xbdf19a0,0xbfce9a0,0xc6e8400,0x139655c0,0xcd9efc0,0xc168da0,0xcc838a0,0x33a7620,0x18742520] 2014-06-09 16:22:11.652739 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.652740 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.652747 7f8692f68700 10 op_submit_finish 3367 2014-06-09 16:22:11.652751 7f86a5610700 10 _finish_op 0x30060300 seq 3324 osr(0.61f 0x2bd7d30)/0x2bd7d30 2014-06-09 16:22:11.652759 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52eb40 (1 active) 2014-06-09 16:22:11.652762 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac45a0 (2 active) 2014-06-09 16:22:11.652753 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.652765 7f86a5610700 5 _do_op 0x30606900 seq 3349 osr(0.4ba 0x2bd7710)/0x2bd7710 start 2014-06-09 16:22:11.652768 7f86a5610700 10 _do_transaction on 0xc05a240 2014-06-09 16:22:11.652768 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652770 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652772 7f869af78700 20 osd.26 243 _dispatch 0x1b0ce1c0 pg_info(1 pgs e243:3.11f(2)) v4 2014-06-09 16:22:11.652775 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.652767 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.652781 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.652785 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.652787 7f869af78700 5 -- op tracker -- , seq: 2373, time: 2014-06-09 16:22:11.330169, event: header_read, op: pg_info(1 pgs e243:3.11f(2)) 2014-06-09 16:22:11.652789 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.886502 7 0.000188 2014-06-09 16:22:11.652795 7f869af78700 5 -- op tracker -- , seq: 2373, time: 2014-06-09 16:22:11.330170, event: throttled, op: pg_info(1 pgs e243:3.11f(2)) 2014-06-09 16:22:11.652801 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.652809 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.652815 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd75e40] 2014-06-09 16:22:11.652815 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.652820 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.652823 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.652835 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.652832 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.652843 7f869af78700 5 -- op tracker -- , seq: 2373, time: 2014-06-09 16:22:11.330194, event: all_read, op: pg_info(1 pgs e243:3.11f(2)) 2014-06-09 16:22:11.652852 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.652859 7f869af78700 5 -- op tracker -- , seq: 2373, time: 2014-06-09 16:22:11.652767, event: dispatched, op: pg_info(1 pgs e243:3.11f(2)) 2014-06-09 16:22:11.652863 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd75e40] 2014-06-09 16:22:11.652864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.652865 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.652867 7f869af78700 5 -- op tracker -- , seq: 2373, time: 2014-06-09 16:22:11.652865, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.11f(2)) 2014-06-09 16:22:11.652871 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.11f(2)) v4 from osd.4 2014-06-09 16:22:11.652875 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1b0ce1c0 2014-06-09 16:22:11.652881 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.652882 7f868df5e700 10 op_submit_start 3368 2014-06-09 16:22:11.652883 7f868df5e700 5 queue_transactions (trailing journal) 3368 2014-06-09 16:22:11.652880 7f869af78700 5 -- op tracker -- , seq: 2373, time: 2014-06-09 16:22:11.652878, event: started, op: pg_info(1 pgs e243:3.11f(2)) 2014-06-09 16:22:11.652886 7f8692f68700 5 queue_transactions existing osr(0.47d 0x2bd7470)/0x2bd7470 2014-06-09 16:22:11.652884 7f868df5e700 5 queue_op 0xd4717a0 seq 3368 osr(2.4a5 0x2ed17f0) 1883 bytes (queue has 37 ops and 1246100 bytes) 2014-06-09 16:22:11.652888 7f868df5e700 10 op_submit_finish 3368 2014-06-09 16:22:11.652892 7f8689f56700 10 op_submit_start 3369 2014-06-09 16:22:11.652893 7f8689f56700 5 queue_transactions (trailing journal) 3369 2014-06-09 16:22:11.652895 7f8689f56700 5 queue_op 0x309a3da0 seq 3369 osr(2.1e3 0x2ed0810) 2001 bytes (queue has 39 ops and 1249984 bytes) 2014-06-09 16:22:11.652898 7f8689f56700 10 op_submit_finish 3369 2014-06-09 16:22:11.652900 7f869af78700 5 -- op tracker -- , seq: 2373, time: 2014-06-09 16:22:11.652897, event: done, op: pg_info(1 pgs e243:3.11f(2)) 2014-06-09 16:22:11.652893 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.5a7( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.652903 7f868ff62700 10 op_submit_start 3370 2014-06-09 16:22:11.652906 7f868ff62700 5 queue_transactions (trailing journal) 3370 2014-06-09 16:22:11.652908 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652907 7f868ff62700 5 queue_op 0x159965a0 seq 3370 osr(1.a 0x384c270) 1877 bytes (queue has 39 ops and 1249984 bytes) 2014-06-09 16:22:11.652910 7f868ff62700 10 op_submit_finish 3370 2014-06-09 16:22:11.652910 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652903 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 1 notify: (query_epoch:242, epoch_sent:243, info:1.446( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.652907 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.652918 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.652919 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0260 (19 active) 2014-06-09 16:22:11.652923 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.652918 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.652926 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.652927 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.652927 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.652927 7f869af78700 20 osd.26 243 _dispatch 0x30d24e00 pg_info(1 pgs e243:1.603) v4 2014-06-09 16:22:11.652923 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:0.5a8( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.652933 7f869af78700 5 -- op tracker -- , seq: 2374, time: 2014-06-09 16:22:11.390475, event: header_read, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.652935 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.652932 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 1 notify: (query_epoch:242, epoch_sent:243, info:3.709( v 213'575 (0'0,213'575] local-les=191 n=111 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.652945 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.652940 7f869af78700 5 -- op tracker -- , seq: 2374, time: 2014-06-09 16:22:11.390475, event: throttled, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.652952 7f869af78700 5 -- op tracker -- , seq: 2374, time: 2014-06-09 16:22:11.390486, event: all_read, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.652948 7f8689f56700 7 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.1 2014-06-09 16:22:11.652950 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:1.2e0( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.652960 7f869af78700 5 -- op tracker -- , seq: 2374, time: 2014-06-09 16:22:11.652923, event: dispatched, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.652963 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.652967 7f869af78700 5 -- op tracker -- , seq: 2374, time: 2014-06-09 16:22:11.652965, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.652971 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.652972 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.603) v4 from osd.7 2014-06-09 16:22:11.652975 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d24e00 2014-06-09 16:22:11.652974 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.5e1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.652960 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.1 info 3.709( v 213'575 (0'0,213'575] local-les=191 n=111 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.652983 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.652980 7f869af78700 5 -- op tracker -- , seq: 2374, time: 2014-06-09 16:22:11.652978, event: started, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.652982 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.652993 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.653000 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (19 active) 2014-06-09 16:22:11.652997 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.653007 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.653014 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.653016 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.653018 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.330661 7 0.000196 2014-06-09 16:22:11.653025 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0320 (20 active) 2014-06-09 16:22:11.653028 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.653034 7f868df5e700 5 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.653031 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.5d1( v 213'644 (0'0,213'644] local-les=212 n=123 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.653037 7f869af78700 5 -- op tracker -- , seq: 2374, time: 2014-06-09 16:22:11.653034, event: done, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.653040 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.653045 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.653049 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.653051 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.653045 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.653050 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.653058 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.653063 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.653065 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.653067 7f869af78700 20 osd.26 243 _dispatch 0x2eef6a80 pg_info(1 pgs e243:1.a3) v4 2014-06-09 16:22:11.653062 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.653072 7f869af78700 5 -- op tracker -- , seq: 2375, time: 2014-06-09 16:22:11.437145, event: header_read, op: pg_info(1 pgs e243:1.a3) 2014-06-09 16:22:11.653079 7f869af78700 5 -- op tracker -- , seq: 2375, time: 2014-06-09 16:22:11.437145, event: throttled, op: pg_info(1 pgs e243:1.a3) 2014-06-09 16:22:11.653073 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 2.213( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.653085 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.653086 7f869af78700 5 -- op tracker -- , seq: 2375, time: 2014-06-09 16:22:11.437157, event: all_read, op: pg_info(1 pgs e243:1.a3) 2014-06-09 16:22:11.653089 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.653092 7f869af78700 5 -- op tracker -- , seq: 2375, time: 2014-06-09 16:22:11.653063, event: dispatched, op: pg_info(1 pgs e243:1.a3) 2014-06-09 16:22:11.653099 7f869af78700 5 -- op tracker -- , seq: 2375, time: 2014-06-09 16:22:11.653097, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a3) 2014-06-09 16:22:11.653101 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.653103 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a3) v4 from osd.3 2014-06-09 16:22:11.653106 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef6a80 2014-06-09 16:22:11.653106 7f868df5e700 5 queue_transactions existing osr(1.26c 0x384cf50)/0x384cf50 2014-06-09 16:22:11.653111 7f868d75d700 10 op_submit_start 3371 2014-06-09 16:22:11.653092 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.213( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.858600 7 0.000201 2014-06-09 16:22:11.656080 7f868d75d700 5 queue_transactions (trailing journal) 3371 2014-06-09 16:22:11.653166 7f86a5e11700 10 _do_op 0x308cb8c0 seq 3348 r = 0, finisher 0xbd6fd00 0 2014-06-09 16:22:11.656083 7f868d75d700 5 queue_op 0x2fc8c120 seq 3371 osr(2.624 0x2ed1c50) 1903 bytes (queue has 40 ops and 1251981 bytes) 2014-06-09 16:22:11.656086 7f868d75d700 10 op_submit_finish 3371 2014-06-09 16:22:11.656081 7f869af78700 5 -- op tracker -- , seq: 2375, time: 2014-06-09 16:22:11.656077, event: started, op: pg_info(1 pgs e243:1.a3) 2014-06-09 16:22:11.653193 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.656091 7f86a5e11700 10 _finish_op 0x308cb8c0 seq 3348 osr(1.8c 0x384c470)/0x384c470 2014-06-09 16:22:11.656087 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.213( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.656096 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ec60 (1 active) 2014-06-09 16:22:11.656098 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780240 (2 active) 2014-06-09 16:22:11.656101 7f86a5e11700 5 _do_op 0x300602a0 seq 3350 osr(3.124 0x3ed07c0)/0x3ed07c0 start 2014-06-09 16:22:11.656091 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.368( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.656103 7f86a5e11700 10 _do_transaction on 0x4801a40 2014-06-09 16:22:11.656100 7f869af78700 5 -- op tracker -- , seq: 2375, time: 2014-06-09 16:22:11.656097, event: done, op: pg_info(1 pgs e243:1.a3) 2014-06-09 16:22:11.656101 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.213( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.656106 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3349.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.656108 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656109 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.656111 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.656110 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656105 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.656116 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.656110 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.656119 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.656126 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656127 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656129 7f869af78700 20 osd.26 243 _dispatch 0x15c94c40 pg_info(1 pgs e243:0.161) v4 2014-06-09 16:22:11.656121 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:2.28b( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.656127 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.656135 7f869af78700 5 -- op tracker -- , seq: 2376, time: 2014-06-09 16:22:11.311291, event: header_read, op: pg_info(1 pgs e243:0.161) 2014-06-09 16:22:11.656133 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.656143 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.656142 7f869af78700 5 -- op tracker -- , seq: 2376, time: 2014-06-09 16:22:11.311291, event: throttled, op: pg_info(1 pgs e243:0.161) 2014-06-09 16:22:11.656145 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6fd00] 2014-06-09 16:22:11.656150 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6fd00] 2014-06-09 16:22:11.656149 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd64fc0] 2014-06-09 16:22:11.656151 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.656152 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.656148 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:0.290( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.656150 7f869af78700 5 -- op tracker -- , seq: 2376, time: 2014-06-09 16:22:11.311299, event: all_read, op: pg_info(1 pgs e243:0.161) 2014-06-09 16:22:11.656159 7f86a5610700 10 _do_op 0x30606900 seq 3349 r = 0, finisher 0xcc83bc0 0 2014-06-09 16:22:11.656153 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.656162 7f86a5610700 10 _finish_op 0x30606900 seq 3349 osr(0.4ba 0x2bd7710)/0x2bd7710 2014-06-09 16:22:11.656159 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.656164 7f869af78700 5 -- op tracker -- , seq: 2376, time: 2014-06-09 16:22:11.656125, event: dispatched, op: pg_info(1 pgs e243:0.161) 2014-06-09 16:22:11.656166 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac45a0 (1 active) 2014-06-09 16:22:11.656169 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.656167 7f86a460e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.656168 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3598c0 (2 active) 2014-06-09 16:22:11.656170 7f86a5610700 5 _do_op 0x309a2000 seq 3351 osr(1.60e 0x384dc30)/0x384dc30 start 2014-06-09 16:22:11.656173 7f86a5610700 10 _do_transaction on 0xc1458c0 2014-06-09 16:22:11.656177 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656171 7f869af78700 5 -- op tracker -- , seq: 2376, time: 2014-06-09 16:22:11.656169, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.161) 2014-06-09 16:22:11.656172 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.656181 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.161) v4 from osd.1 2014-06-09 16:22:11.656184 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15c94c40 2014-06-09 16:22:11.656186 7f8689f56700 5 queue_transactions existing osr(2.213 0x2ed0b70)/0x2ed0b70 2014-06-09 16:22:11.656189 7f8689f56700 10 op_submit_start 3372 2014-06-09 16:22:11.656190 7f8689f56700 5 queue_transactions (trailing journal) 3372 2014-06-09 16:22:11.656184 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.656191 7f8689f56700 5 queue_op 0x309a3740 seq 3372 osr(2.213 0x2ed0b70) 2009 bytes (queue has 39 ops and 1249995 bytes) 2014-06-09 16:22:11.656182 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 2.2d9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.656190 7f869af78700 5 -- op tracker -- , seq: 2376, time: 2014-06-09 16:22:11.656188, event: started, op: pg_info(1 pgs e243:0.161) 2014-06-09 16:22:11.656194 7f8689f56700 10 op_submit_finish 3372 2014-06-09 16:22:11.656195 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.656197 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd64fc0] 2014-06-09 16:22:11.656199 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc40c6a0] 2014-06-09 16:22:11.656199 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.656201 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc40c6a0] 2014-06-09 16:22:11.656202 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.656203 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.656197 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.656207 7f869af78700 5 -- op tracker -- , seq: 2376, time: 2014-06-09 16:22:11.656204, event: done, op: pg_info(1 pgs e243:0.161) 2014-06-09 16:22:11.656203 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.879728 7 0.000191 2014-06-09 16:22:11.656214 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656217 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656210 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.656214 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.656220 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.656230 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656225 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.656232 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656227 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.656234 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.656234 7f869af78700 20 osd.26 243 _dispatch 0x304f9600 pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.656240 7f8689f56700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.656235 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.656242 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc83bc0] 2014-06-09 16:22:11.656242 7f869af78700 5 -- op tracker -- , seq: 2377, time: 2014-06-09 16:22:11.399817, event: header_read, op: pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656244 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.656249 7f869af78700 5 -- op tracker -- , seq: 2377, time: 2014-06-09 16:22:11.399818, event: throttled, op: pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656251 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3350.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.656256 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.656248 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.656253 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.656257 7f869af78700 5 -- op tracker -- , seq: 2377, time: 2014-06-09 16:22:11.399829, event: all_read, op: pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656261 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.656255 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:189 2014-06-09 16:22:11.656265 7f869af78700 5 -- op tracker -- , seq: 2377, time: 2014-06-09 16:22:11.656230, event: dispatched, op: pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656270 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656273 7f869af78700 5 -- op tracker -- , seq: 2377, time: 2014-06-09 16:22:11.656270, event: waiting_for_osdmap, op: pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656275 7f86a5e11700 10 _do_op 0x300602a0 seq 3350 r = 0, finisher 0xbd76ee0 0 2014-06-09 16:22:11.656278 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.656280 7f86a5e11700 10 _finish_op 0x300602a0 seq 3350 osr(3.124 0x3ed07c0)/0x3ed07c0 2014-06-09 16:22:11.656282 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f9600 2014-06-09 16:22:11.656283 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780240 (1 active) 2014-06-09 16:22:11.656278 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.656285 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406d80 (2 active) 2014-06-09 16:22:11.656287 7f86a5e11700 5 _do_op 0x30d97380 seq 3352 osr(0.6c0 0x2bd7f50)/0x2bd7f50 start 2014-06-09 16:22:11.656290 7f86a5e11700 10 _do_transaction on 0x480c300 2014-06-09 16:22:11.656288 7f869af78700 5 -- op tracker -- , seq: 2377, time: 2014-06-09 16:22:11.656285, event: started, op: pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656293 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc83bc0] 2014-06-09 16:22:11.656294 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76ee0] 2014-06-09 16:22:11.656296 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.656298 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76ee0] 2014-06-09 16:22:11.656299 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.656300 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.656301 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.656303 7f868d75d700 5 queue_transactions existing osr(2.2d9 0x2ed0c30)/0x2ed0c30 2014-06-09 16:22:11.656306 7f868d75d700 10 op_submit_start 3373 2014-06-09 16:22:11.656307 7f868d75d700 5 queue_transactions (trailing journal) 3373 2014-06-09 16:22:11.656308 7f868d75d700 5 queue_op 0x2fc8c780 seq 3373 osr(2.2d9 0x2ed0c30) 2001 bytes (queue has 39 ops and 1250567 bytes) 2014-06-09 16:22:11.656299 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:11.656302 7f869af78700 5 -- op tracker -- , seq: 2377, time: 2014-06-09 16:22:11.656298, event: done, op: pg_log(2.1e0 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656311 7f868d75d700 10 op_submit_finish 3373 2014-06-09 16:22:11.656315 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656317 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656313 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.656314 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=191 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.656329 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.656324 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.656330 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656332 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656334 7f869af78700 20 osd.26 243 _dispatch 0x30d25180 pg_notify(1.1eb(20),2.4b5(20) epoch 243) v5 2014-06-09 16:22:11.656332 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.656328 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=191 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.656340 7f869af78700 5 -- op tracker -- , seq: 2378, time: 2014-06-09 16:22:11.399726, event: header_read, op: pg_notify(1.1eb(20),2.4b5(20) epoch 243) 2014-06-09 16:22:11.656346 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.656344 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3352.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.656349 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.656348 7f869af78700 5 -- op tracker -- , seq: 2378, time: 2014-06-09 16:22:11.399727, event: throttled, op: pg_notify(1.1eb(20),2.4b5(20) epoch 243) 2014-06-09 16:22:11.656342 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=191 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.881331 7 0.000179 2014-06-09 16:22:11.656356 7f869af78700 5 -- op tracker -- , seq: 2378, time: 2014-06-09 16:22:11.399774, event: all_read, op: pg_notify(1.1eb(20),2.4b5(20) epoch 243) 2014-06-09 16:22:11.656355 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=191 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.656363 7f869af78700 5 -- op tracker -- , seq: 2378, time: 2014-06-09 16:22:11.656330, event: dispatched, op: pg_notify(1.1eb(20),2.4b5(20) epoch 243) 2014-06-09 16:22:11.656357 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:854 2014-06-09 16:22:11.656373 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656371 7f86a5e11700 10 _do_op 0x30d97380 seq 3352 r = 0, finisher 0x230c0000 0 2014-06-09 16:22:11.656371 7f869af78700 5 -- op tracker -- , seq: 2378, time: 2014-06-09 16:22:11.656368, event: waiting_for_osdmap, op: pg_notify(1.1eb(20),2.4b5(20) epoch 243) 2014-06-09 16:22:11.656375 7f86a5e11700 10 _finish_op 0x30d97380 seq 3352 osr(0.6c0 0x2bd7f50)/0x2bd7f50 2014-06-09 16:22:11.656375 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.656368 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=191 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.656377 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d25180 2014-06-09 16:22:11.656380 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406d80 (1 active) 2014-06-09 16:22:11.656382 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89a480 (2 active) 2014-06-09 16:22:11.656384 7f86a5e11700 5 _do_op 0x30060240 seq 3353 osr(3.587 0x3ed1840)/0x3ed1840 start 2014-06-09 16:22:11.656377 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:2.5ad( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.656383 7f869af78700 5 -- op tracker -- , seq: 2378, time: 2014-06-09 16:22:11.656380, event: started, op: pg_notify(1.1eb(20),2.4b5(20) epoch 243) 2014-06-09 16:22:11.656387 7f86a5e11700 10 _do_transaction on 0xc1f09c0 2014-06-09 16:22:11.656379 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=191 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.656392 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x230c0000] 2014-06-09 16:22:11.656393 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.656389 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.656398 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656391 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'733 -> 213'733 2014-06-09 16:22:11.656401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x230c0000] 2014-06-09 16:22:11.656402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.656403 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.656179 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.656402 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.656408 7f869af78700 5 -- op tracker -- , seq: 2378, time: 2014-06-09 16:22:11.656405, event: done, op: pg_notify(1.1eb(20),2.4b5(20) epoch 243) 2014-06-09 16:22:11.656421 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656423 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656413 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.5c3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.656425 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.656425 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.656428 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.656432 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.805299 7 0.000191 2014-06-09 16:22:11.656441 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3353.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.656444 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.656445 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656436 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.656447 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656448 7f869af78700 20 osd.26 243 _dispatch 0x310cba40 pg_info(1 pgs e243:3.224) v4 2014-06-09 16:22:11.656445 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.656455 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.656454 7f869af78700 5 -- op tracker -- , seq: 2379, time: 2014-06-09 16:22:11.400080, event: header_read, op: pg_info(1 pgs e243:3.224) 2014-06-09 16:22:11.656463 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x4eb9380,0xc6e8960] 2014-06-09 16:22:11.656466 7f869af78700 5 -- op tracker -- , seq: 2379, time: 2014-06-09 16:22:11.400080, event: throttled, op: pg_info(1 pgs e243:3.224) 2014-06-09 16:22:11.656464 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.656466 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.656473 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.656479 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.656473 7f869af78700 5 -- op tracker -- , seq: 2379, time: 2014-06-09 16:22:11.400092, event: all_read, op: pg_info(1 pgs e243:3.224) 2014-06-09 16:22:11.656481 7f86a460e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.656483 7f868d75d700 5 queue_transactions existing osr(3.111 0x3ed0830)/0x3ed0830 2014-06-09 16:22:11.656480 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.656487 7f868d75d700 10 op_submit_start 3374 2014-06-09 16:22:11.656484 7f869af78700 5 -- op tracker -- , seq: 2379, time: 2014-06-09 16:22:11.656445, event: dispatched, op: pg_info(1 pgs e243:3.224) 2014-06-09 16:22:11.656487 7f868d75d700 5 queue_transactions (trailing journal) 3374 2014-06-09 16:22:11.656489 7f868d75d700 5 queue_op 0x2fc8d740 seq 3374 osr(3.111 0x3ed0830) 1429 bytes (queue has 39 ops and 1249995 bytes) 2014-06-09 16:22:11.656492 7f868d75d700 10 op_submit_finish 3374 2014-06-09 16:22:11.656495 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.656494 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.656510 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.656507 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.2cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.656519 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.656521 7f86a460e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.656523 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.656527 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656528 7f86a5e11700 10 _do_op 0x30060240 seq 3353 r = 0, finisher 0xc5a7360 0 2014-06-09 16:22:11.656532 7f86a5e11700 10 _finish_op 0x30060240 seq 3353 osr(3.587 0x3ed1840)/0x3ed1840 2014-06-09 16:22:11.656532 7f8689f56700 5 queue_transactions existing osr(0.5c3 0x2bd7a50)/0x2bd7a50 2014-06-09 16:22:11.656527 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.018438 7 0.000187 2014-06-09 16:22:11.656535 7f8689f56700 10 op_submit_start 3375 2014-06-09 16:22:11.656536 7f8689f56700 5 queue_transactions (trailing journal) 3375 2014-06-09 16:22:11.656491 7f869af78700 5 -- op tracker -- , seq: 2379, time: 2014-06-09 16:22:11.656489, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.224) 2014-06-09 16:22:11.656536 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89a480 (1 active) 2014-06-09 16:22:11.656531 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.656537 7f8689f56700 5 queue_op 0x309a3680 seq 3375 osr(0.5c3 0x2bd7a50) 1997 bytes (queue has 39 ops and 1250657 bytes) 2014-06-09 16:22:11.656538 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa7e0 (2 active) 2014-06-09 16:22:11.656537 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.656543 7f86a5e11700 5 _do_op 0x2fcafda0 seq 3354 osr(2.646 0x2ed1c30)/0x2ed1c30 start 2014-06-09 16:22:11.656543 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x4eb9380,0xc6e8960] 2014-06-09 16:22:11.656545 7f8689f56700 10 op_submit_finish 3375 2014-06-09 16:22:11.656545 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a7980] 2014-06-09 16:22:11.656545 7f86a5e11700 10 _do_transaction on 0xc31ce40 2014-06-09 16:22:11.656547 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a7980] 2014-06-09 16:22:11.656549 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.656549 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.656551 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.656547 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.656548 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.656555 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.656561 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.656564 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.656570 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.656572 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.656538 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.224) v4 from osd.6 2014-06-09 16:22:11.656581 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310cba40 2014-06-09 16:22:11.656578 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.656585 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.656591 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.656618 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.656625 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a7360] 2014-06-09 16:22:11.656623 7f869af78700 5 -- op tracker -- , seq: 2379, time: 2014-06-09 16:22:11.656619, event: started, op: pg_info(1 pgs e243:3.224) 2014-06-09 16:22:11.656636 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.656656 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656629 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.656660 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 3 notify: (query_epoch:242, epoch_sent:243, info:1.1e4( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.656672 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.656673 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.656684 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656690 7f868d75d700 5 queue_transactions existing osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 2014-06-09 16:22:11.656693 7f868d75d700 10 op_submit_start 3376 2014-06-09 16:22:11.656696 7f868d75d700 5 queue_transactions (trailing journal) 3376 2014-06-09 16:22:11.656689 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 3 notify: (query_epoch:242, epoch_sent:243, info:2.777( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.656689 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.656698 7f868d75d700 5 queue_op 0x2fc8d6e0 seq 3376 osr(0.2cb 0x2bd6ed0) 1954 bytes (queue has 40 ops and 1252611 bytes) 2014-06-09 16:22:11.656700 7f868d75d700 10 op_submit_finish 3376 2014-06-09 16:22:11.656705 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a7360] 2014-06-09 16:22:11.656682 7f869af78700 5 -- op tracker -- , seq: 2379, time: 2014-06-09 16:22:11.656680, event: done, op: pg_info(1 pgs e243:3.224) 2014-06-09 16:22:11.656701 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.656712 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656711 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656714 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656716 7f868d75d700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02b0 (19 active) 2014-06-09 16:22:11.656723 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.656716 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.656736 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656738 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656729 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.656739 7f869af78700 20 osd.26 243 _dispatch 0x240e8000 pg_info(1 pgs e243:2.37) v4 2014-06-09 16:22:11.656727 7f868d75d700 10 osd.26 243 dequeue_op 0x230dbf00 prio 63 cost 540672 latency 0.940851 osd_op(client.4573.0:34051 rbd_data.117d6b8b4567.00000000000016ba [set-alloc-hint object_size 4194304 write_size 4194304,write 0~540672] 3.4129182b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.656745 7f869af78700 5 -- op tracker -- , seq: 2380, time: 2014-06-09 16:22:11.371089, event: header_read, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.656750 7f868a757700 10 op_submit_start 3377 2014-06-09 16:22:11.656742 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.966932 7 0.000166 2014-06-09 16:22:11.656752 7f868a757700 5 queue_transactions (trailing journal) 3377 2014-06-09 16:22:11.656753 7f868a757700 5 queue_op 0x306076e0 seq 3377 osr(3.786 0x3ed1d90) 1335 bytes (queue has 40 ops and 1252611 bytes) 2014-06-09 16:22:11.656757 7f868a757700 10 op_submit_finish 3377 2014-06-09 16:22:11.656755 7f869af78700 5 -- op tracker -- , seq: 2380, time: 2014-06-09 16:22:11.371089, event: throttled, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.656754 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.656762 7f869af78700 5 -- op tracker -- , seq: 2380, time: 2014-06-09 16:22:11.371099, event: all_read, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.656765 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.656769 7f869af78700 5 -- op tracker -- , seq: 2380, time: 2014-06-09 16:22:11.656735, event: dispatched, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.656763 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=208 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.656778 7f869af78700 5 -- op tracker -- , seq: 2380, time: 2014-06-09 16:22:11.656776, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.656783 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.37) v4 from osd.0 2014-06-09 16:22:11.656776 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.656786 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e8000 2014-06-09 16:22:11.656776 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=208 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.656707 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.656794 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.656788 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'603 -> 213'603 2014-06-09 16:22:11.656791 7f869af78700 5 -- op tracker -- , seq: 2380, time: 2014-06-09 16:22:11.656789, event: started, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.656791 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=208 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.722192 7 0.000151 2014-06-09 16:22:11.656805 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3354.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.656809 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.656803 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=208 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.656810 7f869af78700 5 -- op tracker -- , seq: 2380, time: 2014-06-09 16:22:11.656807, event: done, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.656818 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656820 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656814 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=208 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.656824 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=208 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.656832 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656834 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656834 7f86a5e11700 10 _do_op 0x2fcafda0 seq 3354 r = 0, finisher 0xbd6eb00 0 2014-06-09 16:22:11.656835 7f869af78700 20 osd.26 243 _dispatch 0x30bd9c00 pg_info(1 pgs e243:3.48d(2)) v4 2014-06-09 16:22:11.656828 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.656834 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'588 -> 213'588 2014-06-09 16:22:11.656841 7f869af78700 5 -- op tracker -- , seq: 2381, time: 2014-06-09 16:22:11.415630, event: header_read, op: pg_info(1 pgs e243:3.48d(2)) 2014-06-09 16:22:11.656849 7f869af78700 5 -- op tracker -- , seq: 2381, time: 2014-06-09 16:22:11.415631, event: throttled, op: pg_info(1 pgs e243:3.48d(2)) 2014-06-09 16:22:11.656855 7f869af78700 5 -- op tracker -- , seq: 2381, time: 2014-06-09 16:22:11.415650, event: all_read, op: pg_info(1 pgs e243:3.48d(2)) 2014-06-09 16:22:11.656861 7f869af78700 5 -- op tracker -- , seq: 2381, time: 2014-06-09 16:22:11.656831, event: dispatched, op: pg_info(1 pgs e243:3.48d(2)) 2014-06-09 16:22:11.656868 7f869af78700 5 -- op tracker -- , seq: 2381, time: 2014-06-09 16:22:11.656866, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.48d(2)) 2014-06-09 16:22:11.656874 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656873 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.48d(2)) v4 from osd.5 2014-06-09 16:22:11.656876 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd9c00 2014-06-09 16:22:11.656879 7f8689f56700 5 queue_transactions existing osr(3.172 0x3ed06e0)/0x3ed06e0 2014-06-09 16:22:11.656872 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.656881 7f8689f56700 10 op_submit_start 3378 2014-06-09 16:22:11.656883 7f8689f56700 5 queue_transactions (trailing journal) 3378 2014-06-09 16:22:11.656884 7f8689f56700 5 queue_op 0x2fcae5a0 seq 3378 osr(3.172 0x3ed06e0) 1429 bytes (queue has 41 ops and 1254040 bytes) 2014-06-09 16:22:11.656882 7f869af78700 5 -- op tracker -- , seq: 2381, time: 2014-06-09 16:22:11.656880, event: started, op: pg_info(1 pgs e243:3.48d(2)) 2014-06-09 16:22:11.656887 7f8689f56700 10 op_submit_finish 3378 2014-06-09 16:22:11.656893 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.656890 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:2.2ef( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.656901 7f869af78700 5 -- op tracker -- , seq: 2381, time: 2014-06-09 16:22:11.656898, event: done, op: pg_info(1 pgs e243:3.48d(2)) 2014-06-09 16:22:11.656910 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656911 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.656904 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.656912 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656914 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.656916 7f868a757700 5 queue_transactions existing osr(3.1f0 0x3ed0990)/0x3ed0990 2014-06-09 16:22:11.656919 7f868a757700 10 op_submit_start 3379 2014-06-09 16:22:11.656920 7f868a757700 5 queue_transactions (trailing journal) 3379 2014-06-09 16:22:11.656921 7f868a757700 5 queue_op 0x30607740 seq 3379 osr(3.1f0 0x3ed0990) 1335 bytes (queue has 42 ops and 1255375 bytes) 2014-06-09 16:22:11.656924 7f868a757700 10 op_submit_finish 3379 2014-06-09 16:22:11.656927 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.656919 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.656929 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.656930 7f869af78700 20 osd.26 243 _dispatch 0x306da600 pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.656927 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.656909 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3351.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.656931 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.656940 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.656937 7f869af78700 5 -- op tracker -- , seq: 2382, time: 2014-06-09 16:22:11.263819, event: header_read, op: pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656940 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.656941 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.656946 7f869af78700 5 -- op tracker -- , seq: 2382, time: 2014-06-09 16:22:11.263819, event: throttled, op: pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656949 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.656949 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.656961 7f8689f56700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.656959 7f869af78700 5 -- op tracker -- , seq: 2382, time: 2014-06-09 16:22:11.263831, event: all_read, op: pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656958 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.656967 7f869af78700 5 -- op tracker -- , seq: 2382, time: 2014-06-09 16:22:11.656926, event: dispatched, op: pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.656971 7f868a757700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.656991 7f86a5610700 10 _do_op 0x309a2000 seq 3351 r = 0, finisher 0xc40c3e0 0 2014-06-09 16:22:11.656987 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:189 2014-06-09 16:22:11.656998 7f86a5e11700 10 _finish_op 0x2fcafda0 seq 3354 osr(2.646 0x2ed1c30)/0x2ed1c30 2014-06-09 16:22:11.657002 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.657002 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa7e0 (1 active) 2014-06-09 16:22:11.657004 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcfc0 (2 active) 2014-06-09 16:22:11.657006 7f86a5e11700 5 _do_op 0x30d96720 seq 3355 osr(0.4c8 0x2bd7670)/0x2bd7670 start 2014-06-09 16:22:11.656977 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:184 2014-06-09 16:22:11.657009 7f86a5e11700 10 _do_transaction on 0xbfec780 2014-06-09 16:22:11.656997 7f869af78700 5 -- op tracker -- , seq: 2382, time: 2014-06-09 16:22:11.656973, event: waiting_for_osdmap, op: pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657006 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.3fa( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.657011 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.657015 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.657016 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306da600 2014-06-09 16:22:11.657018 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657017 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.657027 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.657023 7f869af78700 5 -- op tracker -- , seq: 2382, time: 2014-06-09 16:22:11.657020, event: started, op: pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657023 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.207( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.657032 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a66e0] 2014-06-09 16:22:11.657035 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a66e0] 2014-06-09 16:22:11.657037 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.657038 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.657037 7f869af78700 5 -- op tracker -- , seq: 2382, time: 2014-06-09 16:22:11.657034, event: done, op: pg_log(0.f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657035 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.657045 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657046 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657038 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.553( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.553( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.657048 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657048 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.657051 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.553( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.657050 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.657061 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.657061 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657063 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657065 7f869af78700 20 osd.26 243 _dispatch 0x306dac00 pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.657063 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.657065 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.657072 7f869af78700 5 -- op tracker -- , seq: 2383, time: 2014-06-09 16:22:11.330204, event: header_read, op: pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657072 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.657080 7f869af78700 5 -- op tracker -- , seq: 2383, time: 2014-06-09 16:22:11.330205, event: throttled, op: pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657076 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.2a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.657080 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.657087 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6eb00] 2014-06-09 16:22:11.657088 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.657087 7f869af78700 5 -- op tracker -- , seq: 2383, time: 2014-06-09 16:22:11.330227, event: all_read, op: pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657092 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.657092 7f8689f56700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.657094 7f869af78700 5 -- op tracker -- , seq: 2383, time: 2014-06-09 16:22:11.657061, event: dispatched, op: pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657090 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.657095 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.160307 7 0.000200 2014-06-09 16:22:11.657103 7f869af78700 5 -- op tracker -- , seq: 2383, time: 2014-06-09 16:22:11.657100, event: waiting_for_osdmap, op: pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657107 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.657111 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306dac00 2014-06-09 16:22:11.657108 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.657117 7f869af78700 5 -- op tracker -- , seq: 2383, time: 2014-06-09 16:22:11.657115, event: started, op: pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657119 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.657132 7f86a5610700 10 _finish_op 0x309a2000 seq 3351 osr(1.60e 0x384dc30)/0x384dc30 2014-06-09 16:22:11.657131 7f869af78700 5 -- op tracker -- , seq: 2383, time: 2014-06-09 16:22:11.657128, event: done, op: pg_log(1.5f8 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657129 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.657135 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3598c0 (1 active) 2014-06-09 16:22:11.657137 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a480 (2 active) 2014-06-09 16:22:11.657138 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657140 7f86a5610700 5 _do_op 0x2fcae1e0 seq 3356 osr(3.76f 0x3ed1e20)/0x3ed1e20 start 2014-06-09 16:22:11.657141 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657142 7f86a5610700 10 _do_transaction on 0xc29a780 2014-06-09 16:22:11.657138 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.657136 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:195 2014-06-09 16:22:11.657148 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.657146 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.657153 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657157 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:11.657172 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.657182 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.657192 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.657196 7f868a757700 5 queue_transactions existing osr(1.2a3 0x384d1f0)/0x384d1f0 2014-06-09 16:22:11.657191 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.657199 7f868a757700 10 op_submit_start 3380 2014-06-09 16:22:11.657200 7f868a757700 5 queue_transactions (trailing journal) 3380 2014-06-09 16:22:11.657202 7f868a757700 5 queue_op 0x306078c0 seq 3380 osr(1.2a3 0x384d1f0) 2001 bytes (queue has 41 ops and 1254632 bytes) 2014-06-09 16:22:11.657203 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.657205 7f868a757700 10 op_submit_finish 3380 2014-06-09 16:22:11.657209 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657211 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657212 7f869af78700 20 osd.26 243 _dispatch 0x30d24700 pg_info(1 pgs e243:0.663) v4 2014-06-09 16:22:11.657207 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.77c( v 213'558 (0'0,213'558] local-les=210 n=109 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.657218 7f869af78700 5 -- op tracker -- , seq: 2384, time: 2014-06-09 16:22:11.390490, event: header_read, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.657219 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:974 2014-06-09 16:22:11.657222 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.657229 7f869af78700 5 -- op tracker -- , seq: 2384, time: 2014-06-09 16:22:11.390490, event: throttled, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.657234 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.657235 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657235 7f869af78700 5 -- op tracker -- , seq: 2384, time: 2014-06-09 16:22:11.390500, event: all_read, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.657242 7f869af78700 5 -- op tracker -- , seq: 2384, time: 2014-06-09 16:22:11.657208, event: dispatched, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.657240 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:1.4bc( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.657240 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=191 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.657252 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.657251 7f869af78700 5 -- op tracker -- , seq: 2384, time: 2014-06-09 16:22:11.657249, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.657256 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.663) v4 from osd.7 2014-06-09 16:22:11.657253 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.657259 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d24700 2014-06-09 16:22:11.657262 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.657266 7f868c75b700 10 op_submit_start 3381 2014-06-09 16:22:11.657264 7f869af78700 5 -- op tracker -- , seq: 2384, time: 2014-06-09 16:22:11.657262, event: started, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.657268 7f868c75b700 5 queue_transactions (trailing journal) 3381 2014-06-09 16:22:11.657269 7f868c75b700 5 queue_op 0x2ea52960 seq 3381 osr(1.3c3 0x384d530) 1954 bytes (queue has 41 ops and 1254632 bytes) 2014-06-09 16:22:11.657272 7f868c75b700 10 op_submit_finish 3381 2014-06-09 16:22:11.657269 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.657284 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6eb00] 2014-06-09 16:22:11.657281 7f869af78700 5 -- op tracker -- , seq: 2384, time: 2014-06-09 16:22:11.657278, event: done, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.657286 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc40c3e0] 2014-06-09 16:22:11.657254 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=191 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.657278 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.657289 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc40c3e0] 2014-06-09 16:22:11.657290 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.657290 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657291 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.657292 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657293 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=191 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.950847 7 0.000191 2014-06-09 16:22:11.657290 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.657304 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657306 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657304 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=191 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.657308 7f869af78700 20 osd.26 243 _dispatch 0x2eef68c0 pg_info(1 pgs e243:0.37e) v4 2014-06-09 16:22:11.657304 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.379429 7 0.000191 2014-06-09 16:22:11.657313 7f869af78700 5 -- op tracker -- , seq: 2385, time: 2014-06-09 16:22:11.437164, event: header_read, op: pg_info(1 pgs e243:0.37e) 2014-06-09 16:22:11.657294 7f868e75f700 10 op_submit_start 3382 2014-06-09 16:22:11.657319 7f868e75f700 5 queue_transactions (trailing journal) 3382 2014-06-09 16:22:11.657312 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=191 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.657322 7f868e75f700 5 queue_op 0x30d97020 seq 3382 osr(2.4 0x2ed0310) 1877 bytes (queue has 41 ops and 1254632 bytes) 2014-06-09 16:22:11.657317 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.657325 7f868e75f700 10 op_submit_finish 3382 2014-06-09 16:22:11.657321 7f869af78700 5 -- op tracker -- , seq: 2385, time: 2014-06-09 16:22:11.437164, event: throttled, op: pg_info(1 pgs e243:0.37e) 2014-06-09 16:22:11.657324 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=191 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.657329 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.657333 7f869af78700 5 -- op tracker -- , seq: 2385, time: 2014-06-09 16:22:11.437174, event: all_read, op: pg_info(1 pgs e243:0.37e) 2014-06-09 16:22:11.657330 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.657339 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=208 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.657340 7f869af78700 5 -- op tracker -- , seq: 2385, time: 2014-06-09 16:22:11.657304, event: dispatched, op: pg_info(1 pgs e243:0.37e) 2014-06-09 16:22:11.657343 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.657336 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'548 -> 213'548 2014-06-09 16:22:11.657348 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'654 -> 213'654 2014-06-09 16:22:11.657352 7f869af78700 5 -- op tracker -- , seq: 2385, time: 2014-06-09 16:22:11.657350, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.37e) 2014-06-09 16:22:11.657351 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.657357 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.37e) v4 from osd.3 2014-06-09 16:22:11.657361 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef68c0 2014-06-09 16:22:11.657366 7f869af78700 5 -- op tracker -- , seq: 2385, time: 2014-06-09 16:22:11.657364, event: started, op: pg_info(1 pgs e243:0.37e) 2014-06-09 16:22:11.657382 7f869af78700 5 -- op tracker -- , seq: 2385, time: 2014-06-09 16:22:11.657379, event: done, op: pg_info(1 pgs e243:0.37e) 2014-06-09 16:22:11.657360 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.657379 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.657389 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657391 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657391 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.657404 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657405 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657407 7f869af78700 20 osd.26 243 _dispatch 0x150e1340 pg_info(1 pgs e243:3.3dc) v4 2014-06-09 16:22:11.657412 7f869af78700 5 -- op tracker -- , seq: 2386, time: 2014-06-09 16:22:11.311308, event: header_read, op: pg_info(1 pgs e243:3.3dc) 2014-06-09 16:22:11.657420 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657420 7f869af78700 5 -- op tracker -- , seq: 2386, time: 2014-06-09 16:22:11.311308, event: throttled, op: pg_info(1 pgs e243:3.3dc) 2014-06-09 16:22:11.657425 7f8689f56700 5 queue_transactions existing osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 2014-06-09 16:22:11.657427 7f8689f56700 10 op_submit_start 3383 2014-06-09 16:22:11.657428 7f8689f56700 5 queue_transactions (trailing journal) 3383 2014-06-09 16:22:11.657430 7f8689f56700 5 queue_op 0x308cb200 seq 3383 osr(3.2d3 0x3ed0eb0) 1429 bytes (queue has 42 ops and 1256061 bytes) 2014-06-09 16:22:11.657428 7f869af78700 5 -- op tracker -- , seq: 2386, time: 2014-06-09 16:22:11.311316, event: all_read, op: pg_info(1 pgs e243:3.3dc) 2014-06-09 16:22:11.657432 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657433 7f8689f56700 10 op_submit_finish 3383 2014-06-09 16:22:11.657390 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.657436 7f868c75b700 5 queue_transactions existing osr(3.e8 0x3ed0510)/0x3ed0510 2014-06-09 16:22:11.657435 7f869af78700 5 -- op tracker -- , seq: 2386, time: 2014-06-09 16:22:11.657403, event: dispatched, op: pg_info(1 pgs e243:3.3dc) 2014-06-09 16:22:11.657438 7f868c75b700 10 op_submit_start 3384 2014-06-09 16:22:11.657439 7f868c75b700 5 queue_transactions (trailing journal) 3384 2014-06-09 16:22:11.657441 7f868c75b700 5 queue_op 0x2ea53680 seq 3384 osr(3.e8 0x3ed0510) 1332 bytes (queue has 43 ops and 1257393 bytes) 2014-06-09 16:22:11.657436 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.657443 7f868c75b700 10 op_submit_finish 3384 2014-06-09 16:22:11.657442 7f869af78700 5 -- op tracker -- , seq: 2386, time: 2014-06-09 16:22:11.657440, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3dc) 2014-06-09 16:22:11.657447 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3dc) v4 from osd.1 2014-06-09 16:22:11.657450 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e1340 2014-06-09 16:22:11.657446 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=208 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.688( v 213'675 (0'0,213'675] local-les=243 n=126 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.657450 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 2014-06-09 16:22:11.657455 7f869af78700 5 -- op tracker -- , seq: 2386, time: 2014-06-09 16:22:11.657453, event: started, op: pg_info(1 pgs e243:3.3dc) 2014-06-09 16:22:11.657459 7f868e75f700 20 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.657457 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=208 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.688( v 213'675 (0'0,213'675] local-les=243 n=126 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.657466 7f868e75f700 20 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.657474 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.657469 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=208 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.193002 7 0.000179 2014-06-09 16:22:11.657472 7f869af78700 5 -- op tracker -- , seq: 2386, time: 2014-06-09 16:22:11.657470, event: done, op: pg_info(1 pgs e243:3.3dc) 2014-06-09 16:22:11.657479 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=208 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.657486 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657478 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=213 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.657488 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657489 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=208 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.657500 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657502 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657503 7f869af78700 20 osd.26 243 _dispatch 0x240ea540 pg_info(1 pgs e243:1.299) v4 2014-06-09 16:22:11.657499 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=208 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.657508 7f869af78700 5 -- op tracker -- , seq: 2387, time: 2014-06-09 16:22:11.399833, event: header_read, op: pg_info(1 pgs e243:1.299) 2014-06-09 16:22:11.657509 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=243 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'675 -> 213'675 2014-06-09 16:22:11.657516 7f869af78700 5 -- op tracker -- , seq: 2387, time: 2014-06-09 16:22:11.399834, event: throttled, op: pg_info(1 pgs e243:1.299) 2014-06-09 16:22:11.657489 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=213 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.9 3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) 2014-06-09 16:22:11.657524 7f869af78700 5 -- op tracker -- , seq: 2387, time: 2014-06-09 16:22:11.399844, event: all_read, op: pg_info(1 pgs e243:1.299) 2014-06-09 16:22:11.657482 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.2f7( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.657531 7f869af78700 5 -- op tracker -- , seq: 2387, time: 2014-06-09 16:22:11.657500, event: dispatched, op: pg_info(1 pgs e243:1.299) 2014-06-09 16:22:11.657526 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=213 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.421343 7 0.000174 2014-06-09 16:22:11.657538 7f869af78700 5 -- op tracker -- , seq: 2387, time: 2014-06-09 16:22:11.657536, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.299) 2014-06-09 16:22:11.657542 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.299) v4 from osd.9 2014-06-09 16:22:11.657546 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ea540 2014-06-09 16:22:11.657535 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.657550 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657551 7f869af78700 5 -- op tracker -- , seq: 2387, time: 2014-06-09 16:22:11.657549, event: started, op: pg_info(1 pgs e243:1.299) 2014-06-09 16:22:11.657553 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=243 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.657555 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.657538 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=213 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.657567 7f869af78700 5 -- op tracker -- , seq: 2387, time: 2014-06-09 16:22:11.657564, event: done, op: pg_info(1 pgs e243:1.299) 2014-06-09 16:22:11.657568 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.657575 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657577 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657570 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=213 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.657576 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.657581 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.657581 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=213 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.657590 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657591 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657593 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657591 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.657594 7f868c75b700 5 queue_transactions existing osr(3.688 0x3ed1d20)/0x3ed1d20 2014-06-09 16:22:11.657596 7f868c75b700 10 op_submit_start 3385 2014-06-09 16:22:11.657595 7f869af78700 20 osd.26 243 _dispatch 0x307e0a00 pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.657597 7f868c75b700 5 queue_transactions (trailing journal) 3385 2014-06-09 16:22:11.657591 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'571 -> 213'571 2014-06-09 16:22:11.657599 7f868c75b700 5 queue_op 0x288f9920 seq 3385 osr(3.688 0x3ed1d20) 1335 bytes (queue has 44 ops and 1258728 bytes) 2014-06-09 16:22:11.657601 7f868c75b700 10 op_submit_finish 3385 2014-06-09 16:22:11.657602 7f869af78700 5 -- op tracker -- , seq: 2388, time: 2014-06-09 16:22:11.399780, event: header_read, op: pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657610 7f869af78700 5 -- op tracker -- , seq: 2388, time: 2014-06-09 16:22:11.399780, event: throttled, op: pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657604 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 0 notify: (query_epoch:243, epoch_sent:243, info:3.362( v 213'616 (0'0,213'616] local-les=191 n=123 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.657617 7f869af78700 5 -- op tracker -- , seq: 2388, time: 2014-06-09 16:22:11.399791, event: all_read, op: pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657617 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.657629 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657624 7f869af78700 5 -- op tracker -- , seq: 2388, time: 2014-06-09 16:22:11.657590, event: dispatched, op: pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657639 7f868c75b700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40a20 (19 active) 2014-06-09 16:22:11.657637 7f869af78700 5 -- op tracker -- , seq: 2388, time: 2014-06-09 16:22:11.657634, event: waiting_for_osdmap, op: pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657630 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:178 2014-06-09 16:22:11.657642 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.657646 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e0a00 2014-06-09 16:22:11.657652 7f869af78700 5 -- op tracker -- , seq: 2388, time: 2014-06-09 16:22:11.657649, event: started, op: pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657669 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40a20 (20 active) 2014-06-09 16:22:11.657681 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657676 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.657685 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.657687 7f869af78700 5 -- op tracker -- , seq: 2388, time: 2014-06-09 16:22:11.657683, event: done, op: pg_log(2.552 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.657688 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.657697 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657699 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657697 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.657693 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 0.771( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.657706 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.657705 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.657710 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.657712 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657714 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657715 7f869af78700 20 osd.26 243 _dispatch 0x310dbdc0 pg_info(1 pgs e243:2.6dd) v4 2014-06-09 16:22:11.657716 7f868c75b700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.657710 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.657714 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.771( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.765818 7 0.004670 2014-06-09 16:22:11.657720 7f869af78700 5 -- op tracker -- , seq: 2389, time: 2014-06-09 16:22:11.400097, event: header_read, op: pg_info(1 pgs e243:2.6dd) 2014-06-09 16:22:11.657730 7f869af78700 5 -- op tracker -- , seq: 2389, time: 2014-06-09 16:22:11.400097, event: throttled, op: pg_info(1 pgs e243:2.6dd) 2014-06-09 16:22:11.657730 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.657736 7f869af78700 5 -- op tracker -- , seq: 2389, time: 2014-06-09 16:22:11.400107, event: all_read, op: pg_info(1 pgs e243:2.6dd) 2014-06-09 16:22:11.657744 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657743 7f869af78700 5 -- op tracker -- , seq: 2389, time: 2014-06-09 16:22:11.657711, event: dispatched, op: pg_info(1 pgs e243:2.6dd) 2014-06-09 16:22:11.657748 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.657755 7f869af78700 5 -- op tracker -- , seq: 2389, time: 2014-06-09 16:22:11.657752, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6dd) 2014-06-09 16:22:11.657725 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.771( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.657759 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6dd) v4 from osd.6 2014-06-09 16:22:11.657763 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310dbdc0 2014-06-09 16:22:11.657760 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.657762 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.771( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.657768 7f869af78700 5 -- op tracker -- , seq: 2389, time: 2014-06-09 16:22:11.657766, event: started, op: pg_info(1 pgs e243:2.6dd) 2014-06-09 16:22:11.657768 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.657777 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.657773 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.657776 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.657781 7f868e75f700 5 queue_transactions existing osr(3.375 0x3ed1060)/0x3ed1060 2014-06-09 16:22:11.657783 7f868e75f700 10 op_submit_start 3386 2014-06-09 16:22:11.657784 7f868e75f700 5 queue_transactions (trailing journal) 3386 2014-06-09 16:22:11.657786 7f868e75f700 5 queue_op 0x309a20c0 seq 3386 osr(3.375 0x3ed1060) 1335 bytes (queue has 45 ops and 1260063 bytes) 2014-06-09 16:22:11.657786 7f868c75b700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.657788 7f868e75f700 10 op_submit_finish 3386 2014-06-09 16:22:11.657785 7f869af78700 5 -- op tracker -- , seq: 2389, time: 2014-06-09 16:22:11.657782, event: done, op: pg_info(1 pgs e243:2.6dd) 2014-06-09 16:22:11.657783 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.657795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657797 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657791 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.164( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.657800 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.657810 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657812 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657812 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657814 7f869af78700 20 osd.26 243 _dispatch 0x3044c380 pg_info(1 pgs e243:1.5e) v4 2014-06-09 16:22:11.657792 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.657819 7f869af78700 5 -- op tracker -- , seq: 2390, time: 2014-06-09 16:22:11.371104, event: header_read, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.657816 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 0 notify: (query_epoch:242, epoch_sent:243, info:0.70f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.657826 7f869af78700 5 -- op tracker -- , seq: 2390, time: 2014-06-09 16:22:11.371105, event: throttled, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.657828 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 0, already have info from that osd, ignoring 2014-06-09 16:22:11.657803 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.657833 7f869af78700 5 -- op tracker -- , seq: 2390, time: 2014-06-09 16:22:11.371115, event: all_read, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.657838 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657838 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.657840 7f869af78700 5 -- op tracker -- , seq: 2390, time: 2014-06-09 16:22:11.657810, event: dispatched, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.657847 7f869af78700 5 -- op tracker -- , seq: 2390, time: 2014-06-09 16:22:11.657845, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.657857 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5e) v4 from osd.0 2014-06-09 16:22:11.657860 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044c380 2014-06-09 16:22:11.657851 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.657860 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657866 7f8689f56700 5 queue_transactions existing osr(0.771 0x384c310)/0x384c310 2014-06-09 16:22:11.657869 7f8689f56700 10 op_submit_start 3387 2014-06-09 16:22:11.657870 7f8689f56700 5 queue_transactions (trailing journal) 3387 2014-06-09 16:22:11.657871 7f8689f56700 5 queue_op 0x30d97200 seq 3387 osr(0.771 0x384c310) 1883 bytes (queue has 46 ops and 1261946 bytes) 2014-06-09 16:22:11.657865 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.657866 7f869af78700 5 -- op tracker -- , seq: 2390, time: 2014-06-09 16:22:11.657864, event: started, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.657874 7f8689f56700 10 op_submit_finish 3387 2014-06-09 16:22:11.657877 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657876 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x170014c0] 2014-06-09 16:22:11.657879 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.657880 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.657877 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.505( v 213'732 (0'0,213'732] local-les=213 n=143 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.657887 7f869af78700 5 -- op tracker -- , seq: 2390, time: 2014-06-09 16:22:11.657885, event: done, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.657888 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.657895 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657890 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.657897 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657899 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657893 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.657897 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.60e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:11.657905 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.657906 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x170014c0] 2014-06-09 16:22:11.657907 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.657908 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.657902 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.657910 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657912 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657913 7f869af78700 20 osd.26 243 _dispatch 0x3115ba40 pg_info(1 pgs e243:3.380) v4 2014-06-09 16:22:11.657912 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.657919 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0320 (19 active) 2014-06-09 16:22:11.657922 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657919 7f869af78700 5 -- op tracker -- , seq: 2391, time: 2014-06-09 16:22:11.415659, event: header_read, op: pg_info(1 pgs e243:3.380) 2014-06-09 16:22:11.657926 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.657926 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3355.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.657926 7f869af78700 5 -- op tracker -- , seq: 2391, time: 2014-06-09 16:22:11.415660, event: throttled, op: pg_info(1 pgs e243:3.380) 2014-06-09 16:22:11.657930 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.657925 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.657933 7f869af78700 5 -- op tracker -- , seq: 2391, time: 2014-06-09 16:22:11.415675, event: all_read, op: pg_info(1 pgs e243:3.380) 2014-06-09 16:22:11.657939 7f869af78700 5 -- op tracker -- , seq: 2391, time: 2014-06-09 16:22:11.657910, event: dispatched, op: pg_info(1 pgs e243:3.380) 2014-06-09 16:22:11.657936 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.657945 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657947 7f86a5e11700 10 _do_op 0x30d96720 seq 3355 r = 0, finisher 0x33a7d00 0 2014-06-09 16:22:11.657946 7f869af78700 5 -- op tracker -- , seq: 2391, time: 2014-06-09 16:22:11.657944, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.380) 2014-06-09 16:22:11.657951 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.380) v4 from osd.5 2014-06-09 16:22:11.657952 7f86a5e11700 10 _finish_op 0x30d96720 seq 3355 osr(0.4c8 0x2bd7670)/0x2bd7670 2014-06-09 16:22:11.657954 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115ba40 2014-06-09 16:22:11.657954 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509940] 2014-06-09 16:22:11.657956 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcfc0 (1 active) 2014-06-09 16:22:11.657949 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.657957 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509940] 2014-06-09 16:22:11.657958 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3aea0 (2 active) 2014-06-09 16:22:11.657959 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.657961 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.657960 7f86a5e11700 5 _do_op 0x30d97ec0 seq 3357 osr(1.1fa 0x384cd50)/0x384cd50 start 2014-06-09 16:22:11.657959 7f869af78700 5 -- op tracker -- , seq: 2391, time: 2014-06-09 16:22:11.657957, event: started, op: pg_info(1 pgs e243:3.380) 2014-06-09 16:22:11.657963 7f86a5e11700 10 _do_transaction on 0xc005800 2014-06-09 16:22:11.657960 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.198( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.657969 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.657969 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.657972 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.657976 7f869af78700 5 -- op tracker -- , seq: 2391, time: 2014-06-09 16:22:11.657974, event: done, op: pg_info(1 pgs e243:3.380) 2014-06-09 16:22:11.657985 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.657988 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.657983 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.657992 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.658000 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658002 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658001 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.658003 7f869af78700 20 osd.26 243 _dispatch 0x30b8ddc0 pg_notify(2.b(21) epoch 243) v5 2014-06-09 16:22:11.657999 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.658008 7f869af78700 5 -- op tracker -- , seq: 2392, time: 2014-06-09 16:22:11.263836, event: header_read, op: pg_notify(2.b(21) epoch 243) 2014-06-09 16:22:11.658011 7f868c75b700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.658015 7f869af78700 5 -- op tracker -- , seq: 2392, time: 2014-06-09 16:22:11.263836, event: throttled, op: pg_notify(2.b(21) epoch 243) 2014-06-09 16:22:11.658017 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3357.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.658021 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.658027 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.658040 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.658042 7f86a5e11700 10 _do_op 0x30d97ec0 seq 3357 r = 0, finisher 0xbd74140 0 2014-06-09 16:22:11.658047 7f86a5e11700 10 _finish_op 0x30d97ec0 seq 3357 osr(1.1fa 0x384cd50)/0x384cd50 2014-06-09 16:22:11.658049 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050b9e0] 2014-06-09 16:22:11.658022 7f869af78700 5 -- op tracker -- , seq: 2392, time: 2014-06-09 16:22:11.263863, event: all_read, op: pg_notify(2.b(21) epoch 243) 2014-06-09 16:22:11.658050 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3aea0 (1 active) 2014-06-09 16:22:11.658044 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.658052 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.658053 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050b9e0] 2014-06-09 16:22:11.658054 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.658055 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.658054 7f86a5e11700 5 _do_op 0x30d96ea0 seq 3358 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.658056 7f86a5e11700 10 _do_transaction on 0xc317740 2014-06-09 16:22:11.658061 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.658055 7f869af78700 5 -- op tracker -- , seq: 2392, time: 2014-06-09 16:22:11.658000, event: dispatched, op: pg_notify(2.b(21) epoch 243) 2014-06-09 16:22:11.658054 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.658065 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.658066 7f869af78700 5 -- op tracker -- , seq: 2392, time: 2014-06-09 16:22:11.658063, event: waiting_for_osdmap, op: pg_notify(2.b(21) epoch 243) 2014-06-09 16:22:11.658068 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.658071 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.658073 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8ddc0 2014-06-09 16:22:11.658072 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.493( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] exit Started/Stray 0.863434 7 0.001470 2014-06-09 16:22:11.658078 7f869af78700 5 -- op tracker -- , seq: 2392, time: 2014-06-09 16:22:11.658076, event: started, op: pg_notify(2.b(21) epoch 243) 2014-06-09 16:22:11.658082 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.493( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.658091 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.493( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.658095 7f869af78700 5 -- op tracker -- , seq: 2392, time: 2014-06-09 16:22:11.658093, event: done, op: pg_notify(2.b(21) epoch 243) 2014-06-09 16:22:11.658100 7f86a5e11700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.658105 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658101 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.658107 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658109 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.658115 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.658117 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.658125 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658127 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658128 7f869af78700 20 osd.26 243 _dispatch 0x30e0d800 pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.658135 7f869af78700 5 -- op tracker -- , seq: 2393, time: 2014-06-09 16:22:11.330527, event: header_read, op: pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658143 7f869af78700 5 -- op tracker -- , seq: 2393, time: 2014-06-09 16:22:11.330528, event: throttled, op: pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658146 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.658150 7f869af78700 5 -- op tracker -- , seq: 2393, time: 2014-06-09 16:22:11.330541, event: all_read, op: pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658156 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.658160 7f86a5e11700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.658161 7f868c75b700 5 queue_transactions existing osr(0.493 0x2bd73d0)/0x2bd73d0 2014-06-09 16:22:11.658163 7f868c75b700 10 op_submit_start 3388 2014-06-09 16:22:11.658164 7f868c75b700 5 queue_transactions (trailing journal) 3388 2014-06-09 16:22:11.658158 7f869af78700 5 -- op tracker -- , seq: 2393, time: 2014-06-09 16:22:11.658125, event: dispatched, op: pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658165 7f86a5e11700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.658166 7f868c75b700 5 queue_op 0x2ea52a20 seq 3388 osr(0.493 0x2bd73d0) 1621 bytes (queue has 45 ops and 1259675 bytes) 2014-06-09 16:22:11.658168 7f868c75b700 10 op_submit_finish 3388 2014-06-09 16:22:11.658171 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.658171 7f869af78700 5 -- op tracker -- , seq: 2393, time: 2014-06-09 16:22:11.658168, event: waiting_for_osdmap, op: pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658176 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.658170 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.658180 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e0d800 2014-06-09 16:22:11.658182 7f86a5e11700 15 _touch 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.658186 7f869af78700 5 -- op tracker -- , seq: 2393, time: 2014-06-09 16:22:11.658183, event: started, op: pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658182 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.658191 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.658201 7f869af78700 5 -- op tracker -- , seq: 2393, time: 2014-06-09 16:22:11.658197, event: done, op: pg_log(1.755 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658199 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.658210 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658212 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658211 7f868c75b700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.658214 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.658223 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658225 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658226 7f869af78700 20 osd.26 243 _dispatch 0x30bd9a40 pg_info(1 pgs e243:1.691) v4 2014-06-09 16:22:11.658226 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1007 2014-06-09 16:22:11.658231 7f869af78700 5 -- op tracker -- , seq: 2394, time: 2014-06-09 16:22:11.390504, event: header_read, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.658239 7f86a5e11700 20 write_state seq is 30152 2014-06-09 16:22:11.658241 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.658240 7f869af78700 5 -- op tracker -- , seq: 2394, time: 2014-06-09 16:22:11.390504, event: throttled, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.658246 7f86a5e11700 20 set_header setting 30151 cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 parent seq 0 2014-06-09 16:22:11.658247 7f869af78700 5 -- op tracker -- , seq: 2394, time: 2014-06-09 16:22:11.390514, event: all_read, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.658253 7f869af78700 5 -- op tracker -- , seq: 2394, time: 2014-06-09 16:22:11.658223, event: dispatched, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.658261 7f86a5e11700 10 _touch 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.658255 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.658266 7f86a5e11700 15 _write 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 0~962560 2014-06-09 16:22:11.658266 7f869af78700 5 -- op tracker -- , seq: 2394, time: 2014-06-09 16:22:11.658263, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.658271 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.691) v4 from osd.7 2014-06-09 16:22:11.658261 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.658274 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd9a40 2014-06-09 16:22:11.658269 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.658280 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658282 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.658279 7f869af78700 5 -- op tracker -- , seq: 2394, time: 2014-06-09 16:22:11.658277, event: started, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.658278 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.658297 7f869af78700 5 -- op tracker -- , seq: 2394, time: 2014-06-09 16:22:11.658295, event: done, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.658283 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.4ba( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.658305 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658307 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658304 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.658313 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658319 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658321 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658322 7f869af78700 20 osd.26 243 _dispatch 0x2eef6700 pg_info(1 pgs e243:3.66f) v4 2014-06-09 16:22:11.658317 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.1f4( v 213'599 (0'0,213'599] local-les=208 n=118 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.658327 7f869af78700 5 -- op tracker -- , seq: 2395, time: 2014-06-09 16:22:11.437222, event: header_read, op: pg_info(1 pgs e243:3.66f) 2014-06-09 16:22:11.658329 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.658334 7f869af78700 5 -- op tracker -- , seq: 2395, time: 2014-06-09 16:22:11.437223, event: throttled, op: pg_info(1 pgs e243:3.66f) 2014-06-09 16:22:11.658339 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658341 7f869af78700 5 -- op tracker -- , seq: 2395, time: 2014-06-09 16:22:11.437234, event: all_read, op: pg_info(1 pgs e243:3.66f) 2014-06-09 16:22:11.658288 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.658353 7f869af78700 5 -- op tracker -- , seq: 2395, time: 2014-06-09 16:22:11.658318, event: dispatched, op: pg_info(1 pgs e243:3.66f) 2014-06-09 16:22:11.658342 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.658354 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.658360 7f869af78700 5 -- op tracker -- , seq: 2395, time: 2014-06-09 16:22:11.658358, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.66f) 2014-06-09 16:22:11.658360 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.658367 7f868c75b700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.658371 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658370 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.66f) v4 from osd.3 2014-06-09 16:22:11.658373 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef6700 2014-06-09 16:22:11.658375 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.658378 7f869af78700 5 -- op tracker -- , seq: 2395, time: 2014-06-09 16:22:11.658376, event: started, op: pg_info(1 pgs e243:3.66f) 2014-06-09 16:22:11.658382 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:163 2014-06-09 16:22:11.658386 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.658398 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.658395 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.658399 7f869af78700 5 -- op tracker -- , seq: 2395, time: 2014-06-09 16:22:11.658397, event: done, op: pg_info(1 pgs e243:3.66f) 2014-06-09 16:22:11.658407 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658402 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.658409 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658403 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.658422 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658424 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658414 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.658425 7f868e75f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.658425 7f869af78700 20 osd.26 243 _dispatch 0x30641800 pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.658427 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.658429 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.658432 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.145157 7 0.000195 2014-06-09 16:22:11.658432 7f869af78700 5 -- op tracker -- , seq: 2396, time: 2014-06-09 16:22:11.312143, event: header_read, op: pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658439 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:205 2014-06-09 16:22:11.658441 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.658445 7f869af78700 5 -- op tracker -- , seq: 2396, time: 2014-06-09 16:22:11.312144, event: throttled, op: pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658453 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658453 7f869af78700 5 -- op tracker -- , seq: 2396, time: 2014-06-09 16:22:11.312884, event: all_read, op: pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658451 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.658460 7f869af78700 5 -- op tracker -- , seq: 2396, time: 2014-06-09 16:22:11.658422, event: dispatched, op: pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658467 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0290 (19 active) 2014-06-09 16:22:11.658461 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.658468 7f869af78700 5 -- op tracker -- , seq: 2396, time: 2014-06-09 16:22:11.658465, event: waiting_for_osdmap, op: pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658473 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) v4 from osd.1 2014-06-09 16:22:11.658477 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30641800 2014-06-09 16:22:11.658471 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.658480 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.658483 7f869af78700 5 -- op tracker -- , seq: 2396, time: 2014-06-09 16:22:11.658481, event: started, op: pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658490 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0290 (20 active) 2014-06-09 16:22:11.658494 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:243, info:3.1f4( v 213'599 (0'0,213'599] local-les=208 n=118 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.658499 7f869af78700 5 -- op tracker -- , seq: 2396, time: 2014-06-09 16:22:11.658496, event: done, op: pg_log(3.1aa epoch 243 log log((0'0,213'493], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.658507 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658509 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658505 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.658515 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658520 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658522 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658524 7f869af78700 20 osd.26 243 _dispatch 0x240eb880 pg_info(1 pgs e243:1.4d3) v4 2014-06-09 16:22:11.658519 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.1db( v 214'696 (0'0,214'696] local-les=208 n=139 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.658528 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.658529 7f869af78700 5 -- op tracker -- , seq: 2397, time: 2014-06-09 16:22:11.399857, event: header_read, op: pg_info(1 pgs e243:1.4d3) 2014-06-09 16:22:11.658532 7f868c75b700 5 queue_transactions existing osr(0.3fe 0x2bd7210)/0x2bd7210 2014-06-09 16:22:11.658535 7f868c75b700 10 op_submit_start 3389 2014-06-09 16:22:11.658536 7f868c75b700 5 queue_transactions (trailing journal) 3389 2014-06-09 16:22:11.658536 7f869af78700 5 -- op tracker -- , seq: 2397, time: 2014-06-09 16:22:11.399858, event: throttled, op: pg_info(1 pgs e243:1.4d3) 2014-06-09 16:22:11.658531 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.658538 7f868c75b700 5 queue_op 0x30032660 seq 3389 osr(0.3fe 0x2bd7210) 1883 bytes (queue has 46 ops and 1261558 bytes) 2014-06-09 16:22:11.658541 7f868c75b700 10 op_submit_finish 3389 2014-06-09 16:22:11.658541 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658543 7f869af78700 5 -- op tracker -- , seq: 2397, time: 2014-06-09 16:22:11.399871, event: all_read, op: pg_info(1 pgs e243:1.4d3) 2014-06-09 16:22:11.658549 7f869af78700 5 -- op tracker -- , seq: 2397, time: 2014-06-09 16:22:11.658520, event: dispatched, op: pg_info(1 pgs e243:1.4d3) 2014-06-09 16:22:11.658544 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.658544 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.658556 7f869af78700 5 -- op tracker -- , seq: 2397, time: 2014-06-09 16:22:11.658554, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4d3) 2014-06-09 16:22:11.658561 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4d3) v4 from osd.9 2014-06-09 16:22:11.658557 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.658564 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eb880 2014-06-09 16:22:11.658555 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 2.40c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.658568 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.658565 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.658572 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.658570 7f869af78700 5 -- op tracker -- , seq: 2397, time: 2014-06-09 16:22:11.658568, event: started, op: pg_info(1 pgs e243:1.4d3) 2014-06-09 16:22:11.658573 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.658575 7f868c75b700 5 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.985224 7 0.038418 2014-06-09 16:22:11.658583 7f868e75f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.658586 7f868c75b700 5 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.658590 7f869af78700 5 -- op tracker -- , seq: 2397, time: 2014-06-09 16:22:11.658588, event: done, op: pg_info(1 pgs e243:1.4d3) 2014-06-09 16:22:11.658598 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658600 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658594 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.658597 7f868c75b700 5 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.658607 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658612 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.658614 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.658607 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.658616 7f869af78700 20 osd.26 243 _dispatch 0x30d248c0 pg_info(1 pgs e243:3.555(2)) v4 2014-06-09 16:22:11.658611 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:2.359( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.658617 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.658621 7f869af78700 5 -- op tracker -- , seq: 2398, time: 2014-06-09 16:22:11.399795, event: header_read, op: pg_info(1 pgs e243:3.555(2)) 2014-06-09 16:22:11.658622 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.658632 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658626 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.658633 7f869af78700 5 -- op tracker -- , seq: 2398, time: 2014-06-09 16:22:11.399796, event: throttled, op: pg_info(1 pgs e243:3.555(2)) 2014-06-09 16:22:11.658635 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:0.627( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.658640 7f869af78700 5 -- op tracker -- , seq: 2398, time: 2014-06-09 16:22:11.399809, event: all_read, op: pg_info(1 pgs e243:3.555(2)) 2014-06-09 16:22:11.658646 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.658655 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.658651 7f869af78700 5 -- op tracker -- , seq: 2398, time: 2014-06-09 16:22:11.658612, event: dispatched, op: pg_info(1 pgs e243:3.555(2)) 2014-06-09 16:22:11.658662 7f869af78700 5 -- op tracker -- , seq: 2398, time: 2014-06-09 16:22:11.658660, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.555(2)) 2014-06-09 16:22:11.658658 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.18b( v 213'575 (0'0,213'575] local-les=191 n=113 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.658666 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.555(2)) v4 from osd.8 2014-06-09 16:22:11.658672 7f868bf5a700 10 op_submit_start 3390 2014-06-09 16:22:11.658674 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.663084 7f868bf5a700 5 queue_transactions (trailing journal) 3390 2014-06-09 16:22:11.663088 7f868bf5a700 5 queue_op 0x308caf60 seq 3390 osr(3.13c 0x3ed0760) 1335 bytes (queue has 46 ops and 1261558 bytes) 2014-06-09 16:22:11.663092 7f868bf5a700 10 op_submit_finish 3390 2014-06-09 16:22:11.663081 7f868e75f700 7 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.663096 7f868c75b700 5 queue_transactions existing osr(2.40c 0x2ed1250)/0x2ed1250 2014-06-09 16:22:11.663100 7f868c75b700 10 op_submit_start 3391 2014-06-09 16:22:11.663101 7f868c75b700 5 queue_transactions (trailing journal) 3391 2014-06-09 16:22:11.663103 7f868c75b700 5 queue_op 0x2ea52b40 seq 3391 osr(2.40c 0x2ed1250) 2009 bytes (queue has 47 ops and 1263567 bytes) 2014-06-09 16:22:11.663106 7f868c75b700 10 op_submit_finish 3391 2014-06-09 16:22:11.663099 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.663097 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.18b( v 213'575 (0'0,213'575] local-les=191 n=113 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.663118 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.663111 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.1d4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663110 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 7 2014-06-09 16:22:11.663123 7f868bf5a700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663128 7f868bf5a700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.663124 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.71f( v 213'532 (0'0,213'532] local-les=191 n=101 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.663124 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.7 0.1fd( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663132 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.765436 7 0.000180 2014-06-09 16:22:11.663138 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.7 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663142 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.663136 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.663147 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.663142 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.663146 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.685016 7 0.000169 2014-06-09 16:22:11.663151 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.722( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.663154 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.663157 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.663162 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.663164 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.663172 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.663169 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.663174 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.663176 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.458( v 213'610 (0'0,213'610] local-les=191 n=125 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.663179 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.663183 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.663188 7f868e75f700 7 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.663189 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.663198 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.663197 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.458( v 213'610 (0'0,213'610] local-les=191 n=125 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.663214 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.663218 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.663232 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.663227 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.14c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663236 7f868bf5a700 5 queue_transactions existing osr(0.1d4 0x2bd6940)/0x2bd6940 2014-06-09 16:22:11.663239 7f868bf5a700 10 op_submit_start 3392 2014-06-09 16:22:11.663238 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663240 7f868bf5a700 5 queue_transactions (trailing journal) 3392 2014-06-09 16:22:11.663241 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.663241 7f868bf5a700 5 queue_op 0x308caf00 seq 3392 osr(0.1d4 0x2bd6940) 1883 bytes (queue has 48 ops and 1265450 bytes) 2014-06-09 16:22:11.663244 7f868bf5a700 10 op_submit_finish 3392 2014-06-09 16:22:11.663249 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.663244 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.974045 7 0.010669 2014-06-09 16:22:11.663246 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 5 notify: (query_epoch:242, epoch_sent:243, info:2.446( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.663254 7f868c75b700 5 queue_transactions existing osr(0.1fd 0x2bd6b30)/0x2bd6b30 2014-06-09 16:22:11.663257 7f868c75b700 10 op_submit_start 3393 2014-06-09 16:22:11.663258 7f868c75b700 5 queue_transactions (trailing journal) 3393 2014-06-09 16:22:11.663253 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.663259 7f868c75b700 5 queue_op 0x300326c0 seq 3393 osr(0.1fd 0x2bd6b30) 2009 bytes (queue has 49 ops and 1267459 bytes) 2014-06-09 16:22:11.663262 7f868c75b700 10 op_submit_finish 3393 2014-06-09 16:22:11.663257 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.663267 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.663263 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.663265 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 3 2014-06-09 16:22:11.663272 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.663271 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=191 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.663280 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.663281 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd3 2014-06-09 16:22:11.663283 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=191 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.663287 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.663294 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=191 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.989130 7 0.000207 2014-06-09 16:22:11.663294 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.663304 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.663307 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=191 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.663314 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'610 (0'0,214'610] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.3: log((214'610,214'612], crt=0'0) missing(0) 2014-06-09 16:22:11.663317 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=191 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.663325 7f868c75b700 10 merge_log log((214'610,214'612], crt=0'0) from osd.3 into log((0'0,214'610], crt=0'0) 2014-06-09 16:22:11.663329 7f868c75b700 10 merge_log extending head to 214'612 2014-06-09 16:22:11.663331 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.663326 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=191 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.663335 7f868e75f700 5 queue_transactions existing osr(1.14c 0x384c810)/0x384c810 2014-06-09 16:22:11.663337 7f868e75f700 10 op_submit_start 3394 2014-06-09 16:22:11.663338 7f868e75f700 5 queue_transactions (trailing journal) 3394 2014-06-09 16:22:11.663339 7f868e75f700 5 queue_op 0x30d97e00 seq 3394 osr(1.14c 0x384c810) 1954 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.663331 7f868c75b700 20 ? 214'612 (214'611) modify 7121e06e/rbd_data.115e6b8b4567.00000000000017ff/head//3 by client.4542.0:34501 2014-06-09 10:18:56.197489 2014-06-09 16:22:11.663342 7f868e75f700 10 op_submit_finish 3394 2014-06-09 16:22:11.663335 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'643 -> 213'643 2014-06-09 16:22:11.663343 7f868c75b700 20 ? 214'611 (214'610) modify 7121e06e/rbd_data.115e6b8b4567.00000000000017ff/head//3 by client.4542.0:34500 2014-06-09 10:18:52.730120 2014-06-09 16:22:11.663349 7f868c75b700 20 merge_log 214'611 (214'610) modify 7121e06e/rbd_data.115e6b8b4567.00000000000017ff/head//3 by client.4542.0:34500 2014-06-09 10:18:52.730120 2014-06-09 16:22:11.663347 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 1 2014-06-09 16:22:11.663361 7f868c75b700 20 merge_log 214'612 (214'611) modify 7121e06e/rbd_data.115e6b8b4567.00000000000017ff/head//3 by client.4542.0:34501 2014-06-09 10:18:56.197489 2014-06-09 16:22:11.663361 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.1: 3.709( v 213'575 (0'0,213'575] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) log((0'0,213'575], crt=0'0) missing(0) 2014-06-09 16:22:11.663374 7f868e75f700 10 proc_replica_log for osd.1: 3.709( v 213'575 (0'0,213'575] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) log((0'0,213'575], crt=0'0) missing(0) 2014-06-09 16:22:11.663379 7f868e75f700 20 merge_log cut point (usually last shared) is 213'575 (213'574) modify 3dee7709/rbd_data.11786b8b4567.00000000000016a8/head//3 by client.4568.0:34549 2014-06-09 10:18:03.518687 2014-06-09 16:22:11.663381 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.663418 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.663422 7f868bf5a700 5 queue_transactions existing osr(3.5ce 0x3ed17c0)/0x3ed17c0 2014-06-09 16:22:11.663424 7f868bf5a700 2 waiting 51 > 50 ops || 1270842 > 104857600 2014-06-09 16:22:11.663430 7f8689755700 10 op_submit_start 3395 2014-06-09 16:22:11.663431 7f8689755700 5 queue_transactions (trailing journal) 3395 2014-06-09 16:22:11.663433 7f8689755700 5 queue_op 0x10039740 seq 3395 osr(3.527 0x3ed1660) 1335 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.663436 7f8689755700 10 op_submit_finish 3395 2014-06-09 16:22:11.663441 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=203 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.663454 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=203 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.663466 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=203 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.263212 7 0.000181 2014-06-09 16:22:11.663477 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=203 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.663487 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=203 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.663496 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=203 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.663506 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'697 -> 213'697 2014-06-09 16:22:11.663541 7f868c75b700 10 merge_log result log((0'0,214'612], crt=214'612) missing(1) changed=1 2014-06-09 16:22:11.663548 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.663547 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 peering m=1] peer osd.3 now 3.6e( v 214'612 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) missing(0) 2014-06-09 16:22:11.663567 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.649655 6 0.000269 2014-06-09 16:22:11.663583 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.663587 7f8689755700 5 queue_transactions existing osr(3.52b 0x3ed1640)/0x3ed1640 2014-06-09 16:22:11.663582 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.663590 7f8689755700 2 waiting 51 > 50 ops || 1270748 > 104857600 2014-06-09 16:22:11.663594 7f8690f64700 10 op_submit_start 3396 2014-06-09 16:22:11.663591 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 peering m=1] state: osd.3 has no missing, identical log 2014-06-09 16:22:11.663596 7f8690f64700 5 queue_transactions (trailing journal) 3396 2014-06-09 16:22:11.663597 7f8690f64700 5 queue_op 0x2fcaf800 seq 3396 osr(1.fa 0x384c970) 1880 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.663600 7f8690f64700 10 op_submit_finish 3396 2014-06-09 16:22:11.663599 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000017 0 0.000000 2014-06-09 16:22:11.663606 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.658681 7f868d75d700 5 -- op tracker -- , seq: 626, time: 2014-06-09 16:22:11.656747, event: reached_pg, op: osd_op(client.4573.0:34051 rbd_data.117d6b8b4567.00000000000016ba [set-alloc-hint object_size 4194304 write_size 4194304,write 0~540672] 3.4129182b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.663629 7f868b759700 10 op_submit_start 3397 2014-06-09 16:22:11.663631 7f868b759700 5 queue_transactions (trailing journal) 3397 2014-06-09 16:22:11.663632 7f868b759700 5 queue_op 0x308ca900 seq 3397 osr(0.129 0x2bd66b0) 2099 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.663636 7f868b759700 10 op_submit_finish 3397 2014-06-09 16:22:11.663629 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.484136 0 0.000000 2014-06-09 16:22:11.663641 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.663640 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.663636 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.663649 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.663653 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x230dbf00 2014-06-09 16:22:11.663651 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663662 7f868d75d700 10 osd.26 243 dequeue_op 0x230dbf00 finish 2014-06-09 16:22:11.663662 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663665 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.663666 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.663663 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.663673 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.663668 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.190018 7 0.000188 2014-06-09 16:22:11.663674 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.663677 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.663683 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'610 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.663688 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.663697 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.663704 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.663712 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.663757 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.663762 7f868b759700 5 queue_transactions existing osr(0.6b6 0x2bd7fb0)/0x2bd7fb0 2014-06-09 16:22:11.663764 7f868b759700 2 waiting 51 > 50 ops || 1271422 > 104857600 2014-06-09 16:22:11.663605 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.663772 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663784 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.663787 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.663790 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.648( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.326510 7 0.000197 2014-06-09 16:22:11.663800 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.648( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.663800 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.663810 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.648( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.663812 7f868c75b700 10 osd.26 243 queue_for_recovery queued pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.663817 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x4fce000 (1 active) 2014-06-09 16:22:11.663810 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.1 now 3.709( v 213'575 (0'0,213'575] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.663822 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.663819 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.663824 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.663828 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.663825 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 3.6e( v 214'612 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) 2014-06-09 16:22:11.663835 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.663835 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.636616 7 0.000313 2014-06-09 16:22:11.663838 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 is up to date, queueing in pending_activators 2014-06-09 16:22:11.663845 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.663855 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.947276 0 0.000000 2014-06-09 16:22:11.663856 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 3.6e( v 214'612 (0'0,214'612] local-les=201 n=117 ec=118 les/c 201/201 242/242/242) uptodate 2014-06-09 16:22:11.663867 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.663866 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.663879 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.663883 7f8690f64700 5 queue_transactions existing osr(0.648 0x2bd7bd0)/0x2bd7bd0 2014-06-09 16:22:11.663877 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.663885 7f8690f64700 2 waiting 51 > 50 ops || 1271414 > 104857600 2014-06-09 16:22:11.659630 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a7d00,0xbd74140] 2014-06-09 16:22:11.663879 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 7121e06e/rbd_data.115e6b8b4567.00000000000017ff/head//3 214'612 also missing on osd.26 2014-06-09 16:22:11.663888 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.663891 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {7121e06e/rbd_data.115e6b8b4567.00000000000017ff/head//3=214'612(214'610)} 2014-06-09 16:22:11.663894 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.663897 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.663899 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 3 as a source 2014-06-09 16:22:11.663909 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.663905 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'577 -> 214'577 2014-06-09 16:22:11.663906 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 7121e06e/rbd_data.115e6b8b4567.00000000000017ff/head//3 214'612 is on osd.3 2014-06-09 16:22:11.663919 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {7121e06e/rbd_data.115e6b8b4567.00000000000017ff/head//3=214'612(214'610)} 2014-06-09 16:22:11.663928 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.663932 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.663944 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 201 2014-06-09 16:22:11.663950 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.663949 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.709( v 213'575 (0'0,213'575] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.663962 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.663956 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 3 2014-06-09 16:22:11.663966 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 sending log((213'575,214'577], crt=0'0) 2014-06-09 16:22:11.663969 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.663978 7f868e75f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.663976 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.663980 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.663990 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a7d00,0xbd74140] 2014-06-09 16:22:11.663992 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.663993 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.663988 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.709( v 214'577 lc 213'575 (0'0,214'577] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.664001 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.1 has 1 missing 2014-06-09 16:22:11.664010 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 7d86a709/rbd_data.11666b8b4567.0000000000001836/head//3 214'577 is on osd.26 2014-06-09 16:22:11.664018 7f868c75b700 10 write_log with: dirty_to: 0'0, dirty_from: 214'610, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.664023 7f8692767700 10 op_submit_start 3398 2014-06-09 16:22:11.664020 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {7d86a709/rbd_data.11666b8b4567.0000000000001836/head//3=214'577} 2014-06-09 16:22:11.664029 7f8692767700 5 queue_transactions (trailing journal) 3398 2014-06-09 16:22:11.664030 7f8692767700 5 queue_op 0x30d96f00 seq 3398 osr(1.1cf 0x384ca70) 1974 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664034 7f8692767700 10 op_submit_finish 3398 2014-06-09 16:22:11.664029 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 1 as a source 2014-06-09 16:22:11.664040 7f868cf5c700 10 op_submit_start 3399 2014-06-09 16:22:11.664043 7f868cf5c700 5 queue_transactions (trailing journal) 3399 2014-06-09 16:22:11.664038 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 7d86a709/rbd_data.11666b8b4567.0000000000001836/head//3 214'577 also missing on osd.1 2014-06-09 16:22:11.664045 7f868cf5c700 5 queue_op 0x30d96600 seq 3399 osr(3.753 0x3ed1e50) 1335 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664048 7f868cf5c700 10 op_submit_finish 3399 2014-06-09 16:22:11.664040 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.664047 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {7d86a709/rbd_data.11666b8b4567.0000000000001836/head//3=214'577} 2014-06-09 16:22:11.664055 7f868c75b700 5 queue_transactions existing osr(3.6e 0x3ed02a0)/0x3ed02a0 2014-06-09 16:22:11.664058 7f868c75b700 2 waiting 51 > 50 ops || 1271350 > 104857600 2014-06-09 16:22:11.664051 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664056 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.664062 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664054 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.664066 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.664064 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.664069 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.766553 7 0.000193 2014-06-09 16:22:11.664067 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664078 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664074 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 1 2014-06-09 16:22:11.664082 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.664079 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.664082 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.664085 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.448( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.679793 7 0.000161 2014-06-09 16:22:11.664088 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.664091 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.664094 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.448( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.664097 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.663080 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d248c0 2014-06-09 16:22:11.664103 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.448( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.664106 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.664112 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.664114 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.664118 7f869af78700 5 -- op tracker -- , seq: 2398, time: 2014-06-09 16:22:11.664113, event: started, op: pg_info(1 pgs e243:3.555(2)) 2014-06-09 16:22:11.664119 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.664127 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.664144 7f869af78700 5 -- op tracker -- , seq: 2398, time: 2014-06-09 16:22:11.664142, event: done, op: pg_info(1 pgs e243:3.555(2)) 2014-06-09 16:22:11.664154 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664155 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664159 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.664164 7f8692767700 5 queue_transactions existing osr(0.70e 0x384c0f0)/0x384c0f0 2014-06-09 16:22:11.664167 7f8692767700 2 waiting 51 > 50 ops || 1271296 > 104857600 2014-06-09 16:22:11.664170 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.661259 7f86a5e11700 10 _generic_write 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 0~962560 = 0 2014-06-09 16:22:11.664174 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.664174 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664174 7f868cf5c700 5 queue_transactions existing osr(0.448 0x2bd7190)/0x2bd7190 2014-06-09 16:22:11.664176 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664176 7f868cf5c700 2 waiting 51 > 50 ops || 1271296 > 104857600 2014-06-09 16:22:11.664177 7f868e75f700 5 queue_transactions existing osr(3.709 0x3ed1bc0)/0x3ed1bc0 2014-06-09 16:22:11.664177 7f869af78700 20 osd.26 243 _dispatch 0x310e3dc0 pg_info(1 pgs e243:3.661) v4 2014-06-09 16:22:11.664180 7f868e75f700 2 waiting 51 > 50 ops || 1270466 > 104857600 2014-06-09 16:22:11.664185 7f868ef60700 10 op_submit_start 3400 2014-06-09 16:22:11.664186 7f868ef60700 5 queue_transactions (trailing journal) 3400 2014-06-09 16:22:11.664183 7f869af78700 5 -- op tracker -- , seq: 2399, time: 2014-06-09 16:22:11.400112, event: header_read, op: pg_info(1 pgs e243:3.661) 2014-06-09 16:22:11.664187 7f868ef60700 5 queue_op 0x2fcafc80 seq 3400 osr(3.60 0x3ed02e0) 219476 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664190 7f868ef60700 10 op_submit_finish 3400 2014-06-09 16:22:11.664201 7f869af78700 5 -- op tracker -- , seq: 2399, time: 2014-06-09 16:22:11.400112, event: throttled, op: pg_info(1 pgs e243:3.661) 2014-06-09 16:22:11.664209 7f869af78700 5 -- op tracker -- , seq: 2399, time: 2014-06-09 16:22:11.400140, event: all_read, op: pg_info(1 pgs e243:3.661) 2014-06-09 16:22:11.664197 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc295200 243'638 rep_tid=2 committed?=0 applied?=0 lock=0 op=osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.664216 7f869af78700 5 -- op tracker -- , seq: 2399, time: 2014-06-09 16:22:11.664173, event: dispatched, op: pg_info(1 pgs e243:3.661) 2014-06-09 16:22:11.664221 7f868ef60700 10 osd.26 243 dequeue_op 0x1b0b7200 finish 2014-06-09 16:22:11.664227 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.664223 7f869af78700 5 -- op tracker -- , seq: 2399, time: 2014-06-09 16:22:11.664221, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.661) 2014-06-09 16:22:11.664232 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.661) v4 from osd.6 2014-06-09 16:22:11.664236 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e3dc0 2014-06-09 16:22:11.664241 7f869af78700 5 -- op tracker -- , seq: 2399, time: 2014-06-09 16:22:11.664239, event: started, op: pg_info(1 pgs e243:3.661) 2014-06-09 16:22:11.664231 7f8691765700 10 osd.26 243 dequeue_op 0x230da100 prio 63 cost 516096 latency 0.901292 osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.664257 7f8691765700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.664252, event: reached_pg, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.664262 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:11.664265 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.664267 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 213'662 (0'0,213'662] local-les=213 n=126 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 9 2014-06-09 16:22:11.664275 7f869af78700 5 -- op tracker -- , seq: 2399, time: 2014-06-09 16:22:11.664271, event: done, op: pg_info(1 pgs e243:3.661) 2014-06-09 16:22:11.664277 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x230da100 2014-06-09 16:22:11.664283 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664286 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664280 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 213'662 (0'0,213'662] local-les=213 n=126 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd9 2014-06-09 16:22:11.664284 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.664291 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 213'662 (0'0,213'662] local-les=213 n=126 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.664300 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664301 7f86a5e11700 15 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.664302 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664304 7f869af78700 20 osd.26 243 _dispatch 0x30b8c380 pg_info(1 pgs e243:3.7ce) v4 2014-06-09 16:22:11.664300 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x30d99080 2014-06-09 16:22:11.664299 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 213'662 (0'0,213'662] local-les=213 n=126 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.664310 7f869af78700 5 -- op tracker -- , seq: 2400, time: 2014-06-09 16:22:11.371120, event: header_read, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.664307 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30d99080 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 rwstate(write n=1 w=0) oi: 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3(243'637 client.4570.0:34069 wrlock_by=unknown.0.0:0 dirty s 962560 uv637) ssc: 0x30dd4c40 snapset: 0=[]:[]+head 2014-06-09 16:22:11.664308 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 213'662 (0'0,213'662] local-les=213 n=126 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.9: log((213'662,214'665], crt=0'0) missing(0) 2014-06-09 16:22:11.664316 7f86a5e11700 10 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.664318 7f868ef60700 10 merge_log log((213'662,214'665], crt=0'0) from osd.9 into log((0'0,213'662], crt=0'0) 2014-06-09 16:22:11.664317 7f869af78700 5 -- op tracker -- , seq: 2400, time: 2014-06-09 16:22:11.371120, event: throttled, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.664322 7f868ef60700 10 merge_log extending head to 214'665 2014-06-09 16:22:11.664322 7f86a5e11700 15 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.664317 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 @head oi=886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3(243'637 client.4570.0:34069 wrlock_by=unknown.0.0:0 dirty s 962560 uv637) 2014-06-09 16:22:11.664324 7f869af78700 5 -- op tracker -- , seq: 2400, time: 2014-06-09 16:22:11.371130, event: all_read, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.664323 7f868ef60700 20 ? 214'665 (214'664) modify cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3 by client.4541.0:32284 2014-06-09 10:19:26.113126 2014-06-09 16:22:11.664330 7f86a5e11700 10 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.664330 7f869af78700 5 -- op tracker -- , seq: 2400, time: 2014-06-09 16:22:11.664300, event: dispatched, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.664338 7f8691f66700 10 op_submit_start 3401 2014-06-09 16:22:11.664339 7f8691f66700 5 queue_transactions (trailing journal) 3401 2014-06-09 16:22:11.664331 7f868ef60700 20 ? 214'664 (214'663) modify cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3 by client.4541.0:32283 2014-06-09 10:19:21.866081 2014-06-09 16:22:11.664341 7f8691f66700 5 queue_op 0x308cb860 seq 3401 osr(0.725 0x384c3d0) 1840 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664343 7f8691f66700 10 op_submit_finish 3401 2014-06-09 16:22:11.664341 7f869af78700 5 -- op tracker -- , seq: 2400, time: 2014-06-09 16:22:11.664338, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.664343 7f868ef60700 20 ? 214'663 (0'0) modify cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3 by client.4541.0:32282 2014-06-09 10:19:18.847502 2014-06-09 16:22:11.664347 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7ce) v4 from osd.0 2014-06-09 16:22:11.664349 7f8690763700 10 op_submit_start 3402 2014-06-09 16:22:11.664351 7f8690763700 5 queue_transactions (trailing journal) 3402 2014-06-09 16:22:11.664351 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8c380 2014-06-09 16:22:11.664349 7f868ef60700 20 merge_log 214'663 (0'0) modify cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3 by client.4541.0:32282 2014-06-09 10:19:18.847502 2014-06-09 16:22:11.664352 7f8690763700 5 queue_op 0x2fcaec60 seq 3402 osr(2.3aa 0x2ed1070) 1954 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664355 7f8690763700 10 op_submit_finish 3402 2014-06-09 16:22:11.664350 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.423( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:2.423( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.664356 7f8691765700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.664331, event: started, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.664358 7f868ef60700 20 merge_log 214'664 (214'663) modify cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3 by client.4541.0:32283 2014-06-09 10:19:21.866081 2014-06-09 16:22:11.664363 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.423( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.664360 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.532( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.664366 7f868ef60700 20 merge_log 214'665 (214'664) modify cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3 by client.4541.0:32284 2014-06-09 10:19:26.113126 2014-06-09 16:22:11.664373 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.664370 7f869af78700 5 -- op tracker -- , seq: 2400, time: 2014-06-09 16:22:11.664354, event: started, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.664365 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x308a6c00 2014-06-09 16:22:11.664373 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.664383 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.664377 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.158( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.664385 7f8691765700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.664380, event: started, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.664388 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.664387 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.53d( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.664397 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.664397 7f869af78700 5 -- op tracker -- , seq: 2400, time: 2014-06-09 16:22:11.664387, event: done, op: pg_info(1 pgs e243:3.7ce) 2014-06-09 16:22:11.664392 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] ov 243'637 av 243'639 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.664398 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.664406 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.664406 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664400 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.664409 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664406 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 2014-06-09 16:22:11.664410 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664421 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664416 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.664424 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0250 (19 active) 2014-06-09 16:22:11.664424 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.664425 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664431 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.664432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664434 7f869af78700 20 osd.26 243 _dispatch 0x3115b880 pg_info(1 pgs e243:0.1ca) v4 2014-06-09 16:22:11.664428 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 962560~516096 2014-06-09 16:22:11.664431 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.158088 7 0.000148 2014-06-09 16:22:11.664440 7f869af78700 5 -- op tracker -- , seq: 2401, time: 2014-06-09 16:22:11.415680, event: header_read, op: pg_info(1 pgs e243:0.1ca) 2014-06-09 16:22:11.664442 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.664435 7f8690763700 10 osd.26 243 dequeue_op 0x230dbe00 prio 63 cost 516096 latency 0.941115 osd_op(client.4573.0:34052 rbd_data.117d6b8b4567.00000000000016ba [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~516096] 3.4129182b RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.664454 7f868f761700 10 op_submit_start 3403 2014-06-09 16:22:11.664455 7f868f761700 5 queue_transactions (trailing journal) 3403 2014-06-09 16:22:11.664451 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.664456 7f868f761700 5 queue_op 0x30d96ae0 seq 3403 osr(2.409 0x2ed1270) 1883 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664460 7f868f761700 10 op_submit_finish 3403 2014-06-09 16:22:11.664445 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228a00 2014-06-09 16:22:11.664461 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.664465 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=206 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.664470 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.664466 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 snapset=0x30dd4c78 snapc=0=[] 2014-06-09 16:22:11.664477 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.664477 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=206 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.664479 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.664489 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=206 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.832584 7 0.000278 2014-06-09 16:22:11.664490 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 0x308a6c00 op modify 2014-06-09 16:22:11.664503 7f8692f68700 10 op_submit_start 3404 2014-06-09 16:22:11.664505 7f8692f68700 5 queue_transactions (trailing journal) 3404 2014-06-09 16:22:11.664501 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=206 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.664507 7f8692f68700 5 queue_op 0x309a3380 seq 3404 osr(0.47d 0x2bd7470) 1883 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664511 7f8692f68700 10 op_submit_finish 3404 2014-06-09 16:22:11.664502 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:07.612964 2014-06-09 16:22:11.664511 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=206 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.664521 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.664525 7f8691f66700 5 queue_transactions existing osr(0.5f5 0x2bd7990)/0x2bd7990 2014-06-09 16:22:11.664527 7f8691f66700 2 waiting 51 > 50 ops || 1271296 > 104857600 2014-06-09 16:22:11.664521 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=206 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.664518 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.664523 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.664532 7f868af58700 10 op_submit_start 3405 2014-06-09 16:22:11.664534 7f868af58700 5 queue_transactions (trailing journal) 3405 2014-06-09 16:22:11.664535 7f868af58700 5 queue_op 0x30d97c80 seq 3405 osr(1.77c 0x2ed0330) 2009 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664538 7f868af58700 10 op_submit_finish 3405 2014-06-09 16:22:11.664530 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'621 -> 213'621 2014-06-09 16:22:11.664534 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.664541 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.664543 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.669( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.669( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.664548 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.473918 7 0.000190 2014-06-09 16:22:11.664556 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.669( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.664552 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 3 on osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.664566 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.664561 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.664567 7f8691765700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 3 o 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.664570 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.664574 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.664574 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.664586 7f868ef60700 10 merge_log result log((0'0,214'665], crt=214'665) missing(1) changed=1 2014-06-09 16:22:11.664580 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664591 7f868af58700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664586 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.664594 7f868af58700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.664591 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 peering m=1] peer osd.9 now 3.799( v 214'665 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) missing(0) 2014-06-09 16:22:11.664597 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.399338 7 0.000229 2014-06-09 16:22:11.664597 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'549 -> 213'549 2014-06-09 16:22:11.664611 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.664608 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.664609 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.675925 5 0.000236 2014-06-09 16:22:11.664615 7f868f761700 5 queue_transactions existing osr(3.437 0x3ed11c0)/0x3ed11c0 2014-06-09 16:22:11.664617 7f868f761700 2 waiting 51 > 50 ops || 1270748 > 104857600 2014-06-09 16:22:11.664617 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.664623 7f868df5e700 10 op_submit_start 3406 2014-06-09 16:22:11.664624 7f868df5e700 5 queue_transactions (trailing journal) 3406 2014-06-09 16:22:11.664617 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.664626 7f868df5e700 5 queue_op 0x2fcae120 seq 3406 osr(1.26c 0x384cf50) 1997 bytes (queue has 50 ops and 1269413 bytes) 2014-06-09 16:22:11.664629 7f868df5e700 10 op_submit_finish 3406 2014-06-09 16:22:11.664624 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 peering m=1] state: osd.9 has no missing, identical log 2014-06-09 16:22:11.664627 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.664632 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000015 0 0.000000 2014-06-09 16:22:11.664632 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.664636 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.664634 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.664640 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.664644 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.664645 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.664647 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.463162 0 0.000000 2014-06-09 16:22:11.664653 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.664656 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.664662 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.664661 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.664669 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.664670 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.664678 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.664675 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.664684 7f8692f68700 5 queue_transactions existing osr(3.46f 0x3ed1560)/0x3ed1560 2014-06-09 16:22:11.664678 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.664680 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 213'662 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.664687 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.664687 7f8692f68700 2 waiting 51 > 50 ops || 1270748 > 104857600 2014-06-09 16:22:11.664691 7f868af58700 5 queue_transactions existing osr(0.33a 0x2bd70b0)/0x2bd70b0 2014-06-09 16:22:11.664693 7f868af58700 2 waiting 51 > 50 ops || 1271410 > 104857600 2014-06-09 16:22:11.664694 7f869af78700 5 -- op tracker -- , seq: 2401, time: 2014-06-09 16:22:11.415680, event: throttled, op: pg_info(1 pgs e243:0.1ca) 2014-06-09 16:22:11.664693 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.658733 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.664702 7f868df5e700 20 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.664711 7f8690763700 5 -- op tracker -- , seq: 632, time: 2014-06-09 16:22:11.664450, event: reached_pg, op: osd_op(client.4573.0:34052 rbd_data.117d6b8b4567.00000000000016ba [set-alloc-hint object_size 4194304 write_size 4194304,write 540672~516096] 3.4129182b RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.664704 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664710 7f868df5e700 20 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.664719 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.664718 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.664721 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.664725 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.799342 7 0.000189 2014-06-09 16:22:11.664720 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.664728 7f8691765700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.664585, event: waiting for subops from 7, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.664736 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02a0 (19 active) 2014-06-09 16:22:11.664734 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.664735 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x230dbe00 2014-06-09 16:22:11.664744 7f8690763700 10 osd.26 243 dequeue_op 0x230dbe00 finish 2014-06-09 16:22:11.664748 7f8691765700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.664744 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.664755 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f02a0 (20 active) 2014-06-09 16:22:11.664753 7f869af78700 5 -- op tracker -- , seq: 2401, time: 2014-06-09 16:22:11.415691, event: all_read, op: pg_info(1 pgs e243:0.1ca) 2014-06-09 16:22:11.664752 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.664761 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.664762 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.664767 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.664761 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.664769 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.664775 7f869af78700 5 -- op tracker -- , seq: 2401, time: 2014-06-09 16:22:11.664424, event: dispatched, op: pg_info(1 pgs e243:0.1ca) 2014-06-09 16:22:11.664774 7f868df5e700 15 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.664783 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.664770 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'638 (0'0,243'638] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'638], crt=0'0) [243'639 (243'637) modify 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 by client.4570.0:34070 2014-06-09 10:20:07.612964] 2014-06-09 16:22:11.664781 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.664783 7f869af78700 5 -- op tracker -- , seq: 2401, time: 2014-06-09 16:22:11.664781, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1ca) 2014-06-09 16:22:11.664793 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1ca) v4 from osd.5 2014-06-09 16:22:11.664787 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.664797 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115b880 2014-06-09 16:22:11.664792 7f868ef60700 10 osd.26 243 queue_for_recovery queued pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.664800 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.664803 7f869af78700 5 -- op tracker -- , seq: 2401, time: 2014-06-09 16:22:11.664800, event: started, op: pg_info(1 pgs e243:0.1ca) 2014-06-09 16:22:11.664811 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.664813 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.664806 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 3.799( v 214'665 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) 2014-06-09 16:22:11.664814 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.664818 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 is up to date, queueing in pending_activators 2014-06-09 16:22:11.664816 7f868a757700 5 queue_transactions existing osr(0.1cd 0x2bd6960)/0x2bd6960 2014-06-09 16:22:11.664824 7f869af78700 5 -- op tracker -- , seq: 2401, time: 2014-06-09 16:22:11.664821, event: done, op: pg_info(1 pgs e243:0.1ca) 2014-06-09 16:22:11.664829 7f868a757700 2 waiting 51 > 50 ops || 1271414 > 104857600 2014-06-09 16:22:11.664825 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.289( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.664833 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664834 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.664835 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664833 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 3.799( v 214'665 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242) uptodate 2014-06-09 16:22:11.664839 7f8682ae5700 5 -- op tracker -- , seq: 2402, time: 2014-06-09 16:22:11.661796, event: header_read, op: osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.664838 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.664849 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664850 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664849 7f8682ae5700 5 -- op tracker -- , seq: 2402, time: 2014-06-09 16:22:11.661796, event: throttled, op: osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.664852 7f869af78700 20 osd.26 243 _dispatch 0x304f8400 pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.664845 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.664849 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.664857 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.664857 7f8682ae5700 5 -- op tracker -- , seq: 2402, time: 2014-06-09 16:22:11.661805, event: all_read, op: osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.664857 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3 214'665 also missing on osd.26 2014-06-09 16:22:11.664865 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.664874 7f8682ae5700 5 -- op tracker -- , seq: 2402, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.664868 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3=214'665} 2014-06-09 16:22:11.664874 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.664881 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.664879 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 9 as a source 2014-06-09 16:22:11.664881 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.664887 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.664894 7f8682ae5700 15 osd.26 243 enqueue_op 0x257c4300 prio 196 cost 0 latency 0.003097 osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.664889 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3 214'665 is on osd.9 2014-06-09 16:22:11.664894 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:190 2014-06-09 16:22:11.664900 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {cd13df99/rbd_data.115d6b8b4567.000000000000164d/head//3=214'665} 2014-06-09 16:22:11.664904 7f868df5e700 20 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.664913 7f869af78700 5 -- op tracker -- , seq: 2403, time: 2014-06-09 16:22:11.264934, event: header_read, op: pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.664911 7f868df5e700 20 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.664911 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.664920 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.664921 7f869af78700 5 -- op tracker -- , seq: 2403, time: 2014-06-09 16:22:11.264935, event: throttled, op: pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.664920 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 213 2014-06-09 16:22:11.664927 7f869af78700 5 -- op tracker -- , seq: 2403, time: 2014-06-09 16:22:11.266045, event: all_read, op: pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.664924 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.664934 7f869af78700 5 -- op tracker -- , seq: 2403, time: 2014-06-09 16:22:11.664849, event: dispatched, op: pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.664931 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 9 2014-06-09 16:22:11.664935 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.664941 7f869af78700 5 -- op tracker -- , seq: 2403, time: 2014-06-09 16:22:11.664939, event: waiting_for_osdmap, op: pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.664946 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) v4 from osd.2 2014-06-09 16:22:11.664943 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.664941 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.664950 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x304f8400 2014-06-09 16:22:11.664951 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.664955 7f869af78700 5 -- op tracker -- , seq: 2403, time: 2014-06-09 16:22:11.664953, event: started, op: pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.664952 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.664958 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.664969 7f869af78700 5 -- op tracker -- , seq: 2403, time: 2014-06-09 16:22:11.664966, event: done, op: pg_log(3.582 epoch 243 log log((0'0,213'598], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.664966 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.664978 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:192 2014-06-09 16:22:11.664983 7f868ef60700 10 write_log with: dirty_to: 0'0, dirty_from: 213'662, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.664985 7f868df5e700 20 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.664991 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.664993 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.664992 7f868df5e700 20 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.665000 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665005 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665006 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665007 7f869af78700 20 osd.26 243 _dispatch 0x29ec5c00 pg_notify(0.759(20),1.1c9(23) epoch 243) v5 2014-06-09 16:22:11.665003 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.665013 7f869af78700 5 -- op tracker -- , seq: 2404, time: 2014-06-09 16:22:11.331530, event: header_read, op: pg_notify(0.759(20),1.1c9(23) epoch 243) 2014-06-09 16:22:11.665019 7f868ef60700 5 queue_transactions existing osr(3.799 0x3ed1fa0)/0x3ed1fa0 2014-06-09 16:22:11.665014 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.665022 7f868ef60700 2 waiting 51 > 50 ops || 1271573 > 104857600 2014-06-09 16:22:11.665020 7f869af78700 5 -- op tracker -- , seq: 2404, time: 2014-06-09 16:22:11.331531, event: throttled, op: pg_notify(0.759(20),1.1c9(23) epoch 243) 2014-06-09 16:22:11.665023 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.665026 7f869af78700 5 -- op tracker -- , seq: 2404, time: 2014-06-09 16:22:11.331583, event: all_read, op: pg_notify(0.759(20),1.1c9(23) epoch 243) 2014-06-09 16:22:11.665032 7f869af78700 5 -- op tracker -- , seq: 2404, time: 2014-06-09 16:22:11.665005, event: dispatched, op: pg_notify(0.759(20),1.1c9(23) epoch 243) 2014-06-09 16:22:11.665030 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.665039 7f869af78700 5 -- op tracker -- , seq: 2404, time: 2014-06-09 16:22:11.665037, event: waiting_for_osdmap, op: pg_notify(0.759(20),1.1c9(23) epoch 243) 2014-06-09 16:22:11.665042 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.665043 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.665045 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec5c00 2014-06-09 16:22:11.665050 7f869af78700 5 -- op tracker -- , seq: 2404, time: 2014-06-09 16:22:11.665047, event: started, op: pg_notify(0.759(20),1.1c9(23) epoch 243) 2014-06-09 16:22:11.665057 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.665070 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665074 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.665086 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.665094 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.665099 7f869af78700 5 -- op tracker -- , seq: 2404, time: 2014-06-09 16:22:11.665097, event: done, op: pg_notify(0.759(20),1.1c9(23) epoch 243) 2014-06-09 16:22:11.665101 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.665108 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665109 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665112 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.665126 7f868df5e700 15 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.665141 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665146 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.665159 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.665171 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665172 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665167 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.665173 7f869af78700 20 osd.26 243 _dispatch 0x30bd8fc0 pg_info(1 pgs e243:1.92) v4 2014-06-09 16:22:11.665174 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.665178 7f869af78700 5 -- op tracker -- , seq: 2405, time: 2014-06-09 16:22:11.390519, event: header_read, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.665186 7f868df5e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.665186 7f869af78700 5 -- op tracker -- , seq: 2405, time: 2014-06-09 16:22:11.390519, event: throttled, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.665193 7f869af78700 5 -- op tracker -- , seq: 2405, time: 2014-06-09 16:22:11.390529, event: all_read, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.665202 7f869af78700 5 -- op tracker -- , seq: 2405, time: 2014-06-09 16:22:11.665170, event: dispatched, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.665200 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.665208 7f869af78700 5 -- op tracker -- , seq: 2405, time: 2014-06-09 16:22:11.665206, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.665213 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665214 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.92) v4 from osd.7 2014-06-09 16:22:11.665217 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd8fc0 2014-06-09 16:22:11.665217 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:0.34e( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.665221 7f869af78700 5 -- op tracker -- , seq: 2405, time: 2014-06-09 16:22:11.665219, event: started, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.665229 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.665238 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665238 7f869af78700 5 -- op tracker -- , seq: 2405, time: 2014-06-09 16:22:11.665236, event: done, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.665250 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665242 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:0.89( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.665251 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665253 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.665262 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665264 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665265 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665266 7f869af78700 20 osd.26 243 _dispatch 0x2eef6540 pg_info(1 pgs e243:3.4c9(2)) v4 2014-06-09 16:22:11.665266 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.665271 7f869af78700 5 -- op tracker -- , seq: 2406, time: 2014-06-09 16:22:11.437241, event: header_read, op: pg_info(1 pgs e243:3.4c9(2)) 2014-06-09 16:22:11.665278 7f869af78700 5 -- op tracker -- , seq: 2406, time: 2014-06-09 16:22:11.437241, event: throttled, op: pg_info(1 pgs e243:3.4c9(2)) 2014-06-09 16:22:11.665278 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.665286 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.665293 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.665285 7f869af78700 5 -- op tracker -- , seq: 2406, time: 2014-06-09 16:22:11.437258, event: all_read, op: pg_info(1 pgs e243:3.4c9(2)) 2014-06-09 16:22:11.665304 7f868df5e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.665308 7f869af78700 5 -- op tracker -- , seq: 2406, time: 2014-06-09 16:22:11.665263, event: dispatched, op: pg_info(1 pgs e243:3.4c9(2)) 2014-06-09 16:22:11.665319 7f868df5e700 15 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.665332 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665336 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.665315 7f869af78700 5 -- op tracker -- , seq: 2406, time: 2014-06-09 16:22:11.665313, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c9(2)) 2014-06-09 16:22:11.665353 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c9(2)) v4 from osd.3 2014-06-09 16:22:11.665349 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.665356 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef6540 2014-06-09 16:22:11.665357 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.664793 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'639 (243'637) modify 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 by client.4570.0:34070 2014-06-09 10:20:07.612964 2014-06-09 16:22:11.665361 7f869af78700 5 -- op tracker -- , seq: 2406, time: 2014-06-09 16:22:11.665359, event: started, op: pg_info(1 pgs e243:3.4c9(2)) 2014-06-09 16:22:11.665364 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.665375 7f868df5e700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.665371 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.665389 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.665403 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665379 7f869af78700 5 -- op tracker -- , seq: 2406, time: 2014-06-09 16:22:11.665377, event: done, op: pg_info(1 pgs e243:3.4c9(2)) 2014-06-09 16:22:11.665409 7f8691765700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'639, trimmed: 2014-06-09 16:22:11.665411 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665414 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665407 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.665419 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.665429 7f8691765700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.665443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665446 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665427 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.665447 7f869af78700 20 osd.26 243 _dispatch 0x26000700 pg_info(1 pgs e243:2.1fa) v4 2014-06-09 16:22:11.665448 7f8691765700 2 waiting 51 > 50 ops || 1787793 > 104857600 2014-06-09 16:22:11.665449 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.665462 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.665476 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:161 2014-06-09 16:22:11.665490 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665452 7f869af78700 5 -- op tracker -- , seq: 2407, time: 2014-06-09 16:22:11.312903, event: header_read, op: pg_info(1 pgs e243:2.1fa) 2014-06-09 16:22:11.665494 7f869af78700 5 -- op tracker -- , seq: 2407, time: 2014-06-09 16:22:11.312903, event: throttled, op: pg_info(1 pgs e243:2.1fa) 2014-06-09 16:22:11.665494 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.665502 7f869af78700 5 -- op tracker -- , seq: 2407, time: 2014-06-09 16:22:11.312913, event: all_read, op: pg_info(1 pgs e243:2.1fa) 2014-06-09 16:22:11.665507 7f869af78700 5 -- op tracker -- , seq: 2407, time: 2014-06-09 16:22:11.665443, event: dispatched, op: pg_info(1 pgs e243:2.1fa) 2014-06-09 16:22:11.665504 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.665515 7f868df5e700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.665519 7f868df5e700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.665514 7f869af78700 5 -- op tracker -- , seq: 2407, time: 2014-06-09 16:22:11.665512, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1fa) 2014-06-09 16:22:11.665526 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1fa) v4 from osd.1 2014-06-09 16:22:11.665523 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.737738 7 0.000171 2014-06-09 16:22:11.665533 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.665529 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26000700 2014-06-09 16:22:11.665543 7f868df5e700 5 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.665552 7f869af78700 5 -- op tracker -- , seq: 2407, time: 2014-06-09 16:22:11.665550, event: started, op: pg_info(1 pgs e243:2.1fa) 2014-06-09 16:22:11.665553 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.665561 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.665567 7f869af78700 5 -- op tracker -- , seq: 2407, time: 2014-06-09 16:22:11.665565, event: done, op: pg_info(1 pgs e243:2.1fa) 2014-06-09 16:22:11.665569 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.665580 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665582 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665585 7f8682ae5700 5 -- op tracker -- , seq: 2408, time: 2014-06-09 16:22:11.665547, event: header_read, op: osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.665596 7f8682ae5700 5 -- op tracker -- , seq: 2408, time: 2014-06-09 16:22:11.665547, event: throttled, op: osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.665610 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665611 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665604 7f8682ae5700 5 -- op tracker -- , seq: 2408, time: 2014-06-09 16:22:11.665555, event: all_read, op: osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.665612 7f869af78700 20 osd.26 243 _dispatch 0x310dbc00 pg_info(1 pgs e243:1.46e) v4 2014-06-09 16:22:11.665617 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.665617 7f8682ae5700 5 -- op tracker -- , seq: 2408, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.665622 7f868df5e700 5 queue_transactions existing osr(2.4b9 0x2ed1730)/0x2ed1730 2014-06-09 16:22:11.665624 7f868df5e700 2 waiting 51 > 50 ops || 1271296 > 104857600 2014-06-09 16:22:11.665623 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.665628 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.665626 7f869af78700 5 -- op tracker -- , seq: 2409, time: 2014-06-09 16:22:11.399876, event: header_read, op: pg_info(1 pgs e243:1.46e) 2014-06-09 16:22:11.665635 7f8682ae5700 15 osd.26 243 enqueue_op 0x2f097b00 prio 196 cost 0 latency 0.000088 osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.665638 7f869af78700 5 -- op tracker -- , seq: 2409, time: 2014-06-09 16:22:11.399877, event: throttled, op: pg_info(1 pgs e243:1.46e) 2014-06-09 16:22:11.665644 7f869af78700 5 -- op tracker -- , seq: 2409, time: 2014-06-09 16:22:11.399891, event: all_read, op: pg_info(1 pgs e243:1.46e) 2014-06-09 16:22:11.665664 7f869af78700 5 -- op tracker -- , seq: 2409, time: 2014-06-09 16:22:11.665610, event: dispatched, op: pg_info(1 pgs e243:1.46e) 2014-06-09 16:22:11.665671 7f869af78700 5 -- op tracker -- , seq: 2409, time: 2014-06-09 16:22:11.665669, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.46e) 2014-06-09 16:22:11.665675 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.46e) v4 from osd.9 2014-06-09 16:22:11.665677 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310dbc00 2014-06-09 16:22:11.665682 7f869af78700 5 -- op tracker -- , seq: 2409, time: 2014-06-09 16:22:11.665680, event: started, op: pg_info(1 pgs e243:1.46e) 2014-06-09 16:22:11.665702 7f869af78700 5 -- op tracker -- , seq: 2409, time: 2014-06-09 16:22:11.665700, event: done, op: pg_info(1 pgs e243:1.46e) 2014-06-09 16:22:11.665718 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665720 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665733 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665734 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665735 7f869af78700 20 osd.26 243 _dispatch 0x307e0400 pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.665741 7f869af78700 5 -- op tracker -- , seq: 2410, time: 2014-06-09 16:22:11.408259, event: header_read, op: pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.665755 7f869af78700 5 -- op tracker -- , seq: 2410, time: 2014-06-09 16:22:11.408260, event: throttled, op: pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.665764 7f869af78700 5 -- op tracker -- , seq: 2410, time: 2014-06-09 16:22:11.408274, event: all_read, op: pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.665780 7f869af78700 5 -- op tracker -- , seq: 2410, time: 2014-06-09 16:22:11.665733, event: dispatched, op: pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.665788 7f869af78700 5 -- op tracker -- , seq: 2410, time: 2014-06-09 16:22:11.665786, event: waiting_for_osdmap, op: pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.665793 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.665796 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e0400 2014-06-09 16:22:11.665801 7f869af78700 5 -- op tracker -- , seq: 2410, time: 2014-06-09 16:22:11.665799, event: started, op: pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.665813 7f869af78700 5 -- op tracker -- , seq: 2410, time: 2014-06-09 16:22:11.665810, event: done, op: pg_log(0.492 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.665817 7f86a5e11700 10 cid: oid: 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 not skipping op, *spos 3358.0.0 2014-06-09 16:22:11.665820 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:11.665825 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665827 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665838 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665839 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665840 7f869af78700 20 osd.26 243 _dispatch 0x310e3c00 pg_info(1 pgs e243:2.244) v4 2014-06-09 16:22:11.665844 7f869af78700 5 -- op tracker -- , seq: 2411, time: 2014-06-09 16:22:11.400145, event: header_read, op: pg_info(1 pgs e243:2.244) 2014-06-09 16:22:11.665848 7f86a5e11700 20 set_header setting 30151 cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 parent seq 0 2014-06-09 16:22:11.665851 7f869af78700 5 -- op tracker -- , seq: 2411, time: 2014-06-09 16:22:11.400146, event: throttled, op: pg_info(1 pgs e243:2.244) 2014-06-09 16:22:11.665856 7f869af78700 5 -- op tracker -- , seq: 2411, time: 2014-06-09 16:22:11.400155, event: all_read, op: pg_info(1 pgs e243:2.244) 2014-06-09 16:22:11.665864 7f869af78700 5 -- op tracker -- , seq: 2411, time: 2014-06-09 16:22:11.665837, event: dispatched, op: pg_info(1 pgs e243:2.244) 2014-06-09 16:22:11.665871 7f869af78700 5 -- op tracker -- , seq: 2411, time: 2014-06-09 16:22:11.665869, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.244) 2014-06-09 16:22:11.665875 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.244) v4 from osd.6 2014-06-09 16:22:11.665877 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e3c00 2014-06-09 16:22:11.665882 7f869af78700 5 -- op tracker -- , seq: 2411, time: 2014-06-09 16:22:11.665880, event: started, op: pg_info(1 pgs e243:2.244) 2014-06-09 16:22:11.665901 7f869af78700 5 -- op tracker -- , seq: 2411, time: 2014-06-09 16:22:11.665899, event: done, op: pg_info(1 pgs e243:2.244) 2014-06-09 16:22:11.665908 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665910 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665921 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665923 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.665924 7f869af78700 20 osd.26 243 _dispatch 0x30b8c1c0 pg_info(1 pgs e243:1.30) v4 2014-06-09 16:22:11.665928 7f869af78700 5 -- op tracker -- , seq: 2412, time: 2014-06-09 16:22:11.371134, event: header_read, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.665934 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3358.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.665936 7f869af78700 5 -- op tracker -- , seq: 2412, time: 2014-06-09 16:22:11.371135, event: throttled, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.665940 7f86a5e11700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3358.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.665943 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.665942 7f869af78700 5 -- op tracker -- , seq: 2412, time: 2014-06-09 16:22:11.371144, event: all_read, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.665947 7f869af78700 5 -- op tracker -- , seq: 2412, time: 2014-06-09 16:22:11.665921, event: dispatched, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.665953 7f869af78700 5 -- op tracker -- , seq: 2412, time: 2014-06-09 16:22:11.665951, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.665961 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.30) v4 from osd.0 2014-06-09 16:22:11.665964 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8c1c0 2014-06-09 16:22:11.665968 7f869af78700 5 -- op tracker -- , seq: 2412, time: 2014-06-09 16:22:11.665966, event: started, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.665987 7f869af78700 5 -- op tracker -- , seq: 2412, time: 2014-06-09 16:22:11.665985, event: done, op: pg_info(1 pgs e243:1.30) 2014-06-09 16:22:11.665995 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.665997 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666010 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666011 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666012 7f869af78700 20 osd.26 243 _dispatch 0x30e0f600 pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.666018 7f869af78700 5 -- op tracker -- , seq: 2413, time: 2014-06-09 16:22:11.415695, event: header_read, op: pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666026 7f869af78700 5 -- op tracker -- , seq: 2413, time: 2014-06-09 16:22:11.415695, event: throttled, op: pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666033 7f869af78700 5 -- op tracker -- , seq: 2413, time: 2014-06-09 16:22:11.415713, event: all_read, op: pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666040 7f869af78700 5 -- op tracker -- , seq: 2413, time: 2014-06-09 16:22:11.666009, event: dispatched, op: pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666048 7f869af78700 5 -- op tracker -- , seq: 2413, time: 2014-06-09 16:22:11.666045, event: waiting_for_osdmap, op: pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666053 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) v4 from osd.5 2014-06-09 16:22:11.666056 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e0f600 2014-06-09 16:22:11.666061 7f869af78700 5 -- op tracker -- , seq: 2413, time: 2014-06-09 16:22:11.666059, event: started, op: pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666073 7f869af78700 5 -- op tracker -- , seq: 2413, time: 2014-06-09 16:22:11.666070, event: done, op: pg_log(3.542 epoch 243 log log((214'709,214'710], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666082 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666084 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666096 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666097 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666098 7f869af78700 20 osd.26 243 _dispatch 0x30b8dc00 pg_notify(3.443(12) epoch 243) v5 2014-06-09 16:22:11.666103 7f869af78700 5 -- op tracker -- , seq: 2414, time: 2014-06-09 16:22:11.266053, event: header_read, op: pg_notify(3.443(12) epoch 243) 2014-06-09 16:22:11.666111 7f869af78700 5 -- op tracker -- , seq: 2414, time: 2014-06-09 16:22:11.266053, event: throttled, op: pg_notify(3.443(12) epoch 243) 2014-06-09 16:22:11.666117 7f869af78700 5 -- op tracker -- , seq: 2414, time: 2014-06-09 16:22:11.266068, event: all_read, op: pg_notify(3.443(12) epoch 243) 2014-06-09 16:22:11.666122 7f869af78700 5 -- op tracker -- , seq: 2414, time: 2014-06-09 16:22:11.666095, event: dispatched, op: pg_notify(3.443(12) epoch 243) 2014-06-09 16:22:11.666129 7f869af78700 5 -- op tracker -- , seq: 2414, time: 2014-06-09 16:22:11.666127, event: waiting_for_osdmap, op: pg_notify(3.443(12) epoch 243) 2014-06-09 16:22:11.666132 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.666134 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8dc00 2014-06-09 16:22:11.666139 7f869af78700 5 -- op tracker -- , seq: 2414, time: 2014-06-09 16:22:11.666137, event: started, op: pg_notify(3.443(12) epoch 243) 2014-06-09 16:22:11.666161 7f869af78700 5 -- op tracker -- , seq: 2414, time: 2014-06-09 16:22:11.666159, event: done, op: pg_notify(3.443(12) epoch 243) 2014-06-09 16:22:11.666168 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666170 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666187 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666189 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666190 7f869af78700 20 osd.26 243 _dispatch 0x30a9b600 pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.666197 7f869af78700 5 -- op tracker -- , seq: 2415, time: 2014-06-09 16:22:11.332360, event: header_read, op: pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666207 7f869af78700 5 -- op tracker -- , seq: 2415, time: 2014-06-09 16:22:11.332360, event: throttled, op: pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666215 7f869af78700 5 -- op tracker -- , seq: 2415, time: 2014-06-09 16:22:11.332373, event: all_read, op: pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666222 7f869af78700 5 -- op tracker -- , seq: 2415, time: 2014-06-09 16:22:11.666187, event: dispatched, op: pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666233 7f869af78700 5 -- op tracker -- , seq: 2415, time: 2014-06-09 16:22:11.666230, event: waiting_for_osdmap, op: pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666237 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.666241 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a9b600 2014-06-09 16:22:11.666247 7f869af78700 5 -- op tracker -- , seq: 2415, time: 2014-06-09 16:22:11.666244, event: started, op: pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666261 7f869af78700 5 -- op tracker -- , seq: 2415, time: 2014-06-09 16:22:11.666258, event: done, op: pg_log(1.188 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666267 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666269 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666278 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666280 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666281 7f869af78700 20 osd.26 243 _dispatch 0x30bd9340 pg_info(1 pgs e243:1.631) v4 2014-06-09 16:22:11.666285 7f869af78700 5 -- op tracker -- , seq: 2416, time: 2014-06-09 16:22:11.390533, event: header_read, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.666291 7f869af78700 5 -- op tracker -- , seq: 2416, time: 2014-06-09 16:22:11.390533, event: throttled, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.666297 7f869af78700 5 -- op tracker -- , seq: 2416, time: 2014-06-09 16:22:11.390544, event: all_read, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.666305 7f869af78700 5 -- op tracker -- , seq: 2416, time: 2014-06-09 16:22:11.666278, event: dispatched, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.666312 7f869af78700 5 -- op tracker -- , seq: 2416, time: 2014-06-09 16:22:11.666310, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.666318 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.631) v4 from osd.7 2014-06-09 16:22:11.666321 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd9340 2014-06-09 16:22:11.666325 7f869af78700 5 -- op tracker -- , seq: 2416, time: 2014-06-09 16:22:11.666324, event: started, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.666339 7f869af78700 5 -- op tracker -- , seq: 2416, time: 2014-06-09 16:22:11.666337, event: done, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.666345 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666347 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666360 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666362 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666363 7f869af78700 20 osd.26 243 _dispatch 0x30a8a000 pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.666369 7f869af78700 5 -- op tracker -- , seq: 2417, time: 2014-06-09 16:22:11.437298, event: header_read, op: pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666376 7f869af78700 5 -- op tracker -- , seq: 2417, time: 2014-06-09 16:22:11.437299, event: throttled, op: pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666382 7f869af78700 5 -- op tracker -- , seq: 2417, time: 2014-06-09 16:22:11.437311, event: all_read, op: pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666388 7f869af78700 5 -- op tracker -- , seq: 2417, time: 2014-06-09 16:22:11.666360, event: dispatched, op: pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666395 7f869af78700 5 -- op tracker -- , seq: 2417, time: 2014-06-09 16:22:11.666392, event: waiting_for_osdmap, op: pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666401 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.666405 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a8a000 2014-06-09 16:22:11.666411 7f869af78700 5 -- op tracker -- , seq: 2417, time: 2014-06-09 16:22:11.666408, event: started, op: pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666424 7f869af78700 5 -- op tracker -- , seq: 2417, time: 2014-06-09 16:22:11.666421, event: done, op: pg_log(0.1f9 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.666430 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666442 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666443 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666444 7f869af78700 20 osd.26 243 _dispatch 0x15a28700 pg_info(1 pgs e243:1.4c2) v4 2014-06-09 16:22:11.666449 7f869af78700 5 -- op tracker -- , seq: 2418, time: 2014-06-09 16:22:11.312925, event: header_read, op: pg_info(1 pgs e243:1.4c2) 2014-06-09 16:22:11.666456 7f869af78700 5 -- op tracker -- , seq: 2418, time: 2014-06-09 16:22:11.312925, event: throttled, op: pg_info(1 pgs e243:1.4c2) 2014-06-09 16:22:11.666463 7f869af78700 5 -- op tracker -- , seq: 2418, time: 2014-06-09 16:22:11.312934, event: all_read, op: pg_info(1 pgs e243:1.4c2) 2014-06-09 16:22:11.666469 7f869af78700 5 -- op tracker -- , seq: 2418, time: 2014-06-09 16:22:11.666442, event: dispatched, op: pg_info(1 pgs e243:1.4c2) 2014-06-09 16:22:11.666475 7f869af78700 5 -- op tracker -- , seq: 2418, time: 2014-06-09 16:22:11.666474, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4c2) 2014-06-09 16:22:11.666484 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4c2) v4 from osd.1 2014-06-09 16:22:11.666487 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a28700 2014-06-09 16:22:11.666491 7f869af78700 5 -- op tracker -- , seq: 2418, time: 2014-06-09 16:22:11.666489, event: started, op: pg_info(1 pgs e243:1.4c2) 2014-06-09 16:22:11.666505 7f869af78700 5 -- op tracker -- , seq: 2418, time: 2014-06-09 16:22:11.666503, event: done, op: pg_info(1 pgs e243:1.4c2) 2014-06-09 16:22:11.666515 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666517 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666529 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666530 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666531 7f869af78700 20 osd.26 243 _dispatch 0x310dba40 pg_info(1 pgs e243:0.6ab) v4 2014-06-09 16:22:11.666535 7f869af78700 5 -- op tracker -- , seq: 2419, time: 2014-06-09 16:22:11.399895, event: header_read, op: pg_info(1 pgs e243:0.6ab) 2014-06-09 16:22:11.666543 7f869af78700 5 -- op tracker -- , seq: 2419, time: 2014-06-09 16:22:11.399896, event: throttled, op: pg_info(1 pgs e243:0.6ab) 2014-06-09 16:22:11.666549 7f869af78700 5 -- op tracker -- , seq: 2419, time: 2014-06-09 16:22:11.399905, event: all_read, op: pg_info(1 pgs e243:0.6ab) 2014-06-09 16:22:11.666555 7f869af78700 5 -- op tracker -- , seq: 2419, time: 2014-06-09 16:22:11.666528, event: dispatched, op: pg_info(1 pgs e243:0.6ab) 2014-06-09 16:22:11.666562 7f869af78700 5 -- op tracker -- , seq: 2419, time: 2014-06-09 16:22:11.666560, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6ab) 2014-06-09 16:22:11.666566 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6ab) v4 from osd.9 2014-06-09 16:22:11.666569 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310dba40 2014-06-09 16:22:11.666574 7f869af78700 5 -- op tracker -- , seq: 2419, time: 2014-06-09 16:22:11.666572, event: started, op: pg_info(1 pgs e243:0.6ab) 2014-06-09 16:22:11.666590 7f869af78700 5 -- op tracker -- , seq: 2419, time: 2014-06-09 16:22:11.666588, event: done, op: pg_info(1 pgs e243:0.6ab) 2014-06-09 16:22:11.666598 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666599 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666611 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666612 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666614 7f869af78700 20 osd.26 243 _dispatch 0x30d24380 pg_notify(3.1bd(10),3.750(10) epoch 243) v5 2014-06-09 16:22:11.666619 7f869af78700 5 -- op tracker -- , seq: 2420, time: 2014-06-09 16:22:11.408504, event: header_read, op: pg_notify(3.1bd(10),3.750(10) epoch 243) 2014-06-09 16:22:11.666632 7f869af78700 5 -- op tracker -- , seq: 2420, time: 2014-06-09 16:22:11.408505, event: throttled, op: pg_notify(3.1bd(10),3.750(10) epoch 243) 2014-06-09 16:22:11.666638 7f869af78700 5 -- op tracker -- , seq: 2420, time: 2014-06-09 16:22:11.408549, event: all_read, op: pg_notify(3.1bd(10),3.750(10) epoch 243) 2014-06-09 16:22:11.666645 7f869af78700 5 -- op tracker -- , seq: 2420, time: 2014-06-09 16:22:11.666610, event: dispatched, op: pg_notify(3.1bd(10),3.750(10) epoch 243) 2014-06-09 16:22:11.666652 7f869af78700 5 -- op tracker -- , seq: 2420, time: 2014-06-09 16:22:11.666650, event: waiting_for_osdmap, op: pg_notify(3.1bd(10),3.750(10) epoch 243) 2014-06-09 16:22:11.666657 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.666659 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d24380 2014-06-09 16:22:11.666664 7f869af78700 5 -- op tracker -- , seq: 2420, time: 2014-06-09 16:22:11.666662, event: started, op: pg_notify(3.1bd(10),3.750(10) epoch 243) 2014-06-09 16:22:11.666688 7f869af78700 5 -- op tracker -- , seq: 2420, time: 2014-06-09 16:22:11.666686, event: done, op: pg_notify(3.1bd(10),3.750(10) epoch 243) 2014-06-09 16:22:11.666695 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666697 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666713 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666715 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666716 7f869af78700 20 osd.26 243 _dispatch 0x310e3a40 pg_info(1 pgs e243:2.648) v4 2014-06-09 16:22:11.666721 7f869af78700 5 -- op tracker -- , seq: 2421, time: 2014-06-09 16:22:11.400159, event: header_read, op: pg_info(1 pgs e243:2.648) 2014-06-09 16:22:11.666731 7f869af78700 5 -- op tracker -- , seq: 2421, time: 2014-06-09 16:22:11.400160, event: throttled, op: pg_info(1 pgs e243:2.648) 2014-06-09 16:22:11.666737 7f869af78700 5 -- op tracker -- , seq: 2421, time: 2014-06-09 16:22:11.400170, event: all_read, op: pg_info(1 pgs e243:2.648) 2014-06-09 16:22:11.666743 7f869af78700 5 -- op tracker -- , seq: 2421, time: 2014-06-09 16:22:11.666713, event: dispatched, op: pg_info(1 pgs e243:2.648) 2014-06-09 16:22:11.666749 7f869af78700 5 -- op tracker -- , seq: 2421, time: 2014-06-09 16:22:11.666747, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.648) 2014-06-09 16:22:11.666753 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.648) v4 from osd.6 2014-06-09 16:22:11.666755 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e3a40 2014-06-09 16:22:11.666759 7f869af78700 5 -- op tracker -- , seq: 2421, time: 2014-06-09 16:22:11.666758, event: started, op: pg_info(1 pgs e243:2.648) 2014-06-09 16:22:11.666774 7f869af78700 5 -- op tracker -- , seq: 2421, time: 2014-06-09 16:22:11.666771, event: done, op: pg_info(1 pgs e243:2.648) 2014-06-09 16:22:11.666781 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666783 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666795 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666796 7f869af78700 20 osd.26 243 _dispatch 0x30b8c000 pg_info(1 pgs e243:2.37) v4 2014-06-09 16:22:11.666800 7f869af78700 5 -- op tracker -- , seq: 2422, time: 2014-06-09 16:22:11.371149, event: header_read, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.666806 7f869af78700 5 -- op tracker -- , seq: 2422, time: 2014-06-09 16:22:11.371149, event: throttled, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.666812 7f869af78700 5 -- op tracker -- , seq: 2422, time: 2014-06-09 16:22:11.371160, event: all_read, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.666820 7f869af78700 5 -- op tracker -- , seq: 2422, time: 2014-06-09 16:22:11.666793, event: dispatched, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.666827 7f869af78700 5 -- op tracker -- , seq: 2422, time: 2014-06-09 16:22:11.666825, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.666832 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.37) v4 from osd.0 2014-06-09 16:22:11.666835 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8c000 2014-06-09 16:22:11.666839 7f869af78700 5 -- op tracker -- , seq: 2422, time: 2014-06-09 16:22:11.666837, event: started, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.666854 7f869af78700 5 -- op tracker -- , seq: 2422, time: 2014-06-09 16:22:11.666852, event: done, op: pg_info(1 pgs e243:2.37) 2014-06-09 16:22:11.666861 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666863 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666875 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666876 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666878 7f869af78700 20 osd.26 243 _dispatch 0x310e8700 pg_info(1 pgs e243:3.38c) v4 2014-06-09 16:22:11.666883 7f869af78700 5 -- op tracker -- , seq: 2423, time: 2014-06-09 16:22:11.419629, event: header_read, op: pg_info(1 pgs e243:3.38c) 2014-06-09 16:22:11.666895 7f869af78700 5 -- op tracker -- , seq: 2423, time: 2014-06-09 16:22:11.419629, event: throttled, op: pg_info(1 pgs e243:3.38c) 2014-06-09 16:22:11.666901 7f869af78700 5 -- op tracker -- , seq: 2423, time: 2014-06-09 16:22:11.419641, event: all_read, op: pg_info(1 pgs e243:3.38c) 2014-06-09 16:22:11.666906 7f869af78700 5 -- op tracker -- , seq: 2423, time: 2014-06-09 16:22:11.666874, event: dispatched, op: pg_info(1 pgs e243:3.38c) 2014-06-09 16:22:11.666912 7f869af78700 5 -- op tracker -- , seq: 2423, time: 2014-06-09 16:22:11.666910, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.38c) 2014-06-09 16:22:11.666916 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.38c) v4 from osd.5 2014-06-09 16:22:11.666918 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e8700 2014-06-09 16:22:11.666923 7f869af78700 5 -- op tracker -- , seq: 2423, time: 2014-06-09 16:22:11.666921, event: started, op: pg_info(1 pgs e243:3.38c) 2014-06-09 16:22:11.666944 7f869af78700 5 -- op tracker -- , seq: 2423, time: 2014-06-09 16:22:11.666942, event: done, op: pg_info(1 pgs e243:3.38c) 2014-06-09 16:22:11.666951 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666963 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.666964 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.666965 7f869af78700 20 osd.26 243 _dispatch 0x30b8da40 pg_info(1 pgs e243:3.712(4)) v4 2014-06-09 16:22:11.666969 7f869af78700 5 -- op tracker -- , seq: 2424, time: 2014-06-09 16:22:11.266072, event: header_read, op: pg_info(1 pgs e243:3.712(4)) 2014-06-09 16:22:11.666979 7f869af78700 5 -- op tracker -- , seq: 2424, time: 2014-06-09 16:22:11.266072, event: throttled, op: pg_info(1 pgs e243:3.712(4)) 2014-06-09 16:22:11.666988 7f869af78700 5 -- op tracker -- , seq: 2424, time: 2014-06-09 16:22:11.266081, event: all_read, op: pg_info(1 pgs e243:3.712(4)) 2014-06-09 16:22:11.666996 7f869af78700 5 -- op tracker -- , seq: 2424, time: 2014-06-09 16:22:11.666962, event: dispatched, op: pg_info(1 pgs e243:3.712(4)) 2014-06-09 16:22:11.667002 7f869af78700 5 -- op tracker -- , seq: 2424, time: 2014-06-09 16:22:11.667000, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.712(4)) 2014-06-09 16:22:11.667007 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.712(4)) v4 from osd.2 2014-06-09 16:22:11.667010 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8da40 2014-06-09 16:22:11.667015 7f869af78700 5 -- op tracker -- , seq: 2424, time: 2014-06-09 16:22:11.667013, event: started, op: pg_info(1 pgs e243:3.712(4)) 2014-06-09 16:22:11.667031 7f869af78700 5 -- op tracker -- , seq: 2424, time: 2014-06-09 16:22:11.667029, event: done, op: pg_info(1 pgs e243:3.712(4)) 2014-06-09 16:22:11.667039 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.667041 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.667056 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.667057 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.667058 7f869af78700 20 osd.26 243 _dispatch 0x30951800 pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.667064 7f869af78700 5 -- op tracker -- , seq: 2425, time: 2014-06-09 16:22:11.353203, event: header_read, op: pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.667071 7f869af78700 5 -- op tracker -- , seq: 2425, time: 2014-06-09 16:22:11.353203, event: throttled, op: pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.667078 7f869af78700 5 -- op tracker -- , seq: 2425, time: 2014-06-09 16:22:11.353217, event: all_read, op: pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.667085 7f869af78700 5 -- op tracker -- , seq: 2425, time: 2014-06-09 16:22:11.667056, event: dispatched, op: pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.667093 7f869af78700 5 -- op tracker -- , seq: 2425, time: 2014-06-09 16:22:11.667090, event: waiting_for_osdmap, op: pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.667098 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.667102 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30951800 2014-06-09 16:22:11.667107 7f869af78700 5 -- op tracker -- , seq: 2425, time: 2014-06-09 16:22:11.667104, event: started, op: pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.667118 7f869af78700 5 -- op tracker -- , seq: 2425, time: 2014-06-09 16:22:11.667116, event: done, op: pg_log(2.778 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.667124 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.667126 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.667138 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.667139 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.667140 7f869af78700 20 osd.26 243 _dispatch 0x30bd88c0 pg_info(1 pgs e243:3.62a) v4 2014-06-09 16:22:11.667145 7f869af78700 5 -- op tracker -- , seq: 2426, time: 2014-06-09 16:22:11.390549, event: header_read, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.667157 7f869af78700 5 -- op tracker -- , seq: 2426, time: 2014-06-09 16:22:11.390550, event: throttled, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.667163 7f869af78700 5 -- op tracker -- , seq: 2426, time: 2014-06-09 16:22:11.390559, event: all_read, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.667168 7f869af78700 5 -- op tracker -- , seq: 2426, time: 2014-06-09 16:22:11.667137, event: dispatched, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.667174 7f869af78700 5 -- op tracker -- , seq: 2426, time: 2014-06-09 16:22:11.667172, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.667178 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.62a) v4 from osd.7 2014-06-09 16:22:11.667181 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd88c0 2014-06-09 16:22:11.667186 7f869af78700 5 -- op tracker -- , seq: 2426, time: 2014-06-09 16:22:11.667184, event: started, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.667204 7f869af78700 5 -- op tracker -- , seq: 2426, time: 2014-06-09 16:22:11.667201, event: done, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.667210 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.667212 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.667224 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.667225 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.667226 7f869af78700 20 osd.26 243 _dispatch 0x2eef6380 pg_notify(1.4c3(23) epoch 243) v5 2014-06-09 16:22:11.667231 7f869af78700 5 -- op tracker -- , seq: 2427, time: 2014-06-09 16:22:11.437797, event: header_read, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.667240 7f869af78700 5 -- op tracker -- , seq: 2427, time: 2014-06-09 16:22:11.437798, event: throttled, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.667247 7f869af78700 5 -- op tracker -- , seq: 2427, time: 2014-06-09 16:22:11.437833, event: all_read, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.667259 7f869af78700 5 -- op tracker -- , seq: 2427, time: 2014-06-09 16:22:11.667224, event: dispatched, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.667265 7f869af78700 5 -- op tracker -- , seq: 2427, time: 2014-06-09 16:22:11.667264, event: waiting_for_osdmap, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.667269 7f869af78700 7 osd.26 243 handle_pg_notify from osd.3 2014-06-09 16:22:11.667271 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef6380 2014-06-09 16:22:11.670780 7f869af78700 5 -- op tracker -- , seq: 2427, time: 2014-06-09 16:22:11.670777, event: started, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.670799 7f8682ae5700 5 -- op tracker -- , seq: 2428, time: 2014-06-09 16:22:11.670546, event: header_read, op: osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.670819 7f8682ae5700 5 -- op tracker -- , seq: 2428, time: 2014-06-09 16:22:11.670547, event: throttled, op: osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.670831 7f8682ae5700 5 -- op tracker -- , seq: 2428, time: 2014-06-09 16:22:11.670556, event: all_read, op: osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.670841 7f8682ae5700 5 -- op tracker -- , seq: 2428, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.670848 7f869af78700 5 -- op tracker -- , seq: 2427, time: 2014-06-09 16:22:11.670801, event: done, op: pg_notify(1.4c3(23) epoch 243) 2014-06-09 16:22:11.670848 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.670857 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.670855 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.670859 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.670862 7f8682ae5700 15 osd.26 243 enqueue_op 0x257c4100 prio 196 cost 0 latency 0.000316 osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.670876 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.670877 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.670877 7f869af78700 20 osd.26 243 _dispatch 0x1ac95880 pg_info(1 pgs e243:3.464) v4 2014-06-09 16:22:11.670880 7f869af78700 5 -- op tracker -- , seq: 2429, time: 2014-06-09 16:22:11.312943, event: header_read, op: pg_info(1 pgs e243:3.464) 2014-06-09 16:22:11.670886 7f869af78700 5 -- op tracker -- , seq: 2429, time: 2014-06-09 16:22:11.312943, event: throttled, op: pg_info(1 pgs e243:3.464) 2014-06-09 16:22:11.670890 7f869af78700 5 -- op tracker -- , seq: 2429, time: 2014-06-09 16:22:11.312951, event: all_read, op: pg_info(1 pgs e243:3.464) 2014-06-09 16:22:11.670894 7f869af78700 5 -- op tracker -- , seq: 2429, time: 2014-06-09 16:22:11.670875, event: dispatched, op: pg_info(1 pgs e243:3.464) 2014-06-09 16:22:11.670897 7f869af78700 5 -- op tracker -- , seq: 2429, time: 2014-06-09 16:22:11.670896, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.464) 2014-06-09 16:22:11.670900 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.464) v4 from osd.1 2014-06-09 16:22:11.670902 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac95880 2014-06-09 16:22:11.670905 7f869af78700 5 -- op tracker -- , seq: 2429, time: 2014-06-09 16:22:11.670904, event: started, op: pg_info(1 pgs e243:3.464) 2014-06-09 16:22:11.670916 7f869af78700 5 -- op tracker -- , seq: 2429, time: 2014-06-09 16:22:11.670915, event: done, op: pg_info(1 pgs e243:3.464) 2014-06-09 16:22:11.670921 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.670921 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.670930 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.670930 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.670931 7f869af78700 20 osd.26 243 _dispatch 0x310db880 pg_info(1 pgs e243:0.1a7) v4 2014-06-09 16:22:11.670934 7f869af78700 5 -- op tracker -- , seq: 2430, time: 2014-06-09 16:22:11.399910, event: header_read, op: pg_info(1 pgs e243:0.1a7) 2014-06-09 16:22:11.670938 7f869af78700 5 -- op tracker -- , seq: 2430, time: 2014-06-09 16:22:11.399910, event: throttled, op: pg_info(1 pgs e243:0.1a7) 2014-06-09 16:22:11.670941 7f869af78700 5 -- op tracker -- , seq: 2430, time: 2014-06-09 16:22:11.399924, event: all_read, op: pg_info(1 pgs e243:0.1a7) 2014-06-09 16:22:11.670944 7f869af78700 5 -- op tracker -- , seq: 2430, time: 2014-06-09 16:22:11.670929, event: dispatched, op: pg_info(1 pgs e243:0.1a7) 2014-06-09 16:22:11.670948 7f869af78700 5 -- op tracker -- , seq: 2430, time: 2014-06-09 16:22:11.670947, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1a7) 2014-06-09 16:22:11.670950 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1a7) v4 from osd.9 2014-06-09 16:22:11.670951 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310db880 2014-06-09 16:22:11.670954 7f869af78700 5 -- op tracker -- , seq: 2430, time: 2014-06-09 16:22:11.670953, event: started, op: pg_info(1 pgs e243:0.1a7) 2014-06-09 16:22:11.670963 7f869af78700 5 -- op tracker -- , seq: 2430, time: 2014-06-09 16:22:11.670962, event: done, op: pg_info(1 pgs e243:0.1a7) 2014-06-09 16:22:11.670967 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.670967 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.670976 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.670976 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.670977 7f869af78700 20 osd.26 243 _dispatch 0x310e88c0 pg_notify(0.13a(20),3.41b(10) epoch 243) v5 2014-06-09 16:22:11.670980 7f869af78700 5 -- op tracker -- , seq: 2431, time: 2014-06-09 16:22:11.415215, event: header_read, op: pg_notify(0.13a(20),3.41b(10) epoch 243) 2014-06-09 16:22:11.670984 7f869af78700 5 -- op tracker -- , seq: 2431, time: 2014-06-09 16:22:11.415216, event: throttled, op: pg_notify(0.13a(20),3.41b(10) epoch 243) 2014-06-09 16:22:11.670988 7f869af78700 5 -- op tracker -- , seq: 2431, time: 2014-06-09 16:22:11.415273, event: all_read, op: pg_notify(0.13a(20),3.41b(10) epoch 243) 2014-06-09 16:22:11.670991 7f869af78700 5 -- op tracker -- , seq: 2431, time: 2014-06-09 16:22:11.670976, event: dispatched, op: pg_notify(0.13a(20),3.41b(10) epoch 243) 2014-06-09 16:22:11.670995 7f869af78700 5 -- op tracker -- , seq: 2431, time: 2014-06-09 16:22:11.670994, event: waiting_for_osdmap, op: pg_notify(0.13a(20),3.41b(10) epoch 243) 2014-06-09 16:22:11.670997 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.670998 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e88c0 2014-06-09 16:22:11.671001 7f869af78700 5 -- op tracker -- , seq: 2431, time: 2014-06-09 16:22:11.671000, event: started, op: pg_notify(0.13a(20),3.41b(10) epoch 243) 2014-06-09 16:22:11.671015 7f869af78700 5 -- op tracker -- , seq: 2431, time: 2014-06-09 16:22:11.671014, event: done, op: pg_notify(0.13a(20),3.41b(10) epoch 243) 2014-06-09 16:22:11.671019 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671020 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671036 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671037 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671037 7f869af78700 20 osd.26 243 _dispatch 0x310e3880 pg_info(1 pgs e243:2.521) v4 2014-06-09 16:22:11.671040 7f869af78700 5 -- op tracker -- , seq: 2432, time: 2014-06-09 16:22:11.400176, event: header_read, op: pg_info(1 pgs e243:2.521) 2014-06-09 16:22:11.671044 7f869af78700 5 -- op tracker -- , seq: 2432, time: 2014-06-09 16:22:11.400176, event: throttled, op: pg_info(1 pgs e243:2.521) 2014-06-09 16:22:11.671049 7f869af78700 5 -- op tracker -- , seq: 2432, time: 2014-06-09 16:22:11.400186, event: all_read, op: pg_info(1 pgs e243:2.521) 2014-06-09 16:22:11.671052 7f869af78700 5 -- op tracker -- , seq: 2432, time: 2014-06-09 16:22:11.671036, event: dispatched, op: pg_info(1 pgs e243:2.521) 2014-06-09 16:22:11.671056 7f869af78700 5 -- op tracker -- , seq: 2432, time: 2014-06-09 16:22:11.671054, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.521) 2014-06-09 16:22:11.671058 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.521) v4 from osd.6 2014-06-09 16:22:11.671060 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e3880 2014-06-09 16:22:11.671062 7f869af78700 5 -- op tracker -- , seq: 2432, time: 2014-06-09 16:22:11.671061, event: started, op: pg_info(1 pgs e243:2.521) 2014-06-09 16:22:11.671071 7f869af78700 5 -- op tracker -- , seq: 2432, time: 2014-06-09 16:22:11.671070, event: done, op: pg_info(1 pgs e243:2.521) 2014-06-09 16:22:11.671075 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671076 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671083 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671084 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671085 7f869af78700 20 osd.26 243 _dispatch 0x1ae7c380 pg_info(1 pgs e243:1.5e) v4 2014-06-09 16:22:11.671087 7f869af78700 5 -- op tracker -- , seq: 2433, time: 2014-06-09 16:22:11.371164, event: header_read, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.671091 7f869af78700 5 -- op tracker -- , seq: 2433, time: 2014-06-09 16:22:11.371164, event: throttled, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.671094 7f869af78700 5 -- op tracker -- , seq: 2433, time: 2014-06-09 16:22:11.371177, event: all_read, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.671098 7f869af78700 5 -- op tracker -- , seq: 2433, time: 2014-06-09 16:22:11.671083, event: dispatched, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.671101 7f869af78700 5 -- op tracker -- , seq: 2433, time: 2014-06-09 16:22:11.671100, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.671104 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5e) v4 from osd.0 2014-06-09 16:22:11.671105 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ae7c380 2014-06-09 16:22:11.671108 7f869af78700 5 -- op tracker -- , seq: 2433, time: 2014-06-09 16:22:11.671107, event: started, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.671118 7f869af78700 5 -- op tracker -- , seq: 2433, time: 2014-06-09 16:22:11.671117, event: done, op: pg_info(1 pgs e243:1.5e) 2014-06-09 16:22:11.671122 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671123 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671130 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671131 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671131 7f869af78700 20 osd.26 243 _dispatch 0x30d25500 pg_info(1 pgs e243:3.4fa) v4 2014-06-09 16:22:11.671134 7f869af78700 5 -- op tracker -- , seq: 2434, time: 2014-06-09 16:22:11.419651, event: header_read, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.671138 7f869af78700 5 -- op tracker -- , seq: 2434, time: 2014-06-09 16:22:11.419652, event: throttled, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.671141 7f869af78700 5 -- op tracker -- , seq: 2434, time: 2014-06-09 16:22:11.419660, event: all_read, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.671144 7f869af78700 5 -- op tracker -- , seq: 2434, time: 2014-06-09 16:22:11.671130, event: dispatched, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.671148 7f869af78700 5 -- op tracker -- , seq: 2434, time: 2014-06-09 16:22:11.671147, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.671150 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4fa) v4 from osd.5 2014-06-09 16:22:11.671152 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d25500 2014-06-09 16:22:11.671154 7f869af78700 5 -- op tracker -- , seq: 2434, time: 2014-06-09 16:22:11.671153, event: started, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.671163 7f869af78700 5 -- op tracker -- , seq: 2434, time: 2014-06-09 16:22:11.671162, event: done, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.671167 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671168 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671176 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671177 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671177 7f869af78700 20 osd.26 243 _dispatch 0x305b5000 pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.671181 7f869af78700 5 -- op tracker -- , seq: 2435, time: 2014-06-09 16:22:11.267857, event: header_read, op: pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.671190 7f869af78700 5 -- op tracker -- , seq: 2435, time: 2014-06-09 16:22:11.267857, event: throttled, op: pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.671194 7f869af78700 5 -- op tracker -- , seq: 2435, time: 2014-06-09 16:22:11.268855, event: all_read, op: pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.671198 7f869af78700 5 -- op tracker -- , seq: 2435, time: 2014-06-09 16:22:11.671176, event: dispatched, op: pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.671202 7f869af78700 5 -- op tracker -- , seq: 2435, time: 2014-06-09 16:22:11.671200, event: waiting_for_osdmap, op: pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.671204 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) v4 from osd.2 2014-06-09 16:22:11.671207 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b5000 2014-06-09 16:22:11.671210 7f869af78700 5 -- op tracker -- , seq: 2435, time: 2014-06-09 16:22:11.671208, event: started, op: pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.671222 7f869af78700 5 -- op tracker -- , seq: 2435, time: 2014-06-09 16:22:11.671221, event: done, op: pg_log(3.15c epoch 243 log log((0'0,213'624], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.671227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671228 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671235 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671236 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671236 7f869af78700 20 osd.26 243 _dispatch 0x29ec4e00 pg_notify(0.4b3(23) epoch 243) v5 2014-06-09 16:22:11.671239 7f869af78700 5 -- op tracker -- , seq: 2436, time: 2014-06-09 16:22:11.356866, event: header_read, op: pg_notify(0.4b3(23) epoch 243) 2014-06-09 16:22:11.671243 7f869af78700 5 -- op tracker -- , seq: 2436, time: 2014-06-09 16:22:11.356866, event: throttled, op: pg_notify(0.4b3(23) epoch 243) 2014-06-09 16:22:11.671246 7f869af78700 5 -- op tracker -- , seq: 2436, time: 2014-06-09 16:22:11.356901, event: all_read, op: pg_notify(0.4b3(23) epoch 243) 2014-06-09 16:22:11.671249 7f869af78700 5 -- op tracker -- , seq: 2436, time: 2014-06-09 16:22:11.671235, event: dispatched, op: pg_notify(0.4b3(23) epoch 243) 2014-06-09 16:22:11.671253 7f869af78700 5 -- op tracker -- , seq: 2436, time: 2014-06-09 16:22:11.671252, event: waiting_for_osdmap, op: pg_notify(0.4b3(23) epoch 243) 2014-06-09 16:22:11.671259 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.671261 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec4e00 2014-06-09 16:22:11.671264 7f869af78700 5 -- op tracker -- , seq: 2436, time: 2014-06-09 16:22:11.671262, event: started, op: pg_notify(0.4b3(23) epoch 243) 2014-06-09 16:22:11.671273 7f869af78700 5 -- op tracker -- , seq: 2436, time: 2014-06-09 16:22:11.671272, event: done, op: pg_notify(0.4b3(23) epoch 243) 2014-06-09 16:22:11.671277 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671278 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671291 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671292 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671292 7f869af78700 20 osd.26 243 _dispatch 0x31041c00 pg_info(1 pgs e243:1.4db) v4 2014-06-09 16:22:11.671295 7f869af78700 5 -- op tracker -- , seq: 2437, time: 2014-06-09 16:22:11.390564, event: header_read, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.671299 7f869af78700 5 -- op tracker -- , seq: 2437, time: 2014-06-09 16:22:11.390564, event: throttled, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.671302 7f869af78700 5 -- op tracker -- , seq: 2437, time: 2014-06-09 16:22:11.390575, event: all_read, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.671305 7f869af78700 5 -- op tracker -- , seq: 2437, time: 2014-06-09 16:22:11.671291, event: dispatched, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.671309 7f869af78700 5 -- op tracker -- , seq: 2437, time: 2014-06-09 16:22:11.671308, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.671311 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4db) v4 from osd.7 2014-06-09 16:22:11.671313 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31041c00 2014-06-09 16:22:11.671315 7f869af78700 5 -- op tracker -- , seq: 2437, time: 2014-06-09 16:22:11.671314, event: started, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.671323 7f869af78700 5 -- op tracker -- , seq: 2437, time: 2014-06-09 16:22:11.671322, event: done, op: pg_info(1 pgs e243:1.4db) 2014-06-09 16:22:11.671327 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671328 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671335 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671335 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671336 7f869af78700 20 osd.26 243 _dispatch 0x2eef61c0 pg_info(1 pgs e243:0.7d) v4 2014-06-09 16:22:11.671339 7f869af78700 5 -- op tracker -- , seq: 2438, time: 2014-06-09 16:22:11.438226, event: header_read, op: pg_info(1 pgs e243:0.7d) 2014-06-09 16:22:11.671343 7f869af78700 5 -- op tracker -- , seq: 2438, time: 2014-06-09 16:22:11.438227, event: throttled, op: pg_info(1 pgs e243:0.7d) 2014-06-09 16:22:11.671348 7f869af78700 5 -- op tracker -- , seq: 2438, time: 2014-06-09 16:22:11.438240, event: all_read, op: pg_info(1 pgs e243:0.7d) 2014-06-09 16:22:11.671351 7f869af78700 5 -- op tracker -- , seq: 2438, time: 2014-06-09 16:22:11.671335, event: dispatched, op: pg_info(1 pgs e243:0.7d) 2014-06-09 16:22:11.671356 7f869af78700 5 -- op tracker -- , seq: 2438, time: 2014-06-09 16:22:11.671355, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.7d) 2014-06-09 16:22:11.671358 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.7d) v4 from osd.3 2014-06-09 16:22:11.671360 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef61c0 2014-06-09 16:22:11.671362 7f869af78700 5 -- op tracker -- , seq: 2438, time: 2014-06-09 16:22:11.671361, event: started, op: pg_info(1 pgs e243:0.7d) 2014-06-09 16:22:11.671370 7f869af78700 5 -- op tracker -- , seq: 2438, time: 2014-06-09 16:22:11.671369, event: done, op: pg_info(1 pgs e243:0.7d) 2014-06-09 16:22:11.671374 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671375 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671383 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671384 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671384 7f869af78700 20 osd.26 243 _dispatch 0x2fef4000 pg_info(1 pgs e243:0.1fb) v4 2014-06-09 16:22:11.671387 7f869af78700 5 -- op tracker -- , seq: 2439, time: 2014-06-09 16:22:11.312959, event: header_read, op: pg_info(1 pgs e243:0.1fb) 2014-06-09 16:22:11.671391 7f869af78700 5 -- op tracker -- , seq: 2439, time: 2014-06-09 16:22:11.312960, event: throttled, op: pg_info(1 pgs e243:0.1fb) 2014-06-09 16:22:11.671394 7f869af78700 5 -- op tracker -- , seq: 2439, time: 2014-06-09 16:22:11.312968, event: all_read, op: pg_info(1 pgs e243:0.1fb) 2014-06-09 16:22:11.671397 7f869af78700 5 -- op tracker -- , seq: 2439, time: 2014-06-09 16:22:11.671383, event: dispatched, op: pg_info(1 pgs e243:0.1fb) 2014-06-09 16:22:11.671401 7f869af78700 5 -- op tracker -- , seq: 2439, time: 2014-06-09 16:22:11.671400, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1fb) 2014-06-09 16:22:11.671403 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1fb) v4 from osd.1 2014-06-09 16:22:11.671404 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4000 2014-06-09 16:22:11.671407 7f869af78700 5 -- op tracker -- , seq: 2439, time: 2014-06-09 16:22:11.671406, event: started, op: pg_info(1 pgs e243:0.1fb) 2014-06-09 16:22:11.671415 7f869af78700 5 -- op tracker -- , seq: 2439, time: 2014-06-09 16:22:11.671414, event: done, op: pg_info(1 pgs e243:0.1fb) 2014-06-09 16:22:11.671419 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671426 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671428 7f869af78700 20 osd.26 243 _dispatch 0x310db6c0 pg_info(1 pgs e243:3.19b) v4 2014-06-09 16:22:11.671430 7f869af78700 5 -- op tracker -- , seq: 2440, time: 2014-06-09 16:22:11.404774, event: header_read, op: pg_info(1 pgs e243:3.19b) 2014-06-09 16:22:11.671434 7f869af78700 5 -- op tracker -- , seq: 2440, time: 2014-06-09 16:22:11.404774, event: throttled, op: pg_info(1 pgs e243:3.19b) 2014-06-09 16:22:11.671437 7f869af78700 5 -- op tracker -- , seq: 2440, time: 2014-06-09 16:22:11.404788, event: all_read, op: pg_info(1 pgs e243:3.19b) 2014-06-09 16:22:11.671440 7f869af78700 5 -- op tracker -- , seq: 2440, time: 2014-06-09 16:22:11.671426, event: dispatched, op: pg_info(1 pgs e243:3.19b) 2014-06-09 16:22:11.671444 7f869af78700 5 -- op tracker -- , seq: 2440, time: 2014-06-09 16:22:11.671443, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.19b) 2014-06-09 16:22:11.671446 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.19b) v4 from osd.9 2014-06-09 16:22:11.671448 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310db6c0 2014-06-09 16:22:11.671450 7f869af78700 5 -- op tracker -- , seq: 2440, time: 2014-06-09 16:22:11.671449, event: started, op: pg_info(1 pgs e243:3.19b) 2014-06-09 16:22:11.671461 7f869af78700 5 -- op tracker -- , seq: 2440, time: 2014-06-09 16:22:11.671460, event: done, op: pg_info(1 pgs e243:3.19b) 2014-06-09 16:22:11.671465 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671466 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671473 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671474 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671475 7f869af78700 20 osd.26 243 _dispatch 0x3104a8c0 pg_notify(3.1ff(10),2.1fc(20) epoch 243) v5 2014-06-09 16:22:11.671478 7f869af78700 5 -- op tracker -- , seq: 2441, time: 2014-06-09 16:22:11.415285, event: header_read, op: pg_notify(3.1ff(10),2.1fc(20) epoch 243) 2014-06-09 16:22:11.671482 7f869af78700 5 -- op tracker -- , seq: 2441, time: 2014-06-09 16:22:11.415286, event: throttled, op: pg_notify(3.1ff(10),2.1fc(20) epoch 243) 2014-06-09 16:22:11.671485 7f869af78700 5 -- op tracker -- , seq: 2441, time: 2014-06-09 16:22:11.415327, event: all_read, op: pg_notify(3.1ff(10),2.1fc(20) epoch 243) 2014-06-09 16:22:11.671489 7f869af78700 5 -- op tracker -- , seq: 2441, time: 2014-06-09 16:22:11.671473, event: dispatched, op: pg_notify(3.1ff(10),2.1fc(20) epoch 243) 2014-06-09 16:22:11.671494 7f869af78700 5 -- op tracker -- , seq: 2441, time: 2014-06-09 16:22:11.671493, event: waiting_for_osdmap, op: pg_notify(3.1ff(10),2.1fc(20) epoch 243) 2014-06-09 16:22:11.671496 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.671498 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104a8c0 2014-06-09 16:22:11.671500 7f869af78700 5 -- op tracker -- , seq: 2441, time: 2014-06-09 16:22:11.671499, event: started, op: pg_notify(3.1ff(10),2.1fc(20) epoch 243) 2014-06-09 16:22:11.671514 7f869af78700 5 -- op tracker -- , seq: 2441, time: 2014-06-09 16:22:11.671513, event: done, op: pg_notify(3.1ff(10),2.1fc(20) epoch 243) 2014-06-09 16:22:11.671518 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671519 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671533 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671533 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671534 7f869af78700 20 osd.26 243 _dispatch 0x310e36c0 pg_info(1 pgs e243:2.53d) v4 2014-06-09 16:22:11.671537 7f869af78700 5 -- op tracker -- , seq: 2442, time: 2014-06-09 16:22:11.400191, event: header_read, op: pg_info(1 pgs e243:2.53d) 2014-06-09 16:22:11.671540 7f869af78700 5 -- op tracker -- , seq: 2442, time: 2014-06-09 16:22:11.400191, event: throttled, op: pg_info(1 pgs e243:2.53d) 2014-06-09 16:22:11.671544 7f869af78700 5 -- op tracker -- , seq: 2442, time: 2014-06-09 16:22:11.400203, event: all_read, op: pg_info(1 pgs e243:2.53d) 2014-06-09 16:22:11.671552 7f869af78700 5 -- op tracker -- , seq: 2442, time: 2014-06-09 16:22:11.671532, event: dispatched, op: pg_info(1 pgs e243:2.53d) 2014-06-09 16:22:11.671556 7f869af78700 5 -- op tracker -- , seq: 2442, time: 2014-06-09 16:22:11.671555, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.53d) 2014-06-09 16:22:11.671558 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.53d) v4 from osd.6 2014-06-09 16:22:11.671560 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e36c0 2014-06-09 16:22:11.671562 7f869af78700 5 -- op tracker -- , seq: 2442, time: 2014-06-09 16:22:11.671561, event: started, op: pg_info(1 pgs e243:2.53d) 2014-06-09 16:22:11.671574 7f869af78700 5 -- op tracker -- , seq: 2442, time: 2014-06-09 16:22:11.671573, event: done, op: pg_info(1 pgs e243:2.53d) 2014-06-09 16:22:11.671579 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671580 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671586 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671587 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671588 7f869af78700 20 osd.26 243 _dispatch 0x30d256c0 pg_info(1 pgs e243:2.73) v4 2014-06-09 16:22:11.671590 7f869af78700 5 -- op tracker -- , seq: 2443, time: 2014-06-09 16:22:11.387809, event: header_read, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.671594 7f869af78700 5 -- op tracker -- , seq: 2443, time: 2014-06-09 16:22:11.387810, event: throttled, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.671598 7f869af78700 5 -- op tracker -- , seq: 2443, time: 2014-06-09 16:22:11.387821, event: all_read, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.671601 7f869af78700 5 -- op tracker -- , seq: 2443, time: 2014-06-09 16:22:11.671586, event: dispatched, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.671609 7f869af78700 5 -- op tracker -- , seq: 2443, time: 2014-06-09 16:22:11.671607, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.671611 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.73) v4 from osd.0 2014-06-09 16:22:11.671613 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d256c0 2014-06-09 16:22:11.671624 7f869af78700 5 -- op tracker -- , seq: 2443, time: 2014-06-09 16:22:11.671622, event: started, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.671638 7f869af78700 5 -- op tracker -- , seq: 2443, time: 2014-06-09 16:22:11.671637, event: done, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.671643 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671644 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671652 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671652 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671653 7f869af78700 20 osd.26 243 _dispatch 0x30d24000 pg_info(1 pgs e243:1.82) v4 2014-06-09 16:22:11.671656 7f869af78700 5 -- op tracker -- , seq: 2444, time: 2014-06-09 16:22:11.419672, event: header_read, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.671659 7f869af78700 5 -- op tracker -- , seq: 2444, time: 2014-06-09 16:22:11.419672, event: throttled, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.671663 7f869af78700 5 -- op tracker -- , seq: 2444, time: 2014-06-09 16:22:11.419682, event: all_read, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.671666 7f869af78700 5 -- op tracker -- , seq: 2444, time: 2014-06-09 16:22:11.671652, event: dispatched, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.671670 7f869af78700 5 -- op tracker -- , seq: 2444, time: 2014-06-09 16:22:11.671669, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.671672 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.82) v4 from osd.5 2014-06-09 16:22:11.671673 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d24000 2014-06-09 16:22:11.671676 7f869af78700 5 -- op tracker -- , seq: 2444, time: 2014-06-09 16:22:11.671675, event: started, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.671684 7f869af78700 5 -- op tracker -- , seq: 2444, time: 2014-06-09 16:22:11.671683, event: done, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.671688 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671696 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671697 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671697 7f869af78700 20 osd.26 243 _dispatch 0x305b4a00 pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.671701 7f869af78700 5 -- op tracker -- , seq: 2445, time: 2014-06-09 16:22:11.285992, event: header_read, op: pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671705 7f869af78700 5 -- op tracker -- , seq: 2445, time: 2014-06-09 16:22:11.285993, event: throttled, op: pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671713 7f869af78700 5 -- op tracker -- , seq: 2445, time: 2014-06-09 16:22:11.286006, event: all_read, op: pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671717 7f869af78700 5 -- op tracker -- , seq: 2445, time: 2014-06-09 16:22:11.671696, event: dispatched, op: pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671721 7f869af78700 5 -- op tracker -- , seq: 2445, time: 2014-06-09 16:22:11.671720, event: waiting_for_osdmap, op: pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671724 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.671726 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305b4a00 2014-06-09 16:22:11.671729 7f869af78700 5 -- op tracker -- , seq: 2445, time: 2014-06-09 16:22:11.671727, event: started, op: pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671736 7f869af78700 5 -- op tracker -- , seq: 2445, time: 2014-06-09 16:22:11.671734, event: done, op: pg_log(0.772 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671740 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671741 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671748 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671749 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671749 7f869af78700 20 osd.26 243 _dispatch 0x30a9aa00 pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.671753 7f869af78700 5 -- op tracker -- , seq: 2446, time: 2014-06-09 16:22:11.356908, event: header_read, op: pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671758 7f869af78700 5 -- op tracker -- , seq: 2446, time: 2014-06-09 16:22:11.356908, event: throttled, op: pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671762 7f869af78700 5 -- op tracker -- , seq: 2446, time: 2014-06-09 16:22:11.356921, event: all_read, op: pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671766 7f869af78700 5 -- op tracker -- , seq: 2446, time: 2014-06-09 16:22:11.671748, event: dispatched, op: pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671770 7f869af78700 5 -- op tracker -- , seq: 2446, time: 2014-06-09 16:22:11.671769, event: waiting_for_osdmap, op: pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671772 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.671775 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a9aa00 2014-06-09 16:22:11.671777 7f869af78700 5 -- op tracker -- , seq: 2446, time: 2014-06-09 16:22:11.671776, event: started, op: pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671784 7f869af78700 5 -- op tracker -- , seq: 2446, time: 2014-06-09 16:22:11.671783, event: done, op: pg_log(0.5fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.671788 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671789 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671796 7f869af78700 20 osd.26 243 _dispatch 0x31052a80 pg_info(1 pgs e243:1.603) v4 2014-06-09 16:22:11.671799 7f869af78700 5 -- op tracker -- , seq: 2447, time: 2014-06-09 16:22:11.390579, event: header_read, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.671803 7f869af78700 5 -- op tracker -- , seq: 2447, time: 2014-06-09 16:22:11.390579, event: throttled, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.671806 7f869af78700 5 -- op tracker -- , seq: 2447, time: 2014-06-09 16:22:11.390590, event: all_read, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.671813 7f869af78700 5 -- op tracker -- , seq: 2447, time: 2014-06-09 16:22:11.671795, event: dispatched, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.671817 7f869af78700 5 -- op tracker -- , seq: 2447, time: 2014-06-09 16:22:11.671815, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.671819 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.603) v4 from osd.7 2014-06-09 16:22:11.671821 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31052a80 2014-06-09 16:22:11.671823 7f869af78700 5 -- op tracker -- , seq: 2447, time: 2014-06-09 16:22:11.671822, event: started, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.671831 7f869af78700 5 -- op tracker -- , seq: 2447, time: 2014-06-09 16:22:11.671830, event: done, op: pg_info(1 pgs e243:1.603) 2014-06-09 16:22:11.671837 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671838 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671845 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671845 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671846 7f869af78700 20 osd.26 243 _dispatch 0x2eef6000 pg_info(1 pgs e243:3.583) v4 2014-06-09 16:22:11.671849 7f869af78700 5 -- op tracker -- , seq: 2448, time: 2014-06-09 16:22:11.438314, event: header_read, op: pg_info(1 pgs e243:3.583) 2014-06-09 16:22:11.671852 7f869af78700 5 -- op tracker -- , seq: 2448, time: 2014-06-09 16:22:11.438315, event: throttled, op: pg_info(1 pgs e243:3.583) 2014-06-09 16:22:11.671860 7f869af78700 5 -- op tracker -- , seq: 2448, time: 2014-06-09 16:22:11.438327, event: all_read, op: pg_info(1 pgs e243:3.583) 2014-06-09 16:22:11.671863 7f869af78700 5 -- op tracker -- , seq: 2448, time: 2014-06-09 16:22:11.671844, event: dispatched, op: pg_info(1 pgs e243:3.583) 2014-06-09 16:22:11.671867 7f869af78700 5 -- op tracker -- , seq: 2448, time: 2014-06-09 16:22:11.671866, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.583) 2014-06-09 16:22:11.671869 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.583) v4 from osd.3 2014-06-09 16:22:11.671871 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef6000 2014-06-09 16:22:11.671873 7f869af78700 5 -- op tracker -- , seq: 2448, time: 2014-06-09 16:22:11.671872, event: started, op: pg_info(1 pgs e243:3.583) 2014-06-09 16:22:11.671887 7f869af78700 5 -- op tracker -- , seq: 2448, time: 2014-06-09 16:22:11.671886, event: done, op: pg_info(1 pgs e243:3.583) 2014-06-09 16:22:11.671891 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671892 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671899 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671899 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671900 7f869af78700 20 osd.26 243 _dispatch 0x2fef41c0 pg_info(1 pgs e243:3.308) v4 2014-06-09 16:22:11.671903 7f869af78700 5 -- op tracker -- , seq: 2449, time: 2014-06-09 16:22:11.328688, event: header_read, op: pg_info(1 pgs e243:3.308) 2014-06-09 16:22:11.671906 7f869af78700 5 -- op tracker -- , seq: 2449, time: 2014-06-09 16:22:11.328688, event: throttled, op: pg_info(1 pgs e243:3.308) 2014-06-09 16:22:11.671910 7f869af78700 5 -- op tracker -- , seq: 2449, time: 2014-06-09 16:22:11.328701, event: all_read, op: pg_info(1 pgs e243:3.308) 2014-06-09 16:22:11.671913 7f869af78700 5 -- op tracker -- , seq: 2449, time: 2014-06-09 16:22:11.671898, event: dispatched, op: pg_info(1 pgs e243:3.308) 2014-06-09 16:22:11.671921 7f869af78700 5 -- op tracker -- , seq: 2449, time: 2014-06-09 16:22:11.671919, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.308) 2014-06-09 16:22:11.671923 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.308) v4 from osd.1 2014-06-09 16:22:11.671925 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef41c0 2014-06-09 16:22:11.671927 7f869af78700 5 -- op tracker -- , seq: 2449, time: 2014-06-09 16:22:11.671926, event: started, op: pg_info(1 pgs e243:3.308) 2014-06-09 16:22:11.671936 7f869af78700 5 -- op tracker -- , seq: 2449, time: 2014-06-09 16:22:11.671935, event: done, op: pg_info(1 pgs e243:3.308) 2014-06-09 16:22:11.671940 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671941 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671949 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671950 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.671950 7f869af78700 20 osd.26 243 _dispatch 0x310db500 pg_notify(0.5ed(20) epoch 243) v5 2014-06-09 16:22:11.671953 7f869af78700 5 -- op tracker -- , seq: 2450, time: 2014-06-09 16:22:11.407929, event: header_read, op: pg_notify(0.5ed(20) epoch 243) 2014-06-09 16:22:11.671958 7f869af78700 5 -- op tracker -- , seq: 2450, time: 2014-06-09 16:22:11.407930, event: throttled, op: pg_notify(0.5ed(20) epoch 243) 2014-06-09 16:22:11.671962 7f869af78700 5 -- op tracker -- , seq: 2450, time: 2014-06-09 16:22:11.407960, event: all_read, op: pg_notify(0.5ed(20) epoch 243) 2014-06-09 16:22:11.671966 7f869af78700 5 -- op tracker -- , seq: 2450, time: 2014-06-09 16:22:11.671949, event: dispatched, op: pg_notify(0.5ed(20) epoch 243) 2014-06-09 16:22:11.671970 7f869af78700 5 -- op tracker -- , seq: 2450, time: 2014-06-09 16:22:11.671969, event: waiting_for_osdmap, op: pg_notify(0.5ed(20) epoch 243) 2014-06-09 16:22:11.671972 7f869af78700 7 osd.26 243 handle_pg_notify from osd.9 2014-06-09 16:22:11.671973 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310db500 2014-06-09 16:22:11.671976 7f869af78700 5 -- op tracker -- , seq: 2450, time: 2014-06-09 16:22:11.671974, event: started, op: pg_notify(0.5ed(20) epoch 243) 2014-06-09 16:22:11.671984 7f869af78700 5 -- op tracker -- , seq: 2450, time: 2014-06-09 16:22:11.671983, event: done, op: pg_notify(0.5ed(20) epoch 243) 2014-06-09 16:22:11.671990 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.671991 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672003 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672004 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672005 7f869af78700 20 osd.26 243 _dispatch 0x307dfe00 pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.672008 7f869af78700 5 -- op tracker -- , seq: 2451, time: 2014-06-09 16:22:11.415334, event: header_read, op: pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672013 7f869af78700 5 -- op tracker -- , seq: 2451, time: 2014-06-09 16:22:11.415335, event: throttled, op: pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672016 7f869af78700 5 -- op tracker -- , seq: 2451, time: 2014-06-09 16:22:11.415346, event: all_read, op: pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672020 7f869af78700 5 -- op tracker -- , seq: 2451, time: 2014-06-09 16:22:11.672003, event: dispatched, op: pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672024 7f869af78700 5 -- op tracker -- , seq: 2451, time: 2014-06-09 16:22:11.672022, event: waiting_for_osdmap, op: pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672031 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.672033 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307dfe00 2014-06-09 16:22:11.672036 7f869af78700 5 -- op tracker -- , seq: 2451, time: 2014-06-09 16:22:11.672035, event: started, op: pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672043 7f869af78700 5 -- op tracker -- , seq: 2451, time: 2014-06-09 16:22:11.672042, event: done, op: pg_log(0.4bd epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672053 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672054 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672061 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672061 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672062 7f869af78700 20 osd.26 243 _dispatch 0x310e3500 pg_info(1 pgs e243:3.7e4) v4 2014-06-09 16:22:11.672065 7f869af78700 5 -- op tracker -- , seq: 2452, time: 2014-06-09 16:22:11.400207, event: header_read, op: pg_info(1 pgs e243:3.7e4) 2014-06-09 16:22:11.672070 7f869af78700 5 -- op tracker -- , seq: 2452, time: 2014-06-09 16:22:11.400208, event: throttled, op: pg_info(1 pgs e243:3.7e4) 2014-06-09 16:22:11.672073 7f869af78700 5 -- op tracker -- , seq: 2452, time: 2014-06-09 16:22:11.400217, event: all_read, op: pg_info(1 pgs e243:3.7e4) 2014-06-09 16:22:11.672080 7f869af78700 5 -- op tracker -- , seq: 2452, time: 2014-06-09 16:22:11.672061, event: dispatched, op: pg_info(1 pgs e243:3.7e4) 2014-06-09 16:22:11.672084 7f869af78700 5 -- op tracker -- , seq: 2452, time: 2014-06-09 16:22:11.672083, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7e4) 2014-06-09 16:22:11.672086 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7e4) v4 from osd.6 2014-06-09 16:22:11.672088 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e3500 2014-06-09 16:22:11.672090 7f869af78700 5 -- op tracker -- , seq: 2452, time: 2014-06-09 16:22:11.672089, event: started, op: pg_info(1 pgs e243:3.7e4) 2014-06-09 16:22:11.672099 7f869af78700 5 -- op tracker -- , seq: 2452, time: 2014-06-09 16:22:11.672098, event: done, op: pg_info(1 pgs e243:3.7e4) 2014-06-09 16:22:11.672103 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672104 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672110 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672111 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672112 7f869af78700 20 osd.26 243 _dispatch 0x30d24540 pg_info(1 pgs e243:1.604) v4 2014-06-09 16:22:11.672114 7f869af78700 5 -- op tracker -- , seq: 2453, time: 2014-06-09 16:22:11.387829, event: header_read, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.672118 7f869af78700 5 -- op tracker -- , seq: 2453, time: 2014-06-09 16:22:11.387830, event: throttled, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.672127 7f869af78700 5 -- op tracker -- , seq: 2453, time: 2014-06-09 16:22:11.387840, event: all_read, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.672145 7f869af78700 5 -- op tracker -- , seq: 2453, time: 2014-06-09 16:22:11.672110, event: dispatched, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.672150 7f869af78700 5 -- op tracker -- , seq: 2453, time: 2014-06-09 16:22:11.672149, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.672154 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.604) v4 from osd.0 2014-06-09 16:22:11.672156 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d24540 2014-06-09 16:22:11.672159 7f869af78700 5 -- op tracker -- , seq: 2453, time: 2014-06-09 16:22:11.672158, event: started, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.672174 7f869af78700 5 -- op tracker -- , seq: 2453, time: 2014-06-09 16:22:11.672173, event: done, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.672181 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672183 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672191 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672192 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672193 7f869af78700 20 osd.26 243 _dispatch 0x3115b6c0 pg_info(1 pgs e243:3.31c) v4 2014-06-09 16:22:11.672196 7f869af78700 5 -- op tracker -- , seq: 2454, time: 2014-06-09 16:22:11.419688, event: header_read, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.672205 7f869af78700 5 -- op tracker -- , seq: 2454, time: 2014-06-09 16:22:11.419688, event: throttled, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.672209 7f869af78700 5 -- op tracker -- , seq: 2454, time: 2014-06-09 16:22:11.419698, event: all_read, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.672212 7f869af78700 5 -- op tracker -- , seq: 2454, time: 2014-06-09 16:22:11.672191, event: dispatched, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.672219 7f869af78700 5 -- op tracker -- , seq: 2454, time: 2014-06-09 16:22:11.672218, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.672221 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.31c) v4 from osd.5 2014-06-09 16:22:11.672223 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115b6c0 2014-06-09 16:22:11.672226 7f869af78700 5 -- op tracker -- , seq: 2454, time: 2014-06-09 16:22:11.672225, event: started, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.672235 7f869af78700 5 -- op tracker -- , seq: 2454, time: 2014-06-09 16:22:11.672234, event: done, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.672239 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672240 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672248 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672249 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672249 7f869af78700 20 osd.26 243 _dispatch 0x30727600 pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.672253 7f869af78700 5 -- op tracker -- , seq: 2455, time: 2014-06-09 16:22:11.286055, event: header_read, op: pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672257 7f869af78700 5 -- op tracker -- , seq: 2455, time: 2014-06-09 16:22:11.286056, event: throttled, op: pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672261 7f869af78700 5 -- op tracker -- , seq: 2455, time: 2014-06-09 16:22:11.286069, event: all_read, op: pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672264 7f869af78700 5 -- op tracker -- , seq: 2455, time: 2014-06-09 16:22:11.672248, event: dispatched, op: pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672272 7f869af78700 5 -- op tracker -- , seq: 2455, time: 2014-06-09 16:22:11.672270, event: waiting_for_osdmap, op: pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672275 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.672277 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30727600 2014-06-09 16:22:11.672280 7f869af78700 5 -- op tracker -- , seq: 2455, time: 2014-06-09 16:22:11.672278, event: started, op: pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672288 7f869af78700 5 -- op tracker -- , seq: 2455, time: 2014-06-09 16:22:11.672287, event: done, op: pg_log(1.774 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672293 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672294 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672301 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672302 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672303 7f869af78700 20 osd.26 243 _dispatch 0x29ec5dc0 pg_notify(2.5fb(20) epoch 243) v5 2014-06-09 16:22:11.672305 7f869af78700 5 -- op tracker -- , seq: 2456, time: 2014-06-09 16:22:11.357012, event: header_read, op: pg_notify(2.5fb(20) epoch 243) 2014-06-09 16:22:11.672315 7f869af78700 5 -- op tracker -- , seq: 2456, time: 2014-06-09 16:22:11.357013, event: throttled, op: pg_notify(2.5fb(20) epoch 243) 2014-06-09 16:22:11.672318 7f869af78700 5 -- op tracker -- , seq: 2456, time: 2014-06-09 16:22:11.357041, event: all_read, op: pg_notify(2.5fb(20) epoch 243) 2014-06-09 16:22:11.672321 7f869af78700 5 -- op tracker -- , seq: 2456, time: 2014-06-09 16:22:11.672301, event: dispatched, op: pg_notify(2.5fb(20) epoch 243) 2014-06-09 16:22:11.672325 7f869af78700 5 -- op tracker -- , seq: 2456, time: 2014-06-09 16:22:11.672324, event: waiting_for_osdmap, op: pg_notify(2.5fb(20) epoch 243) 2014-06-09 16:22:11.672327 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.672329 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec5dc0 2014-06-09 16:22:11.672331 7f869af78700 5 -- op tracker -- , seq: 2456, time: 2014-06-09 16:22:11.672330, event: started, op: pg_notify(2.5fb(20) epoch 243) 2014-06-09 16:22:11.672340 7f869af78700 5 -- op tracker -- , seq: 2456, time: 2014-06-09 16:22:11.672339, event: done, op: pg_notify(2.5fb(20) epoch 243) 2014-06-09 16:22:11.672344 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672345 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672358 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672359 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672359 7f869af78700 20 osd.26 243 _dispatch 0x310528c0 pg_info(1 pgs e243:3.9b) v4 2014-06-09 16:22:11.672362 7f869af78700 5 -- op tracker -- , seq: 2457, time: 2014-06-09 16:22:11.390594, event: header_read, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672366 7f869af78700 5 -- op tracker -- , seq: 2457, time: 2014-06-09 16:22:11.390594, event: throttled, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672371 7f869af78700 5 -- op tracker -- , seq: 2457, time: 2014-06-09 16:22:11.390612, event: all_read, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672374 7f869af78700 5 -- op tracker -- , seq: 2457, time: 2014-06-09 16:22:11.672358, event: dispatched, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672378 7f869af78700 5 -- op tracker -- , seq: 2457, time: 2014-06-09 16:22:11.672376, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672380 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.9b) v4 from osd.7 2014-06-09 16:22:11.672381 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310528c0 2014-06-09 16:22:11.672384 7f869af78700 5 -- op tracker -- , seq: 2457, time: 2014-06-09 16:22:11.672383, event: started, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672393 7f869af78700 5 -- op tracker -- , seq: 2457, time: 2014-06-09 16:22:11.672392, event: done, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672397 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672398 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672404 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672405 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672406 7f869af78700 20 osd.26 243 _dispatch 0x2eef7dc0 pg_info(1 pgs e243:1.373) v4 2014-06-09 16:22:11.672408 7f869af78700 5 -- op tracker -- , seq: 2458, time: 2014-06-09 16:22:11.438334, event: header_read, op: pg_info(1 pgs e243:1.373) 2014-06-09 16:22:11.672412 7f869af78700 5 -- op tracker -- , seq: 2458, time: 2014-06-09 16:22:11.438335, event: throttled, op: pg_info(1 pgs e243:1.373) 2014-06-09 16:22:11.672415 7f869af78700 5 -- op tracker -- , seq: 2458, time: 2014-06-09 16:22:11.438353, event: all_read, op: pg_info(1 pgs e243:1.373) 2014-06-09 16:22:11.672418 7f869af78700 5 -- op tracker -- , seq: 2458, time: 2014-06-09 16:22:11.672404, event: dispatched, op: pg_info(1 pgs e243:1.373) 2014-06-09 16:22:11.672422 7f869af78700 5 -- op tracker -- , seq: 2458, time: 2014-06-09 16:22:11.672421, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.373) 2014-06-09 16:22:11.672424 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.373) v4 from osd.3 2014-06-09 16:22:11.672425 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef7dc0 2014-06-09 16:22:11.672428 7f869af78700 5 -- op tracker -- , seq: 2458, time: 2014-06-09 16:22:11.672427, event: started, op: pg_info(1 pgs e243:1.373) 2014-06-09 16:22:11.672436 7f869af78700 5 -- op tracker -- , seq: 2458, time: 2014-06-09 16:22:11.672435, event: done, op: pg_info(1 pgs e243:1.373) 2014-06-09 16:22:11.672440 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672441 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672447 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672448 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672449 7f869af78700 20 osd.26 243 _dispatch 0x3044d340 pg_info(1 pgs e243:3.7d4) v4 2014-06-09 16:22:11.672451 7f869af78700 5 -- op tracker -- , seq: 2459, time: 2014-06-09 16:22:11.328721, event: header_read, op: pg_info(1 pgs e243:3.7d4) 2014-06-09 16:22:11.672455 7f869af78700 5 -- op tracker -- , seq: 2459, time: 2014-06-09 16:22:11.328722, event: throttled, op: pg_info(1 pgs e243:3.7d4) 2014-06-09 16:22:11.672458 7f869af78700 5 -- op tracker -- , seq: 2459, time: 2014-06-09 16:22:11.328736, event: all_read, op: pg_info(1 pgs e243:3.7d4) 2014-06-09 16:22:11.672461 7f869af78700 5 -- op tracker -- , seq: 2459, time: 2014-06-09 16:22:11.672447, event: dispatched, op: pg_info(1 pgs e243:3.7d4) 2014-06-09 16:22:11.672465 7f869af78700 5 -- op tracker -- , seq: 2459, time: 2014-06-09 16:22:11.672464, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7d4) 2014-06-09 16:22:11.672467 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7d4) v4 from osd.1 2014-06-09 16:22:11.672469 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044d340 2014-06-09 16:22:11.672471 7f869af78700 5 -- op tracker -- , seq: 2459, time: 2014-06-09 16:22:11.672470, event: started, op: pg_info(1 pgs e243:3.7d4) 2014-06-09 16:22:11.672480 7f869af78700 5 -- op tracker -- , seq: 2459, time: 2014-06-09 16:22:11.672479, event: done, op: pg_info(1 pgs e243:3.7d4) 2014-06-09 16:22:11.672484 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672485 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672491 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672492 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672493 7f869af78700 20 osd.26 243 _dispatch 0x310e8a80 pg_info(1 pgs e243:2.27) v4 2014-06-09 16:22:11.672495 7f869af78700 5 -- op tracker -- , seq: 2460, time: 2014-06-09 16:22:11.410770, event: header_read, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672499 7f869af78700 5 -- op tracker -- , seq: 2460, time: 2014-06-09 16:22:11.410771, event: throttled, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672502 7f869af78700 5 -- op tracker -- , seq: 2460, time: 2014-06-09 16:22:11.410785, event: all_read, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672505 7f869af78700 5 -- op tracker -- , seq: 2460, time: 2014-06-09 16:22:11.672491, event: dispatched, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672509 7f869af78700 5 -- op tracker -- , seq: 2460, time: 2014-06-09 16:22:11.672508, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672511 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.27) v4 from osd.9 2014-06-09 16:22:11.672513 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e8a80 2014-06-09 16:22:11.672515 7f869af78700 5 -- op tracker -- , seq: 2460, time: 2014-06-09 16:22:11.672514, event: started, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672523 7f869af78700 5 -- op tracker -- , seq: 2460, time: 2014-06-09 16:22:11.672522, event: done, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672527 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672528 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672535 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672536 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672537 7f869af78700 20 osd.26 243 _dispatch 0x306e7200 pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.672540 7f869af78700 5 -- op tracker -- , seq: 2461, time: 2014-06-09 16:22:11.415351, event: header_read, op: pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672546 7f869af78700 5 -- op tracker -- , seq: 2461, time: 2014-06-09 16:22:11.415352, event: throttled, op: pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672550 7f869af78700 5 -- op tracker -- , seq: 2461, time: 2014-06-09 16:22:11.415373, event: all_read, op: pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672554 7f869af78700 5 -- op tracker -- , seq: 2461, time: 2014-06-09 16:22:11.672535, event: dispatched, op: pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672558 7f869af78700 5 -- op tracker -- , seq: 2461, time: 2014-06-09 16:22:11.672557, event: waiting_for_osdmap, op: pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672561 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.672563 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306e7200 2014-06-09 16:22:11.672566 7f869af78700 5 -- op tracker -- , seq: 2461, time: 2014-06-09 16:22:11.672564, event: started, op: pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672573 7f869af78700 5 -- op tracker -- , seq: 2461, time: 2014-06-09 16:22:11.672571, event: done, op: pg_log(2.566 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672578 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672579 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672587 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672588 7f869af78700 20 osd.26 243 _dispatch 0x310e3340 pg_info(1 pgs e243:3.7f) v4 2014-06-09 16:22:11.672591 7f869af78700 5 -- op tracker -- , seq: 2462, time: 2014-06-09 16:22:11.400221, event: header_read, op: pg_info(1 pgs e243:3.7f) 2014-06-09 16:22:11.672595 7f869af78700 5 -- op tracker -- , seq: 2462, time: 2014-06-09 16:22:11.400222, event: throttled, op: pg_info(1 pgs e243:3.7f) 2014-06-09 16:22:11.672598 7f869af78700 5 -- op tracker -- , seq: 2462, time: 2014-06-09 16:22:11.400231, event: all_read, op: pg_info(1 pgs e243:3.7f) 2014-06-09 16:22:11.672601 7f869af78700 5 -- op tracker -- , seq: 2462, time: 2014-06-09 16:22:11.672587, event: dispatched, op: pg_info(1 pgs e243:3.7f) 2014-06-09 16:22:11.672605 7f869af78700 5 -- op tracker -- , seq: 2462, time: 2014-06-09 16:22:11.672604, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7f) 2014-06-09 16:22:11.672607 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7f) v4 from osd.6 2014-06-09 16:22:11.672609 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e3340 2014-06-09 16:22:11.672611 7f869af78700 5 -- op tracker -- , seq: 2462, time: 2014-06-09 16:22:11.672610, event: started, op: pg_info(1 pgs e243:3.7f) 2014-06-09 16:22:11.672620 7f869af78700 5 -- op tracker -- , seq: 2462, time: 2014-06-09 16:22:11.672619, event: done, op: pg_info(1 pgs e243:3.7f) 2014-06-09 16:22:11.672624 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672625 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672632 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672632 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672633 7f869af78700 20 osd.26 243 _dispatch 0x30d241c0 pg_info(1 pgs e243:1.76) v4 2014-06-09 16:22:11.672636 7f869af78700 5 -- op tracker -- , seq: 2463, time: 2014-06-09 16:22:11.387844, event: header_read, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.672639 7f869af78700 5 -- op tracker -- , seq: 2463, time: 2014-06-09 16:22:11.387845, event: throttled, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.672647 7f869af78700 5 -- op tracker -- , seq: 2463, time: 2014-06-09 16:22:11.387855, event: all_read, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.672651 7f869af78700 5 -- op tracker -- , seq: 2463, time: 2014-06-09 16:22:11.672631, event: dispatched, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.672654 7f869af78700 5 -- op tracker -- , seq: 2463, time: 2014-06-09 16:22:11.672653, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.672656 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.76) v4 from osd.0 2014-06-09 16:22:11.672658 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d241c0 2014-06-09 16:22:11.672660 7f869af78700 5 -- op tracker -- , seq: 2463, time: 2014-06-09 16:22:11.672659, event: started, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.672669 7f869af78700 5 -- op tracker -- , seq: 2463, time: 2014-06-09 16:22:11.672668, event: done, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.672673 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672674 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672681 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672682 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672682 7f869af78700 20 osd.26 243 _dispatch 0x3115b500 pg_info(1 pgs e243:3.4fa) v4 2014-06-09 16:22:11.672685 7f869af78700 5 -- op tracker -- , seq: 2464, time: 2014-06-09 16:22:11.419703, event: header_read, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.672689 7f869af78700 5 -- op tracker -- , seq: 2464, time: 2014-06-09 16:22:11.419703, event: throttled, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.672696 7f869af78700 5 -- op tracker -- , seq: 2464, time: 2014-06-09 16:22:11.419711, event: all_read, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.672700 7f869af78700 5 -- op tracker -- , seq: 2464, time: 2014-06-09 16:22:11.672681, event: dispatched, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.672703 7f869af78700 5 -- op tracker -- , seq: 2464, time: 2014-06-09 16:22:11.672702, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.672706 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4fa) v4 from osd.5 2014-06-09 16:22:11.672707 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115b500 2014-06-09 16:22:11.672710 7f869af78700 5 -- op tracker -- , seq: 2464, time: 2014-06-09 16:22:11.672708, event: started, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.672718 7f869af78700 5 -- op tracker -- , seq: 2464, time: 2014-06-09 16:22:11.672716, event: done, op: pg_info(1 pgs e243:3.4fa) 2014-06-09 16:22:11.672722 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672722 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672732 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672733 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672734 7f869af78700 20 osd.26 243 _dispatch 0x30726400 pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.672737 7f869af78700 5 -- op tracker -- , seq: 2465, time: 2014-06-09 16:22:11.286077, event: header_read, op: pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672743 7f869af78700 5 -- op tracker -- , seq: 2465, time: 2014-06-09 16:22:11.286077, event: throttled, op: pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672747 7f869af78700 5 -- op tracker -- , seq: 2465, time: 2014-06-09 16:22:11.286088, event: all_read, op: pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672751 7f869af78700 5 -- op tracker -- , seq: 2465, time: 2014-06-09 16:22:11.672732, event: dispatched, op: pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672756 7f869af78700 5 -- op tracker -- , seq: 2465, time: 2014-06-09 16:22:11.672754, event: waiting_for_osdmap, op: pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672758 7f869af78700 7 osd.26 243 handle_pg_log pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.672760 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30726400 2014-06-09 16:22:11.672763 7f869af78700 5 -- op tracker -- , seq: 2465, time: 2014-06-09 16:22:11.672762, event: started, op: pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672770 7f869af78700 5 -- op tracker -- , seq: 2465, time: 2014-06-09 16:22:11.672768, event: done, op: pg_log(2.4af epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.672774 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672774 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672781 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672782 7f869af78700 20 osd.26 243 _dispatch 0x29ec4700 pg_notify(1.779(20) epoch 243) v5 2014-06-09 16:22:11.672785 7f869af78700 5 -- op tracker -- , seq: 2466, time: 2014-06-09 16:22:11.369607, event: header_read, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.672788 7f869af78700 5 -- op tracker -- , seq: 2466, time: 2014-06-09 16:22:11.369608, event: throttled, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.672791 7f869af78700 5 -- op tracker -- , seq: 2466, time: 2014-06-09 16:22:11.369639, event: all_read, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.672795 7f869af78700 5 -- op tracker -- , seq: 2466, time: 2014-06-09 16:22:11.672780, event: dispatched, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.672798 7f869af78700 5 -- op tracker -- , seq: 2466, time: 2014-06-09 16:22:11.672797, event: waiting_for_osdmap, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.672802 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.672803 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec4700 2014-06-09 16:22:11.672806 7f869af78700 5 -- op tracker -- , seq: 2466, time: 2014-06-09 16:22:11.672805, event: started, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.672815 7f869af78700 5 -- op tracker -- , seq: 2466, time: 2014-06-09 16:22:11.672813, event: done, op: pg_notify(1.779(20) epoch 243) 2014-06-09 16:22:11.672819 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672819 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672829 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672829 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672830 7f869af78700 20 osd.26 243 _dispatch 0x31052700 pg_info(1 pgs e243:3.9b) v4 2014-06-09 16:22:11.672832 7f869af78700 5 -- op tracker -- , seq: 2467, time: 2014-06-09 16:22:11.390616, event: header_read, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672836 7f869af78700 5 -- op tracker -- , seq: 2467, time: 2014-06-09 16:22:11.390616, event: throttled, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672840 7f869af78700 5 -- op tracker -- , seq: 2467, time: 2014-06-09 16:22:11.390627, event: all_read, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672843 7f869af78700 5 -- op tracker -- , seq: 2467, time: 2014-06-09 16:22:11.672828, event: dispatched, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672846 7f869af78700 5 -- op tracker -- , seq: 2467, time: 2014-06-09 16:22:11.672845, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672848 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.9b) v4 from osd.7 2014-06-09 16:22:11.672850 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31052700 2014-06-09 16:22:11.672852 7f869af78700 5 -- op tracker -- , seq: 2467, time: 2014-06-09 16:22:11.672851, event: started, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672861 7f869af78700 5 -- op tracker -- , seq: 2467, time: 2014-06-09 16:22:11.672860, event: done, op: pg_info(1 pgs e243:3.9b) 2014-06-09 16:22:11.672864 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672865 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672872 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672873 7f869af78700 20 osd.26 243 _dispatch 0x2eef7c00 pg_info(1 pgs e243:3.4c4(2)) v4 2014-06-09 16:22:11.672876 7f869af78700 5 -- op tracker -- , seq: 2468, time: 2014-06-09 16:22:11.438390, event: header_read, op: pg_info(1 pgs e243:3.4c4(2)) 2014-06-09 16:22:11.672880 7f869af78700 5 -- op tracker -- , seq: 2468, time: 2014-06-09 16:22:11.438391, event: throttled, op: pg_info(1 pgs e243:3.4c4(2)) 2014-06-09 16:22:11.672883 7f869af78700 5 -- op tracker -- , seq: 2468, time: 2014-06-09 16:22:11.438404, event: all_read, op: pg_info(1 pgs e243:3.4c4(2)) 2014-06-09 16:22:11.672888 7f869af78700 5 -- op tracker -- , seq: 2468, time: 2014-06-09 16:22:11.672872, event: dispatched, op: pg_info(1 pgs e243:3.4c4(2)) 2014-06-09 16:22:11.672892 7f869af78700 5 -- op tracker -- , seq: 2468, time: 2014-06-09 16:22:11.672891, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c4(2)) 2014-06-09 16:22:11.672897 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c4(2)) v4 from osd.3 2014-06-09 16:22:11.672899 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef7c00 2014-06-09 16:22:11.672902 7f869af78700 5 -- op tracker -- , seq: 2468, time: 2014-06-09 16:22:11.672901, event: started, op: pg_info(1 pgs e243:3.4c4(2)) 2014-06-09 16:22:11.672916 7f869af78700 5 -- op tracker -- , seq: 2468, time: 2014-06-09 16:22:11.672915, event: done, op: pg_info(1 pgs e243:3.4c4(2)) 2014-06-09 16:22:11.672920 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672921 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672929 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672929 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672930 7f869af78700 20 osd.26 243 _dispatch 0x30bd9dc0 pg_info(1 pgs e243:2.312) v4 2014-06-09 16:22:11.672932 7f869af78700 5 -- op tracker -- , seq: 2469, time: 2014-06-09 16:22:11.368131, event: header_read, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.672936 7f869af78700 5 -- op tracker -- , seq: 2469, time: 2014-06-09 16:22:11.368132, event: throttled, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.672939 7f869af78700 5 -- op tracker -- , seq: 2469, time: 2014-06-09 16:22:11.368151, event: all_read, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.672942 7f869af78700 5 -- op tracker -- , seq: 2469, time: 2014-06-09 16:22:11.672928, event: dispatched, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.672946 7f869af78700 5 -- op tracker -- , seq: 2469, time: 2014-06-09 16:22:11.672945, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.672948 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.312) v4 from osd.1 2014-06-09 16:22:11.672950 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd9dc0 2014-06-09 16:22:11.672952 7f869af78700 5 -- op tracker -- , seq: 2469, time: 2014-06-09 16:22:11.672951, event: started, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.672960 7f869af78700 5 -- op tracker -- , seq: 2469, time: 2014-06-09 16:22:11.672959, event: done, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.672964 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672965 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672971 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.672972 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.672973 7f869af78700 20 osd.26 243 _dispatch 0x310db340 pg_info(1 pgs e243:2.27) v4 2014-06-09 16:22:11.672975 7f869af78700 5 -- op tracker -- , seq: 2470, time: 2014-06-09 16:22:11.410794, event: header_read, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672979 7f869af78700 5 -- op tracker -- , seq: 2470, time: 2014-06-09 16:22:11.410794, event: throttled, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672982 7f869af78700 5 -- op tracker -- , seq: 2470, time: 2014-06-09 16:22:11.410807, event: all_read, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672987 7f869af78700 5 -- op tracker -- , seq: 2470, time: 2014-06-09 16:22:11.672971, event: dispatched, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672991 7f869af78700 5 -- op tracker -- , seq: 2470, time: 2014-06-09 16:22:11.672990, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.672997 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.27) v4 from osd.9 2014-06-09 16:22:11.672998 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310db340 2014-06-09 16:22:11.673001 7f869af78700 5 -- op tracker -- , seq: 2470, time: 2014-06-09 16:22:11.673000, event: started, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.673009 7f869af78700 5 -- op tracker -- , seq: 2470, time: 2014-06-09 16:22:11.673008, event: done, op: pg_info(1 pgs e243:2.27) 2014-06-09 16:22:11.673013 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673014 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673021 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673022 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673022 7f869af78700 20 osd.26 243 _dispatch 0x3104a700 pg_info(1 pgs e243:2.185) v4 2014-06-09 16:22:11.673025 7f869af78700 5 -- op tracker -- , seq: 2471, time: 2014-06-09 16:22:11.415378, event: header_read, op: pg_info(1 pgs e243:2.185) 2014-06-09 16:22:11.673028 7f869af78700 5 -- op tracker -- , seq: 2471, time: 2014-06-09 16:22:11.415378, event: throttled, op: pg_info(1 pgs e243:2.185) 2014-06-09 16:22:11.673036 7f869af78700 5 -- op tracker -- , seq: 2471, time: 2014-06-09 16:22:11.415388, event: all_read, op: pg_info(1 pgs e243:2.185) 2014-06-09 16:22:11.673039 7f869af78700 5 -- op tracker -- , seq: 2471, time: 2014-06-09 16:22:11.673021, event: dispatched, op: pg_info(1 pgs e243:2.185) 2014-06-09 16:22:11.673043 7f869af78700 5 -- op tracker -- , seq: 2471, time: 2014-06-09 16:22:11.673042, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.185) 2014-06-09 16:22:11.673045 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.185) v4 from osd.8 2014-06-09 16:22:11.673047 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104a700 2014-06-09 16:22:11.673049 7f869af78700 5 -- op tracker -- , seq: 2471, time: 2014-06-09 16:22:11.673048, event: started, op: pg_info(1 pgs e243:2.185) 2014-06-09 16:22:11.673058 7f869af78700 5 -- op tracker -- , seq: 2471, time: 2014-06-09 16:22:11.673057, event: done, op: pg_info(1 pgs e243:2.185) 2014-06-09 16:22:11.673062 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673063 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673069 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673070 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673070 7f869af78700 20 osd.26 243 _dispatch 0x310e3180 pg_info(1 pgs e243:2.102) v4 2014-06-09 16:22:11.673073 7f869af78700 5 -- op tracker -- , seq: 2472, time: 2014-06-09 16:22:11.400236, event: header_read, op: pg_info(1 pgs e243:2.102) 2014-06-09 16:22:11.673077 7f869af78700 5 -- op tracker -- , seq: 2472, time: 2014-06-09 16:22:11.400236, event: throttled, op: pg_info(1 pgs e243:2.102) 2014-06-09 16:22:11.673080 7f869af78700 5 -- op tracker -- , seq: 2472, time: 2014-06-09 16:22:11.400246, event: all_read, op: pg_info(1 pgs e243:2.102) 2014-06-09 16:22:11.673083 7f869af78700 5 -- op tracker -- , seq: 2472, time: 2014-06-09 16:22:11.673069, event: dispatched, op: pg_info(1 pgs e243:2.102) 2014-06-09 16:22:11.673091 7f869af78700 5 -- op tracker -- , seq: 2472, time: 2014-06-09 16:22:11.673090, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.102) 2014-06-09 16:22:11.673096 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.102) v4 from osd.6 2014-06-09 16:22:11.673098 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e3180 2014-06-09 16:22:11.673100 7f869af78700 5 -- op tracker -- , seq: 2472, time: 2014-06-09 16:22:11.673099, event: started, op: pg_info(1 pgs e243:2.102) 2014-06-09 16:22:11.673109 7f869af78700 5 -- op tracker -- , seq: 2472, time: 2014-06-09 16:22:11.673108, event: done, op: pg_info(1 pgs e243:2.102) 2014-06-09 16:22:11.673113 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673114 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673121 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673121 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673122 7f869af78700 20 osd.26 243 _dispatch 0x29ec5180 pg_info(1 pgs e243:2.73) v4 2014-06-09 16:22:11.673125 7f869af78700 5 -- op tracker -- , seq: 2473, time: 2014-06-09 16:22:11.387859, event: header_read, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.673129 7f869af78700 5 -- op tracker -- , seq: 2473, time: 2014-06-09 16:22:11.387859, event: throttled, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.673132 7f869af78700 5 -- op tracker -- , seq: 2473, time: 2014-06-09 16:22:11.387869, event: all_read, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.673135 7f869af78700 5 -- op tracker -- , seq: 2473, time: 2014-06-09 16:22:11.673120, event: dispatched, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.673139 7f869af78700 5 -- op tracker -- , seq: 2473, time: 2014-06-09 16:22:11.673138, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.673145 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.73) v4 from osd.0 2014-06-09 16:22:11.673146 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec5180 2014-06-09 16:22:11.673149 7f869af78700 5 -- op tracker -- , seq: 2473, time: 2014-06-09 16:22:11.673148, event: started, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.673162 7f869af78700 5 -- op tracker -- , seq: 2473, time: 2014-06-09 16:22:11.673161, event: done, op: pg_info(1 pgs e243:2.73) 2014-06-09 16:22:11.673167 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673168 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673175 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673176 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673176 7f869af78700 20 osd.26 243 _dispatch 0x3115b340 pg_info(1 pgs e243:1.82) v4 2014-06-09 16:22:11.673179 7f869af78700 5 -- op tracker -- , seq: 2474, time: 2014-06-09 16:22:11.419715, event: header_read, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.673183 7f869af78700 5 -- op tracker -- , seq: 2474, time: 2014-06-09 16:22:11.419716, event: throttled, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.673186 7f869af78700 5 -- op tracker -- , seq: 2474, time: 2014-06-09 16:22:11.419724, event: all_read, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.673189 7f869af78700 5 -- op tracker -- , seq: 2474, time: 2014-06-09 16:22:11.673175, event: dispatched, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.673193 7f869af78700 5 -- op tracker -- , seq: 2474, time: 2014-06-09 16:22:11.673192, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.673195 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.82) v4 from osd.5 2014-06-09 16:22:11.673197 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115b340 2014-06-09 16:22:11.673199 7f869af78700 5 -- op tracker -- , seq: 2474, time: 2014-06-09 16:22:11.673198, event: started, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.673207 7f869af78700 5 -- op tracker -- , seq: 2474, time: 2014-06-09 16:22:11.673206, event: done, op: pg_info(1 pgs e243:1.82) 2014-06-09 16:22:11.673211 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673212 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673219 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673219 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673220 7f869af78700 20 osd.26 243 _dispatch 0x1205e540 pg_notify(3.4b3(12) epoch 243) v5 2014-06-09 16:22:11.673223 7f869af78700 5 -- op tracker -- , seq: 2475, time: 2014-06-09 16:22:11.286093, event: header_read, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.673226 7f869af78700 5 -- op tracker -- , seq: 2475, time: 2014-06-09 16:22:11.286093, event: throttled, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.673230 7f869af78700 5 -- op tracker -- , seq: 2475, time: 2014-06-09 16:22:11.286108, event: all_read, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.673233 7f869af78700 5 -- op tracker -- , seq: 2475, time: 2014-06-09 16:22:11.673218, event: dispatched, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.673236 7f869af78700 5 -- op tracker -- , seq: 2475, time: 2014-06-09 16:22:11.673235, event: waiting_for_osdmap, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.673238 7f869af78700 7 osd.26 243 handle_pg_notify from osd.2 2014-06-09 16:22:11.673239 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1205e540 2014-06-09 16:22:11.673242 7f869af78700 5 -- op tracker -- , seq: 2475, time: 2014-06-09 16:22:11.673241, event: started, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.673251 7f869af78700 5 -- op tracker -- , seq: 2475, time: 2014-06-09 16:22:11.673250, event: done, op: pg_notify(3.4b3(12) epoch 243) 2014-06-09 16:22:11.673255 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673256 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673266 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673267 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673267 7f869af78700 20 osd.26 243 _dispatch 0x30a9b000 pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.673271 7f869af78700 5 -- op tracker -- , seq: 2476, time: 2014-06-09 16:22:11.369729, event: header_read, op: pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.673279 7f869af78700 5 -- op tracker -- , seq: 2476, time: 2014-06-09 16:22:11.369729, event: throttled, op: pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.673283 7f869af78700 5 -- op tracker -- , seq: 2476, time: 2014-06-09 16:22:11.369749, event: all_read, op: pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.673287 7f869af78700 5 -- op tracker -- , seq: 2476, time: 2014-06-09 16:22:11.673266, event: dispatched, op: pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.673291 7f869af78700 5 -- op tracker -- , seq: 2476, time: 2014-06-09 16:22:11.673289, event: waiting_for_osdmap, op: pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.673293 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) v4 from osd.4 2014-06-09 16:22:11.673295 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a9b000 2014-06-09 16:22:11.673298 7f869af78700 5 -- op tracker -- , seq: 2476, time: 2014-06-09 16:22:11.673297, event: started, op: pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.673305 7f869af78700 5 -- op tracker -- , seq: 2476, time: 2014-06-09 16:22:11.673304, event: done, op: pg_log(3.55b epoch 243 log log((214'714,214'716], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.673309 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673310 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673316 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673317 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673318 7f869af78700 20 osd.26 243 _dispatch 0x31052540 pg_info(1 pgs e243:3.6) v4 2014-06-09 16:22:11.673320 7f869af78700 5 -- op tracker -- , seq: 2477, time: 2014-06-09 16:22:11.393543, event: header_read, op: pg_info(1 pgs e243:3.6) 2014-06-09 16:22:11.673324 7f869af78700 5 -- op tracker -- , seq: 2477, time: 2014-06-09 16:22:11.393544, event: throttled, op: pg_info(1 pgs e243:3.6) 2014-06-09 16:22:11.673331 7f869af78700 5 -- op tracker -- , seq: 2477, time: 2014-06-09 16:22:11.393566, event: all_read, op: pg_info(1 pgs e243:3.6) 2014-06-09 16:22:11.673335 7f869af78700 5 -- op tracker -- , seq: 2477, time: 2014-06-09 16:22:11.673316, event: dispatched, op: pg_info(1 pgs e243:3.6) 2014-06-09 16:22:11.673340 7f869af78700 5 -- op tracker -- , seq: 2477, time: 2014-06-09 16:22:11.673339, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6) 2014-06-09 16:22:11.673342 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6) v4 from osd.7 2014-06-09 16:22:11.673344 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31052540 2014-06-09 16:22:11.673346 7f869af78700 5 -- op tracker -- , seq: 2477, time: 2014-06-09 16:22:11.673345, event: started, op: pg_info(1 pgs e243:3.6) 2014-06-09 16:22:11.673355 7f869af78700 5 -- op tracker -- , seq: 2477, time: 2014-06-09 16:22:11.673354, event: done, op: pg_info(1 pgs e243:3.6) 2014-06-09 16:22:11.673359 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673360 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673367 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673368 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673368 7f869af78700 20 osd.26 243 _dispatch 0x2eef7a40 pg_info(1 pgs e243:0.198) v4 2014-06-09 16:22:11.673371 7f869af78700 5 -- op tracker -- , seq: 2478, time: 2014-06-09 16:22:11.440564, event: header_read, op: pg_info(1 pgs e243:0.198) 2014-06-09 16:22:11.673375 7f869af78700 5 -- op tracker -- , seq: 2478, time: 2014-06-09 16:22:11.440564, event: throttled, op: pg_info(1 pgs e243:0.198) 2014-06-09 16:22:11.673378 7f869af78700 5 -- op tracker -- , seq: 2478, time: 2014-06-09 16:22:11.440579, event: all_read, op: pg_info(1 pgs e243:0.198) 2014-06-09 16:22:11.673386 7f869af78700 5 -- op tracker -- , seq: 2478, time: 2014-06-09 16:22:11.673367, event: dispatched, op: pg_info(1 pgs e243:0.198) 2014-06-09 16:22:11.673389 7f869af78700 5 -- op tracker -- , seq: 2478, time: 2014-06-09 16:22:11.673388, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.198) 2014-06-09 16:22:11.673392 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.198) v4 from osd.3 2014-06-09 16:22:11.673393 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef7a40 2014-06-09 16:22:11.673396 7f869af78700 5 -- op tracker -- , seq: 2478, time: 2014-06-09 16:22:11.673395, event: started, op: pg_info(1 pgs e243:0.198) 2014-06-09 16:22:11.673404 7f869af78700 5 -- op tracker -- , seq: 2478, time: 2014-06-09 16:22:11.673403, event: done, op: pg_info(1 pgs e243:0.198) 2014-06-09 16:22:11.673408 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673409 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673415 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673416 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673417 7f869af78700 20 osd.26 243 _dispatch 0x30bd8e00 pg_info(1 pgs e243:2.312) v4 2014-06-09 16:22:11.673419 7f869af78700 5 -- op tracker -- , seq: 2479, time: 2014-06-09 16:22:11.368161, event: header_read, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.673423 7f869af78700 5 -- op tracker -- , seq: 2479, time: 2014-06-09 16:22:11.368162, event: throttled, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.673426 7f869af78700 5 -- op tracker -- , seq: 2479, time: 2014-06-09 16:22:11.368173, event: all_read, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.673431 7f869af78700 5 -- op tracker -- , seq: 2479, time: 2014-06-09 16:22:11.673415, event: dispatched, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.673435 7f869af78700 5 -- op tracker -- , seq: 2479, time: 2014-06-09 16:22:11.673434, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.673441 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.312) v4 from osd.1 2014-06-09 16:22:11.673442 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd8e00 2014-06-09 16:22:11.673445 7f869af78700 5 -- op tracker -- , seq: 2479, time: 2014-06-09 16:22:11.673444, event: started, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.673453 7f869af78700 5 -- op tracker -- , seq: 2479, time: 2014-06-09 16:22:11.673452, event: done, op: pg_info(1 pgs e243:2.312) 2014-06-09 16:22:11.673458 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673459 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673466 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673467 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673467 7f869af78700 20 osd.26 243 _dispatch 0x30a80600 pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.673471 7f869af78700 5 -- op tracker -- , seq: 2480, time: 2014-06-09 16:22:11.410812, event: header_read, op: pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673475 7f869af78700 5 -- op tracker -- , seq: 2480, time: 2014-06-09 16:22:11.410812, event: throttled, op: pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673479 7f869af78700 5 -- op tracker -- , seq: 2480, time: 2014-06-09 16:22:11.410833, event: all_read, op: pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673483 7f869af78700 5 -- op tracker -- , seq: 2480, time: 2014-06-09 16:22:11.673466, event: dispatched, op: pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673487 7f869af78700 5 -- op tracker -- , seq: 2480, time: 2014-06-09 16:22:11.673485, event: waiting_for_osdmap, op: pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673489 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.673491 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a80600 2014-06-09 16:22:11.673494 7f869af78700 5 -- op tracker -- , seq: 2480, time: 2014-06-09 16:22:11.673493, event: started, op: pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673503 7f869af78700 5 -- op tracker -- , seq: 2480, time: 2014-06-09 16:22:11.673501, event: done, op: pg_log(3.113 epoch 243 log log((213'650,227'654], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673507 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673508 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673514 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673515 7f869af78700 20 osd.26 243 _dispatch 0x3104a540 pg_info(1 pgs e243:2.442) v4 2014-06-09 16:22:11.673518 7f869af78700 5 -- op tracker -- , seq: 2481, time: 2014-06-09 16:22:11.415393, event: header_read, op: pg_info(1 pgs e243:2.442) 2014-06-09 16:22:11.673521 7f869af78700 5 -- op tracker -- , seq: 2481, time: 2014-06-09 16:22:11.415393, event: throttled, op: pg_info(1 pgs e243:2.442) 2014-06-09 16:22:11.673525 7f869af78700 5 -- op tracker -- , seq: 2481, time: 2014-06-09 16:22:11.415403, event: all_read, op: pg_info(1 pgs e243:2.442) 2014-06-09 16:22:11.673528 7f869af78700 5 -- op tracker -- , seq: 2481, time: 2014-06-09 16:22:11.673514, event: dispatched, op: pg_info(1 pgs e243:2.442) 2014-06-09 16:22:11.673531 7f869af78700 5 -- op tracker -- , seq: 2481, time: 2014-06-09 16:22:11.673530, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.442) 2014-06-09 16:22:11.673533 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.442) v4 from osd.8 2014-06-09 16:22:11.673535 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104a540 2014-06-09 16:22:11.673537 7f869af78700 5 -- op tracker -- , seq: 2481, time: 2014-06-09 16:22:11.673536, event: started, op: pg_info(1 pgs e243:2.442) 2014-06-09 16:22:11.673545 7f869af78700 5 -- op tracker -- , seq: 2481, time: 2014-06-09 16:22:11.673544, event: done, op: pg_info(1 pgs e243:2.442) 2014-06-09 16:22:11.673549 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673550 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673557 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673557 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673558 7f869af78700 20 osd.26 243 _dispatch 0x310e2fc0 pg_info(1 pgs e243:2.5e2) v4 2014-06-09 16:22:11.673560 7f869af78700 5 -- op tracker -- , seq: 2482, time: 2014-06-09 16:22:11.400251, event: header_read, op: pg_info(1 pgs e243:2.5e2) 2014-06-09 16:22:11.673564 7f869af78700 5 -- op tracker -- , seq: 2482, time: 2014-06-09 16:22:11.400251, event: throttled, op: pg_info(1 pgs e243:2.5e2) 2014-06-09 16:22:11.673567 7f869af78700 5 -- op tracker -- , seq: 2482, time: 2014-06-09 16:22:11.400260, event: all_read, op: pg_info(1 pgs e243:2.5e2) 2014-06-09 16:22:11.673571 7f869af78700 5 -- op tracker -- , seq: 2482, time: 2014-06-09 16:22:11.673556, event: dispatched, op: pg_info(1 pgs e243:2.5e2) 2014-06-09 16:22:11.673574 7f869af78700 5 -- op tracker -- , seq: 2482, time: 2014-06-09 16:22:11.673573, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5e2) 2014-06-09 16:22:11.673577 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5e2) v4 from osd.6 2014-06-09 16:22:11.673578 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e2fc0 2014-06-09 16:22:11.673581 7f869af78700 5 -- op tracker -- , seq: 2482, time: 2014-06-09 16:22:11.673580, event: started, op: pg_info(1 pgs e243:2.5e2) 2014-06-09 16:22:11.673589 7f869af78700 5 -- op tracker -- , seq: 2482, time: 2014-06-09 16:22:11.673588, event: done, op: pg_info(1 pgs e243:2.5e2) 2014-06-09 16:22:11.673592 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673593 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673600 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673600 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673601 7f869af78700 20 osd.26 243 _dispatch 0x29ec4000 pg_info(1 pgs e243:1.604) v4 2014-06-09 16:22:11.673604 7f869af78700 5 -- op tracker -- , seq: 2483, time: 2014-06-09 16:22:11.387874, event: header_read, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.673607 7f869af78700 5 -- op tracker -- , seq: 2483, time: 2014-06-09 16:22:11.387875, event: throttled, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.673610 7f869af78700 5 -- op tracker -- , seq: 2483, time: 2014-06-09 16:22:11.387885, event: all_read, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.673614 7f869af78700 5 -- op tracker -- , seq: 2483, time: 2014-06-09 16:22:11.673600, event: dispatched, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.673617 7f869af78700 5 -- op tracker -- , seq: 2483, time: 2014-06-09 16:22:11.673616, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.673620 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.604) v4 from osd.0 2014-06-09 16:22:11.673621 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec4000 2014-06-09 16:22:11.673624 7f869af78700 5 -- op tracker -- , seq: 2483, time: 2014-06-09 16:22:11.673623, event: started, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.673635 7f869af78700 5 -- op tracker -- , seq: 2483, time: 2014-06-09 16:22:11.673634, event: done, op: pg_info(1 pgs e243:1.604) 2014-06-09 16:22:11.673639 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673640 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673648 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673648 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673649 7f869af78700 20 osd.26 243 _dispatch 0x3115b180 pg_info(1 pgs e243:0.18b) v4 2014-06-09 16:22:11.673652 7f869af78700 5 -- op tracker -- , seq: 2484, time: 2014-06-09 16:22:11.419729, event: header_read, op: pg_info(1 pgs e243:0.18b) 2014-06-09 16:22:11.673659 7f869af78700 5 -- op tracker -- , seq: 2484, time: 2014-06-09 16:22:11.419730, event: throttled, op: pg_info(1 pgs e243:0.18b) 2014-06-09 16:22:11.673663 7f869af78700 5 -- op tracker -- , seq: 2484, time: 2014-06-09 16:22:11.419745, event: all_read, op: pg_info(1 pgs e243:0.18b) 2014-06-09 16:22:11.673666 7f869af78700 5 -- op tracker -- , seq: 2484, time: 2014-06-09 16:22:11.673648, event: dispatched, op: pg_info(1 pgs e243:0.18b) 2014-06-09 16:22:11.673670 7f869af78700 5 -- op tracker -- , seq: 2484, time: 2014-06-09 16:22:11.673669, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.18b) 2014-06-09 16:22:11.673672 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.18b) v4 from osd.5 2014-06-09 16:22:11.673674 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115b180 2014-06-09 16:22:11.673676 7f869af78700 5 -- op tracker -- , seq: 2484, time: 2014-06-09 16:22:11.673675, event: started, op: pg_info(1 pgs e243:0.18b) 2014-06-09 16:22:11.673684 7f869af78700 5 -- op tracker -- , seq: 2484, time: 2014-06-09 16:22:11.673683, event: done, op: pg_info(1 pgs e243:0.18b) 2014-06-09 16:22:11.673688 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673696 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673697 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673697 7f869af78700 20 osd.26 243 _dispatch 0x240eae00 pg_info(1 pgs e243:3.1e1(4)) v4 2014-06-09 16:22:11.673700 7f869af78700 5 -- op tracker -- , seq: 2485, time: 2014-06-09 16:22:11.286119, event: header_read, op: pg_info(1 pgs e243:3.1e1(4)) 2014-06-09 16:22:11.673704 7f869af78700 5 -- op tracker -- , seq: 2485, time: 2014-06-09 16:22:11.286119, event: throttled, op: pg_info(1 pgs e243:3.1e1(4)) 2014-06-09 16:22:11.673707 7f869af78700 5 -- op tracker -- , seq: 2485, time: 2014-06-09 16:22:11.286128, event: all_read, op: pg_info(1 pgs e243:3.1e1(4)) 2014-06-09 16:22:11.673711 7f869af78700 5 -- op tracker -- , seq: 2485, time: 2014-06-09 16:22:11.673696, event: dispatched, op: pg_info(1 pgs e243:3.1e1(4)) 2014-06-09 16:22:11.673715 7f869af78700 5 -- op tracker -- , seq: 2485, time: 2014-06-09 16:22:11.673714, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1e1(4)) 2014-06-09 16:22:11.673717 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1e1(4)) v4 from osd.2 2014-06-09 16:22:11.673719 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eae00 2014-06-09 16:22:11.673721 7f869af78700 5 -- op tracker -- , seq: 2485, time: 2014-06-09 16:22:11.673720, event: started, op: pg_info(1 pgs e243:3.1e1(4)) 2014-06-09 16:22:11.673730 7f869af78700 5 -- op tracker -- , seq: 2485, time: 2014-06-09 16:22:11.673729, event: done, op: pg_info(1 pgs e243:3.1e1(4)) 2014-06-09 16:22:11.673737 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673738 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673745 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673746 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673747 7f869af78700 20 osd.26 243 _dispatch 0x29ec4380 pg_notify(0.708(23) epoch 243) v5 2014-06-09 16:22:11.673749 7f869af78700 5 -- op tracker -- , seq: 2486, time: 2014-06-09 16:22:11.391701, event: header_read, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.673753 7f869af78700 5 -- op tracker -- , seq: 2486, time: 2014-06-09 16:22:11.391701, event: throttled, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.673756 7f869af78700 5 -- op tracker -- , seq: 2486, time: 2014-06-09 16:22:11.391732, event: all_read, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.673759 7f869af78700 5 -- op tracker -- , seq: 2486, time: 2014-06-09 16:22:11.673745, event: dispatched, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.673763 7f869af78700 5 -- op tracker -- , seq: 2486, time: 2014-06-09 16:22:11.673762, event: waiting_for_osdmap, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.673765 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.673766 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec4380 2014-06-09 16:22:11.673769 7f869af78700 5 -- op tracker -- , seq: 2486, time: 2014-06-09 16:22:11.673768, event: started, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.673779 7f869af78700 5 -- op tracker -- , seq: 2486, time: 2014-06-09 16:22:11.673778, event: done, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.673783 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673784 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673796 7f869af78700 20 osd.26 243 _dispatch 0x31052380 pg_info(1 pgs e243:0.663) v4 2014-06-09 16:22:11.673799 7f869af78700 5 -- op tracker -- , seq: 2487, time: 2014-06-09 16:22:11.398509, event: header_read, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.673802 7f869af78700 5 -- op tracker -- , seq: 2487, time: 2014-06-09 16:22:11.398510, event: throttled, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.673807 7f869af78700 5 -- op tracker -- , seq: 2487, time: 2014-06-09 16:22:11.398528, event: all_read, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.673810 7f869af78700 5 -- op tracker -- , seq: 2487, time: 2014-06-09 16:22:11.673795, event: dispatched, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.673817 7f869af78700 5 -- op tracker -- , seq: 2487, time: 2014-06-09 16:22:11.673816, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.673821 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.663) v4 from osd.7 2014-06-09 16:22:11.673822 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31052380 2014-06-09 16:22:11.673825 7f869af78700 5 -- op tracker -- , seq: 2487, time: 2014-06-09 16:22:11.673824, event: started, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.673836 7f869af78700 5 -- op tracker -- , seq: 2487, time: 2014-06-09 16:22:11.673835, event: done, op: pg_info(1 pgs e243:0.663) 2014-06-09 16:22:11.673840 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673841 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673848 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673849 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673849 7f869af78700 20 osd.26 243 _dispatch 0x30dafdc0 pg_info(1 pgs e243:1.1e4) v4 2014-06-09 16:22:11.673852 7f869af78700 5 -- op tracker -- , seq: 2488, time: 2014-06-09 16:22:11.446108, event: header_read, op: pg_info(1 pgs e243:1.1e4) 2014-06-09 16:22:11.673856 7f869af78700 5 -- op tracker -- , seq: 2488, time: 2014-06-09 16:22:11.446109, event: throttled, op: pg_info(1 pgs e243:1.1e4) 2014-06-09 16:22:11.673859 7f869af78700 5 -- op tracker -- , seq: 2488, time: 2014-06-09 16:22:11.446126, event: all_read, op: pg_info(1 pgs e243:1.1e4) 2014-06-09 16:22:11.673864 7f869af78700 5 -- op tracker -- , seq: 2488, time: 2014-06-09 16:22:11.673848, event: dispatched, op: pg_info(1 pgs e243:1.1e4) 2014-06-09 16:22:11.673867 7f869af78700 5 -- op tracker -- , seq: 2488, time: 2014-06-09 16:22:11.673866, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1e4) 2014-06-09 16:22:11.673873 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1e4) v4 from osd.3 2014-06-09 16:22:11.673874 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dafdc0 2014-06-09 16:22:11.673877 7f869af78700 5 -- op tracker -- , seq: 2488, time: 2014-06-09 16:22:11.673876, event: started, op: pg_info(1 pgs e243:1.1e4) 2014-06-09 16:22:11.673885 7f869af78700 5 -- op tracker -- , seq: 2488, time: 2014-06-09 16:22:11.673884, event: done, op: pg_info(1 pgs e243:1.1e4) 2014-06-09 16:22:11.673889 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673890 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673897 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673897 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673898 7f869af78700 20 osd.26 243 _dispatch 0x30bd9180 pg_info(1 pgs e243:2.364) v4 2014-06-09 16:22:11.673901 7f869af78700 5 -- op tracker -- , seq: 2489, time: 2014-06-09 16:22:11.368464, event: header_read, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.673905 7f869af78700 5 -- op tracker -- , seq: 2489, time: 2014-06-09 16:22:11.368464, event: throttled, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.673908 7f869af78700 5 -- op tracker -- , seq: 2489, time: 2014-06-09 16:22:11.368521, event: all_read, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.673911 7f869af78700 5 -- op tracker -- , seq: 2489, time: 2014-06-09 16:22:11.673897, event: dispatched, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.673915 7f869af78700 5 -- op tracker -- , seq: 2489, time: 2014-06-09 16:22:11.673914, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.673917 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.364) v4 from osd.1 2014-06-09 16:22:11.673918 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd9180 2014-06-09 16:22:11.673921 7f869af78700 5 -- op tracker -- , seq: 2489, time: 2014-06-09 16:22:11.673920, event: started, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.673929 7f869af78700 5 -- op tracker -- , seq: 2489, time: 2014-06-09 16:22:11.673928, event: done, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.673933 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673934 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673941 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673941 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673942 7f869af78700 20 osd.26 243 _dispatch 0x30a80000 pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.673946 7f869af78700 5 -- op tracker -- , seq: 2490, time: 2014-06-09 16:22:11.410838, event: header_read, op: pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673950 7f869af78700 5 -- op tracker -- , seq: 2490, time: 2014-06-09 16:22:11.410839, event: throttled, op: pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673954 7f869af78700 5 -- op tracker -- , seq: 2490, time: 2014-06-09 16:22:11.410868, event: all_read, op: pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673958 7f869af78700 5 -- op tracker -- , seq: 2490, time: 2014-06-09 16:22:11.673941, event: dispatched, op: pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673962 7f869af78700 5 -- op tracker -- , seq: 2490, time: 2014-06-09 16:22:11.673961, event: waiting_for_osdmap, op: pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673964 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) v4 from osd.9 2014-06-09 16:22:11.673966 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a80000 2014-06-09 16:22:11.673969 7f869af78700 5 -- op tracker -- , seq: 2490, time: 2014-06-09 16:22:11.673968, event: started, op: pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673976 7f869af78700 5 -- op tracker -- , seq: 2490, time: 2014-06-09 16:22:11.673975, event: done, op: pg_log(3.450 epoch 243 log log((213'620,221'627], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.673984 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673985 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673991 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.673992 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.673992 7f869af78700 20 osd.26 243 _dispatch 0x306e6600 pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.673996 7f869af78700 5 -- op tracker -- , seq: 2491, time: 2014-06-09 16:22:11.415408, event: header_read, op: pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.674000 7f869af78700 5 -- op tracker -- , seq: 2491, time: 2014-06-09 16:22:11.415408, event: throttled, op: pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.674004 7f869af78700 5 -- op tracker -- , seq: 2491, time: 2014-06-09 16:22:11.415418, event: all_read, op: pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.674007 7f869af78700 5 -- op tracker -- , seq: 2491, time: 2014-06-09 16:22:11.673991, event: dispatched, op: pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.674011 7f869af78700 5 -- op tracker -- , seq: 2491, time: 2014-06-09 16:22:11.674010, event: waiting_for_osdmap, op: pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.674014 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.674016 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x306e6600 2014-06-09 16:22:11.674019 7f869af78700 5 -- op tracker -- , seq: 2491, time: 2014-06-09 16:22:11.674018, event: started, op: pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.674026 7f869af78700 5 -- op tracker -- , seq: 2491, time: 2014-06-09 16:22:11.674025, event: done, op: pg_log(0.4c7 epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.674030 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.674031 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.674037 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.674038 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.674038 7f869af78700 20 osd.26 243 _dispatch 0x310e2e00 pg_info(1 pgs e243:0.138) v4 2014-06-09 16:22:11.674041 7f869af78700 5 -- op tracker -- , seq: 2492, time: 2014-06-09 16:22:11.400265, event: header_read, op: pg_info(1 pgs e243:0.138) 2014-06-09 16:22:11.674045 7f869af78700 5 -- op tracker -- , seq: 2492, time: 2014-06-09 16:22:11.400265, event: throttled, op: pg_info(1 pgs e243:0.138) 2014-06-09 16:22:11.674048 7f869af78700 5 -- op tracker -- , seq: 2492, time: 2014-06-09 16:22:11.400275, event: all_read, op: pg_info(1 pgs e243:0.138) 2014-06-09 16:22:11.674053 7f869af78700 5 -- op tracker -- , seq: 2492, time: 2014-06-09 16:22:11.674037, event: dispatched, op: pg_info(1 pgs e243:0.138) 2014-06-09 16:22:11.674056 7f869af78700 5 -- op tracker -- , seq: 2492, time: 2014-06-09 16:22:11.674055, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.138) 2014-06-09 16:22:11.674060 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.138) v4 from osd.6 2014-06-09 16:22:11.674061 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e2e00 2014-06-09 16:22:11.674063 7f869af78700 5 -- op tracker -- , seq: 2492, time: 2014-06-09 16:22:11.674063, event: started, op: pg_info(1 pgs e243:0.138) 2014-06-09 16:22:11.674072 7f869af78700 5 -- op tracker -- , seq: 2492, time: 2014-06-09 16:22:11.674071, event: done, op: pg_info(1 pgs e243:0.138) 2014-06-09 16:22:11.674076 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.674077 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.674083 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.674084 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.674084 7f869af78700 20 osd.26 243 _dispatch 0x3046ba40 pg_info(1 pgs e243:1.76) v4 2014-06-09 16:22:11.674087 7f869af78700 5 -- op tracker -- , seq: 2493, time: 2014-06-09 16:22:11.387890, event: header_read, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.674091 7f869af78700 5 -- op tracker -- , seq: 2493, time: 2014-06-09 16:22:11.387891, event: throttled, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.674095 7f869af78700 5 -- op tracker -- , seq: 2493, time: 2014-06-09 16:22:11.387901, event: all_read, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.674099 7f869af78700 5 -- op tracker -- , seq: 2493, time: 2014-06-09 16:22:11.674083, event: dispatched, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.674104 7f869af78700 5 -- op tracker -- , seq: 2493, time: 2014-06-09 16:22:11.674103, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.674106 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.76) v4 from osd.0 2014-06-09 16:22:11.674108 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046ba40 2014-06-09 16:22:11.674110 7f869af78700 5 -- op tracker -- , seq: 2493, time: 2014-06-09 16:22:11.674109, event: started, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.674118 7f869af78700 5 -- op tracker -- , seq: 2493, time: 2014-06-09 16:22:11.674117, event: done, op: pg_info(1 pgs e243:1.76) 2014-06-09 16:22:11.674122 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.674123 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.674129 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.674130 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.674130 7f869af78700 20 osd.26 243 _dispatch 0x3115afc0 pg_info(1 pgs e243:2.746) v4 2014-06-09 16:22:11.674133 7f869af78700 5 -- op tracker -- , seq: 2494, time: 2014-06-09 16:22:11.419749, event: header_read, op: pg_info(1 pgs e243:2.746) 2014-06-09 16:22:11.674137 7f869af78700 5 -- op tracker -- , seq: 2494, time: 2014-06-09 16:22:11.419749, event: throttled, op: pg_info(1 pgs e243:2.746) 2014-06-09 16:22:11.674141 7f869af78700 5 -- op tracker -- , seq: 2494, time: 2014-06-09 16:22:11.419758, event: all_read, op: pg_info(1 pgs e243:2.746) 2014-06-09 16:22:11.674144 7f869af78700 5 -- op tracker -- , seq: 2494, time: 2014-06-09 16:22:11.674129, event: dispatched, op: pg_info(1 pgs e243:2.746) 2014-06-09 16:22:11.674148 7f869af78700 5 -- op tracker -- , seq: 2494, time: 2014-06-09 16:22:11.674146, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.746) 2014-06-09 16:22:11.674150 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.746) v4 from osd.5 2014-06-09 16:22:11.674151 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115afc0 2014-06-09 16:22:11.674154 7f869af78700 5 -- op tracker -- , seq: 2494, time: 2014-06-09 16:22:11.674153, event: started, op: pg_info(1 pgs e243:2.746) 2014-06-09 16:22:11.674162 7f869af78700 5 -- op tracker -- , seq: 2494, time: 2014-06-09 16:22:11.674161, event: done, op: pg_info(1 pgs e243:2.746) 2014-06-09 16:22:11.674166 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676454 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676455 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676455 7f869af78700 20 osd.26 243 _dispatch 0x305f2000 pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.676459 7f869af78700 5 -- op tracker -- , seq: 2495, time: 2014-06-09 16:22:11.286131, event: header_read, op: pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.676465 7f869af78700 5 -- op tracker -- , seq: 2495, time: 2014-06-09 16:22:11.286131, event: throttled, op: pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.676469 7f869af78700 5 -- op tracker -- , seq: 2495, time: 2014-06-09 16:22:11.286137, event: all_read, op: pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.676473 7f869af78700 5 -- op tracker -- , seq: 2495, time: 2014-06-09 16:22:11.676454, event: dispatched, op: pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.676477 7f869af78700 5 -- op tracker -- , seq: 2495, time: 2014-06-09 16:22:11.676475, event: waiting_for_osdmap, op: pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.676479 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.676482 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x305f2000 2014-06-09 16:22:11.676485 7f869af78700 5 -- op tracker -- , seq: 2495, time: 2014-06-09 16:22:11.676483, event: started, op: pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.676492 7f869af78700 5 -- op tracker -- , seq: 2495, time: 2014-06-09 16:22:11.676490, event: done, op: pg_log(1.1fb epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.676497 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676498 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676505 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676506 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676506 7f869af78700 20 osd.26 243 _dispatch 0x210b1180 pg_notify(0.708(23) epoch 243) v5 2014-06-09 16:22:11.676509 7f869af78700 5 -- op tracker -- , seq: 2496, time: 2014-06-09 16:22:11.391747, event: header_read, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.676513 7f869af78700 5 -- op tracker -- , seq: 2496, time: 2014-06-09 16:22:11.391747, event: throttled, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.676521 7f869af78700 5 -- op tracker -- , seq: 2496, time: 2014-06-09 16:22:11.391783, event: all_read, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.676525 7f869af78700 5 -- op tracker -- , seq: 2496, time: 2014-06-09 16:22:11.676505, event: dispatched, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.676528 7f869af78700 5 -- op tracker -- , seq: 2496, time: 2014-06-09 16:22:11.676527, event: waiting_for_osdmap, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.676535 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.676536 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x210b1180 2014-06-09 16:22:11.676539 7f869af78700 5 -- op tracker -- , seq: 2496, time: 2014-06-09 16:22:11.676538, event: started, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.676548 7f869af78700 5 -- op tracker -- , seq: 2496, time: 2014-06-09 16:22:11.676547, event: done, op: pg_notify(0.708(23) epoch 243) 2014-06-09 16:22:11.676552 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676553 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676566 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676566 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676567 7f869af78700 20 osd.26 243 _dispatch 0x310521c0 pg_info(1 pgs e243:1.691) v4 2014-06-09 16:22:11.676569 7f869af78700 5 -- op tracker -- , seq: 2497, time: 2014-06-09 16:22:11.398538, event: header_read, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.676573 7f869af78700 5 -- op tracker -- , seq: 2497, time: 2014-06-09 16:22:11.398538, event: throttled, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.676576 7f869af78700 5 -- op tracker -- , seq: 2497, time: 2014-06-09 16:22:11.398550, event: all_read, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.676584 7f869af78700 5 -- op tracker -- , seq: 2497, time: 2014-06-09 16:22:11.676565, event: dispatched, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.676588 7f869af78700 5 -- op tracker -- , seq: 2497, time: 2014-06-09 16:22:11.676587, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.676592 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.691) v4 from osd.7 2014-06-09 16:22:11.676594 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310521c0 2014-06-09 16:22:11.676596 7f869af78700 5 -- op tracker -- , seq: 2497, time: 2014-06-09 16:22:11.676595, event: started, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.676606 7f869af78700 5 -- op tracker -- , seq: 2497, time: 2014-06-09 16:22:11.676605, event: done, op: pg_info(1 pgs e243:1.691) 2014-06-09 16:22:11.676610 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676611 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676618 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676618 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676619 7f869af78700 20 osd.26 243 _dispatch 0x2d740380 pg_info(1 pgs e243:2.777) v4 2014-06-09 16:22:11.676621 7f869af78700 5 -- op tracker -- , seq: 2498, time: 2014-06-09 16:22:11.446135, event: header_read, op: pg_info(1 pgs e243:2.777) 2014-06-09 16:22:11.676625 7f869af78700 5 -- op tracker -- , seq: 2498, time: 2014-06-09 16:22:11.446135, event: throttled, op: pg_info(1 pgs e243:2.777) 2014-06-09 16:22:11.676628 7f869af78700 5 -- op tracker -- , seq: 2498, time: 2014-06-09 16:22:11.446147, event: all_read, op: pg_info(1 pgs e243:2.777) 2014-06-09 16:22:11.676631 7f869af78700 5 -- op tracker -- , seq: 2498, time: 2014-06-09 16:22:11.676617, event: dispatched, op: pg_info(1 pgs e243:2.777) 2014-06-09 16:22:11.676637 7f869af78700 5 -- op tracker -- , seq: 2498, time: 2014-06-09 16:22:11.676635, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.777) 2014-06-09 16:22:11.676639 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.777) v4 from osd.3 2014-06-09 16:22:11.676640 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d740380 2014-06-09 16:22:11.676643 7f869af78700 5 -- op tracker -- , seq: 2498, time: 2014-06-09 16:22:11.676642, event: started, op: pg_info(1 pgs e243:2.777) 2014-06-09 16:22:11.676655 7f869af78700 5 -- op tracker -- , seq: 2498, time: 2014-06-09 16:22:11.676654, event: done, op: pg_info(1 pgs e243:2.777) 2014-06-09 16:22:11.676659 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676660 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676666 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676667 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676668 7f869af78700 20 osd.26 243 _dispatch 0x30bd8700 pg_info(1 pgs e243:0.369) v4 2014-06-09 16:22:11.676670 7f869af78700 5 -- op tracker -- , seq: 2499, time: 2014-06-09 16:22:11.368618, event: header_read, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.676675 7f869af78700 5 -- op tracker -- , seq: 2499, time: 2014-06-09 16:22:11.368619, event: throttled, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.676679 7f869af78700 5 -- op tracker -- , seq: 2499, time: 2014-06-09 16:22:11.368653, event: all_read, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.676683 7f869af78700 5 -- op tracker -- , seq: 2499, time: 2014-06-09 16:22:11.676666, event: dispatched, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.676687 7f869af78700 5 -- op tracker -- , seq: 2499, time: 2014-06-09 16:22:11.676686, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.676694 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.369) v4 from osd.1 2014-06-09 16:22:11.676696 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd8700 2014-06-09 16:22:11.676698 7f869af78700 5 -- op tracker -- , seq: 2499, time: 2014-06-09 16:22:11.676697, event: started, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.676707 7f869af78700 5 -- op tracker -- , seq: 2499, time: 2014-06-09 16:22:11.676706, event: done, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.676716 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676717 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676723 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676724 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676725 7f869af78700 20 osd.26 243 _dispatch 0x310db180 pg_info(1 pgs e243:3.94) v4 2014-06-09 16:22:11.676727 7f869af78700 5 -- op tracker -- , seq: 2500, time: 2014-06-09 16:22:11.410873, event: header_read, op: pg_info(1 pgs e243:3.94) 2014-06-09 16:22:11.676733 7f869af78700 5 -- op tracker -- , seq: 2500, time: 2014-06-09 16:22:11.410874, event: throttled, op: pg_info(1 pgs e243:3.94) 2014-06-09 16:22:11.676736 7f869af78700 5 -- op tracker -- , seq: 2500, time: 2014-06-09 16:22:11.410885, event: all_read, op: pg_info(1 pgs e243:3.94) 2014-06-09 16:22:11.676740 7f869af78700 5 -- op tracker -- , seq: 2500, time: 2014-06-09 16:22:11.676723, event: dispatched, op: pg_info(1 pgs e243:3.94) 2014-06-09 16:22:11.676743 7f869af78700 5 -- op tracker -- , seq: 2500, time: 2014-06-09 16:22:11.676742, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.94) 2014-06-09 16:22:11.676746 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.94) v4 from osd.9 2014-06-09 16:22:11.676747 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310db180 2014-06-09 16:22:11.676750 7f869af78700 5 -- op tracker -- , seq: 2500, time: 2014-06-09 16:22:11.676749, event: started, op: pg_info(1 pgs e243:3.94) 2014-06-09 16:22:11.676760 7f869af78700 5 -- op tracker -- , seq: 2500, time: 2014-06-09 16:22:11.676759, event: done, op: pg_info(1 pgs e243:3.94) 2014-06-09 16:22:11.676764 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676765 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676772 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676772 7f869af78700 20 osd.26 243 _dispatch 0x3104a380 pg_info(1 pgs e243:0.4da) v4 2014-06-09 16:22:11.676775 7f869af78700 5 -- op tracker -- , seq: 2501, time: 2014-06-09 16:22:11.415423, event: header_read, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.676779 7f869af78700 5 -- op tracker -- , seq: 2501, time: 2014-06-09 16:22:11.415423, event: throttled, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.676782 7f869af78700 5 -- op tracker -- , seq: 2501, time: 2014-06-09 16:22:11.415436, event: all_read, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.676786 7f869af78700 5 -- op tracker -- , seq: 2501, time: 2014-06-09 16:22:11.676771, event: dispatched, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.676790 7f869af78700 5 -- op tracker -- , seq: 2501, time: 2014-06-09 16:22:11.676789, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.676793 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4da) v4 from osd.8 2014-06-09 16:22:11.676794 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104a380 2014-06-09 16:22:11.676797 7f869af78700 5 -- op tracker -- , seq: 2501, time: 2014-06-09 16:22:11.676796, event: started, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.676805 7f869af78700 5 -- op tracker -- , seq: 2501, time: 2014-06-09 16:22:11.676804, event: done, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.676809 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676810 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676816 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676816 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676817 7f869af78700 20 osd.26 243 _dispatch 0x310e2c40 pg_info(1 pgs e243:0.532) v4 2014-06-09 16:22:11.676820 7f869af78700 5 -- op tracker -- , seq: 2502, time: 2014-06-09 16:22:11.400279, event: header_read, op: pg_info(1 pgs e243:0.532) 2014-06-09 16:22:11.676823 7f869af78700 5 -- op tracker -- , seq: 2502, time: 2014-06-09 16:22:11.400280, event: throttled, op: pg_info(1 pgs e243:0.532) 2014-06-09 16:22:11.676827 7f869af78700 5 -- op tracker -- , seq: 2502, time: 2014-06-09 16:22:11.400293, event: all_read, op: pg_info(1 pgs e243:0.532) 2014-06-09 16:22:11.676831 7f869af78700 5 -- op tracker -- , seq: 2502, time: 2014-06-09 16:22:11.676816, event: dispatched, op: pg_info(1 pgs e243:0.532) 2014-06-09 16:22:11.676835 7f869af78700 5 -- op tracker -- , seq: 2502, time: 2014-06-09 16:22:11.676834, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.532) 2014-06-09 16:22:11.676842 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.532) v4 from osd.6 2014-06-09 16:22:11.676844 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e2c40 2014-06-09 16:22:11.676846 7f869af78700 5 -- op tracker -- , seq: 2502, time: 2014-06-09 16:22:11.676845, event: started, op: pg_info(1 pgs e243:0.532) 2014-06-09 16:22:11.676855 7f869af78700 5 -- op tracker -- , seq: 2502, time: 2014-06-09 16:22:11.676854, event: done, op: pg_info(1 pgs e243:0.532) 2014-06-09 16:22:11.676859 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676860 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676867 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676868 7f869af78700 20 osd.26 243 _dispatch 0x30bd9880 pg_info(1 pgs e243:2.75) v4 2014-06-09 16:22:11.676871 7f869af78700 5 -- op tracker -- , seq: 2503, time: 2014-06-09 16:22:11.391642, event: header_read, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.676874 7f869af78700 5 -- op tracker -- , seq: 2503, time: 2014-06-09 16:22:11.391643, event: throttled, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.676878 7f869af78700 5 -- op tracker -- , seq: 2503, time: 2014-06-09 16:22:11.391656, event: all_read, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.676881 7f869af78700 5 -- op tracker -- , seq: 2503, time: 2014-06-09 16:22:11.676867, event: dispatched, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.676884 7f869af78700 5 -- op tracker -- , seq: 2503, time: 2014-06-09 16:22:11.676883, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.676886 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.75) v4 from osd.0 2014-06-09 16:22:11.676888 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd9880 2014-06-09 16:22:11.676890 7f869af78700 5 -- op tracker -- , seq: 2503, time: 2014-06-09 16:22:11.676889, event: started, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.676898 7f869af78700 5 -- op tracker -- , seq: 2503, time: 2014-06-09 16:22:11.676897, event: done, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.676902 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676903 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676910 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676911 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676911 7f869af78700 20 osd.26 243 _dispatch 0x3115ae00 pg_info(1 pgs e243:3.31c) v4 2014-06-09 16:22:11.676914 7f869af78700 5 -- op tracker -- , seq: 2504, time: 2014-06-09 16:22:11.419763, event: header_read, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.676918 7f869af78700 5 -- op tracker -- , seq: 2504, time: 2014-06-09 16:22:11.419763, event: throttled, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.676921 7f869af78700 5 -- op tracker -- , seq: 2504, time: 2014-06-09 16:22:11.419773, event: all_read, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.676926 7f869af78700 5 -- op tracker -- , seq: 2504, time: 2014-06-09 16:22:11.676910, event: dispatched, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.676929 7f869af78700 5 -- op tracker -- , seq: 2504, time: 2014-06-09 16:22:11.676928, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.676935 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.31c) v4 from osd.5 2014-06-09 16:22:11.676937 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115ae00 2014-06-09 16:22:11.676939 7f869af78700 5 -- op tracker -- , seq: 2504, time: 2014-06-09 16:22:11.676938, event: started, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.676948 7f869af78700 5 -- op tracker -- , seq: 2504, time: 2014-06-09 16:22:11.676947, event: done, op: pg_info(1 pgs e243:3.31c) 2014-06-09 16:22:11.676952 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676961 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.676961 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.676962 7f869af78700 20 osd.26 243 _dispatch 0x15458e00 pg_info(1 pgs e243:3.793(4)) v4 2014-06-09 16:22:11.676965 7f869af78700 5 -- op tracker -- , seq: 2505, time: 2014-06-09 16:22:11.286140, event: header_read, op: pg_info(1 pgs e243:3.793(4)) 2014-06-09 16:22:11.676969 7f869af78700 5 -- op tracker -- , seq: 2505, time: 2014-06-09 16:22:11.286140, event: throttled, op: pg_info(1 pgs e243:3.793(4)) 2014-06-09 16:22:11.676976 7f869af78700 5 -- op tracker -- , seq: 2505, time: 2014-06-09 16:22:11.286148, event: all_read, op: pg_info(1 pgs e243:3.793(4)) 2014-06-09 16:22:11.676979 7f869af78700 5 -- op tracker -- , seq: 2505, time: 2014-06-09 16:22:11.676960, event: dispatched, op: pg_info(1 pgs e243:3.793(4)) 2014-06-09 16:22:11.676983 7f869af78700 5 -- op tracker -- , seq: 2505, time: 2014-06-09 16:22:11.676982, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.793(4)) 2014-06-09 16:22:11.676985 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.793(4)) v4 from osd.2 2014-06-09 16:22:11.676987 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15458e00 2014-06-09 16:22:11.676989 7f869af78700 5 -- op tracker -- , seq: 2505, time: 2014-06-09 16:22:11.676988, event: started, op: pg_info(1 pgs e243:3.793(4)) 2014-06-09 16:22:11.677001 7f869af78700 5 -- op tracker -- , seq: 2505, time: 2014-06-09 16:22:11.677000, event: done, op: pg_info(1 pgs e243:3.793(4)) 2014-06-09 16:22:11.677006 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677006 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677016 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677016 7f869af78700 20 osd.26 243 _dispatch 0x2fe33e00 pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.677020 7f869af78700 5 -- op tracker -- , seq: 2506, time: 2014-06-09 16:22:11.391790, event: header_read, op: pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.677024 7f869af78700 5 -- op tracker -- , seq: 2506, time: 2014-06-09 16:22:11.391790, event: throttled, op: pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.677028 7f869af78700 5 -- op tracker -- , seq: 2506, time: 2014-06-09 16:22:11.391806, event: all_read, op: pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.677031 7f869af78700 5 -- op tracker -- , seq: 2506, time: 2014-06-09 16:22:11.677015, event: dispatched, op: pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.677035 7f869af78700 5 -- op tracker -- , seq: 2506, time: 2014-06-09 16:22:11.677034, event: waiting_for_osdmap, op: pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.677041 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.677043 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe33e00 2014-06-09 16:22:11.677046 7f869af78700 5 -- op tracker -- , seq: 2506, time: 2014-06-09 16:22:11.677045, event: started, op: pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.677057 7f869af78700 5 -- op tracker -- , seq: 2506, time: 2014-06-09 16:22:11.677055, event: done, op: pg_log(0.71f epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.677062 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677063 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677069 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677070 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677070 7f869af78700 20 osd.26 243 _dispatch 0x31052000 pg_info(1 pgs e243:1.92) v4 2014-06-09 16:22:11.677073 7f869af78700 5 -- op tracker -- , seq: 2507, time: 2014-06-09 16:22:11.398555, event: header_read, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.677077 7f869af78700 5 -- op tracker -- , seq: 2507, time: 2014-06-09 16:22:11.398556, event: throttled, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.677080 7f869af78700 5 -- op tracker -- , seq: 2507, time: 2014-06-09 16:22:11.398566, event: all_read, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.677084 7f869af78700 5 -- op tracker -- , seq: 2507, time: 2014-06-09 16:22:11.677069, event: dispatched, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.677088 7f869af78700 5 -- op tracker -- , seq: 2507, time: 2014-06-09 16:22:11.677087, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.677090 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.92) v4 from osd.7 2014-06-09 16:22:11.677092 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31052000 2014-06-09 16:22:11.677094 7f869af78700 5 -- op tracker -- , seq: 2507, time: 2014-06-09 16:22:11.677093, event: started, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.677105 7f869af78700 5 -- op tracker -- , seq: 2507, time: 2014-06-09 16:22:11.677104, event: done, op: pg_info(1 pgs e243:1.92) 2014-06-09 16:22:11.677109 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677110 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677117 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677117 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677118 7f869af78700 20 osd.26 243 _dispatch 0x2d0948c0 pg_info(1 pgs e243:1.19f) v4 2014-06-09 16:22:11.677121 7f869af78700 5 -- op tracker -- , seq: 2508, time: 2014-06-09 16:22:11.446152, event: header_read, op: pg_info(1 pgs e243:1.19f) 2014-06-09 16:22:11.677124 7f869af78700 5 -- op tracker -- , seq: 2508, time: 2014-06-09 16:22:11.446152, event: throttled, op: pg_info(1 pgs e243:1.19f) 2014-06-09 16:22:11.677129 7f869af78700 5 -- op tracker -- , seq: 2508, time: 2014-06-09 16:22:11.446162, event: all_read, op: pg_info(1 pgs e243:1.19f) 2014-06-09 16:22:11.677133 7f869af78700 5 -- op tracker -- , seq: 2508, time: 2014-06-09 16:22:11.677117, event: dispatched, op: pg_info(1 pgs e243:1.19f) 2014-06-09 16:22:11.677140 7f869af78700 5 -- op tracker -- , seq: 2508, time: 2014-06-09 16:22:11.677139, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.19f) 2014-06-09 16:22:11.677142 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.19f) v4 from osd.3 2014-06-09 16:22:11.677144 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d0948c0 2014-06-09 16:22:11.677146 7f869af78700 5 -- op tracker -- , seq: 2508, time: 2014-06-09 16:22:11.677145, event: started, op: pg_info(1 pgs e243:1.19f) 2014-06-09 16:22:11.677154 7f869af78700 5 -- op tracker -- , seq: 2508, time: 2014-06-09 16:22:11.677153, event: done, op: pg_info(1 pgs e243:1.19f) 2014-06-09 16:22:11.677158 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677159 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677166 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677166 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677167 7f869af78700 20 osd.26 243 _dispatch 0x30bd81c0 pg_info(1 pgs e243:2.372) v4 2014-06-09 16:22:11.677169 7f869af78700 5 -- op tracker -- , seq: 2509, time: 2014-06-09 16:22:11.368667, event: header_read, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.677173 7f869af78700 5 -- op tracker -- , seq: 2509, time: 2014-06-09 16:22:11.368667, event: throttled, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.677176 7f869af78700 5 -- op tracker -- , seq: 2509, time: 2014-06-09 16:22:11.368733, event: all_read, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.677181 7f869af78700 5 -- op tracker -- , seq: 2509, time: 2014-06-09 16:22:11.677165, event: dispatched, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.677185 7f869af78700 5 -- op tracker -- , seq: 2509, time: 2014-06-09 16:22:11.677184, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.677190 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.372) v4 from osd.1 2014-06-09 16:22:11.677192 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd81c0 2014-06-09 16:22:11.677194 7f869af78700 5 -- op tracker -- , seq: 2509, time: 2014-06-09 16:22:11.677193, event: started, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.677203 7f869af78700 5 -- op tracker -- , seq: 2509, time: 2014-06-09 16:22:11.677202, event: done, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.677207 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677208 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677215 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677215 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677216 7f869af78700 20 osd.26 243 _dispatch 0x310dafc0 pg_info(1 pgs e243:3.173) v4 2014-06-09 16:22:11.677219 7f869af78700 5 -- op tracker -- , seq: 2510, time: 2014-06-09 16:22:11.410890, event: header_read, op: pg_info(1 pgs e243:3.173) 2014-06-09 16:22:11.677222 7f869af78700 5 -- op tracker -- , seq: 2510, time: 2014-06-09 16:22:11.410890, event: throttled, op: pg_info(1 pgs e243:3.173) 2014-06-09 16:22:11.677229 7f869af78700 5 -- op tracker -- , seq: 2510, time: 2014-06-09 16:22:11.410901, event: all_read, op: pg_info(1 pgs e243:3.173) 2014-06-09 16:22:11.677232 7f869af78700 5 -- op tracker -- , seq: 2510, time: 2014-06-09 16:22:11.677215, event: dispatched, op: pg_info(1 pgs e243:3.173) 2014-06-09 16:22:11.677236 7f869af78700 5 -- op tracker -- , seq: 2510, time: 2014-06-09 16:22:11.677235, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.173) 2014-06-09 16:22:11.677238 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.173) v4 from osd.9 2014-06-09 16:22:11.677240 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310dafc0 2014-06-09 16:22:11.677242 7f869af78700 5 -- op tracker -- , seq: 2510, time: 2014-06-09 16:22:11.677241, event: started, op: pg_info(1 pgs e243:3.173) 2014-06-09 16:22:11.677253 7f869af78700 5 -- op tracker -- , seq: 2510, time: 2014-06-09 16:22:11.677252, event: done, op: pg_info(1 pgs e243:3.173) 2014-06-09 16:22:11.677257 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677258 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677265 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677265 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677266 7f869af78700 20 osd.26 243 _dispatch 0x3104a1c0 pg_info(1 pgs e243:0.301) v4 2014-06-09 16:22:11.677268 7f869af78700 5 -- op tracker -- , seq: 2511, time: 2014-06-09 16:22:11.415440, event: header_read, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.677273 7f869af78700 5 -- op tracker -- , seq: 2511, time: 2014-06-09 16:22:11.415440, event: throttled, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.677277 7f869af78700 5 -- op tracker -- , seq: 2511, time: 2014-06-09 16:22:11.415453, event: all_read, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.677281 7f869af78700 5 -- op tracker -- , seq: 2511, time: 2014-06-09 16:22:11.677264, event: dispatched, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.677285 7f869af78700 5 -- op tracker -- , seq: 2511, time: 2014-06-09 16:22:11.677284, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.677291 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.301) v4 from osd.8 2014-06-09 16:22:11.677292 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104a1c0 2014-06-09 16:22:11.677295 7f869af78700 5 -- op tracker -- , seq: 2511, time: 2014-06-09 16:22:11.677294, event: started, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.677303 7f869af78700 5 -- op tracker -- , seq: 2511, time: 2014-06-09 16:22:11.677302, event: done, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.677307 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677308 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677315 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677315 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677316 7f869af78700 20 osd.26 243 _dispatch 0x310e2a80 pg_info(1 pgs e243:1.700) v4 2014-06-09 16:22:11.677319 7f869af78700 5 -- op tracker -- , seq: 2512, time: 2014-06-09 16:22:11.400298, event: header_read, op: pg_info(1 pgs e243:1.700) 2014-06-09 16:22:11.677322 7f869af78700 5 -- op tracker -- , seq: 2512, time: 2014-06-09 16:22:11.400298, event: throttled, op: pg_info(1 pgs e243:1.700) 2014-06-09 16:22:11.677325 7f869af78700 5 -- op tracker -- , seq: 2512, time: 2014-06-09 16:22:11.400308, event: all_read, op: pg_info(1 pgs e243:1.700) 2014-06-09 16:22:11.677329 7f869af78700 5 -- op tracker -- , seq: 2512, time: 2014-06-09 16:22:11.677315, event: dispatched, op: pg_info(1 pgs e243:1.700) 2014-06-09 16:22:11.677334 7f869af78700 5 -- op tracker -- , seq: 2512, time: 2014-06-09 16:22:11.677332, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.700) 2014-06-09 16:22:11.677336 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.700) v4 from osd.6 2014-06-09 16:22:11.677337 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e2a80 2014-06-09 16:22:11.677340 7f869af78700 5 -- op tracker -- , seq: 2512, time: 2014-06-09 16:22:11.677339, event: started, op: pg_info(1 pgs e243:1.700) 2014-06-09 16:22:11.677348 7f869af78700 5 -- op tracker -- , seq: 2512, time: 2014-06-09 16:22:11.677347, event: done, op: pg_info(1 pgs e243:1.700) 2014-06-09 16:22:11.677352 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677353 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677359 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677360 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677360 7f869af78700 20 osd.26 243 _dispatch 0x3046bc00 pg_info(1 pgs e243:2.75) v4 2014-06-09 16:22:11.677363 7f869af78700 5 -- op tracker -- , seq: 2513, time: 2014-06-09 16:22:11.391666, event: header_read, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.677367 7f869af78700 5 -- op tracker -- , seq: 2513, time: 2014-06-09 16:22:11.391666, event: throttled, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.677370 7f869af78700 5 -- op tracker -- , seq: 2513, time: 2014-06-09 16:22:11.391688, event: all_read, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.677375 7f869af78700 5 -- op tracker -- , seq: 2513, time: 2014-06-09 16:22:11.677359, event: dispatched, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.677378 7f869af78700 5 -- op tracker -- , seq: 2513, time: 2014-06-09 16:22:11.677377, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.677381 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.75) v4 from osd.0 2014-06-09 16:22:11.677383 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3046bc00 2014-06-09 16:22:11.677386 7f869af78700 5 -- op tracker -- , seq: 2513, time: 2014-06-09 16:22:11.677385, event: started, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.677395 7f869af78700 5 -- op tracker -- , seq: 2513, time: 2014-06-09 16:22:11.677394, event: done, op: pg_info(1 pgs e243:2.75) 2014-06-09 16:22:11.677398 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677406 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677407 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677407 7f869af78700 20 osd.26 243 _dispatch 0x3115ac40 pg_notify(3.4b9(10),1.1(20) epoch 243) v5 2014-06-09 16:22:11.677411 7f869af78700 5 -- op tracker -- , seq: 2514, time: 2014-06-09 16:22:11.419778, event: header_read, op: pg_notify(3.4b9(10),1.1(20) epoch 243) 2014-06-09 16:22:11.677416 7f869af78700 5 -- op tracker -- , seq: 2514, time: 2014-06-09 16:22:11.419778, event: throttled, op: pg_notify(3.4b9(10),1.1(20) epoch 243) 2014-06-09 16:22:11.677419 7f869af78700 5 -- op tracker -- , seq: 2514, time: 2014-06-09 16:22:11.419816, event: all_read, op: pg_notify(3.4b9(10),1.1(20) epoch 243) 2014-06-09 16:22:11.677423 7f869af78700 5 -- op tracker -- , seq: 2514, time: 2014-06-09 16:22:11.677406, event: dispatched, op: pg_notify(3.4b9(10),1.1(20) epoch 243) 2014-06-09 16:22:11.677427 7f869af78700 5 -- op tracker -- , seq: 2514, time: 2014-06-09 16:22:11.677426, event: waiting_for_osdmap, op: pg_notify(3.4b9(10),1.1(20) epoch 243) 2014-06-09 16:22:11.677429 7f869af78700 7 osd.26 243 handle_pg_notify from osd.5 2014-06-09 16:22:11.677430 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115ac40 2014-06-09 16:22:11.677433 7f869af78700 5 -- op tracker -- , seq: 2514, time: 2014-06-09 16:22:11.677431, event: started, op: pg_notify(3.4b9(10),1.1(20) epoch 243) 2014-06-09 16:22:11.677447 7f869af78700 5 -- op tracker -- , seq: 2514, time: 2014-06-09 16:22:11.677446, event: done, op: pg_notify(3.4b9(10),1.1(20) epoch 243) 2014-06-09 16:22:11.677452 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677453 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677465 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677465 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677466 7f869af78700 20 osd.26 243 _dispatch 0x3088c000 pg_info(1 pgs e243:2.614) v4 2014-06-09 16:22:11.677468 7f869af78700 5 -- op tracker -- , seq: 2515, time: 2014-06-09 16:22:11.302799, event: header_read, op: pg_info(1 pgs e243:2.614) 2014-06-09 16:22:11.677476 7f869af78700 5 -- op tracker -- , seq: 2515, time: 2014-06-09 16:22:11.302800, event: throttled, op: pg_info(1 pgs e243:2.614) 2014-06-09 16:22:11.677480 7f869af78700 5 -- op tracker -- , seq: 2515, time: 2014-06-09 16:22:11.302810, event: all_read, op: pg_info(1 pgs e243:2.614) 2014-06-09 16:22:11.677484 7f869af78700 5 -- op tracker -- , seq: 2515, time: 2014-06-09 16:22:11.677464, event: dispatched, op: pg_info(1 pgs e243:2.614) 2014-06-09 16:22:11.677488 7f869af78700 5 -- op tracker -- , seq: 2515, time: 2014-06-09 16:22:11.677487, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.614) 2014-06-09 16:22:11.677491 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.614) v4 from osd.2 2014-06-09 16:22:11.677493 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088c000 2014-06-09 16:22:11.677496 7f869af78700 5 -- op tracker -- , seq: 2515, time: 2014-06-09 16:22:11.677495, event: started, op: pg_info(1 pgs e243:2.614) 2014-06-09 16:22:11.677505 7f869af78700 5 -- op tracker -- , seq: 2515, time: 2014-06-09 16:22:11.677504, event: done, op: pg_info(1 pgs e243:2.614) 2014-06-09 16:22:11.677509 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677510 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677517 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677518 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677518 7f869af78700 20 osd.26 243 _dispatch 0x280c5880 pg_info(1 pgs e243:1.4ef) v4 2014-06-09 16:22:11.677521 7f869af78700 5 -- op tracker -- , seq: 2516, time: 2014-06-09 16:22:11.395941, event: header_read, op: pg_info(1 pgs e243:1.4ef) 2014-06-09 16:22:11.677525 7f869af78700 5 -- op tracker -- , seq: 2516, time: 2014-06-09 16:22:11.395942, event: throttled, op: pg_info(1 pgs e243:1.4ef) 2014-06-09 16:22:11.677528 7f869af78700 5 -- op tracker -- , seq: 2516, time: 2014-06-09 16:22:11.395960, event: all_read, op: pg_info(1 pgs e243:1.4ef) 2014-06-09 16:22:11.677531 7f869af78700 5 -- op tracker -- , seq: 2516, time: 2014-06-09 16:22:11.677517, event: dispatched, op: pg_info(1 pgs e243:1.4ef) 2014-06-09 16:22:11.677537 7f869af78700 5 -- op tracker -- , seq: 2516, time: 2014-06-09 16:22:11.677536, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4ef) 2014-06-09 16:22:11.677539 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4ef) v4 from osd.4 2014-06-09 16:22:11.677541 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c5880 2014-06-09 16:22:11.677543 7f869af78700 5 -- op tracker -- , seq: 2516, time: 2014-06-09 16:22:11.677542, event: started, op: pg_info(1 pgs e243:1.4ef) 2014-06-09 16:22:11.677551 7f869af78700 5 -- op tracker -- , seq: 2516, time: 2014-06-09 16:22:11.677550, event: done, op: pg_info(1 pgs e243:1.4ef) 2014-06-09 16:22:11.677555 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677556 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677563 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677564 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677564 7f869af78700 20 osd.26 243 _dispatch 0x310d7dc0 pg_info(1 pgs e243:1.631) v4 2014-06-09 16:22:11.677567 7f869af78700 5 -- op tracker -- , seq: 2517, time: 2014-06-09 16:22:11.398572, event: header_read, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.677570 7f869af78700 5 -- op tracker -- , seq: 2517, time: 2014-06-09 16:22:11.398572, event: throttled, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.677574 7f869af78700 5 -- op tracker -- , seq: 2517, time: 2014-06-09 16:22:11.398585, event: all_read, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.677577 7f869af78700 5 -- op tracker -- , seq: 2517, time: 2014-06-09 16:22:11.677563, event: dispatched, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.677580 7f869af78700 5 -- op tracker -- , seq: 2517, time: 2014-06-09 16:22:11.677579, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.677582 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.631) v4 from osd.7 2014-06-09 16:22:11.677584 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d7dc0 2014-06-09 16:22:11.677586 7f869af78700 5 -- op tracker -- , seq: 2517, time: 2014-06-09 16:22:11.677585, event: started, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.677594 7f869af78700 5 -- op tracker -- , seq: 2517, time: 2014-06-09 16:22:11.677593, event: done, op: pg_info(1 pgs e243:1.631) 2014-06-09 16:22:11.677602 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677603 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677610 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677611 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677611 7f869af78700 20 osd.26 243 _dispatch 0x16065500 pg_info(1 pgs e243:0.498) v4 2014-06-09 16:22:11.677614 7f869af78700 5 -- op tracker -- , seq: 2518, time: 2014-06-09 16:22:11.446166, event: header_read, op: pg_info(1 pgs e243:0.498) 2014-06-09 16:22:11.677618 7f869af78700 5 -- op tracker -- , seq: 2518, time: 2014-06-09 16:22:11.446167, event: throttled, op: pg_info(1 pgs e243:0.498) 2014-06-09 16:22:11.677621 7f869af78700 5 -- op tracker -- , seq: 2518, time: 2014-06-09 16:22:11.446177, event: all_read, op: pg_info(1 pgs e243:0.498) 2014-06-09 16:22:11.677624 7f869af78700 5 -- op tracker -- , seq: 2518, time: 2014-06-09 16:22:11.677610, event: dispatched, op: pg_info(1 pgs e243:0.498) 2014-06-09 16:22:11.677629 7f869af78700 5 -- op tracker -- , seq: 2518, time: 2014-06-09 16:22:11.677628, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.498) 2014-06-09 16:22:11.677632 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.498) v4 from osd.3 2014-06-09 16:22:11.677633 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16065500 2014-06-09 16:22:11.677636 7f869af78700 5 -- op tracker -- , seq: 2518, time: 2014-06-09 16:22:11.677635, event: started, op: pg_info(1 pgs e243:0.498) 2014-06-09 16:22:11.677644 7f869af78700 5 -- op tracker -- , seq: 2518, time: 2014-06-09 16:22:11.677643, event: done, op: pg_info(1 pgs e243:0.498) 2014-06-09 16:22:11.677648 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677649 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677655 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677656 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677656 7f869af78700 20 osd.26 243 _dispatch 0x2fe71880 pg_info(1 pgs e243:1.38d) v4 2014-06-09 16:22:11.677659 7f869af78700 5 -- op tracker -- , seq: 2519, time: 2014-06-09 16:22:11.368743, event: header_read, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.677663 7f869af78700 5 -- op tracker -- , seq: 2519, time: 2014-06-09 16:22:11.368743, event: throttled, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.677666 7f869af78700 5 -- op tracker -- , seq: 2519, time: 2014-06-09 16:22:11.368755, event: all_read, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.677669 7f869af78700 5 -- op tracker -- , seq: 2519, time: 2014-06-09 16:22:11.677655, event: dispatched, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.677673 7f869af78700 5 -- op tracker -- , seq: 2519, time: 2014-06-09 16:22:11.677672, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.677675 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.38d) v4 from osd.1 2014-06-09 16:22:11.677677 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe71880 2014-06-09 16:22:11.677679 7f869af78700 5 -- op tracker -- , seq: 2519, time: 2014-06-09 16:22:11.677678, event: started, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.677688 7f869af78700 5 -- op tracker -- , seq: 2519, time: 2014-06-09 16:22:11.677687, event: done, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.677696 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677697 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677703 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677704 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677705 7f869af78700 20 osd.26 243 _dispatch 0x310dae00 pg_info(1 pgs e243:3.6d3) v4 2014-06-09 16:22:11.677707 7f869af78700 5 -- op tracker -- , seq: 2520, time: 2014-06-09 16:22:11.410906, event: header_read, op: pg_info(1 pgs e243:3.6d3) 2014-06-09 16:22:11.677711 7f869af78700 5 -- op tracker -- , seq: 2520, time: 2014-06-09 16:22:11.410906, event: throttled, op: pg_info(1 pgs e243:3.6d3) 2014-06-09 16:22:11.677714 7f869af78700 5 -- op tracker -- , seq: 2520, time: 2014-06-09 16:22:11.410921, event: all_read, op: pg_info(1 pgs e243:3.6d3) 2014-06-09 16:22:11.677717 7f869af78700 5 -- op tracker -- , seq: 2520, time: 2014-06-09 16:22:11.677703, event: dispatched, op: pg_info(1 pgs e243:3.6d3) 2014-06-09 16:22:11.677721 7f869af78700 5 -- op tracker -- , seq: 2520, time: 2014-06-09 16:22:11.677720, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6d3) 2014-06-09 16:22:11.677723 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6d3) v4 from osd.9 2014-06-09 16:22:11.677725 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310dae00 2014-06-09 16:22:11.677727 7f869af78700 5 -- op tracker -- , seq: 2520, time: 2014-06-09 16:22:11.677726, event: started, op: pg_info(1 pgs e243:3.6d3) 2014-06-09 16:22:11.677738 7f869af78700 5 -- op tracker -- , seq: 2520, time: 2014-06-09 16:22:11.677737, event: done, op: pg_info(1 pgs e243:3.6d3) 2014-06-09 16:22:11.677742 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677742 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677749 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677749 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677750 7f869af78700 20 osd.26 243 _dispatch 0x3104a000 pg_info(1 pgs e243:2.4e0) v4 2014-06-09 16:22:11.677753 7f869af78700 5 -- op tracker -- , seq: 2521, time: 2014-06-09 16:22:11.415457, event: header_read, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.677758 7f869af78700 5 -- op tracker -- , seq: 2521, time: 2014-06-09 16:22:11.415457, event: throttled, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.677761 7f869af78700 5 -- op tracker -- , seq: 2521, time: 2014-06-09 16:22:11.415467, event: all_read, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.677765 7f869af78700 5 -- op tracker -- , seq: 2521, time: 2014-06-09 16:22:11.677749, event: dispatched, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.677769 7f869af78700 5 -- op tracker -- , seq: 2521, time: 2014-06-09 16:22:11.677768, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.677772 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4e0) v4 from osd.8 2014-06-09 16:22:11.677773 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104a000 2014-06-09 16:22:11.677776 7f869af78700 5 -- op tracker -- , seq: 2521, time: 2014-06-09 16:22:11.677775, event: started, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.677784 7f869af78700 5 -- op tracker -- , seq: 2521, time: 2014-06-09 16:22:11.677783, event: done, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.677788 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677788 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677795 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677796 7f869af78700 20 osd.26 243 _dispatch 0x310e28c0 pg_info(1 pgs e243:1.293) v4 2014-06-09 16:22:11.677799 7f869af78700 5 -- op tracker -- , seq: 2522, time: 2014-06-09 16:22:11.400312, event: header_read, op: pg_info(1 pgs e243:1.293) 2014-06-09 16:22:11.677802 7f869af78700 5 -- op tracker -- , seq: 2522, time: 2014-06-09 16:22:11.400312, event: throttled, op: pg_info(1 pgs e243:1.293) 2014-06-09 16:22:11.677806 7f869af78700 5 -- op tracker -- , seq: 2522, time: 2014-06-09 16:22:11.400322, event: all_read, op: pg_info(1 pgs e243:1.293) 2014-06-09 16:22:11.677809 7f869af78700 5 -- op tracker -- , seq: 2522, time: 2014-06-09 16:22:11.677795, event: dispatched, op: pg_info(1 pgs e243:1.293) 2014-06-09 16:22:11.677812 7f869af78700 5 -- op tracker -- , seq: 2522, time: 2014-06-09 16:22:11.677811, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.293) 2014-06-09 16:22:11.677814 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.293) v4 from osd.6 2014-06-09 16:22:11.677816 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e28c0 2014-06-09 16:22:11.677818 7f869af78700 5 -- op tracker -- , seq: 2522, time: 2014-06-09 16:22:11.677817, event: started, op: pg_info(1 pgs e243:1.293) 2014-06-09 16:22:11.677826 7f869af78700 5 -- op tracker -- , seq: 2522, time: 2014-06-09 16:22:11.677825, event: done, op: pg_info(1 pgs e243:1.293) 2014-06-09 16:22:11.677830 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677831 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677838 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677839 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677840 7f869af78700 20 osd.26 243 _dispatch 0x311661c0 pg_info(1 pgs e243:1.a2) v4 2014-06-09 16:22:11.677842 7f869af78700 5 -- op tracker -- , seq: 2523, time: 2014-06-09 16:22:11.430079, event: header_read, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.677847 7f869af78700 5 -- op tracker -- , seq: 2523, time: 2014-06-09 16:22:11.430080, event: throttled, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.677851 7f869af78700 5 -- op tracker -- , seq: 2523, time: 2014-06-09 16:22:11.430097, event: all_read, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.677858 7f869af78700 5 -- op tracker -- , seq: 2523, time: 2014-06-09 16:22:11.677838, event: dispatched, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.677861 7f869af78700 5 -- op tracker -- , seq: 2523, time: 2014-06-09 16:22:11.677860, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.677865 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a2) v4 from osd.0 2014-06-09 16:22:11.677866 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311661c0 2014-06-09 16:22:11.677869 7f869af78700 5 -- op tracker -- , seq: 2523, time: 2014-06-09 16:22:11.677868, event: started, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.677878 7f869af78700 5 -- op tracker -- , seq: 2523, time: 2014-06-09 16:22:11.677877, event: done, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.677882 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677883 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677890 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677891 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677891 7f869af78700 20 osd.26 243 _dispatch 0x3115aa80 pg_info(1 pgs e243:3.792(2)) v4 2014-06-09 16:22:11.677894 7f869af78700 5 -- op tracker -- , seq: 2524, time: 2014-06-09 16:22:11.419821, event: header_read, op: pg_info(1 pgs e243:3.792(2)) 2014-06-09 16:22:11.677898 7f869af78700 5 -- op tracker -- , seq: 2524, time: 2014-06-09 16:22:11.419821, event: throttled, op: pg_info(1 pgs e243:3.792(2)) 2014-06-09 16:22:11.677902 7f869af78700 5 -- op tracker -- , seq: 2524, time: 2014-06-09 16:22:11.419832, event: all_read, op: pg_info(1 pgs e243:3.792(2)) 2014-06-09 16:22:11.677905 7f869af78700 5 -- op tracker -- , seq: 2524, time: 2014-06-09 16:22:11.677890, event: dispatched, op: pg_info(1 pgs e243:3.792(2)) 2014-06-09 16:22:11.677909 7f869af78700 5 -- op tracker -- , seq: 2524, time: 2014-06-09 16:22:11.677907, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.792(2)) 2014-06-09 16:22:11.677911 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.792(2)) v4 from osd.5 2014-06-09 16:22:11.677913 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115aa80 2014-06-09 16:22:11.677915 7f869af78700 5 -- op tracker -- , seq: 2524, time: 2014-06-09 16:22:11.677914, event: started, op: pg_info(1 pgs e243:3.792(2)) 2014-06-09 16:22:11.677924 7f869af78700 5 -- op tracker -- , seq: 2524, time: 2014-06-09 16:22:11.677923, event: done, op: pg_info(1 pgs e243:3.792(2)) 2014-06-09 16:22:11.677928 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677928 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677935 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677936 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677936 7f869af78700 20 osd.26 243 _dispatch 0x3043a540 pg_info(1 pgs e243:1.669) v4 2014-06-09 16:22:11.677939 7f869af78700 5 -- op tracker -- , seq: 2525, time: 2014-06-09 16:22:11.302817, event: header_read, op: pg_info(1 pgs e243:1.669) 2014-06-09 16:22:11.677943 7f869af78700 5 -- op tracker -- , seq: 2525, time: 2014-06-09 16:22:11.302817, event: throttled, op: pg_info(1 pgs e243:1.669) 2014-06-09 16:22:11.677946 7f869af78700 5 -- op tracker -- , seq: 2525, time: 2014-06-09 16:22:11.302844, event: all_read, op: pg_info(1 pgs e243:1.669) 2014-06-09 16:22:11.677949 7f869af78700 5 -- op tracker -- , seq: 2525, time: 2014-06-09 16:22:11.677935, event: dispatched, op: pg_info(1 pgs e243:1.669) 2014-06-09 16:22:11.677956 7f869af78700 5 -- op tracker -- , seq: 2525, time: 2014-06-09 16:22:11.677955, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.669) 2014-06-09 16:22:11.677959 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.669) v4 from osd.2 2014-06-09 16:22:11.677960 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3043a540 2014-06-09 16:22:11.677963 7f869af78700 5 -- op tracker -- , seq: 2525, time: 2014-06-09 16:22:11.677962, event: started, op: pg_info(1 pgs e243:1.669) 2014-06-09 16:22:11.677971 7f869af78700 5 -- op tracker -- , seq: 2525, time: 2014-06-09 16:22:11.677970, event: done, op: pg_info(1 pgs e243:1.669) 2014-06-09 16:22:11.677975 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677976 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677983 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.677983 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.677984 7f869af78700 20 osd.26 243 _dispatch 0xe06e380 pg_info(1 pgs e243:0.35) v4 2014-06-09 16:22:11.677987 7f869af78700 5 -- op tracker -- , seq: 2526, time: 2014-06-09 16:22:11.395970, event: header_read, op: pg_info(1 pgs e243:0.35) 2014-06-09 16:22:11.677990 7f869af78700 5 -- op tracker -- , seq: 2526, time: 2014-06-09 16:22:11.395971, event: throttled, op: pg_info(1 pgs e243:0.35) 2014-06-09 16:22:11.677994 7f869af78700 5 -- op tracker -- , seq: 2526, time: 2014-06-09 16:22:11.395982, event: all_read, op: pg_info(1 pgs e243:0.35) 2014-06-09 16:22:11.677997 7f869af78700 5 -- op tracker -- , seq: 2526, time: 2014-06-09 16:22:11.677983, event: dispatched, op: pg_info(1 pgs e243:0.35) 2014-06-09 16:22:11.678000 7f869af78700 5 -- op tracker -- , seq: 2526, time: 2014-06-09 16:22:11.677999, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.35) 2014-06-09 16:22:11.678003 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.35) v4 from osd.4 2014-06-09 16:22:11.678004 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xe06e380 2014-06-09 16:22:11.678006 7f869af78700 5 -- op tracker -- , seq: 2526, time: 2014-06-09 16:22:11.678005, event: started, op: pg_info(1 pgs e243:0.35) 2014-06-09 16:22:11.678018 7f869af78700 5 -- op tracker -- , seq: 2526, time: 2014-06-09 16:22:11.678017, event: done, op: pg_info(1 pgs e243:0.35) 2014-06-09 16:22:11.678023 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678024 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678030 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678031 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678031 7f869af78700 20 osd.26 243 _dispatch 0x310d7c00 pg_info(1 pgs e243:3.62a) v4 2014-06-09 16:22:11.678034 7f869af78700 5 -- op tracker -- , seq: 2527, time: 2014-06-09 16:22:11.398590, event: header_read, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.678038 7f869af78700 5 -- op tracker -- , seq: 2527, time: 2014-06-09 16:22:11.398590, event: throttled, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.678042 7f869af78700 5 -- op tracker -- , seq: 2527, time: 2014-06-09 16:22:11.398602, event: all_read, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.678045 7f869af78700 5 -- op tracker -- , seq: 2527, time: 2014-06-09 16:22:11.678030, event: dispatched, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.678049 7f869af78700 5 -- op tracker -- , seq: 2527, time: 2014-06-09 16:22:11.678048, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.678051 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.62a) v4 from osd.7 2014-06-09 16:22:11.678052 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d7c00 2014-06-09 16:22:11.678055 7f869af78700 5 -- op tracker -- , seq: 2527, time: 2014-06-09 16:22:11.678054, event: started, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.678063 7f869af78700 5 -- op tracker -- , seq: 2527, time: 2014-06-09 16:22:11.678062, event: done, op: pg_info(1 pgs e243:3.62a) 2014-06-09 16:22:11.678068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678069 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678077 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678078 7f869af78700 20 osd.26 243 _dispatch 0x2f031dc0 pg_info(1 pgs e243:0.28a) v4 2014-06-09 16:22:11.678081 7f869af78700 5 -- op tracker -- , seq: 2528, time: 2014-06-09 16:22:11.446181, event: header_read, op: pg_info(1 pgs e243:0.28a) 2014-06-09 16:22:11.678085 7f869af78700 5 -- op tracker -- , seq: 2528, time: 2014-06-09 16:22:11.446182, event: throttled, op: pg_info(1 pgs e243:0.28a) 2014-06-09 16:22:11.678092 7f869af78700 5 -- op tracker -- , seq: 2528, time: 2014-06-09 16:22:11.446194, event: all_read, op: pg_info(1 pgs e243:0.28a) 2014-06-09 16:22:11.678095 7f869af78700 5 -- op tracker -- , seq: 2528, time: 2014-06-09 16:22:11.678077, event: dispatched, op: pg_info(1 pgs e243:0.28a) 2014-06-09 16:22:11.678099 7f869af78700 5 -- op tracker -- , seq: 2528, time: 2014-06-09 16:22:11.678098, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.28a) 2014-06-09 16:22:11.678101 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.28a) v4 from osd.3 2014-06-09 16:22:11.678103 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f031dc0 2014-06-09 16:22:11.678105 7f869af78700 5 -- op tracker -- , seq: 2528, time: 2014-06-09 16:22:11.678104, event: started, op: pg_info(1 pgs e243:0.28a) 2014-06-09 16:22:11.678114 7f869af78700 5 -- op tracker -- , seq: 2528, time: 2014-06-09 16:22:11.678113, event: done, op: pg_info(1 pgs e243:0.28a) 2014-06-09 16:22:11.678118 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678118 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678125 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678126 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678126 7f869af78700 20 osd.26 243 _dispatch 0xe06ee00 pg_info(1 pgs e243:2.364) v4 2014-06-09 16:22:11.678129 7f869af78700 5 -- op tracker -- , seq: 2529, time: 2014-06-09 16:22:11.369247, event: header_read, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.678134 7f869af78700 5 -- op tracker -- , seq: 2529, time: 2014-06-09 16:22:11.369248, event: throttled, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.678137 7f869af78700 5 -- op tracker -- , seq: 2529, time: 2014-06-09 16:22:11.369263, event: all_read, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.678144 7f869af78700 5 -- op tracker -- , seq: 2529, time: 2014-06-09 16:22:11.678125, event: dispatched, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.678148 7f869af78700 5 -- op tracker -- , seq: 2529, time: 2014-06-09 16:22:11.678147, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.678152 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.364) v4 from osd.1 2014-06-09 16:22:11.678153 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xe06ee00 2014-06-09 16:22:11.678156 7f869af78700 5 -- op tracker -- , seq: 2529, time: 2014-06-09 16:22:11.678155, event: started, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.678166 7f869af78700 5 -- op tracker -- , seq: 2529, time: 2014-06-09 16:22:11.678165, event: done, op: pg_info(1 pgs e243:2.364) 2014-06-09 16:22:11.678170 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678171 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678177 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678178 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678179 7f869af78700 20 osd.26 243 _dispatch 0x310dac40 pg_info(1 pgs e243:3.4c7) v4 2014-06-09 16:22:11.678181 7f869af78700 5 -- op tracker -- , seq: 2530, time: 2014-06-09 16:22:11.411632, event: header_read, op: pg_info(1 pgs e243:3.4c7) 2014-06-09 16:22:11.678185 7f869af78700 5 -- op tracker -- , seq: 2530, time: 2014-06-09 16:22:11.411633, event: throttled, op: pg_info(1 pgs e243:3.4c7) 2014-06-09 16:22:11.678188 7f869af78700 5 -- op tracker -- , seq: 2530, time: 2014-06-09 16:22:11.411646, event: all_read, op: pg_info(1 pgs e243:3.4c7) 2014-06-09 16:22:11.678191 7f869af78700 5 -- op tracker -- , seq: 2530, time: 2014-06-09 16:22:11.678177, event: dispatched, op: pg_info(1 pgs e243:3.4c7) 2014-06-09 16:22:11.678195 7f869af78700 5 -- op tracker -- , seq: 2530, time: 2014-06-09 16:22:11.678194, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c7) 2014-06-09 16:22:11.678197 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c7) v4 from osd.9 2014-06-09 16:22:11.678199 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310dac40 2014-06-09 16:22:11.678201 7f869af78700 5 -- op tracker -- , seq: 2530, time: 2014-06-09 16:22:11.678200, event: started, op: pg_info(1 pgs e243:3.4c7) 2014-06-09 16:22:11.678211 7f869af78700 5 -- op tracker -- , seq: 2530, time: 2014-06-09 16:22:11.678210, event: done, op: pg_info(1 pgs e243:3.4c7) 2014-06-09 16:22:11.678215 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678216 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678223 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678223 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678224 7f869af78700 20 osd.26 243 _dispatch 0x31157dc0 pg_info(1 pgs e243:0.13a) v4 2014-06-09 16:22:11.678227 7f869af78700 5 -- op tracker -- , seq: 2531, time: 2014-06-09 16:22:11.415471, event: header_read, op: pg_info(1 pgs e243:0.13a) 2014-06-09 16:22:11.678230 7f869af78700 5 -- op tracker -- , seq: 2531, time: 2014-06-09 16:22:11.415471, event: throttled, op: pg_info(1 pgs e243:0.13a) 2014-06-09 16:22:11.678234 7f869af78700 5 -- op tracker -- , seq: 2531, time: 2014-06-09 16:22:11.415485, event: all_read, op: pg_info(1 pgs e243:0.13a) 2014-06-09 16:22:11.678237 7f869af78700 5 -- op tracker -- , seq: 2531, time: 2014-06-09 16:22:11.678223, event: dispatched, op: pg_info(1 pgs e243:0.13a) 2014-06-09 16:22:11.678241 7f869af78700 5 -- op tracker -- , seq: 2531, time: 2014-06-09 16:22:11.678240, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.13a) 2014-06-09 16:22:11.678243 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.13a) v4 from osd.8 2014-06-09 16:22:11.678244 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31157dc0 2014-06-09 16:22:11.678247 7f869af78700 5 -- op tracker -- , seq: 2531, time: 2014-06-09 16:22:11.678246, event: started, op: pg_info(1 pgs e243:0.13a) 2014-06-09 16:22:11.678255 7f869af78700 5 -- op tracker -- , seq: 2531, time: 2014-06-09 16:22:11.678254, event: done, op: pg_info(1 pgs e243:0.13a) 2014-06-09 16:22:11.678259 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678260 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678267 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678268 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678269 7f869af78700 20 osd.26 243 _dispatch 0x30889e00 pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.678272 7f869af78700 5 -- op tracker -- , seq: 2532, time: 2014-06-09 16:22:11.400326, event: header_read, op: pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.678277 7f869af78700 5 -- op tracker -- , seq: 2532, time: 2014-06-09 16:22:11.400326, event: throttled, op: pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.678285 7f869af78700 5 -- op tracker -- , seq: 2532, time: 2014-06-09 16:22:11.400345, event: all_read, op: pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.678292 7f869af78700 5 -- op tracker -- , seq: 2532, time: 2014-06-09 16:22:11.678267, event: dispatched, op: pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.678296 7f869af78700 5 -- op tracker -- , seq: 2532, time: 2014-06-09 16:22:11.678295, event: waiting_for_osdmap, op: pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.678299 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) v4 from osd.6 2014-06-09 16:22:11.678301 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30889e00 2014-06-09 16:22:11.678304 7f869af78700 5 -- op tracker -- , seq: 2532, time: 2014-06-09 16:22:11.678303, event: started, op: pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.678311 7f869af78700 5 -- op tracker -- , seq: 2532, time: 2014-06-09 16:22:11.678310, event: done, op: pg_log(3.428 epoch 243 log log((213'561,214'563], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.678315 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678316 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678323 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678324 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678324 7f869af78700 20 osd.26 243 _dispatch 0x31166700 pg_info(1 pgs e243:2.37a) v4 2014-06-09 16:22:11.678327 7f869af78700 5 -- op tracker -- , seq: 2533, time: 2014-06-09 16:22:11.430105, event: header_read, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.678331 7f869af78700 5 -- op tracker -- , seq: 2533, time: 2014-06-09 16:22:11.430106, event: throttled, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.678334 7f869af78700 5 -- op tracker -- , seq: 2533, time: 2014-06-09 16:22:11.430116, event: all_read, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.678337 7f869af78700 5 -- op tracker -- , seq: 2533, time: 2014-06-09 16:22:11.678323, event: dispatched, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.678341 7f869af78700 5 -- op tracker -- , seq: 2533, time: 2014-06-09 16:22:11.678340, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.678343 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.37a) v4 from osd.0 2014-06-09 16:22:11.678345 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31166700 2014-06-09 16:22:11.678347 7f869af78700 5 -- op tracker -- , seq: 2533, time: 2014-06-09 16:22:11.678346, event: started, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.678357 7f869af78700 5 -- op tracker -- , seq: 2533, time: 2014-06-09 16:22:11.678356, event: done, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.678361 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678362 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678369 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678370 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678371 7f869af78700 20 osd.26 243 _dispatch 0x30e0f000 pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) v4 2014-06-09 16:22:11.678374 7f869af78700 5 -- op tracker -- , seq: 2534, time: 2014-06-09 16:22:11.419839, event: header_read, op: pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.678382 7f869af78700 5 -- op tracker -- , seq: 2534, time: 2014-06-09 16:22:11.419839, event: throttled, op: pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.678386 7f869af78700 5 -- op tracker -- , seq: 2534, time: 2014-06-09 16:22:11.419853, event: all_read, op: pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.678390 7f869af78700 5 -- op tracker -- , seq: 2534, time: 2014-06-09 16:22:11.678369, event: dispatched, op: pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.678399 7f869af78700 5 -- op tracker -- , seq: 2534, time: 2014-06-09 16:22:11.678397, event: waiting_for_osdmap, op: pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.678408 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) v4 from osd.5 2014-06-09 16:22:11.678410 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e0f000 2014-06-09 16:22:11.678414 7f869af78700 5 -- op tracker -- , seq: 2534, time: 2014-06-09 16:22:11.678412, event: started, op: pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.678421 7f869af78700 5 -- op tracker -- , seq: 2534, time: 2014-06-09 16:22:11.678419, event: done, op: pg_log(3.4b9 epoch 243 log log((214'556,214'558], crt=0'0) query_epoch 242) 2014-06-09 16:22:11.678426 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678433 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678434 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678435 7f869af78700 20 osd.26 243 _dispatch 0x19805180 pg_info(1 pgs e243:3.3ae) v4 2014-06-09 16:22:11.678437 7f869af78700 5 -- op tracker -- , seq: 2535, time: 2014-06-09 16:22:11.302852, event: header_read, op: pg_info(1 pgs e243:3.3ae) 2014-06-09 16:22:11.678441 7f869af78700 5 -- op tracker -- , seq: 2535, time: 2014-06-09 16:22:11.302853, event: throttled, op: pg_info(1 pgs e243:3.3ae) 2014-06-09 16:22:11.678444 7f869af78700 5 -- op tracker -- , seq: 2535, time: 2014-06-09 16:22:11.302859, event: all_read, op: pg_info(1 pgs e243:3.3ae) 2014-06-09 16:22:11.678449 7f869af78700 5 -- op tracker -- , seq: 2535, time: 2014-06-09 16:22:11.678433, event: dispatched, op: pg_info(1 pgs e243:3.3ae) 2014-06-09 16:22:11.678453 7f869af78700 5 -- op tracker -- , seq: 2535, time: 2014-06-09 16:22:11.678452, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3ae) 2014-06-09 16:22:11.678459 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3ae) v4 from osd.2 2014-06-09 16:22:11.678461 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19805180 2014-06-09 16:22:11.678463 7f869af78700 5 -- op tracker -- , seq: 2535, time: 2014-06-09 16:22:11.678462, event: started, op: pg_info(1 pgs e243:3.3ae) 2014-06-09 16:22:11.678472 7f869af78700 5 -- op tracker -- , seq: 2535, time: 2014-06-09 16:22:11.678471, event: done, op: pg_info(1 pgs e243:3.3ae) 2014-06-09 16:22:11.678476 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678477 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678484 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678485 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678485 7f869af78700 20 osd.26 243 _dispatch 0x15458540 pg_info(1 pgs e243:2.316) v4 2014-06-09 16:22:11.678488 7f869af78700 5 -- op tracker -- , seq: 2536, time: 2014-06-09 16:22:11.395988, event: header_read, op: pg_info(1 pgs e243:2.316) 2014-06-09 16:22:11.678495 7f869af78700 5 -- op tracker -- , seq: 2536, time: 2014-06-09 16:22:11.395989, event: throttled, op: pg_info(1 pgs e243:2.316) 2014-06-09 16:22:11.678498 7f869af78700 5 -- op tracker -- , seq: 2536, time: 2014-06-09 16:22:11.396005, event: all_read, op: pg_info(1 pgs e243:2.316) 2014-06-09 16:22:11.678503 7f869af78700 5 -- op tracker -- , seq: 2536, time: 2014-06-09 16:22:11.678484, event: dispatched, op: pg_info(1 pgs e243:2.316) 2014-06-09 16:22:11.678507 7f869af78700 5 -- op tracker -- , seq: 2536, time: 2014-06-09 16:22:11.678506, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.316) 2014-06-09 16:22:11.678509 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.316) v4 from osd.4 2014-06-09 16:22:11.678511 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15458540 2014-06-09 16:22:11.678513 7f869af78700 5 -- op tracker -- , seq: 2536, time: 2014-06-09 16:22:11.678512, event: started, op: pg_info(1 pgs e243:2.316) 2014-06-09 16:22:11.678522 7f869af78700 5 -- op tracker -- , seq: 2536, time: 2014-06-09 16:22:11.678521, event: done, op: pg_info(1 pgs e243:2.316) 2014-06-09 16:22:11.678526 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678526 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678533 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678534 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678535 7f869af78700 20 osd.26 243 _dispatch 0x310e9dc0 pg_info(1 pgs e243:2.744) v4 2014-06-09 16:22:11.678537 7f869af78700 5 -- op tracker -- , seq: 2537, time: 2014-06-09 16:22:11.403675, event: header_read, op: pg_info(1 pgs e243:2.744) 2014-06-09 16:22:11.678541 7f869af78700 5 -- op tracker -- , seq: 2537, time: 2014-06-09 16:22:11.403676, event: throttled, op: pg_info(1 pgs e243:2.744) 2014-06-09 16:22:11.678544 7f869af78700 5 -- op tracker -- , seq: 2537, time: 2014-06-09 16:22:11.403691, event: all_read, op: pg_info(1 pgs e243:2.744) 2014-06-09 16:22:11.678548 7f869af78700 5 -- op tracker -- , seq: 2537, time: 2014-06-09 16:22:11.678533, event: dispatched, op: pg_info(1 pgs e243:2.744) 2014-06-09 16:22:11.678552 7f869af78700 5 -- op tracker -- , seq: 2537, time: 2014-06-09 16:22:11.678551, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.744) 2014-06-09 16:22:11.678554 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.744) v4 from osd.7 2014-06-09 16:22:11.678556 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e9dc0 2014-06-09 16:22:11.678558 7f869af78700 5 -- op tracker -- , seq: 2537, time: 2014-06-09 16:22:11.678557, event: started, op: pg_info(1 pgs e243:2.744) 2014-06-09 16:22:11.678568 7f869af78700 5 -- op tracker -- , seq: 2537, time: 2014-06-09 16:22:11.678567, event: done, op: pg_info(1 pgs e243:2.744) 2014-06-09 16:22:11.678572 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678573 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678580 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678581 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678581 7f869af78700 20 osd.26 243 _dispatch 0x2f031c00 pg_info(1 pgs e243:0.158) v4 2014-06-09 16:22:11.678584 7f869af78700 5 -- op tracker -- , seq: 2538, time: 2014-06-09 16:22:11.446699, event: header_read, op: pg_info(1 pgs e243:0.158) 2014-06-09 16:22:11.678588 7f869af78700 5 -- op tracker -- , seq: 2538, time: 2014-06-09 16:22:11.446699, event: throttled, op: pg_info(1 pgs e243:0.158) 2014-06-09 16:22:11.678591 7f869af78700 5 -- op tracker -- , seq: 2538, time: 2014-06-09 16:22:11.446712, event: all_read, op: pg_info(1 pgs e243:0.158) 2014-06-09 16:22:11.678594 7f869af78700 5 -- op tracker -- , seq: 2538, time: 2014-06-09 16:22:11.678580, event: dispatched, op: pg_info(1 pgs e243:0.158) 2014-06-09 16:22:11.678598 7f869af78700 5 -- op tracker -- , seq: 2538, time: 2014-06-09 16:22:11.678597, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.158) 2014-06-09 16:22:11.678600 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.158) v4 from osd.3 2014-06-09 16:22:11.678602 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f031c00 2014-06-09 16:22:11.678604 7f869af78700 5 -- op tracker -- , seq: 2538, time: 2014-06-09 16:22:11.678603, event: started, op: pg_info(1 pgs e243:0.158) 2014-06-09 16:22:11.678612 7f869af78700 5 -- op tracker -- , seq: 2538, time: 2014-06-09 16:22:11.678611, event: done, op: pg_info(1 pgs e243:0.158) 2014-06-09 16:22:11.678616 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678616 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678623 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678624 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678625 7f869af78700 20 osd.26 243 _dispatch 0x1609d880 pg_info(1 pgs e243:0.369) v4 2014-06-09 16:22:11.678627 7f869af78700 5 -- op tracker -- , seq: 2539, time: 2014-06-09 16:22:11.369425, event: header_read, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.678631 7f869af78700 5 -- op tracker -- , seq: 2539, time: 2014-06-09 16:22:11.369426, event: throttled, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.678638 7f869af78700 5 -- op tracker -- , seq: 2539, time: 2014-06-09 16:22:11.369533, event: all_read, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.678642 7f869af78700 5 -- op tracker -- , seq: 2539, time: 2014-06-09 16:22:11.678623, event: dispatched, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.678645 7f869af78700 5 -- op tracker -- , seq: 2539, time: 2014-06-09 16:22:11.678644, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.678648 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.369) v4 from osd.1 2014-06-09 16:22:11.678649 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1609d880 2014-06-09 16:22:11.678652 7f869af78700 5 -- op tracker -- , seq: 2539, time: 2014-06-09 16:22:11.678651, event: started, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.678661 7f869af78700 5 -- op tracker -- , seq: 2539, time: 2014-06-09 16:22:11.678660, event: done, op: pg_info(1 pgs e243:0.369) 2014-06-09 16:22:11.678665 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678666 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678673 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678674 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678674 7f869af78700 20 osd.26 243 _dispatch 0x310daa80 pg_info(1 pgs e243:0.5ed) v4 2014-06-09 16:22:11.678677 7f869af78700 5 -- op tracker -- , seq: 2540, time: 2014-06-09 16:22:11.411655, event: header_read, op: pg_info(1 pgs e243:0.5ed) 2014-06-09 16:22:11.678681 7f869af78700 5 -- op tracker -- , seq: 2540, time: 2014-06-09 16:22:11.411655, event: throttled, op: pg_info(1 pgs e243:0.5ed) 2014-06-09 16:22:11.678684 7f869af78700 5 -- op tracker -- , seq: 2540, time: 2014-06-09 16:22:11.411666, event: all_read, op: pg_info(1 pgs e243:0.5ed) 2014-06-09 16:22:11.678688 7f869af78700 5 -- op tracker -- , seq: 2540, time: 2014-06-09 16:22:11.678673, event: dispatched, op: pg_info(1 pgs e243:0.5ed) 2014-06-09 16:22:11.678692 7f869af78700 5 -- op tracker -- , seq: 2540, time: 2014-06-09 16:22:11.678691, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5ed) 2014-06-09 16:22:11.678697 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5ed) v4 from osd.9 2014-06-09 16:22:11.678699 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310daa80 2014-06-09 16:22:11.678702 7f869af78700 5 -- op tracker -- , seq: 2540, time: 2014-06-09 16:22:11.678701, event: started, op: pg_info(1 pgs e243:0.5ed) 2014-06-09 16:22:11.678710 7f869af78700 5 -- op tracker -- , seq: 2540, time: 2014-06-09 16:22:11.678709, event: done, op: pg_info(1 pgs e243:0.5ed) 2014-06-09 16:22:11.678715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678716 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678723 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678723 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678724 7f869af78700 20 osd.26 243 _dispatch 0x31157c00 pg_info(1 pgs e243:3.41b) v4 2014-06-09 16:22:11.678726 7f869af78700 5 -- op tracker -- , seq: 2541, time: 2014-06-09 16:22:11.415490, event: header_read, op: pg_info(1 pgs e243:3.41b) 2014-06-09 16:22:11.678730 7f869af78700 5 -- op tracker -- , seq: 2541, time: 2014-06-09 16:22:11.415490, event: throttled, op: pg_info(1 pgs e243:3.41b) 2014-06-09 16:22:11.678737 7f869af78700 5 -- op tracker -- , seq: 2541, time: 2014-06-09 16:22:11.415500, event: all_read, op: pg_info(1 pgs e243:3.41b) 2014-06-09 16:22:11.678740 7f869af78700 5 -- op tracker -- , seq: 2541, time: 2014-06-09 16:22:11.678722, event: dispatched, op: pg_info(1 pgs e243:3.41b) 2014-06-09 16:22:11.678744 7f869af78700 5 -- op tracker -- , seq: 2541, time: 2014-06-09 16:22:11.678743, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.41b) 2014-06-09 16:22:11.678746 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.41b) v4 from osd.8 2014-06-09 16:22:11.678748 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31157c00 2014-06-09 16:22:11.678750 7f869af78700 5 -- op tracker -- , seq: 2541, time: 2014-06-09 16:22:11.678749, event: started, op: pg_info(1 pgs e243:3.41b) 2014-06-09 16:22:11.678759 7f869af78700 5 -- op tracker -- , seq: 2541, time: 2014-06-09 16:22:11.678758, event: done, op: pg_info(1 pgs e243:3.41b) 2014-06-09 16:22:11.678763 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678764 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678770 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678771 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678771 7f869af78700 20 osd.26 243 _dispatch 0x310e2700 pg_info(1 pgs e243:0.6b4) v4 2014-06-09 16:22:11.678774 7f869af78700 5 -- op tracker -- , seq: 2542, time: 2014-06-09 16:22:11.418825, event: header_read, op: pg_info(1 pgs e243:0.6b4) 2014-06-09 16:22:11.678778 7f869af78700 5 -- op tracker -- , seq: 2542, time: 2014-06-09 16:22:11.418826, event: throttled, op: pg_info(1 pgs e243:0.6b4) 2014-06-09 16:22:11.678781 7f869af78700 5 -- op tracker -- , seq: 2542, time: 2014-06-09 16:22:11.418845, event: all_read, op: pg_info(1 pgs e243:0.6b4) 2014-06-09 16:22:11.678788 7f869af78700 5 -- op tracker -- , seq: 2542, time: 2014-06-09 16:22:11.678770, event: dispatched, op: pg_info(1 pgs e243:0.6b4) 2014-06-09 16:22:11.678792 7f869af78700 5 -- op tracker -- , seq: 2542, time: 2014-06-09 16:22:11.678791, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6b4) 2014-06-09 16:22:11.678795 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6b4) v4 from osd.6 2014-06-09 16:22:11.678797 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e2700 2014-06-09 16:22:11.678799 7f869af78700 5 -- op tracker -- , seq: 2542, time: 2014-06-09 16:22:11.678798, event: started, op: pg_info(1 pgs e243:0.6b4) 2014-06-09 16:22:11.678809 7f869af78700 5 -- op tracker -- , seq: 2542, time: 2014-06-09 16:22:11.678808, event: done, op: pg_info(1 pgs e243:0.6b4) 2014-06-09 16:22:11.678813 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678814 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678822 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678823 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678823 7f869af78700 20 osd.26 243 _dispatch 0x31167500 pg_info(1 pgs e243:1.a2) v4 2014-06-09 16:22:11.678826 7f869af78700 5 -- op tracker -- , seq: 2543, time: 2014-06-09 16:22:11.430123, event: header_read, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.678830 7f869af78700 5 -- op tracker -- , seq: 2543, time: 2014-06-09 16:22:11.430124, event: throttled, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.678833 7f869af78700 5 -- op tracker -- , seq: 2543, time: 2014-06-09 16:22:11.430133, event: all_read, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.678836 7f869af78700 5 -- op tracker -- , seq: 2543, time: 2014-06-09 16:22:11.678822, event: dispatched, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.678841 7f869af78700 5 -- op tracker -- , seq: 2543, time: 2014-06-09 16:22:11.678840, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.678843 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a2) v4 from osd.0 2014-06-09 16:22:11.678845 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31167500 2014-06-09 16:22:11.678847 7f869af78700 5 -- op tracker -- , seq: 2543, time: 2014-06-09 16:22:11.678846, event: started, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.678855 7f869af78700 5 -- op tracker -- , seq: 2543, time: 2014-06-09 16:22:11.678854, event: done, op: pg_info(1 pgs e243:1.a2) 2014-06-09 16:22:11.678859 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678860 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678866 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678867 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678868 7f869af78700 20 osd.26 243 _dispatch 0x3115a8c0 pg_info(1 pgs e243:2.6cd) v4 2014-06-09 16:22:11.678870 7f869af78700 5 -- op tracker -- , seq: 2544, time: 2014-06-09 16:22:11.419857, event: header_read, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.678874 7f869af78700 5 -- op tracker -- , seq: 2544, time: 2014-06-09 16:22:11.419858, event: throttled, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.678877 7f869af78700 5 -- op tracker -- , seq: 2544, time: 2014-06-09 16:22:11.419867, event: all_read, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.678884 7f869af78700 5 -- op tracker -- , seq: 2544, time: 2014-06-09 16:22:11.678866, event: dispatched, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.678889 7f869af78700 5 -- op tracker -- , seq: 2544, time: 2014-06-09 16:22:11.678888, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.678891 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6cd) v4 from osd.5 2014-06-09 16:22:11.678893 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115a8c0 2014-06-09 16:22:11.678895 7f869af78700 5 -- op tracker -- , seq: 2544, time: 2014-06-09 16:22:11.678894, event: started, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.678904 7f869af78700 5 -- op tracker -- , seq: 2544, time: 2014-06-09 16:22:11.678903, event: done, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.678908 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678909 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678915 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678915 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678916 7f869af78700 20 osd.26 243 _dispatch 0x10d0d340 pg_info(1 pgs e243:0.1f) v4 2014-06-09 16:22:11.678919 7f869af78700 5 -- op tracker -- , seq: 2545, time: 2014-06-09 16:22:11.302863, event: header_read, op: pg_info(1 pgs e243:0.1f) 2014-06-09 16:22:11.678922 7f869af78700 5 -- op tracker -- , seq: 2545, time: 2014-06-09 16:22:11.302871, event: throttled, op: pg_info(1 pgs e243:0.1f) 2014-06-09 16:22:11.678926 7f869af78700 5 -- op tracker -- , seq: 2545, time: 2014-06-09 16:22:11.302885, event: all_read, op: pg_info(1 pgs e243:0.1f) 2014-06-09 16:22:11.678930 7f869af78700 5 -- op tracker -- , seq: 2545, time: 2014-06-09 16:22:11.678915, event: dispatched, op: pg_info(1 pgs e243:0.1f) 2014-06-09 16:22:11.678934 7f869af78700 5 -- op tracker -- , seq: 2545, time: 2014-06-09 16:22:11.678933, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1f) 2014-06-09 16:22:11.678940 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1f) v4 from osd.2 2014-06-09 16:22:11.678942 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d0d340 2014-06-09 16:22:11.678944 7f869af78700 5 -- op tracker -- , seq: 2545, time: 2014-06-09 16:22:11.678943, event: started, op: pg_info(1 pgs e243:0.1f) 2014-06-09 16:22:11.678953 7f869af78700 5 -- op tracker -- , seq: 2545, time: 2014-06-09 16:22:11.678952, event: done, op: pg_info(1 pgs e243:0.1f) 2014-06-09 16:22:11.678957 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678958 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678964 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.678965 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.678965 7f869af78700 20 osd.26 243 _dispatch 0x1405ce00 pg_info(1 pgs e243:1.30e) v4 2014-06-09 16:22:11.678968 7f869af78700 5 -- op tracker -- , seq: 2546, time: 2014-06-09 16:22:11.396010, event: header_read, op: pg_info(1 pgs e243:1.30e) 2014-06-09 16:22:11.678975 7f869af78700 5 -- op tracker -- , seq: 2546, time: 2014-06-09 16:22:11.396010, event: throttled, op: pg_info(1 pgs e243:1.30e) 2014-06-09 16:22:11.678978 7f869af78700 5 -- op tracker -- , seq: 2546, time: 2014-06-09 16:22:11.396020, event: all_read, op: pg_info(1 pgs e243:1.30e) 2014-06-09 16:22:11.678982 7f869af78700 5 -- op tracker -- , seq: 2546, time: 2014-06-09 16:22:11.678964, event: dispatched, op: pg_info(1 pgs e243:1.30e) 2014-06-09 16:22:11.678985 7f869af78700 5 -- op tracker -- , seq: 2546, time: 2014-06-09 16:22:11.678984, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.30e) 2014-06-09 16:22:11.678987 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.30e) v4 from osd.4 2014-06-09 16:22:11.678989 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405ce00 2014-06-09 16:22:11.678992 7f869af78700 5 -- op tracker -- , seq: 2546, time: 2014-06-09 16:22:11.678991, event: started, op: pg_info(1 pgs e243:1.30e) 2014-06-09 16:22:11.679000 7f869af78700 5 -- op tracker -- , seq: 2546, time: 2014-06-09 16:22:11.678999, event: done, op: pg_info(1 pgs e243:1.30e) 2014-06-09 16:22:11.679004 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679005 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679011 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679012 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679013 7f869af78700 20 osd.26 243 _dispatch 0x30bd96c0 pg_info(1 pgs e243:1.676) v4 2014-06-09 16:22:11.679015 7f869af78700 5 -- op tracker -- , seq: 2547, time: 2014-06-09 16:22:11.411545, event: header_read, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679019 7f869af78700 5 -- op tracker -- , seq: 2547, time: 2014-06-09 16:22:11.411546, event: throttled, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679022 7f869af78700 5 -- op tracker -- , seq: 2547, time: 2014-06-09 16:22:11.411559, event: all_read, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679026 7f869af78700 5 -- op tracker -- , seq: 2547, time: 2014-06-09 16:22:11.679011, event: dispatched, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679029 7f869af78700 5 -- op tracker -- , seq: 2547, time: 2014-06-09 16:22:11.679028, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679032 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.676) v4 from osd.7 2014-06-09 16:22:11.679033 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd96c0 2014-06-09 16:22:11.679035 7f869af78700 5 -- op tracker -- , seq: 2547, time: 2014-06-09 16:22:11.679034, event: started, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679044 7f869af78700 5 -- op tracker -- , seq: 2547, time: 2014-06-09 16:22:11.679043, event: done, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679047 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679048 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679054 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679055 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679056 7f869af78700 20 osd.26 243 _dispatch 0x2f031a40 pg_info(1 pgs e243:2.292) v4 2014-06-09 16:22:11.679058 7f869af78700 5 -- op tracker -- , seq: 2548, time: 2014-06-09 16:22:11.446801, event: header_read, op: pg_info(1 pgs e243:2.292) 2014-06-09 16:22:11.679062 7f869af78700 5 -- op tracker -- , seq: 2548, time: 2014-06-09 16:22:11.446802, event: throttled, op: pg_info(1 pgs e243:2.292) 2014-06-09 16:22:11.679068 7f869af78700 5 -- op tracker -- , seq: 2548, time: 2014-06-09 16:22:11.446814, event: all_read, op: pg_info(1 pgs e243:2.292) 2014-06-09 16:22:11.679072 7f869af78700 5 -- op tracker -- , seq: 2548, time: 2014-06-09 16:22:11.679054, event: dispatched, op: pg_info(1 pgs e243:2.292) 2014-06-09 16:22:11.679075 7f869af78700 5 -- op tracker -- , seq: 2548, time: 2014-06-09 16:22:11.679074, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.292) 2014-06-09 16:22:11.679078 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.292) v4 from osd.3 2014-06-09 16:22:11.679079 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f031a40 2014-06-09 16:22:11.679082 7f869af78700 5 -- op tracker -- , seq: 2548, time: 2014-06-09 16:22:11.679081, event: started, op: pg_info(1 pgs e243:2.292) 2014-06-09 16:22:11.679090 7f869af78700 5 -- op tracker -- , seq: 2548, time: 2014-06-09 16:22:11.679089, event: done, op: pg_info(1 pgs e243:2.292) 2014-06-09 16:22:11.679094 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679094 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679101 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679102 7f869af78700 20 osd.26 243 _dispatch 0x19fdaa80 pg_info(1 pgs e243:2.372) v4 2014-06-09 16:22:11.679104 7f869af78700 5 -- op tracker -- , seq: 2549, time: 2014-06-09 16:22:11.369621, event: header_read, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.679108 7f869af78700 5 -- op tracker -- , seq: 2549, time: 2014-06-09 16:22:11.369622, event: throttled, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.679111 7f869af78700 5 -- op tracker -- , seq: 2549, time: 2014-06-09 16:22:11.369634, event: all_read, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.679114 7f869af78700 5 -- op tracker -- , seq: 2549, time: 2014-06-09 16:22:11.679100, event: dispatched, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.679118 7f869af78700 5 -- op tracker -- , seq: 2549, time: 2014-06-09 16:22:11.679117, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.679120 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.372) v4 from osd.1 2014-06-09 16:22:11.679122 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdaa80 2014-06-09 16:22:11.679124 7f869af78700 5 -- op tracker -- , seq: 2549, time: 2014-06-09 16:22:11.679123, event: started, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.679133 7f869af78700 5 -- op tracker -- , seq: 2549, time: 2014-06-09 16:22:11.679132, event: done, op: pg_info(1 pgs e243:2.372) 2014-06-09 16:22:11.679137 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679138 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679145 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679145 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679146 7f869af78700 20 osd.26 243 _dispatch 0x31167180 pg_info(1 pgs e243:1.56) v4 2014-06-09 16:22:11.679148 7f869af78700 5 -- op tracker -- , seq: 2550, time: 2014-06-09 16:22:11.446262, event: header_read, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.679152 7f869af78700 5 -- op tracker -- , seq: 2550, time: 2014-06-09 16:22:11.446263, event: throttled, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.679155 7f869af78700 5 -- op tracker -- , seq: 2550, time: 2014-06-09 16:22:11.446274, event: all_read, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.679159 7f869af78700 5 -- op tracker -- , seq: 2550, time: 2014-06-09 16:22:11.679144, event: dispatched, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.679163 7f869af78700 5 -- op tracker -- , seq: 2550, time: 2014-06-09 16:22:11.679162, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.679165 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.56) v4 from osd.9 2014-06-09 16:22:11.679166 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31167180 2014-06-09 16:22:11.679169 7f869af78700 5 -- op tracker -- , seq: 2550, time: 2014-06-09 16:22:11.679168, event: started, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.679179 7f869af78700 5 -- op tracker -- , seq: 2550, time: 2014-06-09 16:22:11.679178, event: done, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.679183 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679184 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679190 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679191 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679191 7f869af78700 20 osd.26 243 _dispatch 0x31157a40 pg_info(1 pgs e243:3.7e2) v4 2014-06-09 16:22:11.679194 7f869af78700 5 -- op tracker -- , seq: 2551, time: 2014-06-09 16:22:11.415504, event: header_read, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.679197 7f869af78700 5 -- op tracker -- , seq: 2551, time: 2014-06-09 16:22:11.415505, event: throttled, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.679201 7f869af78700 5 -- op tracker -- , seq: 2551, time: 2014-06-09 16:22:11.415514, event: all_read, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.679205 7f869af78700 5 -- op tracker -- , seq: 2551, time: 2014-06-09 16:22:11.679190, event: dispatched, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.679208 7f869af78700 5 -- op tracker -- , seq: 2551, time: 2014-06-09 16:22:11.679207, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.679214 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7e2) v4 from osd.8 2014-06-09 16:22:11.679215 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31157a40 2014-06-09 16:22:11.679218 7f869af78700 5 -- op tracker -- , seq: 2551, time: 2014-06-09 16:22:11.679217, event: started, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.679227 7f869af78700 5 -- op tracker -- , seq: 2551, time: 2014-06-09 16:22:11.679226, event: done, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.679233 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679234 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679241 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679241 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679242 7f869af78700 20 osd.26 243 _dispatch 0x310e2540 pg_info(1 pgs e243:0.420) v4 2014-06-09 16:22:11.679245 7f869af78700 5 -- op tracker -- , seq: 2552, time: 2014-06-09 16:22:11.419495, event: header_read, op: pg_info(1 pgs e243:0.420) 2014-06-09 16:22:11.679250 7f869af78700 5 -- op tracker -- , seq: 2552, time: 2014-06-09 16:22:11.419496, event: throttled, op: pg_info(1 pgs e243:0.420) 2014-06-09 16:22:11.679253 7f869af78700 5 -- op tracker -- , seq: 2552, time: 2014-06-09 16:22:11.419509, event: all_read, op: pg_info(1 pgs e243:0.420) 2014-06-09 16:22:11.679258 7f869af78700 5 -- op tracker -- , seq: 2552, time: 2014-06-09 16:22:11.679240, event: dispatched, op: pg_info(1 pgs e243:0.420) 2014-06-09 16:22:11.679261 7f869af78700 5 -- op tracker -- , seq: 2552, time: 2014-06-09 16:22:11.679260, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.420) 2014-06-09 16:22:11.679264 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.420) v4 from osd.6 2014-06-09 16:22:11.679266 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e2540 2014-06-09 16:22:11.679268 7f869af78700 5 -- op tracker -- , seq: 2552, time: 2014-06-09 16:22:11.679267, event: started, op: pg_info(1 pgs e243:0.420) 2014-06-09 16:22:11.679280 7f869af78700 5 -- op tracker -- , seq: 2552, time: 2014-06-09 16:22:11.679279, event: done, op: pg_info(1 pgs e243:0.420) 2014-06-09 16:22:11.679284 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679285 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679291 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679292 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679292 7f869af78700 20 osd.26 243 _dispatch 0x280c5a40 pg_info(1 pgs e243:2.37a) v4 2014-06-09 16:22:11.679295 7f869af78700 5 -- op tracker -- , seq: 2553, time: 2014-06-09 16:22:11.430137, event: header_read, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.679298 7f869af78700 5 -- op tracker -- , seq: 2553, time: 2014-06-09 16:22:11.430137, event: throttled, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.679302 7f869af78700 5 -- op tracker -- , seq: 2553, time: 2014-06-09 16:22:11.430147, event: all_read, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.679305 7f869af78700 5 -- op tracker -- , seq: 2553, time: 2014-06-09 16:22:11.679291, event: dispatched, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.679308 7f869af78700 5 -- op tracker -- , seq: 2553, time: 2014-06-09 16:22:11.679307, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.679311 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.37a) v4 from osd.0 2014-06-09 16:22:11.679312 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c5a40 2014-06-09 16:22:11.679315 7f869af78700 5 -- op tracker -- , seq: 2553, time: 2014-06-09 16:22:11.679314, event: started, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.679322 7f869af78700 5 -- op tracker -- , seq: 2553, time: 2014-06-09 16:22:11.679321, event: done, op: pg_info(1 pgs e243:2.37a) 2014-06-09 16:22:11.679329 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679330 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679337 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679337 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679338 7f869af78700 20 osd.26 243 _dispatch 0x3115a700 pg_info(1 pgs e243:2.6cd) v4 2014-06-09 16:22:11.679341 7f869af78700 5 -- op tracker -- , seq: 2554, time: 2014-06-09 16:22:11.419872, event: header_read, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.679344 7f869af78700 5 -- op tracker -- , seq: 2554, time: 2014-06-09 16:22:11.419872, event: throttled, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.679348 7f869af78700 5 -- op tracker -- , seq: 2554, time: 2014-06-09 16:22:11.419880, event: all_read, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.679351 7f869af78700 5 -- op tracker -- , seq: 2554, time: 2014-06-09 16:22:11.679337, event: dispatched, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.679358 7f869af78700 5 -- op tracker -- , seq: 2554, time: 2014-06-09 16:22:11.679356, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.679360 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6cd) v4 from osd.5 2014-06-09 16:22:11.679362 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115a700 2014-06-09 16:22:11.679364 7f869af78700 5 -- op tracker -- , seq: 2554, time: 2014-06-09 16:22:11.679363, event: started, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.679372 7f869af78700 5 -- op tracker -- , seq: 2554, time: 2014-06-09 16:22:11.679371, event: done, op: pg_info(1 pgs e243:2.6cd) 2014-06-09 16:22:11.679376 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679377 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679383 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679384 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679384 7f869af78700 20 osd.26 243 _dispatch 0x19fdefc0 pg_info(1 pgs e243:0.c8) v4 2014-06-09 16:22:11.679387 7f869af78700 5 -- op tracker -- , seq: 2555, time: 2014-06-09 16:22:11.302889, event: header_read, op: pg_info(1 pgs e243:0.c8) 2014-06-09 16:22:11.679391 7f869af78700 5 -- op tracker -- , seq: 2555, time: 2014-06-09 16:22:11.302889, event: throttled, op: pg_info(1 pgs e243:0.c8) 2014-06-09 16:22:11.679398 7f869af78700 5 -- op tracker -- , seq: 2555, time: 2014-06-09 16:22:11.302895, event: all_read, op: pg_info(1 pgs e243:0.c8) 2014-06-09 16:22:11.679401 7f869af78700 5 -- op tracker -- , seq: 2555, time: 2014-06-09 16:22:11.679383, event: dispatched, op: pg_info(1 pgs e243:0.c8) 2014-06-09 16:22:11.679406 7f869af78700 5 -- op tracker -- , seq: 2555, time: 2014-06-09 16:22:11.679405, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.c8) 2014-06-09 16:22:11.679409 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.c8) v4 from osd.2 2014-06-09 16:22:11.679410 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdefc0 2014-06-09 16:22:11.679413 7f869af78700 5 -- op tracker -- , seq: 2555, time: 2014-06-09 16:22:11.679412, event: started, op: pg_info(1 pgs e243:0.c8) 2014-06-09 16:22:11.679421 7f869af78700 5 -- op tracker -- , seq: 2555, time: 2014-06-09 16:22:11.679420, event: done, op: pg_info(1 pgs e243:0.c8) 2014-06-09 16:22:11.679425 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679426 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679433 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679433 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679434 7f869af78700 20 osd.26 243 _dispatch 0x1405d880 pg_info(1 pgs e243:2.368) v4 2014-06-09 16:22:11.679437 7f869af78700 5 -- op tracker -- , seq: 2556, time: 2014-06-09 16:22:11.396026, event: header_read, op: pg_info(1 pgs e243:2.368) 2014-06-09 16:22:11.679440 7f869af78700 5 -- op tracker -- , seq: 2556, time: 2014-06-09 16:22:11.396026, event: throttled, op: pg_info(1 pgs e243:2.368) 2014-06-09 16:22:11.679444 7f869af78700 5 -- op tracker -- , seq: 2556, time: 2014-06-09 16:22:11.396036, event: all_read, op: pg_info(1 pgs e243:2.368) 2014-06-09 16:22:11.679447 7f869af78700 5 -- op tracker -- , seq: 2556, time: 2014-06-09 16:22:11.679432, event: dispatched, op: pg_info(1 pgs e243:2.368) 2014-06-09 16:22:11.679452 7f869af78700 5 -- op tracker -- , seq: 2556, time: 2014-06-09 16:22:11.679450, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.368) 2014-06-09 16:22:11.679454 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.368) v4 from osd.4 2014-06-09 16:22:11.679455 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405d880 2014-06-09 16:22:11.679458 7f869af78700 5 -- op tracker -- , seq: 2556, time: 2014-06-09 16:22:11.679457, event: started, op: pg_info(1 pgs e243:2.368) 2014-06-09 16:22:11.679466 7f869af78700 5 -- op tracker -- , seq: 2556, time: 2014-06-09 16:22:11.679465, event: done, op: pg_info(1 pgs e243:2.368) 2014-06-09 16:22:11.679470 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679471 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679477 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679478 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679478 7f869af78700 20 osd.26 243 _dispatch 0x310e9c00 pg_info(1 pgs e243:1.676) v4 2014-06-09 16:22:11.679481 7f869af78700 5 -- op tracker -- , seq: 2557, time: 2014-06-09 16:22:11.411569, event: header_read, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679484 7f869af78700 5 -- op tracker -- , seq: 2557, time: 2014-06-09 16:22:11.411570, event: throttled, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679491 7f869af78700 5 -- op tracker -- , seq: 2557, time: 2014-06-09 16:22:11.411589, event: all_read, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679495 7f869af78700 5 -- op tracker -- , seq: 2557, time: 2014-06-09 16:22:11.679477, event: dispatched, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679499 7f869af78700 5 -- op tracker -- , seq: 2557, time: 2014-06-09 16:22:11.679498, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679501 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.676) v4 from osd.7 2014-06-09 16:22:11.679502 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e9c00 2014-06-09 16:22:11.679505 7f869af78700 5 -- op tracker -- , seq: 2557, time: 2014-06-09 16:22:11.679504, event: started, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679514 7f869af78700 5 -- op tracker -- , seq: 2557, time: 2014-06-09 16:22:11.679513, event: done, op: pg_info(1 pgs e243:1.676) 2014-06-09 16:22:11.679518 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679519 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679526 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679526 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679527 7f869af78700 20 osd.26 243 _dispatch 0x2f031880 pg_info(1 pgs e243:0.3cf) v4 2014-06-09 16:22:11.679530 7f869af78700 5 -- op tracker -- , seq: 2558, time: 2014-06-09 16:22:11.447079, event: header_read, op: pg_info(1 pgs e243:0.3cf) 2014-06-09 16:22:11.679533 7f869af78700 5 -- op tracker -- , seq: 2558, time: 2014-06-09 16:22:11.447079, event: throttled, op: pg_info(1 pgs e243:0.3cf) 2014-06-09 16:22:11.679537 7f869af78700 5 -- op tracker -- , seq: 2558, time: 2014-06-09 16:22:11.447094, event: all_read, op: pg_info(1 pgs e243:0.3cf) 2014-06-09 16:22:11.679541 7f869af78700 5 -- op tracker -- , seq: 2558, time: 2014-06-09 16:22:11.679526, event: dispatched, op: pg_info(1 pgs e243:0.3cf) 2014-06-09 16:22:11.679544 7f869af78700 5 -- op tracker -- , seq: 2558, time: 2014-06-09 16:22:11.679543, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3cf) 2014-06-09 16:22:11.679546 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3cf) v4 from osd.3 2014-06-09 16:22:11.679548 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f031880 2014-06-09 16:22:11.679550 7f869af78700 5 -- op tracker -- , seq: 2558, time: 2014-06-09 16:22:11.679549, event: started, op: pg_info(1 pgs e243:0.3cf) 2014-06-09 16:22:11.679559 7f869af78700 5 -- op tracker -- , seq: 2558, time: 2014-06-09 16:22:11.679558, event: done, op: pg_info(1 pgs e243:0.3cf) 2014-06-09 16:22:11.679563 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679563 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679570 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679570 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679571 7f869af78700 20 osd.26 243 _dispatch 0x280c5c00 pg_info(1 pgs e243:1.38d) v4 2014-06-09 16:22:11.679573 7f869af78700 5 -- op tracker -- , seq: 2559, time: 2014-06-09 16:22:11.369813, event: header_read, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.679577 7f869af78700 5 -- op tracker -- , seq: 2559, time: 2014-06-09 16:22:11.369814, event: throttled, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.679581 7f869af78700 5 -- op tracker -- , seq: 2559, time: 2014-06-09 16:22:11.369829, event: all_read, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.679585 7f869af78700 5 -- op tracker -- , seq: 2559, time: 2014-06-09 16:22:11.679569, event: dispatched, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.679589 7f869af78700 5 -- op tracker -- , seq: 2559, time: 2014-06-09 16:22:11.679588, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.679595 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.38d) v4 from osd.1 2014-06-09 16:22:11.679596 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c5c00 2014-06-09 16:22:11.679599 7f869af78700 5 -- op tracker -- , seq: 2559, time: 2014-06-09 16:22:11.679598, event: started, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.679606 7f869af78700 5 -- op tracker -- , seq: 2559, time: 2014-06-09 16:22:11.679605, event: done, op: pg_info(1 pgs e243:1.38d) 2014-06-09 16:22:11.679610 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679611 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679636 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679638 7f869af78700 20 osd.26 243 _dispatch 0x31166a80 pg_info(1 pgs e243:0.e8) v4 2014-06-09 16:22:11.679641 7f869af78700 5 -- op tracker -- , seq: 2560, time: 2014-06-09 16:22:11.446284, event: header_read, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.679646 7f869af78700 5 -- op tracker -- , seq: 2560, time: 2014-06-09 16:22:11.446284, event: throttled, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.679649 7f869af78700 5 -- op tracker -- , seq: 2560, time: 2014-06-09 16:22:11.446294, event: all_read, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.679656 7f869af78700 5 -- op tracker -- , seq: 2560, time: 2014-06-09 16:22:11.679636, event: dispatched, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.679660 7f869af78700 5 -- op tracker -- , seq: 2560, time: 2014-06-09 16:22:11.679659, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.679663 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.e8) v4 from osd.9 2014-06-09 16:22:11.679665 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31166a80 2014-06-09 16:22:11.679667 7f869af78700 5 -- op tracker -- , seq: 2560, time: 2014-06-09 16:22:11.679666, event: started, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.679676 7f869af78700 5 -- op tracker -- , seq: 2560, time: 2014-06-09 16:22:11.679675, event: done, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.679681 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679682 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679689 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.679689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.679690 7f869af78700 20 osd.26 243 _dispatch 0x31157880 pg_info(1 pgs e243:2.4b5) v4 2014-06-09 16:22:11.679693 7f869af78700 5 -- op tracker -- , seq: 2561, time: 2014-06-09 16:22:11.415518, event: header_read, op: pg_info(1 pgs e243:2.4b5) 2014-06-09 16:22:11.679696 7f869af78700 5 -- op tracker -- , seq: 2561, time: 2014-06-09 16:22:11.415518, event: throttled, op: pg_info(1 pgs e243:2.4b5) 2014-06-09 16:22:11.679702 7f869af78700 5 -- op tracker -- , seq: 2561, time: 2014-06-09 16:22:11.415532, event: all_read, op: pg_info(1 pgs e243:2.4b5) 2014-06-09 16:22:11.679705 7f869af78700 5 -- op tracker -- , seq: 2561, time: 2014-06-09 16:22:11.679688, event: dispatched, op: pg_info(1 pgs e243:2.4b5) 2014-06-09 16:22:11.679710 7f869af78700 5 -- op tracker -- , seq: 2561, time: 2014-06-09 16:22:11.679709, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4b5) 2014-06-09 16:22:11.679712 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4b5) v4 from osd.8 2014-06-09 16:22:11.681372 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31157880 2014-06-09 16:22:11.681380 7f869af78700 5 -- op tracker -- , seq: 2561, time: 2014-06-09 16:22:11.681378, event: started, op: pg_info(1 pgs e243:2.4b5) 2014-06-09 16:22:11.681399 7f869af78700 5 -- op tracker -- , seq: 2561, time: 2014-06-09 16:22:11.681398, event: done, op: pg_info(1 pgs e243:2.4b5) 2014-06-09 16:22:11.681407 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681409 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681419 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681421 7f869af78700 20 osd.26 243 _dispatch 0x310e2380 pg_info(1 pgs e243:0.77a) v4 2014-06-09 16:22:11.681425 7f869af78700 5 -- op tracker -- , seq: 2562, time: 2014-06-09 16:22:11.419550, event: header_read, op: pg_info(1 pgs e243:0.77a) 2014-06-09 16:22:11.681432 7f869af78700 5 -- op tracker -- , seq: 2562, time: 2014-06-09 16:22:11.419551, event: throttled, op: pg_info(1 pgs e243:0.77a) 2014-06-09 16:22:11.681436 7f869af78700 5 -- op tracker -- , seq: 2562, time: 2014-06-09 16:22:11.419564, event: all_read, op: pg_info(1 pgs e243:0.77a) 2014-06-09 16:22:11.681440 7f869af78700 5 -- op tracker -- , seq: 2562, time: 2014-06-09 16:22:11.681419, event: dispatched, op: pg_info(1 pgs e243:0.77a) 2014-06-09 16:22:11.681445 7f869af78700 5 -- op tracker -- , seq: 2562, time: 2014-06-09 16:22:11.681444, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.77a) 2014-06-09 16:22:11.681447 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.77a) v4 from osd.6 2014-06-09 16:22:11.681449 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e2380 2014-06-09 16:22:11.681452 7f869af78700 5 -- op tracker -- , seq: 2562, time: 2014-06-09 16:22:11.681451, event: started, op: pg_info(1 pgs e243:0.77a) 2014-06-09 16:22:11.681469 7f869af78700 5 -- op tracker -- , seq: 2562, time: 2014-06-09 16:22:11.681468, event: done, op: pg_info(1 pgs e243:0.77a) 2014-06-09 16:22:11.681474 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681474 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681482 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681483 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681483 7f869af78700 20 osd.26 243 _dispatch 0x31167dc0 pg_info(1 pgs e243:0.8c) v4 2014-06-09 16:22:11.681486 7f869af78700 5 -- op tracker -- , seq: 2563, time: 2014-06-09 16:22:11.430152, event: header_read, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.681490 7f869af78700 5 -- op tracker -- , seq: 2563, time: 2014-06-09 16:22:11.430152, event: throttled, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.681493 7f869af78700 5 -- op tracker -- , seq: 2563, time: 2014-06-09 16:22:11.430164, event: all_read, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.681497 7f869af78700 5 -- op tracker -- , seq: 2563, time: 2014-06-09 16:22:11.681482, event: dispatched, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.681500 7f869af78700 5 -- op tracker -- , seq: 2563, time: 2014-06-09 16:22:11.681499, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.681503 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.8c) v4 from osd.0 2014-06-09 16:22:11.681504 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31167dc0 2014-06-09 16:22:11.681507 7f869af78700 5 -- op tracker -- , seq: 2563, time: 2014-06-09 16:22:11.681506, event: started, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.681515 7f869af78700 5 -- op tracker -- , seq: 2563, time: 2014-06-09 16:22:11.681514, event: done, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.681519 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681520 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681526 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681527 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681528 7f869af78700 20 osd.26 243 _dispatch 0x3115a540 pg_info(1 pgs e243:2.446) v4 2014-06-09 16:22:11.681530 7f869af78700 5 -- op tracker -- , seq: 2564, time: 2014-06-09 16:22:11.419885, event: header_read, op: pg_info(1 pgs e243:2.446) 2014-06-09 16:22:11.681535 7f869af78700 5 -- op tracker -- , seq: 2564, time: 2014-06-09 16:22:11.419885, event: throttled, op: pg_info(1 pgs e243:2.446) 2014-06-09 16:22:11.681538 7f869af78700 5 -- op tracker -- , seq: 2564, time: 2014-06-09 16:22:11.419894, event: all_read, op: pg_info(1 pgs e243:2.446) 2014-06-09 16:22:11.681541 7f869af78700 5 -- op tracker -- , seq: 2564, time: 2014-06-09 16:22:11.681526, event: dispatched, op: pg_info(1 pgs e243:2.446) 2014-06-09 16:22:11.681545 7f869af78700 5 -- op tracker -- , seq: 2564, time: 2014-06-09 16:22:11.681543, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.446) 2014-06-09 16:22:11.681548 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.446) v4 from osd.5 2014-06-09 16:22:11.681550 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115a540 2014-06-09 16:22:11.681553 7f869af78700 5 -- op tracker -- , seq: 2564, time: 2014-06-09 16:22:11.681552, event: started, op: pg_info(1 pgs e243:2.446) 2014-06-09 16:22:11.681561 7f869af78700 5 -- op tracker -- , seq: 2564, time: 2014-06-09 16:22:11.681560, event: done, op: pg_info(1 pgs e243:2.446) 2014-06-09 16:22:11.681565 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681566 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681573 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681573 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681574 7f869af78700 20 osd.26 243 _dispatch 0x307e6fc0 pg_info(1 pgs e243:1.246) v4 2014-06-09 16:22:11.681577 7f869af78700 5 -- op tracker -- , seq: 2565, time: 2014-06-09 16:22:11.302906, event: header_read, op: pg_info(1 pgs e243:1.246) 2014-06-09 16:22:11.681586 7f869af78700 5 -- op tracker -- , seq: 2565, time: 2014-06-09 16:22:11.302906, event: throttled, op: pg_info(1 pgs e243:1.246) 2014-06-09 16:22:11.681589 7f869af78700 5 -- op tracker -- , seq: 2565, time: 2014-06-09 16:22:11.302926, event: all_read, op: pg_info(1 pgs e243:1.246) 2014-06-09 16:22:11.681594 7f869af78700 5 -- op tracker -- , seq: 2565, time: 2014-06-09 16:22:11.681573, event: dispatched, op: pg_info(1 pgs e243:1.246) 2014-06-09 16:22:11.681597 7f869af78700 5 -- op tracker -- , seq: 2565, time: 2014-06-09 16:22:11.681596, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.246) 2014-06-09 16:22:11.681600 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.246) v4 from osd.2 2014-06-09 16:22:11.681602 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e6fc0 2014-06-09 16:22:11.681604 7f869af78700 5 -- op tracker -- , seq: 2565, time: 2014-06-09 16:22:11.681603, event: started, op: pg_info(1 pgs e243:1.246) 2014-06-09 16:22:11.681613 7f869af78700 5 -- op tracker -- , seq: 2565, time: 2014-06-09 16:22:11.681612, event: done, op: pg_info(1 pgs e243:1.246) 2014-06-09 16:22:11.681618 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681619 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681626 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681627 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681627 7f869af78700 20 osd.26 243 _dispatch 0x10d02000 pg_notify(2.186(20) epoch 243) v5 2014-06-09 16:22:11.681630 7f869af78700 5 -- op tracker -- , seq: 2566, time: 2014-06-09 16:22:11.398486, event: header_read, op: pg_notify(2.186(20) epoch 243) 2014-06-09 16:22:11.681634 7f869af78700 5 -- op tracker -- , seq: 2566, time: 2014-06-09 16:22:11.398486, event: throttled, op: pg_notify(2.186(20) epoch 243) 2014-06-09 16:22:11.681637 7f869af78700 5 -- op tracker -- , seq: 2566, time: 2014-06-09 16:22:11.398520, event: all_read, op: pg_notify(2.186(20) epoch 243) 2014-06-09 16:22:11.681646 7f869af78700 5 -- op tracker -- , seq: 2566, time: 2014-06-09 16:22:11.681626, event: dispatched, op: pg_notify(2.186(20) epoch 243) 2014-06-09 16:22:11.681649 7f869af78700 5 -- op tracker -- , seq: 2566, time: 2014-06-09 16:22:11.681648, event: waiting_for_osdmap, op: pg_notify(2.186(20) epoch 243) 2014-06-09 16:22:11.681656 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.681657 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d02000 2014-06-09 16:22:11.681660 7f869af78700 5 -- op tracker -- , seq: 2566, time: 2014-06-09 16:22:11.681659, event: started, op: pg_notify(2.186(20) epoch 243) 2014-06-09 16:22:11.681670 7f869af78700 5 -- op tracker -- , seq: 2566, time: 2014-06-09 16:22:11.681669, event: done, op: pg_notify(2.186(20) epoch 243) 2014-06-09 16:22:11.681676 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681677 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681688 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681690 7f869af78700 20 osd.26 243 _dispatch 0x310e8540 pg_info(1 pgs e243:0.521) v4 2014-06-09 16:22:11.681692 7f869af78700 5 -- op tracker -- , seq: 2567, time: 2014-06-09 16:22:11.430112, event: header_read, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.681698 7f869af78700 5 -- op tracker -- , seq: 2567, time: 2014-06-09 16:22:11.430112, event: throttled, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.681701 7f869af78700 5 -- op tracker -- , seq: 2567, time: 2014-06-09 16:22:11.430124, event: all_read, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.681704 7f869af78700 5 -- op tracker -- , seq: 2567, time: 2014-06-09 16:22:11.681688, event: dispatched, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.681708 7f869af78700 5 -- op tracker -- , seq: 2567, time: 2014-06-09 16:22:11.681707, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.681710 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.521) v4 from osd.7 2014-06-09 16:22:11.681712 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e8540 2014-06-09 16:22:11.681714 7f869af78700 5 -- op tracker -- , seq: 2567, time: 2014-06-09 16:22:11.681713, event: started, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.681722 7f869af78700 5 -- op tracker -- , seq: 2567, time: 2014-06-09 16:22:11.681721, event: done, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.681727 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681727 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681734 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681735 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681735 7f869af78700 20 osd.26 243 _dispatch 0x2f0316c0 pg_info(1 pgs e243:1.4c3) v4 2014-06-09 16:22:11.681738 7f869af78700 5 -- op tracker -- , seq: 2568, time: 2014-06-09 16:22:11.447133, event: header_read, op: pg_info(1 pgs e243:1.4c3) 2014-06-09 16:22:11.681747 7f869af78700 5 -- op tracker -- , seq: 2568, time: 2014-06-09 16:22:11.447134, event: throttled, op: pg_info(1 pgs e243:1.4c3) 2014-06-09 16:22:11.681750 7f869af78700 5 -- op tracker -- , seq: 2568, time: 2014-06-09 16:22:11.447146, event: all_read, op: pg_info(1 pgs e243:1.4c3) 2014-06-09 16:22:11.681758 7f869af78700 5 -- op tracker -- , seq: 2568, time: 2014-06-09 16:22:11.681734, event: dispatched, op: pg_info(1 pgs e243:1.4c3) 2014-06-09 16:22:11.681761 7f869af78700 5 -- op tracker -- , seq: 2568, time: 2014-06-09 16:22:11.681760, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4c3) 2014-06-09 16:22:11.681766 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4c3) v4 from osd.3 2014-06-09 16:22:11.681768 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f0316c0 2014-06-09 16:22:11.681771 7f869af78700 5 -- op tracker -- , seq: 2568, time: 2014-06-09 16:22:11.681770, event: started, op: pg_info(1 pgs e243:1.4c3) 2014-06-09 16:22:11.681780 7f869af78700 5 -- op tracker -- , seq: 2568, time: 2014-06-09 16:22:11.681779, event: done, op: pg_info(1 pgs e243:1.4c3) 2014-06-09 16:22:11.681787 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681788 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681796 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681797 7f869af78700 20 osd.26 243 _dispatch 0x30d24a80 pg_info(1 pgs e243:1.67a) v4 2014-06-09 16:22:11.681800 7f869af78700 5 -- op tracker -- , seq: 2569, time: 2014-06-09 16:22:11.380142, event: header_read, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.681807 7f869af78700 5 -- op tracker -- , seq: 2569, time: 2014-06-09 16:22:11.380143, event: throttled, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.681813 7f869af78700 5 -- op tracker -- , seq: 2569, time: 2014-06-09 16:22:11.380156, event: all_read, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.681819 7f869af78700 5 -- op tracker -- , seq: 2569, time: 2014-06-09 16:22:11.681795, event: dispatched, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.681823 7f869af78700 5 -- op tracker -- , seq: 2569, time: 2014-06-09 16:22:11.681822, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.681825 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.67a) v4 from osd.1 2014-06-09 16:22:11.681827 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d24a80 2014-06-09 16:22:11.681829 7f869af78700 5 -- op tracker -- , seq: 2569, time: 2014-06-09 16:22:11.681828, event: started, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.681840 7f869af78700 5 -- op tracker -- , seq: 2569, time: 2014-06-09 16:22:11.681839, event: done, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.681845 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681845 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681853 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681853 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681854 7f869af78700 20 osd.26 243 _dispatch 0x310da8c0 pg_info(1 pgs e243:1.56) v4 2014-06-09 16:22:11.681857 7f869af78700 5 -- op tracker -- , seq: 2570, time: 2014-06-09 16:22:11.446299, event: header_read, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.681860 7f869af78700 5 -- op tracker -- , seq: 2570, time: 2014-06-09 16:22:11.446299, event: throttled, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.681864 7f869af78700 5 -- op tracker -- , seq: 2570, time: 2014-06-09 16:22:11.446311, event: all_read, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.681867 7f869af78700 5 -- op tracker -- , seq: 2570, time: 2014-06-09 16:22:11.681852, event: dispatched, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.681871 7f869af78700 5 -- op tracker -- , seq: 2570, time: 2014-06-09 16:22:11.681870, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.681873 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.56) v4 from osd.9 2014-06-09 16:22:11.681875 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310da8c0 2014-06-09 16:22:11.681877 7f869af78700 5 -- op tracker -- , seq: 2570, time: 2014-06-09 16:22:11.681876, event: started, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.681890 7f869af78700 5 -- op tracker -- , seq: 2570, time: 2014-06-09 16:22:11.681889, event: done, op: pg_info(1 pgs e243:1.56) 2014-06-09 16:22:11.681894 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681895 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681902 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681903 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681904 7f869af78700 20 osd.26 243 _dispatch 0x311576c0 pg_info(1 pgs e243:2.4e0) v4 2014-06-09 16:22:11.681906 7f869af78700 5 -- op tracker -- , seq: 2571, time: 2014-06-09 16:22:11.415536, event: header_read, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.681912 7f869af78700 5 -- op tracker -- , seq: 2571, time: 2014-06-09 16:22:11.415537, event: throttled, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.681915 7f869af78700 5 -- op tracker -- , seq: 2571, time: 2014-06-09 16:22:11.415547, event: all_read, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.681922 7f869af78700 5 -- op tracker -- , seq: 2571, time: 2014-06-09 16:22:11.681902, event: dispatched, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.681926 7f869af78700 5 -- op tracker -- , seq: 2571, time: 2014-06-09 16:22:11.681925, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.681930 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4e0) v4 from osd.8 2014-06-09 16:22:11.681931 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311576c0 2014-06-09 16:22:11.681934 7f869af78700 5 -- op tracker -- , seq: 2571, time: 2014-06-09 16:22:11.681933, event: started, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.681942 7f869af78700 5 -- op tracker -- , seq: 2571, time: 2014-06-09 16:22:11.681941, event: done, op: pg_info(1 pgs e243:2.4e0) 2014-06-09 16:22:11.681947 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681948 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681954 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681955 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.681955 7f869af78700 20 osd.26 243 _dispatch 0x310e21c0 pg_info(1 pgs e243:1.1b1) v4 2014-06-09 16:22:11.681958 7f869af78700 5 -- op tracker -- , seq: 2572, time: 2014-06-09 16:22:11.419605, event: header_read, op: pg_info(1 pgs e243:1.1b1) 2014-06-09 16:22:11.681962 7f869af78700 5 -- op tracker -- , seq: 2572, time: 2014-06-09 16:22:11.419605, event: throttled, op: pg_info(1 pgs e243:1.1b1) 2014-06-09 16:22:11.681965 7f869af78700 5 -- op tracker -- , seq: 2572, time: 2014-06-09 16:22:11.423636, event: all_read, op: pg_info(1 pgs e243:1.1b1) 2014-06-09 16:22:11.681970 7f869af78700 5 -- op tracker -- , seq: 2572, time: 2014-06-09 16:22:11.681954, event: dispatched, op: pg_info(1 pgs e243:1.1b1) 2014-06-09 16:22:11.681974 7f869af78700 5 -- op tracker -- , seq: 2572, time: 2014-06-09 16:22:11.681973, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1b1) 2014-06-09 16:22:11.681977 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1b1) v4 from osd.6 2014-06-09 16:22:11.681979 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e21c0 2014-06-09 16:22:11.681981 7f869af78700 5 -- op tracker -- , seq: 2572, time: 2014-06-09 16:22:11.681980, event: started, op: pg_info(1 pgs e243:1.1b1) 2014-06-09 16:22:11.681991 7f869af78700 5 -- op tracker -- , seq: 2572, time: 2014-06-09 16:22:11.681990, event: done, op: pg_info(1 pgs e243:1.1b1) 2014-06-09 16:22:11.681995 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.681996 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682003 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682003 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682004 7f869af78700 20 osd.26 243 _dispatch 0x31167c00 pg_info(1 pgs e243:0.8c) v4 2014-06-09 16:22:11.682007 7f869af78700 5 -- op tracker -- , seq: 2573, time: 2014-06-09 16:22:11.430168, event: header_read, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.682010 7f869af78700 5 -- op tracker -- , seq: 2573, time: 2014-06-09 16:22:11.430168, event: throttled, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.682017 7f869af78700 5 -- op tracker -- , seq: 2573, time: 2014-06-09 16:22:11.430178, event: all_read, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.682020 7f869af78700 5 -- op tracker -- , seq: 2573, time: 2014-06-09 16:22:11.682003, event: dispatched, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.682024 7f869af78700 5 -- op tracker -- , seq: 2573, time: 2014-06-09 16:22:11.682023, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.682026 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.8c) v4 from osd.0 2014-06-09 16:22:11.682028 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31167c00 2014-06-09 16:22:11.682030 7f869af78700 5 -- op tracker -- , seq: 2573, time: 2014-06-09 16:22:11.682029, event: started, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.682040 7f869af78700 5 -- op tracker -- , seq: 2573, time: 2014-06-09 16:22:11.682038, event: done, op: pg_info(1 pgs e243:0.8c) 2014-06-09 16:22:11.682044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682051 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682052 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682052 7f869af78700 20 osd.26 243 _dispatch 0x3115a380 pg_info(1 pgs e243:1.15e) v4 2014-06-09 16:22:11.682055 7f869af78700 5 -- op tracker -- , seq: 2574, time: 2014-06-09 16:22:11.419898, event: header_read, op: pg_info(1 pgs e243:1.15e) 2014-06-09 16:22:11.682059 7f869af78700 5 -- op tracker -- , seq: 2574, time: 2014-06-09 16:22:11.419898, event: throttled, op: pg_info(1 pgs e243:1.15e) 2014-06-09 16:22:11.682062 7f869af78700 5 -- op tracker -- , seq: 2574, time: 2014-06-09 16:22:11.419906, event: all_read, op: pg_info(1 pgs e243:1.15e) 2014-06-09 16:22:11.682067 7f869af78700 5 -- op tracker -- , seq: 2574, time: 2014-06-09 16:22:11.682051, event: dispatched, op: pg_info(1 pgs e243:1.15e) 2014-06-09 16:22:11.682071 7f869af78700 5 -- op tracker -- , seq: 2574, time: 2014-06-09 16:22:11.682070, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.15e) 2014-06-09 16:22:11.682073 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.15e) v4 from osd.5 2014-06-09 16:22:11.682075 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115a380 2014-06-09 16:22:11.682077 7f869af78700 5 -- op tracker -- , seq: 2574, time: 2014-06-09 16:22:11.682076, event: started, op: pg_info(1 pgs e243:1.15e) 2014-06-09 16:22:11.682085 7f869af78700 5 -- op tracker -- , seq: 2574, time: 2014-06-09 16:22:11.682084, event: done, op: pg_info(1 pgs e243:1.15e) 2014-06-09 16:22:11.682090 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682091 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682097 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682098 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682098 7f869af78700 20 osd.26 243 _dispatch 0x3088ce00 pg_info(1 pgs e243:1.29a) v4 2014-06-09 16:22:11.682101 7f869af78700 5 -- op tracker -- , seq: 2575, time: 2014-06-09 16:22:11.302938, event: header_read, op: pg_info(1 pgs e243:1.29a) 2014-06-09 16:22:11.682105 7f869af78700 5 -- op tracker -- , seq: 2575, time: 2014-06-09 16:22:11.302939, event: throttled, op: pg_info(1 pgs e243:1.29a) 2014-06-09 16:22:11.682108 7f869af78700 5 -- op tracker -- , seq: 2575, time: 2014-06-09 16:22:11.302945, event: all_read, op: pg_info(1 pgs e243:1.29a) 2014-06-09 16:22:11.682111 7f869af78700 5 -- op tracker -- , seq: 2575, time: 2014-06-09 16:22:11.682097, event: dispatched, op: pg_info(1 pgs e243:1.29a) 2014-06-09 16:22:11.682115 7f869af78700 5 -- op tracker -- , seq: 2575, time: 2014-06-09 16:22:11.682114, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.29a) 2014-06-09 16:22:11.682118 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.29a) v4 from osd.2 2014-06-09 16:22:11.682119 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3088ce00 2014-06-09 16:22:11.682121 7f869af78700 5 -- op tracker -- , seq: 2575, time: 2014-06-09 16:22:11.682120, event: started, op: pg_info(1 pgs e243:1.29a) 2014-06-09 16:22:11.682131 7f869af78700 5 -- op tracker -- , seq: 2575, time: 2014-06-09 16:22:11.682130, event: done, op: pg_info(1 pgs e243:1.29a) 2014-06-09 16:22:11.682136 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682137 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682144 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682145 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682145 7f869af78700 20 osd.26 243 _dispatch 0x310d7a40 pg_info(1 pgs e243:2.3a3) v4 2014-06-09 16:22:11.682148 7f869af78700 5 -- op tracker -- , seq: 2576, time: 2014-06-09 16:22:11.398535, event: header_read, op: pg_info(1 pgs e243:2.3a3) 2014-06-09 16:22:11.682153 7f869af78700 5 -- op tracker -- , seq: 2576, time: 2014-06-09 16:22:11.398535, event: throttled, op: pg_info(1 pgs e243:2.3a3) 2014-06-09 16:22:11.682156 7f869af78700 5 -- op tracker -- , seq: 2576, time: 2014-06-09 16:22:11.398557, event: all_read, op: pg_info(1 pgs e243:2.3a3) 2014-06-09 16:22:11.682163 7f869af78700 5 -- op tracker -- , seq: 2576, time: 2014-06-09 16:22:11.682144, event: dispatched, op: pg_info(1 pgs e243:2.3a3) 2014-06-09 16:22:11.682170 7f869af78700 5 -- op tracker -- , seq: 2576, time: 2014-06-09 16:22:11.682169, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3a3) 2014-06-09 16:22:11.682172 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3a3) v4 from osd.4 2014-06-09 16:22:11.682174 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d7a40 2014-06-09 16:22:11.682176 7f869af78700 5 -- op tracker -- , seq: 2576, time: 2014-06-09 16:22:11.682175, event: started, op: pg_info(1 pgs e243:2.3a3) 2014-06-09 16:22:11.682185 7f869af78700 5 -- op tracker -- , seq: 2576, time: 2014-06-09 16:22:11.682184, event: done, op: pg_info(1 pgs e243:2.3a3) 2014-06-09 16:22:11.682189 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682190 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682196 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682197 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682197 7f869af78700 20 osd.26 243 _dispatch 0x31167340 pg_info(1 pgs e243:0.521) v4 2014-06-09 16:22:11.682200 7f869af78700 5 -- op tracker -- , seq: 2577, time: 2014-06-09 16:22:11.430167, event: header_read, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.682207 7f869af78700 5 -- op tracker -- , seq: 2577, time: 2014-06-09 16:22:11.430168, event: throttled, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.682211 7f869af78700 5 -- op tracker -- , seq: 2577, time: 2014-06-09 16:22:11.430178, event: all_read, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.682215 7f869af78700 5 -- op tracker -- , seq: 2577, time: 2014-06-09 16:22:11.682196, event: dispatched, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.682219 7f869af78700 5 -- op tracker -- , seq: 2577, time: 2014-06-09 16:22:11.682218, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.682222 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.521) v4 from osd.7 2014-06-09 16:22:11.682223 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31167340 2014-06-09 16:22:11.682226 7f869af78700 5 -- op tracker -- , seq: 2577, time: 2014-06-09 16:22:11.682225, event: started, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.682234 7f869af78700 5 -- op tracker -- , seq: 2577, time: 2014-06-09 16:22:11.682233, event: done, op: pg_info(1 pgs e243:0.521) 2014-06-09 16:22:11.682238 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682239 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682246 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682246 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682247 7f869af78700 20 osd.26 243 _dispatch 0x2f031500 pg_info(1 pgs e243:0.13b) v4 2014-06-09 16:22:11.682249 7f869af78700 5 -- op tracker -- , seq: 2578, time: 2014-06-09 16:22:11.447251, event: header_read, op: pg_info(1 pgs e243:0.13b) 2014-06-09 16:22:11.682253 7f869af78700 5 -- op tracker -- , seq: 2578, time: 2014-06-09 16:22:11.447251, event: throttled, op: pg_info(1 pgs e243:0.13b) 2014-06-09 16:22:11.682256 7f869af78700 5 -- op tracker -- , seq: 2578, time: 2014-06-09 16:22:11.447264, event: all_read, op: pg_info(1 pgs e243:0.13b) 2014-06-09 16:22:11.682263 7f869af78700 5 -- op tracker -- , seq: 2578, time: 2014-06-09 16:22:11.682245, event: dispatched, op: pg_info(1 pgs e243:0.13b) 2014-06-09 16:22:11.682270 7f869af78700 5 -- op tracker -- , seq: 2578, time: 2014-06-09 16:22:11.682268, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.13b) 2014-06-09 16:22:11.682272 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.13b) v4 from osd.3 2014-06-09 16:22:11.682274 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f031500 2014-06-09 16:22:11.682276 7f869af78700 5 -- op tracker -- , seq: 2578, time: 2014-06-09 16:22:11.682275, event: started, op: pg_info(1 pgs e243:0.13b) 2014-06-09 16:22:11.682284 7f869af78700 5 -- op tracker -- , seq: 2578, time: 2014-06-09 16:22:11.682283, event: done, op: pg_info(1 pgs e243:0.13b) 2014-06-09 16:22:11.682288 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682289 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682296 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682296 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682297 7f869af78700 20 osd.26 243 _dispatch 0x3104bdc0 pg_info(1 pgs e243:1.67a) v4 2014-06-09 16:22:11.682300 7f869af78700 5 -- op tracker -- , seq: 2579, time: 2014-06-09 16:22:11.380165, event: header_read, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.682303 7f869af78700 5 -- op tracker -- , seq: 2579, time: 2014-06-09 16:22:11.380165, event: throttled, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.682307 7f869af78700 5 -- op tracker -- , seq: 2579, time: 2014-06-09 16:22:11.380176, event: all_read, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.682310 7f869af78700 5 -- op tracker -- , seq: 2579, time: 2014-06-09 16:22:11.682296, event: dispatched, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.682313 7f869af78700 5 -- op tracker -- , seq: 2579, time: 2014-06-09 16:22:11.682312, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.682315 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.67a) v4 from osd.1 2014-06-09 16:22:11.682317 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104bdc0 2014-06-09 16:22:11.682319 7f869af78700 5 -- op tracker -- , seq: 2579, time: 2014-06-09 16:22:11.682318, event: started, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.682327 7f869af78700 5 -- op tracker -- , seq: 2579, time: 2014-06-09 16:22:11.682326, event: done, op: pg_info(1 pgs e243:1.67a) 2014-06-09 16:22:11.682331 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682332 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682338 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682339 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682339 7f869af78700 20 osd.26 243 _dispatch 0x310da700 pg_info(1 pgs e243:0.e8) v4 2014-06-09 16:22:11.682342 7f869af78700 5 -- op tracker -- , seq: 2580, time: 2014-06-09 16:22:11.446316, event: header_read, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.682346 7f869af78700 5 -- op tracker -- , seq: 2580, time: 2014-06-09 16:22:11.446317, event: throttled, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.682350 7f869af78700 5 -- op tracker -- , seq: 2580, time: 2014-06-09 16:22:11.446328, event: all_read, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.682355 7f869af78700 5 -- op tracker -- , seq: 2580, time: 2014-06-09 16:22:11.682338, event: dispatched, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.682359 7f869af78700 5 -- op tracker -- , seq: 2580, time: 2014-06-09 16:22:11.682357, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.682361 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.e8) v4 from osd.9 2014-06-09 16:22:11.682363 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310da700 2014-06-09 16:22:11.682365 7f869af78700 5 -- op tracker -- , seq: 2580, time: 2014-06-09 16:22:11.682364, event: started, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.682373 7f869af78700 5 -- op tracker -- , seq: 2580, time: 2014-06-09 16:22:11.682372, event: done, op: pg_info(1 pgs e243:0.e8) 2014-06-09 16:22:11.682377 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682378 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682384 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682385 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682385 7f869af78700 20 osd.26 243 _dispatch 0x31157500 pg_info(1 pgs e243:0.4da) v4 2014-06-09 16:22:11.682388 7f869af78700 5 -- op tracker -- , seq: 2581, time: 2014-06-09 16:22:11.415551, event: header_read, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.682391 7f869af78700 5 -- op tracker -- , seq: 2581, time: 2014-06-09 16:22:11.415551, event: throttled, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.682395 7f869af78700 5 -- op tracker -- , seq: 2581, time: 2014-06-09 16:22:11.415561, event: all_read, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.682398 7f869af78700 5 -- op tracker -- , seq: 2581, time: 2014-06-09 16:22:11.682384, event: dispatched, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.682401 7f869af78700 5 -- op tracker -- , seq: 2581, time: 2014-06-09 16:22:11.682400, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.682404 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4da) v4 from osd.8 2014-06-09 16:22:11.682405 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31157500 2014-06-09 16:22:11.682407 7f869af78700 5 -- op tracker -- , seq: 2581, time: 2014-06-09 16:22:11.682406, event: started, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.682415 7f869af78700 5 -- op tracker -- , seq: 2581, time: 2014-06-09 16:22:11.682414, event: done, op: pg_info(1 pgs e243:0.4da) 2014-06-09 16:22:11.682419 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682427 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682428 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682429 7f869af78700 20 osd.26 243 _dispatch 0x310e2000 pg_info(1 pgs e243:1.125) v4 2014-06-09 16:22:11.682431 7f869af78700 5 -- op tracker -- , seq: 2582, time: 2014-06-09 16:22:11.423652, event: header_read, op: pg_info(1 pgs e243:1.125) 2014-06-09 16:22:11.682437 7f869af78700 5 -- op tracker -- , seq: 2582, time: 2014-06-09 16:22:11.423652, event: throttled, op: pg_info(1 pgs e243:1.125) 2014-06-09 16:22:11.682440 7f869af78700 5 -- op tracker -- , seq: 2582, time: 2014-06-09 16:22:11.423667, event: all_read, op: pg_info(1 pgs e243:1.125) 2014-06-09 16:22:11.682444 7f869af78700 5 -- op tracker -- , seq: 2582, time: 2014-06-09 16:22:11.682427, event: dispatched, op: pg_info(1 pgs e243:1.125) 2014-06-09 16:22:11.682448 7f869af78700 5 -- op tracker -- , seq: 2582, time: 2014-06-09 16:22:11.682447, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.125) 2014-06-09 16:22:11.682450 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.125) v4 from osd.6 2014-06-09 16:22:11.682451 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e2000 2014-06-09 16:22:11.682454 7f869af78700 5 -- op tracker -- , seq: 2582, time: 2014-06-09 16:22:11.682453, event: started, op: pg_info(1 pgs e243:1.125) 2014-06-09 16:22:11.682463 7f869af78700 5 -- op tracker -- , seq: 2582, time: 2014-06-09 16:22:11.682462, event: done, op: pg_info(1 pgs e243:1.125) 2014-06-09 16:22:11.682467 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682468 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682475 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682475 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682476 7f869af78700 20 osd.26 243 _dispatch 0x19fda700 pg_info(1 pgs e243:0.35b) v4 2014-06-09 16:22:11.682478 7f869af78700 5 -- op tracker -- , seq: 2583, time: 2014-06-09 16:22:11.477825, event: header_read, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682483 7f869af78700 5 -- op tracker -- , seq: 2583, time: 2014-06-09 16:22:11.477825, event: throttled, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682487 7f869af78700 5 -- op tracker -- , seq: 2583, time: 2014-06-09 16:22:11.477837, event: all_read, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682494 7f869af78700 5 -- op tracker -- , seq: 2583, time: 2014-06-09 16:22:11.682474, event: dispatched, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682497 7f869af78700 5 -- op tracker -- , seq: 2583, time: 2014-06-09 16:22:11.682496, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682503 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.35b) v4 from osd.0 2014-06-09 16:22:11.682504 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fda700 2014-06-09 16:22:11.682507 7f869af78700 5 -- op tracker -- , seq: 2583, time: 2014-06-09 16:22:11.682506, event: started, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682515 7f869af78700 5 -- op tracker -- , seq: 2583, time: 2014-06-09 16:22:11.682514, event: done, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682520 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682521 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682527 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682528 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682529 7f869af78700 20 osd.26 243 _dispatch 0x3115a1c0 pg_info(1 pgs e243:3.18d) v4 2014-06-09 16:22:11.682531 7f869af78700 5 -- op tracker -- , seq: 2584, time: 2014-06-09 16:22:11.419910, event: header_read, op: pg_info(1 pgs e243:3.18d) 2014-06-09 16:22:11.682538 7f869af78700 5 -- op tracker -- , seq: 2584, time: 2014-06-09 16:22:11.419910, event: throttled, op: pg_info(1 pgs e243:3.18d) 2014-06-09 16:22:11.682542 7f869af78700 5 -- op tracker -- , seq: 2584, time: 2014-06-09 16:22:11.419920, event: all_read, op: pg_info(1 pgs e243:3.18d) 2014-06-09 16:22:11.682548 7f869af78700 5 -- op tracker -- , seq: 2584, time: 2014-06-09 16:22:11.682527, event: dispatched, op: pg_info(1 pgs e243:3.18d) 2014-06-09 16:22:11.682552 7f869af78700 5 -- op tracker -- , seq: 2584, time: 2014-06-09 16:22:11.682551, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.18d) 2014-06-09 16:22:11.682557 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.18d) v4 from osd.5 2014-06-09 16:22:11.682559 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115a1c0 2014-06-09 16:22:11.682561 7f869af78700 5 -- op tracker -- , seq: 2584, time: 2014-06-09 16:22:11.682560, event: started, op: pg_info(1 pgs e243:3.18d) 2014-06-09 16:22:11.682570 7f869af78700 5 -- op tracker -- , seq: 2584, time: 2014-06-09 16:22:11.682569, event: done, op: pg_info(1 pgs e243:3.18d) 2014-06-09 16:22:11.682576 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682577 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682584 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682584 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682585 7f869af78700 20 osd.26 243 _dispatch 0x1609d340 pg_info(1 pgs e243:0.553) v4 2014-06-09 16:22:11.682588 7f869af78700 5 -- op tracker -- , seq: 2585, time: 2014-06-09 16:22:11.302957, event: header_read, op: pg_info(1 pgs e243:0.553) 2014-06-09 16:22:11.682595 7f869af78700 5 -- op tracker -- , seq: 2585, time: 2014-06-09 16:22:11.302957, event: throttled, op: pg_info(1 pgs e243:0.553) 2014-06-09 16:22:11.682598 7f869af78700 5 -- op tracker -- , seq: 2585, time: 2014-06-09 16:22:11.302963, event: all_read, op: pg_info(1 pgs e243:0.553) 2014-06-09 16:22:11.682604 7f869af78700 5 -- op tracker -- , seq: 2585, time: 2014-06-09 16:22:11.682584, event: dispatched, op: pg_info(1 pgs e243:0.553) 2014-06-09 16:22:11.682611 7f869af78700 5 -- op tracker -- , seq: 2585, time: 2014-06-09 16:22:11.682610, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.553) 2014-06-09 16:22:11.682614 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.553) v4 from osd.2 2014-06-09 16:22:11.682615 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1609d340 2014-06-09 16:22:11.682618 7f869af78700 5 -- op tracker -- , seq: 2585, time: 2014-06-09 16:22:11.682617, event: started, op: pg_info(1 pgs e243:0.553) 2014-06-09 16:22:11.682626 7f869af78700 5 -- op tracker -- , seq: 2585, time: 2014-06-09 16:22:11.682624, event: done, op: pg_info(1 pgs e243:0.553) 2014-06-09 16:22:11.682633 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682634 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682640 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682641 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682642 7f869af78700 20 osd.26 243 _dispatch 0x310d7880 pg_info(1 pgs e243:0.697) v4 2014-06-09 16:22:11.682644 7f869af78700 5 -- op tracker -- , seq: 2586, time: 2014-06-09 16:22:11.398563, event: header_read, op: pg_info(1 pgs e243:0.697) 2014-06-09 16:22:11.682648 7f869af78700 5 -- op tracker -- , seq: 2586, time: 2014-06-09 16:22:11.398564, event: throttled, op: pg_info(1 pgs e243:0.697) 2014-06-09 16:22:11.682652 7f869af78700 5 -- op tracker -- , seq: 2586, time: 2014-06-09 16:22:11.398575, event: all_read, op: pg_info(1 pgs e243:0.697) 2014-06-09 16:22:11.682655 7f869af78700 5 -- op tracker -- , seq: 2586, time: 2014-06-09 16:22:11.682640, event: dispatched, op: pg_info(1 pgs e243:0.697) 2014-06-09 16:22:11.682661 7f869af78700 5 -- op tracker -- , seq: 2586, time: 2014-06-09 16:22:11.682660, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.697) 2014-06-09 16:22:11.682664 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.697) v4 from osd.4 2014-06-09 16:22:11.682665 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d7880 2014-06-09 16:22:11.682668 7f869af78700 5 -- op tracker -- , seq: 2586, time: 2014-06-09 16:22:11.682667, event: started, op: pg_info(1 pgs e243:0.697) 2014-06-09 16:22:11.682677 7f869af78700 5 -- op tracker -- , seq: 2586, time: 2014-06-09 16:22:11.682675, event: done, op: pg_info(1 pgs e243:0.697) 2014-06-09 16:22:11.682681 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682682 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682688 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682690 7f869af78700 20 osd.26 243 _dispatch 0x310e9a40 pg_info(1 pgs e243:3.e0) v4 2014-06-09 16:22:11.682692 7f869af78700 5 -- op tracker -- , seq: 2587, time: 2014-06-09 16:22:11.434972, event: header_read, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.682696 7f869af78700 5 -- op tracker -- , seq: 2587, time: 2014-06-09 16:22:11.434972, event: throttled, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.682699 7f869af78700 5 -- op tracker -- , seq: 2587, time: 2014-06-09 16:22:11.434985, event: all_read, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.682703 7f869af78700 5 -- op tracker -- , seq: 2587, time: 2014-06-09 16:22:11.682688, event: dispatched, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.682707 7f869af78700 5 -- op tracker -- , seq: 2587, time: 2014-06-09 16:22:11.682706, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.682709 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.e0) v4 from osd.7 2014-06-09 16:22:11.682711 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e9a40 2014-06-09 16:22:11.682713 7f869af78700 5 -- op tracker -- , seq: 2587, time: 2014-06-09 16:22:11.682712, event: started, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.682725 7f869af78700 5 -- op tracker -- , seq: 2587, time: 2014-06-09 16:22:11.682724, event: done, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.682729 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682730 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682736 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682737 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682738 7f869af78700 20 osd.26 243 _dispatch 0x2f031340 pg_info(1 pgs e243:2.423) v4 2014-06-09 16:22:11.682740 7f869af78700 5 -- op tracker -- , seq: 2588, time: 2014-06-09 16:22:11.447384, event: header_read, op: pg_info(1 pgs e243:2.423) 2014-06-09 16:22:11.682744 7f869af78700 5 -- op tracker -- , seq: 2588, time: 2014-06-09 16:22:11.447385, event: throttled, op: pg_info(1 pgs e243:2.423) 2014-06-09 16:22:11.682747 7f869af78700 5 -- op tracker -- , seq: 2588, time: 2014-06-09 16:22:11.447397, event: all_read, op: pg_info(1 pgs e243:2.423) 2014-06-09 16:22:11.682751 7f869af78700 5 -- op tracker -- , seq: 2588, time: 2014-06-09 16:22:11.682736, event: dispatched, op: pg_info(1 pgs e243:2.423) 2014-06-09 16:22:11.682755 7f869af78700 5 -- op tracker -- , seq: 2588, time: 2014-06-09 16:22:11.682754, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.423) 2014-06-09 16:22:11.682757 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.423) v4 from osd.3 2014-06-09 16:22:11.682758 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f031340 2014-06-09 16:22:11.682761 7f869af78700 5 -- op tracker -- , seq: 2588, time: 2014-06-09 16:22:11.682760, event: started, op: pg_info(1 pgs e243:2.423) 2014-06-09 16:22:11.682772 7f869af78700 5 -- op tracker -- , seq: 2588, time: 2014-06-09 16:22:11.682771, event: done, op: pg_info(1 pgs e243:2.423) 2014-06-09 16:22:11.682776 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682777 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682784 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682785 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682785 7f869af78700 20 osd.26 243 _dispatch 0x1c039180 pg_info(1 pgs e243:1.91) v4 2014-06-09 16:22:11.682788 7f869af78700 5 -- op tracker -- , seq: 2589, time: 2014-06-09 16:22:11.380181, event: header_read, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.682793 7f869af78700 5 -- op tracker -- , seq: 2589, time: 2014-06-09 16:22:11.380182, event: throttled, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.682796 7f869af78700 5 -- op tracker -- , seq: 2589, time: 2014-06-09 16:22:11.380192, event: all_read, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.682800 7f869af78700 5 -- op tracker -- , seq: 2589, time: 2014-06-09 16:22:11.682784, event: dispatched, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.682803 7f869af78700 5 -- op tracker -- , seq: 2589, time: 2014-06-09 16:22:11.682802, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.682806 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.91) v4 from osd.1 2014-06-09 16:22:11.682807 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c039180 2014-06-09 16:22:11.682810 7f869af78700 5 -- op tracker -- , seq: 2589, time: 2014-06-09 16:22:11.682809, event: started, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.682818 7f869af78700 5 -- op tracker -- , seq: 2589, time: 2014-06-09 16:22:11.682817, event: done, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.682822 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682823 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682829 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682830 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682830 7f869af78700 20 osd.26 243 _dispatch 0x30dc3500 pg_info(1 pgs e243:2.c9) v4 2014-06-09 16:22:11.682833 7f869af78700 5 -- op tracker -- , seq: 2590, time: 2014-06-09 16:22:11.563335, event: header_read, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.682837 7f869af78700 5 -- op tracker -- , seq: 2590, time: 2014-06-09 16:22:11.563335, event: throttled, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.682840 7f869af78700 5 -- op tracker -- , seq: 2590, time: 2014-06-09 16:22:11.563350, event: all_read, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.682845 7f869af78700 5 -- op tracker -- , seq: 2590, time: 2014-06-09 16:22:11.682829, event: dispatched, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.682848 7f869af78700 5 -- op tracker -- , seq: 2590, time: 2014-06-09 16:22:11.682847, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.682854 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.c9) v4 from osd.9 2014-06-09 16:22:11.682855 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc3500 2014-06-09 16:22:11.682858 7f869af78700 5 -- op tracker -- , seq: 2590, time: 2014-06-09 16:22:11.682857, event: started, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.682866 7f869af78700 5 -- op tracker -- , seq: 2590, time: 2014-06-09 16:22:11.682865, event: done, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.682870 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682871 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682878 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682878 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682879 7f869af78700 20 osd.26 243 _dispatch 0x31157340 pg_info(1 pgs e243:0.301) v4 2014-06-09 16:22:11.682882 7f869af78700 5 -- op tracker -- , seq: 2591, time: 2014-06-09 16:22:11.415565, event: header_read, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.682887 7f869af78700 5 -- op tracker -- , seq: 2591, time: 2014-06-09 16:22:11.415565, event: throttled, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.682890 7f869af78700 5 -- op tracker -- , seq: 2591, time: 2014-06-09 16:22:11.415577, event: all_read, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.682893 7f869af78700 5 -- op tracker -- , seq: 2591, time: 2014-06-09 16:22:11.682878, event: dispatched, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.682897 7f869af78700 5 -- op tracker -- , seq: 2591, time: 2014-06-09 16:22:11.682896, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.682899 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.301) v4 from osd.8 2014-06-09 16:22:11.682901 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31157340 2014-06-09 16:22:11.682903 7f869af78700 5 -- op tracker -- , seq: 2591, time: 2014-06-09 16:22:11.682902, event: started, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.682911 7f869af78700 5 -- op tracker -- , seq: 2591, time: 2014-06-09 16:22:11.682910, event: done, op: pg_info(1 pgs e243:0.301) 2014-06-09 16:22:11.682915 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682916 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682922 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682923 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682923 7f869af78700 20 osd.26 243 _dispatch 0x3115bdc0 pg_info(1 pgs e243:0.6bb) v4 2014-06-09 16:22:11.682926 7f869af78700 5 -- op tracker -- , seq: 2592, time: 2014-06-09 16:22:11.423672, event: header_read, op: pg_info(1 pgs e243:0.6bb) 2014-06-09 16:22:11.682929 7f869af78700 5 -- op tracker -- , seq: 2592, time: 2014-06-09 16:22:11.423672, event: throttled, op: pg_info(1 pgs e243:0.6bb) 2014-06-09 16:22:11.682933 7f869af78700 5 -- op tracker -- , seq: 2592, time: 2014-06-09 16:22:11.423686, event: all_read, op: pg_info(1 pgs e243:0.6bb) 2014-06-09 16:22:11.682940 7f869af78700 5 -- op tracker -- , seq: 2592, time: 2014-06-09 16:22:11.682922, event: dispatched, op: pg_info(1 pgs e243:0.6bb) 2014-06-09 16:22:11.682943 7f869af78700 5 -- op tracker -- , seq: 2592, time: 2014-06-09 16:22:11.682942, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6bb) 2014-06-09 16:22:11.682947 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6bb) v4 from osd.6 2014-06-09 16:22:11.682948 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115bdc0 2014-06-09 16:22:11.682951 7f869af78700 5 -- op tracker -- , seq: 2592, time: 2014-06-09 16:22:11.682950, event: started, op: pg_info(1 pgs e243:0.6bb) 2014-06-09 16:22:11.682960 7f869af78700 5 -- op tracker -- , seq: 2592, time: 2014-06-09 16:22:11.682959, event: done, op: pg_info(1 pgs e243:0.6bb) 2014-06-09 16:22:11.682965 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682966 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682972 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.682973 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.682974 7f869af78700 20 osd.26 243 _dispatch 0x31167a40 pg_info(1 pgs e243:0.35b) v4 2014-06-09 16:22:11.682976 7f869af78700 5 -- op tracker -- , seq: 2593, time: 2014-06-09 16:22:11.477844, event: header_read, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682980 7f869af78700 5 -- op tracker -- , seq: 2593, time: 2014-06-09 16:22:11.477845, event: throttled, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682983 7f869af78700 5 -- op tracker -- , seq: 2593, time: 2014-06-09 16:22:11.477857, event: all_read, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682988 7f869af78700 5 -- op tracker -- , seq: 2593, time: 2014-06-09 16:22:11.682972, event: dispatched, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682992 7f869af78700 5 -- op tracker -- , seq: 2593, time: 2014-06-09 16:22:11.682991, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.682997 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.35b) v4 from osd.0 2014-06-09 16:22:11.682999 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31167a40 2014-06-09 16:22:11.683001 7f869af78700 5 -- op tracker -- , seq: 2593, time: 2014-06-09 16:22:11.683000, event: started, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.683009 7f869af78700 5 -- op tracker -- , seq: 2593, time: 2014-06-09 16:22:11.683008, event: done, op: pg_info(1 pgs e243:0.35b) 2014-06-09 16:22:11.683014 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683015 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683021 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683022 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683022 7f869af78700 20 osd.26 243 _dispatch 0x3115a000 pg_info(1 pgs e243:2.266) v4 2014-06-09 16:22:11.683025 7f869af78700 5 -- op tracker -- , seq: 2594, time: 2014-06-09 16:22:11.435632, event: header_read, op: pg_info(1 pgs e243:2.266) 2014-06-09 16:22:11.683029 7f869af78700 5 -- op tracker -- , seq: 2594, time: 2014-06-09 16:22:11.435633, event: throttled, op: pg_info(1 pgs e243:2.266) 2014-06-09 16:22:11.683032 7f869af78700 5 -- op tracker -- , seq: 2594, time: 2014-06-09 16:22:11.435647, event: all_read, op: pg_info(1 pgs e243:2.266) 2014-06-09 16:22:11.683037 7f869af78700 5 -- op tracker -- , seq: 2594, time: 2014-06-09 16:22:11.683021, event: dispatched, op: pg_info(1 pgs e243:2.266) 2014-06-09 16:22:11.683041 7f869af78700 5 -- op tracker -- , seq: 2594, time: 2014-06-09 16:22:11.683040, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.266) 2014-06-09 16:22:11.683046 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.266) v4 from osd.5 2014-06-09 16:22:11.683048 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115a000 2014-06-09 16:22:11.683050 7f869af78700 5 -- op tracker -- , seq: 2594, time: 2014-06-09 16:22:11.683049, event: started, op: pg_info(1 pgs e243:2.266) 2014-06-09 16:22:11.683059 7f869af78700 5 -- op tracker -- , seq: 2594, time: 2014-06-09 16:22:11.683058, event: done, op: pg_info(1 pgs e243:2.266) 2014-06-09 16:22:11.683063 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683064 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683070 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683071 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683072 7f869af78700 20 osd.26 243 _dispatch 0x10d0ca80 pg_info(1 pgs e243:3.7af) v4 2014-06-09 16:22:11.683074 7f869af78700 5 -- op tracker -- , seq: 2595, time: 2014-06-09 16:22:11.302965, event: header_read, op: pg_info(1 pgs e243:3.7af) 2014-06-09 16:22:11.683079 7f869af78700 5 -- op tracker -- , seq: 2595, time: 2014-06-09 16:22:11.302965, event: throttled, op: pg_info(1 pgs e243:3.7af) 2014-06-09 16:22:11.683083 7f869af78700 5 -- op tracker -- , seq: 2595, time: 2014-06-09 16:22:11.302979, event: all_read, op: pg_info(1 pgs e243:3.7af) 2014-06-09 16:22:11.683090 7f869af78700 5 -- op tracker -- , seq: 2595, time: 2014-06-09 16:22:11.683070, event: dispatched, op: pg_info(1 pgs e243:3.7af) 2014-06-09 16:22:11.683094 7f869af78700 5 -- op tracker -- , seq: 2595, time: 2014-06-09 16:22:11.683093, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7af) 2014-06-09 16:22:11.683097 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7af) v4 from osd.2 2014-06-09 16:22:11.683099 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x10d0ca80 2014-06-09 16:22:11.683101 7f869af78700 5 -- op tracker -- , seq: 2595, time: 2014-06-09 16:22:11.683100, event: started, op: pg_info(1 pgs e243:3.7af) 2014-06-09 16:22:11.683111 7f869af78700 5 -- op tracker -- , seq: 2595, time: 2014-06-09 16:22:11.683110, event: done, op: pg_info(1 pgs e243:3.7af) 2014-06-09 16:22:11.683116 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683117 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683124 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683125 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683126 7f869af78700 20 osd.26 243 _dispatch 0x310d76c0 pg_notify(1.75b(20),1.76c(20) epoch 243) v5 2014-06-09 16:22:11.683129 7f869af78700 5 -- op tracker -- , seq: 2596, time: 2014-06-09 16:22:11.419509, event: header_read, op: pg_notify(1.75b(20),1.76c(20) epoch 243) 2014-06-09 16:22:11.683133 7f869af78700 5 -- op tracker -- , seq: 2596, time: 2014-06-09 16:22:11.419510, event: throttled, op: pg_notify(1.75b(20),1.76c(20) epoch 243) 2014-06-09 16:22:11.683137 7f869af78700 5 -- op tracker -- , seq: 2596, time: 2014-06-09 16:22:11.419564, event: all_read, op: pg_notify(1.75b(20),1.76c(20) epoch 243) 2014-06-09 16:22:11.683142 7f869af78700 5 -- op tracker -- , seq: 2596, time: 2014-06-09 16:22:11.683124, event: dispatched, op: pg_notify(1.75b(20),1.76c(20) epoch 243) 2014-06-09 16:22:11.683146 7f869af78700 5 -- op tracker -- , seq: 2596, time: 2014-06-09 16:22:11.683144, event: waiting_for_osdmap, op: pg_notify(1.75b(20),1.76c(20) epoch 243) 2014-06-09 16:22:11.683149 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.683150 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d76c0 2014-06-09 16:22:11.683152 7f869af78700 5 -- op tracker -- , seq: 2596, time: 2014-06-09 16:22:11.683151, event: started, op: pg_notify(1.75b(20),1.76c(20) epoch 243) 2014-06-09 16:22:11.683165 7f869af78700 5 -- op tracker -- , seq: 2596, time: 2014-06-09 16:22:11.683164, event: done, op: pg_notify(1.75b(20),1.76c(20) epoch 243) 2014-06-09 16:22:11.683170 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683171 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683186 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683187 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683188 7f869af78700 20 osd.26 243 _dispatch 0x310e9880 pg_info(1 pgs e243:3.e0) v4 2014-06-09 16:22:11.683190 7f869af78700 5 -- op tracker -- , seq: 2597, time: 2014-06-09 16:22:11.434994, event: header_read, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.683194 7f869af78700 5 -- op tracker -- , seq: 2597, time: 2014-06-09 16:22:11.434994, event: throttled, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.683201 7f869af78700 5 -- op tracker -- , seq: 2597, time: 2014-06-09 16:22:11.435006, event: all_read, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.683205 7f869af78700 5 -- op tracker -- , seq: 2597, time: 2014-06-09 16:22:11.683186, event: dispatched, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.683208 7f869af78700 5 -- op tracker -- , seq: 2597, time: 2014-06-09 16:22:11.683207, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.683211 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.e0) v4 from osd.7 2014-06-09 16:22:11.683212 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e9880 2014-06-09 16:22:11.683215 7f869af78700 5 -- op tracker -- , seq: 2597, time: 2014-06-09 16:22:11.683214, event: started, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.683223 7f869af78700 5 -- op tracker -- , seq: 2597, time: 2014-06-09 16:22:11.683222, event: done, op: pg_info(1 pgs e243:3.e0) 2014-06-09 16:22:11.683227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683228 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683234 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683235 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683236 7f869af78700 20 osd.26 243 _dispatch 0x2f031180 pg_info(1 pgs e243:1.1bc) v4 2014-06-09 16:22:11.683238 7f869af78700 5 -- op tracker -- , seq: 2598, time: 2014-06-09 16:22:11.447519, event: header_read, op: pg_info(1 pgs e243:1.1bc) 2014-06-09 16:22:11.683243 7f869af78700 5 -- op tracker -- , seq: 2598, time: 2014-06-09 16:22:11.447520, event: throttled, op: pg_info(1 pgs e243:1.1bc) 2014-06-09 16:22:11.683246 7f869af78700 5 -- op tracker -- , seq: 2598, time: 2014-06-09 16:22:11.447532, event: all_read, op: pg_info(1 pgs e243:1.1bc) 2014-06-09 16:22:11.683249 7f869af78700 5 -- op tracker -- , seq: 2598, time: 2014-06-09 16:22:11.683234, event: dispatched, op: pg_info(1 pgs e243:1.1bc) 2014-06-09 16:22:11.683253 7f869af78700 5 -- op tracker -- , seq: 2598, time: 2014-06-09 16:22:11.683252, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1bc) 2014-06-09 16:22:11.683256 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1bc) v4 from osd.3 2014-06-09 16:22:11.683258 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f031180 2014-06-09 16:22:11.683260 7f869af78700 5 -- op tracker -- , seq: 2598, time: 2014-06-09 16:22:11.683259, event: started, op: pg_info(1 pgs e243:1.1bc) 2014-06-09 16:22:11.683270 7f869af78700 5 -- op tracker -- , seq: 2598, time: 2014-06-09 16:22:11.683268, event: done, op: pg_info(1 pgs e243:1.1bc) 2014-06-09 16:22:11.683273 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683281 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683282 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683283 7f869af78700 20 osd.26 243 _dispatch 0x1a039dc0 pg_info(1 pgs e243:1.91) v4 2014-06-09 16:22:11.683285 7f869af78700 5 -- op tracker -- , seq: 2599, time: 2014-06-09 16:22:11.380371, event: header_read, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.683289 7f869af78700 5 -- op tracker -- , seq: 2599, time: 2014-06-09 16:22:11.380372, event: throttled, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.683292 7f869af78700 5 -- op tracker -- , seq: 2599, time: 2014-06-09 16:22:11.380383, event: all_read, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.683295 7f869af78700 5 -- op tracker -- , seq: 2599, time: 2014-06-09 16:22:11.683281, event: dispatched, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.683299 7f869af78700 5 -- op tracker -- , seq: 2599, time: 2014-06-09 16:22:11.683298, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.683301 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.91) v4 from osd.1 2014-06-09 16:22:11.683303 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a039dc0 2014-06-09 16:22:11.683305 7f869af78700 5 -- op tracker -- , seq: 2599, time: 2014-06-09 16:22:11.683304, event: started, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.683314 7f869af78700 5 -- op tracker -- , seq: 2599, time: 2014-06-09 16:22:11.683313, event: done, op: pg_info(1 pgs e243:1.91) 2014-06-09 16:22:11.683319 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683320 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683327 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683328 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683328 7f869af78700 20 osd.26 243 _dispatch 0x30ebac40 pg_info(1 pgs e243:1.6e8) v4 2014-06-09 16:22:11.683331 7f869af78700 5 -- op tracker -- , seq: 2600, time: 2014-06-09 16:22:11.567631, event: header_read, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.683334 7f869af78700 5 -- op tracker -- , seq: 2600, time: 2014-06-09 16:22:11.567632, event: throttled, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.683338 7f869af78700 5 -- op tracker -- , seq: 2600, time: 2014-06-09 16:22:11.567647, event: all_read, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.683341 7f869af78700 5 -- op tracker -- , seq: 2600, time: 2014-06-09 16:22:11.683327, event: dispatched, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.683346 7f869af78700 5 -- op tracker -- , seq: 2600, time: 2014-06-09 16:22:11.683344, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.683348 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6e8) v4 from osd.9 2014-06-09 16:22:11.683349 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebac40 2014-06-09 16:22:11.683352 7f869af78700 5 -- op tracker -- , seq: 2600, time: 2014-06-09 16:22:11.683351, event: started, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.683360 7f869af78700 5 -- op tracker -- , seq: 2600, time: 2014-06-09 16:22:11.683359, event: done, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.683364 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683365 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683371 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683372 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683373 7f869af78700 20 osd.26 243 _dispatch 0x31157180 pg_info(1 pgs e243:0.451) v4 2014-06-09 16:22:11.683375 7f869af78700 5 -- op tracker -- , seq: 2601, time: 2014-06-09 16:22:11.429902, event: header_read, op: pg_info(1 pgs e243:0.451) 2014-06-09 16:22:11.683379 7f869af78700 5 -- op tracker -- , seq: 2601, time: 2014-06-09 16:22:11.429903, event: throttled, op: pg_info(1 pgs e243:0.451) 2014-06-09 16:22:11.683383 7f869af78700 5 -- op tracker -- , seq: 2601, time: 2014-06-09 16:22:11.429921, event: all_read, op: pg_info(1 pgs e243:0.451) 2014-06-09 16:22:11.683387 7f869af78700 5 -- op tracker -- , seq: 2601, time: 2014-06-09 16:22:11.683371, event: dispatched, op: pg_info(1 pgs e243:0.451) 2014-06-09 16:22:11.683391 7f869af78700 5 -- op tracker -- , seq: 2601, time: 2014-06-09 16:22:11.683390, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.451) 2014-06-09 16:22:11.683397 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.451) v4 from osd.8 2014-06-09 16:22:11.683398 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31157180 2014-06-09 16:22:11.683401 7f869af78700 5 -- op tracker -- , seq: 2601, time: 2014-06-09 16:22:11.683400, event: started, op: pg_info(1 pgs e243:0.451) 2014-06-09 16:22:11.683409 7f869af78700 5 -- op tracker -- , seq: 2601, time: 2014-06-09 16:22:11.683408, event: done, op: pg_info(1 pgs e243:0.451) 2014-06-09 16:22:11.683413 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683414 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683421 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683423 7f869af78700 20 osd.26 243 _dispatch 0x3115bc00 pg_info(1 pgs e243:1.685) v4 2014-06-09 16:22:11.683425 7f869af78700 5 -- op tracker -- , seq: 2602, time: 2014-06-09 16:22:11.423691, event: header_read, op: pg_info(1 pgs e243:1.685) 2014-06-09 16:22:11.683432 7f869af78700 5 -- op tracker -- , seq: 2602, time: 2014-06-09 16:22:11.423692, event: throttled, op: pg_info(1 pgs e243:1.685) 2014-06-09 16:22:11.683436 7f869af78700 5 -- op tracker -- , seq: 2602, time: 2014-06-09 16:22:11.423702, event: all_read, op: pg_info(1 pgs e243:1.685) 2014-06-09 16:22:11.683442 7f869af78700 5 -- op tracker -- , seq: 2602, time: 2014-06-09 16:22:11.683421, event: dispatched, op: pg_info(1 pgs e243:1.685) 2014-06-09 16:22:11.683446 7f869af78700 5 -- op tracker -- , seq: 2602, time: 2014-06-09 16:22:11.683445, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.685) 2014-06-09 16:22:11.683451 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.685) v4 from osd.6 2014-06-09 16:22:11.683453 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115bc00 2014-06-09 16:22:11.683455 7f869af78700 5 -- op tracker -- , seq: 2602, time: 2014-06-09 16:22:11.683454, event: started, op: pg_info(1 pgs e243:1.685) 2014-06-09 16:22:11.683464 7f869af78700 5 -- op tracker -- , seq: 2602, time: 2014-06-09 16:22:11.683463, event: done, op: pg_info(1 pgs e243:1.685) 2014-06-09 16:22:11.683468 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683469 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683477 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683477 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683478 7f869af78700 20 osd.26 243 _dispatch 0x3044c8c0 pg_info(1 pgs e243:3.603) v4 2014-06-09 16:22:11.683481 7f869af78700 5 -- op tracker -- , seq: 2603, time: 2014-06-09 16:22:11.545710, event: header_read, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.683484 7f869af78700 5 -- op tracker -- , seq: 2603, time: 2014-06-09 16:22:11.545710, event: throttled, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.683488 7f869af78700 5 -- op tracker -- , seq: 2603, time: 2014-06-09 16:22:11.545722, event: all_read, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.683495 7f869af78700 5 -- op tracker -- , seq: 2603, time: 2014-06-09 16:22:11.683477, event: dispatched, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.683499 7f869af78700 5 -- op tracker -- , seq: 2603, time: 2014-06-09 16:22:11.683498, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.683502 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.603) v4 from osd.0 2014-06-09 16:22:11.683504 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044c8c0 2014-06-09 16:22:11.683506 7f869af78700 5 -- op tracker -- , seq: 2603, time: 2014-06-09 16:22:11.683505, event: started, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.683517 7f869af78700 5 -- op tracker -- , seq: 2603, time: 2014-06-09 16:22:11.683516, event: done, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.683522 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683522 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683529 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683530 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683530 7f869af78700 20 osd.26 243 _dispatch 0x30dafc00 pg_info(1 pgs e243:2.1b1) v4 2014-06-09 16:22:11.683533 7f869af78700 5 -- op tracker -- , seq: 2604, time: 2014-06-09 16:22:11.439640, event: header_read, op: pg_info(1 pgs e243:2.1b1) 2014-06-09 16:22:11.683540 7f869af78700 5 -- op tracker -- , seq: 2604, time: 2014-06-09 16:22:11.439641, event: throttled, op: pg_info(1 pgs e243:2.1b1) 2014-06-09 16:22:11.683544 7f869af78700 5 -- op tracker -- , seq: 2604, time: 2014-06-09 16:22:11.439656, event: all_read, op: pg_info(1 pgs e243:2.1b1) 2014-06-09 16:22:11.683547 7f869af78700 5 -- op tracker -- , seq: 2604, time: 2014-06-09 16:22:11.683529, event: dispatched, op: pg_info(1 pgs e243:2.1b1) 2014-06-09 16:22:11.683551 7f869af78700 5 -- op tracker -- , seq: 2604, time: 2014-06-09 16:22:11.683550, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1b1) 2014-06-09 16:22:11.683554 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1b1) v4 from osd.5 2014-06-09 16:22:11.683556 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dafc00 2014-06-09 16:22:11.683558 7f869af78700 5 -- op tracker -- , seq: 2604, time: 2014-06-09 16:22:11.683557, event: started, op: pg_info(1 pgs e243:2.1b1) 2014-06-09 16:22:11.683568 7f869af78700 5 -- op tracker -- , seq: 2604, time: 2014-06-09 16:22:11.683567, event: done, op: pg_info(1 pgs e243:2.1b1) 2014-06-09 16:22:11.683572 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683573 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683579 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683580 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683580 7f869af78700 20 osd.26 243 _dispatch 0x30b8d880 pg_info(1 pgs e243:0.2c2) v4 2014-06-09 16:22:11.683583 7f869af78700 5 -- op tracker -- , seq: 2605, time: 2014-06-09 16:22:11.302983, event: header_read, op: pg_info(1 pgs e243:0.2c2) 2014-06-09 16:22:11.683587 7f869af78700 5 -- op tracker -- , seq: 2605, time: 2014-06-09 16:22:11.302983, event: throttled, op: pg_info(1 pgs e243:0.2c2) 2014-06-09 16:22:11.683590 7f869af78700 5 -- op tracker -- , seq: 2605, time: 2014-06-09 16:22:11.303011, event: all_read, op: pg_info(1 pgs e243:0.2c2) 2014-06-09 16:22:11.683593 7f869af78700 5 -- op tracker -- , seq: 2605, time: 2014-06-09 16:22:11.683579, event: dispatched, op: pg_info(1 pgs e243:0.2c2) 2014-06-09 16:22:11.683597 7f869af78700 5 -- op tracker -- , seq: 2605, time: 2014-06-09 16:22:11.683596, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2c2) 2014-06-09 16:22:11.683599 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2c2) v4 from osd.2 2014-06-09 16:22:11.683600 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8d880 2014-06-09 16:22:11.683603 7f869af78700 5 -- op tracker -- , seq: 2605, time: 2014-06-09 16:22:11.683602, event: started, op: pg_info(1 pgs e243:0.2c2) 2014-06-09 16:22:11.683618 7f869af78700 5 -- op tracker -- , seq: 2605, time: 2014-06-09 16:22:11.683617, event: done, op: pg_info(1 pgs e243:0.2c2) 2014-06-09 16:22:11.683628 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683629 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683637 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683638 7f869af78700 20 osd.26 243 _dispatch 0x2fe32c00 pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.683642 7f869af78700 5 -- op tracker -- , seq: 2606, time: 2014-06-09 16:22:11.419575, event: header_read, op: pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683648 7f869af78700 5 -- op tracker -- , seq: 2606, time: 2014-06-09 16:22:11.419575, event: throttled, op: pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683651 7f869af78700 5 -- op tracker -- , seq: 2606, time: 2014-06-09 16:22:11.419587, event: all_read, op: pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683655 7f869af78700 5 -- op tracker -- , seq: 2606, time: 2014-06-09 16:22:11.683636, event: dispatched, op: pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683660 7f869af78700 5 -- op tracker -- , seq: 2606, time: 2014-06-09 16:22:11.683659, event: waiting_for_osdmap, op: pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683663 7f869af78700 7 osd.26 243 handle_pg_log pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.683665 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe32c00 2014-06-09 16:22:11.683668 7f869af78700 5 -- op tracker -- , seq: 2606, time: 2014-06-09 16:22:11.683667, event: started, op: pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683675 7f869af78700 5 -- op tracker -- , seq: 2606, time: 2014-06-09 16:22:11.683674, event: done, op: pg_log(0.1da epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683680 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683681 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683688 7f869af78700 20 osd.26 243 _dispatch 0x2d095a40 pg_info(1 pgs e243:0.2c3) v4 2014-06-09 16:22:11.683690 7f869af78700 5 -- op tracker -- , seq: 2607, time: 2014-06-09 16:22:11.479634, event: header_read, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.683699 7f869af78700 5 -- op tracker -- , seq: 2607, time: 2014-06-09 16:22:11.479635, event: throttled, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.683703 7f869af78700 5 -- op tracker -- , seq: 2607, time: 2014-06-09 16:22:11.479647, event: all_read, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.683706 7f869af78700 5 -- op tracker -- , seq: 2607, time: 2014-06-09 16:22:11.683686, event: dispatched, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.683711 7f869af78700 5 -- op tracker -- , seq: 2607, time: 2014-06-09 16:22:11.683710, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.683713 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2c3) v4 from osd.7 2014-06-09 16:22:11.683715 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d095a40 2014-06-09 16:22:11.683717 7f869af78700 5 -- op tracker -- , seq: 2607, time: 2014-06-09 16:22:11.683717, event: started, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.683726 7f869af78700 5 -- op tracker -- , seq: 2607, time: 2014-06-09 16:22:11.683725, event: done, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.683731 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683732 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683740 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683740 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683741 7f869af78700 20 osd.26 243 _dispatch 0x240f2000 pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.683744 7f869af78700 5 -- op tracker -- , seq: 2608, time: 2014-06-09 16:22:11.447636, event: header_read, op: pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683749 7f869af78700 5 -- op tracker -- , seq: 2608, time: 2014-06-09 16:22:11.447638, event: throttled, op: pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683757 7f869af78700 5 -- op tracker -- , seq: 2608, time: 2014-06-09 16:22:11.447657, event: all_read, op: pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683761 7f869af78700 5 -- op tracker -- , seq: 2608, time: 2014-06-09 16:22:11.683739, event: dispatched, op: pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683765 7f869af78700 5 -- op tracker -- , seq: 2608, time: 2014-06-09 16:22:11.683764, event: waiting_for_osdmap, op: pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683768 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) v4 from osd.3 2014-06-09 16:22:11.683770 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240f2000 2014-06-09 16:22:11.683773 7f869af78700 5 -- op tracker -- , seq: 2608, time: 2014-06-09 16:22:11.683772, event: started, op: pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683781 7f869af78700 5 -- op tracker -- , seq: 2608, time: 2014-06-09 16:22:11.683779, event: done, op: pg_log(3.5 epoch 243 log log((214'622,214'624], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.683785 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683786 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683792 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683793 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683793 7f869af78700 20 osd.26 243 _dispatch 0x30bd9500 pg_info(1 pgs e243:2.3f0) v4 2014-06-09 16:22:11.683796 7f869af78700 5 -- op tracker -- , seq: 2609, time: 2014-06-09 16:22:11.403104, event: header_read, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.683800 7f869af78700 5 -- op tracker -- , seq: 2609, time: 2014-06-09 16:22:11.403105, event: throttled, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.683803 7f869af78700 5 -- op tracker -- , seq: 2609, time: 2014-06-09 16:22:11.403118, event: all_read, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.683811 7f869af78700 5 -- op tracker -- , seq: 2609, time: 2014-06-09 16:22:11.683792, event: dispatched, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.683815 7f869af78700 5 -- op tracker -- , seq: 2609, time: 2014-06-09 16:22:11.683814, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.683817 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3f0) v4 from osd.1 2014-06-09 16:22:11.683819 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd9500 2014-06-09 16:22:11.683821 7f869af78700 5 -- op tracker -- , seq: 2609, time: 2014-06-09 16:22:11.683820, event: started, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.683830 7f869af78700 5 -- op tracker -- , seq: 2609, time: 2014-06-09 16:22:11.683829, event: done, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.683834 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683835 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683842 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683842 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683843 7f869af78700 20 osd.26 243 _dispatch 0x310da540 pg_info(1 pgs e243:2.c9) v4 2014-06-09 16:22:11.683845 7f869af78700 5 -- op tracker -- , seq: 2610, time: 2014-06-09 16:22:11.567666, event: header_read, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.683852 7f869af78700 5 -- op tracker -- , seq: 2610, time: 2014-06-09 16:22:11.567666, event: throttled, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.683856 7f869af78700 5 -- op tracker -- , seq: 2610, time: 2014-06-09 16:22:11.567683, event: all_read, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.683860 7f869af78700 5 -- op tracker -- , seq: 2610, time: 2014-06-09 16:22:11.683841, event: dispatched, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.683864 7f869af78700 5 -- op tracker -- , seq: 2610, time: 2014-06-09 16:22:11.683863, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.683871 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.c9) v4 from osd.9 2014-06-09 16:22:11.683873 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310da540 2014-06-09 16:22:11.683875 7f869af78700 5 -- op tracker -- , seq: 2610, time: 2014-06-09 16:22:11.683874, event: started, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.683884 7f869af78700 5 -- op tracker -- , seq: 2610, time: 2014-06-09 16:22:11.683883, event: done, op: pg_info(1 pgs e243:2.c9) 2014-06-09 16:22:11.683891 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683892 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683899 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683900 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683901 7f869af78700 20 osd.26 243 _dispatch 0x31156fc0 pg_info(1 pgs e243:3.3b4) v4 2014-06-09 16:22:11.683903 7f869af78700 5 -- op tracker -- , seq: 2611, time: 2014-06-09 16:22:11.429951, event: header_read, op: pg_info(1 pgs e243:3.3b4) 2014-06-09 16:22:11.683907 7f869af78700 5 -- op tracker -- , seq: 2611, time: 2014-06-09 16:22:11.429951, event: throttled, op: pg_info(1 pgs e243:3.3b4) 2014-06-09 16:22:11.683910 7f869af78700 5 -- op tracker -- , seq: 2611, time: 2014-06-09 16:22:11.429963, event: all_read, op: pg_info(1 pgs e243:3.3b4) 2014-06-09 16:22:11.683914 7f869af78700 5 -- op tracker -- , seq: 2611, time: 2014-06-09 16:22:11.683899, event: dispatched, op: pg_info(1 pgs e243:3.3b4) 2014-06-09 16:22:11.683917 7f869af78700 5 -- op tracker -- , seq: 2611, time: 2014-06-09 16:22:11.683916, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3b4) 2014-06-09 16:22:11.683919 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3b4) v4 from osd.8 2014-06-09 16:22:11.683921 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31156fc0 2014-06-09 16:22:11.683923 7f869af78700 5 -- op tracker -- , seq: 2611, time: 2014-06-09 16:22:11.683922, event: started, op: pg_info(1 pgs e243:3.3b4) 2014-06-09 16:22:11.683932 7f869af78700 5 -- op tracker -- , seq: 2611, time: 2014-06-09 16:22:11.683931, event: done, op: pg_info(1 pgs e243:3.3b4) 2014-06-09 16:22:11.683936 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683937 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683943 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683944 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683945 7f869af78700 20 osd.26 243 _dispatch 0x31163dc0 pg_info(1 pgs e243:1.3f9) v4 2014-06-09 16:22:11.683947 7f869af78700 5 -- op tracker -- , seq: 2612, time: 2014-06-09 16:22:11.423705, event: header_read, op: pg_info(1 pgs e243:1.3f9) 2014-06-09 16:22:11.683951 7f869af78700 5 -- op tracker -- , seq: 2612, time: 2014-06-09 16:22:11.423706, event: throttled, op: pg_info(1 pgs e243:1.3f9) 2014-06-09 16:22:11.683954 7f869af78700 5 -- op tracker -- , seq: 2612, time: 2014-06-09 16:22:11.423731, event: all_read, op: pg_info(1 pgs e243:1.3f9) 2014-06-09 16:22:11.683958 7f869af78700 5 -- op tracker -- , seq: 2612, time: 2014-06-09 16:22:11.683943, event: dispatched, op: pg_info(1 pgs e243:1.3f9) 2014-06-09 16:22:11.683964 7f869af78700 5 -- op tracker -- , seq: 2612, time: 2014-06-09 16:22:11.683963, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3f9) 2014-06-09 16:22:11.683967 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3f9) v4 from osd.6 2014-06-09 16:22:11.683968 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31163dc0 2014-06-09 16:22:11.683971 7f869af78700 5 -- op tracker -- , seq: 2612, time: 2014-06-09 16:22:11.683970, event: started, op: pg_info(1 pgs e243:1.3f9) 2014-06-09 16:22:11.683979 7f869af78700 5 -- op tracker -- , seq: 2612, time: 2014-06-09 16:22:11.683978, event: done, op: pg_info(1 pgs e243:1.3f9) 2014-06-09 16:22:11.683984 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683985 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683992 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.683993 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.683993 7f869af78700 20 osd.26 243 _dispatch 0x31167880 pg_info(1 pgs e243:3.603) v4 2014-06-09 16:22:11.683996 7f869af78700 5 -- op tracker -- , seq: 2613, time: 2014-06-09 16:22:11.546136, event: header_read, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.684000 7f869af78700 5 -- op tracker -- , seq: 2613, time: 2014-06-09 16:22:11.546136, event: throttled, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.684004 7f869af78700 5 -- op tracker -- , seq: 2613, time: 2014-06-09 16:22:11.546149, event: all_read, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.684007 7f869af78700 5 -- op tracker -- , seq: 2613, time: 2014-06-09 16:22:11.683992, event: dispatched, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.684015 7f869af78700 5 -- op tracker -- , seq: 2613, time: 2014-06-09 16:22:11.684014, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.684018 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.603) v4 from osd.0 2014-06-09 16:22:11.684019 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31167880 2014-06-09 16:22:11.684022 7f869af78700 5 -- op tracker -- , seq: 2613, time: 2014-06-09 16:22:11.684021, event: started, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.684031 7f869af78700 5 -- op tracker -- , seq: 2613, time: 2014-06-09 16:22:11.684030, event: done, op: pg_info(1 pgs e243:3.603) 2014-06-09 16:22:11.684035 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684036 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684045 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684046 7f869af78700 20 osd.26 243 _dispatch 0x30dafa40 pg_info(1 pgs e243:1.1) v4 2014-06-09 16:22:11.684048 7f869af78700 5 -- op tracker -- , seq: 2614, time: 2014-06-09 16:22:11.439666, event: header_read, op: pg_info(1 pgs e243:1.1) 2014-06-09 16:22:11.684056 7f869af78700 5 -- op tracker -- , seq: 2614, time: 2014-06-09 16:22:11.439666, event: throttled, op: pg_info(1 pgs e243:1.1) 2014-06-09 16:22:11.684059 7f869af78700 5 -- op tracker -- , seq: 2614, time: 2014-06-09 16:22:11.439684, event: all_read, op: pg_info(1 pgs e243:1.1) 2014-06-09 16:22:11.684064 7f869af78700 5 -- op tracker -- , seq: 2614, time: 2014-06-09 16:22:11.684044, event: dispatched, op: pg_info(1 pgs e243:1.1) 2014-06-09 16:22:11.684068 7f869af78700 5 -- op tracker -- , seq: 2614, time: 2014-06-09 16:22:11.684067, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1) 2014-06-09 16:22:11.684073 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1) v4 from osd.5 2014-06-09 16:22:11.684074 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dafa40 2014-06-09 16:22:11.684077 7f869af78700 5 -- op tracker -- , seq: 2614, time: 2014-06-09 16:22:11.684076, event: started, op: pg_info(1 pgs e243:1.1) 2014-06-09 16:22:11.684085 7f869af78700 5 -- op tracker -- , seq: 2614, time: 2014-06-09 16:22:11.684084, event: done, op: pg_info(1 pgs e243:1.1) 2014-06-09 16:22:11.684089 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684090 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684097 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684098 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684098 7f869af78700 20 osd.26 243 _dispatch 0x30b8d6c0 pg_info(1 pgs e243:3.60e) v4 2014-06-09 16:22:11.684101 7f869af78700 5 -- op tracker -- , seq: 2615, time: 2014-06-09 16:22:11.303016, event: header_read, op: pg_info(1 pgs e243:3.60e) 2014-06-09 16:22:11.684105 7f869af78700 5 -- op tracker -- , seq: 2615, time: 2014-06-09 16:22:11.303016, event: throttled, op: pg_info(1 pgs e243:3.60e) 2014-06-09 16:22:11.684108 7f869af78700 5 -- op tracker -- , seq: 2615, time: 2014-06-09 16:22:11.303022, event: all_read, op: pg_info(1 pgs e243:3.60e) 2014-06-09 16:22:11.684113 7f869af78700 5 -- op tracker -- , seq: 2615, time: 2014-06-09 16:22:11.684097, event: dispatched, op: pg_info(1 pgs e243:3.60e) 2014-06-09 16:22:11.684117 7f869af78700 5 -- op tracker -- , seq: 2615, time: 2014-06-09 16:22:11.684116, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.60e) 2014-06-09 16:22:11.684120 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.60e) v4 from osd.2 2014-06-09 16:22:11.684121 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30b8d6c0 2014-06-09 16:22:11.684124 7f869af78700 5 -- op tracker -- , seq: 2615, time: 2014-06-09 16:22:11.684123, event: started, op: pg_info(1 pgs e243:3.60e) 2014-06-09 16:22:11.684133 7f869af78700 5 -- op tracker -- , seq: 2615, time: 2014-06-09 16:22:11.684132, event: done, op: pg_info(1 pgs e243:3.60e) 2014-06-09 16:22:11.684139 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684146 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684147 7f869af78700 20 osd.26 243 _dispatch 0x310d7500 pg_info(1 pgs e243:1.238) v4 2014-06-09 16:22:11.684150 7f869af78700 5 -- op tracker -- , seq: 2616, time: 2014-06-09 16:22:11.422750, event: header_read, op: pg_info(1 pgs e243:1.238) 2014-06-09 16:22:11.684154 7f869af78700 5 -- op tracker -- , seq: 2616, time: 2014-06-09 16:22:11.422751, event: throttled, op: pg_info(1 pgs e243:1.238) 2014-06-09 16:22:11.684157 7f869af78700 5 -- op tracker -- , seq: 2616, time: 2014-06-09 16:22:11.422768, event: all_read, op: pg_info(1 pgs e243:1.238) 2014-06-09 16:22:11.684162 7f869af78700 5 -- op tracker -- , seq: 2616, time: 2014-06-09 16:22:11.684146, event: dispatched, op: pg_info(1 pgs e243:1.238) 2014-06-09 16:22:11.684165 7f869af78700 5 -- op tracker -- , seq: 2616, time: 2014-06-09 16:22:11.684164, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.238) 2014-06-09 16:22:11.684171 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.238) v4 from osd.4 2014-06-09 16:22:11.684172 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d7500 2014-06-09 16:22:11.684175 7f869af78700 5 -- op tracker -- , seq: 2616, time: 2014-06-09 16:22:11.684174, event: started, op: pg_info(1 pgs e243:1.238) 2014-06-09 16:22:11.684184 7f869af78700 5 -- op tracker -- , seq: 2616, time: 2014-06-09 16:22:11.684183, event: done, op: pg_info(1 pgs e243:1.238) 2014-06-09 16:22:11.684188 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684189 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684196 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684196 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684197 7f869af78700 20 osd.26 243 _dispatch 0x310e96c0 pg_info(1 pgs e243:0.2c3) v4 2014-06-09 16:22:11.684200 7f869af78700 5 -- op tracker -- , seq: 2617, time: 2014-06-09 16:22:11.479655, event: header_read, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.684206 7f869af78700 5 -- op tracker -- , seq: 2617, time: 2014-06-09 16:22:11.479655, event: throttled, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.684212 7f869af78700 5 -- op tracker -- , seq: 2617, time: 2014-06-09 16:22:11.479666, event: all_read, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.684218 7f869af78700 5 -- op tracker -- , seq: 2617, time: 2014-06-09 16:22:11.684195, event: dispatched, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.684224 7f869af78700 5 -- op tracker -- , seq: 2617, time: 2014-06-09 16:22:11.684223, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.684228 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2c3) v4 from osd.7 2014-06-09 16:22:11.684229 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e96c0 2014-06-09 16:22:11.684232 7f869af78700 5 -- op tracker -- , seq: 2617, time: 2014-06-09 16:22:11.684231, event: started, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.684241 7f869af78700 5 -- op tracker -- , seq: 2617, time: 2014-06-09 16:22:11.684240, event: done, op: pg_info(1 pgs e243:0.2c3) 2014-06-09 16:22:11.684245 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684246 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684253 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684254 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684254 7f869af78700 20 osd.26 243 _dispatch 0x2f030fc0 pg_info(1 pgs e243:0.74a) v4 2014-06-09 16:22:11.684257 7f869af78700 5 -- op tracker -- , seq: 2618, time: 2014-06-09 16:22:11.447738, event: header_read, op: pg_info(1 pgs e243:0.74a) 2014-06-09 16:22:11.684261 7f869af78700 5 -- op tracker -- , seq: 2618, time: 2014-06-09 16:22:11.447739, event: throttled, op: pg_info(1 pgs e243:0.74a) 2014-06-09 16:22:11.684264 7f869af78700 5 -- op tracker -- , seq: 2618, time: 2014-06-09 16:22:11.447749, event: all_read, op: pg_info(1 pgs e243:0.74a) 2014-06-09 16:22:11.684267 7f869af78700 5 -- op tracker -- , seq: 2618, time: 2014-06-09 16:22:11.684253, event: dispatched, op: pg_info(1 pgs e243:0.74a) 2014-06-09 16:22:11.684271 7f869af78700 5 -- op tracker -- , seq: 2618, time: 2014-06-09 16:22:11.684270, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.74a) 2014-06-09 16:22:11.684273 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.74a) v4 from osd.3 2014-06-09 16:22:11.684275 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f030fc0 2014-06-09 16:22:11.684277 7f869af78700 5 -- op tracker -- , seq: 2618, time: 2014-06-09 16:22:11.684276, event: started, op: pg_info(1 pgs e243:0.74a) 2014-06-09 16:22:11.684285 7f869af78700 5 -- op tracker -- , seq: 2618, time: 2014-06-09 16:22:11.684284, event: done, op: pg_info(1 pgs e243:0.74a) 2014-06-09 16:22:11.684289 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684290 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684296 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684296 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684297 7f869af78700 20 osd.26 243 _dispatch 0x240ebdc0 pg_info(1 pgs e243:2.3f0) v4 2014-06-09 16:22:11.684300 7f869af78700 5 -- op tracker -- , seq: 2619, time: 2014-06-09 16:22:11.403128, event: header_read, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.684303 7f869af78700 5 -- op tracker -- , seq: 2619, time: 2014-06-09 16:22:11.403129, event: throttled, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.684308 7f869af78700 5 -- op tracker -- , seq: 2619, time: 2014-06-09 16:22:11.403141, event: all_read, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.684312 7f869af78700 5 -- op tracker -- , seq: 2619, time: 2014-06-09 16:22:11.684296, event: dispatched, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.684316 7f869af78700 5 -- op tracker -- , seq: 2619, time: 2014-06-09 16:22:11.684315, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.684318 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3f0) v4 from osd.1 2014-06-09 16:22:11.684320 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ebdc0 2014-06-09 16:22:11.684322 7f869af78700 5 -- op tracker -- , seq: 2619, time: 2014-06-09 16:22:11.684321, event: started, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.684330 7f869af78700 5 -- op tracker -- , seq: 2619, time: 2014-06-09 16:22:11.684329, event: done, op: pg_info(1 pgs e243:2.3f0) 2014-06-09 16:22:11.684334 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684335 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684341 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684342 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684342 7f869af78700 20 osd.26 243 _dispatch 0x310da380 pg_info(1 pgs e243:1.6e8) v4 2014-06-09 16:22:11.684345 7f869af78700 5 -- op tracker -- , seq: 2620, time: 2014-06-09 16:22:11.567697, event: header_read, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.684349 7f869af78700 5 -- op tracker -- , seq: 2620, time: 2014-06-09 16:22:11.567698, event: throttled, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.684356 7f869af78700 5 -- op tracker -- , seq: 2620, time: 2014-06-09 16:22:11.567708, event: all_read, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.684359 7f869af78700 5 -- op tracker -- , seq: 2620, time: 2014-06-09 16:22:11.684341, event: dispatched, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.684364 7f869af78700 5 -- op tracker -- , seq: 2620, time: 2014-06-09 16:22:11.684363, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.684367 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6e8) v4 from osd.9 2014-06-09 16:22:11.684368 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310da380 2014-06-09 16:22:11.684371 7f869af78700 5 -- op tracker -- , seq: 2620, time: 2014-06-09 16:22:11.684370, event: started, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.684378 7f869af78700 5 -- op tracker -- , seq: 2620, time: 2014-06-09 16:22:11.684377, event: done, op: pg_info(1 pgs e243:1.6e8) 2014-06-09 16:22:11.684383 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684383 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684390 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684391 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684392 7f869af78700 20 osd.26 243 _dispatch 0x31166fc0 pg_notify(2.290(23),3.56a(10) epoch 243) v5 2014-06-09 16:22:11.684395 7f869af78700 5 -- op tracker -- , seq: 2621, time: 2014-06-09 16:22:11.431805, event: header_read, op: pg_notify(2.290(23),3.56a(10) epoch 243) 2014-06-09 16:22:11.684399 7f869af78700 5 -- op tracker -- , seq: 2621, time: 2014-06-09 16:22:11.431805, event: throttled, op: pg_notify(2.290(23),3.56a(10) epoch 243) 2014-06-09 16:22:11.684402 7f869af78700 5 -- op tracker -- , seq: 2621, time: 2014-06-09 16:22:11.431858, event: all_read, op: pg_notify(2.290(23),3.56a(10) epoch 243) 2014-06-09 16:22:11.684406 7f869af78700 5 -- op tracker -- , seq: 2621, time: 2014-06-09 16:22:11.684390, event: dispatched, op: pg_notify(2.290(23),3.56a(10) epoch 243) 2014-06-09 16:22:11.684409 7f869af78700 5 -- op tracker -- , seq: 2621, time: 2014-06-09 16:22:11.684408, event: waiting_for_osdmap, op: pg_notify(2.290(23),3.56a(10) epoch 243) 2014-06-09 16:22:11.684411 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.684412 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31166fc0 2014-06-09 16:22:11.684415 7f869af78700 5 -- op tracker -- , seq: 2621, time: 2014-06-09 16:22:11.684414, event: started, op: pg_notify(2.290(23),3.56a(10) epoch 243) 2014-06-09 16:22:11.684428 7f869af78700 5 -- op tracker -- , seq: 2621, time: 2014-06-09 16:22:11.684427, event: done, op: pg_notify(2.290(23),3.56a(10) epoch 243) 2014-06-09 16:22:11.684432 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684433 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684446 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684447 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684447 7f869af78700 20 osd.26 243 _dispatch 0x31163c00 pg_info(1 pgs e243:2.6c7) v4 2014-06-09 16:22:11.684450 7f869af78700 5 -- op tracker -- , seq: 2622, time: 2014-06-09 16:22:11.423735, event: header_read, op: pg_info(1 pgs e243:2.6c7) 2014-06-09 16:22:11.684456 7f869af78700 5 -- op tracker -- , seq: 2622, time: 2014-06-09 16:22:11.423736, event: throttled, op: pg_info(1 pgs e243:2.6c7) 2014-06-09 16:22:11.684459 7f869af78700 5 -- op tracker -- , seq: 2622, time: 2014-06-09 16:22:11.423746, event: all_read, op: pg_info(1 pgs e243:2.6c7) 2014-06-09 16:22:11.684464 7f869af78700 5 -- op tracker -- , seq: 2622, time: 2014-06-09 16:22:11.684446, event: dispatched, op: pg_info(1 pgs e243:2.6c7) 2014-06-09 16:22:11.684467 7f869af78700 5 -- op tracker -- , seq: 2622, time: 2014-06-09 16:22:11.684466, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6c7) 2014-06-09 16:22:11.684473 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6c7) v4 from osd.6 2014-06-09 16:22:11.684474 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31163c00 2014-06-09 16:22:11.684477 7f869af78700 5 -- op tracker -- , seq: 2622, time: 2014-06-09 16:22:11.684476, event: started, op: pg_info(1 pgs e243:2.6c7) 2014-06-09 16:22:11.684485 7f869af78700 5 -- op tracker -- , seq: 2622, time: 2014-06-09 16:22:11.684484, event: done, op: pg_info(1 pgs e243:2.6c7) 2014-06-09 16:22:11.684490 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684490 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684498 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684498 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684499 7f869af78700 20 osd.26 243 _dispatch 0x30ebb880 pg_info(1 pgs e243:0.380) v4 2014-06-09 16:22:11.684502 7f869af78700 5 -- op tracker -- , seq: 2623, time: 2014-06-09 16:22:11.565796, event: header_read, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.684506 7f869af78700 5 -- op tracker -- , seq: 2623, time: 2014-06-09 16:22:11.565796, event: throttled, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.684512 7f869af78700 5 -- op tracker -- , seq: 2623, time: 2014-06-09 16:22:11.565810, event: all_read, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.684515 7f869af78700 5 -- op tracker -- , seq: 2623, time: 2014-06-09 16:22:11.684498, event: dispatched, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.684519 7f869af78700 5 -- op tracker -- , seq: 2623, time: 2014-06-09 16:22:11.684518, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.684521 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.380) v4 from osd.0 2014-06-09 16:22:11.684523 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebb880 2014-06-09 16:22:11.684525 7f869af78700 5 -- op tracker -- , seq: 2623, time: 2014-06-09 16:22:11.684524, event: started, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.684536 7f869af78700 5 -- op tracker -- , seq: 2623, time: 2014-06-09 16:22:11.684535, event: done, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.684540 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684541 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684547 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684548 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684549 7f869af78700 20 osd.26 243 _dispatch 0x310e81c0 pg_info(1 pgs e243:3.6d5) v4 2014-06-09 16:22:11.684551 7f869af78700 5 -- op tracker -- , seq: 2624, time: 2014-06-09 16:22:11.446375, event: header_read, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.684555 7f869af78700 5 -- op tracker -- , seq: 2624, time: 2014-06-09 16:22:11.446376, event: throttled, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.684559 7f869af78700 5 -- op tracker -- , seq: 2624, time: 2014-06-09 16:22:11.446388, event: all_read, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.684562 7f869af78700 5 -- op tracker -- , seq: 2624, time: 2014-06-09 16:22:11.684547, event: dispatched, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.684565 7f869af78700 5 -- op tracker -- , seq: 2624, time: 2014-06-09 16:22:11.684564, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.684568 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6d5) v4 from osd.5 2014-06-09 16:22:11.684569 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e81c0 2014-06-09 16:22:11.684572 7f869af78700 5 -- op tracker -- , seq: 2624, time: 2014-06-09 16:22:11.684571, event: started, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.684582 7f869af78700 5 -- op tracker -- , seq: 2624, time: 2014-06-09 16:22:11.684581, event: done, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.684586 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684587 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684593 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684594 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684595 7f869af78700 20 osd.26 243 _dispatch 0x30cf9a40 pg_info(1 pgs e243:0.69f) v4 2014-06-09 16:22:11.684597 7f869af78700 5 -- op tracker -- , seq: 2625, time: 2014-06-09 16:22:11.303024, event: header_read, op: pg_info(1 pgs e243:0.69f) 2014-06-09 16:22:11.684604 7f869af78700 5 -- op tracker -- , seq: 2625, time: 2014-06-09 16:22:11.303024, event: throttled, op: pg_info(1 pgs e243:0.69f) 2014-06-09 16:22:11.684608 7f869af78700 5 -- op tracker -- , seq: 2625, time: 2014-06-09 16:22:11.303043, event: all_read, op: pg_info(1 pgs e243:0.69f) 2014-06-09 16:22:11.684611 7f869af78700 5 -- op tracker -- , seq: 2625, time: 2014-06-09 16:22:11.684593, event: dispatched, op: pg_info(1 pgs e243:0.69f) 2014-06-09 16:22:11.684615 7f869af78700 5 -- op tracker -- , seq: 2625, time: 2014-06-09 16:22:11.684614, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.69f) 2014-06-09 16:22:11.684617 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.69f) v4 from osd.2 2014-06-09 16:22:11.684618 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf9a40 2014-06-09 16:22:11.684621 7f869af78700 5 -- op tracker -- , seq: 2625, time: 2014-06-09 16:22:11.684620, event: started, op: pg_info(1 pgs e243:0.69f) 2014-06-09 16:22:11.684629 7f869af78700 5 -- op tracker -- , seq: 2625, time: 2014-06-09 16:22:11.684628, event: done, op: pg_info(1 pgs e243:0.69f) 2014-06-09 16:22:11.684634 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684635 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684641 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684642 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684642 7f869af78700 20 osd.26 243 _dispatch 0x310d7340 pg_info(1 pgs e243:2.3d7) v4 2014-06-09 16:22:11.684645 7f869af78700 5 -- op tracker -- , seq: 2626, time: 2014-06-09 16:22:11.422780, event: header_read, op: pg_info(1 pgs e243:2.3d7) 2014-06-09 16:22:11.684650 7f869af78700 5 -- op tracker -- , seq: 2626, time: 2014-06-09 16:22:11.422780, event: throttled, op: pg_info(1 pgs e243:2.3d7) 2014-06-09 16:22:11.684653 7f869af78700 5 -- op tracker -- , seq: 2626, time: 2014-06-09 16:22:11.422790, event: all_read, op: pg_info(1 pgs e243:2.3d7) 2014-06-09 16:22:11.684657 7f869af78700 5 -- op tracker -- , seq: 2626, time: 2014-06-09 16:22:11.684641, event: dispatched, op: pg_info(1 pgs e243:2.3d7) 2014-06-09 16:22:11.684660 7f869af78700 5 -- op tracker -- , seq: 2626, time: 2014-06-09 16:22:11.684659, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3d7) 2014-06-09 16:22:11.684662 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3d7) v4 from osd.4 2014-06-09 16:22:11.684664 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d7340 2014-06-09 16:22:11.684667 7f869af78700 5 -- op tracker -- , seq: 2626, time: 2014-06-09 16:22:11.684665, event: started, op: pg_info(1 pgs e243:2.3d7) 2014-06-09 16:22:11.684675 7f869af78700 5 -- op tracker -- , seq: 2626, time: 2014-06-09 16:22:11.684674, event: done, op: pg_info(1 pgs e243:2.3d7) 2014-06-09 16:22:11.684679 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684680 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684686 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684687 7f869af78700 20 osd.26 243 _dispatch 0x30ebb340 pg_info(1 pgs e243:0.16c) v4 2014-06-09 16:22:11.684690 7f869af78700 5 -- op tracker -- , seq: 2627, time: 2014-06-09 16:22:11.563775, event: header_read, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.684694 7f869af78700 5 -- op tracker -- , seq: 2627, time: 2014-06-09 16:22:11.563776, event: throttled, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.684697 7f869af78700 5 -- op tracker -- , seq: 2627, time: 2014-06-09 16:22:11.563788, event: all_read, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.684700 7f869af78700 5 -- op tracker -- , seq: 2627, time: 2014-06-09 16:22:11.684686, event: dispatched, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.684703 7f869af78700 5 -- op tracker -- , seq: 2627, time: 2014-06-09 16:22:11.684702, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.684706 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.16c) v4 from osd.7 2014-06-09 16:22:11.684707 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebb340 2014-06-09 16:22:11.684709 7f869af78700 5 -- op tracker -- , seq: 2627, time: 2014-06-09 16:22:11.684709, event: started, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.684718 7f869af78700 5 -- op tracker -- , seq: 2627, time: 2014-06-09 16:22:11.684717, event: done, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.684721 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684722 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684728 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.684729 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.684730 7f869af78700 20 osd.26 243 _dispatch 0x2f030e00 pg_info(1 pgs e243:3.4d1) v4 2014-06-09 16:22:11.684732 7f869af78700 5 -- op tracker -- , seq: 2628, time: 2014-06-09 16:22:11.448444, event: header_read, op: pg_info(1 pgs e243:3.4d1) 2014-06-09 16:22:11.684736 7f869af78700 5 -- op tracker -- , seq: 2628, time: 2014-06-09 16:22:11.448444, event: throttled, op: pg_info(1 pgs e243:3.4d1) 2014-06-09 16:22:11.686290 7f869af78700 5 -- op tracker -- , seq: 2628, time: 2014-06-09 16:22:11.448458, event: all_read, op: pg_info(1 pgs e243:3.4d1) 2014-06-09 16:22:11.686294 7f869af78700 5 -- op tracker -- , seq: 2628, time: 2014-06-09 16:22:11.684728, event: dispatched, op: pg_info(1 pgs e243:3.4d1) 2014-06-09 16:22:11.686297 7f869af78700 5 -- op tracker -- , seq: 2628, time: 2014-06-09 16:22:11.686296, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4d1) 2014-06-09 16:22:11.686304 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4d1) v4 from osd.3 2014-06-09 16:22:11.686306 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f030e00 2014-06-09 16:22:11.686309 7f869af78700 5 -- op tracker -- , seq: 2628, time: 2014-06-09 16:22:11.686308, event: started, op: pg_info(1 pgs e243:3.4d1) 2014-06-09 16:22:11.686318 7f869af78700 5 -- op tracker -- , seq: 2628, time: 2014-06-09 16:22:11.686317, event: done, op: pg_info(1 pgs e243:3.4d1) 2014-06-09 16:22:11.686322 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686323 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686330 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686331 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686331 7f869af78700 20 osd.26 243 _dispatch 0x31166540 pg_info(1 pgs e243:2.6ec) v4 2014-06-09 16:22:11.686334 7f869af78700 5 -- op tracker -- , seq: 2629, time: 2014-06-09 16:22:11.429914, event: header_read, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.686338 7f869af78700 5 -- op tracker -- , seq: 2629, time: 2014-06-09 16:22:11.429915, event: throttled, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.686346 7f869af78700 5 -- op tracker -- , seq: 2629, time: 2014-06-09 16:22:11.429928, event: all_read, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.686349 7f869af78700 5 -- op tracker -- , seq: 2629, time: 2014-06-09 16:22:11.686330, event: dispatched, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.686353 7f869af78700 5 -- op tracker -- , seq: 2629, time: 2014-06-09 16:22:11.686352, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.686357 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6ec) v4 from osd.1 2014-06-09 16:22:11.686358 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31166540 2014-06-09 16:22:11.686361 7f869af78700 5 -- op tracker -- , seq: 2629, time: 2014-06-09 16:22:11.686360, event: started, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.686370 7f869af78700 5 -- op tracker -- , seq: 2629, time: 2014-06-09 16:22:11.686369, event: done, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.686375 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686376 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686383 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686383 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686384 7f869af78700 20 osd.26 243 _dispatch 0x2176c8c0 pg_info(1 pgs e243:0.5b7) v4 2014-06-09 16:22:11.686387 7f869af78700 5 -- op tracker -- , seq: 2630, time: 2014-06-09 16:22:11.574342, event: header_read, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686391 7f869af78700 5 -- op tracker -- , seq: 2630, time: 2014-06-09 16:22:11.574343, event: throttled, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686394 7f869af78700 5 -- op tracker -- , seq: 2630, time: 2014-06-09 16:22:11.574355, event: all_read, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686399 7f869af78700 5 -- op tracker -- , seq: 2630, time: 2014-06-09 16:22:11.686383, event: dispatched, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686402 7f869af78700 5 -- op tracker -- , seq: 2630, time: 2014-06-09 16:22:11.686401, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686409 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5b7) v4 from osd.9 2014-06-09 16:22:11.686411 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176c8c0 2014-06-09 16:22:11.686413 7f869af78700 5 -- op tracker -- , seq: 2630, time: 2014-06-09 16:22:11.686412, event: started, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686422 7f869af78700 5 -- op tracker -- , seq: 2630, time: 2014-06-09 16:22:11.686420, event: done, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686426 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686434 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686434 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686435 7f869af78700 20 osd.26 243 _dispatch 0x31166380 pg_info(1 pgs e243:3.292(2)) v4 2014-06-09 16:22:11.686438 7f869af78700 5 -- op tracker -- , seq: 2631, time: 2014-06-09 16:22:11.431868, event: header_read, op: pg_info(1 pgs e243:3.292(2)) 2014-06-09 16:22:11.686443 7f869af78700 5 -- op tracker -- , seq: 2631, time: 2014-06-09 16:22:11.431869, event: throttled, op: pg_info(1 pgs e243:3.292(2)) 2014-06-09 16:22:11.686447 7f869af78700 5 -- op tracker -- , seq: 2631, time: 2014-06-09 16:22:11.431882, event: all_read, op: pg_info(1 pgs e243:3.292(2)) 2014-06-09 16:22:11.686450 7f869af78700 5 -- op tracker -- , seq: 2631, time: 2014-06-09 16:22:11.686433, event: dispatched, op: pg_info(1 pgs e243:3.292(2)) 2014-06-09 16:22:11.686453 7f869af78700 5 -- op tracker -- , seq: 2631, time: 2014-06-09 16:22:11.686452, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.292(2)) 2014-06-09 16:22:11.686456 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.292(2)) v4 from osd.8 2014-06-09 16:22:11.686457 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31166380 2014-06-09 16:22:11.686460 7f869af78700 5 -- op tracker -- , seq: 2631, time: 2014-06-09 16:22:11.686459, event: started, op: pg_info(1 pgs e243:3.292(2)) 2014-06-09 16:22:11.686468 7f869af78700 5 -- op tracker -- , seq: 2631, time: 2014-06-09 16:22:11.686467, event: done, op: pg_info(1 pgs e243:3.292(2)) 2014-06-09 16:22:11.686472 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686473 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686481 7f869af78700 20 osd.26 243 _dispatch 0x31163a40 pg_info(1 pgs e243:0.1dc) v4 2014-06-09 16:22:11.686484 7f869af78700 5 -- op tracker -- , seq: 2632, time: 2014-06-09 16:22:11.423750, event: header_read, op: pg_info(1 pgs e243:0.1dc) 2014-06-09 16:22:11.686487 7f869af78700 5 -- op tracker -- , seq: 2632, time: 2014-06-09 16:22:11.423750, event: throttled, op: pg_info(1 pgs e243:0.1dc) 2014-06-09 16:22:11.686491 7f869af78700 5 -- op tracker -- , seq: 2632, time: 2014-06-09 16:22:11.423759, event: all_read, op: pg_info(1 pgs e243:0.1dc) 2014-06-09 16:22:11.686495 7f869af78700 5 -- op tracker -- , seq: 2632, time: 2014-06-09 16:22:11.686480, event: dispatched, op: pg_info(1 pgs e243:0.1dc) 2014-06-09 16:22:11.686498 7f869af78700 5 -- op tracker -- , seq: 2632, time: 2014-06-09 16:22:11.686497, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1dc) 2014-06-09 16:22:11.686500 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1dc) v4 from osd.6 2014-06-09 16:22:11.686502 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31163a40 2014-06-09 16:22:11.686504 7f869af78700 5 -- op tracker -- , seq: 2632, time: 2014-06-09 16:22:11.686503, event: started, op: pg_info(1 pgs e243:0.1dc) 2014-06-09 16:22:11.686512 7f869af78700 5 -- op tracker -- , seq: 2632, time: 2014-06-09 16:22:11.686511, event: done, op: pg_info(1 pgs e243:0.1dc) 2014-06-09 16:22:11.686516 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686517 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686523 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686524 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686525 7f869af78700 20 osd.26 243 _dispatch 0x30ebbdc0 pg_info(1 pgs e243:2.57a) v4 2014-06-09 16:22:11.686527 7f869af78700 5 -- op tracker -- , seq: 2633, time: 2014-06-09 16:22:11.565821, event: header_read, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.686532 7f869af78700 5 -- op tracker -- , seq: 2633, time: 2014-06-09 16:22:11.565821, event: throttled, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.686537 7f869af78700 5 -- op tracker -- , seq: 2633, time: 2014-06-09 16:22:11.565832, event: all_read, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.686540 7f869af78700 5 -- op tracker -- , seq: 2633, time: 2014-06-09 16:22:11.686523, event: dispatched, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.686548 7f869af78700 5 -- op tracker -- , seq: 2633, time: 2014-06-09 16:22:11.686547, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.686550 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.57a) v4 from osd.0 2014-06-09 16:22:11.686552 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebbdc0 2014-06-09 16:22:11.686555 7f869af78700 5 -- op tracker -- , seq: 2633, time: 2014-06-09 16:22:11.686554, event: started, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.686563 7f869af78700 5 -- op tracker -- , seq: 2633, time: 2014-06-09 16:22:11.686562, event: done, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.686567 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686568 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686575 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686575 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686576 7f869af78700 20 osd.26 243 _dispatch 0x30daf880 pg_info(1 pgs e243:3.6d5) v4 2014-06-09 16:22:11.686579 7f869af78700 5 -- op tracker -- , seq: 2634, time: 2014-06-09 16:22:11.446393, event: header_read, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.686582 7f869af78700 5 -- op tracker -- , seq: 2634, time: 2014-06-09 16:22:11.446394, event: throttled, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.686586 7f869af78700 5 -- op tracker -- , seq: 2634, time: 2014-06-09 16:22:11.446404, event: all_read, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.686593 7f869af78700 5 -- op tracker -- , seq: 2634, time: 2014-06-09 16:22:11.686574, event: dispatched, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.686599 7f869af78700 5 -- op tracker -- , seq: 2634, time: 2014-06-09 16:22:11.686597, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.686601 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6d5) v4 from osd.5 2014-06-09 16:22:11.686602 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daf880 2014-06-09 16:22:11.686605 7f869af78700 5 -- op tracker -- , seq: 2634, time: 2014-06-09 16:22:11.686604, event: started, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.686613 7f869af78700 5 -- op tracker -- , seq: 2634, time: 2014-06-09 16:22:11.686612, event: done, op: pg_info(1 pgs e243:3.6d5) 2014-06-09 16:22:11.686618 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686619 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686626 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686626 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686627 7f869af78700 20 osd.26 243 _dispatch 0x30cf9880 pg_info(1 pgs e243:2.12d) v4 2014-06-09 16:22:11.686629 7f869af78700 5 -- op tracker -- , seq: 2635, time: 2014-06-09 16:22:11.303055, event: header_read, op: pg_info(1 pgs e243:2.12d) 2014-06-09 16:22:11.686638 7f869af78700 5 -- op tracker -- , seq: 2635, time: 2014-06-09 16:22:11.303055, event: throttled, op: pg_info(1 pgs e243:2.12d) 2014-06-09 16:22:11.686641 7f869af78700 5 -- op tracker -- , seq: 2635, time: 2014-06-09 16:22:11.303069, event: all_read, op: pg_info(1 pgs e243:2.12d) 2014-06-09 16:22:11.686646 7f869af78700 5 -- op tracker -- , seq: 2635, time: 2014-06-09 16:22:11.686625, event: dispatched, op: pg_info(1 pgs e243:2.12d) 2014-06-09 16:22:11.686649 7f869af78700 5 -- op tracker -- , seq: 2635, time: 2014-06-09 16:22:11.686648, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.12d) 2014-06-09 16:22:11.686654 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.12d) v4 from osd.2 2014-06-09 16:22:11.686656 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf9880 2014-06-09 16:22:11.686659 7f869af78700 5 -- op tracker -- , seq: 2635, time: 2014-06-09 16:22:11.686658, event: started, op: pg_info(1 pgs e243:2.12d) 2014-06-09 16:22:11.686667 7f869af78700 5 -- op tracker -- , seq: 2635, time: 2014-06-09 16:22:11.686666, event: done, op: pg_info(1 pgs e243:2.12d) 2014-06-09 16:22:11.686671 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686672 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686680 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686680 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686681 7f869af78700 20 osd.26 243 _dispatch 0x310d7180 pg_info(1 pgs e243:3.67f) v4 2014-06-09 16:22:11.686684 7f869af78700 5 -- op tracker -- , seq: 2636, time: 2014-06-09 16:22:11.422795, event: header_read, op: pg_info(1 pgs e243:3.67f) 2014-06-09 16:22:11.686687 7f869af78700 5 -- op tracker -- , seq: 2636, time: 2014-06-09 16:22:11.422795, event: throttled, op: pg_info(1 pgs e243:3.67f) 2014-06-09 16:22:11.686691 7f869af78700 5 -- op tracker -- , seq: 2636, time: 2014-06-09 16:22:11.422805, event: all_read, op: pg_info(1 pgs e243:3.67f) 2014-06-09 16:22:11.686695 7f869af78700 5 -- op tracker -- , seq: 2636, time: 2014-06-09 16:22:11.686679, event: dispatched, op: pg_info(1 pgs e243:3.67f) 2014-06-09 16:22:11.686699 7f869af78700 5 -- op tracker -- , seq: 2636, time: 2014-06-09 16:22:11.686698, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.67f) 2014-06-09 16:22:11.686704 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.67f) v4 from osd.4 2014-06-09 16:22:11.686706 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d7180 2014-06-09 16:22:11.686709 7f869af78700 5 -- op tracker -- , seq: 2636, time: 2014-06-09 16:22:11.686707, event: started, op: pg_info(1 pgs e243:3.67f) 2014-06-09 16:22:11.686718 7f869af78700 5 -- op tracker -- , seq: 2636, time: 2014-06-09 16:22:11.686717, event: done, op: pg_info(1 pgs e243:3.67f) 2014-06-09 16:22:11.686725 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686726 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686732 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686733 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686733 7f869af78700 20 osd.26 243 _dispatch 0x310e9500 pg_info(1 pgs e243:0.16c) v4 2014-06-09 16:22:11.686736 7f869af78700 5 -- op tracker -- , seq: 2637, time: 2014-06-09 16:22:11.563793, event: header_read, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.686740 7f869af78700 5 -- op tracker -- , seq: 2637, time: 2014-06-09 16:22:11.563793, event: throttled, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.686743 7f869af78700 5 -- op tracker -- , seq: 2637, time: 2014-06-09 16:22:11.563805, event: all_read, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.686746 7f869af78700 5 -- op tracker -- , seq: 2637, time: 2014-06-09 16:22:11.686732, event: dispatched, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.686750 7f869af78700 5 -- op tracker -- , seq: 2637, time: 2014-06-09 16:22:11.686749, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.686752 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.16c) v4 from osd.7 2014-06-09 16:22:11.686753 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e9500 2014-06-09 16:22:11.686756 7f869af78700 5 -- op tracker -- , seq: 2637, time: 2014-06-09 16:22:11.686755, event: started, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.686764 7f869af78700 5 -- op tracker -- , seq: 2637, time: 2014-06-09 16:22:11.686763, event: done, op: pg_info(1 pgs e243:0.16c) 2014-06-09 16:22:11.686768 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686769 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686775 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686775 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686776 7f869af78700 20 osd.26 243 _dispatch 0x2f030c40 pg_info(1 pgs e243:2.75d) v4 2014-06-09 16:22:11.686779 7f869af78700 5 -- op tracker -- , seq: 2638, time: 2014-06-09 16:22:11.448550, event: header_read, op: pg_info(1 pgs e243:2.75d) 2014-06-09 16:22:11.686782 7f869af78700 5 -- op tracker -- , seq: 2638, time: 2014-06-09 16:22:11.448551, event: throttled, op: pg_info(1 pgs e243:2.75d) 2014-06-09 16:22:11.686786 7f869af78700 5 -- op tracker -- , seq: 2638, time: 2014-06-09 16:22:11.448564, event: all_read, op: pg_info(1 pgs e243:2.75d) 2014-06-09 16:22:11.686789 7f869af78700 5 -- op tracker -- , seq: 2638, time: 2014-06-09 16:22:11.686775, event: dispatched, op: pg_info(1 pgs e243:2.75d) 2014-06-09 16:22:11.686792 7f869af78700 5 -- op tracker -- , seq: 2638, time: 2014-06-09 16:22:11.686791, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.75d) 2014-06-09 16:22:11.686794 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.75d) v4 from osd.3 2014-06-09 16:22:11.686796 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f030c40 2014-06-09 16:22:11.686798 7f869af78700 5 -- op tracker -- , seq: 2638, time: 2014-06-09 16:22:11.686797, event: started, op: pg_info(1 pgs e243:2.75d) 2014-06-09 16:22:11.686808 7f869af78700 5 -- op tracker -- , seq: 2638, time: 2014-06-09 16:22:11.686807, event: done, op: pg_info(1 pgs e243:2.75d) 2014-06-09 16:22:11.686812 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686812 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686819 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686819 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686820 7f869af78700 20 osd.26 243 _dispatch 0x31166c40 pg_info(1 pgs e243:2.530) v4 2014-06-09 16:22:11.686822 7f869af78700 5 -- op tracker -- , seq: 2639, time: 2014-06-09 16:22:11.429937, event: header_read, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.686827 7f869af78700 5 -- op tracker -- , seq: 2639, time: 2014-06-09 16:22:11.429937, event: throttled, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.686831 7f869af78700 5 -- op tracker -- , seq: 2639, time: 2014-06-09 16:22:11.429947, event: all_read, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.686834 7f869af78700 5 -- op tracker -- , seq: 2639, time: 2014-06-09 16:22:11.686818, event: dispatched, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.686841 7f869af78700 5 -- op tracker -- , seq: 2639, time: 2014-06-09 16:22:11.686840, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.686844 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.530) v4 from osd.1 2014-06-09 16:22:11.686846 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31166c40 2014-06-09 16:22:11.686848 7f869af78700 5 -- op tracker -- , seq: 2639, time: 2014-06-09 16:22:11.686847, event: started, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.686857 7f869af78700 5 -- op tracker -- , seq: 2639, time: 2014-06-09 16:22:11.686856, event: done, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.686862 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686862 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686869 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686870 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686870 7f869af78700 20 osd.26 243 _dispatch 0x2176ca80 pg_info(1 pgs e243:0.5b7) v4 2014-06-09 16:22:11.686873 7f869af78700 5 -- op tracker -- , seq: 2640, time: 2014-06-09 16:22:11.574360, event: header_read, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686879 7f869af78700 5 -- op tracker -- , seq: 2640, time: 2014-06-09 16:22:11.574361, event: throttled, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686883 7f869af78700 5 -- op tracker -- , seq: 2640, time: 2014-06-09 16:22:11.574370, event: all_read, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686886 7f869af78700 5 -- op tracker -- , seq: 2640, time: 2014-06-09 16:22:11.686869, event: dispatched, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686890 7f869af78700 5 -- op tracker -- , seq: 2640, time: 2014-06-09 16:22:11.686889, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686894 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5b7) v4 from osd.9 2014-06-09 16:22:11.686895 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176ca80 2014-06-09 16:22:11.686898 7f869af78700 5 -- op tracker -- , seq: 2640, time: 2014-06-09 16:22:11.686897, event: started, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686909 7f869af78700 5 -- op tracker -- , seq: 2640, time: 2014-06-09 16:22:11.686908, event: done, op: pg_info(1 pgs e243:0.5b7) 2014-06-09 16:22:11.686915 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686916 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686923 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686924 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686924 7f869af78700 20 osd.26 243 _dispatch 0x311668c0 pg_info(1 pgs e243:2.290) v4 2014-06-09 16:22:11.686927 7f869af78700 5 -- op tracker -- , seq: 2641, time: 2014-06-09 16:22:11.434365, event: header_read, op: pg_info(1 pgs e243:2.290) 2014-06-09 16:22:11.686931 7f869af78700 5 -- op tracker -- , seq: 2641, time: 2014-06-09 16:22:11.434366, event: throttled, op: pg_info(1 pgs e243:2.290) 2014-06-09 16:22:11.686934 7f869af78700 5 -- op tracker -- , seq: 2641, time: 2014-06-09 16:22:11.434378, event: all_read, op: pg_info(1 pgs e243:2.290) 2014-06-09 16:22:11.686940 7f869af78700 5 -- op tracker -- , seq: 2641, time: 2014-06-09 16:22:11.686923, event: dispatched, op: pg_info(1 pgs e243:2.290) 2014-06-09 16:22:11.686946 7f869af78700 5 -- op tracker -- , seq: 2641, time: 2014-06-09 16:22:11.686945, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.290) 2014-06-09 16:22:11.686949 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.290) v4 from osd.8 2014-06-09 16:22:11.686950 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311668c0 2014-06-09 16:22:11.686953 7f869af78700 5 -- op tracker -- , seq: 2641, time: 2014-06-09 16:22:11.686952, event: started, op: pg_info(1 pgs e243:2.290) 2014-06-09 16:22:11.686961 7f869af78700 5 -- op tracker -- , seq: 2641, time: 2014-06-09 16:22:11.686960, event: done, op: pg_info(1 pgs e243:2.290) 2014-06-09 16:22:11.686965 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686966 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686972 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.686972 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.686973 7f869af78700 20 osd.26 243 _dispatch 0x31163880 pg_info(1 pgs e243:3.1db) v4 2014-06-09 16:22:11.686976 7f869af78700 5 -- op tracker -- , seq: 2642, time: 2014-06-09 16:22:11.423763, event: header_read, op: pg_info(1 pgs e243:3.1db) 2014-06-09 16:22:11.686979 7f869af78700 5 -- op tracker -- , seq: 2642, time: 2014-06-09 16:22:11.423763, event: throttled, op: pg_info(1 pgs e243:3.1db) 2014-06-09 16:22:11.686983 7f869af78700 5 -- op tracker -- , seq: 2642, time: 2014-06-09 16:22:11.423773, event: all_read, op: pg_info(1 pgs e243:3.1db) 2014-06-09 16:22:11.686986 7f869af78700 5 -- op tracker -- , seq: 2642, time: 2014-06-09 16:22:11.686972, event: dispatched, op: pg_info(1 pgs e243:3.1db) 2014-06-09 16:22:11.686991 7f869af78700 5 -- op tracker -- , seq: 2642, time: 2014-06-09 16:22:11.686990, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1db) 2014-06-09 16:22:11.686993 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1db) v4 from osd.6 2014-06-09 16:22:11.686995 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31163880 2014-06-09 16:22:11.686997 7f869af78700 5 -- op tracker -- , seq: 2642, time: 2014-06-09 16:22:11.686996, event: started, op: pg_info(1 pgs e243:3.1db) 2014-06-09 16:22:11.687006 7f869af78700 5 -- op tracker -- , seq: 2642, time: 2014-06-09 16:22:11.687005, event: done, op: pg_info(1 pgs e243:3.1db) 2014-06-09 16:22:11.687010 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687010 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687017 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687017 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687018 7f869af78700 20 osd.26 243 _dispatch 0x30eba700 pg_info(1 pgs e243:1.382) v4 2014-06-09 16:22:11.687021 7f869af78700 5 -- op tracker -- , seq: 2643, time: 2014-06-09 16:22:11.565837, event: header_read, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687024 7f869af78700 5 -- op tracker -- , seq: 2643, time: 2014-06-09 16:22:11.565837, event: throttled, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687028 7f869af78700 5 -- op tracker -- , seq: 2643, time: 2014-06-09 16:22:11.565847, event: all_read, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687031 7f869af78700 5 -- op tracker -- , seq: 2643, time: 2014-06-09 16:22:11.687017, event: dispatched, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687037 7f869af78700 5 -- op tracker -- , seq: 2643, time: 2014-06-09 16:22:11.687036, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687040 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.382) v4 from osd.0 2014-06-09 16:22:11.687041 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30eba700 2014-06-09 16:22:11.687044 7f869af78700 5 -- op tracker -- , seq: 2643, time: 2014-06-09 16:22:11.687043, event: started, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687052 7f869af78700 5 -- op tracker -- , seq: 2643, time: 2014-06-09 16:22:11.687051, event: done, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687056 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687057 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687063 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687064 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687065 7f869af78700 20 osd.26 243 _dispatch 0x31166000 pg_info(1 pgs e243:0.54c) v4 2014-06-09 16:22:11.687067 7f869af78700 5 -- op tracker -- , seq: 2644, time: 2014-06-09 16:22:11.446412, event: header_read, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687072 7f869af78700 5 -- op tracker -- , seq: 2644, time: 2014-06-09 16:22:11.446412, event: throttled, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687076 7f869af78700 5 -- op tracker -- , seq: 2644, time: 2014-06-09 16:22:11.446422, event: all_read, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687079 7f869af78700 5 -- op tracker -- , seq: 2644, time: 2014-06-09 16:22:11.687063, event: dispatched, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687082 7f869af78700 5 -- op tracker -- , seq: 2644, time: 2014-06-09 16:22:11.687081, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687085 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.54c) v4 from osd.5 2014-06-09 16:22:11.687086 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31166000 2014-06-09 16:22:11.687089 7f869af78700 5 -- op tracker -- , seq: 2644, time: 2014-06-09 16:22:11.687088, event: started, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687097 7f869af78700 5 -- op tracker -- , seq: 2644, time: 2014-06-09 16:22:11.687096, event: done, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687101 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687108 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687109 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687109 7f869af78700 20 osd.26 243 _dispatch 0x30cf96c0 pg_info(1 pgs e243:2.5ad) v4 2014-06-09 16:22:11.687112 7f869af78700 5 -- op tracker -- , seq: 2645, time: 2014-06-09 16:22:11.303080, event: header_read, op: pg_info(1 pgs e243:2.5ad) 2014-06-09 16:22:11.687115 7f869af78700 5 -- op tracker -- , seq: 2645, time: 2014-06-09 16:22:11.303080, event: throttled, op: pg_info(1 pgs e243:2.5ad) 2014-06-09 16:22:11.687119 7f869af78700 5 -- op tracker -- , seq: 2645, time: 2014-06-09 16:22:11.303087, event: all_read, op: pg_info(1 pgs e243:2.5ad) 2014-06-09 16:22:11.687123 7f869af78700 5 -- op tracker -- , seq: 2645, time: 2014-06-09 16:22:11.687108, event: dispatched, op: pg_info(1 pgs e243:2.5ad) 2014-06-09 16:22:11.687127 7f869af78700 5 -- op tracker -- , seq: 2645, time: 2014-06-09 16:22:11.687126, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5ad) 2014-06-09 16:22:11.687130 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5ad) v4 from osd.2 2014-06-09 16:22:11.687131 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf96c0 2014-06-09 16:22:11.687134 7f869af78700 5 -- op tracker -- , seq: 2645, time: 2014-06-09 16:22:11.687133, event: started, op: pg_info(1 pgs e243:2.5ad) 2014-06-09 16:22:11.687143 7f869af78700 5 -- op tracker -- , seq: 2645, time: 2014-06-09 16:22:11.687142, event: done, op: pg_info(1 pgs e243:2.5ad) 2014-06-09 16:22:11.687147 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687148 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687155 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687156 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687156 7f869af78700 20 osd.26 243 _dispatch 0x310d6fc0 pg_info(1 pgs e243:1.359) v4 2014-06-09 16:22:11.687159 7f869af78700 5 -- op tracker -- , seq: 2646, time: 2014-06-09 16:22:11.422808, event: header_read, op: pg_info(1 pgs e243:1.359) 2014-06-09 16:22:11.687163 7f869af78700 5 -- op tracker -- , seq: 2646, time: 2014-06-09 16:22:11.422809, event: throttled, op: pg_info(1 pgs e243:1.359) 2014-06-09 16:22:11.687166 7f869af78700 5 -- op tracker -- , seq: 2646, time: 2014-06-09 16:22:11.422826, event: all_read, op: pg_info(1 pgs e243:1.359) 2014-06-09 16:22:11.687169 7f869af78700 5 -- op tracker -- , seq: 2646, time: 2014-06-09 16:22:11.687155, event: dispatched, op: pg_info(1 pgs e243:1.359) 2014-06-09 16:22:11.687174 7f869af78700 5 -- op tracker -- , seq: 2646, time: 2014-06-09 16:22:11.687173, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.359) 2014-06-09 16:22:11.687177 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.359) v4 from osd.4 2014-06-09 16:22:11.687178 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d6fc0 2014-06-09 16:22:11.687181 7f869af78700 5 -- op tracker -- , seq: 2646, time: 2014-06-09 16:22:11.687180, event: started, op: pg_info(1 pgs e243:1.359) 2014-06-09 16:22:11.687189 7f869af78700 5 -- op tracker -- , seq: 2646, time: 2014-06-09 16:22:11.687188, event: done, op: pg_info(1 pgs e243:1.359) 2014-06-09 16:22:11.687192 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687193 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687200 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687201 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687202 7f869af78700 20 osd.26 243 _dispatch 0x30dc2c40 pg_info(1 pgs e243:1.45d) v4 2014-06-09 16:22:11.687204 7f869af78700 5 -- op tracker -- , seq: 2647, time: 2014-06-09 16:22:11.570777, event: header_read, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.687208 7f869af78700 5 -- op tracker -- , seq: 2647, time: 2014-06-09 16:22:11.570778, event: throttled, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.687211 7f869af78700 5 -- op tracker -- , seq: 2647, time: 2014-06-09 16:22:11.570790, event: all_read, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.687214 7f869af78700 5 -- op tracker -- , seq: 2647, time: 2014-06-09 16:22:11.687200, event: dispatched, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.687219 7f869af78700 5 -- op tracker -- , seq: 2647, time: 2014-06-09 16:22:11.687218, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.687222 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.45d) v4 from osd.7 2014-06-09 16:22:11.687223 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc2c40 2014-06-09 16:22:11.687226 7f869af78700 5 -- op tracker -- , seq: 2647, time: 2014-06-09 16:22:11.687225, event: started, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.687234 7f869af78700 5 -- op tracker -- , seq: 2647, time: 2014-06-09 16:22:11.687233, event: done, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.687240 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687241 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687248 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687248 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687249 7f869af78700 20 osd.26 243 _dispatch 0x2f030a80 pg_info(1 pgs e243:2.606) v4 2014-06-09 16:22:11.687252 7f869af78700 5 -- op tracker -- , seq: 2648, time: 2014-06-09 16:22:11.449000, event: header_read, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.687255 7f869af78700 5 -- op tracker -- , seq: 2648, time: 2014-06-09 16:22:11.449000, event: throttled, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.687259 7f869af78700 5 -- op tracker -- , seq: 2648, time: 2014-06-09 16:22:11.449014, event: all_read, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.687262 7f869af78700 5 -- op tracker -- , seq: 2648, time: 2014-06-09 16:22:11.687248, event: dispatched, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.687267 7f869af78700 5 -- op tracker -- , seq: 2648, time: 2014-06-09 16:22:11.687266, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.687269 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.606) v4 from osd.3 2014-06-09 16:22:11.687271 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f030a80 2014-06-09 16:22:11.687273 7f869af78700 5 -- op tracker -- , seq: 2648, time: 2014-06-09 16:22:11.687272, event: started, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.687281 7f869af78700 5 -- op tracker -- , seq: 2648, time: 2014-06-09 16:22:11.687280, event: done, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.687285 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687286 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687292 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687293 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687293 7f869af78700 20 osd.26 243 _dispatch 0x310e8380 pg_info(1 pgs e243:1.560) v4 2014-06-09 16:22:11.687296 7f869af78700 5 -- op tracker -- , seq: 2649, time: 2014-06-09 16:22:11.429951, event: header_read, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.687299 7f869af78700 5 -- op tracker -- , seq: 2649, time: 2014-06-09 16:22:11.429952, event: throttled, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.687303 7f869af78700 5 -- op tracker -- , seq: 2649, time: 2014-06-09 16:22:11.429964, event: all_read, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.687306 7f869af78700 5 -- op tracker -- , seq: 2649, time: 2014-06-09 16:22:11.687292, event: dispatched, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.687311 7f869af78700 5 -- op tracker -- , seq: 2649, time: 2014-06-09 16:22:11.687310, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.687313 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.560) v4 from osd.1 2014-06-09 16:22:11.687315 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e8380 2014-06-09 16:22:11.687317 7f869af78700 5 -- op tracker -- , seq: 2649, time: 2014-06-09 16:22:11.687316, event: started, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.687326 7f869af78700 5 -- op tracker -- , seq: 2649, time: 2014-06-09 16:22:11.687325, event: done, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.687330 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687330 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687337 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687338 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687338 7f869af78700 20 osd.26 243 _dispatch 0x2176cc40 pg_info(1 pgs e243:2.27a) v4 2014-06-09 16:22:11.687341 7f869af78700 5 -- op tracker -- , seq: 2650, time: 2014-06-09 16:22:11.574420, event: header_read, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.687345 7f869af78700 5 -- op tracker -- , seq: 2650, time: 2014-06-09 16:22:11.574421, event: throttled, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.687348 7f869af78700 5 -- op tracker -- , seq: 2650, time: 2014-06-09 16:22:11.574433, event: all_read, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.687351 7f869af78700 5 -- op tracker -- , seq: 2650, time: 2014-06-09 16:22:11.687337, event: dispatched, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.687355 7f869af78700 5 -- op tracker -- , seq: 2650, time: 2014-06-09 16:22:11.687354, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.687358 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.27a) v4 from osd.9 2014-06-09 16:22:11.687359 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176cc40 2014-06-09 16:22:11.687361 7f869af78700 5 -- op tracker -- , seq: 2650, time: 2014-06-09 16:22:11.687361, event: started, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.687370 7f869af78700 5 -- op tracker -- , seq: 2650, time: 2014-06-09 16:22:11.687368, event: done, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.687377 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687378 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687385 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687385 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687386 7f869af78700 20 osd.26 243 _dispatch 0x31156e00 pg_info(1 pgs e243:3.56a) v4 2014-06-09 16:22:11.687389 7f869af78700 5 -- op tracker -- , seq: 2651, time: 2014-06-09 16:22:11.434414, event: header_read, op: pg_info(1 pgs e243:3.56a) 2014-06-09 16:22:11.687392 7f869af78700 5 -- op tracker -- , seq: 2651, time: 2014-06-09 16:22:11.434415, event: throttled, op: pg_info(1 pgs e243:3.56a) 2014-06-09 16:22:11.687396 7f869af78700 5 -- op tracker -- , seq: 2651, time: 2014-06-09 16:22:11.434427, event: all_read, op: pg_info(1 pgs e243:3.56a) 2014-06-09 16:22:11.687399 7f869af78700 5 -- op tracker -- , seq: 2651, time: 2014-06-09 16:22:11.687384, event: dispatched, op: pg_info(1 pgs e243:3.56a) 2014-06-09 16:22:11.687402 7f869af78700 5 -- op tracker -- , seq: 2651, time: 2014-06-09 16:22:11.687401, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.56a) 2014-06-09 16:22:11.687404 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.56a) v4 from osd.8 2014-06-09 16:22:11.687406 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31156e00 2014-06-09 16:22:11.687408 7f869af78700 5 -- op tracker -- , seq: 2651, time: 2014-06-09 16:22:11.687407, event: started, op: pg_info(1 pgs e243:3.56a) 2014-06-09 16:22:11.687416 7f869af78700 5 -- op tracker -- , seq: 2651, time: 2014-06-09 16:22:11.687415, event: done, op: pg_info(1 pgs e243:3.56a) 2014-06-09 16:22:11.687421 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687428 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687428 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687429 7f869af78700 20 osd.26 243 _dispatch 0x311636c0 pg_info(1 pgs e243:0.1c8) v4 2014-06-09 16:22:11.687431 7f869af78700 5 -- op tracker -- , seq: 2652, time: 2014-06-09 16:22:11.423776, event: header_read, op: pg_info(1 pgs e243:0.1c8) 2014-06-09 16:22:11.687435 7f869af78700 5 -- op tracker -- , seq: 2652, time: 2014-06-09 16:22:11.423777, event: throttled, op: pg_info(1 pgs e243:0.1c8) 2014-06-09 16:22:11.687439 7f869af78700 5 -- op tracker -- , seq: 2652, time: 2014-06-09 16:22:11.423786, event: all_read, op: pg_info(1 pgs e243:0.1c8) 2014-06-09 16:22:11.687445 7f869af78700 5 -- op tracker -- , seq: 2652, time: 2014-06-09 16:22:11.687427, event: dispatched, op: pg_info(1 pgs e243:0.1c8) 2014-06-09 16:22:11.687451 7f869af78700 5 -- op tracker -- , seq: 2652, time: 2014-06-09 16:22:11.687450, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1c8) 2014-06-09 16:22:11.687454 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1c8) v4 from osd.6 2014-06-09 16:22:11.687455 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311636c0 2014-06-09 16:22:11.687458 7f869af78700 5 -- op tracker -- , seq: 2652, time: 2014-06-09 16:22:11.687457, event: started, op: pg_info(1 pgs e243:0.1c8) 2014-06-09 16:22:11.687466 7f869af78700 5 -- op tracker -- , seq: 2652, time: 2014-06-09 16:22:11.687465, event: done, op: pg_info(1 pgs e243:0.1c8) 2014-06-09 16:22:11.687470 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687470 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687477 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687477 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687478 7f869af78700 20 osd.26 243 _dispatch 0x30dc2fc0 pg_info(1 pgs e243:1.382) v4 2014-06-09 16:22:11.687481 7f869af78700 5 -- op tracker -- , seq: 2653, time: 2014-06-09 16:22:11.565855, event: header_read, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687484 7f869af78700 5 -- op tracker -- , seq: 2653, time: 2014-06-09 16:22:11.565856, event: throttled, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687488 7f869af78700 5 -- op tracker -- , seq: 2653, time: 2014-06-09 16:22:11.565866, event: all_read, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687491 7f869af78700 5 -- op tracker -- , seq: 2653, time: 2014-06-09 16:22:11.687477, event: dispatched, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687496 7f869af78700 5 -- op tracker -- , seq: 2653, time: 2014-06-09 16:22:11.687495, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687498 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.382) v4 from osd.0 2014-06-09 16:22:11.687500 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc2fc0 2014-06-09 16:22:11.687502 7f869af78700 5 -- op tracker -- , seq: 2653, time: 2014-06-09 16:22:11.687501, event: started, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687510 7f869af78700 5 -- op tracker -- , seq: 2653, time: 2014-06-09 16:22:11.687509, event: done, op: pg_info(1 pgs e243:1.382) 2014-06-09 16:22:11.687514 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687515 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687521 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687522 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687523 7f869af78700 20 osd.26 243 _dispatch 0x30daf6c0 pg_info(1 pgs e243:0.54c) v4 2014-06-09 16:22:11.687525 7f869af78700 5 -- op tracker -- , seq: 2654, time: 2014-06-09 16:22:11.446427, event: header_read, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687529 7f869af78700 5 -- op tracker -- , seq: 2654, time: 2014-06-09 16:22:11.446427, event: throttled, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687532 7f869af78700 5 -- op tracker -- , seq: 2654, time: 2014-06-09 16:22:11.446438, event: all_read, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687536 7f869af78700 5 -- op tracker -- , seq: 2654, time: 2014-06-09 16:22:11.687521, event: dispatched, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687541 7f869af78700 5 -- op tracker -- , seq: 2654, time: 2014-06-09 16:22:11.687540, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687543 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.54c) v4 from osd.5 2014-06-09 16:22:11.687544 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daf6c0 2014-06-09 16:22:11.687547 7f869af78700 5 -- op tracker -- , seq: 2654, time: 2014-06-09 16:22:11.687546, event: started, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687555 7f869af78700 5 -- op tracker -- , seq: 2654, time: 2014-06-09 16:22:11.687554, event: done, op: pg_info(1 pgs e243:0.54c) 2014-06-09 16:22:11.687559 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687560 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687566 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687567 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687567 7f869af78700 20 osd.26 243 _dispatch 0x30cf9500 pg_info(1 pgs e243:1.16d) v4 2014-06-09 16:22:11.687570 7f869af78700 5 -- op tracker -- , seq: 2655, time: 2014-06-09 16:22:11.303091, event: header_read, op: pg_info(1 pgs e243:1.16d) 2014-06-09 16:22:11.687574 7f869af78700 5 -- op tracker -- , seq: 2655, time: 2014-06-09 16:22:11.303091, event: throttled, op: pg_info(1 pgs e243:1.16d) 2014-06-09 16:22:11.687580 7f869af78700 5 -- op tracker -- , seq: 2655, time: 2014-06-09 16:22:11.303097, event: all_read, op: pg_info(1 pgs e243:1.16d) 2014-06-09 16:22:11.687584 7f869af78700 5 -- op tracker -- , seq: 2655, time: 2014-06-09 16:22:11.687566, event: dispatched, op: pg_info(1 pgs e243:1.16d) 2014-06-09 16:22:11.687588 7f869af78700 5 -- op tracker -- , seq: 2655, time: 2014-06-09 16:22:11.687587, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.16d) 2014-06-09 16:22:11.687590 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.16d) v4 from osd.2 2014-06-09 16:22:11.687591 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf9500 2014-06-09 16:22:11.687594 7f869af78700 5 -- op tracker -- , seq: 2655, time: 2014-06-09 16:22:11.687593, event: started, op: pg_info(1 pgs e243:1.16d) 2014-06-09 16:22:11.687603 7f869af78700 5 -- op tracker -- , seq: 2655, time: 2014-06-09 16:22:11.687602, event: done, op: pg_info(1 pgs e243:1.16d) 2014-06-09 16:22:11.687608 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687609 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687630 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687631 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687632 7f869af78700 20 osd.26 243 _dispatch 0x310d6e00 pg_info(1 pgs e243:2.383) v4 2014-06-09 16:22:11.687635 7f869af78700 5 -- op tracker -- , seq: 2656, time: 2014-06-09 16:22:11.422830, event: header_read, op: pg_info(1 pgs e243:2.383) 2014-06-09 16:22:11.687641 7f869af78700 5 -- op tracker -- , seq: 2656, time: 2014-06-09 16:22:11.422830, event: throttled, op: pg_info(1 pgs e243:2.383) 2014-06-09 16:22:11.687645 7f869af78700 5 -- op tracker -- , seq: 2656, time: 2014-06-09 16:22:11.422845, event: all_read, op: pg_info(1 pgs e243:2.383) 2014-06-09 16:22:11.687648 7f869af78700 5 -- op tracker -- , seq: 2656, time: 2014-06-09 16:22:11.687630, event: dispatched, op: pg_info(1 pgs e243:2.383) 2014-06-09 16:22:11.687652 7f869af78700 5 -- op tracker -- , seq: 2656, time: 2014-06-09 16:22:11.687651, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.383) 2014-06-09 16:22:11.687654 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.383) v4 from osd.4 2014-06-09 16:22:11.687656 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d6e00 2014-06-09 16:22:11.687658 7f869af78700 5 -- op tracker -- , seq: 2656, time: 2014-06-09 16:22:11.687657, event: started, op: pg_info(1 pgs e243:2.383) 2014-06-09 16:22:11.687667 7f869af78700 5 -- op tracker -- , seq: 2656, time: 2014-06-09 16:22:11.687666, event: done, op: pg_info(1 pgs e243:2.383) 2014-06-09 16:22:11.687672 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687672 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687680 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687680 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687681 7f869af78700 20 osd.26 243 _dispatch 0x30ebb6c0 pg_info(1 pgs e243:0.25f) v4 2014-06-09 16:22:11.687683 7f869af78700 5 -- op tracker -- , seq: 2657, time: 2014-06-09 16:22:11.570903, event: header_read, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.687689 7f869af78700 5 -- op tracker -- , seq: 2657, time: 2014-06-09 16:22:11.570904, event: throttled, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.687692 7f869af78700 5 -- op tracker -- , seq: 2657, time: 2014-06-09 16:22:11.570920, event: all_read, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.687700 7f869af78700 5 -- op tracker -- , seq: 2657, time: 2014-06-09 16:22:11.687679, event: dispatched, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.687703 7f869af78700 5 -- op tracker -- , seq: 2657, time: 2014-06-09 16:22:11.687702, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.687709 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.25f) v4 from osd.7 2014-06-09 16:22:11.687711 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebb6c0 2014-06-09 16:22:11.687713 7f869af78700 5 -- op tracker -- , seq: 2657, time: 2014-06-09 16:22:11.687712, event: started, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.687723 7f869af78700 5 -- op tracker -- , seq: 2657, time: 2014-06-09 16:22:11.687722, event: done, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.687728 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687729 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687735 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687736 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687736 7f869af78700 20 osd.26 243 _dispatch 0x2f0308c0 pg_info(1 pgs e243:3.359) v4 2014-06-09 16:22:11.687739 7f869af78700 5 -- op tracker -- , seq: 2658, time: 2014-06-09 16:22:11.449037, event: header_read, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.687743 7f869af78700 5 -- op tracker -- , seq: 2658, time: 2014-06-09 16:22:11.449038, event: throttled, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.687746 7f869af78700 5 -- op tracker -- , seq: 2658, time: 2014-06-09 16:22:11.449050, event: all_read, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.687750 7f869af78700 5 -- op tracker -- , seq: 2658, time: 2014-06-09 16:22:11.687735, event: dispatched, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.687753 7f869af78700 5 -- op tracker -- , seq: 2658, time: 2014-06-09 16:22:11.687752, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.687756 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.359) v4 from osd.3 2014-06-09 16:22:11.687757 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f0308c0 2014-06-09 16:22:11.687760 7f869af78700 5 -- op tracker -- , seq: 2658, time: 2014-06-09 16:22:11.687759, event: started, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.687768 7f869af78700 5 -- op tracker -- , seq: 2658, time: 2014-06-09 16:22:11.687767, event: done, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.687773 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687774 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687780 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687782 7f869af78700 20 osd.26 243 _dispatch 0x240ea1c0 pg_info(1 pgs e243:2.6ec) v4 2014-06-09 16:22:11.687784 7f869af78700 5 -- op tracker -- , seq: 2659, time: 2014-06-09 16:22:11.429972, event: header_read, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.687788 7f869af78700 5 -- op tracker -- , seq: 2659, time: 2014-06-09 16:22:11.429972, event: throttled, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.687791 7f869af78700 5 -- op tracker -- , seq: 2659, time: 2014-06-09 16:22:11.429991, event: all_read, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.687796 7f869af78700 5 -- op tracker -- , seq: 2659, time: 2014-06-09 16:22:11.687780, event: dispatched, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.687799 7f869af78700 5 -- op tracker -- , seq: 2659, time: 2014-06-09 16:22:11.687798, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.687805 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6ec) v4 from osd.1 2014-06-09 16:22:11.687807 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ea1c0 2014-06-09 16:22:11.687809 7f869af78700 5 -- op tracker -- , seq: 2659, time: 2014-06-09 16:22:11.687808, event: started, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.687817 7f869af78700 5 -- op tracker -- , seq: 2659, time: 2014-06-09 16:22:11.687816, event: done, op: pg_info(1 pgs e243:2.6ec) 2014-06-09 16:22:11.687821 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687822 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687829 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687829 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687830 7f869af78700 20 osd.26 243 _dispatch 0x2176d180 pg_info(1 pgs e243:2.d3) v4 2014-06-09 16:22:11.687832 7f869af78700 5 -- op tracker -- , seq: 2660, time: 2014-06-09 16:22:11.574438, event: header_read, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.687836 7f869af78700 5 -- op tracker -- , seq: 2660, time: 2014-06-09 16:22:11.574439, event: throttled, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.687839 7f869af78700 5 -- op tracker -- , seq: 2660, time: 2014-06-09 16:22:11.574448, event: all_read, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.687843 7f869af78700 5 -- op tracker -- , seq: 2660, time: 2014-06-09 16:22:11.687828, event: dispatched, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.687850 7f869af78700 5 -- op tracker -- , seq: 2660, time: 2014-06-09 16:22:11.687849, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.687853 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.d3) v4 from osd.9 2014-06-09 16:22:11.687854 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176d180 2014-06-09 16:22:11.687857 7f869af78700 5 -- op tracker -- , seq: 2660, time: 2014-06-09 16:22:11.687856, event: started, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.687865 7f869af78700 5 -- op tracker -- , seq: 2660, time: 2014-06-09 16:22:11.687864, event: done, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.687869 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687870 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687876 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687877 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687878 7f869af78700 20 osd.26 243 _dispatch 0x31156c40 pg_info(1 pgs e243:3.7e2) v4 2014-06-09 16:22:11.687880 7f869af78700 5 -- op tracker -- , seq: 2661, time: 2014-06-09 16:22:11.434471, event: header_read, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.687884 7f869af78700 5 -- op tracker -- , seq: 2661, time: 2014-06-09 16:22:11.434471, event: throttled, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.687888 7f869af78700 5 -- op tracker -- , seq: 2661, time: 2014-06-09 16:22:11.434483, event: all_read, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.687893 7f869af78700 5 -- op tracker -- , seq: 2661, time: 2014-06-09 16:22:11.687876, event: dispatched, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.687897 7f869af78700 5 -- op tracker -- , seq: 2661, time: 2014-06-09 16:22:11.687896, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.687902 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7e2) v4 from osd.8 2014-06-09 16:22:11.687904 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31156c40 2014-06-09 16:22:11.687907 7f869af78700 5 -- op tracker -- , seq: 2661, time: 2014-06-09 16:22:11.687906, event: started, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.687916 7f869af78700 5 -- op tracker -- , seq: 2661, time: 2014-06-09 16:22:11.687915, event: done, op: pg_info(1 pgs e243:3.7e2) 2014-06-09 16:22:11.687920 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687921 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687927 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687928 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687928 7f869af78700 20 osd.26 243 _dispatch 0x31163500 pg_info(1 pgs e243:3.59f) v4 2014-06-09 16:22:11.687931 7f869af78700 5 -- op tracker -- , seq: 2662, time: 2014-06-09 16:22:11.423795, event: header_read, op: pg_info(1 pgs e243:3.59f) 2014-06-09 16:22:11.687935 7f869af78700 5 -- op tracker -- , seq: 2662, time: 2014-06-09 16:22:11.423796, event: throttled, op: pg_info(1 pgs e243:3.59f) 2014-06-09 16:22:11.687938 7f869af78700 5 -- op tracker -- , seq: 2662, time: 2014-06-09 16:22:11.423805, event: all_read, op: pg_info(1 pgs e243:3.59f) 2014-06-09 16:22:11.687941 7f869af78700 5 -- op tracker -- , seq: 2662, time: 2014-06-09 16:22:11.687927, event: dispatched, op: pg_info(1 pgs e243:3.59f) 2014-06-09 16:22:11.687947 7f869af78700 5 -- op tracker -- , seq: 2662, time: 2014-06-09 16:22:11.687945, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.59f) 2014-06-09 16:22:11.687949 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.59f) v4 from osd.6 2014-06-09 16:22:11.687950 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31163500 2014-06-09 16:22:11.687953 7f869af78700 5 -- op tracker -- , seq: 2662, time: 2014-06-09 16:22:11.687952, event: started, op: pg_info(1 pgs e243:3.59f) 2014-06-09 16:22:11.687961 7f869af78700 5 -- op tracker -- , seq: 2662, time: 2014-06-09 16:22:11.687960, event: done, op: pg_info(1 pgs e243:3.59f) 2014-06-09 16:22:11.687965 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687966 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687972 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.687973 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.687973 7f869af78700 20 osd.26 243 _dispatch 0x311676c0 pg_info(1 pgs e243:0.380) v4 2014-06-09 16:22:11.687976 7f869af78700 5 -- op tracker -- , seq: 2663, time: 2014-06-09 16:22:11.565870, event: header_read, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.687980 7f869af78700 5 -- op tracker -- , seq: 2663, time: 2014-06-09 16:22:11.565870, event: throttled, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.687985 7f869af78700 5 -- op tracker -- , seq: 2663, time: 2014-06-09 16:22:11.565885, event: all_read, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.687988 7f869af78700 5 -- op tracker -- , seq: 2663, time: 2014-06-09 16:22:11.687972, event: dispatched, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.687995 7f869af78700 5 -- op tracker -- , seq: 2663, time: 2014-06-09 16:22:11.687994, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.687998 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.380) v4 from osd.0 2014-06-09 16:22:11.687999 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311676c0 2014-06-09 16:22:11.688002 7f869af78700 5 -- op tracker -- , seq: 2663, time: 2014-06-09 16:22:11.688001, event: started, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.688011 7f869af78700 5 -- op tracker -- , seq: 2663, time: 2014-06-09 16:22:11.688009, event: done, op: pg_info(1 pgs e243:0.380) 2014-06-09 16:22:11.688015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688016 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688022 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688023 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688023 7f869af78700 20 osd.26 243 _dispatch 0x31166e00 pg_info(1 pgs e243:2.3d1) v4 2014-06-09 16:22:11.688026 7f869af78700 5 -- op tracker -- , seq: 2664, time: 2014-06-09 16:22:11.471652, event: header_read, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688030 7f869af78700 5 -- op tracker -- , seq: 2664, time: 2014-06-09 16:22:11.471653, event: throttled, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688033 7f869af78700 5 -- op tracker -- , seq: 2664, time: 2014-06-09 16:22:11.471669, event: all_read, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688040 7f869af78700 5 -- op tracker -- , seq: 2664, time: 2014-06-09 16:22:11.688022, event: dispatched, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688044 7f869af78700 5 -- op tracker -- , seq: 2664, time: 2014-06-09 16:22:11.688043, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688049 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3d1) v4 from osd.5 2014-06-09 16:22:11.688051 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31166e00 2014-06-09 16:22:11.688054 7f869af78700 5 -- op tracker -- , seq: 2664, time: 2014-06-09 16:22:11.688052, event: started, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688062 7f869af78700 5 -- op tracker -- , seq: 2664, time: 2014-06-09 16:22:11.688061, event: done, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688066 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688067 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688074 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688074 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688075 7f869af78700 20 osd.26 243 _dispatch 0x30cf9340 pg_info(1 pgs e243:1.5e1) v4 2014-06-09 16:22:11.688078 7f869af78700 5 -- op tracker -- , seq: 2665, time: 2014-06-09 16:22:11.303104, event: header_read, op: pg_info(1 pgs e243:1.5e1) 2014-06-09 16:22:11.688085 7f869af78700 5 -- op tracker -- , seq: 2665, time: 2014-06-09 16:22:11.303105, event: throttled, op: pg_info(1 pgs e243:1.5e1) 2014-06-09 16:22:11.688088 7f869af78700 5 -- op tracker -- , seq: 2665, time: 2014-06-09 16:22:11.303111, event: all_read, op: pg_info(1 pgs e243:1.5e1) 2014-06-09 16:22:11.688094 7f869af78700 5 -- op tracker -- , seq: 2665, time: 2014-06-09 16:22:11.688074, event: dispatched, op: pg_info(1 pgs e243:1.5e1) 2014-06-09 16:22:11.688098 7f869af78700 5 -- op tracker -- , seq: 2665, time: 2014-06-09 16:22:11.688097, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5e1) 2014-06-09 16:22:11.688102 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5e1) v4 from osd.2 2014-06-09 16:22:11.688103 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf9340 2014-06-09 16:22:11.688106 7f869af78700 5 -- op tracker -- , seq: 2665, time: 2014-06-09 16:22:11.688105, event: started, op: pg_info(1 pgs e243:1.5e1) 2014-06-09 16:22:11.688114 7f869af78700 5 -- op tracker -- , seq: 2665, time: 2014-06-09 16:22:11.688113, event: done, op: pg_info(1 pgs e243:1.5e1) 2014-06-09 16:22:11.688118 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688119 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688126 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688126 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688127 7f869af78700 20 osd.26 243 _dispatch 0x310d6c40 pg_info(1 pgs e243:0.594) v4 2014-06-09 16:22:11.688130 7f869af78700 5 -- op tracker -- , seq: 2666, time: 2014-06-09 16:22:11.422849, event: header_read, op: pg_info(1 pgs e243:0.594) 2014-06-09 16:22:11.688135 7f869af78700 5 -- op tracker -- , seq: 2666, time: 2014-06-09 16:22:11.422849, event: throttled, op: pg_info(1 pgs e243:0.594) 2014-06-09 16:22:11.688138 7f869af78700 5 -- op tracker -- , seq: 2666, time: 2014-06-09 16:22:11.422857, event: all_read, op: pg_info(1 pgs e243:0.594) 2014-06-09 16:22:11.688145 7f869af78700 5 -- op tracker -- , seq: 2666, time: 2014-06-09 16:22:11.688126, event: dispatched, op: pg_info(1 pgs e243:0.594) 2014-06-09 16:22:11.688149 7f869af78700 5 -- op tracker -- , seq: 2666, time: 2014-06-09 16:22:11.688147, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.594) 2014-06-09 16:22:11.688151 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.594) v4 from osd.4 2014-06-09 16:22:11.688152 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d6c40 2014-06-09 16:22:11.688155 7f869af78700 5 -- op tracker -- , seq: 2666, time: 2014-06-09 16:22:11.688154, event: started, op: pg_info(1 pgs e243:0.594) 2014-06-09 16:22:11.688163 7f869af78700 5 -- op tracker -- , seq: 2666, time: 2014-06-09 16:22:11.688162, event: done, op: pg_info(1 pgs e243:0.594) 2014-06-09 16:22:11.688168 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688168 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688175 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688175 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688176 7f869af78700 20 osd.26 243 _dispatch 0x310e9340 pg_info(1 pgs e243:1.45d) v4 2014-06-09 16:22:11.688179 7f869af78700 5 -- op tracker -- , seq: 2667, time: 2014-06-09 16:22:11.571376, event: header_read, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.688183 7f869af78700 5 -- op tracker -- , seq: 2667, time: 2014-06-09 16:22:11.571377, event: throttled, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.688186 7f869af78700 5 -- op tracker -- , seq: 2667, time: 2014-06-09 16:22:11.571390, event: all_read, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.688189 7f869af78700 5 -- op tracker -- , seq: 2667, time: 2014-06-09 16:22:11.688175, event: dispatched, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.688192 7f869af78700 5 -- op tracker -- , seq: 2667, time: 2014-06-09 16:22:11.688191, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.688195 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.45d) v4 from osd.7 2014-06-09 16:22:11.688196 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e9340 2014-06-09 16:22:11.688199 7f869af78700 5 -- op tracker -- , seq: 2667, time: 2014-06-09 16:22:11.688198, event: started, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.688210 7f869af78700 5 -- op tracker -- , seq: 2667, time: 2014-06-09 16:22:11.688209, event: done, op: pg_info(1 pgs e243:1.45d) 2014-06-09 16:22:11.688214 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688215 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688223 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688224 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688224 7f869af78700 20 osd.26 243 _dispatch 0x2f030700 pg_info(1 pgs e243:2.606) v4 2014-06-09 16:22:11.688227 7f869af78700 5 -- op tracker -- , seq: 2668, time: 2014-06-09 16:22:11.449195, event: header_read, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.688230 7f869af78700 5 -- op tracker -- , seq: 2668, time: 2014-06-09 16:22:11.449195, event: throttled, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.688234 7f869af78700 5 -- op tracker -- , seq: 2668, time: 2014-06-09 16:22:11.449209, event: all_read, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.688238 7f869af78700 5 -- op tracker -- , seq: 2668, time: 2014-06-09 16:22:11.688223, event: dispatched, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.688243 7f869af78700 5 -- op tracker -- , seq: 2668, time: 2014-06-09 16:22:11.688242, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.688245 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.606) v4 from osd.3 2014-06-09 16:22:11.688247 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f030700 2014-06-09 16:22:11.688249 7f869af78700 5 -- op tracker -- , seq: 2668, time: 2014-06-09 16:22:11.688248, event: started, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.688257 7f869af78700 5 -- op tracker -- , seq: 2668, time: 2014-06-09 16:22:11.688256, event: done, op: pg_info(1 pgs e243:2.606) 2014-06-09 16:22:11.688261 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688262 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688268 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688269 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688270 7f869af78700 20 osd.26 243 _dispatch 0x240ebc00 pg_info(1 pgs e243:2.530) v4 2014-06-09 16:22:11.688272 7f869af78700 5 -- op tracker -- , seq: 2669, time: 2014-06-09 16:22:11.429997, event: header_read, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.688276 7f869af78700 5 -- op tracker -- , seq: 2669, time: 2014-06-09 16:22:11.429997, event: throttled, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.688279 7f869af78700 5 -- op tracker -- , seq: 2669, time: 2014-06-09 16:22:11.430008, event: all_read, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.688283 7f869af78700 5 -- op tracker -- , seq: 2669, time: 2014-06-09 16:22:11.688268, event: dispatched, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.688286 7f869af78700 5 -- op tracker -- , seq: 2669, time: 2014-06-09 16:22:11.688285, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.688289 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.530) v4 from osd.1 2014-06-09 16:22:11.688290 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ebc00 2014-06-09 16:22:11.688292 7f869af78700 5 -- op tracker -- , seq: 2669, time: 2014-06-09 16:22:11.688291, event: started, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.688302 7f869af78700 5 -- op tracker -- , seq: 2669, time: 2014-06-09 16:22:11.688301, event: done, op: pg_info(1 pgs e243:2.530) 2014-06-09 16:22:11.688306 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688306 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688313 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688313 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688314 7f869af78700 20 osd.26 243 _dispatch 0x30dc2380 pg_info(1 pgs e243:0.6d2) v4 2014-06-09 16:22:11.688317 7f869af78700 5 -- op tracker -- , seq: 2670, time: 2014-06-09 16:22:11.574452, event: header_read, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.688321 7f869af78700 5 -- op tracker -- , seq: 2670, time: 2014-06-09 16:22:11.574453, event: throttled, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.688325 7f869af78700 5 -- op tracker -- , seq: 2670, time: 2014-06-09 16:22:11.574465, event: all_read, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.688332 7f869af78700 5 -- op tracker -- , seq: 2670, time: 2014-06-09 16:22:11.688312, event: dispatched, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.688335 7f869af78700 5 -- op tracker -- , seq: 2670, time: 2014-06-09 16:22:11.688334, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.688339 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6d2) v4 from osd.9 2014-06-09 16:22:11.688340 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc2380 2014-06-09 16:22:11.688343 7f869af78700 5 -- op tracker -- , seq: 2670, time: 2014-06-09 16:22:11.688342, event: started, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.688351 7f869af78700 5 -- op tracker -- , seq: 2670, time: 2014-06-09 16:22:11.688350, event: done, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.688355 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688356 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688362 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688363 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688364 7f869af78700 20 osd.26 243 _dispatch 0x31156a80 pg_info(1 pgs e243:3.19) v4 2014-06-09 16:22:11.688366 7f869af78700 5 -- op tracker -- , seq: 2671, time: 2014-06-09 16:22:11.434520, event: header_read, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.688370 7f869af78700 5 -- op tracker -- , seq: 2671, time: 2014-06-09 16:22:11.434521, event: throttled, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.688373 7f869af78700 5 -- op tracker -- , seq: 2671, time: 2014-06-09 16:22:11.434533, event: all_read, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.688377 7f869af78700 5 -- op tracker -- , seq: 2671, time: 2014-06-09 16:22:11.688362, event: dispatched, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.688380 7f869af78700 5 -- op tracker -- , seq: 2671, time: 2014-06-09 16:22:11.688379, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.688382 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.19) v4 from osd.8 2014-06-09 16:22:11.688384 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31156a80 2014-06-09 16:22:11.688386 7f869af78700 5 -- op tracker -- , seq: 2671, time: 2014-06-09 16:22:11.688385, event: started, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.688395 7f869af78700 5 -- op tracker -- , seq: 2671, time: 2014-06-09 16:22:11.688394, event: done, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.688399 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688400 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688406 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688406 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688407 7f869af78700 20 osd.26 243 _dispatch 0x31163340 pg_info(1 pgs e243:2.4ba) v4 2014-06-09 16:22:11.688410 7f869af78700 5 -- op tracker -- , seq: 2672, time: 2014-06-09 16:22:11.423809, event: header_read, op: pg_info(1 pgs e243:2.4ba) 2014-06-09 16:22:11.688415 7f869af78700 5 -- op tracker -- , seq: 2672, time: 2014-06-09 16:22:11.423810, event: throttled, op: pg_info(1 pgs e243:2.4ba) 2014-06-09 16:22:11.688418 7f869af78700 5 -- op tracker -- , seq: 2672, time: 2014-06-09 16:22:11.423819, event: all_read, op: pg_info(1 pgs e243:2.4ba) 2014-06-09 16:22:11.688425 7f869af78700 5 -- op tracker -- , seq: 2672, time: 2014-06-09 16:22:11.688405, event: dispatched, op: pg_info(1 pgs e243:2.4ba) 2014-06-09 16:22:11.688429 7f869af78700 5 -- op tracker -- , seq: 2672, time: 2014-06-09 16:22:11.688428, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4ba) 2014-06-09 16:22:11.688433 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4ba) v4 from osd.6 2014-06-09 16:22:11.688434 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31163340 2014-06-09 16:22:11.688437 7f869af78700 5 -- op tracker -- , seq: 2672, time: 2014-06-09 16:22:11.688436, event: started, op: pg_info(1 pgs e243:2.4ba) 2014-06-09 16:22:11.688447 7f869af78700 5 -- op tracker -- , seq: 2672, time: 2014-06-09 16:22:11.688446, event: done, op: pg_info(1 pgs e243:2.4ba) 2014-06-09 16:22:11.688451 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688452 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688458 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688459 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688459 7f869af78700 20 osd.26 243 _dispatch 0x1c03fc00 pg_info(1 pgs e243:2.e6) v4 2014-06-09 16:22:11.688462 7f869af78700 5 -- op tracker -- , seq: 2673, time: 2014-06-09 16:22:11.565891, event: header_read, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.688466 7f869af78700 5 -- op tracker -- , seq: 2673, time: 2014-06-09 16:22:11.565891, event: throttled, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.688469 7f869af78700 5 -- op tracker -- , seq: 2673, time: 2014-06-09 16:22:11.565902, event: all_read, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.688472 7f869af78700 5 -- op tracker -- , seq: 2673, time: 2014-06-09 16:22:11.688458, event: dispatched, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.688476 7f869af78700 5 -- op tracker -- , seq: 2673, time: 2014-06-09 16:22:11.688475, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.688478 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.e6) v4 from osd.0 2014-06-09 16:22:11.688479 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03fc00 2014-06-09 16:22:11.688482 7f869af78700 5 -- op tracker -- , seq: 2673, time: 2014-06-09 16:22:11.688481, event: started, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.688490 7f869af78700 5 -- op tracker -- , seq: 2673, time: 2014-06-09 16:22:11.688489, event: done, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.688494 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688495 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688501 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688501 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688502 7f869af78700 20 osd.26 243 _dispatch 0x30daf500 pg_info(1 pgs e243:2.3d1) v4 2014-06-09 16:22:11.688505 7f869af78700 5 -- op tracker -- , seq: 2674, time: 2014-06-09 16:22:11.471690, event: header_read, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688508 7f869af78700 5 -- op tracker -- , seq: 2674, time: 2014-06-09 16:22:11.471691, event: throttled, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688515 7f869af78700 5 -- op tracker -- , seq: 2674, time: 2014-06-09 16:22:11.471718, event: all_read, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688519 7f869af78700 5 -- op tracker -- , seq: 2674, time: 2014-06-09 16:22:11.688501, event: dispatched, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688523 7f869af78700 5 -- op tracker -- , seq: 2674, time: 2014-06-09 16:22:11.688522, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688525 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3d1) v4 from osd.5 2014-06-09 16:22:11.688526 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daf500 2014-06-09 16:22:11.688529 7f869af78700 5 -- op tracker -- , seq: 2674, time: 2014-06-09 16:22:11.688528, event: started, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688537 7f869af78700 5 -- op tracker -- , seq: 2674, time: 2014-06-09 16:22:11.688536, event: done, op: pg_info(1 pgs e243:2.3d1) 2014-06-09 16:22:11.688541 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688542 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688549 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688549 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688550 7f869af78700 20 osd.26 243 _dispatch 0x30cf9180 pg_info(1 pgs e243:1.25c) v4 2014-06-09 16:22:11.688552 7f869af78700 5 -- op tracker -- , seq: 2675, time: 2014-06-09 16:22:11.303115, event: header_read, op: pg_info(1 pgs e243:1.25c) 2014-06-09 16:22:11.688556 7f869af78700 5 -- op tracker -- , seq: 2675, time: 2014-06-09 16:22:11.303115, event: throttled, op: pg_info(1 pgs e243:1.25c) 2014-06-09 16:22:11.688560 7f869af78700 5 -- op tracker -- , seq: 2675, time: 2014-06-09 16:22:11.303121, event: all_read, op: pg_info(1 pgs e243:1.25c) 2014-06-09 16:22:11.688563 7f869af78700 5 -- op tracker -- , seq: 2675, time: 2014-06-09 16:22:11.688548, event: dispatched, op: pg_info(1 pgs e243:1.25c) 2014-06-09 16:22:11.688568 7f869af78700 5 -- op tracker -- , seq: 2675, time: 2014-06-09 16:22:11.688567, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.25c) 2014-06-09 16:22:11.688570 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.25c) v4 from osd.2 2014-06-09 16:22:11.688572 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf9180 2014-06-09 16:22:11.688574 7f869af78700 5 -- op tracker -- , seq: 2675, time: 2014-06-09 16:22:11.688573, event: started, op: pg_info(1 pgs e243:1.25c) 2014-06-09 16:22:11.688582 7f869af78700 5 -- op tracker -- , seq: 2675, time: 2014-06-09 16:22:11.688581, event: done, op: pg_info(1 pgs e243:1.25c) 2014-06-09 16:22:11.688586 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688587 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688594 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688594 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688595 7f869af78700 20 osd.26 243 _dispatch 0x310d6a80 pg_info(1 pgs e243:3.6cd) v4 2014-06-09 16:22:11.688597 7f869af78700 5 -- op tracker -- , seq: 2676, time: 2014-06-09 16:22:11.422861, event: header_read, op: pg_info(1 pgs e243:3.6cd) 2014-06-09 16:22:11.688601 7f869af78700 5 -- op tracker -- , seq: 2676, time: 2014-06-09 16:22:11.422862, event: throttled, op: pg_info(1 pgs e243:3.6cd) 2014-06-09 16:22:11.688604 7f869af78700 5 -- op tracker -- , seq: 2676, time: 2014-06-09 16:22:11.422871, event: all_read, op: pg_info(1 pgs e243:3.6cd) 2014-06-09 16:22:11.688608 7f869af78700 5 -- op tracker -- , seq: 2676, time: 2014-06-09 16:22:11.688593, event: dispatched, op: pg_info(1 pgs e243:3.6cd) 2014-06-09 16:22:11.688612 7f869af78700 5 -- op tracker -- , seq: 2676, time: 2014-06-09 16:22:11.688611, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6cd) 2014-06-09 16:22:11.688614 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6cd) v4 from osd.4 2014-06-09 16:22:11.688616 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d6a80 2014-06-09 16:22:11.688618 7f869af78700 5 -- op tracker -- , seq: 2676, time: 2014-06-09 16:22:11.688617, event: started, op: pg_info(1 pgs e243:3.6cd) 2014-06-09 16:22:11.688629 7f869af78700 5 -- op tracker -- , seq: 2676, time: 2014-06-09 16:22:11.688628, event: done, op: pg_info(1 pgs e243:3.6cd) 2014-06-09 16:22:11.688633 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688633 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688640 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688640 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688641 7f869af78700 20 osd.26 243 _dispatch 0x310e9180 pg_info(1 pgs e243:0.25f) v4 2014-06-09 16:22:11.688644 7f869af78700 5 -- op tracker -- , seq: 2677, time: 2014-06-09 16:22:11.571591, event: header_read, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.688647 7f869af78700 5 -- op tracker -- , seq: 2677, time: 2014-06-09 16:22:11.571591, event: throttled, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.688651 7f869af78700 5 -- op tracker -- , seq: 2677, time: 2014-06-09 16:22:11.571604, event: all_read, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.688657 7f869af78700 5 -- op tracker -- , seq: 2677, time: 2014-06-09 16:22:11.688639, event: dispatched, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.688664 7f869af78700 5 -- op tracker -- , seq: 2677, time: 2014-06-09 16:22:11.688663, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.688666 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.25f) v4 from osd.7 2014-06-09 16:22:11.688668 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e9180 2014-06-09 16:22:11.688671 7f869af78700 5 -- op tracker -- , seq: 2677, time: 2014-06-09 16:22:11.688670, event: started, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.688682 7f869af78700 5 -- op tracker -- , seq: 2677, time: 2014-06-09 16:22:11.688681, event: done, op: pg_info(1 pgs e243:0.25f) 2014-06-09 16:22:11.688686 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688694 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688694 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688695 7f869af78700 20 osd.26 243 _dispatch 0x2f030540 pg_info(1 pgs e243:3.c1) v4 2014-06-09 16:22:11.688697 7f869af78700 5 -- op tracker -- , seq: 2678, time: 2014-06-09 16:22:11.449303, event: header_read, op: pg_info(1 pgs e243:3.c1) 2014-06-09 16:22:11.688701 7f869af78700 5 -- op tracker -- , seq: 2678, time: 2014-06-09 16:22:11.449304, event: throttled, op: pg_info(1 pgs e243:3.c1) 2014-06-09 16:22:11.688704 7f869af78700 5 -- op tracker -- , seq: 2678, time: 2014-06-09 16:22:11.449317, event: all_read, op: pg_info(1 pgs e243:3.c1) 2014-06-09 16:22:11.688711 7f869af78700 5 -- op tracker -- , seq: 2678, time: 2014-06-09 16:22:11.688693, event: dispatched, op: pg_info(1 pgs e243:3.c1) 2014-06-09 16:22:11.688715 7f869af78700 5 -- op tracker -- , seq: 2678, time: 2014-06-09 16:22:11.688714, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.c1) 2014-06-09 16:22:11.688719 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.c1) v4 from osd.3 2014-06-09 16:22:11.688720 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f030540 2014-06-09 16:22:11.688723 7f869af78700 5 -- op tracker -- , seq: 2678, time: 2014-06-09 16:22:11.688722, event: started, op: pg_info(1 pgs e243:3.c1) 2014-06-09 16:22:11.688733 7f869af78700 5 -- op tracker -- , seq: 2678, time: 2014-06-09 16:22:11.688732, event: done, op: pg_info(1 pgs e243:3.c1) 2014-06-09 16:22:11.688737 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688738 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688745 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688746 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688747 7f869af78700 20 osd.26 243 _dispatch 0x15999880 pg_info(1 pgs e243:1.560) v4 2014-06-09 16:22:11.688749 7f869af78700 5 -- op tracker -- , seq: 2679, time: 2014-06-09 16:22:11.430013, event: header_read, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.688754 7f869af78700 5 -- op tracker -- , seq: 2679, time: 2014-06-09 16:22:11.430013, event: throttled, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.688758 7f869af78700 5 -- op tracker -- , seq: 2679, time: 2014-06-09 16:22:11.430024, event: all_read, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.688765 7f869af78700 5 -- op tracker -- , seq: 2679, time: 2014-06-09 16:22:11.688745, event: dispatched, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.688769 7f869af78700 5 -- op tracker -- , seq: 2679, time: 2014-06-09 16:22:11.688768, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.688771 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.560) v4 from osd.1 2014-06-09 16:22:11.688772 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15999880 2014-06-09 16:22:11.688775 7f869af78700 5 -- op tracker -- , seq: 2679, time: 2014-06-09 16:22:11.688774, event: started, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.688787 7f869af78700 5 -- op tracker -- , seq: 2679, time: 2014-06-09 16:22:11.688785, event: done, op: pg_info(1 pgs e243:1.560) 2014-06-09 16:22:11.688794 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688795 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688803 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688804 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688804 7f869af78700 20 osd.26 243 _dispatch 0x30dc28c0 pg_info(1 pgs e243:3.609) v4 2014-06-09 16:22:11.688807 7f869af78700 5 -- op tracker -- , seq: 2680, time: 2014-06-09 16:22:11.574470, event: header_read, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.688811 7f869af78700 5 -- op tracker -- , seq: 2680, time: 2014-06-09 16:22:11.574470, event: throttled, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.688814 7f869af78700 5 -- op tracker -- , seq: 2680, time: 2014-06-09 16:22:11.574479, event: all_read, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.688817 7f869af78700 5 -- op tracker -- , seq: 2680, time: 2014-06-09 16:22:11.688803, event: dispatched, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.688821 7f869af78700 5 -- op tracker -- , seq: 2680, time: 2014-06-09 16:22:11.688820, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.688823 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.609) v4 from osd.9 2014-06-09 16:22:11.688824 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc28c0 2014-06-09 16:22:11.688827 7f869af78700 5 -- op tracker -- , seq: 2680, time: 2014-06-09 16:22:11.688826, event: started, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.688835 7f869af78700 5 -- op tracker -- , seq: 2680, time: 2014-06-09 16:22:11.688834, event: done, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.688839 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688840 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688846 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688847 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688847 7f869af78700 20 osd.26 243 _dispatch 0x311568c0 pg_info(1 pgs e243:2.2b) v4 2014-06-09 16:22:11.688850 7f869af78700 5 -- op tracker -- , seq: 2681, time: 2014-06-09 16:22:11.434580, event: header_read, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.688853 7f869af78700 5 -- op tracker -- , seq: 2681, time: 2014-06-09 16:22:11.434581, event: throttled, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.688857 7f869af78700 5 -- op tracker -- , seq: 2681, time: 2014-06-09 16:22:11.434595, event: all_read, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.688860 7f869af78700 5 -- op tracker -- , seq: 2681, time: 2014-06-09 16:22:11.688846, event: dispatched, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.688863 7f869af78700 5 -- op tracker -- , seq: 2681, time: 2014-06-09 16:22:11.688862, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.688865 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2b) v4 from osd.8 2014-06-09 16:22:11.688867 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311568c0 2014-06-09 16:22:11.688869 7f869af78700 5 -- op tracker -- , seq: 2681, time: 2014-06-09 16:22:11.688868, event: started, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.688878 7f869af78700 5 -- op tracker -- , seq: 2681, time: 2014-06-09 16:22:11.688877, event: done, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.688882 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688883 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688889 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688889 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688890 7f869af78700 20 osd.26 243 _dispatch 0x31163180 pg_info(1 pgs e243:3.1f4) v4 2014-06-09 16:22:11.688893 7f869af78700 5 -- op tracker -- , seq: 2682, time: 2014-06-09 16:22:11.423822, event: header_read, op: pg_info(1 pgs e243:3.1f4) 2014-06-09 16:22:11.688898 7f869af78700 5 -- op tracker -- , seq: 2682, time: 2014-06-09 16:22:11.423823, event: throttled, op: pg_info(1 pgs e243:3.1f4) 2014-06-09 16:22:11.688902 7f869af78700 5 -- op tracker -- , seq: 2682, time: 2014-06-09 16:22:11.423832, event: all_read, op: pg_info(1 pgs e243:3.1f4) 2014-06-09 16:22:11.688909 7f869af78700 5 -- op tracker -- , seq: 2682, time: 2014-06-09 16:22:11.688889, event: dispatched, op: pg_info(1 pgs e243:3.1f4) 2014-06-09 16:22:11.688915 7f869af78700 5 -- op tracker -- , seq: 2682, time: 2014-06-09 16:22:11.688914, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1f4) 2014-06-09 16:22:11.688918 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1f4) v4 from osd.6 2014-06-09 16:22:11.688919 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31163180 2014-06-09 16:22:11.688922 7f869af78700 5 -- op tracker -- , seq: 2682, time: 2014-06-09 16:22:11.688921, event: started, op: pg_info(1 pgs e243:3.1f4) 2014-06-09 16:22:11.688931 7f869af78700 5 -- op tracker -- , seq: 2682, time: 2014-06-09 16:22:11.688930, event: done, op: pg_info(1 pgs e243:3.1f4) 2014-06-09 16:22:11.688935 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688936 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688942 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688943 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688943 7f869af78700 20 osd.26 243 _dispatch 0x210b1340 pg_info(1 pgs e243:2.57a) v4 2014-06-09 16:22:11.688946 7f869af78700 5 -- op tracker -- , seq: 2683, time: 2014-06-09 16:22:11.565906, event: header_read, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.688951 7f869af78700 5 -- op tracker -- , seq: 2683, time: 2014-06-09 16:22:11.565907, event: throttled, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.688954 7f869af78700 5 -- op tracker -- , seq: 2683, time: 2014-06-09 16:22:11.565917, event: all_read, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.688959 7f869af78700 5 -- op tracker -- , seq: 2683, time: 2014-06-09 16:22:11.688942, event: dispatched, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.688962 7f869af78700 5 -- op tracker -- , seq: 2683, time: 2014-06-09 16:22:11.688961, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.688966 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.57a) v4 from osd.0 2014-06-09 16:22:11.688967 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x210b1340 2014-06-09 16:22:11.688970 7f869af78700 5 -- op tracker -- , seq: 2683, time: 2014-06-09 16:22:11.688969, event: started, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.688979 7f869af78700 5 -- op tracker -- , seq: 2683, time: 2014-06-09 16:22:11.688978, event: done, op: pg_info(1 pgs e243:2.57a) 2014-06-09 16:22:11.688983 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688983 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688990 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.688991 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.688991 7f869af78700 20 osd.26 243 _dispatch 0x240e8fc0 pg_info(1 pgs e243:2.620) v4 2014-06-09 16:22:11.688994 7f869af78700 5 -- op tracker -- , seq: 2684, time: 2014-06-09 16:22:11.474032, event: header_read, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.688997 7f869af78700 5 -- op tracker -- , seq: 2684, time: 2014-06-09 16:22:11.474032, event: throttled, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689001 7f869af78700 5 -- op tracker -- , seq: 2684, time: 2014-06-09 16:22:11.474047, event: all_read, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689004 7f869af78700 5 -- op tracker -- , seq: 2684, time: 2014-06-09 16:22:11.688990, event: dispatched, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689008 7f869af78700 5 -- op tracker -- , seq: 2684, time: 2014-06-09 16:22:11.689006, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689010 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.620) v4 from osd.5 2014-06-09 16:22:11.689011 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e8fc0 2014-06-09 16:22:11.689014 7f869af78700 5 -- op tracker -- , seq: 2684, time: 2014-06-09 16:22:11.689013, event: started, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689022 7f869af78700 5 -- op tracker -- , seq: 2684, time: 2014-06-09 16:22:11.689021, event: done, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689029 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689030 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689037 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689037 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689038 7f869af78700 20 osd.26 243 _dispatch 0x30cf8fc0 pg_info(1 pgs e243:1.591) v4 2014-06-09 16:22:11.689041 7f869af78700 5 -- op tracker -- , seq: 2685, time: 2014-06-09 16:22:11.303125, event: header_read, op: pg_info(1 pgs e243:1.591) 2014-06-09 16:22:11.689045 7f869af78700 5 -- op tracker -- , seq: 2685, time: 2014-06-09 16:22:11.303125, event: throttled, op: pg_info(1 pgs e243:1.591) 2014-06-09 16:22:11.689048 7f869af78700 5 -- op tracker -- , seq: 2685, time: 2014-06-09 16:22:11.303131, event: all_read, op: pg_info(1 pgs e243:1.591) 2014-06-09 16:22:11.689051 7f869af78700 5 -- op tracker -- , seq: 2685, time: 2014-06-09 16:22:11.689037, event: dispatched, op: pg_info(1 pgs e243:1.591) 2014-06-09 16:22:11.689058 7f869af78700 5 -- op tracker -- , seq: 2685, time: 2014-06-09 16:22:11.689057, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.591) 2014-06-09 16:22:11.689060 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.591) v4 from osd.2 2014-06-09 16:22:11.689062 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf8fc0 2014-06-09 16:22:11.689064 7f869af78700 5 -- op tracker -- , seq: 2685, time: 2014-06-09 16:22:11.689063, event: started, op: pg_info(1 pgs e243:1.591) 2014-06-09 16:22:11.689072 7f869af78700 5 -- op tracker -- , seq: 2685, time: 2014-06-09 16:22:11.689071, event: done, op: pg_info(1 pgs e243:1.591) 2014-06-09 16:22:11.689076 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689077 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689084 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689084 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689085 7f869af78700 20 osd.26 243 _dispatch 0x310d68c0 pg_info(1 pgs e243:0.f1) v4 2014-06-09 16:22:11.689088 7f869af78700 5 -- op tracker -- , seq: 2686, time: 2014-06-09 16:22:11.422875, event: header_read, op: pg_info(1 pgs e243:0.f1) 2014-06-09 16:22:11.689091 7f869af78700 5 -- op tracker -- , seq: 2686, time: 2014-06-09 16:22:11.422875, event: throttled, op: pg_info(1 pgs e243:0.f1) 2014-06-09 16:22:11.689095 7f869af78700 5 -- op tracker -- , seq: 2686, time: 2014-06-09 16:22:11.422885, event: all_read, op: pg_info(1 pgs e243:0.f1) 2014-06-09 16:22:11.689098 7f869af78700 5 -- op tracker -- , seq: 2686, time: 2014-06-09 16:22:11.689084, event: dispatched, op: pg_info(1 pgs e243:0.f1) 2014-06-09 16:22:11.689103 7f869af78700 5 -- op tracker -- , seq: 2686, time: 2014-06-09 16:22:11.689102, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.f1) 2014-06-09 16:22:11.689106 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.f1) v4 from osd.4 2014-06-09 16:22:11.689107 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d68c0 2014-06-09 16:22:11.689110 7f869af78700 5 -- op tracker -- , seq: 2686, time: 2014-06-09 16:22:11.689109, event: started, op: pg_info(1 pgs e243:0.f1) 2014-06-09 16:22:11.689118 7f869af78700 5 -- op tracker -- , seq: 2686, time: 2014-06-09 16:22:11.689117, event: done, op: pg_info(1 pgs e243:0.f1) 2014-06-09 16:22:11.689122 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689123 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689129 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689130 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689130 7f869af78700 20 osd.26 243 _dispatch 0x2176d340 pg_info(1 pgs e243:2.113) v4 2014-06-09 16:22:11.689133 7f869af78700 5 -- op tracker -- , seq: 2687, time: 2014-06-09 16:22:11.573304, event: header_read, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.689136 7f869af78700 5 -- op tracker -- , seq: 2687, time: 2014-06-09 16:22:11.573304, event: throttled, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.689140 7f869af78700 5 -- op tracker -- , seq: 2687, time: 2014-06-09 16:22:11.573318, event: all_read, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.689143 7f869af78700 5 -- op tracker -- , seq: 2687, time: 2014-06-09 16:22:11.689129, event: dispatched, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.689153 7f869af78700 5 -- op tracker -- , seq: 2687, time: 2014-06-09 16:22:11.689151, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.689159 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.113) v4 from osd.7 2014-06-09 16:22:11.689161 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176d340 2014-06-09 16:22:11.689164 7f869af78700 5 -- op tracker -- , seq: 2687, time: 2014-06-09 16:22:11.689163, event: started, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.689174 7f869af78700 5 -- op tracker -- , seq: 2687, time: 2014-06-09 16:22:11.689173, event: done, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.689178 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689179 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689186 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689187 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689187 7f869af78700 20 osd.26 243 _dispatch 0x2f030380 pg_info(1 pgs e243:3.359) v4 2014-06-09 16:22:11.689190 7f869af78700 5 -- op tracker -- , seq: 2688, time: 2014-06-09 16:22:11.449324, event: header_read, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.689194 7f869af78700 5 -- op tracker -- , seq: 2688, time: 2014-06-09 16:22:11.449325, event: throttled, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.689197 7f869af78700 5 -- op tracker -- , seq: 2688, time: 2014-06-09 16:22:11.449336, event: all_read, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.689200 7f869af78700 5 -- op tracker -- , seq: 2688, time: 2014-06-09 16:22:11.689186, event: dispatched, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.689207 7f869af78700 5 -- op tracker -- , seq: 2688, time: 2014-06-09 16:22:11.689206, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.689210 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.359) v4 from osd.3 2014-06-09 16:22:11.689211 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f030380 2014-06-09 16:22:11.689214 7f869af78700 5 -- op tracker -- , seq: 2688, time: 2014-06-09 16:22:11.689213, event: started, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.689222 7f869af78700 5 -- op tracker -- , seq: 2688, time: 2014-06-09 16:22:11.689221, event: done, op: pg_info(1 pgs e243:3.359) 2014-06-09 16:22:11.689226 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689227 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689234 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689235 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689235 7f869af78700 20 osd.26 243 _dispatch 0x13be48c0 pg_info(1 pgs e243:0.dd) v4 2014-06-09 16:22:11.689238 7f869af78700 5 -- op tracker -- , seq: 2689, time: 2014-06-09 16:22:11.545665, event: header_read, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.689243 7f869af78700 5 -- op tracker -- , seq: 2689, time: 2014-06-09 16:22:11.545667, event: throttled, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.689247 7f869af78700 5 -- op tracker -- , seq: 2689, time: 2014-06-09 16:22:11.545683, event: all_read, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.689250 7f869af78700 5 -- op tracker -- , seq: 2689, time: 2014-06-09 16:22:11.689234, event: dispatched, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.689254 7f869af78700 5 -- op tracker -- , seq: 2689, time: 2014-06-09 16:22:11.689253, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.689256 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.dd) v4 from osd.1 2014-06-09 16:22:11.689258 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x13be48c0 2014-06-09 16:22:11.689260 7f869af78700 5 -- op tracker -- , seq: 2689, time: 2014-06-09 16:22:11.689259, event: started, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.689269 7f869af78700 5 -- op tracker -- , seq: 2689, time: 2014-06-09 16:22:11.689268, event: done, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.689272 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689273 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689280 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689280 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689281 7f869af78700 20 osd.26 243 _dispatch 0x310da1c0 pg_info(1 pgs e243:2.501) v4 2014-06-09 16:22:11.689284 7f869af78700 5 -- op tracker -- , seq: 2690, time: 2014-06-09 16:22:11.574523, event: header_read, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.689287 7f869af78700 5 -- op tracker -- , seq: 2690, time: 2014-06-09 16:22:11.574523, event: throttled, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.689291 7f869af78700 5 -- op tracker -- , seq: 2690, time: 2014-06-09 16:22:11.574536, event: all_read, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.689294 7f869af78700 5 -- op tracker -- , seq: 2690, time: 2014-06-09 16:22:11.689279, event: dispatched, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.689297 7f869af78700 5 -- op tracker -- , seq: 2690, time: 2014-06-09 16:22:11.689296, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.689299 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.501) v4 from osd.9 2014-06-09 16:22:11.689301 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310da1c0 2014-06-09 16:22:11.689303 7f869af78700 5 -- op tracker -- , seq: 2690, time: 2014-06-09 16:22:11.689302, event: started, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.689314 7f869af78700 5 -- op tracker -- , seq: 2690, time: 2014-06-09 16:22:11.689313, event: done, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.689318 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689319 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689325 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689326 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689326 7f869af78700 20 osd.26 243 _dispatch 0x31156700 pg_info(1 pgs e243:2.65) v4 2014-06-09 16:22:11.689329 7f869af78700 5 -- op tracker -- , seq: 2691, time: 2014-06-09 16:22:11.434602, event: header_read, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.689333 7f869af78700 5 -- op tracker -- , seq: 2691, time: 2014-06-09 16:22:11.434602, event: throttled, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.689336 7f869af78700 5 -- op tracker -- , seq: 2691, time: 2014-06-09 16:22:11.434612, event: all_read, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.689343 7f869af78700 5 -- op tracker -- , seq: 2691, time: 2014-06-09 16:22:11.689325, event: dispatched, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.689349 7f869af78700 5 -- op tracker -- , seq: 2691, time: 2014-06-09 16:22:11.689348, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.689352 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.65) v4 from osd.8 2014-06-09 16:22:11.689353 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31156700 2014-06-09 16:22:11.689356 7f869af78700 5 -- op tracker -- , seq: 2691, time: 2014-06-09 16:22:11.689355, event: started, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.689364 7f869af78700 5 -- op tracker -- , seq: 2691, time: 2014-06-09 16:22:11.689363, event: done, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.689368 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689369 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689376 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689377 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689377 7f869af78700 20 osd.26 243 _dispatch 0x31162fc0 pg_info(1 pgs e243:3.491) v4 2014-06-09 16:22:11.689380 7f869af78700 5 -- op tracker -- , seq: 2692, time: 2014-06-09 16:22:11.423836, event: header_read, op: pg_info(1 pgs e243:3.491) 2014-06-09 16:22:11.689384 7f869af78700 5 -- op tracker -- , seq: 2692, time: 2014-06-09 16:22:11.423836, event: throttled, op: pg_info(1 pgs e243:3.491) 2014-06-09 16:22:11.689387 7f869af78700 5 -- op tracker -- , seq: 2692, time: 2014-06-09 16:22:11.423847, event: all_read, op: pg_info(1 pgs e243:3.491) 2014-06-09 16:22:11.689394 7f869af78700 5 -- op tracker -- , seq: 2692, time: 2014-06-09 16:22:11.689376, event: dispatched, op: pg_info(1 pgs e243:3.491) 2014-06-09 16:22:11.689398 7f869af78700 5 -- op tracker -- , seq: 2692, time: 2014-06-09 16:22:11.689397, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.491) 2014-06-09 16:22:11.689402 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.491) v4 from osd.6 2014-06-09 16:22:11.689403 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31162fc0 2014-06-09 16:22:11.689406 7f869af78700 5 -- op tracker -- , seq: 2692, time: 2014-06-09 16:22:11.689405, event: started, op: pg_info(1 pgs e243:3.491) 2014-06-09 16:22:11.689416 7f869af78700 5 -- op tracker -- , seq: 2692, time: 2014-06-09 16:22:11.689415, event: done, op: pg_info(1 pgs e243:3.491) 2014-06-09 16:22:11.689420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689421 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689427 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689428 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689429 7f869af78700 20 osd.26 243 _dispatch 0x210b16c0 pg_info(1 pgs e243:2.e6) v4 2014-06-09 16:22:11.689431 7f869af78700 5 -- op tracker -- , seq: 2693, time: 2014-06-09 16:22:11.565922, event: header_read, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.689435 7f869af78700 5 -- op tracker -- , seq: 2693, time: 2014-06-09 16:22:11.565922, event: throttled, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.689438 7f869af78700 5 -- op tracker -- , seq: 2693, time: 2014-06-09 16:22:11.565933, event: all_read, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.689445 7f869af78700 5 -- op tracker -- , seq: 2693, time: 2014-06-09 16:22:11.689427, event: dispatched, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.689449 7f869af78700 5 -- op tracker -- , seq: 2693, time: 2014-06-09 16:22:11.689448, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.689454 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.e6) v4 from osd.0 2014-06-09 16:22:11.689455 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x210b16c0 2014-06-09 16:22:11.689458 7f869af78700 5 -- op tracker -- , seq: 2693, time: 2014-06-09 16:22:11.689457, event: started, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.689466 7f869af78700 5 -- op tracker -- , seq: 2693, time: 2014-06-09 16:22:11.689465, event: done, op: pg_info(1 pgs e243:2.e6) 2014-06-09 16:22:11.689470 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689471 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689478 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689478 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689479 7f869af78700 20 osd.26 243 _dispatch 0x30daf340 pg_info(1 pgs e243:2.620) v4 2014-06-09 16:22:11.689482 7f869af78700 5 -- op tracker -- , seq: 2694, time: 2014-06-09 16:22:11.477746, event: header_read, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689487 7f869af78700 5 -- op tracker -- , seq: 2694, time: 2014-06-09 16:22:11.477747, event: throttled, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689490 7f869af78700 5 -- op tracker -- , seq: 2694, time: 2014-06-09 16:22:11.477767, event: all_read, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689497 7f869af78700 5 -- op tracker -- , seq: 2694, time: 2014-06-09 16:22:11.689478, event: dispatched, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689501 7f869af78700 5 -- op tracker -- , seq: 2694, time: 2014-06-09 16:22:11.689500, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689505 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.620) v4 from osd.5 2014-06-09 16:22:11.689506 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daf340 2014-06-09 16:22:11.689509 7f869af78700 5 -- op tracker -- , seq: 2694, time: 2014-06-09 16:22:11.689508, event: started, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689518 7f869af78700 5 -- op tracker -- , seq: 2694, time: 2014-06-09 16:22:11.689517, event: done, op: pg_info(1 pgs e243:2.620) 2014-06-09 16:22:11.689522 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.689523 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.689530 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.690798 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.691268 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.690831 7f86a5e11700 10 _do_op 0x30d96ea0 seq 3358 r = 0, finisher 0xbd79ca0 0x309ea5c0 2014-06-09 16:22:11.691273 7f869af78700 20 osd.26 243 _dispatch 0x30cf8e00 pg_info(1 pgs e243:1.4a) v4 2014-06-09 16:22:11.691282 7f86a5e11700 10 _finish_op 0x30d96ea0 seq 3358 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.691286 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189aaa80] 2014-06-09 16:22:11.691280 7f869af78700 5 -- op tracker -- , seq: 2695, time: 2014-06-09 16:22:11.303138, event: header_read, op: pg_info(1 pgs e243:1.4a) 2014-06-09 16:22:11.691294 7f869af78700 5 -- op tracker -- , seq: 2695, time: 2014-06-09 16:22:11.303138, event: throttled, op: pg_info(1 pgs e243:1.4a) 2014-06-09 16:22:11.691297 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3356.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.691303 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.691302 7f869af78700 5 -- op tracker -- , seq: 2695, time: 2014-06-09 16:22:11.303145, event: all_read, op: pg_info(1 pgs e243:1.4a) 2014-06-09 16:22:11.691306 7f8689755700 10 op_submit_start 3407 2014-06-09 16:22:11.691308 7f8689755700 5 queue_transactions (trailing journal) 3407 2014-06-09 16:22:11.691309 7f868b759700 2 waiting 51 > 50 ops || 307809 > 104857600 2014-06-09 16:22:11.691310 7f8689755700 5 queue_op 0x10038480 seq 3407 osr(3.52b 0x3ed1640) 1335 bytes (queue has 50 ops and 305800 bytes) 2014-06-09 16:22:11.691310 7f869af78700 5 -- op tracker -- , seq: 2695, time: 2014-06-09 16:22:11.689529, event: dispatched, op: pg_info(1 pgs e243:1.4a) 2014-06-09 16:22:11.691317 7f8690f64700 2 waiting 51 > 50 ops || 307801 > 104857600 2014-06-09 16:22:11.691319 7f869af78700 5 -- op tracker -- , seq: 2695, time: 2014-06-09 16:22:11.691317, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4a) 2014-06-09 16:22:11.691325 7f868c75b700 2 waiting 51 > 50 ops || 307737 > 104857600 2014-06-09 16:22:11.691325 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.691325 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4a) v4 from osd.2 2014-06-09 16:22:11.691328 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf8e00 2014-06-09 16:22:11.691333 7f8692767700 2 waiting 51 > 50 ops || 307683 > 104857600 2014-06-09 16:22:11.691333 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd79ca0] 2014-06-09 16:22:11.691334 7f869af78700 5 -- op tracker -- , seq: 2695, time: 2014-06-09 16:22:11.691332, event: started, op: pg_info(1 pgs e243:1.4a) 2014-06-09 16:22:11.691329 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89efc0 (2 active) 2014-06-09 16:22:11.691336 7f86a5610700 10 _do_op 0x2fcae1e0 seq 3356 r = 0, finisher 0x189abde0 0 2014-06-09 16:22:11.691340 7f868cf5c700 2 waiting 51 > 50 ops || 307683 > 104857600 2014-06-09 16:22:11.691341 7f86a5610700 10 _finish_op 0x2fcae1e0 seq 3356 osr(3.76f 0x3ed1e20)/0x3ed1e20 2014-06-09 16:22:11.691342 7f86a5e11700 5 _do_op 0x2fc8ca20 seq 3359 osr(0.48b 0x2bd7450)/0x2bd7450 start 2014-06-09 16:22:11.691346 7f86a5e11700 10 _do_transaction on 0x3e635c0 2014-06-09 16:22:11.691356 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a480 (1 active) 2014-06-09 16:22:11.691358 7f8691f66700 2 waiting 51 > 50 ops || 308263 > 104857600 2014-06-09 16:22:11.691360 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f680 (2 active) 2014-06-09 16:22:11.691364 7f868f761700 2 waiting 51 > 50 ops || 307715 > 104857600 2014-06-09 16:22:11.691364 7f86a5610700 5 _do_op 0x308cb2c0 seq 3360 osr(3.27 0x3ed0340)/0x3ed0340 start 2014-06-09 16:22:11.691367 7f86a5610700 10 _do_transaction on 0xbebc6c0 2014-06-09 16:22:11.691366 7f8689755700 10 op_submit_finish 3407 2014-06-09 16:22:11.691362 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.691370 7f868af58700 2 waiting 51 > 50 ops || 308377 > 104857600 2014-06-09 16:22:11.691372 7f868b759700 10 op_submit_start 3408 2014-06-09 16:22:11.691374 7f868b759700 5 queue_transactions (trailing journal) 3408 2014-06-09 16:22:11.691374 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.691376 7f868a757700 2 waiting 51 > 50 ops || 308381 > 104857600 2014-06-09 16:22:11.691375 7f868b759700 5 queue_op 0x308ca8a0 seq 3408 osr(0.6b6 0x2bd7fb0) 2009 bytes (queue has 50 ops and 306380 bytes) 2014-06-09 16:22:11.691378 7f868b759700 10 op_submit_finish 3408 2014-06-09 16:22:11.691364 7f869af78700 5 -- op tracker -- , seq: 2695, time: 2014-06-09 16:22:11.691353, event: done, op: pg_info(1 pgs e243:1.4a) 2014-06-09 16:22:11.691383 7f868ef60700 2 waiting 51 > 50 ops || 308540 > 104857600 2014-06-09 16:22:11.691376 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.4c0( v 214'586 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.691388 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691390 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691390 7f8691765700 2 waiting 51 > 50 ops || 824760 > 104857600 2014-06-09 16:22:11.691383 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.691393 7f8689755700 7 osd.26 pg_epoch: 243 pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.7 2014-06-09 16:22:11.691403 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691405 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691406 7f869af78700 20 osd.26 243 _dispatch 0x310d6700 pg_info(1 pgs e243:2.587) v4 2014-06-09 16:22:11.691399 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.1 3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) 2014-06-09 16:22:11.691400 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.7 info 3.4c0( v 214'586 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242), identical to ours 2014-06-09 16:22:11.691413 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.691419 7f868df5e700 2 waiting 51 > 50 ops || 308263 > 104857600 2014-06-09 16:22:11.691414 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.407614 7 0.000179 2014-06-09 16:22:11.691419 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.0( v 213'622 (0'0,213'622] local-les=210 n=121 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.691411 7f869af78700 5 -- op tracker -- , seq: 2696, time: 2014-06-09 16:22:11.422890, event: header_read, op: pg_info(1 pgs e243:2.587) 2014-06-09 16:22:11.691428 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.691435 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.691433 7f869af78700 5 -- op tracker -- , seq: 2696, time: 2014-06-09 16:22:11.422890, event: throttled, op: pg_info(1 pgs e243:2.587) 2014-06-09 16:22:11.691431 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.691437 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.691439 7f869af78700 5 -- op tracker -- , seq: 2696, time: 2014-06-09 16:22:11.422901, event: all_read, op: pg_info(1 pgs e243:2.587) 2014-06-09 16:22:11.691438 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:3.6( v 213'682 (0'0,213'682] local-les=210 n=136 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.691445 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.691452 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.691444 7f868b759700 5 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.691445 7f869af78700 5 -- op tracker -- , seq: 2696, time: 2014-06-09 16:22:11.691403, event: dispatched, op: pg_info(1 pgs e243:2.587) 2014-06-09 16:22:11.691456 7f86a5610700 15 _omap_rmkeyrange meta/516802bb/pglog_3.27/0//-1 [0000000213.00000000000000000666,4294967295.18446744073709551615] 2014-06-09 16:22:11.691454 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.691459 7f86a5610700 15 get_omap_iterator meta/516802bb/pglog_3.27/0//-1 2014-06-09 16:22:11.691460 7f869af78700 5 -- op tracker -- , seq: 2696, time: 2014-06-09 16:22:11.691458, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.587) 2014-06-09 16:22:11.691465 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.587) v4 from osd.4 2014-06-09 16:22:11.691458 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=191 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.691461 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.691467 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d6700 2014-06-09 16:22:11.691468 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.691471 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.691472 7f869af78700 5 -- op tracker -- , seq: 2696, time: 2014-06-09 16:22:11.691470, event: started, op: pg_info(1 pgs e243:2.587) 2014-06-09 16:22:11.691469 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'589 -> 213'589 2014-06-09 16:22:11.691474 7f8689755700 5 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 0.774449 7 0.000180 2014-06-09 16:22:11.691481 7f8689755700 5 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.691487 7f8689755700 5 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.691489 7f869af78700 5 -- op tracker -- , seq: 2696, time: 2014-06-09 16:22:11.691487, event: done, op: pg_info(1 pgs e243:2.587) 2014-06-09 16:22:11.691493 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.691497 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691498 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691498 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.691502 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.691511 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691512 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691513 7f869af78700 20 osd.26 243 _dispatch 0x30dc2000 pg_info(1 pgs e243:1.589) v4 2014-06-09 16:22:11.691510 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.691518 7f869af78700 5 -- op tracker -- , seq: 2697, time: 2014-06-09 16:22:11.573932, event: header_read, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.691524 7f869af78700 5 -- op tracker -- , seq: 2697, time: 2014-06-09 16:22:11.573932, event: throttled, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.691539 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.691542 7f8689755700 5 queue_transactions existing osr(1.4bb 0x384d970)/0x384d970 2014-06-09 16:22:11.691540 7f869af78700 5 -- op tracker -- , seq: 2697, time: 2014-06-09 16:22:11.573945, event: all_read, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.691544 7f8689755700 2 waiting 51 > 50 ops || 308479 > 104857600 2014-06-09 16:22:11.691546 7f869af78700 5 -- op tracker -- , seq: 2697, time: 2014-06-09 16:22:11.691510, event: dispatched, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.691555 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.691554 7f86a5610700 15 _omap_rmkeys meta/516802bb/pglog_3.27/0//-1 2014-06-09 16:22:11.691559 7f869af78700 5 -- op tracker -- , seq: 2697, time: 2014-06-09 16:22:11.691557, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.691561 7f868b759700 5 queue_transactions existing osr(3.f7 0x3ed0860)/0x3ed0860 2014-06-09 16:22:11.691563 7f868b759700 2 waiting 51 > 50 ops || 307806 > 104857600 2014-06-09 16:22:11.691563 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.589) v4 from osd.7 2014-06-09 16:22:11.691566 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc2000 2014-06-09 16:22:11.691567 7f868bf5a700 2 waiting 51 > 50 ops || 307809 > 104857600 2014-06-09 16:22:11.691573 7f868e75f700 2 waiting 51 > 50 ops || 307433 > 104857600 2014-06-09 16:22:11.691578 7f8690f64700 2 waiting 51 > 50 ops || 308381 > 104857600 2014-06-09 16:22:11.691570 7f869af78700 5 -- op tracker -- , seq: 2697, time: 2014-06-09 16:22:11.691568, event: started, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.691585 7f868c75b700 2 waiting 51 > 50 ops || 308317 > 104857600 2014-06-09 16:22:11.691590 7f86a5610700 10 lookup_strip_header done cid meta oid 516802bb/pglog_3.27/0//-1 2014-06-09 16:22:11.691592 7f8692767700 2 waiting 51 > 50 ops || 308263 > 104857600 2014-06-09 16:22:11.691593 7f869af78700 5 -- op tracker -- , seq: 2697, time: 2014-06-09 16:22:11.691591, event: done, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.691598 7f868cf5c700 2 waiting 51 > 50 ops || 308263 > 104857600 2014-06-09 16:22:11.691600 7f86a5610700 10 _omap_rmkeys meta/516802bb/pglog_3.27/0//-1 r = 0 2014-06-09 16:22:11.691601 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691603 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691605 7f8692f68700 2 waiting 51 > 50 ops || 307715 > 104857600 2014-06-09 16:22:11.691605 7f86a5610700 15 _omap_rmkeys meta/516802bb/pglog_3.27/0//-1 2014-06-09 16:22:11.691619 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691621 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691622 7f869af78700 20 osd.26 243 _dispatch 0x1a039500 pg_info(1 pgs e243:3.3a9) v4 2014-06-09 16:22:11.691611 7f86a5610700 10 _omap_rmkeys meta/516802bb/pglog_3.27/0//-1 r = 0 2014-06-09 16:22:11.691626 7f869af78700 5 -- op tracker -- , seq: 2698, time: 2014-06-09 16:22:11.479450, event: header_read, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.691633 7f86a5610700 15 _omap_setkeys meta/516802bb/pglog_3.27/0//-1 2014-06-09 16:22:11.691633 7f869af78700 5 -- op tracker -- , seq: 2698, time: 2014-06-09 16:22:11.479450, event: throttled, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.691639 7f869af78700 5 -- op tracker -- , seq: 2698, time: 2014-06-09 16:22:11.479461, event: all_read, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.691645 7f869af78700 5 -- op tracker -- , seq: 2698, time: 2014-06-09 16:22:11.691618, event: dispatched, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.691651 7f869af78700 5 -- op tracker -- , seq: 2698, time: 2014-06-09 16:22:11.691649, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.691652 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3360.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.691656 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a9) v4 from osd.3 2014-06-09 16:22:11.691659 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a039500 2014-06-09 16:22:11.691657 7f86a5610700 10 cid: oid: 516802bb/pglog_3.27/0//-1 skipping op, spos 3360.0.0 <= header.spos 149667.0.0 2014-06-09 16:22:11.691663 7f869af78700 5 -- op tracker -- , seq: 2698, time: 2014-06-09 16:22:11.691662, event: started, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.691674 7f869af78700 5 -- op tracker -- , seq: 2698, time: 2014-06-09 16:22:11.691673, event: done, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.691684 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691685 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691693 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691694 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691694 7f869af78700 20 osd.26 243 _dispatch 0x1ac94380 pg_info(1 pgs e243:0.dd) v4 2014-06-09 16:22:11.691697 7f869af78700 5 -- op tracker -- , seq: 2699, time: 2014-06-09 16:22:11.546104, event: header_read, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.691701 7f869af78700 5 -- op tracker -- , seq: 2699, time: 2014-06-09 16:22:11.546105, event: throttled, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.691704 7f869af78700 5 -- op tracker -- , seq: 2699, time: 2014-06-09 16:22:11.546120, event: all_read, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.691708 7f869af78700 5 -- op tracker -- , seq: 2699, time: 2014-06-09 16:22:11.691693, event: dispatched, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.691711 7f869af78700 5 -- op tracker -- , seq: 2699, time: 2014-06-09 16:22:11.691710, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.691714 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.dd) v4 from osd.1 2014-06-09 16:22:11.691715 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac94380 2014-06-09 16:22:11.691717 7f869af78700 5 -- op tracker -- , seq: 2699, time: 2014-06-09 16:22:11.691717, event: started, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.691721 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.691727 7f869af78700 5 -- op tracker -- , seq: 2699, time: 2014-06-09 16:22:11.691726, event: done, op: pg_info(1 pgs e243:0.dd) 2014-06-09 16:22:11.691731 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691732 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691740 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.691740 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.691741 7f869af78700 20 osd.26 243 _dispatch 0x310da000 pg_info(1 pgs e243:2.27a) v4 2014-06-09 16:22:11.691743 7f869af78700 5 -- op tracker -- , seq: 2700, time: 2014-06-09 16:22:11.576061, event: header_read, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.691754 7f869af78700 5 -- op tracker -- , seq: 2700, time: 2014-06-09 16:22:11.576062, event: throttled, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.691760 7f869af78700 5 -- op tracker -- , seq: 2700, time: 2014-06-09 16:22:11.576077, event: all_read, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.691764 7f869af78700 5 -- op tracker -- , seq: 2700, time: 2014-06-09 16:22:11.691739, event: dispatched, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.691767 7f869af78700 5 -- op tracker -- , seq: 2700, time: 2014-06-09 16:22:11.691766, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.691776 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.27a) v4 from osd.9 2014-06-09 16:22:11.691779 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310da000 2014-06-09 16:22:11.691778 7f86a5610700 10 _do_op 0x308cb2c0 seq 3360 r = 0, finisher 0xc5a70e0 0 2014-06-09 16:22:11.691783 7f86a5610700 10 _finish_op 0x308cb2c0 seq 3360 osr(3.27 0x3ed0340)/0x3ed0340 2014-06-09 16:22:11.691784 7f869af78700 5 -- op tracker -- , seq: 2700, time: 2014-06-09 16:22:11.691782, event: started, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.691790 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f680 (1 active) 2014-06-09 16:22:11.691793 7f8691f66700 10 op_submit_start 3409 2014-06-09 16:22:11.691793 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287c20 (2 active) 2014-06-09 16:22:11.691794 7f8691f66700 5 queue_transactions (trailing journal) 3409 2014-06-09 16:22:11.691794 7f868f761700 2 waiting 51 > 50 ops || 307289 > 104857600 2014-06-09 16:22:11.691795 7f86a5610700 5 _do_op 0x2fc8cd20 seq 3361 osr(0.529 0x2bd7930)/0x2bd7930 start 2014-06-09 16:22:11.691796 7f8691f66700 5 queue_op 0x308cb800 seq 3409 osr(0.5f5 0x2bd7990) 1883 bytes (queue has 50 ops and 305954 bytes) 2014-06-09 16:22:11.691798 7f86a5610700 10 _do_transaction on 0xc1609c0 2014-06-09 16:22:11.691799 7f8691f66700 10 op_submit_finish 3409 2014-06-09 16:22:11.691804 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.691815 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (19 active) 2014-06-09 16:22:11.691821 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.691842 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3359.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.691844 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (20 active) 2014-06-09 16:22:11.691846 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.691848 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 0 2014-06-09 16:22:11.691868 7f86a5e11700 10 _do_op 0x2fc8ca20 seq 3359 r = 0, finisher 0xbd5a120 0 2014-06-09 16:22:11.691872 7f86a5e11700 10 _finish_op 0x2fc8ca20 seq 3359 osr(0.48b 0x2bd7450)/0x2bd7450 2014-06-09 16:22:11.691863 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.0: 3.72c( v 213'555 (0'0,213'555] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) log((0'0,213'555], crt=0'0) missing(0) 2014-06-09 16:22:11.691877 7f8691f66700 10 proc_replica_log for osd.0: 3.72c( v 213'555 (0'0,213'555] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) log((0'0,213'555], crt=0'0) missing(0) 2014-06-09 16:22:11.691882 7f868a757700 10 op_submit_start 3410 2014-06-09 16:22:11.691883 7f868a757700 5 queue_transactions (trailing journal) 3410 2014-06-09 16:22:11.691884 7f868ef60700 2 waiting 51 > 50 ops || 308161 > 104857600 2014-06-09 16:22:11.691885 7f868a757700 5 queue_op 0x30607920 seq 3410 osr(0.1cd 0x2bd6960) 2001 bytes (queue has 50 ops and 306001 bytes) 2014-06-09 16:22:11.691891 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89efc0 (1 active) 2014-06-09 16:22:11.691892 7f8691765700 2 waiting 51 > 50 ops || 824381 > 104857600 2014-06-09 16:22:11.691893 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc120 (2 active) 2014-06-09 16:22:11.691897 7f868a757700 10 op_submit_finish 3410 2014-06-09 16:22:11.691898 7f868df5e700 2 waiting 51 > 50 ops || 307884 > 104857600 2014-06-09 16:22:11.691902 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.691882 7f8691f66700 20 merge_log cut point (usually last shared) is 213'555 (213'554) modify 14132f2c/rbd_data.11686b8b4567.000000000000167f/head//3 by client.4552.0:33778 2014-06-09 10:18:09.384938 2014-06-09 16:22:11.691904 7f8689755700 2 waiting 51 > 50 ops || 308100 > 104857600 2014-06-09 16:22:11.691901 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:0.1fb( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.691920 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3361.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.691916 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.691924 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.691926 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.691931 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.691941 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.691951 7f86a5610700 10 _do_op 0x2fc8cd20 seq 3361 r = 0, finisher 0xbebffc0 0 2014-06-09 16:22:11.691949 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.691955 7f86a5610700 10 _finish_op 0x2fc8cd20 seq 3361 osr(0.529 0x2bd7930)/0x2bd7930 2014-06-09 16:22:11.691956 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.691961 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287c20 (1 active) 2014-06-09 16:22:11.691964 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89eb40 (2 active) 2014-06-09 16:22:11.691965 7f868ef60700 10 op_submit_start 3411 2014-06-09 16:22:11.691966 7f86a5610700 5 _do_op 0xd471f80 seq 3363 osr(2.48f 0x2ed1450)/0x2ed1450 start 2014-06-09 16:22:11.691967 7f868ef60700 5 queue_transactions (trailing journal) 3411 2014-06-09 16:22:11.691968 7f86a5610700 10 _do_transaction on 0xbfff080 2014-06-09 16:22:11.691969 7f868ef60700 5 queue_op 0x288f9b00 seq 3411 osr(3.799 0x3ed1fa0) 2160 bytes (queue has 50 ops and 306207 bytes) 2014-06-09 16:22:11.691972 7f868ef60700 10 op_submit_finish 3411 2014-06-09 16:22:11.691971 7f868a757700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.691974 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.691976 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.691990 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.691991 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.692004 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.692005 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692001 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692009 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692007 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:1.3e1( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.692020 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3363.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692024 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692023 7f868ef60700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.692018 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.692027 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.692031 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.692044 7f86a5610700 10 _do_op 0xd471f80 seq 3363 r = 0, finisher 0xc5a7540 0 2014-06-09 16:22:11.692038 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.692047 7f86a5610700 10 _finish_op 0xd471f80 seq 3363 osr(2.48f 0x2ed1450)/0x2ed1450 2014-06-09 16:22:11.692042 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.692049 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89eb40 (1 active) 2014-06-09 16:22:11.692051 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6360 (2 active) 2014-06-09 16:22:11.692052 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692053 7f86a5610700 5 _do_op 0xd470960 seq 3364 osr(3.470 0x3ed1540)/0x3ed1540 start 2014-06-09 16:22:11.692055 7f86a5610700 10 _do_transaction on 0xc29afc0 2014-06-09 16:22:11.692050 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692061 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692057 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692057 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.692069 7f868a757700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.692068 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.692077 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692083 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:189 2014-06-09 16:22:11.692085 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692091 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692097 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.692096 7f868ef60700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.692101 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=206 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.692111 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:205 2014-06-09 16:22:11.692112 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=206 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.692106 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3364.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692126 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692127 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692125 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=206 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.186206 7 0.000156 2014-06-09 16:22:11.692131 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.692138 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=206 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.692148 7f86a5610700 10 _do_op 0xd470960 seq 3364 r = 0, finisher 0xc6e99a0 0 2014-06-09 16:22:11.692151 7f86a5610700 10 _finish_op 0xd470960 seq 3364 osr(3.470 0x3ed1540)/0x3ed1540 2014-06-09 16:22:11.692142 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692153 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6360 (1 active) 2014-06-09 16:22:11.692153 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692155 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5247e0 (2 active) 2014-06-09 16:22:11.692149 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=206 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.692157 7f86a5610700 5 _do_op 0x309a2660 seq 3365 osr(3.58f 0x3ed1850)/0x3ed1850 start 2014-06-09 16:22:11.692157 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.692160 7f86a5610700 10 _do_transaction on 0x2ed52c0 2014-06-09 16:22:11.692165 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692159 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=206 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.692160 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.492( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.756843 7 0.000182 2014-06-09 16:22:11.692169 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'619 -> 213'619 2014-06-09 16:22:11.692170 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.492( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.692181 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.492( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.692191 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.692197 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692199 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.692206 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692207 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692212 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3365.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692216 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692233 7f86a5610700 10 _do_op 0x309a2660 seq 3365 r = 0, finisher 0xbd78440 0 2014-06-09 16:22:11.692236 7f86a5610700 10 _finish_op 0x309a2660 seq 3365 osr(3.58f 0x3ed1850)/0x3ed1850 2014-06-09 16:22:11.692238 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5247e0 (1 active) 2014-06-09 16:22:11.692240 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652c60 (2 active) 2014-06-09 16:22:11.692242 7f86a5610700 5 _do_op 0x309a25a0 seq 3366 osr(1.2f8 0x384d0d0)/0x384d0d0 start 2014-06-09 16:22:11.692244 7f86a5610700 10 _do_transaction on 0xc2d2180 2014-06-09 16:22:11.692244 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.692249 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692249 7f868a757700 5 queue_transactions existing osr(3.6bb 0x3ed1cd0)/0x3ed1cd0 2014-06-09 16:22:11.692252 7f868a757700 10 op_submit_start 3412 2014-06-09 16:22:11.692253 7f868a757700 5 queue_transactions (trailing journal) 3412 2014-06-09 16:22:11.692253 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692254 7f868a757700 5 queue_op 0x30607b00 seq 3412 osr(3.6bb 0x3ed1cd0) 1335 bytes (queue has 48 ops and 302824 bytes) 2014-06-09 16:22:11.692256 7f868a757700 10 op_submit_finish 3412 2014-06-09 16:22:11.692258 7f868ef60700 5 queue_transactions existing osr(2.492 0x2ed1410)/0x2ed1410 2014-06-09 16:22:11.692261 7f868ef60700 10 op_submit_start 3413 2014-06-09 16:22:11.692262 7f868ef60700 5 queue_transactions (trailing journal) 3413 2014-06-09 16:22:11.692263 7f868ef60700 5 queue_op 0x2fcafce0 seq 3413 osr(2.492 0x2ed1410) 1883 bytes (queue has 49 ops and 304707 bytes) 2014-06-09 16:22:11.692266 7f868ef60700 10 op_submit_finish 3413 2014-06-09 16:22:11.692259 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.692268 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.692270 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692279 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692280 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692283 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.692279 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692291 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692287 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.193666 7 0.000199 2014-06-09 16:22:11.692294 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.692294 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3366.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692297 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692296 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.692290 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.0 now 3.72c( v 213'555 (0'0,213'555] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.692298 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.188198 7 0.000152 2014-06-09 16:22:11.692306 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.692306 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.692308 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.692315 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.692318 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.692324 7f86a5610700 10 _do_op 0x309a25a0 seq 3366 r = 0, finisher 0xbd71cc0 0 2014-06-09 16:22:11.692318 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.678104 7 0.000267 2014-06-09 16:22:11.692327 7f86a5610700 10 _finish_op 0x309a25a0 seq 3366 osr(1.2f8 0x384d0d0)/0x384d0d0 2014-06-09 16:22:11.692323 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.692329 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652c60 (1 active) 2014-06-09 16:22:11.692332 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c120 (2 active) 2014-06-09 16:22:11.692333 7f86a5610700 5 _do_op 0x309a33e0 seq 3367 osr(3.577 0x3ed18c0)/0x3ed18c0 start 2014-06-09 16:22:11.692328 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.692336 7f86a5610700 10 _do_transaction on 0xc259500 2014-06-09 16:22:11.692331 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692329 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.692341 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692337 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.692339 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.915149 0 0.000000 2014-06-09 16:22:11.692344 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692354 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.692363 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.692372 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692376 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.692380 7f868a757700 5 queue_transactions existing osr(0.2c6 0x2bd6b80)/0x2bd6b80 2014-06-09 16:22:11.692382 7f868a757700 10 op_submit_start 3414 2014-06-09 16:22:11.692375 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.692383 7f868a757700 5 queue_transactions (trailing journal) 3414 2014-06-09 16:22:11.692384 7f868a757700 5 queue_op 0x2fcae660 seq 3414 osr(0.2c6 0x2bd6b80) 1883 bytes (queue has 49 ops and 304491 bytes) 2014-06-09 16:22:11.692386 7f868a757700 10 op_submit_finish 3414 2014-06-09 16:22:11.692386 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3367.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692390 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692390 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692384 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.692395 7f868ef60700 5 queue_transactions existing osr(1.6b6 0x384dff0)/0x384dff0 2014-06-09 16:22:11.692389 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.692397 7f868ef60700 10 op_submit_start 3415 2014-06-09 16:22:11.692398 7f868ef60700 5 queue_transactions (trailing journal) 3415 2014-06-09 16:22:11.692392 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 213'558 -> 213'558 2014-06-09 16:22:11.692400 7f868ef60700 5 queue_op 0x15996ba0 seq 3415 osr(1.6b6 0x384dff0) 1954 bytes (queue has 50 ops and 306445 bytes) 2014-06-09 16:22:11.692402 7f868ef60700 10 op_submit_finish 3415 2014-06-09 16:22:11.692405 7f86a5610700 10 _do_op 0x309a33e0 seq 3367 r = 0, finisher 0xbe31980 0 2014-06-09 16:22:11.692408 7f86a5610700 10 _finish_op 0x309a33e0 seq 3367 osr(3.577 0x3ed18c0)/0x3ed18c0 2014-06-09 16:22:11.692399 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692410 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c120 (1 active) 2014-06-09 16:22:11.692411 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692412 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922900 (2 active) 2014-06-09 16:22:11.692414 7f86a5610700 5 _do_op 0xd4717a0 seq 3368 osr(2.4a5 0x2ed17f0)/0x2ed17f0 start 2014-06-09 16:22:11.692414 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.692416 7f86a5610700 10 _do_transaction on 0xbf36600 2014-06-09 16:22:11.692405 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.799( v 214'665 (0'0,214'665] local-les=213 n=127 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.692421 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692417 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.582( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.052445 7 0.000200 2014-06-09 16:22:11.692422 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.692427 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.582( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.692434 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692432 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.0 3.72c( v 213'555 (0'0,213'555] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.692437 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.582( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.692439 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.4d3( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.692445 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.692451 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692451 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.692454 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.692451 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.0 sending log((213'555,213'558], crt=0'0) 2014-06-09 16:22:11.692461 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692463 7f8691f66700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.692461 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692465 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3368.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692469 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692465 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 9 notify: (query_epoch:242, epoch_sent:243, info:3.173( v 213'624 (0'0,213'624] local-les=213 n=119 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.692478 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.692477 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.0 3.72c( v 213'558 lc 213'555 (0'0,213'558] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.692489 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692493 7f86a5610700 10 _do_op 0xd4717a0 seq 3368 r = 0, finisher 0x3062bde0 0 2014-06-09 16:22:11.692496 7f86a5610700 10 _finish_op 0xd4717a0 seq 3368 osr(2.4a5 0x2ed17f0)/0x2ed17f0 2014-06-09 16:22:11.692491 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.0 has 1 missing 2014-06-09 16:22:11.692498 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922900 (1 active) 2014-06-09 16:22:11.692494 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.692502 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922480 (2 active) 2014-06-09 16:22:11.692504 7f86a5610700 5 _do_op 0x309a3da0 seq 3369 osr(2.1e3 0x2ed0810)/0x2ed0810 start 2014-06-09 16:22:11.692505 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.692506 7f86a5610700 10 _do_transaction on 0xc320300 2014-06-09 16:22:11.692509 7f868a757700 5 queue_transactions existing osr(1.582 0x384da30)/0x384da30 2014-06-09 16:22:11.692502 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 14132f2c/rbd_data.11686b8b4567.000000000000167f/head//3 213'558 is on osd.26 2014-06-09 16:22:11.692504 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.692512 7f868a757700 10 op_submit_start 3416 2014-06-09 16:22:11.692511 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692513 7f868a757700 5 queue_transactions (trailing journal) 3416 2014-06-09 16:22:11.692514 7f868a757700 5 queue_op 0x2fcaefc0 seq 3416 osr(1.582 0x384da30) 1997 bytes (queue has 49 ops and 305224 bytes) 2014-06-09 16:22:11.692516 7f868a757700 10 op_submit_finish 3416 2014-06-09 16:22:11.692512 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692513 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {14132f2c/rbd_data.11686b8b4567.000000000000167f/head//3=213'558(213'555)} 2014-06-09 16:22:11.692520 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692519 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 7 notify: (query_epoch:242, epoch_sent:243, info:1.4bc( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.692524 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 0 as a source 2014-06-09 16:22:11.692532 7f868ef60700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.692531 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.692532 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 14132f2c/rbd_data.11686b8b4567.000000000000167f/head//3 213'558 also missing on osd.0 2014-06-09 16:22:11.692540 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.692542 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692541 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {14132f2c/rbd_data.11686b8b4567.000000000000167f/head//3=213'558(213'555)} 2014-06-09 16:22:11.692544 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.692556 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (19 active) 2014-06-09 16:22:11.692551 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.692557 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3369.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692562 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692563 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.692559 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.692572 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692574 7f868b759700 10 op_submit_start 3417 2014-06-09 16:22:11.692575 7f868b759700 5 queue_transactions (trailing journal) 3417 2014-06-09 16:22:11.692569 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 0 2014-06-09 16:22:11.692576 7f868b759700 5 queue_op 0x308ca840 seq 3417 osr(3.f7 0x3ed0860) 1426 bytes (queue has 50 ops and 306650 bytes) 2014-06-09 16:22:11.692578 7f868b759700 10 op_submit_finish 3417 2014-06-09 16:22:11.692581 7f86a5610700 10 _do_op 0x309a3da0 seq 3369 r = 0, finisher 0xbd6d6e0 0 2014-06-09 16:22:11.692583 7f86a5610700 10 _finish_op 0x309a3da0 seq 3369 osr(2.1e3 0x2ed0810)/0x2ed0810 2014-06-09 16:22:11.692578 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692586 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922480 (1 active) 2014-06-09 16:22:11.692577 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.692587 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b0e0 (2 active) 2014-06-09 16:22:11.692581 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.692589 7f86a5610700 5 _do_op 0x159965a0 seq 3370 osr(1.a 0x384c270)/0x384c270 start 2014-06-09 16:22:11.692592 7f86a5610700 10 _do_transaction on 0xc3f15c0 2014-06-09 16:22:11.692588 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.692590 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.692598 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692592 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692598 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692603 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692607 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.692605 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692616 7f868ef60700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.692610 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.242029 7 0.000188 2014-06-09 16:22:11.692620 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.692628 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692630 7f868b759700 5 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.692630 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.692644 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3370.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692639 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.692647 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692648 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692654 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.692648 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.692658 7f8691f66700 5 queue_transactions existing osr(3.72c 0x3ed1ee0)/0x3ed1ee0 2014-06-09 16:22:11.692651 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.411( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.692656 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692664 7f8691f66700 10 op_submit_start 3418 2014-06-09 16:22:11.692665 7f8691f66700 5 queue_transactions (trailing journal) 3418 2014-06-09 16:22:11.692666 7f8691f66700 5 queue_op 0xc64b860 seq 3418 osr(3.72c 0x3ed1ee0) 1053 bytes (queue has 50 ops and 305702 bytes) 2014-06-09 16:22:11.692669 7f8691f66700 10 op_submit_finish 3418 2014-06-09 16:22:11.692664 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.692674 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692672 7f86a5610700 10 _do_op 0x159965a0 seq 3370 r = 0, finisher 0xbccd640 0 2014-06-09 16:22:11.692675 7f86a5610700 10 _finish_op 0x159965a0 seq 3370 osr(1.a 0x384c270)/0x384c270 2014-06-09 16:22:11.692678 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b0e0 (1 active) 2014-06-09 16:22:11.692680 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2878c0 (2 active) 2014-06-09 16:22:11.692671 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.692681 7f86a5610700 5 _do_op 0x2fc8c120 seq 3371 osr(2.624 0x2ed1c50)/0x2ed1c50 start 2014-06-09 16:22:11.692684 7f86a5610700 10 _do_transaction on 0xc31e300 2014-06-09 16:22:11.692678 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.6cd( v 214'577 (0'0,214'577] local-les=203 n=111 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.692689 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692685 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.692695 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.692697 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.692690 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.692700 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692700 7f868b759700 5 queue_transactions existing osr(2.55b 0x2ed1930)/0x2ed1930 2014-06-09 16:22:11.692703 7f868b759700 10 op_submit_start 3419 2014-06-09 16:22:11.692703 7f868b759700 5 queue_transactions (trailing journal) 3419 2014-06-09 16:22:11.692704 7f868b759700 5 queue_op 0x308ca7e0 seq 3419 osr(2.55b 0x2ed1930) 1954 bytes (queue has 50 ops and 305779 bytes) 2014-06-09 16:22:11.692699 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.692707 7f868b759700 10 op_submit_finish 3419 2014-06-09 16:22:11.692704 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.692709 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.578( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.692709 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:1.b9( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.692718 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.692719 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692717 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.692720 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.692721 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.692730 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.692725 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692733 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3371.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692737 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692732 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.692733 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692740 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.692734 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.692744 7f868ef60700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.692744 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692744 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.2a5( v 214'635 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.692753 7f86a5610700 10 _do_op 0x2fc8c120 seq 3371 r = 0, finisher 0x13965d60 0 2014-06-09 16:22:11.692755 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.692756 7f86a5610700 10 _finish_op 0x2fc8c120 seq 3371 osr(2.624 0x2ed1c50)/0x2ed1c50 2014-06-09 16:22:11.692758 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.692757 7f8691f66700 7 osd.26 pg_epoch: 243 pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.9 2014-06-09 16:22:11.692761 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.384( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.443266 7 0.000174 2014-06-09 16:22:11.692760 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.692774 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692772 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.384( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.692766 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.9 info 3.2a5( v 214'635 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242), identical to ours 2014-06-09 16:22:11.692784 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.692781 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.384( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.692785 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2878c0 (1 active) 2014-06-09 16:22:11.692790 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb0e0 (2 active) 2014-06-09 16:22:11.692778 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.6d3( v 214'632 (0'0,214'632] local-les=213 n=122 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.692792 7f86a5610700 5 _do_op 0x309a3740 seq 3372 osr(2.213 0x2ed0b70)/0x2ed0b70 start 2014-06-09 16:22:11.692795 7f86a5610700 10 _do_transaction on 0xc2a03c0 2014-06-09 16:22:11.692789 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.692788 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.692800 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692797 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.692794 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.692799 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.692806 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692805 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692806 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692814 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692811 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.692825 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.692823 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.692832 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.692832 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692839 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692840 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:184 2014-06-09 16:22:11.692849 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.692854 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.692856 7f868bf5a700 10 op_submit_start 3420 2014-06-09 16:22:11.692858 7f868bf5a700 5 queue_transactions (trailing journal) 3420 2014-06-09 16:22:11.692858 7f8690f64700 2 waiting 51 > 50 ops || 307306 > 104857600 2014-06-09 16:22:11.692859 7f868bf5a700 5 queue_op 0x308caea0 seq 3420 osr(3.5ce 0x3ed17c0) 1429 bytes (queue has 50 ops and 305305 bytes) 2014-06-09 16:22:11.692862 7f868bf5a700 10 op_submit_finish 3420 2014-06-09 16:22:11.692861 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.692866 7f868c75b700 2 waiting 51 > 50 ops || 307242 > 104857600 2014-06-09 16:22:11.692872 7f8692767700 2 waiting 51 > 50 ops || 307188 > 104857600 2014-06-09 16:22:11.692866 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=191 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.692877 7f868cf5c700 2 waiting 51 > 50 ops || 307188 > 104857600 2014-06-09 16:22:11.692883 7f8692f68700 2 waiting 51 > 50 ops || 306640 > 104857600 2014-06-09 16:22:11.692877 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:151 2014-06-09 16:22:11.692887 7f868f761700 2 waiting 51 > 50 ops || 306640 > 104857600 2014-06-09 16:22:11.692879 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=191 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.692893 7f868af58700 2 waiting 51 > 50 ops || 307302 > 104857600 2014-06-09 16:22:11.692893 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692898 7f8691765700 2 waiting 51 > 50 ops || 823685 > 104857600 2014-06-09 16:22:11.692892 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=191 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.011671 7 0.000829 2014-06-09 16:22:11.692904 7f868df5e700 2 waiting 51 > 50 ops || 307188 > 104857600 2014-06-09 16:22:11.692898 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.692905 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3372.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.692909 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.692910 7f8689755700 2 waiting 51 > 50 ops || 307404 > 104857600 2014-06-09 16:22:11.692904 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=191 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.692916 7f868e75f700 2 waiting 51 > 50 ops || 306358 > 104857600 2014-06-09 16:22:11.692910 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.692917 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=191 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.692919 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.692927 7f86a5e11700 5 _do_op 0x30d97320 seq 3362 osr(0.2a2 0x2bd6c20)/0x2bd6c20 start 2014-06-09 16:22:11.692931 7f86a5e11700 10 _do_transaction on 0xc227080 2014-06-09 16:22:11.692926 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.692927 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=191 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.692937 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.692937 7f868ef60700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.692937 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'565 -> 213'565 2014-06-09 16:22:11.692952 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:169 2014-06-09 16:22:11.692962 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.692921 7f869af78700 5 -- op tracker -- , seq: 2700, time: 2014-06-09 16:22:11.692914, event: done, op: pg_info(1 pgs e243:2.27a) 2014-06-09 16:22:11.692972 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.692984 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.692977 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.692986 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.692987 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.692999 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693000 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693001 7f869af78700 20 osd.26 243 _dispatch 0x31156540 pg_info(1 pgs e243:3.19) v4 2014-06-09 16:22:11.692997 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.693006 7f869af78700 5 -- op tracker -- , seq: 2701, time: 2014-06-09 16:22:11.434625, event: header_read, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.693013 7f869af78700 5 -- op tracker -- , seq: 2701, time: 2014-06-09 16:22:11.434625, event: throttled, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.693017 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.693021 7f868bf5a700 5 queue_transactions existing osr(3.105 0x3ed0820)/0x3ed0820 2014-06-09 16:22:11.693020 7f869af78700 5 -- op tracker -- , seq: 2701, time: 2014-06-09 16:22:11.434636, event: all_read, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.693005 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.693025 7f868bf5a700 2 waiting 51 > 50 ops || 306734 > 104857600 2014-06-09 16:22:11.693025 7f869af78700 5 -- op tracker -- , seq: 2701, time: 2014-06-09 16:22:11.692998, event: dispatched, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.693029 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.693034 7f868b759700 5 queue_transactions existing osr(1.384 0x384d5c0)/0x384d5c0 2014-06-09 16:22:11.693037 7f868b759700 2 waiting 51 > 50 ops || 307306 > 104857600 2014-06-09 16:22:11.693038 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.693044 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.693051 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.693057 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.693060 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.693061 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.693068 7f8691f66700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.693065 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.693079 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.693078 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.693093 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.693092 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.180051 7 0.000343 2014-06-09 16:22:11.693097 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.693103 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.693110 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.693117 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.693115 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.693121 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.693128 7f8691f66700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.693127 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.693138 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:204 2014-06-09 16:22:11.693137 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'596 -> 213'596 2014-06-09 16:22:11.693147 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.693149 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.693156 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.693161 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.693165 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.693171 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.693179 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:160 2014-06-09 16:22:11.693177 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.693187 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.693197 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.693207 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.693210 7f869af78700 5 -- op tracker -- , seq: 2701, time: 2014-06-09 16:22:11.693030, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.693215 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.693215 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.19) v4 from osd.8 2014-06-09 16:22:11.693217 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.693217 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31156540 2014-06-09 16:22:11.693221 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.693219 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.740077 7 0.000187 2014-06-09 16:22:11.693225 7f868ef60700 5 queue_transactions existing osr(3.408 0x3ed1270)/0x3ed1270 2014-06-09 16:22:11.693228 7f868ef60700 2 waiting 51 > 50 ops || 306734 > 104857600 2014-06-09 16:22:11.693226 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.693221 7f869af78700 5 -- op tracker -- , seq: 2701, time: 2014-06-09 16:22:11.693220, event: started, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.693233 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.693245 7f869af78700 5 -- op tracker -- , seq: 2701, time: 2014-06-09 16:22:11.693242, event: done, op: pg_info(1 pgs e243:3.19) 2014-06-09 16:22:11.693242 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.693253 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693236 7f86a5610700 10 _do_op 0x309a3740 seq 3372 r = 0, finisher 0xc2d63c0 0 2014-06-09 16:22:11.693255 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693256 7f86a5610700 10 _finish_op 0x309a3740 seq 3372 osr(2.213 0x2ed0b70)/0x2ed0b70 2014-06-09 16:22:11.693251 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.693259 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.693267 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693268 7f868c75b700 10 op_submit_start 3421 2014-06-09 16:22:11.693267 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.693269 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693270 7f868c75b700 5 queue_transactions (trailing journal) 3421 2014-06-09 16:22:11.693270 7f869af78700 20 osd.26 243 _dispatch 0x1405c8c0 pg_info(1 pgs e243:3.31f) v4 2014-06-09 16:22:11.693272 7f868c75b700 5 queue_op 0x230b8cc0 seq 3421 osr(3.6e 0x3ed02a0) 1937 bytes (queue has 50 ops and 305233 bytes) 2014-06-09 16:22:11.693277 7f8692767700 2 waiting 51 > 50 ops || 307116 > 104857600 2014-06-09 16:22:11.693275 7f869af78700 5 -- op tracker -- , seq: 2702, time: 2014-06-09 16:22:11.479399, event: header_read, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.693283 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb0e0 (1 active) 2014-06-09 16:22:11.693283 7f869af78700 5 -- op tracker -- , seq: 2702, time: 2014-06-09 16:22:11.479399, event: throttled, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.693286 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad7a0 (2 active) 2014-06-09 16:22:11.693286 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3362.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.693290 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.693291 7f868c75b700 10 op_submit_finish 3421 2014-06-09 16:22:11.693289 7f869af78700 5 -- op tracker -- , seq: 2702, time: 2014-06-09 16:22:11.479410, event: all_read, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.693295 7f869af78700 5 -- op tracker -- , seq: 2702, time: 2014-06-09 16:22:11.693267, event: dispatched, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.693303 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.693301 7f869af78700 5 -- op tracker -- , seq: 2702, time: 2014-06-09 16:22:11.693299, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.693303 7f868cf5c700 2 waiting 51 > 50 ops || 307116 > 104857600 2014-06-09 16:22:11.693305 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.31f) v4 from osd.6 2014-06-09 16:22:11.693308 7f8691f66700 5 queue_transactions existing osr(0.4d5 0x2bd7610)/0x2bd7610 2014-06-09 16:22:11.693308 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405c8c0 2014-06-09 16:22:11.693297 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 1 2014-06-09 16:22:11.693310 7f8691f66700 2 waiting 51 > 50 ops || 307116 > 104857600 2014-06-09 16:22:11.693310 7f86a5e11700 10 _do_op 0x30d97320 seq 3362 r = 0, finisher 0xbe306e0 0 2014-06-09 16:22:11.693313 7f86a5e11700 10 _finish_op 0x30d97320 seq 3362 osr(0.2a2 0x2bd6c20)/0x2bd6c20 2014-06-09 16:22:11.693313 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.1: 3.718( v 213'620 (0'0,213'620] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) log((0'0,213'620], crt=0'0) missing(0) 2014-06-09 16:22:11.693326 7f868c75b700 10 proc_replica_log for osd.1: 3.718( v 213'620 (0'0,213'620] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) log((0'0,213'620], crt=0'0) missing(0) 2014-06-09 16:22:11.693313 7f869af78700 5 -- op tracker -- , seq: 2702, time: 2014-06-09 16:22:11.693311, event: started, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.693329 7f868f761700 2 waiting 51 > 50 ops || 306568 > 104857600 2014-06-09 16:22:11.693335 7f868af58700 2 waiting 51 > 50 ops || 307230 > 104857600 2014-06-09 16:22:11.693332 7f868c75b700 20 merge_log cut point (usually last shared) is 213'620 (0'0) modify c8c2c718/rbd_data.11616b8b4567.0000000000001732/head//3 by client.4545.0:34202 2014-06-09 10:18:58.678220 2014-06-09 16:22:11.693340 7f8691765700 2 waiting 51 > 50 ops || 823613 > 104857600 2014-06-09 16:22:11.693344 7f868df5e700 2 waiting 51 > 50 ops || 307116 > 104857600 2014-06-09 16:22:11.693340 7f869af78700 5 -- op tracker -- , seq: 2702, time: 2014-06-09 16:22:11.693339, event: done, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.693349 7f8689755700 2 waiting 51 > 50 ops || 307332 > 104857600 2014-06-09 16:22:11.693354 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693356 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693356 7f868e75f700 2 waiting 51 > 50 ops || 306286 > 104857600 2014-06-09 16:22:11.693362 7f868bf5a700 2 waiting 51 > 50 ops || 306662 > 104857600 2014-06-09 16:22:11.693366 7f868b759700 2 waiting 51 > 50 ops || 307234 > 104857600 2014-06-09 16:22:11.693369 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693370 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693370 7f868ef60700 2 waiting 51 > 50 ops || 306662 > 104857600 2014-06-09 16:22:11.693371 7f869af78700 20 osd.26 243 _dispatch 0x2176d880 pg_info(1 pgs e243:0.681) v4 2014-06-09 16:22:11.693374 7f8690f64700 2 waiting 51 > 50 ops || 307234 > 104857600 2014-06-09 16:22:11.693376 7f869af78700 5 -- op tracker -- , seq: 2703, time: 2014-06-09 16:22:11.573998, event: header_read, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.693385 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc120 (1 active) 2014-06-09 16:22:11.693384 7f869af78700 5 -- op tracker -- , seq: 2703, time: 2014-06-09 16:22:11.573999, event: throttled, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.693387 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653680 (2 active) 2014-06-09 16:22:11.693388 7f8692f68700 10 op_submit_start 3422 2014-06-09 16:22:11.693389 7f86a5e11700 5 _do_op 0x2fc8d740 seq 3374 osr(3.111 0x3ed0830)/0x3ed0830 start 2014-06-09 16:22:11.693390 7f8692f68700 5 queue_transactions (trailing journal) 3422 2014-06-09 16:22:11.693392 7f86a5e11700 10 _do_transaction on 0xbd8ed80 2014-06-09 16:22:11.693390 7f869af78700 5 -- op tracker -- , seq: 2703, time: 2014-06-09 16:22:11.574011, event: all_read, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.693392 7f8692f68700 5 queue_op 0x309a3320 seq 3422 osr(3.46f 0x3ed1560) 1335 bytes (queue has 50 ops and 304559 bytes) 2014-06-09 16:22:11.693396 7f8692f68700 10 op_submit_finish 3422 2014-06-09 16:22:11.693396 7f869af78700 5 -- op tracker -- , seq: 2703, time: 2014-06-09 16:22:11.693368, event: dispatched, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.693398 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.693403 7f869af78700 5 -- op tracker -- , seq: 2703, time: 2014-06-09 16:22:11.693401, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.693407 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.681) v4 from osd.0 2014-06-09 16:22:11.693410 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176d880 2014-06-09 16:22:11.693399 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.69f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.693414 7f869af78700 5 -- op tracker -- , seq: 2703, time: 2014-06-09 16:22:11.693412, event: started, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.693430 7f869af78700 5 -- op tracker -- , seq: 2703, time: 2014-06-09 16:22:11.693427, event: done, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.693432 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.693437 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693439 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693450 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693449 7f868cf5c700 2 waiting 51 > 50 ops || 306442 > 104857600 2014-06-09 16:22:11.693452 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693450 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3374.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.693454 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.693453 7f869af78700 20 osd.26 243 _dispatch 0x1abd4540 pg_info(1 pgs e243:0.5b0) v4 2014-06-09 16:22:11.693414 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.693458 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.693458 7f869af78700 5 -- op tracker -- , seq: 2704, time: 2014-06-09 16:22:11.477778, event: header_read, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.693465 7f869af78700 5 -- op tracker -- , seq: 2704, time: 2014-06-09 16:22:11.477778, event: throttled, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.693471 7f869af78700 5 -- op tracker -- , seq: 2704, time: 2014-06-09 16:22:11.477789, event: all_read, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.693476 7f869af78700 5 -- op tracker -- , seq: 2704, time: 2014-06-09 16:22:11.693450, event: dispatched, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.693481 7f86a5e11700 10 _do_op 0x2fc8d740 seq 3374 r = 0, finisher 0xbd5a920 0 2014-06-09 16:22:11.693484 7f86a5e11700 10 _finish_op 0x2fc8d740 seq 3374 osr(3.111 0x3ed0830)/0x3ed0830 2014-06-09 16:22:11.693485 7f869af78700 5 -- op tracker -- , seq: 2704, time: 2014-06-09 16:22:11.693484, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.693490 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5b0) v4 from osd.5 2014-06-09 16:22:11.693493 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd4540 2014-06-09 16:22:11.693493 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653680 (1 active) 2014-06-09 16:22:11.693495 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652a20 (2 active) 2014-06-09 16:22:11.693464 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.693497 7f86a5e11700 5 _do_op 0x309a3680 seq 3375 osr(0.5c3 0x2bd7a50)/0x2bd7a50 start 2014-06-09 16:22:11.693500 7f86a5e11700 10 _do_transaction on 0x32fbbc0 2014-06-09 16:22:11.693498 7f869af78700 5 -- op tracker -- , seq: 2704, time: 2014-06-09 16:22:11.693496, event: started, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.693506 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.693500 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.693513 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.693517 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.693518 7f869af78700 5 -- op tracker -- , seq: 2704, time: 2014-06-09 16:22:11.693516, event: done, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.693526 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693528 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693521 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.504( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.231395 7 0.000194 2014-06-09 16:22:11.693532 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.504( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.693540 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693540 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.693541 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693542 7f869af78700 20 osd.26 243 _dispatch 0x30cf8c40 pg_info(1 pgs e243:0.4d) v4 2014-06-09 16:22:11.693544 7f8692f68700 5 osd.26 pg_epoch: 243 pg[1.504( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.693547 7f869af78700 5 -- op tracker -- , seq: 2705, time: 2014-06-09 16:22:11.303149, event: header_read, op: pg_info(1 pgs e243:0.4d) 2014-06-09 16:22:11.693556 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3375.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.693560 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.693557 7f869af78700 5 -- op tracker -- , seq: 2705, time: 2014-06-09 16:22:11.303150, event: throttled, op: pg_info(1 pgs e243:0.4d) 2014-06-09 16:22:11.693563 7f869af78700 5 -- op tracker -- , seq: 2705, time: 2014-06-09 16:22:11.303157, event: all_read, op: pg_info(1 pgs e243:0.4d) 2014-06-09 16:22:11.693569 7f869af78700 5 -- op tracker -- , seq: 2705, time: 2014-06-09 16:22:11.693539, event: dispatched, op: pg_info(1 pgs e243:0.4d) 2014-06-09 16:22:11.693577 7f86a5e11700 10 _do_op 0x309a3680 seq 3375 r = 0, finisher 0xbd667a0 0 2014-06-09 16:22:11.693577 7f869af78700 5 -- op tracker -- , seq: 2705, time: 2014-06-09 16:22:11.693575, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4d) 2014-06-09 16:22:11.693582 7f86a5e11700 10 _finish_op 0x309a3680 seq 3375 osr(0.5c3 0x2bd7a50)/0x2bd7a50 2014-06-09 16:22:11.693581 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4d) v4 from osd.2 2014-06-09 16:22:11.693584 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf8c40 2014-06-09 16:22:11.693584 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652a20 (1 active) 2014-06-09 16:22:11.693586 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6b40 (2 active) 2014-06-09 16:22:11.693588 7f86a5e11700 5 _do_op 0x2fc8d6e0 seq 3376 osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 start 2014-06-09 16:22:11.693591 7f86a5e11700 10 _do_transaction on 0xbfb2a80 2014-06-09 16:22:11.693588 7f869af78700 5 -- op tracker -- , seq: 2705, time: 2014-06-09 16:22:11.693586, event: started, op: pg_info(1 pgs e243:0.4d) 2014-06-09 16:22:11.693555 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.693596 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.693605 7f869af78700 5 -- op tracker -- , seq: 2705, time: 2014-06-09 16:22:11.693603, event: done, op: pg_info(1 pgs e243:0.4d) 2014-06-09 16:22:11.693613 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693615 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693611 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.693627 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693627 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.693629 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693630 7f869af78700 20 osd.26 243 _dispatch 0x310d6540 pg_info(1 pgs e243:3.6d8) v4 2014-06-09 16:22:11.693634 7f869af78700 5 -- op tracker -- , seq: 2706, time: 2014-06-09 16:22:11.422905, event: header_read, op: pg_info(1 pgs e243:3.6d8) 2014-06-09 16:22:11.693642 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3376.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.693622 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.693643 7f869af78700 5 -- op tracker -- , seq: 2706, time: 2014-06-09 16:22:11.422906, event: throttled, op: pg_info(1 pgs e243:3.6d8) 2014-06-09 16:22:11.693646 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.693649 7f869af78700 5 -- op tracker -- , seq: 2706, time: 2014-06-09 16:22:11.422916, event: all_read, op: pg_info(1 pgs e243:3.6d8) 2014-06-09 16:22:11.693657 7f869af78700 5 -- op tracker -- , seq: 2706, time: 2014-06-09 16:22:11.693627, event: dispatched, op: pg_info(1 pgs e243:3.6d8) 2014-06-09 16:22:11.693664 7f869af78700 5 -- op tracker -- , seq: 2706, time: 2014-06-09 16:22:11.693662, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6d8) 2014-06-09 16:22:11.693668 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6d8) v4 from osd.4 2014-06-09 16:22:11.693669 7f86a5e11700 10 _do_op 0x2fc8d6e0 seq 3376 r = 0, finisher 0xbd680a0 0 2014-06-09 16:22:11.693670 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d6540 2014-06-09 16:22:11.693672 7f86a5e11700 10 _finish_op 0x2fc8d6e0 seq 3376 osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 2014-06-09 16:22:11.693674 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6b40 (1 active) 2014-06-09 16:22:11.693676 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77bd40 (2 active) 2014-06-09 16:22:11.693678 7f86a5e11700 5 _do_op 0x306076e0 seq 3377 osr(3.786 0x3ed1d90)/0x3ed1d90 start 2014-06-09 16:22:11.693675 7f869af78700 5 -- op tracker -- , seq: 2706, time: 2014-06-09 16:22:11.693673, event: started, op: pg_info(1 pgs e243:3.6d8) 2014-06-09 16:22:11.693680 7f86a5e11700 10 _do_transaction on 0xc256780 2014-06-09 16:22:11.693686 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.693688 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.693693 7f8692f68700 5 queue_transactions existing osr(1.504 0x384dbd0)/0x384dbd0 2014-06-09 16:22:11.693692 7f869af78700 5 -- op tracker -- , seq: 2706, time: 2014-06-09 16:22:11.693690, event: done, op: pg_info(1 pgs e243:3.6d8) 2014-06-09 16:22:11.693696 7f8692f68700 10 op_submit_start 3423 2014-06-09 16:22:11.693697 7f8692f68700 5 queue_transactions (trailing journal) 3423 2014-06-09 16:22:11.693698 7f8692f68700 5 queue_op 0x309a32c0 seq 3423 osr(1.504 0x384dbd0) 1997 bytes (queue has 48 ops and 301176 bytes) 2014-06-09 16:22:11.693699 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693701 7f8692f68700 10 op_submit_finish 3423 2014-06-09 16:22:11.693701 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693713 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693715 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693704 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=210 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.693716 7f869af78700 20 osd.26 243 _dispatch 0x310e8fc0 pg_info(1 pgs e243:2.113) v4 2014-06-09 16:22:11.693717 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.693721 7f869af78700 5 -- op tracker -- , seq: 2707, time: 2014-06-09 16:22:11.573951, event: header_read, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.693718 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=210 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.693728 7f869af78700 5 -- op tracker -- , seq: 2707, time: 2014-06-09 16:22:11.573951, event: throttled, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.693733 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3377.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.693734 7f869af78700 5 -- op tracker -- , seq: 2707, time: 2014-06-09 16:22:11.573963, event: all_read, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.693737 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.693732 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=210 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.860697 7 0.000181 2014-06-09 16:22:11.693740 7f869af78700 5 -- op tracker -- , seq: 2707, time: 2014-06-09 16:22:11.693713, event: dispatched, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.693746 7f869af78700 5 -- op tracker -- , seq: 2707, time: 2014-06-09 16:22:11.693744, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.693750 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.113) v4 from osd.7 2014-06-09 16:22:11.693745 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=210 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.693753 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e8fc0 2014-06-09 16:22:11.693756 7f86a5e11700 10 _do_op 0x306076e0 seq 3377 r = 0, finisher 0xbebf4c0 0 2014-06-09 16:22:11.693758 7f86a5e11700 10 _finish_op 0x306076e0 seq 3377 osr(3.786 0x3ed1d90)/0x3ed1d90 2014-06-09 16:22:11.693757 7f869af78700 5 -- op tracker -- , seq: 2707, time: 2014-06-09 16:22:11.693755, event: started, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.693761 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77bd40 (1 active) 2014-06-09 16:22:11.693763 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780d80 (2 active) 2014-06-09 16:22:11.693764 7f86a5e11700 5 _do_op 0x2fcae5a0 seq 3378 osr(3.172 0x3ed06e0)/0x3ed06e0 start 2014-06-09 16:22:11.693757 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=210 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.693767 7f86a5e11700 10 _do_transaction on 0xc05a480 2014-06-09 16:22:11.693772 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.693772 7f869af78700 5 -- op tracker -- , seq: 2707, time: 2014-06-09 16:22:11.693770, event: done, op: pg_info(1 pgs e243:2.113) 2014-06-09 16:22:11.693769 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=210 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.693778 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693780 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693780 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'543 -> 213'543 2014-06-09 16:22:11.693792 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693794 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693795 7f869af78700 20 osd.26 243 _dispatch 0x2f0301c0 pg_info(1 pgs e243:3.3a9) v4 2014-06-09 16:22:11.693800 7f869af78700 5 -- op tracker -- , seq: 2708, time: 2014-06-09 16:22:11.479468, event: header_read, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.693803 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.693806 7f869af78700 5 -- op tracker -- , seq: 2708, time: 2014-06-09 16:22:11.479469, event: throttled, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.693812 7f869af78700 5 -- op tracker -- , seq: 2708, time: 2014-06-09 16:22:11.479479, event: all_read, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.693818 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3378.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.693822 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.693818 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.693819 7f869af78700 5 -- op tracker -- , seq: 2708, time: 2014-06-09 16:22:11.693792, event: dispatched, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.693816 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.1 now 3.718( v 213'620 (0'0,213'620] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.693832 7f869af78700 5 -- op tracker -- , seq: 2708, time: 2014-06-09 16:22:11.693830, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.693836 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a9) v4 from osd.3 2014-06-09 16:22:11.693838 7f86a5e11700 10 _do_op 0x2fcae5a0 seq 3378 r = 0, finisher 0x17001380 0 2014-06-09 16:22:11.693840 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f0301c0 2014-06-09 16:22:11.693841 7f86a5e11700 10 _finish_op 0x2fcae5a0 seq 3378 osr(3.172 0x3ed06e0)/0x3ed06e0 2014-06-09 16:22:11.693844 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780d80 (1 active) 2014-06-09 16:22:11.693835 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.693845 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77bc20 (2 active) 2014-06-09 16:22:11.693848 7f86a5e11700 5 _do_op 0x30607740 seq 3379 osr(3.1f0 0x3ed0990)/0x3ed0990 start 2014-06-09 16:22:11.693844 7f869af78700 5 -- op tracker -- , seq: 2708, time: 2014-06-09 16:22:11.693842, event: started, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.693850 7f86a5e11700 10 _do_transaction on 0xbf28a80 2014-06-09 16:22:11.693856 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.693848 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.695075 7 0.000324 2014-06-09 16:22:11.693858 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.693863 7f8692f68700 5 queue_transactions existing osr(3.2df 0x3ed0e40)/0x3ed0e40 2014-06-09 16:22:11.693866 7f8692f68700 10 op_submit_start 3424 2014-06-09 16:22:11.693867 7f8692f68700 5 queue_transactions (trailing journal) 3424 2014-06-09 16:22:11.693863 7f869af78700 5 -- op tracker -- , seq: 2708, time: 2014-06-09 16:22:11.693861, event: done, op: pg_info(1 pgs e243:3.3a9) 2014-06-09 16:22:11.693868 7f8692f68700 5 queue_op 0x309a3260 seq 3424 osr(3.2df 0x3ed0e40) 1335 bytes (queue has 47 ops and 299747 bytes) 2014-06-09 16:22:11.693861 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.693871 7f8692f68700 10 op_submit_finish 3424 2014-06-09 16:22:11.693871 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693873 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693872 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.961240 0 0.000000 2014-06-09 16:22:11.693885 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693887 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693887 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.693888 7f869af78700 20 osd.26 243 _dispatch 0x30ebae00 pg_info(1 pgs e243:3.39e) v4 2014-06-09 16:22:11.693888 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (19 active) 2014-06-09 16:22:11.693885 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.693893 7f869af78700 5 -- op tracker -- , seq: 2709, time: 2014-06-09 16:22:11.561981, event: header_read, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.693900 7f869af78700 5 -- op tracker -- , seq: 2709, time: 2014-06-09 16:22:11.561983, event: throttled, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.693895 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=191 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.693902 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3379.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.693906 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.693909 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (20 active) 2014-06-09 16:22:11.693906 7f869af78700 5 -- op tracker -- , seq: 2709, time: 2014-06-09 16:22:11.561999, event: all_read, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.693908 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.693915 7f869af78700 5 -- op tracker -- , seq: 2709, time: 2014-06-09 16:22:11.693885, event: dispatched, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.693922 7f869af78700 5 -- op tracker -- , seq: 2709, time: 2014-06-09 16:22:11.693920, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.693913 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 5 2014-06-09 16:22:11.693919 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.693926 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.39e) v4 from osd.1 2014-06-09 16:22:11.693928 7f86a5e11700 10 _do_op 0x30607740 seq 3379 r = 0, finisher 0x305083e0 0 2014-06-09 16:22:11.693929 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebae00 2014-06-09 16:22:11.693931 7f86a5e11700 10 _finish_op 0x30607740 seq 3379 osr(3.1f0 0x3ed0990)/0x3ed0990 2014-06-09 16:22:11.693933 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77bc20 (1 active) 2014-06-09 16:22:11.693935 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc000 (2 active) 2014-06-09 16:22:11.693933 7f869af78700 5 -- op tracker -- , seq: 2709, time: 2014-06-09 16:22:11.693932, event: started, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.693928 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'639 -> 214'639 2014-06-09 16:22:11.693938 7f86a5e11700 5 _do_op 0x306078c0 seq 3380 osr(1.2a3 0x384d1f0)/0x384d1f0 start 2014-06-09 16:22:11.693928 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd5 2014-06-09 16:22:11.693940 7f86a5e11700 10 _do_transaction on 0xc2a09c0 2014-06-09 16:22:11.693946 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.693942 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.693949 7f869af78700 5 -- op tracker -- , seq: 2709, time: 2014-06-09 16:22:11.693947, event: done, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.693952 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.693958 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693961 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693961 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'647 (0'0,214'647] local-les=206 n=123 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.5: log((214'647,214'648], crt=0'0) missing(0) 2014-06-09 16:22:11.693973 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.693972 7f8692f68700 10 merge_log log((214'647,214'648], crt=0'0) from osd.5 into log((0'0,214'647], crt=0'0) 2014-06-09 16:22:11.693975 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.693976 7f8692f68700 10 merge_log extending head to 214'648 2014-06-09 16:22:11.693988 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x4fce000 (0 active) 2014-06-09 16:22:11.693976 7f869af78700 20 osd.26 243 _dispatch 0x30cf1dc0 pg_info(1 pgs e243:2.d3) v4 2014-06-09 16:22:11.693992 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0xb78a000 (1 active) 2014-06-09 16:22:11.693994 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.693990 7f8692f68700 20 ? 214'648 (0'0) modify 993f1730/rbd_data.11786b8b4567.00000000000016f0/head//3 by client.4568.0:35013 2014-06-09 10:19:13.609863 2014-06-09 16:22:11.693997 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0xb78a000 (0 active) 2014-06-09 16:22:11.693999 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.694005 7f86a5610700 5 _do_op 0x2fc8c780 seq 3373 osr(2.2d9 0x2ed0c30)/0x2ed0c30 start 2014-06-09 16:22:11.694007 7f8691f66700 10 op_submit_start 3425 2014-06-09 16:22:11.694000 7f869af78700 5 -- op tracker -- , seq: 2710, time: 2014-06-09 16:22:11.576131, event: header_read, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.694008 7f86a5610700 10 _do_transaction on 0xc249140 2014-06-09 16:22:11.694009 7f8691f66700 5 queue_transactions (trailing journal) 3425 2014-06-09 16:22:11.693969 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.718( v 213'620 (0'0,213'620] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.694011 7f8691f66700 5 queue_op 0xc64b920 seq 3425 osr(0.4d5 0x2bd7610) 1883 bytes (queue has 47 ops and 300295 bytes) 2014-06-09 16:22:11.694014 7f8691f66700 10 op_submit_finish 3425 2014-06-09 16:22:11.694013 7f869af78700 5 -- op tracker -- , seq: 2710, time: 2014-06-09 16:22:11.576132, event: throttled, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.694022 7f869af78700 5 -- op tracker -- , seq: 2710, time: 2014-06-09 16:22:11.576145, event: all_read, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.694017 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.424( v 214'730 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.694030 7f869af78700 5 -- op tracker -- , seq: 2710, time: 2014-06-09 16:22:11.693973, event: dispatched, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.694032 7f8691f66700 7 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.4 2014-06-09 16:22:11.694039 7f869af78700 5 -- op tracker -- , seq: 2710, time: 2014-06-09 16:22:11.694036, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.694045 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.d3) v4 from osd.9 2014-06-09 16:22:11.694048 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf1dc0 2014-06-09 16:22:11.694038 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 sending log((213'620,214'639], crt=0'0) 2014-06-09 16:22:11.694052 7f868c75b700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.694054 7f869af78700 5 -- op tracker -- , seq: 2710, time: 2014-06-09 16:22:11.694052, event: started, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.694043 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.4 info 3.424( v 214'730 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242), identical to ours 2014-06-09 16:22:11.694066 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.694071 7f869af78700 5 -- op tracker -- , seq: 2710, time: 2014-06-09 16:22:11.694069, event: done, op: pg_info(1 pgs e243:2.d3) 2014-06-09 16:22:11.694077 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.694078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.694072 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.694069 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.718( v 214'639 lc 213'620 (0'0,214'639] local-les=191 n=121 ec=118 les/c 191/191 242/242/242) missing missing(4) 2014-06-09 16:22:11.694087 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.694088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.694089 7f869af78700 20 osd.26 243 _dispatch 0x31156380 pg_info(1 pgs e243:2.1fc) v4 2014-06-09 16:22:11.694084 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.694092 7f869af78700 5 -- op tracker -- , seq: 2711, time: 2014-06-09 16:22:11.434684, event: header_read, op: pg_info(1 pgs e243:2.1fc) 2014-06-09 16:22:11.694086 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.1 has 4 missing 2014-06-09 16:22:11.697850 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.697861 7f869af78700 5 -- op tracker -- , seq: 2711, time: 2014-06-09 16:22:11.434685, event: throttled, op: pg_info(1 pgs e243:2.1fc) 2014-06-09 16:22:11.697862 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.697860 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing b90b0718/rbd_data.11816b8b4567.0000000000001911/head//3 214'639 is on osd.26 2014-06-09 16:22:11.697871 7f869af78700 5 -- op tracker -- , seq: 2711, time: 2014-06-09 16:22:11.434697, event: all_read, op: pg_info(1 pgs e243:2.1fc) 2014-06-09 16:22:11.697876 7f8691f66700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.697879 7f869af78700 5 -- op tracker -- , seq: 2711, time: 2014-06-09 16:22:11.694087, event: dispatched, op: pg_info(1 pgs e243:2.1fc) 2014-06-09 16:22:11.697877 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing c3cb6718/rbd_data.11686b8b4567.0000000000001689/head//3 213'631 is on osd.26 2014-06-09 16:22:11.697887 7f869af78700 5 -- op tracker -- , seq: 2711, time: 2014-06-09 16:22:11.697885, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1fc) 2014-06-09 16:22:11.697892 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1fc) v4 from osd.8 2014-06-09 16:22:11.697895 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31156380 2014-06-09 16:22:11.697888 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing c8c2c718/rbd_data.11616b8b4567.0000000000001732/head//3 213'627 is on osd.26 2014-06-09 16:22:11.697893 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:175 2014-06-09 16:22:11.697902 7f869af78700 5 -- op tracker -- , seq: 2711, time: 2014-06-09 16:22:11.697899, event: started, op: pg_info(1 pgs e243:2.1fc) 2014-06-09 16:22:11.697899 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 5d02bf18/rbd_data.11726b8b4567.0000000000001927/head//3 214'638 is on osd.26 2014-06-09 16:22:11.697908 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.697913 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.697920 7f869af78700 5 -- op tracker -- , seq: 2711, time: 2014-06-09 16:22:11.697917, event: done, op: pg_info(1 pgs e243:2.1fc) 2014-06-09 16:22:11.697910 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {b90b0718/rbd_data.11816b8b4567.0000000000001911/head//3=214'639,c3cb6718/rbd_data.11686b8b4567.0000000000001689/head//3=213'631,c8c2c718/rbd_data.11616b8b4567.0000000000001732/head//3=213'627(213'620),5d02bf18/rbd_data.11726b8b4567.0000000000001927/head//3=214'638} 2014-06-09 16:22:11.697932 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.697934 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.697927 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.697927 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 1 as a source 2014-06-09 16:22:11.697936 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.697938 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing b90b0718/rbd_data.11816b8b4567.0000000000001911/head//3 214'639 also missing on osd.1 2014-06-09 16:22:11.697945 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.697952 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.697954 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.697955 7f869af78700 20 osd.26 243 _dispatch 0x31162e00 pg_info(1 pgs e243:3.31f) v4 2014-06-09 16:22:11.697949 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing c3cb6718/rbd_data.11686b8b4567.0000000000001689/head//3 213'631 also missing on osd.1 2014-06-09 16:22:11.697958 7f8691f66700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.697961 7f869af78700 5 -- op tracker -- , seq: 2712, time: 2014-06-09 16:22:11.479419, event: header_read, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.697961 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing c8c2c718/rbd_data.11616b8b4567.0000000000001732/head//3 213'627 also missing on osd.1 2014-06-09 16:22:11.697970 7f869af78700 5 -- op tracker -- , seq: 2712, time: 2014-06-09 16:22:11.479419, event: throttled, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.697977 7f869af78700 5 -- op tracker -- , seq: 2712, time: 2014-06-09 16:22:11.479432, event: all_read, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.697972 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 5d02bf18/rbd_data.11726b8b4567.0000000000001927/head//3 214'638 also missing on osd.1 2014-06-09 16:22:11.697973 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.697985 7f869af78700 5 -- op tracker -- , seq: 2712, time: 2014-06-09 16:22:11.697951, event: dispatched, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.697989 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.697982 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {b90b0718/rbd_data.11816b8b4567.0000000000001911/head//3=214'639,c3cb6718/rbd_data.11686b8b4567.0000000000001689/head//3=213'631,c8c2c718/rbd_data.11616b8b4567.0000000000001732/head//3=213'627(213'620),5d02bf18/rbd_data.11726b8b4567.0000000000001927/head//3=214'638} 2014-06-09 16:22:11.697992 7f869af78700 5 -- op tracker -- , seq: 2712, time: 2014-06-09 16:22:11.697989, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.697996 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.31f) v4 from osd.6 2014-06-09 16:22:11.697999 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31162e00 2014-06-09 16:22:11.697994 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.697996 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.698005 7f869af78700 5 -- op tracker -- , seq: 2712, time: 2014-06-09 16:22:11.698003, event: started, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.698005 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.698007 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.698016 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.698022 7f869af78700 5 -- op tracker -- , seq: 2712, time: 2014-06-09 16:22:11.698019, event: done, op: pg_info(1 pgs e243:3.31f) 2014-06-09 16:22:11.698016 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 1 2014-06-09 16:22:11.698030 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698032 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698025 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.698027 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.694098 7f868af58700 10 op_submit_start 3426 2014-06-09 16:22:11.698039 7f868af58700 5 queue_transactions (trailing journal) 3426 2014-06-09 16:22:11.698042 7f8691f66700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.698043 7f868af58700 5 queue_op 0x30d97c20 seq 3426 osr(0.33a 0x2bd70b0) 1997 bytes (queue has 50 ops and 822555 bytes) 2014-06-09 16:22:11.698038 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.698046 7f868af58700 10 op_submit_finish 3426 2014-06-09 16:22:11.698051 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698053 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698054 7f869af78700 20 osd.26 243 _dispatch 0x30dc2540 pg_info(1 pgs e243:3.d1) v4 2014-06-09 16:22:11.698054 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:181 2014-06-09 16:22:11.698059 7f869af78700 5 -- op tracker -- , seq: 2713, time: 2014-06-09 16:22:11.574061, event: header_read, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.698052 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.5f9( v 214'611 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.698068 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.698067 7f869af78700 5 -- op tracker -- , seq: 2713, time: 2014-06-09 16:22:11.574061, event: throttled, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.698066 7f868af58700 7 osd.26 pg_epoch: 243 pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.3 2014-06-09 16:22:11.698074 7f869af78700 5 -- op tracker -- , seq: 2713, time: 2014-06-09 16:22:11.574073, event: all_read, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.698072 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:2.154( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.698081 7f869af78700 5 -- op tracker -- , seq: 2713, time: 2014-06-09 16:22:11.698051, event: dispatched, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.698083 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.698091 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.698088 7f869af78700 5 -- op tracker -- , seq: 2713, time: 2014-06-09 16:22:11.698086, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.698077 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.3 info 3.5f9( v 214'611 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242), identical to ours 2014-06-09 16:22:11.698095 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.698096 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.d1) v4 from osd.0 2014-06-09 16:22:11.698099 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc2540 2014-06-09 16:22:11.698095 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.698103 7f869af78700 5 -- op tracker -- , seq: 2713, time: 2014-06-09 16:22:11.698102, event: started, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.698106 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.698112 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (19 active) 2014-06-09 16:22:11.693978 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.698117 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.698114 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.698121 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.698133 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.698138 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.698138 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3380.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.698143 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.698143 7f868c75b700 5 queue_transactions existing osr(3.718 0x3ed1f70)/0x3ed1f70 2014-06-09 16:22:11.698148 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.698162 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.698165 7f86a5e11700 10 _do_op 0x306078c0 seq 3380 r = 0, finisher 0xbcc9f60 0 2014-06-09 16:22:11.698169 7f86a5e11700 10 _finish_op 0x306078c0 seq 3380 osr(1.2a3 0x384d1f0)/0x384d1f0 2014-06-09 16:22:11.698166 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.698179 7f869af78700 5 -- op tracker -- , seq: 2713, time: 2014-06-09 16:22:11.698176, event: done, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.698178 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.698187 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698189 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698188 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.698200 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698195 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.698202 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.694181 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.698203 7f869af78700 20 osd.26 243 _dispatch 0x30daf180 pg_info(1 pgs e243:0.5b0) v4 2014-06-09 16:22:11.698207 7f8691f66700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.698209 7f869af78700 5 -- op tracker -- , seq: 2714, time: 2014-06-09 16:22:11.477794, event: header_read, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.698219 7f869af78700 5 -- op tracker -- , seq: 2714, time: 2014-06-09 16:22:11.477794, event: throttled, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.698226 7f869af78700 5 -- op tracker -- , seq: 2714, time: 2014-06-09 16:22:11.477807, event: all_read, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.698222 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:189 2014-06-09 16:22:11.698232 7f869af78700 5 -- op tracker -- , seq: 2714, time: 2014-06-09 16:22:11.698200, event: dispatched, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.698237 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.698242 7f8691765700 10 op_submit_start 3427 2014-06-09 16:22:11.698240 7f869af78700 5 -- op tracker -- , seq: 2714, time: 2014-06-09 16:22:11.698238, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.698244 7f8691765700 5 queue_transactions (trailing journal) 3427 2014-06-09 16:22:11.698245 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5b0) v4 from osd.5 2014-06-09 16:22:11.698246 7f8691765700 5 queue_op 0x308ca9c0 seq 3427 osr(3.60 0x3ed02e0) 518380 bytes (queue has 50 ops and 822555 bytes) 2014-06-09 16:22:11.698248 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daf180 2014-06-09 16:22:11.693999 7f8692f68700 20 merge_log 214'648 (0'0) modify 993f1730/rbd_data.11786b8b4567.00000000000016f0/head//3 by client.4568.0:35013 2014-06-09 10:19:13.609863 2014-06-09 16:22:11.698253 7f869af78700 5 -- op tracker -- , seq: 2714, time: 2014-06-09 16:22:11.698251, event: started, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.698268 7f869af78700 5 -- op tracker -- , seq: 2714, time: 2014-06-09 16:22:11.698266, event: done, op: pg_info(1 pgs e243:0.5b0) 2014-06-09 16:22:11.698273 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.698276 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698278 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698290 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698292 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698293 7f869af78700 20 osd.26 243 _dispatch 0x30cf8a80 pg_info(1 pgs e243:1.53) v4 2014-06-09 16:22:11.698292 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3373.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.698297 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.698298 7f869af78700 5 -- op tracker -- , seq: 2715, time: 2014-06-09 16:22:11.303160, event: header_read, op: pg_info(1 pgs e243:1.53) 2014-06-09 16:22:11.698306 7f869af78700 5 -- op tracker -- , seq: 2715, time: 2014-06-09 16:22:11.303161, event: throttled, op: pg_info(1 pgs e243:1.53) 2014-06-09 16:22:11.698312 7f869af78700 5 -- op tracker -- , seq: 2715, time: 2014-06-09 16:22:11.303166, event: all_read, op: pg_info(1 pgs e243:1.53) 2014-06-09 16:22:11.698318 7f869af78700 5 -- op tracker -- , seq: 2715, time: 2014-06-09 16:22:11.698290, event: dispatched, op: pg_info(1 pgs e243:1.53) 2014-06-09 16:22:11.698325 7f869af78700 5 -- op tracker -- , seq: 2715, time: 2014-06-09 16:22:11.698323, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.53) 2014-06-09 16:22:11.698330 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.53) v4 from osd.2 2014-06-09 16:22:11.698333 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf8a80 2014-06-09 16:22:11.698333 7f86a5610700 10 _do_op 0x2fc8c780 seq 3373 r = 0, finisher 0x3062bd20 0 2014-06-09 16:22:11.698338 7f869af78700 5 -- op tracker -- , seq: 2715, time: 2014-06-09 16:22:11.698336, event: started, op: pg_info(1 pgs e243:1.53) 2014-06-09 16:22:11.698354 7f869af78700 5 -- op tracker -- , seq: 2715, time: 2014-06-09 16:22:11.698351, event: done, op: pg_info(1 pgs e243:1.53) 2014-06-09 16:22:11.698364 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698366 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698377 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698379 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698380 7f869af78700 20 osd.26 243 _dispatch 0x310d6380 pg_info(1 pgs e243:3.2ac) v4 2014-06-09 16:22:11.698386 7f869af78700 5 -- op tracker -- , seq: 2716, time: 2014-06-09 16:22:11.422920, event: header_read, op: pg_info(1 pgs e243:3.2ac) 2014-06-09 16:22:11.698403 7f869af78700 5 -- op tracker -- , seq: 2716, time: 2014-06-09 16:22:11.422921, event: throttled, op: pg_info(1 pgs e243:3.2ac) 2014-06-09 16:22:11.698409 7f869af78700 5 -- op tracker -- , seq: 2716, time: 2014-06-09 16:22:11.422931, event: all_read, op: pg_info(1 pgs e243:3.2ac) 2014-06-09 16:22:11.698416 7f869af78700 5 -- op tracker -- , seq: 2716, time: 2014-06-09 16:22:11.698377, event: dispatched, op: pg_info(1 pgs e243:3.2ac) 2014-06-09 16:22:11.698426 7f869af78700 5 -- op tracker -- , seq: 2716, time: 2014-06-09 16:22:11.698424, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2ac) 2014-06-09 16:22:11.698430 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2ac) v4 from osd.4 2014-06-09 16:22:11.698433 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d6380 2014-06-09 16:22:11.698439 7f869af78700 5 -- op tracker -- , seq: 2716, time: 2014-06-09 16:22:11.698437, event: started, op: pg_info(1 pgs e243:3.2ac) 2014-06-09 16:22:11.698456 7f869af78700 5 -- op tracker -- , seq: 2716, time: 2014-06-09 16:22:11.698454, event: done, op: pg_info(1 pgs e243:3.2ac) 2014-06-09 16:22:11.698465 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698467 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698468 7f8692f68700 10 merge_log result log((0'0,214'648], crt=214'648) missing(1) changed=1 2014-06-09 16:22:11.698477 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698478 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698478 7f869af78700 20 osd.26 243 _dispatch 0x310e8e00 pg_info(1 pgs e243:1.589) v4 2014-06-09 16:22:11.698483 7f869af78700 5 -- op tracker -- , seq: 2717, time: 2014-06-09 16:22:11.573968, event: header_read, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.698473 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 peering m=1] peer osd.5 now 3.730( v 214'648 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) missing(0) 2014-06-09 16:22:11.698490 7f869af78700 5 -- op tracker -- , seq: 2717, time: 2014-06-09 16:22:11.573968, event: throttled, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.698497 7f869af78700 5 -- op tracker -- , seq: 2717, time: 2014-06-09 16:22:11.573979, event: all_read, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.698494 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.708628 6 0.000271 2014-06-09 16:22:11.698504 7f869af78700 5 -- op tracker -- , seq: 2717, time: 2014-06-09 16:22:11.698477, event: dispatched, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.698512 7f869af78700 5 -- op tracker -- , seq: 2717, time: 2014-06-09 16:22:11.698509, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.698516 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.589) v4 from osd.7 2014-06-09 16:22:11.698509 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.698519 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e8e00 2014-06-09 16:22:11.698525 7f869af78700 5 -- op tracker -- , seq: 2717, time: 2014-06-09 16:22:11.698522, event: started, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.698521 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 peering m=1] state: osd.5 has no missing, identical log 2014-06-09 16:22:11.698540 7f869af78700 5 -- op tracker -- , seq: 2717, time: 2014-06-09 16:22:11.698537, event: done, op: pg_info(1 pgs e243:1.589) 2014-06-09 16:22:11.698535 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000026 0 0.000000 2014-06-09 16:22:11.698548 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698550 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698548 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.698562 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698564 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698566 7f869af78700 20 osd.26 243 _dispatch 0x3044c000 pg_info(1 pgs e243:0.3b6) v4 2014-06-09 16:22:11.698559 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.527829 0 0.000000 2014-06-09 16:22:11.698571 7f869af78700 5 -- op tracker -- , seq: 2718, time: 2014-06-09 16:22:11.549640, event: header_read, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.698573 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.698579 7f869af78700 5 -- op tracker -- , seq: 2718, time: 2014-06-09 16:22:11.549640, event: throttled, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.698582 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.694110 7f868f761700 2 waiting 51 > 50 ops || 823890 > 104857600 2014-06-09 16:22:11.698594 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.698603 7f868e75f700 2 waiting 51 > 50 ops || 823608 > 104857600 2014-06-09 16:22:11.698610 7f868bf5a700 2 waiting 51 > 50 ops || 823984 > 104857600 2014-06-09 16:22:11.698604 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.698615 7f868b759700 2 waiting 51 > 50 ops || 824556 > 104857600 2014-06-09 16:22:11.698587 7f869af78700 5 -- op tracker -- , seq: 2718, time: 2014-06-09 16:22:11.549652, event: all_read, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.698613 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.698620 7f868ef60700 2 waiting 51 > 50 ops || 823984 > 104857600 2014-06-09 16:22:11.698624 7f8690f64700 2 waiting 51 > 50 ops || 824556 > 104857600 2014-06-09 16:22:11.698623 7f869af78700 5 -- op tracker -- , seq: 2718, time: 2014-06-09 16:22:11.698562, event: dispatched, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.698629 7f8692767700 2 waiting 51 > 50 ops || 824438 > 104857600 2014-06-09 16:22:11.698633 7f868cf5c700 2 waiting 51 > 50 ops || 824438 > 104857600 2014-06-09 16:22:11.698641 7f8689755700 2 waiting 51 > 50 ops || 824654 > 104857600 2014-06-09 16:22:11.698630 7f869af78700 5 -- op tracker -- , seq: 2718, time: 2014-06-09 16:22:11.698629, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.698647 7f868c75b700 2 waiting 51 > 50 ops || 823608 > 104857600 2014-06-09 16:22:11.698644 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3b6) v4 from osd.3 2014-06-09 16:22:11.698657 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044c000 2014-06-09 16:22:11.698664 7f868bf5a700 2 waiting 51 > 50 ops || 823036 > 104857600 2014-06-09 16:22:11.698663 7f869af78700 5 -- op tracker -- , seq: 2718, time: 2014-06-09 16:22:11.698660, event: started, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.698670 7f868b759700 2 waiting 51 > 50 ops || 823608 > 104857600 2014-06-09 16:22:11.698675 7f868ef60700 2 waiting 51 > 50 ops || 823036 > 104857600 2014-06-09 16:22:11.698678 7f8690f64700 2 waiting 51 > 50 ops || 823608 > 104857600 2014-06-09 16:22:11.698682 7f8692767700 2 waiting 51 > 50 ops || 823490 > 104857600 2014-06-09 16:22:11.698681 7f869af78700 5 -- op tracker -- , seq: 2718, time: 2014-06-09 16:22:11.698679, event: done, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.698685 7f868cf5c700 2 waiting 51 > 50 ops || 823490 > 104857600 2014-06-09 16:22:11.698688 7f8689755700 2 waiting 51 > 50 ops || 823706 > 104857600 2014-06-09 16:22:11.698692 7f868c75b700 2 waiting 51 > 50 ops || 822660 > 104857600 2014-06-09 16:22:11.698692 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc000 (1 active) 2014-06-09 16:22:11.698689 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698698 7f868f761700 2 waiting 51 > 50 ops || 822942 > 104857600 2014-06-09 16:22:11.698699 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698695 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407d40 (2 active) 2014-06-09 16:22:11.698705 7f86a5e11700 5 _do_op 0x2ea52960 seq 3381 osr(1.3c3 0x384d530)/0x384d530 start 2014-06-09 16:22:11.698708 7f86a5e11700 10 _do_transaction on 0xc134cc0 2014-06-09 16:22:11.698713 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698714 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698716 7f869af78700 20 osd.26 243 _dispatch 0x30ebafc0 pg_info(1 pgs e243:3.66a) v4 2014-06-09 16:22:11.698716 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.698721 7f869af78700 5 -- op tracker -- , seq: 2719, time: 2014-06-09 16:22:11.563609, event: header_read, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.698721 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.698730 7f869af78700 5 -- op tracker -- , seq: 2719, time: 2014-06-09 16:22:11.563610, event: throttled, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.698739 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0xb184000 (1 active) 2014-06-09 16:22:11.698742 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.698737 7f8692f68700 10 osd.26 243 queue_for_recovery queued pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.698747 7f869af78700 5 -- op tracker -- , seq: 2719, time: 2014-06-09 16:22:11.563641, event: all_read, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.698755 7f869af78700 5 -- op tracker -- , seq: 2719, time: 2014-06-09 16:22:11.698712, event: dispatched, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.698756 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.698751 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 3.730( v 214'648 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) 2014-06-09 16:22:11.698762 7f869af78700 5 -- op tracker -- , seq: 2719, time: 2014-06-09 16:22:11.698760, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.698767 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.66a) v4 from osd.1 2014-06-09 16:22:11.698770 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebafc0 2014-06-09 16:22:11.698765 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 is up to date, queueing in pending_activators 2014-06-09 16:22:11.698775 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3381.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.698775 7f869af78700 5 -- op tracker -- , seq: 2719, time: 2014-06-09 16:22:11.698773, event: started, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.698780 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.698781 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 3.730( v 214'648 (0'0,214'648] local-les=206 n=124 ec=118 les/c 206/206 242/242/242) uptodate 2014-06-09 16:22:11.698793 7f869af78700 5 -- op tracker -- , seq: 2719, time: 2014-06-09 16:22:11.698791, event: done, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.698801 7f86a5e11700 10 _do_op 0x2ea52960 seq 3381 r = 0, finisher 0xbd6eaa0 0 2014-06-09 16:22:11.698803 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698805 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698795 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.698806 7f86a5e11700 10 _finish_op 0x2ea52960 seq 3381 osr(1.3c3 0x384d530)/0x384d530 2014-06-09 16:22:11.698808 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 993f1730/rbd_data.11786b8b4567.00000000000016f0/head//3 214'648 also missing on osd.26 2014-06-09 16:22:11.698816 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407d40 (1 active) 2014-06-09 16:22:11.698818 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698820 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698820 7f868ef60700 2 waiting 51 > 50 ops || 823083 > 104857600 2014-06-09 16:22:11.698821 7f869af78700 20 osd.26 243 _dispatch 0xddc5dc0 pg_info(1 pgs e243:0.6d2) v4 2014-06-09 16:22:11.698825 7f8690f64700 2 waiting 51 > 50 ops || 823655 > 104857600 2014-06-09 16:22:11.698818 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {993f1730/rbd_data.11786b8b4567.00000000000016f0/head//3=214'648} 2014-06-09 16:22:11.698827 7f869af78700 5 -- op tracker -- , seq: 2720, time: 2014-06-09 16:22:11.576210, event: header_read, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.698830 7f8692767700 2 waiting 51 > 50 ops || 823537 > 104857600 2014-06-09 16:22:11.698834 7f868cf5c700 2 waiting 51 > 50 ops || 823537 > 104857600 2014-06-09 16:22:11.698828 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 5 as a source 2014-06-09 16:22:11.698834 7f869af78700 5 -- op tracker -- , seq: 2720, time: 2014-06-09 16:22:11.576211, event: throttled, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.698840 7f8689755700 2 waiting 51 > 50 ops || 823753 > 104857600 2014-06-09 16:22:11.698819 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca485a0 (2 active) 2014-06-09 16:22:11.698846 7f868c75b700 2 waiting 51 > 50 ops || 822707 > 104857600 2014-06-09 16:22:11.698838 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 993f1730/rbd_data.11786b8b4567.00000000000016f0/head//3 214'648 is on osd.5 2014-06-09 16:22:11.698848 7f86a5e11700 5 _do_op 0x30d97020 seq 3382 osr(2.4 0x2ed0310)/0x2ed0310 start 2014-06-09 16:22:11.698851 7f86a5e11700 10 _do_transaction on 0xc3269c0 2014-06-09 16:22:11.698841 7f869af78700 5 -- op tracker -- , seq: 2720, time: 2014-06-09 16:22:11.576224, event: all_read, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.698857 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.698849 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {993f1730/rbd_data.11786b8b4567.00000000000016f0/head//3=214'648} 2014-06-09 16:22:11.698859 7f869af78700 5 -- op tracker -- , seq: 2720, time: 2014-06-09 16:22:11.698817, event: dispatched, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.698864 7f868f761700 2 waiting 51 > 50 ops || 822989 > 104857600 2014-06-09 16:22:11.698866 7f869af78700 5 -- op tracker -- , seq: 2720, time: 2014-06-09 16:22:11.698864, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.698861 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.698871 7f868bf5a700 2 waiting 51 > 50 ops || 823083 > 104857600 2014-06-09 16:22:11.698871 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6d2) v4 from osd.9 2014-06-09 16:22:11.698874 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc5dc0 2014-06-09 16:22:11.698872 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 206 2014-06-09 16:22:11.698880 7f869af78700 5 -- op tracker -- , seq: 2720, time: 2014-06-09 16:22:11.698877, event: started, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.698884 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 5 2014-06-09 16:22:11.698898 7f869af78700 5 -- op tracker -- , seq: 2720, time: 2014-06-09 16:22:11.698896, event: done, op: pg_info(1 pgs e243:0.6d2) 2014-06-09 16:22:11.698901 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.698895 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.698906 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698907 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698907 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.698919 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.698921 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.698919 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3382.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.698922 7f869af78700 20 osd.26 243 _dispatch 0x311561c0 pg_info(1 pgs e243:2.2b) v4 2014-06-09 16:22:11.698924 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.698927 7f869af78700 5 -- op tracker -- , seq: 2721, time: 2014-06-09 16:22:11.434704, event: header_read, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.698934 7f869af78700 5 -- op tracker -- , seq: 2721, time: 2014-06-09 16:22:11.434704, event: throttled, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.698940 7f869af78700 5 -- op tracker -- , seq: 2721, time: 2014-06-09 16:22:11.434715, event: all_read, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.698946 7f8692f68700 10 write_log with: dirty_to: 0'0, dirty_from: 214'647, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.698948 7f869af78700 5 -- op tracker -- , seq: 2721, time: 2014-06-09 16:22:11.698919, event: dispatched, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.698953 7f86a5e11700 10 _do_op 0x30d97020 seq 3382 r = 0, finisher 0xbd6a960 0 2014-06-09 16:22:11.698957 7f86a5e11700 10 _finish_op 0x30d97020 seq 3382 osr(2.4 0x2ed0310)/0x2ed0310 2014-06-09 16:22:11.698956 7f869af78700 5 -- op tracker -- , seq: 2721, time: 2014-06-09 16:22:11.698953, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.698960 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2b) v4 from osd.8 2014-06-09 16:22:11.698963 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311561c0 2014-06-09 16:22:11.698970 7f8692767700 2 waiting 51 > 50 ops || 823661 > 104857600 2014-06-09 16:22:11.698969 7f869af78700 5 -- op tracker -- , seq: 2721, time: 2014-06-09 16:22:11.698966, event: started, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.698975 7f868cf5c700 2 waiting 51 > 50 ops || 823661 > 104857600 2014-06-09 16:22:11.698979 7f8689755700 2 waiting 51 > 50 ops || 823877 > 104857600 2014-06-09 16:22:11.698967 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca485a0 (1 active) 2014-06-09 16:22:11.698985 7f868c75b700 2 waiting 51 > 50 ops || 822831 > 104857600 2014-06-09 16:22:11.698985 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6900 (2 active) 2014-06-09 16:22:11.698986 7f8692f68700 5 queue_transactions existing osr(3.730 0x3ed1ef0)/0x3ed1ef0 2014-06-09 16:22:11.698989 7f8692f68700 2 waiting 51 > 50 ops || 823504 > 104857600 2014-06-09 16:22:11.698990 7f86a5e11700 5 _do_op 0x308cb200 seq 3383 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 start 2014-06-09 16:22:11.698992 7f86a5610700 10 _finish_op 0x2fc8c780 seq 3373 osr(2.2d9 0x2ed0c30)/0x2ed0c30 2014-06-09 16:22:11.698991 7f868f761700 2 waiting 51 > 50 ops || 823113 > 104857600 2014-06-09 16:22:11.698983 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.698997 7f868ef60700 2 waiting 51 > 50 ops || 823207 > 104857600 2014-06-09 16:22:11.699001 7f868bf5a700 2 waiting 51 > 50 ops || 823207 > 104857600 2014-06-09 16:22:11.698993 7f86a5e11700 10 _do_transaction on 0xbebbbc0 2014-06-09 16:22:11.698998 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.699011 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad7a0 (1 active) 2014-06-09 16:22:11.699014 7f868cf5c700 2 waiting 51 > 50 ops || 823543 > 104857600 2014-06-09 16:22:11.699008 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.699014 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497440 (2 active) 2014-06-09 16:22:11.699019 7f8689755700 2 waiting 51 > 50 ops || 823759 > 104857600 2014-06-09 16:22:11.699019 7f86a5610700 5 _do_op 0x2ea53680 seq 3384 osr(3.e8 0x3ed0510)/0x3ed0510 start 2014-06-09 16:22:11.699022 7f86a5610700 10 _do_transaction on 0xbf8af00 2014-06-09 16:22:11.699017 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.699025 7f8692f68700 2 waiting 51 > 50 ops || 823386 > 104857600 2014-06-09 16:22:11.699028 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.699028 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.699030 7f868f761700 2 waiting 51 > 50 ops || 822995 > 104857600 2014-06-09 16:22:11.699031 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.699040 7f868ef60700 2 waiting 51 > 50 ops || 823089 > 104857600 2014-06-09 16:22:11.699046 7f868bf5a700 2 waiting 51 > 50 ops || 823089 > 104857600 2014-06-09 16:22:11.699052 7f868c75b700 2 waiting 51 > 50 ops || 822713 > 104857600 2014-06-09 16:22:11.699047 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.699057 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.699065 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.699072 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3384.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.699075 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.699069 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:2.12d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.699082 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.699092 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.699001 7f869af78700 5 -- op tracker -- , seq: 2721, time: 2014-06-09 16:22:11.698999, event: done, op: pg_info(1 pgs e243:2.2b) 2014-06-09 16:22:11.699098 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.699101 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.699097 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.699108 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.699120 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.699121 7f86a5610700 10 _do_op 0x2ea53680 seq 3384 r = 0, finisher 0x189aa660 0 2014-06-09 16:22:11.699123 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.699125 7f86a5610700 10 _finish_op 0x2ea53680 seq 3384 osr(3.e8 0x3ed0510)/0x3ed0510 2014-06-09 16:22:11.699130 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497440 (1 active) 2014-06-09 16:22:11.699133 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa000 (2 active) 2014-06-09 16:22:11.699127 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.247659 7 0.000614 2014-06-09 16:22:11.699135 7f86a5610700 5 _do_op 0x288f9920 seq 3385 osr(3.688 0x3ed1d20)/0x3ed1d20 start 2014-06-09 16:22:11.699135 7f8689755700 2 waiting 51 > 50 ops || 824310 > 104857600 2014-06-09 16:22:11.699138 7f86a5610700 10 _do_transaction on 0xc2f4300 2014-06-09 16:22:11.699142 7f8692f68700 2 waiting 51 > 50 ops || 823937 > 104857600 2014-06-09 16:22:11.699139 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.699145 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.699148 7f868f761700 2 waiting 51 > 50 ops || 823546 > 104857600 2014-06-09 16:22:11.699113 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.699151 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.699153 7f868ef60700 2 waiting 51 > 50 ops || 823640 > 104857600 2014-06-09 16:22:11.699153 7f869af78700 20 osd.26 243 _dispatch 0x3044d180 pg_info(1 pgs e243:2.6f) v4 2014-06-09 16:22:11.699157 7f868bf5a700 2 waiting 51 > 50 ops || 823640 > 104857600 2014-06-09 16:22:11.699151 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.699159 7f869af78700 5 -- op tracker -- , seq: 2722, time: 2014-06-09 16:22:11.549691, event: header_read, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.699162 7f868c75b700 2 waiting 51 > 50 ops || 823264 > 104857600 2014-06-09 16:22:11.699162 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.699166 7f869af78700 5 -- op tracker -- , seq: 2722, time: 2014-06-09 16:22:11.549691, event: throttled, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.699173 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.699172 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.699180 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.699191 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3383.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.699196 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.699210 7f869af78700 5 -- op tracker -- , seq: 2722, time: 2014-06-09 16:22:11.549703, event: all_read, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.699218 7f86a5e11700 10 _do_op 0x308cb200 seq 3383 r = 0, finisher 0xbd76d40 0 2014-06-09 16:22:11.699222 7f86a5e11700 10 _finish_op 0x308cb200 seq 3383 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 2014-06-09 16:22:11.699228 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.699232 7f8691f66700 5 queue_transactions existing osr(0.57b 0x2bd77b0)/0x2bd77b0 2014-06-09 16:22:11.699233 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6900 (1 active) 2014-06-09 16:22:11.699235 7f8691f66700 2 waiting 51 > 50 ops || 824517 > 104857600 2014-06-09 16:22:11.699236 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e120 (2 active) 2014-06-09 16:22:11.699229 7f8670072700 10 osd.26 243 new session 0xc055800 con=0x30ba0420 addr=172.16.96.220:0/1123636 2014-06-09 16:22:11.699239 7f86a5e11700 5 _do_op 0x309a20c0 seq 3386 osr(3.375 0x3ed1060)/0x3ed1060 start 2014-06-09 16:22:11.699239 7f868f761700 2 waiting 51 > 50 ops || 823843 > 104857600 2014-06-09 16:22:11.699241 7f8691765700 10 op_submit_finish 3427 2014-06-09 16:22:11.699241 7f86a5e11700 10 _do_transaction on 0xbf28300 2014-06-09 16:22:11.699245 7f868ef60700 2 waiting 51 > 50 ops || 823937 > 104857600 2014-06-09 16:22:11.699244 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.699249 7f868bf5a700 2 waiting 51 > 50 ops || 823937 > 104857600 2014-06-09 16:22:11.699255 7f868c75b700 2 waiting 51 > 50 ops || 823561 > 104857600 2014-06-09 16:22:11.699259 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.699262 7f8689755700 2 waiting 51 > 50 ops || 824607 > 104857600 2014-06-09 16:22:11.699261 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3385.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.699266 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.699248 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xbfca900 243'639 rep_tid=3 committed?=0 applied?=0 lock=0 op=osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.699275 7f8691765700 10 osd.26 243 dequeue_op 0x230da100 finish 2014-06-09 16:22:11.699284 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.699298 7f86a5610700 10 _do_op 0x288f9920 seq 3385 r = 0, finisher 0x3050bb00 0 2014-06-09 16:22:11.699302 7f86a5610700 10 _finish_op 0x288f9920 seq 3385 osr(3.688 0x3ed1d20)/0x3ed1d20 2014-06-09 16:22:11.699307 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358d10 (20 active) 2014-06-09 16:22:11.699308 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa000 (1 active) 2014-06-09 16:22:11.699310 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9226c0 (2 active) 2014-06-09 16:22:11.699312 7f86a5610700 5 _do_op 0x30d97200 seq 3387 osr(0.771 0x384c310)/0x384c310 start 2014-06-09 16:22:11.699295 7f868ff62700 10 osd.26 243 dequeue_op 0x257c5700 prio 63 cost 2064384 latency 0.965898 osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.699316 7f86a5610700 10 _do_transaction on 0xc317200 2014-06-09 16:22:11.699322 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.699312 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.699323 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.699327 7f868df5e700 10 op_submit_start 3428 2014-06-09 16:22:11.699330 7f868df5e700 5 queue_transactions (trailing journal) 3428 2014-06-09 16:22:11.699326 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.699332 7f868df5e700 5 queue_op 0xc7045a0 seq 3428 osr(2.4b9 0x2ed1730) 1883 bytes (queue has 49 ops and 821173 bytes) 2014-06-09 16:22:11.699336 7f868df5e700 10 op_submit_finish 3428 2014-06-09 16:22:11.699341 7f868e75f700 10 op_submit_start 3429 2014-06-09 16:22:11.699336 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.699343 7f868e75f700 5 queue_transactions (trailing journal) 3429 2014-06-09 16:22:11.699345 7f868e75f700 5 queue_op 0x30d97500 seq 3429 osr(3.709 0x3ed1bc0) 1053 bytes (queue has 49 ops and 821173 bytes) 2014-06-09 16:22:11.699349 7f868e75f700 10 op_submit_finish 3429 2014-06-09 16:22:11.699344 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.699342 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.6d8( v 213'642 (0'0,213'642] local-les=203 n=123 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.699354 7f868b759700 10 op_submit_start 3430 2014-06-09 16:22:11.699357 7f868b759700 5 queue_transactions (trailing journal) 3430 2014-06-09 16:22:11.699357 7f8691765700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.699358 7f868b759700 5 queue_op 0x308ca780 seq 3430 osr(1.384 0x384d5c0) 2001 bytes (queue has 49 ops and 821173 bytes) 2014-06-09 16:22:11.699361 7f868b759700 10 op_submit_finish 3430 2014-06-09 16:22:11.699355 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.699358 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.699371 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.699366 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.699369 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.699380 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.699373 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.699384 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.699377 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.0 1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.699377 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.699389 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699389 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.0 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.699392 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.699387 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.840941 7 0.000195 2014-06-09 16:22:11.699391 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.699395 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.390861 7 0.000160 2014-06-09 16:22:11.699397 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.699394 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) 2014-06-09 16:22:11.699401 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.699405 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.699408 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.699409 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.699408 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.699415 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.699416 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.699422 7f868df5e700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.699359 7f869af78700 5 -- op tracker -- , seq: 2722, time: 2014-06-09 16:22:11.699113, event: dispatched, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.699421 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.699424 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.699425 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.699434 7f869af78700 5 -- op tracker -- , seq: 2722, time: 2014-06-09 16:22:11.699431, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.699434 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.699431 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.699433 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.699440 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6f) v4 from osd.6 2014-06-09 16:22:11.699443 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044d180 2014-06-09 16:22:11.699439 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.699446 7f8691765700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.699442 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.699449 7f869af78700 5 -- op tracker -- , seq: 2722, time: 2014-06-09 16:22:11.699447, event: started, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.699454 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.699463 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:975 2014-06-09 16:22:11.699479 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699480 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.699486 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.699485 7f868e75f700 5 queue_transactions existing osr(2.1c6 0x2ed0870)/0x2ed0870 2014-06-09 16:22:11.699488 7f868e75f700 10 op_submit_start 3431 2014-06-09 16:22:11.699489 7f868e75f700 5 queue_transactions (trailing journal) 3431 2014-06-09 16:22:11.699490 7f868b759700 5 queue_transactions existing osr(1.3bc 0x384d550)/0x384d550 2014-06-09 16:22:11.699490 7f868e75f700 5 queue_op 0x309a2240 seq 3431 osr(2.1c6 0x2ed0870) 1997 bytes (queue has 50 ops and 823170 bytes) 2014-06-09 16:22:11.699492 7f868b759700 2 waiting 51 > 50 ops || 825053 > 104857600 2014-06-09 16:22:11.699492 7f868e75f700 10 op_submit_finish 3431 2014-06-09 16:22:11.699483 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.699496 7f8690f64700 10 op_submit_start 3432 2014-06-09 16:22:11.699497 7f8690f64700 5 queue_transactions (trailing journal) 3432 2014-06-09 16:22:11.699494 7f869af78700 5 -- op tracker -- , seq: 2722, time: 2014-06-09 16:22:11.699490, event: done, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.699499 7f8690f64700 5 queue_op 0x2fcaf7a0 seq 3432 osr(0.648 0x2bd7bd0) 2001 bytes (queue has 50 ops and 823170 bytes) 2014-06-09 16:22:11.699501 7f8690f64700 10 op_submit_finish 3432 2014-06-09 16:22:11.699495 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.699505 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.699499 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.699507 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.699511 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699506 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.699508 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.699521 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.699523 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.699517 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.699516 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.699524 7f869af78700 20 osd.26 243 _dispatch 0x2d094540 pg_info(1 pgs e243:0.681) v4 2014-06-09 16:22:11.699517 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.699529 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.699525 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.699532 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.699528 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.113( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.699537 7f868e75f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.699539 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699530 7f869af78700 5 -- op tracker -- , seq: 2723, time: 2014-06-09 16:22:11.574078, event: header_read, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.699536 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.330( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.478227 7 0.000194 2014-06-09 16:22:11.699546 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.330( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.699543 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.699545 7f869af78700 5 -- op tracker -- , seq: 2723, time: 2014-06-09 16:22:11.574079, event: throttled, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.699552 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.699559 7f869af78700 5 -- op tracker -- , seq: 2723, time: 2014-06-09 16:22:11.574090, event: all_read, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.699556 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.330( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.699556 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.699565 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.699566 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699565 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.699566 7f869af78700 5 -- op tracker -- , seq: 2723, time: 2014-06-09 16:22:11.699520, event: dispatched, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.699569 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.699574 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.699579 7f869af78700 5 -- op tracker -- , seq: 2723, time: 2014-06-09 16:22:11.699576, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.699571 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.699583 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.681) v4 from osd.0 2014-06-09 16:22:11.699580 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.699581 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.699587 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d094540 2014-06-09 16:22:11.699588 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.699585 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.699592 7f869af78700 5 -- op tracker -- , seq: 2723, time: 2014-06-09 16:22:11.699590, event: started, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.699598 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699595 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.699606 7f868e75f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.699602 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.699611 7f869af78700 5 -- op tracker -- , seq: 2723, time: 2014-06-09 16:22:11.699608, event: done, op: pg_info(1 pgs e243:0.681) 2014-06-09 16:22:11.699629 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.699632 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.699627 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.699644 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.699645 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.699647 7f869af78700 20 osd.26 243 _dispatch 0x3044ce00 pg_info(1 pgs e243:1.2d0) v4 2014-06-09 16:22:11.699641 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.699652 7f869af78700 5 -- op tracker -- , seq: 2724, time: 2014-06-09 16:22:11.549533, event: header_read, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.699651 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.699658 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.699654 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.699670 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699659 7f869af78700 5 -- op tracker -- , seq: 2724, time: 2014-06-09 16:22:11.549534, event: throttled, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.699664 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.699673 7f868df5e700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.699681 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.699676 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.699688 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.699698 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699690 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.699706 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.699702 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.699711 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.699714 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.699711 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=206 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.699723 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.699676 7f869af78700 5 -- op tracker -- , seq: 2724, time: 2014-06-09 16:22:11.549549, event: all_read, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.699723 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.699729 7f869af78700 5 -- op tracker -- , seq: 2724, time: 2014-06-09 16:22:11.699643, event: dispatched, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.699726 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=206 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.699732 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.699745 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.699740 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.699728 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 1 2014-06-09 16:22:11.699740 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=206 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.463707 7 0.000200 2014-06-09 16:22:11.699750 7f8690f64700 5 queue_transactions existing osr(0.330 0x2bd70a0)/0x2bd70a0 2014-06-09 16:22:11.699752 7f8690f64700 2 waiting 51 > 50 ops || 825053 > 104857600 2014-06-09 16:22:11.699753 7f868df5e700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.699751 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=206 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.699740 7f869af78700 5 -- op tracker -- , seq: 2724, time: 2014-06-09 16:22:11.699737, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.699751 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.1: 3.55d( v 213'582 (0'0,213'582] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) log((0'0,213'582], crt=0'0) missing(0) 2014-06-09 16:22:11.699762 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=206 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.699767 7f8691765700 10 proc_replica_log for osd.1: 3.55d( v 213'582 (0'0,213'582] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) log((0'0,213'582], crt=0'0) missing(0) 2014-06-09 16:22:11.699768 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.699772 7f868ff62700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.699316, event: reached_pg, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.699772 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=206 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.699782 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.699772 7f8691765700 20 merge_log cut point (usually last shared) is 213'582 (213'581) modify aa89755d/rbd_data.11776b8b4567.0000000000001717/head//3 by client.4567.0:34274 2014-06-09 10:17:51.796447 2014-06-09 16:22:11.699783 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'792 -> 213'792 2014-06-09 16:22:11.699763 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2d0) v4 from osd.5 2014-06-09 16:22:11.699782 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.699794 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044ce00 2014-06-09 16:22:11.699786 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.299( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.699796 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x257c5700 2014-06-09 16:22:11.699799 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.699803 7f869af78700 5 -- op tracker -- , seq: 2724, time: 2014-06-09 16:22:11.699798, event: started, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.699810 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.699806 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.699814 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:1.56c( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.699821 7f869af78700 5 -- op tracker -- , seq: 2724, time: 2014-06-09 16:22:11.699819, event: done, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.699829 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.699831 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.699826 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.699836 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.699829 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x305c6b00 2014-06-09 16:22:11.699844 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.699845 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.699839 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.699847 7f869af78700 20 osd.26 243 _dispatch 0x30cf88c0 pg_info(1 pgs e243:2.621) v4 2014-06-09 16:22:11.699852 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02a0 (19 active) 2014-06-09 16:22:11.699840 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x305c6b00 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 rwstate(write n=1 w=0) oi: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'638 client.4572.0:30632 wrlock_by=unknown.0.0:0 dirty s 217088 uv638) ssc: 0x2e8775e0 snapset: 0=[]:[]+head 2014-06-09 16:22:11.699852 7f869af78700 5 -- op tracker -- , seq: 2725, time: 2014-06-09 16:22:11.303171, event: header_read, op: pg_info(1 pgs e243:2.621) 2014-06-09 16:22:11.699859 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.699859 7f869af78700 5 -- op tracker -- , seq: 2725, time: 2014-06-09 16:22:11.303171, event: throttled, op: pg_info(1 pgs e243:2.621) 2014-06-09 16:22:11.699855 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 @head oi=5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'638 client.4572.0:30632 wrlock_by=unknown.0.0:0 dirty s 217088 uv638) 2014-06-09 16:22:11.699868 7f869af78700 5 -- op tracker -- , seq: 2725, time: 2014-06-09 16:22:11.303182, event: all_read, op: pg_info(1 pgs e243:2.621) 2014-06-09 16:22:11.699875 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.699875 7f869af78700 5 -- op tracker -- , seq: 2725, time: 2014-06-09 16:22:11.699843, event: dispatched, op: pg_info(1 pgs e243:2.621) 2014-06-09 16:22:11.699880 7f868e75f700 5 queue_transactions existing osr(3.9f 0x3ed0600)/0x3ed0600 2014-06-09 16:22:11.699883 7f868e75f700 2 waiting 51 > 50 ops || 824502 > 104857600 2014-06-09 16:22:11.699884 7f869af78700 5 -- op tracker -- , seq: 2725, time: 2014-06-09 16:22:11.699881, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.621) 2014-06-09 16:22:11.699888 7f8692767700 10 op_submit_start 3433 2014-06-09 16:22:11.699889 7f8692767700 5 queue_transactions (trailing journal) 3433 2014-06-09 16:22:11.699888 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.621) v4 from osd.2 2014-06-09 16:22:11.699891 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf88c0 2014-06-09 16:22:11.699891 7f8692767700 5 queue_op 0x30d96f60 seq 3433 osr(0.70e 0x384c0f0) 1883 bytes (queue has 50 ops and 823170 bytes) 2014-06-09 16:22:11.699894 7f8692767700 10 op_submit_finish 3433 2014-06-09 16:22:11.699897 7f869af78700 5 -- op tracker -- , seq: 2725, time: 2014-06-09 16:22:11.699894, event: started, op: pg_info(1 pgs e243:2.621) 2014-06-09 16:22:11.699900 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:2.573( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.699912 7f868ff62700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.699875, event: started, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.699914 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.699926 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.699919 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30724000 2014-06-09 16:22:11.699931 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.2b3( v 214'636 (0'0,214'636] local-les=213 n=123 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.699940 7f868ff62700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.699934, event: started, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.699945 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.699957 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.699946 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] ov 243'638 av 243'640 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.699962 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.221( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.699961 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 2014-06-09 16:22:11.699974 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.699975 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.699986 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.699988 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 217088~2064384 2014-06-09 16:22:11.699990 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:1.223( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.700003 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.700013 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700007 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229400 2014-06-09 16:22:11.700016 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.22d( v 214'683 (0'0,214'683] local-les=212 n=133 ec=118 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.700020 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 snapset=0x2e877618 snapc=0=[] 2014-06-09 16:22:11.700026 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.700033 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.700044 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700045 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 0x30724000 op modify 2014-06-09 16:22:11.700049 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.1b1( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.700060 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.700058 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:19.375622 2014-06-09 16:22:11.700071 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700076 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.700078 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.700089 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.700098 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700098 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.700106 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700119 7f8692767700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.700110 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 4 on osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.700127 7f868ff62700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 4 o 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.700134 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.700150 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700154 7f868ff62700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.700148, event: waiting for subops from 7, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.700155 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.700168 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.700177 7f868ff62700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.700177 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700173 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.1 now 3.55d( v 213'582 (0'0,213'582] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.700185 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700197 7f8692767700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.700191 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.700202 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.716654 7 0.000307 2014-06-09 16:22:11.700199 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'639 (0'0,243'639] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'639], crt=0'0) [243'640 (243'638) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30633 2014-06-09 10:20:19.375622] 2014-06-09 16:22:11.700211 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.700213 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.700225 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700222 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.219063 0 0.000000 2014-06-09 16:22:11.700221 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'640 (243'638) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30633 2014-06-09 10:20:19.375622 2014-06-09 16:22:11.700234 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.700230 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.582( v 213'598 (0'0,213'598] local-les=191 n=118 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.700243 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.700242 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.700245 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.700254 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.700262 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.700257 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.582( v 213'598 (0'0,213'598] local-les=191 n=118 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.700274 7f868ff62700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'640, trimmed: 2014-06-09 16:22:11.700270 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'593 -> 214'593 2014-06-09 16:22:11.700279 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700283 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.2c2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.700293 7f868ff62700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.700296 7f868ff62700 2 waiting 51 > 50 ops || 2889838 > 104857600 2014-06-09 16:22:11.700295 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.700306 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700304 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.55d( v 213'582 (0'0,213'582] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.700312 7f869af78700 5 -- op tracker -- , seq: 2725, time: 2014-06-09 16:22:11.700309, event: done, op: pg_info(1 pgs e243:2.621) 2014-06-09 16:22:11.700320 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.700322 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.700322 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f23c0 (19 active) 2014-06-09 16:22:11.700327 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 sending log((213'582,214'593], crt=0'0) 2014-06-09 16:22:11.700338 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.700338 7f8691765700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.700339 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.700340 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f23c0 (20 active) 2014-06-09 16:22:11.700341 7f869af78700 20 osd.26 243 _dispatch 0x48fd600 pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.700345 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.700352 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.55d( v 214'593 lc 213'582 (0'0,214'593] local-les=191 n=112 ec=118 les/c 191/191 242/242/242) missing missing(2) 2014-06-09 16:22:11.700356 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.700365 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.1 has 2 missing 2014-06-09 16:22:11.700366 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700374 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700374 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing d889555d/rbd_data.117b6b8b4567.00000000000017ea/head//3 214'590 is on osd.26 2014-06-09 16:22:11.700383 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.700384 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing bcc3a55d/rbd_data.11676b8b4567.000000000000176e/head//3 214'593 is on osd.26 2014-06-09 16:22:11.700348 7f869af78700 5 -- op tracker -- , seq: 2726, time: 2014-06-09 16:22:11.422936, event: header_read, op: pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.700397 7f869af78700 5 -- op tracker -- , seq: 2726, time: 2014-06-09 16:22:11.422936, event: throttled, op: pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.700393 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {d889555d/rbd_data.117b6b8b4567.00000000000017ea/head//3=214'590,bcc3a55d/rbd_data.11676b8b4567.000000000000176e/head//3=214'593} 2014-06-09 16:22:11.700398 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.700405 7f869af78700 5 -- op tracker -- , seq: 2726, time: 2014-06-09 16:22:11.422948, event: all_read, op: pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.700404 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 1 as a source 2014-06-09 16:22:11.700414 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700413 7f869af78700 5 -- op tracker -- , seq: 2726, time: 2014-06-09 16:22:11.700337, event: dispatched, op: pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.700414 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing d889555d/rbd_data.117b6b8b4567.00000000000017ea/head//3 214'590 also missing on osd.1 2014-06-09 16:22:11.700421 7f869af78700 5 -- op tracker -- , seq: 2726, time: 2014-06-09 16:22:11.700418, event: waiting_for_osdmap, op: pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.700418 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.700426 7f869af78700 7 osd.26 243 handle_pg_log pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.700430 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x48fd600 2014-06-09 16:22:11.700425 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing bcc3a55d/rbd_data.11676b8b4567.000000000000176e/head//3 214'593 also missing on osd.1 2014-06-09 16:22:11.700431 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.700437 7f869af78700 5 -- op tracker -- , seq: 2726, time: 2014-06-09 16:22:11.700434, event: started, op: pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.700434 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {d889555d/rbd_data.117b6b8b4567.00000000000017ea/head//3=214'590,bcc3a55d/rbd_data.11676b8b4567.000000000000176e/head//3=214'593} 2014-06-09 16:22:11.700441 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700444 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.700450 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700452 7f869af78700 5 -- op tracker -- , seq: 2726, time: 2014-06-09 16:22:11.700449, event: done, op: pg_log(1.76e epoch 243 log log((0'0,0'0], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.700452 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.700461 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.700461 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.700463 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.700462 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 1 2014-06-09 16:22:11.700476 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.700470 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.700477 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.700479 7f869af78700 20 osd.26 243 _dispatch 0x30cf1c00 pg_info(1 pgs e243:2.5d9) v4 2014-06-09 16:22:11.700473 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1059 2014-06-09 16:22:11.700478 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.700484 7f869af78700 5 -- op tracker -- , seq: 2727, time: 2014-06-09 16:22:11.577905, event: header_read, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.700491 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700492 7f869af78700 5 -- op tracker -- , seq: 2727, time: 2014-06-09 16:22:11.577906, event: throttled, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.700498 7f869af78700 5 -- op tracker -- , seq: 2727, time: 2014-06-09 16:22:11.577919, event: all_read, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.700495 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.700505 7f869af78700 5 -- op tracker -- , seq: 2727, time: 2014-06-09 16:22:11.700475, event: dispatched, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.700508 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.700518 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700525 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700537 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.700553 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:151 2014-06-09 16:22:11.700566 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.700569 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700571 7f8691765700 5 queue_transactions existing osr(3.55d 0x3ed1940)/0x3ed1940 2014-06-09 16:22:11.700574 7f8691765700 2 waiting 51 > 50 ops || 824223 > 104857600 2014-06-09 16:22:11.700512 7f869af78700 5 -- op tracker -- , seq: 2727, time: 2014-06-09 16:22:11.700509, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.700578 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5d9) v4 from osd.7 2014-06-09 16:22:11.700574 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.700583 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf1c00 2014-06-09 16:22:11.700585 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.700588 7f869af78700 5 -- op tracker -- , seq: 2727, time: 2014-06-09 16:22:11.700586, event: started, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.700591 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700598 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700610 7f8691f66700 2 waiting 51 > 50 ops || 825179 > 104857600 2014-06-09 16:22:11.700610 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.700615 7f868f761700 2 waiting 51 > 50 ops || 824505 > 104857600 2014-06-09 16:22:11.700614 7f869af78700 5 -- op tracker -- , seq: 2727, time: 2014-06-09 16:22:11.700605, event: done, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.700619 7f868ef60700 2 waiting 51 > 50 ops || 824599 > 104857600 2014-06-09 16:22:11.700623 7f868bf5a700 2 waiting 51 > 50 ops || 824599 > 104857600 2014-06-09 16:22:11.700628 7f868c75b700 2 waiting 51 > 50 ops || 824223 > 104857600 2014-06-09 16:22:11.700624 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:151 2014-06-09 16:22:11.700636 7f8689755700 2 waiting 51 > 50 ops || 825269 > 104857600 2014-06-09 16:22:11.700639 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700643 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.700654 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.700665 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700674 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700687 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.700687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.700690 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.700701 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1139 2014-06-09 16:22:11.700716 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700721 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.700733 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.700740 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700744 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700750 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.700752 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.700754 7f869af78700 20 osd.26 243 _dispatch 0x2f030000 pg_info(1 pgs e243:0.3b6) v4 2014-06-09 16:22:11.700755 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.700760 7f869af78700 5 -- op tracker -- , seq: 2728, time: 2014-06-09 16:22:11.549661, event: header_read, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.700767 7f869af78700 5 -- op tracker -- , seq: 2728, time: 2014-06-09 16:22:11.549661, event: throttled, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.700774 7f869af78700 5 -- op tracker -- , seq: 2728, time: 2014-06-09 16:22:11.549673, event: all_read, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.700769 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.700784 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700788 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.700801 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.700810 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700781 7f869af78700 5 -- op tracker -- , seq: 2728, time: 2014-06-09 16:22:11.700749, event: dispatched, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.700818 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700825 7f869af78700 5 -- op tracker -- , seq: 2728, time: 2014-06-09 16:22:11.700823, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.700829 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.700831 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3b6) v4 from osd.3 2014-06-09 16:22:11.700834 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f030000 2014-06-09 16:22:11.700843 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.700858 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700862 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.700875 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.700884 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700893 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700905 7f8692767700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.700898 7f869af78700 5 -- op tracker -- , seq: 2728, time: 2014-06-09 16:22:11.700895, event: started, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.700919 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:166 2014-06-09 16:22:11.700933 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.700938 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.700932 7f869af78700 5 -- op tracker -- , seq: 2728, time: 2014-06-09 16:22:11.700929, event: done, op: pg_info(1 pgs e243:0.3b6) 2014-06-09 16:22:11.700954 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.700957 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.700949 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.700959 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.700968 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.700976 7f8692767700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.700978 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.700980 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.700981 7f869af78700 20 osd.26 243 _dispatch 0x30dc36c0 pg_info(1 pgs e243:3.39e) v4 2014-06-09 16:22:11.700986 7f869af78700 5 -- op tracker -- , seq: 2729, time: 2014-06-09 16:22:11.563654, event: header_read, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.700987 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:189 2014-06-09 16:22:11.700994 7f869af78700 5 -- op tracker -- , seq: 2729, time: 2014-06-09 16:22:11.563654, event: throttled, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.701002 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.701000 7f869af78700 5 -- op tracker -- , seq: 2729, time: 2014-06-09 16:22:11.563665, event: all_read, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.701011 7f869af78700 5 -- op tracker -- , seq: 2729, time: 2014-06-09 16:22:11.700977, event: dispatched, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.701006 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:11.701019 7f869af78700 5 -- op tracker -- , seq: 2729, time: 2014-06-09 16:22:11.701017, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.701023 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.39e) v4 from osd.1 2014-06-09 16:22:11.701026 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc36c0 2014-06-09 16:22:11.701019 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.701031 7f869af78700 5 -- op tracker -- , seq: 2729, time: 2014-06-09 16:22:11.701029, event: started, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.701030 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.701040 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.701052 7f8692767700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.701057 7f869af78700 5 -- op tracker -- , seq: 2729, time: 2014-06-09 16:22:11.701054, event: done, op: pg_info(1 pgs e243:3.39e) 2014-06-09 16:22:11.701065 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.701067 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.701062 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:986 2014-06-09 16:22:11.701078 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.701087 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.701089 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.701082 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=191 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.701091 7f869af78700 20 osd.26 243 _dispatch 0xddc5c00 pg_info(1 pgs e243:3.609) v4 2014-06-09 16:22:11.701097 7f869af78700 5 -- op tracker -- , seq: 2730, time: 2014-06-09 16:22:11.576231, event: header_read, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.701094 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=191 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.701106 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=191 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.958479 7 0.004467 2014-06-09 16:22:11.701118 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=191 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.701131 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=191 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.701142 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=191 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.701104 7f869af78700 5 -- op tracker -- , seq: 2730, time: 2014-06-09 16:22:11.576231, event: throttled, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.701153 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'625 -> 213'625 2014-06-09 16:22:11.701164 7f869af78700 5 -- op tracker -- , seq: 2730, time: 2014-06-09 16:22:11.576243, event: all_read, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.701171 7f869af78700 5 -- op tracker -- , seq: 2730, time: 2014-06-09 16:22:11.701087, event: dispatched, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.701178 7f869af78700 5 -- op tracker -- , seq: 2730, time: 2014-06-09 16:22:11.701176, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.701182 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.609) v4 from osd.9 2014-06-09 16:22:11.701186 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc5c00 2014-06-09 16:22:11.701191 7f869af78700 5 -- op tracker -- , seq: 2730, time: 2014-06-09 16:22:11.701189, event: started, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.701194 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.701239 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.701244 7f8692767700 5 queue_transactions existing osr(3.251 0x3ed0c20)/0x3ed0c20 2014-06-09 16:22:11.701247 7f8692767700 2 waiting 51 > 50 ops || 824599 > 104857600 2014-06-09 16:22:11.701252 7f868cf5c700 10 op_submit_start 3434 2014-06-09 16:22:11.701254 7f868cf5c700 5 queue_transactions (trailing journal) 3434 2014-06-09 16:22:11.701255 7f868cf5c700 5 queue_op 0x30d965a0 seq 3434 osr(0.448 0x2bd7190) 1883 bytes (queue has 50 ops and 823170 bytes) 2014-06-09 16:22:11.701259 7f868cf5c700 10 op_submit_finish 3434 2014-06-09 16:22:11.701265 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.701279 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.701294 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.701299 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.701304 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.245015 7 0.000203 2014-06-09 16:22:11.701319 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.701331 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.701341 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.701350 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.701356 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.701407 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.701412 7f868cf5c700 5 queue_transactions existing osr(2.5a4 0x2ed1810)/0x2ed1810 2014-06-09 16:22:11.701414 7f868cf5c700 2 waiting 51 > 50 ops || 825073 > 104857600 2014-06-09 16:22:11.701419 7f8692f68700 10 op_submit_start 3435 2014-06-09 16:22:11.701421 7f8692f68700 5 queue_transactions (trailing journal) 3435 2014-06-09 16:22:11.701422 7f8692f68700 5 queue_op 0x2ea53920 seq 3435 osr(3.730 0x3ed1ef0) 1726 bytes (queue has 50 ops and 823170 bytes) 2014-06-09 16:22:11.701425 7f8692f68700 10 op_submit_finish 3435 2014-06-09 16:22:11.701432 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.701436 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0xb184000 (0 active) 2014-06-09 16:22:11.701445 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.701444 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.701454 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x19fe0a00,0x306a400,0x19fe0900,0x306aa00,0x19fe0800,0x306a800,0x19fe0700,0xe033200,0x19fe0600,0xe033100,0x1c66b200,0xe033000 2014-06-09 16:22:11.701495 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.701507 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.701517 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.701533 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:963 2014-06-09 16:22:11.701543 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.701553 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.701564 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.701569 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.701581 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.701591 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.701599 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.701611 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.701627 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.701643 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.701648 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 1 notify: (query_epoch:242, epoch_sent:243, info:0.1fb( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.701660 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.701671 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.701675 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.701687 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.701694 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.701701 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.701713 7f8692f68700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.701724 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.701740 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.701744 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.701757 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.701766 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.701773 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.701785 7f8692f68700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.701800 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.701814 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.701826 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.701840 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.701849 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.701857 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.701868 7f8692f68700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.701882 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.701896 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.701901 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.701913 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.701921 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.701929 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.701942 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.701955 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.701971 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.705341 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.4d7( v 214'562 (0'0,214'562] local-les=212 n=109 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.701976 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3386.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.705358 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.705366 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.705366 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.705370 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.d( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.705377 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.705383 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.705386 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.705398 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.705402 7f86a5e11700 10 _do_op 0x309a20c0 seq 3386 r = 0, finisher 0xc411880 0 2014-06-09 16:22:11.705403 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.705408 7f86a5e11700 10 _finish_op 0x309a20c0 seq 3386 osr(3.375 0x3ed1060)/0x3ed1060 2014-06-09 16:22:11.705407 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.705416 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e120 (1 active) 2014-06-09 16:22:11.705419 7f8692f68700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.705420 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd320 (2 active) 2014-06-09 16:22:11.705420 7f868b759700 10 op_submit_start 3436 2014-06-09 16:22:11.705423 7f868b759700 5 queue_transactions (trailing journal) 3436 2014-06-09 16:22:11.705423 7f86a5e11700 5 _do_op 0x2ea52a20 seq 3388 osr(0.493 0x2bd73d0)/0x2bd73d0 start 2014-06-09 16:22:11.705424 7f868b759700 5 queue_op 0x308ca720 seq 3436 osr(1.3bc 0x384d550) 1883 bytes (queue has 50 ops and 823718 bytes) 2014-06-09 16:22:11.705426 7f86a5e11700 10 _do_transaction on 0xc336b40 2014-06-09 16:22:11.705428 7f868b759700 10 op_submit_finish 3436 2014-06-09 16:22:11.705436 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.705432 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.705438 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.705451 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.705444 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.705456 7f868b759700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.705460 7f868b759700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.705455 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.705466 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.705465 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.286( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.397505 7 0.000199 2014-06-09 16:22:11.705473 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.705479 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.705480 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.286( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.705490 7f8692f68700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.705491 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.286( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.705504 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:199 2014-06-09 16:22:11.705521 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.705521 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.705522 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.705526 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.705533 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.705536 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.705540 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.705544 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.705551 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.705561 7f8692f68700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.705564 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3387.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.705570 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.705575 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:161 2014-06-09 16:22:11.705588 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.705589 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.705594 7f868b759700 5 queue_transactions existing osr(1.286 0x384ced0)/0x384ced0 2014-06-09 16:22:11.705596 7f868b759700 2 waiting 51 > 50 ops || 825727 > 104857600 2014-06-09 16:22:11.705591 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.705601 7f8690f64700 2 waiting 51 > 50 ops || 825601 > 104857600 2014-06-09 16:22:11.705601 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.705612 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.705615 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.705616 7f868e75f700 2 waiting 51 > 50 ops || 825050 > 104857600 2014-06-09 16:22:11.705619 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.179( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.961393 7 0.000165 2014-06-09 16:22:11.705626 7f868ff62700 2 waiting 51 > 50 ops || 2890386 > 104857600 2014-06-09 16:22:11.705629 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.179( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.705633 7f8691765700 2 waiting 51 > 50 ops || 824771 > 104857600 2014-06-09 16:22:11.705640 7f86a5610700 10 _do_op 0x30d97200 seq 3387 r = 0, finisher 0xbd6bf80 0 2014-06-09 16:22:11.705639 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.179( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.705642 7f8691f66700 2 waiting 51 > 50 ops || 825727 > 104857600 2014-06-09 16:22:11.705645 7f86a5610700 10 _finish_op 0x30d97200 seq 3387 osr(0.771 0x384c310)/0x384c310 2014-06-09 16:22:11.705645 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.705655 7f868ef60700 2 waiting 51 > 50 ops || 825273 > 104857600 2014-06-09 16:22:11.705651 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.705659 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.705673 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9226c0 (1 active) 2014-06-09 16:22:11.705675 7f868b759700 10 op_submit_start 3437 2014-06-09 16:22:11.705677 7f868b759700 5 queue_transactions (trailing journal) 3437 2014-06-09 16:22:11.705677 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524120 (2 active) 2014-06-09 16:22:11.705678 7f868b759700 5 queue_op 0x308ca6c0 seq 3437 osr(1.286 0x384ced0) 2009 bytes (queue has 50 ops and 823844 bytes) 2014-06-09 16:22:11.705679 7f86a5610700 5 _do_op 0x30032660 seq 3389 osr(0.3fe 0x2bd7210)/0x2bd7210 start 2014-06-09 16:22:11.705681 7f868b759700 10 op_submit_finish 3437 2014-06-09 16:22:11.705680 7f868bf5a700 2 waiting 51 > 50 ops || 825273 > 104857600 2014-06-09 16:22:11.705683 7f86a5610700 10 _do_transaction on 0xbe88a80 2014-06-09 16:22:11.705688 7f8689755700 2 waiting 51 > 50 ops || 825943 > 104857600 2014-06-09 16:22:11.705690 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.705701 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.705702 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02f0 (19 active) 2014-06-09 16:22:11.705705 7f8692f68700 5 queue_transactions existing osr(0.179 0x2bd6800)/0x2bd6800 2014-06-09 16:22:11.705708 7f8692f68700 2 waiting 51 > 50 ops || 825727 > 104857600 2014-06-09 16:22:11.705709 7f868b759700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.705713 7f8692767700 2 waiting 51 > 50 ops || 825273 > 104857600 2014-06-09 16:22:11.705719 7f868cf5c700 2 waiting 51 > 50 ops || 825747 > 104857600 2014-06-09 16:22:11.705724 7f8690f64700 2 waiting 51 > 50 ops || 825727 > 104857600 2014-06-09 16:22:11.705726 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.705730 7f868e75f700 2 waiting 51 > 50 ops || 825176 > 104857600 2014-06-09 16:22:11.705743 7f868ff62700 2 waiting 51 > 50 ops || 2890512 > 104857600 2014-06-09 16:22:11.705743 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3389.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.705748 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.705757 7f8691765700 2 waiting 51 > 50 ops || 824897 > 104857600 2014-06-09 16:22:11.705762 7f8691f66700 2 waiting 51 > 50 ops || 825853 > 104857600 2014-06-09 16:22:11.705765 7f86a5610700 10 _do_op 0x30032660 seq 3389 r = 0, finisher 0xbd7ca20 0 2014-06-09 16:22:11.705768 7f868f761700 2 waiting 51 > 50 ops || 825179 > 104857600 2014-06-09 16:22:11.705769 7f86a5610700 10 _finish_op 0x30032660 seq 3389 osr(0.3fe 0x2bd7210)/0x2bd7210 2014-06-09 16:22:11.705795 7f868f761700 2 waiting 51 > 50 ops || 824725 > 104857600 2014-06-09 16:22:11.705796 7f868ef60700 10 op_submit_start 3438 2014-06-09 16:22:11.705795 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524120 (1 active) 2014-06-09 16:22:11.705798 7f868ef60700 5 queue_transactions (trailing journal) 3438 2014-06-09 16:22:11.705798 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca499e0 (2 active) 2014-06-09 16:22:11.705799 7f8689755700 2 waiting 51 > 50 ops || 825489 > 104857600 2014-06-09 16:22:11.705800 7f86a5610700 5 _do_op 0x308caf60 seq 3390 osr(3.13c 0x3ed0760)/0x3ed0760 start 2014-06-09 16:22:11.705799 7f868ef60700 5 queue_op 0x2c06efc0 seq 3438 osr(3.408 0x3ed1270) 1429 bytes (queue has 50 ops and 823390 bytes) 2014-06-09 16:22:11.705804 7f868ef60700 10 op_submit_finish 3438 2014-06-09 16:22:11.705803 7f86a5610700 10 _do_transaction on 0xbebc900 2014-06-09 16:22:11.705805 7f8692f68700 2 waiting 51 > 50 ops || 825273 > 104857600 2014-06-09 16:22:11.705809 7f8692767700 2 waiting 51 > 50 ops || 824819 > 104857600 2014-06-09 16:22:11.705810 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.705814 7f868cf5c700 2 waiting 51 > 50 ops || 825293 > 104857600 2014-06-09 16:22:11.705807 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.5d3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.705818 7f8690f64700 2 waiting 51 > 50 ops || 825273 > 104857600 2014-06-09 16:22:11.705823 7f868e75f700 2 waiting 51 > 50 ops || 824722 > 104857600 2014-06-09 16:22:11.705827 7f868ff62700 2 waiting 51 > 50 ops || 2890058 > 104857600 2014-06-09 16:22:11.705820 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.705832 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.705832 7f8691765700 2 waiting 51 > 50 ops || 824443 > 104857600 2014-06-09 16:22:11.705837 7f8691f66700 2 waiting 51 > 50 ops || 825399 > 104857600 2014-06-09 16:22:11.705841 7f868c75b700 2 waiting 51 > 50 ops || 824443 > 104857600 2014-06-09 16:22:11.705844 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.705838 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=210 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.705852 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=210 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.705862 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3390.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.705867 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.705865 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=210 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.728157 7 0.000160 2014-06-09 16:22:11.705877 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=210 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.705888 7f86a5610700 10 _do_op 0x308caf60 seq 3390 r = 0, finisher 0x33a7380 0 2014-06-09 16:22:11.705892 7f86a5610700 10 _finish_op 0x308caf60 seq 3390 osr(3.13c 0x3ed0760)/0x3ed0760 2014-06-09 16:22:11.705888 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=210 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.705903 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca499e0 (1 active) 2014-06-09 16:22:11.705905 7f8689755700 10 op_submit_start 3439 2014-06-09 16:22:11.705906 7f8692f68700 2 waiting 51 > 50 ops || 826037 > 104857600 2014-06-09 16:22:11.705906 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4aca20 (2 active) 2014-06-09 16:22:11.705907 7f8689755700 5 queue_transactions (trailing journal) 3439 2014-06-09 16:22:11.705900 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=210 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.705908 7f86a5610700 5 _do_op 0x2ea52b40 seq 3391 osr(2.40c 0x2ed1250)/0x2ed1250 start 2014-06-09 16:22:11.705911 7f86a5610700 10 _do_transaction on 0xc22aa80 2014-06-09 16:22:11.705909 7f8689755700 5 queue_op 0x15996d20 seq 3439 osr(1.4bb 0x384d970) 2099 bytes (queue has 50 ops and 824154 bytes) 2014-06-09 16:22:11.705912 7f8689755700 10 op_submit_finish 3439 2014-06-09 16:22:11.705917 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.705911 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'605 -> 213'605 2014-06-09 16:22:11.705915 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.705926 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.705934 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.705937 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:3.78a( v 213'545 (0'0,213'545] local-les=191 n=110 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.705946 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.705950 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.705954 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.705949 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.705956 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:2.1fa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.705964 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.705969 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.705966 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3391.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.705971 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.705972 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 1 notify: (query_epoch:243, epoch_sent:243, info:1.4c2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.705978 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.705983 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.705986 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.705991 7f86a5610700 10 _do_op 0x2ea52b40 seq 3391 r = 0, finisher 0xc2d6c40 0 2014-06-09 16:22:11.705993 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.705992 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.705995 7f86a5610700 10 _finish_op 0x2ea52b40 seq 3391 osr(2.40c 0x2ed1250)/0x2ed1250 2014-06-09 16:22:11.705997 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.705999 7f868ef60700 5 queue_transactions existing osr(3.4c5 0x3ed1430)/0x3ed1430 2014-06-09 16:22:11.706001 7f868ef60700 10 op_submit_start 3440 2014-06-09 16:22:11.706001 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4aca20 (1 active) 2014-06-09 16:22:11.706003 7f868ef60700 5 queue_transactions (trailing journal) 3440 2014-06-09 16:22:11.706002 7f8692f68700 2 waiting 51 > 50 ops || 825363 > 104857600 2014-06-09 16:22:11.706004 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb7560 (2 active) 2014-06-09 16:22:11.706004 7f868ef60700 5 queue_op 0x2c06ef00 seq 3440 osr(3.4c5 0x3ed1430) 1335 bytes (queue has 50 ops and 823480 bytes) 2014-06-09 16:22:11.706007 7f868ef60700 10 op_submit_finish 3440 2014-06-09 16:22:11.706006 7f86a5610700 5 _do_op 0x308caf00 seq 3392 osr(0.1d4 0x2bd6940)/0x2bd6940 start 2014-06-09 16:22:11.705999 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.706008 7f868cf5c700 2 waiting 51 > 50 ops || 825383 > 104857600 2014-06-09 16:22:11.706009 7f86a5610700 10 _do_transaction on 0x36b3e00 2014-06-09 16:22:11.706012 7f8690f64700 2 waiting 51 > 50 ops || 825363 > 104857600 2014-06-09 16:22:11.706010 7f868ef60700 20 osd.26 243 Sending info 3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) to shard 9 2014-06-09 16:22:11.706015 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.706016 7f868e75f700 2 waiting 51 > 50 ops || 824812 > 104857600 2014-06-09 16:22:11.706011 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.706018 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.706019 7f868ff62700 2 waiting 51 > 50 ops || 2890148 > 104857600 2014-06-09 16:22:11.706023 7f8691765700 2 waiting 51 > 50 ops || 824533 > 104857600 2014-06-09 16:22:11.706020 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.706028 7f8691f66700 2 waiting 51 > 50 ops || 825489 > 104857600 2014-06-09 16:22:11.706031 7f868c75b700 2 waiting 51 > 50 ops || 824533 > 104857600 2014-06-09 16:22:11.706035 7f868f761700 2 waiting 51 > 50 ops || 824815 > 104857600 2014-06-09 16:22:11.706029 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.706038 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.706044 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.706047 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.706053 7f8692767700 2 waiting 51 > 50 ops || 824909 > 104857600 2014-06-09 16:22:11.706053 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41ef0 (19 active) 2014-06-09 16:22:11.706064 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3392.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.706068 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.706075 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:11.706084 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.706082 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706091 7f8689755700 20 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.706095 7f8689755700 20 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.706101 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.706096 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.706090 7f86a5610700 10 _do_op 0x308caf00 seq 3392 r = 0, finisher 0x314ac380 0 2014-06-09 16:22:11.706105 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706104 7f86a5610700 10 _finish_op 0x308caf00 seq 3392 osr(0.1d4 0x2bd6940)/0x2bd6940 2014-06-09 16:22:11.706104 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706112 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb7560 (1 active) 2014-06-09 16:22:11.706113 7f8692f68700 10 op_submit_start 3441 2014-06-09 16:22:11.706115 7f8692f68700 5 queue_transactions (trailing journal) 3441 2014-06-09 16:22:11.706114 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb6a20 (2 active) 2014-06-09 16:22:11.706117 7f86a5610700 5 _do_op 0x300326c0 seq 3393 osr(0.1fd 0x2bd6b30)/0x2bd6b30 start 2014-06-09 16:22:11.706117 7f8692f68700 5 queue_op 0xc2ea900 seq 3441 osr(0.179 0x2bd6800) 1883 bytes (queue has 50 ops and 823480 bytes) 2014-06-09 16:22:11.706109 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706113 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.706120 7f8692f68700 10 op_submit_finish 3441 2014-06-09 16:22:11.706120 7f86a5610700 10 _do_transaction on 0xc3ec240 2014-06-09 16:22:11.706122 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.706126 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.706122 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.706132 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706126 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706122 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.573( v 213'607 (0'0,213'607] local-les=203 n=120 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.706136 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.67( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.706136 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.706145 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.706147 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.706137 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706149 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.706151 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.706152 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:3.2ac( v 213'663 (0'0,213'663] local-les=203 n=133 ec=118 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.706160 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.706163 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706160 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.706163 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.706166 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.706173 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.706167 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706176 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.706177 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3393.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.706174 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.706181 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706182 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.706177 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.706182 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.706184 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706189 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.706189 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.706199 7f86a5610700 10 _do_op 0x300326c0 seq 3393 r = 0, finisher 0x31479d80 0 2014-06-09 16:22:11.706203 7f86a5610700 10 _finish_op 0x300326c0 seq 3393 osr(0.1fd 0x2bd6b30)/0x2bd6b30 2014-06-09 16:22:11.706198 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706195 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.706205 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb6a20 (1 active) 2014-06-09 16:22:11.706200 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.706208 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706208 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fdd40 (2 active) 2014-06-09 16:22:11.706210 7f86a5610700 5 _do_op 0x30d97e00 seq 3394 osr(1.14c 0x384c810)/0x384c810 start 2014-06-09 16:22:11.706206 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706213 7f86a5610700 10 _do_transaction on 0xc20f200 2014-06-09 16:22:11.706209 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.706218 7f8692f68700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.706216 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.706212 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706224 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.706227 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.706228 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706239 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706232 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1056 2014-06-09 16:22:11.706245 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.706239 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.706248 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.706251 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.706243 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.706248 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.706251 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.706256 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.706259 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.706262 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.458( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.706270 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.706267 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706266 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706274 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706275 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706284 7f8692f68700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.706285 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0200 (19 active) 2014-06-09 16:22:11.706285 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.706287 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.706291 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.706298 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:179 2014-06-09 16:22:11.706311 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.706295 7f8689755700 10 osd.26 243 dequeue_op 0x306a400 prio 63 cost 425984 latency 1.770516 osd_op(client.4573.0:33015 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~425984] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.706304 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:185 2014-06-09 16:22:11.706321 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706315 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.706327 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3394.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.706327 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.706332 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.706325 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.706333 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706338 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706345 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.706338 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.706347 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706352 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1221 2014-06-09 16:22:11.706362 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.706355 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706364 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=212 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) 2014-06-09 16:22:11.706368 7f868ef60700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.706370 7f86a5610700 10 _do_op 0x30d97e00 seq 3394 r = 0, finisher 0x307f07e0 0 2014-06-09 16:22:11.706374 7f86a5610700 10 _finish_op 0x30d97e00 seq 3394 osr(1.14c 0x384c810)/0x384c810 2014-06-09 16:22:11.706372 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=212 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) 2014-06-09 16:22:11.706377 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fdd40 (1 active) 2014-06-09 16:22:11.706380 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525320 (2 active) 2014-06-09 16:22:11.706379 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=212 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.803087 7 0.000192 2014-06-09 16:22:11.706386 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=212 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.706385 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.706393 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=212 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.706401 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706399 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=212 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.706382 7f86a5610700 5 _do_op 0x10039740 seq 3395 osr(3.527 0x3ed1660)/0x3ed1660 start 2014-06-09 16:22:11.706404 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'642 -> 213'642 2014-06-09 16:22:11.706409 7f86a5610700 10 _do_transaction on 0xc0d7e00 2014-06-09 16:22:11.706405 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.706416 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.706419 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.706428 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706436 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706439 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.706448 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.706450 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.706471 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.706473 7f8692f68700 5 queue_transactions existing osr(3.4c8 0x3ed13c0)/0x3ed13c0 2014-06-09 16:22:11.706475 7f8692f68700 10 op_submit_start 3442 2014-06-09 16:22:11.706476 7f8692f68700 5 queue_transactions (trailing journal) 3442 2014-06-09 16:22:11.706467 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.706476 7f8692f68700 5 queue_op 0xc2eacc0 seq 3442 osr(3.4c8 0x3ed13c0) 1335 bytes (queue has 49 ops and 820852 bytes) 2014-06-09 16:22:11.706478 7f8692f68700 10 op_submit_finish 3442 2014-06-09 16:22:11.706480 7f8692f68700 20 osd.26 243 Sending info 3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) to shard 5 2014-06-09 16:22:11.706482 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706484 7f8692f68700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.706487 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.706498 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.706504 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (19 active) 2014-06-09 16:22:11.706507 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706517 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (20 active) 2014-06-09 16:22:11.706515 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706519 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 7 2014-06-09 16:22:11.706527 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.706466 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3395.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.706528 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd7 2014-06-09 16:22:11.706534 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.706536 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.706541 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.706539 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.706546 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'583 (0'0,214'583] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.7: log((214'583,214'586], crt=0'0) missing(0) 2014-06-09 16:22:11.706552 7f8692f68700 10 merge_log log((214'583,214'586], crt=0'0) from osd.7 into log((0'0,214'583], crt=0'0) 2014-06-09 16:22:11.706554 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706555 7f8692f68700 10 merge_log extending head to 214'586 2014-06-09 16:22:11.706554 7f86a5610700 10 _do_op 0x10039740 seq 3395 r = 0, finisher 0x18742ec0 0 2014-06-09 16:22:11.706557 7f86a5610700 10 _finish_op 0x10039740 seq 3395 osr(3.527 0x3ed1660)/0x3ed1660 2014-06-09 16:22:11.706556 7f8692f68700 20 ? 214'586 (214'585) modify c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3 by client.4544.0:34433 2014-06-09 10:19:30.524067 2014-06-09 16:22:11.706560 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525320 (1 active) 2014-06-09 16:22:11.706563 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358fc0 (2 active) 2014-06-09 16:22:11.706561 7f8692f68700 20 ? 214'585 (214'584) modify c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3 by client.4544.0:34432 2014-06-09 10:19:30.523110 2014-06-09 16:22:11.706559 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.706565 7f86a5610700 5 _do_op 0x2fcaf800 seq 3396 osr(1.fa 0x384c970)/0x384c970 start 2014-06-09 16:22:11.706565 7f8692f68700 20 ? 214'584 (214'583) modify c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3 by client.4544.0:34431 2014-06-09 10:19:30.401010 2014-06-09 16:22:11.706567 7f86a5610700 10 _do_transaction on 0xc05bbc0 2014-06-09 16:22:11.706568 7f8692f68700 20 merge_log 214'584 (214'583) modify c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3 by client.4544.0:34431 2014-06-09 10:19:30.401010 2014-06-09 16:22:11.706568 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.706574 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.706575 7f8692f68700 20 merge_log 214'585 (214'584) modify c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3 by client.4544.0:34432 2014-06-09 10:19:30.523110 2014-06-09 16:22:11.706576 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706581 7f8692f68700 20 merge_log 214'586 (214'585) modify c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3 by client.4544.0:34433 2014-06-09 10:19:30.524067 2014-06-09 16:22:11.706584 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706596 7f868ef60700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.706609 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.706623 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706627 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.706640 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.706648 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.706649 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706657 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706665 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3396.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.706668 7f868ef60700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.706670 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.706682 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.706697 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706699 7f86a5610700 10 _do_op 0x2fcaf800 seq 3396 r = 0, finisher 0x3061e2c0 0 2014-06-09 16:22:11.706704 7f86a5610700 10 _finish_op 0x2fcaf800 seq 3396 osr(1.fa 0x384c970)/0x384c970 2014-06-09 16:22:11.706706 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358fc0 (1 active) 2014-06-09 16:22:11.706707 7f8692f68700 10 merge_log result log((0'0,214'586], crt=214'586) missing(1) changed=1 2014-06-09 16:22:11.706708 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780000 (2 active) 2014-06-09 16:22:11.706701 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.706711 7f86a5610700 5 _do_op 0x308ca900 seq 3397 osr(0.129 0x2bd66b0)/0x2bd66b0 start 2014-06-09 16:22:11.706714 7f86a5610700 10 _do_transaction on 0xc20e9c0 2014-06-09 16:22:11.706710 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 peering m=1] peer osd.7 now 3.4c0( v 214'586 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) missing(0) 2014-06-09 16:22:11.706713 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.706723 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.685782 5 0.000211 2014-06-09 16:22:11.706723 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706732 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.706732 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706738 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 peering m=1] state: osd.7 has no missing, identical log 2014-06-09 16:22:11.706744 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.706747 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000015 0 0.000000 2014-06-09 16:22:11.706754 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.706759 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 0.803225 0 0.000000 2014-06-09 16:22:11.706757 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:161 2014-06-09 16:22:11.706772 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706769 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.706775 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.706781 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.706787 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.706790 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.706783 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.1ae( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.706792 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'583 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.706799 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.706809 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706813 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.1b3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.706826 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.706824 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.706834 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706839 7f868bf5a700 10 op_submit_start 3443 2014-06-09 16:22:11.706841 7f868bf5a700 5 queue_transactions (trailing journal) 3443 2014-06-09 16:22:11.706842 7f868bf5a700 5 queue_op 0xc34b7a0 seq 3443 osr(3.105 0x3ed0820) 1429 bytes (queue has 49 ops and 820949 bytes) 2014-06-09 16:22:11.706838 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.706845 7f868bf5a700 10 op_submit_finish 3443 2014-06-09 16:22:11.706849 7f868ff62700 2 waiting 51 > 50 ops || 2888949 > 104857600 2014-06-09 16:22:11.706855 7f8690f64700 10 op_submit_start 3444 2014-06-09 16:22:11.706849 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.706857 7f8690f64700 5 queue_transactions (trailing journal) 3444 2014-06-09 16:22:11.706854 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3397.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.706859 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.706858 7f8690f64700 5 queue_op 0x2fcaf740 seq 3444 osr(0.330 0x2bd70a0) 1883 bytes (queue has 50 ops and 822281 bytes) 2014-06-09 16:22:11.706862 7f8690f64700 10 op_submit_finish 3444 2014-06-09 16:22:11.706857 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706865 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f02e0 (19 active) 2014-06-09 16:22:11.706864 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706871 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.706875 7f868ef60700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.706866 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.35( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:0.35( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.706881 7f86a5610700 10 _do_op 0x308ca900 seq 3397 r = 0, finisher 0x306cef60 0 2014-06-09 16:22:11.706884 7f86a5610700 10 _finish_op 0x308ca900 seq 3397 osr(0.129 0x2bd66b0)/0x2bd66b0 2014-06-09 16:22:11.706880 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.35( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.706880 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.706890 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.706887 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.706898 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x8f3e000 (1 active) 2014-06-09 16:22:11.706893 7f8692f68700 10 osd.26 243 queue_for_recovery queued pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.706901 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706901 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.706896 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=210 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.706890 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780000 (1 active) 2014-06-09 16:22:11.706907 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3bc20 (2 active) 2014-06-09 16:22:11.706909 7f86a5610700 5 _do_op 0x30d96f00 seq 3398 osr(1.1cf 0x384ca70)/0x384ca70 start 2014-06-09 16:22:11.706912 7f86a5610700 10 _do_transaction on 0xbdea780 2014-06-09 16:22:11.706906 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.706906 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.7 3.4c0( v 214'586 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) 2014-06-09 16:22:11.706908 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=210 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.7 3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) 2014-06-09 16:22:11.706918 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.706919 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.706918 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.7 is up to date, queueing in pending_activators 2014-06-09 16:22:11.706920 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=210 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.025471 7 0.000602 2014-06-09 16:22:11.706928 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706934 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706930 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=210 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.706941 7f868ef60700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.706933 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.7 3.4c0( v 214'586 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242) uptodate 2014-06-09 16:22:11.706945 7f8690f64700 5 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=210 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.706946 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.706953 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.706955 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=210 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.706964 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.706956 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3 214'586 also missing on osd.26 2014-06-09 16:22:11.706965 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'639 -> 213'639 2014-06-09 16:22:11.706966 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) 2014-06-09 16:22:11.706966 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3=214'586(214'583)} 2014-06-09 16:22:11.706975 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.706981 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.706977 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 7 as a source 2014-06-09 16:22:11.706986 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.706994 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.706988 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3 214'586 is on osd.7 2014-06-09 16:22:11.707000 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {c77024c0/rbd_data.11606b8b4567.000000000000183a/head//3=214'586(214'583)} 2014-06-09 16:22:11.707003 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1093 2014-06-09 16:22:11.707017 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.707011 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.707012 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.707021 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.707021 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 210 2014-06-09 16:22:11.707032 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] state: got info+log from osd.6 1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.707040 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.6 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.707033 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 7 2014-06-09 16:22:11.707042 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.707044 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.233( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] exit Started/Stray 1.392522 7 0.000224 2014-06-09 16:22:11.707049 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.707043 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.707050 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.233( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.707053 7f8690f64700 5 queue_transactions existing osr(3.148 0x3ed0740)/0x3ed0740 2014-06-09 16:22:11.707056 7f8690f64700 10 op_submit_start 3445 2014-06-09 16:22:11.707057 7f8690f64700 5 queue_transactions (trailing journal) 3445 2014-06-09 16:22:11.707058 7f8690f64700 5 queue_op 0x2fcaf6e0 seq 3445 osr(3.148 0x3ed0740) 1335 bytes (queue has 50 ops and 821517 bytes) 2014-06-09 16:22:11.707056 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.233( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.707061 7f8690f64700 10 op_submit_finish 3445 2014-06-09 16:22:11.707054 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.707062 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.707067 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.707063 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.6b7( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.707072 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.707076 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.707085 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.707093 7f8692f68700 10 write_log with: dirty_to: 0'0, dirty_from: 214'583, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.707089 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:3.3f5( v 213'565 (0'0,213'565] local-les=201 n=106 ec=118 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.707105 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.707108 7f868ef60700 5 queue_transactions existing osr(1.233 0x384cc30)/0x384cc30 2014-06-09 16:22:11.707101 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.707110 7f868ef60700 2 waiting 51 > 50 ops || 823306 > 104857600 2014-06-09 16:22:11.707111 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.706951 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.707115 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:0.3f9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.707126 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.707133 7f8692f68700 5 queue_transactions existing osr(3.4c0 0x3ed1440)/0x3ed1440 2014-06-09 16:22:11.707135 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.707136 7f8692f68700 2 waiting 51 > 50 ops || 823677 > 104857600 2014-06-09 16:22:11.707135 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3398.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.707140 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.707139 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.707149 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.1 1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.707160 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.1 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.707163 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.707164 7f86a5610700 10 _do_op 0x30d96f00 seq 3398 r = 0, finisher 0x3050bec0 0 2014-06-09 16:22:11.707168 7f86a5610700 10 _finish_op 0x30d96f00 seq 3398 osr(1.1cf 0x384ca70)/0x384ca70 2014-06-09 16:22:11.707167 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.963345 7 0.000185 2014-06-09 16:22:11.707174 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3bc20 (1 active) 2014-06-09 16:22:11.707176 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7320 (2 active) 2014-06-09 16:22:11.707177 7f868cf5c700 2 waiting 51 > 50 ops || 823235 > 104857600 2014-06-09 16:22:11.707179 7f86a5610700 5 _do_op 0x30d96600 seq 3399 osr(3.753 0x3ed1e50)/0x3ed1e50 start 2014-06-09 16:22:11.707179 7f868ef60700 10 op_submit_start 3446 2014-06-09 16:22:11.707181 7f86a5610700 10 _do_transaction on 0x394f500 2014-06-09 16:22:11.707182 7f868ef60700 5 queue_transactions (trailing journal) 3446 2014-06-09 16:22:11.707182 7f868c75b700 2 waiting 51 > 50 ops || 822385 > 104857600 2014-06-09 16:22:11.707177 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.707184 7f868ef60700 5 queue_op 0x2fc8df20 seq 3446 osr(1.233 0x384cc30) 1789 bytes (queue has 50 ops and 821332 bytes) 2014-06-09 16:22:11.707187 7f868ef60700 10 op_submit_finish 3446 2014-06-09 16:22:11.707187 7f8691765700 2 waiting 51 > 50 ops || 822385 > 104857600 2014-06-09 16:22:11.707187 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.707192 7f8691f66700 2 waiting 51 > 50 ops || 823341 > 104857600 2014-06-09 16:22:11.707196 7f868f761700 2 waiting 51 > 50 ops || 822667 > 104857600 2014-06-09 16:22:11.707199 7f8692f68700 2 waiting 51 > 50 ops || 823492 > 104857600 2014-06-09 16:22:11.707190 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.707195 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.707205 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.707205 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.707212 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.707217 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.707215 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.707223 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.707235 7f868ef60700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.707246 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.707248 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.707251 7f8690f64700 5 queue_transactions existing osr(1.10d 0x384c930)/0x384c930 2014-06-09 16:22:11.707253 7f8690f64700 2 waiting 51 > 50 ops || 823215 > 104857600 2014-06-09 16:22:11.707256 7f868ff62700 2 waiting 51 > 50 ops || 2888000 > 104857600 2014-06-09 16:22:11.707258 7f868e75f700 10 op_submit_start 3447 2014-06-09 16:22:11.707258 7f868e75f700 5 queue_transactions (trailing journal) 3447 2014-06-09 16:22:11.707251 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.707259 7f868e75f700 5 queue_op 0x309a2300 seq 3447 osr(3.9f 0x3ed0600) 1332 bytes (queue has 50 ops and 821332 bytes) 2014-06-09 16:22:11.707262 7f868e75f700 10 op_submit_finish 3447 2014-06-09 16:22:11.707268 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.707266 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3399.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.707270 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.707264 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=206 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.707273 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=206 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.707281 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41ef0 (19 active) 2014-06-09 16:22:11.707281 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=206 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.470584 7 0.000173 2014-06-09 16:22:11.707290 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=206 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.707295 7f86a5610700 10 _do_op 0x30d96600 seq 3399 r = 0, finisher 0xcc838e0 0 2014-06-09 16:22:11.707297 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:11.707299 7f86a5610700 10 _finish_op 0x30d96600 seq 3399 osr(3.753 0x3ed1e50)/0x3ed1e50 2014-06-09 16:22:11.707304 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7320 (1 active) 2014-06-09 16:22:11.707307 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.707308 7f868cf5c700 10 op_submit_start 3448 2014-06-09 16:22:11.707302 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=206 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.707301 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.707310 7f868cf5c700 5 queue_transactions (trailing journal) 3448 2014-06-09 16:22:11.707310 7f86a5610700 5 _do_op 0x2fcafc80 seq 3400 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.707312 7f86a5610700 10 _do_transaction on 0xc2d8240 2014-06-09 16:22:11.707312 7f868cf5c700 5 queue_op 0x30d96540 seq 3448 osr(2.5a4 0x2ed1810) 1903 bytes (queue has 50 ops and 821900 bytes) 2014-06-09 16:22:11.707315 7f868cf5c700 10 op_submit_finish 3448 2014-06-09 16:22:11.707317 7f86a5610700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.707312 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=206 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.707313 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.707318 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.707322 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.707322 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'578 -> 213'578 2014-06-09 16:22:11.707328 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.707331 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.707340 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.707344 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.707340 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.707347 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 1.195381 7 0.000187 2014-06-09 16:22:11.707349 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.707357 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.707362 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.707363 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.707374 7f86a5610700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.707373 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.707378 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.707382 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.707391 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.707390 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.707400 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.707388 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.707395 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.707404 7f868e75f700 5 queue_transactions existing osr(3.ad 0x3ed05e0)/0x3ed05e0 2014-06-09 16:22:11.707406 7f868e75f700 2 waiting 51 > 50 ops || 823232 > 104857600 2014-06-09 16:22:11.707411 7f8692767700 2 waiting 51 > 50 ops || 823329 > 104857600 2014-06-09 16:22:11.707404 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.707408 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.494( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.707416 7f8691765700 2 waiting 51 > 50 ops || 822953 > 104857600 2014-06-09 16:22:11.707418 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.707413 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.707420 7f8691f66700 2 waiting 51 > 50 ops || 823909 > 104857600 2014-06-09 16:22:11.707422 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.707424 7f868f761700 2 waiting 51 > 50 ops || 823235 > 104857600 2014-06-09 16:22:11.707428 7f8692f68700 2 waiting 51 > 50 ops || 824060 > 104857600 2014-06-09 16:22:11.707422 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.707432 7f8690f64700 2 waiting 51 > 50 ops || 823783 > 104857600 2014-06-09 16:22:11.707435 7f868ff62700 2 waiting 51 > 50 ops || 2888568 > 104857600 2014-06-09 16:22:11.707437 7f86a5610700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.707434 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.707443 7f86a5610700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.707442 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.707454 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.707450 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.707457 7f868cf5c700 5 queue_transactions existing osr(2.40a 0x2ed1280)/0x2ed1280 2014-06-09 16:22:11.707459 7f868cf5c700 2 waiting 51 > 50 ops || 823783 > 104857600 2014-06-09 16:22:11.707458 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.707467 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.707479 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.707489 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.707449 7f86a5610700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.707497 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.707505 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.707515 7f86a5610700 15 _touch 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.707515 7f868c75b700 2 waiting 51 > 50 ops || 822953 > 104857600 2014-06-09 16:22:11.707509 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.707522 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.707531 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.707535 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 1 2014-06-09 16:22:11.707554 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.707549 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.1: 3.75f( v 213'591 (0'0,213'591] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) log((0'0,213'591], crt=0'0) missing(0) 2014-06-09 16:22:11.707564 7f868ef60700 10 proc_replica_log for osd.1: 3.75f( v 213'591 (0'0,213'591] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) log((0'0,213'591], crt=0'0) missing(0) 2014-06-09 16:22:11.707569 7f868ef60700 20 merge_log cut point (usually last shared) is 213'591 (213'589) modify d952275f/rbd_data.116f6b8b4567.00000000000017dd/head//3 by client.4559.0:35423 2014-06-09 10:18:08.292271 2014-06-09 16:22:11.707591 7f86a5610700 20 write_state seq is 30153 2014-06-09 16:22:11.707599 7f86a5610700 20 set_header setting 30152 cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 parent seq 0 2014-06-09 16:22:11.707621 7f86a5610700 10 _touch 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.707628 7f86a5610700 15 _write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 0~217088 2014-06-09 16:22:11.707637 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.707958 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.1 now 3.75f( v 213'591 (0'0,213'591] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.707973 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.707985 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.717375 7 0.000378 2014-06-09 16:22:11.707997 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.708006 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.975341 0 0.000000 2014-06-09 16:22:11.708020 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.708030 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=191 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.708039 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.708049 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.708057 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 213'605 -> 213'605 2014-06-09 16:22:11.708096 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.75f( v 213'591 (0'0,213'591] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.708128 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 sending log((213'591,213'605], crt=0'0) 2014-06-09 16:22:11.708141 7f868ef60700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.708157 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.75f( v 213'605 lc 213'591 (0'0,213'605] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) missing missing(3) 2014-06-09 16:22:11.708173 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.1 has 3 missing 2014-06-09 16:22:11.708184 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 6d01975f/rbd_data.116a6b8b4567.0000000000001653/head//3 213'603 is on osd.26 2014-06-09 16:22:11.708195 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 8a9b9f5f/rbd_data.11806b8b4567.0000000000001803/head//3 213'596 is on osd.26 2014-06-09 16:22:11.708206 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 26d0ef5f/rbd_data.117d6b8b4567.0000000000001653/head//3 213'605 is on osd.26 2014-06-09 16:22:11.708218 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {6d01975f/rbd_data.116a6b8b4567.0000000000001653/head//3=213'603,8a9b9f5f/rbd_data.11806b8b4567.0000000000001803/head//3=213'596(213'590),26d0ef5f/rbd_data.117d6b8b4567.0000000000001653/head//3=213'605} 2014-06-09 16:22:11.708241 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 1 as a source 2014-06-09 16:22:11.708252 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 6d01975f/rbd_data.116a6b8b4567.0000000000001653/head//3 213'603 also missing on osd.1 2014-06-09 16:22:11.708263 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 8a9b9f5f/rbd_data.11806b8b4567.0000000000001803/head//3 213'596 also missing on osd.1 2014-06-09 16:22:11.708274 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 26d0ef5f/rbd_data.117d6b8b4567.0000000000001653/head//3 213'605 also missing on osd.1 2014-06-09 16:22:11.708284 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {6d01975f/rbd_data.116a6b8b4567.0000000000001653/head//3=213'603,8a9b9f5f/rbd_data.11806b8b4567.0000000000001803/head//3=213'596(213'590),26d0ef5f/rbd_data.117d6b8b4567.0000000000001653/head//3=213'605} 2014-06-09 16:22:11.708295 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.708304 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.708310 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 1 2014-06-09 16:22:11.708318 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.708327 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.708426 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.708431 7f868ef60700 5 queue_transactions existing osr(3.75f 0x3ed1e60)/0x3ed1e60 2014-06-09 16:22:11.708434 7f868ef60700 2 waiting 51 > 50 ops || 822953 > 104857600 2014-06-09 16:22:11.708707 7f86a5610700 10 _generic_write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 0~217088 = 0 2014-06-09 16:22:11.708728 7f86a5610700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.708739 7f86a5610700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.708745 7f86a5610700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.708754 7f86a5610700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.708783 7f86a5610700 10 cid: oid: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 not skipping op, *spos 3400.0.0 2014-06-09 16:22:11.708787 7f86a5610700 10 > header.spos 0.0.0 2014-06-09 16:22:11.708797 7f86a5610700 20 set_header setting 30152 cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 parent seq 0 2014-06-09 16:22:11.708826 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3400.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.708831 7f86a5610700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3400.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.708834 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.709672 7f86a5610700 10 _do_op 0x2fcafc80 seq 3400 r = 0, finisher 0xbd6ddc0 0x2fc5d680 2014-06-09 16:22:11.709677 7f86a5610700 10 _finish_op 0x2fcafc80 seq 3400 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.709685 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.709688 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c18c0 (2 active) 2014-06-09 16:22:11.709688 7f868e75f700 10 op_submit_start 3449 2014-06-09 16:22:11.709689 7f8692767700 2 waiting 51 > 50 ops || 605185 > 104857600 2014-06-09 16:22:11.709691 7f868e75f700 5 queue_transactions (trailing journal) 3449 2014-06-09 16:22:11.709691 7f86a5610700 5 _do_op 0x308cb860 seq 3401 osr(0.725 0x384c3d0)/0x384c3d0 start 2014-06-09 16:22:11.709694 7f86a5610700 10 _do_transaction on 0x331ed80 2014-06-09 16:22:11.709692 7f868e75f700 5 queue_op 0x309a2360 seq 3449 osr(3.ad 0x3ed05e0) 1332 bytes (queue has 50 ops and 603756 bytes) 2014-06-09 16:22:11.709695 7f868e75f700 10 op_submit_finish 3449 2014-06-09 16:22:11.709701 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.709706 7f8691765700 2 waiting 51 > 50 ops || 604809 > 104857600 2014-06-09 16:22:11.709699 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.709713 7f8691f66700 2 waiting 51 > 50 ops || 605765 > 104857600 2014-06-09 16:22:11.709718 7f868f761700 2 waiting 51 > 50 ops || 605091 > 104857600 2014-06-09 16:22:11.709712 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.709724 7f8692f68700 2 waiting 51 > 50 ops || 605916 > 104857600 2014-06-09 16:22:11.709724 7f868e75f700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.709728 7f868e75f700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.709729 7f8690f64700 2 waiting 51 > 50 ops || 605639 > 104857600 2014-06-09 16:22:11.709735 7f868ff62700 2 waiting 51 > 50 ops || 2670424 > 104857600 2014-06-09 16:22:11.709739 7f868cf5c700 2 waiting 51 > 50 ops || 605639 > 104857600 2014-06-09 16:22:11.709732 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.465( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.941063 7 0.000159 2014-06-09 16:22:11.709743 7f868c75b700 2 waiting 51 > 50 ops || 604809 > 104857600 2014-06-09 16:22:11.709747 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.709743 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.465( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.709754 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.465( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.709762 7f868ef60700 2 waiting 51 > 50 ops || 604809 > 104857600 2014-06-09 16:22:11.709765 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3401.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.709770 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.709764 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.709774 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.709782 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.709802 7f86a5610700 10 _do_op 0x308cb860 seq 3401 r = 0, finisher 0xcdb2820 0 2014-06-09 16:22:11.709806 7f86a5610700 10 _finish_op 0x308cb860 seq 3401 osr(0.725 0x384c3d0)/0x384c3d0 2014-06-09 16:22:11.709812 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c18c0 (1 active) 2014-06-09 16:22:11.709814 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd680 (2 active) 2014-06-09 16:22:11.709814 7f8692767700 10 op_submit_start 3450 2014-06-09 16:22:11.709816 7f8692767700 5 queue_transactions (trailing journal) 3450 2014-06-09 16:22:11.709817 7f86a5610700 5 _do_op 0x2fcaec60 seq 3402 osr(2.3aa 0x2ed1070)/0x2ed1070 start 2014-06-09 16:22:11.709819 7f86a5610700 10 _do_transaction on 0xc12c540 2014-06-09 16:22:11.709818 7f8692767700 5 queue_op 0x30061920 seq 3450 osr(3.251 0x3ed0c20) 1429 bytes (queue has 50 ops and 603345 bytes) 2014-06-09 16:22:11.709822 7f8692767700 10 op_submit_finish 3450 2014-06-09 16:22:11.709826 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.709828 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.709832 7f868e75f700 5 queue_transactions existing osr(1.465 0x384d650)/0x384d650 2014-06-09 16:22:11.709824 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.709835 7f868e75f700 2 waiting 51 > 50 ops || 605228 > 104857600 2014-06-09 16:22:11.709836 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.709847 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.709851 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.709853 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.719( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.976709 7 0.000930 2014-06-09 16:22:11.709860 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.709860 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.719( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.709866 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.719( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.709872 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.709877 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3402.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.709877 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.709882 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.709882 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.709902 7f86a5610700 10 _do_op 0x2fcaec60 seq 3402 r = 0, finisher 0x292820a0 0 2014-06-09 16:22:11.709906 7f86a5610700 10 _finish_op 0x2fcaec60 seq 3402 osr(2.3aa 0x2ed1070)/0x2ed1070 2014-06-09 16:22:11.709911 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd680 (1 active) 2014-06-09 16:22:11.709913 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4acc60 (2 active) 2014-06-09 16:22:11.709915 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.709916 7f868e75f700 10 op_submit_start 3451 2014-06-09 16:22:11.709916 7f86a5610700 5 _do_op 0x30d96ae0 seq 3403 osr(2.409 0x2ed1270)/0x2ed1270 start 2014-06-09 16:22:11.709917 7f868e75f700 5 queue_transactions (trailing journal) 3451 2014-06-09 16:22:11.709918 7f86a5610700 10 _do_transaction on 0x36b2840 2014-06-09 16:22:11.709919 7f8692767700 5 queue_transactions existing osr(2.719 0x3ed01d0)/0x3ed01d0 2014-06-09 16:22:11.709919 7f868e75f700 5 queue_op 0x309a23c0 seq 3451 osr(1.465 0x384d650) 1883 bytes (queue has 50 ops and 603274 bytes) 2014-06-09 16:22:11.709922 7f868e75f700 10 op_submit_finish 3451 2014-06-09 16:22:11.709921 7f8692767700 2 waiting 51 > 50 ops || 605275 > 104857600 2014-06-09 16:22:11.709924 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.709925 7f8691765700 2 waiting 51 > 50 ops || 604327 > 104857600 2014-06-09 16:22:11.709930 7f8691f66700 2 waiting 51 > 50 ops || 605283 > 104857600 2014-06-09 16:22:11.709933 7f868f761700 2 waiting 51 > 50 ops || 604609 > 104857600 2014-06-09 16:22:11.709924 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.43b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.709937 7f8690f64700 2 waiting 51 > 50 ops || 605157 > 104857600 2014-06-09 16:22:11.709940 7f868ff62700 2 waiting 51 > 50 ops || 2669942 > 104857600 2014-06-09 16:22:11.709944 7f868cf5c700 2 waiting 51 > 50 ops || 605157 > 104857600 2014-06-09 16:22:11.709938 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.709947 7f868c75b700 2 waiting 51 > 50 ops || 604327 > 104857600 2014-06-09 16:22:11.709948 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.709951 7f868ef60700 2 waiting 51 > 50 ops || 604327 > 104857600 2014-06-09 16:22:11.709954 7f8692f68700 2 waiting 51 > 50 ops || 605434 > 104857600 2014-06-09 16:22:11.709956 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.709953 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.709964 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.709970 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.709973 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3403.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.709974 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.709978 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.709982 7f868e75f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.709993 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:173 2014-06-09 16:22:11.710001 7f86a5610700 10 _do_op 0x30d96ae0 seq 3403 r = 0, finisher 0x19941120 0 2014-06-09 16:22:11.710005 7f86a5610700 10 _finish_op 0x30d96ae0 seq 3403 osr(2.409 0x2ed1270)/0x2ed1270 2014-06-09 16:22:11.710009 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.710011 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4acc60 (1 active) 2014-06-09 16:22:11.710014 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac50e0 (2 active) 2014-06-09 16:22:11.710016 7f86a5610700 5 _do_op 0x309a3380 seq 3404 osr(0.47d 0x2bd7470)/0x2bd7470 start 2014-06-09 16:22:11.710019 7f86a5610700 10 _do_transaction on 0x390ca80 2014-06-09 16:22:11.710019 7f8692767700 10 op_submit_start 3452 2014-06-09 16:22:11.710021 7f8692767700 5 queue_transactions (trailing journal) 3452 2014-06-09 16:22:11.710014 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 3 notify: (query_epoch:243, epoch_sent:243, info:1.4c3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.710022 7f8692767700 5 queue_op 0x30061980 seq 3452 osr(2.719 0x3ed01d0) 2001 bytes (queue has 50 ops and 603392 bytes) 2014-06-09 16:22:11.710025 7f8692767700 10 op_submit_finish 3452 2014-06-09 16:22:11.710025 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.710026 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.710034 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.710028 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.710039 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.710052 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0290 (19 active) 2014-06-09 16:22:11.710048 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.710059 7f868e75f700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.710058 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.710054 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.710067 7f8692767700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.710074 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3404.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.710079 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.710085 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.710100 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.710098 7f86a5610700 10 _do_op 0x309a3380 seq 3404 r = 0, finisher 0xbd6b880 0 2014-06-09 16:22:11.710102 7f86a5610700 10 _finish_op 0x309a3380 seq 3404 osr(0.47d 0x2bd7470)/0x2bd7470 2014-06-09 16:22:11.710104 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac50e0 (1 active) 2014-06-09 16:22:11.710106 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7817a0 (2 active) 2014-06-09 16:22:11.710108 7f86a5610700 5 _do_op 0x30d97c80 seq 3405 osr(1.77c 0x2ed0330)/0x2ed0330 start 2014-06-09 16:22:11.710111 7f86a5610700 10 _do_transaction on 0x390c000 2014-06-09 16:22:11.710104 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.710117 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.710117 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.710123 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.710127 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.710134 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.710149 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.710144 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.710157 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.710164 7f8691765700 10 op_submit_start 3453 2014-06-09 16:22:11.710165 7f8691765700 5 queue_transactions (trailing journal) 3453 2014-06-09 16:22:11.710167 7f8691765700 5 queue_op 0xc2eb9e0 seq 3453 osr(3.55d 0x3ed1940) 1053 bytes (queue has 50 ops and 602562 bytes) 2014-06-09 16:22:11.710166 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3405.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.710171 7f8691765700 10 op_submit_finish 3453 2014-06-09 16:22:11.710171 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.710174 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 3 2014-06-09 16:22:11.710187 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd3 2014-06-09 16:22:11.710194 7f86a5610700 10 _do_op 0x30d97c80 seq 3405 r = 0, finisher 0xcc95360 0 2014-06-09 16:22:11.710197 7f86a5610700 10 _finish_op 0x30d97c80 seq 3405 osr(1.77c 0x2ed0330)/0x2ed0330 2014-06-09 16:22:11.710195 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.710200 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7817a0 (1 active) 2014-06-09 16:22:11.710202 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0d80 (2 active) 2014-06-09 16:22:11.710204 7f86a5610700 5 _do_op 0x2fcae120 seq 3406 osr(1.26c 0x384cf50)/0x384cf50 start 2014-06-09 16:22:11.710207 7f86a5610700 10 _do_transaction on 0xbe89140 2014-06-09 16:22:11.710202 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.710213 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.710209 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 213'610 (0'0,213'610] local-les=201 n=122 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.3: log((213'610,214'611], crt=0'0) missing(0) 2014-06-09 16:22:11.710216 7f8691765700 10 merge_log log((213'610,214'611], crt=0'0) from osd.3 into log((0'0,213'610], crt=0'0) 2014-06-09 16:22:11.710218 7f8691765700 10 merge_log extending head to 214'611 2014-06-09 16:22:11.710219 7f8691765700 20 ? 214'611 (0'0) modify b67825f9/rbd_data.115e6b8b4567.0000000000001803/head//3 by client.4542.0:34527 2014-06-09 10:18:59.156187 2014-06-09 16:22:11.710225 7f8691765700 20 merge_log 214'611 (0'0) modify b67825f9/rbd_data.115e6b8b4567.0000000000001803/head//3 by client.4542.0:34527 2014-06-09 10:18:59.156187 2014-06-09 16:22:11.710243 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.710265 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3388.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.710270 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.710288 7f86a5e11700 10 _do_op 0x2ea52a20 seq 3388 r = 0, finisher 0xbdbb760 0 2014-06-09 16:22:11.710293 7f86a5e11700 10 _finish_op 0x2ea52a20 seq 3388 osr(0.493 0x2bd73d0)/0x2bd73d0 2014-06-09 16:22:11.710296 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd320 (1 active) 2014-06-09 16:22:11.710298 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286360 (2 active) 2014-06-09 16:22:11.710300 7f86a5e11700 5 _do_op 0x10038480 seq 3407 osr(3.52b 0x3ed1640)/0x3ed1640 start 2014-06-09 16:22:11.710303 7f86a5e11700 10 _do_transaction on 0xc294540 2014-06-09 16:22:11.710309 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.710344 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.710361 7f8691765700 10 merge_log result log((0'0,214'611], crt=214'611) missing(1) changed=1 2014-06-09 16:22:11.710361 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3407.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.710366 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.710365 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 peering m=1] peer osd.3 now 3.5f9( v 214'611 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) missing(0) 2014-06-09 16:22:11.710377 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.698352 6 0.000257 2014-06-09 16:22:11.710386 7f86a5e11700 10 _do_op 0x10038480 seq 3407 r = 0, finisher 0xbfce100 0 2014-06-09 16:22:11.710386 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.710390 7f86a5e11700 10 _finish_op 0x10038480 seq 3407 osr(3.52b 0x3ed1640)/0x3ed1640 2014-06-09 16:22:11.710393 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286360 (1 active) 2014-06-09 16:22:11.710395 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115d40 (2 active) 2014-06-09 16:22:11.710392 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 peering m=1] state: osd.3 has no missing, identical log 2014-06-09 16:22:11.710397 7f86a5e11700 5 _do_op 0x308ca8a0 seq 3408 osr(0.6b6 0x2bd7fb0)/0x2bd7fb0 start 2014-06-09 16:22:11.710400 7f86a5e11700 10 _do_transaction on 0xbf906c0 2014-06-09 16:22:11.710399 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000014 0 0.000000 2014-06-09 16:22:11.710406 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.710407 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.710412 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 0.987121 0 0.000000 2014-06-09 16:22:11.710421 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.710426 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.710434 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.710439 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.710439 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.710444 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.710456 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3408.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.710460 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.701208 7f869af78700 5 -- op tracker -- , seq: 2730, time: 2014-06-09 16:22:11.701205, event: done, op: pg_info(1 pgs e243:3.609) 2014-06-09 16:22:11.710484 7f86a5e11700 10 _do_op 0x308ca8a0 seq 3408 r = 0, finisher 0x314acfc0 0 2014-06-09 16:22:11.710488 7f86a5e11700 10 _finish_op 0x308ca8a0 seq 3408 osr(0.6b6 0x2bd7fb0)/0x2bd7fb0 2014-06-09 16:22:11.710491 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115d40 (1 active) 2014-06-09 16:22:11.710493 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac6c0 (2 active) 2014-06-09 16:22:11.710495 7f86a5e11700 5 _do_op 0x308cb800 seq 3409 osr(0.5f5 0x2bd7990)/0x2bd7990 start 2014-06-09 16:22:11.710491 7f8670072700 5 -- op tracker -- , seq: 2731, time: 2014-06-09 16:22:11.699561, event: header_read, op: osd_op(client.4538.0:36624 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~1548288] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.710498 7f86a5e11700 10 _do_transaction on 0xc2e0780 2014-06-09 16:22:11.710504 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.710507 7f8670072700 5 -- op tracker -- , seq: 2731, time: 2014-06-09 16:22:11.699563, event: throttled, op: osd_op(client.4538.0:36624 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~1548288] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.710520 7f8670072700 5 -- op tracker -- , seq: 2731, time: 2014-06-09 16:22:11.710462, event: all_read, op: osd_op(client.4538.0:36624 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~1548288] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.710520 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.710529 7f8691765700 10 osd.26 243 queue_for_recovery queued pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.710534 7f8670072700 5 -- op tracker -- , seq: 2731, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36624 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~1548288] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.710537 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.710540 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 3.5f9( v 214'611 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) 2014-06-09 16:22:11.710544 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.710548 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.710550 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.710547 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 is up to date, queueing in pending_activators 2014-06-09 16:22:11.710553 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3409.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.710558 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.710557 7f8689755700 5 -- op tracker -- , seq: 5, time: 2014-06-09 16:22:11.706315, event: reached_pg, op: osd_op(client.4573.0:33015 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~425984] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.710565 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.710567 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.710568 7f869af78700 20 osd.26 243 _dispatch 0x31156000 pg_info(1 pgs e243:2.65) v4 2014-06-09 16:22:11.710562 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 3.5f9( v 214'611 (0'0,214'611] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) uptodate 2014-06-09 16:22:11.710569 7f8670072700 15 osd.26 243 enqueue_op 0x2f097f00 prio 63 cost 1548288 latency 0.011008 osd_op(client.4538.0:36624 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~1548288] 3.923772b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.710573 7f869af78700 5 -- op tracker -- , seq: 2732, time: 2014-06-09 16:22:11.434735, event: header_read, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.710569 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.710580 7f869af78700 5 -- op tracker -- , seq: 2732, time: 2014-06-09 16:22:11.434736, event: throttled, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.710575 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.710583 7f86a5e11700 10 _do_op 0x308cb800 seq 3409 r = 0, finisher 0xc2d7760 0 2014-06-09 16:22:11.710587 7f86a5e11700 10 _finish_op 0x308cb800 seq 3409 osr(0.5f5 0x2bd7990)/0x2bd7990 2014-06-09 16:22:11.710586 7f869af78700 5 -- op tracker -- , seq: 2732, time: 2014-06-09 16:22:11.434747, event: all_read, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.710582 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x306a400 2014-06-09 16:22:11.710590 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac6c0 (1 active) 2014-06-09 16:22:11.710592 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3bd40 (2 active) 2014-06-09 16:22:11.710595 7f86a5e11700 5 _do_op 0x30607920 seq 3410 osr(0.1cd 0x2bd6960)/0x2bd6960 start 2014-06-09 16:22:11.710593 7f869af78700 5 -- op tracker -- , seq: 2732, time: 2014-06-09 16:22:11.710564, event: dispatched, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.710602 7f86a5e11700 10 _do_transaction on 0xbe8ab40 2014-06-09 16:22:11.710602 7f8691f66700 10 op_submit_start 3454 2014-06-09 16:22:11.710604 7f8691f66700 5 queue_transactions (trailing journal) 3454 2014-06-09 16:22:11.710592 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:33015 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~425984] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.710606 7f8691f66700 5 queue_op 0x24104060 seq 3454 osr(0.57b 0x2bd77b0) 2009 bytes (queue has 46 ops and 595714 bytes) 2014-06-09 16:22:11.710605 7f869af78700 5 -- op tracker -- , seq: 2732, time: 2014-06-09 16:22:11.710603, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.710609 7f8691f66700 10 op_submit_finish 3454 2014-06-09 16:22:11.710608 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.710610 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.65) v4 from osd.8 2014-06-09 16:22:11.710614 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31156000 2014-06-09 16:22:11.710619 7f869af78700 5 -- op tracker -- , seq: 2732, time: 2014-06-09 16:22:11.710617, event: started, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.710611 7f8689755700 3 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4573.0:33015 was 213'583 2014-06-09 16:22:11.710625 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (19 active) 2014-06-09 16:22:11.710640 7f8689755700 10 osd.26 243 dequeue_op 0x306a400 finish 2014-06-09 16:22:11.710641 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (20 active) 2014-06-09 16:22:11.710642 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.710647 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.710651 7f8689755700 5 -- op tracker -- , seq: 5, time: 2014-06-09 16:22:11.710646, event: done, op: osd_op(client.4573.0:33015 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~425984] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.710660 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.710665 7f869af78700 5 -- op tracker -- , seq: 2732, time: 2014-06-09 16:22:11.710649, event: done, op: pg_info(1 pgs e243:2.65) 2014-06-09 16:22:11.710651 7f868bf5a700 10 osd.26 243 dequeue_op 0x19fe0a00 prio 63 cost 352256 latency 1.290056 osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.710669 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.710676 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.710677 7f868bf5a700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.710671, event: reached_pg, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.710692 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.710692 7f868f761700 10 op_submit_start 3455 2014-06-09 16:22:11.710695 7f868f761700 5 queue_transactions (trailing journal) 3455 2014-06-09 16:22:11.710695 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.710697 7f868f761700 5 queue_op 0x30d96a80 seq 3455 osr(3.437 0x3ed11c0) 1335 bytes (queue has 47 ops and 597049 bytes) 2014-06-09 16:22:11.710700 7f868f761700 10 op_submit_finish 3455 2014-06-09 16:22:11.710689 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.710710 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.713051 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713055 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713056 7f869af78700 20 osd.26 243 _dispatch 0x31162c40 pg_info(1 pgs e243:2.6f) v4 2014-06-09 16:22:11.713060 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.713048 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x19fe0a00 2014-06-09 16:22:11.710704 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=191 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.713062 7f869af78700 5 -- op tracker -- , seq: 2733, time: 2014-06-09 16:22:11.549711, event: header_read, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.713071 7f869af78700 5 -- op tracker -- , seq: 2733, time: 2014-06-09 16:22:11.549711, event: throttled, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.713064 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.713079 7f869af78700 5 -- op tracker -- , seq: 2733, time: 2014-06-09 16:22:11.549722, event: all_read, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.713070 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=191 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.713085 7f869af78700 5 -- op tracker -- , seq: 2733, time: 2014-06-09 16:22:11.713049, event: dispatched, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.713088 7f868bf5a700 15 getattr 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 '_' 2014-06-09 16:22:11.713085 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=191 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.532451 7 0.000175 2014-06-09 16:22:11.713093 7f869af78700 5 -- op tracker -- , seq: 2733, time: 2014-06-09 16:22:11.713092, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.713098 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6f) v4 from osd.6 2014-06-09 16:22:11.713101 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31162c40 2014-06-09 16:22:11.713097 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=191 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.713105 7f869af78700 5 -- op tracker -- , seq: 2733, time: 2014-06-09 16:22:11.713104, event: started, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.713110 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=191 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.713121 7f869af78700 5 -- op tracker -- , seq: 2733, time: 2014-06-09 16:22:11.713119, event: done, op: pg_info(1 pgs e243:2.6f) 2014-06-09 16:22:11.713121 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=191 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.713134 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713136 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713131 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'649 -> 213'649 2014-06-09 16:22:11.713153 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713155 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713156 7f869af78700 20 osd.26 243 _dispatch 0x1a0381c0 pg_info(1 pgs e243:3.d1) v4 2014-06-09 16:22:11.713161 7f869af78700 5 -- op tracker -- , seq: 2734, time: 2014-06-09 16:22:11.574095, event: header_read, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.713167 7f869af78700 5 -- op tracker -- , seq: 2734, time: 2014-06-09 16:22:11.574095, event: throttled, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.713173 7f869af78700 5 -- op tracker -- , seq: 2734, time: 2014-06-09 16:22:11.574107, event: all_read, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.713180 7f869af78700 5 -- op tracker -- , seq: 2734, time: 2014-06-09 16:22:11.713153, event: dispatched, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.713176 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.713187 7f869af78700 5 -- op tracker -- , seq: 2734, time: 2014-06-09 16:22:11.713185, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.713193 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.d1) v4 from osd.0 2014-06-09 16:22:11.713195 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a0381c0 2014-06-09 16:22:11.713200 7f869af78700 5 -- op tracker -- , seq: 2734, time: 2014-06-09 16:22:11.713198, event: started, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.713216 7f869af78700 5 -- op tracker -- , seq: 2734, time: 2014-06-09 16:22:11.713214, event: done, op: pg_info(1 pgs e243:3.d1) 2014-06-09 16:22:11.713219 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.713223 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713224 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713225 7f868f761700 5 queue_transactions existing osr(3.342 0x3ed10e0)/0x3ed10e0 2014-06-09 16:22:11.710719 7f8690f64700 10 op_submit_start 3456 2014-06-09 16:22:11.713233 7f8690f64700 5 queue_transactions (trailing journal) 3456 2014-06-09 16:22:11.713235 7f8690f64700 5 queue_op 0x2fcaf680 seq 3456 osr(1.10d 0x384c930) 1883 bytes (queue has 50 ops and 2667483 bytes) 2014-06-09 16:22:11.713237 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713238 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713239 7f8690f64700 10 op_submit_finish 3456 2014-06-09 16:22:11.713239 7f869af78700 20 osd.26 243 _dispatch 0x30daefc0 pg_info(1 pgs e243:1.2d0) v4 2014-06-09 16:22:11.713244 7f869af78700 5 -- op tracker -- , seq: 2735, time: 2014-06-09 16:22:11.549562, event: header_read, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.713245 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.713250 7f869af78700 5 -- op tracker -- , seq: 2735, time: 2014-06-09 16:22:11.549563, event: throttled, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.713258 7f869af78700 5 -- op tracker -- , seq: 2735, time: 2014-06-09 16:22:11.549575, event: all_read, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.713257 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.713269 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.713272 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.713264 7f869af78700 5 -- op tracker -- , seq: 2735, time: 2014-06-09 16:22:11.713236, event: dispatched, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.713276 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.095174 7 0.000180 2014-06-09 16:22:11.713283 7f869af78700 5 -- op tracker -- , seq: 2735, time: 2014-06-09 16:22:11.713279, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.713288 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2d0) v4 from osd.5 2014-06-09 16:22:11.713291 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daefc0 2014-06-09 16:22:11.713291 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.713297 7f869af78700 5 -- op tracker -- , seq: 2735, time: 2014-06-09 16:22:11.713295, event: started, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.713304 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.713315 7f869af78700 5 -- op tracker -- , seq: 2735, time: 2014-06-09 16:22:11.713312, event: done, op: pg_info(1 pgs e243:1.2d0) 2014-06-09 16:22:11.713314 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.713324 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713326 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713324 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.713333 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.713341 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713343 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713344 7f869af78700 20 osd.26 243 _dispatch 0x30cf8700 pg_info(1 pgs e243:0.38c) v4 2014-06-09 16:22:11.713349 7f869af78700 5 -- op tracker -- , seq: 2736, time: 2014-06-09 16:22:11.303188, event: header_read, op: pg_info(1 pgs e243:0.38c) 2014-06-09 16:22:11.713357 7f869af78700 5 -- op tracker -- , seq: 2736, time: 2014-06-09 16:22:11.303188, event: throttled, op: pg_info(1 pgs e243:0.38c) 2014-06-09 16:22:11.713364 7f869af78700 5 -- op tracker -- , seq: 2736, time: 2014-06-09 16:22:11.303194, event: all_read, op: pg_info(1 pgs e243:0.38c) 2014-06-09 16:22:11.713370 7f869af78700 5 -- op tracker -- , seq: 2736, time: 2014-06-09 16:22:11.713340, event: dispatched, op: pg_info(1 pgs e243:0.38c) 2014-06-09 16:22:11.713377 7f869af78700 5 -- op tracker -- , seq: 2736, time: 2014-06-09 16:22:11.713375, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.38c) 2014-06-09 16:22:11.713390 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.38c) v4 from osd.2 2014-06-09 16:22:11.713393 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf8700 2014-06-09 16:22:11.713394 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.713399 7f8690f64700 5 queue_transactions existing osr(1.5cb 0x384dd70)/0x384dd70 2014-06-09 16:22:11.713398 7f869af78700 5 -- op tracker -- , seq: 2736, time: 2014-06-09 16:22:11.713396, event: started, op: pg_info(1 pgs e243:0.38c) 2014-06-09 16:22:11.713405 7f868ff62700 10 op_submit_start 3457 2014-06-09 16:22:11.713406 7f868ff62700 5 queue_transactions (trailing journal) 3457 2014-06-09 16:22:11.713408 7f868ff62700 5 queue_op 0x288f91a0 seq 3457 osr(3.60 0x3ed02e0) 2066668 bytes (queue has 50 ops and 2667483 bytes) 2014-06-09 16:22:11.713411 7f868ff62700 10 op_submit_finish 3457 2014-06-09 16:22:11.713414 7f869af78700 5 -- op tracker -- , seq: 2736, time: 2014-06-09 16:22:11.713412, event: done, op: pg_info(1 pgs e243:0.38c) 2014-06-09 16:22:11.713422 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713424 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713438 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713439 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713418 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc139740 243'640 rep_tid=4 committed?=0 applied?=0 lock=0 op=osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.713441 7f869af78700 20 osd.26 243 _dispatch 0x310d61c0 pg_notify(1.76c(20) epoch 243) v5 2014-06-09 16:22:11.713444 7f868ff62700 10 osd.26 243 dequeue_op 0x257c5700 finish 2014-06-09 16:22:11.713446 7f869af78700 5 -- op tracker -- , seq: 2737, time: 2014-06-09 16:22:11.422953, event: header_read, op: pg_notify(1.76c(20) epoch 243) 2014-06-09 16:22:11.713453 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.713453 7f869af78700 5 -- op tracker -- , seq: 2737, time: 2014-06-09 16:22:11.422953, event: throttled, op: pg_notify(1.76c(20) epoch 243) 2014-06-09 16:22:11.713460 7f869af78700 5 -- op tracker -- , seq: 2737, time: 2014-06-09 16:22:11.422981, event: all_read, op: pg_notify(1.76c(20) epoch 243) 2014-06-09 16:22:11.713466 7f869af78700 5 -- op tracker -- , seq: 2737, time: 2014-06-09 16:22:11.713437, event: dispatched, op: pg_notify(1.76c(20) epoch 243) 2014-06-09 16:22:11.713457 7f8689f56700 10 osd.26 243 dequeue_op 0x230da000 prio 63 cost 1548288 latency 0.949553 osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.713474 7f869af78700 5 -- op tracker -- , seq: 2737, time: 2014-06-09 16:22:11.713472, event: waiting_for_osdmap, op: pg_notify(1.76c(20) epoch 243) 2014-06-09 16:22:11.713477 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e130 (20 active) 2014-06-09 16:22:11.713479 7f868bf5a700 10 getattr got.size() is 0 2014-06-09 16:22:11.713481 7f869af78700 7 osd.26 243 handle_pg_notify from osd.4 2014-06-09 16:22:11.713484 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d61c0 2014-06-09 16:22:11.713481 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.713482 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 but can_create 2014-06-09 16:22:11.713489 7f869af78700 5 -- op tracker -- , seq: 2737, time: 2014-06-09 16:22:11.713487, event: started, op: pg_notify(1.76c(20) epoch 243) 2014-06-09 16:22:11.713497 7f868bf5a700 15 getattr 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 'snapset' 2014-06-09 16:22:11.713495 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.713505 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.713509 7f869af78700 5 -- op tracker -- , seq: 2737, time: 2014-06-09 16:22:11.713507, event: done, op: pg_notify(1.76c(20) epoch 243) 2014-06-09 16:22:11.713517 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713519 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713513 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.713526 7f868bf5a700 10 getattr got.size() is 0 2014-06-09 16:22:11.713528 7f868ff62700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.713529 7f868bf5a700 15 getattr 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/snapdir//3 'snapset' 2014-06-09 16:22:11.713540 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713542 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713543 7f869af78700 20 osd.26 243 _dispatch 0x2176c540 pg_info(1 pgs e243:0.1a4) v4 2014-06-09 16:22:11.713548 7f869af78700 5 -- op tracker -- , seq: 2738, time: 2014-06-09 16:22:11.577987, event: header_read, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.713545 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:151 2014-06-09 16:22:11.713556 7f868bf5a700 10 getattr got.size() is 0 2014-06-09 16:22:11.713555 7f869af78700 5 -- op tracker -- , seq: 2738, time: 2014-06-09 16:22:11.577987, event: throttled, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.713562 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.713561 7f869af78700 5 -- op tracker -- , seq: 2738, time: 2014-06-09 16:22:11.577995, event: all_read, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.713566 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.713568 7f869af78700 5 -- op tracker -- , seq: 2738, time: 2014-06-09 16:22:11.713540, event: dispatched, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.713566 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30b5e580 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.713584 7f869af78700 5 -- op tracker -- , seq: 2738, time: 2014-06-09 16:22:11.713582, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.713580 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.350( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.713581 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.713589 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1a4) v4 from osd.7 2014-06-09 16:22:11.713593 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.713593 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176c540 2014-06-09 16:22:11.713592 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.713598 7f869af78700 5 -- op tracker -- , seq: 2738, time: 2014-06-09 16:22:11.713596, event: started, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.713597 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.713602 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30b5e580 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 rwstate(none n=0 w=0) oi: 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2e8769a0 snapset: 0=[]:[] 2014-06-09 16:22:11.713610 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.636866 6 0.000303 2014-06-09 16:22:11.713617 7f869af78700 5 -- op tracker -- , seq: 2738, time: 2014-06-09 16:22:11.713615, event: done, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.713625 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713627 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713618 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 @head oi=78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.713623 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.713637 7f868bf5a700 15 getattr 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/snapdir//3 '_' 2014-06-09 16:22:11.713640 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713642 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713635 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.713644 7f869af78700 20 osd.26 243 _dispatch 0x30ebb500 pg_info(1 pgs e243:1.697) v4 2014-06-09 16:22:11.713649 7f869af78700 5 -- op tracker -- , seq: 2739, time: 2014-06-09 16:22:11.570637, event: header_read, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.713647 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000025 0 0.000000 2014-06-09 16:22:11.713657 7f869af78700 5 -- op tracker -- , seq: 2739, time: 2014-06-09 16:22:11.570637, event: throttled, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.713658 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.713665 7f868bf5a700 10 getattr got.size() is 0 2014-06-09 16:22:11.713663 7f869af78700 5 -- op tracker -- , seq: 2739, time: 2014-06-09 16:22:11.570656, event: all_read, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.713670 7f869af78700 5 -- op tracker -- , seq: 2739, time: 2014-06-09 16:22:11.713640, event: dispatched, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.713668 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.713668 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/snapdir//3 and !can_create 2014-06-09 16:22:11.713677 7f869af78700 5 -- op tracker -- , seq: 2739, time: 2014-06-09 16:22:11.713675, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.713677 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.713687 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.697) v4 from osd.3 2014-06-09 16:22:11.713690 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebb500 2014-06-09 16:22:11.713695 7f869af78700 5 -- op tracker -- , seq: 2739, time: 2014-06-09 16:22:11.713693, event: started, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.713690 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:11.713702 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:11.713711 7f869af78700 5 -- op tracker -- , seq: 2739, time: 2014-06-09 16:22:11.713709, event: done, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.713712 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.713723 7f868ff62700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.713726 7f868bf5a700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.713682, event: started, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.713742 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713745 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713738 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 2014-06-09 16:22:11.713737 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x3070d200 2014-06-09 16:22:11.713756 7f868bf5a700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.713751, event: started, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.713768 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.713770 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.713771 7f869af78700 20 osd.26 243 _dispatch 0x2d7321c0 pg_info(1 pgs e243:3.66a) v4 2014-06-09 16:22:11.713773 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.713763 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] ov 0'0 av 243'593 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.713777 7f868ff62700 5 queue_transactions existing osr(2.349 0x2ed0e90)/0x2ed0e90 2014-06-09 16:22:11.713776 7f869af78700 5 -- op tracker -- , seq: 2740, time: 2014-06-09 16:22:11.563670, event: header_read, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.713782 7f868cf5c700 10 op_submit_start 3458 2014-06-09 16:22:11.713784 7f868cf5c700 5 queue_transactions (trailing journal) 3458 2014-06-09 16:22:11.713786 7f868cf5c700 5 queue_op 0x30d964e0 seq 3458 osr(2.40a 0x2ed1280) 1883 bytes (queue has 50 ops and 2667483 bytes) 2014-06-09 16:22:11.713779 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 2014-06-09 16:22:11.713786 7f869af78700 5 -- op tracker -- , seq: 2740, time: 2014-06-09 16:22:11.563671, event: throttled, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.713791 7f868cf5c700 10 op_submit_finish 3458 2014-06-09 16:22:11.713793 7f869af78700 5 -- op tracker -- , seq: 2740, time: 2014-06-09 16:22:11.563682, event: all_read, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.710727 7f868c75b700 2 waiting 51 > 50 ops || 2668536 > 104857600 2014-06-09 16:22:11.710731 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.713794 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.713799 7f869af78700 5 -- op tracker -- , seq: 2740, time: 2014-06-09 16:22:11.713767, event: dispatched, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.713795 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.713809 7f869af78700 5 -- op tracker -- , seq: 2740, time: 2014-06-09 16:22:11.713807, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.713808 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.713816 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.66a) v4 from osd.1 2014-06-09 16:22:11.713820 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d7321c0 2014-06-09 16:22:11.713811 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~352256 2014-06-09 16:22:11.713820 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.713824 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.713820 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.713826 7f869af78700 5 -- op tracker -- , seq: 2740, time: 2014-06-09 16:22:11.713824, event: started, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.713828 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.512( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.096126 7 0.000205 2014-06-09 16:22:11.713836 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.713831 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228e00 2014-06-09 16:22:11.713841 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.512( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.713840 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.713842 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 snapset=0x2e8769d8 snapc=0=[] 2014-06-09 16:22:11.713853 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.512( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.713852 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.713853 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.713862 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.713862 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.713863 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.713870 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.713871 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.713874 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 0x3070d200 op modify 2014-06-09 16:22:11.713882 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.713878 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.713893 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1034 2014-06-09 16:22:11.713907 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.713913 7f868ef60700 2 waiting 51 > 50 ops || 2668536 > 104857600 2014-06-09 16:22:11.710735 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.713924 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.713928 7f868cf5c700 5 queue_transactions existing osr(0.512 0x2bd7590)/0x2bd7590 2014-06-09 16:22:11.713930 7f868cf5c700 2 waiting 51 > 50 ops || 2669437 > 104857600 2014-06-09 16:22:11.713921 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.713919 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.700( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.713932 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.713936 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.713940 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.713948 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.713887 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:41.403464 2014-06-09 16:22:11.713952 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.710586 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing b67825f9/rbd_data.115e6b8b4567.0000000000001803/head//3 214'611 also missing on osd.26 2014-06-09 16:22:11.713953 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:0.18b( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.713960 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=110 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.713965 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.713967 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:182 2014-06-09 16:22:11.713974 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.713959 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {b67825f9/rbd_data.115e6b8b4567.0000000000001803/head//3=214'611} 2014-06-09 16:22:11.713984 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.713978 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 3 as a source 2014-06-09 16:22:11.713981 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.713979 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.18d( v 213'668 (0'0,213'668] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.18d( v 213'668 (0'0,213'668] local-les=206 n=124 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.713988 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.713988 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing b67825f9/rbd_data.115e6b8b4567.0000000000001803/head//3 214'611 is on osd.3 2014-06-09 16:22:11.713992 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.18d( v 213'668 (0'0,213'668] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.714001 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.713991 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 5 on osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.714000 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.714000 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {b67825f9/rbd_data.115e6b8b4567.0000000000001803/head//3=214'611} 2014-06-09 16:22:11.714005 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:2.740( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.714006 7f868bf5a700 7 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 5 o 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.714010 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.714012 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.714016 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.714018 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.714031 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.714022 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 201 2014-06-09 16:22:11.714031 7f8692767700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.710719 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3410.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.714033 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 3 2014-06-09 16:22:11.714035 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.714046 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.714044 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1010 2014-06-09 16:22:11.714045 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.714046 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.714058 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.714058 7f8691f66700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.714062 7f8691f66700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.714063 7f8692f68700 2 waiting 51 > 50 ops || 2669643 > 104857600 2014-06-09 16:22:11.714057 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.714062 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.714065 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.207344 7 0.000215 2014-06-09 16:22:11.714076 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.714073 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.714081 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.714088 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.714091 7f8691765700 10 write_log with: dirty_to: 0'0, dirty_from: 213'610, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.714088 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.714095 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.714099 7f8692767700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.714100 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.714105 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.714114 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.714130 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.714132 7f8691765700 5 queue_transactions existing osr(3.5f9 0x3ed1b40)/0x3ed1b40 2014-06-09 16:22:11.714135 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.714134 7f8691765700 2 waiting 51 > 50 ops || 2669209 > 104857600 2014-06-09 16:22:11.714138 7f8691f66700 5 queue_transactions existing osr(1.54a 0x384daf0)/0x384daf0 2014-06-09 16:22:11.714140 7f8691f66700 2 waiting 51 > 50 ops || 2669492 > 104857600 2014-06-09 16:22:11.714134 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.714144 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.714155 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.714159 7f868f761700 2 waiting 51 > 50 ops || 2668912 > 104857600 2014-06-09 16:22:11.714159 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.714166 7f8690f64700 2 waiting 51 > 50 ops || 2669492 > 104857600 2014-06-09 16:22:11.714162 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.482( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.913097 7 0.000225 2014-06-09 16:22:11.714171 7f868ff62700 2 waiting 51 > 50 ops || 2668344 > 104857600 2014-06-09 16:22:11.714172 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.482( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.714183 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.482( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.714189 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.714196 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.714201 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.714236 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.714239 7f8692767700 5 queue_transactions existing osr(2.482 0x2ed14f0)/0x2ed14f0 2014-06-09 16:22:11.714240 7f8692767700 2 waiting 51 > 50 ops || 2669480 > 104857600 2014-06-09 16:22:11.713845 7f869af78700 5 -- op tracker -- , seq: 2740, time: 2014-06-09 16:22:11.713842, event: done, op: pg_info(1 pgs e243:3.66a) 2014-06-09 16:22:11.714272 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.714274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.714289 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.714291 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.714292 7f869af78700 20 osd.26 243 _dispatch 0xddc5a40 pg_info(1 pgs e243:2.501) v4 2014-06-09 16:22:11.714297 7f869af78700 5 -- op tracker -- , seq: 2741, time: 2014-06-09 16:22:11.576282, event: header_read, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.714304 7f869af78700 5 -- op tracker -- , seq: 2741, time: 2014-06-09 16:22:11.576283, event: throttled, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.714311 7f869af78700 5 -- op tracker -- , seq: 2741, time: 2014-06-09 16:22:11.576295, event: all_read, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.714317 7f869af78700 5 -- op tracker -- , seq: 2741, time: 2014-06-09 16:22:11.714289, event: dispatched, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.714327 7f869af78700 5 -- op tracker -- , seq: 2741, time: 2014-06-09 16:22:11.714325, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.714332 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.501) v4 from osd.9 2014-06-09 16:22:11.714335 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc5a40 2014-06-09 16:22:11.714340 7f869af78700 5 -- op tracker -- , seq: 2741, time: 2014-06-09 16:22:11.714338, event: started, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.714359 7f869af78700 5 -- op tracker -- , seq: 2741, time: 2014-06-09 16:22:11.714356, event: done, op: pg_info(1 pgs e243:2.501) 2014-06-09 16:22:11.714367 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.714368 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.714381 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.714383 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.714384 7f869af78700 20 osd.26 243 _dispatch 0x2eee7dc0 pg_info(1 pgs e243:3.4d7) v4 2014-06-09 16:22:11.714389 7f869af78700 5 -- op tracker -- , seq: 2742, time: 2014-06-09 16:22:11.434796, event: header_read, op: pg_info(1 pgs e243:3.4d7) 2014-06-09 16:22:11.714404 7f869af78700 5 -- op tracker -- , seq: 2742, time: 2014-06-09 16:22:11.434797, event: throttled, op: pg_info(1 pgs e243:3.4d7) 2014-06-09 16:22:11.714436 7f869af78700 5 -- op tracker -- , seq: 2742, time: 2014-06-09 16:22:11.434809, event: all_read, op: pg_info(1 pgs e243:3.4d7) 2014-06-09 16:22:11.714443 7f869af78700 5 -- op tracker -- , seq: 2742, time: 2014-06-09 16:22:11.714381, event: dispatched, op: pg_info(1 pgs e243:3.4d7) 2014-06-09 16:22:11.714450 7f869af78700 5 -- op tracker -- , seq: 2742, time: 2014-06-09 16:22:11.714448, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4d7) 2014-06-09 16:22:11.714461 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4d7) v4 from osd.8 2014-06-09 16:22:11.714465 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee7dc0 2014-06-09 16:22:11.714471 7f868bf5a700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.714023, event: waiting for subops from 9, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.714493 7f868bf5a700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.714491 7f8689f56700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.713477, event: reached_pg, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.714501 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.714516 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x230da000 2014-06-09 16:22:11.714511 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 214'592 (0'0,214'592] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'592], crt=0'0) [243'593 (0'0) modify 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 by client.4571.0:35947 2014-06-09 10:19:41.403464] 2014-06-09 16:22:11.714527 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.714538 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'593 (0'0) modify 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 by client.4571.0:35947 2014-06-09 10:19:41.403464 2014-06-09 16:22:11.714550 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x30d99080 2014-06-09 16:22:11.714560 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30d99080 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 rwstate(write n=2 w=0) oi: 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3(243'639 client.4570.0:34070 wrlock_by=unknown.0.0:0 dirty s 1478656 uv639) ssc: 0x30dd4c40 snapset: 0=[]:[]+head 2014-06-09 16:22:11.714569 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.714575 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 @head oi=886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3(243'639 client.4570.0:34070 wrlock_by=unknown.0.0:0 dirty s 1478656 uv639) 2014-06-09 16:22:11.714597 7f868bf5a700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'593, trimmed: 2014-06-09 16:22:11.714596 7f8689f56700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.714591, event: started, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.714602 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30645e00 2014-06-09 16:22:11.714621 7f868bf5a700 5 queue_transactions existing osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.714619 7f8689f56700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.714614, event: started, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.714623 7f868bf5a700 2 waiting 51 > 50 ops || 3022127 > 104857600 2014-06-09 16:22:11.714625 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] ov 243'639 av 243'641 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.714639 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 2014-06-09 16:22:11.714651 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.714663 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 1478656~1548288 2014-06-09 16:22:11.714678 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228a00 2014-06-09 16:22:11.714689 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 snapset=0x30dd4c78 snapc=0=[] 2014-06-09 16:22:11.714770 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.714781 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 0x30645e00 op modify 2014-06-09 16:22:11.714809 7f869af78700 5 -- op tracker -- , seq: 2742, time: 2014-06-09 16:22:11.714468, event: started, op: pg_info(1 pgs e243:3.4d7) 2014-06-09 16:22:11.714849 7f869af78700 5 -- op tracker -- , seq: 2742, time: 2014-06-09 16:22:11.714846, event: done, op: pg_info(1 pgs e243:3.4d7) 2014-06-09 16:22:11.714857 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.714860 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.714868 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.714881 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.714883 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.714884 7f869af78700 20 osd.26 243 _dispatch 0x30dc3180 pg_info(1 pgs e243:3.6f) v4 2014-06-09 16:22:11.714889 7f869af78700 5 -- op tracker -- , seq: 2743, time: 2014-06-09 16:22:11.566206, event: header_read, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.714893 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3406.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.714898 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.714896 7f869af78700 5 -- op tracker -- , seq: 2743, time: 2014-06-09 16:22:11.566207, event: throttled, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.714794 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:07.616563 2014-06-09 16:22:11.714917 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.714933 7f86a5610700 10 _do_op 0x2fcae120 seq 3406 r = 0, finisher 0x3062a640 0 2014-06-09 16:22:11.714938 7f86a5610700 10 _finish_op 0x2fcae120 seq 3406 osr(1.26c 0x384cf50)/0x384cf50 2014-06-09 16:22:11.714943 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0d80 (1 active) 2014-06-09 16:22:11.714945 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca48480 (2 active) 2014-06-09 16:22:11.714948 7f86a5610700 5 _do_op 0x288f9b00 seq 3411 osr(3.799 0x3ed1fa0)/0x3ed1fa0 start 2014-06-09 16:22:11.714940 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.714948 7f868c75b700 10 op_submit_start 3459 2014-06-09 16:22:11.714951 7f86a5610700 10 _do_transaction on 0xc31e0c0 2014-06-09 16:22:11.714951 7f868c75b700 5 queue_transactions (trailing journal) 3459 2014-06-09 16:22:11.714953 7f868c75b700 5 queue_op 0x24104f60 seq 3459 osr(3.718 0x3ed1f70) 1053 bytes (queue has 50 ops and 2666539 bytes) 2014-06-09 16:22:11.714957 7f868c75b700 10 op_submit_finish 3459 2014-06-09 16:22:11.714959 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.714951 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 6 on osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.714960 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.714972 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.714985 7f868c75b700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.714968 7f8689f56700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 6 o 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.714990 7f868c75b700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.714994 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.714994 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.906881 7 0.000199 2014-06-09 16:22:11.714903 7f869af78700 5 -- op tracker -- , seq: 2743, time: 2014-06-09 16:22:11.566220, event: all_read, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.715009 7f869af78700 5 -- op tracker -- , seq: 2743, time: 2014-06-09 16:22:11.714880, event: dispatched, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.715005 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.715012 7f86a5610700 15 _omap_rmkeyrange meta/1fe0baaa/pglog_3.799/0//-1 [0000000213.00000000000000000662,4294967295.18446744073709551615] 2014-06-09 16:22:11.715015 7f86a5610700 15 get_omap_iterator meta/1fe0baaa/pglog_3.799/0//-1 2014-06-09 16:22:11.715016 7f869af78700 5 -- op tracker -- , seq: 2743, time: 2014-06-09 16:22:11.715014, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.715017 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.715021 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6f) v4 from osd.6 2014-06-09 16:22:11.715025 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc3180 2014-06-09 16:22:11.715025 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.715038 7f869af78700 5 -- op tracker -- , seq: 2743, time: 2014-06-09 16:22:11.715035, event: started, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.715035 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.715045 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.715055 7f869af78700 5 -- op tracker -- , seq: 2743, time: 2014-06-09 16:22:11.715053, event: done, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.715064 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.715065 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.715080 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.715082 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.715083 7f869af78700 20 osd.26 243 _dispatch 0x2176da40 pg_info(1 pgs e243:2.6b0) v4 2014-06-09 16:22:11.715092 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.715088 7f869af78700 5 -- op tracker -- , seq: 2744, time: 2014-06-09 16:22:11.575059, event: header_read, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.715098 7f868c75b700 5 queue_transactions existing osr(0.6f0 0x384c170)/0x384c170 2014-06-09 16:22:11.715100 7f868c75b700 2 waiting 51 > 50 ops || 2668493 > 104857600 2014-06-09 16:22:11.715105 7f868ef60700 2 waiting 51 > 50 ops || 2667592 > 104857600 2014-06-09 16:22:11.715103 7f86a5610700 15 _omap_rmkeys meta/1fe0baaa/pglog_3.799/0//-1 2014-06-09 16:22:11.715103 7f869af78700 5 -- op tracker -- , seq: 2744, time: 2014-06-09 16:22:11.575060, event: throttled, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.715110 7f868cf5c700 2 waiting 51 > 50 ops || 2668493 > 104857600 2014-06-09 16:22:11.715115 7f8692f68700 2 waiting 51 > 50 ops || 2668699 > 104857600 2014-06-09 16:22:11.715119 7f8691765700 2 waiting 51 > 50 ops || 2668265 > 104857600 2014-06-09 16:22:11.715125 7f8691f66700 2 waiting 51 > 50 ops || 2668548 > 104857600 2014-06-09 16:22:11.715110 7f869af78700 5 -- op tracker -- , seq: 2744, time: 2014-06-09 16:22:11.575071, event: all_read, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.715133 7f868f761700 2 waiting 51 > 50 ops || 2667968 > 104857600 2014-06-09 16:22:11.715136 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe0baaa/pglog_3.799/0//-1 2014-06-09 16:22:11.715139 7f8690f64700 2 waiting 51 > 50 ops || 2668548 > 104857600 2014-06-09 16:22:11.715137 7f869af78700 5 -- op tracker -- , seq: 2744, time: 2014-06-09 16:22:11.715080, event: dispatched, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.715145 7f868ff62700 2 waiting 51 > 50 ops || 2667400 > 104857600 2014-06-09 16:22:11.715147 7f86a5610700 10 _omap_rmkeys meta/1fe0baaa/pglog_3.799/0//-1 r = 0 2014-06-09 16:22:11.715147 7f869af78700 5 -- op tracker -- , seq: 2744, time: 2014-06-09 16:22:11.715145, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.715150 7f8692767700 2 waiting 51 > 50 ops || 2668536 > 104857600 2014-06-09 16:22:11.715153 7f86a5610700 15 _omap_rmkeys meta/1fe0baaa/pglog_3.799/0//-1 2014-06-09 16:22:11.715152 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6b0) v4 from osd.0 2014-06-09 16:22:11.715155 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176da40 2014-06-09 16:22:11.715156 7f868bf5a700 2 waiting 51 > 50 ops || 3021183 > 104857600 2014-06-09 16:22:11.715157 7f86a5610700 10 _omap_rmkeys meta/1fe0baaa/pglog_3.799/0//-1 r = 0 2014-06-09 16:22:11.715160 7f869af78700 5 -- op tracker -- , seq: 2744, time: 2014-06-09 16:22:11.715158, event: started, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.715167 7f86a5610700 15 _omap_setkeys meta/1fe0baaa/pglog_3.799/0//-1 2014-06-09 16:22:11.715179 7f869af78700 5 -- op tracker -- , seq: 2744, time: 2014-06-09 16:22:11.715177, event: done, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.715188 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3411.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.715192 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.715194 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.715195 7f86a5610700 10 cid: oid: 1fe0baaa/pglog_3.799/0//-1 skipping op, spos 3411.0.0 <= header.spos 149904.0.0 2014-06-09 16:22:11.715199 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.715206 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.715208 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.715210 7f869af78700 20 osd.26 243 _dispatch 0x30dc2e00 pg_info(1 pgs e243:2.487) v4 2014-06-09 16:22:11.715215 7f869af78700 5 -- op tracker -- , seq: 2745, time: 2014-06-09 16:22:11.567083, event: header_read, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.715225 7f869af78700 5 -- op tracker -- , seq: 2745, time: 2014-06-09 16:22:11.567083, event: throttled, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.715233 7f86a5610700 10 _do_op 0x288f9b00 seq 3411 r = 0, finisher 0x30c543a0 0 2014-06-09 16:22:11.715237 7f86a5610700 10 _finish_op 0x288f9b00 seq 3411 osr(3.799 0x3ed1fa0)/0x3ed1fa0 2014-06-09 16:22:11.715243 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca48480 (1 active) 2014-06-09 16:22:11.715246 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301680 (2 active) 2014-06-09 16:22:11.715246 7f868c75b700 10 op_submit_start 3460 2014-06-09 16:22:11.715248 7f868c75b700 5 queue_transactions (trailing journal) 3460 2014-06-09 16:22:11.715248 7f86a5610700 5 _do_op 0x30607b00 seq 3412 osr(3.6bb 0x3ed1cd0)/0x3ed1cd0 start 2014-06-09 16:22:11.715251 7f86a5610700 10 _do_transaction on 0xbffef00 2014-06-09 16:22:11.715250 7f868c75b700 5 queue_op 0x2ea532c0 seq 3460 osr(0.6f0 0x384c170) 1954 bytes (queue has 50 ops and 2666333 bytes) 2014-06-09 16:22:11.715253 7f868c75b700 10 op_submit_finish 3460 2014-06-09 16:22:11.715256 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.715257 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.1c8( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.715271 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.715283 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.715291 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.715288 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.491( v 214'640 (0'0,214'640] local-les=208 n=123 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.715308 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3412.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.715302 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.715312 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.715313 7f868c75b700 10 log is not dirty 2014-06-09 16:22:11.715319 7f868c75b700 20 osd.26 243 Sending info 3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) to shard 3 2014-06-09 16:22:11.715324 7f868c75b700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.715339 7f86a5610700 10 _do_op 0x30607b00 seq 3412 r = 0, finisher 0xbef19e0 0 2014-06-09 16:22:11.715343 7f86a5610700 10 _finish_op 0x30607b00 seq 3412 osr(3.6bb 0x3ed1cd0)/0x3ed1cd0 2014-06-09 16:22:11.715345 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301680 (1 active) 2014-06-09 16:22:11.715346 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c0480 (2 active) 2014-06-09 16:22:11.715348 7f86a5610700 5 _do_op 0x2fcafce0 seq 3413 osr(2.492 0x2ed1410)/0x2ed1410 start 2014-06-09 16:22:11.715350 7f86a5610700 10 _do_transaction on 0xc3f2000 2014-06-09 16:22:11.715355 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.715355 7f868c75b700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40a20 (19 active) 2014-06-09 16:22:11.715363 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.715368 7f868c75b700 10 osd.26 243 dequeue_op 0x2f097f00 prio 63 cost 1548288 latency 0.015806 osd_op(client.4538.0:36624 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~1548288] 3.923772b RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.715390 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.715392 7f868ef60700 10 op_submit_start 3461 2014-06-09 16:22:11.715394 7f868ef60700 5 queue_transactions (trailing journal) 3461 2014-06-09 16:22:11.715395 7f868ef60700 5 queue_op 0x30032cc0 seq 3461 osr(3.75f 0x3ed1e60) 1053 bytes (queue has 50 ops and 2666051 bytes) 2014-06-09 16:22:11.715398 7f868ef60700 10 op_submit_finish 3461 2014-06-09 16:22:11.715407 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3413.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.715412 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.715402 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.715416 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.715425 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.715432 7f86a5610700 10 _do_op 0x2fcafce0 seq 3413 r = 0, finisher 0x31456c80 0 2014-06-09 16:22:11.715436 7f86a5610700 10 _finish_op 0x2fcafce0 seq 3413 osr(2.492 0x2ed1410)/0x2ed1410 2014-06-09 16:22:11.715432 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.715439 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c0480 (1 active) 2014-06-09 16:22:11.715441 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6ea0 (2 active) 2014-06-09 16:22:11.715443 7f86a5610700 5 _do_op 0x2fcae660 seq 3414 osr(0.2c6 0x2bd6b80)/0x2bd6b80 start 2014-06-09 16:22:11.715446 7f86a5610700 10 _do_transaction on 0x2c81200 2014-06-09 16:22:11.715445 7f868ef60700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.715450 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.715462 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:173 2014-06-09 16:22:11.715477 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.715483 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.715481 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.715493 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.715500 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3414.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.715505 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.715503 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.715511 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.715524 7f868ef60700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.715539 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.715556 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.715561 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.715569 7f86a5610700 10 _do_op 0x2fcae660 seq 3414 r = 0, finisher 0x307f0d20 0 2014-06-09 16:22:11.715577 7f86a5610700 10 _finish_op 0x2fcae660 seq 3414 osr(0.2c6 0x2bd6b80)/0x2bd6b80 2014-06-09 16:22:11.715579 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6ea0 (1 active) 2014-06-09 16:22:11.715581 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286900 (2 active) 2014-06-09 16:22:11.715574 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.715582 7f86a5610700 5 _do_op 0x15996ba0 seq 3415 osr(1.6b6 0x384dff0)/0x384dff0 start 2014-06-09 16:22:11.715584 7f86a5610700 10 _do_transaction on 0xc226fc0 2014-06-09 16:22:11.715584 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.715590 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.715592 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.715603 7f868ef60700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.715637 7f868cf5c700 10 op_submit_start 3462 2014-06-09 16:22:11.715640 7f868cf5c700 5 queue_transactions (trailing journal) 3462 2014-06-09 16:22:11.715641 7f868cf5c700 5 queue_op 0x30d96480 seq 3462 osr(0.512 0x2bd7590) 1954 bytes (queue has 49 ops and 2664239 bytes) 2014-06-09 16:22:11.715644 7f868cf5c700 10 op_submit_finish 3462 2014-06-09 16:22:11.715648 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.715662 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (19 active) 2014-06-09 16:22:11.715666 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3415.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.715658 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.715671 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.715679 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (20 active) 2014-06-09 16:22:11.715691 7f86a5610700 10 _do_op 0x15996ba0 seq 3415 r = 0, finisher 0x307f1f00 0 2014-06-09 16:22:11.715695 7f86a5610700 10 _finish_op 0x15996ba0 seq 3415 osr(1.6b6 0x384dff0)/0x384dff0 2014-06-09 16:22:11.715685 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.715697 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286900 (1 active) 2014-06-09 16:22:11.715698 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.715699 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89a5a0 (2 active) 2014-06-09 16:22:11.715701 7f86a5610700 5 _do_op 0x2fcaefc0 seq 3416 osr(1.582 0x384da30)/0x384da30 start 2014-06-09 16:22:11.715704 7f86a5610700 10 _do_transaction on 0xbf53500 2014-06-09 16:22:11.715700 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.715709 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.715702 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.715711 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.715717 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.715721 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.715727 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.715731 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.715736 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.715742 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.715741 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.715748 7f868ef60700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.715756 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3416.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.715761 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.715757 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:972 2014-06-09 16:22:11.715767 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.715771 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.715777 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.715786 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.715788 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.715789 7f86a5610700 10 _do_op 0x2fcaefc0 seq 3416 r = 0, finisher 0xcd325a0 0 2014-06-09 16:22:11.715231 7f869af78700 5 -- op tracker -- , seq: 2745, time: 2014-06-09 16:22:11.567096, event: all_read, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.715793 7f86a5610700 10 _finish_op 0x2fcaefc0 seq 3416 osr(1.582 0x384da30)/0x384da30 2014-06-09 16:22:11.715796 7f8692f68700 10 op_submit_start 3463 2014-06-09 16:22:11.715796 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89a5a0 (1 active) 2014-06-09 16:22:11.715797 7f8692f68700 5 queue_transactions (trailing journal) 3463 2014-06-09 16:22:11.715798 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1146c0 (2 active) 2014-06-09 16:22:11.715799 7f8692f68700 5 queue_op 0xd04f620 seq 3463 osr(3.4c0 0x3ed1440) 2160 bytes (queue has 48 ops and 2662448 bytes) 2014-06-09 16:22:11.715800 7f86a5610700 5 _do_op 0x308ca840 seq 3417 osr(3.f7 0x3ed0860)/0x3ed0860 start 2014-06-09 16:22:11.715802 7f8692f68700 10 op_submit_finish 3463 2014-06-09 16:22:11.715803 7f86a5610700 10 _do_transaction on 0xbe1c9c0 2014-06-09 16:22:11.715809 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.715800 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.715811 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x8f3e000 (0 active) 2014-06-09 16:22:11.715815 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0xa160000 (1 active) 2014-06-09 16:22:11.715817 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.715814 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.715824 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.715825 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.715832 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.715841 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.715845 7f868ef60700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.715839 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.715848 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.715859 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3417.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.715857 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.715864 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.715860 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.715865 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.715877 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.715874 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.715887 7f86a5610700 10 _do_op 0x308ca840 seq 3417 r = 0, finisher 0x314acce0 0 2014-06-09 16:22:11.715891 7f86a5610700 10 _finish_op 0x308ca840 seq 3417 osr(3.f7 0x3ed0860)/0x3ed0860 2014-06-09 16:22:11.715882 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.715893 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1146c0 (1 active) 2014-06-09 16:22:11.715887 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.715896 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b320 (2 active) 2014-06-09 16:22:11.715898 7f86a5610700 5 _do_op 0xc64b860 seq 3418 osr(3.72c 0x3ed1ee0)/0x3ed1ee0 start 2014-06-09 16:22:11.715901 7f86a5610700 10 _do_transaction on 0xc33da40 2014-06-09 16:22:11.715895 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.715897 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.715906 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.715904 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.715906 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.715915 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.715912 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.715923 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.715919 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 3 notify: (query_epoch:242, epoch_sent:243, info:1.1bc( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.715932 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.715942 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.715940 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:183 2014-06-09 16:22:11.715955 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.715947 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.715961 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.715959 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.715960 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.428( v 214'563 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.715968 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.715977 7f868ef60700 7 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.6 2014-06-09 16:22:11.715984 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3418.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.715990 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.715977 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.715996 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.715989 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.6 info 3.428( v 214'563 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242), identical to ours 2014-06-09 16:22:11.716008 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.716007 7f86a5610700 10 _do_op 0xc64b860 seq 3418 r = 0, finisher 0x307f1360 0 2014-06-09 16:22:11.716015 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:189 2014-06-09 16:22:11.716030 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.716032 7f86a5610700 10 _finish_op 0xc64b860 seq 3418 osr(3.72c 0x3ed1ee0)/0x3ed1ee0 2014-06-09 16:22:11.716039 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b320 (1 active) 2014-06-09 16:22:11.716041 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52fb00 (2 active) 2014-06-09 16:22:11.716034 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.716044 7f86a5610700 5 _do_op 0x308ca7e0 seq 3419 osr(2.55b 0x2ed1930)/0x2ed1930 start 2014-06-09 16:22:11.716045 7f8691765700 10 op_submit_start 3464 2014-06-09 16:22:11.716047 7f86a5610700 10 _do_transaction on 0xbf7c9c0 2014-06-09 16:22:11.716048 7f8691765700 5 queue_transactions (trailing journal) 3464 2014-06-09 16:22:11.716049 7f8691765700 5 queue_op 0x2fc8c6c0 seq 3464 osr(3.5f9 0x3ed1b40) 1726 bytes (queue has 48 ops and 2663704 bytes) 2014-06-09 16:22:11.716053 7f8691765700 10 op_submit_finish 3464 2014-06-09 16:22:11.716053 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.716055 7f868f761700 10 op_submit_start 3465 2014-06-09 16:22:11.716048 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.716057 7f868f761700 5 queue_transactions (trailing journal) 3465 2014-06-09 16:22:11.716059 7f868f761700 5 queue_op 0x30d96a20 seq 3465 osr(3.342 0x3ed10e0) 1429 bytes (queue has 50 ops and 2667142 bytes) 2014-06-09 16:22:11.716060 7f868ff62700 2 waiting 51 > 50 ops || 2668003 > 104857600 2014-06-09 16:22:11.716062 7f868f761700 10 op_submit_finish 3465 2014-06-09 16:22:11.716066 7f8690f64700 10 op_submit_start 3466 2014-06-09 16:22:11.716058 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716067 7f8690f64700 5 queue_transactions (trailing journal) 3466 2014-06-09 16:22:11.716059 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.716068 7f8690f64700 5 queue_op 0x2fcaf620 seq 3466 osr(1.5cb 0x384dd70) 2009 bytes (queue has 50 ops and 2667142 bytes) 2014-06-09 16:22:11.716071 7f8690f64700 10 op_submit_finish 3466 2014-06-09 16:22:11.716067 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716067 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.697( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:0.697( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.716072 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.716074 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.716082 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.716084 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.716080 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716082 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.697( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.716088 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716094 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.716085 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.716096 7f8690f64700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.716100 7f8690f64700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.716100 7f8691765700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.716096 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1089 2014-06-09 16:22:11.716106 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3419.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.716099 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.3d7( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.716111 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.716103 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.466007 7 0.000197 2014-06-09 16:22:11.716113 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.716112 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:167 2014-06-09 16:22:11.716114 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.716113 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.716125 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.716127 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.716117 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.716124 7f8690f64700 5 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.716128 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.716133 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.716131 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.716138 7f86a5610700 10 _do_op 0x308ca7e0 seq 3419 r = 0, finisher 0xcd32f60 0 2014-06-09 16:22:11.716131 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 7 notify: (query_epoch:243, epoch_sent:243, info:0.537( empty local-les=210 n=0 ec=1 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.716141 7f86a5610700 10 _finish_op 0x308ca7e0 seq 3419 osr(2.55b 0x2ed1930)/0x2ed1930 2014-06-09 16:22:11.716140 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.716141 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.716141 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716143 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.716152 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52fb00 (1 active) 2014-06-09 16:22:11.716154 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.716149 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716149 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.716150 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716156 7f868bf5a700 10 op_submit_start 3467 2014-06-09 16:22:11.716157 7f868ff62700 2 waiting 51 > 50 ops || 3020693 > 104857600 2014-06-09 16:22:11.716158 7f868bf5a700 5 queue_transactions (trailing journal) 3467 2014-06-09 16:22:11.716154 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc360 (2 active) 2014-06-09 16:22:11.716160 7f868bf5a700 5 queue_op 0xc34ae40 seq 3467 osr(3.1e 0x3ed0380) 354644 bytes (queue has 50 ops and 3019832 bytes) 2014-06-09 16:22:11.716156 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716162 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.716163 7f868bf5a700 10 op_submit_finish 3467 2014-06-09 16:22:11.716162 7f86a5610700 5 _do_op 0x308caea0 seq 3420 osr(3.5ce 0x3ed17c0)/0x3ed17c0 start 2014-06-09 16:22:11.716165 7f86a5610700 10 _do_transaction on 0xc2dba40 2014-06-09 16:22:11.716158 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=201 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.716169 7f8691765700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.716170 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.716170 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=201 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.716175 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:205 2014-06-09 16:22:11.716166 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xbf2e6c0 243'593 rep_tid=5 committed?=0 applied?=0 lock=0 op=osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.716186 7f868bf5a700 10 osd.26 243 dequeue_op 0x19fe0a00 finish 2014-06-09 16:22:11.716188 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.716182 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.716192 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.716183 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=201 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.025566 7 0.000191 2014-06-09 16:22:11.716197 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.716198 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.716192 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 0 2014-06-09 16:22:11.716202 7f8690f64700 5 queue_transactions existing osr(1.35c 0x384d2b0)/0x384d2b0 2014-06-09 16:22:11.716196 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=201 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.716204 7f8690f64700 2 waiting 51 > 50 ops || 3021806 > 104857600 2014-06-09 16:22:11.716209 7f8692767700 2 waiting 51 > 50 ops || 3021829 > 104857600 2014-06-09 16:22:11.716207 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=201 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.716203 7f868bf5a700 10 osd.26 243 dequeue_op 0x306aa00 prio 63 cost 516096 latency 1.759825 osd_op(client.4573.0:33016 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 425984~516096] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.716215 7f8691f66700 10 op_submit_start 3468 2014-06-09 16:22:11.716206 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.0: 3.74f( v 213'530 (0'0,213'530] local-les=191 n=106 ec=118 les/c 191/191 242/242/242) log((0'0,213'530], crt=0'0) missing(0) 2014-06-09 16:22:11.716218 7f8691f66700 5 queue_transactions (trailing journal) 3468 2014-06-09 16:22:11.716219 7f868cf5c700 10 proc_replica_log for osd.0: 3.74f( v 213'530 (0'0,213'530] local-les=191 n=106 ec=118 les/c 191/191 242/242/242) log((0'0,213'530], crt=0'0) missing(0) 2014-06-09 16:22:11.716222 7f8691f66700 5 queue_op 0x230b81e0 seq 3468 osr(1.54a 0x384daf0) 2009 bytes (queue has 50 ops and 3019832 bytes) 2014-06-09 16:22:11.716225 7f8691f66700 10 op_submit_finish 3468 2014-06-09 16:22:11.716218 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=201 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.716222 7f868bf5a700 5 -- op tracker -- , seq: 6, time: 2014-06-09 16:22:11.716217, event: reached_pg, op: osd_op(client.4573.0:33016 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 425984~516096] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.716226 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.716224 7f868cf5c700 20 merge_log cut point (usually last shared) is 213'530 (213'529) modify 3ca15f4f/rbd_data.115b6b8b4567.00000000000018a9/head//3 by client.4539.0:34388 2014-06-09 10:18:06.783418 2014-06-09 16:22:11.716228 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'567 -> 213'567 2014-06-09 16:22:11.716229 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.716229 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.716242 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3420.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.716246 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.716241 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x306aa00 2014-06-09 16:22:11.716246 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.716250 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:33016 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 425984~516096] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.716257 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716268 7f86a5610700 10 _do_op 0x308caea0 seq 3420 r = 0, finisher 0x314ac100 0 2014-06-09 16:22:11.716272 7f86a5610700 10 _finish_op 0x308caea0 seq 3420 osr(3.5ce 0x3ed17c0)/0x3ed17c0 2014-06-09 16:22:11.716264 7f868bf5a700 3 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4573.0:33016 was 213'584 2014-06-09 16:22:11.716266 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716269 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.716283 7f8690f64700 10 op_submit_start 3469 2014-06-09 16:22:11.716284 7f8690f64700 5 queue_transactions (trailing journal) 3469 2014-06-09 16:22:11.716286 7f8690f64700 5 queue_op 0x2fcaf5c0 seq 3469 osr(1.35c 0x384d2b0) 1974 bytes (queue has 50 ops and 3020377 bytes) 2014-06-09 16:22:11.716287 7f868bf5a700 10 osd.26 243 dequeue_op 0x306aa00 finish 2014-06-09 16:22:11.716281 7f8691f66700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.716291 7f8692767700 2 waiting 51 > 50 ops || 3022374 > 104857600 2014-06-09 16:22:11.716295 7f868ff62700 2 waiting 51 > 50 ops || 3021238 > 104857600 2014-06-09 16:22:11.716295 7f868bf5a700 5 -- op tracker -- , seq: 6, time: 2014-06-09 16:22:11.716290, event: done, op: osd_op(client.4573.0:33016 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 425984~516096] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.716303 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc360 (1 active) 2014-06-09 16:22:11.716305 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300360 (2 active) 2014-06-09 16:22:11.716308 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.716310 7f8690f64700 10 op_submit_finish 3469 2014-06-09 16:22:11.716303 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1023 2014-06-09 16:22:11.716320 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.716321 7f86a5610700 5 _do_op 0x230b8cc0 seq 3421 osr(3.6e 0x3ed02a0)/0x3ed02a0 start 2014-06-09 16:22:11.716324 7f86a5610700 10 _do_transaction on 0x3562f00 2014-06-09 16:22:11.716325 7f868f761700 5 queue_transactions existing osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 2014-06-09 16:22:11.716327 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) 2014-06-09 16:22:11.716329 7f868f761700 2 waiting 51 > 50 ops || 3021712 > 104857600 2014-06-09 16:22:11.716330 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.716332 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.716346 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.716345 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.716351 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0240 (19 active) 2014-06-09 16:22:11.716358 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.716354 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716352 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.716361 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716365 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.716372 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.716375 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716387 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.716383 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716386 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.716399 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.716402 7f8691f66700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.716405 7f86a5610700 15 _omap_rmkeyrange meta/516fccbc/pglog_3.6e/0//-1 [0000000214.00000000000000000610,4294967295.18446744073709551615] 2014-06-09 16:22:11.716408 7f86a5610700 15 get_omap_iterator meta/516fccbc/pglog_3.6e/0//-1 2014-06-09 16:22:11.716418 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.716434 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.716439 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.716436 7f8689755700 10 osd.26 243 dequeue_op 0x19fe0900 prio 63 cost 516096 latency 1.261495 osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.716451 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.716458 7f8689755700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.716453, event: reached_pg, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.716461 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716473 7f86a5610700 15 _omap_rmkeys meta/516fccbc/pglog_3.6e/0//-1 2014-06-09 16:22:11.716465 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.716469 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716481 7f8691f66700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.716477 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x19fe0900 2014-06-09 16:22:11.716492 7f86a5610700 10 lookup_strip_header done cid meta oid 516fccbc/pglog_3.6e/0//-1 2014-06-09 16:22:11.716498 7f86a5610700 10 _omap_rmkeys meta/516fccbc/pglog_3.6e/0//-1 r = 0 2014-06-09 16:22:11.716486 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.716501 7f86a5610700 15 _omap_rmkeys meta/516fccbc/pglog_3.6e/0//-1 2014-06-09 16:22:11.716494 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.716504 7f86a5610700 10 _omap_rmkeys meta/516fccbc/pglog_3.6e/0//-1 r = 0 2014-06-09 16:22:11.716509 7f86a5610700 15 _omap_setkeys meta/516fccbc/pglog_3.6e/0//-1 2014-06-09 16:22:11.716510 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.716505 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x30b5e580 2014-06-09 16:22:11.716519 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3421.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.716523 7f86a5610700 10 cid: oid: 516fccbc/pglog_3.6e/0//-1 skipping op, spos 3421.0.0 <= header.spos 150420.0.0 2014-06-09 16:22:11.716524 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.716514 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.716515 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30b5e580 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 rwstate(write n=1 w=0) oi: 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3(243'593 client.4571.0:35947 wrlock_by=unknown.0.0:0 dirty s 352256 uv593) ssc: 0x2e8769a0 snapset: 0=[]:[]+head 2014-06-09 16:22:11.716528 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.716530 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 @head oi=78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3(243'593 client.4571.0:35947 wrlock_by=unknown.0.0:0 dirty s 352256 uv593) 2014-06-09 16:22:11.716537 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716549 7f86a5610700 10 _do_op 0x230b8cc0 seq 3421 r = 0, finisher 0x314ac9c0 0 2014-06-09 16:22:11.716552 7f86a5610700 10 _finish_op 0x230b8cc0 seq 3421 osr(3.6e 0x3ed02a0)/0x3ed02a0 2014-06-09 16:22:11.716545 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716551 7f8689755700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.716546, event: started, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.716556 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300360 (1 active) 2014-06-09 16:22:11.716557 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.716558 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9239e0 (2 active) 2014-06-09 16:22:11.716561 7f86a5610700 5 _do_op 0x309a3320 seq 3422 osr(3.46f 0x3ed1560)/0x3ed1560 start 2014-06-09 16:22:11.716561 7f868ff62700 10 op_submit_start 3470 2014-06-09 16:22:11.716563 7f868ff62700 5 queue_transactions (trailing journal) 3470 2014-06-09 16:22:11.716563 7f86a5610700 10 _do_transaction on 0x336acc0 2014-06-09 16:22:11.716564 7f868ff62700 5 queue_op 0x30d970e0 seq 3470 osr(2.349 0x2ed0e90) 861 bytes (queue has 50 ops and 3019301 bytes) 2014-06-09 16:22:11.716557 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30718c00 2014-06-09 16:22:11.716567 7f868ff62700 10 op_submit_finish 3470 2014-06-09 16:22:11.716568 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.716575 7f8689755700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.716570, event: started, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.716571 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.716572 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.716587 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.716584 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.236( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.716581 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] ov 243'593 av 243'594 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.716595 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.716597 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.716591 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:243, info:1.700( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.716596 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 2014-06-09 16:22:11.716600 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.716602 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.716613 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.716612 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3422.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.716607 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.716616 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.716611 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.716620 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.716619 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 352256~516096 2014-06-09 16:22:11.716617 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.17( v 213'672 (0'0,213'672] local-les=243 n=137 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.17( v 213'672 (0'0,213'672] local-les=243 n=137 ec=118 les/c 243/191 242/242/220) 2014-06-09 16:22:11.716624 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.716633 7f86a5610700 10 _do_op 0x309a3320 seq 3422 r = 0, finisher 0x314adf20 0 2014-06-09 16:22:11.716636 7f86a5610700 10 _finish_op 0x309a3320 seq 3422 osr(3.46f 0x3ed1560)/0x3ed1560 2014-06-09 16:22:11.716638 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9239e0 (1 active) 2014-06-09 16:22:11.716640 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f680 (2 active) 2014-06-09 16:22:11.716636 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.716634 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228e00 2014-06-09 16:22:11.716642 7f86a5610700 5 _do_op 0x309a32c0 seq 3423 osr(1.504 0x384dbd0)/0x384dbd0 start 2014-06-09 16:22:11.716645 7f86a5610700 10 _do_transaction on 0x399de00 2014-06-09 16:22:11.716643 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.716650 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.716645 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 snapset=0x2e8769d8 snapc=0=[] 2014-06-09 16:22:11.716651 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.716664 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.716656 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.716660 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.716669 7f8691f66700 5 queue_transactions existing osr(3.17 0x3ed03b0)/0x3ed03b0 2014-06-09 16:22:11.716672 7f8691f66700 10 op_submit_start 3471 2014-06-09 16:22:11.716673 7f8691f66700 5 queue_transactions (trailing journal) 3471 2014-06-09 16:22:11.716668 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.716666 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 0x30718c00 op modify 2014-06-09 16:22:11.716674 7f8691f66700 5 queue_op 0x288f9080 seq 3471 osr(3.17 0x3ed03b0) 826 bytes (queue has 50 ops and 3018792 bytes) 2014-06-09 16:22:11.716677 7f8691f66700 10 op_submit_finish 3471 2014-06-09 16:22:11.716679 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.716681 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:189 2014-06-09 16:22:11.716680 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:1.b6( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.716694 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3423.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.716697 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.716686 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:41.407582 2014-06-09 16:22:11.716692 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.716694 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.716705 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.716700 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.716709 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.716698 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.0 now 3.74f( v 213'530 (0'0,213'530] local-les=191 n=106 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.716706 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.716717 7f86a5610700 10 _do_op 0x309a32c0 seq 3423 r = 0, finisher 0x314adca0 0 2014-06-09 16:22:11.716709 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.716713 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.716721 7f86a5610700 10 _finish_op 0x309a32c0 seq 3423 osr(1.504 0x384dbd0)/0x384dbd0 2014-06-09 16:22:11.716715 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.716724 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f680 (1 active) 2014-06-09 16:22:11.716726 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad680 (2 active) 2014-06-09 16:22:11.716728 7f86a5610700 5 _do_op 0x309a3260 seq 3424 osr(3.2df 0x3ed0e40)/0x3ed0e40 start 2014-06-09 16:22:11.716722 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.716731 7f86a5610700 10 _do_transaction on 0x39da900 2014-06-09 16:22:11.716724 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.456( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.716734 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.716722 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.716727 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.494577 4 0.003550 2014-06-09 16:22:11.716735 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.716732 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 7 on osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.716739 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.716737 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.948760 7 0.000231 2014-06-09 16:22:11.716738 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:1.293( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.716747 7f8689755700 7 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 7 o 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.716748 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.850979 0 0.000000 2014-06-09 16:22:11.716751 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.716750 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.716761 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.716765 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.716761 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.716763 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.716769 7f8689755700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.716762, event: waiting for subops from 9, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.716765 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.4f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.716770 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=191 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.716780 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3424.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.716783 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.716775 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.716785 7f8689755700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.716787 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'636 -> 213'636 2014-06-09 16:22:11.716798 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.716791 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.716807 7f8692767700 10 op_submit_start 3472 2014-06-09 16:22:11.716808 7f8692767700 5 queue_transactions (trailing journal) 3472 2014-06-09 16:22:11.716802 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.716810 7f8692767700 5 queue_op 0xc2d4720 seq 3472 osr(2.482 0x2ed14f0) 1997 bytes (queue has 50 ops and 3018792 bytes) 2014-06-09 16:22:11.716811 7f86a5610700 10 _do_op 0x309a3260 seq 3424 r = 0, finisher 0x314127e0 0 2014-06-09 16:22:11.716813 7f8692767700 10 op_submit_finish 3472 2014-06-09 16:22:11.716814 7f86a5610700 10 _finish_op 0x309a3260 seq 3424 osr(3.2df 0x3ed0e40)/0x3ed0e40 2014-06-09 16:22:11.716810 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 213'540 -> 213'540 2014-06-09 16:22:11.716819 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad680 (1 active) 2014-06-09 16:22:11.716821 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89afc0 (2 active) 2014-06-09 16:22:11.716807 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'593 (0'0,243'593] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'593], crt=0'0) [243'594 (243'593) modify 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 by client.4571.0:35948 2014-06-09 10:19:41.407582] 2014-06-09 16:22:11.716823 7f86a5610700 5 _do_op 0xc64b920 seq 3425 osr(0.4d5 0x2bd7610)/0x2bd7610 start 2014-06-09 16:22:11.716825 7f86a5610700 10 _do_transaction on 0xbfcde00 2014-06-09 16:22:11.716816 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.716830 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.716831 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.716827 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'594 (243'593) modify 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 by client.4571.0:35948 2014-06-09 10:19:41.407582 2014-06-09 16:22:11.716833 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.716840 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.716847 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.716849 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716849 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.0 3.74f( v 213'530 (0'0,213'530] local-les=191 n=106 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.716861 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.716863 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.716873 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.716875 7f8689755700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'594, trimmed: 2014-06-09 16:22:11.716876 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3425.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.716880 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.716875 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.0 sending log((213'530,213'540], crt=0'0) 2014-06-09 16:22:11.716880 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1021 2014-06-09 16:22:11.716894 7f8689755700 5 queue_transactions existing osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.720105 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.716895 7f868ff62700 5 queue_transactions existing osr(2.4f4 0x2ed16b0)/0x2ed16b0 2014-06-09 16:22:11.720112 7f8689755700 2 waiting 51 > 50 ops || 3537172 > 104857600 2014-06-09 16:22:11.720114 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720116 7f868ff62700 2 waiting 51 > 50 ops || 3019620 > 104857600 2014-06-09 16:22:11.716897 7f86a5610700 10 _do_op 0xc64b920 seq 3425 r = 0, finisher 0x307f1060 0 2014-06-09 16:22:11.716899 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0xa160000 (0 active) 2014-06-09 16:22:11.720119 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.720129 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.720124 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.0 3.74f( v 213'540 lc 213'530 (0'0,213'540] local-les=191 n=106 ec=118 les/c 191/191 242/242/242) missing missing(2) 2014-06-09 16:22:11.720133 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.716902 7f86a5e11700 10 _do_op 0x30607920 seq 3410 r = 0, finisher 0xc2d7840 0 2014-06-09 16:22:11.716911 7f868f761700 10 op_submit_start 3473 2014-06-09 16:22:11.720143 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720152 7f868f761700 5 queue_transactions (trailing journal) 3473 2014-06-09 16:22:11.720155 7f868f761700 5 queue_op 0xbe7cba0 seq 3473 osr(3.1a0 0x3ed0ab0) 1335 bytes (queue has 50 ops and 3018792 bytes) 2014-06-09 16:22:11.720151 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.716912 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.720142 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.0 has 2 missing 2014-06-09 16:22:11.720123 7f86a5610700 10 _finish_op 0xc64b920 seq 3425 osr(0.4d5 0x2bd7610)/0x2bd7610 2014-06-09 16:22:11.720165 7f8692767700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.720164 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 3ca15f4f/rbd_data.115b6b8b4567.00000000000018a9/head//3 213'534 is on osd.26 2014-06-09 16:22:11.720173 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89afc0 (1 active) 2014-06-09 16:22:11.720178 7f868ff62700 2 waiting 51 > 50 ops || 3536117 > 104857600 2014-06-09 16:22:11.720166 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.246( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.716930 7f8691f66700 5 queue_transactions existing osr(3.1b5 0x3ed0a80)/0x3ed0a80 2014-06-09 16:22:11.720176 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52fd40 (2 active) 2014-06-09 16:22:11.720184 7f8691f66700 2 waiting 51 > 50 ops || 3536718 > 104857600 2014-06-09 16:22:11.720184 7f86a5610700 5 _do_op 0x30d97c20 seq 3426 osr(0.33a 0x2bd70b0)/0x2bd70b0 start 2014-06-09 16:22:11.720176 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing f6bfff4f/rbd_data.115e6b8b4567.00000000000017fa/head//3 213'540 is on osd.26 2014-06-09 16:22:11.720187 7f86a5610700 10 _do_transaction on 0xbfe7a40 2014-06-09 16:22:11.720182 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.720194 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.720194 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.716925 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.720188 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {3ca15f4f/rbd_data.115b6b8b4567.00000000000018a9/head//3=213'534(213'530),f6bfff4f/rbd_data.115e6b8b4567.00000000000017fa/head//3=213'540} 2014-06-09 16:22:11.720200 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 0 as a source 2014-06-09 16:22:11.720203 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.720210 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 3ca15f4f/rbd_data.115b6b8b4567.00000000000018a9/head//3 213'534 also missing on osd.0 2014-06-09 16:22:11.720213 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720214 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1112 2014-06-09 16:22:11.720222 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.720229 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.720222 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing f6bfff4f/rbd_data.115e6b8b4567.00000000000017fa/head//3 213'540 also missing on osd.0 2014-06-09 16:22:11.720232 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720236 7f868ef60700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.720233 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {3ca15f4f/rbd_data.115b6b8b4567.00000000000018a9/head//3=213'534(213'530),f6bfff4f/rbd_data.115e6b8b4567.00000000000017fa/head//3=213'540} 2014-06-09 16:22:11.720245 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3426.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.720250 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.720250 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f23c0 (19 active) 2014-06-09 16:22:11.720244 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.720251 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.720253 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.720263 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f23c0 (20 active) 2014-06-09 16:22:11.716936 7f869af78700 5 -- op tracker -- , seq: 2745, time: 2014-06-09 16:22:11.715206, event: dispatched, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.720264 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 0 2014-06-09 16:22:11.720274 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.720270 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.720279 7f86a5610700 10 _do_op 0x30d97c20 seq 3426 r = 0, finisher 0x33a7a60 0 2014-06-09 16:22:11.720274 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.720283 7f86a5610700 10 _finish_op 0x30d97c20 seq 3426 osr(0.33a 0x2bd70b0)/0x2bd70b0 2014-06-09 16:22:11.720279 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.720284 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.720285 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.720292 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52fd40 (1 active) 2014-06-09 16:22:11.720294 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720295 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.720296 7f86a5610700 5 _do_op 0x308ca9c0 seq 3427 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.720299 7f86a5610700 10 _do_transaction on 0xbf58300 2014-06-09 16:22:11.720292 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.720298 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.720302 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720310 7f86a5610700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.720310 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.720310 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.720319 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720322 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.720323 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.720334 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.720342 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720337 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.720349 7f86a5610700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.720353 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.720346 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.720362 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.720357 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.720358 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.720367 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720367 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.720379 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.720375 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.720382 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.720386 7f8692767700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.720391 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.720385 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.763( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.828972 7 0.000190 2014-06-09 16:22:11.720397 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.720396 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.763( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.720408 7f868ff62700 2 waiting 51 > 50 ops || 3535549 > 104857600 2014-06-09 16:22:11.720405 7f86a5610700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.720407 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.763( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.720417 7f86a5610700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.720423 7f86a5610700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.720417 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.720427 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.720437 7f86a5610700 15 _write 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 962560~516096 2014-06-09 16:22:11.720436 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.720436 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.720452 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720456 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.720468 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.720476 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720485 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.720483 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.720490 7f868ef60700 5 queue_transactions existing osr(0.763 0x384c350)/0x384c350 2014-06-09 16:22:11.720492 7f868ef60700 2 waiting 51 > 50 ops || 3536604 > 104857600 2014-06-09 16:22:11.720493 7f8692767700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.720496 7f86a5610700 10 lookup_strip_header done cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.716947 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 1 notify: (query_epoch:242, epoch_sent:243, info:1.4c2( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.720511 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.720508 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.720522 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.720528 7f869af78700 5 -- op tracker -- , seq: 2745, time: 2014-06-09 16:22:11.720525, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.720535 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.720550 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720546 7f868c75b700 5 -- op tracker -- , seq: 2731, time: 2014-06-09 16:22:11.715387, event: reached_pg, op: osd_op(client.4538.0:36624 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~1548288] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.720554 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.720556 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.720566 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.720565 7f8689f56700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.714997, event: waiting for subops from 7, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.720575 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720571 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x2f097f00 2014-06-09 16:22:11.720584 7f868c75b700 10 osd.26 243 dequeue_op 0x2f097f00 finish 2014-06-09 16:22:11.720582 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.720589 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.720593 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.720589 7f8682de8700 5 -- op tracker -- , seq: 2746, time: 2014-06-09 16:22:11.720412, event: header_read, op: osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.720593 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.720597 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.487) v4 from osd.5 2014-06-09 16:22:11.720600 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc2e00 2014-06-09 16:22:11.720598 7f8682de8700 5 -- op tracker -- , seq: 2746, time: 2014-06-09 16:22:11.720413, event: throttled, op: osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.720591 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.72b( v 213'642 (0'0,213'642] local-les=191 n=121 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.720604 7f86a5e11700 10 _finish_op 0x30607920 seq 3410 osr(0.1cd 0x2bd6960)/0x2bd6960 2014-06-09 16:22:11.720610 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3bd40 (1 active) 2014-06-09 16:22:11.720608 7f8682de8700 5 -- op tracker -- , seq: 2746, time: 2014-06-09 16:22:11.720422, event: all_read, op: osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.720612 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77be60 (2 active) 2014-06-09 16:22:11.720605 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.720606 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.720616 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.720616 7f868f761700 10 op_submit_finish 3473 2014-06-09 16:22:11.720615 7f868ef60700 2 waiting 51 > 50 ops || 3535431 > 104857600 2014-06-09 16:22:11.720619 7f8682de8700 5 -- op tracker -- , seq: 2746, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.720621 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720625 7f8682de8700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.720622 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.720628 7f869af78700 5 -- op tracker -- , seq: 2745, time: 2014-06-09 16:22:11.720602, event: started, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.720624 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.720630 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:11.720634 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (19 active) 2014-06-09 16:22:11.720637 7f8682de8700 15 osd.26 243 enqueue_op 0x22009c00 prio 196 cost 0 latency 0.000225 osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.720636 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.720633 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.720646 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0270 (20 active) 2014-06-09 16:22:11.720645 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720646 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.927684 7 0.000156 2014-06-09 16:22:11.720652 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.720658 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.720652 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 5 notify: (query_epoch:242, epoch_sent:243, info:3.701( v 213'541 (0'0,213'541] local-les=206 n=108 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.720663 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.720659 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.720670 7f869af78700 5 -- op tracker -- , seq: 2745, time: 2014-06-09 16:22:11.720668, event: done, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.720667 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.720671 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.720678 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.720673 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.720680 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.720682 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.720686 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720681 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.720683 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.720677 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'640 (0'0,243'640] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'640], crt=0'0) [243'641 (243'639) modify 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 by client.4570.0:34071 2014-06-09 10:20:07.616563] 2014-06-09 16:22:11.720697 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.720699 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.720691 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'631 -> 213'631 2014-06-09 16:22:11.720690 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.720700 7f869af78700 20 osd.26 243 _dispatch 0x30cf8540 pg_info(1 pgs e243:0.31b) v4 2014-06-09 16:22:11.720695 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.720705 7f869af78700 5 -- op tracker -- , seq: 2747, time: 2014-06-09 16:22:11.303197, event: header_read, op: pg_info(1 pgs e243:0.31b) 2014-06-09 16:22:11.720703 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.720704 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.720699 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'641 (243'639) modify 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 by client.4570.0:34071 2014-06-09 10:20:07.616563 2014-06-09 16:22:11.720712 7f869af78700 5 -- op tracker -- , seq: 2747, time: 2014-06-09 16:22:11.303198, event: throttled, op: pg_info(1 pgs e243:0.31b) 2014-06-09 16:22:11.720711 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.720718 7f869af78700 5 -- op tracker -- , seq: 2747, time: 2014-06-09 16:22:11.303203, event: all_read, op: pg_info(1 pgs e243:0.31b) 2014-06-09 16:22:11.720714 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720719 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.720724 7f869af78700 5 -- op tracker -- , seq: 2747, time: 2014-06-09 16:22:11.720697, event: dispatched, op: pg_info(1 pgs e243:0.31b) 2014-06-09 16:22:11.720719 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.720723 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.720730 7f869af78700 5 -- op tracker -- , seq: 2747, time: 2014-06-09 16:22:11.720729, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.31b) 2014-06-09 16:22:11.720727 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.720735 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.31b) v4 from osd.2 2014-06-09 16:22:11.720735 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.720738 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf8540 2014-06-09 16:22:11.720734 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.720743 7f869af78700 5 -- op tracker -- , seq: 2747, time: 2014-06-09 16:22:11.720741, event: started, op: pg_info(1 pgs e243:0.31b) 2014-06-09 16:22:11.720740 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.720749 7f868f761700 20 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.720749 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:885 2014-06-09 16:22:11.720758 7f869af78700 5 -- op tracker -- , seq: 2747, time: 2014-06-09 16:22:11.720756, event: done, op: pg_info(1 pgs e243:0.31b) 2014-06-09 16:22:11.720757 7f868f761700 20 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.720764 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.720765 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.720766 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.720767 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.720774 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.720770 7f8689f56700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'641, trimmed: 2014-06-09 16:22:11.720769 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.720768 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.720779 7f868bf5a700 5 queue_transactions existing osr(3.465 0x3ed1580)/0x3ed1580 2014-06-09 16:22:11.720781 7f868bf5a700 2 waiting 51 > 50 ops || 3534883 > 104857600 2014-06-09 16:22:11.720784 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.720786 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.720780 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.720786 7f8689755700 10 op_submit_start 3474 2014-06-09 16:22:11.720788 7f8689755700 5 queue_transactions (trailing journal) 3474 2014-06-09 16:22:11.720780 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.720787 7f869af78700 20 osd.26 243 _dispatch 0x48fca00 pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.720789 7f8689755700 5 queue_op 0x2fcae480 seq 3474 osr(3.1e 0x3ed0380) 518380 bytes (queue has 50 ops and 3533548 bytes) 2014-06-09 16:22:11.720792 7f8689755700 10 op_submit_finish 3474 2014-06-09 16:22:11.720792 7f8689f56700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.720787 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.720794 7f8689f56700 2 waiting 51 > 50 ops || 5084120 > 104857600 2014-06-09 16:22:11.720790 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.720794 7f869af78700 5 -- op tracker -- , seq: 2748, time: 2014-06-09 16:22:11.422986, event: header_read, op: pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.720795 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.720799 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.720803 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.720811 7f8692767700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.720811 7f869af78700 5 -- op tracker -- , seq: 2748, time: 2014-06-09 16:22:11.422986, event: throttled, op: pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.720798 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc049740 243'594 rep_tid=7 committed?=0 applied?=0 lock=0 op=osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.720810 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.720819 7f8689755700 10 osd.26 243 dequeue_op 0x19fe0900 finish 2014-06-09 16:22:11.720820 7f869af78700 5 -- op tracker -- , seq: 2748, time: 2014-06-09 16:22:11.423002, event: all_read, op: pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.720825 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.720822 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.720828 7f869af78700 5 -- op tracker -- , seq: 2748, time: 2014-06-09 16:22:11.720784, event: dispatched, op: pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.720831 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.720831 7f868f761700 20 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.720837 7f869af78700 5 -- op tracker -- , seq: 2748, time: 2014-06-09 16:22:11.720835, event: waiting_for_osdmap, op: pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.720839 7f868f761700 20 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.720844 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.720835 7f8689755700 10 osd.26 243 dequeue_op 0x306a800 prio 63 cost 1548288 latency 1.762365 osd_op(client.4573.0:33017 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1548288] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.720848 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.720847 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x48fca00 2014-06-09 16:22:11.720853 7f86a5e11700 5 _do_op 0xc7045a0 seq 3428 osr(2.4b9 0x2ed1730)/0x2ed1730 start 2014-06-09 16:22:11.720857 7f86a5e11700 10 _do_transaction on 0xbf90840 2014-06-09 16:22:11.720855 7f8670072700 5 -- op tracker -- , seq: 2749, time: 2014-06-09 16:22:11.710595, event: header_read, op: osd_op(client.4538.0:36625 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1032192] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.720865 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.720870 7f8670072700 5 -- op tracker -- , seq: 2749, time: 2014-06-09 16:22:11.710596, event: throttled, op: osd_op(client.4538.0:36625 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1032192] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.720882 7f8670072700 5 -- op tracker -- , seq: 2749, time: 2014-06-09 16:22:11.712242, event: all_read, op: osd_op(client.4538.0:36625 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1032192] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.720893 7f8670072700 5 -- op tracker -- , seq: 2749, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36625 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1032192] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.720900 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.720921 7f8670072700 15 osd.26 243 enqueue_op 0x2f097e00 prio 63 cost 1032192 latency 0.010326 osd_op(client.4538.0:36625 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1032192] 3.923772b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.721176 7f8670072700 5 -- op tracker -- , seq: 2750, time: 2014-06-09 16:22:11.720944, event: header_read, op: osd_op(client.4538.0:36626 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 4055040~139264] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.721198 7f8670072700 5 -- op tracker -- , seq: 2750, time: 2014-06-09 16:22:11.720946, event: throttled, op: osd_op(client.4538.0:36626 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 4055040~139264] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.721209 7f8670072700 5 -- op tracker -- , seq: 2750, time: 2014-06-09 16:22:11.721149, event: all_read, op: osd_op(client.4538.0:36626 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 4055040~139264] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.721220 7f8670072700 5 -- op tracker -- , seq: 2750, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36626 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 4055040~139264] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.721225 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.721249 7f8670072700 15 osd.26 243 enqueue_op 0x257c4200 prio 63 cost 139264 latency 0.000305 osd_op(client.4538.0:36626 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 4055040~139264] 3.923772b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.721886 7f8691f66700 10 op_submit_start 3475 2014-06-09 16:22:11.721889 7f8691f66700 5 queue_transactions (trailing journal) 3475 2014-06-09 16:22:11.721892 7f8691f66700 5 queue_op 0x288f9e00 seq 3475 osr(3.1b5 0x3ed0a80) 1429 bytes (queue has 50 ops and 3533548 bytes) 2014-06-09 16:22:11.721896 7f8691f66700 10 op_submit_finish 3475 2014-06-09 16:22:11.721903 7f868ff62700 10 op_submit_start 3476 2014-06-09 16:22:11.721905 7f868ff62700 5 queue_transactions (trailing journal) 3476 2014-06-09 16:22:11.721907 7f868ff62700 5 queue_op 0x30607bc0 seq 3476 osr(2.4f4 0x2ed16b0) 828 bytes (queue has 50 ops and 3533548 bytes) 2014-06-09 16:22:11.721910 7f868ff62700 10 op_submit_finish 3476 2014-06-09 16:22:11.721904 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:0.19a( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.721894 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:914 2014-06-09 16:22:11.721914 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.721921 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.721931 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.721933 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.721930 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.721940 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.721936 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.721938 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.721948 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.721944 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.721956 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.721951 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.721957 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.721960 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.721961 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.721965 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.721969 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.721972 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.721974 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.721983 7f8691f66700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.721984 7f868ff62700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.721998 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.722000 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.722009 7f868cf5c700 5 queue_transactions existing osr(3.74f 0x3ed1ea0)/0x3ed1ea0 2014-06-09 16:22:11.722013 7f868cf5c700 2 waiting 51 > 50 ops || 3534601 > 104857600 2014-06-09 16:22:11.722015 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.722007 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 1 notify: (query_epoch:242, epoch_sent:243, info:2.1fa( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.722023 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.722026 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:164 2014-06-09 16:22:11.722022 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.51e( v 213'674 (0'0,213'674] local-les=191 n=127 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.722040 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.722036 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.722039 7f8691765700 7 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.722044 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.722045 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.722053 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.722055 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.722064 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.722066 7f8691f66700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.722052 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.722050 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.51e( v 213'674 (0'0,213'674] local-les=191 n=127 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.722074 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.722068 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.722073 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.722085 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722079 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.722089 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.722083 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:175 2014-06-09 16:22:11.722099 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.722090 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.722093 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.722104 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.722111 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722106 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.677501 6 0.000629 2014-06-09 16:22:11.722114 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.722119 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.722121 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.722119 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.25a( v 213'576 (0'0,213'576] local-les=191 n=111 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.722130 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.722129 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.722132 7f8691765700 7 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.722138 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.722151 7f868f761700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.722141 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:11.722159 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.722164 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:173 2014-06-09 16:22:11.722168 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.722141 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.25a( v 213'576 (0'0,213'576] local-les=191 n=111 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.722177 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.722179 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722175 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.722182 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=203 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.722184 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:243, info:1.1da( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.722187 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.722193 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=203 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.722197 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.722196 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.722207 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722206 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.722206 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=203 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.997444 7 0.000218 2014-06-09 16:22:11.722217 7f868ff62700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.722212 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.722217 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=203 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.722224 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.722227 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:180 2014-06-09 16:22:11.722234 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.722229 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=203 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.722242 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.722254 7f868f761700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.722248 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=203 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.722258 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'660 -> 213'660 2014-06-09 16:22:11.722268 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.722271 7f868ff62700 5 queue_transactions existing osr(2.4d 0x2ed05d0)/0x2ed05d0 2014-06-09 16:22:11.722265 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.722273 7f868ff62700 2 waiting 51 > 50 ops || 3534406 > 104857600 2014-06-09 16:22:11.722280 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722284 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:11.722297 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.722303 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.722307 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.722317 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.722330 7f868f761700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.722344 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:932 2014-06-09 16:22:11.722360 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722363 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.722374 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.722383 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.722391 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.722405 7f868f761700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.722423 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.722438 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722442 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.722458 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.722468 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.722483 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.722477 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.722488 7f8691765700 5 queue_transactions existing osr(3.2a9 0x3ed0b40)/0x3ed0b40 2014-06-09 16:22:11.722489 7f868f761700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.722491 7f8691765700 2 waiting 51 > 50 ops || 3534883 > 104857600 2014-06-09 16:22:11.722498 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.722501 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:916 2014-06-09 16:22:11.722519 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722514 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.117( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.722524 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.722523 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.7ce( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.7ce( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 243/191 242/242/220) 2014-06-09 16:22:11.722528 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.722540 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.722549 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.722553 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.4b3( v 213'517 (0'0,213'517] local-les=191 n=98 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.722569 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.722567 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.722574 7f868f761700 5 queue_transactions existing osr(3.7ce 0xb7f01a0)/0xb7f01a0 2014-06-09 16:22:11.722577 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.722577 7f868f761700 2 waiting 51 > 50 ops || 3534376 > 104857600 2014-06-09 16:22:11.722581 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.722591 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.722603 7f8692767700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.722607 7f8692767700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.722024 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.722611 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.438( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.033250 7 0.000197 2014-06-09 16:22:11.722619 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.722622 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.438( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.722624 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 1 notify: (query_epoch:242, epoch_sent:243, info:3.78a( v 213'545 (0'0,213'545] local-les=191 n=110 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.722632 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.438( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.722638 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 1, already have info from that osd, ignoring 2014-06-09 16:22:11.722641 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.722649 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.722650 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.722658 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.722654 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 7 notify: (query_epoch:242, epoch_sent:243, info:3.4c0( v 214'586 (0'0,214'586] local-les=210 n=110 ec=118 les/c 210/210 242/242/242)) 2014-06-09 16:22:11.722679 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] state: Active: got notify from 7, already have info from that osd, ignoring 2014-06-09 16:22:11.722692 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.722696 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.722708 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.722718 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.722727 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.722732 7f8692767700 5 queue_transactions existing osr(1.438 0x384d710)/0x384d710 2014-06-09 16:22:11.722734 7f8692767700 2 waiting 51 > 50 ops || 3535545 > 104857600 2014-06-09 16:22:11.722727 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.722740 7f8692f68700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.722756 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:871 2014-06-09 16:22:11.722772 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.722776 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.722789 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.722798 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.722806 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.722818 7f8692f68700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.722753 7f869af78700 5 -- op tracker -- , seq: 2748, time: 2014-06-09 16:22:11.720850, event: started, op: pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.722856 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.722870 7f8689755700 5 -- op tracker -- , seq: 19, time: 2014-06-09 16:22:11.720848, event: reached_pg, op: osd_op(client.4573.0:33017 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1548288] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.722876 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.722881 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:2.621( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.722880 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.722894 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.722906 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.722897 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x306a800 2014-06-09 16:22:11.722910 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=203 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.722909 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:33017 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1548288] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.722928 7f869af78700 5 -- op tracker -- , seq: 2748, time: 2014-06-09 16:22:11.722924, event: done, op: pg_log(3.424 epoch 243 log log((214'729,214'730], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.722924 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=203 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.722928 7f8689755700 3 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4573.0:33017 was 213'585 2014-06-09 16:22:11.722941 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.722944 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.722938 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=203 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.958827 7 0.000191 2014-06-09 16:22:11.722956 7f8689755700 10 osd.26 243 dequeue_op 0x306a800 finish 2014-06-09 16:22:11.722950 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=203 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.722958 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.722960 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.722961 7f869af78700 20 osd.26 243 _dispatch 0x310e8c40 pg_info(1 pgs e243:2.5d9) v4 2014-06-09 16:22:11.722967 7f8689755700 5 -- op tracker -- , seq: 19, time: 2014-06-09 16:22:11.722962, event: done, op: osd_op(client.4573.0:33017 rbd_data.117d6b8b4567.0000000000001613 [set-alloc-hint object_size 4194304 write_size 4194304,write 942080~1548288] 3.ceb8701e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.722967 7f868c75b700 10 osd.26 243 dequeue_op 0x22009c00 prio 196 cost 0 latency 0.002554 osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 pg pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.722988 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.722986 7f868c75b700 5 -- op tracker -- , seq: 2746, time: 2014-06-09 16:22:11.722983, event: reached_pg, op: osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.723001 7f869af78700 5 -- op tracker -- , seq: 2751, time: 2014-06-09 16:22:11.578875, event: header_read, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.722997 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x22009c00 2014-06-09 16:22:11.723009 7f869af78700 5 -- op tracker -- , seq: 2751, time: 2014-06-09 16:22:11.578875, event: throttled, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.723011 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e770 (20 active) 2014-06-09 16:22:11.723017 7f869af78700 5 -- op tracker -- , seq: 2751, time: 2014-06-09 16:22:11.578890, event: all_read, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.723023 7f869af78700 5 -- op tracker -- , seq: 2751, time: 2014-06-09 16:22:11.722957, event: dispatched, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.723016 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.723030 7f869af78700 5 -- op tracker -- , seq: 2751, time: 2014-06-09 16:22:11.723028, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.723034 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5d9) v4 from osd.7 2014-06-09 16:22:11.723030 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.723037 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310e8c40 2014-06-09 16:22:11.723041 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723043 7f869af78700 5 -- op tracker -- , seq: 2751, time: 2014-06-09 16:22:11.723041, event: started, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.723046 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.723058 7f869af78700 5 -- op tracker -- , seq: 2751, time: 2014-06-09 16:22:11.723056, event: done, op: pg_info(1 pgs e243:2.5d9) 2014-06-09 16:22:11.723058 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.723068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.723070 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.723067 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.723074 7f868c75b700 5 -- op tracker -- , seq: 2746, time: 2014-06-09 16:22:11.723009, event: started, op: osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.723074 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.723085 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.723086 7f8689755700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.723087 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.723088 7f869af78700 20 osd.26 243 _dispatch 0x30eba380 pg_info(1 pgs e243:2.5f8) v4 2014-06-09 16:22:11.723082 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 5 op ack_type 4 from 9 2014-06-09 16:22:11.723093 7f869af78700 5 -- op tracker -- , seq: 2752, time: 2014-06-09 16:22:11.570668, event: header_read, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.723100 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=203 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.723101 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.723110 7f868c75b700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.723097, event: sub_op_commit_rec, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.723116 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723112 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=203 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.723119 7f86a5610700 10 _generic_write 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 962560~516096 = 0 2014-06-09 16:22:11.723123 7f868c75b700 10 osd.26 243 dequeue_op 0x22009c00 finish 2014-06-09 16:22:11.723122 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'653 -> 213'653 2014-06-09 16:22:11.723129 7f868c75b700 5 -- op tracker -- , seq: 2746, time: 2014-06-09 16:22:11.723126, event: done, op: osd_sub_op_reply(client.4571.0:35947 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.723143 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.723147 7f869af78700 5 -- op tracker -- , seq: 2752, time: 2014-06-09 16:22:11.570669, event: throttled, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.723151 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.723155 7f869af78700 5 -- op tracker -- , seq: 2752, time: 2014-06-09 16:22:11.570680, event: all_read, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.723150 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.723162 7f869af78700 5 -- op tracker -- , seq: 2752, time: 2014-06-09 16:22:11.723085, event: dispatched, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.723164 7f86a5610700 15 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.723155 7f868c75b700 10 osd.26 243 dequeue_op 0x19fe0800 prio 63 cost 516096 latency 1.242586 osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.723164 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.431( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.723168 7f869af78700 5 -- op tracker -- , seq: 2752, time: 2014-06-09 16:22:11.723166, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.723174 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723167 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.723180 7f86a5610700 10 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.723178 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.723186 7f86a5610700 15 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.723188 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.375( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.723197 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723196 7f86a5610700 10 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.723210 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.723200 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.723215 7f8692f68700 5 queue_transactions existing osr(3.1b1 0x3ed0a70)/0x3ed0a70 2014-06-09 16:22:11.723218 7f8692f68700 2 waiting 51 > 50 ops || 3534883 > 104857600 2014-06-09 16:22:11.723213 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.723217 7f868c75b700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.723171, event: reached_pg, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.723224 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723227 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.723225 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.723239 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.723239 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x19fe0800 2014-06-09 16:22:11.723250 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723254 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.723250 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.723265 7f86a5610700 10 cid: oid: 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 not skipping op, *spos 3427.0.0 2014-06-09 16:22:11.723269 7f86a5610700 10 > header.spos 3358.0.0 2014-06-09 16:22:11.723264 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.723273 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723272 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x30b5e580 2014-06-09 16:22:11.723276 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.723289 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.723299 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723282 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30b5e580 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 rwstate(write n=2 w=0) oi: 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3(243'594 client.4571.0:35948 wrlock_by=unknown.0.0:0 dirty s 868352 uv594) ssc: 0x2e8769a0 snapset: 0=[]:[]+head 2014-06-09 16:22:11.723302 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.723312 7f86a5610700 20 set_header setting 30151 cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 parent seq 0 2014-06-09 16:22:11.723313 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.723322 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723310 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 @head oi=78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3(243'594 client.4571.0:35948 wrlock_by=unknown.0.0:0 dirty s 868352 uv594) 2014-06-09 16:22:11.723325 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.723336 7f868c75b700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.723331, event: started, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.723336 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.723347 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.723351 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.723343 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30a81e00 2014-06-09 16:22:11.723354 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.298( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.415327 7 0.000739 2014-06-09 16:22:11.723360 7f868c75b700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.723355, event: started, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.723367 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3427.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.723364 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.298( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.723372 7f86a5610700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3427.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.723375 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.723366 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] ov 243'594 av 243'595 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.723374 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.298( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.723383 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.723391 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.723379 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 2014-06-09 16:22:11.723399 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.723404 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.723418 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 868352~516096 2014-06-09 16:22:11.723433 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228e00 2014-06-09 16:22:11.723453 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.723445 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 snapset=0x2e8769d8 snapc=0=[] 2014-06-09 16:22:11.723457 7f8689755700 5 queue_transactions existing osr(0.298 0x2bd6c60)/0x2bd6c60 2014-06-09 16:22:11.723460 7f8689755700 2 waiting 51 > 50 ops || 3535424 > 104857600 2014-06-09 16:22:11.723457 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.723467 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 0x30a81e00 op modify 2014-06-09 16:22:11.723479 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:41.414980 2014-06-09 16:22:11.723503 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.723520 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.723530 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 8 on osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.723548 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 8 o 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.723569 7f868c75b700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.723564, event: waiting for subops from 9, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.723586 7f868c75b700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.723609 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'594 (0'0,243'594] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'594], crt=0'0) [243'595 (243'594) modify 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 by client.4571.0:35949 2014-06-09 10:19:41.414980] 2014-06-09 16:22:11.723651 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'595 (243'594) modify 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 by client.4571.0:35949 2014-06-09 10:19:41.414980 2014-06-09 16:22:11.723671 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.723700 7f868c75b700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'595, trimmed: 2014-06-09 16:22:11.723730 7f868c75b700 5 queue_transactions existing osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.723733 7f868c75b700 2 waiting 51 > 50 ops || 4051928 > 104857600 2014-06-09 16:22:11.723835 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5f8) v4 from osd.3 2014-06-09 16:22:11.723840 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30eba380 2014-06-09 16:22:11.723845 7f869af78700 5 -- op tracker -- , seq: 2752, time: 2014-06-09 16:22:11.723843, event: started, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.723862 7f869af78700 5 -- op tracker -- , seq: 2752, time: 2014-06-09 16:22:11.723860, event: done, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.723875 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.723878 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.723893 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.723894 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.723895 7f869af78700 20 osd.26 243 _dispatch 0x15458700 pg_info(1 pgs e243:3.7fc) v4 2014-06-09 16:22:11.723900 7f869af78700 5 -- op tracker -- , seq: 2753, time: 2014-06-09 16:22:11.563687, event: header_read, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.723909 7f869af78700 5 -- op tracker -- , seq: 2753, time: 2014-06-09 16:22:11.563687, event: throttled, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.723915 7f869af78700 5 -- op tracker -- , seq: 2753, time: 2014-06-09 16:22:11.563700, event: all_read, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.723922 7f869af78700 5 -- op tracker -- , seq: 2753, time: 2014-06-09 16:22:11.723892, event: dispatched, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.723936 7f869af78700 5 -- op tracker -- , seq: 2753, time: 2014-06-09 16:22:11.723934, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.723941 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7fc) v4 from osd.1 2014-06-09 16:22:11.723944 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15458700 2014-06-09 16:22:11.723949 7f869af78700 5 -- op tracker -- , seq: 2753, time: 2014-06-09 16:22:11.723947, event: started, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.723966 7f869af78700 5 -- op tracker -- , seq: 2753, time: 2014-06-09 16:22:11.723964, event: done, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.723999 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724001 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724017 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724018 7f869af78700 20 osd.26 243 _dispatch 0x30cf1500 pg_info(1 pgs e243:2.220) v4 2014-06-09 16:22:11.724022 7f869af78700 5 -- op tracker -- , seq: 2754, time: 2014-06-09 16:22:11.577683, event: header_read, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.724029 7f869af78700 5 -- op tracker -- , seq: 2754, time: 2014-06-09 16:22:11.577684, event: throttled, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.724036 7f869af78700 5 -- op tracker -- , seq: 2754, time: 2014-06-09 16:22:11.577696, event: all_read, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.724043 7f869af78700 5 -- op tracker -- , seq: 2754, time: 2014-06-09 16:22:11.724015, event: dispatched, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.724050 7f869af78700 5 -- op tracker -- , seq: 2754, time: 2014-06-09 16:22:11.724048, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.724055 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.220) v4 from osd.9 2014-06-09 16:22:11.724058 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf1500 2014-06-09 16:22:11.724062 7f869af78700 5 -- op tracker -- , seq: 2754, time: 2014-06-09 16:22:11.724060, event: started, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.724079 7f869af78700 5 -- op tracker -- , seq: 2754, time: 2014-06-09 16:22:11.724077, event: done, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.724086 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724102 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724104 7f869af78700 20 osd.26 243 _dispatch 0x2eef1dc0 pg_info(1 pgs e243:3.1bd) v4 2014-06-09 16:22:11.724108 7f869af78700 5 -- op tracker -- , seq: 2755, time: 2014-06-09 16:22:11.434855, event: header_read, op: pg_info(1 pgs e243:3.1bd) 2014-06-09 16:22:11.724115 7f869af78700 5 -- op tracker -- , seq: 2755, time: 2014-06-09 16:22:11.434856, event: throttled, op: pg_info(1 pgs e243:3.1bd) 2014-06-09 16:22:11.724121 7f869af78700 5 -- op tracker -- , seq: 2755, time: 2014-06-09 16:22:11.434868, event: all_read, op: pg_info(1 pgs e243:3.1bd) 2014-06-09 16:22:11.724126 7f869af78700 5 -- op tracker -- , seq: 2755, time: 2014-06-09 16:22:11.724101, event: dispatched, op: pg_info(1 pgs e243:3.1bd) 2014-06-09 16:22:11.724132 7f869af78700 5 -- op tracker -- , seq: 2755, time: 2014-06-09 16:22:11.724130, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1bd) 2014-06-09 16:22:11.724142 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1bd) v4 from osd.8 2014-06-09 16:22:11.724145 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef1dc0 2014-06-09 16:22:11.724150 7f869af78700 5 -- op tracker -- , seq: 2755, time: 2014-06-09 16:22:11.724148, event: started, op: pg_info(1 pgs e243:3.1bd) 2014-06-09 16:22:11.724167 7f869af78700 5 -- op tracker -- , seq: 2755, time: 2014-06-09 16:22:11.724165, event: done, op: pg_info(1 pgs e243:3.1bd) 2014-06-09 16:22:11.724174 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724176 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724187 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724188 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724189 7f869af78700 20 osd.26 243 _dispatch 0x30eba000 pg_info(1 pgs e243:2.f2) v4 2014-06-09 16:22:11.724193 7f8670072700 5 -- op tracker -- , seq: 2756, time: 2014-06-09 16:22:11.721269, event: header_read, op: osd_op(client.4538.0:36645 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1019904] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.724207 7f8670072700 5 -- op tracker -- , seq: 2756, time: 2014-06-09 16:22:11.721269, event: throttled, op: osd_op(client.4538.0:36645 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1019904] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.724219 7f8670072700 5 -- op tracker -- , seq: 2756, time: 2014-06-09 16:22:11.724165, event: all_read, op: osd_op(client.4538.0:36645 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1019904] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.724232 7f8670072700 5 -- op tracker -- , seq: 2756, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36645 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1019904] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.724240 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.724241 7f869af78700 5 -- op tracker -- , seq: 2757, time: 2014-06-09 16:22:11.566233, event: header_read, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.724250 7f869af78700 5 -- op tracker -- , seq: 2757, time: 2014-06-09 16:22:11.566233, event: throttled, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.724256 7f869af78700 5 -- op tracker -- , seq: 2757, time: 2014-06-09 16:22:11.566244, event: all_read, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.724262 7f869af78700 5 -- op tracker -- , seq: 2757, time: 2014-06-09 16:22:11.724186, event: dispatched, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.724264 7f8670072700 15 osd.26 243 enqueue_op 0x1b0d7800 prio 63 cost 1019904 latency 0.002995 osd_op(client.4538.0:36645 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1019904] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.724271 7f869af78700 5 -- op tracker -- , seq: 2757, time: 2014-06-09 16:22:11.724269, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.724276 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.f2) v4 from osd.6 2014-06-09 16:22:11.724279 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30eba000 2014-06-09 16:22:11.724283 7f869af78700 5 -- op tracker -- , seq: 2757, time: 2014-06-09 16:22:11.724282, event: started, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.724304 7f869af78700 5 -- op tracker -- , seq: 2757, time: 2014-06-09 16:22:11.724302, event: done, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.724311 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724312 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724325 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724326 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724327 7f869af78700 20 osd.26 243 _dispatch 0x2fe71dc0 pg_info(1 pgs e243:2.6b0) v4 2014-06-09 16:22:11.724332 7f869af78700 5 -- op tracker -- , seq: 2758, time: 2014-06-09 16:22:11.575079, event: header_read, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.724340 7f869af78700 5 -- op tracker -- , seq: 2758, time: 2014-06-09 16:22:11.575080, event: throttled, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.724347 7f869af78700 5 -- op tracker -- , seq: 2758, time: 2014-06-09 16:22:11.575093, event: all_read, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.724358 7f869af78700 5 -- op tracker -- , seq: 2758, time: 2014-06-09 16:22:11.724324, event: dispatched, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.724365 7f869af78700 5 -- op tracker -- , seq: 2758, time: 2014-06-09 16:22:11.724363, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.724371 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.6b0) v4 from osd.0 2014-06-09 16:22:11.724374 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe71dc0 2014-06-09 16:22:11.724378 7f869af78700 5 -- op tracker -- , seq: 2758, time: 2014-06-09 16:22:11.724376, event: started, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.724395 7f869af78700 5 -- op tracker -- , seq: 2758, time: 2014-06-09 16:22:11.724393, event: done, op: pg_info(1 pgs e243:2.6b0) 2014-06-09 16:22:11.724402 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724404 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724418 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724419 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724421 7f869af78700 20 osd.26 243 _dispatch 0x30ebbc00 pg_info(1 pgs e243:2.fd) v4 2014-06-09 16:22:11.724425 7f869af78700 5 -- op tracker -- , seq: 2759, time: 2014-06-09 16:22:11.567159, event: header_read, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.724432 7f869af78700 5 -- op tracker -- , seq: 2759, time: 2014-06-09 16:22:11.567159, event: throttled, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.724440 7f869af78700 5 -- op tracker -- , seq: 2759, time: 2014-06-09 16:22:11.567171, event: all_read, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.724446 7f869af78700 5 -- op tracker -- , seq: 2759, time: 2014-06-09 16:22:11.724417, event: dispatched, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.724453 7f869af78700 5 -- op tracker -- , seq: 2759, time: 2014-06-09 16:22:11.724452, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.724462 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.fd) v4 from osd.5 2014-06-09 16:22:11.724465 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebbc00 2014-06-09 16:22:11.724470 7f869af78700 5 -- op tracker -- , seq: 2759, time: 2014-06-09 16:22:11.724468, event: started, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.724485 7f869af78700 5 -- op tracker -- , seq: 2759, time: 2014-06-09 16:22:11.724483, event: done, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.724493 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724495 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724507 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724509 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724510 7f869af78700 20 osd.26 243 _dispatch 0x30cf8380 pg_info(1 pgs e243:1.345) v4 2014-06-09 16:22:11.724514 7f869af78700 5 -- op tracker -- , seq: 2760, time: 2014-06-09 16:22:11.303207, event: header_read, op: pg_info(1 pgs e243:1.345) 2014-06-09 16:22:11.724521 7f869af78700 5 -- op tracker -- , seq: 2760, time: 2014-06-09 16:22:11.303207, event: throttled, op: pg_info(1 pgs e243:1.345) 2014-06-09 16:22:11.724527 7f869af78700 5 -- op tracker -- , seq: 2760, time: 2014-06-09 16:22:11.303215, event: all_read, op: pg_info(1 pgs e243:1.345) 2014-06-09 16:22:11.724536 7f869af78700 5 -- op tracker -- , seq: 2760, time: 2014-06-09 16:22:11.724507, event: dispatched, op: pg_info(1 pgs e243:1.345) 2014-06-09 16:22:11.724543 7f869af78700 5 -- op tracker -- , seq: 2760, time: 2014-06-09 16:22:11.724541, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.345) 2014-06-09 16:22:11.724547 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.345) v4 from osd.2 2014-06-09 16:22:11.724550 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf8380 2014-06-09 16:22:11.724555 7f869af78700 5 -- op tracker -- , seq: 2760, time: 2014-06-09 16:22:11.724553, event: started, op: pg_info(1 pgs e243:1.345) 2014-06-09 16:22:11.724574 7f869af78700 5 -- op tracker -- , seq: 2760, time: 2014-06-09 16:22:11.724572, event: done, op: pg_info(1 pgs e243:1.345) 2014-06-09 16:22:11.724587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724589 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724602 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724603 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724604 7f869af78700 20 osd.26 243 _dispatch 0x310d6000 pg_info(1 pgs e243:3.327) v4 2014-06-09 16:22:11.724609 7f869af78700 5 -- op tracker -- , seq: 2761, time: 2014-06-09 16:22:11.430043, event: header_read, op: pg_info(1 pgs e243:3.327) 2014-06-09 16:22:11.724617 7f869af78700 5 -- op tracker -- , seq: 2761, time: 2014-06-09 16:22:11.430044, event: throttled, op: pg_info(1 pgs e243:3.327) 2014-06-09 16:22:11.724628 7f869af78700 5 -- op tracker -- , seq: 2761, time: 2014-06-09 16:22:11.430055, event: all_read, op: pg_info(1 pgs e243:3.327) 2014-06-09 16:22:11.724638 7f869af78700 5 -- op tracker -- , seq: 2761, time: 2014-06-09 16:22:11.724601, event: dispatched, op: pg_info(1 pgs e243:3.327) 2014-06-09 16:22:11.724648 7f869af78700 5 -- op tracker -- , seq: 2761, time: 2014-06-09 16:22:11.724646, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.327) 2014-06-09 16:22:11.724652 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.327) v4 from osd.4 2014-06-09 16:22:11.724655 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310d6000 2014-06-09 16:22:11.724660 7f869af78700 5 -- op tracker -- , seq: 2761, time: 2014-06-09 16:22:11.724658, event: started, op: pg_info(1 pgs e243:3.327) 2014-06-09 16:22:11.724676 7f869af78700 5 -- op tracker -- , seq: 2761, time: 2014-06-09 16:22:11.724674, event: done, op: pg_info(1 pgs e243:3.327) 2014-06-09 16:22:11.724683 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724684 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724695 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724697 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724698 7f869af78700 20 osd.26 243 _dispatch 0x2f047dc0 pg_info(1 pgs e243:0.1a4) v4 2014-06-09 16:22:11.724702 7f869af78700 5 -- op tracker -- , seq: 2762, time: 2014-06-09 16:22:11.578911, event: header_read, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.724709 7f869af78700 5 -- op tracker -- , seq: 2762, time: 2014-06-09 16:22:11.578911, event: throttled, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.724715 7f869af78700 5 -- op tracker -- , seq: 2762, time: 2014-06-09 16:22:11.578922, event: all_read, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.724721 7f869af78700 5 -- op tracker -- , seq: 2762, time: 2014-06-09 16:22:11.724695, event: dispatched, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.724729 7f869af78700 5 -- op tracker -- , seq: 2762, time: 2014-06-09 16:22:11.724727, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.724734 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1a4) v4 from osd.7 2014-06-09 16:22:11.724737 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f047dc0 2014-06-09 16:22:11.724741 7f869af78700 5 -- op tracker -- , seq: 2762, time: 2014-06-09 16:22:11.724740, event: started, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.724755 7f869af78700 5 -- op tracker -- , seq: 2762, time: 2014-06-09 16:22:11.724753, event: done, op: pg_info(1 pgs e243:0.1a4) 2014-06-09 16:22:11.724767 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724769 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724780 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724782 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724783 7f869af78700 20 osd.26 243 _dispatch 0x30ebb180 pg_info(1 pgs e243:2.69c) v4 2014-06-09 16:22:11.724788 7f869af78700 5 -- op tracker -- , seq: 2763, time: 2014-06-09 16:22:11.570785, event: header_read, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.724794 7f869af78700 5 -- op tracker -- , seq: 2763, time: 2014-06-09 16:22:11.570786, event: throttled, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.724800 7f869af78700 5 -- op tracker -- , seq: 2763, time: 2014-06-09 16:22:11.570798, event: all_read, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.724810 7f869af78700 5 -- op tracker -- , seq: 2763, time: 2014-06-09 16:22:11.724780, event: dispatched, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.724820 7f869af78700 5 -- op tracker -- , seq: 2763, time: 2014-06-09 16:22:11.724818, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.724825 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.69c) v4 from osd.3 2014-06-09 16:22:11.724828 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebb180 2014-06-09 16:22:11.724832 7f869af78700 5 -- op tracker -- , seq: 2763, time: 2014-06-09 16:22:11.724831, event: started, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.724846 7f869af78700 5 -- op tracker -- , seq: 2763, time: 2014-06-09 16:22:11.724844, event: done, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.724853 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724854 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724866 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724867 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724868 7f869af78700 20 osd.26 243 _dispatch 0x2a070380 pg_info(1 pgs e243:3.7fc) v4 2014-06-09 16:22:11.724873 7f869af78700 5 -- op tracker -- , seq: 2764, time: 2014-06-09 16:22:11.568604, event: header_read, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.724879 7f869af78700 5 -- op tracker -- , seq: 2764, time: 2014-06-09 16:22:11.568605, event: throttled, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.724885 7f869af78700 5 -- op tracker -- , seq: 2764, time: 2014-06-09 16:22:11.568619, event: all_read, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.724891 7f869af78700 5 -- op tracker -- , seq: 2764, time: 2014-06-09 16:22:11.724866, event: dispatched, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.724899 7f869af78700 5 -- op tracker -- , seq: 2764, time: 2014-06-09 16:22:11.724898, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.724904 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7fc) v4 from osd.1 2014-06-09 16:22:11.724906 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a070380 2014-06-09 16:22:11.724911 7f869af78700 5 -- op tracker -- , seq: 2764, time: 2014-06-09 16:22:11.724909, event: started, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.724924 7f869af78700 5 -- op tracker -- , seq: 2764, time: 2014-06-09 16:22:11.724922, event: done, op: pg_info(1 pgs e243:3.7fc) 2014-06-09 16:22:11.724931 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724933 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724943 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.724945 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.724946 7f869af78700 20 osd.26 243 _dispatch 0x30cf16c0 pg_info(1 pgs e243:1.289) v4 2014-06-09 16:22:11.724950 7f869af78700 5 -- op tracker -- , seq: 2765, time: 2014-06-09 16:22:11.577741, event: header_read, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.724957 7f869af78700 5 -- op tracker -- , seq: 2765, time: 2014-06-09 16:22:11.577742, event: throttled, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.724962 7f869af78700 5 -- op tracker -- , seq: 2765, time: 2014-06-09 16:22:11.577754, event: all_read, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.724973 7f869af78700 5 -- op tracker -- , seq: 2765, time: 2014-06-09 16:22:11.724943, event: dispatched, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.724984 7f869af78700 5 -- op tracker -- , seq: 2765, time: 2014-06-09 16:22:11.724982, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.724988 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.289) v4 from osd.9 2014-06-09 16:22:11.724991 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf16c0 2014-06-09 16:22:11.724995 7f869af78700 5 -- op tracker -- , seq: 2765, time: 2014-06-09 16:22:11.724993, event: started, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.725009 7f869af78700 5 -- op tracker -- , seq: 2765, time: 2014-06-09 16:22:11.725007, event: done, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.725016 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725017 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725029 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725031 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725032 7f869af78700 20 osd.26 243 _dispatch 0x3070d800 pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.725039 7f869af78700 5 -- op tracker -- , seq: 2766, time: 2014-06-09 16:22:11.434875, event: header_read, op: pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725046 7f869af78700 5 -- op tracker -- , seq: 2766, time: 2014-06-09 16:22:11.434875, event: throttled, op: pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725053 7f869af78700 5 -- op tracker -- , seq: 2766, time: 2014-06-09 16:22:11.434898, event: all_read, op: pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725059 7f869af78700 5 -- op tracker -- , seq: 2766, time: 2014-06-09 16:22:11.725029, event: dispatched, op: pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725067 7f869af78700 5 -- op tracker -- , seq: 2766, time: 2014-06-09 16:22:11.725064, event: waiting_for_osdmap, op: pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725076 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) v4 from osd.8 2014-06-09 16:22:11.725080 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3070d800 2014-06-09 16:22:11.725086 7f869af78700 5 -- op tracker -- , seq: 2766, time: 2014-06-09 16:22:11.725083, event: started, op: pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725100 7f869af78700 5 -- op tracker -- , seq: 2766, time: 2014-06-09 16:22:11.725097, event: done, op: pg_log(3.1ff epoch 243 log log((214'681,214'683], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725107 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725109 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725119 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725120 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725121 7f869af78700 20 osd.26 243 _dispatch 0x30eba8c0 pg_info(1 pgs e243:2.34d) v4 2014-06-09 16:22:11.725126 7f869af78700 5 -- op tracker -- , seq: 2767, time: 2014-06-09 16:22:11.566249, event: header_read, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.725134 7f869af78700 5 -- op tracker -- , seq: 2767, time: 2014-06-09 16:22:11.566250, event: throttled, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.725140 7f869af78700 5 -- op tracker -- , seq: 2767, time: 2014-06-09 16:22:11.566262, event: all_read, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.725153 7f869af78700 5 -- op tracker -- , seq: 2767, time: 2014-06-09 16:22:11.725118, event: dispatched, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.725161 7f869af78700 5 -- op tracker -- , seq: 2767, time: 2014-06-09 16:22:11.725159, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.725167 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.34d) v4 from osd.6 2014-06-09 16:22:11.725170 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30eba8c0 2014-06-09 16:22:11.725175 7f8682de8700 5 -- op tracker -- , seq: 2768, time: 2014-06-09 16:22:11.725107, event: header_read, op: osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.725193 7f8682de8700 5 -- op tracker -- , seq: 2768, time: 2014-06-09 16:22:11.725108, event: throttled, op: osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.725205 7f8682de8700 5 -- op tracker -- , seq: 2768, time: 2014-06-09 16:22:11.725119, event: all_read, op: osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.725214 7f8682de8700 5 -- op tracker -- , seq: 2768, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.725221 7f8682de8700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.725222 7f869af78700 5 -- op tracker -- , seq: 2767, time: 2014-06-09 16:22:11.725173, event: started, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.725227 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:11.725235 7f8682de8700 15 osd.26 243 enqueue_op 0xdfa6100 prio 196 cost 0 latency 0.000128 osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.725242 7f869af78700 5 -- op tracker -- , seq: 2767, time: 2014-06-09 16:22:11.725240, event: done, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.725250 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725251 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725262 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725264 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725265 7f869af78700 20 osd.26 243 _dispatch 0x30cf0fc0 pg_info(1 pgs e243:1.3f0) v4 2014-06-09 16:22:11.725270 7f869af78700 5 -- op tracker -- , seq: 2769, time: 2014-06-09 16:22:11.582955, event: header_read, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.725276 7f869af78700 5 -- op tracker -- , seq: 2769, time: 2014-06-09 16:22:11.582956, event: throttled, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.725282 7f869af78700 5 -- op tracker -- , seq: 2769, time: 2014-06-09 16:22:11.582967, event: all_read, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.725288 7f869af78700 5 -- op tracker -- , seq: 2769, time: 2014-06-09 16:22:11.725262, event: dispatched, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.725294 7f869af78700 5 -- op tracker -- , seq: 2769, time: 2014-06-09 16:22:11.725292, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.725299 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3f0) v4 from osd.0 2014-06-09 16:22:11.725302 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf0fc0 2014-06-09 16:22:11.725307 7f869af78700 5 -- op tracker -- , seq: 2769, time: 2014-06-09 16:22:11.725305, event: started, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.725323 7f869af78700 5 -- op tracker -- , seq: 2769, time: 2014-06-09 16:22:11.725321, event: done, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.725330 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725332 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725343 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725344 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725345 7f869af78700 20 osd.26 243 _dispatch 0x30daee00 pg_info(1 pgs e243:2.487) v4 2014-06-09 16:22:11.725350 7f869af78700 5 -- op tracker -- , seq: 2770, time: 2014-06-09 16:22:11.568252, event: header_read, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.725359 7f869af78700 5 -- op tracker -- , seq: 2770, time: 2014-06-09 16:22:11.568252, event: throttled, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.725365 7f869af78700 5 -- op tracker -- , seq: 2770, time: 2014-06-09 16:22:11.568266, event: all_read, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.725370 7f869af78700 5 -- op tracker -- , seq: 2770, time: 2014-06-09 16:22:11.725342, event: dispatched, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.725377 7f869af78700 5 -- op tracker -- , seq: 2770, time: 2014-06-09 16:22:11.725375, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.725382 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.487) v4 from osd.5 2014-06-09 16:22:11.725384 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daee00 2014-06-09 16:22:11.725389 7f869af78700 5 -- op tracker -- , seq: 2770, time: 2014-06-09 16:22:11.725387, event: started, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.725402 7f869af78700 5 -- op tracker -- , seq: 2770, time: 2014-06-09 16:22:11.725400, event: done, op: pg_info(1 pgs e243:2.487) 2014-06-09 16:22:11.725410 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725411 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725425 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725428 7f869af78700 20 osd.26 243 _dispatch 0x30951e00 pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.725435 7f869af78700 5 -- op tracker -- , seq: 2771, time: 2014-06-09 16:22:11.313559, event: header_read, op: pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725442 7f869af78700 5 -- op tracker -- , seq: 2771, time: 2014-06-09 16:22:11.313560, event: throttled, op: pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725449 7f869af78700 5 -- op tracker -- , seq: 2771, time: 2014-06-09 16:22:11.314506, event: all_read, op: pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725455 7f869af78700 5 -- op tracker -- , seq: 2771, time: 2014-06-09 16:22:11.725425, event: dispatched, op: pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725463 7f869af78700 5 -- op tracker -- , seq: 2771, time: 2014-06-09 16:22:11.725460, event: waiting_for_osdmap, op: pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725468 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.725472 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30951e00 2014-06-09 16:22:11.725477 7f869af78700 5 -- op tracker -- , seq: 2771, time: 2014-06-09 16:22:11.725474, event: started, op: pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725489 7f869af78700 5 -- op tracker -- , seq: 2771, time: 2014-06-09 16:22:11.725487, event: done, op: pg_log(3.18b epoch 243 log log((0'0,213'575], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.725496 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725498 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725552 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725554 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725556 7f869af78700 20 osd.26 243 _dispatch 0x2eee7c00 pg_info(1 pgs e243:3.6d) v4 2014-06-09 16:22:11.725561 7f869af78700 5 -- op tracker -- , seq: 2772, time: 2014-06-09 16:22:11.430400, event: header_read, op: pg_info(1 pgs e243:3.6d) 2014-06-09 16:22:11.725568 7f869af78700 5 -- op tracker -- , seq: 2772, time: 2014-06-09 16:22:11.430401, event: throttled, op: pg_info(1 pgs e243:3.6d) 2014-06-09 16:22:11.725574 7f869af78700 5 -- op tracker -- , seq: 2772, time: 2014-06-09 16:22:11.430412, event: all_read, op: pg_info(1 pgs e243:3.6d) 2014-06-09 16:22:11.725580 7f869af78700 5 -- op tracker -- , seq: 2772, time: 2014-06-09 16:22:11.725552, event: dispatched, op: pg_info(1 pgs e243:3.6d) 2014-06-09 16:22:11.725586 7f869af78700 5 -- op tracker -- , seq: 2772, time: 2014-06-09 16:22:11.725584, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6d) 2014-06-09 16:22:11.725594 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6d) v4 from osd.4 2014-06-09 16:22:11.725597 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee7c00 2014-06-09 16:22:11.725601 7f869af78700 5 -- op tracker -- , seq: 2772, time: 2014-06-09 16:22:11.725600, event: started, op: pg_info(1 pgs e243:3.6d) 2014-06-09 16:22:11.725619 7f869af78700 5 -- op tracker -- , seq: 2772, time: 2014-06-09 16:22:11.725617, event: done, op: pg_info(1 pgs e243:3.6d) 2014-06-09 16:22:11.725629 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725631 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725645 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725646 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725647 7f869af78700 20 osd.26 243 _dispatch 0x2fef56c0 pg_info(1 pgs e243:1.24c) v4 2014-06-09 16:22:11.725652 7f869af78700 5 -- op tracker -- , seq: 2773, time: 2014-06-09 16:22:11.597282, event: header_read, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.725659 7f869af78700 5 -- op tracker -- , seq: 2773, time: 2014-06-09 16:22:11.597283, event: throttled, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.725665 7f869af78700 5 -- op tracker -- , seq: 2773, time: 2014-06-09 16:22:11.597295, event: all_read, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.725675 7f869af78700 5 -- op tracker -- , seq: 2773, time: 2014-06-09 16:22:11.725644, event: dispatched, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.725682 7f869af78700 5 -- op tracker -- , seq: 2773, time: 2014-06-09 16:22:11.725681, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.725687 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.24c) v4 from osd.7 2014-06-09 16:22:11.725689 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef56c0 2014-06-09 16:22:11.725694 7f869af78700 5 -- op tracker -- , seq: 2773, time: 2014-06-09 16:22:11.725693, event: started, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.725710 7f869af78700 5 -- op tracker -- , seq: 2773, time: 2014-06-09 16:22:11.725708, event: done, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.725744 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725747 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725761 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725763 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725763 7f869af78700 20 osd.26 243 _dispatch 0x30dc3dc0 pg_info(1 pgs e243:1.697) v4 2014-06-09 16:22:11.725768 7f869af78700 5 -- op tracker -- , seq: 2774, time: 2014-06-09 16:22:11.571442, event: header_read, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.725777 7f869af78700 5 -- op tracker -- , seq: 2774, time: 2014-06-09 16:22:11.571443, event: throttled, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.725810 7f869af78700 5 -- op tracker -- , seq: 2774, time: 2014-06-09 16:22:11.571460, event: all_read, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.725818 7f869af78700 5 -- op tracker -- , seq: 2774, time: 2014-06-09 16:22:11.725761, event: dispatched, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.725840 7f869af78700 5 -- op tracker -- , seq: 2774, time: 2014-06-09 16:22:11.725837, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.725844 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.697) v4 from osd.3 2014-06-09 16:22:11.725847 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc3dc0 2014-06-09 16:22:11.725851 7f869af78700 5 -- op tracker -- , seq: 2774, time: 2014-06-09 16:22:11.725850, event: started, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.725867 7f869af78700 5 -- op tracker -- , seq: 2774, time: 2014-06-09 16:22:11.725865, event: done, op: pg_info(1 pgs e243:1.697) 2014-06-09 16:22:11.725874 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725876 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725887 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.725888 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.725890 7f869af78700 20 osd.26 243 _dispatch 0x2176d6c0 pg_info(1 pgs e243:0.3b7) v4 2014-06-09 16:22:11.725894 7f869af78700 5 -- op tracker -- , seq: 2775, time: 2014-06-09 16:22:11.574146, event: header_read, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.725902 7f869af78700 5 -- op tracker -- , seq: 2775, time: 2014-06-09 16:22:11.574146, event: throttled, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.725908 7f869af78700 5 -- op tracker -- , seq: 2775, time: 2014-06-09 16:22:11.574158, event: all_read, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.725913 7f869af78700 5 -- op tracker -- , seq: 2775, time: 2014-06-09 16:22:11.725886, event: dispatched, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.725947 7f869af78700 5 -- op tracker -- , seq: 2775, time: 2014-06-09 16:22:11.725944, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.725952 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3b7) v4 from osd.1 2014-06-09 16:22:11.725955 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176d6c0 2014-06-09 16:22:11.725960 7f869af78700 5 -- op tracker -- , seq: 2775, time: 2014-06-09 16:22:11.725958, event: started, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.725999 7f869af78700 5 -- op tracker -- , seq: 2775, time: 2014-06-09 16:22:11.725997, event: done, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.726007 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.726009 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.726019 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.726021 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.726022 7f869af78700 20 osd.26 243 _dispatch 0x2176c000 pg_info(1 pgs e243:1.289) v4 2014-06-09 16:22:11.726026 7f869af78700 5 -- op tracker -- , seq: 2776, time: 2014-06-09 16:22:11.578063, event: header_read, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.726032 7f869af78700 5 -- op tracker -- , seq: 2776, time: 2014-06-09 16:22:11.578063, event: throttled, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.726043 7f869af78700 5 -- op tracker -- , seq: 2776, time: 2014-06-09 16:22:11.578076, event: all_read, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.726049 7f869af78700 5 -- op tracker -- , seq: 2776, time: 2014-06-09 16:22:11.726019, event: dispatched, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.726086 7f869af78700 5 -- op tracker -- , seq: 2776, time: 2014-06-09 16:22:11.726084, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.726122 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.289) v4 from osd.9 2014-06-09 16:22:11.726125 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176c000 2014-06-09 16:22:11.726143 7f869af78700 5 -- op tracker -- , seq: 2776, time: 2014-06-09 16:22:11.726141, event: started, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.726183 7f86a5610700 10 _do_op 0x308ca9c0 seq 3427 r = 0, finisher 0x30c54e80 0x2eefaa40 2014-06-09 16:22:11.726189 7f86a5610700 10 _finish_op 0x308ca9c0 seq 3427 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.726184 7f869af78700 5 -- op tracker -- , seq: 2776, time: 2014-06-09 16:22:11.726181, event: done, op: pg_info(1 pgs e243:1.289) 2014-06-09 16:22:11.726224 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.726240 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.726266 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.726294 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3428.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.726298 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.726311 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.726314 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.726326 7f86a5e11700 10 _do_op 0xc7045a0 seq 3428 r = 0, finisher 0xbccc060 0 2014-06-09 16:22:11.726338 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.726343 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcada7e0 (2 active) 2014-06-09 16:22:11.726343 7f868bf5a700 10 op_submit_start 3477 2014-06-09 16:22:11.726346 7f868bf5a700 5 queue_transactions (trailing journal) 3477 2014-06-09 16:22:11.726329 7f869af78700 20 osd.26 243 _dispatch 0x2eef1c00 pg_notify(2.1fc(20),3.1ff(10) epoch 243) v5 2014-06-09 16:22:11.726347 7f86a5610700 5 _do_op 0x30d97500 seq 3429 osr(3.709 0x3ed1bc0)/0x3ed1bc0 start 2014-06-09 16:22:11.726348 7f868bf5a700 5 queue_op 0xc34a180 seq 3477 osr(3.465 0x3ed1580) 1335 bytes (queue has 50 ops and 3016503 bytes) 2014-06-09 16:22:11.726350 7f86a5610700 10 _do_transaction on 0xc2c4900 2014-06-09 16:22:11.726352 7f868bf5a700 10 op_submit_finish 3477 2014-06-09 16:22:11.726358 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.726357 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.726374 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.726382 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.726390 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.726395 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.728956 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.728970 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.728984 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3429.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.728990 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.728988 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:190 2014-06-09 16:22:11.728999 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.729007 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.729018 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729018 7f86a5610700 10 _do_op 0x30d97500 seq 3429 r = 0, finisher 0x189aa860 0 2014-06-09 16:22:11.729027 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.729041 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.729050 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.729060 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.729066 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.729074 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.729087 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1075 2014-06-09 16:22:11.729097 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.729105 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.729116 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729120 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.729133 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.729142 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.729150 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.729166 7f868bf5a700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.729183 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:146 2014-06-09 16:22:11.729197 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729202 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.729215 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.729224 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.729233 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.729246 7f868bf5a700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.729261 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1042 2014-06-09 16:22:11.729276 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.726401 7f86a5e11700 10 _finish_op 0xc7045a0 seq 3428 osr(2.4b9 0x2ed1730)/0x2ed1730 2014-06-09 16:22:11.726400 7f8689f56700 2 waiting 51 > 50 ops || 4567075 > 104857600 2014-06-09 16:22:11.729297 7f868cf5c700 2 waiting 51 > 50 ops || 3017556 > 104857600 2014-06-09 16:22:11.726383 7f869af78700 5 -- op tracker -- , seq: 2777, time: 2014-06-09 16:22:11.434968, event: header_read, op: pg_notify(2.1fc(20),3.1ff(10) epoch 243) 2014-06-09 16:22:11.729298 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:3.4c7( v 213'593 (0'0,213'593] local-les=213 n=118 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.729312 7f869af78700 5 -- op tracker -- , seq: 2777, time: 2014-06-09 16:22:11.434969, event: throttled, op: pg_notify(2.1fc(20),3.1ff(10) epoch 243) 2014-06-09 16:22:11.729313 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.729324 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729323 7f869af78700 5 -- op tracker -- , seq: 2777, time: 2014-06-09 16:22:11.435012, event: all_read, op: pg_notify(2.1fc(20),3.1ff(10) epoch 243) 2014-06-09 16:22:11.729330 7f869af78700 5 -- op tracker -- , seq: 2777, time: 2014-06-09 16:22:11.726297, event: dispatched, op: pg_notify(2.1fc(20),3.1ff(10) epoch 243) 2014-06-09 16:22:11.729328 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.1dc( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.729337 7f869af78700 5 -- op tracker -- , seq: 2777, time: 2014-06-09 16:22:11.729335, event: waiting_for_osdmap, op: pg_notify(2.1fc(20),3.1ff(10) epoch 243) 2014-06-09 16:22:11.729342 7f869af78700 7 osd.26 243 handle_pg_notify from osd.8 2014-06-09 16:22:11.729344 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef1c00 2014-06-09 16:22:11.729339 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.729348 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729350 7f869af78700 5 -- op tracker -- , seq: 2777, time: 2014-06-09 16:22:11.729347, event: started, op: pg_notify(2.1fc(20),3.1ff(10) epoch 243) 2014-06-09 16:22:11.729352 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.77a( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.729363 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.729371 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729377 7f869af78700 5 -- op tracker -- , seq: 2777, time: 2014-06-09 16:22:11.729375, event: done, op: pg_notify(2.1fc(20),3.1ff(10) epoch 243) 2014-06-09 16:22:11.729375 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.729387 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.729389 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.729387 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.729395 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x10c55d00,0x10c55c00,0x10c55b00,0x10c55a00,0x14053900 2014-06-09 16:22:11.729410 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.729411 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.729413 7f869af78700 20 osd.26 243 _dispatch 0x31162a80 pg_info(1 pgs e243:2.34d) v4 2014-06-09 16:22:11.729417 7f869af78700 5 -- op tracker -- , seq: 2778, time: 2014-06-09 16:22:11.566267, event: header_read, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.729418 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.729424 7f869af78700 5 -- op tracker -- , seq: 2778, time: 2014-06-09 16:22:11.566267, event: throttled, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.729429 7f869af78700 5 -- op tracker -- , seq: 2778, time: 2014-06-09 16:22:11.566278, event: all_read, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.729427 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.729435 7f869af78700 5 -- op tracker -- , seq: 2778, time: 2014-06-09 16:22:11.729409, event: dispatched, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.729436 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.729441 7f869af78700 5 -- op tracker -- , seq: 2778, time: 2014-06-09 16:22:11.729439, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.729449 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:974 2014-06-09 16:22:11.729456 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.34d) v4 from osd.6 2014-06-09 16:22:11.729459 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31162a80 2014-06-09 16:22:11.729459 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.729464 7f869af78700 5 -- op tracker -- , seq: 2778, time: 2014-06-09 16:22:11.729462, event: started, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.729467 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.729476 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729479 7f8682ae5700 5 -- op tracker -- , seq: 2779, time: 2014-06-09 16:22:11.727883, event: header_read, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.729481 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729492 7f8682ae5700 5 -- op tracker -- , seq: 2779, time: 2014-06-09 16:22:11.727884, event: throttled, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.729493 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.555( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729501 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729502 7f8682ae5700 5 -- op tracker -- , seq: 2779, time: 2014-06-09 16:22:11.727893, event: all_read, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.729505 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729516 7f8682ae5700 5 -- op tracker -- , seq: 2779, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.729512 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729520 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729522 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) v2 epoch 243 2014-06-09 16:22:11.729528 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.729524 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729531 7f8670072700 5 -- op tracker -- , seq: 2780, time: 2014-06-09 16:22:11.724290, event: header_read, op: osd_op(client.4538.0:36646 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 1019904~1548288] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.729535 7f8682ae5700 15 osd.26 243 enqueue_op 0xdfa6400 prio 196 cost 0 latency 0.001651 osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) v2 2014-06-09 16:22:11.729535 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729544 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729544 7f8670072700 5 -- op tracker -- , seq: 2780, time: 2014-06-09 16:22:11.724292, event: throttled, op: osd_op(client.4538.0:36646 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 1019904~1548288] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.729547 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729555 7f8670072700 5 -- op tracker -- , seq: 2780, time: 2014-06-09 16:22:11.726932, event: all_read, op: osd_op(client.4538.0:36646 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 1019904~1548288] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.729558 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729567 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729564 7f8670072700 5 -- op tracker -- , seq: 2780, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36646 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 1019904~1548288] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.729572 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.729570 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729583 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.729592 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729588 7f8682ae5700 5 -- op tracker -- , seq: 2781, time: 2014-06-09 16:22:11.729554, event: header_read, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.729595 7f8670072700 15 osd.26 243 enqueue_op 0x1b0b6800 prio 63 cost 1548288 latency 0.005305 osd_op(client.4538.0:36646 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 1019904~1548288] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.729596 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729599 7f8682ae5700 5 -- op tracker -- , seq: 2781, time: 2014-06-09 16:22:11.729555, event: throttled, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.729606 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729610 7f8682ae5700 5 -- op tracker -- , seq: 2781, time: 2014-06-09 16:22:11.729566, event: all_read, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.729615 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729619 7f8682ae5700 5 -- op tracker -- , seq: 2781, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.729623 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.729619 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729627 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.729632 7f8682ae5700 15 osd.26 243 enqueue_op 0xdfa6300 prio 196 cost 0 latency 0.000078 osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.729632 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.729642 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729645 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729656 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729664 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729669 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729679 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729688 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729711 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41980 (19 active) 2014-06-09 16:22:11.729729 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) 2014-06-09 16:22:11.729732 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 6 notify: (query_epoch:242, epoch_sent:243, info:0.1dc( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.729742 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.729751 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729754 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 9 notify: (query_epoch:242, epoch_sent:243, info:3.4c7( v 213'593 (0'0,213'593] local-les=213 n=118 ec=118 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.729765 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.729775 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729778 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:0.1ca( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.729790 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.729799 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729803 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729813 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729822 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729826 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729837 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.729847 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729850 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.601( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729860 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.601( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729868 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729871 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:0.1f1( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.729881 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.729890 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729893 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:1.1f0( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.729904 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.729913 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729917 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729929 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.729939 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729942 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.729952 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.729959 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.729963 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=203 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.729975 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=203 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.729987 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=203 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.024776 7 0.004485 2014-06-09 16:22:11.729999 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=203 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.730010 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=203 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.730019 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=203 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.730029 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'644 -> 213'644 2014-06-09 16:22:11.730067 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.730107 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.730112 7f868bf5a700 5 queue_transactions existing osr(3.11f 0x3ed07e0)/0x3ed07e0 2014-06-09 16:22:11.730114 7f868bf5a700 2 waiting 51 > 50 ops || 3017838 > 104857600 2014-06-09 16:22:11.730120 7f868ff62700 2 waiting 51 > 50 ops || 3017361 > 104857600 2014-06-09 16:22:11.730128 7f8691765700 2 waiting 51 > 50 ops || 3017838 > 104857600 2014-06-09 16:22:11.730143 7f869af78700 5 -- op tracker -- , seq: 2778, time: 2014-06-09 16:22:11.729476, event: done, op: pg_info(1 pgs e243:2.34d) 2014-06-09 16:22:11.730148 7f868f761700 2 waiting 51 > 50 ops || 3017331 > 104857600 2014-06-09 16:22:11.730155 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.730155 7f8692767700 2 waiting 51 > 50 ops || 3018500 > 104857600 2014-06-09 16:22:11.730157 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.730162 7f8692f68700 2 waiting 51 > 50 ops || 3017838 > 104857600 2014-06-09 16:22:11.730170 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.730172 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.730169 7f8689755700 2 waiting 51 > 50 ops || 3018379 > 104857600 2014-06-09 16:22:11.730172 7f869af78700 20 osd.26 243 _dispatch 0x30cafdc0 pg_info(1 pgs e243:1.3f0) v4 2014-06-09 16:22:11.730177 7f869af78700 5 -- op tracker -- , seq: 2782, time: 2014-06-09 16:22:11.582972, event: header_read, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.730184 7f869af78700 5 -- op tracker -- , seq: 2782, time: 2014-06-09 16:22:11.582973, event: throttled, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.730177 7f868c75b700 2 waiting 51 > 50 ops || 3534883 > 104857600 2014-06-09 16:22:11.730194 7f868ef60700 2 waiting 51 > 50 ops || 3018386 > 104857600 2014-06-09 16:22:11.730204 7f868cf5c700 2 waiting 51 > 50 ops || 4566245 > 104857600 2014-06-09 16:22:11.730205 7f8689f56700 10 op_submit_start 3478 2014-06-09 16:22:11.730207 7f8689f56700 5 queue_transactions (trailing journal) 3478 2014-06-09 16:22:11.730208 7f868ff62700 2 waiting 51 > 50 ops || 4566050 > 104857600 2014-06-09 16:22:11.730209 7f8689f56700 5 queue_op 0x30d97260 seq 3478 osr(3.60 0x3ed02e0) 1550572 bytes (queue has 50 ops and 4565192 bytes) 2014-06-09 16:22:11.730207 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77be60 (1 active) 2014-06-09 16:22:11.730213 7f8691765700 2 waiting 51 > 50 ops || 4566527 > 104857600 2014-06-09 16:22:11.730215 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496000 (2 active) 2014-06-09 16:22:11.730219 7f86a5e11700 5 _do_op 0x308ca780 seq 3430 osr(1.384 0x384d5c0)/0x384d5c0 start 2014-06-09 16:22:11.730220 7f868f761700 2 waiting 51 > 50 ops || 4566020 > 104857600 2014-06-09 16:22:11.730222 7f86a5e11700 10 _do_transaction on 0xc1fd740 2014-06-09 16:22:11.730225 7f8692767700 2 waiting 51 > 50 ops || 4567189 > 104857600 2014-06-09 16:22:11.730227 7f86a5610700 10 _finish_op 0x30d97500 seq 3429 osr(3.709 0x3ed1bc0)/0x3ed1bc0 2014-06-09 16:22:11.730229 7f8692f68700 2 waiting 51 > 50 ops || 4566527 > 104857600 2014-06-09 16:22:11.730230 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.730246 7f8689755700 2 waiting 51 > 50 ops || 4567068 > 104857600 2014-06-09 16:22:11.730251 7f868c75b700 2 waiting 51 > 50 ops || 5083572 > 104857600 2014-06-09 16:22:11.730257 7f868ef60700 2 waiting 51 > 50 ops || 4567075 > 104857600 2014-06-09 16:22:11.730262 7f868bf5a700 2 waiting 51 > 50 ops || 4566527 > 104857600 2014-06-09 16:22:11.730271 7f868ff62700 2 waiting 51 > 50 ops || 4566050 > 104857600 2014-06-09 16:22:11.730276 7f868f761700 2 waiting 51 > 50 ops || 4566020 > 104857600 2014-06-09 16:22:11.730281 7f8692767700 2 waiting 51 > 50 ops || 4567189 > 104857600 2014-06-09 16:22:11.730283 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcada7e0 (1 active) 2014-06-09 16:22:11.730287 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89ea20 (2 active) 2014-06-09 16:22:11.730291 7f86a5610700 5 _do_op 0x309a2240 seq 3431 osr(2.1c6 0x2ed0870)/0x2ed0870 start 2014-06-09 16:22:11.730291 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.730293 7f8689f56700 10 op_submit_finish 3478 2014-06-09 16:22:11.730294 7f86a5610700 10 _do_transaction on 0x3859a40 2014-06-09 16:22:11.730286 7f8692f68700 2 waiting 51 > 50 ops || 4566527 > 104857600 2014-06-09 16:22:11.730301 7f868cf5c700 10 op_submit_start 3479 2014-06-09 16:22:11.730303 7f868cf5c700 5 queue_transactions (trailing journal) 3479 2014-06-09 16:22:11.730302 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.730304 7f868cf5c700 5 queue_op 0x30d96060 seq 3479 osr(3.74f 0x3ed1ea0) 1053 bytes (queue has 50 ops and 4565192 bytes) 2014-06-09 16:22:11.730308 7f868cf5c700 10 op_submit_finish 3479 2014-06-09 16:22:11.730311 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3430.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.730315 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.730301 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc1f0540 243'641 rep_tid=6 committed?=0 applied?=0 lock=0 op=osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.730312 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.730328 7f8689f56700 10 osd.26 243 dequeue_op 0x230da000 finish 2014-06-09 16:22:11.730328 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.730335 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.730339 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730342 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.730347 7f8689755700 2 waiting 51 > 50 ops || 4567068 > 104857600 2014-06-09 16:22:11.730339 7f868d75d700 10 osd.26 243 dequeue_op 0x257c5600 prio 63 cost 516096 latency 0.979778 osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.730344 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.730363 7f868c75b700 2 waiting 51 > 50 ops || 5083572 > 104857600 2014-06-09 16:22:11.730369 7f868ef60700 2 waiting 51 > 50 ops || 4567075 > 104857600 2014-06-09 16:22:11.730365 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.730191 7f869af78700 5 -- op tracker -- , seq: 2782, time: 2014-06-09 16:22:11.582980, event: all_read, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.730374 7f868bf5a700 2 waiting 51 > 50 ops || 4566527 > 104857600 2014-06-09 16:22:11.730373 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.730380 7f8691765700 2 waiting 51 > 50 ops || 4566527 > 104857600 2014-06-09 16:22:11.730378 7f869af78700 5 -- op tracker -- , seq: 2782, time: 2014-06-09 16:22:11.730170, event: dispatched, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.730381 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.730387 7f869af78700 5 -- op tracker -- , seq: 2782, time: 2014-06-09 16:22:11.730385, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.730389 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.730394 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3f0) v4 from osd.0 2014-06-09 16:22:11.730397 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cafdc0 2014-06-09 16:22:11.730396 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.730402 7f868d75d700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.730361, event: reached_pg, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.730409 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.730418 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.730415 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.730425 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.730435 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730431 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x257c5600 2014-06-09 16:22:11.730439 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.730451 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.730442 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.730458 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.730462 7f8682de8700 5 -- op tracker -- , seq: 2783, time: 2014-06-09 16:22:11.730427, event: header_read, op: osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.730466 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.730473 7f8682de8700 5 -- op tracker -- , seq: 2783, time: 2014-06-09 16:22:11.730428, event: throttled, op: osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.730467 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x305c6b00 2014-06-09 16:22:11.730473 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.730480 7f8682de8700 5 -- op tracker -- , seq: 2783, time: 2014-06-09 16:22:11.730439, event: all_read, op: osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.730481 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.730489 7f8682de8700 5 -- op tracker -- , seq: 2783, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.730479 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x305c6b00 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 rwstate(write n=2 w=0) oi: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'640 client.4572.0:30633 wrlock_by=unknown.0.0:0 dirty s 2281472 uv640) ssc: 0x2e8775e0 snapset: 0=[]:[]+head 2014-06-09 16:22:11.730493 7f8682de8700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.730497 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:11.730493 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:182 2014-06-09 16:22:11.730503 7f8682de8700 15 osd.26 243 enqueue_op 0xdfa6000 prio 196 cost 0 latency 0.000075 osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.730497 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 @head oi=5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'640 client.4572.0:30633 wrlock_by=unknown.0.0:0 dirty s 2281472 uv640) 2014-06-09 16:22:11.730507 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.730515 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.730524 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730520 7f869af78700 5 -- op tracker -- , seq: 2782, time: 2014-06-09 16:22:11.730400, event: started, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.730528 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.730536 7f868d75d700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.730518, event: started, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.730539 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.730549 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730551 7f869af78700 5 -- op tracker -- , seq: 2782, time: 2014-06-09 16:22:11.730541, event: done, op: pg_info(1 pgs e243:1.3f0) 2014-06-09 16:22:11.730548 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30a83000 2014-06-09 16:22:11.730560 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.730553 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.730562 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.730568 7f868d75d700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.730562, event: started, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.730575 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.730576 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.730577 7f869af78700 20 osd.26 243 _dispatch 0x30daec40 pg_info(1 pgs e243:2.fd) v4 2014-06-09 16:22:11.730565 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.730584 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730582 7f869af78700 5 -- op tracker -- , seq: 2784, time: 2014-06-09 16:22:11.568700, event: header_read, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.730574 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] ov 243'640 av 243'642 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.730589 7f869af78700 5 -- op tracker -- , seq: 2784, time: 2014-06-09 16:22:11.568701, event: throttled, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.730590 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 2014-06-09 16:22:11.730588 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.21b( v 214'612 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.730602 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.730604 7f869af78700 5 -- op tracker -- , seq: 2784, time: 2014-06-09 16:22:11.568713, event: all_read, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.730606 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.7 activated and committed 2014-06-09 16:22:11.730615 7f869af78700 5 -- op tracker -- , seq: 2784, time: 2014-06-09 16:22:11.730574, event: dispatched, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.730616 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 2281472~516096 2014-06-09 16:22:11.730617 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:11.730629 7f869af78700 5 -- op tracker -- , seq: 2784, time: 2014-06-09 16:22:11.730627, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.730633 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.fd) v4 from osd.5 2014-06-09 16:22:11.730627 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:11.730636 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daec40 2014-06-09 16:22:11.730641 7f869af78700 5 -- op tracker -- , seq: 2784, time: 2014-06-09 16:22:11.730639, event: started, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.730635 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229400 2014-06-09 16:22:11.730645 7f868cf5c700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.730648 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 snapset=0x2e877618 snapc=0=[] 2014-06-09 16:22:11.730661 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.730660 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1049 2014-06-09 16:22:11.730678 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730672 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 0x30a83000 op modify 2014-06-09 16:22:11.730655 7f869af78700 5 -- op tracker -- , seq: 2784, time: 2014-06-09 16:22:11.730653, event: done, op: pg_info(1 pgs e243:2.fd) 2014-06-09 16:22:11.730691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.730683 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.730692 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.730685 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:23.267173 2014-06-09 16:22:11.730695 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.730706 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.730707 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.730703 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.730708 7f869af78700 20 osd.26 243 _dispatch 0x308d1200 pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.730710 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.730707 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.730715 7f869af78700 5 -- op tracker -- , seq: 2785, time: 2014-06-09 16:22:11.321619, event: header_read, op: pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.730721 7f868cf5c700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.730723 7f869af78700 5 -- op tracker -- , seq: 2785, time: 2014-06-09 16:22:11.321620, event: throttled, op: pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.730730 7f869af78700 5 -- op tracker -- , seq: 2785, time: 2014-06-09 16:22:11.322886, event: all_read, op: pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.730728 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.730737 7f869af78700 5 -- op tracker -- , seq: 2785, time: 2014-06-09 16:22:11.730705, event: dispatched, op: pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.730735 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:144 2014-06-09 16:22:11.730744 7f869af78700 5 -- op tracker -- , seq: 2785, time: 2014-06-09 16:22:11.730741, event: waiting_for_osdmap, op: pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.730748 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.730751 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730752 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d1200 2014-06-09 16:22:11.730740 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 9 on osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.730757 7f869af78700 5 -- op tracker -- , seq: 2785, time: 2014-06-09 16:22:11.730755, event: started, op: pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.730755 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.730758 7f868d75d700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 9 o 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.730770 7f869af78700 5 -- op tracker -- , seq: 2785, time: 2014-06-09 16:22:11.730767, event: done, op: pg_log(3.458 epoch 243 log log((0'0,213'610], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.730766 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.730774 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.730783 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.730785 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.730782 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.730792 7f868cf5c700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.730799 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.730800 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.730802 7f869af78700 20 osd.26 243 _dispatch 0x2eee7a40 pg_info(1 pgs e243:1.623) v4 2014-06-09 16:22:11.730805 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:173 2014-06-09 16:22:11.730818 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730815 7f868d75d700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.730779, event: waiting for subops from 7, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.730828 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.730838 7f868d75d700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.730839 7f869af78700 5 -- op tracker -- , seq: 2786, time: 2014-06-09 16:22:11.430418, event: header_read, op: pg_info(1 pgs e243:1.623) 2014-06-09 16:22:11.730840 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.730847 7f869af78700 5 -- op tracker -- , seq: 2786, time: 2014-06-09 16:22:11.430418, event: throttled, op: pg_info(1 pgs e243:1.623) 2014-06-09 16:22:11.730848 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.730856 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.730863 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.730859 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'641 (0'0,243'641] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'641], crt=0'0) [243'642 (243'640) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30634 2014-06-09 10:20:23.267173] 2014-06-09 16:22:11.730870 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.730883 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.730881 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'642 (243'640) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30634 2014-06-09 10:20:23.267173 2014-06-09 16:22:11.730891 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.730897 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.730905 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.730902 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.730933 7f868d75d700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'642, trimmed: 2014-06-09 16:22:11.730954 7f868d75d700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.730957 7f868d75d700 2 waiting 51 > 50 ops || 5083572 > 104857600 2014-06-09 16:22:11.730975 7f86a5e11700 10 _do_op 0x308ca780 seq 3430 r = 0, finisher 0xcd33460 0 2014-06-09 16:22:11.730979 7f86a5e11700 10 _finish_op 0x308ca780 seq 3430 osr(1.384 0x384d5c0)/0x384d5c0 2014-06-09 16:22:11.730990 7f868f761700 10 op_submit_start 3480 2014-06-09 16:22:11.730991 7f868f761700 5 queue_transactions (trailing journal) 3480 2014-06-09 16:22:11.730991 7f8692767700 2 waiting 51 > 50 ops || 4566016 > 104857600 2014-06-09 16:22:11.730993 7f868f761700 5 queue_op 0xbe7d860 seq 3480 osr(3.7ce 0xb7f01a0) 828 bytes (queue has 50 ops and 4564019 bytes) 2014-06-09 16:22:11.730996 7f8692f68700 2 waiting 51 > 50 ops || 4565354 > 104857600 2014-06-09 16:22:11.730999 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496000 (1 active) 2014-06-09 16:22:11.731001 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407200 (2 active) 2014-06-09 16:22:11.731003 7f8689755700 2 waiting 51 > 50 ops || 4565895 > 104857600 2014-06-09 16:22:11.731006 7f868f761700 10 op_submit_finish 3480 2014-06-09 16:22:11.731011 7f868c75b700 2 waiting 51 > 50 ops || 5082399 > 104857600 2014-06-09 16:22:11.731015 7f868ef60700 2 waiting 51 > 50 ops || 4565902 > 104857600 2014-06-09 16:22:11.731010 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.731020 7f868bf5a700 2 waiting 51 > 50 ops || 4565354 > 104857600 2014-06-09 16:22:11.731025 7f8691765700 2 waiting 51 > 50 ops || 4565354 > 104857600 2014-06-09 16:22:11.731030 7f868d75d700 2 waiting 51 > 50 ops || 5082399 > 104857600 2014-06-09 16:22:11.731023 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.5 2.4a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.731035 7f868ff62700 2 waiting 51 > 50 ops || 4564877 > 104857600 2014-06-09 16:22:11.731037 7f868f761700 10 merge_log log((0'0,0'0], crt=0'0) from osd.5 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.731041 7f868f761700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.731045 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.945014 7 0.000186 2014-06-09 16:22:11.731056 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.731067 7f868f761700 5 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.731077 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.731087 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.731096 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.731131 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 9 2014-06-09 16:22:11.731148 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.731146 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd9 2014-06-09 16:22:11.731153 7f868f761700 5 queue_transactions existing osr(2.4a7 0x2ed17d0)/0x2ed17d0 2014-06-09 16:22:11.731156 7f868f761700 2 waiting 51 > 50 ops || 4566020 > 104857600 2014-06-09 16:22:11.731161 7f86a5e11700 5 _do_op 0x2fcaf7a0 seq 3432 osr(0.648 0x2bd7bd0)/0x2bd7bd0 start 2014-06-09 16:22:11.731157 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.731165 7f86a5e11700 10 _do_transaction on 0xc10bb00 2014-06-09 16:22:11.731171 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.731166 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.731175 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'632 (0'0,214'632] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.9: log((214'632,214'635], crt=0'0) missing(0) 2014-06-09 16:22:11.731184 7f868cf5c700 10 merge_log log((214'632,214'635], crt=0'0) from osd.9 into log((0'0,214'632], crt=0'0) 2014-06-09 16:22:11.731188 7f868cf5c700 10 merge_log extending head to 214'635 2014-06-09 16:22:11.731190 7f868cf5c700 20 ? 214'635 (214'634) modify f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3 by client.4574.0:31806 2014-06-09 10:18:54.867368 2014-06-09 16:22:11.731204 7f868cf5c700 20 ? 214'634 (214'633) modify f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3 by client.4574.0:31805 2014-06-09 10:18:54.850752 2014-06-09 16:22:11.731209 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.731210 7f868cf5c700 20 ? 214'633 (214'632) modify f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3 by client.4574.0:31804 2014-06-09 10:18:54.850128 2014-06-09 16:22:11.731216 7f868cf5c700 20 merge_log 214'633 (214'632) modify f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3 by client.4574.0:31804 2014-06-09 10:18:54.850128 2014-06-09 16:22:11.731228 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3432.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.731228 7f868cf5c700 20 merge_log 214'634 (214'633) modify f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3 by client.4574.0:31805 2014-06-09 10:18:54.850752 2014-06-09 16:22:11.731233 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.731237 7f868cf5c700 20 merge_log 214'635 (214'634) modify f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3 by client.4574.0:31806 2014-06-09 10:18:54.867368 2014-06-09 16:22:11.731268 7f86a5e11700 10 _do_op 0x2fcaf7a0 seq 3432 r = 0, finisher 0x314ac580 0 2014-06-09 16:22:11.731272 7f86a5e11700 10 _finish_op 0x2fcaf7a0 seq 3432 osr(0.648 0x2bd7bd0)/0x2bd7bd0 2014-06-09 16:22:11.731279 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407200 (1 active) 2014-06-09 16:22:11.731281 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1680 (2 active) 2014-06-09 16:22:11.731284 7f86a5e11700 5 _do_op 0x30d96f60 seq 3433 osr(0.70e 0x384c0f0)/0x384c0f0 start 2014-06-09 16:22:11.731286 7f86a5e11700 10 _do_transaction on 0xc28f8c0 2014-06-09 16:22:11.730855 7f869af78700 5 -- op tracker -- , seq: 2786, time: 2014-06-09 16:22:11.430427, event: all_read, op: pg_info(1 pgs e243:1.623) 2014-06-09 16:22:11.731293 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.731295 7f869af78700 5 -- op tracker -- , seq: 2786, time: 2014-06-09 16:22:11.730798, event: dispatched, op: pg_info(1 pgs e243:1.623) 2014-06-09 16:22:11.731301 7f869af78700 5 -- op tracker -- , seq: 2786, time: 2014-06-09 16:22:11.731300, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.623) 2014-06-09 16:22:11.731305 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.623) v4 from osd.4 2014-06-09 16:22:11.731308 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee7a40 2014-06-09 16:22:11.731312 7f869af78700 5 -- op tracker -- , seq: 2786, time: 2014-06-09 16:22:11.731310, event: started, op: pg_info(1 pgs e243:1.623) 2014-06-09 16:22:11.731327 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.731330 7f869af78700 5 -- op tracker -- , seq: 2786, time: 2014-06-09 16:22:11.731328, event: done, op: pg_info(1 pgs e243:1.623) 2014-06-09 16:22:11.731339 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731341 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731345 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3433.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.731350 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.731355 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731356 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731357 7f869af78700 20 osd.26 243 _dispatch 0x308d9dc0 pg_info(1 pgs e243:1.24c) v4 2014-06-09 16:22:11.731362 7f869af78700 5 -- op tracker -- , seq: 2787, time: 2014-06-09 16:22:11.597783, event: header_read, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.731368 7f869af78700 5 -- op tracker -- , seq: 2787, time: 2014-06-09 16:22:11.597783, event: throttled, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.731373 7f86a5e11700 10 _do_op 0x30d96f60 seq 3433 r = 0, finisher 0xc6e97e0 0 2014-06-09 16:22:11.731374 7f869af78700 5 -- op tracker -- , seq: 2787, time: 2014-06-09 16:22:11.597791, event: all_read, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.731377 7f86a5e11700 10 _finish_op 0x30d96f60 seq 3433 osr(0.70e 0x384c0f0)/0x384c0f0 2014-06-09 16:22:11.731379 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1680 (1 active) 2014-06-09 16:22:11.731382 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d10e0 (2 active) 2014-06-09 16:22:11.731380 7f869af78700 5 -- op tracker -- , seq: 2787, time: 2014-06-09 16:22:11.731354, event: dispatched, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.731384 7f86a5e11700 5 _do_op 0x30d965a0 seq 3434 osr(0.448 0x2bd7190)/0x2bd7190 start 2014-06-09 16:22:11.731386 7f86a5e11700 10 _do_transaction on 0xc29b380 2014-06-09 16:22:11.731386 7f869af78700 5 -- op tracker -- , seq: 2787, time: 2014-06-09 16:22:11.731384, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.731390 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.24c) v4 from osd.7 2014-06-09 16:22:11.731393 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d9dc0 2014-06-09 16:22:11.731393 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.731397 7f869af78700 5 -- op tracker -- , seq: 2787, time: 2014-06-09 16:22:11.731396, event: started, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.731414 7f869af78700 5 -- op tracker -- , seq: 2787, time: 2014-06-09 16:22:11.731412, event: done, op: pg_info(1 pgs e243:1.24c) 2014-06-09 16:22:11.731421 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731423 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731425 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.731436 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731438 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731439 7f869af78700 20 osd.26 243 _dispatch 0x30dc3c00 pg_info(1 pgs e243:2.5f8) v4 2014-06-09 16:22:11.731444 7f869af78700 5 -- op tracker -- , seq: 2788, time: 2014-06-09 16:22:11.571630, event: header_read, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.731450 7f869af78700 5 -- op tracker -- , seq: 2788, time: 2014-06-09 16:22:11.571631, event: throttled, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.731457 7f869af78700 5 -- op tracker -- , seq: 2788, time: 2014-06-09 16:22:11.571645, event: all_read, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.731463 7f869af78700 5 -- op tracker -- , seq: 2788, time: 2014-06-09 16:22:11.731435, event: dispatched, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.731468 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3434.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.731471 7f869af78700 5 -- op tracker -- , seq: 2788, time: 2014-06-09 16:22:11.731469, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.731474 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.731475 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5f8) v4 from osd.3 2014-06-09 16:22:11.731478 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc3c00 2014-06-09 16:22:11.731482 7f869af78700 5 -- op tracker -- , seq: 2788, time: 2014-06-09 16:22:11.731480, event: started, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.731499 7f86a5e11700 10 _do_op 0x30d965a0 seq 3434 r = 0, finisher 0x314ad360 0 2014-06-09 16:22:11.731502 7f86a5e11700 10 _finish_op 0x30d965a0 seq 3434 osr(0.448 0x2bd7190)/0x2bd7190 2014-06-09 16:22:11.731505 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d10e0 (1 active) 2014-06-09 16:22:11.731507 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc923c20 (2 active) 2014-06-09 16:22:11.731510 7f86a5e11700 5 _do_op 0x2ea53920 seq 3435 osr(3.730 0x3ed1ef0)/0x3ed1ef0 start 2014-06-09 16:22:11.731512 7f86a5e11700 10 _do_transaction on 0xc149980 2014-06-09 16:22:11.731496 7f869af78700 5 -- op tracker -- , seq: 2788, time: 2014-06-09 16:22:11.731494, event: done, op: pg_info(1 pgs e243:2.5f8) 2014-06-09 16:22:11.731517 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731519 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731518 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.731530 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731531 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731532 7f869af78700 20 osd.26 243 _dispatch 0x30dc2a80 pg_info(1 pgs e243:0.3b7) v4 2014-06-09 16:22:11.731537 7f869af78700 5 -- op tracker -- , seq: 2789, time: 2014-06-09 16:22:11.574164, event: header_read, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.731544 7f869af78700 5 -- op tracker -- , seq: 2789, time: 2014-06-09 16:22:11.574164, event: throttled, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.731550 7f869af78700 5 -- op tracker -- , seq: 2789, time: 2014-06-09 16:22:11.574175, event: all_read, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.731553 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.731558 7f869af78700 5 -- op tracker -- , seq: 2789, time: 2014-06-09 16:22:11.731530, event: dispatched, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.731564 7f869af78700 5 -- op tracker -- , seq: 2789, time: 2014-06-09 16:22:11.731563, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.731566 7f868cf5c700 10 merge_log result log((0'0,214'635], crt=214'635) missing(1) changed=1 2014-06-09 16:22:11.731569 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3b7) v4 from osd.1 2014-06-09 16:22:11.731571 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc2a80 2014-06-09 16:22:11.731572 7f86a5e11700 15 _omap_rmkeyrange meta/1fe0f63a/pglog_3.730/0//-1 [0000000214.00000000000000000647,4294967295.18446744073709551615] 2014-06-09 16:22:11.731576 7f86a5e11700 15 get_omap_iterator meta/1fe0f63a/pglog_3.730/0//-1 2014-06-09 16:22:11.731576 7f869af78700 5 -- op tracker -- , seq: 2789, time: 2014-06-09 16:22:11.731574, event: started, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.731570 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 peering m=1] peer osd.9 now 3.2a5( v 214'635 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) missing(0) 2014-06-09 16:22:11.731591 7f869af78700 5 -- op tracker -- , seq: 2789, time: 2014-06-09 16:22:11.731589, event: done, op: pg_info(1 pgs e243:0.3b7) 2014-06-09 16:22:11.731599 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731600 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731592 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.725846 6 0.000204 2014-06-09 16:22:11.731612 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731626 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731628 7f869af78700 20 osd.26 243 _dispatch 0xddc5880 pg_info(1 pgs e243:0.714) v4 2014-06-09 16:22:11.731633 7f869af78700 5 -- op tracker -- , seq: 2790, time: 2014-06-09 16:22:11.579060, event: header_read, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.731644 7f869af78700 5 -- op tracker -- , seq: 2790, time: 2014-06-09 16:22:11.579061, event: throttled, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.731652 7f869af78700 5 -- op tracker -- , seq: 2790, time: 2014-06-09 16:22:11.579075, event: all_read, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.731605 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.731658 7f869af78700 5 -- op tracker -- , seq: 2790, time: 2014-06-09 16:22:11.731611, event: dispatched, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.731666 7f869af78700 5 -- op tracker -- , seq: 2790, time: 2014-06-09 16:22:11.731665, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.731659 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 peering m=1] state: osd.9 has no missing, identical log 2014-06-09 16:22:11.731668 7f86a5e11700 15 _omap_rmkeys meta/1fe0f63a/pglog_3.730/0//-1 2014-06-09 16:22:11.731671 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.714) v4 from osd.9 2014-06-09 16:22:11.731673 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc5880 2014-06-09 16:22:11.731678 7f869af78700 5 -- op tracker -- , seq: 2790, time: 2014-06-09 16:22:11.731676, event: started, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.731675 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000070 0 0.000000 2014-06-09 16:22:11.731686 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.731693 7f869af78700 5 -- op tracker -- , seq: 2790, time: 2014-06-09 16:22:11.731691, event: done, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.731696 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.190620 0 0.000000 2014-06-09 16:22:11.731704 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731703 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe0f63a/pglog_3.730/0//-1 2014-06-09 16:22:11.731705 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731715 7f86a5e11700 10 _omap_rmkeys meta/1fe0f63a/pglog_3.730/0//-1 r = 0 2014-06-09 16:22:11.731709 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.731717 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731719 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731720 7f869af78700 20 osd.26 243 _dispatch 0x2eef1a40 pg_info(1 pgs e243:3.750) v4 2014-06-09 16:22:11.731721 7f86a5e11700 15 _omap_rmkeys meta/1fe0f63a/pglog_3.730/0//-1 2014-06-09 16:22:11.731725 7f86a5e11700 10 _omap_rmkeys meta/1fe0f63a/pglog_3.730/0//-1 r = 0 2014-06-09 16:22:11.731719 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.731725 7f869af78700 5 -- op tracker -- , seq: 2791, time: 2014-06-09 16:22:11.445858, event: header_read, op: pg_info(1 pgs e243:3.750) 2014-06-09 16:22:11.731733 7f86a5e11700 15 _omap_setkeys meta/1fe0f63a/pglog_3.730/0//-1 2014-06-09 16:22:11.731733 7f869af78700 5 -- op tracker -- , seq: 2791, time: 2014-06-09 16:22:11.445860, event: throttled, op: pg_info(1 pgs e243:3.750) 2014-06-09 16:22:11.731731 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.731739 7f869af78700 5 -- op tracker -- , seq: 2791, time: 2014-06-09 16:22:11.445878, event: all_read, op: pg_info(1 pgs e243:3.750) 2014-06-09 16:22:11.731744 7f869af78700 5 -- op tracker -- , seq: 2791, time: 2014-06-09 16:22:11.731716, event: dispatched, op: pg_info(1 pgs e243:3.750) 2014-06-09 16:22:11.731741 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.731748 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3435.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.731752 7f869af78700 5 -- op tracker -- , seq: 2791, time: 2014-06-09 16:22:11.731750, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.750) 2014-06-09 16:22:11.731754 7f86a5e11700 10 cid: oid: 1fe0f63a/pglog_3.730/0//-1 skipping op, spos 3435.0.0 <= header.spos 150267.0.0 2014-06-09 16:22:11.731750 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'632 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.731756 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.750) v4 from osd.8 2014-06-09 16:22:11.731757 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.731759 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef1a40 2014-06-09 16:22:11.731763 7f869af78700 5 -- op tracker -- , seq: 2791, time: 2014-06-09 16:22:11.731762, event: started, op: pg_info(1 pgs e243:3.750) 2014-06-09 16:22:11.731780 7f869af78700 5 -- op tracker -- , seq: 2791, time: 2014-06-09 16:22:11.731778, event: done, op: pg_info(1 pgs e243:3.750) 2014-06-09 16:22:11.731786 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731788 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731788 7f86a5e11700 10 _do_op 0x2ea53920 seq 3435 r = 0, finisher 0x314121a0 0 2014-06-09 16:22:11.731792 7f86a5e11700 10 _finish_op 0x2ea53920 seq 3435 osr(3.730 0x3ed1ef0)/0x3ed1ef0 2014-06-09 16:22:11.731795 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc923c20 (1 active) 2014-06-09 16:22:11.731797 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34967e0 (2 active) 2014-06-09 16:22:11.731799 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731800 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731799 7f86a5e11700 5 _do_op 0x308ca720 seq 3436 osr(1.3bc 0x384d550)/0x384d550 start 2014-06-09 16:22:11.731802 7f86a5e11700 10 _do_transaction on 0x2eaa780 2014-06-09 16:22:11.731801 7f869af78700 20 osd.26 243 _dispatch 0x311628c0 pg_info(1 pgs e243:3.6f) v4 2014-06-09 16:22:11.731807 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.731806 7f869af78700 5 -- op tracker -- , seq: 2792, time: 2014-06-09 16:22:11.566337, event: header_read, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.731812 7f869af78700 5 -- op tracker -- , seq: 2792, time: 2014-06-09 16:22:11.566337, event: throttled, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.731817 7f869af78700 5 -- op tracker -- , seq: 2792, time: 2014-06-09 16:22:11.566351, event: all_read, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.731823 7f869af78700 5 -- op tracker -- , seq: 2792, time: 2014-06-09 16:22:11.731799, event: dispatched, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.731829 7f869af78700 5 -- op tracker -- , seq: 2792, time: 2014-06-09 16:22:11.731827, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.731833 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6f) v4 from osd.6 2014-06-09 16:22:11.731836 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311628c0 2014-06-09 16:22:11.731840 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.731840 7f869af78700 5 -- op tracker -- , seq: 2792, time: 2014-06-09 16:22:11.731838, event: started, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.731854 7f869af78700 5 -- op tracker -- , seq: 2792, time: 2014-06-09 16:22:11.731852, event: done, op: pg_info(1 pgs e243:3.6f) 2014-06-09 16:22:11.731857 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3436.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.731861 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731861 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.731862 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731873 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731874 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731875 7f869af78700 20 osd.26 243 _dispatch 0x30cf0380 pg_info(1 pgs e243:0.55d) v4 2014-06-09 16:22:11.731872 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.731879 7f86a5e11700 10 _do_op 0x308ca720 seq 3436 r = 0, finisher 0xcd33220 0 2014-06-09 16:22:11.731882 7f86a5e11700 10 _finish_op 0x308ca720 seq 3436 osr(1.3bc 0x384d550)/0x384d550 2014-06-09 16:22:11.731880 7f869af78700 5 -- op tracker -- , seq: 2793, time: 2014-06-09 16:22:11.590854, event: header_read, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.731890 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x7142000 (1 active) 2014-06-09 16:22:11.731886 7f868cf5c700 10 osd.26 243 queue_for_recovery queued pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.731892 7f869af78700 5 -- op tracker -- , seq: 2793, time: 2014-06-09 16:22:11.590855, event: throttled, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.731895 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.731899 7f869af78700 5 -- op tracker -- , seq: 2793, time: 2014-06-09 16:22:11.590868, event: all_read, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.731901 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34967e0 (1 active) 2014-06-09 16:22:11.731904 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e5a0 (2 active) 2014-06-09 16:22:11.731905 7f869af78700 5 -- op tracker -- , seq: 2793, time: 2014-06-09 16:22:11.731873, event: dispatched, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.731906 7f86a5e11700 5 _do_op 0x308ca6c0 seq 3437 osr(1.286 0x384ced0)/0x384ced0 start 2014-06-09 16:22:11.731909 7f86a5e11700 10 _do_transaction on 0xc3f1500 2014-06-09 16:22:11.731900 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 3.2a5( v 214'635 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) 2014-06-09 16:22:11.731911 7f869af78700 5 -- op tracker -- , seq: 2793, time: 2014-06-09 16:22:11.731909, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.731915 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.731917 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.55d) v4 from osd.0 2014-06-09 16:22:11.731911 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 is up to date, queueing in pending_activators 2014-06-09 16:22:11.731919 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf0380 2014-06-09 16:22:11.731924 7f869af78700 5 -- op tracker -- , seq: 2793, time: 2014-06-09 16:22:11.731922, event: started, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.731928 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.9 3.2a5( v 214'635 (0'0,214'635] local-les=213 n=128 ec=118 les/c 213/213 242/242/242) uptodate 2014-06-09 16:22:11.731937 7f869af78700 5 -- op tracker -- , seq: 2793, time: 2014-06-09 16:22:11.731935, event: done, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.731947 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731948 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731941 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.731949 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.731952 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3 214'635 also missing on osd.26 2014-06-09 16:22:11.731963 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.731964 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.731965 7f869af78700 20 osd.26 243 _dispatch 0x2176d500 pg_info(1 pgs e243:2.3c0) v4 2014-06-09 16:22:11.731966 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3437.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.731970 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.731962 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3=214'635(214'632)} 2014-06-09 16:22:11.731969 7f869af78700 5 -- op tracker -- , seq: 2794, time: 2014-06-09 16:22:11.575020, event: header_read, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.731976 7f869af78700 5 -- op tracker -- , seq: 2794, time: 2014-06-09 16:22:11.575021, event: throttled, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.731973 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 9 as a source 2014-06-09 16:22:11.731981 7f869af78700 5 -- op tracker -- , seq: 2794, time: 2014-06-09 16:22:11.575034, event: all_read, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.731989 7f869af78700 5 -- op tracker -- , seq: 2794, time: 2014-06-09 16:22:11.731962, event: dispatched, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.731983 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3 214'635 is on osd.9 2014-06-09 16:22:11.731996 7f869af78700 5 -- op tracker -- , seq: 2794, time: 2014-06-09 16:22:11.731994, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.731998 7f86a5e11700 10 _do_op 0x308ca6c0 seq 3437 r = 0, finisher 0x314567e0 0 2014-06-09 16:22:11.732001 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3c0) v4 from osd.5 2014-06-09 16:22:11.732002 7f86a5e11700 10 _finish_op 0x308ca6c0 seq 3437 osr(1.286 0x384ced0)/0x384ced0 2014-06-09 16:22:11.732004 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176d500 2014-06-09 16:22:11.731995 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {f7ee42a5/rbd_data.117e6b8b4567.00000000000014d0/head//3=214'635(214'632)} 2014-06-09 16:22:11.732005 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e5a0 (1 active) 2014-06-09 16:22:11.732007 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4acd80 (2 active) 2014-06-09 16:22:11.732010 7f86a5e11700 5 _do_op 0x2c06efc0 seq 3438 osr(3.408 0x3ed1270)/0x3ed1270 start 2014-06-09 16:22:11.732009 7f869af78700 5 -- op tracker -- , seq: 2794, time: 2014-06-09 16:22:11.732007, event: started, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.732012 7f86a5e11700 10 _do_transaction on 0xbfca600 2014-06-09 16:22:11.732007 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.732019 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.732023 7f869af78700 5 -- op tracker -- , seq: 2794, time: 2014-06-09 16:22:11.732021, event: done, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.732017 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 213 2014-06-09 16:22:11.732029 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732031 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732028 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 9 2014-06-09 16:22:11.732042 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732043 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732044 7f869af78700 20 osd.26 243 _dispatch 0x307e7dc0 pg_info(1 pgs e243:0.722) v4 2014-06-09 16:22:11.732038 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.732048 7f869af78700 5 -- op tracker -- , seq: 2795, time: 2014-06-09 16:22:11.322897, event: header_read, op: pg_info(1 pgs e243:0.722) 2014-06-09 16:22:11.732052 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.732054 7f869af78700 5 -- op tracker -- , seq: 2795, time: 2014-06-09 16:22:11.322898, event: throttled, op: pg_info(1 pgs e243:0.722) 2014-06-09 16:22:11.732049 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.732060 7f869af78700 5 -- op tracker -- , seq: 2795, time: 2014-06-09 16:22:11.322910, event: all_read, op: pg_info(1 pgs e243:0.722) 2014-06-09 16:22:11.732066 7f869af78700 5 -- op tracker -- , seq: 2795, time: 2014-06-09 16:22:11.732041, event: dispatched, op: pg_info(1 pgs e243:0.722) 2014-06-09 16:22:11.732068 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3438.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.732073 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.732072 7f869af78700 5 -- op tracker -- , seq: 2795, time: 2014-06-09 16:22:11.732070, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.722) 2014-06-09 16:22:11.732076 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.722) v4 from osd.2 2014-06-09 16:22:11.732079 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x307e7dc0 2014-06-09 16:22:11.732084 7f868cf5c700 10 write_log with: dirty_to: 0'0, dirty_from: 214'632, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.732083 7f869af78700 5 -- op tracker -- , seq: 2795, time: 2014-06-09 16:22:11.732082, event: started, op: pg_info(1 pgs e243:0.722) 2014-06-09 16:22:11.732091 7f86a5e11700 10 _do_op 0x2c06efc0 seq 3438 r = 0, finisher 0x307f1760 0 2014-06-09 16:22:11.732095 7f86a5e11700 10 _finish_op 0x2c06efc0 seq 3438 osr(3.408 0x3ed1270)/0x3ed1270 2014-06-09 16:22:11.732097 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4acd80 (1 active) 2014-06-09 16:22:11.732099 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4480 (2 active) 2014-06-09 16:22:11.732102 7f86a5e11700 5 _do_op 0x15996d20 seq 3439 osr(1.4bb 0x384d970)/0x384d970 start 2014-06-09 16:22:11.732100 7f869af78700 5 -- op tracker -- , seq: 2795, time: 2014-06-09 16:22:11.732098, event: done, op: pg_info(1 pgs e243:0.722) 2014-06-09 16:22:11.732105 7f86a5e11700 10 _do_transaction on 0x35629c0 2014-06-09 16:22:11.732107 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732109 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732110 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.732120 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732122 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732123 7f869af78700 20 osd.26 243 _dispatch 0x2eee7880 pg_info(1 pgs e243:1.3e2) v4 2014-06-09 16:22:11.732125 7f868cf5c700 5 queue_transactions existing osr(3.2a5 0x3ed0b80)/0x3ed0b80 2014-06-09 16:22:11.732128 7f868cf5c700 10 op_submit_start 3481 2014-06-09 16:22:11.732129 7f868cf5c700 5 queue_transactions (trailing journal) 3481 2014-06-09 16:22:11.732130 7f868cf5c700 5 queue_op 0x10038000 seq 3481 osr(3.2a5 0x3ed0b80) 2160 bytes (queue has 44 ops and 4553365 bytes) 2014-06-09 16:22:11.732127 7f869af78700 5 -- op tracker -- , seq: 2796, time: 2014-06-09 16:22:11.430431, event: header_read, op: pg_info(1 pgs e243:1.3e2) 2014-06-09 16:22:11.732133 7f868cf5c700 10 op_submit_finish 3481 2014-06-09 16:22:11.732135 7f869af78700 5 -- op tracker -- , seq: 2796, time: 2014-06-09 16:22:11.430431, event: throttled, op: pg_info(1 pgs e243:1.3e2) 2014-06-09 16:22:11.732141 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x7142000 (0 active) 2014-06-09 16:22:11.732141 7f869af78700 5 -- op tracker -- , seq: 2796, time: 2014-06-09 16:22:11.430440, event: all_read, op: pg_info(1 pgs e243:1.3e2) 2014-06-09 16:22:11.732144 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.732138 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.442( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.732147 7f869af78700 5 -- op tracker -- , seq: 2796, time: 2014-06-09 16:22:11.732120, event: dispatched, op: pg_info(1 pgs e243:1.3e2) 2014-06-09 16:22:11.732153 7f869af78700 5 -- op tracker -- , seq: 2796, time: 2014-06-09 16:22:11.732151, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3e2) 2014-06-09 16:22:11.732151 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.732156 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.732159 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3e2) v4 from osd.4 2014-06-09 16:22:11.732160 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732161 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee7880 2014-06-09 16:22:11.732166 7f869af78700 5 -- op tracker -- , seq: 2796, time: 2014-06-09 16:22:11.732164, event: started, op: pg_info(1 pgs e243:1.3e2) 2014-06-09 16:22:11.732164 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.185( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.732172 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3439.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.732177 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.732176 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.732180 7f869af78700 5 -- op tracker -- , seq: 2796, time: 2014-06-09 16:22:11.732178, event: done, op: pg_info(1 pgs e243:1.3e2) 2014-06-09 16:22:11.732185 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732188 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732190 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732188 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.732203 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732201 7f86a5e11700 10 _do_op 0x15996d20 seq 3439 r = 0, finisher 0x31435400 0 2014-06-09 16:22:11.732204 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732205 7f86a5e11700 10 _finish_op 0x15996d20 seq 3439 osr(1.4bb 0x384d970)/0x384d970 2014-06-09 16:22:11.732205 7f869af78700 20 osd.26 243 _dispatch 0x30a3a000 pg_info(1 pgs e243:3.2fd) v4 2014-06-09 16:22:11.732208 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4480 (1 active) 2014-06-09 16:22:11.732200 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.6 3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) 2014-06-09 16:22:11.732210 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3ac60 (2 active) 2014-06-09 16:22:11.732213 7f86a5e11700 5 _do_op 0x2c06ef00 seq 3440 osr(3.4c5 0x3ed1430)/0x3ed1430 start 2014-06-09 16:22:11.732210 7f869af78700 5 -- op tracker -- , seq: 2797, time: 2014-06-09 16:22:11.606059, event: header_read, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.732216 7f86a5e11700 10 _do_transaction on 0x2ed4e40 2014-06-09 16:22:11.732212 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.227629 7 0.000190 2014-06-09 16:22:11.732218 7f869af78700 5 -- op tracker -- , seq: 2797, time: 2014-06-09 16:22:11.606060, event: throttled, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.732221 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.732224 7f869af78700 5 -- op tracker -- , seq: 2797, time: 2014-06-09 16:22:11.606068, event: all_read, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.732223 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.732232 7f869af78700 5 -- op tracker -- , seq: 2797, time: 2014-06-09 16:22:11.732203, event: dispatched, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.732234 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.732238 7f869af78700 5 -- op tracker -- , seq: 2797, time: 2014-06-09 16:22:11.732236, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.732244 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2fd) v4 from osd.7 2014-06-09 16:22:11.732246 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3a000 2014-06-09 16:22:11.732244 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=208 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.732251 7f869af78700 5 -- op tracker -- , seq: 2797, time: 2014-06-09 16:22:11.732249, event: started, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.732254 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.732253 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'569 -> 213'569 2014-06-09 16:22:11.732265 7f869af78700 5 -- op tracker -- , seq: 2797, time: 2014-06-09 16:22:11.732263, event: done, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.732273 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732270 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3440.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.732274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732275 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.732285 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732286 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732287 7f869af78700 20 osd.26 243 _dispatch 0x30dc3a40 pg_info(1 pgs e243:2.69c) v4 2014-06-09 16:22:11.732291 7f869af78700 5 -- op tracker -- , seq: 2798, time: 2014-06-09 16:22:11.571944, event: header_read, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.732294 7f86a5e11700 10 _do_op 0x2c06ef00 seq 3440 r = 0, finisher 0x31456f20 0 2014-06-09 16:22:11.732298 7f86a5e11700 10 _finish_op 0x2c06ef00 seq 3440 osr(3.4c5 0x3ed1430)/0x3ed1430 2014-06-09 16:22:11.732291 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.732300 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3ac60 (1 active) 2014-06-09 16:22:11.732298 7f869af78700 5 -- op tracker -- , seq: 2798, time: 2014-06-09 16:22:11.571945, event: throttled, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.732303 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcada6c0 (2 active) 2014-06-09 16:22:11.732305 7f86a5e11700 5 _do_op 0xc2ea900 seq 3441 osr(0.179 0x2bd6800)/0x2bd6800 start 2014-06-09 16:22:11.732308 7f86a5e11700 10 _do_transaction on 0x97bab40 2014-06-09 16:22:11.732306 7f869af78700 5 -- op tracker -- , seq: 2798, time: 2014-06-09 16:22:11.571958, event: all_read, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.732313 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.732312 7f869af78700 5 -- op tracker -- , seq: 2798, time: 2014-06-09 16:22:11.732284, event: dispatched, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.732318 7f869af78700 5 -- op tracker -- , seq: 2798, time: 2014-06-09 16:22:11.732317, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.732328 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.69c) v4 from osd.3 2014-06-09 16:22:11.732331 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732331 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc3a40 2014-06-09 16:22:11.732335 7f868cf5c700 5 queue_transactions existing osr(3.544 0x3ed19a0)/0x3ed19a0 2014-06-09 16:22:11.732338 7f868cf5c700 10 op_submit_start 3482 2014-06-09 16:22:11.732339 7f868cf5c700 5 queue_transactions (trailing journal) 3482 2014-06-09 16:22:11.732335 7f869af78700 5 -- op tracker -- , seq: 2798, time: 2014-06-09 16:22:11.732333, event: started, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.732340 7f868cf5c700 5 queue_op 0x16fd55c0 seq 3482 osr(3.544 0x3ed19a0) 1335 bytes (queue has 43 ops and 4551266 bytes) 2014-06-09 16:22:11.732342 7f868cf5c700 10 op_submit_finish 3482 2014-06-09 16:22:11.732346 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.732345 7f868cf5c700 20 osd.26 243 Sending info 3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) to shard 9 2014-06-09 16:22:11.732351 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.732351 7f869af78700 5 -- op tracker -- , seq: 2798, time: 2014-06-09 16:22:11.732349, event: done, op: pg_info(1 pgs e243:2.69c) 2014-06-09 16:22:11.732359 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732361 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732372 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732374 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732375 7f869af78700 20 osd.26 243 _dispatch 0x2176c1c0 pg_info(1 pgs e243:2.f4) v4 2014-06-09 16:22:11.732381 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (19 active) 2014-06-09 16:22:11.732379 7f869af78700 5 -- op tracker -- , seq: 2799, time: 2014-06-09 16:22:11.577958, event: header_read, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.732388 7f869af78700 5 -- op tracker -- , seq: 2799, time: 2014-06-09 16:22:11.577958, event: throttled, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.732363 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3441.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.732397 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (20 active) 2014-06-09 16:22:11.732397 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.732393 7f869af78700 5 -- op tracker -- , seq: 2799, time: 2014-06-09 16:22:11.577971, event: all_read, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.732405 7f869af78700 5 -- op tracker -- , seq: 2799, time: 2014-06-09 16:22:11.732372, event: dispatched, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.732401 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:243, info:2.442( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.732412 7f869af78700 5 -- op tracker -- , seq: 2799, time: 2014-06-09 16:22:11.732410, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.732412 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.732418 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.f4) v4 from osd.1 2014-06-09 16:22:11.732421 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176c1c0 2014-06-09 16:22:11.732424 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732424 7f86a5e11700 10 _do_op 0xc2ea900 seq 3441 r = 0, finisher 0x31434b20 0 2014-06-09 16:22:11.732428 7f86a5e11700 10 _finish_op 0xc2ea900 seq 3441 osr(0.179 0x2bd6800)/0x2bd6800 2014-06-09 16:22:11.732425 7f869af78700 5 -- op tracker -- , seq: 2799, time: 2014-06-09 16:22:11.732423, event: started, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.732431 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcada6c0 (1 active) 2014-06-09 16:22:11.732433 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dca20 (2 active) 2014-06-09 16:22:11.732435 7f86a5e11700 5 _do_op 0xc2eacc0 seq 3442 osr(3.4c8 0x3ed13c0)/0x3ed13c0 start 2014-06-09 16:22:11.732429 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.4b5( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.732438 7f86a5e11700 10 _do_transaction on 0xc143800 2014-06-09 16:22:11.732441 7f869af78700 5 -- op tracker -- , seq: 2799, time: 2014-06-09 16:22:11.732439, event: done, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.732444 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.732440 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.732449 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732448 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732450 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732461 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732453 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.732463 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732464 7f869af78700 20 osd.26 243 _dispatch 0xddc56c0 pg_info(1 pgs e243:2.220) v4 2014-06-09 16:22:11.732468 7f869af78700 5 -- op tracker -- , seq: 2800, time: 2014-06-09 16:22:11.580251, event: header_read, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.732466 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.732476 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732475 7f869af78700 5 -- op tracker -- , seq: 2800, time: 2014-06-09 16:22:11.580252, event: throttled, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.732478 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.732481 7f869af78700 5 -- op tracker -- , seq: 2800, time: 2014-06-09 16:22:11.580270, event: all_read, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.732480 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.732486 7f869af78700 5 -- op tracker -- , seq: 2800, time: 2014-06-09 16:22:11.732461, event: dispatched, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.732492 7f869af78700 5 -- op tracker -- , seq: 2800, time: 2014-06-09 16:22:11.732491, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.732491 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.732495 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3442.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.732497 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.220) v4 from osd.9 2014-06-09 16:22:11.732499 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732500 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.732499 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc56c0 2014-06-09 16:22:11.732504 7f869af78700 5 -- op tracker -- , seq: 2800, time: 2014-06-09 16:22:11.732502, event: started, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.732503 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.732517 7f86a5e11700 10 _do_op 0xc2eacc0 seq 3442 r = 0, finisher 0x314571a0 0 2014-06-09 16:22:11.732514 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.732517 7f869af78700 5 -- op tracker -- , seq: 2800, time: 2014-06-09 16:22:11.732516, event: done, op: pg_info(1 pgs e243:2.220) 2014-06-09 16:22:11.732520 7f86a5e11700 10 _finish_op 0xc2eacc0 seq 3442 osr(3.4c8 0x3ed13c0)/0x3ed13c0 2014-06-09 16:22:11.732523 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dca20 (1 active) 2014-06-09 16:22:11.732524 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732525 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc120 (2 active) 2014-06-09 16:22:11.732526 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732521 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.732528 7f86a5e11700 5 _do_op 0xc34b7a0 seq 3443 osr(3.105 0x3ed0820)/0x3ed0820 start 2014-06-09 16:22:11.732530 7f86a5e11700 10 _do_transaction on 0x3562180 2014-06-09 16:22:11.732529 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.732536 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.732537 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732539 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732540 7f869af78700 20 osd.26 243 _dispatch 0x2eef1880 pg_info(1 pgs e243:1.1eb) v4 2014-06-09 16:22:11.732538 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.732544 7f869af78700 5 -- op tracker -- , seq: 2801, time: 2014-06-09 16:22:11.446209, event: header_read, op: pg_info(1 pgs e243:1.1eb) 2014-06-09 16:22:11.732545 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.732553 7f869af78700 5 -- op tracker -- , seq: 2801, time: 2014-06-09 16:22:11.446209, event: throttled, op: pg_info(1 pgs e243:1.1eb) 2014-06-09 16:22:11.732559 7f869af78700 5 -- op tracker -- , seq: 2801, time: 2014-06-09 16:22:11.446220, event: all_read, op: pg_info(1 pgs e243:1.1eb) 2014-06-09 16:22:11.732557 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.732564 7f869af78700 5 -- op tracker -- , seq: 2801, time: 2014-06-09 16:22:11.732537, event: dispatched, op: pg_info(1 pgs e243:1.1eb) 2014-06-09 16:22:11.732568 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.732566 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.732571 7f869af78700 5 -- op tracker -- , seq: 2801, time: 2014-06-09 16:22:11.732569, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1eb) 2014-06-09 16:22:11.732575 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1eb) v4 from osd.8 2014-06-09 16:22:11.732578 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef1880 2014-06-09 16:22:11.732573 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.732581 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732584 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3443.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.732582 7f869af78700 5 -- op tracker -- , seq: 2801, time: 2014-06-09 16:22:11.732580, event: started, op: pg_info(1 pgs e243:1.1eb) 2014-06-09 16:22:11.732588 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.732585 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.732598 7f869af78700 5 -- op tracker -- , seq: 2801, time: 2014-06-09 16:22:11.732596, event: done, op: pg_info(1 pgs e243:1.1eb) 2014-06-09 16:22:11.732598 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.732606 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732607 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732608 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732615 7f86a5e11700 10 _do_op 0xc34b7a0 seq 3443 r = 0, finisher 0xcd33f20 0 2014-06-09 16:22:11.732619 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732619 7f86a5e11700 10 _finish_op 0xc34b7a0 seq 3443 osr(3.105 0x3ed0820)/0x3ed0820 2014-06-09 16:22:11.732612 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.37( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.732620 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732622 7f869af78700 20 osd.26 243 _dispatch 0x31162700 pg_info(1 pgs e243:2.f2) v4 2014-06-09 16:22:11.732621 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc120 (1 active) 2014-06-09 16:22:11.732624 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3006c0 (2 active) 2014-06-09 16:22:11.732626 7f86a5e11700 5 _do_op 0x2fcaf740 seq 3444 osr(0.330 0x2bd70a0)/0x2bd70a0 start 2014-06-09 16:22:11.732629 7f86a5e11700 10 _do_transaction on 0xc253500 2014-06-09 16:22:11.732626 7f869af78700 5 -- op tracker -- , seq: 2802, time: 2014-06-09 16:22:11.566627, event: header_read, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.732635 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.732633 7f869af78700 5 -- op tracker -- , seq: 2802, time: 2014-06-09 16:22:11.566627, event: throttled, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.732641 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732645 7f868cf5c700 5 queue_transactions existing osr(2.37 0x2ed05f0)/0x2ed05f0 2014-06-09 16:22:11.732639 7f869af78700 5 -- op tracker -- , seq: 2802, time: 2014-06-09 16:22:11.566701, event: all_read, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.732648 7f868cf5c700 10 op_submit_start 3483 2014-06-09 16:22:11.732649 7f868cf5c700 5 queue_transactions (trailing journal) 3483 2014-06-09 16:22:11.732650 7f868cf5c700 5 queue_op 0x2fcae4e0 seq 3483 osr(2.37 0x2ed05f0) 826 bytes (queue has 41 ops and 4547445 bytes) 2014-06-09 16:22:11.732653 7f868cf5c700 10 op_submit_finish 3483 2014-06-09 16:22:11.732650 7f869af78700 5 -- op tracker -- , seq: 2802, time: 2014-06-09 16:22:11.732618, event: dispatched, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.732660 7f869af78700 5 -- op tracker -- , seq: 2802, time: 2014-06-09 16:22:11.732658, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.732656 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.732664 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.f2) v4 from osd.6 2014-06-09 16:22:11.732667 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31162700 2014-06-09 16:22:11.732667 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.732671 7f869af78700 5 -- op tracker -- , seq: 2802, time: 2014-06-09 16:22:11.732669, event: started, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.732668 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.732684 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3444.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.732680 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.849361 7 0.000755 2014-06-09 16:22:11.732688 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.732687 7f869af78700 5 -- op tracker -- , seq: 2802, time: 2014-06-09 16:22:11.732685, event: done, op: pg_info(1 pgs e243:2.f2) 2014-06-09 16:22:11.732694 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732696 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732691 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.732706 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732708 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732702 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.732709 7f869af78700 20 osd.26 243 _dispatch 0x30cf0a80 pg_info(1 pgs e243:0.25b) v4 2014-06-09 16:22:11.732709 7f86a5e11700 10 _do_op 0x2fcaf740 seq 3444 r = 0, finisher 0x31435620 0 2014-06-09 16:22:11.732712 7f86a5e11700 10 _finish_op 0x2fcaf740 seq 3444 osr(0.330 0x2bd70a0)/0x2bd70a0 2014-06-09 16:22:11.732715 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3006c0 (1 active) 2014-06-09 16:22:11.732713 7f869af78700 5 -- op tracker -- , seq: 2803, time: 2014-06-09 16:22:11.591388, event: header_read, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.732717 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fde60 (2 active) 2014-06-09 16:22:11.732711 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=206 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.732720 7f86a5e11700 5 _do_op 0x2fcaf6e0 seq 3445 osr(3.148 0x3ed0740)/0x3ed0740 start 2014-06-09 16:22:11.732723 7f86a5e11700 10 _do_transaction on 0xbf0ef00 2014-06-09 16:22:11.732720 7f869af78700 5 -- op tracker -- , seq: 2803, time: 2014-06-09 16:22:11.591388, event: throttled, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.732721 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'551 -> 213'551 2014-06-09 16:22:11.732729 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.732726 7f869af78700 5 -- op tracker -- , seq: 2803, time: 2014-06-09 16:22:11.591399, event: all_read, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.732738 7f869af78700 5 -- op tracker -- , seq: 2803, time: 2014-06-09 16:22:11.732706, event: dispatched, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.732745 7f869af78700 5 -- op tracker -- , seq: 2803, time: 2014-06-09 16:22:11.732743, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.732756 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.25b) v4 from osd.0 2014-06-09 16:22:11.732759 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf0a80 2014-06-09 16:22:11.732761 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.732757 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.732764 7f869af78700 5 -- op tracker -- , seq: 2803, time: 2014-06-09 16:22:11.732762, event: started, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.732777 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3445.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.732779 7f869af78700 5 -- op tracker -- , seq: 2803, time: 2014-06-09 16:22:11.732777, event: done, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.732782 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.732785 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732787 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732793 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732797 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732797 7f868cf5c700 5 queue_transactions existing osr(3.48d 0x3ed14e0)/0x3ed14e0 2014-06-09 16:22:11.732799 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732797 7f86a5e11700 10 _do_op 0x2fcaf6e0 seq 3445 r = 0, finisher 0x31435320 0 2014-06-09 16:22:11.732800 7f868cf5c700 10 op_submit_start 3484 2014-06-09 16:22:11.732800 7f868cf5c700 5 queue_transactions (trailing journal) 3484 2014-06-09 16:22:11.732800 7f869af78700 20 osd.26 243 _dispatch 0x30dc2700 pg_info(1 pgs e243:2.52e) v4 2014-06-09 16:22:11.732801 7f86a5e11700 10 _finish_op 0x2fcaf6e0 seq 3445 osr(3.148 0x3ed0740)/0x3ed0740 2014-06-09 16:22:11.732802 7f868cf5c700 5 queue_op 0x16fd5b00 seq 3484 osr(3.48d 0x3ed14e0) 1335 bytes (queue has 41 ops and 4546897 bytes) 2014-06-09 16:22:11.732804 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fde60 (1 active) 2014-06-09 16:22:11.732806 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1440 (2 active) 2014-06-09 16:22:11.732804 7f869af78700 5 -- op tracker -- , seq: 2804, time: 2014-06-09 16:22:11.575047, event: header_read, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.732813 7f868cf5c700 10 op_submit_finish 3484 2014-06-09 16:22:11.732812 7f869af78700 5 -- op tracker -- , seq: 2804, time: 2014-06-09 16:22:11.575047, event: throttled, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.732817 7f8689755700 10 op_submit_start 3485 2014-06-09 16:22:11.732818 7f8689755700 5 queue_transactions (trailing journal) 3485 2014-06-09 16:22:11.732819 7f8689755700 5 queue_op 0x16fd4cc0 seq 3485 osr(0.298 0x2bd6c60) 1876 bytes (queue has 44 ops and 5069150 bytes) 2014-06-09 16:22:11.732823 7f8689755700 10 op_submit_finish 3485 2014-06-09 16:22:11.732818 7f869af78700 5 -- op tracker -- , seq: 2804, time: 2014-06-09 16:22:11.575066, event: all_read, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.732826 7f869af78700 5 -- op tracker -- , seq: 2804, time: 2014-06-09 16:22:11.732797, event: dispatched, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.732819 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.732826 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.732833 7f869af78700 5 -- op tracker -- , seq: 2804, time: 2014-06-09 16:22:11.732831, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.732833 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.732842 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.52e) v4 from osd.5 2014-06-09 16:22:11.732845 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc2700 2014-06-09 16:22:11.732839 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.732848 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.732842 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.732836 7f868c75b700 10 op_submit_start 3486 2014-06-09 16:22:11.732849 7f869af78700 5 -- op tracker -- , seq: 2804, time: 2014-06-09 16:22:11.732847, event: started, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.732851 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.732853 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.732852 7f868c75b700 5 queue_transactions (trailing journal) 3486 2014-06-09 16:22:11.732860 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.732866 7f869af78700 5 -- op tracker -- , seq: 2804, time: 2014-06-09 16:22:11.732864, event: done, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.732873 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732866 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.732875 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732869 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.732877 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.732869 7f868c75b700 5 queue_op 0x230b96e0 seq 3486 osr(3.1e 0x3ed0380) 518380 bytes (queue has 44 ops and 5069150 bytes) 2014-06-09 16:22:11.732886 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.732888 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.732881 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.732889 7f869af78700 20 osd.26 243 _dispatch 0x240e8380 pg_info(1 pgs e243:2.740) v4 2014-06-09 16:22:11.732883 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:190 2014-06-09 16:22:11.732893 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.732893 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.732894 7f869af78700 5 -- op tracker -- , seq: 2805, time: 2014-06-09 16:22:11.322914, event: header_read, op: pg_info(1 pgs e243:2.740) 2014-06-09 16:22:11.732888 7f868c75b700 10 op_submit_finish 3486 2014-06-09 16:22:11.732905 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.732902 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.732906 7f869af78700 5 -- op tracker -- , seq: 2805, time: 2014-06-09 16:22:11.322915, event: throttled, op: pg_info(1 pgs e243:2.740) 2014-06-09 16:22:11.732909 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.732912 7f869af78700 5 -- op tracker -- , seq: 2805, time: 2014-06-09 16:22:11.322924, event: all_read, op: pg_info(1 pgs e243:2.740) 2014-06-09 16:22:11.732909 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.732921 7f8692767700 10 op_submit_start 3487 2014-06-09 16:22:11.732914 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.735178 7f8692767700 5 queue_transactions (trailing journal) 3487 2014-06-09 16:22:11.735176 7f869af78700 5 -- op tracker -- , seq: 2805, time: 2014-06-09 16:22:11.732886, event: dispatched, op: pg_info(1 pgs e243:2.740) 2014-06-09 16:22:11.735181 7f8692767700 5 queue_op 0xbc2dec0 seq 3487 osr(1.438 0x384d710) 1997 bytes (queue has 50 ops and 5594942 bytes) 2014-06-09 16:22:11.735184 7f8692767700 10 op_submit_finish 3487 2014-06-09 16:22:11.735177 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.735189 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.735188 7f869af78700 5 -- op tracker -- , seq: 2805, time: 2014-06-09 16:22:11.735186, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.740) 2014-06-09 16:22:11.735184 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.735193 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.740) v4 from osd.2 2014-06-09 16:22:11.735197 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e8380 2014-06-09 16:22:11.735189 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.735196 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.735193 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.735203 7f869af78700 5 -- op tracker -- , seq: 2805, time: 2014-06-09 16:22:11.735201, event: started, op: pg_info(1 pgs e243:2.740) 2014-06-09 16:22:11.735205 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.735204 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.735207 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.732907 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc128a80 243'595 rep_tid=8 committed?=0 applied?=0 lock=0 op=osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.735217 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.735214 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.735221 7f868c75b700 10 osd.26 243 dequeue_op 0x19fe0800 finish 2014-06-09 16:22:11.735216 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735222 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.735229 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.735222 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.30( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.30( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.735225 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735235 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.732927 7f86a5e11700 5 _do_op 0x2fc8df20 seq 3446 osr(1.233 0x384cc30)/0x384cc30 start 2014-06-09 16:22:11.735239 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.732975 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.735233 7f8689f56700 10 osd.26 243 dequeue_op 0xdfa6100 prio 196 cost 0 latency 0.010125 osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 pg pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.735243 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:185 2014-06-09 16:22:11.735252 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.735256 7f8689755700 5 queue_transactions existing osr(1.30 0x384c280)/0x384c280 2014-06-09 16:22:11.735258 7f8689755700 2 waiting 51 > 50 ops || 5595768 > 104857600 2014-06-09 16:22:11.735253 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.735262 7f8692f68700 10 op_submit_start 3488 2014-06-09 16:22:11.735253 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:934 2014-06-09 16:22:11.735260 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3431.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.735253 7f8689f56700 5 -- op tracker -- , seq: 2768, time: 2014-06-09 16:22:11.735249, event: reached_pg, op: osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.735263 7f8692f68700 5 queue_transactions (trailing journal) 3488 2014-06-09 16:22:11.735264 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.735265 7f8692f68700 5 queue_op 0xc2eba40 seq 3488 osr(3.1b1 0x3ed0a70) 1335 bytes (queue has 50 ops and 5594942 bytes) 2014-06-09 16:22:11.735268 7f8692f68700 10 op_submit_finish 3488 2014-06-09 16:22:11.735269 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.735262 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.735272 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.735273 7f868ef60700 10 op_submit_start 3489 2014-06-09 16:22:11.735274 7f868ef60700 5 queue_transactions (trailing journal) 3489 2014-06-09 16:22:11.735266 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0xdfa6100 2014-06-09 16:22:11.735276 7f868ef60700 5 queue_op 0xc2d5da0 seq 3489 osr(0.763 0x384c350) 1883 bytes (queue has 50 ops and 5594942 bytes) 2014-06-09 16:22:11.735279 7f868ef60700 10 op_submit_finish 3489 2014-06-09 16:22:11.735273 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.735273 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.735281 7f8689f56700 5 -- op tracker -- , seq: 2768, time: 2014-06-09 16:22:11.735277, event: started, op: osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.735277 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.735285 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.735285 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.735284 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.735287 7f8689f56700 7 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 7 op ack_type 4 from 9 2014-06-09 16:22:11.735294 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735294 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735290 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.735304 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.735296 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.4c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.735304 7f8689f56700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.735298, event: sub_op_commit_rec, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.735308 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.735308 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.735303 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735303 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735311 7f8689f56700 10 osd.26 243 dequeue_op 0xdfa6100 finish 2014-06-09 16:22:11.735311 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.735313 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.776419 7 0.000187 2014-06-09 16:22:11.735318 7f8689f56700 5 -- op tracker -- , seq: 2768, time: 2014-06-09 16:22:11.735314, event: done, op: osd_sub_op_reply(client.4571.0:35948 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.735315 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.889826 7 0.000195 2014-06-09 16:22:11.735322 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.735328 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.735327 7f8692767700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.735325 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.735335 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.735345 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f25d0 (20 active) 2014-06-09 16:22:11.735344 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.735342 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.735348 7f8692f68700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.735348 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.735357 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.735353 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.735334 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.735360 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.735361 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.735361 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.735365 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.735368 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.735379 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.735374 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.735372 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.735376 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.735382 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735383 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.735384 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.735385 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.735390 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735392 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.735395 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.735403 7f8692767700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.735393 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.735407 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.735404 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:179 2014-06-09 16:22:11.735412 7f868ef60700 5 queue_transactions existing osr(1.4c9 0x384d8b0)/0x384d8b0 2014-06-09 16:22:11.735414 7f868ef60700 2 waiting 51 > 50 ops || 5596896 > 104857600 2014-06-09 16:22:11.735408 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735419 7f868bf5a700 10 op_submit_start 3490 2014-06-09 16:22:11.735420 7f868bf5a700 5 queue_transactions (trailing journal) 3490 2014-06-09 16:22:11.735413 7f8689f56700 20 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.735421 7f868bf5a700 5 queue_op 0x309a2cc0 seq 3490 osr(3.11f 0x3ed07e0) 1335 bytes (queue has 50 ops and 5594942 bytes) 2014-06-09 16:22:11.735417 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735424 7f868bf5a700 10 op_submit_finish 3490 2014-06-09 16:22:11.735421 7f8689f56700 20 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.735430 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.735429 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.603( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.603( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.735448 7f8692f68700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.735452 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.735445 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.735457 7f868cf5c700 5 queue_transactions existing osr(0.f 0x2bd60e0)/0x2bd60e0 2014-06-09 16:22:11.735460 7f868cf5c700 2 waiting 51 > 50 ops || 5596819 > 104857600 2014-06-09 16:22:11.735453 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.552( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.735462 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.735464 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.735238 7f86a5e11700 10 _do_transaction on 0xc28ba40 2014-06-09 16:22:11.735466 7f868bf5a700 5 queue_transactions existing osr(1.603 0x384dc90)/0x384dc90 2014-06-09 16:22:11.735469 7f868bf5a700 2 waiting 51 > 50 ops || 5595770 > 104857600 2014-06-09 16:22:11.735464 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.735471 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.735473 7f8691765700 10 op_submit_start 3491 2014-06-09 16:22:11.735464 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.552( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.552( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.735474 7f8691765700 5 queue_transactions (trailing journal) 3491 2014-06-09 16:22:11.735475 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.735468 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.735476 7f8691765700 5 queue_op 0xc34a360 seq 3491 osr(3.2a9 0x3ed0b40) 1335 bytes (queue has 50 ops and 5594942 bytes) 2014-06-09 16:22:11.735479 7f8691765700 10 op_submit_finish 3491 2014-06-09 16:22:11.735479 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.735479 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.735480 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.735482 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.552( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 1.127806 7 0.011643 2014-06-09 16:22:11.735483 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.735483 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.735489 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735493 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.735495 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.552( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.735496 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735497 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.735501 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735507 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.735505 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.552( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.735507 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735509 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735514 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.552( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.735516 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735521 7f8670072700 5 -- op tracker -- , seq: 2806, time: 2014-06-09 16:22:11.729617, event: header_read, op: osd_op(client.4538.0:36647 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 2568192~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.735529 7f8691765700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.735523 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.552( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.735523 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:190 2014-06-09 16:22:11.735531 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.552( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.735538 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.735535 7f8670072700 5 -- op tracker -- , seq: 2806, time: 2014-06-09 16:22:11.729618, event: throttled, op: osd_op(client.4538.0:36647 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 2568192~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.735546 7f8692f68700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.735545 7f8670072700 5 -- op tracker -- , seq: 2806, time: 2014-06-09 16:22:11.730921, event: all_read, op: osd_op(client.4538.0:36647 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 2568192~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.735546 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:968 2014-06-09 16:22:11.735555 7f8670072700 5 -- op tracker -- , seq: 2806, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36647 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 2568192~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.735561 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.735562 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.735565 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.735562 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:181 2014-06-09 16:22:11.735575 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.735579 7f8689f56700 5 queue_transactions existing osr(2.552 0x2ed1950)/0x2ed1950 2014-06-09 16:22:11.735581 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.735581 7f8689f56700 2 waiting 51 > 50 ops || 5596818 > 104857600 2014-06-09 16:22:11.735576 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.735584 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735586 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6900 prio 63 cost 516096 latency 0.005969 osd_op(client.4538.0:36647 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 2568192~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.735585 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.735592 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735604 7f8691765700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.735599 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.735625 7f868d75d700 10 op_submit_start 3492 2014-06-09 16:22:11.735627 7f868d75d700 5 queue_transactions (trailing journal) 3492 2014-06-09 16:22:11.735628 7f868d75d700 5 queue_op 0x2fc8d7a0 seq 3492 osr(3.60 0x3ed02e0) 518380 bytes (queue has 50 ops and 5594942 bytes) 2014-06-09 16:22:11.735631 7f868d75d700 10 op_submit_finish 3492 2014-06-09 16:22:11.735637 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=212 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.735610 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.735636 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc2af740 243'642 rep_tid=9 committed?=0 applied?=0 lock=0 op=osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.735658 7f868d75d700 10 osd.26 243 dequeue_op 0x257c5600 finish 2014-06-09 16:22:11.735653 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.735653 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=212 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.735665 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.735666 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.735671 7f868ff62700 10 op_submit_start 3493 2014-06-09 16:22:11.735672 7f868ff62700 5 queue_transactions (trailing journal) 3493 2014-06-09 16:22:11.735674 7f868ff62700 5 queue_op 0x2fcae180 seq 3493 osr(2.4d 0x2ed05d0) 858 bytes (queue has 50 ops and 5594942 bytes) 2014-06-09 16:22:11.735677 7f868ff62700 10 op_submit_finish 3493 2014-06-09 16:22:11.735682 7f868f761700 10 op_submit_start 3494 2014-06-09 16:22:11.735685 7f868f761700 5 queue_transactions (trailing journal) 3494 2014-06-09 16:22:11.735679 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1111 2014-06-09 16:22:11.735686 7f868f761700 5 queue_op 0xbe7cf60 seq 3494 osr(2.4a7 0x2ed17d0) 2001 bytes (queue has 50 ops and 5594942 bytes) 2014-06-09 16:22:11.735690 7f868f761700 10 op_submit_finish 3494 2014-06-09 16:22:11.735677 7f8690763700 10 osd.26 243 dequeue_op 0x270bff00 prio 63 cost 1167360 latency 0.949732 osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.735683 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:3.1c7( v 213'647 (0'0,213'647] local-les=191 n=131 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.735694 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=191 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.735699 7f868ff62700 7 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.2 2014-06-09 16:22:11.735702 7f8690763700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.735697, event: reached_pg, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.735708 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=191 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.735715 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.735711 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.2 info 3.1c7( v 213'647 (0'0,213'647] local-les=191 n=131 ec=118 les/c 191/191 242/242/242), identical to ours 2014-06-09 16:22:11.735722 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=191 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.989561 7 0.000196 2014-06-09 16:22:11.735732 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.735728 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x270bff00 2014-06-09 16:22:11.735734 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=191 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.735735 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.735737 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:2.755( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.735746 7f868f761700 5 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=191 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.735750 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x30d99080 2014-06-09 16:22:11.735750 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.735760 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.735757 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=191 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.735757 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30d99080 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 rwstate(write n=3 w=0) oi: 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3(243'641 client.4570.0:34071 wrlock_by=unknown.0.0:0 dirty s 3026944 uv641) ssc: 0x30dd4c40 snapset: 0=[]:[]+head 2014-06-09 16:22:11.735764 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.735768 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'563 -> 213'563 2014-06-09 16:22:11.735769 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 @head oi=886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3(243'641 client.4570.0:34071 wrlock_by=unknown.0.0:0 dirty s 3026944 uv641) 2014-06-09 16:22:11.735776 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.735786 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.735787 7f8690763700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.735783, event: started, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.735792 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30718600 2014-06-09 16:22:11.735790 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.735802 7f8690763700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.735799, event: started, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.735806 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] ov 243'641 av 243'643 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.735805 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.735816 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 2014-06-09 16:22:11.735826 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.735834 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 3026944~1167360 2014-06-09 16:22:11.735845 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228a00 2014-06-09 16:22:11.735854 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.735852 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 snapset=0x30dd4c78 snapc=0=[] 2014-06-09 16:22:11.735858 7f868f761700 5 queue_transactions existing osr(3.1eb 0x3ed09c0)/0x3ed09c0 2014-06-09 16:22:11.735861 7f868f761700 2 waiting 51 > 50 ops || 5596371 > 104857600 2014-06-09 16:22:11.735859 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.735804 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.735867 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 0x30718600 op modify 2014-06-09 16:22:11.735877 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.735878 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:07.645738 2014-06-09 16:22:11.735881 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=203 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.50d( v 213'657 (0'0,213'657] local-les=243 n=126 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.735892 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.735893 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=203 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.50d( v 213'657 (0'0,213'657] local-les=243 n=126 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.735905 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.735907 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=203 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.237875 7 0.000181 2014-06-09 16:22:11.735912 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 10 on osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.735918 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=203 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.735925 7f8690763700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 10 o 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.735930 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=203 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.735940 7f8690763700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.735937, event: waiting for subops from 7, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.735941 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=203 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.735955 7f8690763700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.735951 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=243 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'657 -> 213'657 2014-06-09 16:22:11.735966 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'642 (0'0,243'642] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'642], crt=0'0) [243'643 (243'641) modify 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 by client.4570.0:34072 2014-06-09 10:20:07.645738] 2014-06-09 16:22:11.735987 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'643 (243'641) modify 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 by client.4570.0:34072 2014-06-09 10:20:07.645738 2014-06-09 16:22:11.736003 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=243 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.736005 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.736031 7f8690763700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'643, trimmed: 2014-06-09 16:22:11.736048 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.736049 7f8690763700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.736051 7f8690763700 2 waiting 51 > 50 ops || 6764586 > 104857600 2014-06-09 16:22:11.736053 7f868ff62700 5 queue_transactions existing osr(3.50d 0x3ed16c0)/0x3ed16c0 2014-06-09 16:22:11.736056 7f868ff62700 2 waiting 51 > 50 ops || 5596277 > 104857600 2014-06-09 16:22:11.736059 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.736064 7f868d75d700 10 osd.26 243 dequeue_op 0xdfa6000 prio 196 cost 0 latency 0.005637 osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 pg pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.736068 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.736084 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.736080 7f868d75d700 5 -- op tracker -- , seq: 2783, time: 2014-06-09 16:22:11.736077, event: reached_pg, op: osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.736086 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0xdfa6000 2014-06-09 16:22:11.736088 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.736099 7f868d75d700 5 -- op tracker -- , seq: 2783, time: 2014-06-09 16:22:11.736095, event: started, op: osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.736101 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.736103 7f868d75d700 7 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 8 op ack_type 4 from 9 2014-06-09 16:22:11.736110 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.736118 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.736119 7f868d75d700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.736114, event: sub_op_commit_rec, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.736130 7f868d75d700 10 osd.26 243 dequeue_op 0xdfa6000 finish 2014-06-09 16:22:11.736131 7f8691765700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.736135 7f868d75d700 5 -- op tracker -- , seq: 2783, time: 2014-06-09 16:22:11.736132, event: done, op: osd_sub_op_reply(client.4571.0:35949 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.736144 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.736146 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.736163 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15359b40 (20 active) 2014-06-09 16:22:11.736174 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.736170 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.736180 7f8691765700 20 osd.26 243 Sending info 3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) to shard 3 2014-06-09 16:22:11.736187 7f8691765700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.736189 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.736185 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.736195 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.736195 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.736203 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.736216 7f868d75d700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.736217 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358d10 (19 active) 2014-06-09 16:22:11.736223 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.736206 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.736228 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.736232 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:924 2014-06-09 16:22:11.736236 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.736246 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.736246 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.736250 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.736260 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.736261 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.691693 6 0.000464 2014-06-09 16:22:11.736264 7f869af78700 5 -- op tracker -- , seq: 2805, time: 2014-06-09 16:22:11.736258, event: done, op: pg_info(1 pgs e243:2.740) 2014-06-09 16:22:11.736272 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.736277 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.736280 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.736286 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.736282 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.736294 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.736295 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.736297 7f869af78700 20 osd.26 243 _dispatch 0x2eee76c0 pg_info(1 pgs e243:3.6b4) v4 2014-06-09 16:22:11.736290 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:2.6dd( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.736293 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:11.736302 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.736301 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.736303 7f869af78700 5 -- op tracker -- , seq: 2807, time: 2014-06-09 16:22:11.430444, event: header_read, op: pg_info(1 pgs e243:3.6b4) 2014-06-09 16:22:11.736310 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.736309 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.736313 7f869af78700 5 -- op tracker -- , seq: 2807, time: 2014-06-09 16:22:11.430444, event: throttled, op: pg_info(1 pgs e243:3.6b4) 2014-06-09 16:22:11.736317 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.736314 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:3.224( v 214'739 (0'0,214'739] local-les=208 n=135 ec=118 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.736320 7f869af78700 5 -- op tracker -- , seq: 2807, time: 2014-06-09 16:22:11.430453, event: all_read, op: pg_info(1 pgs e243:3.6b4) 2014-06-09 16:22:11.736326 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.736336 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.736328 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.736337 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.736346 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.736349 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 6 notify: (query_epoch:243, epoch_sent:243, info:0.230( empty local-les=208 n=0 ec=1 les/c 208/208 242/242/242)) 2014-06-09 16:22:11.736358 7f86a5610700 10 _do_op 0x309a2240 seq 3431 r = 0, finisher 0x3050a3a0 0 2014-06-09 16:22:11.736363 7f86a5610700 10 _finish_op 0x309a2240 seq 3431 osr(2.1c6 0x2ed0870)/0x2ed0870 2014-06-09 16:22:11.736361 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 6, already have info from that osd, ignoring 2014-06-09 16:22:11.736371 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.736376 7f868ef60700 10 op_submit_start 3495 2014-06-09 16:22:11.736377 7f868ef60700 5 queue_transactions (trailing journal) 3495 2014-06-09 16:22:11.736378 7f868cf5c700 2 waiting 51 > 50 ops || 5596776 > 104857600 2014-06-09 16:22:11.736378 7f868ef60700 5 queue_op 0xc2d5b00 seq 3495 osr(1.4c9 0x384d8b0) 1954 bytes (queue has 50 ops and 5594899 bytes) 2014-06-09 16:22:11.736386 7f868bf5a700 2 waiting 51 > 50 ops || 5595727 > 104857600 2014-06-09 16:22:11.736384 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89ea20 (1 active) 2014-06-09 16:22:11.736391 7f8689f56700 2 waiting 51 > 50 ops || 5596775 > 104857600 2014-06-09 16:22:11.736393 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62ec60 (2 active) 2014-06-09 16:22:11.736123 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=212 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.876581 7 0.000183 2014-06-09 16:22:11.736398 7f868ef60700 10 op_submit_finish 3495 2014-06-09 16:22:11.736398 7f868f761700 2 waiting 51 > 50 ops || 5596328 > 104857600 2014-06-09 16:22:11.736404 7f8690763700 2 waiting 51 > 50 ops || 6764543 > 104857600 2014-06-09 16:22:11.736399 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=212 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.736402 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.736418 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=212 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.736418 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.736431 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.736428 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=212 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.736429 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.736438 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.736443 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'600 -> 213'600 2014-06-09 16:22:11.736451 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3446.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.736454 7f868ff62700 2 waiting 51 > 50 ops || 5596234 > 104857600 2014-06-09 16:22:11.736456 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.736460 7f8689755700 2 waiting 51 > 50 ops || 5595725 > 104857600 2014-06-09 16:22:11.736464 7f86a5610700 5 _do_op 0x309a2300 seq 3447 osr(3.9f 0x3ed0600)/0x3ed0600 start 2014-06-09 16:22:11.736467 7f86a5610700 10 _do_transaction on 0xc2ac0c0 2014-06-09 16:22:11.736474 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.736476 7f86a5e11700 10 _do_op 0x2fc8df20 seq 3446 r = 0, finisher 0x31456820 0 2014-06-09 16:22:11.736480 7f86a5e11700 10 _finish_op 0x2fc8df20 seq 3446 osr(1.233 0x384cc30)/0x384cc30 2014-06-09 16:22:11.736477 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.736490 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1440 (1 active) 2014-06-09 16:22:11.736493 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad440 (2 active) 2014-06-09 16:22:11.736493 7f868bf5a700 10 op_submit_start 3496 2014-06-09 16:22:11.736495 7f8689f56700 2 waiting 51 > 50 ops || 5595814 > 104857600 2014-06-09 16:22:11.736495 7f868bf5a700 5 queue_transactions (trailing journal) 3496 2014-06-09 16:22:11.736495 7f86a5e11700 5 _do_op 0x30d96540 seq 3448 osr(2.5a4 0x2ed1810)/0x2ed1810 start 2014-06-09 16:22:11.736497 7f868bf5a700 5 queue_op 0xc64a240 seq 3496 osr(1.603 0x384dc90) 828 bytes (queue has 50 ops and 5593938 bytes) 2014-06-09 16:22:11.736501 7f868bf5a700 10 op_submit_finish 3496 2014-06-09 16:22:11.736500 7f868f761700 2 waiting 51 > 50 ops || 5595367 > 104857600 2014-06-09 16:22:11.736506 7f86a5e11700 10 _do_transaction on 0xc1495c0 2014-06-09 16:22:11.736506 7f8690763700 2 waiting 51 > 50 ops || 6763582 > 104857600 2014-06-09 16:22:11.736512 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.736504 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.736520 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.736517 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.736527 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.736534 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.736550 7f868ff62700 2 waiting 51 > 50 ops || 5595273 > 104857600 2014-06-09 16:22:11.736555 7f8689755700 2 waiting 51 > 50 ops || 5594764 > 104857600 2014-06-09 16:22:11.736561 7f868cf5c700 2 waiting 51 > 50 ops || 5595815 > 104857600 2014-06-09 16:22:11.736327 7f869af78700 5 -- op tracker -- , seq: 2807, time: 2014-06-09 16:22:11.736293, event: dispatched, op: pg_info(1 pgs e243:3.6b4) 2014-06-09 16:22:11.736570 7f869af78700 5 -- op tracker -- , seq: 2807, time: 2014-06-09 16:22:11.736567, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6b4) 2014-06-09 16:22:11.736575 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6b4) v4 from osd.4 2014-06-09 16:22:11.736578 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee76c0 2014-06-09 16:22:11.736584 7f869af78700 5 -- op tracker -- , seq: 2807, time: 2014-06-09 16:22:11.736582, event: started, op: pg_info(1 pgs e243:3.6b4) 2014-06-09 16:22:11.736601 7f869af78700 5 -- op tracker -- , seq: 2807, time: 2014-06-09 16:22:11.736598, event: done, op: pg_info(1 pgs e243:3.6b4) 2014-06-09 16:22:11.736611 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.736614 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.736627 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.736628 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.736630 7f869af78700 20 osd.26 243 _dispatch 0x30cae380 pg_info(1 pgs e243:3.2fd) v4 2014-06-09 16:22:11.736635 7f869af78700 5 -- op tracker -- , seq: 2808, time: 2014-06-09 16:22:11.606157, event: header_read, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.736646 7f869af78700 5 -- op tracker -- , seq: 2808, time: 2014-06-09 16:22:11.606158, event: throttled, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.736656 7f869af78700 5 -- op tracker -- , seq: 2808, time: 2014-06-09 16:22:11.606170, event: all_read, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.736665 7f869af78700 5 -- op tracker -- , seq: 2808, time: 2014-06-09 16:22:11.736626, event: dispatched, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.736673 7f869af78700 5 -- op tracker -- , seq: 2808, time: 2014-06-09 16:22:11.736671, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.736680 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2fd) v4 from osd.7 2014-06-09 16:22:11.736683 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cae380 2014-06-09 16:22:11.736690 7f8670072700 5 -- op tracker -- , seq: 2809, time: 2014-06-09 16:22:11.735624, event: header_read, op: osd_op(client.4538.0:36648 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3084288~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.736704 7f8670072700 5 -- op tracker -- , seq: 2809, time: 2014-06-09 16:22:11.735626, event: throttled, op: osd_op(client.4538.0:36648 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3084288~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.736719 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.736715 7f8670072700 5 -- op tracker -- , seq: 2809, time: 2014-06-09 16:22:11.736621, event: all_read, op: osd_op(client.4538.0:36648 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3084288~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.736726 7f8670072700 5 -- op tracker -- , seq: 2809, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36648 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3084288~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.736730 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.736739 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3447.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.736743 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.736746 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6800 prio 63 cost 516096 latency 0.001121 osd_op(client.4538.0:36648 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3084288~516096] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.736774 7f86a5610700 10 _do_op 0x309a2300 seq 3447 r = 0, finisher 0xbfcea60 0 2014-06-09 16:22:11.736778 7f86a5610700 10 _finish_op 0x309a2300 seq 3447 osr(3.9f 0x3ed0600)/0x3ed0600 2014-06-09 16:22:11.736785 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62ec60 (1 active) 2014-06-09 16:22:11.736787 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e000 (2 active) 2014-06-09 16:22:11.736788 7f8689f56700 10 op_submit_start 3497 2014-06-09 16:22:11.736790 7f8689f56700 5 queue_transactions (trailing journal) 3497 2014-06-09 16:22:11.736789 7f86a5610700 5 _do_op 0x309a2360 seq 3449 osr(3.ad 0x3ed05e0)/0x3ed05e0 start 2014-06-09 16:22:11.736792 7f86a5610700 10 _do_transaction on 0xc05ae40 2014-06-09 16:22:11.736790 7f868f761700 2 waiting 51 > 50 ops || 5595911 > 104857600 2014-06-09 16:22:11.736791 7f8689f56700 5 queue_op 0x308ca300 seq 3497 osr(2.552 0x2ed1950) 1876 bytes (queue has 50 ops and 5594482 bytes) 2014-06-09 16:22:11.736793 7f8689f56700 10 op_submit_finish 3497 2014-06-09 16:22:11.736797 7f8690763700 2 waiting 51 > 50 ops || 6764126 > 104857600 2014-06-09 16:22:11.736797 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.736803 7f868ff62700 2 waiting 51 > 50 ops || 5595817 > 104857600 2014-06-09 16:22:11.736796 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.736807 7f8689755700 2 waiting 51 > 50 ops || 5595308 > 104857600 2014-06-09 16:22:11.736812 7f868cf5c700 2 waiting 51 > 50 ops || 5596359 > 104857600 2014-06-09 16:22:11.736813 7f868d75d700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.736809 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.736814 7f869af78700 5 -- op tracker -- , seq: 2808, time: 2014-06-09 16:22:11.736686, event: started, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.736817 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.736825 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.736831 7f869af78700 5 -- op tracker -- , seq: 2808, time: 2014-06-09 16:22:11.736829, event: done, op: pg_info(1 pgs e243:3.2fd) 2014-06-09 16:22:11.736826 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:11.736834 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.736838 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.736833 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.736840 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.736841 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.736854 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.736855 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.736853 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3448.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.736856 7f869af78700 20 osd.26 243 _dispatch 0x30cf1880 pg_info(1 pgs e243:3.2c9) v4 2014-06-09 16:22:11.736858 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.736853 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.736861 7f869af78700 5 -- op tracker -- , seq: 2810, time: 2014-06-09 16:22:11.577787, event: header_read, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.736862 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.736868 7f869af78700 5 -- op tracker -- , seq: 2810, time: 2014-06-09 16:22:11.577787, event: throttled, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.736869 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.736873 7f869af78700 5 -- op tracker -- , seq: 2810, time: 2014-06-09 16:22:11.577800, event: all_read, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.736878 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.736879 7f869af78700 5 -- op tracker -- , seq: 2810, time: 2014-06-09 16:22:11.736854, event: dispatched, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.736886 7f86a5e11700 10 _do_op 0x30d96540 seq 3448 r = 0, finisher 0x314ad080 0 2014-06-09 16:22:11.736885 7f869af78700 5 -- op tracker -- , seq: 2810, time: 2014-06-09 16:22:11.736883, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.736889 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2c9) v4 from osd.3 2014-06-09 16:22:11.736889 7f86a5e11700 10 _finish_op 0x30d96540 seq 3448 osr(2.5a4 0x2ed1810)/0x2ed1810 2014-06-09 16:22:11.736891 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf1880 2014-06-09 16:22:11.736883 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.736896 7f869af78700 5 -- op tracker -- , seq: 2810, time: 2014-06-09 16:22:11.736894, event: started, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.736900 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad440 (1 active) 2014-06-09 16:22:11.736896 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.736903 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3385a0 (2 active) 2014-06-09 16:22:11.736905 7f8690763700 10 op_submit_start 3498 2014-06-09 16:22:11.736905 7f86a5e11700 5 _do_op 0x30061920 seq 3450 osr(3.251 0x3ed0c20)/0x3ed0c20 start 2014-06-09 16:22:11.736906 7f868ff62700 2 waiting 51 > 50 ops || 6763558 > 104857600 2014-06-09 16:22:11.736907 7f8690763700 5 queue_transactions (trailing journal) 3498 2014-06-09 16:22:11.736908 7f86a5e11700 10 _do_transaction on 0xc2ada40 2014-06-09 16:22:11.736903 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.736909 7f8690763700 5 queue_op 0x2fcaec00 seq 3498 osr(3.60 0x3ed02e0) 1169644 bytes (queue has 50 ops and 6762223 bytes) 2014-06-09 16:22:11.736912 7f8690763700 10 op_submit_finish 3498 2014-06-09 16:22:11.736911 7f8689755700 2 waiting 51 > 50 ops || 6763049 > 104857600 2014-06-09 16:22:11.736915 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.736916 7f868cf5c700 2 waiting 51 > 50 ops || 6764100 > 104857600 2014-06-09 16:22:11.736912 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.736921 7f868f761700 2 waiting 51 > 50 ops || 6763652 > 104857600 2014-06-09 16:22:11.736920 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.736915 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc232840 243'643 rep_tid=10 committed?=0 applied?=0 lock=0 op=osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.736929 7f869af78700 5 -- op tracker -- , seq: 2810, time: 2014-06-09 16:22:11.736927, event: done, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.736927 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.736934 7f8690763700 10 osd.26 243 dequeue_op 0x270bff00 finish 2014-06-09 16:22:11.736937 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.736939 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.736941 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.736940 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.736952 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.736951 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.736953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.736948 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.736955 7f869af78700 20 osd.26 243 _dispatch 0x1405c1c0 pg_info(1 pgs e243:2.69d) v4 2014-06-09 16:22:11.736959 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8fe40 (20 active) 2014-06-09 16:22:11.736956 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.736959 7f869af78700 5 -- op tracker -- , seq: 2811, time: 2014-06-09 16:22:11.577988, event: header_read, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.736964 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.736965 7f869af78700 5 -- op tracker -- , seq: 2811, time: 2014-06-09 16:22:11.577989, event: throttled, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.736969 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3450.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.736973 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.736963 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 2 2014-06-09 16:22:11.736976 7f868ef60700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.736968 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.5e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.736974 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.736978 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.25a( v 213'576 (0'0,213'576] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) log((0'0,213'576], crt=0'0) missing(0) 2014-06-09 16:22:11.736990 7f8690763700 10 proc_replica_log for osd.2: 3.25a( v 213'576 (0'0,213'576] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) log((0'0,213'576], crt=0'0) missing(0) 2014-06-09 16:22:11.736987 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1004 2014-06-09 16:22:11.736999 7f86a5e11700 10 _do_op 0x30061920 seq 3450 r = 0, finisher 0x17001540 0 2014-06-09 16:22:11.737002 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.736995 7f8690763700 20 merge_log cut point (usually last shared) is 213'576 (213'575) modify 8894225a/rbd_data.11776b8b4567.000000000000171d/head//3 by client.4567.0:34302 2014-06-09 10:17:58.128268 2014-06-09 16:22:11.737002 7f86a5e11700 10 _finish_op 0x30061920 seq 3450 osr(3.251 0x3ed0c20)/0x3ed0c20 2014-06-09 16:22:11.737006 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3385a0 (1 active) 2014-06-09 16:22:11.737008 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b440 (2 active) 2014-06-09 16:22:11.736988 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.737010 7f86a5e11700 5 _do_op 0x309a23c0 seq 3451 osr(1.465 0x384d650)/0x384d650 start 2014-06-09 16:22:11.737012 7f86a5e11700 10 _do_transaction on 0xc3f1ec0 2014-06-09 16:22:11.737005 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.737011 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.737018 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.737018 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.737020 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.737030 7f8692f68700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.737031 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.737040 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.737044 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:904 2014-06-09 16:22:11.737052 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.737053 7f868ef60700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.737057 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.737061 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.737069 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3451.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.737073 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.737068 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:973 2014-06-09 16:22:11.737084 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.737071 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.737093 7f86a5e11700 10 _do_op 0x309a23c0 seq 3451 r = 0, finisher 0xbd69cc0 0 2014-06-09 16:22:11.737088 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.737097 7f86a5e11700 10 _finish_op 0x309a23c0 seq 3451 osr(1.465 0x384d650)/0x384d650 2014-06-09 16:22:11.737099 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b440 (1 active) 2014-06-09 16:22:11.737094 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.285092 7 0.000163 2014-06-09 16:22:11.737102 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4add40 (2 active) 2014-06-09 16:22:11.737104 7f86a5e11700 5 _do_op 0x30061980 seq 3452 osr(2.719 0x3ed01d0)/0x3ed01d0 start 2014-06-09 16:22:11.737107 7f86a5e11700 10 _do_transaction on 0x97bb980 2014-06-09 16:22:11.737101 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.737105 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.737113 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.737110 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.737115 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.737119 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.737124 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=191 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.737130 7f868ef60700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.737133 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'611 -> 213'611 2014-06-09 16:22:11.737146 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.737145 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.737159 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.737163 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3452.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.737167 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.737163 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.737177 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.737178 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.737190 7f86a5e11700 10 _do_op 0x30061980 seq 3452 r = 0, finisher 0xc6e9080 0 2014-06-09 16:22:11.737187 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.737194 7f86a5e11700 10 _finish_op 0x30061980 seq 3452 osr(2.719 0x3ed01d0)/0x3ed01d0 2014-06-09 16:22:11.737197 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4add40 (1 active) 2014-06-09 16:22:11.737199 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6539e0 (2 active) 2014-06-09 16:22:11.737201 7f86a5e11700 5 _do_op 0xc2eb9e0 seq 3453 osr(3.55d 0x3ed1940)/0x3ed1940 start 2014-06-09 16:22:11.737196 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.737204 7f86a5e11700 10 _do_transaction on 0xc014e40 2014-06-09 16:22:11.737209 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.737210 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.737216 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.737225 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1048 2014-06-09 16:22:11.737240 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.737243 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.737247 7f868ff62700 10 op_submit_start 3499 2014-06-09 16:22:11.737249 7f868ff62700 5 queue_transactions (trailing journal) 3499 2014-06-09 16:22:11.737251 7f868ff62700 5 queue_op 0x2c06f1a0 seq 3499 osr(3.50d 0x3ed16c0) 1335 bytes (queue has 48 ops and 6758245 bytes) 2014-06-09 16:22:11.737253 7f868ff62700 10 op_submit_finish 3499 2014-06-09 16:22:11.737261 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3453.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.737264 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.737256 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.737270 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.4 3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) 2014-06-09 16:22:11.737281 7f86a5e11700 10 _do_op 0xc2eb9e0 seq 3453 r = 0, finisher 0x30c54ac0 0 2014-06-09 16:22:11.737284 7f86a5e11700 10 _finish_op 0xc2eb9e0 seq 3453 osr(3.55d 0x3ed1940)/0x3ed1940 2014-06-09 16:22:11.737287 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6539e0 (1 active) 2014-06-09 16:22:11.737289 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0120 (2 active) 2014-06-09 16:22:11.737291 7f86a5e11700 5 _do_op 0x24104060 seq 3454 osr(0.57b 0x2bd77b0)/0x2bd77b0 start 2014-06-09 16:22:11.737283 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.101786 7 0.000205 2014-06-09 16:22:11.737294 7f86a5e11700 10 _do_transaction on 0xc28b500 2014-06-09 16:22:11.737299 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.737295 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.737308 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.737318 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=203 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.737333 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.737329 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'643 -> 213'643 2014-06-09 16:22:11.737349 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3454.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.737354 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.737390 7f86a5e11700 10 _do_op 0x24104060 seq 3454 r = 0, finisher 0x31413320 0 2014-06-09 16:22:11.737395 7f86a5e11700 10 _finish_op 0x24104060 seq 3454 osr(0.57b 0x2bd77b0)/0x2bd77b0 2014-06-09 16:22:11.737388 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.737398 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0120 (1 active) 2014-06-09 16:22:11.737400 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0b40 (2 active) 2014-06-09 16:22:11.737402 7f86a5e11700 5 _do_op 0x30d96a80 seq 3455 osr(3.437 0x3ed11c0)/0x3ed11c0 start 2014-06-09 16:22:11.737405 7f86a5e11700 10 _do_transaction on 0xbf538c0 2014-06-09 16:22:11.737393 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 1 2014-06-09 16:22:11.737411 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.737431 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.737439 7f8689755700 10 op_submit_start 3500 2014-06-09 16:22:11.737441 7f8689755700 5 queue_transactions (trailing journal) 3500 2014-06-09 16:22:11.737442 7f8689755700 5 queue_op 0x16fd5140 seq 3500 osr(1.30 0x384c280) 826 bytes (queue has 47 ops and 6756009 bytes) 2014-06-09 16:22:11.737434 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.25a( v 213'576 (0'0,213'576] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.737443 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.737445 7f8689755700 10 op_submit_finish 3500 2014-06-09 16:22:11.737447 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.737449 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.737455 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.731222 7 0.000327 2014-06-09 16:22:11.737460 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3455.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.737464 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.737464 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.737476 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.737465 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.737484 7f868cf5c700 10 op_submit_start 3501 2014-06-09 16:22:11.737486 7f868cf5c700 5 queue_transactions (trailing journal) 3501 2014-06-09 16:22:11.737482 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.440215 0 0.000000 2014-06-09 16:22:11.737487 7f868cf5c700 5 queue_op 0x2fcafec0 seq 3501 osr(0.f 0x2bd60e0) 1877 bytes (queue has 48 ops and 6757886 bytes) 2014-06-09 16:22:11.737490 7f868cf5c700 10 op_submit_finish 3501 2014-06-09 16:22:11.737493 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.737493 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.737498 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.737509 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.737513 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.737505 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.737510 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.737517 7f868cf5c700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.737522 7f868cf5c700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.737520 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.737525 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.030469 7 0.000203 2014-06-09 16:22:11.737527 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'581 -> 214'581 2014-06-09 16:22:11.737536 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.737550 7f8692f68700 5 queue_transactions existing osr(3.366 0x3ed1080)/0x3ed1080 2014-06-09 16:22:11.737547 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.737557 7f8692f68700 10 op_submit_start 3502 2014-06-09 16:22:11.737558 7f8692f68700 5 queue_transactions (trailing journal) 3502 2014-06-09 16:22:11.737559 7f8692f68700 5 queue_op 0x2e21d40 seq 3502 osr(3.366 0x3ed1080) 1429 bytes (queue has 49 ops and 6759315 bytes) 2014-06-09 16:22:11.737562 7f8692f68700 10 op_submit_finish 3502 2014-06-09 16:22:11.737556 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.737563 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41ef0 (19 active) 2014-06-09 16:22:11.737570 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.737565 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.737565 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.779( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.737573 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.737568 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.25a( v 213'576 (0'0,213'576] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.737579 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.737588 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.737575 7f868ef60700 10 osd.26 243 dequeue_op 0x10c55d00 prio 63 cost 311296 latency 1.598702 osd_op(client.4573.0:33491 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~311296] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.737592 7f8692f68700 20 osd.26 243 Sending info 3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) to shard 7 2014-06-09 16:22:11.737598 7f8692f68700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.737591 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'576,214'581], crt=0'0) 2014-06-09 16:22:11.737602 7f8690763700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.737619 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.737624 7f868cf5c700 5 queue_transactions existing osr(1.5f8 0x384dcf0)/0x384dcf0 2014-06-09 16:22:11.737627 7f868cf5c700 10 op_submit_start 3503 2014-06-09 16:22:11.737616 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.25a( v 214'581 lc 213'576 (0'0,214'581] local-les=191 n=111 ec=118 les/c 191/191 242/242/242) missing missing(2) 2014-06-09 16:22:11.737628 7f868cf5c700 5 queue_transactions (trailing journal) 3503 2014-06-09 16:22:11.737629 7f868cf5c700 5 queue_op 0x16fd4f60 seq 3503 osr(1.5f8 0x384dcf0) 1997 bytes (queue has 50 ops and 6761312 bytes) 2014-06-09 16:22:11.737632 7f868cf5c700 10 op_submit_finish 3503 2014-06-09 16:22:11.737630 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 2 missing 2014-06-09 16:22:11.737640 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (19 active) 2014-06-09 16:22:11.737636 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.737641 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing f421125a/rbd_data.11636b8b4567.0000000000001802/head//3 214'579 is on osd.26 2014-06-09 16:22:11.737656 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (20 active) 2014-06-09 16:22:11.737649 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.737652 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 904cc25a/rbd_data.115c6b8b4567.00000000000017f4/head//3 214'581 is on osd.26 2014-06-09 16:22:11.737658 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.737659 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.737661 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {f421125a/rbd_data.11636b8b4567.0000000000001802/head//3=214'579,904cc25a/rbd_data.115c6b8b4567.00000000000017f4/head//3=214'581} 2014-06-09 16:22:11.737667 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.737669 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.737671 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.737676 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.737678 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.737681 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing f421125a/rbd_data.11636b8b4567.0000000000001802/head//3 214'579 also missing on osd.2 2014-06-09 16:22:11.737686 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.737685 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.737698 7f8692f68700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.737691 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 904cc25a/rbd_data.115c6b8b4567.00000000000017f4/head//3 214'581 also missing on osd.2 2014-06-09 16:22:11.737700 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:196 2014-06-09 16:22:11.737700 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {f421125a/rbd_data.11636b8b4567.0000000000001802/head//3=214'579,904cc25a/rbd_data.115c6b8b4567.00000000000017f4/head//3=214'581} 2014-06-09 16:22:11.737710 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.737711 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.737720 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.737730 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.737739 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.737740 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.737753 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.737749 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.737757 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.737768 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.737777 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.737784 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.737796 7f8692f68700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.737718 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.737801 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.737806 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.737821 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.737840 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.737830 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16059200,0x16059100,0x16059000,0x16058f00,0x1c768600,0x1c768500,0x1c54bb00,0x1c54ba00,0x1c54b900,0x1c54b800,0x1c54b700,0x1c54b600,0x1c768400 2014-06-09 16:22:11.737833 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1235 2014-06-09 16:22:11.737844 7f8690763700 5 queue_transactions existing osr(3.25a 0x3ed0c00)/0x3ed0c00 2014-06-09 16:22:11.737846 7f8690763700 2 waiting 51 > 50 ops || 6762365 > 104857600 2014-06-09 16:22:11.737855 7f868ff62700 5 queue_transactions existing osr(3.516 0x3ed16d0)/0x3ed16d0 2014-06-09 16:22:11.737858 7f868ff62700 2 waiting 51 > 50 ops || 6762647 > 104857600 2014-06-09 16:22:11.737864 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.737867 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.737878 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.737877 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.737888 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.737889 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.737895 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.737903 7f8692f68700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.737898 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.737913 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.737913 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:975 2014-06-09 16:22:11.737927 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.737924 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.737933 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.737934 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.737943 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.737946 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.737952 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.737948 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.663( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.737956 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.737964 7f8692f68700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.737980 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.737974 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.737985 7f868cf5c700 5 queue_transactions existing osr(0.663 0x2bd7f30)/0x2bd7f30 2014-06-09 16:22:11.737987 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.737987 7f868cf5c700 2 waiting 51 > 50 ops || 6762140 > 104857600 2014-06-09 16:22:11.737991 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.737997 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.194( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.738003 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.738009 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.738014 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.738011 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.194( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.738020 7f8692f68700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.738022 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.738026 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.738038 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.738047 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.738050 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.738051 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.738064 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.738063 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.738072 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.738068 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 2 2014-06-09 16:22:11.738088 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e770 (19 active) 2014-06-09 16:22:11.738096 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.738092 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.15c( v 213'624 (0'0,213'624] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) log((0'0,213'624], crt=0'0) missing(0) 2014-06-09 16:22:11.738104 7f8692f68700 10 proc_replica_log for osd.2: 3.15c( v 213'624 (0'0,213'624] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) log((0'0,213'624], crt=0'0) missing(0) 2014-06-09 16:22:11.738100 7f8689755700 10 osd.26 243 dequeue_op 0x2f097e00 prio 63 cost 1032192 latency 0.027505 osd_op(client.4538.0:36625 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1032192] 3.923772b RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.738110 7f8692f68700 20 merge_log cut point (usually last shared) is 213'624 (213'623) modify 8af7715c/rbd_data.117a6b8b4567.000000000000167f/head//3 by client.4570.0:33450 2014-06-09 10:17:36.095447 2014-06-09 16:22:11.738126 7f868bf5a700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.738143 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:146 2014-06-09 16:22:11.738159 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.738164 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.738172 7f868d75d700 5 queue_transactions existing osr(1.300 0x384d050)/0x384d050 2014-06-09 16:22:11.738175 7f868d75d700 2 waiting 51 > 50 ops || 6762173 > 104857600 2014-06-09 16:22:11.738175 7f8689f56700 5 queue_transactions existing osr(1.5e 0x384c5d0)/0x384c5d0 2014-06-09 16:22:11.738178 7f8689f56700 2 waiting 51 > 50 ops || 6762138 > 104857600 2014-06-09 16:22:11.738176 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.8c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.738185 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.738189 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.738202 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.738211 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.738219 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.738231 7f868bf5a700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.738246 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:175 2014-06-09 16:22:11.738261 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.738265 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.738277 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.738285 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.738294 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.738302 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.738310 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.738319 7f86a5e11700 10 _do_op 0x30d96a80 seq 3455 r = 0, finisher 0xcdb3de0 0 2014-06-09 16:22:11.738324 7f86a5e11700 10 _finish_op 0x30d96a80 seq 3455 osr(3.437 0x3ed11c0)/0x3ed11c0 2014-06-09 16:22:11.738323 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:190 2014-06-09 16:22:11.738330 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0b40 (1 active) 2014-06-09 16:22:11.738333 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca48fc0 (2 active) 2014-06-09 16:22:11.738335 7f86a5e11700 5 _do_op 0x2fcaf680 seq 3456 osr(1.10d 0x384c930)/0x384c930 start 2014-06-09 16:22:11.738335 7f8690763700 10 op_submit_start 3504 2014-06-09 16:22:11.738338 7f8690763700 5 queue_transactions (trailing journal) 3504 2014-06-09 16:22:11.738338 7f86a5e11700 10 _do_transaction on 0x3562000 2014-06-09 16:22:11.738333 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.738340 7f8690763700 5 queue_op 0x2fcaeae0 seq 3504 osr(3.25a 0x3ed0c00) 1053 bytes (queue has 50 ops and 6761030 bytes) 2014-06-09 16:22:11.738343 7f8690763700 10 op_submit_finish 3504 2014-06-09 16:22:11.738345 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.738341 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.738351 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.738350 7f868ff62700 2 waiting 51 > 50 ops || 6762365 > 104857600 2014-06-09 16:22:11.738346 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 2 2014-06-09 16:22:11.738355 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.738361 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.51e( v 213'674 (0'0,213'674] local-les=191 n=127 ec=118 les/c 191/191 242/242/242) log((0'0,213'674], crt=0'0) missing(0) 2014-06-09 16:22:11.738368 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.738374 7f8690763700 10 proc_replica_log for osd.2: 3.51e( v 213'674 (0'0,213'674] local-les=191 n=127 ec=118 les/c 191/191 242/242/242) log((0'0,213'674], crt=0'0) missing(0) 2014-06-09 16:22:11.738378 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.738379 7f8690763700 20 merge_log cut point (usually last shared) is 213'674 (213'673) modify 37eaf51e/rbd_data.11626b8b4567.0000000000001854/head//3 by client.4546.0:35399 2014-06-09 10:17:51.794691 2014-06-09 16:22:11.738387 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.738382 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.738393 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.9 2.1e0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.738405 7f868bf5a700 10 merge_log log((0'0,0'0], crt=0'0) from osd.9 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.738405 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3456.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.738410 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.738409 7f868bf5a700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.738413 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 0.828173 7 0.000157 2014-06-09 16:22:11.738424 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.738428 7f86a5e11700 10 _do_op 0x2fcaf680 seq 3456 r = 0, finisher 0x31457ea0 0 2014-06-09 16:22:11.738433 7f86a5e11700 10 _finish_op 0x2fcaf680 seq 3456 osr(1.10d 0x384c930)/0x384c930 2014-06-09 16:22:11.738435 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.738445 7f868d75d700 10 op_submit_start 3505 2014-06-09 16:22:11.738446 7f868d75d700 5 queue_transactions (trailing journal) 3505 2014-06-09 16:22:11.738446 7f8689f56700 2 waiting 51 > 50 ops || 6760834 > 104857600 2014-06-09 16:22:11.738448 7f868d75d700 5 queue_op 0x2fc8c840 seq 3505 osr(1.300 0x384d050) 861 bytes (queue has 50 ops and 6760008 bytes) 2014-06-09 16:22:11.738445 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.738454 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca48fc0 (1 active) 2014-06-09 16:22:11.738453 7f868ff62700 2 waiting 51 > 50 ops || 6761343 > 104857600 2014-06-09 16:22:11.738456 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.738461 7f868d75d700 10 op_submit_finish 3505 2014-06-09 16:22:11.738454 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.738463 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.738465 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.738480 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.628142 7 0.000339 2014-06-09 16:22:11.738493 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.738506 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.738512 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.738517 7f868bf5a700 5 queue_transactions existing osr(2.1e0 0x2ed0850)/0x2ed0850 2014-06-09 16:22:11.738519 7f868bf5a700 2 waiting 51 > 50 ops || 6761962 > 104857600 2014-06-09 16:22:11.738516 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:11.738525 7f868f761700 2 waiting 51 > 50 ops || 6761437 > 104857600 2014-06-09 16:22:11.738529 7f868cf5c700 2 waiting 51 > 50 ops || 6760836 > 104857600 2014-06-09 16:22:11.738527 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.738536 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.738543 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.738551 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:11.738563 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:11.738573 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.738572 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.738586 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.738594 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3449.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.738598 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.738602 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 2014-06-09 16:22:11.738627 7f86a5610700 10 _do_op 0x309a2360 seq 3449 r = 0, finisher 0xc5a75a0 0 2014-06-09 16:22:11.738631 7f86a5610700 10 _finish_op 0x309a2360 seq 3449 osr(3.ad 0x3ed05e0)/0x3ed05e0 2014-06-09 16:22:11.738637 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.738637 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e000 (1 active) 2014-06-09 16:22:11.738640 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca498c0 (2 active) 2014-06-09 16:22:11.738640 7f8689f56700 10 op_submit_start 3506 2014-06-09 16:22:11.738642 7f8689f56700 5 queue_transactions (trailing journal) 3506 2014-06-09 16:22:11.738642 7f868d75d700 5 queue_transactions existing osr(1.99 0x384c7d0)/0x384c7d0 2014-06-09 16:22:11.738642 7f86a5610700 5 _do_op 0x30d964e0 seq 3458 osr(2.40a 0x2ed1280)/0x2ed1280 start 2014-06-09 16:22:11.738645 7f86a5610700 10 _do_transaction on 0xc20f800 2014-06-09 16:22:11.738643 7f8689f56700 5 queue_op 0x2ea536e0 seq 3506 osr(1.5e 0x384c5d0) 826 bytes (queue has 50 ops and 6759502 bytes) 2014-06-09 16:22:11.738645 7f868d75d700 2 waiting 51 > 50 ops || 6760360 > 104857600 2014-06-09 16:22:11.738646 7f8689f56700 10 op_submit_finish 3506 2014-06-09 16:22:11.738649 7f86a5e11700 5 _do_op 0x288f91a0 seq 3457 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.738651 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.738653 7f86a5e11700 10 _do_transaction on 0xbf7ccc0 2014-06-09 16:22:11.738658 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.738649 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:11.738671 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.738687 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.738674 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.15c( v 213'624 (0'0,213'624] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.738682 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.738690 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.738692 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.738703 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.738703 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3458.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.738707 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.738704 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.718811 7 0.000309 2014-06-09 16:22:11.738715 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.738718 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1103 2014-06-09 16:22:11.738724 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.198057 0 0.000000 2014-06-09 16:22:11.738735 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.738737 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.738739 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.738746 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=191 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.738750 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.738756 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.738760 7f86a5e11700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.738763 7f86a5610700 10 _do_op 0x30d964e0 seq 3458 r = 0, finisher 0x31478820 0 2014-06-09 16:22:11.738767 7f86a5610700 10 _finish_op 0x30d964e0 seq 3458 osr(2.40a 0x2ed1280)/0x2ed1280 2014-06-09 16:22:11.738759 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.738762 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.738772 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca498c0 (1 active) 2014-06-09 16:22:11.738774 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcadaea0 (2 active) 2014-06-09 16:22:11.738769 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.738776 7f86a5610700 5 _do_op 0x24104f60 seq 3459 osr(3.718 0x3ed1f70)/0x3ed1f70 start 2014-06-09 16:22:11.738770 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.738779 7f86a5610700 10 _do_transaction on 0x344cf00 2014-06-09 16:22:11.738781 7f868d75d700 10 op_submit_start 3507 2014-06-09 16:22:11.738782 7f868bf5a700 2 waiting 51 > 50 ops || 6760431 > 104857600 2014-06-09 16:22:11.738783 7f868d75d700 5 queue_transactions (trailing journal) 3507 2014-06-09 16:22:11.738777 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'638 -> 214'638 2014-06-09 16:22:11.738779 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.738785 7f868d75d700 5 queue_op 0x309a22a0 seq 3507 osr(1.99 0x384c7d0) 858 bytes (queue has 50 ops and 6758477 bytes) 2014-06-09 16:22:11.738787 7f868f761700 2 waiting 51 > 50 ops || 6759906 > 104857600 2014-06-09 16:22:11.738788 7f868d75d700 10 op_submit_finish 3507 2014-06-09 16:22:11.738790 7f868cf5c700 2 waiting 51 > 50 ops || 6759305 > 104857600 2014-06-09 16:22:11.738791 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.738799 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.738790 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.51e( v 213'674 (0'0,213'674] local-les=191 n=127 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.738792 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 2 2014-06-09 16:22:11.738801 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.738805 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.738809 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.738818 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.738808 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.582( v 213'598 (0'0,213'598] local-les=191 n=118 ec=118 les/c 191/191 242/242/242) log((0'0,213'598], crt=0'0) missing(0) 2014-06-09 16:22:11.738822 7f868d75d700 10 proc_replica_log for osd.2: 3.582( v 213'598 (0'0,213'598] local-les=191 n=118 ec=118 les/c 191/191 242/242/242) log((0'0,213'598], crt=0'0) missing(0) 2014-06-09 16:22:11.738816 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.732774 7 0.000322 2014-06-09 16:22:11.738820 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.15c( v 213'624 (0'0,213'624] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.738822 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 2 notify: (query_epoch:243, epoch_sent:243, info:2.b( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.738832 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.738827 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.738838 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.738834 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.738844 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.738836 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 0.971710 0 0.000000 2014-06-09 16:22:11.738847 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3459.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.738851 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.738848 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.755( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.738848 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.738849 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'624,214'638], crt=0'0) 2014-06-09 16:22:11.738860 7f8692f68700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.738858 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=191 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.738858 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.755( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.755( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.738868 7f8689f56700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.738872 7f8689f56700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.738870 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.738875 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.755( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.969944 7 0.000236 2014-06-09 16:22:11.738874 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.15c( v 214'638 lc 213'624 (0'0,214'638] local-les=191 n=123 ec=118 les/c 191/191 242/242/242) missing missing(3) 2014-06-09 16:22:11.738879 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.738886 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.755( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.738887 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'681 -> 214'681 2014-06-09 16:22:11.738888 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 3 missing 2014-06-09 16:22:11.738896 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.755( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.738897 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 569b715c/rbd_data.115c6b8b4567.00000000000017ea/head//3 214'638 is on osd.26 2014-06-09 16:22:11.738905 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.755( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.738908 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 9f12c15c/rbd_data.11796b8b4567.0000000000001a6e/head//3 213'629 is on osd.26 2014-06-09 16:22:11.738914 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.755( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.738920 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.755( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.738919 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing d23f495c/rbd_data.116c6b8b4567.000000000000177c/head//3 214'634 is on osd.26 2014-06-09 16:22:11.738929 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {569b715c/rbd_data.115c6b8b4567.00000000000017ea/head//3=214'638,9f12c15c/rbd_data.11796b8b4567.0000000000001a6e/head//3=213'629,d23f495c/rbd_data.116c6b8b4567.000000000000177c/head//3=214'634} 2014-06-09 16:22:11.738934 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.51e( v 213'674 (0'0,213'674] local-les=191 n=127 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.738940 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.738952 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.742509 7f8689f56700 5 queue_transactions existing osr(1.755 0x2ed0090)/0x2ed0090 2014-06-09 16:22:11.738953 7f8692767700 5 queue_transactions existing osr(3.70e 0x3ed1ba0)/0x3ed1ba0 2014-06-09 16:22:11.742501 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 569b715c/rbd_data.115c6b8b4567.00000000000017ea/head//3 214'638 also missing on osd.2 2014-06-09 16:22:11.738955 7f868ff62700 2 waiting 51 > 50 ops || 6759812 > 104857600 2014-06-09 16:22:11.738954 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'674,214'681], crt=0'0) 2014-06-09 16:22:11.742525 7f8689f56700 2 waiting 51 > 50 ops || 6760486 > 104857600 2014-06-09 16:22:11.742518 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 9f12c15c/rbd_data.11796b8b4567.0000000000001a6e/head//3 213'629 also missing on osd.2 2014-06-09 16:22:11.742530 7f8690763700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.742532 7f8692767700 2 waiting 51 > 50 ops || 6759812 > 104857600 2014-06-09 16:22:11.742529 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing d23f495c/rbd_data.116c6b8b4567.000000000000177c/head//3 214'634 also missing on osd.2 2014-06-09 16:22:11.742539 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {569b715c/rbd_data.115c6b8b4567.00000000000017ea/head//3=214'638,9f12c15c/rbd_data.11796b8b4567.0000000000001a6e/head//3=213'629,d23f495c/rbd_data.116c6b8b4567.000000000000177c/head//3=214'634} 2014-06-09 16:22:11.742552 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.742548 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.51e( v 214'681 lc 213'674 (0'0,214'681] local-les=191 n=127 ec=118 les/c 191/191 242/242/242) missing missing(2) 2014-06-09 16:22:11.742561 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.742566 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 2 missing 2014-06-09 16:22:11.742572 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.742577 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 1561f51e/rbd_data.11676b8b4567.0000000000001767/head//3 214'681 is on osd.26 2014-06-09 16:22:11.742583 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.742589 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 356aed1e/rbd_data.115a6b8b4567.0000000000001a30/head//3 213'679 is on osd.26 2014-06-09 16:22:11.742594 7f8692f68700 5 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.742601 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {1561f51e/rbd_data.11676b8b4567.0000000000001767/head//3=214'681,356aed1e/rbd_data.115a6b8b4567.0000000000001a30/head//3=213'679} 2014-06-09 16:22:11.742612 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.742622 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 1561f51e/rbd_data.11676b8b4567.0000000000001767/head//3 214'681 also missing on osd.2 2014-06-09 16:22:11.742634 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 356aed1e/rbd_data.115a6b8b4567.0000000000001a30/head//3 213'679 also missing on osd.2 2014-06-09 16:22:11.742643 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {1561f51e/rbd_data.11676b8b4567.0000000000001767/head//3=214'681,356aed1e/rbd_data.115a6b8b4567.0000000000001a30/head//3=213'679} 2014-06-09 16:22:11.742654 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.742663 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.742674 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.742683 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.742693 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.742709 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.742715 7f8692f68700 5 queue_transactions existing osr(3.15c 0x3ed0720)/0x3ed0720 2014-06-09 16:22:11.742717 7f8692f68700 2 waiting 51 > 50 ops || 6759530 > 104857600 2014-06-09 16:22:11.736971 7f869af78700 5 -- op tracker -- , seq: 2811, time: 2014-06-09 16:22:11.578013, event: all_read, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.738826 7f868d75d700 20 merge_log cut point (usually last shared) is 213'598 (213'597) modify a877d82/rbd_data.117b6b8b4567.00000000000017a1/head//3 by client.4571.0:35414 2014-06-09 10:17:58.936423 2014-06-09 16:22:11.739281 7f86a5610700 10 _do_op 0x24104f60 seq 3459 r = 0, finisher 0x31413ce0 0 2014-06-09 16:22:11.742732 7f86a5610700 10 _finish_op 0x24104f60 seq 3459 osr(3.718 0x3ed1f70)/0x3ed1f70 2014-06-09 16:22:11.742739 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcadaea0 (1 active) 2014-06-09 16:22:11.742742 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338120 (2 active) 2014-06-09 16:22:11.742745 7f86a5610700 5 _do_op 0x2ea532c0 seq 3460 osr(0.6f0 0x384c170)/0x384c170 start 2014-06-09 16:22:11.742740 7f86a460e700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.737408, event: op_commit, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.742748 7f86a5610700 10 _do_transaction on 0xbf88180 2014-06-09 16:22:11.742755 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.742757 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189aaa80] 2014-06-09 16:22:11.742761 7f868ef60700 5 -- op tracker -- , seq: 106, time: 2014-06-09 16:22:11.737593, event: reached_pg, op: osd_op(client.4573.0:33491 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~311296] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.742759 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x13964040,0xc5a75e0,0x2d738200,0xbe304c0,0xbcc9980,0xc40dfe0,0xbccd800,0xbd6b4a0,0x29283d00,0xbd6b780,0xbd6b180,0xbdbbd60,0xbd798c0,0xbd687e0,0x306ce5e0,0xc6e85e0,0xbfcf860,0xbd708a0,0xbd6d0e0,0xbdbb960,0x30629c20,0x189aa200,0x307f0280,0xbd71f60,0x29282060,0x13964cc0,0xcdb2420,0xbd79dc0,0xbd64980,0x13964900,0x17001c60,0xc2d6600,0xbce68c0,0x314ac360,0x31479d60,0x20051e00,0xbd75a00,0xc6e9240,0x30628f60,0x3050b620,0xc5a7e00,0xbccd0a0,0xcdb31a0,0x19941400,0xbce79c0,0xbd6a100,0xc40cda0,0xbd7a5e0,0xbef12c0,0x314acfa0,0xc2d6c80,0xc6e8d40,0x30c543c0,0xbef1080,0x31456cc0,0x307f0d60,0x307f1f40,0xcd325e0,0x314accc0,0x307f13a0,0xcd32fa0,0x314ac0e0,0x314ac9e0,0x314adf00,0x314adc80,0x31412020,0x307f10a0,0xc2d6800,0xbebffa0,0x3050abc0,0xbd78880,0x189ab540,0xcd334a0,0x314ac560,0xbebff40,0x314ad340,0xbd6a640,0xcd33260,0x31456020,0x307f17a0,0x31435440,0x31456f60,0x31434b60,0x314571e0,0xcd33f60,0x31435660,0x31435360,0xc5a7d80,0x31456fe0,0x30509f80,0x314ad060,0xbd72de0,0x189aa9c0,0xbe67580,0x30c54aa0,0x31413360,0xbd67840,0x31457ee0,0x3062a440,0x31478860,0x31413d20] 2014-06-09 16:22:11.742773 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.742788 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x10c55d00 2014-06-09 16:22:11.742797 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:33491 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~311296] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.742813 7f868ef60700 3 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4573.0:33491 was 214'577 2014-06-09 16:22:11.742815 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 2 2014-06-09 16:22:11.742833 7f86a460e700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.742827, event: op_commit, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.742840 7f868ef60700 10 osd.26 243 dequeue_op 0x10c55d00 finish 2014-06-09 16:22:11.742848 7f868ef60700 5 -- op tracker -- , seq: 106, time: 2014-06-09 16:22:11.742844, event: done, op: osd_op(client.4573.0:33491 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~311296] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.742867 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.742867 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.742860 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 3 2014-06-09 16:22:11.742873 7f8690763700 5 queue_transactions existing osr(3.51e 0x3ed16a0)/0x3ed16a0 2014-06-09 16:22:11.742876 7f8690763700 10 op_submit_start 3508 2014-06-09 16:22:11.742877 7f8690763700 5 queue_transactions (trailing journal) 3508 2014-06-09 16:22:11.742876 7f86a460e700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.742870, event: op_commit, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.742879 7f8690763700 5 queue_op 0x309a29c0 seq 3508 osr(3.51e 0x3ed16a0) 1053 bytes (queue has 50 ops and 6758477 bytes) 2014-06-09 16:22:11.742882 7f8690763700 10 op_submit_finish 3508 2014-06-09 16:22:11.742888 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:11.742886 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.742892 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.30( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.30( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.742899 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.742912 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.742916 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.742904 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x13964040,0xc5a75e0,0x2d738200,0xbe304c0,0xbcc9980,0xc40dfe0,0xbccd800,0xbd6b4a0,0x29283d00,0xbd6b780,0xbd6b180,0xbdbbd60,0xbd798c0,0xbd687e0,0x306ce5e0,0xc6e85e0,0xbfcf860,0xbd708a0,0xbd6d0e0,0xbdbb960,0x30629c20,0x189aa200,0x307f0280,0xbd71f60,0x29282060,0x13964cc0,0xcdb2420,0xbd79dc0,0xbd64980,0x13964900,0x17001c60,0xc2d6600,0xbce68c0,0x314ac360,0x31479d60,0x20051e00,0xbd75a00,0xc6e9240,0x30628f60,0x3050b620,0xc5a7e00,0xbccd0a0,0xcdb31a0,0x19941400,0xbce79c0,0xbd6a100,0xc40cda0,0xbd7a5e0,0xbef12c0,0x314acfa0,0xc2d6c80,0xc6e8d40,0x30c543c0,0xbef1080,0x31456cc0,0x307f0d60,0x307f1f40,0xcd325e0,0x314accc0,0x307f13a0,0xcd32fa0,0x314ac0e0,0x314ac9e0,0x314adf00,0x314adc80,0x31412020,0x307f10a0,0xc2d6800,0xbebffa0,0x3050abc0,0xbd78880,0x189ab540,0xcd334a0,0x314ac560,0xbebff40,0x314ad340,0xbd6a640,0xcd33260,0x31456020,0x307f17a0,0x31435440,0x31456f60,0x31434b60,0x314571e0,0xcd33f60,0x31435660,0x31435360,0xc5a7d80,0x31456fe0,0x30509f80,0x314ad060,0xbd72de0,0x189aa9c0,0xbe67580,0x30c54aa0,0x31413360,0xbd67840,0x31457ee0,0x3062a440,0x31478860,0x31413d20] 2014-06-09 16:22:11.742920 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.742922 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.742922 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.742927 7f868ef60700 5 queue_transactions existing osr(1.30 0x384c280)/0x384c280 2014-06-09 16:22:11.742920 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.188( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.001739 7 0.000282 2014-06-09 16:22:11.742930 7f868ef60700 2 waiting 51 > 50 ops || 6759303 > 104857600 2014-06-09 16:22:11.742933 7f8689755700 5 -- op tracker -- , seq: 2749, time: 2014-06-09 16:22:11.738121, event: reached_pg, op: osd_op(client.4538.0:36625 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1032192] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.742932 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.188( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.742943 7f8690763700 5 osd.26 pg_epoch: 243 pg[1.188( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.742943 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.742953 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.742958 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x2f097e00 2014-06-09 16:22:11.742968 7f8689755700 10 osd.26 243 dequeue_op 0x2f097e00 finish 2014-06-09 16:22:11.742963 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.742972 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.742972 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.742979 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.742984 7f8689755700 10 osd.26 243 dequeue_op 0x16059200 prio 63 cost 385024 latency 1.451996 osd_op(client.4567.0:35112 rbd_data.11776b8b4567.00000000000017ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.8984db19 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.743006 7f8689755700 5 -- op tracker -- , seq: 229, time: 2014-06-09 16:22:11.743001, event: reached_pg, op: osd_op(client.4567.0:35112 rbd_data.11776b8b4567.00000000000017ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.8984db19 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743020 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.743013 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.743025 7f8690763700 5 queue_transactions existing osr(1.188 0x384cb70)/0x384cb70 2014-06-09 16:22:11.743029 7f8690763700 2 waiting 51 > 50 ops || 6760486 > 104857600 2014-06-09 16:22:11.743027 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x16059200 2014-06-09 16:22:11.743035 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 1 2014-06-09 16:22:11.743032 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.582( v 213'598 (0'0,213'598] local-les=191 n=118 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.743037 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4567.0:35112 rbd_data.11776b8b4567.00000000000017ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.8984db19 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.743057 7f86a4e0f700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.743049, event: op_applied, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743053 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.743055 7f8689755700 3 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4567.0:35112 was 214'604 2014-06-09 16:22:11.743065 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'637 2014-06-09 16:22:11.743067 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.777650 7 0.000322 2014-06-09 16:22:11.743082 7f8689755700 10 osd.26 243 dequeue_op 0x16059200 finish 2014-06-09 16:22:11.743080 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.743097 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd79ca0] 2014-06-09 16:22:11.743090 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.148612 0 0.000000 2014-06-09 16:22:11.743102 7f8689755700 5 -- op tracker -- , seq: 229, time: 2014-06-09 16:22:11.743085, event: done, op: osd_op(client.4567.0:35112 rbd_data.11776b8b4567.00000000000017ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.8984db19 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743104 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.743100 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189abde0,0xc5a70e0,0xbd5a120,0xbebffc0,0xc5a7540,0xc6e99a0,0xbd78440,0xbd71cc0,0xbe31980,0x3062bde0,0xbd6d6e0,0xbccd640,0x13965d60,0xc2d63c0,0xbe306e0,0xbd5a920,0xbd667a0,0xbd680a0,0xbebf4c0,0x17001380,0x305083e0,0xbcc9f60,0xbd6eaa0,0xbd6a960,0x3062bd20,0x189aa660,0xbd76d40,0x3050bb00,0xc411880,0xbd6bf80,0xbd7ca20,0x33a7380,0xc2d6c40,0x314ac380,0x31479d80,0x307f07e0,0x18742ec0,0x3061e2c0,0x306cef60,0x3050bec0,0xcc838e0,0xbd6ddc0,0xcdb2820,0x292820a0,0x19941120,0xbd6b880,0xcc95360,0xbdbb760,0xbfce100,0x314acfc0,0xc2d7760,0x3062a640,0x30c543a0,0xbef19e0,0x31456c80,0x307f0d20,0x307f1f00,0xcd325a0,0x314acce0,0x307f1360,0xcd32f60,0x314ac100,0x314ac9c0,0x314adf20,0x314adca0,0x314127e0,0x307f1060,0x33a7a60,0xc2d7840,0x30c54e80,0xbccc060,0x189aa860,0xcd33460,0x314ac580,0xc6e97e0,0x314ad360,0x314121a0,0xcd33220,0x314567e0,0x307f1760,0x31435400,0x31456f20,0x31434b20,0x314571a0,0xcd33f20,0x31435620,0x31435320,0x3050a3a0,0x31456820,0xbfcea60,0x314ad080,0x17001540,0xbd69cc0,0xc6e9080,0x30c54ac0,0x31413320,0xcdb3de0,0x31457ea0,0xc5a75a0,0x31478820,0x31413ce0] 2014-06-09 16:22:11.743117 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.743115 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=191 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.743117 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743135 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e770 (20 active) 2014-06-09 16:22:11.743135 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.743128 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.743139 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.743139 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.743148 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'602 -> 214'602 2014-06-09 16:22:11.743153 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.743152 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.743163 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.743172 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.743171 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=215'668 lcod 0'0 inactive m=1] _activate_committed 243 telling primary 2014-06-09 16:22:11.743186 7f8689755700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.743190 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.743188 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.582( v 213'598 (0'0,213'598] local-les=191 n=118 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.743206 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.743208 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'598,214'602], crt=0'0) 2014-06-09 16:22:11.743206 7f868a757700 10 osd.26 243 dequeue_op 0x257c4300 prio 196 cost 0 latency 0.081409 osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.743222 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.743221 7f868d75d700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.743225 7f868a757700 5 -- op tracker -- , seq: 2402, time: 2014-06-09 16:22:11.743221, event: reached_pg, op: osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743226 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.743232 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x257c4300 2014-06-09 16:22:11.743237 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.743236 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.582( v 214'602 lc 213'598 (0'0,214'602] local-les=191 n=118 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.743247 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.743248 7f868a757700 5 -- op tracker -- , seq: 2402, time: 2014-06-09 16:22:11.743245, event: started, op: osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743251 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:11.743255 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.743266 7f8689755700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.743260 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing fa167d82/rbd_data.115a6b8b4567.0000000000001a53/head//3 214'602 is on osd.26 2014-06-09 16:22:11.743261 7f868a757700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 1 op ack_type 4 from 7 2014-06-09 16:22:11.743273 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {fa167d82/rbd_data.115a6b8b4567.0000000000001a53/head//3=214'602} 2014-06-09 16:22:11.743279 7f868a757700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.743274, event: sub_op_commit_rec, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743280 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.743283 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.743292 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.743287 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_applied: repop tid 1 all applied 2014-06-09 16:22:11.743294 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing fa167d82/rbd_data.115a6b8b4567.0000000000001a53/head//3 214'602 also missing on osd.2 2014-06-09 16:22:11.743296 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 4 2014-06-09 16:22:11.743297 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc295140 243'637 rep_tid=1 committed?=0 applied?=1 lock=0 op=osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.743305 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {fa167d82/rbd_data.115a6b8b4567.0000000000001a53/head//3=214'602} 2014-06-09 16:22:11.743308 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd4 2014-06-09 16:22:11.743316 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.743312 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] sending ack on repgather(0xc295140 243'637 rep_tid=1 committed?=0 applied?=1 lock=0 op=osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 0x257c6a00 2014-06-09 16:22:11.743319 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.743325 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.743327 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.743321 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] flushed 2014-06-09 16:22:11.743336 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.743336 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'714 (0'0,214'714] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.4: log((214'714,214'716], crt=0'0) missing(0) 2014-06-09 16:22:11.743347 7f8689755700 10 merge_log log((214'714,214'716], crt=0'0) from osd.4 into log((0'0,214'714], crt=0'0) 2014-06-09 16:22:11.743346 7f8682ae5700 5 -- op tracker -- , seq: 2812, time: 2014-06-09 16:22:11.743315, event: header_read, op: osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743346 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.743352 7f8689755700 10 merge_log extending head to 214'716 2014-06-09 16:22:11.743340 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 1 all committed 2014-06-09 16:22:11.743353 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.743354 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743355 7f8682ae5700 5 -- op tracker -- , seq: 2812, time: 2014-06-09 16:22:11.743315, event: throttled, op: osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743369 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.743357 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 0'0 active] eval_repop repgather(0xc295140 243'637 rep_tid=1 committed?=1 applied?=1 lock=0 op=osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.743368 7f8682ae5700 5 -- op tracker -- , seq: 2812, time: 2014-06-09 16:22:11.743324, event: all_read, op: osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743377 7f8682ae5700 5 -- op tracker -- , seq: 2812, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743382 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.743373 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 0'0 active] log_op_stats osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 inb 962941 outb 0 rlat 0.981958 lat 0.981991 2014-06-09 16:22:11.743383 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.743387 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.743393 7f8682ae5700 15 osd.26 243 enqueue_op 0xdfa6200 prio 196 cost 0 latency 0.000077 osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.743390 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 0'0 active] publish_stats_to_osd 243:1017 2014-06-09 16:22:11.743353 7f8689755700 20 ? 214'716 (214'715) modify b4f7cd5b/rbd_data.117c6b8b4567.000000000000143d/head//3 by client.4572.0:30293 2014-06-09 10:19:21.026047 2014-06-09 16:22:11.743397 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743405 7f8689755700 20 ? 214'715 (214'714) modify b4f7cd5b/rbd_data.117c6b8b4567.000000000000143d/head//3 by client.4572.0:30292 2014-06-09 10:19:16.812009 2014-06-09 16:22:11.743410 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.743411 7f8689755700 20 merge_log 214'715 (214'714) modify b4f7cd5b/rbd_data.117c6b8b4567.000000000000143d/head//3 by client.4572.0:30292 2014-06-09 10:19:16.812009 2014-06-09 16:22:11.743404 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 0'0 active] sending commit on repgather(0xc295140 243'637 rep_tid=1 committed?=1 applied?=1 lock=0 op=osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 0xdfb1680 2014-06-09 16:22:11.743420 7f8689755700 20 merge_log 214'716 (214'715) modify b4f7cd5b/rbd_data.117c6b8b4567.000000000000143d/head//3 by client.4572.0:30293 2014-06-09 10:19:21.026047 2014-06-09 16:22:11.743425 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.743435 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.743433 7f868a757700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.743428, event: commit_sent, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743440 7f868d75d700 5 queue_transactions existing osr(3.582 0x3ed18a0)/0x3ed18a0 2014-06-09 16:22:11.743443 7f868d75d700 2 waiting 51 > 50 ops || 6759530 > 104857600 2014-06-09 16:22:11.743439 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.743442 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743456 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.743448 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 214'636 active] removing repgather(0xc295140 243'637 rep_tid=1 committed?=1 applied?=1 op=osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.743460 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 214'636 active] q front is repgather(0xc295140 243'637 rep_tid=1 committed?=1 applied?=1 op=osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.743468 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.743473 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 214'636 active] remove_repop repgather(0xc295140 243'637 rep_tid=1 committed?=1 applied?=1 op=osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.743484 7f8670072700 5 -- op tracker -- , seq: 2813, time: 2014-06-09 16:22:11.736767, event: header_read, op: osd_op(client.4538.0:36649 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3600384~593920] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.743483 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743487 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 214'636 active] requeue_ops 2014-06-09 16:22:11.743498 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.743497 7f8670072700 5 -- op tracker -- , seq: 2813, time: 2014-06-09 16:22:11.736768, event: throttled, op: osd_op(client.4538.0:36649 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3600384~593920] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.743516 7f868a757700 5 -- op tracker -- , seq: 689, time: 2014-06-09 16:22:11.743504, event: done, op: osd_op(client.4570.0:34069 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~962560] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743512 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.743530 7f868a757700 10 osd.26 243 dequeue_op 0x257c4300 finish 2014-06-09 16:22:11.743533 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743541 7f868a757700 5 -- op tracker -- , seq: 2402, time: 2014-06-09 16:22:11.743536, event: done, op: osd_sub_op_reply(client.4570.0:34069 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743547 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.743553 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.743559 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.743561 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.743563 7f868a757700 10 osd.26 243 dequeue_op 0x2f097b00 prio 196 cost 0 latency 0.078016 osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 214'636 active] 2014-06-09 16:22:11.743581 7f868a757700 5 -- op tracker -- , seq: 2408, time: 2014-06-09 16:22:11.743578, event: reached_pg, op: osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743588 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743590 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 214'636 active] handle_message: 0x2f097b00 2014-06-09 16:22:11.743602 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.743605 7f868a757700 5 -- op tracker -- , seq: 2408, time: 2014-06-09 16:22:11.743601, event: started, op: osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743611 7f868a757700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 214'636 active] sub_op_modify_reply: tid 2 op ack_type 4 from 7 2014-06-09 16:22:11.743622 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.743632 7f868a757700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.743628, event: sub_op_commit_rec, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743637 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'637 crt=0'0 lcod 214'636 mlcod 214'636 active] repop_all_committed: repop tid 2 all committed 2014-06-09 16:22:11.743638 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743652 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.743648 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'638 crt=0'0 lcod 243'637 mlcod 214'636 active] eval_repop repgather(0xc295200 243'638 rep_tid=2 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.743663 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.743665 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'638 crt=0'0 lcod 243'637 mlcod 214'636 active] log_op_stats osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 217469 outb 0 rlat 0.000000 lat 1.010715 2014-06-09 16:22:11.743678 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743680 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'638 crt=0'0 lcod 243'637 mlcod 214'636 active] publish_stats_to_osd 243:1018 2014-06-09 16:22:11.743691 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.743690 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'638 crt=0'0 lcod 243'637 mlcod 214'636 active] sending commit on repgather(0xc295200 243'638 rep_tid=2 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 0x257c6f00 2014-06-09 16:22:11.743705 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.743721 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.743727 7f868a757700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.743721, event: commit_sent, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743734 7f868a757700 10 osd.26 243 dequeue_op 0x2f097b00 finish 2014-06-09 16:22:11.743734 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.743739 7f8689755700 10 merge_log result log((0'0,214'716], crt=214'716) missing(1) changed=1 2014-06-09 16:22:11.743742 7f868a757700 5 -- op tracker -- , seq: 2408, time: 2014-06-09 16:22:11.743739, event: done, op: osd_sub_op_reply(client.4572.0:30632 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743752 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.743743 7f868df5e700 10 osd.26 243 dequeue_op 0x257c4100 prio 196 cost 0 latency 0.073197 osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'638 crt=0'0 lcod 243'637 mlcod 214'636 active] 2014-06-09 16:22:11.743744 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 peering m=1] peer osd.4 now 3.55b( v 214'716 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) missing(0) 2014-06-09 16:22:11.743751 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.743762 7f868df5e700 5 -- op tracker -- , seq: 2428, time: 2014-06-09 16:22:11.743758, event: reached_pg, op: osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743770 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (20 active) 2014-06-09 16:22:11.743765 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.753711 6 0.000240 2014-06-09 16:22:11.743768 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'638 crt=0'0 lcod 243'637 mlcod 214'636 active] handle_message: 0x257c4100 2014-06-09 16:22:11.743776 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.4da( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.743783 7f868df5e700 5 -- op tracker -- , seq: 2428, time: 2014-06-09 16:22:11.743779, event: started, op: osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743780 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.743790 7f868df5e700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'638 crt=0'0 lcod 243'637 mlcod 214'636 active] sub_op_modify_reply: tid 3 op ack_type 4 from 7 2014-06-09 16:22:11.743792 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 peering m=1] state: osd.4 has no missing, identical log 2014-06-09 16:22:11.743806 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.743810 7f868a757700 5 queue_transactions existing osr(0.4da 0x2bd75f0)/0x2bd75f0 2014-06-09 16:22:11.743807 7f868df5e700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.743802, event: sub_op_commit_rec, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743813 7f868a757700 2 waiting 51 > 50 ops || 6759305 > 104857600 2014-06-09 16:22:11.743806 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000026 0 0.000000 2014-06-09 16:22:11.743813 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'638 crt=0'0 lcod 243'637 mlcod 214'636 active] repop_all_committed: repop tid 3 all committed 2014-06-09 16:22:11.743819 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.743824 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] eval_repop repgather(0xbfca900 243'639 rep_tid=3 committed?=1 applied?=0 lock=0 op=osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.743830 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.293770 0 0.000000 2014-06-09 16:22:11.743844 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.743840 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] log_op_stats osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 0.980901 2014-06-09 16:22:11.743855 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.743857 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] publish_stats_to_osd 243:1019 2014-06-09 16:22:11.743867 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.743867 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] sending commit on repgather(0xbfca900 243'639 rep_tid=3 committed?=1 applied?=0 lock=0 op=osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 0x257c7400 2014-06-09 16:22:11.743878 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.743887 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'714 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.743904 7f868df5e700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.743898, event: commit_sent, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743912 7f868df5e700 10 osd.26 243 dequeue_op 0x257c4100 finish 2014-06-09 16:22:11.743921 7f868df5e700 5 -- op tracker -- , seq: 2428, time: 2014-06-09 16:22:11.743917, event: done, op: osd_sub_op_reply(client.4570.0:34070 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.743932 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.743938 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.743942 7f868df5e700 10 osd.26 243 dequeue_op 0x1c54bb00 prio 63 cost 61440 latency 1.262178 osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.743950 7f868b759700 10 osd.26 243 dequeue_op 0x257c5500 prio 63 cost 516096 latency 0.988688 osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] 2014-06-09 16:22:11.743962 7f868df5e700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.743958, event: reached_pg, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743972 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.743979 7f868b759700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.743967, event: reached_pg, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.743985 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1c54bb00 2014-06-09 16:22:11.743988 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.743999 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] handle_message: 0x257c5500 2014-06-09 16:22:11.743994 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.744015 7f868df5e700 15 getattr 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 '_' 2014-06-09 16:22:11.744008 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_op osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.744027 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.744028 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] get_object_context: found obc in cache: 0x305c6b00 2014-06-09 16:22:11.744046 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x998a000 (1 active) 2014-06-09 16:22:11.744043 7f8689755700 10 osd.26 243 queue_for_recovery queued pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.744051 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.744040 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] get_object_context: 0x305c6b00 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 rwstate(write n=3 w=0) oi: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'642 client.4572.0:30634 wrlock_by=unknown.0.0:0 dirty s 2797568 uv642) ssc: 0x2e8775e0 snapset: 0=[]:[]+head 2014-06-09 16:22:11.744057 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 3.55b( v 214'716 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) 2014-06-09 16:22:11.744056 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] find_object_context 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 @head oi=5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'642 client.4572.0:30634 wrlock_by=unknown.0.0:0 dirty s 2797568 uv642) 2014-06-09 16:22:11.744070 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 is up to date, queueing in pending_activators 2014-06-09 16:22:11.744084 7f868b759700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.744077, event: started, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.744086 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 3.55b( v 214'716 (0'0,214'716] local-les=203 n=138 ec=118 les/c 203/203 242/242/242) uptodate 2014-06-09 16:22:11.744091 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] execute_ctx 0x3088b000 2014-06-09 16:22:11.744100 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.744109 7f868b759700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.744104, event: started, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.744112 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing b4f7cd5b/rbd_data.117c6b8b4567.000000000000143d/head//3 214'716 also missing on osd.26 2014-06-09 16:22:11.744115 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] ov 243'642 av 243'644 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.744124 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {b4f7cd5b/rbd_data.117c6b8b4567.000000000000143d/head//3=214'716(214'714)} 2014-06-09 16:22:11.744130 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 2014-06-09 16:22:11.744136 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 4 as a source 2014-06-09 16:22:11.744143 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.744147 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing b4f7cd5b/rbd_data.117c6b8b4567.000000000000143d/head//3 214'716 is on osd.4 2014-06-09 16:22:11.744157 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op write 2797568~516096 2014-06-09 16:22:11.744160 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {b4f7cd5b/rbd_data.117c6b8b4567.000000000000143d/head//3=214'716(214'714)} 2014-06-09 16:22:11.744173 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.744174 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op_effects on session 0xd229400 2014-06-09 16:22:11.744183 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 203 2014-06-09 16:22:11.744186 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 snapset=0x2e877618 snapc=0=[] 2014-06-09 16:22:11.744195 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 4 2014-06-09 16:22:11.744200 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.744206 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.744211 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] finish_ctx 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 0x3088b000 op modify 2014-06-09 16:22:11.744218 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.744224 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] set mtime to 2014-06-09 10:20:23.788498 2014-06-09 16:22:11.744246 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] final snapset 0=[]:[]+head in 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.744256 7f8689755700 10 write_log with: dirty_to: 0'0, dirty_from: 214'714, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.744265 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] zeroing write result code 0 2014-06-09 16:22:11.744277 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] new_repop rep_tid 11 on osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.744292 7f8689755700 5 queue_transactions existing osr(3.55b 0x3ed1980)/0x3ed1980 2014-06-09 16:22:11.744295 7f8689755700 2 waiting 51 > 50 ops || 6760420 > 104857600 2014-06-09 16:22:11.744301 7f868bf5a700 2 waiting 51 > 50 ops || 6760431 > 104857600 2014-06-09 16:22:11.744294 7f868b759700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] issue_repop rep_tid 11 o 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.744307 7f868f761700 2 waiting 51 > 50 ops || 6759906 > 104857600 2014-06-09 16:22:11.744314 7f868cf5c700 2 waiting 51 > 50 ops || 6759305 > 104857600 2014-06-09 16:22:11.744321 7f868ff62700 2 waiting 51 > 50 ops || 6759812 > 104857600 2014-06-09 16:22:11.744311 7f869af78700 5 -- op tracker -- , seq: 2811, time: 2014-06-09 16:22:11.736952, event: dispatched, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.744328 7f8689f56700 2 waiting 51 > 50 ops || 6760486 > 104857600 2014-06-09 16:22:11.744323 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744331 7f869af78700 5 -- op tracker -- , seq: 2811, time: 2014-06-09 16:22:11.744329, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.744333 7f8692767700 2 waiting 51 > 50 ops || 6759812 > 104857600 2014-06-09 16:22:11.744339 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.744346 7f868b759700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.744314, event: waiting for subops from 7, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.744355 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.744364 7f868b759700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.744371 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744390 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.744390 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'643 (0'0,243'643] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] append_log log((0'0,243'643], crt=0'0) [243'644 (243'642) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30635 2014-06-09 10:20:23.788498] 2014-06-09 16:22:11.744405 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.744411 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] add_log_entry 243'644 (243'642) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30635 2014-06-09 10:20:23.788498 2014-06-09 16:22:11.744421 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744435 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.744433 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] append_log adding 1 keys 2014-06-09 16:22:11.744449 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.744462 7f868b759700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'644, trimmed: 2014-06-09 16:22:11.744466 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744478 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.744480 7f868b759700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.744485 7f868b759700 2 waiting 51 > 50 ops || 7276857 > 104857600 2014-06-09 16:22:11.744490 7f8692f68700 2 waiting 51 > 50 ops || 6759530 > 104857600 2014-06-09 16:22:11.744495 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.69d) v4 from osd.1 2014-06-09 16:22:11.744491 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.744499 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405c1c0 2014-06-09 16:22:11.744507 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744522 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.744536 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.744554 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744567 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.744579 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.744593 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744606 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.744621 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.744637 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744651 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.744665 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.744683 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744698 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.744712 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.744729 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744743 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.744757 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.744774 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744787 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.744798 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.744814 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744827 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.744841 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.744851 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744858 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.744870 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.744886 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744900 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.744915 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.744930 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744944 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.744957 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.744974 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.744989 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.745003 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.745020 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=243 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745036 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.745050 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=243 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.745067 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745082 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.745096 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.745113 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745126 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.745140 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.745157 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745170 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.745184 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.745199 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745214 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.745223 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.745239 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745252 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.745263 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.745278 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745291 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.745306 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.745321 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745334 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.745349 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.745365 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745378 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.745389 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.745404 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745418 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.745429 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.745445 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745457 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.745472 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.745490 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745504 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.745519 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.745534 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745547 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.745561 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.745575 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=243 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.745586 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.745600 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=243 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.745698 7f8670072700 5 -- op tracker -- , seq: 2813, time: 2014-06-09 16:22:11.738261, event: all_read, op: osd_op(client.4538.0:36649 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3600384~593920] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.745715 7f8670072700 5 -- op tracker -- , seq: 2813, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36649 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3600384~593920] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.745721 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.745724 7f869af78700 5 -- op tracker -- , seq: 2811, time: 2014-06-09 16:22:11.744503, event: started, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.745742 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6700 prio 63 cost 593920 latency 0.008975 osd_op(client.4538.0:36649 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 3600384~593920] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.745748 7f869af78700 5 -- op tracker -- , seq: 2811, time: 2014-06-09 16:22:11.745746, event: done, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.745758 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.745760 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.745775 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.745776 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.745778 7f869af78700 20 osd.26 243 _dispatch 0xddc5500 pg_info(1 pgs e243:0.714) v4 2014-06-09 16:22:11.745784 7f869af78700 5 -- op tracker -- , seq: 2814, time: 2014-06-09 16:22:11.580521, event: header_read, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.745798 7f869af78700 5 -- op tracker -- , seq: 2814, time: 2014-06-09 16:22:11.580522, event: throttled, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.745805 7f869af78700 5 -- op tracker -- , seq: 2814, time: 2014-06-09 16:22:11.580535, event: all_read, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.745812 7f869af78700 5 -- op tracker -- , seq: 2814, time: 2014-06-09 16:22:11.745774, event: dispatched, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.745820 7f869af78700 5 -- op tracker -- , seq: 2814, time: 2014-06-09 16:22:11.745818, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.745827 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.714) v4 from osd.9 2014-06-09 16:22:11.745830 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc5500 2014-06-09 16:22:11.745836 7f869af78700 5 -- op tracker -- , seq: 2814, time: 2014-06-09 16:22:11.745834, event: started, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.745859 7f869af78700 5 -- op tracker -- , seq: 2814, time: 2014-06-09 16:22:11.745856, event: done, op: pg_info(1 pgs e243:0.714) 2014-06-09 16:22:11.745872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.745874 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.745891 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.745893 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.745894 7f869af78700 20 osd.26 243 _dispatch 0x1607afc0 pg_info(1 pgs e243:0.33f) v4 2014-06-09 16:22:11.745900 7f869af78700 5 -- op tracker -- , seq: 2815, time: 2014-06-09 16:22:11.479345, event: header_read, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.745906 7f869af78700 5 -- op tracker -- , seq: 2815, time: 2014-06-09 16:22:11.479346, event: throttled, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.745913 7f869af78700 5 -- op tracker -- , seq: 2815, time: 2014-06-09 16:22:11.479359, event: all_read, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.745920 7f869af78700 5 -- op tracker -- , seq: 2815, time: 2014-06-09 16:22:11.745891, event: dispatched, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.745927 7f869af78700 5 -- op tracker -- , seq: 2815, time: 2014-06-09 16:22:11.745925, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.745932 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.33f) v4 from osd.8 2014-06-09 16:22:11.745935 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1607afc0 2014-06-09 16:22:11.745940 7f869af78700 5 -- op tracker -- , seq: 2815, time: 2014-06-09 16:22:11.745938, event: started, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.745961 7f869af78700 5 -- op tracker -- , seq: 2815, time: 2014-06-09 16:22:11.745958, event: done, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.745969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.745971 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.745984 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.745986 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.745987 7f869af78700 20 osd.26 243 _dispatch 0x2176ce00 pg_info(1 pgs e243:1.a7) v4 2014-06-09 16:22:11.745992 7f869af78700 5 -- op tracker -- , seq: 2816, time: 2014-06-09 16:22:11.573977, event: header_read, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.745999 7f869af78700 5 -- op tracker -- , seq: 2816, time: 2014-06-09 16:22:11.573978, event: throttled, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746005 7f869af78700 5 -- op tracker -- , seq: 2816, time: 2014-06-09 16:22:11.573991, event: all_read, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746012 7f869af78700 5 -- op tracker -- , seq: 2816, time: 2014-06-09 16:22:11.745983, event: dispatched, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746020 7f869af78700 5 -- op tracker -- , seq: 2816, time: 2014-06-09 16:22:11.746017, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746025 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a7) v4 from osd.6 2014-06-09 16:22:11.746028 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176ce00 2014-06-09 16:22:11.746033 7f869af78700 5 -- op tracker -- , seq: 2816, time: 2014-06-09 16:22:11.746031, event: started, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746049 7f869af78700 5 -- op tracker -- , seq: 2816, time: 2014-06-09 16:22:11.746046, event: done, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746057 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746058 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746073 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746074 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746076 7f869af78700 20 osd.26 243 _dispatch 0x30cafc00 pg_info(1 pgs e243:0.55d) v4 2014-06-09 16:22:11.746081 7f869af78700 5 -- op tracker -- , seq: 2817, time: 2014-06-09 16:22:11.591470, event: header_read, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.746088 7f869af78700 5 -- op tracker -- , seq: 2817, time: 2014-06-09 16:22:11.591470, event: throttled, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.746094 7f869af78700 5 -- op tracker -- , seq: 2817, time: 2014-06-09 16:22:11.591485, event: all_read, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.746102 7f869af78700 5 -- op tracker -- , seq: 2817, time: 2014-06-09 16:22:11.746072, event: dispatched, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.746109 7f869af78700 5 -- op tracker -- , seq: 2817, time: 2014-06-09 16:22:11.746107, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.746114 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.55d) v4 from osd.0 2014-06-09 16:22:11.746117 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cafc00 2014-06-09 16:22:11.746122 7f869af78700 5 -- op tracker -- , seq: 2817, time: 2014-06-09 16:22:11.746120, event: started, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.746137 7f869af78700 5 -- op tracker -- , seq: 2817, time: 2014-06-09 16:22:11.746135, event: done, op: pg_info(1 pgs e243:0.55d) 2014-06-09 16:22:11.746145 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746160 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746162 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746163 7f869af78700 20 osd.26 243 _dispatch 0x30daea80 pg_info(1 pgs e243:2.3c0) v4 2014-06-09 16:22:11.746168 7f869af78700 5 -- op tracker -- , seq: 2818, time: 2014-06-09 16:22:11.575083, event: header_read, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.746175 7f869af78700 5 -- op tracker -- , seq: 2818, time: 2014-06-09 16:22:11.575084, event: throttled, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.746185 7f869af78700 5 -- op tracker -- , seq: 2818, time: 2014-06-09 16:22:11.575095, event: all_read, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.746191 7f869af78700 5 -- op tracker -- , seq: 2818, time: 2014-06-09 16:22:11.746160, event: dispatched, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.746198 7f869af78700 5 -- op tracker -- , seq: 2818, time: 2014-06-09 16:22:11.746196, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.746208 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3c0) v4 from osd.5 2014-06-09 16:22:11.746211 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30daea80 2014-06-09 16:22:11.746216 7f869af78700 5 -- op tracker -- , seq: 2818, time: 2014-06-09 16:22:11.746214, event: started, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.746232 7f869af78700 5 -- op tracker -- , seq: 2818, time: 2014-06-09 16:22:11.746230, event: done, op: pg_info(1 pgs e243:2.3c0) 2014-06-09 16:22:11.746241 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746243 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746256 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746257 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746259 7f869af78700 20 osd.26 243 _dispatch 0x30cf81c0 pg_info(1 pgs e243:3.443) v4 2014-06-09 16:22:11.746264 7f869af78700 5 -- op tracker -- , seq: 2819, time: 2014-06-09 16:22:11.322928, event: header_read, op: pg_info(1 pgs e243:3.443) 2014-06-09 16:22:11.746271 7f869af78700 5 -- op tracker -- , seq: 2819, time: 2014-06-09 16:22:11.322928, event: throttled, op: pg_info(1 pgs e243:3.443) 2014-06-09 16:22:11.746277 7f869af78700 5 -- op tracker -- , seq: 2819, time: 2014-06-09 16:22:11.322938, event: all_read, op: pg_info(1 pgs e243:3.443) 2014-06-09 16:22:11.746284 7f869af78700 5 -- op tracker -- , seq: 2819, time: 2014-06-09 16:22:11.746255, event: dispatched, op: pg_info(1 pgs e243:3.443) 2014-06-09 16:22:11.746291 7f869af78700 5 -- op tracker -- , seq: 2819, time: 2014-06-09 16:22:11.746289, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.443) 2014-06-09 16:22:11.746297 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.443) v4 from osd.2 2014-06-09 16:22:11.746301 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf81c0 2014-06-09 16:22:11.746306 7f869af78700 5 -- op tracker -- , seq: 2819, time: 2014-06-09 16:22:11.746304, event: started, op: pg_info(1 pgs e243:3.443) 2014-06-09 16:22:11.746323 7f869af78700 5 -- op tracker -- , seq: 2819, time: 2014-06-09 16:22:11.746321, event: done, op: pg_info(1 pgs e243:3.443) 2014-06-09 16:22:11.746336 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746338 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746351 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746353 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746354 7f869af78700 20 osd.26 243 _dispatch 0x2eee7500 pg_info(1 pgs e243:2.2f7) v4 2014-06-09 16:22:11.746359 7f869af78700 5 -- op tracker -- , seq: 2820, time: 2014-06-09 16:22:11.439764, event: header_read, op: pg_info(1 pgs e243:2.2f7) 2014-06-09 16:22:11.746371 7f869af78700 5 -- op tracker -- , seq: 2820, time: 2014-06-09 16:22:11.439765, event: throttled, op: pg_info(1 pgs e243:2.2f7) 2014-06-09 16:22:11.746378 7f869af78700 5 -- op tracker -- , seq: 2820, time: 2014-06-09 16:22:11.439778, event: all_read, op: pg_info(1 pgs e243:2.2f7) 2014-06-09 16:22:11.746389 7f869af78700 5 -- op tracker -- , seq: 2820, time: 2014-06-09 16:22:11.746351, event: dispatched, op: pg_info(1 pgs e243:2.2f7) 2014-06-09 16:22:11.746400 7f869af78700 5 -- op tracker -- , seq: 2820, time: 2014-06-09 16:22:11.746398, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2f7) 2014-06-09 16:22:11.746405 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2f7) v4 from osd.4 2014-06-09 16:22:11.746408 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee7500 2014-06-09 16:22:11.746413 7f869af78700 5 -- op tracker -- , seq: 2820, time: 2014-06-09 16:22:11.746411, event: started, op: pg_info(1 pgs e243:2.2f7) 2014-06-09 16:22:11.746436 7f869af78700 5 -- op tracker -- , seq: 2820, time: 2014-06-09 16:22:11.746434, event: done, op: pg_info(1 pgs e243:2.2f7) 2014-06-09 16:22:11.746444 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746446 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746460 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746458 7f8670072700 5 -- op tracker -- , seq: 2821, time: 2014-06-09 16:22:11.745759, event: header_read, op: osd_op(client.4538.0:36794 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~798720] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.746461 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746463 7f869af78700 20 osd.26 243 _dispatch 0x2fc94a80 pg_info(1 pgs e243:2.298) v4 2014-06-09 16:22:11.746465 7f8670072700 5 -- op tracker -- , seq: 2821, time: 2014-06-09 16:22:11.745760, event: throttled, op: osd_op(client.4538.0:36794 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~798720] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.746472 7f8670072700 5 -- op tracker -- , seq: 2821, time: 2014-06-09 16:22:11.746436, event: all_read, op: osd_op(client.4538.0:36794 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~798720] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.746486 7f8670072700 5 -- op tracker -- , seq: 2821, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36794 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~798720] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.746490 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.746494 7f869af78700 5 -- op tracker -- , seq: 2822, time: 2014-06-09 16:22:11.610598, event: header_read, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.746501 7f869af78700 5 -- op tracker -- , seq: 2822, time: 2014-06-09 16:22:11.610599, event: throttled, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.746502 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6600 prio 63 cost 798720 latency 0.000743 osd_op(client.4538.0:36794 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~798720] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.746508 7f869af78700 5 -- op tracker -- , seq: 2822, time: 2014-06-09 16:22:11.610619, event: all_read, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.746519 7f869af78700 5 -- op tracker -- , seq: 2822, time: 2014-06-09 16:22:11.746459, event: dispatched, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.746530 7f869af78700 5 -- op tracker -- , seq: 2822, time: 2014-06-09 16:22:11.746528, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.746534 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.298) v4 from osd.7 2014-06-09 16:22:11.746538 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc94a80 2014-06-09 16:22:11.746543 7f869af78700 5 -- op tracker -- , seq: 2822, time: 2014-06-09 16:22:11.746541, event: started, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.746558 7f869af78700 5 -- op tracker -- , seq: 2822, time: 2014-06-09 16:22:11.746555, event: done, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.746573 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746575 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746589 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746590 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746592 7f869af78700 20 osd.26 243 _dispatch 0x30dc3880 pg_info(1 pgs e243:3.2c9) v4 2014-06-09 16:22:11.746597 7f869af78700 5 -- op tracker -- , seq: 2823, time: 2014-06-09 16:22:11.577831, event: header_read, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.746604 7f869af78700 5 -- op tracker -- , seq: 2823, time: 2014-06-09 16:22:11.577831, event: throttled, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.746610 7f869af78700 5 -- op tracker -- , seq: 2823, time: 2014-06-09 16:22:11.577844, event: all_read, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.746619 7f869af78700 5 -- op tracker -- , seq: 2823, time: 2014-06-09 16:22:11.746588, event: dispatched, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.746627 7f869af78700 5 -- op tracker -- , seq: 2823, time: 2014-06-09 16:22:11.746625, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.746631 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2c9) v4 from osd.3 2014-06-09 16:22:11.746634 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc3880 2014-06-09 16:22:11.746639 7f869af78700 5 -- op tracker -- , seq: 2823, time: 2014-06-09 16:22:11.746637, event: started, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.746656 7f869af78700 5 -- op tracker -- , seq: 2823, time: 2014-06-09 16:22:11.746654, event: done, op: pg_info(1 pgs e243:3.2c9) 2014-06-09 16:22:11.746663 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746665 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746678 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746680 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746681 7f869af78700 20 osd.26 243 _dispatch 0x1405d340 pg_info(1 pgs e243:2.f4) v4 2014-06-09 16:22:11.746686 7f869af78700 5 -- op tracker -- , seq: 2824, time: 2014-06-09 16:22:11.578020, event: header_read, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.746694 7f869af78700 5 -- op tracker -- , seq: 2824, time: 2014-06-09 16:22:11.578020, event: throttled, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.746700 7f869af78700 5 -- op tracker -- , seq: 2824, time: 2014-06-09 16:22:11.578032, event: all_read, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.746707 7f869af78700 5 -- op tracker -- , seq: 2824, time: 2014-06-09 16:22:11.746678, event: dispatched, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.746719 7f869af78700 5 -- op tracker -- , seq: 2824, time: 2014-06-09 16:22:11.746717, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.746724 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.f4) v4 from osd.1 2014-06-09 16:22:11.746727 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405d340 2014-06-09 16:22:11.746731 7f869af78700 5 -- op tracker -- , seq: 2824, time: 2014-06-09 16:22:11.746730, event: started, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.746747 7f869af78700 5 -- op tracker -- , seq: 2824, time: 2014-06-09 16:22:11.746745, event: done, op: pg_info(1 pgs e243:2.f4) 2014-06-09 16:22:11.746754 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746756 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746768 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746770 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746771 7f869af78700 20 osd.26 243 _dispatch 0x30cf01c0 pg_info(1 pgs e243:3.77) v4 2014-06-09 16:22:11.746776 7f869af78700 5 -- op tracker -- , seq: 2825, time: 2014-06-09 16:22:11.593016, event: header_read, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.746785 7f869af78700 5 -- op tracker -- , seq: 2825, time: 2014-06-09 16:22:11.593016, event: throttled, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.746792 7f869af78700 5 -- op tracker -- , seq: 2825, time: 2014-06-09 16:22:11.593029, event: all_read, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.746799 7f869af78700 5 -- op tracker -- , seq: 2825, time: 2014-06-09 16:22:11.746768, event: dispatched, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.746807 7f869af78700 5 -- op tracker -- , seq: 2825, time: 2014-06-09 16:22:11.746805, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.746812 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.77) v4 from osd.9 2014-06-09 16:22:11.746815 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf01c0 2014-06-09 16:22:11.746820 7f869af78700 5 -- op tracker -- , seq: 2825, time: 2014-06-09 16:22:11.746818, event: started, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.746841 7f869af78700 5 -- op tracker -- , seq: 2825, time: 2014-06-09 16:22:11.746838, event: done, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.746848 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746850 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746862 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746864 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746865 7f869af78700 20 osd.26 243 _dispatch 0x2eef16c0 pg_info(1 pgs e243:0.33f) v4 2014-06-09 16:22:11.746870 7f869af78700 5 -- op tracker -- , seq: 2826, time: 2014-06-09 16:22:11.479369, event: header_read, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.746877 7f869af78700 5 -- op tracker -- , seq: 2826, time: 2014-06-09 16:22:11.479369, event: throttled, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.746884 7f869af78700 5 -- op tracker -- , seq: 2826, time: 2014-06-09 16:22:11.479382, event: all_read, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.746890 7f869af78700 5 -- op tracker -- , seq: 2826, time: 2014-06-09 16:22:11.746861, event: dispatched, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.746902 7f869af78700 5 -- op tracker -- , seq: 2826, time: 2014-06-09 16:22:11.746899, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.746907 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.33f) v4 from osd.8 2014-06-09 16:22:11.746910 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef16c0 2014-06-09 16:22:11.746915 7f869af78700 5 -- op tracker -- , seq: 2826, time: 2014-06-09 16:22:11.746913, event: started, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.746930 7f869af78700 5 -- op tracker -- , seq: 2826, time: 2014-06-09 16:22:11.746928, event: done, op: pg_info(1 pgs e243:0.33f) 2014-06-09 16:22:11.746939 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746942 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746955 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.746956 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.746958 7f869af78700 20 osd.26 243 _dispatch 0x30d24fc0 pg_info(1 pgs e243:1.a7) v4 2014-06-09 16:22:11.746963 7f869af78700 5 -- op tracker -- , seq: 2827, time: 2014-06-09 16:22:11.574166, event: header_read, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746970 7f869af78700 5 -- op tracker -- , seq: 2827, time: 2014-06-09 16:22:11.574166, event: throttled, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746978 7f869af78700 5 -- op tracker -- , seq: 2827, time: 2014-06-09 16:22:11.574178, event: all_read, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746985 7f869af78700 5 -- op tracker -- , seq: 2827, time: 2014-06-09 16:22:11.746954, event: dispatched, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746992 7f869af78700 5 -- op tracker -- , seq: 2827, time: 2014-06-09 16:22:11.746990, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.746998 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a7) v4 from osd.6 2014-06-09 16:22:11.747001 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d24fc0 2014-06-09 16:22:11.747006 7f869af78700 5 -- op tracker -- , seq: 2827, time: 2014-06-09 16:22:11.747004, event: started, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.747021 7f869af78700 5 -- op tracker -- , seq: 2827, time: 2014-06-09 16:22:11.747019, event: done, op: pg_info(1 pgs e243:1.a7) 2014-06-09 16:22:11.747033 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747036 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747048 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747050 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747051 7f869af78700 20 osd.26 243 _dispatch 0x30cafa40 pg_info(1 pgs e243:0.25b) v4 2014-06-09 16:22:11.747056 7f869af78700 5 -- op tracker -- , seq: 2828, time: 2014-06-09 16:22:11.591501, event: header_read, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.747063 7f869af78700 5 -- op tracker -- , seq: 2828, time: 2014-06-09 16:22:11.591501, event: throttled, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.747069 7f869af78700 5 -- op tracker -- , seq: 2828, time: 2014-06-09 16:22:11.591521, event: all_read, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.747076 7f869af78700 5 -- op tracker -- , seq: 2828, time: 2014-06-09 16:22:11.747048, event: dispatched, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.747083 7f869af78700 5 -- op tracker -- , seq: 2828, time: 2014-06-09 16:22:11.747081, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.747090 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.25b) v4 from osd.0 2014-06-09 16:22:11.747093 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cafa40 2014-06-09 16:22:11.747098 7f869af78700 5 -- op tracker -- , seq: 2828, time: 2014-06-09 16:22:11.747096, event: started, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.747114 7f869af78700 5 -- op tracker -- , seq: 2828, time: 2014-06-09 16:22:11.747112, event: done, op: pg_info(1 pgs e243:0.25b) 2014-06-09 16:22:11.747122 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747124 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747136 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747138 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747139 7f869af78700 20 osd.26 243 _dispatch 0x30dae8c0 pg_info(1 pgs e243:2.52e) v4 2014-06-09 16:22:11.747144 7f869af78700 5 -- op tracker -- , seq: 2829, time: 2014-06-09 16:22:11.575100, event: header_read, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.747151 7f869af78700 5 -- op tracker -- , seq: 2829, time: 2014-06-09 16:22:11.575100, event: throttled, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.747157 7f869af78700 5 -- op tracker -- , seq: 2829, time: 2014-06-09 16:22:11.575114, event: all_read, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.747164 7f869af78700 5 -- op tracker -- , seq: 2829, time: 2014-06-09 16:22:11.747135, event: dispatched, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.747171 7f869af78700 5 -- op tracker -- , seq: 2829, time: 2014-06-09 16:22:11.747169, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.747180 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.52e) v4 from osd.5 2014-06-09 16:22:11.747183 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dae8c0 2014-06-09 16:22:11.747189 7f869af78700 5 -- op tracker -- , seq: 2829, time: 2014-06-09 16:22:11.747187, event: started, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.747207 7f869af78700 5 -- op tracker -- , seq: 2829, time: 2014-06-09 16:22:11.747204, event: done, op: pg_info(1 pgs e243:2.52e) 2014-06-09 16:22:11.747214 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747216 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747228 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747229 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747230 7f869af78700 20 osd.26 243 _dispatch 0x30cf8000 pg_info(1 pgs e243:3.4b3) v4 2014-06-09 16:22:11.747235 7f869af78700 5 -- op tracker -- , seq: 2830, time: 2014-06-09 16:22:11.322942, event: header_read, op: pg_info(1 pgs e243:3.4b3) 2014-06-09 16:22:11.747243 7f869af78700 5 -- op tracker -- , seq: 2830, time: 2014-06-09 16:22:11.322942, event: throttled, op: pg_info(1 pgs e243:3.4b3) 2014-06-09 16:22:11.747249 7f869af78700 5 -- op tracker -- , seq: 2830, time: 2014-06-09 16:22:11.322952, event: all_read, op: pg_info(1 pgs e243:3.4b3) 2014-06-09 16:22:11.747256 7f869af78700 5 -- op tracker -- , seq: 2830, time: 2014-06-09 16:22:11.747227, event: dispatched, op: pg_info(1 pgs e243:3.4b3) 2014-06-09 16:22:11.747262 7f869af78700 5 -- op tracker -- , seq: 2830, time: 2014-06-09 16:22:11.747260, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4b3) 2014-06-09 16:22:11.747269 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4b3) v4 from osd.2 2014-06-09 16:22:11.747272 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf8000 2014-06-09 16:22:11.747277 7f869af78700 5 -- op tracker -- , seq: 2830, time: 2014-06-09 16:22:11.747275, event: started, op: pg_info(1 pgs e243:3.4b3) 2014-06-09 16:22:11.747293 7f869af78700 5 -- op tracker -- , seq: 2830, time: 2014-06-09 16:22:11.747291, event: done, op: pg_info(1 pgs e243:3.4b3) 2014-06-09 16:22:11.747301 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747365 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747382 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747383 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747385 7f869af78700 20 osd.26 243 _dispatch 0x2eee7340 pg_info(1 pgs e243:1.779) v4 2014-06-09 16:22:11.747390 7f869af78700 5 -- op tracker -- , seq: 2831, time: 2014-06-09 16:22:11.439788, event: header_read, op: pg_info(1 pgs e243:1.779) 2014-06-09 16:22:11.747397 7f869af78700 5 -- op tracker -- , seq: 2831, time: 2014-06-09 16:22:11.439788, event: throttled, op: pg_info(1 pgs e243:1.779) 2014-06-09 16:22:11.747403 7f869af78700 5 -- op tracker -- , seq: 2831, time: 2014-06-09 16:22:11.439799, event: all_read, op: pg_info(1 pgs e243:1.779) 2014-06-09 16:22:11.747416 7f869af78700 5 -- op tracker -- , seq: 2831, time: 2014-06-09 16:22:11.747381, event: dispatched, op: pg_info(1 pgs e243:1.779) 2014-06-09 16:22:11.747427 7f869af78700 5 -- op tracker -- , seq: 2831, time: 2014-06-09 16:22:11.747424, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.779) 2014-06-09 16:22:11.747432 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.779) v4 from osd.4 2014-06-09 16:22:11.747435 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee7340 2014-06-09 16:22:11.747440 7f869af78700 5 -- op tracker -- , seq: 2831, time: 2014-06-09 16:22:11.747438, event: started, op: pg_info(1 pgs e243:1.779) 2014-06-09 16:22:11.747456 7f869af78700 5 -- op tracker -- , seq: 2831, time: 2014-06-09 16:22:11.747454, event: done, op: pg_info(1 pgs e243:1.779) 2014-06-09 16:22:11.747464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747466 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747482 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747483 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747484 7f869af78700 20 osd.26 243 _dispatch 0x2fcaac40 pg_info(1 pgs e243:2.298) v4 2014-06-09 16:22:11.747489 7f869af78700 5 -- op tracker -- , seq: 2832, time: 2014-06-09 16:22:11.610885, event: header_read, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.747496 7f869af78700 5 -- op tracker -- , seq: 2832, time: 2014-06-09 16:22:11.610885, event: throttled, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.747503 7f869af78700 5 -- op tracker -- , seq: 2832, time: 2014-06-09 16:22:11.610898, event: all_read, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.747515 7f869af78700 5 -- op tracker -- , seq: 2832, time: 2014-06-09 16:22:11.747481, event: dispatched, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.747526 7f869af78700 5 -- op tracker -- , seq: 2832, time: 2014-06-09 16:22:11.747524, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.747531 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.298) v4 from osd.7 2014-06-09 16:22:11.747534 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaac40 2014-06-09 16:22:11.747538 7f869af78700 5 -- op tracker -- , seq: 2832, time: 2014-06-09 16:22:11.747536, event: started, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.747554 7f869af78700 5 -- op tracker -- , seq: 2832, time: 2014-06-09 16:22:11.747551, event: done, op: pg_info(1 pgs e243:2.298) 2014-06-09 16:22:11.747564 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747566 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747578 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747580 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747581 7f869af78700 20 osd.26 243 _dispatch 0x30cf0c40 pg_info(1 pgs e243:1.211) v4 2014-06-09 16:22:11.747586 7f869af78700 5 -- op tracker -- , seq: 2833, time: 2014-06-09 16:22:11.590383, event: header_read, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.747593 7f869af78700 5 -- op tracker -- , seq: 2833, time: 2014-06-09 16:22:11.590384, event: throttled, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.747606 7f869af78700 5 -- op tracker -- , seq: 2833, time: 2014-06-09 16:22:11.590400, event: all_read, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.747613 7f869af78700 5 -- op tracker -- , seq: 2833, time: 2014-06-09 16:22:11.747578, event: dispatched, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.747636 7f869af78700 5 -- op tracker -- , seq: 2833, time: 2014-06-09 16:22:11.747634, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.747641 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.211) v4 from osd.3 2014-06-09 16:22:11.747644 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf0c40 2014-06-09 16:22:11.747649 7f869af78700 5 -- op tracker -- , seq: 2833, time: 2014-06-09 16:22:11.747647, event: started, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.747670 7f869af78700 5 -- op tracker -- , seq: 2833, time: 2014-06-09 16:22:11.747668, event: done, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.747677 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747680 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747692 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747694 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747695 7f869af78700 20 osd.26 243 _dispatch 0x1609d6c0 pg_info(1 pgs e243:2.69d) v4 2014-06-09 16:22:11.747700 7f869af78700 5 -- op tracker -- , seq: 2834, time: 2014-06-09 16:22:11.578036, event: header_read, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.747707 7f869af78700 5 -- op tracker -- , seq: 2834, time: 2014-06-09 16:22:11.578037, event: throttled, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.747713 7f869af78700 5 -- op tracker -- , seq: 2834, time: 2014-06-09 16:22:11.578048, event: all_read, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.747719 7f869af78700 5 -- op tracker -- , seq: 2834, time: 2014-06-09 16:22:11.747692, event: dispatched, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.747729 7f869af78700 5 -- op tracker -- , seq: 2834, time: 2014-06-09 16:22:11.747727, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.747734 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.69d) v4 from osd.1 2014-06-09 16:22:11.747737 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1609d6c0 2014-06-09 16:22:11.747742 7f869af78700 5 -- op tracker -- , seq: 2834, time: 2014-06-09 16:22:11.747740, event: started, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.747763 7f869af78700 5 -- op tracker -- , seq: 2834, time: 2014-06-09 16:22:11.747760, event: done, op: pg_info(1 pgs e243:2.69d) 2014-06-09 16:22:11.747771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747773 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747797 7f869af78700 20 osd.26 243 _dispatch 0x30cf0540 pg_info(1 pgs e243:3.77) v4 2014-06-09 16:22:11.747803 7f869af78700 5 -- op tracker -- , seq: 2835, time: 2014-06-09 16:22:11.593063, event: header_read, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.747815 7f869af78700 5 -- op tracker -- , seq: 2835, time: 2014-06-09 16:22:11.593063, event: throttled, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.747822 7f869af78700 5 -- op tracker -- , seq: 2835, time: 2014-06-09 16:22:11.593076, event: all_read, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.747828 7f869af78700 5 -- op tracker -- , seq: 2835, time: 2014-06-09 16:22:11.747794, event: dispatched, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.747835 7f869af78700 5 -- op tracker -- , seq: 2835, time: 2014-06-09 16:22:11.747833, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.747840 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.77) v4 from osd.9 2014-06-09 16:22:11.747843 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf0540 2014-06-09 16:22:11.747848 7f869af78700 5 -- op tracker -- , seq: 2835, time: 2014-06-09 16:22:11.747846, event: started, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.747865 7f869af78700 5 -- op tracker -- , seq: 2835, time: 2014-06-09 16:22:11.747862, event: done, op: pg_info(1 pgs e243:3.77) 2014-06-09 16:22:11.747872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747874 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747886 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747888 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747889 7f869af78700 20 osd.26 243 _dispatch 0x3044d880 pg_info(1 pgs e243:3.76) v4 2014-06-09 16:22:11.747894 7f869af78700 5 -- op tracker -- , seq: 2836, time: 2014-06-09 16:22:11.549584, event: header_read, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.747903 7f869af78700 5 -- op tracker -- , seq: 2836, time: 2014-06-09 16:22:11.549584, event: throttled, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.747910 7f869af78700 5 -- op tracker -- , seq: 2836, time: 2014-06-09 16:22:11.549597, event: all_read, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.747916 7f869af78700 5 -- op tracker -- , seq: 2836, time: 2014-06-09 16:22:11.747886, event: dispatched, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.747925 7f869af78700 5 -- op tracker -- , seq: 2836, time: 2014-06-09 16:22:11.747923, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.747930 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.76) v4 from osd.8 2014-06-09 16:22:11.747933 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3044d880 2014-06-09 16:22:11.747938 7f869af78700 5 -- op tracker -- , seq: 2836, time: 2014-06-09 16:22:11.747936, event: started, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.747955 7f869af78700 5 -- op tracker -- , seq: 2836, time: 2014-06-09 16:22:11.747953, event: done, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.747964 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747967 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747978 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.747980 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.747981 7f869af78700 20 osd.26 243 _dispatch 0x2176dc00 pg_info(1 pgs e243:3.7a) v4 2014-06-09 16:22:11.747986 7f869af78700 5 -- op tracker -- , seq: 2837, time: 2014-06-09 16:22:11.574379, event: header_read, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.747993 7f869af78700 5 -- op tracker -- , seq: 2837, time: 2014-06-09 16:22:11.574379, event: throttled, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748001 7f869af78700 5 -- op tracker -- , seq: 2837, time: 2014-06-09 16:22:11.574391, event: all_read, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748008 7f869af78700 5 -- op tracker -- , seq: 2837, time: 2014-06-09 16:22:11.747978, event: dispatched, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748015 7f869af78700 5 -- op tracker -- , seq: 2837, time: 2014-06-09 16:22:11.748013, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748021 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7a) v4 from osd.6 2014-06-09 16:22:11.748024 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176dc00 2014-06-09 16:22:11.748030 7f869af78700 5 -- op tracker -- , seq: 2837, time: 2014-06-09 16:22:11.748028, event: started, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748046 7f869af78700 5 -- op tracker -- , seq: 2837, time: 2014-06-09 16:22:11.748044, event: done, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748058 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748060 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748072 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748074 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748076 7f869af78700 20 osd.26 243 _dispatch 0x190b61c0 pg_info(1 pgs e243:2.5be) v4 2014-06-09 16:22:11.748081 7f869af78700 5 -- op tracker -- , seq: 2838, time: 2014-06-09 16:22:11.604524, event: header_read, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.748090 7f869af78700 5 -- op tracker -- , seq: 2838, time: 2014-06-09 16:22:11.604525, event: throttled, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.748097 7f869af78700 5 -- op tracker -- , seq: 2838, time: 2014-06-09 16:22:11.604537, event: all_read, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.748106 7f869af78700 5 -- op tracker -- , seq: 2838, time: 2014-06-09 16:22:11.748072, event: dispatched, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.748113 7f869af78700 5 -- op tracker -- , seq: 2838, time: 2014-06-09 16:22:11.748111, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.748118 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5be) v4 from osd.0 2014-06-09 16:22:11.748121 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b61c0 2014-06-09 16:22:11.748126 7f869af78700 5 -- op tracker -- , seq: 2838, time: 2014-06-09 16:22:11.748124, event: started, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.748144 7f869af78700 5 -- op tracker -- , seq: 2838, time: 2014-06-09 16:22:11.748141, event: done, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.748152 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748154 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748165 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748167 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748168 7f869af78700 20 osd.26 243 _dispatch 0x30cf1340 pg_info(1 pgs e243:0.186) v4 2014-06-09 16:22:11.748173 7f869af78700 5 -- op tracker -- , seq: 2839, time: 2014-06-09 16:22:11.577342, event: header_read, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.748180 7f869af78700 5 -- op tracker -- , seq: 2839, time: 2014-06-09 16:22:11.577342, event: throttled, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.748192 7f869af78700 5 -- op tracker -- , seq: 2839, time: 2014-06-09 16:22:11.577355, event: all_read, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.748199 7f869af78700 5 -- op tracker -- , seq: 2839, time: 2014-06-09 16:22:11.748165, event: dispatched, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.748209 7f869af78700 5 -- op tracker -- , seq: 2839, time: 2014-06-09 16:22:11.748207, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.748214 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.186) v4 from osd.5 2014-06-09 16:22:11.748217 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf1340 2014-06-09 16:22:11.748222 7f869af78700 5 -- op tracker -- , seq: 2839, time: 2014-06-09 16:22:11.748220, event: started, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.748238 7f869af78700 5 -- op tracker -- , seq: 2839, time: 2014-06-09 16:22:11.748235, event: done, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.748249 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748251 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748264 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748266 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748267 7f869af78700 20 osd.26 243 _dispatch 0x280c56c0 pg_info(1 pgs e243:0.1f1) v4 2014-06-09 16:22:11.748272 7f869af78700 5 -- op tracker -- , seq: 2840, time: 2014-06-09 16:22:11.322955, event: header_read, op: pg_info(1 pgs e243:0.1f1) 2014-06-09 16:22:11.748280 7f869af78700 5 -- op tracker -- , seq: 2840, time: 2014-06-09 16:22:11.322955, event: throttled, op: pg_info(1 pgs e243:0.1f1) 2014-06-09 16:22:11.748287 7f869af78700 5 -- op tracker -- , seq: 2840, time: 2014-06-09 16:22:11.322968, event: all_read, op: pg_info(1 pgs e243:0.1f1) 2014-06-09 16:22:11.748293 7f869af78700 5 -- op tracker -- , seq: 2840, time: 2014-06-09 16:22:11.748264, event: dispatched, op: pg_info(1 pgs e243:0.1f1) 2014-06-09 16:22:11.748300 7f869af78700 5 -- op tracker -- , seq: 2840, time: 2014-06-09 16:22:11.748298, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1f1) 2014-06-09 16:22:11.748305 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1f1) v4 from osd.2 2014-06-09 16:22:11.748308 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c56c0 2014-06-09 16:22:11.748312 7f869af78700 5 -- op tracker -- , seq: 2840, time: 2014-06-09 16:22:11.748310, event: started, op: pg_info(1 pgs e243:0.1f1) 2014-06-09 16:22:11.748332 7f869af78700 5 -- op tracker -- , seq: 2840, time: 2014-06-09 16:22:11.748329, event: done, op: pg_info(1 pgs e243:0.1f1) 2014-06-09 16:22:11.748342 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748344 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748358 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748360 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748361 7f869af78700 20 osd.26 243 _dispatch 0x2d735e00 pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.748368 7f869af78700 5 -- op tracker -- , seq: 2841, time: 2014-06-09 16:22:11.439804, event: header_read, op: pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.748376 7f869af78700 5 -- op tracker -- , seq: 2841, time: 2014-06-09 16:22:11.439804, event: throttled, op: pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.748384 7f869af78700 5 -- op tracker -- , seq: 2841, time: 2014-06-09 16:22:11.439819, event: all_read, op: pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.748391 7f869af78700 5 -- op tracker -- , seq: 2841, time: 2014-06-09 16:22:11.748358, event: dispatched, op: pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.748404 7f869af78700 5 -- op tracker -- , seq: 2841, time: 2014-06-09 16:22:11.748401, event: waiting_for_osdmap, op: pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.748409 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) v4 from osd.4 2014-06-09 16:22:11.748413 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d735e00 2014-06-09 16:22:11.748420 7f869af78700 5 -- op tracker -- , seq: 2841, time: 2014-06-09 16:22:11.748417, event: started, op: pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.748434 7f869af78700 5 -- op tracker -- , seq: 2841, time: 2014-06-09 16:22:11.748431, event: done, op: pg_log(3.21e epoch 243 log log((213'615,221'617], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.748446 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748448 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748460 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748462 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748463 7f869af78700 20 osd.26 243 _dispatch 0xfadc540 pg_info(1 pgs e243:3.461) v4 2014-06-09 16:22:11.748468 7f869af78700 5 -- op tracker -- , seq: 2842, time: 2014-06-09 16:22:11.610921, event: header_read, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.748478 7f869af78700 5 -- op tracker -- , seq: 2842, time: 2014-06-09 16:22:11.610922, event: throttled, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.748484 7f869af78700 5 -- op tracker -- , seq: 2842, time: 2014-06-09 16:22:11.610934, event: all_read, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.748491 7f869af78700 5 -- op tracker -- , seq: 2842, time: 2014-06-09 16:22:11.748459, event: dispatched, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.748498 7f869af78700 5 -- op tracker -- , seq: 2842, time: 2014-06-09 16:22:11.748496, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.748502 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.461) v4 from osd.7 2014-06-09 16:22:11.748505 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xfadc540 2014-06-09 16:22:11.748510 7f869af78700 5 -- op tracker -- , seq: 2842, time: 2014-06-09 16:22:11.748508, event: started, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.748527 7f869af78700 5 -- op tracker -- , seq: 2842, time: 2014-06-09 16:22:11.748525, event: done, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.748535 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748537 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748549 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748551 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748553 7f869af78700 20 osd.26 243 _dispatch 0x30c31dc0 pg_info(1 pgs e243:1.211) v4 2014-06-09 16:22:11.748558 7f869af78700 5 -- op tracker -- , seq: 2843, time: 2014-06-09 16:22:11.590451, event: header_read, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.748565 7f869af78700 5 -- op tracker -- , seq: 2843, time: 2014-06-09 16:22:11.590452, event: throttled, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.748572 7f869af78700 5 -- op tracker -- , seq: 2843, time: 2014-06-09 16:22:11.590465, event: all_read, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.748583 7f869af78700 5 -- op tracker -- , seq: 2843, time: 2014-06-09 16:22:11.748549, event: dispatched, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.748594 7f869af78700 5 -- op tracker -- , seq: 2843, time: 2014-06-09 16:22:11.748592, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.748600 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.211) v4 from osd.3 2014-06-09 16:22:11.748603 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c31dc0 2014-06-09 16:22:11.748607 7f869af78700 5 -- op tracker -- , seq: 2843, time: 2014-06-09 16:22:11.748606, event: started, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.748622 7f869af78700 5 -- op tracker -- , seq: 2843, time: 2014-06-09 16:22:11.748620, event: done, op: pg_info(1 pgs e243:1.211) 2014-06-09 16:22:11.748630 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748632 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748644 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748645 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748647 7f869af78700 20 osd.26 243 _dispatch 0x30cf08c0 pg_info(1 pgs e243:2.126) v4 2014-06-09 16:22:11.748652 7f869af78700 5 -- op tracker -- , seq: 2844, time: 2014-06-09 16:22:11.590896, event: header_read, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.748659 7f869af78700 5 -- op tracker -- , seq: 2844, time: 2014-06-09 16:22:11.590897, event: throttled, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.748666 7f869af78700 5 -- op tracker -- , seq: 2844, time: 2014-06-09 16:22:11.590910, event: all_read, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.748675 7f869af78700 5 -- op tracker -- , seq: 2844, time: 2014-06-09 16:22:11.748643, event: dispatched, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.748682 7f869af78700 5 -- op tracker -- , seq: 2844, time: 2014-06-09 16:22:11.748680, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.748686 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.126) v4 from osd.1 2014-06-09 16:22:11.748689 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf08c0 2014-06-09 16:22:11.748694 7f869af78700 5 -- op tracker -- , seq: 2844, time: 2014-06-09 16:22:11.748692, event: started, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.748710 7f869af78700 5 -- op tracker -- , seq: 2844, time: 2014-06-09 16:22:11.748708, event: done, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.748717 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748719 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748735 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748736 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748738 7f869af78700 20 osd.26 243 _dispatch 0xddc5340 pg_info(1 pgs e243:0.219) v4 2014-06-09 16:22:11.748743 7f869af78700 5 -- op tracker -- , seq: 2845, time: 2014-06-09 16:22:11.593112, event: header_read, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.748750 7f869af78700 5 -- op tracker -- , seq: 2845, time: 2014-06-09 16:22:11.593112, event: throttled, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.748756 7f869af78700 5 -- op tracker -- , seq: 2845, time: 2014-06-09 16:22:11.593125, event: all_read, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.748762 7f869af78700 5 -- op tracker -- , seq: 2845, time: 2014-06-09 16:22:11.748734, event: dispatched, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.748769 7f869af78700 5 -- op tracker -- , seq: 2845, time: 2014-06-09 16:22:11.748767, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.748773 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.219) v4 from osd.9 2014-06-09 16:22:11.748776 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc5340 2014-06-09 16:22:11.748781 7f869af78700 5 -- op tracker -- , seq: 2845, time: 2014-06-09 16:22:11.748779, event: started, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.748798 7f869af78700 5 -- op tracker -- , seq: 2845, time: 2014-06-09 16:22:11.748796, event: done, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.748806 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748808 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748819 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748821 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748822 7f869af78700 20 osd.26 243 _dispatch 0x2eef1500 pg_info(1 pgs e243:3.76) v4 2014-06-09 16:22:11.748827 7f869af78700 5 -- op tracker -- , seq: 2846, time: 2014-06-09 16:22:11.549607, event: header_read, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.748834 7f869af78700 5 -- op tracker -- , seq: 2846, time: 2014-06-09 16:22:11.549607, event: throttled, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.748840 7f869af78700 5 -- op tracker -- , seq: 2846, time: 2014-06-09 16:22:11.549619, event: all_read, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.748847 7f869af78700 5 -- op tracker -- , seq: 2846, time: 2014-06-09 16:22:11.748819, event: dispatched, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.748854 7f869af78700 5 -- op tracker -- , seq: 2846, time: 2014-06-09 16:22:11.748852, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.748859 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.76) v4 from osd.8 2014-06-09 16:22:11.748862 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef1500 2014-06-09 16:22:11.748875 7f869af78700 5 -- op tracker -- , seq: 2846, time: 2014-06-09 16:22:11.748864, event: started, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.748898 7f869af78700 5 -- op tracker -- , seq: 2846, time: 2014-06-09 16:22:11.748895, event: done, op: pg_info(1 pgs e243:3.76) 2014-06-09 16:22:11.748906 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748908 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748920 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.748922 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.748923 7f869af78700 20 osd.26 243 _dispatch 0x31162540 pg_info(1 pgs e243:3.7a) v4 2014-06-09 16:22:11.748929 7f869af78700 5 -- op tracker -- , seq: 2847, time: 2014-06-09 16:22:11.574893, event: header_read, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748938 7f869af78700 5 -- op tracker -- , seq: 2847, time: 2014-06-09 16:22:11.574893, event: throttled, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748945 7f869af78700 5 -- op tracker -- , seq: 2847, time: 2014-06-09 16:22:11.574906, event: all_read, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748951 7f869af78700 5 -- op tracker -- , seq: 2847, time: 2014-06-09 16:22:11.748920, event: dispatched, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748959 7f869af78700 5 -- op tracker -- , seq: 2847, time: 2014-06-09 16:22:11.748957, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748966 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7a) v4 from osd.6 2014-06-09 16:22:11.748969 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31162540 2014-06-09 16:22:11.748974 7f869af78700 5 -- op tracker -- , seq: 2847, time: 2014-06-09 16:22:11.748972, event: started, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748990 7f869af78700 5 -- op tracker -- , seq: 2847, time: 2014-06-09 16:22:11.748988, event: done, op: pg_info(1 pgs e243:3.7a) 2014-06-09 16:22:11.748998 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.749000 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.749012 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.749014 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750505 7f869af78700 20 osd.26 243 _dispatch 0x2fc95dc0 pg_info(1 pgs e243:2.5be) v4 2014-06-09 16:22:11.750516 7f8670072700 5 -- op tracker -- , seq: 2848, time: 2014-06-09 16:22:11.746517, event: header_read, op: osd_op(client.4538.0:36795 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 798720~1548288] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.750529 7f8670072700 5 -- op tracker -- , seq: 2848, time: 2014-06-09 16:22:11.746517, event: throttled, op: osd_op(client.4538.0:36795 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 798720~1548288] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.750539 7f8670072700 5 -- op tracker -- , seq: 2848, time: 2014-06-09 16:22:11.749518, event: all_read, op: osd_op(client.4538.0:36795 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 798720~1548288] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.750549 7f8670072700 5 -- op tracker -- , seq: 2848, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36795 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 798720~1548288] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.750556 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.750557 7f869af78700 5 -- op tracker -- , seq: 2849, time: 2014-06-09 16:22:11.604595, event: header_read, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.750564 7f869af78700 5 -- op tracker -- , seq: 2849, time: 2014-06-09 16:22:11.604596, event: throttled, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.750570 7f869af78700 5 -- op tracker -- , seq: 2849, time: 2014-06-09 16:22:11.604611, event: all_read, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.750576 7f869af78700 5 -- op tracker -- , seq: 2849, time: 2014-06-09 16:22:11.749012, event: dispatched, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.750577 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6500 prio 63 cost 1548288 latency 0.004060 osd_op(client.4538.0:36795 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 798720~1548288] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.750583 7f869af78700 5 -- op tracker -- , seq: 2849, time: 2014-06-09 16:22:11.750581, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.750588 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5be) v4 from osd.0 2014-06-09 16:22:11.750591 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc95dc0 2014-06-09 16:22:11.750595 7f869af78700 5 -- op tracker -- , seq: 2849, time: 2014-06-09 16:22:11.750593, event: started, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.750611 7f869af78700 5 -- op tracker -- , seq: 2849, time: 2014-06-09 16:22:11.750609, event: done, op: pg_info(1 pgs e243:2.5be) 2014-06-09 16:22:11.750622 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750624 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750639 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750640 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750641 7f869af78700 20 osd.26 243 _dispatch 0x30dae700 pg_info(1 pgs e243:0.186) v4 2014-06-09 16:22:11.750646 7f869af78700 5 -- op tracker -- , seq: 2850, time: 2014-06-09 16:22:11.577414, event: header_read, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.750654 7f869af78700 5 -- op tracker -- , seq: 2850, time: 2014-06-09 16:22:11.577415, event: throttled, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.750660 7f869af78700 5 -- op tracker -- , seq: 2850, time: 2014-06-09 16:22:11.577427, event: all_read, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.750666 7f869af78700 5 -- op tracker -- , seq: 2850, time: 2014-06-09 16:22:11.750638, event: dispatched, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.750678 7f869af78700 5 -- op tracker -- , seq: 2850, time: 2014-06-09 16:22:11.750676, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.750684 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.186) v4 from osd.5 2014-06-09 16:22:11.750696 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dae700 2014-06-09 16:22:11.750702 7f869af78700 5 -- op tracker -- , seq: 2850, time: 2014-06-09 16:22:11.750700, event: started, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.750717 7f869af78700 5 -- op tracker -- , seq: 2850, time: 2014-06-09 16:22:11.750715, event: done, op: pg_info(1 pgs e243:0.186) 2014-06-09 16:22:11.750724 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750726 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750742 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750744 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750745 7f869af78700 20 osd.26 243 _dispatch 0x280c5340 pg_info(1 pgs e243:2.b) v4 2014-06-09 16:22:11.750750 7f869af78700 5 -- op tracker -- , seq: 2851, time: 2014-06-09 16:22:11.322972, event: header_read, op: pg_info(1 pgs e243:2.b) 2014-06-09 16:22:11.750759 7f869af78700 5 -- op tracker -- , seq: 2851, time: 2014-06-09 16:22:11.322972, event: throttled, op: pg_info(1 pgs e243:2.b) 2014-06-09 16:22:11.750766 7f869af78700 5 -- op tracker -- , seq: 2851, time: 2014-06-09 16:22:11.322981, event: all_read, op: pg_info(1 pgs e243:2.b) 2014-06-09 16:22:11.750772 7f869af78700 5 -- op tracker -- , seq: 2851, time: 2014-06-09 16:22:11.750742, event: dispatched, op: pg_info(1 pgs e243:2.b) 2014-06-09 16:22:11.750778 7f869af78700 5 -- op tracker -- , seq: 2851, time: 2014-06-09 16:22:11.750776, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.b) 2014-06-09 16:22:11.750784 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.b) v4 from osd.2 2014-06-09 16:22:11.750787 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c5340 2014-06-09 16:22:11.750792 7f869af78700 5 -- op tracker -- , seq: 2851, time: 2014-06-09 16:22:11.750790, event: started, op: pg_info(1 pgs e243:2.b) 2014-06-09 16:22:11.750810 7f869af78700 5 -- op tracker -- , seq: 2851, time: 2014-06-09 16:22:11.750808, event: done, op: pg_info(1 pgs e243:2.b) 2014-06-09 16:22:11.750818 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750819 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750831 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750833 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750834 7f869af78700 20 osd.26 243 _dispatch 0x2eee7180 pg_info(1 pgs e243:0.19a) v4 2014-06-09 16:22:11.750838 7f869af78700 5 -- op tracker -- , seq: 2852, time: 2014-06-09 16:22:11.439824, event: header_read, op: pg_info(1 pgs e243:0.19a) 2014-06-09 16:22:11.750845 7f869af78700 5 -- op tracker -- , seq: 2852, time: 2014-06-09 16:22:11.439824, event: throttled, op: pg_info(1 pgs e243:0.19a) 2014-06-09 16:22:11.750853 7f869af78700 5 -- op tracker -- , seq: 2852, time: 2014-06-09 16:22:11.439834, event: all_read, op: pg_info(1 pgs e243:0.19a) 2014-06-09 16:22:11.750860 7f869af78700 5 -- op tracker -- , seq: 2852, time: 2014-06-09 16:22:11.750831, event: dispatched, op: pg_info(1 pgs e243:0.19a) 2014-06-09 16:22:11.750868 7f869af78700 5 -- op tracker -- , seq: 2852, time: 2014-06-09 16:22:11.750866, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.19a) 2014-06-09 16:22:11.750872 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.19a) v4 from osd.4 2014-06-09 16:22:11.750875 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee7180 2014-06-09 16:22:11.750880 7f869af78700 5 -- op tracker -- , seq: 2852, time: 2014-06-09 16:22:11.750878, event: started, op: pg_info(1 pgs e243:0.19a) 2014-06-09 16:22:11.750894 7f869af78700 5 -- op tracker -- , seq: 2852, time: 2014-06-09 16:22:11.750892, event: done, op: pg_info(1 pgs e243:0.19a) 2014-06-09 16:22:11.750902 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750903 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750915 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750917 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.750918 7f869af78700 20 osd.26 243 _dispatch 0x2fcaa380 pg_info(1 pgs e243:3.461) v4 2014-06-09 16:22:11.750922 7f869af78700 5 -- op tracker -- , seq: 2853, time: 2014-06-09 16:22:11.611386, event: header_read, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.750934 7f869af78700 5 -- op tracker -- , seq: 2853, time: 2014-06-09 16:22:11.611386, event: throttled, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.750942 7f869af78700 5 -- op tracker -- , seq: 2853, time: 2014-06-09 16:22:11.611397, event: all_read, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.750949 7f869af78700 5 -- op tracker -- , seq: 2853, time: 2014-06-09 16:22:11.750915, event: dispatched, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.750957 7f869af78700 5 -- op tracker -- , seq: 2853, time: 2014-06-09 16:22:11.750955, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.750961 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.461) v4 from osd.7 2014-06-09 16:22:11.750964 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaa380 2014-06-09 16:22:11.750968 7f869af78700 5 -- op tracker -- , seq: 2853, time: 2014-06-09 16:22:11.750967, event: started, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.750986 7f869af78700 5 -- op tracker -- , seq: 2853, time: 2014-06-09 16:22:11.750984, event: done, op: pg_info(1 pgs e243:3.461) 2014-06-09 16:22:11.750993 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.750995 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751007 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751008 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751009 7f869af78700 20 osd.26 243 _dispatch 0x30de56c0 pg_info(1 pgs e243:0.220) v4 2014-06-09 16:22:11.751014 7f869af78700 5 -- op tracker -- , seq: 2854, time: 2014-06-09 16:22:11.592730, event: header_read, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751023 7f869af78700 5 -- op tracker -- , seq: 2854, time: 2014-06-09 16:22:11.592731, event: throttled, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751028 7f869af78700 5 -- op tracker -- , seq: 2854, time: 2014-06-09 16:22:11.592744, event: all_read, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751035 7f869af78700 5 -- op tracker -- , seq: 2854, time: 2014-06-09 16:22:11.751006, event: dispatched, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751043 7f869af78700 5 -- op tracker -- , seq: 2854, time: 2014-06-09 16:22:11.751041, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751047 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.220) v4 from osd.3 2014-06-09 16:22:11.751050 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de56c0 2014-06-09 16:22:11.751055 7f869af78700 5 -- op tracker -- , seq: 2854, time: 2014-06-09 16:22:11.751053, event: started, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751069 7f869af78700 5 -- op tracker -- , seq: 2854, time: 2014-06-09 16:22:11.751067, event: done, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751076 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751090 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751091 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751093 7f869af78700 20 osd.26 243 _dispatch 0x2d095dc0 pg_info(1 pgs e243:2.126) v4 2014-06-09 16:22:11.751098 7f869af78700 5 -- op tracker -- , seq: 2855, time: 2014-06-09 16:22:11.591533, event: header_read, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.751110 7f869af78700 5 -- op tracker -- , seq: 2855, time: 2014-06-09 16:22:11.591533, event: throttled, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.751119 7f869af78700 5 -- op tracker -- , seq: 2855, time: 2014-06-09 16:22:11.591545, event: all_read, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.751125 7f869af78700 5 -- op tracker -- , seq: 2855, time: 2014-06-09 16:22:11.751090, event: dispatched, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.751132 7f869af78700 5 -- op tracker -- , seq: 2855, time: 2014-06-09 16:22:11.751130, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.751138 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.126) v4 from osd.1 2014-06-09 16:22:11.751141 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d095dc0 2014-06-09 16:22:11.751145 7f869af78700 5 -- op tracker -- , seq: 2855, time: 2014-06-09 16:22:11.751144, event: started, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.751170 7f869af78700 5 -- op tracker -- , seq: 2855, time: 2014-06-09 16:22:11.751168, event: done, op: pg_info(1 pgs e243:2.126) 2014-06-09 16:22:11.751190 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751192 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751204 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751206 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751207 7f869af78700 20 osd.26 243 _dispatch 0xddc5180 pg_info(1 pgs e243:0.219) v4 2014-06-09 16:22:11.751211 7f869af78700 5 -- op tracker -- , seq: 2856, time: 2014-06-09 16:22:11.593165, event: header_read, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.751218 7f869af78700 5 -- op tracker -- , seq: 2856, time: 2014-06-09 16:22:11.593166, event: throttled, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.751224 7f869af78700 5 -- op tracker -- , seq: 2856, time: 2014-06-09 16:22:11.593178, event: all_read, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.751232 7f869af78700 5 -- op tracker -- , seq: 2856, time: 2014-06-09 16:22:11.751204, event: dispatched, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.751238 7f869af78700 5 -- op tracker -- , seq: 2856, time: 2014-06-09 16:22:11.751237, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.751244 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.219) v4 from osd.9 2014-06-09 16:22:11.751247 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc5180 2014-06-09 16:22:11.751252 7f869af78700 5 -- op tracker -- , seq: 2856, time: 2014-06-09 16:22:11.751250, event: started, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.751266 7f869af78700 5 -- op tracker -- , seq: 2856, time: 2014-06-09 16:22:11.751264, event: done, op: pg_info(1 pgs e243:0.219) 2014-06-09 16:22:11.751275 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751277 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751290 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751291 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751292 7f869af78700 20 osd.26 243 _dispatch 0x30dc3340 pg_info(1 pgs e243:1.348) v4 2014-06-09 16:22:11.751297 7f869af78700 5 -- op tracker -- , seq: 2857, time: 2014-06-09 16:22:11.567634, event: header_read, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.751304 7f869af78700 5 -- op tracker -- , seq: 2857, time: 2014-06-09 16:22:11.567635, event: throttled, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.751312 7f869af78700 5 -- op tracker -- , seq: 2857, time: 2014-06-09 16:22:11.567648, event: all_read, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.751319 7f869af78700 5 -- op tracker -- , seq: 2857, time: 2014-06-09 16:22:11.751289, event: dispatched, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.751328 7f869af78700 5 -- op tracker -- , seq: 2857, time: 2014-06-09 16:22:11.751325, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.751333 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.348) v4 from osd.8 2014-06-09 16:22:11.751336 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc3340 2014-06-09 16:22:11.751341 7f869af78700 5 -- op tracker -- , seq: 2857, time: 2014-06-09 16:22:11.751339, event: started, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.751356 7f869af78700 5 -- op tracker -- , seq: 2857, time: 2014-06-09 16:22:11.751354, event: done, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.751373 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751375 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751386 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751387 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751388 7f869af78700 20 osd.26 243 _dispatch 0x31162380 pg_info(1 pgs e243:1.11d) v4 2014-06-09 16:22:11.751392 7f869af78700 5 -- op tracker -- , seq: 2858, time: 2014-06-09 16:22:11.574940, event: header_read, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.751398 7f869af78700 5 -- op tracker -- , seq: 2858, time: 2014-06-09 16:22:11.574940, event: throttled, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.751404 7f869af78700 5 -- op tracker -- , seq: 2858, time: 2014-06-09 16:22:11.574955, event: all_read, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.751409 7f869af78700 5 -- op tracker -- , seq: 2858, time: 2014-06-09 16:22:11.751386, event: dispatched, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.751421 7f869af78700 5 -- op tracker -- , seq: 2858, time: 2014-06-09 16:22:11.751415, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.751426 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.11d) v4 from osd.6 2014-06-09 16:22:11.751428 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31162380 2014-06-09 16:22:11.751433 7f869af78700 5 -- op tracker -- , seq: 2858, time: 2014-06-09 16:22:11.751431, event: started, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.751455 7f869af78700 5 -- op tracker -- , seq: 2858, time: 2014-06-09 16:22:11.751453, event: done, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.751464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751466 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751479 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751480 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751481 7f869af78700 20 osd.26 243 _dispatch 0x30a3b500 pg_info(1 pgs e243:1.71e) v4 2014-06-09 16:22:11.751485 7f869af78700 5 -- op tracker -- , seq: 2859, time: 2014-06-09 16:22:11.605216, event: header_read, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.751492 7f869af78700 5 -- op tracker -- , seq: 2859, time: 2014-06-09 16:22:11.605216, event: throttled, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.751497 7f869af78700 5 -- op tracker -- , seq: 2859, time: 2014-06-09 16:22:11.605228, event: all_read, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.751504 7f869af78700 5 -- op tracker -- , seq: 2859, time: 2014-06-09 16:22:11.751478, event: dispatched, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.751511 7f869af78700 5 -- op tracker -- , seq: 2859, time: 2014-06-09 16:22:11.751509, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.751515 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.71e) v4 from osd.0 2014-06-09 16:22:11.751518 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3b500 2014-06-09 16:22:11.751523 7f869af78700 5 -- op tracker -- , seq: 2859, time: 2014-06-09 16:22:11.751521, event: started, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.751540 7f869af78700 5 -- op tracker -- , seq: 2859, time: 2014-06-09 16:22:11.751538, event: done, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.751549 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751551 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751564 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751565 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751567 7f869af78700 20 osd.26 243 _dispatch 0x30cae8c0 pg_info(1 pgs e243:0.151) v4 2014-06-09 16:22:11.751572 7f869af78700 5 -- op tracker -- , seq: 2860, time: 2014-06-09 16:22:11.582929, event: header_read, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.751580 7f869af78700 5 -- op tracker -- , seq: 2860, time: 2014-06-09 16:22:11.582929, event: throttled, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.751587 7f869af78700 5 -- op tracker -- , seq: 2860, time: 2014-06-09 16:22:11.582943, event: all_read, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.751595 7f869af78700 5 -- op tracker -- , seq: 2860, time: 2014-06-09 16:22:11.751563, event: dispatched, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.751602 7f869af78700 5 -- op tracker -- , seq: 2860, time: 2014-06-09 16:22:11.751599, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.751612 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.151) v4 from osd.5 2014-06-09 16:22:11.751632 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cae8c0 2014-06-09 16:22:11.751638 7f869af78700 5 -- op tracker -- , seq: 2860, time: 2014-06-09 16:22:11.751636, event: started, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.751655 7f869af78700 5 -- op tracker -- , seq: 2860, time: 2014-06-09 16:22:11.751653, event: done, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.751666 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751668 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751692 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751694 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751696 7f869af78700 20 osd.26 243 _dispatch 0x280c5500 pg_info(1 pgs e243:2.755) v4 2014-06-09 16:22:11.751701 7f869af78700 5 -- op tracker -- , seq: 2861, time: 2014-06-09 16:22:11.322984, event: header_read, op: pg_info(1 pgs e243:2.755) 2014-06-09 16:22:11.751708 7f869af78700 5 -- op tracker -- , seq: 2861, time: 2014-06-09 16:22:11.322985, event: throttled, op: pg_info(1 pgs e243:2.755) 2014-06-09 16:22:11.751714 7f869af78700 5 -- op tracker -- , seq: 2861, time: 2014-06-09 16:22:11.322999, event: all_read, op: pg_info(1 pgs e243:2.755) 2014-06-09 16:22:11.751720 7f869af78700 5 -- op tracker -- , seq: 2861, time: 2014-06-09 16:22:11.751692, event: dispatched, op: pg_info(1 pgs e243:2.755) 2014-06-09 16:22:11.751726 7f869af78700 5 -- op tracker -- , seq: 2861, time: 2014-06-09 16:22:11.751724, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.755) 2014-06-09 16:22:11.751731 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.755) v4 from osd.2 2014-06-09 16:22:11.751734 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c5500 2014-06-09 16:22:11.751739 7f869af78700 5 -- op tracker -- , seq: 2861, time: 2014-06-09 16:22:11.751737, event: started, op: pg_info(1 pgs e243:2.755) 2014-06-09 16:22:11.751780 7f869af78700 5 -- op tracker -- , seq: 2861, time: 2014-06-09 16:22:11.751777, event: done, op: pg_info(1 pgs e243:2.755) 2014-06-09 16:22:11.751794 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751805 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751806 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751807 7f869af78700 20 osd.26 243 _dispatch 0x2eee6fc0 pg_info(1 pgs e243:2.5fb) v4 2014-06-09 16:22:11.751810 7f869af78700 5 -- op tracker -- , seq: 2862, time: 2014-06-09 16:22:11.439838, event: header_read, op: pg_info(1 pgs e243:2.5fb) 2014-06-09 16:22:11.751820 7f869af78700 5 -- op tracker -- , seq: 2862, time: 2014-06-09 16:22:11.439839, event: throttled, op: pg_info(1 pgs e243:2.5fb) 2014-06-09 16:22:11.751825 7f869af78700 5 -- op tracker -- , seq: 2862, time: 2014-06-09 16:22:11.439849, event: all_read, op: pg_info(1 pgs e243:2.5fb) 2014-06-09 16:22:11.751829 7f869af78700 5 -- op tracker -- , seq: 2862, time: 2014-06-09 16:22:11.751805, event: dispatched, op: pg_info(1 pgs e243:2.5fb) 2014-06-09 16:22:11.751833 7f869af78700 5 -- op tracker -- , seq: 2862, time: 2014-06-09 16:22:11.751832, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5fb) 2014-06-09 16:22:11.751835 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5fb) v4 from osd.4 2014-06-09 16:22:11.751837 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee6fc0 2014-06-09 16:22:11.751840 7f869af78700 5 -- op tracker -- , seq: 2862, time: 2014-06-09 16:22:11.751839, event: started, op: pg_info(1 pgs e243:2.5fb) 2014-06-09 16:22:11.751849 7f869af78700 5 -- op tracker -- , seq: 2862, time: 2014-06-09 16:22:11.751848, event: done, op: pg_info(1 pgs e243:2.5fb) 2014-06-09 16:22:11.751854 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751855 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751862 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751863 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751863 7f869af78700 20 osd.26 243 _dispatch 0x15c94a80 pg_info(1 pgs e243:3.468) v4 2014-06-09 16:22:11.751866 7f869af78700 5 -- op tracker -- , seq: 2863, time: 2014-06-09 16:22:11.611450, event: header_read, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.751870 7f869af78700 5 -- op tracker -- , seq: 2863, time: 2014-06-09 16:22:11.611451, event: throttled, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.751873 7f869af78700 5 -- op tracker -- , seq: 2863, time: 2014-06-09 16:22:11.611464, event: all_read, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.751876 7f869af78700 5 -- op tracker -- , seq: 2863, time: 2014-06-09 16:22:11.751862, event: dispatched, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.751880 7f869af78700 5 -- op tracker -- , seq: 2863, time: 2014-06-09 16:22:11.751879, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.751882 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.468) v4 from osd.7 2014-06-09 16:22:11.751884 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15c94a80 2014-06-09 16:22:11.751898 7f869af78700 5 -- op tracker -- , seq: 2863, time: 2014-06-09 16:22:11.751885, event: started, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.751916 7f869af78700 5 -- op tracker -- , seq: 2863, time: 2014-06-09 16:22:11.751914, event: done, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.751927 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751928 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751936 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751937 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.751938 7f869af78700 20 osd.26 243 _dispatch 0x30c31c00 pg_info(1 pgs e243:0.220) v4 2014-06-09 16:22:11.751941 7f869af78700 5 -- op tracker -- , seq: 2864, time: 2014-06-09 16:22:11.592777, event: header_read, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751944 7f869af78700 5 -- op tracker -- , seq: 2864, time: 2014-06-09 16:22:11.592778, event: throttled, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751948 7f869af78700 5 -- op tracker -- , seq: 2864, time: 2014-06-09 16:22:11.592791, event: all_read, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751951 7f869af78700 5 -- op tracker -- , seq: 2864, time: 2014-06-09 16:22:11.751936, event: dispatched, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751954 7f869af78700 5 -- op tracker -- , seq: 2864, time: 2014-06-09 16:22:11.751953, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751966 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.220) v4 from osd.3 2014-06-09 16:22:11.751969 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c31c00 2014-06-09 16:22:11.751974 7f869af78700 5 -- op tracker -- , seq: 2864, time: 2014-06-09 16:22:11.751972, event: started, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751989 7f869af78700 5 -- op tracker -- , seq: 2864, time: 2014-06-09 16:22:11.751987, event: done, op: pg_info(1 pgs e243:0.220) 2014-06-09 16:22:11.751997 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.751999 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752012 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752013 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752014 7f869af78700 20 osd.26 243 _dispatch 0x30cae1c0 pg_info(1 pgs e243:1.73b) v4 2014-06-09 16:22:11.752017 7f869af78700 5 -- op tracker -- , seq: 2865, time: 2014-06-09 16:22:11.596932, event: header_read, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.752023 7f869af78700 5 -- op tracker -- , seq: 2865, time: 2014-06-09 16:22:11.596932, event: throttled, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.752043 7f869af78700 5 -- op tracker -- , seq: 2865, time: 2014-06-09 16:22:11.596946, event: all_read, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.752050 7f869af78700 5 -- op tracker -- , seq: 2865, time: 2014-06-09 16:22:11.752012, event: dispatched, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.752057 7f869af78700 5 -- op tracker -- , seq: 2865, time: 2014-06-09 16:22:11.752055, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.752061 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.73b) v4 from osd.1 2014-06-09 16:22:11.752064 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cae1c0 2014-06-09 16:22:11.752069 7f869af78700 5 -- op tracker -- , seq: 2865, time: 2014-06-09 16:22:11.752067, event: started, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.752085 7f869af78700 5 -- op tracker -- , seq: 2865, time: 2014-06-09 16:22:11.752083, event: done, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.752093 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752094 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752102 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752115 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752117 7f869af78700 20 osd.26 243 _dispatch 0x2a070000 pg_info(1 pgs e243:3.537) v4 2014-06-09 16:22:11.752122 7f869af78700 5 -- op tracker -- , seq: 2866, time: 2014-06-09 16:22:11.596930, event: header_read, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.752129 7f869af78700 5 -- op tracker -- , seq: 2866, time: 2014-06-09 16:22:11.596930, event: throttled, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.752135 7f869af78700 5 -- op tracker -- , seq: 2866, time: 2014-06-09 16:22:11.596944, event: all_read, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.752142 7f869af78700 5 -- op tracker -- , seq: 2866, time: 2014-06-09 16:22:11.752102, event: dispatched, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.752149 7f869af78700 5 -- op tracker -- , seq: 2866, time: 2014-06-09 16:22:11.752147, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.752153 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.537) v4 from osd.9 2014-06-09 16:22:11.752156 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a070000 2014-06-09 16:22:11.752162 7f869af78700 5 -- op tracker -- , seq: 2866, time: 2014-06-09 16:22:11.752160, event: started, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.752178 7f869af78700 5 -- op tracker -- , seq: 2866, time: 2014-06-09 16:22:11.752176, event: done, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.752192 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752194 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752205 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752207 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752208 7f869af78700 20 osd.26 243 _dispatch 0x30eba1c0 pg_info(1 pgs e243:1.2ef) v4 2014-06-09 16:22:11.752213 7f869af78700 5 -- op tracker -- , seq: 2867, time: 2014-06-09 16:22:11.567698, event: header_read, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.752221 7f869af78700 5 -- op tracker -- , seq: 2867, time: 2014-06-09 16:22:11.567699, event: throttled, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.752228 7f869af78700 5 -- op tracker -- , seq: 2867, time: 2014-06-09 16:22:11.567711, event: all_read, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.752234 7f869af78700 5 -- op tracker -- , seq: 2867, time: 2014-06-09 16:22:11.752205, event: dispatched, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.752238 7f869af78700 5 -- op tracker -- , seq: 2867, time: 2014-06-09 16:22:11.752237, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.752245 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2ef) v4 from osd.8 2014-06-09 16:22:11.752247 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30eba1c0 2014-06-09 16:22:11.752250 7f869af78700 5 -- op tracker -- , seq: 2867, time: 2014-06-09 16:22:11.752249, event: started, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.752278 7f869af78700 5 -- op tracker -- , seq: 2867, time: 2014-06-09 16:22:11.752276, event: done, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.752287 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752289 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752301 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752302 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752303 7f869af78700 20 osd.26 243 _dispatch 0x311621c0 pg_info(1 pgs e243:1.11d) v4 2014-06-09 16:22:11.752310 7f8670072700 5 -- op tracker -- , seq: 2868, time: 2014-06-09 16:22:11.750598, event: header_read, op: osd_op(client.4538.0:36796 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 2347008~1032192] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.752345 7f8670072700 5 -- op tracker -- , seq: 2868, time: 2014-06-09 16:22:11.750599, event: throttled, op: osd_op(client.4538.0:36796 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 2347008~1032192] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.752360 7f8670072700 5 -- op tracker -- , seq: 2868, time: 2014-06-09 16:22:11.752276, event: all_read, op: osd_op(client.4538.0:36796 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 2347008~1032192] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.752371 7f8670072700 5 -- op tracker -- , seq: 2868, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36796 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 2347008~1032192] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.752381 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.752400 7f869af78700 5 -- op tracker -- , seq: 2869, time: 2014-06-09 16:22:11.574995, event: header_read, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.752401 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6d00 prio 63 cost 1032192 latency 0.001803 osd_op(client.4538.0:36796 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 2347008~1032192] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.752412 7f869af78700 5 -- op tracker -- , seq: 2869, time: 2014-06-09 16:22:11.574995, event: throttled, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.752420 7f869af78700 5 -- op tracker -- , seq: 2869, time: 2014-06-09 16:22:11.575007, event: all_read, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.752424 7f869af78700 5 -- op tracker -- , seq: 2869, time: 2014-06-09 16:22:11.752300, event: dispatched, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.752428 7f869af78700 5 -- op tracker -- , seq: 2869, time: 2014-06-09 16:22:11.752427, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.752430 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.11d) v4 from osd.6 2014-06-09 16:22:11.752432 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x311621c0 2014-06-09 16:22:11.752435 7f869af78700 5 -- op tracker -- , seq: 2869, time: 2014-06-09 16:22:11.752434, event: started, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.752456 7f869af78700 5 -- op tracker -- , seq: 2869, time: 2014-06-09 16:22:11.752454, event: done, op: pg_info(1 pgs e243:1.11d) 2014-06-09 16:22:11.752465 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752467 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752482 7f869af78700 20 osd.26 243 _dispatch 0x2fc95c00 pg_info(1 pgs e243:1.71e) v4 2014-06-09 16:22:11.752491 7f869af78700 5 -- op tracker -- , seq: 2870, time: 2014-06-09 16:22:11.606102, event: header_read, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.752496 7f869af78700 5 -- op tracker -- , seq: 2870, time: 2014-06-09 16:22:11.606104, event: throttled, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.752500 7f869af78700 5 -- op tracker -- , seq: 2870, time: 2014-06-09 16:22:11.606115, event: all_read, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.752504 7f869af78700 5 -- op tracker -- , seq: 2870, time: 2014-06-09 16:22:11.752480, event: dispatched, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.752507 7f869af78700 5 -- op tracker -- , seq: 2870, time: 2014-06-09 16:22:11.752506, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.752510 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.71e) v4 from osd.0 2014-06-09 16:22:11.752511 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc95c00 2014-06-09 16:22:11.752514 7f869af78700 5 -- op tracker -- , seq: 2870, time: 2014-06-09 16:22:11.752513, event: started, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.752523 7f869af78700 5 -- op tracker -- , seq: 2870, time: 2014-06-09 16:22:11.752522, event: done, op: pg_info(1 pgs e243:1.71e) 2014-06-09 16:22:11.752529 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752531 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752543 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752544 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752546 7f869af78700 20 osd.26 243 _dispatch 0x30dae540 pg_info(1 pgs e243:0.151) v4 2014-06-09 16:22:11.752551 7f869af78700 5 -- op tracker -- , seq: 2871, time: 2014-06-09 16:22:11.582952, event: header_read, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.752557 7f869af78700 5 -- op tracker -- , seq: 2871, time: 2014-06-09 16:22:11.582952, event: throttled, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.752563 7f869af78700 5 -- op tracker -- , seq: 2871, time: 2014-06-09 16:22:11.582973, event: all_read, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.752569 7f869af78700 5 -- op tracker -- , seq: 2871, time: 2014-06-09 16:22:11.752543, event: dispatched, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.752576 7f869af78700 5 -- op tracker -- , seq: 2871, time: 2014-06-09 16:22:11.752573, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.752582 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.151) v4 from osd.5 2014-06-09 16:22:11.752584 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dae540 2014-06-09 16:22:11.752587 7f869af78700 5 -- op tracker -- , seq: 2871, time: 2014-06-09 16:22:11.752586, event: started, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.752597 7f869af78700 5 -- op tracker -- , seq: 2871, time: 2014-06-09 16:22:11.752596, event: done, op: pg_info(1 pgs e243:0.151) 2014-06-09 16:22:11.752602 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752603 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752611 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752612 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752612 7f869af78700 20 osd.26 243 _dispatch 0x2d732700 pg_info(1 pgs e243:1.1f0) v4 2014-06-09 16:22:11.752615 7f869af78700 5 -- op tracker -- , seq: 2872, time: 2014-06-09 16:22:11.323003, event: header_read, op: pg_info(1 pgs e243:1.1f0) 2014-06-09 16:22:11.752620 7f869af78700 5 -- op tracker -- , seq: 2872, time: 2014-06-09 16:22:11.323003, event: throttled, op: pg_info(1 pgs e243:1.1f0) 2014-06-09 16:22:11.752626 7f869af78700 5 -- op tracker -- , seq: 2872, time: 2014-06-09 16:22:11.323012, event: all_read, op: pg_info(1 pgs e243:1.1f0) 2014-06-09 16:22:11.752635 7f869af78700 5 -- op tracker -- , seq: 2872, time: 2014-06-09 16:22:11.752611, event: dispatched, op: pg_info(1 pgs e243:1.1f0) 2014-06-09 16:22:11.752642 7f869af78700 5 -- op tracker -- , seq: 2872, time: 2014-06-09 16:22:11.752640, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1f0) 2014-06-09 16:22:11.752656 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1f0) v4 from osd.2 2014-06-09 16:22:11.752660 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d732700 2014-06-09 16:22:11.752667 7f869af78700 5 -- op tracker -- , seq: 2872, time: 2014-06-09 16:22:11.752664, event: started, op: pg_info(1 pgs e243:1.1f0) 2014-06-09 16:22:11.752689 7f869af78700 5 -- op tracker -- , seq: 2872, time: 2014-06-09 16:22:11.752687, event: done, op: pg_info(1 pgs e243:1.1f0) 2014-06-09 16:22:11.752698 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752701 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752717 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752718 7f869af78700 20 osd.26 243 _dispatch 0x2eee6e00 pg_info(1 pgs e243:0.4b3) v4 2014-06-09 16:22:11.752723 7f869af78700 5 -- op tracker -- , seq: 2873, time: 2014-06-09 16:22:11.447636, event: header_read, op: pg_info(1 pgs e243:0.4b3) 2014-06-09 16:22:11.752732 7f869af78700 5 -- op tracker -- , seq: 2873, time: 2014-06-09 16:22:11.447638, event: throttled, op: pg_info(1 pgs e243:0.4b3) 2014-06-09 16:22:11.752741 7f869af78700 5 -- op tracker -- , seq: 2873, time: 2014-06-09 16:22:11.447653, event: all_read, op: pg_info(1 pgs e243:0.4b3) 2014-06-09 16:22:11.752747 7f869af78700 5 -- op tracker -- , seq: 2873, time: 2014-06-09 16:22:11.752714, event: dispatched, op: pg_info(1 pgs e243:0.4b3) 2014-06-09 16:22:11.752755 7f869af78700 5 -- op tracker -- , seq: 2873, time: 2014-06-09 16:22:11.752753, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4b3) 2014-06-09 16:22:11.752765 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4b3) v4 from osd.4 2014-06-09 16:22:11.752768 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee6e00 2014-06-09 16:22:11.752773 7f869af78700 5 -- op tracker -- , seq: 2873, time: 2014-06-09 16:22:11.752771, event: started, op: pg_info(1 pgs e243:0.4b3) 2014-06-09 16:22:11.752790 7f869af78700 5 -- op tracker -- , seq: 2873, time: 2014-06-09 16:22:11.752788, event: done, op: pg_info(1 pgs e243:0.4b3) 2014-06-09 16:22:11.752801 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752803 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752816 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752818 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752820 7f869af78700 20 osd.26 243 _dispatch 0x30caf880 pg_info(1 pgs e243:3.468) v4 2014-06-09 16:22:11.752825 7f869af78700 5 -- op tracker -- , seq: 2874, time: 2014-06-09 16:22:11.611914, event: header_read, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.752832 7f869af78700 5 -- op tracker -- , seq: 2874, time: 2014-06-09 16:22:11.611915, event: throttled, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.752838 7f869af78700 5 -- op tracker -- , seq: 2874, time: 2014-06-09 16:22:11.611926, event: all_read, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.752851 7f869af78700 5 -- op tracker -- , seq: 2874, time: 2014-06-09 16:22:11.752816, event: dispatched, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.752858 7f869af78700 5 -- op tracker -- , seq: 2874, time: 2014-06-09 16:22:11.752856, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.752863 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.468) v4 from osd.7 2014-06-09 16:22:11.752866 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30caf880 2014-06-09 16:22:11.752871 7f869af78700 5 -- op tracker -- , seq: 2874, time: 2014-06-09 16:22:11.752869, event: started, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.752888 7f869af78700 5 -- op tracker -- , seq: 2874, time: 2014-06-09 16:22:11.752886, event: done, op: pg_info(1 pgs e243:3.468) 2014-06-09 16:22:11.752897 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752899 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752911 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.752913 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.752914 7f869af78700 20 osd.26 243 _dispatch 0x30de5340 pg_info(1 pgs e243:1.5c1) v4 2014-06-09 16:22:11.752919 7f869af78700 5 -- op tracker -- , seq: 2875, time: 2014-06-09 16:22:11.603373, event: header_read, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.752929 7f869af78700 5 -- op tracker -- , seq: 2875, time: 2014-06-09 16:22:11.603374, event: throttled, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.753019 7f869af78700 5 -- op tracker -- , seq: 2875, time: 2014-06-09 16:22:11.603388, event: all_read, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.755651 7f869af78700 5 -- op tracker -- , seq: 2875, time: 2014-06-09 16:22:11.752911, event: dispatched, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.755661 7f869af78700 5 -- op tracker -- , seq: 2875, time: 2014-06-09 16:22:11.755659, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.755667 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5c1) v4 from osd.3 2014-06-09 16:22:11.755671 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5340 2014-06-09 16:22:11.755679 7f8670072700 5 -- op tracker -- , seq: 2876, time: 2014-06-09 16:22:11.752426, event: header_read, op: osd_op(client.4538.0:36798 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 3379200~815104] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.755694 7f8670072700 5 -- op tracker -- , seq: 2876, time: 2014-06-09 16:22:11.752428, event: throttled, op: osd_op(client.4538.0:36798 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 3379200~815104] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.755705 7f8670072700 5 -- op tracker -- , seq: 2876, time: 2014-06-09 16:22:11.753809, event: all_read, op: osd_op(client.4538.0:36798 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 3379200~815104] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.755717 7f8670072700 5 -- op tracker -- , seq: 2876, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36798 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 3379200~815104] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.755728 7f8682ae5700 5 -- op tracker -- , seq: 2877, time: 2014-06-09 16:22:11.755353, event: header_read, op: osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.755738 7f8682ae5700 5 -- op tracker -- , seq: 2877, time: 2014-06-09 16:22:11.755354, event: throttled, op: osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.755748 7f8682ae5700 5 -- op tracker -- , seq: 2877, time: 2014-06-09 16:22:11.755363, event: all_read, op: osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.755757 7f8682ae5700 5 -- op tracker -- , seq: 2877, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.755767 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.755769 7f869af78700 5 -- op tracker -- , seq: 2875, time: 2014-06-09 16:22:11.755675, event: started, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.755773 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.755780 7f8682ae5700 15 osd.26 243 enqueue_op 0xdfa7100 prio 196 cost 0 latency 0.000427 osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.755790 7f869af78700 5 -- op tracker -- , seq: 2875, time: 2014-06-09 16:22:11.755788, event: done, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.755800 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.755803 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.755801 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.755818 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.755820 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.755821 7f869af78700 20 osd.26 243 _dispatch 0x2d094700 pg_info(1 pgs e243:1.73b) v4 2014-06-09 16:22:11.755827 7f869af78700 5 -- op tracker -- , seq: 2878, time: 2014-06-09 16:22:11.596971, event: header_read, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.755825 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6c00 prio 63 cost 815104 latency 0.003399 osd_op(client.4538.0:36798 rbd_data.115a6b8b4567.0000000000001a53 [set-alloc-hint object_size 4194304 write_size 4194304,write 3379200~815104] 3.fa167d82 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.755834 7f869af78700 5 -- op tracker -- , seq: 2878, time: 2014-06-09 16:22:11.596972, event: throttled, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.755841 7f869af78700 5 -- op tracker -- , seq: 2878, time: 2014-06-09 16:22:11.596984, event: all_read, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.755848 7f869af78700 5 -- op tracker -- , seq: 2878, time: 2014-06-09 16:22:11.755817, event: dispatched, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.755855 7f869af78700 5 -- op tracker -- , seq: 2878, time: 2014-06-09 16:22:11.755853, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.755860 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.73b) v4 from osd.1 2014-06-09 16:22:11.755863 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d094700 2014-06-09 16:22:11.755868 7f869af78700 5 -- op tracker -- , seq: 2878, time: 2014-06-09 16:22:11.755866, event: started, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.755883 7f869af78700 5 -- op tracker -- , seq: 2878, time: 2014-06-09 16:22:11.755881, event: done, op: pg_info(1 pgs e243:1.73b) 2014-06-09 16:22:11.755894 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.755896 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.755909 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.755911 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.755912 7f869af78700 20 osd.26 243 _dispatch 0x1fc048c0 pg_info(1 pgs e243:3.537) v4 2014-06-09 16:22:11.755917 7f869af78700 5 -- op tracker -- , seq: 2879, time: 2014-06-09 16:22:11.596956, event: header_read, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.755924 7f869af78700 5 -- op tracker -- , seq: 2879, time: 2014-06-09 16:22:11.596956, event: throttled, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.755933 7f869af78700 5 -- op tracker -- , seq: 2879, time: 2014-06-09 16:22:11.596967, event: all_read, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.755940 7f869af78700 5 -- op tracker -- , seq: 2879, time: 2014-06-09 16:22:11.755908, event: dispatched, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.755947 7f869af78700 5 -- op tracker -- , seq: 2879, time: 2014-06-09 16:22:11.755945, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.755953 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.537) v4 from osd.9 2014-06-09 16:22:11.755956 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc048c0 2014-06-09 16:22:11.755961 7f869af78700 5 -- op tracker -- , seq: 2879, time: 2014-06-09 16:22:11.755960, event: started, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.755984 7f869af78700 5 -- op tracker -- , seq: 2879, time: 2014-06-09 16:22:11.755982, event: done, op: pg_info(1 pgs e243:3.537) 2014-06-09 16:22:11.755992 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.755994 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756007 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756008 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756010 7f869af78700 20 osd.26 243 _dispatch 0x30ebaa80 pg_info(1 pgs e243:3.616) v4 2014-06-09 16:22:11.756015 7f869af78700 5 -- op tracker -- , seq: 2880, time: 2014-06-09 16:22:11.567826, event: header_read, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.756023 7f869af78700 5 -- op tracker -- , seq: 2880, time: 2014-06-09 16:22:11.567827, event: throttled, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.756030 7f869af78700 5 -- op tracker -- , seq: 2880, time: 2014-06-09 16:22:11.567839, event: all_read, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.756036 7f869af78700 5 -- op tracker -- , seq: 2880, time: 2014-06-09 16:22:11.756006, event: dispatched, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.756051 7f869af78700 5 -- op tracker -- , seq: 2880, time: 2014-06-09 16:22:11.756048, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.756055 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.616) v4 from osd.8 2014-06-09 16:22:11.756058 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebaa80 2014-06-09 16:22:11.756064 7f869af78700 5 -- op tracker -- , seq: 2880, time: 2014-06-09 16:22:11.756062, event: started, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.756080 7f869af78700 5 -- op tracker -- , seq: 2880, time: 2014-06-09 16:22:11.756078, event: done, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.756088 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756090 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756106 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756107 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756109 7f869af78700 20 osd.26 243 _dispatch 0x30cf1180 pg_info(1 pgs e243:0.5a2) v4 2014-06-09 16:22:11.756114 7f869af78700 5 -- op tracker -- , seq: 2881, time: 2014-06-09 16:22:11.582600, event: header_read, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.756121 7f869af78700 5 -- op tracker -- , seq: 2881, time: 2014-06-09 16:22:11.582601, event: throttled, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.756145 7f869af78700 5 -- op tracker -- , seq: 2881, time: 2014-06-09 16:22:11.582614, event: all_read, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.756153 7f869af78700 5 -- op tracker -- , seq: 2881, time: 2014-06-09 16:22:11.756105, event: dispatched, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.756160 7f869af78700 5 -- op tracker -- , seq: 2881, time: 2014-06-09 16:22:11.756158, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.756165 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5a2) v4 from osd.6 2014-06-09 16:22:11.756168 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf1180 2014-06-09 16:22:11.756173 7f869af78700 5 -- op tracker -- , seq: 2881, time: 2014-06-09 16:22:11.756171, event: started, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.756189 7f869af78700 5 -- op tracker -- , seq: 2881, time: 2014-06-09 16:22:11.756187, event: done, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.756197 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756199 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756213 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756214 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756216 7f869af78700 20 osd.26 243 _dispatch 0x30de48c0 pg_info(1 pgs e243:1.113) v4 2014-06-09 16:22:11.756221 7f869af78700 5 -- op tracker -- , seq: 2882, time: 2014-06-09 16:22:11.614717, event: header_read, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.756239 7f869af78700 5 -- op tracker -- , seq: 2882, time: 2014-06-09 16:22:11.614717, event: throttled, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.756246 7f869af78700 5 -- op tracker -- , seq: 2882, time: 2014-06-09 16:22:11.614725, event: all_read, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.756252 7f869af78700 5 -- op tracker -- , seq: 2882, time: 2014-06-09 16:22:11.756212, event: dispatched, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.756261 7f869af78700 5 -- op tracker -- , seq: 2882, time: 2014-06-09 16:22:11.756259, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.756266 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.113) v4 from osd.0 2014-06-09 16:22:11.756269 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de48c0 2014-06-09 16:22:11.756275 7f869af78700 5 -- op tracker -- , seq: 2882, time: 2014-06-09 16:22:11.756273, event: started, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.756293 7f869af78700 5 -- op tracker -- , seq: 2882, time: 2014-06-09 16:22:11.756291, event: done, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.756301 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756303 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756316 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756317 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756319 7f869af78700 20 osd.26 243 _dispatch 0x30cf0000 pg_info(1 pgs e243:1.3cb) v4 2014-06-09 16:22:11.756324 7f869af78700 5 -- op tracker -- , seq: 2883, time: 2014-06-09 16:22:11.593085, event: header_read, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.756331 7f869af78700 5 -- op tracker -- , seq: 2883, time: 2014-06-09 16:22:11.593086, event: throttled, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.756343 7f869af78700 5 -- op tracker -- , seq: 2883, time: 2014-06-09 16:22:11.593099, event: all_read, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.756353 7f869af78700 5 -- op tracker -- , seq: 2883, time: 2014-06-09 16:22:11.756315, event: dispatched, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.756365 7f869af78700 5 -- op tracker -- , seq: 2883, time: 2014-06-09 16:22:11.756363, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.756370 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3cb) v4 from osd.5 2014-06-09 16:22:11.756373 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf0000 2014-06-09 16:22:11.756379 7f869af78700 5 -- op tracker -- , seq: 2883, time: 2014-06-09 16:22:11.756377, event: started, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.756395 7f869af78700 5 -- op tracker -- , seq: 2883, time: 2014-06-09 16:22:11.756393, event: done, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.756403 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756405 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756418 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756421 7f869af78700 20 osd.26 243 _dispatch 0x2d732a80 pg_info(1 pgs e243:1.449) v4 2014-06-09 16:22:11.756426 7f869af78700 5 -- op tracker -- , seq: 2884, time: 2014-06-09 16:22:11.323016, event: header_read, op: pg_info(1 pgs e243:1.449) 2014-06-09 16:22:11.756434 7f869af78700 5 -- op tracker -- , seq: 2884, time: 2014-06-09 16:22:11.323016, event: throttled, op: pg_info(1 pgs e243:1.449) 2014-06-09 16:22:11.756446 7f869af78700 5 -- op tracker -- , seq: 2884, time: 2014-06-09 16:22:11.323026, event: all_read, op: pg_info(1 pgs e243:1.449) 2014-06-09 16:22:11.756453 7f869af78700 5 -- op tracker -- , seq: 2884, time: 2014-06-09 16:22:11.756418, event: dispatched, op: pg_info(1 pgs e243:1.449) 2014-06-09 16:22:11.756465 7f869af78700 5 -- op tracker -- , seq: 2884, time: 2014-06-09 16:22:11.756463, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.449) 2014-06-09 16:22:11.756473 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.449) v4 from osd.2 2014-06-09 16:22:11.756477 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d732a80 2014-06-09 16:22:11.756482 7f869af78700 5 -- op tracker -- , seq: 2884, time: 2014-06-09 16:22:11.756480, event: started, op: pg_info(1 pgs e243:1.449) 2014-06-09 16:22:11.756501 7f869af78700 5 -- op tracker -- , seq: 2884, time: 2014-06-09 16:22:11.756499, event: done, op: pg_info(1 pgs e243:1.449) 2014-06-09 16:22:11.756513 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756516 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756533 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756535 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756536 7f869af78700 20 osd.26 243 _dispatch 0x2eee6c40 pg_info(1 pgs e243:2.186) v4 2014-06-09 16:22:11.756556 7f869af78700 5 -- op tracker -- , seq: 2885, time: 2014-06-09 16:22:11.447665, event: header_read, op: pg_info(1 pgs e243:2.186) 2014-06-09 16:22:11.756564 7f869af78700 5 -- op tracker -- , seq: 2885, time: 2014-06-09 16:22:11.447665, event: throttled, op: pg_info(1 pgs e243:2.186) 2014-06-09 16:22:11.756571 7f869af78700 5 -- op tracker -- , seq: 2885, time: 2014-06-09 16:22:11.447676, event: all_read, op: pg_info(1 pgs e243:2.186) 2014-06-09 16:22:11.756577 7f869af78700 5 -- op tracker -- , seq: 2885, time: 2014-06-09 16:22:11.756532, event: dispatched, op: pg_info(1 pgs e243:2.186) 2014-06-09 16:22:11.756587 7f869af78700 5 -- op tracker -- , seq: 2885, time: 2014-06-09 16:22:11.756585, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.186) 2014-06-09 16:22:11.756592 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.186) v4 from osd.4 2014-06-09 16:22:11.756595 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee6c40 2014-06-09 16:22:11.756600 7f869af78700 5 -- op tracker -- , seq: 2885, time: 2014-06-09 16:22:11.756598, event: started, op: pg_info(1 pgs e243:2.186) 2014-06-09 16:22:11.756619 7f869af78700 5 -- op tracker -- , seq: 2885, time: 2014-06-09 16:22:11.756617, event: done, op: pg_info(1 pgs e243:2.186) 2014-06-09 16:22:11.756631 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756633 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756654 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756656 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756657 7f869af78700 20 osd.26 243 _dispatch 0x30de4a80 pg_info(1 pgs e243:0.4a7) v4 2014-06-09 16:22:11.756662 7f869af78700 5 -- op tracker -- , seq: 2886, time: 2014-06-09 16:22:11.612420, event: header_read, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.756670 7f869af78700 5 -- op tracker -- , seq: 2886, time: 2014-06-09 16:22:11.612420, event: throttled, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.756676 7f869af78700 5 -- op tracker -- , seq: 2886, time: 2014-06-09 16:22:11.612429, event: all_read, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.756683 7f869af78700 5 -- op tracker -- , seq: 2886, time: 2014-06-09 16:22:11.756653, event: dispatched, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.756695 7f869af78700 5 -- op tracker -- , seq: 2886, time: 2014-06-09 16:22:11.756693, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.756700 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4a7) v4 from osd.7 2014-06-09 16:22:11.756703 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de4a80 2014-06-09 16:22:11.756708 7f869af78700 5 -- op tracker -- , seq: 2886, time: 2014-06-09 16:22:11.756706, event: started, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.756725 7f869af78700 5 -- op tracker -- , seq: 2886, time: 2014-06-09 16:22:11.756723, event: done, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.756733 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756735 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756748 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756750 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756751 7f869af78700 20 osd.26 243 _dispatch 0x30c31a40 pg_info(1 pgs e243:1.5c1) v4 2014-06-09 16:22:11.756756 7f869af78700 5 -- op tracker -- , seq: 2887, time: 2014-06-09 16:22:11.603396, event: header_read, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.756766 7f869af78700 5 -- op tracker -- , seq: 2887, time: 2014-06-09 16:22:11.603396, event: throttled, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.756773 7f869af78700 5 -- op tracker -- , seq: 2887, time: 2014-06-09 16:22:11.603409, event: all_read, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.756779 7f869af78700 5 -- op tracker -- , seq: 2887, time: 2014-06-09 16:22:11.756748, event: dispatched, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.756788 7f869af78700 5 -- op tracker -- , seq: 2887, time: 2014-06-09 16:22:11.756786, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.756793 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5c1) v4 from osd.3 2014-06-09 16:22:11.756796 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c31a40 2014-06-09 16:22:11.756801 7f869af78700 5 -- op tracker -- , seq: 2887, time: 2014-06-09 16:22:11.756799, event: started, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.756818 7f869af78700 5 -- op tracker -- , seq: 2887, time: 2014-06-09 16:22:11.756816, event: done, op: pg_info(1 pgs e243:1.5c1) 2014-06-09 16:22:11.756826 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756828 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756841 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756843 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.756844 7f869af78700 20 osd.26 243 _dispatch 0x30a3b180 pg_info(1 pgs e243:1.27c) v4 2014-06-09 16:22:11.756849 7f869af78700 5 -- op tracker -- , seq: 2888, time: 2014-06-09 16:22:11.607185, event: header_read, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.756856 7f869af78700 5 -- op tracker -- , seq: 2888, time: 2014-06-09 16:22:11.607185, event: throttled, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.756863 7f869af78700 5 -- op tracker -- , seq: 2888, time: 2014-06-09 16:22:11.607196, event: all_read, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.756871 7f869af78700 5 -- op tracker -- , seq: 2888, time: 2014-06-09 16:22:11.756841, event: dispatched, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.756878 7f869af78700 5 -- op tracker -- , seq: 2888, time: 2014-06-09 16:22:11.756876, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.756947 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.27c) v4 from osd.1 2014-06-09 16:22:11.756950 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3b180 2014-06-09 16:22:11.756956 7f869af78700 5 -- op tracker -- , seq: 2888, time: 2014-06-09 16:22:11.756954, event: started, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.756978 7f869af78700 5 -- op tracker -- , seq: 2888, time: 2014-06-09 16:22:11.756976, event: done, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.756987 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.756989 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757001 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757003 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757004 7f869af78700 20 osd.26 243 _dispatch 0x190b6a80 pg_info(1 pgs e243:3.53d) v4 2014-06-09 16:22:11.757008 7f869af78700 5 -- op tracker -- , seq: 2889, time: 2014-06-09 16:22:11.597094, event: header_read, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757014 7f869af78700 5 -- op tracker -- , seq: 2889, time: 2014-06-09 16:22:11.597094, event: throttled, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757020 7f869af78700 5 -- op tracker -- , seq: 2889, time: 2014-06-09 16:22:11.597105, event: all_read, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757025 7f869af78700 5 -- op tracker -- , seq: 2889, time: 2014-06-09 16:22:11.757001, event: dispatched, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757031 7f869af78700 5 -- op tracker -- , seq: 2889, time: 2014-06-09 16:22:11.757029, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757035 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.53d) v4 from osd.9 2014-06-09 16:22:11.757038 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b6a80 2014-06-09 16:22:11.757042 7f869af78700 5 -- op tracker -- , seq: 2889, time: 2014-06-09 16:22:11.757040, event: started, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757056 7f869af78700 5 -- op tracker -- , seq: 2889, time: 2014-06-09 16:22:11.757054, event: done, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757062 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757064 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757074 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757075 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757077 7f869af78700 20 osd.26 243 _dispatch 0x2eef1340 pg_info(1 pgs e243:1.348) v4 2014-06-09 16:22:11.757081 7f869af78700 5 -- op tracker -- , seq: 2890, time: 2014-06-09 16:22:11.567901, event: header_read, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.757087 7f869af78700 5 -- op tracker -- , seq: 2890, time: 2014-06-09 16:22:11.567902, event: throttled, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.757092 7f869af78700 5 -- op tracker -- , seq: 2890, time: 2014-06-09 16:22:11.567915, event: all_read, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.757097 7f869af78700 5 -- op tracker -- , seq: 2890, time: 2014-06-09 16:22:11.757074, event: dispatched, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.757102 7f869af78700 5 -- op tracker -- , seq: 2890, time: 2014-06-09 16:22:11.757101, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.757106 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.348) v4 from osd.8 2014-06-09 16:22:11.757109 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef1340 2014-06-09 16:22:11.757113 7f869af78700 5 -- op tracker -- , seq: 2890, time: 2014-06-09 16:22:11.757111, event: started, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.757128 7f869af78700 5 -- op tracker -- , seq: 2890, time: 2014-06-09 16:22:11.757126, event: done, op: pg_info(1 pgs e243:1.348) 2014-06-09 16:22:11.757143 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757145 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757155 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757157 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757158 7f869af78700 20 osd.26 243 _dispatch 0x31162000 pg_info(1 pgs e243:0.5a2) v4 2014-06-09 16:22:11.757161 7f869af78700 5 -- op tracker -- , seq: 2891, time: 2014-06-09 16:22:11.582645, event: header_read, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.757167 7f869af78700 5 -- op tracker -- , seq: 2891, time: 2014-06-09 16:22:11.582646, event: throttled, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.757172 7f869af78700 5 -- op tracker -- , seq: 2891, time: 2014-06-09 16:22:11.582658, event: all_read, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.757177 7f869af78700 5 -- op tracker -- , seq: 2891, time: 2014-06-09 16:22:11.757155, event: dispatched, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.757183 7f869af78700 5 -- op tracker -- , seq: 2891, time: 2014-06-09 16:22:11.757181, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.757187 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5a2) v4 from osd.6 2014-06-09 16:22:11.757189 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31162000 2014-06-09 16:22:11.757193 7f869af78700 5 -- op tracker -- , seq: 2891, time: 2014-06-09 16:22:11.757192, event: started, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.757206 7f869af78700 5 -- op tracker -- , seq: 2891, time: 2014-06-09 16:22:11.757204, event: done, op: pg_info(1 pgs e243:0.5a2) 2014-06-09 16:22:11.757212 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757214 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757228 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757229 7f869af78700 20 osd.26 243 _dispatch 0x2fcabdc0 pg_info(1 pgs e243:3.778) v4 2014-06-09 16:22:11.757234 7f869af78700 5 -- op tracker -- , seq: 2892, time: 2014-06-09 16:22:11.614751, event: header_read, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.757239 7f869af78700 5 -- op tracker -- , seq: 2892, time: 2014-06-09 16:22:11.614751, event: throttled, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.757245 7f869af78700 5 -- op tracker -- , seq: 2892, time: 2014-06-09 16:22:11.614765, event: all_read, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.757250 7f869af78700 5 -- op tracker -- , seq: 2892, time: 2014-06-09 16:22:11.757227, event: dispatched, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.757262 7f869af78700 5 -- op tracker -- , seq: 2892, time: 2014-06-09 16:22:11.757260, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.757269 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.778) v4 from osd.0 2014-06-09 16:22:11.757272 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcabdc0 2014-06-09 16:22:11.757277 7f869af78700 5 -- op tracker -- , seq: 2892, time: 2014-06-09 16:22:11.757275, event: started, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.757294 7f869af78700 5 -- op tracker -- , seq: 2892, time: 2014-06-09 16:22:11.757291, event: done, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.757302 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757304 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757316 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757318 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757319 7f869af78700 20 osd.26 243 _dispatch 0x30cf0700 pg_info(1 pgs e243:0.652) v4 2014-06-09 16:22:11.757324 7f869af78700 5 -- op tracker -- , seq: 2893, time: 2014-06-09 16:22:11.593139, event: header_read, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.757331 7f869af78700 5 -- op tracker -- , seq: 2893, time: 2014-06-09 16:22:11.593140, event: throttled, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.757340 7f869af78700 5 -- op tracker -- , seq: 2893, time: 2014-06-09 16:22:11.593151, event: all_read, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.757346 7f869af78700 5 -- op tracker -- , seq: 2893, time: 2014-06-09 16:22:11.757316, event: dispatched, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.757353 7f869af78700 5 -- op tracker -- , seq: 2893, time: 2014-06-09 16:22:11.757351, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.757359 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.652) v4 from osd.5 2014-06-09 16:22:11.757362 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf0700 2014-06-09 16:22:11.757367 7f869af78700 5 -- op tracker -- , seq: 2893, time: 2014-06-09 16:22:11.757365, event: started, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.757383 7f869af78700 5 -- op tracker -- , seq: 2893, time: 2014-06-09 16:22:11.757380, event: done, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.757392 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757394 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757407 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757408 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757410 7f869af78700 20 osd.26 243 _dispatch 0x2d732c40 pg_info(1 pgs e243:3.71f) v4 2014-06-09 16:22:11.757415 7f869af78700 5 -- op tracker -- , seq: 2894, time: 2014-06-09 16:22:11.323029, event: header_read, op: pg_info(1 pgs e243:3.71f) 2014-06-09 16:22:11.757427 7f869af78700 5 -- op tracker -- , seq: 2894, time: 2014-06-09 16:22:11.323029, event: throttled, op: pg_info(1 pgs e243:3.71f) 2014-06-09 16:22:11.757434 7f869af78700 5 -- op tracker -- , seq: 2894, time: 2014-06-09 16:22:11.323038, event: all_read, op: pg_info(1 pgs e243:3.71f) 2014-06-09 16:22:11.757442 7f869af78700 5 -- op tracker -- , seq: 2894, time: 2014-06-09 16:22:11.757406, event: dispatched, op: pg_info(1 pgs e243:3.71f) 2014-06-09 16:22:11.757449 7f869af78700 5 -- op tracker -- , seq: 2894, time: 2014-06-09 16:22:11.757447, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.71f) 2014-06-09 16:22:11.757454 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.71f) v4 from osd.2 2014-06-09 16:22:11.757457 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d732c40 2014-06-09 16:22:11.757461 7f869af78700 5 -- op tracker -- , seq: 2894, time: 2014-06-09 16:22:11.757460, event: started, op: pg_info(1 pgs e243:3.71f) 2014-06-09 16:22:11.757478 7f869af78700 5 -- op tracker -- , seq: 2894, time: 2014-06-09 16:22:11.757476, event: done, op: pg_info(1 pgs e243:3.71f) 2014-06-09 16:22:11.757486 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757488 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757500 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757501 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757502 7f869af78700 20 osd.26 243 _dispatch 0x2eee6a80 pg_info(1 pgs e243:1.76c) v4 2014-06-09 16:22:11.757507 7f869af78700 5 -- op tracker -- , seq: 2895, time: 2014-06-09 16:22:11.447681, event: header_read, op: pg_info(1 pgs e243:1.76c) 2014-06-09 16:22:11.757517 7f869af78700 5 -- op tracker -- , seq: 2895, time: 2014-06-09 16:22:11.447681, event: throttled, op: pg_info(1 pgs e243:1.76c) 2014-06-09 16:22:11.757524 7f869af78700 5 -- op tracker -- , seq: 2895, time: 2014-06-09 16:22:11.447694, event: all_read, op: pg_info(1 pgs e243:1.76c) 2014-06-09 16:22:11.757531 7f869af78700 5 -- op tracker -- , seq: 2895, time: 2014-06-09 16:22:11.757499, event: dispatched, op: pg_info(1 pgs e243:1.76c) 2014-06-09 16:22:11.757540 7f869af78700 5 -- op tracker -- , seq: 2895, time: 2014-06-09 16:22:11.757538, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.76c) 2014-06-09 16:22:11.757545 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.76c) v4 from osd.4 2014-06-09 16:22:11.757548 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee6a80 2014-06-09 16:22:11.757553 7f869af78700 5 -- op tracker -- , seq: 2895, time: 2014-06-09 16:22:11.757551, event: started, op: pg_info(1 pgs e243:1.76c) 2014-06-09 16:22:11.757573 7f869af78700 5 -- op tracker -- , seq: 2895, time: 2014-06-09 16:22:11.757571, event: done, op: pg_info(1 pgs e243:1.76c) 2014-06-09 16:22:11.757581 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757583 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757596 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757598 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757599 7f869af78700 20 osd.26 243 _dispatch 0x2f047c00 pg_info(1 pgs e243:0.4a7) v4 2014-06-09 16:22:11.757604 7f869af78700 5 -- op tracker -- , seq: 2896, time: 2014-06-09 16:22:11.612641, event: header_read, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.757610 7f869af78700 5 -- op tracker -- , seq: 2896, time: 2014-06-09 16:22:11.612641, event: throttled, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.757617 7f869af78700 5 -- op tracker -- , seq: 2896, time: 2014-06-09 16:22:11.612650, event: all_read, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.757624 7f869af78700 5 -- op tracker -- , seq: 2896, time: 2014-06-09 16:22:11.757595, event: dispatched, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.757631 7f869af78700 5 -- op tracker -- , seq: 2896, time: 2014-06-09 16:22:11.757629, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.757635 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4a7) v4 from osd.7 2014-06-09 16:22:11.757638 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f047c00 2014-06-09 16:22:11.757656 7f869af78700 5 -- op tracker -- , seq: 2896, time: 2014-06-09 16:22:11.757653, event: started, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.757671 7f8670072700 5 -- op tracker -- , seq: 2897, time: 2014-06-09 16:22:11.755848, event: header_read, op: osd_op(client.4538.0:36826 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1249280] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.757685 7f8670072700 5 -- op tracker -- , seq: 2897, time: 2014-06-09 16:22:11.755849, event: throttled, op: osd_op(client.4538.0:36826 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1249280] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.757710 7f8670072700 5 -- op tracker -- , seq: 2897, time: 2014-06-09 16:22:11.757636, event: all_read, op: osd_op(client.4538.0:36826 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1249280] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.757725 7f8670072700 5 -- op tracker -- , seq: 2897, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36826 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1249280] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.757733 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.757734 7f869af78700 5 -- op tracker -- , seq: 2896, time: 2014-06-09 16:22:11.757669, event: done, op: pg_info(1 pgs e243:0.4a7) 2014-06-09 16:22:11.757743 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757745 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757757 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757759 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757760 7f869af78700 20 osd.26 243 _dispatch 0x30a3a1c0 pg_info(1 pgs e243:2.463) v4 2014-06-09 16:22:11.757765 7f869af78700 5 -- op tracker -- , seq: 2898, time: 2014-06-09 16:22:11.603831, event: header_read, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.757772 7f869af78700 5 -- op tracker -- , seq: 2898, time: 2014-06-09 16:22:11.603832, event: throttled, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.757769 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6b00 prio 63 cost 1249280 latency 0.001921 osd_op(client.4538.0:36826 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1249280] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.757778 7f869af78700 5 -- op tracker -- , seq: 2898, time: 2014-06-09 16:22:11.603844, event: all_read, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.757787 7f869af78700 5 -- op tracker -- , seq: 2898, time: 2014-06-09 16:22:11.757757, event: dispatched, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.757794 7f869af78700 5 -- op tracker -- , seq: 2898, time: 2014-06-09 16:22:11.757792, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.757798 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.463) v4 from osd.3 2014-06-09 16:22:11.757801 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3a1c0 2014-06-09 16:22:11.757806 7f869af78700 5 -- op tracker -- , seq: 2898, time: 2014-06-09 16:22:11.757804, event: started, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.757822 7f869af78700 5 -- op tracker -- , seq: 2898, time: 2014-06-09 16:22:11.757820, event: done, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.757829 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757831 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757843 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757845 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757846 7f869af78700 20 osd.26 243 _dispatch 0x30a3b6c0 pg_info(1 pgs e243:2.759) v4 2014-06-09 16:22:11.757851 7f869af78700 5 -- op tracker -- , seq: 2899, time: 2014-06-09 16:22:11.607241, event: header_read, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.757860 7f869af78700 5 -- op tracker -- , seq: 2899, time: 2014-06-09 16:22:11.607241, event: throttled, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.757867 7f869af78700 5 -- op tracker -- , seq: 2899, time: 2014-06-09 16:22:11.607249, event: all_read, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.757873 7f869af78700 5 -- op tracker -- , seq: 2899, time: 2014-06-09 16:22:11.757843, event: dispatched, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.757880 7f869af78700 5 -- op tracker -- , seq: 2899, time: 2014-06-09 16:22:11.757878, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.757885 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.759) v4 from osd.1 2014-06-09 16:22:11.757887 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3b6c0 2014-06-09 16:22:11.757892 7f869af78700 5 -- op tracker -- , seq: 2899, time: 2014-06-09 16:22:11.757890, event: started, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.757908 7f869af78700 5 -- op tracker -- , seq: 2899, time: 2014-06-09 16:22:11.757906, event: done, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.757915 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757917 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757929 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.757931 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.757932 7f869af78700 20 osd.26 243 _dispatch 0x2ea5b880 pg_info(1 pgs e243:3.53d) v4 2014-06-09 16:22:11.757937 7f869af78700 5 -- op tracker -- , seq: 2900, time: 2014-06-09 16:22:11.597436, event: header_read, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757949 7f869af78700 5 -- op tracker -- , seq: 2900, time: 2014-06-09 16:22:11.597437, event: throttled, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757955 7f869af78700 5 -- op tracker -- , seq: 2900, time: 2014-06-09 16:22:11.597450, event: all_read, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757964 7f869af78700 5 -- op tracker -- , seq: 2900, time: 2014-06-09 16:22:11.757929, event: dispatched, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757971 7f869af78700 5 -- op tracker -- , seq: 2900, time: 2014-06-09 16:22:11.757969, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757975 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.53d) v4 from osd.9 2014-06-09 16:22:11.757978 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5b880 2014-06-09 16:22:11.757983 7f869af78700 5 -- op tracker -- , seq: 2900, time: 2014-06-09 16:22:11.757981, event: started, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.757999 7f869af78700 5 -- op tracker -- , seq: 2900, time: 2014-06-09 16:22:11.757997, event: done, op: pg_info(1 pgs e243:3.53d) 2014-06-09 16:22:11.758007 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758009 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758023 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758025 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758026 7f869af78700 20 osd.26 243 _dispatch 0x2eef1180 pg_info(1 pgs e243:1.2ef) v4 2014-06-09 16:22:11.758031 7f869af78700 5 -- op tracker -- , seq: 2901, time: 2014-06-09 16:22:11.568824, event: header_read, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.758037 7f869af78700 5 -- op tracker -- , seq: 2901, time: 2014-06-09 16:22:11.568825, event: throttled, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.758049 7f869af78700 5 -- op tracker -- , seq: 2901, time: 2014-06-09 16:22:11.568838, event: all_read, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.758058 7f869af78700 5 -- op tracker -- , seq: 2901, time: 2014-06-09 16:22:11.758023, event: dispatched, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.758067 7f869af78700 5 -- op tracker -- , seq: 2901, time: 2014-06-09 16:22:11.758065, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.758072 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2ef) v4 from osd.8 2014-06-09 16:22:11.758075 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef1180 2014-06-09 16:22:11.758080 7f869af78700 5 -- op tracker -- , seq: 2901, time: 2014-06-09 16:22:11.758078, event: started, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.758098 7f869af78700 5 -- op tracker -- , seq: 2901, time: 2014-06-09 16:22:11.758095, event: done, op: pg_info(1 pgs e243:1.2ef) 2014-06-09 16:22:11.758105 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758107 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758119 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758121 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758122 7f869af78700 20 osd.26 243 _dispatch 0x30cf0e00 pg_info(1 pgs e243:0.3a6) v4 2014-06-09 16:22:11.758127 7f869af78700 5 -- op tracker -- , seq: 2902, time: 2014-06-09 16:22:11.592610, event: header_read, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.758138 7f869af78700 5 -- op tracker -- , seq: 2902, time: 2014-06-09 16:22:11.592611, event: throttled, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.758146 7f869af78700 5 -- op tracker -- , seq: 2902, time: 2014-06-09 16:22:11.592623, event: all_read, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.758153 7f869af78700 5 -- op tracker -- , seq: 2902, time: 2014-06-09 16:22:11.758119, event: dispatched, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.758160 7f869af78700 5 -- op tracker -- , seq: 2902, time: 2014-06-09 16:22:11.758158, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.758167 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3a6) v4 from osd.6 2014-06-09 16:22:11.758170 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf0e00 2014-06-09 16:22:11.758175 7f869af78700 5 -- op tracker -- , seq: 2902, time: 2014-06-09 16:22:11.758173, event: started, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.758192 7f869af78700 5 -- op tracker -- , seq: 2902, time: 2014-06-09 16:22:11.758190, event: done, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.758202 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758204 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758217 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758218 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758219 7f869af78700 20 osd.26 243 _dispatch 0x2fcaa700 pg_info(1 pgs e243:3.1fc) v4 2014-06-09 16:22:11.758225 7f869af78700 5 -- op tracker -- , seq: 2903, time: 2014-06-09 16:22:11.614782, event: header_read, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.758232 7f869af78700 5 -- op tracker -- , seq: 2903, time: 2014-06-09 16:22:11.614783, event: throttled, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.758243 7f869af78700 5 -- op tracker -- , seq: 2903, time: 2014-06-09 16:22:11.614795, event: all_read, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.758253 7f869af78700 5 -- op tracker -- , seq: 2903, time: 2014-06-09 16:22:11.758216, event: dispatched, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.758264 7f869af78700 5 -- op tracker -- , seq: 2903, time: 2014-06-09 16:22:11.758262, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.758272 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1fc) v4 from osd.0 2014-06-09 16:22:11.758275 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaa700 2014-06-09 16:22:11.758280 7f869af78700 5 -- op tracker -- , seq: 2903, time: 2014-06-09 16:22:11.758278, event: started, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.758297 7f869af78700 5 -- op tracker -- , seq: 2903, time: 2014-06-09 16:22:11.758295, event: done, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.758305 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758307 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758319 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758320 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758321 7f869af78700 20 osd.26 243 _dispatch 0x30dae380 pg_info(1 pgs e243:1.3cb) v4 2014-06-09 16:22:11.758326 7f869af78700 5 -- op tracker -- , seq: 2904, time: 2014-06-09 16:22:11.593163, event: header_read, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.758338 7f869af78700 5 -- op tracker -- , seq: 2904, time: 2014-06-09 16:22:11.593164, event: throttled, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.758348 7f869af78700 5 -- op tracker -- , seq: 2904, time: 2014-06-09 16:22:11.593176, event: all_read, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.758358 7f869af78700 5 -- op tracker -- , seq: 2904, time: 2014-06-09 16:22:11.758318, event: dispatched, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.758370 7f869af78700 5 -- op tracker -- , seq: 2904, time: 2014-06-09 16:22:11.758368, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.758374 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3cb) v4 from osd.5 2014-06-09 16:22:11.758377 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dae380 2014-06-09 16:22:11.758382 7f869af78700 5 -- op tracker -- , seq: 2904, time: 2014-06-09 16:22:11.758380, event: started, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.758398 7f869af78700 5 -- op tracker -- , seq: 2904, time: 2014-06-09 16:22:11.758396, event: done, op: pg_info(1 pgs e243:1.3cb) 2014-06-09 16:22:11.758406 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758408 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758421 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758423 7f869af78700 20 osd.26 243 _dispatch 0x240ef340 pg_info(1 pgs e243:3.72b) v4 2014-06-09 16:22:11.758428 7f869af78700 5 -- op tracker -- , seq: 2905, time: 2014-06-09 16:22:11.323041, event: header_read, op: pg_info(1 pgs e243:3.72b) 2014-06-09 16:22:11.758435 7f869af78700 5 -- op tracker -- , seq: 2905, time: 2014-06-09 16:22:11.323042, event: throttled, op: pg_info(1 pgs e243:3.72b) 2014-06-09 16:22:11.758441 7f869af78700 5 -- op tracker -- , seq: 2905, time: 2014-06-09 16:22:11.323051, event: all_read, op: pg_info(1 pgs e243:3.72b) 2014-06-09 16:22:11.758450 7f869af78700 5 -- op tracker -- , seq: 2905, time: 2014-06-09 16:22:11.758419, event: dispatched, op: pg_info(1 pgs e243:3.72b) 2014-06-09 16:22:11.758457 7f869af78700 5 -- op tracker -- , seq: 2905, time: 2014-06-09 16:22:11.758455, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.72b) 2014-06-09 16:22:11.758461 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.72b) v4 from osd.2 2014-06-09 16:22:11.758464 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ef340 2014-06-09 16:22:11.758469 7f869af78700 5 -- op tracker -- , seq: 2905, time: 2014-06-09 16:22:11.758467, event: started, op: pg_info(1 pgs e243:3.72b) 2014-06-09 16:22:11.758486 7f869af78700 5 -- op tracker -- , seq: 2905, time: 2014-06-09 16:22:11.758484, event: done, op: pg_info(1 pgs e243:3.72b) 2014-06-09 16:22:11.758494 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758496 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758507 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758509 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758511 7f869af78700 20 osd.26 243 _dispatch 0x2eee68c0 pg_info(1 pgs e243:2.5d3) v4 2014-06-09 16:22:11.758515 7f869af78700 5 -- op tracker -- , seq: 2906, time: 2014-06-09 16:22:11.447698, event: header_read, op: pg_info(1 pgs e243:2.5d3) 2014-06-09 16:22:11.758523 7f869af78700 5 -- op tracker -- , seq: 2906, time: 2014-06-09 16:22:11.447699, event: throttled, op: pg_info(1 pgs e243:2.5d3) 2014-06-09 16:22:11.758529 7f869af78700 5 -- op tracker -- , seq: 2906, time: 2014-06-09 16:22:11.447710, event: all_read, op: pg_info(1 pgs e243:2.5d3) 2014-06-09 16:22:11.758536 7f869af78700 5 -- op tracker -- , seq: 2906, time: 2014-06-09 16:22:11.758507, event: dispatched, op: pg_info(1 pgs e243:2.5d3) 2014-06-09 16:22:11.758543 7f869af78700 5 -- op tracker -- , seq: 2906, time: 2014-06-09 16:22:11.758541, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5d3) 2014-06-09 16:22:11.758550 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5d3) v4 from osd.4 2014-06-09 16:22:11.758553 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee68c0 2014-06-09 16:22:11.758558 7f869af78700 5 -- op tracker -- , seq: 2906, time: 2014-06-09 16:22:11.758556, event: started, op: pg_info(1 pgs e243:2.5d3) 2014-06-09 16:22:11.758575 7f869af78700 5 -- op tracker -- , seq: 2906, time: 2014-06-09 16:22:11.758573, event: done, op: pg_info(1 pgs e243:2.5d3) 2014-06-09 16:22:11.758582 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758584 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758597 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758599 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758600 7f869af78700 20 osd.26 243 _dispatch 0x3130b500 pg_info(1 pgs e243:3.5b3) v4 2014-06-09 16:22:11.758605 7f869af78700 5 -- op tracker -- , seq: 2907, time: 2014-06-09 16:22:11.641790, event: header_read, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.758614 7f869af78700 5 -- op tracker -- , seq: 2907, time: 2014-06-09 16:22:11.641791, event: throttled, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.758621 7f869af78700 5 -- op tracker -- , seq: 2907, time: 2014-06-09 16:22:11.641803, event: all_read, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.758627 7f869af78700 5 -- op tracker -- , seq: 2907, time: 2014-06-09 16:22:11.758597, event: dispatched, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.758634 7f869af78700 5 -- op tracker -- , seq: 2907, time: 2014-06-09 16:22:11.758632, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.758644 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5b3) v4 from osd.7 2014-06-09 16:22:11.758647 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130b500 2014-06-09 16:22:11.758652 7f869af78700 5 -- op tracker -- , seq: 2907, time: 2014-06-09 16:22:11.758650, event: started, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.758668 7f869af78700 5 -- op tracker -- , seq: 2907, time: 2014-06-09 16:22:11.758665, event: done, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.758676 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758678 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758690 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758692 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758693 7f869af78700 20 osd.26 243 _dispatch 0x30a3bdc0 pg_info(1 pgs e243:3.5c6) v4 2014-06-09 16:22:11.758698 7f869af78700 5 -- op tracker -- , seq: 2908, time: 2014-06-09 16:22:11.605208, event: header_read, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.758705 7f869af78700 5 -- op tracker -- , seq: 2908, time: 2014-06-09 16:22:11.605209, event: throttled, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.758711 7f869af78700 5 -- op tracker -- , seq: 2908, time: 2014-06-09 16:22:11.605222, event: all_read, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.758722 7f869af78700 5 -- op tracker -- , seq: 2908, time: 2014-06-09 16:22:11.758690, event: dispatched, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.758729 7f869af78700 5 -- op tracker -- , seq: 2908, time: 2014-06-09 16:22:11.758727, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.758739 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5c6) v4 from osd.3 2014-06-09 16:22:11.758742 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3bdc0 2014-06-09 16:22:11.758747 7f869af78700 5 -- op tracker -- , seq: 2908, time: 2014-06-09 16:22:11.758745, event: started, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.758765 7f869af78700 5 -- op tracker -- , seq: 2908, time: 2014-06-09 16:22:11.758763, event: done, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.758773 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758775 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758786 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758788 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758789 7f869af78700 20 osd.26 243 _dispatch 0x30a3b880 pg_info(1 pgs e243:1.27c) v4 2014-06-09 16:22:11.758794 7f869af78700 5 -- op tracker -- , seq: 2909, time: 2014-06-09 16:22:11.607254, event: header_read, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.758869 7f869af78700 5 -- op tracker -- , seq: 2909, time: 2014-06-09 16:22:11.607255, event: throttled, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.758878 7f869af78700 5 -- op tracker -- , seq: 2909, time: 2014-06-09 16:22:11.607266, event: all_read, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.758885 7f869af78700 5 -- op tracker -- , seq: 2909, time: 2014-06-09 16:22:11.758786, event: dispatched, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.758892 7f869af78700 5 -- op tracker -- , seq: 2909, time: 2014-06-09 16:22:11.758890, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.758901 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.27c) v4 from osd.1 2014-06-09 16:22:11.758904 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3b880 2014-06-09 16:22:11.758909 7f869af78700 5 -- op tracker -- , seq: 2909, time: 2014-06-09 16:22:11.758907, event: started, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.758925 7f869af78700 5 -- op tracker -- , seq: 2909, time: 2014-06-09 16:22:11.758923, event: done, op: pg_info(1 pgs e243:1.27c) 2014-06-09 16:22:11.758932 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758934 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758946 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.758948 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.758949 7f869af78700 20 osd.26 243 _dispatch 0x30de5180 pg_info(1 pgs e243:0.458) v4 2014-06-09 16:22:11.758954 7f869af78700 5 -- op tracker -- , seq: 2910, time: 2014-06-09 16:22:11.603429, event: header_read, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.758961 7f869af78700 5 -- op tracker -- , seq: 2910, time: 2014-06-09 16:22:11.603429, event: throttled, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.758968 7f869af78700 5 -- op tracker -- , seq: 2910, time: 2014-06-09 16:22:11.603494, event: all_read, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.758977 7f869af78700 5 -- op tracker -- , seq: 2910, time: 2014-06-09 16:22:11.758946, event: dispatched, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.758984 7f869af78700 5 -- op tracker -- , seq: 2910, time: 2014-06-09 16:22:11.758982, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.758988 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.458) v4 from osd.9 2014-06-09 16:22:11.758991 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5180 2014-06-09 16:22:11.758996 7f869af78700 5 -- op tracker -- , seq: 2910, time: 2014-06-09 16:22:11.758994, event: started, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759015 7f869af78700 5 -- op tracker -- , seq: 2910, time: 2014-06-09 16:22:11.759013, event: done, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759023 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759025 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759036 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759038 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759039 7f869af78700 20 osd.26 243 _dispatch 0x2eef0fc0 pg_info(1 pgs e243:3.616) v4 2014-06-09 16:22:11.759044 7f869af78700 5 -- op tracker -- , seq: 2911, time: 2014-06-09 16:22:11.568870, event: header_read, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.759051 7f869af78700 5 -- op tracker -- , seq: 2911, time: 2014-06-09 16:22:11.568871, event: throttled, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.759057 7f869af78700 5 -- op tracker -- , seq: 2911, time: 2014-06-09 16:22:11.568883, event: all_read, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.759064 7f869af78700 5 -- op tracker -- , seq: 2911, time: 2014-06-09 16:22:11.759036, event: dispatched, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.759071 7f869af78700 5 -- op tracker -- , seq: 2911, time: 2014-06-09 16:22:11.759069, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.759076 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.616) v4 from osd.8 2014-06-09 16:22:11.759079 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef0fc0 2014-06-09 16:22:11.759083 7f869af78700 5 -- op tracker -- , seq: 2911, time: 2014-06-09 16:22:11.759082, event: started, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.759100 7f869af78700 5 -- op tracker -- , seq: 2911, time: 2014-06-09 16:22:11.759098, event: done, op: pg_info(1 pgs e243:3.616) 2014-06-09 16:22:11.759107 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759109 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759122 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759123 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759125 7f869af78700 20 osd.26 243 _dispatch 0x30d9ddc0 pg_info(1 pgs e243:0.3a6) v4 2014-06-09 16:22:11.759130 7f869af78700 5 -- op tracker -- , seq: 2912, time: 2014-06-09 16:22:11.592665, event: header_read, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.759137 7f869af78700 5 -- op tracker -- , seq: 2912, time: 2014-06-09 16:22:11.592665, event: throttled, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.759148 7f869af78700 5 -- op tracker -- , seq: 2912, time: 2014-06-09 16:22:11.592679, event: all_read, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.759160 7f869af78700 5 -- op tracker -- , seq: 2912, time: 2014-06-09 16:22:11.759121, event: dispatched, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.759167 7f869af78700 5 -- op tracker -- , seq: 2912, time: 2014-06-09 16:22:11.759164, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.759173 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3a6) v4 from osd.6 2014-06-09 16:22:11.759176 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9ddc0 2014-06-09 16:22:11.759181 7f869af78700 5 -- op tracker -- , seq: 2912, time: 2014-06-09 16:22:11.759179, event: started, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.759197 7f869af78700 5 -- op tracker -- , seq: 2912, time: 2014-06-09 16:22:11.759195, event: done, op: pg_info(1 pgs e243:0.3a6) 2014-06-09 16:22:11.759206 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759209 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759221 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759223 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759224 7f869af78700 20 osd.26 243 _dispatch 0x2fc95a40 pg_info(1 pgs e243:1.113) v4 2014-06-09 16:22:11.759229 7f869af78700 5 -- op tracker -- , seq: 2913, time: 2014-06-09 16:22:11.614805, event: header_read, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.759236 7f869af78700 5 -- op tracker -- , seq: 2913, time: 2014-06-09 16:22:11.614805, event: throttled, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.759248 7f869af78700 5 -- op tracker -- , seq: 2913, time: 2014-06-09 16:22:11.614817, event: all_read, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.759259 7f869af78700 5 -- op tracker -- , seq: 2913, time: 2014-06-09 16:22:11.759220, event: dispatched, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.759269 7f869af78700 5 -- op tracker -- , seq: 2913, time: 2014-06-09 16:22:11.759267, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.759278 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.113) v4 from osd.0 2014-06-09 16:22:11.759281 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc95a40 2014-06-09 16:22:11.759286 7f869af78700 5 -- op tracker -- , seq: 2913, time: 2014-06-09 16:22:11.759284, event: started, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.759302 7f869af78700 5 -- op tracker -- , seq: 2913, time: 2014-06-09 16:22:11.759300, event: done, op: pg_info(1 pgs e243:1.113) 2014-06-09 16:22:11.759310 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759312 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759324 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759326 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759327 7f869af78700 20 osd.26 243 _dispatch 0x30dae1c0 pg_info(1 pgs e243:0.652) v4 2014-06-09 16:22:11.759332 7f869af78700 5 -- op tracker -- , seq: 2914, time: 2014-06-09 16:22:11.593190, event: header_read, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.759339 7f869af78700 5 -- op tracker -- , seq: 2914, time: 2014-06-09 16:22:11.593190, event: throttled, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.759345 7f869af78700 5 -- op tracker -- , seq: 2914, time: 2014-06-09 16:22:11.593202, event: all_read, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.759354 7f869af78700 5 -- op tracker -- , seq: 2914, time: 2014-06-09 16:22:11.759323, event: dispatched, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.759362 7f869af78700 5 -- op tracker -- , seq: 2914, time: 2014-06-09 16:22:11.759359, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.759366 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.652) v4 from osd.5 2014-06-09 16:22:11.759369 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dae1c0 2014-06-09 16:22:11.759374 7f869af78700 5 -- op tracker -- , seq: 2914, time: 2014-06-09 16:22:11.759372, event: started, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.759392 7f869af78700 5 -- op tracker -- , seq: 2914, time: 2014-06-09 16:22:11.759390, event: done, op: pg_info(1 pgs e243:0.652) 2014-06-09 16:22:11.759400 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759402 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759416 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759418 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759419 7f869af78700 20 osd.26 243 _dispatch 0x30647000 pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) v4 2014-06-09 16:22:11.759426 7f869af78700 5 -- op tracker -- , seq: 2915, time: 2014-06-09 16:22:11.323198, event: header_read, op: pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.759439 7f869af78700 5 -- op tracker -- , seq: 2915, time: 2014-06-09 16:22:11.323198, event: throttled, op: pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.759446 7f869af78700 5 -- op tracker -- , seq: 2915, time: 2014-06-09 16:22:11.324409, event: all_read, op: pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.759457 7f869af78700 5 -- op tracker -- , seq: 2915, time: 2014-06-09 16:22:11.759415, event: dispatched, op: pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.759466 7f869af78700 5 -- op tracker -- , seq: 2915, time: 2014-06-09 16:22:11.759463, event: waiting_for_osdmap, op: pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.759475 7f869af78700 7 osd.26 243 handle_pg_log pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) v4 from osd.2 2014-06-09 16:22:11.759479 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30647000 2014-06-09 16:22:11.759485 7f869af78700 5 -- op tracker -- , seq: 2915, time: 2014-06-09 16:22:11.759482, event: started, op: pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.759504 7f869af78700 5 -- op tracker -- , seq: 2915, time: 2014-06-09 16:22:11.759501, event: done, op: pg_log(3.1c7 epoch 243 log log((0'0,213'647], crt=0'0) query_epoch 243) 2014-06-09 16:22:11.759512 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759525 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759527 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759528 7f869af78700 20 osd.26 243 _dispatch 0x2eee6700 pg_info(1 pgs e243:3.573) v4 2014-06-09 16:22:11.759533 7f869af78700 5 -- op tracker -- , seq: 2916, time: 2014-06-09 16:22:11.447715, event: header_read, op: pg_info(1 pgs e243:3.573) 2014-06-09 16:22:11.759543 7f869af78700 5 -- op tracker -- , seq: 2916, time: 2014-06-09 16:22:11.447716, event: throttled, op: pg_info(1 pgs e243:3.573) 2014-06-09 16:22:11.759550 7f869af78700 5 -- op tracker -- , seq: 2916, time: 2014-06-09 16:22:11.447726, event: all_read, op: pg_info(1 pgs e243:3.573) 2014-06-09 16:22:11.759557 7f869af78700 5 -- op tracker -- , seq: 2916, time: 2014-06-09 16:22:11.759525, event: dispatched, op: pg_info(1 pgs e243:3.573) 2014-06-09 16:22:11.759564 7f869af78700 5 -- op tracker -- , seq: 2916, time: 2014-06-09 16:22:11.759562, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.573) 2014-06-09 16:22:11.759569 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.573) v4 from osd.4 2014-06-09 16:22:11.759572 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee6700 2014-06-09 16:22:11.759577 7f869af78700 5 -- op tracker -- , seq: 2916, time: 2014-06-09 16:22:11.759575, event: started, op: pg_info(1 pgs e243:3.573) 2014-06-09 16:22:11.759594 7f869af78700 5 -- op tracker -- , seq: 2916, time: 2014-06-09 16:22:11.759592, event: done, op: pg_info(1 pgs e243:3.573) 2014-06-09 16:22:11.759601 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759603 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759629 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759631 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759633 7f869af78700 20 osd.26 243 _dispatch 0x16fdbc00 pg_info(1 pgs e243:3.5b3) v4 2014-06-09 16:22:11.759638 7f869af78700 5 -- op tracker -- , seq: 2917, time: 2014-06-09 16:22:11.641827, event: header_read, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.759645 7f869af78700 5 -- op tracker -- , seq: 2917, time: 2014-06-09 16:22:11.641828, event: throttled, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.759652 7f869af78700 5 -- op tracker -- , seq: 2917, time: 2014-06-09 16:22:11.641839, event: all_read, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.759658 7f869af78700 5 -- op tracker -- , seq: 2917, time: 2014-06-09 16:22:11.759629, event: dispatched, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.759665 7f869af78700 5 -- op tracker -- , seq: 2917, time: 2014-06-09 16:22:11.759663, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.759674 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5b3) v4 from osd.7 2014-06-09 16:22:11.759678 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16fdbc00 2014-06-09 16:22:11.759683 7f869af78700 5 -- op tracker -- , seq: 2917, time: 2014-06-09 16:22:11.759681, event: started, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.759699 7f869af78700 5 -- op tracker -- , seq: 2917, time: 2014-06-09 16:22:11.759697, event: done, op: pg_info(1 pgs e243:3.5b3) 2014-06-09 16:22:11.759707 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759709 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759722 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759723 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759724 7f869af78700 20 osd.26 243 _dispatch 0x30a3aa80 pg_info(1 pgs e243:2.182) v4 2014-06-09 16:22:11.759729 7f869af78700 5 -- op tracker -- , seq: 2918, time: 2014-06-09 16:22:11.606291, event: header_read, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.759736 7f869af78700 5 -- op tracker -- , seq: 2918, time: 2014-06-09 16:22:11.606292, event: throttled, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.759743 7f869af78700 5 -- op tracker -- , seq: 2918, time: 2014-06-09 16:22:11.606306, event: all_read, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.759752 7f869af78700 5 -- op tracker -- , seq: 2918, time: 2014-06-09 16:22:11.759721, event: dispatched, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.759759 7f869af78700 5 -- op tracker -- , seq: 2918, time: 2014-06-09 16:22:11.759757, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.759763 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.182) v4 from osd.3 2014-06-09 16:22:11.759766 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3aa80 2014-06-09 16:22:11.759771 7f869af78700 5 -- op tracker -- , seq: 2918, time: 2014-06-09 16:22:11.759769, event: started, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.759787 7f869af78700 5 -- op tracker -- , seq: 2918, time: 2014-06-09 16:22:11.759785, event: done, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.759796 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759798 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759812 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759813 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759815 7f869af78700 20 osd.26 243 _dispatch 0x30a3a380 pg_info(1 pgs e243:3.254) v4 2014-06-09 16:22:11.759820 7f869af78700 5 -- op tracker -- , seq: 2919, time: 2014-06-09 16:22:11.607275, event: header_read, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.759827 7f869af78700 5 -- op tracker -- , seq: 2919, time: 2014-06-09 16:22:11.607276, event: throttled, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.759833 7f869af78700 5 -- op tracker -- , seq: 2919, time: 2014-06-09 16:22:11.607295, event: all_read, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.759840 7f869af78700 5 -- op tracker -- , seq: 2919, time: 2014-06-09 16:22:11.759811, event: dispatched, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.759847 7f869af78700 5 -- op tracker -- , seq: 2919, time: 2014-06-09 16:22:11.759845, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.759855 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.254) v4 from osd.1 2014-06-09 16:22:11.759858 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3a380 2014-06-09 16:22:11.759863 7f869af78700 5 -- op tracker -- , seq: 2919, time: 2014-06-09 16:22:11.759861, event: started, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.759879 7f869af78700 5 -- op tracker -- , seq: 2919, time: 2014-06-09 16:22:11.759877, event: done, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.759887 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759889 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759901 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759903 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759904 7f869af78700 20 osd.26 243 _dispatch 0x19805c00 pg_info(1 pgs e243:0.458) v4 2014-06-09 16:22:11.759909 7f869af78700 5 -- op tracker -- , seq: 2920, time: 2014-06-09 16:22:11.607747, event: header_read, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759916 7f869af78700 5 -- op tracker -- , seq: 2920, time: 2014-06-09 16:22:11.607748, event: throttled, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759923 7f869af78700 5 -- op tracker -- , seq: 2920, time: 2014-06-09 16:22:11.607760, event: all_read, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759929 7f869af78700 5 -- op tracker -- , seq: 2920, time: 2014-06-09 16:22:11.759901, event: dispatched, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759939 7f869af78700 5 -- op tracker -- , seq: 2920, time: 2014-06-09 16:22:11.759936, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759944 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.458) v4 from osd.9 2014-06-09 16:22:11.759947 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19805c00 2014-06-09 16:22:11.759951 7f869af78700 5 -- op tracker -- , seq: 2920, time: 2014-06-09 16:22:11.759949, event: started, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759972 7f869af78700 5 -- op tracker -- , seq: 2920, time: 2014-06-09 16:22:11.759969, event: done, op: pg_info(1 pgs e243:0.458) 2014-06-09 16:22:11.759979 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759981 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759992 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.759994 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.759995 7f869af78700 20 osd.26 243 _dispatch 0x2176ddc0 pg_info(1 pgs e243:1.a6) v4 2014-06-09 16:22:11.760000 7f869af78700 5 -- op tracker -- , seq: 2921, time: 2014-06-09 16:22:11.574286, event: header_read, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.760010 7f869af78700 5 -- op tracker -- , seq: 2921, time: 2014-06-09 16:22:11.574287, event: throttled, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.760016 7f869af78700 5 -- op tracker -- , seq: 2921, time: 2014-06-09 16:22:11.574299, event: all_read, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.760023 7f869af78700 5 -- op tracker -- , seq: 2921, time: 2014-06-09 16:22:11.759992, event: dispatched, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.760032 7f869af78700 5 -- op tracker -- , seq: 2921, time: 2014-06-09 16:22:11.760030, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.760037 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a6) v4 from osd.8 2014-06-09 16:22:11.760040 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176ddc0 2014-06-09 16:22:11.760045 7f869af78700 5 -- op tracker -- , seq: 2921, time: 2014-06-09 16:22:11.760043, event: started, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.760062 7f869af78700 5 -- op tracker -- , seq: 2921, time: 2014-06-09 16:22:11.760060, event: done, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.760070 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760072 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760084 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760086 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760087 7f869af78700 20 osd.26 243 _dispatch 0x190b6540 pg_info(1 pgs e243:0.2fd) v4 2014-06-09 16:22:11.760092 7f869af78700 5 -- op tracker -- , seq: 2922, time: 2014-06-09 16:22:11.595958, event: header_read, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.760104 7f869af78700 5 -- op tracker -- , seq: 2922, time: 2014-06-09 16:22:11.595959, event: throttled, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.760111 7f869af78700 5 -- op tracker -- , seq: 2922, time: 2014-06-09 16:22:11.595971, event: all_read, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.760117 7f869af78700 5 -- op tracker -- , seq: 2922, time: 2014-06-09 16:22:11.760084, event: dispatched, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.760124 7f869af78700 5 -- op tracker -- , seq: 2922, time: 2014-06-09 16:22:11.760122, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.760128 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2fd) v4 from osd.6 2014-06-09 16:22:11.760131 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b6540 2014-06-09 16:22:11.760136 7f869af78700 5 -- op tracker -- , seq: 2922, time: 2014-06-09 16:22:11.760134, event: started, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.760154 7f869af78700 5 -- op tracker -- , seq: 2922, time: 2014-06-09 16:22:11.760152, event: done, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.760162 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760164 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760176 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760177 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760179 7f869af78700 20 osd.26 243 _dispatch 0x2fc95880 pg_info(1 pgs e243:3.778) v4 2014-06-09 16:22:11.760184 7f869af78700 5 -- op tracker -- , seq: 2923, time: 2014-06-09 16:22:11.614827, event: header_read, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.760190 7f869af78700 5 -- op tracker -- , seq: 2923, time: 2014-06-09 16:22:11.614827, event: throttled, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.760199 7f869af78700 5 -- op tracker -- , seq: 2923, time: 2014-06-09 16:22:11.614834, event: all_read, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.760206 7f869af78700 5 -- op tracker -- , seq: 2923, time: 2014-06-09 16:22:11.760175, event: dispatched, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.760213 7f869af78700 5 -- op tracker -- , seq: 2923, time: 2014-06-09 16:22:11.760210, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.760238 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.778) v4 from osd.0 2014-06-09 16:22:11.760241 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc95880 2014-06-09 16:22:11.760247 7f869af78700 5 -- op tracker -- , seq: 2923, time: 2014-06-09 16:22:11.760245, event: started, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.760264 7f869af78700 5 -- op tracker -- , seq: 2923, time: 2014-06-09 16:22:11.760261, event: done, op: pg_info(1 pgs e243:3.778) 2014-06-09 16:22:11.760273 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760275 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760287 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760285 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.760289 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760290 7f869af78700 20 osd.26 243 _dispatch 0x190b6380 pg_info(1 pgs e243:1.408) v4 2014-06-09 16:22:11.760295 7f869af78700 5 -- op tracker -- , seq: 2924, time: 2014-06-09 16:22:11.597041, event: header_read, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.760308 7f869af78700 5 -- op tracker -- , seq: 2924, time: 2014-06-09 16:22:11.597042, event: throttled, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.760314 7f86a5e11700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.760315 7f869af78700 5 -- op tracker -- , seq: 2924, time: 2014-06-09 16:22:11.597054, event: all_read, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.760321 7f86a5e11700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.760321 7f869af78700 5 -- op tracker -- , seq: 2924, time: 2014-06-09 16:22:11.760286, event: dispatched, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.760329 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.760328 7f869af78700 5 -- op tracker -- , seq: 2924, time: 2014-06-09 16:22:11.760326, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.760335 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.408) v4 from osd.5 2014-06-09 16:22:11.760338 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b6380 2014-06-09 16:22:11.760343 7f869af78700 5 -- op tracker -- , seq: 2924, time: 2014-06-09 16:22:11.760341, event: started, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.760345 7f86a5e11700 15 _write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 217088~2064384 2014-06-09 16:22:11.760358 7f869af78700 5 -- op tracker -- , seq: 2924, time: 2014-06-09 16:22:11.760355, event: done, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.760367 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760370 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760382 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760384 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760386 7f869af78700 20 osd.26 243 _dispatch 0x240eefc0 pg_info(1 pgs e243:0.411) v4 2014-06-09 16:22:11.760391 7f869af78700 5 -- op tracker -- , seq: 2925, time: 2014-06-09 16:22:11.331599, event: header_read, op: pg_info(1 pgs e243:0.411) 2014-06-09 16:22:11.760398 7f869af78700 5 -- op tracker -- , seq: 2925, time: 2014-06-09 16:22:11.331599, event: throttled, op: pg_info(1 pgs e243:0.411) 2014-06-09 16:22:11.760404 7f869af78700 5 -- op tracker -- , seq: 2925, time: 2014-06-09 16:22:11.331619, event: all_read, op: pg_info(1 pgs e243:0.411) 2014-06-09 16:22:11.760411 7f869af78700 5 -- op tracker -- , seq: 2925, time: 2014-06-09 16:22:11.760382, event: dispatched, op: pg_info(1 pgs e243:0.411) 2014-06-09 16:22:11.760421 7f869af78700 5 -- op tracker -- , seq: 2925, time: 2014-06-09 16:22:11.760419, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.411) 2014-06-09 16:22:11.760426 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.411) v4 from osd.2 2014-06-09 16:22:11.760429 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240eefc0 2014-06-09 16:22:11.760434 7f869af78700 5 -- op tracker -- , seq: 2925, time: 2014-06-09 16:22:11.760432, event: started, op: pg_info(1 pgs e243:0.411) 2014-06-09 16:22:11.760451 7f869af78700 5 -- op tracker -- , seq: 2925, time: 2014-06-09 16:22:11.760449, event: done, op: pg_info(1 pgs e243:0.411) 2014-06-09 16:22:11.760458 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760460 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760472 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760473 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760475 7f869af78700 20 osd.26 243 _dispatch 0x2eee6540 pg_info(1 pgs e243:2.1ae) v4 2014-06-09 16:22:11.760489 7f869af78700 5 -- op tracker -- , seq: 2926, time: 2014-06-09 16:22:11.447730, event: header_read, op: pg_info(1 pgs e243:2.1ae) 2014-06-09 16:22:11.760497 7f869af78700 5 -- op tracker -- , seq: 2926, time: 2014-06-09 16:22:11.447730, event: throttled, op: pg_info(1 pgs e243:2.1ae) 2014-06-09 16:22:11.760511 7f869af78700 5 -- op tracker -- , seq: 2926, time: 2014-06-09 16:22:11.447741, event: all_read, op: pg_info(1 pgs e243:2.1ae) 2014-06-09 16:22:11.760519 7f869af78700 5 -- op tracker -- , seq: 2926, time: 2014-06-09 16:22:11.760472, event: dispatched, op: pg_info(1 pgs e243:2.1ae) 2014-06-09 16:22:11.760528 7f869af78700 5 -- op tracker -- , seq: 2926, time: 2014-06-09 16:22:11.760526, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1ae) 2014-06-09 16:22:11.760534 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1ae) v4 from osd.4 2014-06-09 16:22:11.760538 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee6540 2014-06-09 16:22:11.760545 7f8670072700 5 -- op tracker -- , seq: 2927, time: 2014-06-09 16:22:11.757795, event: header_read, op: osd_op(client.4538.0:36827 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 1249280~1548288] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.760560 7f8670072700 5 -- op tracker -- , seq: 2927, time: 2014-06-09 16:22:11.757797, event: throttled, op: osd_op(client.4538.0:36827 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 1249280~1548288] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.760571 7f8670072700 5 -- op tracker -- , seq: 2927, time: 2014-06-09 16:22:11.760473, event: all_read, op: osd_op(client.4538.0:36827 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 1249280~1548288] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.760583 7f8670072700 5 -- op tracker -- , seq: 2927, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36827 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 1249280~1548288] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.760592 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.760592 7f869af78700 5 -- op tracker -- , seq: 2926, time: 2014-06-09 16:22:11.760541, event: started, op: pg_info(1 pgs e243:2.1ae) 2014-06-09 16:22:11.760609 7f869af78700 5 -- op tracker -- , seq: 2926, time: 2014-06-09 16:22:11.760607, event: done, op: pg_info(1 pgs e243:2.1ae) 2014-06-09 16:22:11.760620 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760622 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760615 7f8670072700 15 osd.26 243 enqueue_op 0xdfa6a00 prio 63 cost 1548288 latency 0.002820 osd_op(client.4538.0:36827 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 1249280~1548288] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.760635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760637 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760638 7f869af78700 20 osd.26 243 _dispatch 0x2fc57180 pg_info(1 pgs e243:3.271) v4 2014-06-09 16:22:11.760643 7f869af78700 5 -- op tracker -- , seq: 2928, time: 2014-06-09 16:22:11.651243, event: header_read, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.760650 7f869af78700 5 -- op tracker -- , seq: 2928, time: 2014-06-09 16:22:11.651243, event: throttled, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.760657 7f869af78700 5 -- op tracker -- , seq: 2928, time: 2014-06-09 16:22:11.651257, event: all_read, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.760668 7f869af78700 5 -- op tracker -- , seq: 2928, time: 2014-06-09 16:22:11.760635, event: dispatched, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.760680 7f869af78700 5 -- op tracker -- , seq: 2928, time: 2014-06-09 16:22:11.760678, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.760684 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.271) v4 from osd.7 2014-06-09 16:22:11.760687 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc57180 2014-06-09 16:22:11.760693 7f869af78700 5 -- op tracker -- , seq: 2928, time: 2014-06-09 16:22:11.760690, event: started, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.760710 7f869af78700 5 -- op tracker -- , seq: 2928, time: 2014-06-09 16:22:11.760708, event: done, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.760778 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760778 7f868df5e700 10 getattr got.size() is 0 2014-06-09 16:22:11.760781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760794 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760795 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760797 7f869af78700 20 osd.26 243 _dispatch 0x30c31880 pg_info(1 pgs e243:2.182) v4 2014-06-09 16:22:11.760783 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 but can_create 2014-06-09 16:22:11.760802 7f868df5e700 15 getattr 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 'snapset' 2014-06-09 16:22:11.760802 7f869af78700 5 -- op tracker -- , seq: 2929, time: 2014-06-09 16:22:11.606391, event: header_read, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.760809 7f869af78700 5 -- op tracker -- , seq: 2929, time: 2014-06-09 16:22:11.606392, event: throttled, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.760816 7f869af78700 5 -- op tracker -- , seq: 2929, time: 2014-06-09 16:22:11.606405, event: all_read, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.760828 7f869af78700 5 -- op tracker -- , seq: 2929, time: 2014-06-09 16:22:11.760793, event: dispatched, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.760835 7f86a5e11700 10 lookup_strip_header done cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.760836 7f869af78700 5 -- op tracker -- , seq: 2929, time: 2014-06-09 16:22:11.760834, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.760845 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.182) v4 from osd.3 2014-06-09 16:22:11.760848 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c31880 2014-06-09 16:22:11.760856 7f868df5e700 10 getattr got.size() is 0 2014-06-09 16:22:11.760859 7f868df5e700 15 getattr 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/snapdir//3 'snapset' 2014-06-09 16:22:11.760854 7f869af78700 5 -- op tracker -- , seq: 2929, time: 2014-06-09 16:22:11.760852, event: started, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.760874 7f869af78700 5 -- op tracker -- , seq: 2929, time: 2014-06-09 16:22:11.760872, event: done, op: pg_info(1 pgs e243:2.182) 2014-06-09 16:22:11.760882 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760884 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760885 7f868df5e700 10 getattr got.size() is 0 2014-06-09 16:22:11.760895 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.760897 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760899 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760900 7f869af78700 20 osd.26 243 _dispatch 0x2fe71500 pg_info(1 pgs e243:1.494) v4 2014-06-09 16:22:11.760896 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30df7b80 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.760906 7f869af78700 5 -- op tracker -- , seq: 2930, time: 2014-06-09 16:22:11.607308, event: header_read, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.760915 7f869af78700 5 -- op tracker -- , seq: 2930, time: 2014-06-09 16:22:11.607308, event: throttled, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.760910 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.760922 7f869af78700 5 -- op tracker -- , seq: 2930, time: 2014-06-09 16:22:11.607321, event: all_read, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.760920 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.760929 7f869af78700 5 -- op tracker -- , seq: 2930, time: 2014-06-09 16:22:11.760897, event: dispatched, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.760937 7f869af78700 5 -- op tracker -- , seq: 2930, time: 2014-06-09 16:22:11.760935, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.760943 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.494) v4 from osd.1 2014-06-09 16:22:11.760931 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30df7b80 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 rwstate(none n=0 w=0) oi: 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2e876e00 snapset: 0=[]:[] 2014-06-09 16:22:11.760946 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fe71500 2014-06-09 16:22:11.760951 7f869af78700 5 -- op tracker -- , seq: 2930, time: 2014-06-09 16:22:11.760949, event: started, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.760948 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 @head oi=8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.760967 7f869af78700 5 -- op tracker -- , seq: 2930, time: 2014-06-09 16:22:11.760964, event: done, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.760970 7f868df5e700 15 getattr 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/snapdir//3 '_' 2014-06-09 16:22:11.760974 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760976 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760988 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.760990 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.760991 7f869af78700 20 osd.26 243 _dispatch 0x2fc94540 pg_info(1 pgs e243:3.35c) v4 2014-06-09 16:22:11.760999 7f868df5e700 10 getattr got.size() is 0 2014-06-09 16:22:11.760997 7f869af78700 5 -- op tracker -- , seq: 2931, time: 2014-06-09 16:22:11.613500, event: header_read, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.761004 7f869af78700 5 -- op tracker -- , seq: 2931, time: 2014-06-09 16:22:11.613501, event: throttled, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.761001 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 8721c319/rbd_data.11796b8b4567.0000000000001aab/snapdir//3 and !can_create 2014-06-09 16:22:11.761010 7f869af78700 5 -- op tracker -- , seq: 2931, time: 2014-06-09 16:22:11.613514, event: all_read, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.761017 7f869af78700 5 -- op tracker -- , seq: 2931, time: 2014-06-09 16:22:11.760988, event: dispatched, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.761027 7f869af78700 5 -- op tracker -- , seq: 2931, time: 2014-06-09 16:22:11.761025, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.761033 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.35c) v4 from osd.9 2014-06-09 16:22:11.761036 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc94540 2014-06-09 16:22:11.761040 7f868df5e700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.761017, event: started, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.761051 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x305f3800 2014-06-09 16:22:11.761054 7f869af78700 5 -- op tracker -- , seq: 2931, time: 2014-06-09 16:22:11.761039, event: started, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.761069 7f868df5e700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.761063, event: started, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.761082 7f869af78700 5 -- op tracker -- , seq: 2931, time: 2014-06-09 16:22:11.761074, event: done, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.761091 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761079 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] ov 0'0 av 243'608 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.761093 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761095 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 2014-06-09 16:22:11.761105 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761106 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761108 7f869af78700 20 osd.26 243 _dispatch 0x2176c700 pg_info(1 pgs e243:0.36d) v4 2014-06-09 16:22:11.761106 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.761113 7f869af78700 5 -- op tracker -- , seq: 2932, time: 2014-06-09 16:22:11.574326, event: header_read, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.761120 7f869af78700 5 -- op tracker -- , seq: 2932, time: 2014-06-09 16:22:11.574326, event: throttled, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.761127 7f869af78700 5 -- op tracker -- , seq: 2932, time: 2014-06-09 16:22:11.574338, event: all_read, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.761122 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~61440 2014-06-09 16:22:11.761136 7f869af78700 5 -- op tracker -- , seq: 2932, time: 2014-06-09 16:22:11.761104, event: dispatched, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.761144 7f869af78700 5 -- op tracker -- , seq: 2932, time: 2014-06-09 16:22:11.761142, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.761149 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.36d) v4 from osd.8 2014-06-09 16:22:11.761142 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228c00 2014-06-09 16:22:11.761152 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176c700 2014-06-09 16:22:11.761157 7f869af78700 5 -- op tracker -- , seq: 2932, time: 2014-06-09 16:22:11.761155, event: started, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.761155 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 snapset=0x2e876e38 snapc=0=[] 2014-06-09 16:22:11.761166 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.761175 7f869af78700 5 -- op tracker -- , seq: 2932, time: 2014-06-09 16:22:11.761173, event: done, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.761184 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761186 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761177 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.761188 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 0x305f3800 op modify 2014-06-09 16:22:11.761198 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761200 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761201 7f869af78700 20 osd.26 243 _dispatch 0x30cae540 pg_info(1 pgs e243:0.2fd) v4 2014-06-09 16:22:11.761201 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:00.648411 2014-06-09 16:22:11.761206 7f869af78700 5 -- op tracker -- , seq: 2933, time: 2014-06-09 16:22:11.595980, event: header_read, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.761223 7f869af78700 5 -- op tracker -- , seq: 2933, time: 2014-06-09 16:22:11.595981, event: throttled, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.761230 7f869af78700 5 -- op tracker -- , seq: 2933, time: 2014-06-09 16:22:11.595992, event: all_read, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.761227 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.761237 7f869af78700 5 -- op tracker -- , seq: 2933, time: 2014-06-09 16:22:11.761198, event: dispatched, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.761244 7f869af78700 5 -- op tracker -- , seq: 2933, time: 2014-06-09 16:22:11.761242, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.761253 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2fd) v4 from osd.6 2014-06-09 16:22:11.761256 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cae540 2014-06-09 16:22:11.761247 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.761261 7f869af78700 5 -- op tracker -- , seq: 2933, time: 2014-06-09 16:22:11.761259, event: started, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.761259 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 12 on osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.761276 7f869af78700 5 -- op tracker -- , seq: 2933, time: 2014-06-09 16:22:11.761274, event: done, op: pg_info(1 pgs e243:0.2fd) 2014-06-09 16:22:11.761284 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761276 7f868df5e700 7 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 12 o 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.761286 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761297 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761299 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761300 7f869af78700 20 osd.26 243 _dispatch 0x2fc956c0 pg_info(1 pgs e243:3.1fc) v4 2014-06-09 16:22:11.761302 7f868df5e700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.761296, event: waiting for subops from 5, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.761316 7f869af78700 5 -- op tracker -- , seq: 2934, time: 2014-06-09 16:22:11.614841, event: header_read, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.761324 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.761323 7f869af78700 5 -- op tracker -- , seq: 2934, time: 2014-06-09 16:22:11.614841, event: throttled, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.761332 7f869af78700 5 -- op tracker -- , seq: 2934, time: 2014-06-09 16:22:11.614849, event: all_read, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.761339 7f869af78700 5 -- op tracker -- , seq: 2934, time: 2014-06-09 16:22:11.761297, event: dispatched, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.761346 7f869af78700 5 -- op tracker -- , seq: 2934, time: 2014-06-09 16:22:11.761344, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.761354 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1fc) v4 from osd.0 2014-06-09 16:22:11.761357 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc956c0 2014-06-09 16:22:11.761346 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 214'607 (0'0,214'607] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'607], crt=0'0) [243'608 (0'0) modify 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 by client.4569.0:38637 2014-06-09 10:20:00.648411] 2014-06-09 16:22:11.761363 7f869af78700 5 -- op tracker -- , seq: 2934, time: 2014-06-09 16:22:11.761360, event: started, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.761380 7f869af78700 5 -- op tracker -- , seq: 2934, time: 2014-06-09 16:22:11.761377, event: done, op: pg_info(1 pgs e243:3.1fc) 2014-06-09 16:22:11.761370 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'608 (0'0) modify 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 by client.4569.0:38637 2014-06-09 10:20:00.648411 2014-06-09 16:22:11.761389 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761392 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761393 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.761406 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761408 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761409 7f869af78700 20 osd.26 243 _dispatch 0x30cae000 pg_info(1 pgs e243:2.654) v4 2014-06-09 16:22:11.761414 7f869af78700 5 -- op tracker -- , seq: 2935, time: 2014-06-09 16:22:11.597063, event: header_read, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.761421 7f869af78700 5 -- op tracker -- , seq: 2935, time: 2014-06-09 16:22:11.597063, event: throttled, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.761425 7f868df5e700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'608, trimmed: 2014-06-09 16:22:11.761427 7f869af78700 5 -- op tracker -- , seq: 2935, time: 2014-06-09 16:22:11.597077, event: all_read, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.761434 7f869af78700 5 -- op tracker -- , seq: 2935, time: 2014-06-09 16:22:11.761406, event: dispatched, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.761447 7f868df5e700 5 queue_transactions existing osr(3.319 0x3ed0d60)/0x3ed0d60 2014-06-09 16:22:11.761446 7f869af78700 5 -- op tracker -- , seq: 2935, time: 2014-06-09 16:22:11.761444, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.761452 7f868df5e700 2 waiting 51 > 50 ops || 6822315 > 104857600 2014-06-09 16:22:11.761451 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.654) v4 from osd.5 2014-06-09 16:22:11.761455 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cae000 2014-06-09 16:22:11.761460 7f869af78700 5 -- op tracker -- , seq: 2935, time: 2014-06-09 16:22:11.761458, event: started, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.761476 7f869af78700 5 -- op tracker -- , seq: 2935, time: 2014-06-09 16:22:11.761474, event: done, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.761486 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761488 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761502 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761504 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761505 7f869af78700 20 osd.26 243 _dispatch 0x1ae7c1c0 pg_info(1 pgs e243:3.67d) v4 2014-06-09 16:22:11.761510 7f869af78700 5 -- op tracker -- , seq: 2936, time: 2014-06-09 16:22:11.435028, event: header_read, op: pg_info(1 pgs e243:3.67d) 2014-06-09 16:22:11.761517 7f869af78700 5 -- op tracker -- , seq: 2936, time: 2014-06-09 16:22:11.435029, event: throttled, op: pg_info(1 pgs e243:3.67d) 2014-06-09 16:22:11.761524 7f869af78700 5 -- op tracker -- , seq: 2936, time: 2014-06-09 16:22:11.435040, event: all_read, op: pg_info(1 pgs e243:3.67d) 2014-06-09 16:22:11.761531 7f869af78700 5 -- op tracker -- , seq: 2936, time: 2014-06-09 16:22:11.761502, event: dispatched, op: pg_info(1 pgs e243:3.67d) 2014-06-09 16:22:11.761537 7f869af78700 5 -- op tracker -- , seq: 2936, time: 2014-06-09 16:22:11.761535, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.67d) 2014-06-09 16:22:11.761547 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.67d) v4 from osd.2 2014-06-09 16:22:11.761550 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ae7c1c0 2014-06-09 16:22:11.761555 7f869af78700 5 -- op tracker -- , seq: 2936, time: 2014-06-09 16:22:11.761553, event: started, op: pg_info(1 pgs e243:3.67d) 2014-06-09 16:22:11.761572 7f869af78700 5 -- op tracker -- , seq: 2936, time: 2014-06-09 16:22:11.761570, event: done, op: pg_info(1 pgs e243:3.67d) 2014-06-09 16:22:11.761580 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761582 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761595 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761597 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761598 7f869af78700 20 osd.26 243 _dispatch 0x2eee6380 pg_info(1 pgs e243:2.1b3) v4 2014-06-09 16:22:11.761603 7f869af78700 5 -- op tracker -- , seq: 2937, time: 2014-06-09 16:22:11.447745, event: header_read, op: pg_info(1 pgs e243:2.1b3) 2014-06-09 16:22:11.761610 7f869af78700 5 -- op tracker -- , seq: 2937, time: 2014-06-09 16:22:11.447745, event: throttled, op: pg_info(1 pgs e243:2.1b3) 2014-06-09 16:22:11.761617 7f869af78700 5 -- op tracker -- , seq: 2937, time: 2014-06-09 16:22:11.447755, event: all_read, op: pg_info(1 pgs e243:2.1b3) 2014-06-09 16:22:11.761626 7f869af78700 5 -- op tracker -- , seq: 2937, time: 2014-06-09 16:22:11.761594, event: dispatched, op: pg_info(1 pgs e243:2.1b3) 2014-06-09 16:22:11.761633 7f869af78700 5 -- op tracker -- , seq: 2937, time: 2014-06-09 16:22:11.761631, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1b3) 2014-06-09 16:22:11.761637 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1b3) v4 from osd.4 2014-06-09 16:22:11.761640 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee6380 2014-06-09 16:22:11.761645 7f869af78700 5 -- op tracker -- , seq: 2937, time: 2014-06-09 16:22:11.761643, event: started, op: pg_info(1 pgs e243:2.1b3) 2014-06-09 16:22:11.761660 7f869af78700 5 -- op tracker -- , seq: 2937, time: 2014-06-09 16:22:11.761658, event: done, op: pg_info(1 pgs e243:2.1b3) 2014-06-09 16:22:11.761668 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761670 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761682 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761684 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761685 7f869af78700 20 osd.26 243 _dispatch 0x308d8c40 pg_info(1 pgs e243:3.271) v4 2014-06-09 16:22:11.761690 7f869af78700 5 -- op tracker -- , seq: 2938, time: 2014-06-09 16:22:11.651277, event: header_read, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.761697 7f869af78700 5 -- op tracker -- , seq: 2938, time: 2014-06-09 16:22:11.651278, event: throttled, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.761705 7f869af78700 5 -- op tracker -- , seq: 2938, time: 2014-06-09 16:22:11.651292, event: all_read, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.761712 7f869af78700 5 -- op tracker -- , seq: 2938, time: 2014-06-09 16:22:11.761681, event: dispatched, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.761719 7f869af78700 5 -- op tracker -- , seq: 2938, time: 2014-06-09 16:22:11.761717, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.761724 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.271) v4 from osd.7 2014-06-09 16:22:11.761727 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d8c40 2014-06-09 16:22:11.761732 7f869af78700 5 -- op tracker -- , seq: 2938, time: 2014-06-09 16:22:11.761730, event: started, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.761748 7f869af78700 5 -- op tracker -- , seq: 2938, time: 2014-06-09 16:22:11.761746, event: done, op: pg_info(1 pgs e243:3.271) 2014-06-09 16:22:11.761757 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761759 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761773 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761774 7f869af78700 20 osd.26 243 _dispatch 0x30c316c0 pg_info(1 pgs e243:2.463) v4 2014-06-09 16:22:11.761779 7f869af78700 5 -- op tracker -- , seq: 2939, time: 2014-06-09 16:22:11.607251, event: header_read, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.761786 7f869af78700 5 -- op tracker -- , seq: 2939, time: 2014-06-09 16:22:11.607251, event: throttled, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.761793 7f869af78700 5 -- op tracker -- , seq: 2939, time: 2014-06-09 16:22:11.607265, event: all_read, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.761802 7f869af78700 5 -- op tracker -- , seq: 2939, time: 2014-06-09 16:22:11.761771, event: dispatched, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.761810 7f869af78700 5 -- op tracker -- , seq: 2939, time: 2014-06-09 16:22:11.761808, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.761814 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.463) v4 from osd.3 2014-06-09 16:22:11.761817 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c316c0 2014-06-09 16:22:11.761823 7f869af78700 5 -- op tracker -- , seq: 2939, time: 2014-06-09 16:22:11.761820, event: started, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.761838 7f869af78700 5 -- op tracker -- , seq: 2939, time: 2014-06-09 16:22:11.761836, event: done, op: pg_info(1 pgs e243:2.463) 2014-06-09 16:22:11.761846 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761848 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761860 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761861 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761863 7f869af78700 20 osd.26 243 _dispatch 0x240ef6c0 pg_info(1 pgs e243:2.759) v4 2014-06-09 16:22:11.761868 7f869af78700 5 -- op tracker -- , seq: 2940, time: 2014-06-09 16:22:11.607330, event: header_read, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.761875 7f869af78700 5 -- op tracker -- , seq: 2940, time: 2014-06-09 16:22:11.607330, event: throttled, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.761882 7f869af78700 5 -- op tracker -- , seq: 2940, time: 2014-06-09 16:22:11.607340, event: all_read, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.761889 7f869af78700 5 -- op tracker -- , seq: 2940, time: 2014-06-09 16:22:11.761859, event: dispatched, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.761901 7f869af78700 5 -- op tracker -- , seq: 2940, time: 2014-06-09 16:22:11.761899, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.761906 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.759) v4 from osd.1 2014-06-09 16:22:11.761909 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ef6c0 2014-06-09 16:22:11.761914 7f869af78700 5 -- op tracker -- , seq: 2940, time: 2014-06-09 16:22:11.761912, event: started, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.761929 7f869af78700 5 -- op tracker -- , seq: 2940, time: 2014-06-09 16:22:11.761927, event: done, op: pg_info(1 pgs e243:2.759) 2014-06-09 16:22:11.761936 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761938 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761949 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.761951 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.761953 7f869af78700 20 osd.26 243 _dispatch 0x2fc94c40 pg_info(1 pgs e243:3.98) v4 2014-06-09 16:22:11.761958 7f869af78700 5 -- op tracker -- , seq: 2941, time: 2014-06-09 16:22:11.613541, event: header_read, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.761970 7f869af78700 5 -- op tracker -- , seq: 2941, time: 2014-06-09 16:22:11.613542, event: throttled, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.761980 7f869af78700 5 -- op tracker -- , seq: 2941, time: 2014-06-09 16:22:11.613554, event: all_read, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.761990 7f869af78700 5 -- op tracker -- , seq: 2941, time: 2014-06-09 16:22:11.761949, event: dispatched, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.761999 7f869af78700 5 -- op tracker -- , seq: 2941, time: 2014-06-09 16:22:11.761997, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.762004 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.98) v4 from osd.9 2014-06-09 16:22:11.762007 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc94c40 2014-06-09 16:22:11.762011 7f869af78700 5 -- op tracker -- , seq: 2941, time: 2014-06-09 16:22:11.762009, event: started, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.762029 7f869af78700 5 -- op tracker -- , seq: 2941, time: 2014-06-09 16:22:11.762027, event: done, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.762037 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762039 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762051 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762054 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762055 7f869af78700 20 osd.26 243 _dispatch 0x30dc21c0 pg_info(1 pgs e243:1.a6) v4 2014-06-09 16:22:11.762060 7f869af78700 5 -- op tracker -- , seq: 2942, time: 2014-06-09 16:22:11.574406, event: header_read, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.762067 7f869af78700 5 -- op tracker -- , seq: 2942, time: 2014-06-09 16:22:11.574406, event: throttled, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.762073 7f869af78700 5 -- op tracker -- , seq: 2942, time: 2014-06-09 16:22:11.574420, event: all_read, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.762080 7f869af78700 5 -- op tracker -- , seq: 2942, time: 2014-06-09 16:22:11.762051, event: dispatched, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.762087 7f869af78700 5 -- op tracker -- , seq: 2942, time: 2014-06-09 16:22:11.762085, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.762096 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a6) v4 from osd.8 2014-06-09 16:22:11.762100 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dc21c0 2014-06-09 16:22:11.762105 7f869af78700 5 -- op tracker -- , seq: 2942, time: 2014-06-09 16:22:11.762103, event: started, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.762122 7f869af78700 5 -- op tracker -- , seq: 2942, time: 2014-06-09 16:22:11.762120, event: done, op: pg_info(1 pgs e243:1.a6) 2014-06-09 16:22:11.762130 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762132 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762145 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762146 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762148 7f869af78700 20 osd.26 243 _dispatch 0x30d9dc00 pg_info(1 pgs e243:0.4c9) v4 2014-06-09 16:22:11.762152 7f869af78700 5 -- op tracker -- , seq: 2943, time: 2014-06-09 16:22:11.596960, event: header_read, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.762159 7f869af78700 5 -- op tracker -- , seq: 2943, time: 2014-06-09 16:22:11.596960, event: throttled, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.762171 7f869af78700 5 -- op tracker -- , seq: 2943, time: 2014-06-09 16:22:11.596972, event: all_read, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.762177 7f869af78700 5 -- op tracker -- , seq: 2943, time: 2014-06-09 16:22:11.762144, event: dispatched, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.762184 7f869af78700 5 -- op tracker -- , seq: 2943, time: 2014-06-09 16:22:11.762182, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.762189 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4c9) v4 from osd.6 2014-06-09 16:22:11.762192 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9dc00 2014-06-09 16:22:11.762197 7f869af78700 5 -- op tracker -- , seq: 2943, time: 2014-06-09 16:22:11.762195, event: started, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.762212 7f869af78700 5 -- op tracker -- , seq: 2943, time: 2014-06-09 16:22:11.762210, event: done, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.762221 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762224 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762236 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762238 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762239 7f869af78700 20 osd.26 243 _dispatch 0x3130b6c0 pg_info(1 pgs e243:1.15b) v4 2014-06-09 16:22:11.762244 7f869af78700 5 -- op tracker -- , seq: 2944, time: 2014-06-09 16:22:11.640954, event: header_read, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.762251 7f869af78700 5 -- op tracker -- , seq: 2944, time: 2014-06-09 16:22:11.640955, event: throttled, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.762263 7f869af78700 5 -- op tracker -- , seq: 2944, time: 2014-06-09 16:22:11.640966, event: all_read, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.762269 7f869af78700 5 -- op tracker -- , seq: 2944, time: 2014-06-09 16:22:11.762235, event: dispatched, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.762278 7f869af78700 5 -- op tracker -- , seq: 2944, time: 2014-06-09 16:22:11.762276, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.762283 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.15b) v4 from osd.0 2014-06-09 16:22:11.762286 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130b6c0 2014-06-09 16:22:11.762291 7f869af78700 5 -- op tracker -- , seq: 2944, time: 2014-06-09 16:22:11.762289, event: started, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.762313 7f869af78700 5 -- op tracker -- , seq: 2944, time: 2014-06-09 16:22:11.762310, event: done, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.762320 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762322 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762334 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762335 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762337 7f869af78700 20 osd.26 243 _dispatch 0x30dae000 pg_info(1 pgs e243:1.408) v4 2014-06-09 16:22:11.762342 7f869af78700 5 -- op tracker -- , seq: 2945, time: 2014-06-09 16:22:11.597093, event: header_read, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.762349 7f869af78700 5 -- op tracker -- , seq: 2945, time: 2014-06-09 16:22:11.597093, event: throttled, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.762355 7f869af78700 5 -- op tracker -- , seq: 2945, time: 2014-06-09 16:22:11.597105, event: all_read, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.762362 7f869af78700 5 -- op tracker -- , seq: 2945, time: 2014-06-09 16:22:11.762333, event: dispatched, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.762374 7f869af78700 5 -- op tracker -- , seq: 2945, time: 2014-06-09 16:22:11.762372, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.762383 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.408) v4 from osd.5 2014-06-09 16:22:11.762386 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30dae000 2014-06-09 16:22:11.762391 7f869af78700 5 -- op tracker -- , seq: 2945, time: 2014-06-09 16:22:11.762389, event: started, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.762406 7f869af78700 5 -- op tracker -- , seq: 2945, time: 2014-06-09 16:22:11.762404, event: done, op: pg_info(1 pgs e243:1.408) 2014-06-09 16:22:11.762414 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762416 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762427 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762429 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762430 7f869af78700 20 osd.26 243 _dispatch 0x1405dc00 pg_info(1 pgs e243:3.562) v4 2014-06-09 16:22:11.762435 7f869af78700 5 -- op tracker -- , seq: 2946, time: 2014-06-09 16:22:11.463641, event: header_read, op: pg_info(1 pgs e243:3.562) 2014-06-09 16:22:11.762441 7f869af78700 5 -- op tracker -- , seq: 2946, time: 2014-06-09 16:22:11.463643, event: throttled, op: pg_info(1 pgs e243:3.562) 2014-06-09 16:22:11.762453 7f869af78700 5 -- op tracker -- , seq: 2946, time: 2014-06-09 16:22:11.463660, event: all_read, op: pg_info(1 pgs e243:3.562) 2014-06-09 16:22:11.762460 7f869af78700 5 -- op tracker -- , seq: 2946, time: 2014-06-09 16:22:11.762427, event: dispatched, op: pg_info(1 pgs e243:3.562) 2014-06-09 16:22:11.762467 7f869af78700 5 -- op tracker -- , seq: 2946, time: 2014-06-09 16:22:11.762465, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.562) 2014-06-09 16:22:11.762472 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.562) v4 from osd.2 2014-06-09 16:22:11.762475 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405dc00 2014-06-09 16:22:11.762481 7f869af78700 5 -- op tracker -- , seq: 2946, time: 2014-06-09 16:22:11.762479, event: started, op: pg_info(1 pgs e243:3.562) 2014-06-09 16:22:11.762497 7f869af78700 5 -- op tracker -- , seq: 2946, time: 2014-06-09 16:22:11.762495, event: done, op: pg_info(1 pgs e243:3.562) 2014-06-09 16:22:11.762505 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762507 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762519 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762521 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762523 7f869af78700 20 osd.26 243 _dispatch 0x2eee61c0 pg_info(1 pgs e243:1.43b) v4 2014-06-09 16:22:11.762528 7f869af78700 5 -- op tracker -- , seq: 2947, time: 2014-06-09 16:22:11.447759, event: header_read, op: pg_info(1 pgs e243:1.43b) 2014-06-09 16:22:11.762534 7f869af78700 5 -- op tracker -- , seq: 2947, time: 2014-06-09 16:22:11.447760, event: throttled, op: pg_info(1 pgs e243:1.43b) 2014-06-09 16:22:11.762546 7f869af78700 5 -- op tracker -- , seq: 2947, time: 2014-06-09 16:22:11.447770, event: all_read, op: pg_info(1 pgs e243:1.43b) 2014-06-09 16:22:11.762556 7f869af78700 5 -- op tracker -- , seq: 2947, time: 2014-06-09 16:22:11.762519, event: dispatched, op: pg_info(1 pgs e243:1.43b) 2014-06-09 16:22:11.762567 7f869af78700 5 -- op tracker -- , seq: 2947, time: 2014-06-09 16:22:11.762565, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.43b) 2014-06-09 16:22:11.762575 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.43b) v4 from osd.4 2014-06-09 16:22:11.762578 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee61c0 2014-06-09 16:22:11.762583 7f869af78700 5 -- op tracker -- , seq: 2947, time: 2014-06-09 16:22:11.762581, event: started, op: pg_info(1 pgs e243:1.43b) 2014-06-09 16:22:11.762598 7f869af78700 5 -- op tracker -- , seq: 2947, time: 2014-06-09 16:22:11.762596, event: done, op: pg_info(1 pgs e243:1.43b) 2014-06-09 16:22:11.762607 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762609 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762622 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762623 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762625 7f869af78700 20 osd.26 243 _dispatch 0xcd78a80 pg_info(1 pgs e243:3.579) v4 2014-06-09 16:22:11.762685 7f869af78700 5 -- op tracker -- , seq: 2948, time: 2014-06-09 16:22:11.658708, event: header_read, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.762694 7f869af78700 5 -- op tracker -- , seq: 2948, time: 2014-06-09 16:22:11.658708, event: throttled, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.762700 7f869af78700 5 -- op tracker -- , seq: 2948, time: 2014-06-09 16:22:11.658721, event: all_read, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.762705 7f869af78700 5 -- op tracker -- , seq: 2948, time: 2014-06-09 16:22:11.762622, event: dispatched, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.762711 7f869af78700 5 -- op tracker -- , seq: 2948, time: 2014-06-09 16:22:11.762709, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.762715 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.579) v4 from osd.7 2014-06-09 16:22:11.762717 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd78a80 2014-06-09 16:22:11.762721 7f869af78700 5 -- op tracker -- , seq: 2948, time: 2014-06-09 16:22:11.762720, event: started, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.762736 7f869af78700 5 -- op tracker -- , seq: 2948, time: 2014-06-09 16:22:11.762734, event: done, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.762743 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762744 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762755 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762756 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762757 7f869af78700 20 osd.26 243 _dispatch 0x30c31500 pg_info(1 pgs e243:3.5c6) v4 2014-06-09 16:22:11.762761 7f869af78700 5 -- op tracker -- , seq: 2949, time: 2014-06-09 16:22:11.607343, event: header_read, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.762767 7f869af78700 5 -- op tracker -- , seq: 2949, time: 2014-06-09 16:22:11.607343, event: throttled, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.762772 7f869af78700 5 -- op tracker -- , seq: 2949, time: 2014-06-09 16:22:11.607358, event: all_read, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.762776 7f869af78700 5 -- op tracker -- , seq: 2949, time: 2014-06-09 16:22:11.762754, event: dispatched, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.762782 7f869af78700 5 -- op tracker -- , seq: 2949, time: 2014-06-09 16:22:11.762781, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.762786 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5c6) v4 from osd.3 2014-06-09 16:22:11.762788 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c31500 2014-06-09 16:22:11.762792 7f869af78700 5 -- op tracker -- , seq: 2949, time: 2014-06-09 16:22:11.762790, event: started, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.762805 7f869af78700 5 -- op tracker -- , seq: 2949, time: 2014-06-09 16:22:11.762803, event: done, op: pg_info(1 pgs e243:3.5c6) 2014-06-09 16:22:11.762812 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762813 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762823 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762824 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762825 7f869af78700 20 osd.26 243 _dispatch 0x240ef500 pg_info(1 pgs e243:3.254) v4 2014-06-09 16:22:11.762829 7f869af78700 5 -- op tracker -- , seq: 2950, time: 2014-06-09 16:22:11.607348, event: header_read, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.762838 7f869af78700 5 -- op tracker -- , seq: 2950, time: 2014-06-09 16:22:11.607349, event: throttled, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.762845 7f869af78700 5 -- op tracker -- , seq: 2950, time: 2014-06-09 16:22:11.607356, event: all_read, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.762851 7f869af78700 5 -- op tracker -- , seq: 2950, time: 2014-06-09 16:22:11.762823, event: dispatched, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.762858 7f869af78700 5 -- op tracker -- , seq: 2950, time: 2014-06-09 16:22:11.762856, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.762863 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.254) v4 from osd.1 2014-06-09 16:22:11.762866 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240ef500 2014-06-09 16:22:11.762871 7f869af78700 5 -- op tracker -- , seq: 2950, time: 2014-06-09 16:22:11.762869, event: started, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.762889 7f869af78700 5 -- op tracker -- , seq: 2950, time: 2014-06-09 16:22:11.762887, event: done, op: pg_info(1 pgs e243:3.254) 2014-06-09 16:22:11.762896 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762898 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762910 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762911 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762913 7f869af78700 20 osd.26 243 _dispatch 0x1003a1c0 pg_info(1 pgs e243:3.35c) v4 2014-06-09 16:22:11.762918 7f869af78700 5 -- op tracker -- , seq: 2951, time: 2014-06-09 16:22:11.613583, event: header_read, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.762925 7f869af78700 5 -- op tracker -- , seq: 2951, time: 2014-06-09 16:22:11.613583, event: throttled, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.762931 7f869af78700 5 -- op tracker -- , seq: 2951, time: 2014-06-09 16:22:11.613596, event: all_read, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.762940 7f869af78700 5 -- op tracker -- , seq: 2951, time: 2014-06-09 16:22:11.762909, event: dispatched, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.762947 7f869af78700 5 -- op tracker -- , seq: 2951, time: 2014-06-09 16:22:11.762945, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.762951 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.35c) v4 from osd.9 2014-06-09 16:22:11.762954 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1003a1c0 2014-06-09 16:22:11.762959 7f869af78700 5 -- op tracker -- , seq: 2951, time: 2014-06-09 16:22:11.762957, event: started, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.762973 7f869af78700 5 -- op tracker -- , seq: 2951, time: 2014-06-09 16:22:11.762971, event: done, op: pg_info(1 pgs e243:3.35c) 2014-06-09 16:22:11.762982 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762984 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762996 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.762998 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.762999 7f869af78700 20 osd.26 243 _dispatch 0x2eef0e00 pg_info(1 pgs e243:0.36d) v4 2014-06-09 16:22:11.763005 7f869af78700 5 -- op tracker -- , seq: 2952, time: 2014-06-09 16:22:11.575390, event: header_read, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.763012 7f869af78700 5 -- op tracker -- , seq: 2952, time: 2014-06-09 16:22:11.575391, event: throttled, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.763021 7f869af78700 5 -- op tracker -- , seq: 2952, time: 2014-06-09 16:22:11.575404, event: all_read, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.763028 7f869af78700 5 -- op tracker -- , seq: 2952, time: 2014-06-09 16:22:11.762996, event: dispatched, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.763034 7f869af78700 5 -- op tracker -- , seq: 2952, time: 2014-06-09 16:22:11.763032, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.763043 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.36d) v4 from osd.8 2014-06-09 16:22:11.763046 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef0e00 2014-06-09 16:22:11.763051 7f869af78700 5 -- op tracker -- , seq: 2952, time: 2014-06-09 16:22:11.763049, event: started, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.763068 7f869af78700 5 -- op tracker -- , seq: 2952, time: 2014-06-09 16:22:11.763065, event: done, op: pg_info(1 pgs e243:0.36d) 2014-06-09 16:22:11.763075 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763077 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763089 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763090 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763092 7f869af78700 20 osd.26 243 _dispatch 0x30d9da40 pg_info(1 pgs e243:0.4c9) v4 2014-06-09 16:22:11.763097 7f869af78700 5 -- op tracker -- , seq: 2953, time: 2014-06-09 16:22:11.596982, event: header_read, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.763104 7f869af78700 5 -- op tracker -- , seq: 2953, time: 2014-06-09 16:22:11.596982, event: throttled, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.763110 7f869af78700 5 -- op tracker -- , seq: 2953, time: 2014-06-09 16:22:11.596994, event: all_read, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.763119 7f869af78700 5 -- op tracker -- , seq: 2953, time: 2014-06-09 16:22:11.763088, event: dispatched, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.763126 7f869af78700 5 -- op tracker -- , seq: 2953, time: 2014-06-09 16:22:11.763124, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.763130 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4c9) v4 from osd.6 2014-06-09 16:22:11.763133 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9da40 2014-06-09 16:22:11.763137 7f869af78700 5 -- op tracker -- , seq: 2953, time: 2014-06-09 16:22:11.763135, event: started, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.763174 7f869af78700 5 -- op tracker -- , seq: 2953, time: 2014-06-09 16:22:11.763153, event: done, op: pg_info(1 pgs e243:0.4c9) 2014-06-09 16:22:11.763187 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763189 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763200 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763202 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763203 7f869af78700 20 osd.26 243 _dispatch 0x2fc95500 pg_info(1 pgs e243:1.15b) v4 2014-06-09 16:22:11.763208 7f869af78700 5 -- op tracker -- , seq: 2954, time: 2014-06-09 16:22:11.641204, event: header_read, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.763215 7f869af78700 5 -- op tracker -- , seq: 2954, time: 2014-06-09 16:22:11.641205, event: throttled, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.763222 7f869af78700 5 -- op tracker -- , seq: 2954, time: 2014-06-09 16:22:11.641215, event: all_read, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.763228 7f869af78700 5 -- op tracker -- , seq: 2954, time: 2014-06-09 16:22:11.763200, event: dispatched, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.763240 7f869af78700 5 -- op tracker -- , seq: 2954, time: 2014-06-09 16:22:11.763237, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.763244 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.15b) v4 from osd.0 2014-06-09 16:22:11.763247 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc95500 2014-06-09 16:22:11.763251 7f869af78700 5 -- op tracker -- , seq: 2954, time: 2014-06-09 16:22:11.763250, event: started, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.763266 7f869af78700 5 -- op tracker -- , seq: 2954, time: 2014-06-09 16:22:11.763264, event: done, op: pg_info(1 pgs e243:1.15b) 2014-06-09 16:22:11.763274 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763276 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763287 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763289 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763290 7f869af78700 20 osd.26 243 _dispatch 0x30d9fdc0 pg_info(1 pgs e243:2.654) v4 2014-06-09 16:22:11.763295 7f869af78700 5 -- op tracker -- , seq: 2955, time: 2014-06-09 16:22:11.597114, event: header_read, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.763305 7f869af78700 5 -- op tracker -- , seq: 2955, time: 2014-06-09 16:22:11.597115, event: throttled, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.763311 7f869af78700 5 -- op tracker -- , seq: 2955, time: 2014-06-09 16:22:11.597134, event: all_read, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.763317 7f869af78700 5 -- op tracker -- , seq: 2955, time: 2014-06-09 16:22:11.763287, event: dispatched, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.763326 7f869af78700 5 -- op tracker -- , seq: 2955, time: 2014-06-09 16:22:11.763324, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.763331 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.654) v4 from osd.5 2014-06-09 16:22:11.763334 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9fdc0 2014-06-09 16:22:11.763339 7f869af78700 5 -- op tracker -- , seq: 2955, time: 2014-06-09 16:22:11.763337, event: started, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.763356 7f869af78700 5 -- op tracker -- , seq: 2955, time: 2014-06-09 16:22:11.763354, event: done, op: pg_info(1 pgs e243:2.654) 2014-06-09 16:22:11.763363 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763365 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763377 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763379 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763380 7f869af78700 20 osd.26 243 _dispatch 0x29ec41c0 pg_info(1 pgs e243:3.74) v4 2014-06-09 16:22:11.763385 7f869af78700 5 -- op tracker -- , seq: 2956, time: 2014-06-09 16:22:11.471659, event: header_read, op: pg_info(1 pgs e243:3.74) 2014-06-09 16:22:11.763392 7f869af78700 5 -- op tracker -- , seq: 2956, time: 2014-06-09 16:22:11.471659, event: throttled, op: pg_info(1 pgs e243:3.74) 2014-06-09 16:22:11.763399 7f869af78700 5 -- op tracker -- , seq: 2956, time: 2014-06-09 16:22:11.471670, event: all_read, op: pg_info(1 pgs e243:3.74) 2014-06-09 16:22:11.763405 7f869af78700 5 -- op tracker -- , seq: 2956, time: 2014-06-09 16:22:11.763377, event: dispatched, op: pg_info(1 pgs e243:3.74) 2014-06-09 16:22:11.763412 7f869af78700 5 -- op tracker -- , seq: 2956, time: 2014-06-09 16:22:11.763410, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.74) 2014-06-09 16:22:11.763416 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.74) v4 from osd.2 2014-06-09 16:22:11.763419 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec41c0 2014-06-09 16:22:11.763424 7f869af78700 5 -- op tracker -- , seq: 2956, time: 2014-06-09 16:22:11.763422, event: started, op: pg_info(1 pgs e243:3.74) 2014-06-09 16:22:11.763441 7f869af78700 5 -- op tracker -- , seq: 2956, time: 2014-06-09 16:22:11.763439, event: done, op: pg_info(1 pgs e243:3.74) 2014-06-09 16:22:11.763458 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763460 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763472 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763475 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763476 7f869af78700 20 osd.26 243 _dispatch 0x2eee6000 pg_info(1 pgs e243:0.708) v4 2014-06-09 16:22:11.763479 7f8682ae5700 5 -- op tracker -- , seq: 2957, time: 2014-06-09 16:22:11.763446, event: header_read, op: osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.763492 7f8682ae5700 5 -- op tracker -- , seq: 2957, time: 2014-06-09 16:22:11.763447, event: throttled, op: osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.763505 7f8682ae5700 5 -- op tracker -- , seq: 2957, time: 2014-06-09 16:22:11.763456, event: all_read, op: osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.763517 7f8682ae5700 5 -- op tracker -- , seq: 2957, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.763523 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.763529 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.763529 7f869af78700 5 -- op tracker -- , seq: 2959, time: 2014-06-09 16:22:11.447775, event: header_read, op: pg_info(1 pgs e243:0.708) 2014-06-09 16:22:11.763537 7f8682ae5700 15 osd.26 243 enqueue_op 0xdfa7000 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.763540 7f869af78700 5 -- op tracker -- , seq: 2959, time: 2014-06-09 16:22:11.447775, event: throttled, op: pg_info(1 pgs e243:0.708) 2014-06-09 16:22:11.763549 7f869af78700 5 -- op tracker -- , seq: 2959, time: 2014-06-09 16:22:11.447786, event: all_read, op: pg_info(1 pgs e243:0.708) 2014-06-09 16:22:11.763560 7f8682be6700 5 -- op tracker -- , seq: 2958, time: 2014-06-09 16:22:11.763472, event: header_read, op: osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.763570 7f8682be6700 5 -- op tracker -- , seq: 2958, time: 2014-06-09 16:22:11.763473, event: throttled, op: osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.763586 7f8682be6700 5 -- op tracker -- , seq: 2958, time: 2014-06-09 16:22:11.763485, event: all_read, op: osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.763594 7f8682be6700 5 -- op tracker -- , seq: 2958, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.763599 7f8682be6700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.763604 7f8682be6700 20 osd.26 243 should_share_map osd.5 172.16.96.201:6837/51639 243 2014-06-09 16:22:11.763610 7f8682be6700 15 osd.26 243 enqueue_op 0xdfa7200 prio 196 cost 0 latency 0.000138 osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.763634 7f869af78700 5 -- op tracker -- , seq: 2959, time: 2014-06-09 16:22:11.763472, event: dispatched, op: pg_info(1 pgs e243:0.708) 2014-06-09 16:22:11.763645 7f869af78700 5 -- op tracker -- , seq: 2959, time: 2014-06-09 16:22:11.763642, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.708) 2014-06-09 16:22:11.763649 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.708) v4 from osd.4 2014-06-09 16:22:11.763653 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eee6000 2014-06-09 16:22:11.763658 7f869af78700 5 -- op tracker -- , seq: 2959, time: 2014-06-09 16:22:11.763656, event: started, op: pg_info(1 pgs e243:0.708) 2014-06-09 16:22:11.763679 7f869af78700 5 -- op tracker -- , seq: 2959, time: 2014-06-09 16:22:11.763676, event: done, op: pg_info(1 pgs e243:0.708) 2014-06-09 16:22:11.763686 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763688 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763701 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763703 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763704 7f869af78700 20 osd.26 243 _dispatch 0xcd78c40 pg_info(1 pgs e243:3.2b6) v4 2014-06-09 16:22:11.763709 7f869af78700 5 -- op tracker -- , seq: 2960, time: 2014-06-09 16:22:11.658747, event: header_read, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.763721 7f869af78700 5 -- op tracker -- , seq: 2960, time: 2014-06-09 16:22:11.658747, event: throttled, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.763731 7f869af78700 5 -- op tracker -- , seq: 2960, time: 2014-06-09 16:22:11.658759, event: all_read, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.763740 7f869af78700 5 -- op tracker -- , seq: 2960, time: 2014-06-09 16:22:11.763701, event: dispatched, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.763746 7f869af78700 5 -- op tracker -- , seq: 2960, time: 2014-06-09 16:22:11.763744, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.763751 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2b6) v4 from osd.7 2014-06-09 16:22:11.763754 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd78c40 2014-06-09 16:22:11.763759 7f869af78700 5 -- op tracker -- , seq: 2960, time: 2014-06-09 16:22:11.763757, event: started, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.763775 7f869af78700 5 -- op tracker -- , seq: 2960, time: 2014-06-09 16:22:11.763773, event: done, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.763783 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763785 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763796 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763798 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763800 7f869af78700 20 osd.26 243 _dispatch 0x2fcaa540 pg_info(1 pgs e243:2.699) v4 2014-06-09 16:22:11.763804 7f869af78700 5 -- op tracker -- , seq: 2961, time: 2014-06-09 16:22:11.612260, event: header_read, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.763811 7f869af78700 5 -- op tracker -- , seq: 2961, time: 2014-06-09 16:22:11.612261, event: throttled, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.763819 7f869af78700 5 -- op tracker -- , seq: 2961, time: 2014-06-09 16:22:11.612274, event: all_read, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.763825 7f869af78700 5 -- op tracker -- , seq: 2961, time: 2014-06-09 16:22:11.763796, event: dispatched, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.763837 7f869af78700 5 -- op tracker -- , seq: 2961, time: 2014-06-09 16:22:11.763835, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.763842 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.699) v4 from osd.3 2014-06-09 16:22:11.763845 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaa540 2014-06-09 16:22:11.763850 7f869af78700 5 -- op tracker -- , seq: 2961, time: 2014-06-09 16:22:11.763848, event: started, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.763865 7f869af78700 5 -- op tracker -- , seq: 2961, time: 2014-06-09 16:22:11.763863, event: done, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.763872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763874 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763886 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763888 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763889 7f869af78700 20 osd.26 243 _dispatch 0x240efc00 pg_info(1 pgs e243:1.494) v4 2014-06-09 16:22:11.763894 7f869af78700 5 -- op tracker -- , seq: 2962, time: 2014-06-09 16:22:11.607359, event: header_read, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.763901 7f869af78700 5 -- op tracker -- , seq: 2962, time: 2014-06-09 16:22:11.607360, event: throttled, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.763910 7f869af78700 5 -- op tracker -- , seq: 2962, time: 2014-06-09 16:22:11.607370, event: all_read, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.763916 7f869af78700 5 -- op tracker -- , seq: 2962, time: 2014-06-09 16:22:11.763885, event: dispatched, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.763928 7f869af78700 5 -- op tracker -- , seq: 2962, time: 2014-06-09 16:22:11.763925, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.763932 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.494) v4 from osd.1 2014-06-09 16:22:11.763935 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240efc00 2014-06-09 16:22:11.763940 7f869af78700 5 -- op tracker -- , seq: 2962, time: 2014-06-09 16:22:11.763938, event: started, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.763955 7f869af78700 5 -- op tracker -- , seq: 2962, time: 2014-06-09 16:22:11.763952, event: done, op: pg_info(1 pgs e243:1.494) 2014-06-09 16:22:11.763963 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763965 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763977 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.763978 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.763980 7f869af78700 20 osd.26 243 _dispatch 0x190b7dc0 pg_info(1 pgs e243:3.98) v4 2014-06-09 16:22:11.763984 7f869af78700 5 -- op tracker -- , seq: 2963, time: 2014-06-09 16:22:11.613628, event: header_read, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.763992 7f869af78700 5 -- op tracker -- , seq: 2963, time: 2014-06-09 16:22:11.613629, event: throttled, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.763998 7f869af78700 5 -- op tracker -- , seq: 2963, time: 2014-06-09 16:22:11.613641, event: all_read, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.764005 7f869af78700 5 -- op tracker -- , seq: 2963, time: 2014-06-09 16:22:11.763976, event: dispatched, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.764014 7f869af78700 5 -- op tracker -- , seq: 2963, time: 2014-06-09 16:22:11.764012, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.764018 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.98) v4 from osd.9 2014-06-09 16:22:11.764021 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b7dc0 2014-06-09 16:22:11.764037 7f869af78700 5 -- op tracker -- , seq: 2963, time: 2014-06-09 16:22:11.764024, event: started, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.764051 7f8670072700 5 -- op tracker -- , seq: 2964, time: 2014-06-09 16:22:11.760640, event: header_read, op: osd_op(client.4538.0:36828 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.764067 7f8670072700 5 -- op tracker -- , seq: 2964, time: 2014-06-09 16:22:11.760641, event: throttled, op: osd_op(client.4538.0:36828 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.764098 7f8670072700 5 -- op tracker -- , seq: 2964, time: 2014-06-09 16:22:11.764012, event: all_read, op: osd_op(client.4538.0:36828 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.764111 7f8670072700 5 -- op tracker -- , seq: 2964, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36828 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.764117 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.764153 7f8670072700 15 osd.26 243 enqueue_op 0x257c4300 prio 63 cost 516096 latency 0.003513 osd_op(client.4538.0:36828 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.764447 7f869af78700 5 -- op tracker -- , seq: 2963, time: 2014-06-09 16:22:11.764051, event: done, op: pg_info(1 pgs e243:3.98) 2014-06-09 16:22:11.764458 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.764460 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.764473 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.764475 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.764476 7f869af78700 20 osd.26 243 _dispatch 0x2eef0c40 pg_info(1 pgs e243:1.7a) v4 2014-06-09 16:22:11.764481 7f869af78700 5 -- op tracker -- , seq: 2965, time: 2014-06-09 16:22:11.575438, event: header_read, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.764488 7f869af78700 5 -- op tracker -- , seq: 2965, time: 2014-06-09 16:22:11.575438, event: throttled, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.764500 7f869af78700 5 -- op tracker -- , seq: 2965, time: 2014-06-09 16:22:11.575450, event: all_read, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.764511 7f869af78700 5 -- op tracker -- , seq: 2965, time: 2014-06-09 16:22:11.764473, event: dispatched, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.764541 7f869af78700 5 -- op tracker -- , seq: 2965, time: 2014-06-09 16:22:11.764519, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.764615 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.7a) v4 from osd.8 2014-06-09 16:22:11.764619 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef0c40 2014-06-09 16:22:11.764625 7f869af78700 5 -- op tracker -- , seq: 2965, time: 2014-06-09 16:22:11.764622, event: started, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.764653 7f869af78700 5 -- op tracker -- , seq: 2965, time: 2014-06-09 16:22:11.764651, event: done, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.764661 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.764663 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.764676 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.764677 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.764679 7f869af78700 20 osd.26 243 _dispatch 0x30de4e00 pg_info(1 pgs e243:1.122) v4 2014-06-09 16:22:11.764684 7f869af78700 5 -- op tracker -- , seq: 2966, time: 2014-06-09 16:22:11.603294, event: header_read, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.764696 7f869af78700 5 -- op tracker -- , seq: 2966, time: 2014-06-09 16:22:11.603295, event: throttled, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.764705 7f869af78700 5 -- op tracker -- , seq: 2966, time: 2014-06-09 16:22:11.603307, event: all_read, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.764713 7f869af78700 5 -- op tracker -- , seq: 2966, time: 2014-06-09 16:22:11.764675, event: dispatched, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.764723 7f869af78700 5 -- op tracker -- , seq: 2966, time: 2014-06-09 16:22:11.764720, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.764729 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.122) v4 from osd.6 2014-06-09 16:22:11.764732 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de4e00 2014-06-09 16:22:11.764734 7f8670072700 5 -- op tracker -- , seq: 2967, time: 2014-06-09 16:22:11.764176, event: header_read, op: osd_op(client.4538.0:36829 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.764748 7f8670072700 5 -- op tracker -- , seq: 2967, time: 2014-06-09 16:22:11.764177, event: throttled, op: osd_op(client.4538.0:36829 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.764754 7f8670072700 5 -- op tracker -- , seq: 2967, time: 2014-06-09 16:22:11.764667, event: all_read, op: osd_op(client.4538.0:36829 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.764759 7f8670072700 5 -- op tracker -- , seq: 2967, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36829 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.764764 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.764778 7f8670072700 15 osd.26 243 enqueue_op 0x1b0d7c00 prio 63 cost 516096 latency 0.000602 osd_op(client.4538.0:36829 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.764810 7f869af78700 5 -- op tracker -- , seq: 2966, time: 2014-06-09 16:22:11.764735, event: started, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.764830 7f869af78700 5 -- op tracker -- , seq: 2966, time: 2014-06-09 16:22:11.764828, event: done, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.764838 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.764840 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.764854 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.764856 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.764858 7f869af78700 20 osd.26 243 _dispatch 0x2fc568c0 pg_info(1 pgs e243:0.117) v4 2014-06-09 16:22:11.764863 7f869af78700 5 -- op tracker -- , seq: 2968, time: 2014-06-09 16:22:11.652028, event: header_read, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.764869 7f869af78700 5 -- op tracker -- , seq: 2968, time: 2014-06-09 16:22:11.652029, event: throttled, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.764876 7f869af78700 5 -- op tracker -- , seq: 2968, time: 2014-06-09 16:22:11.652041, event: all_read, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.764888 7f869af78700 5 -- op tracker -- , seq: 2968, time: 2014-06-09 16:22:11.764854, event: dispatched, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.764895 7f869af78700 5 -- op tracker -- , seq: 2968, time: 2014-06-09 16:22:11.764893, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.764902 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.117) v4 from osd.0 2014-06-09 16:22:11.764905 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc568c0 2014-06-09 16:22:11.764911 7f869af78700 5 -- op tracker -- , seq: 2968, time: 2014-06-09 16:22:11.764909, event: started, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.764940 7f869af78700 5 -- op tracker -- , seq: 2968, time: 2014-06-09 16:22:11.764938, event: done, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.764951 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.764953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.764968 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.764970 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.764971 7f869af78700 20 osd.26 243 _dispatch 0x30a3a8c0 pg_info(1 pgs e243:3.139) v4 2014-06-09 16:22:11.764976 7f869af78700 5 -- op tracker -- , seq: 2969, time: 2014-06-09 16:22:11.604547, event: header_read, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.764983 7f869af78700 5 -- op tracker -- , seq: 2969, time: 2014-06-09 16:22:11.604548, event: throttled, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.764990 7f869af78700 5 -- op tracker -- , seq: 2969, time: 2014-06-09 16:22:11.604560, event: all_read, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.764996 7f869af78700 5 -- op tracker -- , seq: 2969, time: 2014-06-09 16:22:11.764967, event: dispatched, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.765003 7f869af78700 5 -- op tracker -- , seq: 2969, time: 2014-06-09 16:22:11.765001, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.765008 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.139) v4 from osd.5 2014-06-09 16:22:11.765011 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3a8c0 2014-06-09 16:22:11.765016 7f869af78700 5 -- op tracker -- , seq: 2969, time: 2014-06-09 16:22:11.765014, event: started, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.765034 7f869af78700 5 -- op tracker -- , seq: 2969, time: 2014-06-09 16:22:11.765031, event: done, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.765608 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.765611 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.765618 7f8670072700 5 -- op tracker -- , seq: 2970, time: 2014-06-09 16:22:11.764797, event: header_read, op: osd_op(client.4538.0:36830 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.765627 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.765629 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.765631 7f869af78700 20 osd.26 243 _dispatch 0x2d740540 pg_info(1 pgs e243:3.2b) v4 2014-06-09 16:22:11.765630 7f8670072700 5 -- op tracker -- , seq: 2970, time: 2014-06-09 16:22:11.764798, event: throttled, op: osd_op(client.4538.0:36830 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.765647 7f8670072700 5 -- op tracker -- , seq: 2970, time: 2014-06-09 16:22:11.765229, event: all_read, op: osd_op(client.4538.0:36830 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.765660 7f8670072700 5 -- op tracker -- , seq: 2970, time: 0.000000, event: dispatched, op: osd_op(client.4538.0:36830 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.765667 7f8670072700 20 osd.26 243 should_share_map client.4538 172.16.96.220:0/1123636 243 2014-06-09 16:22:11.765684 7f8670072700 15 osd.26 243 enqueue_op 0xdfa7900 prio 63 cost 364544 latency 0.000887 osd_op(client.4538.0:36830 rbd_data.115a6b8b4567.0000000000001a5c [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.8b1d12a5 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.767702 7f8682ae5700 5 -- op tracker -- , seq: 2971, time: 2014-06-09 16:22:11.767662, event: header_read, op: osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.767711 7f8682ae5700 5 -- op tracker -- , seq: 2971, time: 2014-06-09 16:22:11.767664, event: throttled, op: osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.767720 7f8682ae5700 5 -- op tracker -- , seq: 2971, time: 2014-06-09 16:22:11.767677, event: all_read, op: osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.767732 7f8682ae5700 5 -- op tracker -- , seq: 2971, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.767737 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.767742 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.767743 7f869af78700 5 -- op tracker -- , seq: 2972, time: 2014-06-09 16:22:11.507644, event: header_read, op: pg_info(1 pgs e243:3.2b) 2014-06-09 16:22:11.767749 7f8682ae5700 15 osd.26 243 enqueue_op 0xdfa6f00 prio 196 cost 0 latency 0.000087 osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.767756 7f869af78700 5 -- op tracker -- , seq: 2972, time: 2014-06-09 16:22:11.507645, event: throttled, op: pg_info(1 pgs e243:3.2b) 2014-06-09 16:22:11.767764 7f869af78700 5 -- op tracker -- , seq: 2972, time: 2014-06-09 16:22:11.507663, event: all_read, op: pg_info(1 pgs e243:3.2b) 2014-06-09 16:22:11.767775 7f869af78700 5 -- op tracker -- , seq: 2972, time: 2014-06-09 16:22:11.765627, event: dispatched, op: pg_info(1 pgs e243:3.2b) 2014-06-09 16:22:11.767783 7f869af78700 5 -- op tracker -- , seq: 2972, time: 2014-06-09 16:22:11.767781, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2b) 2014-06-09 16:22:11.767792 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2b) v4 from osd.2 2014-06-09 16:22:11.767796 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d740540 2014-06-09 16:22:11.767802 7f869af78700 5 -- op tracker -- , seq: 2972, time: 2014-06-09 16:22:11.767799, event: started, op: pg_info(1 pgs e243:3.2b) 2014-06-09 16:22:11.767824 7f869af78700 5 -- op tracker -- , seq: 2972, time: 2014-06-09 16:22:11.767822, event: done, op: pg_info(1 pgs e243:3.2b) 2014-06-09 16:22:11.767834 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.767837 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.767856 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.767858 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.767859 7f869af78700 20 osd.26 243 _dispatch 0x2f039dc0 pg_info(1 pgs e243:0.759) v4 2014-06-09 16:22:11.767864 7f869af78700 5 -- op tracker -- , seq: 2973, time: 2014-06-09 16:22:11.447790, event: header_read, op: pg_info(1 pgs e243:0.759) 2014-06-09 16:22:11.767872 7f869af78700 5 -- op tracker -- , seq: 2973, time: 2014-06-09 16:22:11.447791, event: throttled, op: pg_info(1 pgs e243:0.759) 2014-06-09 16:22:11.767882 7f869af78700 5 -- op tracker -- , seq: 2973, time: 2014-06-09 16:22:11.447802, event: all_read, op: pg_info(1 pgs e243:0.759) 2014-06-09 16:22:11.767889 7f869af78700 5 -- op tracker -- , seq: 2973, time: 2014-06-09 16:22:11.767855, event: dispatched, op: pg_info(1 pgs e243:0.759) 2014-06-09 16:22:11.767897 7f869af78700 5 -- op tracker -- , seq: 2973, time: 2014-06-09 16:22:11.767895, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.759) 2014-06-09 16:22:11.767902 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.759) v4 from osd.4 2014-06-09 16:22:11.767905 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f039dc0 2014-06-09 16:22:11.767910 7f869af78700 5 -- op tracker -- , seq: 2973, time: 2014-06-09 16:22:11.767908, event: started, op: pg_info(1 pgs e243:0.759) 2014-06-09 16:22:11.767927 7f869af78700 5 -- op tracker -- , seq: 2973, time: 2014-06-09 16:22:11.767924, event: done, op: pg_info(1 pgs e243:0.759) 2014-06-09 16:22:11.767935 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.767937 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.767951 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.767952 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.767954 7f869af78700 20 osd.26 243 _dispatch 0xcd79180 pg_info(1 pgs e243:3.136) v4 2014-06-09 16:22:11.767959 7f869af78700 5 -- op tracker -- , seq: 2974, time: 2014-06-09 16:22:11.658766, event: header_read, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.767966 7f869af78700 5 -- op tracker -- , seq: 2974, time: 2014-06-09 16:22:11.658766, event: throttled, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.767972 7f869af78700 5 -- op tracker -- , seq: 2974, time: 2014-06-09 16:22:11.658776, event: all_read, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.767979 7f869af78700 5 -- op tracker -- , seq: 2974, time: 2014-06-09 16:22:11.767950, event: dispatched, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.767986 7f869af78700 5 -- op tracker -- , seq: 2974, time: 2014-06-09 16:22:11.767984, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.767994 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.136) v4 from osd.7 2014-06-09 16:22:11.767997 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd79180 2014-06-09 16:22:11.768002 7f869af78700 5 -- op tracker -- , seq: 2974, time: 2014-06-09 16:22:11.768000, event: started, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.768018 7f869af78700 5 -- op tracker -- , seq: 2974, time: 2014-06-09 16:22:11.768016, event: done, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.768026 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768028 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768044 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768046 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768047 7f869af78700 20 osd.26 243 _dispatch 0x30c31340 pg_info(1 pgs e243:2.699) v4 2014-06-09 16:22:11.768052 7f869af78700 5 -- op tracker -- , seq: 2975, time: 2014-06-09 16:22:11.613278, event: header_read, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.768059 7f869af78700 5 -- op tracker -- , seq: 2975, time: 2014-06-09 16:22:11.613279, event: throttled, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.768066 7f869af78700 5 -- op tracker -- , seq: 2975, time: 2014-06-09 16:22:11.613293, event: all_read, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.768072 7f869af78700 5 -- op tracker -- , seq: 2975, time: 2014-06-09 16:22:11.768044, event: dispatched, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.768085 7f869af78700 5 -- op tracker -- , seq: 2975, time: 2014-06-09 16:22:11.768082, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.768089 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.699) v4 from osd.3 2014-06-09 16:22:11.768092 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c31340 2014-06-09 16:22:11.768097 7f869af78700 5 -- op tracker -- , seq: 2975, time: 2014-06-09 16:22:11.768095, event: started, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.768112 7f869af78700 5 -- op tracker -- , seq: 2975, time: 2014-06-09 16:22:11.768110, event: done, op: pg_info(1 pgs e243:2.699) 2014-06-09 16:22:11.768121 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768123 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768136 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768138 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768139 7f869af78700 20 osd.26 243 _dispatch 0x2fcaafc0 pg_info(1 pgs e243:0.644) v4 2014-06-09 16:22:11.768144 7f869af78700 5 -- op tracker -- , seq: 2976, time: 2014-06-09 16:22:11.611892, event: header_read, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.768151 7f869af78700 5 -- op tracker -- , seq: 2976, time: 2014-06-09 16:22:11.611892, event: throttled, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.768157 7f869af78700 5 -- op tracker -- , seq: 2976, time: 2014-06-09 16:22:11.611905, event: all_read, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.768164 7f869af78700 5 -- op tracker -- , seq: 2976, time: 2014-06-09 16:22:11.768136, event: dispatched, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.768176 7f869af78700 5 -- op tracker -- , seq: 2976, time: 2014-06-09 16:22:11.768174, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.768182 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.644) v4 from osd.1 2014-06-09 16:22:11.768185 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaafc0 2014-06-09 16:22:11.768190 7f869af78700 5 -- op tracker -- , seq: 2976, time: 2014-06-09 16:22:11.768188, event: started, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.768206 7f869af78700 5 -- op tracker -- , seq: 2976, time: 2014-06-09 16:22:11.768204, event: done, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.768214 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768216 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768229 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768231 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768232 7f869af78700 20 osd.26 243 _dispatch 0x2fcab6c0 pg_info(1 pgs e243:2.67c) v4 2014-06-09 16:22:11.768237 7f869af78700 5 -- op tracker -- , seq: 2977, time: 2014-06-09 16:22:11.618109, event: header_read, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.768244 7f869af78700 5 -- op tracker -- , seq: 2977, time: 2014-06-09 16:22:11.618110, event: throttled, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.768255 7f869af78700 5 -- op tracker -- , seq: 2977, time: 2014-06-09 16:22:11.618123, event: all_read, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.768267 7f869af78700 5 -- op tracker -- , seq: 2977, time: 2014-06-09 16:22:11.768229, event: dispatched, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.768274 7f869af78700 5 -- op tracker -- , seq: 2977, time: 2014-06-09 16:22:11.768272, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.768281 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.67c) v4 from osd.9 2014-06-09 16:22:11.768284 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcab6c0 2014-06-09 16:22:11.768289 7f869af78700 5 -- op tracker -- , seq: 2977, time: 2014-06-09 16:22:11.768288, event: started, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.768307 7f869af78700 5 -- op tracker -- , seq: 2977, time: 2014-06-09 16:22:11.768304, event: done, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.768314 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768316 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768329 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768331 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768332 7f869af78700 20 osd.26 243 _dispatch 0x2eef0a80 pg_info(1 pgs e243:1.7a) v4 2014-06-09 16:22:11.768338 7f869af78700 5 -- op tracker -- , seq: 2978, time: 2014-06-09 16:22:11.575491, event: header_read, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.768345 7f869af78700 5 -- op tracker -- , seq: 2978, time: 2014-06-09 16:22:11.575492, event: throttled, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.768357 7f869af78700 5 -- op tracker -- , seq: 2978, time: 2014-06-09 16:22:11.575504, event: all_read, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.768367 7f869af78700 5 -- op tracker -- , seq: 2978, time: 2014-06-09 16:22:11.768329, event: dispatched, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.768441 7f869af78700 5 -- op tracker -- , seq: 2978, time: 2014-06-09 16:22:11.768376, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.768446 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.7a) v4 from osd.8 2014-06-09 16:22:11.768449 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef0a80 2014-06-09 16:22:11.768455 7f869af78700 5 -- op tracker -- , seq: 2978, time: 2014-06-09 16:22:11.768453, event: started, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.768471 7f869af78700 5 -- op tracker -- , seq: 2978, time: 2014-06-09 16:22:11.768469, event: done, op: pg_info(1 pgs e243:1.7a) 2014-06-09 16:22:11.768480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768495 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768497 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768498 7f869af78700 20 osd.26 243 _dispatch 0x30d9d880 pg_info(1 pgs e243:1.122) v4 2014-06-09 16:22:11.768503 7f869af78700 5 -- op tracker -- , seq: 2979, time: 2014-06-09 16:22:11.608048, event: header_read, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.768534 7f869af78700 5 -- op tracker -- , seq: 2979, time: 2014-06-09 16:22:11.608048, event: throttled, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.768541 7f869af78700 5 -- op tracker -- , seq: 2979, time: 2014-06-09 16:22:11.608061, event: all_read, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.768548 7f869af78700 5 -- op tracker -- , seq: 2979, time: 2014-06-09 16:22:11.768495, event: dispatched, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.768555 7f869af78700 5 -- op tracker -- , seq: 2979, time: 2014-06-09 16:22:11.768553, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.768560 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.122) v4 from osd.6 2014-06-09 16:22:11.768563 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9d880 2014-06-09 16:22:11.768569 7f869af78700 5 -- op tracker -- , seq: 2979, time: 2014-06-09 16:22:11.768567, event: started, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.768585 7f869af78700 5 -- op tracker -- , seq: 2979, time: 2014-06-09 16:22:11.768582, event: done, op: pg_info(1 pgs e243:1.122) 2014-06-09 16:22:11.768593 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768595 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768607 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768609 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768611 7f869af78700 20 osd.26 243 _dispatch 0x308d9340 pg_info(1 pgs e243:0.117) v4 2014-06-09 16:22:11.768616 7f869af78700 5 -- op tracker -- , seq: 2980, time: 2014-06-09 16:22:11.652052, event: header_read, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.768623 7f869af78700 5 -- op tracker -- , seq: 2980, time: 2014-06-09 16:22:11.652052, event: throttled, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.768630 7f869af78700 5 -- op tracker -- , seq: 2980, time: 2014-06-09 16:22:11.652064, event: all_read, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.768636 7f869af78700 5 -- op tracker -- , seq: 2980, time: 2014-06-09 16:22:11.768607, event: dispatched, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.768649 7f869af78700 5 -- op tracker -- , seq: 2980, time: 2014-06-09 16:22:11.768647, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.768654 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.117) v4 from osd.0 2014-06-09 16:22:11.768656 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d9340 2014-06-09 16:22:11.768661 7f869af78700 5 -- op tracker -- , seq: 2980, time: 2014-06-09 16:22:11.768659, event: started, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.768678 7f869af78700 5 -- op tracker -- , seq: 2980, time: 2014-06-09 16:22:11.768675, event: done, op: pg_info(1 pgs e243:0.117) 2014-06-09 16:22:11.768685 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768700 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768702 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768703 7f869af78700 20 osd.26 243 _dispatch 0x190b6000 pg_info(1 pgs e243:2.2f2) v4 2014-06-09 16:22:11.768708 7f869af78700 5 -- op tracker -- , seq: 2981, time: 2014-06-09 16:22:11.604627, event: header_read, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.768718 7f869af78700 5 -- op tracker -- , seq: 2981, time: 2014-06-09 16:22:11.604627, event: throttled, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.768725 7f869af78700 5 -- op tracker -- , seq: 2981, time: 2014-06-09 16:22:11.604640, event: all_read, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.768733 7f869af78700 5 -- op tracker -- , seq: 2981, time: 2014-06-09 16:22:11.768700, event: dispatched, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.768740 7f869af78700 5 -- op tracker -- , seq: 2981, time: 2014-06-09 16:22:11.768738, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.768744 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2f2) v4 from osd.5 2014-06-09 16:22:11.768747 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b6000 2014-06-09 16:22:11.768752 7f869af78700 5 -- op tracker -- , seq: 2981, time: 2014-06-09 16:22:11.768750, event: started, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.768768 7f869af78700 5 -- op tracker -- , seq: 2981, time: 2014-06-09 16:22:11.768766, event: done, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.768776 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768778 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768791 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768793 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768794 7f869af78700 20 osd.26 243 _dispatch 0x30ebba40 pg_info(1 pgs e243:3.37a) v4 2014-06-09 16:22:11.768799 7f869af78700 5 -- op tracker -- , seq: 2982, time: 2014-06-09 16:22:11.564387, event: header_read, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.768806 7f869af78700 5 -- op tracker -- , seq: 2982, time: 2014-06-09 16:22:11.564387, event: throttled, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.768812 7f869af78700 5 -- op tracker -- , seq: 2982, time: 2014-06-09 16:22:11.564403, event: all_read, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.768824 7f869af78700 5 -- op tracker -- , seq: 2982, time: 2014-06-09 16:22:11.768790, event: dispatched, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.768832 7f869af78700 5 -- op tracker -- , seq: 2982, time: 2014-06-09 16:22:11.768830, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.768840 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.37a) v4 from osd.2 2014-06-09 16:22:11.768843 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ebba40 2014-06-09 16:22:11.768849 7f869af78700 5 -- op tracker -- , seq: 2982, time: 2014-06-09 16:22:11.768847, event: started, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.768868 7f869af78700 5 -- op tracker -- , seq: 2982, time: 2014-06-09 16:22:11.768866, event: done, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.768883 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768885 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768898 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768900 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768901 7f869af78700 20 osd.26 243 _dispatch 0x2f039c00 pg_info(1 pgs e243:1.1c9) v4 2014-06-09 16:22:11.768906 7f869af78700 5 -- op tracker -- , seq: 2983, time: 2014-06-09 16:22:11.447807, event: header_read, op: pg_info(1 pgs e243:1.1c9) 2014-06-09 16:22:11.768913 7f869af78700 5 -- op tracker -- , seq: 2983, time: 2014-06-09 16:22:11.447807, event: throttled, op: pg_info(1 pgs e243:1.1c9) 2014-06-09 16:22:11.768925 7f869af78700 5 -- op tracker -- , seq: 2983, time: 2014-06-09 16:22:11.447818, event: all_read, op: pg_info(1 pgs e243:1.1c9) 2014-06-09 16:22:11.768935 7f869af78700 5 -- op tracker -- , seq: 2983, time: 2014-06-09 16:22:11.768897, event: dispatched, op: pg_info(1 pgs e243:1.1c9) 2014-06-09 16:22:11.768944 7f869af78700 5 -- op tracker -- , seq: 2983, time: 2014-06-09 16:22:11.768941, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1c9) 2014-06-09 16:22:11.768948 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1c9) v4 from osd.4 2014-06-09 16:22:11.768951 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f039c00 2014-06-09 16:22:11.768957 7f869af78700 5 -- op tracker -- , seq: 2983, time: 2014-06-09 16:22:11.768955, event: started, op: pg_info(1 pgs e243:1.1c9) 2014-06-09 16:22:11.768978 7f869af78700 5 -- op tracker -- , seq: 2983, time: 2014-06-09 16:22:11.768975, event: done, op: pg_info(1 pgs e243:1.1c9) 2014-06-09 16:22:11.768985 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.768987 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.768999 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769001 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769002 7f869af78700 20 osd.26 243 _dispatch 0x308d8e00 pg_info(1 pgs e243:3.579) v4 2014-06-09 16:22:11.769007 7f869af78700 5 -- op tracker -- , seq: 2984, time: 2014-06-09 16:22:11.658780, event: header_read, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.769013 7f869af78700 5 -- op tracker -- , seq: 2984, time: 2014-06-09 16:22:11.658780, event: throttled, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.769019 7f869af78700 5 -- op tracker -- , seq: 2984, time: 2014-06-09 16:22:11.658790, event: all_read, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.769025 7f869af78700 5 -- op tracker -- , seq: 2984, time: 2014-06-09 16:22:11.768999, event: dispatched, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.769037 7f869af78700 5 -- op tracker -- , seq: 2984, time: 2014-06-09 16:22:11.769035, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.769042 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.579) v4 from osd.7 2014-06-09 16:22:11.769045 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d8e00 2014-06-09 16:22:11.769050 7f869af78700 5 -- op tracker -- , seq: 2984, time: 2014-06-09 16:22:11.769048, event: started, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.769066 7f869af78700 5 -- op tracker -- , seq: 2984, time: 2014-06-09 16:22:11.769064, event: done, op: pg_info(1 pgs e243:3.579) 2014-06-09 16:22:11.769074 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769076 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769090 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769091 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769093 7f869af78700 20 osd.26 243 _dispatch 0x3130ba40 pg_info(1 pgs e243:1.206) v4 2014-06-09 16:22:11.769098 7f869af78700 5 -- op tracker -- , seq: 2985, time: 2014-06-09 16:22:11.618899, event: header_read, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.769106 7f869af78700 5 -- op tracker -- , seq: 2985, time: 2014-06-09 16:22:11.618899, event: throttled, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.769112 7f869af78700 5 -- op tracker -- , seq: 2985, time: 2014-06-09 16:22:11.618912, event: all_read, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.769119 7f869af78700 5 -- op tracker -- , seq: 2985, time: 2014-06-09 16:22:11.769089, event: dispatched, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.769125 7f869af78700 5 -- op tracker -- , seq: 2985, time: 2014-06-09 16:22:11.769123, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.769133 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.206) v4 from osd.3 2014-06-09 16:22:11.769136 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130ba40 2014-06-09 16:22:11.769141 7f869af78700 5 -- op tracker -- , seq: 2985, time: 2014-06-09 16:22:11.769139, event: started, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.769161 7f869af78700 5 -- op tracker -- , seq: 2985, time: 2014-06-09 16:22:11.769159, event: done, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.769171 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769173 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769185 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769186 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769187 7f869af78700 20 osd.26 243 _dispatch 0x2fcab340 pg_info(1 pgs e243:2.26a) v4 2014-06-09 16:22:11.769192 7f869af78700 5 -- op tracker -- , seq: 2986, time: 2014-06-09 16:22:11.611990, event: header_read, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.769198 7f869af78700 5 -- op tracker -- , seq: 2986, time: 2014-06-09 16:22:11.611990, event: throttled, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.769203 7f869af78700 5 -- op tracker -- , seq: 2986, time: 2014-06-09 16:22:11.612001, event: all_read, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.769209 7f869af78700 5 -- op tracker -- , seq: 2986, time: 2014-06-09 16:22:11.769185, event: dispatched, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.769214 7f869af78700 5 -- op tracker -- , seq: 2986, time: 2014-06-09 16:22:11.769213, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.769218 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.26a) v4 from osd.1 2014-06-09 16:22:11.769221 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcab340 2014-06-09 16:22:11.769225 7f869af78700 5 -- op tracker -- , seq: 2986, time: 2014-06-09 16:22:11.769223, event: started, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.769238 7f869af78700 5 -- op tracker -- , seq: 2986, time: 2014-06-09 16:22:11.769237, event: done, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.769245 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769246 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769256 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769258 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769258 7f869af78700 20 osd.26 243 _dispatch 0x308e2c40 pg_info(1 pgs e243:2.67c) v4 2014-06-09 16:22:11.769262 7f869af78700 5 -- op tracker -- , seq: 2987, time: 2014-06-09 16:22:11.618294, event: header_read, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.769268 7f869af78700 5 -- op tracker -- , seq: 2987, time: 2014-06-09 16:22:11.618296, event: throttled, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.769273 7f869af78700 5 -- op tracker -- , seq: 2987, time: 2014-06-09 16:22:11.618309, event: all_read, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.769278 7f869af78700 5 -- op tracker -- , seq: 2987, time: 2014-06-09 16:22:11.769256, event: dispatched, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.769284 7f869af78700 5 -- op tracker -- , seq: 2987, time: 2014-06-09 16:22:11.769282, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.769287 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.67c) v4 from osd.9 2014-06-09 16:22:11.769290 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e2c40 2014-06-09 16:22:11.769294 7f869af78700 5 -- op tracker -- , seq: 2987, time: 2014-06-09 16:22:11.769292, event: started, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.769306 7f869af78700 5 -- op tracker -- , seq: 2987, time: 2014-06-09 16:22:11.769304, event: done, op: pg_info(1 pgs e243:2.67c) 2014-06-09 16:22:11.769312 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769313 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769324 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769325 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769326 7f869af78700 20 osd.26 243 _dispatch 0x30cae700 pg_info(1 pgs e243:1.c6) v4 2014-06-09 16:22:11.769330 7f869af78700 5 -- op tracker -- , seq: 2988, time: 2014-06-09 16:22:11.582914, event: header_read, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.769335 7f869af78700 5 -- op tracker -- , seq: 2988, time: 2014-06-09 16:22:11.582914, event: throttled, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.769341 7f869af78700 5 -- op tracker -- , seq: 2988, time: 2014-06-09 16:22:11.582927, event: all_read, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.769346 7f869af78700 5 -- op tracker -- , seq: 2988, time: 2014-06-09 16:22:11.769323, event: dispatched, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.769351 7f869af78700 5 -- op tracker -- , seq: 2988, time: 2014-06-09 16:22:11.769349, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.769355 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.c6) v4 from osd.8 2014-06-09 16:22:11.769357 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cae700 2014-06-09 16:22:11.769361 7f869af78700 5 -- op tracker -- , seq: 2988, time: 2014-06-09 16:22:11.769359, event: started, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.769374 7f869af78700 5 -- op tracker -- , seq: 2988, time: 2014-06-09 16:22:11.769372, event: done, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.769380 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769382 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769393 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769395 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769396 7f869af78700 20 osd.26 243 _dispatch 0x2fc94e00 pg_info(1 pgs e243:2.5f2) v4 2014-06-09 16:22:11.769401 7f869af78700 5 -- op tracker -- , seq: 2989, time: 2014-06-09 16:22:11.609372, event: header_read, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.769408 7f869af78700 5 -- op tracker -- , seq: 2989, time: 2014-06-09 16:22:11.609373, event: throttled, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.769414 7f869af78700 5 -- op tracker -- , seq: 2989, time: 2014-06-09 16:22:11.609386, event: all_read, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.769420 7f869af78700 5 -- op tracker -- , seq: 2989, time: 2014-06-09 16:22:11.769393, event: dispatched, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.769431 7f869af78700 5 -- op tracker -- , seq: 2989, time: 2014-06-09 16:22:11.769429, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.769435 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5f2) v4 from osd.6 2014-06-09 16:22:11.769438 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc94e00 2014-06-09 16:22:11.769443 7f869af78700 5 -- op tracker -- , seq: 2989, time: 2014-06-09 16:22:11.769441, event: started, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.769459 7f869af78700 5 -- op tracker -- , seq: 2989, time: 2014-06-09 16:22:11.769457, event: done, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.769469 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769471 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769486 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769487 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769489 7f869af78700 20 osd.26 243 _dispatch 0xcd796c0 pg_info(1 pgs e243:0.3a9) v4 2014-06-09 16:22:11.769494 7f869af78700 5 -- op tracker -- , seq: 2990, time: 2014-06-09 16:22:11.653729, event: header_read, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.769500 7f869af78700 5 -- op tracker -- , seq: 2990, time: 2014-06-09 16:22:11.653730, event: throttled, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.769507 7f869af78700 5 -- op tracker -- , seq: 2990, time: 2014-06-09 16:22:11.653746, event: all_read, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.769513 7f869af78700 5 -- op tracker -- , seq: 2990, time: 2014-06-09 16:22:11.769485, event: dispatched, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.769520 7f869af78700 5 -- op tracker -- , seq: 2990, time: 2014-06-09 16:22:11.769517, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.769524 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3a9) v4 from osd.0 2014-06-09 16:22:11.769527 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd796c0 2014-06-09 16:22:11.769532 7f869af78700 5 -- op tracker -- , seq: 2990, time: 2014-06-09 16:22:11.769530, event: started, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.769549 7f869af78700 5 -- op tracker -- , seq: 2990, time: 2014-06-09 16:22:11.769547, event: done, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.769556 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769558 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769568 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769570 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769571 7f869af78700 20 osd.26 243 _dispatch 0x2fc95180 pg_info(1 pgs e243:3.139) v4 2014-06-09 16:22:11.769576 7f869af78700 5 -- op tracker -- , seq: 2991, time: 2014-06-09 16:22:11.606677, event: header_read, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.769583 7f869af78700 5 -- op tracker -- , seq: 2991, time: 2014-06-09 16:22:11.606677, event: throttled, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.769590 7f869af78700 5 -- op tracker -- , seq: 2991, time: 2014-06-09 16:22:11.606685, event: all_read, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.769596 7f869af78700 5 -- op tracker -- , seq: 2991, time: 2014-06-09 16:22:11.769568, event: dispatched, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.769602 7f869af78700 5 -- op tracker -- , seq: 2991, time: 2014-06-09 16:22:11.769600, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.769609 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.139) v4 from osd.5 2014-06-09 16:22:11.769613 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc95180 2014-06-09 16:22:11.769617 7f869af78700 5 -- op tracker -- , seq: 2991, time: 2014-06-09 16:22:11.769615, event: started, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.769634 7f869af78700 5 -- op tracker -- , seq: 2991, time: 2014-06-09 16:22:11.769631, event: done, op: pg_info(1 pgs e243:3.139) 2014-06-09 16:22:11.769646 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769648 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769660 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769662 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769663 7f869af78700 20 osd.26 243 _dispatch 0x1a038c40 pg_info(1 pgs e243:3.37a) v4 2014-06-09 16:22:11.769668 7f869af78700 5 -- op tracker -- , seq: 2992, time: 2014-06-09 16:22:11.566198, event: header_read, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.769677 7f869af78700 5 -- op tracker -- , seq: 2992, time: 2014-06-09 16:22:11.566199, event: throttled, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.769684 7f869af78700 5 -- op tracker -- , seq: 2992, time: 2014-06-09 16:22:11.566213, event: all_read, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.769690 7f869af78700 5 -- op tracker -- , seq: 2992, time: 2014-06-09 16:22:11.769660, event: dispatched, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.769698 7f869af78700 5 -- op tracker -- , seq: 2992, time: 2014-06-09 16:22:11.769695, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.769702 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.37a) v4 from osd.2 2014-06-09 16:22:11.769705 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a038c40 2014-06-09 16:22:11.769710 7f869af78700 5 -- op tracker -- , seq: 2992, time: 2014-06-09 16:22:11.769708, event: started, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.769730 7f869af78700 5 -- op tracker -- , seq: 2992, time: 2014-06-09 16:22:11.769728, event: done, op: pg_info(1 pgs e243:3.37a) 2014-06-09 16:22:11.769741 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769744 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769755 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769757 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769758 7f869af78700 20 osd.26 243 _dispatch 0x2f039a40 pg_info(1 pgs e243:1.75b) v4 2014-06-09 16:22:11.769763 7f869af78700 5 -- op tracker -- , seq: 2993, time: 2014-06-09 16:22:11.447823, event: header_read, op: pg_info(1 pgs e243:1.75b) 2014-06-09 16:22:11.769773 7f869af78700 5 -- op tracker -- , seq: 2993, time: 2014-06-09 16:22:11.447823, event: throttled, op: pg_info(1 pgs e243:1.75b) 2014-06-09 16:22:11.769779 7f869af78700 5 -- op tracker -- , seq: 2993, time: 2014-06-09 16:22:11.447835, event: all_read, op: pg_info(1 pgs e243:1.75b) 2014-06-09 16:22:11.769786 7f869af78700 5 -- op tracker -- , seq: 2993, time: 2014-06-09 16:22:11.769755, event: dispatched, op: pg_info(1 pgs e243:1.75b) 2014-06-09 16:22:11.769794 7f869af78700 5 -- op tracker -- , seq: 2993, time: 2014-06-09 16:22:11.769792, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.75b) 2014-06-09 16:22:11.769799 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.75b) v4 from osd.4 2014-06-09 16:22:11.769802 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f039a40 2014-06-09 16:22:11.769807 7f869af78700 5 -- op tracker -- , seq: 2993, time: 2014-06-09 16:22:11.769805, event: started, op: pg_info(1 pgs e243:1.75b) 2014-06-09 16:22:11.769824 7f869af78700 5 -- op tracker -- , seq: 2993, time: 2014-06-09 16:22:11.769822, event: done, op: pg_info(1 pgs e243:1.75b) 2014-06-09 16:22:11.769832 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769834 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769846 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769848 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769849 7f869af78700 20 osd.26 243 _dispatch 0x1607ac40 pg_info(1 pgs e243:3.2b6) v4 2014-06-09 16:22:11.769854 7f869af78700 5 -- op tracker -- , seq: 2994, time: 2014-06-09 16:22:11.658794, event: header_read, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.769861 7f869af78700 5 -- op tracker -- , seq: 2994, time: 2014-06-09 16:22:11.658794, event: throttled, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.769868 7f869af78700 5 -- op tracker -- , seq: 2994, time: 2014-06-09 16:22:11.658804, event: all_read, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.769874 7f869af78700 5 -- op tracker -- , seq: 2994, time: 2014-06-09 16:22:11.769845, event: dispatched, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.769881 7f869af78700 5 -- op tracker -- , seq: 2994, time: 2014-06-09 16:22:11.769879, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.769890 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2b6) v4 from osd.7 2014-06-09 16:22:11.769893 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1607ac40 2014-06-09 16:22:11.769898 7f869af78700 5 -- op tracker -- , seq: 2994, time: 2014-06-09 16:22:11.769896, event: started, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.769915 7f869af78700 5 -- op tracker -- , seq: 2994, time: 2014-06-09 16:22:11.769913, event: done, op: pg_info(1 pgs e243:3.2b6) 2014-06-09 16:22:11.769931 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769933 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769945 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.769947 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.769948 7f869af78700 20 osd.26 243 _dispatch 0x2fcaba40 pg_info(1 pgs e243:2.13c) v4 2014-06-09 16:22:11.769953 7f869af78700 5 -- op tracker -- , seq: 2995, time: 2014-06-09 16:22:11.618944, event: header_read, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.769962 7f869af78700 5 -- op tracker -- , seq: 2995, time: 2014-06-09 16:22:11.618944, event: throttled, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.769969 7f869af78700 5 -- op tracker -- , seq: 2995, time: 2014-06-09 16:22:11.618957, event: all_read, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.769975 7f869af78700 5 -- op tracker -- , seq: 2995, time: 2014-06-09 16:22:11.769945, event: dispatched, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.769984 7f869af78700 5 -- op tracker -- , seq: 2995, time: 2014-06-09 16:22:11.769982, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.769989 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.13c) v4 from osd.3 2014-06-09 16:22:11.769992 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaba40 2014-06-09 16:22:11.769997 7f869af78700 5 -- op tracker -- , seq: 2995, time: 2014-06-09 16:22:11.769995, event: started, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.770018 7f869af78700 5 -- op tracker -- , seq: 2995, time: 2014-06-09 16:22:11.770015, event: done, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.770025 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770027 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770039 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770041 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770042 7f869af78700 20 osd.26 243 _dispatch 0x2fc941c0 pg_info(1 pgs e243:2.33a) v4 2014-06-09 16:22:11.770047 7f869af78700 5 -- op tracker -- , seq: 2996, time: 2014-06-09 16:22:11.612005, event: header_read, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.770054 7f869af78700 5 -- op tracker -- , seq: 2996, time: 2014-06-09 16:22:11.612006, event: throttled, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.770061 7f869af78700 5 -- op tracker -- , seq: 2996, time: 2014-06-09 16:22:11.612018, event: all_read, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.770067 7f869af78700 5 -- op tracker -- , seq: 2996, time: 2014-06-09 16:22:11.770039, event: dispatched, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.770074 7f869af78700 5 -- op tracker -- , seq: 2996, time: 2014-06-09 16:22:11.770072, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.770083 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.33a) v4 from osd.1 2014-06-09 16:22:11.770086 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc941c0 2014-06-09 16:22:11.770092 7f869af78700 5 -- op tracker -- , seq: 2996, time: 2014-06-09 16:22:11.770089, event: started, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.770106 7f869af78700 5 -- op tracker -- , seq: 2996, time: 2014-06-09 16:22:11.770104, event: done, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.770114 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770116 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770128 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770129 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770131 7f869af78700 20 osd.26 243 _dispatch 0xcd79340 pg_info(1 pgs e243:0.473) v4 2014-06-09 16:22:11.770136 7f869af78700 5 -- op tracker -- , seq: 2997, time: 2014-06-09 16:22:11.652717, event: header_read, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.770143 7f869af78700 5 -- op tracker -- , seq: 2997, time: 2014-06-09 16:22:11.652718, event: throttled, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.770149 7f869af78700 5 -- op tracker -- , seq: 2997, time: 2014-06-09 16:22:11.652729, event: all_read, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.770158 7f869af78700 5 -- op tracker -- , seq: 2997, time: 2014-06-09 16:22:11.770127, event: dispatched, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.770165 7f869af78700 5 -- op tracker -- , seq: 2997, time: 2014-06-09 16:22:11.770163, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.770169 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.473) v4 from osd.9 2014-06-09 16:22:11.770172 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd79340 2014-06-09 16:22:11.770177 7f869af78700 5 -- op tracker -- , seq: 2997, time: 2014-06-09 16:22:11.770175, event: started, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.770193 7f869af78700 5 -- op tracker -- , seq: 2997, time: 2014-06-09 16:22:11.770190, event: done, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.770200 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770202 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770213 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770215 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770216 7f869af78700 20 osd.26 243 _dispatch 0x2eef08c0 pg_info(1 pgs e243:1.c6) v4 2014-06-09 16:22:11.770221 7f869af78700 5 -- op tracker -- , seq: 2998, time: 2014-06-09 16:22:11.582964, event: header_read, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.770228 7f869af78700 5 -- op tracker -- , seq: 2998, time: 2014-06-09 16:22:11.582965, event: throttled, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.770235 7f869af78700 5 -- op tracker -- , seq: 2998, time: 2014-06-09 16:22:11.582980, event: all_read, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.770242 7f869af78700 5 -- op tracker -- , seq: 2998, time: 2014-06-09 16:22:11.770213, event: dispatched, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.770248 7f869af78700 5 -- op tracker -- , seq: 2998, time: 2014-06-09 16:22:11.770246, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.770258 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.c6) v4 from osd.8 2014-06-09 16:22:11.770261 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef08c0 2014-06-09 16:22:11.770266 7f869af78700 5 -- op tracker -- , seq: 2998, time: 2014-06-09 16:22:11.770264, event: started, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.770281 7f869af78700 5 -- op tracker -- , seq: 2998, time: 2014-06-09 16:22:11.770279, event: done, op: pg_info(1 pgs e243:1.c6) 2014-06-09 16:22:11.770289 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770291 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770364 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770366 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770368 7f869af78700 20 osd.26 243 _dispatch 0x30a3ae00 pg_info(1 pgs e243:0.640) v4 2014-06-09 16:22:11.770373 7f869af78700 5 -- op tracker -- , seq: 2999, time: 2014-06-09 16:22:11.609411, event: header_read, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.770383 7f869af78700 5 -- op tracker -- , seq: 2999, time: 2014-06-09 16:22:11.609412, event: throttled, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.770390 7f869af78700 5 -- op tracker -- , seq: 2999, time: 2014-06-09 16:22:11.609424, event: all_read, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.770396 7f869af78700 5 -- op tracker -- , seq: 2999, time: 2014-06-09 16:22:11.770364, event: dispatched, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.770403 7f869af78700 5 -- op tracker -- , seq: 2999, time: 2014-06-09 16:22:11.770401, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.770407 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.640) v4 from osd.6 2014-06-09 16:22:11.770410 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3ae00 2014-06-09 16:22:11.770415 7f869af78700 5 -- op tracker -- , seq: 2999, time: 2014-06-09 16:22:11.770413, event: started, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.770435 7f869af78700 5 -- op tracker -- , seq: 2999, time: 2014-06-09 16:22:11.770433, event: done, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.770447 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770449 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770461 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770463 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770464 7f869af78700 20 osd.26 243 _dispatch 0x2fc95340 pg_info(1 pgs e243:0.3a9) v4 2014-06-09 16:22:11.770469 7f869af78700 5 -- op tracker -- , seq: 3000, time: 2014-06-09 16:22:11.653772, event: header_read, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.770475 7f869af78700 5 -- op tracker -- , seq: 3000, time: 2014-06-09 16:22:11.653773, event: throttled, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.770486 7f869af78700 5 -- op tracker -- , seq: 3000, time: 2014-06-09 16:22:11.653781, event: all_read, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.770495 7f869af78700 5 -- op tracker -- , seq: 3000, time: 2014-06-09 16:22:11.770461, event: dispatched, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.770502 7f869af78700 5 -- op tracker -- , seq: 3000, time: 2014-06-09 16:22:11.770500, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.770506 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3a9) v4 from osd.0 2014-06-09 16:22:11.770509 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc95340 2014-06-09 16:22:11.770514 7f869af78700 5 -- op tracker -- , seq: 3000, time: 2014-06-09 16:22:11.770512, event: started, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.770529 7f869af78700 5 -- op tracker -- , seq: 3000, time: 2014-06-09 16:22:11.770527, event: done, op: pg_info(1 pgs e243:0.3a9) 2014-06-09 16:22:11.770537 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770539 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770550 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770552 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770553 7f869af78700 20 osd.26 243 _dispatch 0x30d9fc00 pg_info(1 pgs e243:0.530) v4 2014-06-09 16:22:11.770558 7f869af78700 5 -- op tracker -- , seq: 3001, time: 2014-06-09 16:22:11.608028, event: header_read, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.770567 7f869af78700 5 -- op tracker -- , seq: 3001, time: 2014-06-09 16:22:11.608029, event: throttled, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.770573 7f869af78700 5 -- op tracker -- , seq: 3001, time: 2014-06-09 16:22:11.608038, event: all_read, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.770580 7f869af78700 5 -- op tracker -- , seq: 3001, time: 2014-06-09 16:22:11.770550, event: dispatched, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.770588 7f869af78700 5 -- op tracker -- , seq: 3001, time: 2014-06-09 16:22:11.770586, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.770593 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.530) v4 from osd.5 2014-06-09 16:22:11.770596 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9fc00 2014-06-09 16:22:11.770600 7f869af78700 5 -- op tracker -- , seq: 3001, time: 2014-06-09 16:22:11.770598, event: started, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.770618 7f869af78700 5 -- op tracker -- , seq: 3001, time: 2014-06-09 16:22:11.770616, event: done, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.770626 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770627 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770639 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770640 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770642 7f869af78700 20 osd.26 243 _dispatch 0x2176cfc0 pg_info(1 pgs e243:1.32f) v4 2014-06-09 16:22:11.770647 7f869af78700 5 -- op tracker -- , seq: 3002, time: 2014-06-09 16:22:11.574346, event: header_read, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.770659 7f869af78700 5 -- op tracker -- , seq: 3002, time: 2014-06-09 16:22:11.574346, event: throttled, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.770665 7f869af78700 5 -- op tracker -- , seq: 3002, time: 2014-06-09 16:22:11.574359, event: all_read, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.770671 7f869af78700 5 -- op tracker -- , seq: 3002, time: 2014-06-09 16:22:11.770638, event: dispatched, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.770680 7f869af78700 5 -- op tracker -- , seq: 3002, time: 2014-06-09 16:22:11.770678, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.770685 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.32f) v4 from osd.2 2014-06-09 16:22:11.770688 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176cfc0 2014-06-09 16:22:11.770693 7f869af78700 5 -- op tracker -- , seq: 3002, time: 2014-06-09 16:22:11.770691, event: started, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.770713 7f869af78700 5 -- op tracker -- , seq: 3002, time: 2014-06-09 16:22:11.770711, event: done, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.770720 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770723 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770735 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770736 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770738 7f869af78700 20 osd.26 243 _dispatch 0x30eba540 pg_info(1 pgs e243:0.31f) v4 2014-06-09 16:22:11.770743 7f869af78700 5 -- op tracker -- , seq: 3003, time: 2014-06-09 16:22:11.566979, event: header_read, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.770749 7f869af78700 5 -- op tracker -- , seq: 3003, time: 2014-06-09 16:22:11.566980, event: throttled, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.770756 7f869af78700 5 -- op tracker -- , seq: 3003, time: 2014-06-09 16:22:11.566993, event: all_read, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.770762 7f869af78700 5 -- op tracker -- , seq: 3003, time: 2014-06-09 16:22:11.770734, event: dispatched, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.770770 7f869af78700 5 -- op tracker -- , seq: 3003, time: 2014-06-09 16:22:11.770768, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.770775 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.31f) v4 from osd.4 2014-06-09 16:22:11.770778 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30eba540 2014-06-09 16:22:11.770783 7f869af78700 5 -- op tracker -- , seq: 3003, time: 2014-06-09 16:22:11.770781, event: started, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.770800 7f869af78700 5 -- op tracker -- , seq: 3003, time: 2014-06-09 16:22:11.770798, event: done, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.770808 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770810 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770821 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770823 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770824 7f869af78700 20 osd.26 243 _dispatch 0x19fdea80 pg_info(1 pgs e243:3.136) v4 2014-06-09 16:22:11.770829 7f869af78700 5 -- op tracker -- , seq: 3004, time: 2014-06-09 16:22:11.658808, event: header_read, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.770837 7f869af78700 5 -- op tracker -- , seq: 3004, time: 2014-06-09 16:22:11.658808, event: throttled, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.770846 7f869af78700 5 -- op tracker -- , seq: 3004, time: 2014-06-09 16:22:11.658819, event: all_read, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.770852 7f869af78700 5 -- op tracker -- , seq: 3004, time: 2014-06-09 16:22:11.770821, event: dispatched, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.770859 7f869af78700 5 -- op tracker -- , seq: 3004, time: 2014-06-09 16:22:11.770857, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.770865 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.136) v4 from osd.7 2014-06-09 16:22:11.770868 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdea80 2014-06-09 16:22:11.770873 7f869af78700 5 -- op tracker -- , seq: 3004, time: 2014-06-09 16:22:11.770871, event: started, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.770889 7f869af78700 5 -- op tracker -- , seq: 3004, time: 2014-06-09 16:22:11.770887, event: done, op: pg_info(1 pgs e243:3.136) 2014-06-09 16:22:11.770897 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770899 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770913 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.770916 7f869af78700 20 osd.26 243 _dispatch 0x30c31180 pg_info(1 pgs e243:1.206) v4 2014-06-09 16:22:11.770921 7f869af78700 5 -- op tracker -- , seq: 3005, time: 2014-06-09 16:22:11.618986, event: header_read, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.770927 7f869af78700 5 -- op tracker -- , seq: 3005, time: 2014-06-09 16:22:11.618987, event: throttled, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.770934 7f869af78700 5 -- op tracker -- , seq: 3005, time: 2014-06-09 16:22:11.618999, event: all_read, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.770940 7f869af78700 5 -- op tracker -- , seq: 3005, time: 2014-06-09 16:22:11.770912, event: dispatched, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.770953 7f869af78700 5 -- op tracker -- , seq: 3005, time: 2014-06-09 16:22:11.770950, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.770957 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.206) v4 from osd.3 2014-06-09 16:22:11.770960 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c31180 2014-06-09 16:22:11.770966 7f869af78700 5 -- op tracker -- , seq: 3005, time: 2014-06-09 16:22:11.770963, event: started, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.770980 7f869af78700 5 -- op tracker -- , seq: 3005, time: 2014-06-09 16:22:11.770978, event: done, op: pg_info(1 pgs e243:1.206) 2014-06-09 16:22:11.770988 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.770990 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771002 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771004 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771005 7f869af78700 20 osd.26 243 _dispatch 0x30de4700 pg_info(1 pgs e243:1.198) v4 2014-06-09 16:22:11.771010 7f869af78700 5 -- op tracker -- , seq: 3006, time: 2014-06-09 16:22:11.613190, event: header_read, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771017 7f869af78700 5 -- op tracker -- , seq: 3006, time: 2014-06-09 16:22:11.613190, event: throttled, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771024 7f869af78700 5 -- op tracker -- , seq: 3006, time: 2014-06-09 16:22:11.613200, event: all_read, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771031 7f869af78700 5 -- op tracker -- , seq: 3006, time: 2014-06-09 16:22:11.771002, event: dispatched, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771038 7f869af78700 5 -- op tracker -- , seq: 3006, time: 2014-06-09 16:22:11.771036, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771047 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.198) v4 from osd.1 2014-06-09 16:22:11.771050 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de4700 2014-06-09 16:22:11.771055 7f869af78700 5 -- op tracker -- , seq: 3006, time: 2014-06-09 16:22:11.771053, event: started, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771071 7f869af78700 5 -- op tracker -- , seq: 3006, time: 2014-06-09 16:22:11.771068, event: done, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771081 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771084 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771098 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771100 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771101 7f869af78700 20 osd.26 243 _dispatch 0x26000c40 pg_info(1 pgs e243:0.473) v4 2014-06-09 16:22:11.771107 7f869af78700 5 -- op tracker -- , seq: 3007, time: 2014-06-09 16:22:11.652813, event: header_read, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.771115 7f869af78700 5 -- op tracker -- , seq: 3007, time: 2014-06-09 16:22:11.652814, event: throttled, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.771121 7f869af78700 5 -- op tracker -- , seq: 3007, time: 2014-06-09 16:22:11.652828, event: all_read, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.771128 7f869af78700 5 -- op tracker -- , seq: 3007, time: 2014-06-09 16:22:11.771098, event: dispatched, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.771134 7f869af78700 5 -- op tracker -- , seq: 3007, time: 2014-06-09 16:22:11.771132, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.771139 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.473) v4 from osd.9 2014-06-09 16:22:11.771142 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26000c40 2014-06-09 16:22:11.771147 7f869af78700 5 -- op tracker -- , seq: 3007, time: 2014-06-09 16:22:11.771145, event: started, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.771161 7f869af78700 5 -- op tracker -- , seq: 3007, time: 2014-06-09 16:22:11.771159, event: done, op: pg_info(1 pgs e243:0.473) 2014-06-09 16:22:11.771168 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771170 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771183 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771184 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771185 7f869af78700 20 osd.26 243 _dispatch 0x30de5dc0 pg_info(1 pgs e243:2.5b7) v4 2014-06-09 16:22:11.771190 7f869af78700 5 -- op tracker -- , seq: 3008, time: 2014-06-09 16:22:11.591691, event: header_read, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.771197 7f869af78700 5 -- op tracker -- , seq: 3008, time: 2014-06-09 16:22:11.591691, event: throttled, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.771203 7f869af78700 5 -- op tracker -- , seq: 3008, time: 2014-06-09 16:22:11.591705, event: all_read, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.771209 7f869af78700 5 -- op tracker -- , seq: 3008, time: 2014-06-09 16:22:11.771182, event: dispatched, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.771216 7f869af78700 5 -- op tracker -- , seq: 3008, time: 2014-06-09 16:22:11.771214, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.771220 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5b7) v4 from osd.8 2014-06-09 16:22:11.771223 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5dc0 2014-06-09 16:22:11.771227 7f869af78700 5 -- op tracker -- , seq: 3008, time: 2014-06-09 16:22:11.771226, event: started, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.771241 7f869af78700 5 -- op tracker -- , seq: 3008, time: 2014-06-09 16:22:11.771239, event: done, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.771248 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771250 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771262 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771264 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771265 7f869af78700 20 osd.26 243 _dispatch 0x30d9d6c0 pg_info(1 pgs e243:2.5f2) v4 2014-06-09 16:22:11.771270 7f869af78700 5 -- op tracker -- , seq: 3009, time: 2014-06-09 16:22:11.609458, event: header_read, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.771277 7f869af78700 5 -- op tracker -- , seq: 3009, time: 2014-06-09 16:22:11.609459, event: throttled, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.771284 7f869af78700 5 -- op tracker -- , seq: 3009, time: 2014-06-09 16:22:11.609474, event: all_read, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.771290 7f869af78700 5 -- op tracker -- , seq: 3009, time: 2014-06-09 16:22:11.771262, event: dispatched, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.771297 7f869af78700 5 -- op tracker -- , seq: 3009, time: 2014-06-09 16:22:11.771295, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.771301 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5f2) v4 from osd.6 2014-06-09 16:22:11.771304 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9d6c0 2014-06-09 16:22:11.771309 7f869af78700 5 -- op tracker -- , seq: 3009, time: 2014-06-09 16:22:11.771307, event: started, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.771324 7f869af78700 5 -- op tracker -- , seq: 3009, time: 2014-06-09 16:22:11.771322, event: done, op: pg_info(1 pgs e243:2.5f2) 2014-06-09 16:22:11.771331 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771333 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771346 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771348 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771349 7f869af78700 20 osd.26 243 _dispatch 0x30c12e00 pg_info(1 pgs e243:2.646) v4 2014-06-09 16:22:11.771354 7f869af78700 5 -- op tracker -- , seq: 3010, time: 2014-06-09 16:22:11.660713, event: header_read, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.771361 7f869af78700 5 -- op tracker -- , seq: 3010, time: 2014-06-09 16:22:11.660713, event: throttled, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.771367 7f869af78700 5 -- op tracker -- , seq: 3010, time: 2014-06-09 16:22:11.660727, event: all_read, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.771379 7f869af78700 5 -- op tracker -- , seq: 3010, time: 2014-06-09 16:22:11.771345, event: dispatched, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.771389 7f869af78700 5 -- op tracker -- , seq: 3010, time: 2014-06-09 16:22:11.771387, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.771397 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.646) v4 from osd.0 2014-06-09 16:22:11.771401 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c12e00 2014-06-09 16:22:11.771406 7f869af78700 5 -- op tracker -- , seq: 3010, time: 2014-06-09 16:22:11.771404, event: started, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.771422 7f869af78700 5 -- op tracker -- , seq: 3010, time: 2014-06-09 16:22:11.771419, event: done, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.771430 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771432 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771446 7f869af78700 20 osd.26 243 _dispatch 0x30d9fa40 pg_info(1 pgs e243:2.2f2) v4 2014-06-09 16:22:11.771451 7f869af78700 5 -- op tracker -- , seq: 3011, time: 2014-06-09 16:22:11.608079, event: header_read, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.771463 7f869af78700 5 -- op tracker -- , seq: 3011, time: 2014-06-09 16:22:11.608080, event: throttled, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.771474 7f869af78700 5 -- op tracker -- , seq: 3011, time: 2014-06-09 16:22:11.608088, event: all_read, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.771481 7f869af78700 5 -- op tracker -- , seq: 3011, time: 2014-06-09 16:22:11.771443, event: dispatched, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.771488 7f869af78700 5 -- op tracker -- , seq: 3011, time: 2014-06-09 16:22:11.771486, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.771492 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2f2) v4 from osd.5 2014-06-09 16:22:11.771495 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9fa40 2014-06-09 16:22:11.771500 7f869af78700 5 -- op tracker -- , seq: 3011, time: 2014-06-09 16:22:11.771498, event: started, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.771516 7f869af78700 5 -- op tracker -- , seq: 3011, time: 2014-06-09 16:22:11.771513, event: done, op: pg_info(1 pgs e243:2.2f2) 2014-06-09 16:22:11.771523 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771525 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771536 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771538 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771539 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a000 pg_info(1 pgs e243:1.32f) v4 2014-06-09 16:22:11.771545 7f869af78700 5 -- op tracker -- , seq: 3012, time: 2014-06-09 16:22:11.574365, event: header_read, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.771551 7f869af78700 5 -- op tracker -- , seq: 3012, time: 2014-06-09 16:22:11.574365, event: throttled, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.771561 7f869af78700 5 -- op tracker -- , seq: 3012, time: 2014-06-09 16:22:11.574377, event: all_read, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.771567 7f869af78700 5 -- op tracker -- , seq: 3012, time: 2014-06-09 16:22:11.771536, event: dispatched, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.771575 7f869af78700 5 -- op tracker -- , seq: 3012, time: 2014-06-09 16:22:11.771573, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.771581 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.32f) v4 from osd.2 2014-06-09 16:22:11.771584 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5a000 2014-06-09 16:22:11.771589 7f869af78700 5 -- op tracker -- , seq: 3012, time: 2014-06-09 16:22:11.771587, event: started, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.771604 7f869af78700 5 -- op tracker -- , seq: 3012, time: 2014-06-09 16:22:11.771602, event: done, op: pg_info(1 pgs e243:1.32f) 2014-06-09 16:22:11.771627 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771629 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771642 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771643 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771645 7f869af78700 20 osd.26 243 _dispatch 0x2f039880 pg_info(1 pgs e243:0.31f) v4 2014-06-09 16:22:11.771650 7f869af78700 5 -- op tracker -- , seq: 3013, time: 2014-06-09 16:22:11.567727, event: header_read, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.771657 7f869af78700 5 -- op tracker -- , seq: 3013, time: 2014-06-09 16:22:11.567728, event: throttled, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.771664 7f869af78700 5 -- op tracker -- , seq: 3013, time: 2014-06-09 16:22:11.567860, event: all_read, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.771670 7f869af78700 5 -- op tracker -- , seq: 3013, time: 2014-06-09 16:22:11.771641, event: dispatched, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.771677 7f869af78700 5 -- op tracker -- , seq: 3013, time: 2014-06-09 16:22:11.771675, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.771684 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.31f) v4 from osd.4 2014-06-09 16:22:11.771687 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f039880 2014-06-09 16:22:11.771692 7f869af78700 5 -- op tracker -- , seq: 3013, time: 2014-06-09 16:22:11.771690, event: started, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.771713 7f869af78700 5 -- op tracker -- , seq: 3013, time: 2014-06-09 16:22:11.771711, event: done, op: pg_info(1 pgs e243:0.31f) 2014-06-09 16:22:11.771720 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771722 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771735 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771736 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771738 7f869af78700 20 osd.26 243 _dispatch 0x1ac748c0 pg_info(1 pgs e243:1.1fa) v4 2014-06-09 16:22:11.771743 7f869af78700 5 -- op tracker -- , seq: 3014, time: 2014-06-09 16:22:11.665324, event: header_read, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.771750 7f869af78700 5 -- op tracker -- , seq: 3014, time: 2014-06-09 16:22:11.665324, event: throttled, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.771756 7f869af78700 5 -- op tracker -- , seq: 3014, time: 2014-06-09 16:22:11.665335, event: all_read, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.771763 7f869af78700 5 -- op tracker -- , seq: 3014, time: 2014-06-09 16:22:11.771734, event: dispatched, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.771774 7f869af78700 5 -- op tracker -- , seq: 3014, time: 2014-06-09 16:22:11.771772, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.771778 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1fa) v4 from osd.7 2014-06-09 16:22:11.771781 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac748c0 2014-06-09 16:22:11.771787 7f869af78700 5 -- op tracker -- , seq: 3014, time: 2014-06-09 16:22:11.771785, event: started, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.771807 7f869af78700 5 -- op tracker -- , seq: 3014, time: 2014-06-09 16:22:11.771805, event: done, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.771814 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771816 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771828 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771830 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771831 7f869af78700 20 osd.26 243 _dispatch 0x30c30fc0 pg_info(1 pgs e243:2.13c) v4 2014-06-09 16:22:11.771836 7f869af78700 5 -- op tracker -- , seq: 3015, time: 2014-06-09 16:22:11.619590, event: header_read, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.771845 7f869af78700 5 -- op tracker -- , seq: 3015, time: 2014-06-09 16:22:11.619591, event: throttled, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.771852 7f869af78700 5 -- op tracker -- , seq: 3015, time: 2014-06-09 16:22:11.619604, event: all_read, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.771862 7f869af78700 5 -- op tracker -- , seq: 3015, time: 2014-06-09 16:22:11.771828, event: dispatched, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.771873 7f869af78700 5 -- op tracker -- , seq: 3015, time: 2014-06-09 16:22:11.771871, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.771878 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.13c) v4 from osd.3 2014-06-09 16:22:11.771881 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c30fc0 2014-06-09 16:22:11.771886 7f869af78700 5 -- op tracker -- , seq: 3015, time: 2014-06-09 16:22:11.771884, event: started, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.771901 7f869af78700 5 -- op tracker -- , seq: 3015, time: 2014-06-09 16:22:11.771899, event: done, op: pg_info(1 pgs e243:2.13c) 2014-06-09 16:22:11.771909 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771911 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771922 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.771924 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.771925 7f869af78700 20 osd.26 243 _dispatch 0x230baa80 pg_info(1 pgs e243:1.198) v4 2014-06-09 16:22:11.771930 7f869af78700 5 -- op tracker -- , seq: 3016, time: 2014-06-09 16:22:11.613213, event: header_read, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771942 7f869af78700 5 -- op tracker -- , seq: 3016, time: 2014-06-09 16:22:11.613213, event: throttled, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771949 7f869af78700 5 -- op tracker -- , seq: 3016, time: 2014-06-09 16:22:11.613223, event: all_read, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771957 7f869af78700 5 -- op tracker -- , seq: 3016, time: 2014-06-09 16:22:11.771922, event: dispatched, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771965 7f869af78700 5 -- op tracker -- , seq: 3016, time: 2014-06-09 16:22:11.771962, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771969 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.198) v4 from osd.1 2014-06-09 16:22:11.771972 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230baa80 2014-06-09 16:22:11.771977 7f869af78700 5 -- op tracker -- , seq: 3016, time: 2014-06-09 16:22:11.771975, event: started, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.771992 7f869af78700 5 -- op tracker -- , seq: 3016, time: 2014-06-09 16:22:11.771990, event: done, op: pg_info(1 pgs e243:1.198) 2014-06-09 16:22:11.772000 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772002 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772017 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772018 7f869af78700 20 osd.26 243 _dispatch 0xcd78000 pg_info(1 pgs e243:3.27b) v4 2014-06-09 16:22:11.772023 7f869af78700 5 -- op tracker -- , seq: 3017, time: 2014-06-09 16:22:11.654703, event: header_read, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.772030 7f869af78700 5 -- op tracker -- , seq: 3017, time: 2014-06-09 16:22:11.654704, event: throttled, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.772036 7f869af78700 5 -- op tracker -- , seq: 3017, time: 2014-06-09 16:22:11.654719, event: all_read, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.772043 7f869af78700 5 -- op tracker -- , seq: 3017, time: 2014-06-09 16:22:11.772014, event: dispatched, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.772050 7f869af78700 5 -- op tracker -- , seq: 3017, time: 2014-06-09 16:22:11.772048, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.772055 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.27b) v4 from osd.9 2014-06-09 16:22:11.772057 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd78000 2014-06-09 16:22:11.772062 7f869af78700 5 -- op tracker -- , seq: 3017, time: 2014-06-09 16:22:11.772060, event: started, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.772079 7f869af78700 5 -- op tracker -- , seq: 3017, time: 2014-06-09 16:22:11.772077, event: done, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.772086 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772099 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772101 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772103 7f869af78700 20 osd.26 243 _dispatch 0x2eef0700 pg_info(1 pgs e243:2.5b7) v4 2014-06-09 16:22:11.772107 7f869af78700 5 -- op tracker -- , seq: 3018, time: 2014-06-09 16:22:11.592039, event: header_read, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.772114 7f869af78700 5 -- op tracker -- , seq: 3018, time: 2014-06-09 16:22:11.592039, event: throttled, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.772121 7f869af78700 5 -- op tracker -- , seq: 3018, time: 2014-06-09 16:22:11.592052, event: all_read, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.772127 7f869af78700 5 -- op tracker -- , seq: 3018, time: 2014-06-09 16:22:11.772099, event: dispatched, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.772134 7f869af78700 5 -- op tracker -- , seq: 3018, time: 2014-06-09 16:22:11.772132, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.772144 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5b7) v4 from osd.8 2014-06-09 16:22:11.772148 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef0700 2014-06-09 16:22:11.772153 7f869af78700 5 -- op tracker -- , seq: 3018, time: 2014-06-09 16:22:11.772151, event: started, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.772168 7f869af78700 5 -- op tracker -- , seq: 3018, time: 2014-06-09 16:22:11.772166, event: done, op: pg_info(1 pgs e243:2.5b7) 2014-06-09 16:22:11.772176 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772178 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772189 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772191 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772192 7f869af78700 20 osd.26 243 _dispatch 0x30d9d500 pg_info(1 pgs e243:0.640) v4 2014-06-09 16:22:11.772197 7f869af78700 5 -- op tracker -- , seq: 3019, time: 2014-06-09 16:22:11.609505, event: header_read, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.772205 7f869af78700 5 -- op tracker -- , seq: 3019, time: 2014-06-09 16:22:11.609506, event: throttled, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.772273 7f869af78700 5 -- op tracker -- , seq: 3019, time: 2014-06-09 16:22:11.609519, event: all_read, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.772280 7f869af78700 5 -- op tracker -- , seq: 3019, time: 2014-06-09 16:22:11.772189, event: dispatched, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.772288 7f869af78700 5 -- op tracker -- , seq: 3019, time: 2014-06-09 16:22:11.772286, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.772292 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.640) v4 from osd.6 2014-06-09 16:22:11.772295 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9d500 2014-06-09 16:22:11.772300 7f869af78700 5 -- op tracker -- , seq: 3019, time: 2014-06-09 16:22:11.772298, event: started, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.772318 7f869af78700 5 -- op tracker -- , seq: 3019, time: 2014-06-09 16:22:11.772316, event: done, op: pg_info(1 pgs e243:0.640) 2014-06-09 16:22:11.772331 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772333 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772345 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772347 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772348 7f869af78700 20 osd.26 243 _dispatch 0x30c12fc0 pg_info(1 pgs e243:2.646) v4 2014-06-09 16:22:11.772353 7f869af78700 5 -- op tracker -- , seq: 3020, time: 2014-06-09 16:22:11.661090, event: header_read, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.772360 7f869af78700 5 -- op tracker -- , seq: 3020, time: 2014-06-09 16:22:11.661091, event: throttled, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.772367 7f869af78700 5 -- op tracker -- , seq: 3020, time: 2014-06-09 16:22:11.661104, event: all_read, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.772374 7f869af78700 5 -- op tracker -- , seq: 3020, time: 2014-06-09 16:22:11.772345, event: dispatched, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.772380 7f869af78700 5 -- op tracker -- , seq: 3020, time: 2014-06-09 16:22:11.772378, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.772389 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.646) v4 from osd.0 2014-06-09 16:22:11.772393 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c12fc0 2014-06-09 16:22:11.772398 7f869af78700 5 -- op tracker -- , seq: 3020, time: 2014-06-09 16:22:11.772396, event: started, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.772413 7f869af78700 5 -- op tracker -- , seq: 3020, time: 2014-06-09 16:22:11.772411, event: done, op: pg_info(1 pgs e243:2.646) 2014-06-09 16:22:11.772421 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772423 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772434 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772436 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772437 7f869af78700 20 osd.26 243 _dispatch 0x30d9f880 pg_info(1 pgs e243:0.530) v4 2014-06-09 16:22:11.772442 7f869af78700 5 -- op tracker -- , seq: 3021, time: 2014-06-09 16:22:11.608127, event: header_read, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.772449 7f869af78700 5 -- op tracker -- , seq: 3021, time: 2014-06-09 16:22:11.608127, event: throttled, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.772458 7f869af78700 5 -- op tracker -- , seq: 3021, time: 2014-06-09 16:22:11.608134, event: all_read, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.772465 7f869af78700 5 -- op tracker -- , seq: 3021, time: 2014-06-09 16:22:11.772434, event: dispatched, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.772472 7f869af78700 5 -- op tracker -- , seq: 3021, time: 2014-06-09 16:22:11.772470, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.772482 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.530) v4 from osd.5 2014-06-09 16:22:11.772485 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9f880 2014-06-09 16:22:11.772490 7f869af78700 5 -- op tracker -- , seq: 3021, time: 2014-06-09 16:22:11.772488, event: started, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.772505 7f869af78700 5 -- op tracker -- , seq: 3021, time: 2014-06-09 16:22:11.772503, event: done, op: pg_info(1 pgs e243:0.530) 2014-06-09 16:22:11.772514 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772516 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772534 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772536 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772537 7f869af78700 20 osd.26 243 _dispatch 0x2176c380 pg_info(1 pgs e243:3.7ae) v4 2014-06-09 16:22:11.772542 7f869af78700 5 -- op tracker -- , seq: 3022, time: 2014-06-09 16:22:11.578007, event: header_read, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.772550 7f869af78700 5 -- op tracker -- , seq: 3022, time: 2014-06-09 16:22:11.578008, event: throttled, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.772556 7f869af78700 5 -- op tracker -- , seq: 3022, time: 2014-06-09 16:22:11.578021, event: all_read, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.772563 7f869af78700 5 -- op tracker -- , seq: 3022, time: 2014-06-09 16:22:11.772534, event: dispatched, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.772571 7f869af78700 5 -- op tracker -- , seq: 3022, time: 2014-06-09 16:22:11.772569, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.772575 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7ae) v4 from osd.2 2014-06-09 16:22:11.772578 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176c380 2014-06-09 16:22:11.772583 7f869af78700 5 -- op tracker -- , seq: 3022, time: 2014-06-09 16:22:11.772581, event: started, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.772601 7f869af78700 5 -- op tracker -- , seq: 3022, time: 2014-06-09 16:22:11.772598, event: done, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.772609 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772610 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772623 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772625 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772626 7f869af78700 20 osd.26 243 _dispatch 0x30cf1a40 pg_info(1 pgs e243:0.5e) v4 2014-06-09 16:22:11.772631 7f869af78700 5 -- op tracker -- , seq: 3023, time: 2014-06-09 16:22:11.577311, event: header_read, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.772638 7f869af78700 5 -- op tracker -- , seq: 3023, time: 2014-06-09 16:22:11.577311, event: throttled, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.772644 7f869af78700 5 -- op tracker -- , seq: 3023, time: 2014-06-09 16:22:11.577324, event: all_read, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.772656 7f869af78700 5 -- op tracker -- , seq: 3023, time: 2014-06-09 16:22:11.772623, event: dispatched, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.772667 7f869af78700 5 -- op tracker -- , seq: 3023, time: 2014-06-09 16:22:11.772664, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.772671 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5e) v4 from osd.4 2014-06-09 16:22:11.772674 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30cf1a40 2014-06-09 16:22:11.772679 7f869af78700 5 -- op tracker -- , seq: 3023, time: 2014-06-09 16:22:11.772677, event: started, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.772695 7f869af78700 5 -- op tracker -- , seq: 3023, time: 2014-06-09 16:22:11.772693, event: done, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.772704 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772707 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772719 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772720 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772722 7f869af78700 20 osd.26 243 _dispatch 0x1fc041c0 pg_info(1 pgs e243:1.1fa) v4 2014-06-09 16:22:11.772727 7f869af78700 5 -- op tracker -- , seq: 3024, time: 2014-06-09 16:22:11.665345, event: header_read, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.772734 7f869af78700 5 -- op tracker -- , seq: 3024, time: 2014-06-09 16:22:11.665346, event: throttled, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.772740 7f869af78700 5 -- op tracker -- , seq: 3024, time: 2014-06-09 16:22:11.665362, event: all_read, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.772747 7f869af78700 5 -- op tracker -- , seq: 3024, time: 2014-06-09 16:22:11.772718, event: dispatched, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.772754 7f869af78700 5 -- op tracker -- , seq: 3024, time: 2014-06-09 16:22:11.772752, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.772759 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1fa) v4 from osd.7 2014-06-09 16:22:11.772762 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1fc041c0 2014-06-09 16:22:11.772767 7f869af78700 5 -- op tracker -- , seq: 3024, time: 2014-06-09 16:22:11.772765, event: started, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.772781 7f869af78700 5 -- op tracker -- , seq: 3024, time: 2014-06-09 16:22:11.772779, event: done, op: pg_info(1 pgs e243:1.1fa) 2014-06-09 16:22:11.772793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772795 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772807 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772808 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772810 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a1c0 pg_info(1 pgs e243:0.456) v4 2014-06-09 16:22:11.772815 7f869af78700 5 -- op tracker -- , seq: 3025, time: 2014-06-09 16:22:11.640042, event: header_read, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.772822 7f869af78700 5 -- op tracker -- , seq: 3025, time: 2014-06-09 16:22:11.640043, event: throttled, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.772829 7f869af78700 5 -- op tracker -- , seq: 3025, time: 2014-06-09 16:22:11.640054, event: all_read, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.772835 7f869af78700 5 -- op tracker -- , seq: 3025, time: 2014-06-09 16:22:11.772806, event: dispatched, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.772844 7f869af78700 5 -- op tracker -- , seq: 3025, time: 2014-06-09 16:22:11.772842, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.772849 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.456) v4 from osd.3 2014-06-09 16:22:11.772851 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5a1c0 2014-06-09 16:22:11.772857 7f869af78700 5 -- op tracker -- , seq: 3025, time: 2014-06-09 16:22:11.772854, event: started, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.772877 7f869af78700 5 -- op tracker -- , seq: 3025, time: 2014-06-09 16:22:11.772875, event: done, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.772885 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772887 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772899 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772901 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772902 7f869af78700 20 osd.26 243 _dispatch 0x19fda8c0 pg_info(1 pgs e243:0.644) v4 2014-06-09 16:22:11.772907 7f869af78700 5 -- op tracker -- , seq: 3026, time: 2014-06-09 16:22:11.613236, event: header_read, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.772913 7f869af78700 5 -- op tracker -- , seq: 3026, time: 2014-06-09 16:22:11.613236, event: throttled, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.772922 7f869af78700 5 -- op tracker -- , seq: 3026, time: 2014-06-09 16:22:11.613249, event: all_read, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.772928 7f869af78700 5 -- op tracker -- , seq: 3026, time: 2014-06-09 16:22:11.772899, event: dispatched, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.772940 7f869af78700 5 -- op tracker -- , seq: 3026, time: 2014-06-09 16:22:11.772938, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.772945 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.644) v4 from osd.1 2014-06-09 16:22:11.772948 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fda8c0 2014-06-09 16:22:11.772953 7f869af78700 5 -- op tracker -- , seq: 3026, time: 2014-06-09 16:22:11.772951, event: started, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.772968 7f869af78700 5 -- op tracker -- , seq: 3026, time: 2014-06-09 16:22:11.772966, event: done, op: pg_info(1 pgs e243:0.644) 2014-06-09 16:22:11.772983 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.772985 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.772998 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773000 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773001 7f869af78700 20 osd.26 243 _dispatch 0x3130ae00 pg_info(1 pgs e243:3.27b) v4 2014-06-09 16:22:11.773006 7f869af78700 5 -- op tracker -- , seq: 3027, time: 2014-06-09 16:22:11.654814, event: header_read, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.773018 7f869af78700 5 -- op tracker -- , seq: 3027, time: 2014-06-09 16:22:11.654815, event: throttled, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.773028 7f869af78700 5 -- op tracker -- , seq: 3027, time: 2014-06-09 16:22:11.654828, event: all_read, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.773038 7f869af78700 5 -- op tracker -- , seq: 3027, time: 2014-06-09 16:22:11.772997, event: dispatched, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.773050 7f869af78700 5 -- op tracker -- , seq: 3027, time: 2014-06-09 16:22:11.773048, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.773055 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.27b) v4 from osd.9 2014-06-09 16:22:11.773057 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130ae00 2014-06-09 16:22:11.773062 7f869af78700 5 -- op tracker -- , seq: 3027, time: 2014-06-09 16:22:11.773060, event: started, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.773077 7f869af78700 5 -- op tracker -- , seq: 3027, time: 2014-06-09 16:22:11.773075, event: done, op: pg_info(1 pgs e243:3.27b) 2014-06-09 16:22:11.773085 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773086 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773099 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773100 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773102 7f869af78700 20 osd.26 243 _dispatch 0x190b68c0 pg_info(1 pgs e243:1.517) v4 2014-06-09 16:22:11.773107 7f869af78700 5 -- op tracker -- , seq: 3028, time: 2014-06-09 16:22:11.596929, event: header_read, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.773114 7f869af78700 5 -- op tracker -- , seq: 3028, time: 2014-06-09 16:22:11.596930, event: throttled, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.773120 7f869af78700 5 -- op tracker -- , seq: 3028, time: 2014-06-09 16:22:11.596942, event: all_read, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.773127 7f869af78700 5 -- op tracker -- , seq: 3028, time: 2014-06-09 16:22:11.773098, event: dispatched, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.773138 7f869af78700 5 -- op tracker -- , seq: 3028, time: 2014-06-09 16:22:11.773136, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.773143 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.517) v4 from osd.8 2014-06-09 16:22:11.773146 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b68c0 2014-06-09 16:22:11.773151 7f869af78700 5 -- op tracker -- , seq: 3028, time: 2014-06-09 16:22:11.773149, event: started, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.773166 7f869af78700 5 -- op tracker -- , seq: 3028, time: 2014-06-09 16:22:11.773164, event: done, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.773174 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773176 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773188 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773190 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773191 7f869af78700 20 osd.26 243 _dispatch 0x2fcaaa80 pg_info(1 pgs e243:0.631) v4 2014-06-09 16:22:11.773196 7f869af78700 5 -- op tracker -- , seq: 3029, time: 2014-06-09 16:22:11.611979, event: header_read, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.773203 7f869af78700 5 -- op tracker -- , seq: 3029, time: 2014-06-09 16:22:11.611980, event: throttled, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.773209 7f869af78700 5 -- op tracker -- , seq: 3029, time: 2014-06-09 16:22:11.611992, event: all_read, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.773215 7f869af78700 5 -- op tracker -- , seq: 3029, time: 2014-06-09 16:22:11.773188, event: dispatched, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.773223 7f869af78700 5 -- op tracker -- , seq: 3029, time: 2014-06-09 16:22:11.773220, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.773227 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.631) v4 from osd.6 2014-06-09 16:22:11.773230 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaaa80 2014-06-09 16:22:11.773235 7f869af78700 5 -- op tracker -- , seq: 3029, time: 2014-06-09 16:22:11.773233, event: started, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.773250 7f869af78700 5 -- op tracker -- , seq: 3029, time: 2014-06-09 16:22:11.773248, event: done, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.773257 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773259 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773271 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773273 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773274 7f869af78700 20 osd.26 243 _dispatch 0x240e8700 pg_info(1 pgs e243:3.72c) v4 2014-06-09 16:22:11.773279 7f869af78700 5 -- op tracker -- , seq: 3030, time: 2014-06-09 16:22:11.694028, event: header_read, op: pg_info(1 pgs e243:3.72c) 2014-06-09 16:22:11.773291 7f869af78700 5 -- op tracker -- , seq: 3030, time: 2014-06-09 16:22:11.694029, event: throttled, op: pg_info(1 pgs e243:3.72c) 2014-06-09 16:22:11.773297 7f869af78700 5 -- op tracker -- , seq: 3030, time: 2014-06-09 16:22:11.694048, event: all_read, op: pg_info(1 pgs e243:3.72c) 2014-06-09 16:22:11.773297 7f86a5e11700 10 _generic_write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 217088~2064384 = 0 2014-06-09 16:22:11.773304 7f869af78700 5 -- op tracker -- , seq: 3030, time: 2014-06-09 16:22:11.773271, event: dispatched, op: pg_info(1 pgs e243:3.72c) 2014-06-09 16:22:11.773311 7f869af78700 5 -- op tracker -- , seq: 3030, time: 2014-06-09 16:22:11.773309, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.72c) 2014-06-09 16:22:11.773316 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.72c) v4 from osd.0 2014-06-09 16:22:11.773318 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240e8700 2014-06-09 16:22:11.773323 7f869af78700 5 -- op tracker -- , seq: 3030, time: 2014-06-09 16:22:11.773321, event: started, op: pg_info(1 pgs e243:3.72c) 2014-06-09 16:22:11.773345 7f869af78700 5 -- op tracker -- , seq: 3030, time: 2014-06-09 16:22:11.773343, event: done, op: pg_info(1 pgs e243:3.72c) 2014-06-09 16:22:11.773352 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773354 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773367 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773368 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773370 7f869af78700 20 osd.26 243 _dispatch 0x3130b880 pg_info(1 pgs e243:0.2d9) v4 2014-06-09 16:22:11.773374 7f869af78700 5 -- op tracker -- , seq: 3031, time: 2014-06-09 16:22:11.618020, event: header_read, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.773384 7f869af78700 5 -- op tracker -- , seq: 3031, time: 2014-06-09 16:22:11.618020, event: throttled, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.773391 7f869af78700 5 -- op tracker -- , seq: 3031, time: 2014-06-09 16:22:11.618041, event: all_read, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.773400 7f86a5e11700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.773398 7f869af78700 5 -- op tracker -- , seq: 3031, time: 2014-06-09 16:22:11.773366, event: dispatched, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.773407 7f869af78700 5 -- op tracker -- , seq: 3031, time: 2014-06-09 16:22:11.773405, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.773411 7f86a5e11700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.773412 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2d9) v4 from osd.5 2014-06-09 16:22:11.773414 7f86a5e11700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.773415 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130b880 2014-06-09 16:22:11.773419 7f86a5e11700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.773420 7f869af78700 5 -- op tracker -- , seq: 3031, time: 2014-06-09 16:22:11.773418, event: started, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.773455 7f869af78700 5 -- op tracker -- , seq: 3031, time: 2014-06-09 16:22:11.773453, event: done, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.773463 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773465 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773477 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773478 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773480 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a700 pg_info(1 pgs e243:3.7ae) v4 2014-06-09 16:22:11.773563 7f86a5e11700 10 cid: oid: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 not skipping op, *spos 3457.0.0 2014-06-09 16:22:11.773566 7f86a5e11700 10 > header.spos 3400.0.0 2014-06-09 16:22:11.773610 7f86a5e11700 20 set_header setting 30152 cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 parent seq 0 2014-06-09 16:22:11.773733 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3457.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.773739 7f86a5e11700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3457.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.773741 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.773485 7f869af78700 5 -- op tracker -- , seq: 3032, time: 2014-06-09 16:22:11.578030, event: header_read, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.773772 7f869af78700 5 -- op tracker -- , seq: 3032, time: 2014-06-09 16:22:11.578031, event: throttled, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.773778 7f869af78700 5 -- op tracker -- , seq: 3032, time: 2014-06-09 16:22:11.578043, event: all_read, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.773784 7f869af78700 5 -- op tracker -- , seq: 3032, time: 2014-06-09 16:22:11.773476, event: dispatched, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.773790 7f869af78700 5 -- op tracker -- , seq: 3032, time: 2014-06-09 16:22:11.773788, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.773795 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7ae) v4 from osd.2 2014-06-09 16:22:11.773797 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5a700 2014-06-09 16:22:11.773802 7f869af78700 5 -- op tracker -- , seq: 3032, time: 2014-06-09 16:22:11.773800, event: started, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.773817 7f869af78700 5 -- op tracker -- , seq: 3032, time: 2014-06-09 16:22:11.773815, event: done, op: pg_info(1 pgs e243:3.7ae) 2014-06-09 16:22:11.773824 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773826 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773837 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773839 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773840 7f869af78700 20 osd.26 243 _dispatch 0x2f0396c0 pg_info(1 pgs e243:0.5e) v4 2014-06-09 16:22:11.773844 7f869af78700 5 -- op tracker -- , seq: 3033, time: 2014-06-09 16:22:11.578157, event: header_read, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.773850 7f869af78700 5 -- op tracker -- , seq: 3033, time: 2014-06-09 16:22:11.578157, event: throttled, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.773856 7f869af78700 5 -- op tracker -- , seq: 3033, time: 2014-06-09 16:22:11.578174, event: all_read, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.773862 7f869af78700 5 -- op tracker -- , seq: 3033, time: 2014-06-09 16:22:11.773837, event: dispatched, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.773868 7f869af78700 5 -- op tracker -- , seq: 3033, time: 2014-06-09 16:22:11.773866, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.773872 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5e) v4 from osd.4 2014-06-09 16:22:11.773875 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f0396c0 2014-06-09 16:22:11.773879 7f869af78700 5 -- op tracker -- , seq: 3033, time: 2014-06-09 16:22:11.773878, event: started, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.773893 7f869af78700 5 -- op tracker -- , seq: 3033, time: 2014-06-09 16:22:11.773891, event: done, op: pg_info(1 pgs e243:0.5e) 2014-06-09 16:22:11.773899 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773901 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773912 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773915 7f869af78700 20 osd.26 243 _dispatch 0x3104a8c0 pg_info(1 pgs e243:0.1fd) v4 2014-06-09 16:22:11.773919 7f869af78700 5 -- op tracker -- , seq: 3034, time: 2014-06-09 16:22:11.752050, event: header_read, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.773926 7f869af78700 5 -- op tracker -- , seq: 3034, time: 2014-06-09 16:22:11.752050, event: throttled, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.773932 7f869af78700 5 -- op tracker -- , seq: 3034, time: 2014-06-09 16:22:11.752061, event: all_read, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.773936 7f869af78700 5 -- op tracker -- , seq: 3034, time: 2014-06-09 16:22:11.773912, event: dispatched, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.773942 7f869af78700 5 -- op tracker -- , seq: 3034, time: 2014-06-09 16:22:11.773941, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.773945 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1fd) v4 from osd.7 2014-06-09 16:22:11.773948 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104a8c0 2014-06-09 16:22:11.773952 7f869af78700 5 -- op tracker -- , seq: 3034, time: 2014-06-09 16:22:11.773951, event: started, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.773966 7f869af78700 5 -- op tracker -- , seq: 3034, time: 2014-06-09 16:22:11.773964, event: done, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.773974 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773975 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773988 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.773989 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.773990 7f869af78700 20 osd.26 243 _dispatch 0x30c30e00 pg_info(1 pgs e243:0.456) v4 2014-06-09 16:22:11.773995 7f869af78700 5 -- op tracker -- , seq: 3035, time: 2014-06-09 16:22:11.640373, event: header_read, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.774001 7f869af78700 5 -- op tracker -- , seq: 3035, time: 2014-06-09 16:22:11.640374, event: throttled, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.774007 7f869af78700 5 -- op tracker -- , seq: 3035, time: 2014-06-09 16:22:11.640388, event: all_read, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.774013 7f869af78700 5 -- op tracker -- , seq: 3035, time: 2014-06-09 16:22:11.773987, event: dispatched, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.774020 7f869af78700 5 -- op tracker -- , seq: 3035, time: 2014-06-09 16:22:11.774018, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.774025 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.456) v4 from osd.3 2014-06-09 16:22:11.774028 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c30e00 2014-06-09 16:22:11.774033 7f869af78700 5 -- op tracker -- , seq: 3035, time: 2014-06-09 16:22:11.774031, event: started, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.774049 7f869af78700 5 -- op tracker -- , seq: 3035, time: 2014-06-09 16:22:11.774047, event: done, op: pg_info(1 pgs e243:0.456) 2014-06-09 16:22:11.774058 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774060 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774073 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774075 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774076 7f869af78700 20 osd.26 243 _dispatch 0x1003a000 pg_info(1 pgs e243:2.26a) v4 2014-06-09 16:22:11.774082 7f869af78700 5 -- op tracker -- , seq: 3036, time: 2014-06-09 16:22:11.613254, event: header_read, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.774089 7f869af78700 5 -- op tracker -- , seq: 3036, time: 2014-06-09 16:22:11.613254, event: throttled, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.774095 7f869af78700 5 -- op tracker -- , seq: 3036, time: 2014-06-09 16:22:11.613264, event: all_read, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.774102 7f869af78700 5 -- op tracker -- , seq: 3036, time: 2014-06-09 16:22:11.774073, event: dispatched, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.774109 7f869af78700 5 -- op tracker -- , seq: 3036, time: 2014-06-09 16:22:11.774107, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.774113 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.26a) v4 from osd.1 2014-06-09 16:22:11.774117 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1003a000 2014-06-09 16:22:11.774122 7f869af78700 5 -- op tracker -- , seq: 3036, time: 2014-06-09 16:22:11.774120, event: started, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.774196 7f869af78700 5 -- op tracker -- , seq: 3036, time: 2014-06-09 16:22:11.774193, event: done, op: pg_info(1 pgs e243:2.26a) 2014-06-09 16:22:11.774204 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774206 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774220 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774222 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774223 7f869af78700 20 osd.26 243 _dispatch 0x30adbc00 pg_info(1 pgs e243:3.799) v4 2014-06-09 16:22:11.774229 7f869af78700 5 -- op tracker -- , seq: 3037, time: 2014-06-09 16:22:11.708307, event: header_read, op: pg_info(1 pgs e243:3.799) 2014-06-09 16:22:11.774236 7f869af78700 5 -- op tracker -- , seq: 3037, time: 2014-06-09 16:22:11.708307, event: throttled, op: pg_info(1 pgs e243:3.799) 2014-06-09 16:22:11.774242 7f869af78700 5 -- op tracker -- , seq: 3037, time: 2014-06-09 16:22:11.708316, event: all_read, op: pg_info(1 pgs e243:3.799) 2014-06-09 16:22:11.774249 7f869af78700 5 -- op tracker -- , seq: 3037, time: 2014-06-09 16:22:11.774220, event: dispatched, op: pg_info(1 pgs e243:3.799) 2014-06-09 16:22:11.774256 7f869af78700 5 -- op tracker -- , seq: 3037, time: 2014-06-09 16:22:11.774254, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.799) 2014-06-09 16:22:11.774261 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.799) v4 from osd.9 2014-06-09 16:22:11.774264 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adbc00 2014-06-09 16:22:11.774269 7f869af78700 5 -- op tracker -- , seq: 3037, time: 2014-06-09 16:22:11.774267, event: started, op: pg_info(1 pgs e243:3.799) 2014-06-09 16:22:11.774286 7f869af78700 5 -- op tracker -- , seq: 3037, time: 2014-06-09 16:22:11.774283, event: done, op: pg_info(1 pgs e243:3.799) 2014-06-09 16:22:11.774293 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774295 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774307 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774309 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774310 7f869af78700 20 osd.26 243 _dispatch 0x2eef0540 pg_info(1 pgs e243:1.517) v4 2014-06-09 16:22:11.774315 7f869af78700 5 -- op tracker -- , seq: 3038, time: 2014-06-09 16:22:11.596951, event: header_read, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.774322 7f869af78700 5 -- op tracker -- , seq: 3038, time: 2014-06-09 16:22:11.596952, event: throttled, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.774329 7f869af78700 5 -- op tracker -- , seq: 3038, time: 2014-06-09 16:22:11.597013, event: all_read, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.774335 7f869af78700 5 -- op tracker -- , seq: 3038, time: 2014-06-09 16:22:11.774307, event: dispatched, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.774342 7f869af78700 5 -- op tracker -- , seq: 3038, time: 2014-06-09 16:22:11.774340, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.774347 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.517) v4 from osd.8 2014-06-09 16:22:11.774350 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef0540 2014-06-09 16:22:11.774355 7f869af78700 5 -- op tracker -- , seq: 3038, time: 2014-06-09 16:22:11.774353, event: started, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.774370 7f869af78700 5 -- op tracker -- , seq: 3038, time: 2014-06-09 16:22:11.774368, event: done, op: pg_info(1 pgs e243:1.517) 2014-06-09 16:22:11.774377 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774380 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774392 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774394 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774395 7f869af78700 20 osd.26 243 _dispatch 0x30d9d340 pg_info(1 pgs e243:0.631) v4 2014-06-09 16:22:11.774400 7f869af78700 5 -- op tracker -- , seq: 3039, time: 2014-06-09 16:22:11.612014, event: header_read, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.774408 7f869af78700 5 -- op tracker -- , seq: 3039, time: 2014-06-09 16:22:11.612014, event: throttled, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.774414 7f869af78700 5 -- op tracker -- , seq: 3039, time: 2014-06-09 16:22:11.612026, event: all_read, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.774421 7f869af78700 5 -- op tracker -- , seq: 3039, time: 2014-06-09 16:22:11.774392, event: dispatched, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.774428 7f869af78700 5 -- op tracker -- , seq: 3039, time: 2014-06-09 16:22:11.774426, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.774433 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.631) v4 from osd.6 2014-06-09 16:22:11.774436 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9d340 2014-06-09 16:22:11.774441 7f869af78700 5 -- op tracker -- , seq: 3039, time: 2014-06-09 16:22:11.774439, event: started, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.774455 7f869af78700 5 -- op tracker -- , seq: 3039, time: 2014-06-09 16:22:11.774453, event: done, op: pg_info(1 pgs e243:0.631) 2014-06-09 16:22:11.774462 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774464 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774475 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774476 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774477 7f869af78700 20 osd.26 243 _dispatch 0xddc4fc0 pg_info(1 pgs e243:3.74f) v4 2014-06-09 16:22:11.774483 7f869af78700 5 -- op tracker -- , seq: 3040, time: 2014-06-09 16:22:11.721836, event: header_read, op: pg_info(1 pgs e243:3.74f) 2014-06-09 16:22:11.774490 7f869af78700 5 -- op tracker -- , seq: 3040, time: 2014-06-09 16:22:11.721837, event: throttled, op: pg_info(1 pgs e243:3.74f) 2014-06-09 16:22:11.774497 7f869af78700 5 -- op tracker -- , seq: 3040, time: 2014-06-09 16:22:11.721856, event: all_read, op: pg_info(1 pgs e243:3.74f) 2014-06-09 16:22:11.774503 7f869af78700 5 -- op tracker -- , seq: 3040, time: 2014-06-09 16:22:11.774474, event: dispatched, op: pg_info(1 pgs e243:3.74f) 2014-06-09 16:22:11.774510 7f869af78700 5 -- op tracker -- , seq: 3040, time: 2014-06-09 16:22:11.774509, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.74f) 2014-06-09 16:22:11.774515 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.74f) v4 from osd.0 2014-06-09 16:22:11.774518 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc4fc0 2014-06-09 16:22:11.774523 7f869af78700 5 -- op tracker -- , seq: 3040, time: 2014-06-09 16:22:11.774521, event: started, op: pg_info(1 pgs e243:3.74f) 2014-06-09 16:22:11.774541 7f869af78700 5 -- op tracker -- , seq: 3040, time: 2014-06-09 16:22:11.774538, event: done, op: pg_info(1 pgs e243:3.74f) 2014-06-09 16:22:11.774549 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774551 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774562 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774564 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774565 7f869af78700 20 osd.26 243 _dispatch 0x3130bdc0 pg_info(1 pgs e243:1.48e) v4 2014-06-09 16:22:11.774570 7f869af78700 5 -- op tracker -- , seq: 3041, time: 2014-06-09 16:22:11.618167, event: header_read, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.774577 7f869af78700 5 -- op tracker -- , seq: 3041, time: 2014-06-09 16:22:11.618167, event: throttled, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.774584 7f869af78700 5 -- op tracker -- , seq: 3041, time: 2014-06-09 16:22:11.618187, event: all_read, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.774590 7f869af78700 5 -- op tracker -- , seq: 3041, time: 2014-06-09 16:22:11.774562, event: dispatched, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.774597 7f869af78700 5 -- op tracker -- , seq: 3041, time: 2014-06-09 16:22:11.774595, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.774602 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.48e) v4 from osd.5 2014-06-09 16:22:11.774605 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130bdc0 2014-06-09 16:22:11.774610 7f869af78700 5 -- op tracker -- , seq: 3041, time: 2014-06-09 16:22:11.774608, event: started, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.774625 7f869af78700 5 -- op tracker -- , seq: 3041, time: 2014-06-09 16:22:11.774623, event: done, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.774632 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774634 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774646 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774648 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774649 7f869af78700 20 osd.26 243 _dispatch 0x30de5880 pg_info(1 pgs e243:0.616) v4 2014-06-09 16:22:11.774654 7f869af78700 5 -- op tracker -- , seq: 3042, time: 2014-06-09 16:22:11.591291, event: header_read, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.774660 7f869af78700 5 -- op tracker -- , seq: 3042, time: 2014-06-09 16:22:11.591292, event: throttled, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.774666 7f869af78700 5 -- op tracker -- , seq: 3042, time: 2014-06-09 16:22:11.591305, event: all_read, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.774673 7f869af78700 5 -- op tracker -- , seq: 3042, time: 2014-06-09 16:22:11.774646, event: dispatched, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.774680 7f869af78700 5 -- op tracker -- , seq: 3042, time: 2014-06-09 16:22:11.774678, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.774685 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.616) v4 from osd.2 2014-06-09 16:22:11.774688 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5880 2014-06-09 16:22:11.774693 7f869af78700 5 -- op tracker -- , seq: 3042, time: 2014-06-09 16:22:11.774691, event: started, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.774715 7f869af78700 5 -- op tracker -- , seq: 3042, time: 2014-06-09 16:22:11.774712, event: done, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.774723 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774725 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774736 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774738 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774739 7f869af78700 20 osd.26 243 _dispatch 0x30de5500 pg_info(1 pgs e243:1.4fb) v4 2014-06-09 16:22:11.774744 7f869af78700 5 -- op tracker -- , seq: 3043, time: 2014-06-09 16:22:11.592425, event: header_read, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.774751 7f869af78700 5 -- op tracker -- , seq: 3043, time: 2014-06-09 16:22:11.592426, event: throttled, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.774762 7f869af78700 5 -- op tracker -- , seq: 3043, time: 2014-06-09 16:22:11.592438, event: all_read, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.774774 7f869af78700 5 -- op tracker -- , seq: 3043, time: 2014-06-09 16:22:11.774736, event: dispatched, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.774785 7f869af78700 5 -- op tracker -- , seq: 3043, time: 2014-06-09 16:22:11.774783, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.774793 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4fb) v4 from osd.4 2014-06-09 16:22:11.774797 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5500 2014-06-09 16:22:11.774802 7f869af78700 5 -- op tracker -- , seq: 3043, time: 2014-06-09 16:22:11.774800, event: started, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.774819 7f869af78700 5 -- op tracker -- , seq: 3043, time: 2014-06-09 16:22:11.774816, event: done, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.774829 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774831 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774844 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774846 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774847 7f869af78700 20 osd.26 243 _dispatch 0xcd76a80 pg_info(1 pgs e243:0.129) v4 2014-06-09 16:22:11.774853 7f869af78700 5 -- op tracker -- , seq: 3044, time: 2014-06-09 16:22:11.752131, event: header_read, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.774859 7f869af78700 5 -- op tracker -- , seq: 3044, time: 2014-06-09 16:22:11.752132, event: throttled, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.774866 7f869af78700 5 -- op tracker -- , seq: 3044, time: 2014-06-09 16:22:11.752149, event: all_read, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.774872 7f869af78700 5 -- op tracker -- , seq: 3044, time: 2014-06-09 16:22:11.774843, event: dispatched, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.774879 7f869af78700 5 -- op tracker -- , seq: 3044, time: 2014-06-09 16:22:11.774877, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.774884 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.129) v4 from osd.7 2014-06-09 16:22:11.774887 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd76a80 2014-06-09 16:22:11.774892 7f869af78700 5 -- op tracker -- , seq: 3044, time: 2014-06-09 16:22:11.774890, event: started, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.774907 7f869af78700 5 -- op tracker -- , seq: 3044, time: 2014-06-09 16:22:11.774904, event: done, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.774914 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774916 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774928 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.774930 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.774931 7f869af78700 20 osd.26 243 _dispatch 0xcd79dc0 pg_info(1 pgs e243:2.5b6) v4 2014-06-09 16:22:11.774936 7f869af78700 5 -- op tracker -- , seq: 3045, time: 2014-06-09 16:22:11.643512, event: header_read, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.774943 7f869af78700 5 -- op tracker -- , seq: 3045, time: 2014-06-09 16:22:11.643513, event: throttled, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.774950 7f869af78700 5 -- op tracker -- , seq: 3045, time: 2014-06-09 16:22:11.643526, event: all_read, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.774956 7f869af78700 5 -- op tracker -- , seq: 3045, time: 2014-06-09 16:22:11.774928, event: dispatched, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.774963 7f869af78700 5 -- op tracker -- , seq: 3045, time: 2014-06-09 16:22:11.774961, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.774968 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5b6) v4 from osd.3 2014-06-09 16:22:11.774971 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd79dc0 2014-06-09 16:22:11.774976 7f869af78700 5 -- op tracker -- , seq: 3045, time: 2014-06-09 16:22:11.774974, event: started, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.774991 7f869af78700 5 -- op tracker -- , seq: 3045, time: 2014-06-09 16:22:11.774989, event: done, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.774998 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775000 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775012 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775014 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775015 7f869af78700 20 osd.26 243 _dispatch 0x19eb9500 pg_info(1 pgs e243:2.33a) v4 2014-06-09 16:22:11.775020 7f869af78700 5 -- op tracker -- , seq: 3046, time: 2014-06-09 16:22:11.613276, event: header_read, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.775027 7f869af78700 5 -- op tracker -- , seq: 3046, time: 2014-06-09 16:22:11.613277, event: throttled, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.775033 7f869af78700 5 -- op tracker -- , seq: 3046, time: 2014-06-09 16:22:11.613292, event: all_read, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.775040 7f869af78700 5 -- op tracker -- , seq: 3046, time: 2014-06-09 16:22:11.775012, event: dispatched, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.775047 7f869af78700 5 -- op tracker -- , seq: 3046, time: 2014-06-09 16:22:11.775045, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.775052 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.33a) v4 from osd.1 2014-06-09 16:22:11.775054 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19eb9500 2014-06-09 16:22:11.775059 7f869af78700 5 -- op tracker -- , seq: 3046, time: 2014-06-09 16:22:11.775057, event: started, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.775073 7f869af78700 5 -- op tracker -- , seq: 3046, time: 2014-06-09 16:22:11.775071, event: done, op: pg_info(1 pgs e243:2.33a) 2014-06-09 16:22:11.775080 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775082 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775094 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775096 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775097 7f869af78700 20 osd.26 243 _dispatch 0x2d0956c0 pg_info(1 pgs e243:3.2a5) v4 2014-06-09 16:22:11.775102 7f869af78700 5 -- op tracker -- , seq: 3047, time: 2014-06-09 16:22:11.733231, event: header_read, op: pg_info(1 pgs e243:3.2a5) 2014-06-09 16:22:11.775114 7f869af78700 5 -- op tracker -- , seq: 3047, time: 2014-06-09 16:22:11.733232, event: throttled, op: pg_info(1 pgs e243:3.2a5) 2014-06-09 16:22:11.775121 7f869af78700 5 -- op tracker -- , seq: 3047, time: 2014-06-09 16:22:11.733247, event: all_read, op: pg_info(1 pgs e243:3.2a5) 2014-06-09 16:22:11.775129 7f869af78700 5 -- op tracker -- , seq: 3047, time: 2014-06-09 16:22:11.775094, event: dispatched, op: pg_info(1 pgs e243:3.2a5) 2014-06-09 16:22:11.775139 7f869af78700 5 -- op tracker -- , seq: 3047, time: 2014-06-09 16:22:11.775137, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2a5) 2014-06-09 16:22:11.775143 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2a5) v4 from osd.9 2014-06-09 16:22:11.775147 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d0956c0 2014-06-09 16:22:11.775152 7f869af78700 5 -- op tracker -- , seq: 3047, time: 2014-06-09 16:22:11.775150, event: started, op: pg_info(1 pgs e243:3.2a5) 2014-06-09 16:22:11.775168 7f869af78700 5 -- op tracker -- , seq: 3047, time: 2014-06-09 16:22:11.775166, event: done, op: pg_info(1 pgs e243:3.2a5) 2014-06-09 16:22:11.775176 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775178 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775190 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775191 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775193 7f869af78700 20 osd.26 243 _dispatch 0x30de4fc0 pg_info(1 pgs e243:0.6e7) v4 2014-06-09 16:22:11.775198 7f869af78700 5 -- op tracker -- , seq: 3048, time: 2014-06-09 16:22:11.603129, event: header_read, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.775204 7f869af78700 5 -- op tracker -- , seq: 3048, time: 2014-06-09 16:22:11.603130, event: throttled, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.775212 7f869af78700 5 -- op tracker -- , seq: 3048, time: 2014-06-09 16:22:11.603143, event: all_read, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.775220 7f869af78700 5 -- op tracker -- , seq: 3048, time: 2014-06-09 16:22:11.775189, event: dispatched, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.775228 7f869af78700 5 -- op tracker -- , seq: 3048, time: 2014-06-09 16:22:11.775226, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.775232 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6e7) v4 from osd.8 2014-06-09 16:22:11.775235 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de4fc0 2014-06-09 16:22:11.775241 7f869af78700 5 -- op tracker -- , seq: 3048, time: 2014-06-09 16:22:11.775239, event: started, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.775256 7f869af78700 5 -- op tracker -- , seq: 3048, time: 2014-06-09 16:22:11.775253, event: done, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.775263 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775265 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775277 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775279 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775280 7f869af78700 20 osd.26 243 _dispatch 0x31040700 pg_info(1 pgs e243:1.6f5) v4 2014-06-09 16:22:11.775285 7f869af78700 5 -- op tracker -- , seq: 3049, time: 2014-06-09 16:22:11.640755, event: header_read, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.775292 7f869af78700 5 -- op tracker -- , seq: 3049, time: 2014-06-09 16:22:11.640755, event: throttled, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.775299 7f869af78700 5 -- op tracker -- , seq: 3049, time: 2014-06-09 16:22:11.640766, event: all_read, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.775305 7f869af78700 5 -- op tracker -- , seq: 3049, time: 2014-06-09 16:22:11.775277, event: dispatched, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.775312 7f869af78700 5 -- op tracker -- , seq: 3049, time: 2014-06-09 16:22:11.775310, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.775317 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6f5) v4 from osd.6 2014-06-09 16:22:11.775320 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31040700 2014-06-09 16:22:11.775325 7f869af78700 5 -- op tracker -- , seq: 3049, time: 2014-06-09 16:22:11.775323, event: started, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.775341 7f869af78700 5 -- op tracker -- , seq: 3049, time: 2014-06-09 16:22:11.775339, event: done, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.775350 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775352 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775365 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775367 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775368 7f869af78700 20 osd.26 243 _dispatch 0x30c91dc0 pg_info(1 pgs e243:2.4a5) v4 2014-06-09 16:22:11.775374 7f869af78700 5 -- op tracker -- , seq: 3050, time: 2014-06-09 16:22:11.745456, event: header_read, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.775381 7f869af78700 5 -- op tracker -- , seq: 3050, time: 2014-06-09 16:22:11.745457, event: throttled, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.775387 7f869af78700 5 -- op tracker -- , seq: 3050, time: 2014-06-09 16:22:11.745469, event: all_read, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.775399 7f869af78700 5 -- op tracker -- , seq: 3050, time: 2014-06-09 16:22:11.775365, event: dispatched, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.775406 7f869af78700 5 -- op tracker -- , seq: 3050, time: 2014-06-09 16:22:11.775404, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.775413 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4a5) v4 from osd.0 2014-06-09 16:22:11.775416 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c91dc0 2014-06-09 16:22:11.775422 7f869af78700 5 -- op tracker -- , seq: 3050, time: 2014-06-09 16:22:11.775419, event: started, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.775437 7f869af78700 5 -- op tracker -- , seq: 3050, time: 2014-06-09 16:22:11.775434, event: done, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.775446 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775449 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775462 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775464 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775465 7f869af78700 20 osd.26 243 _dispatch 0x2fcab880 pg_info(1 pgs e243:3.3a5) v4 2014-06-09 16:22:11.775470 7f869af78700 5 -- op tracker -- , seq: 3051, time: 2014-06-09 16:22:11.618343, event: header_read, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.775478 7f869af78700 5 -- op tracker -- , seq: 3051, time: 2014-06-09 16:22:11.618343, event: throttled, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.775484 7f869af78700 5 -- op tracker -- , seq: 3051, time: 2014-06-09 16:22:11.618356, event: all_read, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.775490 7f869af78700 5 -- op tracker -- , seq: 3051, time: 2014-06-09 16:22:11.775462, event: dispatched, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.775500 7f869af78700 5 -- op tracker -- , seq: 3051, time: 2014-06-09 16:22:11.775497, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.775504 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a5) v4 from osd.5 2014-06-09 16:22:11.775507 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcab880 2014-06-09 16:22:11.775512 7f869af78700 5 -- op tracker -- , seq: 3051, time: 2014-06-09 16:22:11.775511, event: started, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.775534 7f869af78700 5 -- op tracker -- , seq: 3051, time: 2014-06-09 16:22:11.775532, event: done, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.775542 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775543 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775556 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775557 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775559 7f869af78700 20 osd.26 243 _dispatch 0x30de5c00 pg_info(1 pgs e243:1.ab) v4 2014-06-09 16:22:11.775564 7f869af78700 5 -- op tracker -- , seq: 3052, time: 2014-06-09 16:22:11.591324, event: header_read, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.775573 7f869af78700 5 -- op tracker -- , seq: 3052, time: 2014-06-09 16:22:11.591325, event: throttled, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.775579 7f869af78700 5 -- op tracker -- , seq: 3052, time: 2014-06-09 16:22:11.591336, event: all_read, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.775586 7f869af78700 5 -- op tracker -- , seq: 3052, time: 2014-06-09 16:22:11.775555, event: dispatched, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.775595 7f869af78700 5 -- op tracker -- , seq: 3052, time: 2014-06-09 16:22:11.775592, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.775599 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.ab) v4 from osd.2 2014-06-09 16:22:11.775602 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5c00 2014-06-09 16:22:11.775607 7f869af78700 5 -- op tracker -- , seq: 3052, time: 2014-06-09 16:22:11.775605, event: started, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.775628 7f869af78700 5 -- op tracker -- , seq: 3052, time: 2014-06-09 16:22:11.775625, event: done, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.775636 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775649 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775650 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775651 7f869af78700 20 osd.26 243 _dispatch 0x30de5a40 pg_info(1 pgs e243:0.b9) v4 2014-06-09 16:22:11.775655 7f869af78700 5 -- op tracker -- , seq: 3053, time: 2014-06-09 16:22:11.592527, event: header_read, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.775661 7f869af78700 5 -- op tracker -- , seq: 3053, time: 2014-06-09 16:22:11.592528, event: throttled, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.775666 7f869af78700 5 -- op tracker -- , seq: 3053, time: 2014-06-09 16:22:11.592543, event: all_read, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.775671 7f869af78700 5 -- op tracker -- , seq: 3053, time: 2014-06-09 16:22:11.775648, event: dispatched, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.775677 7f869af78700 5 -- op tracker -- , seq: 3053, time: 2014-06-09 16:22:11.775675, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.775680 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.b9) v4 from osd.4 2014-06-09 16:22:11.775683 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5a40 2014-06-09 16:22:11.775687 7f869af78700 5 -- op tracker -- , seq: 3053, time: 2014-06-09 16:22:11.775685, event: started, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.775700 7f869af78700 5 -- op tracker -- , seq: 3053, time: 2014-06-09 16:22:11.775698, event: done, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.775711 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775713 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775728 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775730 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775731 7f869af78700 20 osd.26 243 _dispatch 0x30acc1c0 pg_info(1 pgs e243:0.1fd) v4 2014-06-09 16:22:11.775735 7f869af78700 5 -- op tracker -- , seq: 3054, time: 2014-06-09 16:22:11.752200, event: header_read, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.775741 7f869af78700 5 -- op tracker -- , seq: 3054, time: 2014-06-09 16:22:11.752201, event: throttled, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.775746 7f869af78700 5 -- op tracker -- , seq: 3054, time: 2014-06-09 16:22:11.752214, event: all_read, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.775752 7f869af78700 5 -- op tracker -- , seq: 3054, time: 2014-06-09 16:22:11.775728, event: dispatched, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.775758 7f869af78700 5 -- op tracker -- , seq: 3054, time: 2014-06-09 16:22:11.775756, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.775762 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1fd) v4 from osd.7 2014-06-09 16:22:11.775765 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acc1c0 2014-06-09 16:22:11.775770 7f869af78700 5 -- op tracker -- , seq: 3054, time: 2014-06-09 16:22:11.775768, event: started, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.775786 7f869af78700 5 -- op tracker -- , seq: 3054, time: 2014-06-09 16:22:11.775784, event: done, op: pg_info(1 pgs e243:0.1fd) 2014-06-09 16:22:11.775793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775795 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775807 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775809 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775810 7f869af78700 20 osd.26 243 _dispatch 0x30c30c40 pg_info(1 pgs e243:2.5b6) v4 2014-06-09 16:22:11.775814 7f869af78700 5 -- op tracker -- , seq: 3055, time: 2014-06-09 16:22:11.643837, event: header_read, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.775821 7f869af78700 5 -- op tracker -- , seq: 3055, time: 2014-06-09 16:22:11.643838, event: throttled, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.775827 7f869af78700 5 -- op tracker -- , seq: 3055, time: 2014-06-09 16:22:11.643854, event: all_read, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.775833 7f869af78700 5 -- op tracker -- , seq: 3055, time: 2014-06-09 16:22:11.775807, event: dispatched, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.775840 7f869af78700 5 -- op tracker -- , seq: 3055, time: 2014-06-09 16:22:11.775838, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.775844 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5b6) v4 from osd.3 2014-06-09 16:22:11.775847 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c30c40 2014-06-09 16:22:11.775852 7f869af78700 5 -- op tracker -- , seq: 3055, time: 2014-06-09 16:22:11.775850, event: started, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.775866 7f869af78700 5 -- op tracker -- , seq: 3055, time: 2014-06-09 16:22:11.775863, event: done, op: pg_info(1 pgs e243:2.5b6) 2014-06-09 16:22:11.775872 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775874 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775886 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775887 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775889 7f869af78700 20 osd.26 243 _dispatch 0x2fcab500 pg_info(1 pgs e243:3.18e) v4 2014-06-09 16:22:11.775894 7f869af78700 5 -- op tracker -- , seq: 3056, time: 2014-06-09 16:22:11.618143, event: header_read, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.775900 7f869af78700 5 -- op tracker -- , seq: 3056, time: 2014-06-09 16:22:11.618144, event: throttled, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.775906 7f869af78700 5 -- op tracker -- , seq: 3056, time: 2014-06-09 16:22:11.618156, event: all_read, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.775913 7f869af78700 5 -- op tracker -- , seq: 3056, time: 2014-06-09 16:22:11.775885, event: dispatched, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.775922 7f869af78700 5 -- op tracker -- , seq: 3056, time: 2014-06-09 16:22:11.775920, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.775927 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.18e) v4 from osd.1 2014-06-09 16:22:11.775930 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcab500 2014-06-09 16:22:11.775935 7f869af78700 5 -- op tracker -- , seq: 3056, time: 2014-06-09 16:22:11.775933, event: started, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.775952 7f869af78700 5 -- op tracker -- , seq: 3056, time: 2014-06-09 16:22:11.775950, event: done, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.775960 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775962 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775973 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.775975 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.775976 7f869af78700 20 osd.26 243 _dispatch 0x1ac74380 pg_info(1 pgs e243:0.48b) v4 2014-06-09 16:22:11.775981 7f869af78700 5 -- op tracker -- , seq: 3057, time: 2014-06-09 16:22:11.748850, event: header_read, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.775990 7f869af78700 5 -- op tracker -- , seq: 3057, time: 2014-06-09 16:22:11.748850, event: throttled, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.775997 7f869af78700 5 -- op tracker -- , seq: 3057, time: 2014-06-09 16:22:11.748863, event: all_read, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.776004 7f869af78700 5 -- op tracker -- , seq: 3057, time: 2014-06-09 16:22:11.775973, event: dispatched, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.776012 7f869af78700 5 -- op tracker -- , seq: 3057, time: 2014-06-09 16:22:11.776010, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.776017 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.48b) v4 from osd.9 2014-06-09 16:22:11.776020 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac74380 2014-06-09 16:22:11.776025 7f869af78700 5 -- op tracker -- , seq: 3057, time: 2014-06-09 16:22:11.776023, event: started, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.776105 7f869af78700 5 -- op tracker -- , seq: 3057, time: 2014-06-09 16:22:11.776102, event: done, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.776114 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776116 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776128 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776129 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776130 7f869af78700 20 osd.26 243 _dispatch 0x2eef0380 pg_info(1 pgs e243:0.6e7) v4 2014-06-09 16:22:11.776135 7f869af78700 5 -- op tracker -- , seq: 3058, time: 2014-06-09 16:22:11.603156, event: header_read, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.776145 7f869af78700 5 -- op tracker -- , seq: 3058, time: 2014-06-09 16:22:11.603157, event: throttled, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.776151 7f869af78700 5 -- op tracker -- , seq: 3058, time: 2014-06-09 16:22:11.603169, event: all_read, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.776157 7f869af78700 5 -- op tracker -- , seq: 3058, time: 2014-06-09 16:22:11.776127, event: dispatched, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.776166 7f869af78700 5 -- op tracker -- , seq: 3058, time: 2014-06-09 16:22:11.776164, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.776171 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6e7) v4 from osd.8 2014-06-09 16:22:11.776174 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef0380 2014-06-09 16:22:11.776179 7f869af78700 5 -- op tracker -- , seq: 3058, time: 2014-06-09 16:22:11.776177, event: started, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.776200 7f869af78700 5 -- op tracker -- , seq: 3058, time: 2014-06-09 16:22:11.776197, event: done, op: pg_info(1 pgs e243:0.6e7) 2014-06-09 16:22:11.776212 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776214 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776226 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776227 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776229 7f869af78700 20 osd.26 243 _dispatch 0x30d9d180 pg_info(1 pgs e243:1.6f5) v4 2014-06-09 16:22:11.776234 7f869af78700 5 -- op tracker -- , seq: 3059, time: 2014-06-09 16:22:11.641171, event: header_read, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.776241 7f869af78700 5 -- op tracker -- , seq: 3059, time: 2014-06-09 16:22:11.641172, event: throttled, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.776252 7f869af78700 5 -- op tracker -- , seq: 3059, time: 2014-06-09 16:22:11.641186, event: all_read, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.776259 7f869af78700 5 -- op tracker -- , seq: 3059, time: 2014-06-09 16:22:11.776225, event: dispatched, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.776269 7f869af78700 5 -- op tracker -- , seq: 3059, time: 2014-06-09 16:22:11.776266, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.776273 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6f5) v4 from osd.6 2014-06-09 16:22:11.776276 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9d180 2014-06-09 16:22:11.776281 7f869af78700 5 -- op tracker -- , seq: 3059, time: 2014-06-09 16:22:11.776280, event: started, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.776301 7f869af78700 5 -- op tracker -- , seq: 3059, time: 2014-06-09 16:22:11.776298, event: done, op: pg_info(1 pgs e243:1.6f5) 2014-06-09 16:22:11.776312 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776314 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776328 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776330 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776331 7f869af78700 20 osd.26 243 _dispatch 0x30acc380 pg_info(1 pgs e243:2.4a5) v4 2014-06-09 16:22:11.776336 7f869af78700 5 -- op tracker -- , seq: 3060, time: 2014-06-09 16:22:11.745475, event: header_read, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.776348 7f869af78700 5 -- op tracker -- , seq: 3060, time: 2014-06-09 16:22:11.745475, event: throttled, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.776358 7f869af78700 5 -- op tracker -- , seq: 3060, time: 2014-06-09 16:22:11.745489, event: all_read, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.776367 7f869af78700 5 -- op tracker -- , seq: 3060, time: 2014-06-09 16:22:11.776327, event: dispatched, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.776374 7f869af78700 5 -- op tracker -- , seq: 3060, time: 2014-06-09 16:22:11.776372, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.776379 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4a5) v4 from osd.0 2014-06-09 16:22:11.776382 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acc380 2014-06-09 16:22:11.776387 7f869af78700 5 -- op tracker -- , seq: 3060, time: 2014-06-09 16:22:11.776385, event: started, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.776402 7f869af78700 5 -- op tracker -- , seq: 3060, time: 2014-06-09 16:22:11.776399, event: done, op: pg_info(1 pgs e243:2.4a5) 2014-06-09 16:22:11.776414 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776416 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776428 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776429 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776431 7f869af78700 20 osd.26 243 _dispatch 0x2fcabc00 pg_info(1 pgs e243:1.66e) v4 2014-06-09 16:22:11.776436 7f869af78700 5 -- op tracker -- , seq: 3061, time: 2014-06-09 16:22:11.618472, event: header_read, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.776445 7f869af78700 5 -- op tracker -- , seq: 3061, time: 2014-06-09 16:22:11.618472, event: throttled, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.776451 7f869af78700 5 -- op tracker -- , seq: 3061, time: 2014-06-09 16:22:11.618480, event: all_read, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.776458 7f869af78700 5 -- op tracker -- , seq: 3061, time: 2014-06-09 16:22:11.776427, event: dispatched, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.776466 7f869af78700 5 -- op tracker -- , seq: 3061, time: 2014-06-09 16:22:11.776464, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.776471 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.66e) v4 from osd.5 2014-06-09 16:22:11.776474 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcabc00 2014-06-09 16:22:11.776479 7f869af78700 5 -- op tracker -- , seq: 3061, time: 2014-06-09 16:22:11.776477, event: started, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.776495 7f869af78700 5 -- op tracker -- , seq: 3061, time: 2014-06-09 16:22:11.776492, event: done, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.776504 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776506 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776517 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776519 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776520 7f869af78700 20 osd.26 243 _dispatch 0x15458fc0 pg_info(1 pgs e243:0.616) v4 2014-06-09 16:22:11.776525 7f869af78700 5 -- op tracker -- , seq: 3062, time: 2014-06-09 16:22:11.591351, event: header_read, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.776531 7f869af78700 5 -- op tracker -- , seq: 3062, time: 2014-06-09 16:22:11.591352, event: throttled, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.776538 7f869af78700 5 -- op tracker -- , seq: 3062, time: 2014-06-09 16:22:11.591364, event: all_read, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.776546 7f869af78700 5 -- op tracker -- , seq: 3062, time: 2014-06-09 16:22:11.776517, event: dispatched, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.776553 7f869af78700 5 -- op tracker -- , seq: 3062, time: 2014-06-09 16:22:11.776551, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.776558 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.616) v4 from osd.2 2014-06-09 16:22:11.776561 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15458fc0 2014-06-09 16:22:11.776565 7f869af78700 5 -- op tracker -- , seq: 3062, time: 2014-06-09 16:22:11.776563, event: started, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.776580 7f869af78700 5 -- op tracker -- , seq: 3062, time: 2014-06-09 16:22:11.776578, event: done, op: pg_info(1 pgs e243:0.616) 2014-06-09 16:22:11.776587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776589 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776601 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776602 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776603 7f869af78700 20 osd.26 243 _dispatch 0x2f039500 pg_info(1 pgs e243:1.4fb) v4 2014-06-09 16:22:11.776608 7f869af78700 5 -- op tracker -- , seq: 3063, time: 2014-06-09 16:22:11.592919, event: header_read, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.776615 7f869af78700 5 -- op tracker -- , seq: 3063, time: 2014-06-09 16:22:11.592920, event: throttled, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.776621 7f869af78700 5 -- op tracker -- , seq: 3063, time: 2014-06-09 16:22:11.592932, event: all_read, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.776628 7f869af78700 5 -- op tracker -- , seq: 3063, time: 2014-06-09 16:22:11.776600, event: dispatched, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.776640 7f869af78700 5 -- op tracker -- , seq: 3063, time: 2014-06-09 16:22:11.776637, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.776644 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4fb) v4 from osd.4 2014-06-09 16:22:11.776647 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f039500 2014-06-09 16:22:11.776652 7f869af78700 5 -- op tracker -- , seq: 3063, time: 2014-06-09 16:22:11.776650, event: started, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.776666 7f869af78700 5 -- op tracker -- , seq: 3063, time: 2014-06-09 16:22:11.776664, event: done, op: pg_info(1 pgs e243:1.4fb) 2014-06-09 16:22:11.776673 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776675 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776690 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776692 7f869af78700 20 osd.26 243 _dispatch 0xddc4c40 pg_info(1 pgs e243:0.129) v4 2014-06-09 16:22:11.776697 7f869af78700 5 -- op tracker -- , seq: 3064, time: 2014-06-09 16:22:11.752222, event: header_read, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.776704 7f869af78700 5 -- op tracker -- , seq: 3064, time: 2014-06-09 16:22:11.752223, event: throttled, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.776715 7f869af78700 5 -- op tracker -- , seq: 3064, time: 2014-06-09 16:22:11.752238, event: all_read, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.776726 7f869af78700 5 -- op tracker -- , seq: 3064, time: 2014-06-09 16:22:11.776686, event: dispatched, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.776738 7f869af78700 5 -- op tracker -- , seq: 3064, time: 2014-06-09 16:22:11.776736, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.776742 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.129) v4 from osd.7 2014-06-09 16:22:11.776745 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc4c40 2014-06-09 16:22:11.776750 7f869af78700 5 -- op tracker -- , seq: 3064, time: 2014-06-09 16:22:11.776748, event: started, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.776765 7f869af78700 5 -- op tracker -- , seq: 3064, time: 2014-06-09 16:22:11.776763, event: done, op: pg_info(1 pgs e243:0.129) 2014-06-09 16:22:11.776773 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776775 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776787 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776789 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776790 7f869af78700 20 osd.26 243 _dispatch 0x2fc56a80 pg_info(1 pgs e243:1.18a) v4 2014-06-09 16:22:11.776795 7f869af78700 5 -- op tracker -- , seq: 3065, time: 2014-06-09 16:22:11.651117, event: header_read, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.776807 7f869af78700 5 -- op tracker -- , seq: 3065, time: 2014-06-09 16:22:11.651118, event: throttled, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.776814 7f869af78700 5 -- op tracker -- , seq: 3065, time: 2014-06-09 16:22:11.651132, event: all_read, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.776823 7f869af78700 5 -- op tracker -- , seq: 3065, time: 2014-06-09 16:22:11.776787, event: dispatched, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.776830 7f869af78700 5 -- op tracker -- , seq: 3065, time: 2014-06-09 16:22:11.776828, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.776834 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.18a) v4 from osd.3 2014-06-09 16:22:11.776837 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc56a80 2014-06-09 16:22:11.776842 7f869af78700 5 -- op tracker -- , seq: 3065, time: 2014-06-09 16:22:11.776840, event: started, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.776859 7f869af78700 5 -- op tracker -- , seq: 3065, time: 2014-06-09 16:22:11.776857, event: done, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.776867 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776869 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776881 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776882 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776884 7f869af78700 20 osd.26 243 _dispatch 0x1607b180 pg_info(1 pgs e243:3.18e) v4 2014-06-09 16:22:11.776889 7f869af78700 5 -- op tracker -- , seq: 3066, time: 2014-06-09 16:22:11.618164, event: header_read, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.776896 7f869af78700 5 -- op tracker -- , seq: 3066, time: 2014-06-09 16:22:11.618164, event: throttled, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.776903 7f869af78700 5 -- op tracker -- , seq: 3066, time: 2014-06-09 16:22:11.618177, event: all_read, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.776912 7f869af78700 5 -- op tracker -- , seq: 3066, time: 2014-06-09 16:22:11.776880, event: dispatched, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.776919 7f869af78700 5 -- op tracker -- , seq: 3066, time: 2014-06-09 16:22:11.776917, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.776928 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.18e) v4 from osd.1 2014-06-09 16:22:11.776931 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1607b180 2014-06-09 16:22:11.776936 7f869af78700 5 -- op tracker -- , seq: 3066, time: 2014-06-09 16:22:11.776934, event: started, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.776954 7f869af78700 5 -- op tracker -- , seq: 3066, time: 2014-06-09 16:22:11.776952, event: done, op: pg_info(1 pgs e243:3.18e) 2014-06-09 16:22:11.776961 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776963 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776975 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.776977 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.776978 7f869af78700 20 osd.26 243 _dispatch 0x31777c00 pg_info(1 pgs e243:0.48b) v4 2014-06-09 16:22:11.776982 7f869af78700 5 -- op tracker -- , seq: 3067, time: 2014-06-09 16:22:11.748915, event: header_read, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.776990 7f869af78700 5 -- op tracker -- , seq: 3067, time: 2014-06-09 16:22:11.748915, event: throttled, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.776996 7f869af78700 5 -- op tracker -- , seq: 3067, time: 2014-06-09 16:22:11.748930, event: all_read, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.777002 7f869af78700 5 -- op tracker -- , seq: 3067, time: 2014-06-09 16:22:11.776975, event: dispatched, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.777012 7f869af78700 5 -- op tracker -- , seq: 3067, time: 2014-06-09 16:22:11.777010, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.777017 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.48b) v4 from osd.9 2014-06-09 16:22:11.777019 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31777c00 2014-06-09 16:22:11.777024 7f869af78700 5 -- op tracker -- , seq: 3067, time: 2014-06-09 16:22:11.777022, event: started, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.777042 7f869af78700 5 -- op tracker -- , seq: 3067, time: 2014-06-09 16:22:11.777040, event: done, op: pg_info(1 pgs e243:0.48b) 2014-06-09 16:22:11.777050 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777052 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777064 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777066 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777067 7f869af78700 20 osd.26 243 _dispatch 0x30a3afc0 pg_info(1 pgs e243:1.3cd) v4 2014-06-09 16:22:11.777072 7f869af78700 5 -- op tracker -- , seq: 3068, time: 2014-06-09 16:22:11.608286, event: header_read, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.777079 7f869af78700 5 -- op tracker -- , seq: 3068, time: 2014-06-09 16:22:11.608286, event: throttled, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.777087 7f869af78700 5 -- op tracker -- , seq: 3068, time: 2014-06-09 16:22:11.608293, event: all_read, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.777094 7f869af78700 5 -- op tracker -- , seq: 3068, time: 2014-06-09 16:22:11.777064, event: dispatched, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.777101 7f869af78700 5 -- op tracker -- , seq: 3068, time: 2014-06-09 16:22:11.777098, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.777107 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3cd) v4 from osd.8 2014-06-09 16:22:11.777110 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3afc0 2014-06-09 16:22:11.777115 7f869af78700 5 -- op tracker -- , seq: 3068, time: 2014-06-09 16:22:11.777113, event: started, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.777135 7f869af78700 5 -- op tracker -- , seq: 3068, time: 2014-06-09 16:22:11.777133, event: done, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.777143 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777145 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777159 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777160 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777161 7f869af78700 20 osd.26 243 _dispatch 0x1ac75a40 pg_info(1 pgs e243:3.6ba) v4 2014-06-09 16:22:11.777166 7f869af78700 5 -- op tracker -- , seq: 3069, time: 2014-06-09 16:22:11.656842, event: header_read, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.777178 7f869af78700 5 -- op tracker -- , seq: 3069, time: 2014-06-09 16:22:11.656843, event: throttled, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.777184 7f869af78700 5 -- op tracker -- , seq: 3069, time: 2014-06-09 16:22:11.656855, event: all_read, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.777191 7f869af78700 5 -- op tracker -- , seq: 3069, time: 2014-06-09 16:22:11.777158, event: dispatched, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.777198 7f869af78700 5 -- op tracker -- , seq: 3069, time: 2014-06-09 16:22:11.777196, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.777203 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6ba) v4 from osd.6 2014-06-09 16:22:11.777206 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac75a40 2014-06-09 16:22:11.777211 7f869af78700 5 -- op tracker -- , seq: 3069, time: 2014-06-09 16:22:11.777209, event: started, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.777231 7f869af78700 5 -- op tracker -- , seq: 3069, time: 2014-06-09 16:22:11.777229, event: done, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.777238 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777240 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777252 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777254 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777255 7f869af78700 20 osd.26 243 _dispatch 0x30e11a40 pg_info(1 pgs e243:0.1d4) v4 2014-06-09 16:22:11.777260 7f869af78700 5 -- op tracker -- , seq: 3070, time: 2014-06-09 16:22:11.746125, event: header_read, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.777267 7f869af78700 5 -- op tracker -- , seq: 3070, time: 2014-06-09 16:22:11.746125, event: throttled, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.777273 7f869af78700 5 -- op tracker -- , seq: 3070, time: 2014-06-09 16:22:11.746138, event: all_read, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.777279 7f869af78700 5 -- op tracker -- , seq: 3070, time: 2014-06-09 16:22:11.777252, event: dispatched, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.777288 7f869af78700 5 -- op tracker -- , seq: 3070, time: 2014-06-09 16:22:11.777286, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.777293 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1d4) v4 from osd.0 2014-06-09 16:22:11.777296 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e11a40 2014-06-09 16:22:11.777301 7f869af78700 5 -- op tracker -- , seq: 3070, time: 2014-06-09 16:22:11.777299, event: started, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.777318 7f869af78700 5 -- op tracker -- , seq: 3070, time: 2014-06-09 16:22:11.777316, event: done, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.777326 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777328 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777341 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777342 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777344 7f869af78700 20 osd.26 243 _dispatch 0x30d9f6c0 pg_info(1 pgs e243:0.2d9) v4 2014-06-09 16:22:11.777349 7f869af78700 5 -- op tracker -- , seq: 3071, time: 2014-06-09 16:22:11.619160, event: header_read, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.777356 7f869af78700 5 -- op tracker -- , seq: 3071, time: 2014-06-09 16:22:11.619160, event: throttled, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.777363 7f869af78700 5 -- op tracker -- , seq: 3071, time: 2014-06-09 16:22:11.619172, event: all_read, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.777369 7f869af78700 5 -- op tracker -- , seq: 3071, time: 2014-06-09 16:22:11.777340, event: dispatched, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.777376 7f869af78700 5 -- op tracker -- , seq: 3071, time: 2014-06-09 16:22:11.777373, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.777386 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2d9) v4 from osd.5 2014-06-09 16:22:11.777389 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9f6c0 2014-06-09 16:22:11.777394 7f869af78700 5 -- op tracker -- , seq: 3071, time: 2014-06-09 16:22:11.777392, event: started, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.777410 7f869af78700 5 -- op tracker -- , seq: 3071, time: 2014-06-09 16:22:11.777408, event: done, op: pg_info(1 pgs e243:0.2d9) 2014-06-09 16:22:11.777418 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777432 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777434 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777435 7f869af78700 20 osd.26 243 _dispatch 0x15a28000 pg_info(1 pgs e243:1.ab) v4 2014-06-09 16:22:11.777440 7f869af78700 5 -- op tracker -- , seq: 3072, time: 2014-06-09 16:22:11.591431, event: header_read, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.777446 7f869af78700 5 -- op tracker -- , seq: 3072, time: 2014-06-09 16:22:11.591431, event: throttled, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.777457 7f869af78700 5 -- op tracker -- , seq: 3072, time: 2014-06-09 16:22:11.591446, event: all_read, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.777469 7f869af78700 5 -- op tracker -- , seq: 3072, time: 2014-06-09 16:22:11.777432, event: dispatched, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.777476 7f869af78700 5 -- op tracker -- , seq: 3072, time: 2014-06-09 16:22:11.777474, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.777483 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.ab) v4 from osd.2 2014-06-09 16:22:11.777486 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a28000 2014-06-09 16:22:11.777491 7f869af78700 5 -- op tracker -- , seq: 3072, time: 2014-06-09 16:22:11.777489, event: started, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.777510 7f869af78700 5 -- op tracker -- , seq: 3072, time: 2014-06-09 16:22:11.777508, event: done, op: pg_info(1 pgs e243:1.ab) 2014-06-09 16:22:11.777522 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777524 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777535 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777537 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777538 7f869af78700 20 osd.26 243 _dispatch 0x2f039340 pg_info(1 pgs e243:0.b9) v4 2014-06-09 16:22:11.777543 7f869af78700 5 -- op tracker -- , seq: 3073, time: 2014-06-09 16:22:11.593341, event: header_read, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.777550 7f869af78700 5 -- op tracker -- , seq: 3073, time: 2014-06-09 16:22:11.593342, event: throttled, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.777557 7f869af78700 5 -- op tracker -- , seq: 3073, time: 2014-06-09 16:22:11.593355, event: all_read, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.777563 7f869af78700 5 -- op tracker -- , seq: 3073, time: 2014-06-09 16:22:11.777535, event: dispatched, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.777575 7f869af78700 5 -- op tracker -- , seq: 3073, time: 2014-06-09 16:22:11.777573, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.777583 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.b9) v4 from osd.4 2014-06-09 16:22:11.777586 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f039340 2014-06-09 16:22:11.777592 7f869af78700 5 -- op tracker -- , seq: 3073, time: 2014-06-09 16:22:11.777590, event: started, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.777607 7f869af78700 5 -- op tracker -- , seq: 3073, time: 2014-06-09 16:22:11.777605, event: done, op: pg_info(1 pgs e243:0.b9) 2014-06-09 16:22:11.777614 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777616 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777628 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777630 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777632 7f869af78700 20 osd.26 243 _dispatch 0x31777dc0 pg_info(1 pgs e243:3.4c0) v4 2014-06-09 16:22:11.777636 7f869af78700 5 -- op tracker -- , seq: 3074, time: 2014-06-09 16:22:11.755833, event: header_read, op: pg_info(1 pgs e243:3.4c0) 2014-06-09 16:22:11.777643 7f869af78700 5 -- op tracker -- , seq: 3074, time: 2014-06-09 16:22:11.755834, event: throttled, op: pg_info(1 pgs e243:3.4c0) 2014-06-09 16:22:11.777649 7f869af78700 5 -- op tracker -- , seq: 3074, time: 2014-06-09 16:22:11.755848, event: all_read, op: pg_info(1 pgs e243:3.4c0) 2014-06-09 16:22:11.777656 7f869af78700 5 -- op tracker -- , seq: 3074, time: 2014-06-09 16:22:11.777628, event: dispatched, op: pg_info(1 pgs e243:3.4c0) 2014-06-09 16:22:11.777664 7f869af78700 5 -- op tracker -- , seq: 3074, time: 2014-06-09 16:22:11.777662, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c0) 2014-06-09 16:22:11.777670 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c0) v4 from osd.7 2014-06-09 16:22:11.777674 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31777dc0 2014-06-09 16:22:11.777679 7f869af78700 5 -- op tracker -- , seq: 3074, time: 2014-06-09 16:22:11.777677, event: started, op: pg_info(1 pgs e243:3.4c0) 2014-06-09 16:22:11.777699 7f869af78700 5 -- op tracker -- , seq: 3074, time: 2014-06-09 16:22:11.777697, event: done, op: pg_info(1 pgs e243:3.4c0) 2014-06-09 16:22:11.777707 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777709 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777720 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777722 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777723 7f869af78700 20 osd.26 243 _dispatch 0x2fc56fc0 pg_info(1 pgs e243:1.1ef) v4 2014-06-09 16:22:11.777728 7f869af78700 5 -- op tracker -- , seq: 3075, time: 2014-06-09 16:22:11.651163, event: header_read, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.777738 7f869af78700 5 -- op tracker -- , seq: 3075, time: 2014-06-09 16:22:11.651163, event: throttled, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.777744 7f869af78700 5 -- op tracker -- , seq: 3075, time: 2014-06-09 16:22:11.651175, event: all_read, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.777750 7f869af78700 5 -- op tracker -- , seq: 3075, time: 2014-06-09 16:22:11.777720, event: dispatched, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.777758 7f869af78700 5 -- op tracker -- , seq: 3075, time: 2014-06-09 16:22:11.777756, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.777762 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1ef) v4 from osd.3 2014-06-09 16:22:11.777765 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc56fc0 2014-06-09 16:22:11.777770 7f869af78700 5 -- op tracker -- , seq: 3075, time: 2014-06-09 16:22:11.777768, event: started, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.777787 7f869af78700 5 -- op tracker -- , seq: 3075, time: 2014-06-09 16:22:11.777785, event: done, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.777795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777797 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777809 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777811 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777813 7f869af78700 20 osd.26 243 _dispatch 0x30de41c0 pg_info(1 pgs e243:1.65e) v4 2014-06-09 16:22:11.777817 7f869af78700 5 -- op tracker -- , seq: 3076, time: 2014-06-09 16:22:11.640405, event: header_read, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.777827 7f869af78700 5 -- op tracker -- , seq: 3076, time: 2014-06-09 16:22:11.640406, event: throttled, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.777834 7f869af78700 5 -- op tracker -- , seq: 3076, time: 2014-06-09 16:22:11.640419, event: all_read, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.777845 7f869af78700 5 -- op tracker -- , seq: 3076, time: 2014-06-09 16:22:11.777809, event: dispatched, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.777856 7f869af78700 5 -- op tracker -- , seq: 3076, time: 2014-06-09 16:22:11.777854, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.777861 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.65e) v4 from osd.1 2014-06-09 16:22:11.777864 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de41c0 2014-06-09 16:22:11.777869 7f869af78700 5 -- op tracker -- , seq: 3076, time: 2014-06-09 16:22:11.777867, event: started, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.777885 7f869af78700 5 -- op tracker -- , seq: 3076, time: 2014-06-09 16:22:11.777883, event: done, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.777893 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777895 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777906 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.777908 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.777909 7f869af78700 20 osd.26 243 _dispatch 0x30acddc0 pg_info(1 pgs e243:0.2cb) v4 2014-06-09 16:22:11.777915 7f869af78700 5 -- op tracker -- , seq: 3077, time: 2014-06-09 16:22:11.750446, event: header_read, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.777922 7f869af78700 5 -- op tracker -- , seq: 3077, time: 2014-06-09 16:22:11.750447, event: throttled, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.777928 7f869af78700 5 -- op tracker -- , seq: 3077, time: 2014-06-09 16:22:11.750456, event: all_read, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.777935 7f869af78700 5 -- op tracker -- , seq: 3077, time: 2014-06-09 16:22:11.777906, event: dispatched, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.777941 7f869af78700 5 -- op tracker -- , seq: 3077, time: 2014-06-09 16:22:11.777939, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.777948 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2cb) v4 from osd.9 2014-06-09 16:22:11.777951 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acddc0 2014-06-09 16:22:11.778018 7f869af78700 5 -- op tracker -- , seq: 3077, time: 2014-06-09 16:22:11.777954, event: started, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.778034 7f869af78700 5 -- op tracker -- , seq: 3077, time: 2014-06-09 16:22:11.778032, event: done, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.778041 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778043 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778055 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778057 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778058 7f869af78700 20 osd.26 243 _dispatch 0x1d039180 pg_info(1 pgs e243:0.6a8) v4 2014-06-09 16:22:11.778063 7f869af78700 5 -- op tracker -- , seq: 3078, time: 2014-06-09 16:22:11.608763, event: header_read, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778072 7f869af78700 5 -- op tracker -- , seq: 3078, time: 2014-06-09 16:22:11.608764, event: throttled, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778080 7f869af78700 5 -- op tracker -- , seq: 3078, time: 2014-06-09 16:22:11.608773, event: all_read, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778086 7f869af78700 5 -- op tracker -- , seq: 3078, time: 2014-06-09 16:22:11.778055, event: dispatched, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778098 7f869af78700 5 -- op tracker -- , seq: 3078, time: 2014-06-09 16:22:11.778096, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778103 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6a8) v4 from osd.8 2014-06-09 16:22:11.778106 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d039180 2014-06-09 16:22:11.778111 7f869af78700 5 -- op tracker -- , seq: 3078, time: 2014-06-09 16:22:11.778109, event: started, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778126 7f869af78700 5 -- op tracker -- , seq: 3078, time: 2014-06-09 16:22:11.778124, event: done, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778134 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778136 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778149 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778150 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778152 7f869af78700 20 osd.26 243 _dispatch 0x2fc57a40 pg_info(1 pgs e243:3.6cb) v4 2014-06-09 16:22:11.778157 7f869af78700 5 -- op tracker -- , seq: 3079, time: 2014-06-09 16:22:11.656880, event: header_read, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.778164 7f869af78700 5 -- op tracker -- , seq: 3079, time: 2014-06-09 16:22:11.656881, event: throttled, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.778171 7f869af78700 5 -- op tracker -- , seq: 3079, time: 2014-06-09 16:22:11.656892, event: all_read, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.778177 7f869af78700 5 -- op tracker -- , seq: 3079, time: 2014-06-09 16:22:11.778148, event: dispatched, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.778186 7f869af78700 5 -- op tracker -- , seq: 3079, time: 2014-06-09 16:22:11.778184, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.778191 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6cb) v4 from osd.6 2014-06-09 16:22:11.778194 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc57a40 2014-06-09 16:22:11.778199 7f869af78700 5 -- op tracker -- , seq: 3079, time: 2014-06-09 16:22:11.778197, event: started, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.778220 7f869af78700 5 -- op tracker -- , seq: 3079, time: 2014-06-09 16:22:11.778218, event: done, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.778227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778229 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778241 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778243 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778244 7f869af78700 20 osd.26 243 _dispatch 0xcd768c0 pg_info(1 pgs e243:1.fa) v4 2014-06-09 16:22:11.778249 7f869af78700 5 -- op tracker -- , seq: 3080, time: 2014-06-09 16:22:11.746153, event: header_read, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.778256 7f869af78700 5 -- op tracker -- , seq: 3080, time: 2014-06-09 16:22:11.746153, event: throttled, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.778264 7f869af78700 5 -- op tracker -- , seq: 3080, time: 2014-06-09 16:22:11.746165, event: all_read, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.778270 7f869af78700 5 -- op tracker -- , seq: 3080, time: 2014-06-09 16:22:11.778241, event: dispatched, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.778279 7f869af78700 5 -- op tracker -- , seq: 3080, time: 2014-06-09 16:22:11.778277, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.778284 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.fa) v4 from osd.0 2014-06-09 16:22:11.778287 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd768c0 2014-06-09 16:22:11.778293 7f869af78700 5 -- op tracker -- , seq: 3080, time: 2014-06-09 16:22:11.778290, event: started, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.778309 7f869af78700 5 -- op tracker -- , seq: 3080, time: 2014-06-09 16:22:11.778307, event: done, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.778317 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778318 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778332 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778333 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778334 7f869af78700 20 osd.26 243 _dispatch 0x30d9f500 pg_info(1 pgs e243:3.3a5) v4 2014-06-09 16:22:11.778340 7f869af78700 5 -- op tracker -- , seq: 3081, time: 2014-06-09 16:22:11.619204, event: header_read, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.778352 7f869af78700 5 -- op tracker -- , seq: 3081, time: 2014-06-09 16:22:11.619204, event: throttled, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.778358 7f869af78700 5 -- op tracker -- , seq: 3081, time: 2014-06-09 16:22:11.619215, event: all_read, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.778367 7f869af78700 5 -- op tracker -- , seq: 3081, time: 2014-06-09 16:22:11.778331, event: dispatched, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.778374 7f869af78700 5 -- op tracker -- , seq: 3081, time: 2014-06-09 16:22:11.778372, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.778379 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a5) v4 from osd.5 2014-06-09 16:22:11.778382 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9f500 2014-06-09 16:22:11.778387 7f869af78700 5 -- op tracker -- , seq: 3081, time: 2014-06-09 16:22:11.778385, event: started, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.778403 7f869af78700 5 -- op tracker -- , seq: 3081, time: 2014-06-09 16:22:11.778401, event: done, op: pg_info(1 pgs e243:3.3a5) 2014-06-09 16:22:11.778410 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778412 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778424 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778426 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778427 7f869af78700 20 osd.26 243 _dispatch 0x190b6c40 pg_info(1 pgs e243:2.38a) v4 2014-06-09 16:22:11.778432 7f869af78700 5 -- op tracker -- , seq: 3082, time: 2014-06-09 16:22:11.597043, event: header_read, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.778439 7f869af78700 5 -- op tracker -- , seq: 3082, time: 2014-06-09 16:22:11.597044, event: throttled, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.778445 7f869af78700 5 -- op tracker -- , seq: 3082, time: 2014-06-09 16:22:11.597056, event: all_read, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.778454 7f869af78700 5 -- op tracker -- , seq: 3082, time: 2014-06-09 16:22:11.778424, event: dispatched, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.778461 7f869af78700 5 -- op tracker -- , seq: 3082, time: 2014-06-09 16:22:11.778459, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.778466 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.38a) v4 from osd.2 2014-06-09 16:22:11.778468 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b6c40 2014-06-09 16:22:11.778473 7f869af78700 5 -- op tracker -- , seq: 3082, time: 2014-06-09 16:22:11.778471, event: started, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.778491 7f869af78700 5 -- op tracker -- , seq: 3082, time: 2014-06-09 16:22:11.778489, event: done, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.778498 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778501 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778512 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778515 7f869af78700 20 osd.26 243 _dispatch 0x150e0fc0 pg_info(1 pgs e243:1.67) v4 2014-06-09 16:22:11.778520 7f869af78700 5 -- op tracker -- , seq: 3083, time: 2014-06-09 16:22:11.597626, event: header_read, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.778527 7f869af78700 5 -- op tracker -- , seq: 3083, time: 2014-06-09 16:22:11.597627, event: throttled, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.778536 7f869af78700 5 -- op tracker -- , seq: 3083, time: 2014-06-09 16:22:11.597639, event: all_read, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.778542 7f869af78700 5 -- op tracker -- , seq: 3083, time: 2014-06-09 16:22:11.778512, event: dispatched, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.778549 7f869af78700 5 -- op tracker -- , seq: 3083, time: 2014-06-09 16:22:11.778547, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.778554 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.67) v4 from osd.4 2014-06-09 16:22:11.778557 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e0fc0 2014-06-09 16:22:11.778562 7f869af78700 5 -- op tracker -- , seq: 3083, time: 2014-06-09 16:22:11.778560, event: started, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.778578 7f869af78700 5 -- op tracker -- , seq: 3083, time: 2014-06-09 16:22:11.778576, event: done, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.778587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778590 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778602 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778603 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778605 7f869af78700 20 osd.26 243 _dispatch 0x30c30a80 pg_info(1 pgs e243:1.18a) v4 2014-06-09 16:22:11.778609 7f869af78700 5 -- op tracker -- , seq: 3084, time: 2014-06-09 16:22:11.651218, event: header_read, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.778616 7f869af78700 5 -- op tracker -- , seq: 3084, time: 2014-06-09 16:22:11.651219, event: throttled, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.778623 7f869af78700 5 -- op tracker -- , seq: 3084, time: 2014-06-09 16:22:11.651233, event: all_read, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.778632 7f869af78700 5 -- op tracker -- , seq: 3084, time: 2014-06-09 16:22:11.778601, event: dispatched, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.778639 7f869af78700 5 -- op tracker -- , seq: 3084, time: 2014-06-09 16:22:11.778637, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.778644 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.18a) v4 from osd.3 2014-06-09 16:22:11.778647 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c30a80 2014-06-09 16:22:11.778652 7f869af78700 5 -- op tracker -- , seq: 3084, time: 2014-06-09 16:22:11.778650, event: started, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.778667 7f869af78700 5 -- op tracker -- , seq: 3084, time: 2014-06-09 16:22:11.778665, event: done, op: pg_info(1 pgs e243:1.18a) 2014-06-09 16:22:11.778674 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778676 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778688 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778690 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778691 7f869af78700 20 osd.26 243 _dispatch 0xcd77dc0 pg_info(1 pgs e243:1.65e) v4 2014-06-09 16:22:11.778696 7f869af78700 5 -- op tracker -- , seq: 3085, time: 2014-06-09 16:22:11.640448, event: header_read, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.778708 7f869af78700 5 -- op tracker -- , seq: 3085, time: 2014-06-09 16:22:11.640449, event: throttled, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.778714 7f869af78700 5 -- op tracker -- , seq: 3085, time: 2014-06-09 16:22:11.640462, event: all_read, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.778723 7f869af78700 5 -- op tracker -- , seq: 3085, time: 2014-06-09 16:22:11.778688, event: dispatched, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.778730 7f869af78700 5 -- op tracker -- , seq: 3085, time: 2014-06-09 16:22:11.778728, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.778735 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.65e) v4 from osd.1 2014-06-09 16:22:11.778738 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd77dc0 2014-06-09 16:22:11.778743 7f869af78700 5 -- op tracker -- , seq: 3085, time: 2014-06-09 16:22:11.778741, event: started, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.778759 7f869af78700 5 -- op tracker -- , seq: 3085, time: 2014-06-09 16:22:11.778756, event: done, op: pg_info(1 pgs e243:1.65e) 2014-06-09 16:22:11.778770 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778772 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778787 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778788 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778789 7f869af78700 20 osd.26 243 _dispatch 0x30c90fc0 pg_info(1 pgs e243:3.375) v4 2014-06-09 16:22:11.778795 7f869af78700 5 -- op tracker -- , seq: 3086, time: 2014-06-09 16:22:11.750484, event: header_read, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.778801 7f869af78700 5 -- op tracker -- , seq: 3086, time: 2014-06-09 16:22:11.750485, event: throttled, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.778814 7f869af78700 5 -- op tracker -- , seq: 3086, time: 2014-06-09 16:22:11.750493, event: all_read, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.778824 7f869af78700 5 -- op tracker -- , seq: 3086, time: 2014-06-09 16:22:11.778786, event: dispatched, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.778836 7f869af78700 5 -- op tracker -- , seq: 3086, time: 2014-06-09 16:22:11.778834, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.778841 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.375) v4 from osd.9 2014-06-09 16:22:11.778845 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c90fc0 2014-06-09 16:22:11.778850 7f869af78700 5 -- op tracker -- , seq: 3086, time: 2014-06-09 16:22:11.778848, event: started, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.778867 7f869af78700 5 -- op tracker -- , seq: 3086, time: 2014-06-09 16:22:11.778864, event: done, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.778874 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778876 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778888 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778890 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778891 7f869af78700 20 osd.26 243 _dispatch 0x30a3bc00 pg_info(1 pgs e243:0.6a8) v4 2014-06-09 16:22:11.778896 7f869af78700 5 -- op tracker -- , seq: 3087, time: 2014-06-09 16:22:11.608811, event: header_read, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778906 7f869af78700 5 -- op tracker -- , seq: 3087, time: 2014-06-09 16:22:11.608812, event: throttled, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778912 7f869af78700 5 -- op tracker -- , seq: 3087, time: 2014-06-09 16:22:11.608819, event: all_read, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778919 7f869af78700 5 -- op tracker -- , seq: 3087, time: 2014-06-09 16:22:11.778888, event: dispatched, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778926 7f869af78700 5 -- op tracker -- , seq: 3087, time: 2014-06-09 16:22:11.778924, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778931 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6a8) v4 from osd.8 2014-06-09 16:22:11.778934 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3bc00 2014-06-09 16:22:11.778939 7f869af78700 5 -- op tracker -- , seq: 3087, time: 2014-06-09 16:22:11.778937, event: started, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778953 7f869af78700 5 -- op tracker -- , seq: 3087, time: 2014-06-09 16:22:11.778951, event: done, op: pg_info(1 pgs e243:0.6a8) 2014-06-09 16:22:11.778966 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778968 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778980 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.778982 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.778984 7f869af78700 20 osd.26 243 _dispatch 0xcd781c0 pg_info(1 pgs e243:3.2d6) v4 2014-06-09 16:22:11.778989 7f869af78700 5 -- op tracker -- , seq: 3088, time: 2014-06-09 16:22:11.656926, event: header_read, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.778998 7f869af78700 5 -- op tracker -- , seq: 3088, time: 2014-06-09 16:22:11.656926, event: throttled, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.779005 7f869af78700 5 -- op tracker -- , seq: 3088, time: 2014-06-09 16:22:11.656938, event: all_read, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.779011 7f869af78700 5 -- op tracker -- , seq: 3088, time: 2014-06-09 16:22:11.778980, event: dispatched, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.779018 7f869af78700 5 -- op tracker -- , seq: 3088, time: 2014-06-09 16:22:11.779016, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.779024 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2d6) v4 from osd.6 2014-06-09 16:22:11.779027 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd781c0 2014-06-09 16:22:11.779033 7f869af78700 5 -- op tracker -- , seq: 3088, time: 2014-06-09 16:22:11.779031, event: started, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.779049 7f869af78700 5 -- op tracker -- , seq: 3088, time: 2014-06-09 16:22:11.779047, event: done, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.779058 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779060 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779072 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779074 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779075 7f869af78700 20 osd.26 243 _dispatch 0x2fef4000 pg_info(1 pgs e243:0.1d4) v4 2014-06-09 16:22:11.779080 7f869af78700 5 -- op tracker -- , seq: 3089, time: 2014-06-09 16:22:11.746287, event: header_read, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.779087 7f869af78700 5 -- op tracker -- , seq: 3089, time: 2014-06-09 16:22:11.746287, event: throttled, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.779096 7f869af78700 5 -- op tracker -- , seq: 3089, time: 2014-06-09 16:22:11.746304, event: all_read, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.779103 7f869af78700 5 -- op tracker -- , seq: 3089, time: 2014-06-09 16:22:11.779072, event: dispatched, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.779115 7f869af78700 5 -- op tracker -- , seq: 3089, time: 2014-06-09 16:22:11.779113, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.779120 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1d4) v4 from osd.0 2014-06-09 16:22:11.779123 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4000 2014-06-09 16:22:11.779128 7f869af78700 5 -- op tracker -- , seq: 3089, time: 2014-06-09 16:22:11.779126, event: started, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.779143 7f869af78700 5 -- op tracker -- , seq: 3089, time: 2014-06-09 16:22:11.779140, event: done, op: pg_info(1 pgs e243:0.1d4) 2014-06-09 16:22:11.779152 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779154 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779165 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779167 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779169 7f869af78700 20 osd.26 243 _dispatch 0x30d9f340 pg_info(1 pgs e243:1.48e) v4 2014-06-09 16:22:11.779174 7f869af78700 5 -- op tracker -- , seq: 3090, time: 2014-06-09 16:22:11.619279, event: header_read, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.779183 7f869af78700 5 -- op tracker -- , seq: 3090, time: 2014-06-09 16:22:11.619280, event: throttled, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.779190 7f869af78700 5 -- op tracker -- , seq: 3090, time: 2014-06-09 16:22:11.619289, event: all_read, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.779197 7f869af78700 5 -- op tracker -- , seq: 3090, time: 2014-06-09 16:22:11.779165, event: dispatched, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.779206 7f869af78700 5 -- op tracker -- , seq: 3090, time: 2014-06-09 16:22:11.779204, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.779211 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.48e) v4 from osd.5 2014-06-09 16:22:11.779214 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9f340 2014-06-09 16:22:11.779219 7f869af78700 5 -- op tracker -- , seq: 3090, time: 2014-06-09 16:22:11.779217, event: started, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.779235 7f869af78700 5 -- op tracker -- , seq: 3090, time: 2014-06-09 16:22:11.779233, event: done, op: pg_info(1 pgs e243:1.48e) 2014-06-09 16:22:11.779244 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779246 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779259 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779260 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779262 7f869af78700 20 osd.26 243 _dispatch 0x30da3dc0 pg_info(1 pgs e243:2.38a) v4 2014-06-09 16:22:11.779267 7f869af78700 5 -- op tracker -- , seq: 3091, time: 2014-06-09 16:22:11.597083, event: header_read, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.779274 7f869af78700 5 -- op tracker -- , seq: 3091, time: 2014-06-09 16:22:11.597084, event: throttled, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.779281 7f869af78700 5 -- op tracker -- , seq: 3091, time: 2014-06-09 16:22:11.597154, event: all_read, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.779287 7f869af78700 5 -- op tracker -- , seq: 3091, time: 2014-06-09 16:22:11.779258, event: dispatched, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.779296 7f869af78700 5 -- op tracker -- , seq: 3091, time: 2014-06-09 16:22:11.779294, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.779301 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.38a) v4 from osd.2 2014-06-09 16:22:11.779304 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da3dc0 2014-06-09 16:22:11.779309 7f869af78700 5 -- op tracker -- , seq: 3091, time: 2014-06-09 16:22:11.779307, event: started, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.779361 7f869af78700 5 -- op tracker -- , seq: 3091, time: 2014-06-09 16:22:11.779358, event: done, op: pg_info(1 pgs e243:2.38a) 2014-06-09 16:22:11.779369 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779371 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779383 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779385 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779386 7f869af78700 20 osd.26 243 _dispatch 0x190b6700 pg_info(1 pgs e243:3.65d) v4 2014-06-09 16:22:11.779391 7f869af78700 5 -- op tracker -- , seq: 3092, time: 2014-06-09 16:22:11.598152, event: header_read, op: pg_info(1 pgs e243:3.65d) 2014-06-09 16:22:11.779397 7f869af78700 5 -- op tracker -- , seq: 3092, time: 2014-06-09 16:22:11.598152, event: throttled, op: pg_info(1 pgs e243:3.65d) 2014-06-09 16:22:11.779403 7f869af78700 5 -- op tracker -- , seq: 3092, time: 2014-06-09 16:22:11.598163, event: all_read, op: pg_info(1 pgs e243:3.65d) 2014-06-09 16:22:11.779410 7f869af78700 5 -- op tracker -- , seq: 3092, time: 2014-06-09 16:22:11.779383, event: dispatched, op: pg_info(1 pgs e243:3.65d) 2014-06-09 16:22:11.779417 7f869af78700 5 -- op tracker -- , seq: 3092, time: 2014-06-09 16:22:11.779414, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.65d) 2014-06-09 16:22:11.779421 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.65d) v4 from osd.4 2014-06-09 16:22:11.779424 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b6700 2014-06-09 16:22:11.779449 7f869af78700 5 -- op tracker -- , seq: 3092, time: 2014-06-09 16:22:11.779427, event: started, op: pg_info(1 pgs e243:3.65d) 2014-06-09 16:22:11.779467 7f869af78700 5 -- op tracker -- , seq: 3092, time: 2014-06-09 16:22:11.779465, event: done, op: pg_info(1 pgs e243:3.65d) 2014-06-09 16:22:11.779475 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779478 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779490 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779492 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779493 7f869af78700 20 osd.26 243 _dispatch 0x30c308c0 pg_info(1 pgs e243:1.1ef) v4 2014-06-09 16:22:11.779498 7f869af78700 5 -- op tracker -- , seq: 3093, time: 2014-06-09 16:22:11.651276, event: header_read, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.779506 7f869af78700 5 -- op tracker -- , seq: 3093, time: 2014-06-09 16:22:11.651277, event: throttled, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.779513 7f869af78700 5 -- op tracker -- , seq: 3093, time: 2014-06-09 16:22:11.651290, event: all_read, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.779519 7f869af78700 5 -- op tracker -- , seq: 3093, time: 2014-06-09 16:22:11.779490, event: dispatched, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.779528 7f869af78700 5 -- op tracker -- , seq: 3093, time: 2014-06-09 16:22:11.779525, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.779535 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1ef) v4 from osd.3 2014-06-09 16:22:11.779538 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c308c0 2014-06-09 16:22:11.779543 7f869af78700 5 -- op tracker -- , seq: 3093, time: 2014-06-09 16:22:11.779541, event: started, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.779643 7f869af78700 5 -- op tracker -- , seq: 3093, time: 2014-06-09 16:22:11.779641, event: done, op: pg_info(1 pgs e243:1.1ef) 2014-06-09 16:22:11.779651 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779654 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779690 7f869af78700 20 osd.26 243 _dispatch 0xcd79a40 pg_info(1 pgs e243:1.1e8) v4 2014-06-09 16:22:11.779695 7f869af78700 5 -- op tracker -- , seq: 3094, time: 2014-06-09 16:22:11.644609, event: header_read, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.779703 7f869af78700 5 -- op tracker -- , seq: 3094, time: 2014-06-09 16:22:11.644610, event: throttled, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.779709 7f869af78700 5 -- op tracker -- , seq: 3094, time: 2014-06-09 16:22:11.644621, event: all_read, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.779716 7f869af78700 5 -- op tracker -- , seq: 3094, time: 2014-06-09 16:22:11.779687, event: dispatched, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.779729 7f869af78700 5 -- op tracker -- , seq: 3094, time: 2014-06-09 16:22:11.779727, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.779738 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1e8) v4 from osd.1 2014-06-09 16:22:11.779741 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd79a40 2014-06-09 16:22:11.779747 7f869af78700 5 -- op tracker -- , seq: 3094, time: 2014-06-09 16:22:11.779745, event: started, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.779767 7f869af78700 5 -- op tracker -- , seq: 3094, time: 2014-06-09 16:22:11.779765, event: done, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.779778 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779780 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779801 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.779803 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.779804 7f869af78700 20 osd.26 243 _dispatch 0xddc4700 pg_info(1 pgs e243:0.2cb) v4 2014-06-09 16:22:11.779809 7f869af78700 5 -- op tracker -- , seq: 3095, time: 2014-06-09 16:22:11.750531, event: header_read, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.779816 7f869af78700 5 -- op tracker -- , seq: 3095, time: 2014-06-09 16:22:11.750532, event: throttled, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.779823 7f869af78700 5 -- op tracker -- , seq: 3095, time: 2014-06-09 16:22:11.750545, event: all_read, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.779831 7f869af78700 5 -- op tracker -- , seq: 3095, time: 2014-06-09 16:22:11.779800, event: dispatched, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.779838 7f869af78700 5 -- op tracker -- , seq: 3095, time: 2014-06-09 16:22:11.779836, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.779843 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2cb) v4 from osd.9 2014-06-09 16:22:11.779846 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc4700 2014-06-09 16:22:11.779852 7f869af78700 5 -- op tracker -- , seq: 3095, time: 2014-06-09 16:22:11.779850, event: started, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.779867 7f869af78700 5 -- op tracker -- , seq: 3095, time: 2014-06-09 16:22:11.779865, event: done, op: pg_info(1 pgs e243:0.2cb) 2014-06-09 16:22:11.780025 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780031 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780045 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780047 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780048 7f869af78700 20 osd.26 243 _dispatch 0x2eef01c0 pg_info(1 pgs e243:1.3cd) v4 2014-06-09 16:22:11.780054 7f869af78700 5 -- op tracker -- , seq: 3096, time: 2014-06-09 16:22:11.609130, event: header_read, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.780061 7f869af78700 5 -- op tracker -- , seq: 3096, time: 2014-06-09 16:22:11.609130, event: throttled, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.780068 7f869af78700 5 -- op tracker -- , seq: 3096, time: 2014-06-09 16:22:11.609138, event: all_read, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.780080 7f869af78700 5 -- op tracker -- , seq: 3096, time: 2014-06-09 16:22:11.780045, event: dispatched, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.780087 7f869af78700 5 -- op tracker -- , seq: 3096, time: 2014-06-09 16:22:11.780085, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.780092 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3cd) v4 from osd.8 2014-06-09 16:22:11.780095 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef01c0 2014-06-09 16:22:11.780100 7f869af78700 5 -- op tracker -- , seq: 3096, time: 2014-06-09 16:22:11.780098, event: started, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.780117 7f869af78700 5 -- op tracker -- , seq: 3096, time: 2014-06-09 16:22:11.780114, event: done, op: pg_info(1 pgs e243:1.3cd) 2014-06-09 16:22:11.780124 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780126 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780138 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780141 7f869af78700 20 osd.26 243 _dispatch 0xcd78700 pg_info(1 pgs e243:0.66a) v4 2014-06-09 16:22:11.780371 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.780401 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3460.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.780406 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.780628 7f86a5e11700 10 _do_op 0x288f91a0 seq 3457 r = 0, finisher 0xbd6ef20 0x30aeac00 2014-06-09 16:22:11.780643 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd653a0] 2014-06-09 16:22:11.780649 7f86a5e11700 10 _finish_op 0x288f91a0 seq 3457 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.780662 7f868ef60700 10 op_submit_start 3509 2014-06-09 16:22:11.780664 7f8690763700 2 waiting 51 > 50 ops || 4694644 > 104857600 2014-06-09 16:22:11.780665 7f868ef60700 5 queue_transactions (trailing journal) 3509 2014-06-09 16:22:11.780667 7f868ef60700 5 queue_op 0x230b9b00 seq 3509 osr(1.30 0x384c280) 826 bytes (queue has 50 ops and 4692635 bytes) 2014-06-09 16:22:11.780671 7f868d75d700 2 waiting 51 > 50 ops || 4693688 > 104857600 2014-06-09 16:22:11.780677 7f868a757700 2 waiting 51 > 50 ops || 4693463 > 104857600 2014-06-09 16:22:11.780659 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.780685 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd560 (2 active) 2014-06-09 16:22:11.780685 7f8689755700 2 waiting 51 > 50 ops || 4694578 > 104857600 2014-06-09 16:22:11.780690 7f86a5e11700 5 _do_op 0x30032cc0 seq 3461 osr(3.75f 0x3ed1e60)/0x3ed1e60 start 2014-06-09 16:22:11.780692 7f868ef60700 10 op_submit_finish 3509 2014-06-09 16:22:11.780692 7f868bf5a700 2 waiting 51 > 50 ops || 4694589 > 104857600 2014-06-09 16:22:11.780693 7f86a5e11700 10 _do_transaction on 0xbf7ca80 2014-06-09 16:22:11.780698 7f868f761700 2 waiting 51 > 50 ops || 4694064 > 104857600 2014-06-09 16:22:11.780703 7f868cf5c700 2 waiting 51 > 50 ops || 4693463 > 104857600 2014-06-09 16:22:11.780697 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.780712 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.780726 7f868ff62700 2 waiting 51 > 50 ops || 4693970 > 104857600 2014-06-09 16:22:11.780732 7f8689f56700 2 waiting 51 > 50 ops || 4694644 > 104857600 2014-06-09 16:22:11.780738 7f8692767700 2 waiting 51 > 50 ops || 4693970 > 104857600 2014-06-09 16:22:11.780745 7f868b759700 2 waiting 51 > 50 ops || 5211015 > 104857600 2014-06-09 16:22:11.780752 7f8692f68700 2 waiting 51 > 50 ops || 4693688 > 104857600 2014-06-09 16:22:11.780146 7f869af78700 5 -- op tracker -- , seq: 3097, time: 2014-06-09 16:22:11.656975, event: header_read, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.780758 7f868df5e700 2 waiting 51 > 50 ops || 4756473 > 104857600 2014-06-09 16:22:11.780758 7f869af78700 5 -- op tracker -- , seq: 3097, time: 2014-06-09 16:22:11.656976, event: throttled, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.780764 7f869af78700 5 -- op tracker -- , seq: 3097, time: 2014-06-09 16:22:11.656987, event: all_read, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.780770 7f869af78700 5 -- op tracker -- , seq: 3097, time: 2014-06-09 16:22:11.780138, event: dispatched, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.780776 7f869af78700 5 -- op tracker -- , seq: 3097, time: 2014-06-09 16:22:11.780775, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.780781 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.66a) v4 from osd.6 2014-06-09 16:22:11.780784 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd78700 2014-06-09 16:22:11.780788 7f869af78700 5 -- op tracker -- , seq: 3097, time: 2014-06-09 16:22:11.780787, event: started, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.780804 7f869af78700 5 -- op tracker -- , seq: 3097, time: 2014-06-09 16:22:11.780802, event: done, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.780812 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780813 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780826 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780828 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780829 7f869af78700 20 osd.26 243 _dispatch 0x2fef4e00 pg_info(1 pgs e243:1.fa) v4 2014-06-09 16:22:11.780833 7f869af78700 5 -- op tracker -- , seq: 3098, time: 2014-06-09 16:22:11.746592, event: header_read, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.780840 7f869af78700 5 -- op tracker -- , seq: 3098, time: 2014-06-09 16:22:11.746593, event: throttled, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.780846 7f869af78700 5 -- op tracker -- , seq: 3098, time: 2014-06-09 16:22:11.746610, event: all_read, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.780851 7f869af78700 5 -- op tracker -- , seq: 3098, time: 2014-06-09 16:22:11.780826, event: dispatched, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.780858 7f869af78700 5 -- op tracker -- , seq: 3098, time: 2014-06-09 16:22:11.780856, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.780862 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.fa) v4 from osd.0 2014-06-09 16:22:11.780865 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef4e00 2014-06-09 16:22:11.780869 7f869af78700 5 -- op tracker -- , seq: 3098, time: 2014-06-09 16:22:11.780867, event: started, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.780883 7f869af78700 5 -- op tracker -- , seq: 3098, time: 2014-06-09 16:22:11.780881, event: done, op: pg_info(1 pgs e243:1.fa) 2014-06-09 16:22:11.780890 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780892 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780904 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780905 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780906 7f869af78700 20 osd.26 243 _dispatch 0x30d9f180 pg_info(1 pgs e243:1.66e) v4 2014-06-09 16:22:11.780911 7f869af78700 5 -- op tracker -- , seq: 3099, time: 2014-06-09 16:22:11.619322, event: header_read, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.780917 7f869af78700 5 -- op tracker -- , seq: 3099, time: 2014-06-09 16:22:11.619323, event: throttled, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.780923 7f869af78700 5 -- op tracker -- , seq: 3099, time: 2014-06-09 16:22:11.619330, event: all_read, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.780929 7f869af78700 5 -- op tracker -- , seq: 3099, time: 2014-06-09 16:22:11.780904, event: dispatched, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.780935 7f869af78700 5 -- op tracker -- , seq: 3099, time: 2014-06-09 16:22:11.780933, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.780940 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.66e) v4 from osd.5 2014-06-09 16:22:11.780942 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9f180 2014-06-09 16:22:11.780947 7f869af78700 5 -- op tracker -- , seq: 3099, time: 2014-06-09 16:22:11.780945, event: started, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.780961 7f869af78700 5 -- op tracker -- , seq: 3099, time: 2014-06-09 16:22:11.780959, event: done, op: pg_info(1 pgs e243:1.66e) 2014-06-09 16:22:11.780969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780970 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780713 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.780973 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.780977 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.780981 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.780982 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.780983 7f869af78700 20 osd.26 243 _dispatch 0x30de4c40 pg_info(1 pgs e243:1.d1) v4 2014-06-09 16:22:11.780982 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.012898 7 0.000158 2014-06-09 16:22:11.780988 7f869af78700 5 -- op tracker -- , seq: 3100, time: 2014-06-09 16:22:11.604419, event: header_read, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.780994 7f869af78700 5 -- op tracker -- , seq: 3100, time: 2014-06-09 16:22:11.604420, event: throttled, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.780994 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.781000 7f869af78700 5 -- op tracker -- , seq: 3100, time: 2014-06-09 16:22:11.604443, event: all_read, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781005 7f869af78700 5 -- op tracker -- , seq: 3100, time: 2014-06-09 16:22:11.780981, event: dispatched, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781005 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.781012 7f869af78700 5 -- op tracker -- , seq: 3100, time: 2014-06-09 16:22:11.781010, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781016 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.d1) v4 from osd.2 2014-06-09 16:22:11.781018 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de4c40 2014-06-09 16:22:11.781016 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.781023 7f869af78700 5 -- op tracker -- , seq: 3100, time: 2014-06-09 16:22:11.781021, event: started, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781026 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.781036 7f869af78700 5 -- op tracker -- , seq: 3100, time: 2014-06-09 16:22:11.781035, event: done, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781034 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.781043 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781061 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781063 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781064 7f869af78700 20 osd.26 243 _dispatch 0x2f039180 pg_info(1 pgs e243:1.67) v4 2014-06-09 16:22:11.781068 7f869af78700 5 -- op tracker -- , seq: 3101, time: 2014-06-09 16:22:11.598200, event: header_read, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.781075 7f869af78700 5 -- op tracker -- , seq: 3101, time: 2014-06-09 16:22:11.598201, event: throttled, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.781083 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.781080 7f869af78700 5 -- op tracker -- , seq: 3101, time: 2014-06-09 16:22:11.598213, event: all_read, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.781086 7f869af78700 5 -- op tracker -- , seq: 3101, time: 2014-06-09 16:22:11.781061, event: dispatched, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.781089 7f868ef60700 5 queue_transactions existing osr(0.4bd 0x2bd76f0)/0x2bd76f0 2014-06-09 16:22:11.781092 7f868ef60700 2 waiting 51 > 50 ops || 4694636 > 104857600 2014-06-09 16:22:11.781092 7f869af78700 5 -- op tracker -- , seq: 3101, time: 2014-06-09 16:22:11.781090, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.781097 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.67) v4 from osd.4 2014-06-09 16:22:11.781100 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f039180 2014-06-09 16:22:11.781105 7f869af78700 5 -- op tracker -- , seq: 3101, time: 2014-06-09 16:22:11.781103, event: started, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.781120 7f869af78700 5 -- op tracker -- , seq: 3101, time: 2014-06-09 16:22:11.781118, event: done, op: pg_info(1 pgs e243:1.67) 2014-06-09 16:22:11.781134 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781136 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781149 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781150 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781152 7f869af78700 20 osd.26 243 _dispatch 0x1ac75340 pg_info(1 pgs e243:2.112) v4 2014-06-09 16:22:11.781157 7f869af78700 5 -- op tracker -- , seq: 3102, time: 2014-06-09 16:22:11.655074, event: header_read, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.781167 7f869af78700 5 -- op tracker -- , seq: 3102, time: 2014-06-09 16:22:11.655074, event: throttled, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.781173 7f869af78700 5 -- op tracker -- , seq: 3102, time: 2014-06-09 16:22:11.655087, event: all_read, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.781180 7f869af78700 5 -- op tracker -- , seq: 3102, time: 2014-06-09 16:22:11.781148, event: dispatched, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.781187 7f869af78700 5 -- op tracker -- , seq: 3102, time: 2014-06-09 16:22:11.781185, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.781192 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.112) v4 from osd.3 2014-06-09 16:22:11.781195 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac75340 2014-06-09 16:22:11.781200 7f869af78700 5 -- op tracker -- , seq: 3102, time: 2014-06-09 16:22:11.781198, event: started, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.781215 7f869af78700 5 -- op tracker -- , seq: 3102, time: 2014-06-09 16:22:11.781213, event: done, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.781223 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781224 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781239 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781241 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781242 7f869af78700 20 osd.26 243 _dispatch 0xcd77c00 pg_info(1 pgs e243:1.1e8) v4 2014-06-09 16:22:11.781247 7f869af78700 5 -- op tracker -- , seq: 3103, time: 2014-06-09 16:22:11.644955, event: header_read, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.781254 7f869af78700 5 -- op tracker -- , seq: 3103, time: 2014-06-09 16:22:11.644956, event: throttled, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.781263 7f869af78700 5 -- op tracker -- , seq: 3103, time: 2014-06-09 16:22:11.644970, event: all_read, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.781269 7f869af78700 5 -- op tracker -- , seq: 3103, time: 2014-06-09 16:22:11.781239, event: dispatched, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.781276 7f869af78700 5 -- op tracker -- , seq: 3103, time: 2014-06-09 16:22:11.781274, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.781283 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1e8) v4 from osd.1 2014-06-09 16:22:11.781286 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd77c00 2014-06-09 16:22:11.781291 7f869af78700 5 -- op tracker -- , seq: 3103, time: 2014-06-09 16:22:11.781289, event: started, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.781306 7f869af78700 5 -- op tracker -- , seq: 3103, time: 2014-06-09 16:22:11.781304, event: done, op: pg_info(1 pgs e243:1.1e8) 2014-06-09 16:22:11.781320 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781322 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781336 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781338 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781339 7f869af78700 20 osd.26 243 _dispatch 0x30acc700 pg_info(1 pgs e243:3.375) v4 2014-06-09 16:22:11.781344 7f869af78700 5 -- op tracker -- , seq: 3104, time: 2014-06-09 16:22:11.750576, event: header_read, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.781351 7f869af78700 5 -- op tracker -- , seq: 3104, time: 2014-06-09 16:22:11.750577, event: throttled, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.781360 7f869af78700 5 -- op tracker -- , seq: 3104, time: 2014-06-09 16:22:11.750590, event: all_read, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.781366 7f869af78700 5 -- op tracker -- , seq: 3104, time: 2014-06-09 16:22:11.781335, event: dispatched, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.781373 7f869af78700 5 -- op tracker -- , seq: 3104, time: 2014-06-09 16:22:11.781371, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.781382 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.375) v4 from osd.9 2014-06-09 16:22:11.781385 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acc700 2014-06-09 16:22:11.781391 7f869af78700 5 -- op tracker -- , seq: 3104, time: 2014-06-09 16:22:11.781389, event: started, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.781409 7f869af78700 5 -- op tracker -- , seq: 3104, time: 2014-06-09 16:22:11.781407, event: done, op: pg_info(1 pgs e243:3.375) 2014-06-09 16:22:11.781416 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781418 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781431 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781433 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781434 7f869af78700 20 osd.26 243 _dispatch 0x3130afc0 pg_info(1 pgs e243:0.578) v4 2014-06-09 16:22:11.781439 7f869af78700 5 -- op tracker -- , seq: 3105, time: 2014-06-09 16:22:11.613770, event: header_read, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.781446 7f869af78700 5 -- op tracker -- , seq: 3105, time: 2014-06-09 16:22:11.613771, event: throttled, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.781452 7f869af78700 5 -- op tracker -- , seq: 3105, time: 2014-06-09 16:22:11.613782, event: all_read, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.781462 7f869af78700 5 -- op tracker -- , seq: 3105, time: 2014-06-09 16:22:11.781430, event: dispatched, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.781469 7f869af78700 5 -- op tracker -- , seq: 3105, time: 2014-06-09 16:22:11.781467, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.781474 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.578) v4 from osd.8 2014-06-09 16:22:11.781477 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130afc0 2014-06-09 16:22:11.781482 7f869af78700 5 -- op tracker -- , seq: 3105, time: 2014-06-09 16:22:11.781480, event: started, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.781497 7f869af78700 5 -- op tracker -- , seq: 3105, time: 2014-06-09 16:22:11.781495, event: done, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.781504 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781506 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781518 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781520 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781521 7f869af78700 20 osd.26 243 _dispatch 0x30d9cfc0 pg_info(1 pgs e243:3.6ba) v4 2014-06-09 16:22:11.781526 7f869af78700 5 -- op tracker -- , seq: 3106, time: 2014-06-09 16:22:11.657020, event: header_read, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.781533 7f869af78700 5 -- op tracker -- , seq: 3106, time: 2014-06-09 16:22:11.657021, event: throttled, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.781540 7f869af78700 5 -- op tracker -- , seq: 3106, time: 2014-06-09 16:22:11.657033, event: all_read, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.781546 7f869af78700 5 -- op tracker -- , seq: 3106, time: 2014-06-09 16:22:11.781518, event: dispatched, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.781554 7f869af78700 5 -- op tracker -- , seq: 3106, time: 2014-06-09 16:22:11.781552, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.781560 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6ba) v4 from osd.6 2014-06-09 16:22:11.781564 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9cfc0 2014-06-09 16:22:11.781569 7f869af78700 5 -- op tracker -- , seq: 3106, time: 2014-06-09 16:22:11.781567, event: started, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.781585 7f869af78700 5 -- op tracker -- , seq: 3106, time: 2014-06-09 16:22:11.781583, event: done, op: pg_info(1 pgs e243:3.6ba) 2014-06-09 16:22:11.781593 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781595 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781607 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781608 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781610 7f869af78700 20 osd.26 243 _dispatch 0x3130b180 pg_info(1 pgs e243:0.71b) v4 2014-06-09 16:22:11.781615 7f869af78700 5 -- op tracker -- , seq: 3107, time: 2014-06-09 16:22:11.645201, event: header_read, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.781622 7f869af78700 5 -- op tracker -- , seq: 3107, time: 2014-06-09 16:22:11.645201, event: throttled, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.781628 7f869af78700 5 -- op tracker -- , seq: 3107, time: 2014-06-09 16:22:11.645215, event: all_read, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.781634 7f869af78700 5 -- op tracker -- , seq: 3107, time: 2014-06-09 16:22:11.781606, event: dispatched, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.781647 7f869af78700 5 -- op tracker -- , seq: 3107, time: 2014-06-09 16:22:11.781645, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.781651 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.71b) v4 from osd.5 2014-06-09 16:22:11.781654 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130b180 2014-06-09 16:22:11.781660 7f869af78700 5 -- op tracker -- , seq: 3107, time: 2014-06-09 16:22:11.781658, event: started, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.781677 7f869af78700 5 -- op tracker -- , seq: 3107, time: 2014-06-09 16:22:11.781675, event: done, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.781687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781700 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781702 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781703 7f869af78700 20 osd.26 243 _dispatch 0x30da3c00 pg_info(1 pgs e243:1.d1) v4 2014-06-09 16:22:11.781708 7f869af78700 5 -- op tracker -- , seq: 3108, time: 2014-06-09 16:22:11.604449, event: header_read, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781720 7f869af78700 5 -- op tracker -- , seq: 3108, time: 2014-06-09 16:22:11.604450, event: throttled, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781727 7f869af78700 5 -- op tracker -- , seq: 3108, time: 2014-06-09 16:22:11.604459, event: all_read, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781736 7f869af78700 5 -- op tracker -- , seq: 3108, time: 2014-06-09 16:22:11.781700, event: dispatched, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781743 7f869af78700 5 -- op tracker -- , seq: 3108, time: 2014-06-09 16:22:11.781741, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781748 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.d1) v4 from osd.2 2014-06-09 16:22:11.781750 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da3c00 2014-06-09 16:22:11.781755 7f869af78700 5 -- op tracker -- , seq: 3108, time: 2014-06-09 16:22:11.781753, event: started, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781770 7f869af78700 5 -- op tracker -- , seq: 3108, time: 2014-06-09 16:22:11.781768, event: done, op: pg_info(1 pgs e243:1.d1) 2014-06-09 16:22:11.781783 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781785 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781859 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781861 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781863 7f869af78700 20 osd.26 243 _dispatch 0x30a3ba40 pg_info(1 pgs e243:1.6f) v4 2014-06-09 16:22:11.781868 7f869af78700 5 -- op tracker -- , seq: 3109, time: 2014-06-09 16:22:11.604317, event: header_read, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.781876 7f869af78700 5 -- op tracker -- , seq: 3109, time: 2014-06-09 16:22:11.604318, event: throttled, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.781883 7f869af78700 5 -- op tracker -- , seq: 3109, time: 2014-06-09 16:22:11.604331, event: all_read, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.781889 7f869af78700 5 -- op tracker -- , seq: 3109, time: 2014-06-09 16:22:11.781859, event: dispatched, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.781896 7f869af78700 5 -- op tracker -- , seq: 3109, time: 2014-06-09 16:22:11.781894, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.781903 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6f) v4 from osd.4 2014-06-09 16:22:11.781906 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3ba40 2014-06-09 16:22:11.781912 7f869af78700 5 -- op tracker -- , seq: 3109, time: 2014-06-09 16:22:11.781910, event: started, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.781932 7f869af78700 5 -- op tracker -- , seq: 3109, time: 2014-06-09 16:22:11.781930, event: done, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.781940 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781942 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781954 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.781956 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.781957 7f869af78700 20 osd.26 243 _dispatch 0x1ac756c0 pg_info(1 pgs e243:0.765) v4 2014-06-09 16:22:11.781962 7f869af78700 5 -- op tracker -- , seq: 3110, time: 2014-06-09 16:22:11.655110, event: header_read, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.781969 7f869af78700 5 -- op tracker -- , seq: 3110, time: 2014-06-09 16:22:11.655111, event: throttled, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.781975 7f869af78700 5 -- op tracker -- , seq: 3110, time: 2014-06-09 16:22:11.655123, event: all_read, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.781982 7f869af78700 5 -- op tracker -- , seq: 3110, time: 2014-06-09 16:22:11.781953, event: dispatched, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.787655 7f86a5610700 10 _do_op 0x2ea532c0 seq 3460 r = 0, finisher 0x31413aa0 0 2014-06-09 16:22:11.787665 7f86a5610700 10 _finish_op 0x2ea532c0 seq 3460 osr(0.6f0 0x384c170)/0x384c170 2014-06-09 16:22:11.787675 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338120 (1 active) 2014-06-09 16:22:11.787679 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300240 (2 active) 2014-06-09 16:22:11.787679 7f8690763700 10 op_submit_start 3510 2014-06-09 16:22:11.787680 7f868d75d700 2 waiting 51 > 50 ops || 4693743 > 104857600 2014-06-09 16:22:11.787682 7f8690763700 5 queue_transactions (trailing journal) 3510 2014-06-09 16:22:11.787682 7f86a5610700 5 _do_op 0x30d96480 seq 3462 osr(0.512 0x2bd7590)/0x2bd7590 start 2014-06-09 16:22:11.787686 7f86a5610700 10 _do_transaction on 0xbf46180 2014-06-09 16:22:11.787685 7f8690763700 5 queue_op 0x2e21200 seq 3510 osr(1.188 0x384cb70) 2009 bytes (queue has 50 ops and 4692690 bytes) 2014-06-09 16:22:11.787688 7f868a757700 2 waiting 51 > 50 ops || 4693518 > 104857600 2014-06-09 16:22:11.787690 7f8690763700 10 op_submit_finish 3510 2014-06-09 16:22:11.787694 7f8689755700 2 waiting 51 > 50 ops || 4694633 > 104857600 2014-06-09 16:22:11.787695 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.787700 7f868bf5a700 2 waiting 51 > 50 ops || 4694644 > 104857600 2014-06-09 16:22:11.787705 7f868f761700 2 waiting 51 > 50 ops || 4694119 > 104857600 2014-06-09 16:22:11.787696 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.631( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.787711 7f868cf5c700 2 waiting 51 > 50 ops || 4693518 > 104857600 2014-06-09 16:22:11.787716 7f868ff62700 2 waiting 51 > 50 ops || 4694025 > 104857600 2014-06-09 16:22:11.787722 7f8689f56700 2 waiting 51 > 50 ops || 4694699 > 104857600 2014-06-09 16:22:11.787731 7f8692767700 2 waiting 51 > 50 ops || 4694025 > 104857600 2014-06-09 16:22:11.787739 7f868b759700 2 waiting 51 > 50 ops || 5211070 > 104857600 2014-06-09 16:22:11.787743 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.787748 7f8692f68700 2 waiting 51 > 50 ops || 4693743 > 104857600 2014-06-09 16:22:11.787752 7f8690763700 5 queue_transactions existing osr(1.631 0x384dfb0)/0x384dfb0 2014-06-09 16:22:11.787754 7f8690763700 2 waiting 51 > 50 ops || 4693518 > 104857600 2014-06-09 16:22:11.787756 7f868df5e700 2 waiting 51 > 50 ops || 4756528 > 104857600 2014-06-09 16:22:11.787764 7f868ef60700 2 waiting 51 > 50 ops || 4694691 > 104857600 2014-06-09 16:22:11.787781 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.787809 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3461.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.787814 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.787831 7f86a5e11700 10 _do_op 0x30032cc0 seq 3461 r = 0, finisher 0x31456ae0 0 2014-06-09 16:22:11.787835 7f86a5e11700 10 _finish_op 0x30032cc0 seq 3461 osr(3.75f 0x3ed1e60)/0x3ed1e60 2014-06-09 16:22:11.787847 7f868a757700 10 op_submit_start 3511 2014-06-09 16:22:11.787849 7f868a757700 5 queue_transactions (trailing journal) 3511 2014-06-09 16:22:11.787850 7f868a757700 5 queue_op 0x2fcaf020 seq 3511 osr(0.4da 0x2bd75f0) 828 bytes (queue has 50 ops and 4692465 bytes) 2014-06-09 16:22:11.787856 7f8689755700 2 waiting 51 > 50 ops || 4694408 > 104857600 2014-06-09 16:22:11.787859 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.787861 7f868bf5a700 2 waiting 51 > 50 ops || 4694419 > 104857600 2014-06-09 16:22:11.787865 7f868f761700 2 waiting 51 > 50 ops || 4693894 > 104857600 2014-06-09 16:22:11.787870 7f868cf5c700 2 waiting 51 > 50 ops || 4693293 > 104857600 2014-06-09 16:22:11.787874 7f868ff62700 2 waiting 51 > 50 ops || 4693800 > 104857600 2014-06-09 16:22:11.787880 7f8689f56700 2 waiting 51 > 50 ops || 4694474 > 104857600 2014-06-09 16:22:11.787880 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd560 (1 active) 2014-06-09 16:22:11.787883 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b440 (2 active) 2014-06-09 16:22:11.787887 7f8692767700 2 waiting 51 > 50 ops || 4693800 > 104857600 2014-06-09 16:22:11.787888 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3462.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.787893 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.787893 7f868b759700 2 waiting 51 > 50 ops || 5210845 > 104857600 2014-06-09 16:22:11.787895 7f868a757700 10 op_submit_finish 3511 2014-06-09 16:22:11.787901 7f8692f68700 2 waiting 51 > 50 ops || 4693518 > 104857600 2014-06-09 16:22:11.787899 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.787912 7f8690763700 2 waiting 51 > 50 ops || 4693293 > 104857600 2014-06-09 16:22:11.787919 7f868df5e700 2 waiting 51 > 50 ops || 4756303 > 104857600 2014-06-09 16:22:11.787916 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.787926 7f868ef60700 2 waiting 51 > 50 ops || 4694466 > 104857600 2014-06-09 16:22:11.787927 7f86a5610700 10 _do_op 0x30d96480 seq 3462 r = 0, finisher 0x3042aac0 0 2014-06-09 16:22:11.787931 7f86a5610700 10 _finish_op 0x30d96480 seq 3462 osr(0.512 0x2bd7590)/0x2bd7590 2014-06-09 16:22:11.787926 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.787935 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.787943 7f868bf5a700 10 op_submit_start 3512 2014-06-09 16:22:11.787945 7f868bf5a700 5 queue_transactions (trailing journal) 3512 2014-06-09 16:22:11.787947 7f868bf5a700 5 queue_op 0x309a37a0 seq 3512 osr(2.1e0 0x2ed0850) 1954 bytes (queue has 50 ops and 4692465 bytes) 2014-06-09 16:22:11.787952 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300240 (1 active) 2014-06-09 16:22:11.787953 7f868a757700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.787954 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac120 (2 active) 2014-06-09 16:22:11.787960 7f868bf5a700 10 op_submit_finish 3512 2014-06-09 16:22:11.787964 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.787972 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:171 2014-06-09 16:22:11.787979 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.787990 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.787990 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0xe2adb00,0xe2ada00,0xe2ad900 2014-06-09 16:22:11.787996 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.75( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.788015 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.788032 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.788026 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.788038 7f868a757700 5 queue_transactions existing osr(2.75 0x2ed04f0)/0x2ed04f0 2014-06-09 16:22:11.788041 7f868a757700 2 waiting 51 > 50 ops || 4693291 > 104857600 2014-06-09 16:22:11.788036 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.788045 7f868d75d700 2 waiting 51 > 50 ops || 4693518 > 104857600 2014-06-09 16:22:11.788050 7f868f761700 2 waiting 51 > 50 ops || 4693894 > 104857600 2014-06-09 16:22:11.788055 7f868cf5c700 2 waiting 51 > 50 ops || 4693293 > 104857600 2014-06-09 16:22:11.788060 7f86a5e11700 5 _do_op 0xd04f620 seq 3463 osr(3.4c0 0x3ed1440)/0x3ed1440 start 2014-06-09 16:22:11.788053 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:855 2014-06-09 16:22:11.788064 7f86a5e11700 10 _do_transaction on 0xc209740 2014-06-09 16:22:11.788071 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.788064 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.788075 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.788086 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.788092 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:1.1eb( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.788108 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.788105 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.788115 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.788131 7f86a5e11700 15 _omap_rmkeyrange meta/1fe06515/pglog_3.4c0/0//-1 [0000000214.00000000000000000583,4294967295.18446744073709551615] 2014-06-09 16:22:11.788135 7f86a5e11700 15 get_omap_iterator meta/1fe06515/pglog_3.4c0/0//-1 2014-06-09 16:22:11.788143 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41980 (19 active) 2014-06-09 16:22:11.788166 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) 2014-06-09 16:22:11.788171 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.788183 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.788193 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.788200 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.788213 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.788231 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:162 2014-06-09 16:22:11.788245 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.788250 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.603( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.603( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.788267 7f86a5e11700 15 _omap_rmkeys meta/1fe06515/pglog_3.4c0/0//-1 2014-06-09 16:22:11.788282 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.788286 7f868bf5a700 5 queue_transactions existing osr(1.603 0x384dc90)/0x384dc90 2014-06-09 16:22:11.788289 7f868bf5a700 2 waiting 51 > 50 ops || 4693293 > 104857600 2014-06-09 16:22:11.788293 7f8689f56700 2 waiting 51 > 50 ops || 4694474 > 104857600 2014-06-09 16:22:11.788298 7f86a5610700 5 _do_op 0x2fc8c6c0 seq 3464 osr(3.5f9 0x3ed1b40)/0x3ed1b40 start 2014-06-09 16:22:11.788301 7f86a5610700 10 _do_transaction on 0xc0da3c0 2014-06-09 16:22:11.788299 7f8692767700 2 waiting 51 > 50 ops || 4693800 > 104857600 2014-06-09 16:22:11.788307 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.788307 7f868b759700 2 waiting 51 > 50 ops || 5210845 > 104857600 2014-06-09 16:22:11.788317 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe06515/pglog_3.4c0/0//-1 2014-06-09 16:22:11.788318 7f8692f68700 2 waiting 51 > 50 ops || 4693518 > 104857600 2014-06-09 16:22:11.788325 7f8690763700 2 waiting 51 > 50 ops || 4693293 > 104857600 2014-06-09 16:22:11.788329 7f86a5e11700 10 _omap_rmkeys meta/1fe06515/pglog_3.4c0/0//-1 r = 0 2014-06-09 16:22:11.788332 7f868df5e700 2 waiting 51 > 50 ops || 4756303 > 104857600 2014-06-09 16:22:11.788334 7f86a5e11700 15 _omap_rmkeys meta/1fe06515/pglog_3.4c0/0//-1 2014-06-09 16:22:11.788338 7f86a5e11700 10 _omap_rmkeys meta/1fe06515/pglog_3.4c0/0//-1 r = 0 2014-06-09 16:22:11.788340 7f868ef60700 2 waiting 51 > 50 ops || 4694466 > 104857600 2014-06-09 16:22:11.788348 7f86a5e11700 15 _omap_setkeys meta/1fe06515/pglog_3.4c0/0//-1 2014-06-09 16:22:11.788348 7f8689755700 2 waiting 51 > 50 ops || 4694408 > 104857600 2014-06-09 16:22:11.788354 7f868ff62700 2 waiting 51 > 50 ops || 4693800 > 104857600 2014-06-09 16:22:11.788369 7f86a5e11700 10 cid: oid: 1fe06515/pglog_3.4c0/0//-1 skipping op, spos 3463.0.0 <= header.spos 150422.0.0 2014-06-09 16:22:11.788375 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3463.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.788378 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.788428 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.788431 7f86a5e11700 10 _do_op 0xd04f620 seq 3463 r = 0, finisher 0x31478520 0 2014-06-09 16:22:11.788435 7f86a5e11700 10 _finish_op 0xd04f620 seq 3463 osr(3.4c0 0x3ed1440)/0x3ed1440 2014-06-09 16:22:11.788443 7f868a757700 10 op_submit_start 3513 2014-06-09 16:22:11.788443 7f868d75d700 2 waiting 51 > 50 ops || 4692184 > 104857600 2014-06-09 16:22:11.788445 7f868a757700 5 queue_transactions (trailing journal) 3513 2014-06-09 16:22:11.788448 7f868f761700 2 waiting 51 > 50 ops || 4692560 > 104857600 2014-06-09 16:22:11.788446 7f868a757700 5 queue_op 0x2fcaf0e0 seq 3513 osr(2.75 0x2ed04f0) 826 bytes (queue has 50 ops and 4691131 bytes) 2014-06-09 16:22:11.788449 7f86a5610700 15 _omap_rmkeyrange meta/1fe7d505/pglog_3.5f9/0//-1 [0000000213.00000000000000000610,4294967295.18446744073709551615] 2014-06-09 16:22:11.788451 7f868cf5c700 2 waiting 51 > 50 ops || 4691959 > 104857600 2014-06-09 16:22:11.788452 7f86a5610700 15 get_omap_iterator meta/1fe7d505/pglog_3.5f9/0//-1 2014-06-09 16:22:11.788441 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b440 (1 active) 2014-06-09 16:22:11.788457 7f868bf5a700 2 waiting 51 > 50 ops || 4691959 > 104857600 2014-06-09 16:22:11.788458 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcea0 (2 active) 2014-06-09 16:22:11.788463 7f868a757700 10 op_submit_finish 3513 2014-06-09 16:22:11.788463 7f8689f56700 2 waiting 51 > 50 ops || 4693140 > 104857600 2014-06-09 16:22:11.788467 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=191 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.793( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.788482 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=191 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.793( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.788497 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=191 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.809909 7 0.000153 2014-06-09 16:22:11.788511 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=191 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.788525 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=191 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.788537 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=191 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.788545 7f86a5610700 15 _omap_rmkeys meta/1fe7d505/pglog_3.5f9/0//-1 2014-06-09 16:22:11.788546 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'642 -> 213'642 2014-06-09 16:22:11.788579 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe7d505/pglog_3.5f9/0//-1 2014-06-09 16:22:11.788591 7f86a5610700 10 _omap_rmkeys meta/1fe7d505/pglog_3.5f9/0//-1 r = 0 2014-06-09 16:22:11.788588 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.788596 7f86a5610700 15 _omap_rmkeys meta/1fe7d505/pglog_3.5f9/0//-1 2014-06-09 16:22:11.788601 7f86a5610700 10 _omap_rmkeys meta/1fe7d505/pglog_3.5f9/0//-1 r = 0 2014-06-09 16:22:11.788608 7f86a5610700 15 _omap_setkeys meta/1fe7d505/pglog_3.5f9/0//-1 2014-06-09 16:22:11.788623 7f86a5610700 10 cid: oid: 1fe7d505/pglog_3.5f9/0//-1 skipping op, spos 3464.0.0 <= header.spos 149584.0.0 2014-06-09 16:22:11.788629 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.788629 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3464.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.788632 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.788634 7f868a757700 5 queue_transactions existing osr(3.793 0x3ed1fd0)/0x3ed1fd0 2014-06-09 16:22:11.788637 7f868a757700 2 waiting 51 > 50 ops || 4692560 > 104857600 2014-06-09 16:22:11.788642 7f8692767700 2 waiting 51 > 50 ops || 4692466 > 104857600 2014-06-09 16:22:11.788648 7f868b759700 2 waiting 51 > 50 ops || 5209511 > 104857600 2014-06-09 16:22:11.788648 7f86a5e11700 5 _do_op 0x30d96a20 seq 3465 osr(3.342 0x3ed10e0)/0x3ed10e0 start 2014-06-09 16:22:11.788653 7f86a5e11700 10 _do_transaction on 0xbf29ec0 2014-06-09 16:22:11.788656 7f8692f68700 2 waiting 51 > 50 ops || 4692184 > 104857600 2014-06-09 16:22:11.788659 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.788659 7f86a5610700 10 _do_op 0x2fc8c6c0 seq 3464 r = 0, finisher 0x2d738fe0 0 2014-06-09 16:22:11.788663 7f86a5610700 10 _finish_op 0x2fc8c6c0 seq 3464 osr(3.5f9 0x3ed1b40)/0x3ed1b40 2014-06-09 16:22:11.788671 7f868df5e700 2 waiting 51 > 50 ops || 4754296 > 104857600 2014-06-09 16:22:11.788672 7f868d75d700 10 op_submit_start 3514 2014-06-09 16:22:11.788674 7f868d75d700 5 queue_transactions (trailing journal) 3514 2014-06-09 16:22:11.788675 7f868d75d700 5 queue_op 0x300336e0 seq 3514 osr(3.582 0x3ed18a0) 1053 bytes (queue has 50 ops and 4690458 bytes) 2014-06-09 16:22:11.788677 7f8689755700 2 waiting 51 > 50 ops || 4692401 > 104857600 2014-06-09 16:22:11.788682 7f868ff62700 2 waiting 51 > 50 ops || 4691793 > 104857600 2014-06-09 16:22:11.788688 7f868f761700 2 waiting 51 > 50 ops || 4691887 > 104857600 2014-06-09 16:22:11.788688 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac120 (1 active) 2014-06-09 16:22:11.788692 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc5a0 (2 active) 2014-06-09 16:22:11.788695 7f868cf5c700 2 waiting 51 > 50 ops || 4691286 > 104857600 2014-06-09 16:22:11.788697 7f868d75d700 10 op_submit_finish 3514 2014-06-09 16:22:11.788701 7f868bf5a700 2 waiting 51 > 50 ops || 4691286 > 104857600 2014-06-09 16:22:11.788701 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.788717 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.788715 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.647476 7 0.000351 2014-06-09 16:22:11.788728 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.788739 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3465.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.788744 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.788740 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.788756 7f86a5e11700 10 _do_op 0x30d96a20 seq 3465 r = 0, finisher 0x30d61dc0 0 2014-06-09 16:22:11.788752 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 2014-06-09 16:22:11.788760 7f86a5e11700 10 _finish_op 0x30d96a20 seq 3465 osr(3.342 0x3ed10e0)/0x3ed10e0 2014-06-09 16:22:11.788763 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.788770 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcea0 (1 active) 2014-06-09 16:22:11.788772 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4a20 (2 active) 2014-06-09 16:22:11.788773 7f868a757700 10 op_submit_start 3515 2014-06-09 16:22:11.788775 7f868a757700 5 queue_transactions (trailing journal) 3515 2014-06-09 16:22:11.788774 7f86a5e11700 5 _do_op 0xc34ae40 seq 3467 osr(3.1e 0x3ed0380)/0x3ed0380 start 2014-06-09 16:22:11.788777 7f86a5e11700 10 _do_transaction on 0xc23ee40 2014-06-09 16:22:11.788776 7f868a757700 5 queue_op 0x2fcaf140 seq 3515 osr(3.793 0x3ed1fd0) 1429 bytes (queue has 50 ops and 4690458 bytes) 2014-06-09 16:22:11.788779 7f868a757700 10 op_submit_finish 3515 2014-06-09 16:22:11.788773 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.788781 7f86a5e11700 15 _omap_setkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.788782 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.788782 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.788794 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.71f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.788794 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:11.788807 7f868a757700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.788811 7f868a757700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.788806 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:11.788819 7f86a5e11700 10 lookup_strip_header done cid meta oid 51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.788815 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.891084 7 0.000195 2014-06-09 16:22:11.788816 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.788825 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.788832 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.788833 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.788838 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.788847 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.788848 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:11.788856 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.788863 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.788864 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.788876 7f86a5e11700 15 _omap_rmkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.788881 7f86a5e11700 10 _omap_rmkeys meta/51682acc/pglog_3.1e/0//-1 r = 0 2014-06-09 16:22:11.788884 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.788886 7f86a5e11700 15 _omap_setkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.788889 7f868d75d700 5 queue_transactions existing osr(2.36c 0x2ed1170)/0x2ed1170 2014-06-09 16:22:11.788892 7f868d75d700 2 waiting 51 > 50 ops || 4691319 > 104857600 2014-06-09 16:22:11.788896 7f8689f56700 2 waiting 51 > 50 ops || 4692467 > 104857600 2014-06-09 16:22:11.788897 7f86a5e11700 15 _touch 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.788902 7f86a5610700 5 _do_op 0x2fcaf620 seq 3466 osr(1.5cb 0x384dd70)/0x384dd70 start 2014-06-09 16:22:11.788902 7f868b759700 2 waiting 51 > 50 ops || 5208838 > 104857600 2014-06-09 16:22:11.788906 7f86a5610700 10 _do_transaction on 0xbdbd980 2014-06-09 16:22:11.788911 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.788912 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.788916 7f868a757700 5 queue_transactions existing osr(0.71f 0x384c060)/0x384c060 2014-06-09 16:22:11.788918 7f868a757700 2 waiting 51 > 50 ops || 4692361 > 104857600 2014-06-09 16:22:11.788923 7f8692767700 2 waiting 51 > 50 ops || 4691793 > 104857600 2014-06-09 16:22:11.788927 7f8690763700 2 waiting 51 > 50 ops || 4691286 > 104857600 2014-06-09 16:22:11.788934 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.788940 7f868ef60700 2 waiting 51 > 50 ops || 4692459 > 104857600 2014-06-09 16:22:11.788947 7f8689755700 2 waiting 51 > 50 ops || 4692401 > 104857600 2014-06-09 16:22:11.788952 7f868ff62700 2 waiting 51 > 50 ops || 4691793 > 104857600 2014-06-09 16:22:11.788959 7f868f761700 2 waiting 51 > 50 ops || 4691887 > 104857600 2014-06-09 16:22:11.788965 7f868cf5c700 2 waiting 51 > 50 ops || 4691286 > 104857600 2014-06-09 16:22:11.788972 7f868bf5a700 2 waiting 51 > 50 ops || 4691286 > 104857600 2014-06-09 16:22:11.788976 7f868df5e700 2 waiting 51 > 50 ops || 4754296 > 104857600 2014-06-09 16:22:11.788962 7f86a5e11700 20 write_state seq is 30154 2014-06-09 16:22:11.788982 7f8692f68700 2 waiting 51 > 50 ops || 4691511 > 104857600 2014-06-09 16:22:11.788988 7f86a5e11700 20 set_header setting 30153 cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 parent seq 0 2014-06-09 16:22:11.789036 7f86a5e11700 10 _touch 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.789043 7f86a5e11700 15 _write 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 0~352256 2014-06-09 16:22:11.789056 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.790210 7f86a5e11700 10 _generic_write 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 0~352256 = 0 2014-06-09 16:22:11.790233 7f86a5e11700 15 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.790244 7f86a5e11700 10 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.790251 7f86a5e11700 15 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.790259 7f86a5e11700 10 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.790318 7f86a5e11700 10 cid: oid: 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 not skipping op, *spos 3467.0.0 2014-06-09 16:22:11.790323 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:11.790336 7f86a5e11700 20 set_header setting 30153 cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 parent seq 0 2014-06-09 16:22:11.790372 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3467.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.790376 7f86a5e11700 10 cid: oid: 51682acc/pglog_3.1e/0//-1 skipping op, spos 3467.0.0 <= header.spos 150326.0.0 2014-06-09 16:22:11.790378 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.781989 7f869af78700 5 -- op tracker -- , seq: 3110, time: 2014-06-09 16:22:11.781987, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.790515 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.765) v4 from osd.3 2014-06-09 16:22:11.790524 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac756c0 2014-06-09 16:22:11.790533 7f869af78700 5 -- op tracker -- , seq: 3110, time: 2014-06-09 16:22:11.790529, event: started, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.790556 7f869af78700 5 -- op tracker -- , seq: 3110, time: 2014-06-09 16:22:11.790553, event: done, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.790567 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790569 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790590 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790592 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790593 7f869af78700 20 osd.26 243 _dispatch 0x2fc56700 pg_info(1 pgs e243:2.48d) v4 2014-06-09 16:22:11.790599 7f869af78700 5 -- op tracker -- , seq: 3111, time: 2014-06-09 16:22:11.650629, event: header_read, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.790607 7f869af78700 5 -- op tracker -- , seq: 3111, time: 2014-06-09 16:22:11.650630, event: throttled, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.790614 7f869af78700 5 -- op tracker -- , seq: 3111, time: 2014-06-09 16:22:11.650640, event: all_read, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.790620 7f869af78700 5 -- op tracker -- , seq: 3111, time: 2014-06-09 16:22:11.790589, event: dispatched, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.790627 7f869af78700 5 -- op tracker -- , seq: 3111, time: 2014-06-09 16:22:11.790625, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.790636 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.48d) v4 from osd.1 2014-06-09 16:22:11.790639 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc56700 2014-06-09 16:22:11.790644 7f869af78700 5 -- op tracker -- , seq: 3111, time: 2014-06-09 16:22:11.790642, event: started, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.790661 7f869af78700 5 -- op tracker -- , seq: 3111, time: 2014-06-09 16:22:11.790658, event: done, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.790668 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790670 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790685 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790688 7f869af78700 20 osd.26 243 _dispatch 0x1405da40 pg_info(1 pgs e243:0.578) v4 2014-06-09 16:22:11.790693 7f869af78700 5 -- op tracker -- , seq: 3112, time: 2014-06-09 16:22:11.613801, event: header_read, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.790700 7f869af78700 5 -- op tracker -- , seq: 3112, time: 2014-06-09 16:22:11.613802, event: throttled, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.790707 7f869af78700 5 -- op tracker -- , seq: 3112, time: 2014-06-09 16:22:11.613813, event: all_read, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.790716 7f869af78700 5 -- op tracker -- , seq: 3112, time: 2014-06-09 16:22:11.790684, event: dispatched, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.790724 7f869af78700 5 -- op tracker -- , seq: 3112, time: 2014-06-09 16:22:11.790722, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.790728 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.578) v4 from osd.8 2014-06-09 16:22:11.790731 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1405da40 2014-06-09 16:22:11.790736 7f869af78700 5 -- op tracker -- , seq: 3112, time: 2014-06-09 16:22:11.790734, event: started, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.790753 7f869af78700 5 -- op tracker -- , seq: 3112, time: 2014-06-09 16:22:11.790751, event: done, op: pg_info(1 pgs e243:0.578) 2014-06-09 16:22:11.790761 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790763 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790780 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790783 7f869af78700 20 osd.26 243 _dispatch 0x30d9ce00 pg_info(1 pgs e243:3.2d6) v4 2014-06-09 16:22:11.790788 7f869af78700 5 -- op tracker -- , seq: 3113, time: 2014-06-09 16:22:11.657070, event: header_read, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.790798 7f869af78700 5 -- op tracker -- , seq: 3113, time: 2014-06-09 16:22:11.657070, event: throttled, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.790807 7f869af78700 5 -- op tracker -- , seq: 3113, time: 2014-06-09 16:22:11.657082, event: all_read, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.790816 7f869af78700 5 -- op tracker -- , seq: 3113, time: 2014-06-09 16:22:11.790779, event: dispatched, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.790823 7f869af78700 5 -- op tracker -- , seq: 3113, time: 2014-06-09 16:22:11.790821, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.790828 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2d6) v4 from osd.6 2014-06-09 16:22:11.790831 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9ce00 2014-06-09 16:22:11.790836 7f869af78700 5 -- op tracker -- , seq: 3113, time: 2014-06-09 16:22:11.790834, event: started, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.790853 7f869af78700 5 -- op tracker -- , seq: 3113, time: 2014-06-09 16:22:11.790851, event: done, op: pg_info(1 pgs e243:3.2d6) 2014-06-09 16:22:11.790863 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790865 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790877 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790879 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790880 7f869af78700 20 osd.26 243 _dispatch 0x30d9efc0 pg_info(1 pgs e243:0.71b) v4 2014-06-09 16:22:11.790885 7f869af78700 5 -- op tracker -- , seq: 3114, time: 2014-06-09 16:22:11.647627, event: header_read, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.790892 7f869af78700 5 -- op tracker -- , seq: 3114, time: 2014-06-09 16:22:11.647627, event: throttled, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.790905 7f869af78700 5 -- op tracker -- , seq: 3114, time: 2014-06-09 16:22:11.647642, event: all_read, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.790911 7f869af78700 5 -- op tracker -- , seq: 3114, time: 2014-06-09 16:22:11.790877, event: dispatched, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.790932 7f869af78700 5 -- op tracker -- , seq: 3114, time: 2014-06-09 16:22:11.790929, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.790936 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.71b) v4 from osd.5 2014-06-09 16:22:11.790939 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9efc0 2014-06-09 16:22:11.790945 7f869af78700 5 -- op tracker -- , seq: 3114, time: 2014-06-09 16:22:11.790943, event: started, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.790962 7f869af78700 5 -- op tracker -- , seq: 3114, time: 2014-06-09 16:22:11.790960, event: done, op: pg_info(1 pgs e243:0.71b) 2014-06-09 16:22:11.790969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790971 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790985 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.790986 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.790988 7f869af78700 20 osd.26 243 _dispatch 0x30a3a540 pg_info(1 pgs e243:0.654) v4 2014-06-09 16:22:11.790993 7f869af78700 5 -- op tracker -- , seq: 3115, time: 2014-06-09 16:22:11.606442, event: header_read, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.791002 7f869af78700 5 -- op tracker -- , seq: 3115, time: 2014-06-09 16:22:11.606442, event: throttled, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.791009 7f869af78700 5 -- op tracker -- , seq: 3115, time: 2014-06-09 16:22:11.606450, event: all_read, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.791015 7f869af78700 5 -- op tracker -- , seq: 3115, time: 2014-06-09 16:22:11.790984, event: dispatched, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.791028 7f869af78700 5 -- op tracker -- , seq: 3115, time: 2014-06-09 16:22:11.791025, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.791033 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.654) v4 from osd.2 2014-06-09 16:22:11.791036 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3a540 2014-06-09 16:22:11.791041 7f869af78700 5 -- op tracker -- , seq: 3115, time: 2014-06-09 16:22:11.791039, event: started, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.791056 7f869af78700 5 -- op tracker -- , seq: 3115, time: 2014-06-09 16:22:11.791054, event: done, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.791064 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791066 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791078 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791080 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791081 7f869af78700 20 osd.26 243 _dispatch 0x30a3a700 pg_info(1 pgs e243:3.f0) v4 2014-06-09 16:22:11.791086 7f869af78700 5 -- op tracker -- , seq: 3116, time: 2014-06-09 16:22:11.605009, event: header_read, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.791093 7f869af78700 5 -- op tracker -- , seq: 3116, time: 2014-06-09 16:22:11.605010, event: throttled, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.791100 7f869af78700 5 -- op tracker -- , seq: 3116, time: 2014-06-09 16:22:11.605019, event: all_read, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.791107 7f869af78700 5 -- op tracker -- , seq: 3116, time: 2014-06-09 16:22:11.791078, event: dispatched, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.791116 7f869af78700 5 -- op tracker -- , seq: 3116, time: 2014-06-09 16:22:11.791114, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.791121 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.f0) v4 from osd.4 2014-06-09 16:22:11.791125 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3a700 2014-06-09 16:22:11.791130 7f869af78700 5 -- op tracker -- , seq: 3116, time: 2014-06-09 16:22:11.791128, event: started, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.791146 7f869af78700 5 -- op tracker -- , seq: 3116, time: 2014-06-09 16:22:11.791144, event: done, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.791154 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791156 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791167 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791169 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791169 7f869af78700 20 osd.26 243 _dispatch 0xcd78380 pg_info(1 pgs e243:3.1d5) v4 2014-06-09 16:22:11.791174 7f869af78700 5 -- op tracker -- , seq: 3117, time: 2014-06-09 16:22:11.655152, event: header_read, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.791180 7f869af78700 5 -- op tracker -- , seq: 3117, time: 2014-06-09 16:22:11.655152, event: throttled, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.791185 7f869af78700 5 -- op tracker -- , seq: 3117, time: 2014-06-09 16:22:11.655165, event: all_read, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.791191 7f869af78700 5 -- op tracker -- , seq: 3117, time: 2014-06-09 16:22:11.791167, event: dispatched, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.791196 7f869af78700 5 -- op tracker -- , seq: 3117, time: 2014-06-09 16:22:11.791195, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.791200 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1d5) v4 from osd.3 2014-06-09 16:22:11.791203 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd78380 2014-06-09 16:22:11.791207 7f869af78700 5 -- op tracker -- , seq: 3117, time: 2014-06-09 16:22:11.791205, event: started, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.791221 7f869af78700 5 -- op tracker -- , seq: 3117, time: 2014-06-09 16:22:11.791219, event: done, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.791230 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791232 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791245 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791247 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791248 7f869af78700 20 osd.26 243 _dispatch 0xcd77a40 pg_info(1 pgs e243:2.48d) v4 2014-06-09 16:22:11.791252 7f869af78700 5 -- op tracker -- , seq: 3118, time: 2014-06-09 16:22:11.650672, event: header_read, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.791258 7f869af78700 5 -- op tracker -- , seq: 3118, time: 2014-06-09 16:22:11.650672, event: throttled, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.791263 7f869af78700 5 -- op tracker -- , seq: 3118, time: 2014-06-09 16:22:11.650687, event: all_read, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.791269 7f869af78700 5 -- op tracker -- , seq: 3118, time: 2014-06-09 16:22:11.791245, event: dispatched, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.791275 7f869af78700 5 -- op tracker -- , seq: 3118, time: 2014-06-09 16:22:11.791273, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.791278 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.48d) v4 from osd.1 2014-06-09 16:22:11.791281 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd77a40 2014-06-09 16:22:11.791285 7f869af78700 5 -- op tracker -- , seq: 3118, time: 2014-06-09 16:22:11.791283, event: started, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.791298 7f869af78700 5 -- op tracker -- , seq: 3118, time: 2014-06-09 16:22:11.791296, event: done, op: pg_info(1 pgs e243:2.48d) 2014-06-09 16:22:11.791304 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791306 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791317 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791318 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791319 7f869af78700 20 osd.26 243 _dispatch 0x2fcaa8c0 pg_info(1 pgs e243:2.216) v4 2014-06-09 16:22:11.791323 7f869af78700 5 -- op tracker -- , seq: 3119, time: 2014-06-09 16:22:11.614517, event: header_read, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.791328 7f869af78700 5 -- op tracker -- , seq: 3119, time: 2014-06-09 16:22:11.614518, event: throttled, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.791333 7f869af78700 5 -- op tracker -- , seq: 3119, time: 2014-06-09 16:22:11.614526, event: all_read, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.791339 7f869af78700 5 -- op tracker -- , seq: 3119, time: 2014-06-09 16:22:11.791316, event: dispatched, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.791344 7f869af78700 5 -- op tracker -- , seq: 3119, time: 2014-06-09 16:22:11.791342, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.791347 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.216) v4 from osd.8 2014-06-09 16:22:11.791349 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaa8c0 2014-06-09 16:22:11.791353 7f869af78700 5 -- op tracker -- , seq: 3119, time: 2014-06-09 16:22:11.791352, event: started, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.791433 7f869af78700 5 -- op tracker -- , seq: 3119, time: 2014-06-09 16:22:11.791366, event: done, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.791447 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791449 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791463 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791464 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791465 7f869af78700 20 osd.26 243 _dispatch 0x30d9cc40 pg_info(1 pgs e243:3.6cb) v4 2014-06-09 16:22:11.791471 7f869af78700 5 -- op tracker -- , seq: 3120, time: 2014-06-09 16:22:11.657114, event: header_read, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.791478 7f869af78700 5 -- op tracker -- , seq: 3120, time: 2014-06-09 16:22:11.657114, event: throttled, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.791490 7f869af78700 5 -- op tracker -- , seq: 3120, time: 2014-06-09 16:22:11.657126, event: all_read, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.791496 7f869af78700 5 -- op tracker -- , seq: 3120, time: 2014-06-09 16:22:11.791462, event: dispatched, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.791505 7f869af78700 5 -- op tracker -- , seq: 3120, time: 2014-06-09 16:22:11.791501, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.791509 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6cb) v4 from osd.6 2014-06-09 16:22:11.791512 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9cc40 2014-06-09 16:22:11.791517 7f869af78700 5 -- op tracker -- , seq: 3120, time: 2014-06-09 16:22:11.791515, event: started, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.791533 7f869af78700 5 -- op tracker -- , seq: 3120, time: 2014-06-09 16:22:11.791531, event: done, op: pg_info(1 pgs e243:3.6cb) 2014-06-09 16:22:11.791541 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791543 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791554 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791556 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791557 7f869af78700 20 osd.26 243 _dispatch 0x2fc57340 pg_info(1 pgs e243:2.5c6) v4 2014-06-09 16:22:11.791563 7f869af78700 5 -- op tracker -- , seq: 3121, time: 2014-06-09 16:22:11.653202, event: header_read, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.791569 7f869af78700 5 -- op tracker -- , seq: 3121, time: 2014-06-09 16:22:11.653202, event: throttled, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.791576 7f869af78700 5 -- op tracker -- , seq: 3121, time: 2014-06-09 16:22:11.653214, event: all_read, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.791582 7f869af78700 5 -- op tracker -- , seq: 3121, time: 2014-06-09 16:22:11.791554, event: dispatched, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.791589 7f869af78700 5 -- op tracker -- , seq: 3121, time: 2014-06-09 16:22:11.791587, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.791594 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5c6) v4 from osd.5 2014-06-09 16:22:11.791597 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc57340 2014-06-09 16:22:11.791602 7f869af78700 5 -- op tracker -- , seq: 3121, time: 2014-06-09 16:22:11.791600, event: started, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.791625 7f869af78700 5 -- op tracker -- , seq: 3121, time: 2014-06-09 16:22:11.791622, event: done, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.791633 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791634 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791646 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791647 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791649 7f869af78700 20 osd.26 243 _dispatch 0x30a3ac40 pg_info(1 pgs e243:1.2c0) v4 2014-06-09 16:22:11.791654 7f869af78700 5 -- op tracker -- , seq: 3122, time: 2014-06-09 16:22:11.606499, event: header_read, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.791663 7f869af78700 5 -- op tracker -- , seq: 3122, time: 2014-06-09 16:22:11.606500, event: throttled, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.791670 7f869af78700 5 -- op tracker -- , seq: 3122, time: 2014-06-09 16:22:11.606511, event: all_read, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.791676 7f869af78700 5 -- op tracker -- , seq: 3122, time: 2014-06-09 16:22:11.791645, event: dispatched, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.791683 7f869af78700 5 -- op tracker -- , seq: 3122, time: 2014-06-09 16:22:11.791681, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.791687 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2c0) v4 from osd.2 2014-06-09 16:22:11.791690 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3ac40 2014-06-09 16:22:11.791695 7f869af78700 5 -- op tracker -- , seq: 3122, time: 2014-06-09 16:22:11.791693, event: started, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.791711 7f869af78700 5 -- op tracker -- , seq: 3122, time: 2014-06-09 16:22:11.791708, event: done, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.791718 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791720 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791731 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791733 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791734 7f869af78700 20 osd.26 243 _dispatch 0x2f038fc0 pg_info(1 pgs e243:1.6f) v4 2014-06-09 16:22:11.791739 7f869af78700 5 -- op tracker -- , seq: 3123, time: 2014-06-09 16:22:11.606102, event: header_read, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.791745 7f869af78700 5 -- op tracker -- , seq: 3123, time: 2014-06-09 16:22:11.606104, event: throttled, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.791757 7f869af78700 5 -- op tracker -- , seq: 3123, time: 2014-06-09 16:22:11.606115, event: all_read, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.791768 7f869af78700 5 -- op tracker -- , seq: 3123, time: 2014-06-09 16:22:11.791731, event: dispatched, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.791776 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.791779 7f869af78700 5 -- op tracker -- , seq: 3123, time: 2014-06-09 16:22:11.791777, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.791783 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6f) v4 from osd.4 2014-06-09 16:22:11.791786 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f038fc0 2014-06-09 16:22:11.791791 7f869af78700 5 -- op tracker -- , seq: 3123, time: 2014-06-09 16:22:11.791789, event: started, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.791799 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3466.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.791803 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.791808 7f869af78700 5 -- op tracker -- , seq: 3123, time: 2014-06-09 16:22:11.791805, event: done, op: pg_info(1 pgs e243:1.6f) 2014-06-09 16:22:11.791815 7f86a5610700 10 _do_op 0x2fcaf620 seq 3466 r = 0, finisher 0x3042a380 0 2014-06-09 16:22:11.791818 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791819 7f86a5610700 10 _finish_op 0x2fcaf620 seq 3466 osr(1.5cb 0x384dd70)/0x384dd70 2014-06-09 16:22:11.791820 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791827 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc5a0 (1 active) 2014-06-09 16:22:11.791830 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fdc20 (2 active) 2014-06-09 16:22:11.791832 7f86a5610700 5 _do_op 0x230b81e0 seq 3468 osr(1.54a 0x384daf0)/0x384daf0 start 2014-06-09 16:22:11.791832 7f868d75d700 10 op_submit_start 3516 2014-06-09 16:22:11.791834 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791835 7f86a5610700 10 _do_transaction on 0xc2bec00 2014-06-09 16:22:11.791835 7f868d75d700 5 queue_transactions (trailing journal) 3516 2014-06-09 16:22:11.791836 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791837 7f869af78700 20 osd.26 243 _dispatch 0xcd78540 pg_info(1 pgs e243:3.52a) v4 2014-06-09 16:22:11.791838 7f868d75d700 5 queue_op 0x2fcae360 seq 3516 osr(2.36c 0x2ed1170) 861 bytes (queue has 50 ops and 4689310 bytes) 2014-06-09 16:22:11.791840 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.791841 7f868d75d700 10 op_submit_finish 3516 2014-06-09 16:22:11.791842 7f869af78700 5 -- op tracker -- , seq: 3124, time: 2014-06-09 16:22:11.655443, event: header_read, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.791849 7f869af78700 5 -- op tracker -- , seq: 3124, time: 2014-06-09 16:22:11.655444, event: throttled, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.791847 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.791856 7f869af78700 5 -- op tracker -- , seq: 3124, time: 2014-06-09 16:22:11.655456, event: all_read, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.791863 7f869af78700 5 -- op tracker -- , seq: 3124, time: 2014-06-09 16:22:11.791833, event: dispatched, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.791864 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.791873 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.791875 7f869af78700 5 -- op tracker -- , seq: 3124, time: 2014-06-09 16:22:11.791873, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.791873 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.791880 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.52a) v4 from osd.3 2014-06-09 16:22:11.791882 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd78540 2014-06-09 16:22:11.791887 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3468.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.791884 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.791888 7f869af78700 5 -- op tracker -- , seq: 3124, time: 2014-06-09 16:22:11.791886, event: started, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.791891 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.791891 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.791895 7f86a5e11700 10 _do_op 0xc34ae40 seq 3467 r = 0, finisher 0x3042b8a0 0x30717f40 2014-06-09 16:22:11.791900 7f86a5e11700 10 _finish_op 0xc34ae40 seq 3467 osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.791898 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.791905 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4a20 (1 active) 2014-06-09 16:22:11.791904 7f869af78700 5 -- op tracker -- , seq: 3124, time: 2014-06-09 16:22:11.791901, event: done, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.791907 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb680 (2 active) 2014-06-09 16:22:11.791910 7f86a5e11700 5 _do_op 0x2fcaf5c0 seq 3469 osr(1.35c 0x384d2b0)/0x384d2b0 start 2014-06-09 16:22:11.791912 7f86a5e11700 10 _do_transaction on 0x39da3c0 2014-06-09 16:22:11.791911 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791915 7f86a5610700 10 _do_op 0x230b81e0 seq 3468 r = 0, finisher 0xbd7d8a0 0 2014-06-09 16:22:11.791919 7f86a5610700 10 _finish_op 0x230b81e0 seq 3468 osr(1.54a 0x384daf0)/0x384daf0 2014-06-09 16:22:11.791919 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.791921 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fdc20 (1 active) 2014-06-09 16:22:11.791915 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.791923 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4c60 (2 active) 2014-06-09 16:22:11.791925 7f86a5610700 5 _do_op 0x30d970e0 seq 3470 osr(2.349 0x2ed0e90)/0x2ed0e90 start 2014-06-09 16:22:11.791927 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.791928 7f86a5610700 10 _do_transaction on 0x2e63500 2014-06-09 16:22:11.791929 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.791931 7f869af78700 20 osd.26 243 _dispatch 0xcd79500 pg_info(1 pgs e243:3.761) v4 2014-06-09 16:22:11.791926 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.791934 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.791936 7f869af78700 5 -- op tracker -- , seq: 3125, time: 2014-06-09 16:22:11.653664, event: header_read, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.791935 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.791946 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.791943 7f869af78700 5 -- op tracker -- , seq: 3125, time: 2014-06-09 16:22:11.653664, event: throttled, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.791950 7f869af78700 5 -- op tracker -- , seq: 3125, time: 2014-06-09 16:22:11.653676, event: all_read, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.791956 7f869af78700 5 -- op tracker -- , seq: 3125, time: 2014-06-09 16:22:11.791927, event: dispatched, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.791952 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:0.759( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.791964 7f869af78700 5 -- op tracker -- , seq: 3125, time: 2014-06-09 16:22:11.791962, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.791968 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.761) v4 from osd.1 2014-06-09 16:22:11.791971 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd79500 2014-06-09 16:22:11.791967 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.791976 7f869af78700 5 -- op tracker -- , seq: 3125, time: 2014-06-09 16:22:11.791974, event: started, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.791980 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.791991 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.791984 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.1c9( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.791992 7f869af78700 5 -- op tracker -- , seq: 3125, time: 2014-06-09 16:22:11.791990, event: done, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.791999 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792001 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792011 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3469.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.792013 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792016 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.791997 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.792028 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792030 7f86a5e11700 10 _do_op 0x2fcaf5c0 seq 3469 r = 0, finisher 0x3042a0a0 0 2014-06-09 16:22:11.792034 7f86a5e11700 10 _finish_op 0x2fcaf5c0 seq 3469 osr(1.35c 0x384d2b0)/0x384d2b0 2014-06-09 16:22:11.792036 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb680 (1 active) 2014-06-09 16:22:11.792038 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383fb00 (2 active) 2014-06-09 16:22:11.792040 7f86a5e11700 5 _do_op 0x288f9080 seq 3471 osr(3.17 0x3ed03b0)/0x3ed03b0 start 2014-06-09 16:22:11.792033 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.792043 7f86a5e11700 10 _do_transaction on 0xbd8f140 2014-06-09 16:22:11.792048 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.792046 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.792054 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.792068 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.792063 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.792015 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792082 7f869af78700 20 osd.26 243 _dispatch 0x2fcab180 pg_info(1 pgs e243:3.218) v4 2014-06-09 16:22:11.792084 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3470.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.792089 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.792072 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.792092 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.792108 7f86a5610700 10 _do_op 0x30d970e0 seq 3470 r = 0, finisher 0x4eb8aa0 0 2014-06-09 16:22:11.792112 7f86a5610700 10 _finish_op 0x30d970e0 seq 3470 osr(2.349 0x2ed0e90)/0x2ed0e90 2014-06-09 16:22:11.792107 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.792114 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4c60 (1 active) 2014-06-09 16:22:11.792117 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4d80 (2 active) 2014-06-09 16:22:11.792119 7f86a5610700 5 _do_op 0xc2d4720 seq 3472 osr(2.482 0x2ed14f0)/0x2ed14f0 start 2014-06-09 16:22:11.792122 7f86a5610700 10 _do_transaction on 0x34e8000 2014-06-09 16:22:11.792117 7f868d75d700 20 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.792128 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.792125 7f868d75d700 20 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.792134 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792133 7f869af78700 5 -- op tracker -- , seq: 3126, time: 2014-06-09 16:22:11.614557, event: header_read, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.792137 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.792143 7f869af78700 5 -- op tracker -- , seq: 3126, time: 2014-06-09 16:22:11.614557, event: throttled, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.792150 7f869af78700 5 -- op tracker -- , seq: 3126, time: 2014-06-09 16:22:11.614565, event: all_read, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.792149 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.792157 7f869af78700 5 -- op tracker -- , seq: 3126, time: 2014-06-09 16:22:11.792013, event: dispatched, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.792157 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.792164 7f869af78700 5 -- op tracker -- , seq: 3126, time: 2014-06-09 16:22:11.792162, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.792167 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.792169 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.218) v4 from osd.8 2014-06-09 16:22:11.792172 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcab180 2014-06-09 16:22:11.792166 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.792182 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3471.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.792187 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.792182 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.792177 7f869af78700 5 -- op tracker -- , seq: 3126, time: 2014-06-09 16:22:11.792175, event: started, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.792191 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.792205 7f869af78700 5 -- op tracker -- , seq: 3126, time: 2014-06-09 16:22:11.792203, event: done, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.792205 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.792213 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792215 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792214 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.792222 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.792231 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792231 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792230 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.792233 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792235 7f869af78700 20 osd.26 243 _dispatch 0x30d9ca80 pg_info(1 pgs e243:0.66a) v4 2014-06-09 16:22:11.792240 7f869af78700 5 -- op tracker -- , seq: 3127, time: 2014-06-09 16:22:11.657162, event: header_read, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.792236 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.92( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.792247 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3472.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.792247 7f869af78700 5 -- op tracker -- , seq: 3127, time: 2014-06-09 16:22:11.657163, event: throttled, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.792252 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.792255 7f869af78700 5 -- op tracker -- , seq: 3127, time: 2014-06-09 16:22:11.657174, event: all_read, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.792263 7f86a5610700 10 _do_op 0xc2d4720 seq 3472 r = 0, finisher 0x3042b3a0 0 2014-06-09 16:22:11.792267 7f86a5610700 10 _finish_op 0xc2d4720 seq 3472 osr(2.482 0x2ed14f0)/0x2ed14f0 2014-06-09 16:22:11.792270 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4d80 (1 active) 2014-06-09 16:22:11.792272 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9238c0 (2 active) 2014-06-09 16:22:11.792274 7f86a5610700 5 _do_op 0xbe7cba0 seq 3473 osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 start 2014-06-09 16:22:11.792276 7f86a5610700 10 _do_transaction on 0xc0cdb00 2014-06-09 16:22:11.792282 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.792292 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792298 7f868d75d700 5 queue_transactions existing osr(1.92 0x384c410)/0x384c410 2014-06-09 16:22:11.792301 7f868d75d700 10 op_submit_start 3517 2014-06-09 16:22:11.792302 7f868d75d700 5 queue_transactions (trailing journal) 3517 2014-06-09 16:22:11.792304 7f868d75d700 5 queue_op 0xc64bda0 seq 3517 osr(1.92 0x384c410) 826 bytes (queue has 46 ops and 4328651 bytes) 2014-06-09 16:22:11.792261 7f869af78700 5 -- op tracker -- , seq: 3127, time: 2014-06-09 16:22:11.792231, event: dispatched, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.792307 7f868d75d700 10 op_submit_finish 3517 2014-06-09 16:22:11.792315 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.792312 7f869af78700 5 -- op tracker -- , seq: 3127, time: 2014-06-09 16:22:11.792309, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.792309 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.792320 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.66a) v4 from osd.6 2014-06-09 16:22:11.792323 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9ca80 2014-06-09 16:22:11.792323 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.792328 7f869af78700 5 -- op tracker -- , seq: 3127, time: 2014-06-09 16:22:11.792326, event: started, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.792331 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3473.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.792336 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.792331 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.792340 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.792348 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.792353 7f86a5610700 10 _do_op 0xbe7cba0 seq 3473 r = 0, finisher 0x30e18880 0 2014-06-09 16:22:11.792357 7f86a5610700 10 _finish_op 0xbe7cba0 seq 3473 osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 2014-06-09 16:22:11.792344 7f869af78700 5 -- op tracker -- , seq: 3127, time: 2014-06-09 16:22:11.792342, event: done, op: pg_info(1 pgs e243:0.66a) 2014-06-09 16:22:11.792360 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9238c0 (1 active) 2014-06-09 16:22:11.792362 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4a20 (2 active) 2014-06-09 16:22:11.792356 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.792363 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792364 7f86a5610700 5 _do_op 0x2fcae480 seq 3474 osr(3.1e 0x3ed0380)/0x3ed0380 start 2014-06-09 16:22:11.792365 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792367 7f86a5610700 10 _do_transaction on 0xc227800 2014-06-09 16:22:11.792372 7f86a5610700 15 _omap_setkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.792370 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.792378 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792379 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792381 7f869af78700 20 osd.26 243 _dispatch 0x308d9c00 pg_info(1 pgs e243:2.5c6) v4 2014-06-09 16:22:11.792380 7f868d75d700 20 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.792386 7f869af78700 5 -- op tracker -- , seq: 3128, time: 2014-06-09 16:22:11.653234, event: header_read, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.792387 7f868d75d700 20 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.792397 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792396 7f869af78700 5 -- op tracker -- , seq: 3128, time: 2014-06-09 16:22:11.653234, event: throttled, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.792402 7f869af78700 5 -- op tracker -- , seq: 3128, time: 2014-06-09 16:22:11.653249, event: all_read, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.792401 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=201 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.4c9( v 213'646 (0'0,213'646] local-les=243 n=124 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.792409 7f869af78700 5 -- op tracker -- , seq: 3128, time: 2014-06-09 16:22:11.792377, event: dispatched, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.792412 7f86a5610700 10 lookup_strip_header done cid meta oid 51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.792417 7f869af78700 5 -- op tracker -- , seq: 3128, time: 2014-06-09 16:22:11.792414, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.792415 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=201 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.4c9( v 213'646 (0'0,213'646] local-les=243 n=124 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.792426 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5c6) v4 from osd.5 2014-06-09 16:22:11.792428 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.792429 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308d9c00 2014-06-09 16:22:11.792430 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=201 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.895414 7 0.000326 2014-06-09 16:22:11.792435 7f869af78700 5 -- op tracker -- , seq: 3128, time: 2014-06-09 16:22:11.792433, event: started, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.792442 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=201 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.792456 7f869af78700 5 -- op tracker -- , seq: 3128, time: 2014-06-09 16:22:11.792453, event: done, op: pg_info(1 pgs e243:2.5c6) 2014-06-09 16:22:11.792459 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.792455 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=201 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.792463 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792466 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792474 7f86a5610700 15 _omap_rmkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.792467 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=201 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.792477 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792479 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792480 7f86a5610700 10 _omap_rmkeys meta/51682acc/pglog_3.1e/0//-1 r = 0 2014-06-09 16:22:11.792481 7f869af78700 20 osd.26 243 _dispatch 0x30da3a40 pg_info(1 pgs e243:0.654) v4 2014-06-09 16:22:11.792487 7f86a5610700 15 _omap_setkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.792479 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=243 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'646 -> 213'646 2014-06-09 16:22:11.792487 7f869af78700 5 -- op tracker -- , seq: 3129, time: 2014-06-09 16:22:11.606519, event: header_read, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.792493 7f869af78700 5 -- op tracker -- , seq: 3129, time: 2014-06-09 16:22:11.606519, event: throttled, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.792500 7f869af78700 5 -- op tracker -- , seq: 3129, time: 2014-06-09 16:22:11.606530, event: all_read, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.792502 7f86a5610700 15 _write 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 352256~516096 2014-06-09 16:22:11.792507 7f869af78700 5 -- op tracker -- , seq: 3129, time: 2014-06-09 16:22:11.792477, event: dispatched, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.792515 7f869af78700 5 -- op tracker -- , seq: 3129, time: 2014-06-09 16:22:11.792512, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.792519 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.654) v4 from osd.2 2014-06-09 16:22:11.792522 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da3a40 2014-06-09 16:22:11.792520 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=243 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.792527 7f869af78700 5 -- op tracker -- , seq: 3129, time: 2014-06-09 16:22:11.792525, event: started, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.792547 7f869af78700 5 -- op tracker -- , seq: 3129, time: 2014-06-09 16:22:11.792545, event: done, op: pg_info(1 pgs e243:0.654) 2014-06-09 16:22:11.792550 7f86a5610700 10 lookup_strip_header done cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.792554 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792556 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792562 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792565 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.792567 7f868d75d700 5 queue_transactions existing osr(3.4c9 0x3ed13d0)/0x3ed13d0 2014-06-09 16:22:11.792568 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792570 7f868d75d700 10 op_submit_start 3518 2014-06-09 16:22:11.792570 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792571 7f868d75d700 5 queue_transactions (trailing journal) 3518 2014-06-09 16:22:11.792571 7f869af78700 20 osd.26 243 _dispatch 0x2f038e00 pg_info(1 pgs e243:3.f0) v4 2014-06-09 16:22:11.792572 7f868d75d700 5 queue_op 0xc64bc20 seq 3518 osr(3.4c9 0x3ed13d0) 1335 bytes (queue has 46 ops and 4328651 bytes) 2014-06-09 16:22:11.792575 7f868d75d700 10 op_submit_finish 3518 2014-06-09 16:22:11.792577 7f869af78700 5 -- op tracker -- , seq: 3130, time: 2014-06-09 16:22:11.606184, event: header_read, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.792578 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.792587 7f869af78700 5 -- op tracker -- , seq: 3130, time: 2014-06-09 16:22:11.606184, event: throttled, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.792594 7f869af78700 5 -- op tracker -- , seq: 3130, time: 2014-06-09 16:22:11.606199, event: all_read, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.792591 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.792600 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.792600 7f869af78700 5 -- op tracker -- , seq: 3130, time: 2014-06-09 16:22:11.792568, event: dispatched, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.792608 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.792612 7f869af78700 5 -- op tracker -- , seq: 3130, time: 2014-06-09 16:22:11.792610, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.792620 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.f0) v4 from osd.4 2014-06-09 16:22:11.792616 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.792623 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f038e00 2014-06-09 16:22:11.792625 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.792628 7f869af78700 5 -- op tracker -- , seq: 3130, time: 2014-06-09 16:22:11.792626, event: started, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.792637 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.792645 7f869af78700 5 -- op tracker -- , seq: 3130, time: 2014-06-09 16:22:11.792642, event: done, op: pg_info(1 pgs e243:3.f0) 2014-06-09 16:22:11.792647 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.792654 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792657 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792655 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.792664 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792669 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792671 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792672 7f869af78700 20 osd.26 243 _dispatch 0x30c30700 pg_info(1 pgs e243:2.112) v4 2014-06-09 16:22:11.792669 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.792677 7f869af78700 5 -- op tracker -- , seq: 3131, time: 2014-06-09 16:22:11.656507, event: header_read, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.792681 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.792688 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.792690 7f869af78700 5 -- op tracker -- , seq: 3131, time: 2014-06-09 16:22:11.656508, event: throttled, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.792697 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.792700 7f869af78700 5 -- op tracker -- , seq: 3131, time: 2014-06-09 16:22:11.656522, event: all_read, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.792703 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.792710 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.792711 7f869af78700 5 -- op tracker -- , seq: 3131, time: 2014-06-09 16:22:11.792669, event: dispatched, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.792722 7f869af78700 5 -- op tracker -- , seq: 3131, time: 2014-06-09 16:22:11.792719, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.792726 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.112) v4 from osd.3 2014-06-09 16:22:11.792723 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:162 2014-06-09 16:22:11.792730 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c30700 2014-06-09 16:22:11.792732 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.792735 7f869af78700 5 -- op tracker -- , seq: 3131, time: 2014-06-09 16:22:11.792733, event: started, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.792740 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.792748 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792751 7f869af78700 5 -- op tracker -- , seq: 3131, time: 2014-06-09 16:22:11.792749, event: done, op: pg_info(1 pgs e243:2.112) 2014-06-09 16:22:11.792759 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792760 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792753 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.792765 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.792772 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792774 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792776 7f869af78700 20 osd.26 243 _dispatch 0xcd77880 pg_info(1 pgs e243:3.761) v4 2014-06-09 16:22:11.792775 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.792781 7f869af78700 5 -- op tracker -- , seq: 3132, time: 2014-06-09 16:22:11.653711, event: header_read, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.792783 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.792788 7f869af78700 5 -- op tracker -- , seq: 3132, time: 2014-06-09 16:22:11.653712, event: throttled, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.792794 7f869af78700 5 -- op tracker -- , seq: 3132, time: 2014-06-09 16:22:11.653726, event: all_read, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.792798 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.792801 7f869af78700 5 -- op tracker -- , seq: 3132, time: 2014-06-09 16:22:11.792772, event: dispatched, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.792817 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.792832 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792836 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.492( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.792808 7f869af78700 5 -- op tracker -- , seq: 3132, time: 2014-06-09 16:22:11.792806, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.792851 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.761) v4 from osd.1 2014-06-09 16:22:11.792855 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd77880 2014-06-09 16:22:11.792848 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.492( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.492( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.792860 7f868d75d700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.792861 7f869af78700 5 -- op tracker -- , seq: 3132, time: 2014-06-09 16:22:11.792859, event: started, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.792864 7f868d75d700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.792869 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.492( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] exit Started/Stray 1.060719 7 0.000179 2014-06-09 16:22:11.792877 7f869af78700 5 -- op tracker -- , seq: 3132, time: 2014-06-09 16:22:11.792875, event: done, op: pg_info(1 pgs e243:3.761) 2014-06-09 16:22:11.792890 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.492( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.792897 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792899 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792900 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.492( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.792916 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.792910 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.492( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.792918 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.792919 7f869af78700 20 osd.26 243 _dispatch 0x2fc94700 pg_info(1 pgs e243:2.216) v4 2014-06-09 16:22:11.792920 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.492( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.792928 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.492( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.792925 7f869af78700 5 -- op tracker -- , seq: 3133, time: 2014-06-09 16:22:11.614860, event: header_read, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.792959 7f869af78700 5 -- op tracker -- , seq: 3133, time: 2014-06-09 16:22:11.614860, event: throttled, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.793000 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.792968 7f869af78700 5 -- op tracker -- , seq: 3133, time: 2014-06-09 16:22:11.614867, event: all_read, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.793006 7f868d75d700 5 queue_transactions existing osr(0.492 0x2bd7410)/0x2bd7410 2014-06-09 16:22:11.793010 7f868d75d700 10 op_submit_start 3519 2014-06-09 16:22:11.793011 7f868d75d700 5 queue_transactions (trailing journal) 3519 2014-06-09 16:22:11.793012 7f868d75d700 5 queue_op 0x14dd6300 seq 3519 osr(0.492 0x2bd7410) 2099 bytes (queue has 47 ops and 4330750 bytes) 2014-06-09 16:22:11.793015 7f868d75d700 10 op_submit_finish 3519 2014-06-09 16:22:11.793013 7f869af78700 5 -- op tracker -- , seq: 3133, time: 2014-06-09 16:22:11.792916, event: dispatched, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.793018 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.793025 7f869af78700 5 -- op tracker -- , seq: 3133, time: 2014-06-09 16:22:11.793023, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.793031 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.216) v4 from osd.8 2014-06-09 16:22:11.793034 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc94700 2014-06-09 16:22:11.793032 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.793040 7f869af78700 5 -- op tracker -- , seq: 3133, time: 2014-06-09 16:22:11.793038, event: started, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.793041 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.793049 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.793076 7f868d75d700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.793057 7f869af78700 5 -- op tracker -- , seq: 3133, time: 2014-06-09 16:22:11.793054, event: done, op: pg_info(1 pgs e243:2.216) 2014-06-09 16:22:11.793093 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:169 2014-06-09 16:22:11.793108 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.793090 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.793113 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 5 2014-06-09 16:22:11.793122 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.793125 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd5 2014-06-09 16:22:11.793136 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.793137 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.793139 7f869af78700 20 osd.26 243 _dispatch 0x30c916c0 pg_info(1 pgs e243:3.470) v4 2014-06-09 16:22:11.793145 7f869af78700 5 -- op tracker -- , seq: 3134, time: 2014-06-09 16:22:11.746222, event: header_read, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.793138 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.793165 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.793177 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'709 (0'0,214'709] local-les=206 n=129 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.5: log((214'709,214'710], crt=0'0) missing(0) 2014-06-09 16:22:11.793189 7f868d75d700 10 merge_log log((214'709,214'710], crt=0'0) from osd.5 into log((0'0,214'709], crt=0'0) 2014-06-09 16:22:11.793193 7f868d75d700 10 merge_log extending head to 214'710 2014-06-09 16:22:11.793195 7f868d75d700 20 ? 214'710 (0'0) modify 357c0d42/rbd_data.11696b8b4567.0000000000001696/head//3 by client.4553.0:33729 2014-06-09 10:19:21.025451 2014-06-09 16:22:11.793204 7f868d75d700 20 merge_log 214'710 (0'0) modify 357c0d42/rbd_data.11696b8b4567.0000000000001696/head//3 by client.4553.0:33729 2014-06-09 10:19:21.025451 2014-06-09 16:22:11.793552 7f868d75d700 10 merge_log result log((0'0,214'710], crt=214'710) missing(1) changed=1 2014-06-09 16:22:11.793558 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 peering m=1] peer osd.5 now 3.542( v 214'710 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) missing(0) 2014-06-09 16:22:11.793579 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.537332 3 0.000216 2014-06-09 16:22:11.793594 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.793605 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 peering m=1] state: osd.5 has no missing, identical log 2014-06-09 16:22:11.793621 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000027 0 0.000000 2014-06-09 16:22:11.793635 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.793644 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.329007 0 0.000000 2014-06-09 16:22:11.793657 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.793667 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.793681 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.793691 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.793700 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.793829 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.793841 7f868d75d700 10 osd.26 243 queue_for_recovery queued pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.793855 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 3.542( v 214'710 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) 2014-06-09 16:22:11.793869 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 is up to date, queueing in pending_activators 2014-06-09 16:22:11.793886 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 3.542( v 214'710 (0'0,214'710] local-les=206 n=130 ec=118 les/c 206/206 242/242/242) uptodate 2014-06-09 16:22:11.793901 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.793914 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 357c0d42/rbd_data.11696b8b4567.0000000000001696/head//3 214'710 also missing on osd.26 2014-06-09 16:22:11.793927 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {357c0d42/rbd_data.11696b8b4567.0000000000001696/head//3=214'710} 2014-06-09 16:22:11.793936 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 5 as a source 2014-06-09 16:22:11.793947 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 357c0d42/rbd_data.11696b8b4567.0000000000001696/head//3 214'710 is on osd.5 2014-06-09 16:22:11.793960 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {357c0d42/rbd_data.11696b8b4567.0000000000001696/head//3=214'710} 2014-06-09 16:22:11.793973 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.793984 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 206 2014-06-09 16:22:11.793996 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 5 2014-06-09 16:22:11.794005 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.794013 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.794055 7f868d75d700 10 write_log with: dirty_to: 0'0, dirty_from: 214'709, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.794092 7f868d75d700 5 queue_transactions existing osr(3.542 0x3ed15e0)/0x3ed15e0 2014-06-09 16:22:11.794096 7f868d75d700 10 op_submit_start 3520 2014-06-09 16:22:11.794097 7f868d75d700 5 queue_transactions (trailing journal) 3520 2014-06-09 16:22:11.794099 7f868d75d700 5 queue_op 0x308cad20 seq 3520 osr(3.542 0x3ed15e0) 1726 bytes (queue has 48 ops and 4332476 bytes) 2014-06-09 16:22:11.794102 7f868d75d700 10 op_submit_finish 3520 2014-06-09 16:22:11.794105 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:243, info:3.443( v 213'673 (0'0,213'673] local-les=191 n=125 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.794122 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.794134 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.794141 7f868d75d700 20 osd.26 243 Sending info 3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) to shard 5 2014-06-09 16:22:11.794149 7f868d75d700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.794190 7f868d75d700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15359b40 (19 active) 2014-06-09 16:22:11.794199 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.794211 7f8689f56700 10 op_submit_start 3521 2014-06-09 16:22:11.794213 7f8689f56700 5 queue_transactions (trailing journal) 3521 2014-06-09 16:22:11.794214 7f8689f56700 5 queue_op 0x1e08b560 seq 3521 osr(1.755 0x2ed0090) 2009 bytes (queue has 50 ops and 4852865 bytes) 2014-06-09 16:22:11.794217 7f8689f56700 10 op_submit_finish 3521 2014-06-09 16:22:11.794218 7f868b759700 10 op_submit_start 3522 2014-06-09 16:22:11.794221 7f868b759700 5 queue_transactions (trailing journal) 3522 2014-06-09 16:22:11.794224 7f868b759700 5 queue_op 0xbf7f3e0 seq 3522 osr(3.60 0x3ed02e0) 518380 bytes (queue has 50 ops and 4852865 bytes) 2014-06-09 16:22:11.794227 7f868b759700 10 op_submit_finish 3522 2014-06-09 16:22:11.794220 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.794237 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.794248 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.794234 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] eval_repop repgather(0x30d1740 243'644 rep_tid=11 committed?=0 applied?=0 lock=0 op=osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.794262 7f868b759700 10 osd.26 243 dequeue_op 0x257c5500 finish 2014-06-09 16:22:11.794258 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.794269 7f868b759700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.794269 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.794279 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.794293 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15359d20 (20 active) 2014-06-09 16:22:11.794299 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.92( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.92( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.794340 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.794345 7f868b759700 5 queue_transactions existing osr(1.92 0x384c410)/0x384c410 2014-06-09 16:22:11.794348 7f868b759700 2 waiting 51 > 50 ops || 4853691 > 104857600 2014-06-09 16:22:11.794341 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1008 2014-06-09 16:22:11.794355 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.794361 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.794367 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.794356 7f868e75f700 10 osd.26 243 dequeue_op 0x257c5400 prio 63 cost 516096 latency 1.037896 osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] 2014-06-09 16:22:11.794377 7f868a757700 2 waiting 51 > 50 ops || 4854768 > 104857600 2014-06-09 16:22:11.794372 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.691( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.794383 7f8692767700 2 waiting 51 > 50 ops || 4854200 > 104857600 2014-06-09 16:22:11.794392 7f8690763700 2 waiting 51 > 50 ops || 4853693 > 104857600 2014-06-09 16:22:11.794397 7f868ef60700 2 waiting 51 > 50 ops || 4854866 > 104857600 2014-06-09 16:22:11.794403 7f8689755700 2 waiting 51 > 50 ops || 4854808 > 104857600 2014-06-09 16:22:11.794404 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.794408 7f8689f56700 5 queue_transactions existing osr(1.691 0x384de30)/0x384de30 2014-06-09 16:22:11.794411 7f8689f56700 2 waiting 51 > 50 ops || 4853693 > 104857600 2014-06-09 16:22:11.794416 7f868ff62700 2 waiting 51 > 50 ops || 4854200 > 104857600 2014-06-09 16:22:11.794424 7f868f761700 2 waiting 51 > 50 ops || 4854294 > 104857600 2014-06-09 16:22:11.794433 7f868cf5c700 2 waiting 51 > 50 ops || 4853693 > 104857600 2014-06-09 16:22:11.794439 7f868bf5a700 2 waiting 51 > 50 ops || 4853693 > 104857600 2014-06-09 16:22:11.794444 7f868df5e700 2 waiting 51 > 50 ops || 4916703 > 104857600 2014-06-09 16:22:11.794450 7f8692f68700 2 waiting 51 > 50 ops || 4853918 > 104857600 2014-06-09 16:22:11.793152 7f869af78700 5 -- op tracker -- , seq: 3134, time: 2014-06-09 16:22:11.746223, event: throttled, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.794515 7f869af78700 5 -- op tracker -- , seq: 3134, time: 2014-06-09 16:22:11.746238, event: all_read, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.794524 7f869af78700 5 -- op tracker -- , seq: 3134, time: 2014-06-09 16:22:11.793135, event: dispatched, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.794532 7f869af78700 5 -- op tracker -- , seq: 3134, time: 2014-06-09 16:22:11.794530, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.794538 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.470) v4 from osd.6 2014-06-09 16:22:11.794541 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c916c0 2014-06-09 16:22:11.794546 7f868e75f700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.794373, event: reached_pg, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.794558 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.794574 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] handle_message: 0x257c5400 2014-06-09 16:22:11.794585 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_op osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.794610 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] get_object_context: found obc in cache: 0x305c6b00 2014-06-09 16:22:11.794621 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] get_object_context: 0x305c6b00 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 rwstate(write n=4 w=0) oi: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'644 client.4572.0:30635 wrlock_by=unknown.0.0:0 dirty s 3313664 uv644) ssc: 0x2e8775e0 snapset: 0=[]:[]+head 2014-06-09 16:22:11.794637 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] find_object_context 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 @head oi=5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'644 client.4572.0:30635 wrlock_by=unknown.0.0:0 dirty s 3313664 uv644) 2014-06-09 16:22:11.794660 7f868e75f700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.794655, event: started, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.794667 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] execute_ctx 0x307df800 2014-06-09 16:22:11.794683 7f868e75f700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.794678, event: started, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.794689 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] ov 243'644 av 243'645 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.794702 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 2014-06-09 16:22:11.794726 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.794739 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op write 3313664~516096 2014-06-09 16:22:11.794758 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op_effects on session 0xd229400 2014-06-09 16:22:11.794770 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 snapset=0x2e877618 snapc=0=[] 2014-06-09 16:22:11.794784 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.794795 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] finish_ctx 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 0x307df800 op modify 2014-06-09 16:22:11.794808 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] set mtime to 2014-06-09 10:20:27.059734 2014-06-09 16:22:11.794832 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] final snapset 0=[]:[]+head in 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.794851 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] zeroing write result code 0 2014-06-09 16:22:11.794863 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] new_repop rep_tid 13 on osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.794880 7f868e75f700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] issue_repop rep_tid 13 o 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.794906 7f868e75f700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.794900, event: waiting for subops from 7, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.794926 7f868e75f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.794950 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'644 (0'0,243'644] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] append_log log((0'0,243'644], crt=0'0) [243'645 (243'644) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30636 2014-06-09 10:20:27.059734] 2014-06-09 16:22:11.794971 7f869af78700 5 -- op tracker -- , seq: 3134, time: 2014-06-09 16:22:11.794544, event: started, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.794974 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] add_log_entry 243'645 (243'644) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30636 2014-06-09 10:20:27.059734 2014-06-09 16:22:11.794991 7f869af78700 5 -- op tracker -- , seq: 3134, time: 2014-06-09 16:22:11.794989, event: done, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.794993 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] append_log adding 1 keys 2014-06-09 16:22:11.795001 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795003 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795021 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795022 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795021 7f868e75f700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'645, trimmed: 2014-06-09 16:22:11.795024 7f869af78700 20 osd.26 243 _dispatch 0x30c00a80 pg_info(1 pgs e243:0.4ba) v4 2014-06-09 16:22:11.795029 7f869af78700 5 -- op tracker -- , seq: 3135, time: 2014-06-09 16:22:11.660102, event: header_read, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.795036 7f869af78700 5 -- op tracker -- , seq: 3135, time: 2014-06-09 16:22:11.660103, event: throttled, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.795039 7f868e75f700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.795042 7f868e75f700 2 waiting 51 > 50 ops || 5371245 > 104857600 2014-06-09 16:22:11.795041 7f869af78700 5 -- op tracker -- , seq: 3135, time: 2014-06-09 16:22:11.660115, event: all_read, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.795048 7f869af78700 5 -- op tracker -- , seq: 3135, time: 2014-06-09 16:22:11.795020, event: dispatched, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.795054 7f869af78700 5 -- op tracker -- , seq: 3135, time: 2014-06-09 16:22:11.795052, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.795058 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4ba) v4 from osd.5 2014-06-09 16:22:11.795061 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c00a80 2014-06-09 16:22:11.795066 7f869af78700 5 -- op tracker -- , seq: 3135, time: 2014-06-09 16:22:11.795064, event: started, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.795080 7f869af78700 5 -- op tracker -- , seq: 3135, time: 2014-06-09 16:22:11.795078, event: done, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.795087 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795099 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795100 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795101 7f869af78700 20 osd.26 243 _dispatch 0x30da3880 pg_info(1 pgs e243:1.2c0) v4 2014-06-09 16:22:11.795106 7f869af78700 5 -- op tracker -- , seq: 3136, time: 2014-06-09 16:22:11.606552, event: header_read, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.795111 7f869af78700 5 -- op tracker -- , seq: 3136, time: 2014-06-09 16:22:11.606552, event: throttled, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.795116 7f869af78700 5 -- op tracker -- , seq: 3136, time: 2014-06-09 16:22:11.606560, event: all_read, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.795121 7f869af78700 5 -- op tracker -- , seq: 3136, time: 2014-06-09 16:22:11.795099, event: dispatched, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.795127 7f869af78700 5 -- op tracker -- , seq: 3136, time: 2014-06-09 16:22:11.795125, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.795130 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2c0) v4 from osd.2 2014-06-09 16:22:11.795133 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da3880 2014-06-09 16:22:11.795137 7f869af78700 5 -- op tracker -- , seq: 3136, time: 2014-06-09 16:22:11.795135, event: started, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.795149 7f869af78700 5 -- op tracker -- , seq: 3136, time: 2014-06-09 16:22:11.795147, event: done, op: pg_info(1 pgs e243:1.2c0) 2014-06-09 16:22:11.795155 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795156 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795166 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795168 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795169 7f869af78700 20 osd.26 243 _dispatch 0x2fcaa000 pg_info(1 pgs e243:0.9d) v4 2014-06-09 16:22:11.795173 7f869af78700 5 -- op tracker -- , seq: 3137, time: 2014-06-09 16:22:11.611560, event: header_read, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.795178 7f869af78700 5 -- op tracker -- , seq: 3137, time: 2014-06-09 16:22:11.611561, event: throttled, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.795183 7f869af78700 5 -- op tracker -- , seq: 3137, time: 2014-06-09 16:22:11.611580, event: all_read, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.795188 7f869af78700 5 -- op tracker -- , seq: 3137, time: 2014-06-09 16:22:11.795166, event: dispatched, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.795194 7f869af78700 5 -- op tracker -- , seq: 3137, time: 2014-06-09 16:22:11.795192, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.795197 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.9d) v4 from osd.4 2014-06-09 16:22:11.795200 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaa000 2014-06-09 16:22:11.795204 7f869af78700 5 -- op tracker -- , seq: 3137, time: 2014-06-09 16:22:11.795202, event: started, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.795218 7f869af78700 5 -- op tracker -- , seq: 3137, time: 2014-06-09 16:22:11.795216, event: done, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.795225 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795226 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795236 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795238 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795239 7f869af78700 20 osd.26 243 _dispatch 0x30c30540 pg_info(1 pgs e243:0.765) v4 2014-06-09 16:22:11.795243 7f869af78700 5 -- op tracker -- , seq: 3138, time: 2014-06-09 16:22:11.656552, event: header_read, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.795249 7f869af78700 5 -- op tracker -- , seq: 3138, time: 2014-06-09 16:22:11.656552, event: throttled, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.795255 7f869af78700 5 -- op tracker -- , seq: 3138, time: 2014-06-09 16:22:11.656565, event: all_read, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.795260 7f869af78700 5 -- op tracker -- , seq: 3138, time: 2014-06-09 16:22:11.795236, event: dispatched, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.795266 7f869af78700 5 -- op tracker -- , seq: 3138, time: 2014-06-09 16:22:11.795265, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.795270 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.765) v4 from osd.3 2014-06-09 16:22:11.795273 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c30540 2014-06-09 16:22:11.795277 7f869af78700 5 -- op tracker -- , seq: 3138, time: 2014-06-09 16:22:11.795275, event: started, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.795290 7f869af78700 5 -- op tracker -- , seq: 3138, time: 2014-06-09 16:22:11.795288, event: done, op: pg_info(1 pgs e243:0.765) 2014-06-09 16:22:11.795296 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795292 7f86a5610700 10 _generic_write 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 352256~516096 = 0 2014-06-09 16:22:11.795297 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795307 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795308 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795309 7f869af78700 20 osd.26 243 _dispatch 0x2fc56380 pg_info(1 pgs e243:3.124) v4 2014-06-09 16:22:11.795313 7f869af78700 5 -- op tracker -- , seq: 3139, time: 2014-06-09 16:22:11.657452, event: header_read, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795319 7f869af78700 5 -- op tracker -- , seq: 3139, time: 2014-06-09 16:22:11.657453, event: throttled, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795324 7f869af78700 5 -- op tracker -- , seq: 3139, time: 2014-06-09 16:22:11.657466, event: all_read, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795326 7f86a5610700 15 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.795329 7f869af78700 5 -- op tracker -- , seq: 3139, time: 2014-06-09 16:22:11.795307, event: dispatched, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795335 7f869af78700 5 -- op tracker -- , seq: 3139, time: 2014-06-09 16:22:11.795333, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795338 7f86a5610700 10 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.795338 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.124) v4 from osd.1 2014-06-09 16:22:11.795341 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc56380 2014-06-09 16:22:11.795344 7f86a5610700 15 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.795345 7f869af78700 5 -- op tracker -- , seq: 3139, time: 2014-06-09 16:22:11.795343, event: started, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795352 7f86a5610700 10 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.795358 7f869af78700 5 -- op tracker -- , seq: 3139, time: 2014-06-09 16:22:11.795356, event: done, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795364 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795365 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795376 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795377 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795378 7f869af78700 20 osd.26 243 _dispatch 0x2eef0000 pg_info(1 pgs e243:3.218) v4 2014-06-09 16:22:11.795382 7f869af78700 5 -- op tracker -- , seq: 3140, time: 2014-06-09 16:22:11.615061, event: header_read, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.795388 7f869af78700 5 -- op tracker -- , seq: 3140, time: 2014-06-09 16:22:11.615061, event: throttled, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.795393 7f869af78700 5 -- op tracker -- , seq: 3140, time: 2014-06-09 16:22:11.615077, event: all_read, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.795399 7f869af78700 5 -- op tracker -- , seq: 3140, time: 2014-06-09 16:22:11.795376, event: dispatched, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.795405 7f869af78700 5 -- op tracker -- , seq: 3140, time: 2014-06-09 16:22:11.795403, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.795409 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.218) v4 from osd.8 2014-06-09 16:22:11.795412 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2eef0000 2014-06-09 16:22:11.795417 7f869af78700 5 -- op tracker -- , seq: 3140, time: 2014-06-09 16:22:11.795415, event: started, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.795432 7f869af78700 5 -- op tracker -- , seq: 3140, time: 2014-06-09 16:22:11.795430, event: done, op: pg_info(1 pgs e243:3.218) 2014-06-09 16:22:11.795440 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795442 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795455 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795453 7f86a5610700 10 cid: oid: 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 not skipping op, *spos 3474.0.0 2014-06-09 16:22:11.795457 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795457 7f86a5610700 10 > header.spos 3467.0.0 2014-06-09 16:22:11.795458 7f869af78700 20 osd.26 243 _dispatch 0xcd77500 pg_info(1 pgs e243:3.577) v4 2014-06-09 16:22:11.795463 7f869af78700 5 -- op tracker -- , seq: 3141, time: 2014-06-09 16:22:11.746256, event: header_read, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.795469 7f869af78700 5 -- op tracker -- , seq: 3141, time: 2014-06-09 16:22:11.746256, event: throttled, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.795475 7f869af78700 5 -- op tracker -- , seq: 3141, time: 2014-06-09 16:22:11.746268, event: all_read, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.795481 7f869af78700 5 -- op tracker -- , seq: 3141, time: 2014-06-09 16:22:11.795455, event: dispatched, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.795483 7f86a5610700 20 set_header setting 30153 cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 parent seq 0 2014-06-09 16:22:11.795488 7f869af78700 5 -- op tracker -- , seq: 3141, time: 2014-06-09 16:22:11.795486, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.795492 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.577) v4 from osd.6 2014-06-09 16:22:11.795495 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd77500 2014-06-09 16:22:11.795499 7f869af78700 5 -- op tracker -- , seq: 3141, time: 2014-06-09 16:22:11.795498, event: started, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.795515 7f869af78700 5 -- op tracker -- , seq: 3141, time: 2014-06-09 16:22:11.795513, event: done, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.795522 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795524 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795536 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795537 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795538 7f869af78700 20 osd.26 243 _dispatch 0x2fc561c0 pg_info(1 pgs e243:0.6c0) v4 2014-06-09 16:22:11.795543 7f869af78700 5 -- op tracker -- , seq: 3142, time: 2014-06-09 16:22:11.660126, event: header_read, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.795545 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3474.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.795549 7f869af78700 5 -- op tracker -- , seq: 3142, time: 2014-06-09 16:22:11.660127, event: throttled, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.795551 7f86a5610700 10 cid: oid: 51682acc/pglog_3.1e/0//-1 skipping op, spos 3474.0.0 <= header.spos 150326.0.0 2014-06-09 16:22:11.795555 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.795556 7f869af78700 5 -- op tracker -- , seq: 3142, time: 2014-06-09 16:22:11.660143, event: all_read, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.795561 7f869af78700 5 -- op tracker -- , seq: 3142, time: 2014-06-09 16:22:11.795535, event: dispatched, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.795567 7f869af78700 5 -- op tracker -- , seq: 3142, time: 2014-06-09 16:22:11.795565, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.795571 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6c0) v4 from osd.5 2014-06-09 16:22:11.795574 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc561c0 2014-06-09 16:22:11.795578 7f869af78700 5 -- op tracker -- , seq: 3142, time: 2014-06-09 16:22:11.795576, event: started, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.795593 7f869af78700 5 -- op tracker -- , seq: 3142, time: 2014-06-09 16:22:11.795591, event: done, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.795602 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795604 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795612 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795630 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795632 7f869af78700 20 osd.26 243 _dispatch 0x2fc94fc0 pg_info(1 pgs e243:3.7bc) v4 2014-06-09 16:22:11.795637 7f869af78700 5 -- op tracker -- , seq: 3143, time: 2014-06-09 16:22:11.607417, event: header_read, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.795644 7f869af78700 5 -- op tracker -- , seq: 3143, time: 2014-06-09 16:22:11.607418, event: throttled, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.795650 7f869af78700 5 -- op tracker -- , seq: 3143, time: 2014-06-09 16:22:11.607428, event: all_read, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.795657 7f869af78700 5 -- op tracker -- , seq: 3143, time: 2014-06-09 16:22:11.795612, event: dispatched, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.795666 7f869af78700 5 -- op tracker -- , seq: 3143, time: 2014-06-09 16:22:11.795664, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.795671 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7bc) v4 from osd.2 2014-06-09 16:22:11.795675 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc94fc0 2014-06-09 16:22:11.795680 7f869af78700 5 -- op tracker -- , seq: 3143, time: 2014-06-09 16:22:11.795678, event: started, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.795698 7f869af78700 5 -- op tracker -- , seq: 3143, time: 2014-06-09 16:22:11.795696, event: done, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.795707 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795708 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795721 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795723 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795725 7f869af78700 20 osd.26 243 _dispatch 0x2fcaae00 pg_info(1 pgs e243:1.c9) v4 2014-06-09 16:22:11.795729 7f869af78700 5 -- op tracker -- , seq: 3144, time: 2014-06-09 16:22:11.611953, event: header_read, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.795736 7f869af78700 5 -- op tracker -- , seq: 3144, time: 2014-06-09 16:22:11.611954, event: throttled, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.795743 7f869af78700 5 -- op tracker -- , seq: 3144, time: 2014-06-09 16:22:11.611965, event: all_read, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.795753 7f869af78700 5 -- op tracker -- , seq: 3144, time: 2014-06-09 16:22:11.795721, event: dispatched, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.795761 7f869af78700 5 -- op tracker -- , seq: 3144, time: 2014-06-09 16:22:11.795759, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.795766 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.c9) v4 from osd.4 2014-06-09 16:22:11.795769 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaae00 2014-06-09 16:22:11.795774 7f869af78700 5 -- op tracker -- , seq: 3144, time: 2014-06-09 16:22:11.795772, event: started, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.795789 7f869af78700 5 -- op tracker -- , seq: 3144, time: 2014-06-09 16:22:11.795787, event: done, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.795797 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795799 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795812 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795813 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795815 7f869af78700 20 osd.26 243 _dispatch 0x30c30380 pg_info(1 pgs e243:3.1d5) v4 2014-06-09 16:22:11.795820 7f869af78700 5 -- op tracker -- , seq: 3145, time: 2014-06-09 16:22:11.656595, event: header_read, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.795833 7f869af78700 5 -- op tracker -- , seq: 3145, time: 2014-06-09 16:22:11.656596, event: throttled, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.795839 7f869af78700 5 -- op tracker -- , seq: 3145, time: 2014-06-09 16:22:11.656608, event: all_read, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.795848 7f869af78700 5 -- op tracker -- , seq: 3145, time: 2014-06-09 16:22:11.795811, event: dispatched, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.795856 7f869af78700 5 -- op tracker -- , seq: 3145, time: 2014-06-09 16:22:11.795853, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.795860 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1d5) v4 from osd.3 2014-06-09 16:22:11.795863 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c30380 2014-06-09 16:22:11.795868 7f869af78700 5 -- op tracker -- , seq: 3145, time: 2014-06-09 16:22:11.795866, event: started, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.795886 7f869af78700 5 -- op tracker -- , seq: 3145, time: 2014-06-09 16:22:11.795884, event: done, op: pg_info(1 pgs e243:3.1d5) 2014-06-09 16:22:11.795894 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795896 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795908 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795910 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.795911 7f869af78700 20 osd.26 243 _dispatch 0xcd776c0 pg_info(1 pgs e243:3.124) v4 2014-06-09 16:22:11.795916 7f869af78700 5 -- op tracker -- , seq: 3146, time: 2014-06-09 16:22:11.657498, event: header_read, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795923 7f869af78700 5 -- op tracker -- , seq: 3146, time: 2014-06-09 16:22:11.657499, event: throttled, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795935 7f869af78700 5 -- op tracker -- , seq: 3146, time: 2014-06-09 16:22:11.657512, event: all_read, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795942 7f869af78700 5 -- op tracker -- , seq: 3146, time: 2014-06-09 16:22:11.795908, event: dispatched, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795949 7f869af78700 5 -- op tracker -- , seq: 3146, time: 2014-06-09 16:22:11.795947, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795956 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.124) v4 from osd.1 2014-06-09 16:22:11.795959 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd776c0 2014-06-09 16:22:11.795964 7f869af78700 5 -- op tracker -- , seq: 3146, time: 2014-06-09 16:22:11.795962, event: started, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795983 7f869af78700 5 -- op tracker -- , seq: 3146, time: 2014-06-09 16:22:11.795980, event: done, op: pg_info(1 pgs e243:3.124) 2014-06-09 16:22:11.795992 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.795994 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796007 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796009 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796010 7f869af78700 20 osd.26 243 _dispatch 0x3130bc00 pg_info(1 pgs e243:2.26d) v4 2014-06-09 16:22:11.796016 7f869af78700 5 -- op tracker -- , seq: 3147, time: 2014-06-09 16:22:11.618978, event: header_read, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796022 7f869af78700 5 -- op tracker -- , seq: 3147, time: 2014-06-09 16:22:11.618979, event: throttled, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796034 7f869af78700 5 -- op tracker -- , seq: 3147, time: 2014-06-09 16:22:11.618991, event: all_read, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796041 7f869af78700 5 -- op tracker -- , seq: 3147, time: 2014-06-09 16:22:11.796007, event: dispatched, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796051 7f869af78700 5 -- op tracker -- , seq: 3147, time: 2014-06-09 16:22:11.796049, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796056 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.26d) v4 from osd.8 2014-06-09 16:22:11.796059 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130bc00 2014-06-09 16:22:11.796064 7f869af78700 5 -- op tracker -- , seq: 3147, time: 2014-06-09 16:22:11.796062, event: started, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796080 7f869af78700 5 -- op tracker -- , seq: 3147, time: 2014-06-09 16:22:11.796078, event: done, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796093 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796095 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796108 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796110 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796111 7f869af78700 20 osd.26 243 _dispatch 0x31ac3dc0 pg_info(1 pgs e243:2.213) v4 2014-06-09 16:22:11.796116 7f869af78700 5 -- op tracker -- , seq: 3148, time: 2014-06-09 16:22:11.747557, event: header_read, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.796123 7f869af78700 5 -- op tracker -- , seq: 3148, time: 2014-06-09 16:22:11.747558, event: throttled, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.796132 7f869af78700 5 -- op tracker -- , seq: 3148, time: 2014-06-09 16:22:11.747572, event: all_read, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.796139 7f869af78700 5 -- op tracker -- , seq: 3148, time: 2014-06-09 16:22:11.796108, event: dispatched, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.796147 7f869af78700 5 -- op tracker -- , seq: 3148, time: 2014-06-09 16:22:11.796144, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.796153 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.213) v4 from osd.6 2014-06-09 16:22:11.796156 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31ac3dc0 2014-06-09 16:22:11.796161 7f869af78700 5 -- op tracker -- , seq: 3148, time: 2014-06-09 16:22:11.796159, event: started, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.796182 7f869af78700 5 -- op tracker -- , seq: 3148, time: 2014-06-09 16:22:11.796180, event: done, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.796190 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796192 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796205 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796206 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796207 7f869af78700 20 osd.26 243 _dispatch 0x30c01180 pg_info(1 pgs e243:0.4ba) v4 2014-06-09 16:22:11.796212 7f869af78700 5 -- op tracker -- , seq: 3149, time: 2014-06-09 16:22:11.660158, event: header_read, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.796219 7f869af78700 5 -- op tracker -- , seq: 3149, time: 2014-06-09 16:22:11.660159, event: throttled, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.796226 7f869af78700 5 -- op tracker -- , seq: 3149, time: 2014-06-09 16:22:11.660169, event: all_read, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.796232 7f869af78700 5 -- op tracker -- , seq: 3149, time: 2014-06-09 16:22:11.796204, event: dispatched, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.796241 7f869af78700 5 -- op tracker -- , seq: 3149, time: 2014-06-09 16:22:11.796239, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.796246 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4ba) v4 from osd.5 2014-06-09 16:22:11.796249 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c01180 2014-06-09 16:22:11.796254 7f869af78700 5 -- op tracker -- , seq: 3149, time: 2014-06-09 16:22:11.796252, event: started, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.796270 7f869af78700 5 -- op tracker -- , seq: 3149, time: 2014-06-09 16:22:11.796268, event: done, op: pg_info(1 pgs e243:0.4ba) 2014-06-09 16:22:11.796285 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796287 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796303 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796305 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796306 7f869af78700 20 osd.26 243 _dispatch 0x30a3b340 pg_info(1 pgs e243:3.7bc) v4 2014-06-09 16:22:11.796311 7f869af78700 5 -- op tracker -- , seq: 3150, time: 2014-06-09 16:22:11.607444, event: header_read, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.796318 7f869af78700 5 -- op tracker -- , seq: 3150, time: 2014-06-09 16:22:11.607444, event: throttled, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.796324 7f869af78700 5 -- op tracker -- , seq: 3150, time: 2014-06-09 16:22:11.607452, event: all_read, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.796331 7f869af78700 5 -- op tracker -- , seq: 3150, time: 2014-06-09 16:22:11.796302, event: dispatched, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.796338 7f869af78700 5 -- op tracker -- , seq: 3150, time: 2014-06-09 16:22:11.796336, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.796343 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.7bc) v4 from osd.2 2014-06-09 16:22:11.796346 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a3b340 2014-06-09 16:22:11.796351 7f869af78700 5 -- op tracker -- , seq: 3150, time: 2014-06-09 16:22:11.796349, event: started, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.796367 7f869af78700 5 -- op tracker -- , seq: 3150, time: 2014-06-09 16:22:11.796365, event: done, op: pg_info(1 pgs e243:3.7bc) 2014-06-09 16:22:11.796374 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796376 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796389 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796391 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796392 7f869af78700 20 osd.26 243 _dispatch 0x2fc94380 pg_info(1 pgs e243:3.4fe) v4 2014-06-09 16:22:11.796397 7f869af78700 5 -- op tracker -- , seq: 3151, time: 2014-06-09 16:22:11.611969, event: header_read, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.796404 7f869af78700 5 -- op tracker -- , seq: 3151, time: 2014-06-09 16:22:11.611969, event: throttled, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.796411 7f869af78700 5 -- op tracker -- , seq: 3151, time: 2014-06-09 16:22:11.611979, event: all_read, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.796418 7f869af78700 5 -- op tracker -- , seq: 3151, time: 2014-06-09 16:22:11.796388, event: dispatched, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.796426 7f869af78700 5 -- op tracker -- , seq: 3151, time: 2014-06-09 16:22:11.796423, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.796431 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4fe) v4 from osd.4 2014-06-09 16:22:11.796434 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc94380 2014-06-09 16:22:11.796439 7f869af78700 5 -- op tracker -- , seq: 3151, time: 2014-06-09 16:22:11.796437, event: started, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.796460 7f869af78700 5 -- op tracker -- , seq: 3151, time: 2014-06-09 16:22:11.796457, event: done, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.796471 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796473 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796485 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796486 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796488 7f869af78700 20 osd.26 243 _dispatch 0x30c301c0 pg_info(1 pgs e243:3.52a) v4 2014-06-09 16:22:11.796493 7f869af78700 5 -- op tracker -- , seq: 3152, time: 2014-06-09 16:22:11.656642, event: header_read, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.796500 7f869af78700 5 -- op tracker -- , seq: 3152, time: 2014-06-09 16:22:11.656642, event: throttled, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.796506 7f869af78700 5 -- op tracker -- , seq: 3152, time: 2014-06-09 16:22:11.656654, event: all_read, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.796518 7f869af78700 5 -- op tracker -- , seq: 3152, time: 2014-06-09 16:22:11.796484, event: dispatched, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.796527 7f869af78700 5 -- op tracker -- , seq: 3152, time: 2014-06-09 16:22:11.796525, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.796532 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.52a) v4 from osd.3 2014-06-09 16:22:11.796535 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c301c0 2014-06-09 16:22:11.796540 7f869af78700 5 -- op tracker -- , seq: 3152, time: 2014-06-09 16:22:11.796538, event: started, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.796559 7f869af78700 5 -- op tracker -- , seq: 3152, time: 2014-06-09 16:22:11.796557, event: done, op: pg_info(1 pgs e243:3.52a) 2014-06-09 16:22:11.796572 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796574 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796589 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796590 7f869af78700 20 osd.26 243 _dispatch 0x1ac74a80 pg_info(1 pgs e243:0.4c8) v4 2014-06-09 16:22:11.796595 7f869af78700 5 -- op tracker -- , seq: 3153, time: 2014-06-09 16:22:11.664468, event: header_read, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.796602 7f869af78700 5 -- op tracker -- , seq: 3153, time: 2014-06-09 16:22:11.664469, event: throttled, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.796611 7f869af78700 5 -- op tracker -- , seq: 3153, time: 2014-06-09 16:22:11.664485, event: all_read, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.796617 7f869af78700 5 -- op tracker -- , seq: 3153, time: 2014-06-09 16:22:11.796587, event: dispatched, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.796625 7f869af78700 5 -- op tracker -- , seq: 3153, time: 2014-06-09 16:22:11.796623, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.796634 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4c8) v4 from osd.1 2014-06-09 16:22:11.796637 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac74a80 2014-06-09 16:22:11.796642 7f869af78700 5 -- op tracker -- , seq: 3153, time: 2014-06-09 16:22:11.796640, event: started, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.796659 7f869af78700 5 -- op tracker -- , seq: 3153, time: 2014-06-09 16:22:11.796656, event: done, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.796668 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796670 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796683 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796685 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796687 7f869af78700 20 osd.26 243 _dispatch 0x31389dc0 pg_info(1 pgs e243:2.26d) v4 2014-06-09 16:22:11.796692 7f869af78700 5 -- op tracker -- , seq: 3154, time: 2014-06-09 16:22:11.619583, event: header_read, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796699 7f869af78700 5 -- op tracker -- , seq: 3154, time: 2014-06-09 16:22:11.619584, event: throttled, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796705 7f869af78700 5 -- op tracker -- , seq: 3154, time: 2014-06-09 16:22:11.619601, event: all_read, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796712 7f869af78700 5 -- op tracker -- , seq: 3154, time: 2014-06-09 16:22:11.796683, event: dispatched, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796722 7f869af78700 5 -- op tracker -- , seq: 3154, time: 2014-06-09 16:22:11.796719, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796727 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.26d) v4 from osd.8 2014-06-09 16:22:11.796730 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31389dc0 2014-06-09 16:22:11.796735 7f869af78700 5 -- op tracker -- , seq: 3154, time: 2014-06-09 16:22:11.796733, event: started, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796750 7f869af78700 5 -- op tracker -- , seq: 3154, time: 2014-06-09 16:22:11.796748, event: done, op: pg_info(1 pgs e243:2.26d) 2014-06-09 16:22:11.796760 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796762 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796775 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796776 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796778 7f869af78700 20 osd.26 243 _dispatch 0x1ac74e00 pg_info(1 pgs e243:3.786) v4 2014-06-09 16:22:11.796783 7f869af78700 5 -- op tracker -- , seq: 3155, time: 2014-06-09 16:22:11.748759, event: header_read, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.796790 7f869af78700 5 -- op tracker -- , seq: 3155, time: 2014-06-09 16:22:11.748759, event: throttled, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.796801 7f869af78700 5 -- op tracker -- , seq: 3155, time: 2014-06-09 16:22:11.748777, event: all_read, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.796808 7f869af78700 5 -- op tracker -- , seq: 3155, time: 2014-06-09 16:22:11.796774, event: dispatched, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.796816 7f869af78700 5 -- op tracker -- , seq: 3155, time: 2014-06-09 16:22:11.796814, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.796821 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.786) v4 from osd.6 2014-06-09 16:22:11.796824 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac74e00 2014-06-09 16:22:11.796829 7f869af78700 5 -- op tracker -- , seq: 3155, time: 2014-06-09 16:22:11.796827, event: started, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.796848 7f869af78700 5 -- op tracker -- , seq: 3155, time: 2014-06-09 16:22:11.796846, event: done, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.796856 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796858 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796873 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796875 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796876 7f869af78700 20 osd.26 243 _dispatch 0x1ac74fc0 pg_info(1 pgs e243:0.6c0) v4 2014-06-09 16:22:11.796882 7f869af78700 5 -- op tracker -- , seq: 3156, time: 2014-06-09 16:22:11.660186, event: header_read, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.796890 7f869af78700 5 -- op tracker -- , seq: 3156, time: 2014-06-09 16:22:11.660186, event: throttled, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.796899 7f869af78700 5 -- op tracker -- , seq: 3156, time: 2014-06-09 16:22:11.660197, event: all_read, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.796906 7f869af78700 5 -- op tracker -- , seq: 3156, time: 2014-06-09 16:22:11.796873, event: dispatched, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.796913 7f869af78700 5 -- op tracker -- , seq: 3156, time: 2014-06-09 16:22:11.796911, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.796922 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6c0) v4 from osd.5 2014-06-09 16:22:11.796925 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac74fc0 2014-06-09 16:22:11.796930 7f869af78700 5 -- op tracker -- , seq: 3156, time: 2014-06-09 16:22:11.796928, event: started, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.796951 7f869af78700 5 -- op tracker -- , seq: 3156, time: 2014-06-09 16:22:11.796948, event: done, op: pg_info(1 pgs e243:0.6c0) 2014-06-09 16:22:11.796960 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796963 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796975 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.796976 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.796978 7f869af78700 20 osd.26 243 _dispatch 0x30da36c0 pg_info(1 pgs e243:0.3ab) v4 2014-06-09 16:22:11.796983 7f869af78700 5 -- op tracker -- , seq: 3157, time: 2014-06-09 16:22:11.608186, event: header_read, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.796990 7f869af78700 5 -- op tracker -- , seq: 3157, time: 2014-06-09 16:22:11.608187, event: throttled, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.796996 7f869af78700 5 -- op tracker -- , seq: 3157, time: 2014-06-09 16:22:11.608194, event: all_read, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797002 7f869af78700 5 -- op tracker -- , seq: 3157, time: 2014-06-09 16:22:11.796974, event: dispatched, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797010 7f869af78700 5 -- op tracker -- , seq: 3157, time: 2014-06-09 16:22:11.797008, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797014 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3ab) v4 from osd.2 2014-06-09 16:22:11.797017 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da36c0 2014-06-09 16:22:11.797022 7f869af78700 5 -- op tracker -- , seq: 3157, time: 2014-06-09 16:22:11.797020, event: started, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797040 7f869af78700 5 -- op tracker -- , seq: 3157, time: 2014-06-09 16:22:11.797037, event: done, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797047 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797062 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797063 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797065 7f869af78700 20 osd.26 243 _dispatch 0x2fc948c0 pg_info(1 pgs e243:3.4fe) v4 2014-06-09 16:22:11.797070 7f869af78700 5 -- op tracker -- , seq: 3158, time: 2014-06-09 16:22:11.613235, event: header_read, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.797076 7f869af78700 5 -- op tracker -- , seq: 3158, time: 2014-06-09 16:22:11.613236, event: throttled, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.797082 7f869af78700 5 -- op tracker -- , seq: 3158, time: 2014-06-09 16:22:11.613250, event: all_read, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.797088 7f869af78700 5 -- op tracker -- , seq: 3158, time: 2014-06-09 16:22:11.797061, event: dispatched, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.797095 7f869af78700 5 -- op tracker -- , seq: 3158, time: 2014-06-09 16:22:11.797093, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.797099 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4fe) v4 from osd.4 2014-06-09 16:22:11.797102 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc948c0 2014-06-09 16:22:11.797108 7f869af78700 5 -- op tracker -- , seq: 3158, time: 2014-06-09 16:22:11.797105, event: started, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.797130 7f869af78700 5 -- op tracker -- , seq: 3158, time: 2014-06-09 16:22:11.797128, event: done, op: pg_info(1 pgs e243:3.4fe) 2014-06-09 16:22:11.797137 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797138 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797149 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797151 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797152 7f869af78700 20 osd.26 243 _dispatch 0x16064540 pg_info(1 pgs e243:3.6e) v4 2014-06-09 16:22:11.797156 7f869af78700 5 -- op tracker -- , seq: 3159, time: 2014-06-09 16:22:11.716271, event: header_read, op: pg_info(1 pgs e243:3.6e) 2014-06-09 16:22:11.797162 7f869af78700 5 -- op tracker -- , seq: 3159, time: 2014-06-09 16:22:11.716276, event: throttled, op: pg_info(1 pgs e243:3.6e) 2014-06-09 16:22:11.797168 7f869af78700 5 -- op tracker -- , seq: 3159, time: 2014-06-09 16:22:11.716293, event: all_read, op: pg_info(1 pgs e243:3.6e) 2014-06-09 16:22:11.797174 7f869af78700 5 -- op tracker -- , seq: 3159, time: 2014-06-09 16:22:11.797149, event: dispatched, op: pg_info(1 pgs e243:3.6e) 2014-06-09 16:22:11.797181 7f869af78700 5 -- op tracker -- , seq: 3159, time: 2014-06-09 16:22:11.797178, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6e) 2014-06-09 16:22:11.797186 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6e) v4 from osd.3 2014-06-09 16:22:11.797188 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16064540 2014-06-09 16:22:11.797194 7f869af78700 5 -- op tracker -- , seq: 3159, time: 2014-06-09 16:22:11.797192, event: started, op: pg_info(1 pgs e243:3.6e) 2014-06-09 16:22:11.797211 7f869af78700 5 -- op tracker -- , seq: 3159, time: 2014-06-09 16:22:11.797209, event: done, op: pg_info(1 pgs e243:3.6e) 2014-06-09 16:22:11.797219 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797221 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797233 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797235 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797236 7f869af78700 20 osd.26 243 _dispatch 0x3115b880 pg_info(1 pgs e243:0.4c8) v4 2014-06-09 16:22:11.797241 7f869af78700 5 -- op tracker -- , seq: 3160, time: 2014-06-09 16:22:11.665037, event: header_read, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.797251 7f869af78700 5 -- op tracker -- , seq: 3160, time: 2014-06-09 16:22:11.665038, event: throttled, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.797258 7f869af78700 5 -- op tracker -- , seq: 3160, time: 2014-06-09 16:22:11.665052, event: all_read, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.797264 7f869af78700 5 -- op tracker -- , seq: 3160, time: 2014-06-09 16:22:11.797233, event: dispatched, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.797273 7f869af78700 5 -- op tracker -- , seq: 3160, time: 2014-06-09 16:22:11.797271, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.797278 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4c8) v4 from osd.1 2014-06-09 16:22:11.797281 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115b880 2014-06-09 16:22:11.797286 7f869af78700 5 -- op tracker -- , seq: 3160, time: 2014-06-09 16:22:11.797285, event: started, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.797301 7f869af78700 5 -- op tracker -- , seq: 3160, time: 2014-06-09 16:22:11.797299, event: done, op: pg_info(1 pgs e243:0.4c8) 2014-06-09 16:22:11.797309 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797311 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797322 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797324 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797326 7f869af78700 20 osd.26 243 _dispatch 0x30de4540 pg_info(1 pgs e243:2.350) v4 2014-06-09 16:22:11.797330 7f869af78700 5 -- op tracker -- , seq: 3161, time: 2014-06-09 16:22:11.640289, event: header_read, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.797337 7f869af78700 5 -- op tracker -- , seq: 3161, time: 2014-06-09 16:22:11.640290, event: throttled, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.797344 7f869af78700 5 -- op tracker -- , seq: 3161, time: 2014-06-09 16:22:11.640302, event: all_read, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.797351 7f869af78700 5 -- op tracker -- , seq: 3161, time: 2014-06-09 16:22:11.797322, event: dispatched, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.797363 7f869af78700 5 -- op tracker -- , seq: 3161, time: 2014-06-09 16:22:11.797361, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.797368 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.350) v4 from osd.8 2014-06-09 16:22:11.797371 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de4540 2014-06-09 16:22:11.797375 7f869af78700 5 -- op tracker -- , seq: 3161, time: 2014-06-09 16:22:11.797374, event: started, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.797391 7f869af78700 5 -- op tracker -- , seq: 3161, time: 2014-06-09 16:22:11.797389, event: done, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.797399 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797400 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797413 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797414 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797416 7f869af78700 20 osd.26 243 _dispatch 0x308e2540 pg_info(1 pgs e243:3.1f0) v4 2014-06-09 16:22:11.797421 7f869af78700 5 -- op tracker -- , seq: 3162, time: 2014-06-09 16:22:11.748790, event: header_read, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.797432 7f869af78700 5 -- op tracker -- , seq: 3162, time: 2014-06-09 16:22:11.748790, event: throttled, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.797439 7f869af78700 5 -- op tracker -- , seq: 3162, time: 2014-06-09 16:22:11.748803, event: all_read, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.797446 7f869af78700 5 -- op tracker -- , seq: 3162, time: 2014-06-09 16:22:11.797412, event: dispatched, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.797454 7f869af78700 5 -- op tracker -- , seq: 3162, time: 2014-06-09 16:22:11.797452, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.797460 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1f0) v4 from osd.6 2014-06-09 16:22:11.797463 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x308e2540 2014-06-09 16:22:11.797468 7f869af78700 5 -- op tracker -- , seq: 3162, time: 2014-06-09 16:22:11.797466, event: started, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.797485 7f869af78700 5 -- op tracker -- , seq: 3162, time: 2014-06-09 16:22:11.797483, event: done, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.797497 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797500 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797513 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797516 7f869af78700 20 osd.26 243 _dispatch 0x30adb500 pg_info(1 pgs e243:3.730) v4 2014-06-09 16:22:11.797521 7f869af78700 5 -- op tracker -- , seq: 3163, time: 2014-06-09 16:22:11.707490, event: header_read, op: pg_info(1 pgs e243:3.730) 2014-06-09 16:22:11.797530 7f869af78700 5 -- op tracker -- , seq: 3163, time: 2014-06-09 16:22:11.707491, event: throttled, op: pg_info(1 pgs e243:3.730) 2014-06-09 16:22:11.797537 7f869af78700 5 -- op tracker -- , seq: 3163, time: 2014-06-09 16:22:11.707503, event: all_read, op: pg_info(1 pgs e243:3.730) 2014-06-09 16:22:11.797543 7f869af78700 5 -- op tracker -- , seq: 3163, time: 2014-06-09 16:22:11.797512, event: dispatched, op: pg_info(1 pgs e243:3.730) 2014-06-09 16:22:11.797552 7f869af78700 5 -- op tracker -- , seq: 3163, time: 2014-06-09 16:22:11.797550, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.730) 2014-06-09 16:22:11.797557 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.730) v4 from osd.5 2014-06-09 16:22:11.797560 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adb500 2014-06-09 16:22:11.797565 7f869af78700 5 -- op tracker -- , seq: 3163, time: 2014-06-09 16:22:11.797563, event: started, op: pg_info(1 pgs e243:3.730) 2014-06-09 16:22:11.797587 7f869af78700 5 -- op tracker -- , seq: 3163, time: 2014-06-09 16:22:11.797585, event: done, op: pg_info(1 pgs e243:3.730) 2014-06-09 16:22:11.797595 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797597 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797609 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797611 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797612 7f869af78700 20 osd.26 243 _dispatch 0x30da3500 pg_info(1 pgs e243:0.3ab) v4 2014-06-09 16:22:11.797617 7f869af78700 5 -- op tracker -- , seq: 3164, time: 2014-06-09 16:22:11.608248, event: header_read, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797624 7f869af78700 5 -- op tracker -- , seq: 3164, time: 2014-06-09 16:22:11.608249, event: throttled, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797636 7f869af78700 5 -- op tracker -- , seq: 3164, time: 2014-06-09 16:22:11.608256, event: all_read, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797646 7f869af78700 5 -- op tracker -- , seq: 3164, time: 2014-06-09 16:22:11.797609, event: dispatched, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797657 7f869af78700 5 -- op tracker -- , seq: 3164, time: 2014-06-09 16:22:11.797655, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797666 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3ab) v4 from osd.2 2014-06-09 16:22:11.797669 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da3500 2014-06-09 16:22:11.797674 7f869af78700 5 -- op tracker -- , seq: 3164, time: 2014-06-09 16:22:11.797672, event: started, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797689 7f869af78700 5 -- op tracker -- , seq: 3164, time: 2014-06-09 16:22:11.797687, event: done, op: pg_info(1 pgs e243:0.3ab) 2014-06-09 16:22:11.797697 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797699 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797717 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797718 7f869af78700 20 osd.26 243 _dispatch 0x2f038c40 pg_info(1 pgs e243:2.30b) v4 2014-06-09 16:22:11.797723 7f869af78700 5 -- op tracker -- , seq: 3165, time: 2014-06-09 16:22:11.613296, event: header_read, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.797730 7f869af78700 5 -- op tracker -- , seq: 3165, time: 2014-06-09 16:22:11.613297, event: throttled, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.797737 7f869af78700 5 -- op tracker -- , seq: 3165, time: 2014-06-09 16:22:11.613310, event: all_read, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.797744 7f869af78700 5 -- op tracker -- , seq: 3165, time: 2014-06-09 16:22:11.797714, event: dispatched, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.797753 7f869af78700 5 -- op tracker -- , seq: 3165, time: 2014-06-09 16:22:11.797751, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.797758 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.30b) v4 from osd.4 2014-06-09 16:22:11.797761 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f038c40 2014-06-09 16:22:11.797766 7f869af78700 5 -- op tracker -- , seq: 3165, time: 2014-06-09 16:22:11.797764, event: started, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.797787 7f869af78700 5 -- op tracker -- , seq: 3165, time: 2014-06-09 16:22:11.797785, event: done, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.797795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797797 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797810 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797812 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797813 7f869af78700 20 osd.26 243 _dispatch 0xcd76700 pg_info(1 pgs e243:3.5f9) v4 2014-06-09 16:22:11.797818 7f869af78700 5 -- op tracker -- , seq: 3166, time: 2014-06-09 16:22:11.737439, event: header_read, op: pg_info(1 pgs e243:3.5f9) 2014-06-09 16:22:11.797830 7f869af78700 5 -- op tracker -- , seq: 3166, time: 2014-06-09 16:22:11.737439, event: throttled, op: pg_info(1 pgs e243:3.5f9) 2014-06-09 16:22:11.797837 7f869af78700 5 -- op tracker -- , seq: 3166, time: 2014-06-09 16:22:11.737455, event: all_read, op: pg_info(1 pgs e243:3.5f9) 2014-06-09 16:22:11.797848 7f869af78700 5 -- op tracker -- , seq: 3166, time: 2014-06-09 16:22:11.797809, event: dispatched, op: pg_info(1 pgs e243:3.5f9) 2014-06-09 16:22:11.797859 7f869af78700 5 -- op tracker -- , seq: 3166, time: 2014-06-09 16:22:11.797857, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5f9) 2014-06-09 16:22:11.797864 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5f9) v4 from osd.3 2014-06-09 16:22:11.797867 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd76700 2014-06-09 16:22:11.797872 7f869af78700 5 -- op tracker -- , seq: 3166, time: 2014-06-09 16:22:11.797870, event: started, op: pg_info(1 pgs e243:3.5f9) 2014-06-09 16:22:11.797888 7f869af78700 5 -- op tracker -- , seq: 3166, time: 2014-06-09 16:22:11.797886, event: done, op: pg_info(1 pgs e243:3.5f9) 2014-06-09 16:22:11.797896 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797898 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797910 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797912 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.797914 7f869af78700 20 osd.26 243 _dispatch 0x30e10fc0 pg_info(1 pgs e243:3.709) v4 2014-06-09 16:22:11.797919 7f869af78700 5 -- op tracker -- , seq: 3167, time: 2014-06-09 16:22:11.665440, event: header_read, op: pg_info(1 pgs e243:3.709) 2014-06-09 16:22:11.797926 7f869af78700 5 -- op tracker -- , seq: 3167, time: 2014-06-09 16:22:11.665440, event: throttled, op: pg_info(1 pgs e243:3.709) 2014-06-09 16:22:11.797932 7f869af78700 5 -- op tracker -- , seq: 3167, time: 2014-06-09 16:22:11.665452, event: all_read, op: pg_info(1 pgs e243:3.709) 2014-06-09 16:22:11.797939 7f869af78700 5 -- op tracker -- , seq: 3167, time: 2014-06-09 16:22:11.797910, event: dispatched, op: pg_info(1 pgs e243:3.709) 2014-06-09 16:22:11.797951 7f869af78700 5 -- op tracker -- , seq: 3167, time: 2014-06-09 16:22:11.797949, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.709) 2014-06-09 16:22:11.797956 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.709) v4 from osd.1 2014-06-09 16:22:11.797959 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e10fc0 2014-06-09 16:22:11.797964 7f869af78700 5 -- op tracker -- , seq: 3167, time: 2014-06-09 16:22:11.797962, event: started, op: pg_info(1 pgs e243:3.709) 2014-06-09 16:22:11.797980 7f869af78700 5 -- op tracker -- , seq: 3167, time: 2014-06-09 16:22:11.797978, event: done, op: pg_info(1 pgs e243:3.709) 2014-06-09 16:22:11.797988 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.797990 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798003 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798004 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798005 7f869af78700 20 osd.26 243 _dispatch 0x31389c00 pg_info(1 pgs e243:2.350) v4 2014-06-09 16:22:11.798011 7f869af78700 5 -- op tracker -- , seq: 3168, time: 2014-06-09 16:22:11.640363, event: header_read, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.798020 7f869af78700 5 -- op tracker -- , seq: 3168, time: 2014-06-09 16:22:11.640365, event: throttled, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.798026 7f869af78700 5 -- op tracker -- , seq: 3168, time: 2014-06-09 16:22:11.640384, event: all_read, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.798033 7f869af78700 5 -- op tracker -- , seq: 3168, time: 2014-06-09 16:22:11.798002, event: dispatched, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.798046 7f869af78700 5 -- op tracker -- , seq: 3168, time: 2014-06-09 16:22:11.798043, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.798050 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.350) v4 from osd.8 2014-06-09 16:22:11.798053 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31389c00 2014-06-09 16:22:11.798053 7f86a5610700 10 _do_op 0x2fcae480 seq 3474 r = 0, finisher 0x30629020 0x12c0ba80 2014-06-09 16:22:11.798060 7f86a5610700 10 _finish_op 0x2fcae480 seq 3474 osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.798058 7f869af78700 5 -- op tracker -- , seq: 3168, time: 2014-06-09 16:22:11.798056, event: started, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.798071 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4a20 (1 active) 2014-06-09 16:22:11.798074 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4fc0 (2 active) 2014-06-09 16:22:11.798074 7f868b759700 10 op_submit_start 3523 2014-06-09 16:22:11.798073 7f869af78700 5 -- op tracker -- , seq: 3168, time: 2014-06-09 16:22:11.798070, event: done, op: pg_info(1 pgs e243:2.350) 2014-06-09 16:22:11.798077 7f868b759700 5 queue_transactions (trailing journal) 3523 2014-06-09 16:22:11.798077 7f86a5610700 5 _do_op 0x288f9e00 seq 3475 osr(3.1b5 0x3ed0a80)/0x3ed0a80 start 2014-06-09 16:22:11.798080 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798078 7f868b759700 5 queue_op 0xbf7f800 seq 3523 osr(1.92 0x384c410) 826 bytes (queue has 50 ops and 4335311 bytes) 2014-06-09 16:22:11.798081 7f86a5610700 10 _do_transaction on 0xc2f6180 2014-06-09 16:22:11.798081 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798082 7f868b759700 10 op_submit_finish 3523 2014-06-09 16:22:11.798089 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.798092 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798094 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798095 7f869af78700 20 osd.26 243 _dispatch 0x30caf500 pg_info(1 pgs e243:3.470) v4 2014-06-09 16:22:11.798087 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.798099 7f869af78700 5 -- op tracker -- , seq: 3169, time: 2014-06-09 16:22:11.749285, event: header_read, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.798105 7f869af78700 5 -- op tracker -- , seq: 3169, time: 2014-06-09 16:22:11.749285, event: throttled, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.798111 7f869af78700 5 -- op tracker -- , seq: 3169, time: 2014-06-09 16:22:11.749299, event: all_read, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.798107 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.798117 7f869af78700 5 -- op tracker -- , seq: 3169, time: 2014-06-09 16:22:11.798092, event: dispatched, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.798116 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.798123 7f869af78700 5 -- op tracker -- , seq: 3169, time: 2014-06-09 16:22:11.798121, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.798127 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.470) v4 from osd.6 2014-06-09 16:22:11.798130 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30caf500 2014-06-09 16:22:11.798124 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.798134 7f869af78700 5 -- op tracker -- , seq: 3169, time: 2014-06-09 16:22:11.798132, event: started, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.798142 7f868b759700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.798149 7f869af78700 5 -- op tracker -- , seq: 3169, time: 2014-06-09 16:22:11.798147, event: done, op: pg_info(1 pgs e243:3.470) 2014-06-09 16:22:11.798156 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798157 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798165 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.798168 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798161 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.798170 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798171 7f869af78700 20 osd.26 243 _dispatch 0x1ac74540 pg_info(1 pgs e243:3.27) v4 2014-06-09 16:22:11.798178 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798176 7f869af78700 5 -- op tracker -- , seq: 3170, time: 2014-06-09 16:22:11.743926, event: header_read, op: pg_info(1 pgs e243:3.27) 2014-06-09 16:22:11.798182 7f869af78700 5 -- op tracker -- , seq: 3170, time: 2014-06-09 16:22:11.743926, event: throttled, op: pg_info(1 pgs e243:3.27) 2014-06-09 16:22:11.798185 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3475.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.798190 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.798188 7f869af78700 5 -- op tracker -- , seq: 3170, time: 2014-06-09 16:22:11.743939, event: all_read, op: pg_info(1 pgs e243:3.27) 2014-06-09 16:22:11.798184 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.798193 7f869af78700 5 -- op tracker -- , seq: 3170, time: 2014-06-09 16:22:11.798168, event: dispatched, op: pg_info(1 pgs e243:3.27) 2014-06-09 16:22:11.798196 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.798200 7f869af78700 5 -- op tracker -- , seq: 3170, time: 2014-06-09 16:22:11.798198, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.27) 2014-06-09 16:22:11.798205 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.27) v4 from osd.5 2014-06-09 16:22:11.798208 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac74540 2014-06-09 16:22:11.798208 7f86a5610700 10 _do_op 0x288f9e00 seq 3475 r = 0, finisher 0x30d618a0 0 2014-06-09 16:22:11.798205 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.798212 7f86a5610700 10 _finish_op 0x288f9e00 seq 3475 osr(3.1b5 0x3ed0a80)/0x3ed0a80 2014-06-09 16:22:11.798212 7f869af78700 5 -- op tracker -- , seq: 3170, time: 2014-06-09 16:22:11.798210, event: started, op: pg_info(1 pgs e243:3.27) 2014-06-09 16:22:11.798215 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4fc0 (1 active) 2014-06-09 16:22:11.798212 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.798218 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f200 (2 active) 2014-06-09 16:22:11.798220 7f86a5610700 5 _do_op 0x30607bc0 seq 3476 osr(2.4f4 0x2ed16b0)/0x2ed16b0 start 2014-06-09 16:22:11.798223 7f86a5610700 10 _do_transaction on 0x3e62780 2014-06-09 16:22:11.798226 7f868b759700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.798228 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.798228 7f869af78700 5 -- op tracker -- , seq: 3170, time: 2014-06-09 16:22:11.798225, event: done, op: pg_info(1 pgs e243:3.27) 2014-06-09 16:22:11.798235 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798237 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798240 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.798248 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798249 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798250 7f869af78700 20 osd.26 243 _dispatch 0x2fcaa1c0 pg_info(1 pgs e243:1.b5) v4 2014-06-09 16:22:11.798254 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798255 7f869af78700 5 -- op tracker -- , seq: 3171, time: 2014-06-09 16:22:11.613121, event: header_read, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.798260 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.798261 7f869af78700 5 -- op tracker -- , seq: 3171, time: 2014-06-09 16:22:11.613121, event: throttled, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.798258 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.798267 7f869af78700 5 -- op tracker -- , seq: 3171, time: 2014-06-09 16:22:11.613129, event: all_read, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.798273 7f869af78700 5 -- op tracker -- , seq: 3171, time: 2014-06-09 16:22:11.798248, event: dispatched, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.798269 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.798274 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3476.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.798278 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.798279 7f869af78700 5 -- op tracker -- , seq: 3171, time: 2014-06-09 16:22:11.798277, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.798278 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.798283 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.b5) v4 from osd.2 2014-06-09 16:22:11.798285 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fcaa1c0 2014-06-09 16:22:11.798288 7f86a5610700 10 _do_op 0x30607bc0 seq 3476 r = 0, finisher 0xc410a60 0 2014-06-09 16:22:11.798285 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.798291 7f86a5610700 10 _finish_op 0x30607bc0 seq 3476 osr(2.4f4 0x2ed16b0)/0x2ed16b0 2014-06-09 16:22:11.798290 7f869af78700 5 -- op tracker -- , seq: 3171, time: 2014-06-09 16:22:11.798288, event: started, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.798295 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f200 (1 active) 2014-06-09 16:22:11.798297 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b200 (2 active) 2014-06-09 16:22:11.798298 7f868b759700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.798299 7f86a5610700 5 _do_op 0xc34a180 seq 3477 osr(3.465 0x3ed1580)/0x3ed1580 start 2014-06-09 16:22:11.798302 7f86a5610700 10 _do_transaction on 0xc218d80 2014-06-09 16:22:11.798304 7f869af78700 5 -- op tracker -- , seq: 3171, time: 2014-06-09 16:22:11.798302, event: done, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.798307 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.798312 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798314 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798312 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:156 2014-06-09 16:22:11.798325 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798326 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798326 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798327 7f869af78700 20 osd.26 243 _dispatch 0x2f038a80 pg_info(1 pgs e243:2.30b) v4 2014-06-09 16:22:11.798332 7f869af78700 5 -- op tracker -- , seq: 3172, time: 2014-06-09 16:22:11.613317, event: header_read, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.798329 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.798338 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.798339 7f869af78700 5 -- op tracker -- , seq: 3172, time: 2014-06-09 16:22:11.613318, event: throttled, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.798344 7f869af78700 5 -- op tracker -- , seq: 3172, time: 2014-06-09 16:22:11.613325, event: all_read, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.798341 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.798350 7f869af78700 5 -- op tracker -- , seq: 3172, time: 2014-06-09 16:22:11.798324, event: dispatched, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.798349 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.798354 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3477.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.798358 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.798356 7f869af78700 5 -- op tracker -- , seq: 3172, time: 2014-06-09 16:22:11.798355, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.798356 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.798361 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.30b) v4 from osd.4 2014-06-09 16:22:11.798364 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f038a80 2014-06-09 16:22:11.798367 7f868b759700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.798369 7f86a5610700 10 _do_op 0xc34a180 seq 3477 r = 0, finisher 0x314790e0 0 2014-06-09 16:22:11.798368 7f869af78700 5 -- op tracker -- , seq: 3172, time: 2014-06-09 16:22:11.798367, event: started, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.798373 7f86a5610700 10 _finish_op 0xc34a180 seq 3477 osr(3.465 0x3ed1580)/0x3ed1580 2014-06-09 16:22:11.798375 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b200 (1 active) 2014-06-09 16:22:11.798377 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.798379 7f86a5610700 5 _do_op 0x30d97260 seq 3478 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.798381 7f86a5610700 10 _do_transaction on 0xc12af00 2014-06-09 16:22:11.798385 7f86a5610700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.798383 7f869af78700 5 -- op tracker -- , seq: 3172, time: 2014-06-09 16:22:11.798381, event: done, op: pg_info(1 pgs e243:2.30b) 2014-06-09 16:22:11.798383 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.798391 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798393 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798396 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798403 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798405 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798406 7f869af78700 20 osd.26 243 _dispatch 0x16fdbdc0 pg_info(1 pgs e243:2.1e3) v4 2014-06-09 16:22:11.798400 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.e8( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.798410 7f869af78700 5 -- op tracker -- , seq: 3173, time: 2014-06-09 16:22:11.746666, event: header_read, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.798417 7f869af78700 5 -- op tracker -- , seq: 3173, time: 2014-06-09 16:22:11.746667, event: throttled, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.798423 7f86a5610700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.798422 7f869af78700 5 -- op tracker -- , seq: 3173, time: 2014-06-09 16:22:11.746684, event: all_read, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.798429 7f869af78700 5 -- op tracker -- , seq: 3173, time: 2014-06-09 16:22:11.798403, event: dispatched, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.798435 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798435 7f869af78700 5 -- op tracker -- , seq: 3173, time: 2014-06-09 16:22:11.798433, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.798437 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.798439 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1e3) v4 from osd.3 2014-06-09 16:22:11.798441 7f868b759700 5 queue_transactions existing osr(0.e8 0x2bd6510)/0x2bd6510 2014-06-09 16:22:11.798442 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16fdbdc0 2014-06-09 16:22:11.798443 7f868b759700 10 op_submit_start 3524 2014-06-09 16:22:11.798444 7f868b759700 5 queue_transactions (trailing journal) 3524 2014-06-09 16:22:11.798446 7f868b759700 5 queue_op 0xbf7faa0 seq 3524 osr(0.e8 0x2bd6510) 826 bytes (queue has 48 ops and 4332545 bytes) 2014-06-09 16:22:11.798448 7f868b759700 10 op_submit_finish 3524 2014-06-09 16:22:11.798446 7f869af78700 5 -- op tracker -- , seq: 3173, time: 2014-06-09 16:22:11.798445, event: started, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.798451 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.798460 7f869af78700 5 -- op tracker -- , seq: 3173, time: 2014-06-09 16:22:11.798458, event: done, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.798467 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798467 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.798469 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798464 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.798472 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.798480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798481 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798481 7f86a5610700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.798482 7f869af78700 20 osd.26 243 _dispatch 0x3115b6c0 pg_info(1 pgs e243:3.718) v4 2014-06-09 16:22:11.798479 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.798487 7f86a5610700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.798486 7f869af78700 5 -- op tracker -- , seq: 3174, time: 2014-06-09 16:22:11.696076, event: header_read, op: pg_info(1 pgs e243:3.718) 2014-06-09 16:22:11.798490 7f868b759700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.798493 7f86a5610700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.798493 7f869af78700 5 -- op tracker -- , seq: 3174, time: 2014-06-09 16:22:11.696077, event: throttled, op: pg_info(1 pgs e243:3.718) 2014-06-09 16:22:11.798499 7f869af78700 5 -- op tracker -- , seq: 3174, time: 2014-06-09 16:22:11.696093, event: all_read, op: pg_info(1 pgs e243:3.718) 2014-06-09 16:22:11.798505 7f86a5610700 15 _write 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 1478656~1548288 2014-06-09 16:22:11.798505 7f869af78700 5 -- op tracker -- , seq: 3174, time: 2014-06-09 16:22:11.798479, event: dispatched, op: pg_info(1 pgs e243:3.718) 2014-06-09 16:22:11.798501 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:166 2014-06-09 16:22:11.798514 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798512 7f869af78700 5 -- op tracker -- , seq: 3174, time: 2014-06-09 16:22:11.798510, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.718) 2014-06-09 16:22:11.798516 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.718) v4 from osd.1 2014-06-09 16:22:11.798519 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115b6c0 2014-06-09 16:22:11.798523 7f869af78700 5 -- op tracker -- , seq: 3174, time: 2014-06-09 16:22:11.798522, event: started, op: pg_info(1 pgs e243:3.718) 2014-06-09 16:22:11.798519 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.8c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.798539 7f869af78700 5 -- op tracker -- , seq: 3174, time: 2014-06-09 16:22:11.798537, event: done, op: pg_info(1 pgs e243:3.718) 2014-06-09 16:22:11.798546 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798547 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798548 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798551 7f868b759700 5 queue_transactions existing osr(0.8c 0x2bd6480)/0x2bd6480 2014-06-09 16:22:11.798554 7f868b759700 10 op_submit_start 3525 2014-06-09 16:22:11.798555 7f868b759700 5 queue_transactions (trailing journal) 3525 2014-06-09 16:22:11.798556 7f868b759700 5 queue_op 0xbf7e120 seq 3525 osr(0.8c 0x2bd6480) 826 bytes (queue has 49 ops and 4333371 bytes) 2014-06-09 16:22:11.798558 7f868b759700 10 op_submit_finish 3525 2014-06-09 16:22:11.798560 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798561 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798562 7f869af78700 20 osd.26 243 _dispatch 0xcd79880 pg_info(1 pgs e243:3.3a8) v4 2014-06-09 16:22:11.798567 7f869af78700 5 -- op tracker -- , seq: 3175, time: 2014-06-09 16:22:11.646309, event: header_read, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.798561 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.798573 7f869af78700 5 -- op tracker -- , seq: 3175, time: 2014-06-09 16:22:11.646310, event: throttled, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.798577 7f86a5610700 10 lookup_strip_header done cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.798573 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.798579 7f869af78700 5 -- op tracker -- , seq: 3175, time: 2014-06-09 16:22:11.646323, event: all_read, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.798581 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.798585 7f869af78700 5 -- op tracker -- , seq: 3175, time: 2014-06-09 16:22:11.798560, event: dispatched, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.798591 7f869af78700 5 -- op tracker -- , seq: 3175, time: 2014-06-09 16:22:11.798589, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.798589 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.798595 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a8) v4 from osd.8 2014-06-09 16:22:11.798598 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd79880 2014-06-09 16:22:11.798600 7f868b759700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.798602 7f869af78700 5 -- op tracker -- , seq: 3175, time: 2014-06-09 16:22:11.798600, event: started, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.798617 7f869af78700 5 -- op tracker -- , seq: 3175, time: 2014-06-09 16:22:11.798615, event: done, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.798614 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.798625 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798626 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798627 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798637 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798638 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798631 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.798639 7f869af78700 20 osd.26 243 _dispatch 0x30c901c0 pg_info(1 pgs e243:3.e8) v4 2014-06-09 16:22:11.798644 7f869af78700 5 -- op tracker -- , seq: 3176, time: 2014-06-09 16:22:11.749317, event: header_read, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.798643 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.798651 7f869af78700 5 -- op tracker -- , seq: 3176, time: 2014-06-09 16:22:11.749318, event: throttled, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.798654 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.798651 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.798657 7f869af78700 5 -- op tracker -- , seq: 3176, time: 2014-06-09 16:22:11.749333, event: all_read, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.798658 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.798663 7f869af78700 5 -- op tracker -- , seq: 3176, time: 2014-06-09 16:22:11.798637, event: dispatched, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.798668 7f868b759700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.798669 7f869af78700 5 -- op tracker -- , seq: 3176, time: 2014-06-09 16:22:11.798667, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.798673 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.e8) v4 from osd.6 2014-06-09 16:22:11.798676 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c901c0 2014-06-09 16:22:11.798680 7f869af78700 5 -- op tracker -- , seq: 3176, time: 2014-06-09 16:22:11.798678, event: started, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.798678 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.798691 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798695 7f869af78700 5 -- op tracker -- , seq: 3176, time: 2014-06-09 16:22:11.798693, event: done, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.798702 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798704 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798696 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.186( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.798707 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.798715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798717 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798717 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798718 7f869af78700 20 osd.26 243 _dispatch 0x30c91880 pg_info(1 pgs e243:2.624) v4 2014-06-09 16:22:11.798723 7f869af78700 5 -- op tracker -- , seq: 3177, time: 2014-06-09 16:22:11.747293, event: header_read, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.798721 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.521( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.798729 7f869af78700 5 -- op tracker -- , seq: 3177, time: 2014-06-09 16:22:11.747293, event: throttled, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.798735 7f869af78700 5 -- op tracker -- , seq: 3177, time: 2014-06-09 16:22:11.747302, event: all_read, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.798741 7f869af78700 5 -- op tracker -- , seq: 3177, time: 2014-06-09 16:22:11.798715, event: dispatched, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.798747 7f869af78700 5 -- op tracker -- , seq: 3177, time: 2014-06-09 16:22:11.798745, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.798750 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798751 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.624) v4 from osd.5 2014-06-09 16:22:11.798754 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c91880 2014-06-09 16:22:11.798754 7f868b759700 5 queue_transactions existing osr(0.521 0x2bd7570)/0x2bd7570 2014-06-09 16:22:11.798756 7f868b759700 10 op_submit_start 3526 2014-06-09 16:22:11.798757 7f868b759700 5 queue_transactions (trailing journal) 3526 2014-06-09 16:22:11.798758 7f868b759700 5 queue_op 0x33a2d20 seq 3526 osr(0.521 0x2bd7570) 828 bytes (queue has 50 ops and 4334199 bytes) 2014-06-09 16:22:11.798761 7f868b759700 10 op_submit_finish 3526 2014-06-09 16:22:11.798758 7f869af78700 5 -- op tracker -- , seq: 3177, time: 2014-06-09 16:22:11.798756, event: started, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.798763 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.67a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.798772 7f869af78700 5 -- op tracker -- , seq: 3177, time: 2014-06-09 16:22:11.798770, event: done, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.798779 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798792 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.798792 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798793 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798794 7f869af78700 20 osd.26 243 _dispatch 0x2fc94000 pg_info(1 pgs e243:2.c6) v4 2014-06-09 16:22:11.798795 7f868b759700 5 queue_transactions existing osr(1.67a 0x384de90)/0x384de90 2014-06-09 16:22:11.798797 7f868b759700 2 waiting 51 > 50 ops || 4335027 > 104857600 2014-06-09 16:22:11.798799 7f869af78700 5 -- op tracker -- , seq: 3178, time: 2014-06-09 16:22:11.613184, event: header_read, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.798803 7f868a757700 2 waiting 51 > 50 ops || 4336102 > 104857600 2014-06-09 16:22:11.798806 7f869af78700 5 -- op tracker -- , seq: 3178, time: 2014-06-09 16:22:11.613185, event: throttled, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.798812 7f869af78700 5 -- op tracker -- , seq: 3178, time: 2014-06-09 16:22:11.613196, event: all_read, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.798818 7f869af78700 5 -- op tracker -- , seq: 3178, time: 2014-06-09 16:22:11.798792, event: dispatched, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.798824 7f869af78700 5 -- op tracker -- , seq: 3178, time: 2014-06-09 16:22:11.798822, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.798828 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.c6) v4 from osd.2 2014-06-09 16:22:11.798831 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc94000 2014-06-09 16:22:11.798836 7f869af78700 5 -- op tracker -- , seq: 3178, time: 2014-06-09 16:22:11.798834, event: started, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.798852 7f8690763700 2 waiting 51 > 50 ops || 4335027 > 104857600 2014-06-09 16:22:11.798851 7f869af78700 5 -- op tracker -- , seq: 3178, time: 2014-06-09 16:22:11.798849, event: done, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.798859 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798859 7f868ef60700 2 waiting 51 > 50 ops || 4336200 > 104857600 2014-06-09 16:22:11.798861 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798865 7f8689755700 2 waiting 51 > 50 ops || 4336142 > 104857600 2014-06-09 16:22:11.798873 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798874 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798876 7f869af78700 20 osd.26 243 _dispatch 0x2f0388c0 pg_info(1 pgs e243:0.9d) v4 2014-06-09 16:22:11.798909 7f8692767700 2 waiting 51 > 50 ops || 4335534 > 104857600 2014-06-09 16:22:11.798907 7f869af78700 5 -- op tracker -- , seq: 3179, time: 2014-06-09 16:22:11.613692, event: header_read, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.798915 7f869af78700 5 -- op tracker -- , seq: 3179, time: 2014-06-09 16:22:11.613693, event: throttled, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.798922 7f869af78700 5 -- op tracker -- , seq: 3179, time: 2014-06-09 16:22:11.613705, event: all_read, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.798928 7f8689f56700 2 waiting 51 > 50 ops || 4335027 > 104857600 2014-06-09 16:22:11.798931 7f869af78700 5 -- op tracker -- , seq: 3179, time: 2014-06-09 16:22:11.798872, event: dispatched, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.798937 7f868ff62700 2 waiting 51 > 50 ops || 4335534 > 104857600 2014-06-09 16:22:11.798939 7f869af78700 5 -- op tracker -- , seq: 3179, time: 2014-06-09 16:22:11.798936, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.798943 7f868f761700 2 waiting 51 > 50 ops || 4335628 > 104857600 2014-06-09 16:22:11.798944 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.9d) v4 from osd.4 2014-06-09 16:22:11.798947 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f0388c0 2014-06-09 16:22:11.798949 7f868cf5c700 2 waiting 51 > 50 ops || 4335027 > 104857600 2014-06-09 16:22:11.798952 7f869af78700 5 -- op tracker -- , seq: 3179, time: 2014-06-09 16:22:11.798950, event: started, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.798955 7f868bf5a700 2 waiting 51 > 50 ops || 4335027 > 104857600 2014-06-09 16:22:11.798960 7f868df5e700 2 waiting 51 > 50 ops || 4398037 > 104857600 2014-06-09 16:22:11.798968 7f8692f68700 2 waiting 51 > 50 ops || 4335252 > 104857600 2014-06-09 16:22:11.798969 7f869af78700 5 -- op tracker -- , seq: 3179, time: 2014-06-09 16:22:11.798966, event: done, op: pg_info(1 pgs e243:0.9d) 2014-06-09 16:22:11.798974 7f868e75f700 2 waiting 51 > 50 ops || 4852579 > 104857600 2014-06-09 16:22:11.798976 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798978 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798991 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.798992 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.798994 7f869af78700 20 osd.26 243 _dispatch 0x30c90380 pg_info(1 pgs e243:2.1e3) v4 2014-06-09 16:22:11.798999 7f869af78700 5 -- op tracker -- , seq: 3180, time: 2014-06-09 16:22:11.746710, event: header_read, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.799008 7f869af78700 5 -- op tracker -- , seq: 3180, time: 2014-06-09 16:22:11.746710, event: throttled, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.799015 7f869af78700 5 -- op tracker -- , seq: 3180, time: 2014-06-09 16:22:11.746722, event: all_read, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.799021 7f869af78700 5 -- op tracker -- , seq: 3180, time: 2014-06-09 16:22:11.798990, event: dispatched, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.799035 7f869af78700 5 -- op tracker -- , seq: 3180, time: 2014-06-09 16:22:11.799033, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.799039 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1e3) v4 from osd.3 2014-06-09 16:22:11.799042 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c90380 2014-06-09 16:22:11.799047 7f869af78700 5 -- op tracker -- , seq: 3180, time: 2014-06-09 16:22:11.799045, event: started, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.799060 7f869af78700 5 -- op tracker -- , seq: 3180, time: 2014-06-09 16:22:11.799058, event: done, op: pg_info(1 pgs e243:2.1e3) 2014-06-09 16:22:11.799066 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799068 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799079 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799080 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799081 7f869af78700 20 osd.26 243 _dispatch 0x30adac40 pg_info(1 pgs e243:3.55d) v4 2014-06-09 16:22:11.799086 7f869af78700 5 -- op tracker -- , seq: 3181, time: 2014-06-09 16:22:11.702497, event: header_read, op: pg_info(1 pgs e243:3.55d) 2014-06-09 16:22:11.799092 7f869af78700 5 -- op tracker -- , seq: 3181, time: 2014-06-09 16:22:11.702498, event: throttled, op: pg_info(1 pgs e243:3.55d) 2014-06-09 16:22:11.799097 7f869af78700 5 -- op tracker -- , seq: 3181, time: 2014-06-09 16:22:11.702513, event: all_read, op: pg_info(1 pgs e243:3.55d) 2014-06-09 16:22:11.799103 7f869af78700 5 -- op tracker -- , seq: 3181, time: 2014-06-09 16:22:11.799079, event: dispatched, op: pg_info(1 pgs e243:3.55d) 2014-06-09 16:22:11.799111 7f869af78700 5 -- op tracker -- , seq: 3181, time: 2014-06-09 16:22:11.799109, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.55d) 2014-06-09 16:22:11.799115 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.55d) v4 from osd.1 2014-06-09 16:22:11.799118 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adac40 2014-06-09 16:22:11.799123 7f869af78700 5 -- op tracker -- , seq: 3181, time: 2014-06-09 16:22:11.799121, event: started, op: pg_info(1 pgs e243:3.55d) 2014-06-09 16:22:11.799141 7f869af78700 5 -- op tracker -- , seq: 3181, time: 2014-06-09 16:22:11.799139, event: done, op: pg_info(1 pgs e243:3.55d) 2014-06-09 16:22:11.799151 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799154 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799167 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799169 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799170 7f869af78700 20 osd.26 243 _dispatch 0xcd79c00 pg_info(1 pgs e243:3.381) v4 2014-06-09 16:22:11.799175 7f869af78700 5 -- op tracker -- , seq: 3182, time: 2014-06-09 16:22:11.646332, event: header_read, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.799182 7f869af78700 5 -- op tracker -- , seq: 3182, time: 2014-06-09 16:22:11.646332, event: throttled, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.799189 7f869af78700 5 -- op tracker -- , seq: 3182, time: 2014-06-09 16:22:11.646343, event: all_read, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.799195 7f869af78700 5 -- op tracker -- , seq: 3182, time: 2014-06-09 16:22:11.799167, event: dispatched, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.799203 7f869af78700 5 -- op tracker -- , seq: 3182, time: 2014-06-09 16:22:11.799201, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.799210 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.381) v4 from osd.8 2014-06-09 16:22:11.799214 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd79c00 2014-06-09 16:22:11.799219 7f869af78700 5 -- op tracker -- , seq: 3182, time: 2014-06-09 16:22:11.799217, event: started, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.799236 7f869af78700 5 -- op tracker -- , seq: 3182, time: 2014-06-09 16:22:11.799233, event: done, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.799243 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799245 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799258 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799260 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799262 7f869af78700 20 osd.26 243 _dispatch 0x30c90a80 pg_info(1 pgs e243:3.688) v4 2014-06-09 16:22:11.799266 7f869af78700 5 -- op tracker -- , seq: 3183, time: 2014-06-09 16:22:11.750667, event: header_read, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.799275 7f869af78700 5 -- op tracker -- , seq: 3183, time: 2014-06-09 16:22:11.750668, event: throttled, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.799282 7f869af78700 5 -- op tracker -- , seq: 3183, time: 2014-06-09 16:22:11.750681, event: all_read, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.799289 7f869af78700 5 -- op tracker -- , seq: 3183, time: 2014-06-09 16:22:11.799258, event: dispatched, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.799298 7f869af78700 5 -- op tracker -- , seq: 3183, time: 2014-06-09 16:22:11.799295, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.799303 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.688) v4 from osd.6 2014-06-09 16:22:11.799306 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c90a80 2014-06-09 16:22:11.799311 7f869af78700 5 -- op tracker -- , seq: 3183, time: 2014-06-09 16:22:11.799309, event: started, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.799333 7f869af78700 5 -- op tracker -- , seq: 3183, time: 2014-06-09 16:22:11.799330, event: done, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.799346 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799348 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799361 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799363 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799365 7f869af78700 20 osd.26 243 _dispatch 0x30c00540 pg_info(1 pgs e243:2.624) v4 2014-06-09 16:22:11.799370 7f869af78700 5 -- op tracker -- , seq: 3184, time: 2014-06-09 16:22:11.747331, event: header_read, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.799377 7f869af78700 5 -- op tracker -- , seq: 3184, time: 2014-06-09 16:22:11.747332, event: throttled, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.799389 7f869af78700 5 -- op tracker -- , seq: 3184, time: 2014-06-09 16:22:11.747341, event: all_read, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.799399 7f869af78700 5 -- op tracker -- , seq: 3184, time: 2014-06-09 16:22:11.799361, event: dispatched, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.799410 7f869af78700 5 -- op tracker -- , seq: 3184, time: 2014-06-09 16:22:11.799408, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.799415 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.624) v4 from osd.5 2014-06-09 16:22:11.799418 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c00540 2014-06-09 16:22:11.799423 7f869af78700 5 -- op tracker -- , seq: 3184, time: 2014-06-09 16:22:11.799421, event: started, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.799438 7f869af78700 5 -- op tracker -- , seq: 3184, time: 2014-06-09 16:22:11.799436, event: done, op: pg_info(1 pgs e243:2.624) 2014-06-09 16:22:11.799445 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799448 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799460 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799462 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799463 7f869af78700 20 osd.26 243 _dispatch 0x30da3340 pg_info(1 pgs e243:1.b5) v4 2014-06-09 16:22:11.799468 7f869af78700 5 -- op tracker -- , seq: 3185, time: 2014-06-09 16:22:11.613212, event: header_read, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.799475 7f869af78700 5 -- op tracker -- , seq: 3185, time: 2014-06-09 16:22:11.613213, event: throttled, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.799482 7f869af78700 5 -- op tracker -- , seq: 3185, time: 2014-06-09 16:22:11.613233, event: all_read, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.799489 7f869af78700 5 -- op tracker -- , seq: 3185, time: 2014-06-09 16:22:11.799459, event: dispatched, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.799496 7f869af78700 5 -- op tracker -- , seq: 3185, time: 2014-06-09 16:22:11.799494, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.799501 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.b5) v4 from osd.2 2014-06-09 16:22:11.799504 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da3340 2014-06-09 16:22:11.799509 7f869af78700 5 -- op tracker -- , seq: 3185, time: 2014-06-09 16:22:11.799507, event: started, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.799524 7f869af78700 5 -- op tracker -- , seq: 3185, time: 2014-06-09 16:22:11.799522, event: done, op: pg_info(1 pgs e243:1.b5) 2014-06-09 16:22:11.799531 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799533 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799544 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799546 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799548 7f869af78700 20 osd.26 243 _dispatch 0x2f038700 pg_info(1 pgs e243:1.c9) v4 2014-06-09 16:22:11.799552 7f869af78700 5 -- op tracker -- , seq: 3186, time: 2014-06-09 16:22:11.613885, event: header_read, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.799562 7f869af78700 5 -- op tracker -- , seq: 3186, time: 2014-06-09 16:22:11.613885, event: throttled, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.799568 7f869af78700 5 -- op tracker -- , seq: 3186, time: 2014-06-09 16:22:11.613894, event: all_read, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.799574 7f869af78700 5 -- op tracker -- , seq: 3186, time: 2014-06-09 16:22:11.799544, event: dispatched, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.799583 7f869af78700 5 -- op tracker -- , seq: 3186, time: 2014-06-09 16:22:11.799581, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.799588 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.c9) v4 from osd.4 2014-06-09 16:22:11.799591 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f038700 2014-06-09 16:22:11.799596 7f869af78700 5 -- op tracker -- , seq: 3186, time: 2014-06-09 16:22:11.799594, event: started, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.799618 7f869af78700 5 -- op tracker -- , seq: 3186, time: 2014-06-09 16:22:11.799611, event: done, op: pg_info(1 pgs e243:1.c9) 2014-06-09 16:22:11.799626 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799628 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799640 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799642 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799643 7f869af78700 20 osd.26 243 _dispatch 0x30c91c00 pg_info(1 pgs e243:2.2d9) v4 2014-06-09 16:22:11.799648 7f869af78700 5 -- op tracker -- , seq: 3187, time: 2014-06-09 16:22:11.747642, event: header_read, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.799655 7f869af78700 5 -- op tracker -- , seq: 3187, time: 2014-06-09 16:22:11.747643, event: throttled, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.799662 7f869af78700 5 -- op tracker -- , seq: 3187, time: 2014-06-09 16:22:11.747658, event: all_read, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.799671 7f869af78700 5 -- op tracker -- , seq: 3187, time: 2014-06-09 16:22:11.799639, event: dispatched, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.799678 7f869af78700 5 -- op tracker -- , seq: 3187, time: 2014-06-09 16:22:11.799676, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.799682 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2d9) v4 from osd.3 2014-06-09 16:22:11.799685 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c91c00 2014-06-09 16:22:11.799690 7f869af78700 5 -- op tracker -- , seq: 3187, time: 2014-06-09 16:22:11.799688, event: started, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.799708 7f869af78700 5 -- op tracker -- , seq: 3187, time: 2014-06-09 16:22:11.799706, event: done, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.799715 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799717 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799729 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799731 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799732 7f869af78700 20 osd.26 243 _dispatch 0x30adb180 pg_info(1 pgs e243:3.75f) v4 2014-06-09 16:22:11.799737 7f869af78700 5 -- op tracker -- , seq: 3188, time: 2014-06-09 16:22:11.709730, event: header_read, op: pg_info(1 pgs e243:3.75f) 2014-06-09 16:22:11.799744 7f869af78700 5 -- op tracker -- , seq: 3188, time: 2014-06-09 16:22:11.709731, event: throttled, op: pg_info(1 pgs e243:3.75f) 2014-06-09 16:22:11.799754 7f869af78700 5 -- op tracker -- , seq: 3188, time: 2014-06-09 16:22:11.709743, event: all_read, op: pg_info(1 pgs e243:3.75f) 2014-06-09 16:22:11.799760 7f869af78700 5 -- op tracker -- , seq: 3188, time: 2014-06-09 16:22:11.799729, event: dispatched, op: pg_info(1 pgs e243:3.75f) 2014-06-09 16:22:11.799767 7f869af78700 5 -- op tracker -- , seq: 3188, time: 2014-06-09 16:22:11.799765, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.75f) 2014-06-09 16:22:11.799777 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.75f) v4 from osd.1 2014-06-09 16:22:11.799780 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adb180 2014-06-09 16:22:11.799785 7f869af78700 5 -- op tracker -- , seq: 3188, time: 2014-06-09 16:22:11.799783, event: started, op: pg_info(1 pgs e243:3.75f) 2014-06-09 16:22:11.799804 7f869af78700 5 -- op tracker -- , seq: 3188, time: 2014-06-09 16:22:11.799802, event: done, op: pg_info(1 pgs e243:3.75f) 2014-06-09 16:22:11.799813 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799815 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799827 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.799829 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.799831 7f869af78700 20 osd.26 243 _dispatch 0x2d095500 pg_info(1 pgs e243:3.3a8) v4 2014-06-09 16:22:11.799836 7f869af78700 5 -- op tracker -- , seq: 3189, time: 2014-06-09 16:22:11.646347, event: header_read, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.799843 7f869af78700 5 -- op tracker -- , seq: 3189, time: 2014-06-09 16:22:11.646348, event: throttled, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.799849 7f869af78700 5 -- op tracker -- , seq: 3189, time: 2014-06-09 16:22:11.646358, event: all_read, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.799856 7f869af78700 5 -- op tracker -- , seq: 3189, time: 2014-06-09 16:22:11.799827, event: dispatched, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.799868 7f869af78700 5 -- op tracker -- , seq: 3189, time: 2014-06-09 16:22:11.799866, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.799873 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.3a8) v4 from osd.8 2014-06-09 16:22:11.800709 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2d095500 2014-06-09 16:22:11.800720 7f869af78700 5 -- op tracker -- , seq: 3189, time: 2014-06-09 16:22:11.800717, event: started, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.800742 7f869af78700 5 -- op tracker -- , seq: 3189, time: 2014-06-09 16:22:11.800739, event: done, op: pg_info(1 pgs e243:3.3a8) 2014-06-09 16:22:11.800754 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.800756 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.800771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.800772 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.800774 7f869af78700 20 osd.26 243 _dispatch 0x30c90e00 pg_info(1 pgs e243:3.13c) v4 2014-06-09 16:22:11.800779 7f869af78700 5 -- op tracker -- , seq: 3190, time: 2014-06-09 16:22:11.750713, event: header_read, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.800786 7f869af78700 5 -- op tracker -- , seq: 3190, time: 2014-06-09 16:22:11.750713, event: throttled, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.800793 7f869af78700 5 -- op tracker -- , seq: 3190, time: 2014-06-09 16:22:11.750725, event: all_read, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.800799 7f869af78700 5 -- op tracker -- , seq: 3190, time: 2014-06-09 16:22:11.800770, event: dispatched, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.800812 7f869af78700 5 -- op tracker -- , seq: 3190, time: 2014-06-09 16:22:11.800810, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.800817 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.13c) v4 from osd.6 2014-06-09 16:22:11.800820 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c90e00 2014-06-09 16:22:11.800825 7f869af78700 5 -- op tracker -- , seq: 3190, time: 2014-06-09 16:22:11.800823, event: started, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.800842 7f869af78700 5 -- op tracker -- , seq: 3190, time: 2014-06-09 16:22:11.800840, event: done, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.800851 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.800853 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.800866 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.800868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.800869 7f869af78700 20 osd.26 243 _dispatch 0x30da3180 pg_info(1 pgs e243:2.c6) v4 2014-06-09 16:22:11.800874 7f869af78700 5 -- op tracker -- , seq: 3191, time: 2014-06-09 16:22:11.613239, event: header_read, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.800884 7f869af78700 5 -- op tracker -- , seq: 3191, time: 2014-06-09 16:22:11.613239, event: throttled, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.800891 7f869af78700 5 -- op tracker -- , seq: 3191, time: 2014-06-09 16:22:11.613263, event: all_read, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.800897 7f869af78700 5 -- op tracker -- , seq: 3191, time: 2014-06-09 16:22:11.800866, event: dispatched, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.800906 7f869af78700 5 -- op tracker -- , seq: 3191, time: 2014-06-09 16:22:11.800904, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.800911 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.c6) v4 from osd.2 2014-06-09 16:22:11.800914 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da3180 2014-06-09 16:22:11.800919 7f869af78700 5 -- op tracker -- , seq: 3191, time: 2014-06-09 16:22:11.800917, event: started, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.800941 7f869af78700 5 -- op tracker -- , seq: 3191, time: 2014-06-09 16:22:11.800938, event: done, op: pg_info(1 pgs e243:2.c6) 2014-06-09 16:22:11.800948 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.800950 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.800961 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.800963 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.800964 7f869af78700 20 osd.26 243 _dispatch 0x30de4380 pg_info(1 pgs e243:2.236) v4 2014-06-09 16:22:11.800968 7f869af78700 5 -- op tracker -- , seq: 3192, time: 2014-06-09 16:22:11.640377, event: header_read, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.800975 7f869af78700 5 -- op tracker -- , seq: 3192, time: 2014-06-09 16:22:11.640378, event: throttled, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.800980 7f869af78700 5 -- op tracker -- , seq: 3192, time: 2014-06-09 16:22:11.640391, event: all_read, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.800986 7f869af78700 5 -- op tracker -- , seq: 3192, time: 2014-06-09 16:22:11.800961, event: dispatched, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.800992 7f869af78700 5 -- op tracker -- , seq: 3192, time: 2014-06-09 16:22:11.800990, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.800995 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.236) v4 from osd.4 2014-06-09 16:22:11.800998 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de4380 2014-06-09 16:22:11.801002 7f869af78700 5 -- op tracker -- , seq: 3192, time: 2014-06-09 16:22:11.801000, event: started, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801019 7f869af78700 5 -- op tracker -- , seq: 3192, time: 2014-06-09 16:22:11.801016, event: done, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801032 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801034 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801047 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801050 7f869af78700 20 osd.26 243 _dispatch 0x31ac3340 pg_info(1 pgs e243:1.1cf) v4 2014-06-09 16:22:11.801055 7f869af78700 5 -- op tracker -- , seq: 3193, time: 2014-06-09 16:22:11.747724, event: header_read, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.801066 7f869af78700 5 -- op tracker -- , seq: 3193, time: 2014-06-09 16:22:11.747730, event: throttled, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.801073 7f869af78700 5 -- op tracker -- , seq: 3193, time: 2014-06-09 16:22:11.747742, event: all_read, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.801082 7f869af78700 5 -- op tracker -- , seq: 3193, time: 2014-06-09 16:22:11.801047, event: dispatched, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.801089 7f869af78700 5 -- op tracker -- , seq: 3193, time: 2014-06-09 16:22:11.801087, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.801093 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1cf) v4 from osd.3 2014-06-09 16:22:11.801096 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31ac3340 2014-06-09 16:22:11.801101 7f869af78700 5 -- op tracker -- , seq: 3193, time: 2014-06-09 16:22:11.801099, event: started, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.801117 7f869af78700 5 -- op tracker -- , seq: 3193, time: 2014-06-09 16:22:11.801115, event: done, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.801125 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801126 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801138 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801141 7f869af78700 20 osd.26 243 _dispatch 0x1ac74700 pg_info(1 pgs e243:3.76f) v4 2014-06-09 16:22:11.801146 7f869af78700 5 -- op tracker -- , seq: 3194, time: 2014-06-09 16:22:11.743845, event: header_read, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801153 7f869af78700 5 -- op tracker -- , seq: 3194, time: 2014-06-09 16:22:11.743846, event: throttled, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801159 7f869af78700 5 -- op tracker -- , seq: 3194, time: 2014-06-09 16:22:11.743858, event: all_read, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801168 7f869af78700 5 -- op tracker -- , seq: 3194, time: 2014-06-09 16:22:11.801138, event: dispatched, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801175 7f869af78700 5 -- op tracker -- , seq: 3194, time: 2014-06-09 16:22:11.801173, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801179 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.76f) v4 from osd.1 2014-06-09 16:22:11.801182 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac74700 2014-06-09 16:22:11.801187 7f869af78700 5 -- op tracker -- , seq: 3194, time: 2014-06-09 16:22:11.801185, event: started, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801203 7f869af78700 5 -- op tracker -- , seq: 3194, time: 2014-06-09 16:22:11.801201, event: done, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801210 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801212 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801224 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801226 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801227 7f869af78700 20 osd.26 243 _dispatch 0x31389a40 pg_info(1 pgs e243:3.381) v4 2014-06-09 16:22:11.801232 7f869af78700 5 -- op tracker -- , seq: 3195, time: 2014-06-09 16:22:11.646363, event: header_read, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.801238 7f869af78700 5 -- op tracker -- , seq: 3195, time: 2014-06-09 16:22:11.646363, event: throttled, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.801245 7f869af78700 5 -- op tracker -- , seq: 3195, time: 2014-06-09 16:22:11.646375, event: all_read, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.801252 7f869af78700 5 -- op tracker -- , seq: 3195, time: 2014-06-09 16:22:11.801224, event: dispatched, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.801258 7f869af78700 5 -- op tracker -- , seq: 3195, time: 2014-06-09 16:22:11.801256, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.801269 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.381) v4 from osd.8 2014-06-09 16:22:11.801272 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31389a40 2014-06-09 16:22:11.801277 7f869af78700 5 -- op tracker -- , seq: 3195, time: 2014-06-09 16:22:11.801275, event: started, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.801295 7f869af78700 5 -- op tracker -- , seq: 3195, time: 2014-06-09 16:22:11.801293, event: done, op: pg_info(1 pgs e243:3.381) 2014-06-09 16:22:11.801303 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801305 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801317 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801318 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801320 7f869af78700 20 osd.26 243 _dispatch 0xddc4a80 pg_info(1 pgs e243:3.753) v4 2014-06-09 16:22:11.801325 7f869af78700 5 -- op tracker -- , seq: 3196, time: 2014-06-09 16:22:11.751033, event: header_read, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.801331 7f869af78700 5 -- op tracker -- , seq: 3196, time: 2014-06-09 16:22:11.751033, event: throttled, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.801338 7f869af78700 5 -- op tracker -- , seq: 3196, time: 2014-06-09 16:22:11.751046, event: all_read, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.801347 7f869af78700 5 -- op tracker -- , seq: 3196, time: 2014-06-09 16:22:11.801316, event: dispatched, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.801355 7f869af78700 5 -- op tracker -- , seq: 3196, time: 2014-06-09 16:22:11.801352, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.801359 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.753) v4 from osd.6 2014-06-09 16:22:11.801362 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc4a80 2014-06-09 16:22:11.801367 7f869af78700 5 -- op tracker -- , seq: 3196, time: 2014-06-09 16:22:11.801365, event: started, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.801383 7f869af78700 5 -- op tracker -- , seq: 3196, time: 2014-06-09 16:22:11.801380, event: done, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.801390 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801392 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801407 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801408 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801410 7f869af78700 20 osd.26 243 _dispatch 0x30de4000 pg_info(1 pgs e243:0.25d) v4 2014-06-09 16:22:11.801415 7f869af78700 5 -- op tracker -- , seq: 3197, time: 2014-06-09 16:22:11.639817, event: header_read, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801421 7f869af78700 5 -- op tracker -- , seq: 3197, time: 2014-06-09 16:22:11.639818, event: throttled, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801428 7f869af78700 5 -- op tracker -- , seq: 3197, time: 2014-06-09 16:22:11.639835, event: all_read, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801434 7f869af78700 5 -- op tracker -- , seq: 3197, time: 2014-06-09 16:22:11.801406, event: dispatched, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801443 7f869af78700 5 -- op tracker -- , seq: 3197, time: 2014-06-09 16:22:11.801441, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801448 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.25d) v4 from osd.2 2014-06-09 16:22:11.801450 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de4000 2014-06-09 16:22:11.801455 7f869af78700 5 -- op tracker -- , seq: 3197, time: 2014-06-09 16:22:11.801453, event: started, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801473 7f869af78700 5 -- op tracker -- , seq: 3197, time: 2014-06-09 16:22:11.801471, event: done, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801483 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801495 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801497 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801498 7f869af78700 20 osd.26 243 _dispatch 0x2f038540 pg_info(1 pgs e243:2.236) v4 2014-06-09 16:22:11.801503 7f869af78700 5 -- op tracker -- , seq: 3198, time: 2014-06-09 16:22:11.640475, event: header_read, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801509 7f869af78700 5 -- op tracker -- , seq: 3198, time: 2014-06-09 16:22:11.640476, event: throttled, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801521 7f869af78700 5 -- op tracker -- , seq: 3198, time: 2014-06-09 16:22:11.640491, event: all_read, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801532 7f869af78700 5 -- op tracker -- , seq: 3198, time: 2014-06-09 16:22:11.801495, event: dispatched, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801539 7f869af78700 5 -- op tracker -- , seq: 3198, time: 2014-06-09 16:22:11.801537, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801544 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.236) v4 from osd.4 2014-06-09 16:22:11.801546 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f038540 2014-06-09 16:22:11.801551 7f869af78700 5 -- op tracker -- , seq: 3198, time: 2014-06-09 16:22:11.801550, event: started, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801569 7f869af78700 5 -- op tracker -- , seq: 3198, time: 2014-06-09 16:22:11.801566, event: done, op: pg_info(1 pgs e243:2.236) 2014-06-09 16:22:11.801576 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801578 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801595 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801596 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801598 7f869af78700 20 osd.26 243 _dispatch 0x31ac3c00 pg_info(1 pgs e243:2.2d9) v4 2014-06-09 16:22:11.801603 7f869af78700 5 -- op tracker -- , seq: 3199, time: 2014-06-09 16:22:11.747749, event: header_read, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.801610 7f869af78700 5 -- op tracker -- , seq: 3199, time: 2014-06-09 16:22:11.747750, event: throttled, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.801617 7f869af78700 5 -- op tracker -- , seq: 3199, time: 2014-06-09 16:22:11.747761, event: all_read, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.801623 7f869af78700 5 -- op tracker -- , seq: 3199, time: 2014-06-09 16:22:11.801594, event: dispatched, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.801630 7f869af78700 5 -- op tracker -- , seq: 3199, time: 2014-06-09 16:22:11.801628, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.801640 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.2d9) v4 from osd.3 2014-06-09 16:22:11.801643 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31ac3c00 2014-06-09 16:22:11.801648 7f869af78700 5 -- op tracker -- , seq: 3199, time: 2014-06-09 16:22:11.801646, event: started, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.801665 7f869af78700 5 -- op tracker -- , seq: 3199, time: 2014-06-09 16:22:11.801663, event: done, op: pg_info(1 pgs e243:2.2d9) 2014-06-09 16:22:11.801673 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801675 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801689 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801690 7f869af78700 20 osd.26 243 _dispatch 0x31ac3180 pg_info(1 pgs e243:3.76f) v4 2014-06-09 16:22:11.801695 7f869af78700 5 -- op tracker -- , seq: 3200, time: 2014-06-09 16:22:11.743946, event: header_read, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801701 7f869af78700 5 -- op tracker -- , seq: 3200, time: 2014-06-09 16:22:11.743946, event: throttled, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801710 7f869af78700 5 -- op tracker -- , seq: 3200, time: 2014-06-09 16:22:11.743967, event: all_read, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801717 7f869af78700 5 -- op tracker -- , seq: 3200, time: 2014-06-09 16:22:11.801687, event: dispatched, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801724 7f869af78700 5 -- op tracker -- , seq: 3200, time: 2014-06-09 16:22:11.801722, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801733 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.76f) v4 from osd.1 2014-06-09 16:22:11.801736 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31ac3180 2014-06-09 16:22:11.801740 7f869af78700 5 -- op tracker -- , seq: 3200, time: 2014-06-09 16:22:11.801739, event: started, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801756 7f869af78700 5 -- op tracker -- , seq: 3200, time: 2014-06-09 16:22:11.801754, event: done, op: pg_info(1 pgs e243:3.76f) 2014-06-09 16:22:11.801767 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801769 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801781 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801783 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801784 7f869af78700 20 osd.26 243 _dispatch 0x2fc56e00 pg_info(1 pgs e243:2.24d) v4 2014-06-09 16:22:11.801789 7f869af78700 5 -- op tracker -- , seq: 3201, time: 2014-06-09 16:22:11.652447, event: header_read, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.801796 7f869af78700 5 -- op tracker -- , seq: 3201, time: 2014-06-09 16:22:11.652447, event: throttled, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.801804 7f869af78700 5 -- op tracker -- , seq: 3201, time: 2014-06-09 16:22:11.652455, event: all_read, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.801811 7f869af78700 5 -- op tracker -- , seq: 3201, time: 2014-06-09 16:22:11.801781, event: dispatched, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.801818 7f869af78700 5 -- op tracker -- , seq: 3201, time: 2014-06-09 16:22:11.801816, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.801823 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.24d) v4 from osd.8 2014-06-09 16:22:11.801826 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc56e00 2014-06-09 16:22:11.801831 7f869af78700 5 -- op tracker -- , seq: 3201, time: 2014-06-09 16:22:11.801829, event: started, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.801847 7f869af78700 5 -- op tracker -- , seq: 3201, time: 2014-06-09 16:22:11.801844, event: done, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.801854 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801856 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801868 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801870 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801871 7f869af78700 20 osd.26 243 _dispatch 0x30adaa80 pg_info(1 pgs e243:3.577) v4 2014-06-09 16:22:11.801876 7f869af78700 5 -- op tracker -- , seq: 3202, time: 2014-06-09 16:22:11.751462, event: header_read, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.801885 7f869af78700 5 -- op tracker -- , seq: 3202, time: 2014-06-09 16:22:11.751463, event: throttled, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.801892 7f869af78700 5 -- op tracker -- , seq: 3202, time: 2014-06-09 16:22:11.751475, event: all_read, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.801898 7f869af78700 5 -- op tracker -- , seq: 3202, time: 2014-06-09 16:22:11.801868, event: dispatched, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.801908 7f869af78700 5 -- op tracker -- , seq: 3202, time: 2014-06-09 16:22:11.801906, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.801913 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.577) v4 from osd.6 2014-06-09 16:22:11.801916 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adaa80 2014-06-09 16:22:11.801921 7f869af78700 5 -- op tracker -- , seq: 3202, time: 2014-06-09 16:22:11.801919, event: started, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.801938 7f869af78700 5 -- op tracker -- , seq: 3202, time: 2014-06-09 16:22:11.801935, event: done, op: pg_info(1 pgs e243:3.577) 2014-06-09 16:22:11.801946 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801948 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801959 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.801961 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.801962 7f869af78700 20 osd.26 243 _dispatch 0x30da2fc0 pg_info(1 pgs e243:0.25d) v4 2014-06-09 16:22:11.801967 7f869af78700 5 -- op tracker -- , seq: 3203, time: 2014-06-09 16:22:11.640093, event: header_read, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801974 7f869af78700 5 -- op tracker -- , seq: 3203, time: 2014-06-09 16:22:11.640094, event: throttled, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801981 7f869af78700 5 -- op tracker -- , seq: 3203, time: 2014-06-09 16:22:11.640111, event: all_read, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801988 7f869af78700 5 -- op tracker -- , seq: 3203, time: 2014-06-09 16:22:11.801959, event: dispatched, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.801995 7f869af78700 5 -- op tracker -- , seq: 3203, time: 2014-06-09 16:22:11.801993, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.802000 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.25d) v4 from osd.2 2014-06-09 16:22:11.802003 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da2fc0 2014-06-09 16:22:11.802008 7f869af78700 5 -- op tracker -- , seq: 3203, time: 2014-06-09 16:22:11.802006, event: started, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.802026 7f869af78700 5 -- op tracker -- , seq: 3203, time: 2014-06-09 16:22:11.802023, event: done, op: pg_info(1 pgs e243:0.25d) 2014-06-09 16:22:11.802036 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802038 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802050 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802051 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802053 7f869af78700 20 osd.26 243 _dispatch 0xcd78e00 pg_info(1 pgs e243:1.375) v4 2014-06-09 16:22:11.802058 7f869af78700 5 -- op tracker -- , seq: 3204, time: 2014-06-09 16:22:11.653117, event: header_read, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802065 7f869af78700 5 -- op tracker -- , seq: 3204, time: 2014-06-09 16:22:11.653118, event: throttled, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802078 7f869af78700 5 -- op tracker -- , seq: 3204, time: 2014-06-09 16:22:11.653133, event: all_read, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802084 7f869af78700 5 -- op tracker -- , seq: 3204, time: 2014-06-09 16:22:11.802049, event: dispatched, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802096 7f869af78700 5 -- op tracker -- , seq: 3204, time: 2014-06-09 16:22:11.802094, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802101 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.375) v4 from osd.4 2014-06-09 16:22:11.802104 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd78e00 2014-06-09 16:22:11.802108 7f869af78700 5 -- op tracker -- , seq: 3204, time: 2014-06-09 16:22:11.802106, event: started, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802124 7f869af78700 5 -- op tracker -- , seq: 3204, time: 2014-06-09 16:22:11.802122, event: done, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802132 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802134 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802147 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802148 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802150 7f869af78700 20 osd.26 243 _dispatch 0x30c77dc0 pg_info(1 pgs e243:1.1cf) v4 2014-06-09 16:22:11.802155 7f869af78700 5 -- op tracker -- , seq: 3205, time: 2014-06-09 16:22:11.747809, event: header_read, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.802162 7f869af78700 5 -- op tracker -- , seq: 3205, time: 2014-06-09 16:22:11.747810, event: throttled, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.802168 7f869af78700 5 -- op tracker -- , seq: 3205, time: 2014-06-09 16:22:11.747823, event: all_read, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.802175 7f869af78700 5 -- op tracker -- , seq: 3205, time: 2014-06-09 16:22:11.802146, event: dispatched, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.802182 7f869af78700 5 -- op tracker -- , seq: 3205, time: 2014-06-09 16:22:11.802180, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.802187 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1cf) v4 from osd.3 2014-06-09 16:22:11.802190 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c77dc0 2014-06-09 16:22:11.802194 7f869af78700 5 -- op tracker -- , seq: 3205, time: 2014-06-09 16:22:11.802193, event: started, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.802210 7f869af78700 5 -- op tracker -- , seq: 3205, time: 2014-06-09 16:22:11.802208, event: done, op: pg_info(1 pgs e243:1.1cf) 2014-06-09 16:22:11.802220 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802222 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802234 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802236 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802237 7f869af78700 20 osd.26 243 _dispatch 0x30c90540 pg_info(1 pgs e243:1.a) v4 2014-06-09 16:22:11.802242 7f869af78700 5 -- op tracker -- , seq: 3206, time: 2014-06-09 16:22:11.745315, event: header_read, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802249 7f869af78700 5 -- op tracker -- , seq: 3206, time: 2014-06-09 16:22:11.745315, event: throttled, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802256 7f869af78700 5 -- op tracker -- , seq: 3206, time: 2014-06-09 16:22:11.745326, event: all_read, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802263 7f869af78700 5 -- op tracker -- , seq: 3206, time: 2014-06-09 16:22:11.802234, event: dispatched, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802273 7f869af78700 5 -- op tracker -- , seq: 3206, time: 2014-06-09 16:22:11.802271, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802277 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a) v4 from osd.1 2014-06-09 16:22:11.802280 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c90540 2014-06-09 16:22:11.802285 7f869af78700 5 -- op tracker -- , seq: 3206, time: 2014-06-09 16:22:11.802283, event: started, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802306 7f869af78700 5 -- op tracker -- , seq: 3206, time: 2014-06-09 16:22:11.802304, event: done, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802314 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802316 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802329 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802330 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802332 7f869af78700 20 osd.26 243 _dispatch 0x2fc576c0 pg_info(1 pgs e243:3.5f8) v4 2014-06-09 16:22:11.802337 7f869af78700 5 -- op tracker -- , seq: 3207, time: 2014-06-09 16:22:11.652550, event: header_read, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.802346 7f869af78700 5 -- op tracker -- , seq: 3207, time: 2014-06-09 16:22:11.652551, event: throttled, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.802352 7f869af78700 5 -- op tracker -- , seq: 3207, time: 2014-06-09 16:22:11.652561, event: all_read, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.802359 7f869af78700 5 -- op tracker -- , seq: 3207, time: 2014-06-09 16:22:11.802328, event: dispatched, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.802368 7f869af78700 5 -- op tracker -- , seq: 3207, time: 2014-06-09 16:22:11.802366, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.802373 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5f8) v4 from osd.8 2014-06-09 16:22:11.802376 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc576c0 2014-06-09 16:22:11.802381 7f869af78700 5 -- op tracker -- , seq: 3207, time: 2014-06-09 16:22:11.802379, event: started, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.802398 7f869af78700 5 -- op tracker -- , seq: 3207, time: 2014-06-09 16:22:11.802396, event: done, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.802405 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802407 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802418 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802421 7f869af78700 20 osd.26 243 _dispatch 0x30adb340 pg_info(1 pgs e243:2.213) v4 2014-06-09 16:22:11.802426 7f869af78700 5 -- op tracker -- , seq: 3208, time: 2014-06-09 16:22:11.751509, event: header_read, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.802434 7f869af78700 5 -- op tracker -- , seq: 3208, time: 2014-06-09 16:22:11.751510, event: throttled, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.802441 7f869af78700 5 -- op tracker -- , seq: 3208, time: 2014-06-09 16:22:11.751521, event: all_read, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.802447 7f869af78700 5 -- op tracker -- , seq: 3208, time: 2014-06-09 16:22:11.802418, event: dispatched, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.802454 7f869af78700 5 -- op tracker -- , seq: 3208, time: 2014-06-09 16:22:11.802452, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.802459 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.213) v4 from osd.6 2014-06-09 16:22:11.802462 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adb340 2014-06-09 16:22:11.802467 7f869af78700 5 -- op tracker -- , seq: 3208, time: 2014-06-09 16:22:11.802465, event: started, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.802482 7f869af78700 5 -- op tracker -- , seq: 3208, time: 2014-06-09 16:22:11.802479, event: done, op: pg_info(1 pgs e243:2.213) 2014-06-09 16:22:11.802491 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802494 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802506 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802507 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802509 7f869af78700 20 osd.26 243 _dispatch 0x3130b340 pg_info(1 pgs e243:1.508) v4 2014-06-09 16:22:11.802514 7f869af78700 5 -- op tracker -- , seq: 3209, time: 2014-06-09 16:22:11.644668, event: header_read, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.802521 7f869af78700 5 -- op tracker -- , seq: 3209, time: 2014-06-09 16:22:11.644668, event: throttled, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.802528 7f869af78700 5 -- op tracker -- , seq: 3209, time: 2014-06-09 16:22:11.644683, event: all_read, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.802534 7f869af78700 5 -- op tracker -- , seq: 3209, time: 2014-06-09 16:22:11.802505, event: dispatched, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.802545 7f869af78700 5 -- op tracker -- , seq: 3209, time: 2014-06-09 16:22:11.802542, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.802550 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.508) v4 from osd.2 2014-06-09 16:22:11.802553 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130b340 2014-06-09 16:22:11.802558 7f869af78700 5 -- op tracker -- , seq: 3209, time: 2014-06-09 16:22:11.802556, event: started, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.802580 7f869af78700 5 -- op tracker -- , seq: 3209, time: 2014-06-09 16:22:11.802577, event: done, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.802587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802589 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802601 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802602 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802604 7f869af78700 20 osd.26 243 _dispatch 0x2f038380 pg_info(1 pgs e243:1.375) v4 2014-06-09 16:22:11.802609 7f869af78700 5 -- op tracker -- , seq: 3210, time: 2014-06-09 16:22:11.653204, event: header_read, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802621 7f869af78700 5 -- op tracker -- , seq: 3210, time: 2014-06-09 16:22:11.653204, event: throttled, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802628 7f869af78700 5 -- op tracker -- , seq: 3210, time: 2014-06-09 16:22:11.653219, event: all_read, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802639 7f869af78700 5 -- op tracker -- , seq: 3210, time: 2014-06-09 16:22:11.802601, event: dispatched, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802646 7f869af78700 5 -- op tracker -- , seq: 3210, time: 2014-06-09 16:22:11.802644, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802654 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.375) v4 from osd.4 2014-06-09 16:22:11.802657 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f038380 2014-06-09 16:22:11.802663 7f869af78700 5 -- op tracker -- , seq: 3210, time: 2014-06-09 16:22:11.802661, event: started, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802680 7f869af78700 5 -- op tracker -- , seq: 3210, time: 2014-06-09 16:22:11.802678, event: done, op: pg_info(1 pgs e243:1.375) 2014-06-09 16:22:11.802688 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802690 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802702 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802704 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802705 7f869af78700 20 osd.26 243 _dispatch 0x30c90c40 pg_info(1 pgs e243:1.a) v4 2014-06-09 16:22:11.802710 7f869af78700 5 -- op tracker -- , seq: 3211, time: 2014-06-09 16:22:11.745334, event: header_read, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802717 7f869af78700 5 -- op tracker -- , seq: 3211, time: 2014-06-09 16:22:11.745334, event: throttled, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802723 7f869af78700 5 -- op tracker -- , seq: 3211, time: 2014-06-09 16:22:11.745346, event: all_read, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802732 7f869af78700 5 -- op tracker -- , seq: 3211, time: 2014-06-09 16:22:11.802702, event: dispatched, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802739 7f869af78700 5 -- op tracker -- , seq: 3211, time: 2014-06-09 16:22:11.802737, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802743 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.a) v4 from osd.1 2014-06-09 16:22:11.802746 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c90c40 2014-06-09 16:22:11.802752 7f869af78700 5 -- op tracker -- , seq: 3211, time: 2014-06-09 16:22:11.802750, event: started, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802768 7f869af78700 5 -- op tracker -- , seq: 3211, time: 2014-06-09 16:22:11.802766, event: done, op: pg_info(1 pgs e243:1.a) 2014-06-09 16:22:11.802778 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802780 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802794 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802795 7f869af78700 20 osd.26 243 _dispatch 0x31389880 pg_info(1 pgs e243:2.24d) v4 2014-06-09 16:22:11.802801 7f869af78700 5 -- op tracker -- , seq: 3212, time: 2014-06-09 16:22:11.652657, event: header_read, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.802808 7f869af78700 5 -- op tracker -- , seq: 3212, time: 2014-06-09 16:22:11.652658, event: throttled, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.802819 7f869af78700 5 -- op tracker -- , seq: 3212, time: 2014-06-09 16:22:11.652668, event: all_read, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.802830 7f869af78700 5 -- op tracker -- , seq: 3212, time: 2014-06-09 16:22:11.802792, event: dispatched, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.802841 7f869af78700 5 -- op tracker -- , seq: 3212, time: 2014-06-09 16:22:11.802838, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.802845 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.24d) v4 from osd.8 2014-06-09 16:22:11.802849 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31389880 2014-06-09 16:22:11.802854 7f869af78700 5 -- op tracker -- , seq: 3212, time: 2014-06-09 16:22:11.802852, event: started, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.802868 7f869af78700 5 -- op tracker -- , seq: 3212, time: 2014-06-09 16:22:11.802866, event: done, op: pg_info(1 pgs e243:2.24d) 2014-06-09 16:22:11.802876 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802877 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802888 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802889 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802890 7f869af78700 20 osd.26 243 _dispatch 0x30ada380 pg_info(1 pgs e243:3.786) v4 2014-06-09 16:22:11.802895 7f869af78700 5 -- op tracker -- , seq: 3213, time: 2014-06-09 16:22:11.752944, event: header_read, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.802901 7f869af78700 5 -- op tracker -- , seq: 3213, time: 2014-06-09 16:22:11.752945, event: throttled, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.802907 7f869af78700 5 -- op tracker -- , seq: 3213, time: 2014-06-09 16:22:11.752958, event: all_read, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.802912 7f869af78700 5 -- op tracker -- , seq: 3213, time: 2014-06-09 16:22:11.802888, event: dispatched, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.802918 7f869af78700 5 -- op tracker -- , seq: 3213, time: 2014-06-09 16:22:11.802916, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.802922 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.786) v4 from osd.6 2014-06-09 16:22:11.802925 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ada380 2014-06-09 16:22:11.802929 7f869af78700 5 -- op tracker -- , seq: 3213, time: 2014-06-09 16:22:11.802928, event: started, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.802945 7f869af78700 5 -- op tracker -- , seq: 3213, time: 2014-06-09 16:22:11.802943, event: done, op: pg_info(1 pgs e243:3.786) 2014-06-09 16:22:11.802960 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802962 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802977 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.802979 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.802980 7f869af78700 20 osd.26 243 _dispatch 0x30da2e00 pg_info(1 pgs e243:1.508) v4 2014-06-09 16:22:11.802985 7f869af78700 5 -- op tracker -- , seq: 3214, time: 2014-06-09 16:22:11.644702, event: header_read, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.802998 7f869af78700 5 -- op tracker -- , seq: 3214, time: 2014-06-09 16:22:11.644702, event: throttled, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.803004 7f869af78700 5 -- op tracker -- , seq: 3214, time: 2014-06-09 16:22:11.644717, event: all_read, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.803013 7f869af78700 5 -- op tracker -- , seq: 3214, time: 2014-06-09 16:22:11.802976, event: dispatched, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.803020 7f869af78700 5 -- op tracker -- , seq: 3214, time: 2014-06-09 16:22:11.803018, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.803025 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.508) v4 from osd.2 2014-06-09 16:22:11.803028 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da2e00 2014-06-09 16:22:11.803032 7f869af78700 5 -- op tracker -- , seq: 3214, time: 2014-06-09 16:22:11.803030, event: started, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.803050 7f869af78700 5 -- op tracker -- , seq: 3214, time: 2014-06-09 16:22:11.803048, event: done, op: pg_info(1 pgs e243:1.508) 2014-06-09 16:22:11.803059 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803061 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803073 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803075 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803076 7f869af78700 20 osd.26 243 _dispatch 0x1ac75180 pg_info(1 pgs e243:1.302) v4 2014-06-09 16:22:11.803081 7f869af78700 5 -- op tracker -- , seq: 3215, time: 2014-06-09 16:22:11.655523, event: header_read, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.803088 7f869af78700 5 -- op tracker -- , seq: 3215, time: 2014-06-09 16:22:11.655523, event: throttled, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.803095 7f869af78700 5 -- op tracker -- , seq: 3215, time: 2014-06-09 16:22:11.655535, event: all_read, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.803101 7f869af78700 5 -- op tracker -- , seq: 3215, time: 2014-06-09 16:22:11.803073, event: dispatched, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.803109 7f869af78700 5 -- op tracker -- , seq: 3215, time: 2014-06-09 16:22:11.803106, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.803114 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.302) v4 from osd.4 2014-06-09 16:22:11.803117 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac75180 2014-06-09 16:22:11.803122 7f869af78700 5 -- op tracker -- , seq: 3215, time: 2014-06-09 16:22:11.803120, event: started, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.803138 7f869af78700 5 -- op tracker -- , seq: 3215, time: 2014-06-09 16:22:11.803136, event: done, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.803146 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803148 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803160 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803161 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803163 7f869af78700 20 osd.26 243 _dispatch 0x30c91340 pg_info(1 pgs e243:3.111) v4 2014-06-09 16:22:11.803168 7f869af78700 5 -- op tracker -- , seq: 3216, time: 2014-06-09 16:22:11.745351, event: header_read, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803177 7f869af78700 5 -- op tracker -- , seq: 3216, time: 2014-06-09 16:22:11.745352, event: throttled, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803184 7f869af78700 5 -- op tracker -- , seq: 3216, time: 2014-06-09 16:22:11.745367, event: all_read, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803190 7f869af78700 5 -- op tracker -- , seq: 3216, time: 2014-06-09 16:22:11.803159, event: dispatched, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803197 7f869af78700 5 -- op tracker -- , seq: 3216, time: 2014-06-09 16:22:11.803195, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803207 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.111) v4 from osd.1 2014-06-09 16:22:11.803210 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c91340 2014-06-09 16:22:11.803215 7f869af78700 5 -- op tracker -- , seq: 3216, time: 2014-06-09 16:22:11.803213, event: started, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803236 7f869af78700 5 -- op tracker -- , seq: 3216, time: 2014-06-09 16:22:11.803234, event: done, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803243 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803246 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803257 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803259 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803260 7f869af78700 20 osd.26 243 _dispatch 0x313896c0 pg_info(1 pgs e243:3.5f8) v4 2014-06-09 16:22:11.803265 7f869af78700 5 -- op tracker -- , seq: 3217, time: 2014-06-09 16:22:11.652761, event: header_read, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.803277 7f869af78700 5 -- op tracker -- , seq: 3217, time: 2014-06-09 16:22:11.652762, event: throttled, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.803283 7f869af78700 5 -- op tracker -- , seq: 3217, time: 2014-06-09 16:22:11.652773, event: all_read, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.803295 7f869af78700 5 -- op tracker -- , seq: 3217, time: 2014-06-09 16:22:11.803257, event: dispatched, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.803306 7f869af78700 5 -- op tracker -- , seq: 3217, time: 2014-06-09 16:22:11.803303, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.803310 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5f8) v4 from osd.8 2014-06-09 16:22:11.803313 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x313896c0 2014-06-09 16:22:11.803318 7f869af78700 5 -- op tracker -- , seq: 3217, time: 2014-06-09 16:22:11.803316, event: started, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.803334 7f869af78700 5 -- op tracker -- , seq: 3217, time: 2014-06-09 16:22:11.803332, event: done, op: pg_info(1 pgs e243:3.5f8) 2014-06-09 16:22:11.803344 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803346 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803358 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803359 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803361 7f869af78700 20 osd.26 243 _dispatch 0x30adae00 pg_info(1 pgs e243:3.1f0) v4 2014-06-09 16:22:11.803366 7f869af78700 5 -- op tracker -- , seq: 3218, time: 2014-06-09 16:22:11.752986, event: header_read, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.803373 7f869af78700 5 -- op tracker -- , seq: 3218, time: 2014-06-09 16:22:11.752986, event: throttled, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.803379 7f869af78700 5 -- op tracker -- , seq: 3218, time: 2014-06-09 16:22:11.752999, event: all_read, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.803391 7f869af78700 5 -- op tracker -- , seq: 3218, time: 2014-06-09 16:22:11.803357, event: dispatched, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.803399 7f869af78700 5 -- op tracker -- , seq: 3218, time: 2014-06-09 16:22:11.803397, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.803407 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1f0) v4 from osd.6 2014-06-09 16:22:11.803410 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adae00 2014-06-09 16:22:11.803415 7f869af78700 5 -- op tracker -- , seq: 3218, time: 2014-06-09 16:22:11.803413, event: started, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.803435 7f869af78700 5 -- op tracker -- , seq: 3218, time: 2014-06-09 16:22:11.803432, event: done, op: pg_info(1 pgs e243:3.1f0) 2014-06-09 16:22:11.803443 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803445 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803457 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803458 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803460 7f869af78700 20 osd.26 243 _dispatch 0x2fc56c40 pg_info(1 pgs e243:1.555) v4 2014-06-09 16:22:11.803465 7f869af78700 5 -- op tracker -- , seq: 3219, time: 2014-06-09 16:22:11.650736, event: header_read, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.803477 7f869af78700 5 -- op tracker -- , seq: 3219, time: 2014-06-09 16:22:11.650736, event: throttled, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.803484 7f869af78700 5 -- op tracker -- , seq: 3219, time: 2014-06-09 16:22:11.650745, event: all_read, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.803494 7f869af78700 5 -- op tracker -- , seq: 3219, time: 2014-06-09 16:22:11.803456, event: dispatched, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.803505 7f869af78700 5 -- op tracker -- , seq: 3219, time: 2014-06-09 16:22:11.803503, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.803510 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.555) v4 from osd.2 2014-06-09 16:22:11.803513 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc56c40 2014-06-09 16:22:11.803518 7f869af78700 5 -- op tracker -- , seq: 3219, time: 2014-06-09 16:22:11.803516, event: started, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.803534 7f869af78700 5 -- op tracker -- , seq: 3219, time: 2014-06-09 16:22:11.803532, event: done, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.803541 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803543 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803556 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803558 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803559 7f869af78700 20 osd.26 243 _dispatch 0x1ac75880 pg_info(1 pgs e243:0.61f) v4 2014-06-09 16:22:11.803559 7f86a5610700 10 _generic_write 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 1478656~1548288 = 0 2014-06-09 16:22:11.803564 7f869af78700 5 -- op tracker -- , seq: 3220, time: 2014-06-09 16:22:11.655574, event: header_read, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.803571 7f869af78700 5 -- op tracker -- , seq: 3220, time: 2014-06-09 16:22:11.655575, event: throttled, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.803578 7f869af78700 5 -- op tracker -- , seq: 3220, time: 2014-06-09 16:22:11.655586, event: all_read, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.803584 7f869af78700 5 -- op tracker -- , seq: 3220, time: 2014-06-09 16:22:11.803555, event: dispatched, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.803592 7f869af78700 5 -- op tracker -- , seq: 3220, time: 2014-06-09 16:22:11.803590, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.803596 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.61f) v4 from osd.4 2014-06-09 16:22:11.803600 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac75880 2014-06-09 16:22:11.803605 7f869af78700 5 -- op tracker -- , seq: 3220, time: 2014-06-09 16:22:11.803603, event: started, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.803630 7f869af78700 5 -- op tracker -- , seq: 3220, time: 2014-06-09 16:22:11.803627, event: done, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.803635 7f86a5610700 15 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.803637 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803640 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803647 7f86a5e11700 10 _do_op 0x288f9080 seq 3471 r = 0, finisher 0xc6e9fc0 0 2014-06-09 16:22:11.803652 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803654 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803653 7f86a5e11700 10 _finish_op 0x288f9080 seq 3471 osr(3.17 0x3ed03b0)/0x3ed03b0 2014-06-09 16:22:11.803654 7f86a5610700 10 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.803656 7f869af78700 20 osd.26 243 _dispatch 0x30adba40 pg_info(1 pgs e243:3.111) v4 2014-06-09 16:22:11.803660 7f86a5610700 15 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.803663 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383fb00 (1 active) 2014-06-09 16:22:11.803666 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f77a0 (2 active) 2014-06-09 16:22:11.803667 7f86a5610700 10 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.803667 7f868b759700 10 op_submit_start 3527 2014-06-09 16:22:11.803670 7f868b759700 5 queue_transactions (trailing journal) 3527 2014-06-09 16:22:11.803669 7f86a5e11700 5 _do_op 0x30d96060 seq 3479 osr(3.74f 0x3ed1ea0)/0x3ed1ea0 start 2014-06-09 16:22:11.803669 7f868a757700 2 waiting 51 > 50 ops || 4336104 > 104857600 2014-06-09 16:22:11.803673 7f86a5e11700 10 _do_transaction on 0xc001440 2014-06-09 16:22:11.803672 7f868b759700 5 queue_op 0x33a3f80 seq 3527 osr(1.67a 0x384de90) 828 bytes (queue has 50 ops and 4334201 bytes) 2014-06-09 16:22:11.803677 7f868b759700 10 op_submit_finish 3527 2014-06-09 16:22:11.803680 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.803682 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.803695 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.803701 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.803706 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.803718 7f868b759700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.803732 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.803748 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.803753 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.8c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.8c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.803786 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.803790 7f868b759700 5 queue_transactions existing osr(0.8c 0x2bd6480)/0x2bd6480 2014-06-09 16:22:11.803792 7f868b759700 2 waiting 51 > 50 ops || 4335027 > 104857600 2014-06-09 16:22:11.803797 7f8690763700 2 waiting 51 > 50 ops || 4335029 > 104857600 2014-06-09 16:22:11.803802 7f868ef60700 2 waiting 51 > 50 ops || 4336202 > 104857600 2014-06-09 16:22:11.803811 7f8689755700 2 waiting 51 > 50 ops || 4336144 > 104857600 2014-06-09 16:22:11.803661 7f869af78700 5 -- op tracker -- , seq: 3221, time: 2014-06-09 16:22:11.747708, event: header_read, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803818 7f8692767700 2 waiting 51 > 50 ops || 4335536 > 104857600 2014-06-09 16:22:11.803818 7f869af78700 5 -- op tracker -- , seq: 3221, time: 2014-06-09 16:22:11.747709, event: throttled, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803825 7f8689f56700 2 waiting 51 > 50 ops || 4335029 > 104857600 2014-06-09 16:22:11.803825 7f869af78700 5 -- op tracker -- , seq: 3221, time: 2014-06-09 16:22:11.747734, event: all_read, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803833 7f868ff62700 2 waiting 51 > 50 ops || 4335536 > 104857600 2014-06-09 16:22:11.803839 7f868f761700 2 waiting 51 > 50 ops || 4335630 > 104857600 2014-06-09 16:22:11.803844 7f868cf5c700 2 waiting 51 > 50 ops || 4335029 > 104857600 2014-06-09 16:22:11.803849 7f868bf5a700 2 waiting 51 > 50 ops || 4335029 > 104857600 2014-06-09 16:22:11.803853 7f868df5e700 2 waiting 51 > 50 ops || 4398039 > 104857600 2014-06-09 16:22:11.803859 7f8692f68700 2 waiting 51 > 50 ops || 4335254 > 104857600 2014-06-09 16:22:11.803865 7f868e75f700 2 waiting 51 > 50 ops || 4852581 > 104857600 2014-06-09 16:22:11.803832 7f869af78700 5 -- op tracker -- , seq: 3221, time: 2014-06-09 16:22:11.803652, event: dispatched, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803875 7f869af78700 5 -- op tracker -- , seq: 3221, time: 2014-06-09 16:22:11.803872, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803880 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.111) v4 from osd.1 2014-06-09 16:22:11.803883 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adba40 2014-06-09 16:22:11.803889 7f86a5610700 10 cid: oid: 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 not skipping op, *spos 3478.0.0 2014-06-09 16:22:11.803894 7f86a5610700 10 > header.spos 3427.0.0 2014-06-09 16:22:11.803888 7f869af78700 5 -- op tracker -- , seq: 3221, time: 2014-06-09 16:22:11.803886, event: started, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803910 7f869af78700 5 -- op tracker -- , seq: 3221, time: 2014-06-09 16:22:11.803908, event: done, op: pg_info(1 pgs e243:3.111) 2014-06-09 16:22:11.803918 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803920 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803934 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.803936 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.803937 7f869af78700 20 osd.26 243 _dispatch 0x1ac75500 pg_info(1 pgs e243:0.58a) v4 2014-06-09 16:22:11.803942 7f869af78700 5 -- op tracker -- , seq: 3222, time: 2014-06-09 16:22:11.658850, event: header_read, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.803949 7f869af78700 5 -- op tracker -- , seq: 3222, time: 2014-06-09 16:22:11.658851, event: throttled, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.803956 7f869af78700 5 -- op tracker -- , seq: 3222, time: 2014-06-09 16:22:11.658864, event: all_read, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.803963 7f869af78700 5 -- op tracker -- , seq: 3222, time: 2014-06-09 16:22:11.803933, event: dispatched, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.803970 7f869af78700 5 -- op tracker -- , seq: 3222, time: 2014-06-09 16:22:11.803968, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.803977 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.58a) v4 from osd.8 2014-06-09 16:22:11.803981 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac75500 2014-06-09 16:22:11.803980 7f86a5610700 20 set_header setting 30151 cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 parent seq 0 2014-06-09 16:22:11.803986 7f869af78700 5 -- op tracker -- , seq: 3222, time: 2014-06-09 16:22:11.803984, event: started, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.804002 7f869af78700 5 -- op tracker -- , seq: 3222, time: 2014-06-09 16:22:11.804000, event: done, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.804014 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804016 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804029 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804031 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804032 7f869af78700 20 osd.26 243 _dispatch 0x30adb880 pg_info(1 pgs e243:3.e8) v4 2014-06-09 16:22:11.804037 7f869af78700 5 -- op tracker -- , seq: 3223, time: 2014-06-09 16:22:11.753034, event: header_read, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.804046 7f869af78700 5 -- op tracker -- , seq: 3223, time: 2014-06-09 16:22:11.753034, event: throttled, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.804053 7f869af78700 5 -- op tracker -- , seq: 3223, time: 2014-06-09 16:22:11.753046, event: all_read, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.804059 7f869af78700 5 -- op tracker -- , seq: 3223, time: 2014-06-09 16:22:11.804029, event: dispatched, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.804070 7f869af78700 5 -- op tracker -- , seq: 3223, time: 2014-06-09 16:22:11.804067, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.804074 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.e8) v4 from osd.6 2014-06-09 16:22:11.804077 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adb880 2014-06-09 16:22:11.804083 7f869af78700 5 -- op tracker -- , seq: 3223, time: 2014-06-09 16:22:11.804081, event: started, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.804099 7f869af78700 5 -- op tracker -- , seq: 3223, time: 2014-06-09 16:22:11.804097, event: done, op: pg_info(1 pgs e243:3.e8) 2014-06-09 16:22:11.804107 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804109 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804115 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3478.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.804121 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804122 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804122 7f86a5610700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3478.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.804124 7f869af78700 20 osd.26 243 _dispatch 0x2fc57500 pg_info(1 pgs e243:1.3d6) v4 2014-06-09 16:22:11.804126 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.804129 7f869af78700 5 -- op tracker -- , seq: 3224, time: 2014-06-09 16:22:11.650784, event: header_read, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.804136 7f869af78700 5 -- op tracker -- , seq: 3224, time: 2014-06-09 16:22:11.650784, event: throttled, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.804142 7f869af78700 5 -- op tracker -- , seq: 3224, time: 2014-06-09 16:22:11.650795, event: all_read, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.804148 7f869af78700 5 -- op tracker -- , seq: 3224, time: 2014-06-09 16:22:11.804120, event: dispatched, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.804156 7f869af78700 5 -- op tracker -- , seq: 3224, time: 2014-06-09 16:22:11.804153, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.804161 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3d6) v4 from osd.2 2014-06-09 16:22:11.804164 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc57500 2014-06-09 16:22:11.804169 7f869af78700 5 -- op tracker -- , seq: 3224, time: 2014-06-09 16:22:11.804167, event: started, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.804185 7f869af78700 5 -- op tracker -- , seq: 3224, time: 2014-06-09 16:22:11.804183, event: done, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.804193 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804195 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804207 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804209 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804210 7f869af78700 20 osd.26 243 _dispatch 0x2f0381c0 pg_info(1 pgs e243:1.302) v4 2014-06-09 16:22:11.804215 7f869af78700 5 -- op tracker -- , seq: 3225, time: 2014-06-09 16:22:11.655624, event: header_read, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.804223 7f869af78700 5 -- op tracker -- , seq: 3225, time: 2014-06-09 16:22:11.655624, event: throttled, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.804229 7f869af78700 5 -- op tracker -- , seq: 3225, time: 2014-06-09 16:22:11.655635, event: all_read, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.804235 7f869af78700 5 -- op tracker -- , seq: 3225, time: 2014-06-09 16:22:11.804207, event: dispatched, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.804245 7f869af78700 5 -- op tracker -- , seq: 3225, time: 2014-06-09 16:22:11.804243, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.804251 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.302) v4 from osd.4 2014-06-09 16:22:11.804253 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f0381c0 2014-06-09 16:22:11.804261 7f869af78700 5 -- op tracker -- , seq: 3225, time: 2014-06-09 16:22:11.804259, event: started, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.804282 7f869af78700 5 -- op tracker -- , seq: 3225, time: 2014-06-09 16:22:11.804280, event: done, op: pg_info(1 pgs e243:1.302) 2014-06-09 16:22:11.804291 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804293 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804304 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804306 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804307 7f869af78700 20 osd.26 243 _dispatch 0x30c91a40 pg_info(1 pgs e243:3.172) v4 2014-06-09 16:22:11.804312 7f869af78700 5 -- op tracker -- , seq: 3226, time: 2014-06-09 16:22:11.747743, event: header_read, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.804319 7f869af78700 5 -- op tracker -- , seq: 3226, time: 2014-06-09 16:22:11.747744, event: throttled, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.804326 7f869af78700 5 -- op tracker -- , seq: 3226, time: 2014-06-09 16:22:11.747755, event: all_read, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.804333 7f869af78700 5 -- op tracker -- , seq: 3226, time: 2014-06-09 16:22:11.804304, event: dispatched, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.804342 7f869af78700 5 -- op tracker -- , seq: 3226, time: 2014-06-09 16:22:11.804340, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.804347 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.172) v4 from osd.1 2014-06-09 16:22:11.804350 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c91a40 2014-06-09 16:22:11.804355 7f869af78700 5 -- op tracker -- , seq: 3226, time: 2014-06-09 16:22:11.804353, event: started, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.804376 7f869af78700 5 -- op tracker -- , seq: 3226, time: 2014-06-09 16:22:11.804373, event: done, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.804383 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804385 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804397 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804400 7f869af78700 20 osd.26 243 _dispatch 0x1ac75dc0 pg_info(1 pgs e243:3.400) v4 2014-06-09 16:22:11.804405 7f869af78700 5 -- op tracker -- , seq: 3227, time: 2014-06-09 16:22:11.658887, event: header_read, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.804414 7f869af78700 5 -- op tracker -- , seq: 3227, time: 2014-06-09 16:22:11.658888, event: throttled, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.804421 7f869af78700 5 -- op tracker -- , seq: 3227, time: 2014-06-09 16:22:11.658904, event: all_read, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.804428 7f869af78700 5 -- op tracker -- , seq: 3227, time: 2014-06-09 16:22:11.804397, event: dispatched, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.804439 7f869af78700 5 -- op tracker -- , seq: 3227, time: 2014-06-09 16:22:11.804436, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.804447 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.400) v4 from osd.8 2014-06-09 16:22:11.804451 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac75dc0 2014-06-09 16:22:11.804456 7f869af78700 5 -- op tracker -- , seq: 3227, time: 2014-06-09 16:22:11.804454, event: started, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.804474 7f869af78700 5 -- op tracker -- , seq: 3227, time: 2014-06-09 16:22:11.804471, event: done, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.804483 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804485 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804497 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804499 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804500 7f869af78700 20 osd.26 243 _dispatch 0x30ada8c0 pg_info(1 pgs e243:3.688) v4 2014-06-09 16:22:11.804506 7f869af78700 5 -- op tracker -- , seq: 3228, time: 2014-06-09 16:22:11.753957, event: header_read, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.804513 7f869af78700 5 -- op tracker -- , seq: 3228, time: 2014-06-09 16:22:11.753957, event: throttled, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.804519 7f869af78700 5 -- op tracker -- , seq: 3228, time: 2014-06-09 16:22:11.753969, event: all_read, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.804528 7f869af78700 5 -- op tracker -- , seq: 3228, time: 2014-06-09 16:22:11.804497, event: dispatched, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.804535 7f869af78700 5 -- op tracker -- , seq: 3228, time: 2014-06-09 16:22:11.804533, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.804540 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.688) v4 from osd.6 2014-06-09 16:22:11.804543 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ada8c0 2014-06-09 16:22:11.804548 7f869af78700 5 -- op tracker -- , seq: 3228, time: 2014-06-09 16:22:11.804546, event: started, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.804564 7f869af78700 5 -- op tracker -- , seq: 3228, time: 2014-06-09 16:22:11.804562, event: done, op: pg_info(1 pgs e243:3.688) 2014-06-09 16:22:11.804571 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804573 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804587 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804589 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804590 7f869af78700 20 osd.26 243 _dispatch 0x2fc57880 pg_info(1 pgs e243:2.101) v4 2014-06-09 16:22:11.804595 7f869af78700 5 -- op tracker -- , seq: 3229, time: 2014-06-09 16:22:11.650809, event: header_read, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.804602 7f869af78700 5 -- op tracker -- , seq: 3229, time: 2014-06-09 16:22:11.650809, event: throttled, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.804611 7f869af78700 5 -- op tracker -- , seq: 3229, time: 2014-06-09 16:22:11.650821, event: all_read, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.804618 7f869af78700 5 -- op tracker -- , seq: 3229, time: 2014-06-09 16:22:11.804587, event: dispatched, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.804625 7f869af78700 5 -- op tracker -- , seq: 3229, time: 2014-06-09 16:22:11.804623, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.804631 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.101) v4 from osd.2 2014-06-09 16:22:11.804634 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc57880 2014-06-09 16:22:11.804639 7f869af78700 5 -- op tracker -- , seq: 3229, time: 2014-06-09 16:22:11.804638, event: started, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.804655 7f869af78700 5 -- op tracker -- , seq: 3229, time: 2014-06-09 16:22:11.804652, event: done, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.804667 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804669 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804681 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804683 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804684 7f869af78700 20 osd.26 243 _dispatch 0x2f038000 pg_info(1 pgs e243:0.61f) v4 2014-06-09 16:22:11.804689 7f869af78700 5 -- op tracker -- , seq: 3230, time: 2014-06-09 16:22:11.655676, event: header_read, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.804701 7f869af78700 5 -- op tracker -- , seq: 3230, time: 2014-06-09 16:22:11.655676, event: throttled, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.804711 7f869af78700 5 -- op tracker -- , seq: 3230, time: 2014-06-09 16:22:11.655688, event: all_read, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.804719 7f869af78700 5 -- op tracker -- , seq: 3230, time: 2014-06-09 16:22:11.804681, event: dispatched, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.804727 7f869af78700 5 -- op tracker -- , seq: 3230, time: 2014-06-09 16:22:11.804724, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.804731 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.61f) v4 from osd.4 2014-06-09 16:22:11.804734 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f038000 2014-06-09 16:22:11.804740 7f869af78700 5 -- op tracker -- , seq: 3230, time: 2014-06-09 16:22:11.804738, event: started, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.804755 7f869af78700 5 -- op tracker -- , seq: 3230, time: 2014-06-09 16:22:11.804753, event: done, op: pg_info(1 pgs e243:0.61f) 2014-06-09 16:22:11.804763 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804765 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804780 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804783 7f869af78700 20 osd.26 243 _dispatch 0x309e7340 pg_info(1 pgs e243:2.4) v4 2014-06-09 16:22:11.804787 7f869af78700 5 -- op tracker -- , seq: 3231, time: 2014-06-09 16:22:11.747760, event: header_read, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.804794 7f869af78700 5 -- op tracker -- , seq: 3231, time: 2014-06-09 16:22:11.747760, event: throttled, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.804799 7f869af78700 5 -- op tracker -- , seq: 3231, time: 2014-06-09 16:22:11.747771, event: all_read, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.804805 7f869af78700 5 -- op tracker -- , seq: 3231, time: 2014-06-09 16:22:11.804780, event: dispatched, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.804810 7f869af78700 5 -- op tracker -- , seq: 3231, time: 2014-06-09 16:22:11.804809, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.804814 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4) v4 from osd.1 2014-06-09 16:22:11.804817 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e7340 2014-06-09 16:22:11.804821 7f869af78700 5 -- op tracker -- , seq: 3231, time: 2014-06-09 16:22:11.804819, event: started, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.804834 7f869af78700 5 -- op tracker -- , seq: 3231, time: 2014-06-09 16:22:11.804832, event: done, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.804842 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804844 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804855 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804856 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804858 7f869af78700 20 osd.26 243 _dispatch 0x2fc57c00 pg_info(1 pgs e243:2.194) v4 2014-06-09 16:22:11.804863 7f869af78700 5 -- op tracker -- , seq: 3232, time: 2014-06-09 16:22:11.658932, event: header_read, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.804875 7f869af78700 5 -- op tracker -- , seq: 3232, time: 2014-06-09 16:22:11.658933, event: throttled, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.804886 7f869af78700 5 -- op tracker -- , seq: 3232, time: 2014-06-09 16:22:11.658946, event: all_read, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.804896 7f869af78700 5 -- op tracker -- , seq: 3232, time: 2014-06-09 16:22:11.804854, event: dispatched, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.804906 7f869af78700 5 -- op tracker -- , seq: 3232, time: 2014-06-09 16:22:11.804904, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.804915 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.194) v4 from osd.8 2014-06-09 16:22:11.804918 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc57c00 2014-06-09 16:22:11.804923 7f869af78700 5 -- op tracker -- , seq: 3232, time: 2014-06-09 16:22:11.804921, event: started, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.804940 7f869af78700 5 -- op tracker -- , seq: 3232, time: 2014-06-09 16:22:11.804937, event: done, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.804947 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804949 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804961 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.804963 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.804965 7f869af78700 20 osd.26 243 _dispatch 0x30acca80 pg_info(1 pgs e243:3.13c) v4 2014-06-09 16:22:11.804970 7f869af78700 5 -- op tracker -- , seq: 3233, time: 2014-06-09 16:22:11.754000, event: header_read, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.804980 7f869af78700 5 -- op tracker -- , seq: 3233, time: 2014-06-09 16:22:11.754001, event: throttled, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.804986 7f869af78700 5 -- op tracker -- , seq: 3233, time: 2014-06-09 16:22:11.754012, event: all_read, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.804993 7f869af78700 5 -- op tracker -- , seq: 3233, time: 2014-06-09 16:22:11.804961, event: dispatched, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.805002 7f869af78700 5 -- op tracker -- , seq: 3233, time: 2014-06-09 16:22:11.805000, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.805007 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.13c) v4 from osd.6 2014-06-09 16:22:11.805010 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acca80 2014-06-09 16:22:11.805015 7f869af78700 5 -- op tracker -- , seq: 3233, time: 2014-06-09 16:22:11.805013, event: started, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.805036 7f869af78700 5 -- op tracker -- , seq: 3233, time: 2014-06-09 16:22:11.805033, event: done, op: pg_info(1 pgs e243:3.13c) 2014-06-09 16:22:11.805043 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805058 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805060 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805061 7f869af78700 20 osd.26 243 _dispatch 0x30da2c40 pg_info(1 pgs e243:1.3d6) v4 2014-06-09 16:22:11.805066 7f869af78700 5 -- op tracker -- , seq: 3234, time: 2014-06-09 16:22:11.650839, event: header_read, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.805073 7f869af78700 5 -- op tracker -- , seq: 3234, time: 2014-06-09 16:22:11.650840, event: throttled, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.805080 7f869af78700 5 -- op tracker -- , seq: 3234, time: 2014-06-09 16:22:11.650851, event: all_read, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.805086 7f869af78700 5 -- op tracker -- , seq: 3234, time: 2014-06-09 16:22:11.805058, event: dispatched, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.805094 7f869af78700 5 -- op tracker -- , seq: 3234, time: 2014-06-09 16:22:11.805092, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.805101 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3d6) v4 from osd.2 2014-06-09 16:22:11.805104 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da2c40 2014-06-09 16:22:11.805109 7f869af78700 5 -- op tracker -- , seq: 3234, time: 2014-06-09 16:22:11.805107, event: started, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.805128 7f869af78700 5 -- op tracker -- , seq: 3234, time: 2014-06-09 16:22:11.805126, event: done, op: pg_info(1 pgs e243:1.3d6) 2014-06-09 16:22:11.805136 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805138 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805150 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805152 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805153 7f869af78700 20 osd.26 243 _dispatch 0x2fc56540 pg_info(1 pgs e243:1.8c) v4 2014-06-09 16:22:11.805158 7f869af78700 5 -- op tracker -- , seq: 3235, time: 2014-06-09 16:22:11.660043, event: header_read, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805165 7f869af78700 5 -- op tracker -- , seq: 3235, time: 2014-06-09 16:22:11.660044, event: throttled, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805172 7f869af78700 5 -- op tracker -- , seq: 3235, time: 2014-06-09 16:22:11.660057, event: all_read, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805179 7f869af78700 5 -- op tracker -- , seq: 3235, time: 2014-06-09 16:22:11.805150, event: dispatched, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805186 7f869af78700 5 -- op tracker -- , seq: 3235, time: 2014-06-09 16:22:11.805184, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805193 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.8c) v4 from osd.4 2014-06-09 16:22:11.805196 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc56540 2014-06-09 16:22:11.805201 7f869af78700 5 -- op tracker -- , seq: 3235, time: 2014-06-09 16:22:11.805199, event: started, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805218 7f869af78700 5 -- op tracker -- , seq: 3235, time: 2014-06-09 16:22:11.805215, event: done, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805225 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805227 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805239 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805241 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805243 7f869af78700 20 osd.26 243 _dispatch 0x309e7500 pg_info(1 pgs e243:0.771) v4 2014-06-09 16:22:11.805248 7f869af78700 5 -- op tracker -- , seq: 3236, time: 2014-06-09 16:22:11.747787, event: header_read, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.805258 7f869af78700 5 -- op tracker -- , seq: 3236, time: 2014-06-09 16:22:11.747788, event: throttled, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.805265 7f869af78700 5 -- op tracker -- , seq: 3236, time: 2014-06-09 16:22:11.747801, event: all_read, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.805271 7f869af78700 5 -- op tracker -- , seq: 3236, time: 2014-06-09 16:22:11.805239, event: dispatched, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.805283 7f869af78700 5 -- op tracker -- , seq: 3236, time: 2014-06-09 16:22:11.805281, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.805288 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.771) v4 from osd.1 2014-06-09 16:22:11.805291 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e7500 2014-06-09 16:22:11.805297 7f869af78700 5 -- op tracker -- , seq: 3236, time: 2014-06-09 16:22:11.805294, event: started, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.805313 7f869af78700 5 -- op tracker -- , seq: 3236, time: 2014-06-09 16:22:11.805311, event: done, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.805321 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805323 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805334 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805336 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805337 7f869af78700 20 osd.26 243 _dispatch 0x2fc57dc0 pg_info(1 pgs e243:1.601) v4 2014-06-09 16:22:11.805343 7f869af78700 5 -- op tracker -- , seq: 3237, time: 2014-06-09 16:22:11.658981, event: header_read, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.805349 7f869af78700 5 -- op tracker -- , seq: 3237, time: 2014-06-09 16:22:11.658982, event: throttled, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.805357 7f869af78700 5 -- op tracker -- , seq: 3237, time: 2014-06-09 16:22:11.658994, event: all_read, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.805366 7f869af78700 5 -- op tracker -- , seq: 3237, time: 2014-06-09 16:22:11.805334, event: dispatched, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.805374 7f869af78700 5 -- op tracker -- , seq: 3237, time: 2014-06-09 16:22:11.805371, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.805378 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.601) v4 from osd.8 2014-06-09 16:22:11.805381 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc57dc0 2014-06-09 16:22:11.805386 7f869af78700 5 -- op tracker -- , seq: 3237, time: 2014-06-09 16:22:11.805384, event: started, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.805402 7f869af78700 5 -- op tracker -- , seq: 3237, time: 2014-06-09 16:22:11.805400, event: done, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.805410 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805412 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805425 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805428 7f869af78700 20 osd.26 243 _dispatch 0x30acd180 pg_info(1 pgs e243:3.753) v4 2014-06-09 16:22:11.805433 7f869af78700 5 -- op tracker -- , seq: 3238, time: 2014-06-09 16:22:11.754049, event: header_read, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.805440 7f869af78700 5 -- op tracker -- , seq: 3238, time: 2014-06-09 16:22:11.754050, event: throttled, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.805447 7f869af78700 5 -- op tracker -- , seq: 3238, time: 2014-06-09 16:22:11.754059, event: all_read, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.805454 7f869af78700 5 -- op tracker -- , seq: 3238, time: 2014-06-09 16:22:11.805424, event: dispatched, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.805476 7f8682be6700 5 -- op tracker -- , seq: 3239, time: 2014-06-09 16:22:11.792030, event: header_read, op: MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:11.805490 7f8682be6700 5 -- op tracker -- , seq: 3239, time: 2014-06-09 16:22:11.792031, event: throttled, op: MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:11.805502 7f8682be6700 5 -- op tracker -- , seq: 3239, time: 2014-06-09 16:22:11.805424, event: all_read, op: MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:11.805515 7f8682be6700 5 -- op tracker -- , seq: 3239, time: 0.000000, event: dispatched, op: MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:11.805521 7f8682be6700 10 osd.26 243 handle_replica_op MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 epoch 243 2014-06-09 16:22:11.805532 7f8682be6700 20 osd.26 243 should_share_map osd.5 172.16.96.201:6837/51639 243 2014-06-09 16:22:11.805540 7f8682be6700 15 osd.26 243 enqueue_op 0x32826100 prio 63 cost 4195304 latency 0.013510 MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 2014-06-09 16:22:11.805628 7f869af78700 5 -- op tracker -- , seq: 3238, time: 2014-06-09 16:22:11.805624, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.805634 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.753) v4 from osd.6 2014-06-09 16:22:11.805637 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acd180 2014-06-09 16:22:11.805643 7f869af78700 5 -- op tracker -- , seq: 3238, time: 2014-06-09 16:22:11.805640, event: started, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.805660 7f869af78700 5 -- op tracker -- , seq: 3238, time: 2014-06-09 16:22:11.805658, event: done, op: pg_info(1 pgs e243:3.753) 2014-06-09 16:22:11.805673 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805675 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805687 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805688 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805690 7f869af78700 20 osd.26 243 _dispatch 0x30da2a80 pg_info(1 pgs e243:1.555) v4 2014-06-09 16:22:11.805695 7f869af78700 5 -- op tracker -- , seq: 3240, time: 2014-06-09 16:22:11.651482, event: header_read, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.805702 7f869af78700 5 -- op tracker -- , seq: 3240, time: 2014-06-09 16:22:11.651483, event: throttled, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.805708 7f869af78700 5 -- op tracker -- , seq: 3240, time: 2014-06-09 16:22:11.651492, event: all_read, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.805720 7f869af78700 5 -- op tracker -- , seq: 3240, time: 2014-06-09 16:22:11.805686, event: dispatched, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.805731 7f869af78700 5 -- op tracker -- , seq: 3240, time: 2014-06-09 16:22:11.805728, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.805739 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.555) v4 from osd.2 2014-06-09 16:22:11.805742 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da2a80 2014-06-09 16:22:11.805748 7f869af78700 5 -- op tracker -- , seq: 3240, time: 2014-06-09 16:22:11.805746, event: started, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.805767 7f869af78700 5 -- op tracker -- , seq: 3240, time: 2014-06-09 16:22:11.805764, event: done, op: pg_info(1 pgs e243:1.555) 2014-06-09 16:22:11.805774 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805776 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805789 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805791 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805792 7f869af78700 20 osd.26 243 _dispatch 0x30c13dc0 pg_info(1 pgs e243:1.8c) v4 2014-06-09 16:22:11.805797 7f869af78700 5 -- op tracker -- , seq: 3241, time: 2014-06-09 16:22:11.660066, event: header_read, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805804 7f869af78700 5 -- op tracker -- , seq: 3241, time: 2014-06-09 16:22:11.660067, event: throttled, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805811 7f869af78700 5 -- op tracker -- , seq: 3241, time: 2014-06-09 16:22:11.660088, event: all_read, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805817 7f869af78700 5 -- op tracker -- , seq: 3241, time: 2014-06-09 16:22:11.805788, event: dispatched, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805824 7f869af78700 5 -- op tracker -- , seq: 3241, time: 2014-06-09 16:22:11.805822, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805830 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.8c) v4 from osd.4 2014-06-09 16:22:11.805834 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c13dc0 2014-06-09 16:22:11.805839 7f869af78700 5 -- op tracker -- , seq: 3241, time: 2014-06-09 16:22:11.805837, event: started, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805854 7f869af78700 5 -- op tracker -- , seq: 3241, time: 2014-06-09 16:22:11.805852, event: done, op: pg_info(1 pgs e243:1.8c) 2014-06-09 16:22:11.805866 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805881 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805882 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805884 7f869af78700 20 osd.26 243 _dispatch 0xddc48c0 pg_info(1 pgs e243:3.172) v4 2014-06-09 16:22:11.805889 7f869af78700 5 -- op tracker -- , seq: 3242, time: 2014-06-09 16:22:11.747806, event: header_read, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.805896 7f869af78700 5 -- op tracker -- , seq: 3242, time: 2014-06-09 16:22:11.747807, event: throttled, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.805903 7f869af78700 5 -- op tracker -- , seq: 3242, time: 2014-06-09 16:22:11.747817, event: all_read, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.805912 7f869af78700 5 -- op tracker -- , seq: 3242, time: 2014-06-09 16:22:11.805880, event: dispatched, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.805919 7f869af78700 5 -- op tracker -- , seq: 3242, time: 2014-06-09 16:22:11.805917, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.805924 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.172) v4 from osd.1 2014-06-09 16:22:11.805927 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc48c0 2014-06-09 16:22:11.805932 7f869af78700 5 -- op tracker -- , seq: 3242, time: 2014-06-09 16:22:11.805930, event: started, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.805948 7f869af78700 5 -- op tracker -- , seq: 3242, time: 2014-06-09 16:22:11.805946, event: done, op: pg_info(1 pgs e243:3.172) 2014-06-09 16:22:11.805955 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805957 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.805971 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.805972 7f869af78700 20 osd.26 243 _dispatch 0x31389500 pg_info(1 pgs e243:0.58a) v4 2014-06-09 16:22:11.805978 7f869af78700 5 -- op tracker -- , seq: 3243, time: 2014-06-09 16:22:11.659026, event: header_read, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.805987 7f869af78700 5 -- op tracker -- , seq: 3243, time: 2014-06-09 16:22:11.659027, event: throttled, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.805994 7f869af78700 5 -- op tracker -- , seq: 3243, time: 2014-06-09 16:22:11.659039, event: all_read, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.806001 7f869af78700 5 -- op tracker -- , seq: 3243, time: 2014-06-09 16:22:11.805969, event: dispatched, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.806008 7f869af78700 5 -- op tracker -- , seq: 3243, time: 2014-06-09 16:22:11.806006, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.806013 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.58a) v4 from osd.8 2014-06-09 16:22:11.806017 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31389500 2014-06-09 16:22:11.806021 7f869af78700 5 -- op tracker -- , seq: 3243, time: 2014-06-09 16:22:11.806020, event: started, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.806041 7f869af78700 5 -- op tracker -- , seq: 3243, time: 2014-06-09 16:22:11.806039, event: done, op: pg_info(1 pgs e243:0.58a) 2014-06-09 16:22:11.806051 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806053 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806065 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806067 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806069 7f869af78700 20 osd.26 243 _dispatch 0x1607ac40 MRecoveryReserve REQUEST pgid: 3.27, query_epoch: 243 v2 2014-06-09 16:22:11.806074 7f869af78700 5 -- op tracker -- , seq: 3244, time: 2014-06-09 16:22:11.805578, event: header_read, op: MRecoveryReserve REQUEST pgid: 3.27, query_epoch: 243 2014-06-09 16:22:11.806081 7f869af78700 5 -- op tracker -- , seq: 3244, time: 2014-06-09 16:22:11.805579, event: throttled, op: MRecoveryReserve REQUEST pgid: 3.27, query_epoch: 243 2014-06-09 16:22:11.806087 7f869af78700 5 -- op tracker -- , seq: 3244, time: 2014-06-09 16:22:11.805585, event: all_read, op: MRecoveryReserve REQUEST pgid: 3.27, query_epoch: 243 2014-06-09 16:22:11.806093 7f869af78700 5 -- op tracker -- , seq: 3244, time: 2014-06-09 16:22:11.806065, event: dispatched, op: MRecoveryReserve REQUEST pgid: 3.27, query_epoch: 243 2014-06-09 16:22:11.806101 7f869af78700 5 -- op tracker -- , seq: 3244, time: 2014-06-09 16:22:11.806099, event: waiting_for_osdmap, op: MRecoveryReserve REQUEST pgid: 3.27, query_epoch: 243 2014-06-09 16:22:11.806105 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1607ac40 2014-06-09 16:22:11.806116 7f869af78700 5 -- op tracker -- , seq: 3244, time: 2014-06-09 16:22:11.806114, event: done, op: MRecoveryReserve REQUEST pgid: 3.27, query_epoch: 243 2014-06-09 16:22:11.806123 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806125 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806137 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806138 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806140 7f869af78700 20 osd.26 243 _dispatch 0x30da28c0 pg_info(1 pgs e243:2.101) v4 2014-06-09 16:22:11.806145 7f869af78700 5 -- op tracker -- , seq: 3245, time: 2014-06-09 16:22:11.651985, event: header_read, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.806152 7f869af78700 5 -- op tracker -- , seq: 3245, time: 2014-06-09 16:22:11.651986, event: throttled, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.806158 7f869af78700 5 -- op tracker -- , seq: 3245, time: 2014-06-09 16:22:11.651995, event: all_read, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.806165 7f869af78700 5 -- op tracker -- , seq: 3245, time: 2014-06-09 16:22:11.806136, event: dispatched, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.806172 7f869af78700 5 -- op tracker -- , seq: 3245, time: 2014-06-09 16:22:11.806170, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.806176 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.101) v4 from osd.2 2014-06-09 16:22:11.806180 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da28c0 2014-06-09 16:22:11.806185 7f869af78700 5 -- op tracker -- , seq: 3245, time: 2014-06-09 16:22:11.806182, event: started, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.806200 7f869af78700 5 -- op tracker -- , seq: 3245, time: 2014-06-09 16:22:11.806198, event: done, op: pg_info(1 pgs e243:2.101) 2014-06-09 16:22:11.806212 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806214 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806229 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806230 7f869af78700 20 osd.26 243 _dispatch 0x3130a700 pg_info(1 pgs e243:1.2f8) v4 2014-06-09 16:22:11.806235 7f869af78700 5 -- op tracker -- , seq: 3246, time: 2014-06-09 16:22:11.744579, event: header_read, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806242 7f869af78700 5 -- op tracker -- , seq: 3246, time: 2014-06-09 16:22:11.744579, event: throttled, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806249 7f869af78700 5 -- op tracker -- , seq: 3246, time: 2014-06-09 16:22:11.744591, event: all_read, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806255 7f869af78700 5 -- op tracker -- , seq: 3246, time: 2014-06-09 16:22:11.806227, event: dispatched, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806262 7f869af78700 5 -- op tracker -- , seq: 3246, time: 2014-06-09 16:22:11.806260, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806267 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2f8) v4 from osd.4 2014-06-09 16:22:11.806270 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130a700 2014-06-09 16:22:11.806275 7f869af78700 5 -- op tracker -- , seq: 3246, time: 2014-06-09 16:22:11.806273, event: started, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806293 7f869af78700 5 -- op tracker -- , seq: 3246, time: 2014-06-09 16:22:11.806291, event: done, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806301 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806303 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806317 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806318 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806320 7f869af78700 20 osd.26 243 _dispatch 0xcd76000 pg_info(1 pgs e243:2.4) v4 2014-06-09 16:22:11.806326 7f869af78700 5 -- op tracker -- , seq: 3247, time: 2014-06-09 16:22:11.747821, event: header_read, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.806334 7f869af78700 5 -- op tracker -- , seq: 3247, time: 2014-06-09 16:22:11.747821, event: throttled, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.806341 7f869af78700 5 -- op tracker -- , seq: 3247, time: 2014-06-09 16:22:11.747831, event: all_read, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.806352 7f869af78700 5 -- op tracker -- , seq: 3247, time: 2014-06-09 16:22:11.806316, event: dispatched, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.806363 7f869af78700 5 -- op tracker -- , seq: 3247, time: 2014-06-09 16:22:11.806360, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.806369 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4) v4 from osd.1 2014-06-09 16:22:11.806372 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd76000 2014-06-09 16:22:11.806377 7f869af78700 5 -- op tracker -- , seq: 3247, time: 2014-06-09 16:22:11.806375, event: started, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.806399 7f869af78700 5 -- op tracker -- , seq: 3247, time: 2014-06-09 16:22:11.806397, event: done, op: pg_info(1 pgs e243:2.4) 2014-06-09 16:22:11.806409 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806411 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806423 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806425 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806426 7f869af78700 20 osd.26 243 _dispatch 0x31389340 pg_info(1 pgs e243:3.400) v4 2014-06-09 16:22:11.806431 7f869af78700 5 -- op tracker -- , seq: 3248, time: 2014-06-09 16:22:11.659144, event: header_read, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.806438 7f869af78700 5 -- op tracker -- , seq: 3248, time: 2014-06-09 16:22:11.659145, event: throttled, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.806445 7f869af78700 5 -- op tracker -- , seq: 3248, time: 2014-06-09 16:22:11.659160, event: all_read, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.806451 7f869af78700 5 -- op tracker -- , seq: 3248, time: 2014-06-09 16:22:11.806423, event: dispatched, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.806459 7f869af78700 5 -- op tracker -- , seq: 3248, time: 2014-06-09 16:22:11.806457, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.806464 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.400) v4 from osd.8 2014-06-09 16:22:11.806467 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31389340 2014-06-09 16:22:11.806472 7f869af78700 5 -- op tracker -- , seq: 3248, time: 2014-06-09 16:22:11.806470, event: started, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.806491 7f869af78700 5 -- op tracker -- , seq: 3248, time: 2014-06-09 16:22:11.806489, event: done, op: pg_info(1 pgs e243:3.400) 2014-06-09 16:22:11.806499 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806501 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806512 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806513 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806514 7f869af78700 20 osd.26 243 _dispatch 0xddc4fc0 pg_info(1 pgs e243:3.542) v4 2014-06-09 16:22:11.806519 7f869af78700 5 -- op tracker -- , seq: 3249, time: 2014-06-09 16:22:11.805596, event: header_read, op: pg_info(1 pgs e243:3.542) 2014-06-09 16:22:11.806531 7f869af78700 5 -- op tracker -- , seq: 3249, time: 2014-06-09 16:22:11.805597, event: throttled, op: pg_info(1 pgs e243:3.542) 2014-06-09 16:22:11.806539 7f869af78700 5 -- op tracker -- , seq: 3249, time: 2014-06-09 16:22:11.805612, event: all_read, op: pg_info(1 pgs e243:3.542) 2014-06-09 16:22:11.806546 7f869af78700 5 -- op tracker -- , seq: 3249, time: 2014-06-09 16:22:11.806511, event: dispatched, op: pg_info(1 pgs e243:3.542) 2014-06-09 16:22:11.806557 7f869af78700 5 -- op tracker -- , seq: 3249, time: 2014-06-09 16:22:11.806555, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.542) 2014-06-09 16:22:11.806565 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.542) v4 from osd.5 2014-06-09 16:22:11.806568 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc4fc0 2014-06-09 16:22:11.806574 7f869af78700 5 -- op tracker -- , seq: 3249, time: 2014-06-09 16:22:11.806571, event: started, op: pg_info(1 pgs e243:3.542) 2014-06-09 16:22:11.806590 7f869af78700 5 -- op tracker -- , seq: 3249, time: 2014-06-09 16:22:11.806588, event: done, op: pg_info(1 pgs e243:3.542) 2014-06-09 16:22:11.806598 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806600 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806612 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806614 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806615 7f869af78700 20 osd.26 243 _dispatch 0x1ac75c00 pg_info(1 pgs e243:2.431) v4 2014-06-09 16:22:11.806620 7f869af78700 5 -- op tracker -- , seq: 3250, time: 2014-06-09 16:22:11.655887, event: header_read, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.806628 7f869af78700 5 -- op tracker -- , seq: 3250, time: 2014-06-09 16:22:11.655887, event: throttled, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.806634 7f869af78700 5 -- op tracker -- , seq: 3250, time: 2014-06-09 16:22:11.655899, event: all_read, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.806640 7f869af78700 5 -- op tracker -- , seq: 3250, time: 2014-06-09 16:22:11.806612, event: dispatched, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.806647 7f869af78700 5 -- op tracker -- , seq: 3250, time: 2014-06-09 16:22:11.806645, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.806656 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.431) v4 from osd.2 2014-06-09 16:22:11.806659 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac75c00 2014-06-09 16:22:11.806666 7f869af78700 5 -- op tracker -- , seq: 3250, time: 2014-06-09 16:22:11.806662, event: started, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.806683 7f869af78700 5 -- op tracker -- , seq: 3250, time: 2014-06-09 16:22:11.806681, event: done, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.806691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806693 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806704 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806705 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806706 7f869af78700 20 osd.26 243 _dispatch 0x30c13c00 pg_info(1 pgs e243:1.2f8) v4 2014-06-09 16:22:11.806711 7f869af78700 5 -- op tracker -- , seq: 3251, time: 2014-06-09 16:22:11.744753, event: header_read, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806717 7f869af78700 5 -- op tracker -- , seq: 3251, time: 2014-06-09 16:22:11.744754, event: throttled, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806723 7f869af78700 5 -- op tracker -- , seq: 3251, time: 2014-06-09 16:22:11.744766, event: all_read, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806728 7f869af78700 5 -- op tracker -- , seq: 3251, time: 2014-06-09 16:22:11.806703, event: dispatched, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806734 7f869af78700 5 -- op tracker -- , seq: 3251, time: 2014-06-09 16:22:11.806732, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806738 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2f8) v4 from osd.4 2014-06-09 16:22:11.806741 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c13c00 2014-06-09 16:22:11.806745 7f869af78700 5 -- op tracker -- , seq: 3251, time: 2014-06-09 16:22:11.806743, event: started, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806758 7f869af78700 5 -- op tracker -- , seq: 3251, time: 2014-06-09 16:22:11.806756, event: done, op: pg_info(1 pgs e243:1.2f8) 2014-06-09 16:22:11.806766 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806768 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806781 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806782 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806784 7f869af78700 20 osd.26 243 _dispatch 0x30acc8c0 pg_info(1 pgs e243:0.771) v4 2014-06-09 16:22:11.806789 7f869af78700 5 -- op tracker -- , seq: 3252, time: 2014-06-09 16:22:11.747834, event: header_read, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.806796 7f869af78700 5 -- op tracker -- , seq: 3252, time: 2014-06-09 16:22:11.747835, event: throttled, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.806806 7f869af78700 5 -- op tracker -- , seq: 3252, time: 2014-06-09 16:22:11.747844, event: all_read, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.806813 7f869af78700 5 -- op tracker -- , seq: 3252, time: 2014-06-09 16:22:11.806780, event: dispatched, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.806820 7f869af78700 5 -- op tracker -- , seq: 3252, time: 2014-06-09 16:22:11.806818, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.806825 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.771) v4 from osd.1 2014-06-09 16:22:11.806828 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acc8c0 2014-06-09 16:22:11.806833 7f869af78700 5 -- op tracker -- , seq: 3252, time: 2014-06-09 16:22:11.806831, event: started, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.806850 7f869af78700 5 -- op tracker -- , seq: 3252, time: 2014-06-09 16:22:11.806847, event: done, op: pg_info(1 pgs e243:0.771) 2014-06-09 16:22:11.806857 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806858 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806874 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.806875 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.806877 7f869af78700 20 osd.26 243 _dispatch 0x31389180 pg_info(1 pgs e243:2.194) v4 2014-06-09 16:22:11.806882 7f869af78700 5 -- op tracker -- , seq: 3253, time: 2014-06-09 16:22:11.659167, event: header_read, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.806889 7f869af78700 5 -- op tracker -- , seq: 3253, time: 2014-06-09 16:22:11.659168, event: throttled, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.806896 7f869af78700 5 -- op tracker -- , seq: 3253, time: 2014-06-09 16:22:11.659179, event: all_read, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.806904 7f869af78700 5 -- op tracker -- , seq: 3253, time: 2014-06-09 16:22:11.806873, event: dispatched, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.806912 7f869af78700 5 -- op tracker -- , seq: 3253, time: 2014-06-09 16:22:11.806910, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.806916 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.194) v4 from osd.8 2014-06-09 16:22:11.806919 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31389180 2014-06-09 16:22:11.806924 7f869af78700 5 -- op tracker -- , seq: 3253, time: 2014-06-09 16:22:11.806922, event: started, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.806940 7f869af78700 5 -- op tracker -- , seq: 3253, time: 2014-06-09 16:22:11.806938, event: done, op: pg_info(1 pgs e243:2.194) 2014-06-09 16:22:11.808953 7f8682ae5700 5 -- op tracker -- , seq: 3254, time: 2014-06-09 16:22:11.807721, event: header_read, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.808963 7f8682ae5700 5 -- op tracker -- , seq: 3254, time: 2014-06-09 16:22:11.807721, event: throttled, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.808978 7f8682ae5700 5 -- op tracker -- , seq: 3254, time: 2014-06-09 16:22:11.807731, event: all_read, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.808986 7f8682ae5700 5 -- op tracker -- , seq: 3254, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.808991 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) v2 epoch 243 2014-06-09 16:22:11.808996 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.809003 7f8682ae5700 15 osd.26 243 enqueue_op 0xdfa6e00 prio 196 cost 0 latency 0.001282 osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) v2 2014-06-09 16:22:11.809050 7f8682ae5700 5 -- op tracker -- , seq: 3255, time: 2014-06-09 16:22:11.809021, event: header_read, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.809059 7f8682ae5700 5 -- op tracker -- , seq: 3255, time: 2014-06-09 16:22:11.809022, event: throttled, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.809068 7f8682ae5700 5 -- op tracker -- , seq: 3255, time: 2014-06-09 16:22:11.809030, event: all_read, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.809076 7f8682ae5700 5 -- op tracker -- , seq: 3255, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.809080 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.809084 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.809089 7f8682ae5700 15 osd.26 243 enqueue_op 0x32826600 prio 196 cost 0 latency 0.000068 osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.809120 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809123 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809140 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809142 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809143 7f869af78700 20 osd.26 243 _dispatch 0xcd788c0 pg_info(1 pgs e243:2.3e8) v4 2014-06-09 16:22:11.809149 7f869af78700 5 -- op tracker -- , seq: 3256, time: 2014-06-09 16:22:11.655940, event: header_read, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809157 7f869af78700 5 -- op tracker -- , seq: 3256, time: 2014-06-09 16:22:11.655940, event: throttled, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809163 7f869af78700 5 -- op tracker -- , seq: 3256, time: 2014-06-09 16:22:11.655952, event: all_read, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809170 7f869af78700 5 -- op tracker -- , seq: 3256, time: 2014-06-09 16:22:11.809139, event: dispatched, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809183 7f869af78700 5 -- op tracker -- , seq: 3256, time: 2014-06-09 16:22:11.809180, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809188 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3e8) v4 from osd.2 2014-06-09 16:22:11.809191 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd788c0 2014-06-09 16:22:11.809197 7f869af78700 5 -- op tracker -- , seq: 3256, time: 2014-06-09 16:22:11.809194, event: started, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809218 7f869af78700 5 -- op tracker -- , seq: 3256, time: 2014-06-09 16:22:11.809215, event: done, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809229 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809231 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809245 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809247 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809248 7f869af78700 20 osd.26 243 _dispatch 0x30c01500 pg_info(1 pgs e243:0.2a2) v4 2014-06-09 16:22:11.809253 7f869af78700 5 -- op tracker -- , seq: 3257, time: 2014-06-09 16:22:11.748763, event: header_read, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.809260 7f869af78700 5 -- op tracker -- , seq: 3257, time: 2014-06-09 16:22:11.748763, event: throttled, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.809266 7f869af78700 5 -- op tracker -- , seq: 3257, time: 2014-06-09 16:22:11.748775, event: all_read, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.809278 7f869af78700 5 -- op tracker -- , seq: 3257, time: 2014-06-09 16:22:11.809245, event: dispatched, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.809287 7f869af78700 5 -- op tracker -- , seq: 3257, time: 2014-06-09 16:22:11.809285, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.809292 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2a2) v4 from osd.4 2014-06-09 16:22:11.809295 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c01500 2014-06-09 16:22:11.809300 7f869af78700 5 -- op tracker -- , seq: 3257, time: 2014-06-09 16:22:11.809298, event: started, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.809316 7f869af78700 5 -- op tracker -- , seq: 3257, time: 2014-06-09 16:22:11.809314, event: done, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.809323 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809325 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809338 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809339 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809341 7f869af78700 20 osd.26 243 _dispatch 0x31388fc0 pg_info(1 pgs e243:1.601) v4 2014-06-09 16:22:11.809346 7f869af78700 5 -- op tracker -- , seq: 3258, time: 2014-06-09 16:22:11.659184, event: header_read, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.809353 7f869af78700 5 -- op tracker -- , seq: 3258, time: 2014-06-09 16:22:11.659184, event: throttled, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.809362 7f869af78700 5 -- op tracker -- , seq: 3258, time: 2014-06-09 16:22:11.659196, event: all_read, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.809369 7f869af78700 5 -- op tracker -- , seq: 3258, time: 2014-06-09 16:22:11.809338, event: dispatched, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.809376 7f869af78700 5 -- op tracker -- , seq: 3258, time: 2014-06-09 16:22:11.809374, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.809382 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.601) v4 from osd.8 2014-06-09 16:22:11.809385 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31388fc0 2014-06-09 16:22:11.809390 7f869af78700 5 -- op tracker -- , seq: 3258, time: 2014-06-09 16:22:11.809388, event: started, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.809412 7f869af78700 5 -- op tracker -- , seq: 3258, time: 2014-06-09 16:22:11.809410, event: done, op: pg_info(1 pgs e243:1.601) 2014-06-09 16:22:11.809420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809434 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809436 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809437 7f869af78700 20 osd.26 243 _dispatch 0xcd78fc0 pg_info(1 pgs e243:2.431) v4 2014-06-09 16:22:11.809442 7f869af78700 5 -- op tracker -- , seq: 3259, time: 2014-06-09 16:22:11.655960, event: header_read, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.809452 7f869af78700 5 -- op tracker -- , seq: 3259, time: 2014-06-09 16:22:11.655960, event: throttled, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.809459 7f869af78700 5 -- op tracker -- , seq: 3259, time: 2014-06-09 16:22:11.655971, event: all_read, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.809465 7f869af78700 5 -- op tracker -- , seq: 3259, time: 2014-06-09 16:22:11.809433, event: dispatched, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.809472 7f869af78700 5 -- op tracker -- , seq: 3259, time: 2014-06-09 16:22:11.809470, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.809477 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.431) v4 from osd.2 2014-06-09 16:22:11.809480 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd78fc0 2014-06-09 16:22:11.809484 7f869af78700 5 -- op tracker -- , seq: 3259, time: 2014-06-09 16:22:11.809482, event: started, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.809506 7f869af78700 5 -- op tracker -- , seq: 3259, time: 2014-06-09 16:22:11.809504, event: done, op: pg_info(1 pgs e243:2.431) 2014-06-09 16:22:11.809514 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809516 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809528 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809530 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809531 7f869af78700 20 osd.26 243 _dispatch 0xcd76c40 pg_info(1 pgs e243:0.5c3) v4 2014-06-09 16:22:11.809536 7f869af78700 5 -- op tracker -- , seq: 3260, time: 2014-06-09 16:22:11.749857, event: header_read, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.809543 7f869af78700 5 -- op tracker -- , seq: 3260, time: 2014-06-09 16:22:11.749858, event: throttled, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.809549 7f869af78700 5 -- op tracker -- , seq: 3260, time: 2014-06-09 16:22:11.749866, event: all_read, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.809557 7f869af78700 5 -- op tracker -- , seq: 3260, time: 2014-06-09 16:22:11.809528, event: dispatched, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.809564 7f869af78700 5 -- op tracker -- , seq: 3260, time: 2014-06-09 16:22:11.809562, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.809569 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5c3) v4 from osd.4 2014-06-09 16:22:11.809572 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd76c40 2014-06-09 16:22:11.809577 7f869af78700 5 -- op tracker -- , seq: 3260, time: 2014-06-09 16:22:11.809575, event: started, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.809598 7f869af78700 5 -- op tracker -- , seq: 3260, time: 2014-06-09 16:22:11.809596, event: done, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.809605 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809607 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809619 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809621 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809622 7f869af78700 20 osd.26 243 _dispatch 0x30c01dc0 pg_info(1 pgs e243:3.587) v4 2014-06-09 16:22:11.809627 7f869af78700 5 -- op tracker -- , seq: 3261, time: 2014-06-09 16:22:11.659744, event: header_read, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809634 7f869af78700 5 -- op tracker -- , seq: 3261, time: 2014-06-09 16:22:11.659745, event: throttled, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809640 7f869af78700 5 -- op tracker -- , seq: 3261, time: 2014-06-09 16:22:11.659757, event: all_read, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809646 7f869af78700 5 -- op tracker -- , seq: 3261, time: 2014-06-09 16:22:11.809619, event: dispatched, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809656 7f869af78700 5 -- op tracker -- , seq: 3261, time: 2014-06-09 16:22:11.809654, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809661 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.587) v4 from osd.8 2014-06-09 16:22:11.809663 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c01dc0 2014-06-09 16:22:11.809669 7f869af78700 5 -- op tracker -- , seq: 3261, time: 2014-06-09 16:22:11.809667, event: started, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809685 7f869af78700 5 -- op tracker -- , seq: 3261, time: 2014-06-09 16:22:11.809683, event: done, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809693 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809694 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809706 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809708 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809709 7f869af78700 20 osd.26 243 _dispatch 0x30da2700 pg_info(1 pgs e243:2.3e8) v4 2014-06-09 16:22:11.809714 7f869af78700 5 -- op tracker -- , seq: 3262, time: 2014-06-09 16:22:11.655991, event: header_read, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809721 7f869af78700 5 -- op tracker -- , seq: 3262, time: 2014-06-09 16:22:11.655992, event: throttled, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809733 7f869af78700 5 -- op tracker -- , seq: 3262, time: 2014-06-09 16:22:11.656003, event: all_read, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809744 7f869af78700 5 -- op tracker -- , seq: 3262, time: 2014-06-09 16:22:11.809706, event: dispatched, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809752 7f869af78700 5 -- op tracker -- , seq: 3262, time: 2014-06-09 16:22:11.809750, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809758 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3e8) v4 from osd.2 2014-06-09 16:22:11.809762 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da2700 2014-06-09 16:22:11.809767 7f869af78700 5 -- op tracker -- , seq: 3262, time: 2014-06-09 16:22:11.809765, event: started, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809782 7f869af78700 5 -- op tracker -- , seq: 3262, time: 2014-06-09 16:22:11.809780, event: done, op: pg_info(1 pgs e243:2.3e8) 2014-06-09 16:22:11.809794 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809797 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809809 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809810 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809811 7f869af78700 20 osd.26 243 _dispatch 0xcd76540 pg_info(1 pgs e243:1.2a3) v4 2014-06-09 16:22:11.809817 7f869af78700 5 -- op tracker -- , seq: 3263, time: 2014-06-09 16:22:11.749917, event: header_read, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.809825 7f869af78700 5 -- op tracker -- , seq: 3263, time: 2014-06-09 16:22:11.749917, event: throttled, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.809832 7f869af78700 5 -- op tracker -- , seq: 3263, time: 2014-06-09 16:22:11.749925, event: all_read, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.809838 7f869af78700 5 -- op tracker -- , seq: 3263, time: 2014-06-09 16:22:11.809808, event: dispatched, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.809845 7f869af78700 5 -- op tracker -- , seq: 3263, time: 2014-06-09 16:22:11.809843, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.809850 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2a3) v4 from osd.4 2014-06-09 16:22:11.809853 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xcd76540 2014-06-09 16:22:11.809857 7f869af78700 5 -- op tracker -- , seq: 3263, time: 2014-06-09 16:22:11.809856, event: started, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.809873 7f869af78700 5 -- op tracker -- , seq: 3263, time: 2014-06-09 16:22:11.809870, event: done, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.809880 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809882 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809895 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809896 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809897 7f869af78700 20 osd.26 243 _dispatch 0x30c00e00 pg_info(1 pgs e243:3.587) v4 2014-06-09 16:22:11.809903 7f869af78700 5 -- op tracker -- , seq: 3264, time: 2014-06-09 16:22:11.660193, event: header_read, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809909 7f869af78700 5 -- op tracker -- , seq: 3264, time: 2014-06-09 16:22:11.660194, event: throttled, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809918 7f869af78700 5 -- op tracker -- , seq: 3264, time: 2014-06-09 16:22:11.660207, event: all_read, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809924 7f869af78700 5 -- op tracker -- , seq: 3264, time: 2014-06-09 16:22:11.809894, event: dispatched, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809931 7f869af78700 5 -- op tracker -- , seq: 3264, time: 2014-06-09 16:22:11.809929, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809937 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.587) v4 from osd.8 2014-06-09 16:22:11.809940 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c00e00 2014-06-09 16:22:11.809945 7f869af78700 5 -- op tracker -- , seq: 3264, time: 2014-06-09 16:22:11.809943, event: started, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809960 7f869af78700 5 -- op tracker -- , seq: 3264, time: 2014-06-09 16:22:11.809958, event: done, op: pg_info(1 pgs e243:3.587) 2014-06-09 16:22:11.809970 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809971 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809987 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.809989 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.809990 7f869af78700 20 osd.26 243 _dispatch 0x30da2540 pg_info(1 pgs e243:2.52d) v4 2014-06-09 16:22:11.809995 7f869af78700 5 -- op tracker -- , seq: 3265, time: 2014-06-09 16:22:11.656011, event: header_read, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810002 7f869af78700 5 -- op tracker -- , seq: 3265, time: 2014-06-09 16:22:11.656012, event: throttled, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810009 7f869af78700 5 -- op tracker -- , seq: 3265, time: 2014-06-09 16:22:11.656022, event: all_read, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810015 7f869af78700 5 -- op tracker -- , seq: 3265, time: 2014-06-09 16:22:11.809987, event: dispatched, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810025 7f869af78700 5 -- op tracker -- , seq: 3265, time: 2014-06-09 16:22:11.810023, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810031 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.52d) v4 from osd.2 2014-06-09 16:22:11.810033 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da2540 2014-06-09 16:22:11.810038 7f869af78700 5 -- op tracker -- , seq: 3265, time: 2014-06-09 16:22:11.810036, event: started, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810060 7f869af78700 5 -- op tracker -- , seq: 3265, time: 2014-06-09 16:22:11.810058, event: done, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810069 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810082 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810084 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810085 7f869af78700 20 osd.26 243 _dispatch 0x30c908c0 pg_info(1 pgs e243:2.40c) v4 2014-06-09 16:22:11.810090 7f869af78700 5 -- op tracker -- , seq: 3266, time: 2014-06-09 16:22:11.749980, event: header_read, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.810102 7f869af78700 5 -- op tracker -- , seq: 3266, time: 2014-06-09 16:22:11.749981, event: throttled, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.810109 7f869af78700 5 -- op tracker -- , seq: 3266, time: 2014-06-09 16:22:11.749988, event: all_read, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.810116 7f869af78700 5 -- op tracker -- , seq: 3266, time: 2014-06-09 16:22:11.810082, event: dispatched, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.810123 7f869af78700 5 -- op tracker -- , seq: 3266, time: 2014-06-09 16:22:11.810121, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.810128 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.40c) v4 from osd.4 2014-06-09 16:22:11.810131 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c908c0 2014-06-09 16:22:11.810136 7f869af78700 5 -- op tracker -- , seq: 3266, time: 2014-06-09 16:22:11.810133, event: started, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.810151 7f869af78700 5 -- op tracker -- , seq: 3266, time: 2014-06-09 16:22:11.810149, event: done, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.810159 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810160 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810172 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810174 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810175 7f869af78700 20 osd.26 243 _dispatch 0x1ac74000 pg_info(1 pgs e243:0.529) v4 2014-06-09 16:22:11.810180 7f869af78700 5 -- op tracker -- , seq: 3267, time: 2014-06-09 16:22:11.744530, event: header_read, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810187 7f869af78700 5 -- op tracker -- , seq: 3267, time: 2014-06-09 16:22:11.744530, event: throttled, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810198 7f869af78700 5 -- op tracker -- , seq: 3267, time: 2014-06-09 16:22:11.744543, event: all_read, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810209 7f869af78700 5 -- op tracker -- , seq: 3267, time: 2014-06-09 16:22:11.810172, event: dispatched, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810216 7f869af78700 5 -- op tracker -- , seq: 3267, time: 2014-06-09 16:22:11.810214, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810221 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.529) v4 from osd.8 2014-06-09 16:22:11.810224 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac74000 2014-06-09 16:22:11.810229 7f869af78700 5 -- op tracker -- , seq: 3267, time: 2014-06-09 16:22:11.810227, event: started, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810246 7f869af78700 5 -- op tracker -- , seq: 3267, time: 2014-06-09 16:22:11.810243, event: done, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810253 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810255 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810267 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810269 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810270 7f869af78700 20 osd.26 243 _dispatch 0x30da2380 pg_info(1 pgs e243:2.52d) v4 2014-06-09 16:22:11.810275 7f869af78700 5 -- op tracker -- , seq: 3268, time: 2014-06-09 16:22:11.656032, event: header_read, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810285 7f869af78700 5 -- op tracker -- , seq: 3268, time: 2014-06-09 16:22:11.656033, event: throttled, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810291 7f869af78700 5 -- op tracker -- , seq: 3268, time: 2014-06-09 16:22:11.656044, event: all_read, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810298 7f869af78700 5 -- op tracker -- , seq: 3268, time: 2014-06-09 16:22:11.810267, event: dispatched, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810307 7f869af78700 5 -- op tracker -- , seq: 3268, time: 2014-06-09 16:22:11.810304, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810311 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.52d) v4 from osd.2 2014-06-09 16:22:11.810314 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da2380 2014-06-09 16:22:11.810319 7f869af78700 5 -- op tracker -- , seq: 3268, time: 2014-06-09 16:22:11.810317, event: started, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810336 7f869af78700 5 -- op tracker -- , seq: 3268, time: 2014-06-09 16:22:11.810334, event: done, op: pg_info(1 pgs e243:2.52d) 2014-06-09 16:22:11.810343 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810345 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810357 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810359 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810360 7f869af78700 20 osd.26 243 _dispatch 0x30c91180 pg_info(1 pgs e243:1.14c) v4 2014-06-09 16:22:11.810366 7f869af78700 5 -- op tracker -- , seq: 3269, time: 2014-06-09 16:22:11.749993, event: header_read, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.810374 7f869af78700 5 -- op tracker -- , seq: 3269, time: 2014-06-09 16:22:11.749994, event: throttled, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.810385 7f869af78700 5 -- op tracker -- , seq: 3269, time: 2014-06-09 16:22:11.750002, event: all_read, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.810396 7f869af78700 5 -- op tracker -- , seq: 3269, time: 2014-06-09 16:22:11.810357, event: dispatched, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.810407 7f869af78700 5 -- op tracker -- , seq: 3269, time: 2014-06-09 16:22:11.810405, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.810412 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.14c) v4 from osd.4 2014-06-09 16:22:11.810415 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c91180 2014-06-09 16:22:11.810420 7f869af78700 5 -- op tracker -- , seq: 3269, time: 2014-06-09 16:22:11.810418, event: started, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.810438 7f869af78700 5 -- op tracker -- , seq: 3269, time: 2014-06-09 16:22:11.810436, event: done, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.810452 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810455 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810468 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810470 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810472 7f869af78700 20 osd.26 243 _dispatch 0x1ac741c0 pg_info(1 pgs e243:2.48f) v4 2014-06-09 16:22:11.810477 7f869af78700 5 -- op tracker -- , seq: 3270, time: 2014-06-09 16:22:11.744551, event: header_read, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.810484 7f869af78700 5 -- op tracker -- , seq: 3270, time: 2014-06-09 16:22:11.744551, event: throttled, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.810491 7f869af78700 5 -- op tracker -- , seq: 3270, time: 2014-06-09 16:22:11.744566, event: all_read, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.810498 7f869af78700 5 -- op tracker -- , seq: 3270, time: 2014-06-09 16:22:11.810468, event: dispatched, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.810507 7f869af78700 5 -- op tracker -- , seq: 3270, time: 2014-06-09 16:22:11.810505, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.810512 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.48f) v4 from osd.8 2014-06-09 16:22:11.810515 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac741c0 2014-06-09 16:22:11.810521 7f869af78700 5 -- op tracker -- , seq: 3270, time: 2014-06-09 16:22:11.810519, event: started, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.810543 7f869af78700 5 -- op tracker -- , seq: 3270, time: 2014-06-09 16:22:11.810541, event: done, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.810550 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810552 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810562 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810564 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810565 7f869af78700 20 osd.26 243 _dispatch 0x30da21c0 pg_info(1 pgs e243:3.25a) v4 2014-06-09 16:22:11.810569 7f869af78700 5 -- op tracker -- , seq: 3271, time: 2014-06-09 16:22:11.739424, event: header_read, op: pg_info(1 pgs e243:3.25a) 2014-06-09 16:22:11.810575 7f869af78700 5 -- op tracker -- , seq: 3271, time: 2014-06-09 16:22:11.739425, event: throttled, op: pg_info(1 pgs e243:3.25a) 2014-06-09 16:22:11.810581 7f869af78700 5 -- op tracker -- , seq: 3271, time: 2014-06-09 16:22:11.739438, event: all_read, op: pg_info(1 pgs e243:3.25a) 2014-06-09 16:22:11.810586 7f869af78700 5 -- op tracker -- , seq: 3271, time: 2014-06-09 16:22:11.810562, event: dispatched, op: pg_info(1 pgs e243:3.25a) 2014-06-09 16:22:11.810592 7f869af78700 5 -- op tracker -- , seq: 3271, time: 2014-06-09 16:22:11.810590, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.25a) 2014-06-09 16:22:11.810596 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.25a) v4 from osd.2 2014-06-09 16:22:11.810599 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da21c0 2014-06-09 16:22:11.810604 7f869af78700 5 -- op tracker -- , seq: 3271, time: 2014-06-09 16:22:11.810602, event: started, op: pg_info(1 pgs e243:3.25a) 2014-06-09 16:22:11.810621 7f869af78700 5 -- op tracker -- , seq: 3271, time: 2014-06-09 16:22:11.810619, event: done, op: pg_info(1 pgs e243:3.25a) 2014-06-09 16:22:11.810628 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810631 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810643 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810644 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810646 7f869af78700 20 osd.26 243 _dispatch 0x30c91500 pg_info(1 pgs e243:3.527) v4 2014-06-09 16:22:11.810651 7f869af78700 5 -- op tracker -- , seq: 3272, time: 2014-06-09 16:22:11.750039, event: header_read, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.810659 7f869af78700 5 -- op tracker -- , seq: 3272, time: 2014-06-09 16:22:11.750039, event: throttled, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.810668 7f869af78700 5 -- op tracker -- , seq: 3272, time: 2014-06-09 16:22:11.750047, event: all_read, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.810675 7f869af78700 5 -- op tracker -- , seq: 3272, time: 2014-06-09 16:22:11.810642, event: dispatched, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.810682 7f869af78700 5 -- op tracker -- , seq: 3272, time: 2014-06-09 16:22:11.810680, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.810691 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.527) v4 from osd.4 2014-06-09 16:22:11.810695 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c91500 2014-06-09 16:22:11.810700 7f869af78700 5 -- op tracker -- , seq: 3272, time: 2014-06-09 16:22:11.810698, event: started, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.810719 7f869af78700 5 -- op tracker -- , seq: 3272, time: 2014-06-09 16:22:11.810716, event: done, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.810726 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810728 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810740 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810742 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810743 7f869af78700 20 osd.26 243 _dispatch 0x240efa40 pg_info(1 pgs e243:0.529) v4 2014-06-09 16:22:11.810749 7f869af78700 5 -- op tracker -- , seq: 3273, time: 2014-06-09 16:22:11.744959, event: header_read, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810756 7f869af78700 5 -- op tracker -- , seq: 3273, time: 2014-06-09 16:22:11.744960, event: throttled, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810763 7f869af78700 5 -- op tracker -- , seq: 3273, time: 2014-06-09 16:22:11.744972, event: all_read, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810769 7f869af78700 5 -- op tracker -- , seq: 3273, time: 2014-06-09 16:22:11.810740, event: dispatched, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810782 7f869af78700 5 -- op tracker -- , seq: 3273, time: 2014-06-09 16:22:11.810779, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810786 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.529) v4 from osd.8 2014-06-09 16:22:11.810789 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x240efa40 2014-06-09 16:22:11.810794 7f869af78700 5 -- op tracker -- , seq: 3273, time: 2014-06-09 16:22:11.810792, event: started, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810808 7f869af78700 5 -- op tracker -- , seq: 3273, time: 2014-06-09 16:22:11.810806, event: done, op: pg_info(1 pgs e243:0.529) 2014-06-09 16:22:11.810815 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810817 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810829 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.810831 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.810832 7f869af78700 20 osd.26 243 _dispatch 0x30da2000 pg_info(1 pgs e243:3.15c) v4 2014-06-09 16:22:11.810837 7f869af78700 5 -- op tracker -- , seq: 3274, time: 2014-06-09 16:22:11.740481, event: header_read, op: pg_info(1 pgs e243:3.15c) 2014-06-09 16:22:11.810844 7f869af78700 5 -- op tracker -- , seq: 3274, time: 2014-06-09 16:22:11.740481, event: throttled, op: pg_info(1 pgs e243:3.15c) 2014-06-09 16:22:11.810850 7f869af78700 5 -- op tracker -- , seq: 3274, time: 2014-06-09 16:22:11.740490, event: all_read, op: pg_info(1 pgs e243:3.15c) 2014-06-09 16:22:11.810859 7f869af78700 5 -- op tracker -- , seq: 3274, time: 2014-06-09 16:22:11.810829, event: dispatched, op: pg_info(1 pgs e243:3.15c) 2014-06-09 16:22:11.810867 7f869af78700 5 -- op tracker -- , seq: 3274, time: 2014-06-09 16:22:11.810864, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.15c) 2014-06-09 16:22:11.810871 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.15c) v4 from osd.2 2014-06-09 16:22:11.810874 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30da2000 2014-06-09 16:22:11.810879 7f869af78700 5 -- op tracker -- , seq: 3274, time: 2014-06-09 16:22:11.810877, event: started, op: pg_info(1 pgs e243:3.15c) 2014-06-09 16:22:11.811952 7f86a5610700 10 _do_op 0x30d97260 seq 3478 r = 0, finisher 0x30e19680 0x2a0939c0 2014-06-09 16:22:11.811960 7f86a5610700 10 _finish_op 0x30d97260 seq 3478 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.811971 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.811975 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2870e0 (2 active) 2014-06-09 16:22:11.811978 7f86a5610700 5 _do_op 0xbe7d860 seq 3480 osr(3.7ce 0xb7f01a0)/0xb7f01a0 start 2014-06-09 16:22:11.811981 7f86a5610700 10 _do_transaction on 0xc038cc0 2014-06-09 16:22:11.811988 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.812063 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.812081 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3480.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.812085 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.812098 7f86a5610700 10 _do_op 0xbe7d860 seq 3480 r = 0, finisher 0xbccde40 0 2014-06-09 16:22:11.812102 7f86a5610700 10 _finish_op 0xbe7d860 seq 3480 osr(3.7ce 0xb7f01a0)/0xb7f01a0 2014-06-09 16:22:11.812104 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2870e0 (1 active) 2014-06-09 16:22:11.812106 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c5a0 (2 active) 2014-06-09 16:22:11.812108 7f86a5610700 5 _do_op 0x10038000 seq 3481 osr(3.2a5 0x3ed0b80)/0x3ed0b80 start 2014-06-09 16:22:11.812110 7f86a5610700 10 _do_transaction on 0x2ed5ec0 2014-06-09 16:22:11.812116 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.812148 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.812169 7f86a5610700 15 _omap_rmkeyrange meta/1fe137e5/pglog_3.2a5/0//-1 [0000000214.00000000000000000632,4294967295.18446744073709551615] 2014-06-09 16:22:11.812173 7f86a5610700 15 get_omap_iterator meta/1fe137e5/pglog_3.2a5/0//-1 2014-06-09 16:22:11.812273 7f86a5610700 15 _omap_rmkeys meta/1fe137e5/pglog_3.2a5/0//-1 2014-06-09 16:22:11.812306 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe137e5/pglog_3.2a5/0//-1 2014-06-09 16:22:11.812316 7f86a5610700 10 _omap_rmkeys meta/1fe137e5/pglog_3.2a5/0//-1 r = 0 2014-06-09 16:22:11.812321 7f86a5610700 15 _omap_rmkeys meta/1fe137e5/pglog_3.2a5/0//-1 2014-06-09 16:22:11.812324 7f86a5610700 10 _omap_rmkeys meta/1fe137e5/pglog_3.2a5/0//-1 r = 0 2014-06-09 16:22:11.812332 7f86a5610700 15 _omap_setkeys meta/1fe137e5/pglog_3.2a5/0//-1 2014-06-09 16:22:11.812352 7f86a5610700 10 cid: oid: 1fe137e5/pglog_3.2a5/0//-1 skipping op, spos 3481.0.0 <= header.spos 150410.0.0 2014-06-09 16:22:11.812357 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3481.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.812359 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.812396 7f86a5610700 10 _do_op 0x10038000 seq 3481 r = 0, finisher 0x318c2fa0 0 2014-06-09 16:22:11.812400 7f86a5610700 10 _finish_op 0x10038000 seq 3481 osr(3.2a5 0x3ed0b80)/0x3ed0b80 2014-06-09 16:22:11.812402 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c5a0 (1 active) 2014-06-09 16:22:11.812404 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fdd40 (2 active) 2014-06-09 16:22:11.812406 7f86a5610700 5 _do_op 0x16fd55c0 seq 3482 osr(3.544 0x3ed19a0)/0x3ed19a0 start 2014-06-09 16:22:11.812409 7f86a5610700 10 _do_transaction on 0xc2563c0 2014-06-09 16:22:11.812415 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.812447 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.812466 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3482.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.812470 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.812486 7f86a5610700 10 _do_op 0x16fd55c0 seq 3482 r = 0, finisher 0x318c3720 0 2014-06-09 16:22:11.812489 7f86a5610700 10 _finish_op 0x16fd55c0 seq 3482 osr(3.544 0x3ed19a0)/0x3ed19a0 2014-06-09 16:22:11.812491 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fdd40 (1 active) 2014-06-09 16:22:11.812493 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1150e0 (2 active) 2014-06-09 16:22:11.812495 7f86a5610700 5 _do_op 0x2fcae4e0 seq 3483 osr(2.37 0x2ed05f0)/0x2ed05f0 start 2014-06-09 16:22:11.812497 7f86a5610700 10 _do_transaction on 0xc04f200 2014-06-09 16:22:11.812502 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.812533 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.812547 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3483.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.812550 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.812563 7f86a5610700 10 _do_op 0x2fcae4e0 seq 3483 r = 0, finisher 0xc6e80e0 0 2014-06-09 16:22:11.812567 7f86a5610700 10 _finish_op 0x2fcae4e0 seq 3483 osr(2.37 0x2ed05f0)/0x2ed05f0 2014-06-09 16:22:11.812570 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1150e0 (1 active) 2014-06-09 16:22:11.812572 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c0900 (2 active) 2014-06-09 16:22:11.812574 7f86a5610700 5 _do_op 0x16fd5b00 seq 3484 osr(3.48d 0x3ed14e0)/0x3ed14e0 start 2014-06-09 16:22:11.812577 7f86a5610700 10 _do_transaction on 0xbe88480 2014-06-09 16:22:11.812582 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.812622 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.812647 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3484.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.812653 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.812674 7f86a5610700 10 _do_op 0x16fd5b00 seq 3484 r = 0, finisher 0xc1681c0 0 2014-06-09 16:22:11.812680 7f86a5610700 10 _finish_op 0x16fd5b00 seq 3484 osr(3.48d 0x3ed14e0)/0x3ed14e0 2014-06-09 16:22:11.812684 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c0900 (1 active) 2014-06-09 16:22:11.812687 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6537a0 (2 active) 2014-06-09 16:22:11.812690 7f86a5610700 5 _do_op 0x16fd4cc0 seq 3485 osr(0.298 0x2bd6c60)/0x2bd6c60 start 2014-06-09 16:22:11.812695 7f86a5610700 10 _do_transaction on 0xc22de00 2014-06-09 16:22:11.812700 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.812733 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.812747 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3485.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.812750 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.812765 7f86a5610700 10 _do_op 0x16fd4cc0 seq 3485 r = 0, finisher 0xbd71340 0 2014-06-09 16:22:11.812768 7f86a5610700 10 _finish_op 0x16fd4cc0 seq 3485 osr(0.298 0x2bd6c60)/0x2bd6c60 2014-06-09 16:22:11.812771 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6537a0 (1 active) 2014-06-09 16:22:11.812773 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4a20 (2 active) 2014-06-09 16:22:11.812775 7f86a5610700 5 _do_op 0x230b96e0 seq 3486 osr(3.1e 0x3ed0380)/0x3ed0380 start 2014-06-09 16:22:11.812777 7f86a5610700 10 _do_transaction on 0xbfa46c0 2014-06-09 16:22:11.812781 7f86a5610700 15 _omap_setkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.812822 7f86a5610700 10 lookup_strip_header done cid meta oid 51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.812836 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.812865 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.812879 7f86a5610700 15 _omap_rmkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.812884 7f86a5610700 10 _omap_rmkeys meta/51682acc/pglog_3.1e/0//-1 r = 0 2014-06-09 16:22:11.812890 7f86a5610700 15 _omap_setkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.812903 7f86a5610700 15 _write 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 868352~516096 2014-06-09 16:22:11.812961 7f86a5610700 10 lookup_strip_header done cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.812977 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.813059 7f866f5f0700 10 osd.26 243 new session 0xcbdd600 con=0x30ba06e0 addr=172.16.96.220:0/1123664 2014-06-09 16:22:11.814881 7f86a5610700 10 _generic_write 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 868352~516096 = 0 2014-06-09 16:22:11.814920 7f86a5610700 15 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.814935 7f86a5610700 10 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.814942 7f86a5610700 15 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.814951 7f86a5610700 10 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.815027 7f86a5610700 10 cid: oid: 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 not skipping op, *spos 3486.0.0 2014-06-09 16:22:11.815031 7f86a5610700 10 > header.spos 3474.0.0 2014-06-09 16:22:11.815060 7f86a5610700 20 set_header setting 30153 cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 parent seq 0 2014-06-09 16:22:11.815093 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3486.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.815097 7f86a5610700 10 cid: oid: 51682acc/pglog_3.1e/0//-1 skipping op, spos 3486.0.0 <= header.spos 150326.0.0 2014-06-09 16:22:11.815099 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.817017 7f86a5610700 10 _do_op 0x230b96e0 seq 3486 r = 0, finisher 0x30508660 0x30c40600 2014-06-09 16:22:11.817024 7f86a5610700 10 _finish_op 0x230b96e0 seq 3486 osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.817029 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4a20 (1 active) 2014-06-09 16:22:11.817032 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780ea0 (2 active) 2014-06-09 16:22:11.817034 7f86a5610700 5 _do_op 0xbc2dec0 seq 3487 osr(1.438 0x384d710)/0x384d710 start 2014-06-09 16:22:11.817036 7f86a5610700 10 _do_transaction on 0xc252300 2014-06-09 16:22:11.817043 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.817096 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.817112 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3487.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.817115 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.817130 7f86a5610700 10 _do_op 0xbc2dec0 seq 3487 r = 0, finisher 0xbd7ce00 0 2014-06-09 16:22:11.817132 7f86a5610700 10 _finish_op 0xbc2dec0 seq 3487 osr(1.438 0x384d710)/0x384d710 2014-06-09 16:22:11.817134 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780ea0 (1 active) 2014-06-09 16:22:11.817135 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca49e60 (2 active) 2014-06-09 16:22:11.817137 7f86a5610700 5 _do_op 0xc2eba40 seq 3488 osr(3.1b1 0x3ed0a70)/0x3ed0a70 start 2014-06-09 16:22:11.817139 7f86a5610700 10 _do_transaction on 0xc23f140 2014-06-09 16:22:11.817138 7f866f5f0700 5 -- op tracker -- , seq: 3275, time: 2014-06-09 16:22:11.813308, event: header_read, op: osd_op(client.4539.0:34389 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.817145 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.817152 7f866f5f0700 5 -- op tracker -- , seq: 3275, time: 2014-06-09 16:22:11.813311, event: throttled, op: osd_op(client.4539.0:34389 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.817175 7f866f5f0700 5 -- op tracker -- , seq: 3275, time: 2014-06-09 16:22:11.817100, event: all_read, op: osd_op(client.4539.0:34389 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.817180 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.817182 7f868a757700 10 op_submit_start 3528 2014-06-09 16:22:11.817185 7f868a757700 5 queue_transactions (trailing journal) 3528 2014-06-09 16:22:11.817185 7f866f5f0700 5 -- op tracker -- , seq: 3275, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34389 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.817187 7f868a757700 5 queue_op 0x308caa20 seq 3528 osr(0.71f 0x384c060) 1903 bytes (queue has 42 ops and 2256795 bytes) 2014-06-09 16:22:11.817191 7f868a757700 10 op_submit_finish 3528 2014-06-09 16:22:11.817197 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3488.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.817195 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.817200 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.817197 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.817212 7f86a5610700 10 _do_op 0xc2eba40 seq 3488 r = 0, finisher 0x2e0ffc0 0 2014-06-09 16:22:11.817215 7f86a5610700 10 _finish_op 0xc2eba40 seq 3488 osr(3.1b1 0x3ed0a70)/0x3ed0a70 2014-06-09 16:22:11.817217 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49e60 (1 active) 2014-06-09 16:22:11.817219 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b8c0 (2 active) 2014-06-09 16:22:11.817221 7f86a5610700 5 _do_op 0xc2d5da0 seq 3489 osr(0.763 0x384c350)/0x384c350 start 2014-06-09 16:22:11.817214 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.817223 7f86a5610700 10 _do_transaction on 0xbf278c0 2014-06-09 16:22:11.817219 7f866f5f0700 15 osd.26 243 enqueue_op 0xdfa7d00 prio 63 cost 516096 latency 0.003911 osd_op(client.4539.0:34389 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.817228 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.817223 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817231 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817250 7f868a757700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.817259 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.817274 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3489.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.817278 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.817271 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.817288 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817295 7f86a5610700 10 _do_op 0xc2d5da0 seq 3489 r = 0, finisher 0x30e19940 0 2014-06-09 16:22:11.817298 7f86a5610700 10 _finish_op 0xc2d5da0 seq 3489 osr(0.763 0x384c350)/0x384c350 2014-06-09 16:22:11.817300 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b8c0 (1 active) 2014-06-09 16:22:11.817302 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780900 (2 active) 2014-06-09 16:22:11.817294 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.372( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.817304 7f86a5610700 5 _do_op 0x309a2cc0 seq 3490 osr(3.11f 0x3ed07e0)/0x3ed07e0 start 2014-06-09 16:22:11.817306 7f86a5610700 10 _do_transaction on 0xc336900 2014-06-09 16:22:11.817312 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.817331 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817338 7f868a757700 5 queue_transactions existing osr(2.372 0x2ed1150)/0x2ed1150 2014-06-09 16:22:11.817344 7f868a757700 10 op_submit_start 3529 2014-06-09 16:22:11.817343 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.817345 7f868a757700 5 queue_transactions (trailing journal) 3529 2014-06-09 16:22:11.817346 7f868a757700 5 queue_op 0xcdab860 seq 3529 osr(2.372 0x2ed1150) 828 bytes (queue has 41 ops and 2254405 bytes) 2014-06-09 16:22:11.817348 7f868a757700 10 op_submit_finish 3529 2014-06-09 16:22:11.817358 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3490.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.817362 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.817351 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.817366 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.817377 7f86a5610700 10 _do_op 0x309a2cc0 seq 3490 r = 0, finisher 0xbd5b6a0 0 2014-06-09 16:22:11.817380 7f86a5610700 10 _finish_op 0x309a2cc0 seq 3490 osr(3.11f 0x3ed07e0)/0x3ed07e0 2014-06-09 16:22:11.817382 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780900 (1 active) 2014-06-09 16:22:11.817376 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817383 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c240 (2 active) 2014-06-09 16:22:11.817385 7f86a5610700 5 _do_op 0xc34a360 seq 3491 osr(3.2a9 0x3ed0b40)/0x3ed0b40 start 2014-06-09 16:22:11.817388 7f86a5610700 10 _do_transaction on 0xbf455c0 2014-06-09 16:22:11.817384 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817393 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.817396 7f868a757700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.817413 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1051 2014-06-09 16:22:11.817425 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.817429 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817440 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3491.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.817434 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.301( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.817444 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.817456 7f86a5610700 10 _do_op 0xc34a360 seq 3491 r = 0, finisher 0x3042ace0 0 2014-06-09 16:22:11.817459 7f86a5610700 10 _finish_op 0xc34a360 seq 3491 osr(3.2a9 0x3ed0b40)/0x3ed0b40 2014-06-09 16:22:11.817461 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c240 (1 active) 2014-06-09 16:22:11.817464 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817463 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.817466 7f86a5610700 5 _do_op 0x2fc8d7a0 seq 3492 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.817468 7f86a5610700 10 _do_transaction on 0xc2af680 2014-06-09 16:22:11.817468 7f868a757700 5 queue_transactions existing osr(0.301 0x2bd6e10)/0x2bd6e10 2014-06-09 16:22:11.817471 7f868a757700 10 op_submit_start 3530 2014-06-09 16:22:11.817472 7f868a757700 5 queue_transactions (trailing journal) 3530 2014-06-09 16:22:11.817472 7f86a5610700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.817473 7f868a757700 5 queue_op 0xcdabf20 seq 3530 osr(0.301 0x2bd6e10) 828 bytes (queue has 40 ops and 2252563 bytes) 2014-06-09 16:22:11.817475 7f868a757700 10 op_submit_finish 3530 2014-06-09 16:22:11.817478 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.75( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.75( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.817508 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817513 7f868a757700 5 queue_transactions existing osr(2.75 0x2ed04f0)/0x2ed04f0 2014-06-09 16:22:11.817513 7f86a5610700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.817515 7f868a757700 10 op_submit_start 3531 2014-06-09 16:22:11.817516 7f868a757700 5 queue_transactions (trailing journal) 3531 2014-06-09 16:22:11.817518 7f868a757700 5 queue_op 0xcdaa3c0 seq 3531 osr(2.75 0x2ed04f0) 826 bytes (queue has 41 ops and 2253389 bytes) 2014-06-09 16:22:11.817520 7f868a757700 10 op_submit_finish 3531 2014-06-09 16:22:11.817527 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.817523 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:3.4b9( v 214'558 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.817539 7f868a757700 7 osd.26 pg_epoch: 243 pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.5 2014-06-09 16:22:11.817555 7f868b759700 10 op_submit_start 3532 2014-06-09 16:22:11.817558 7f868b759700 5 queue_transactions (trailing journal) 3532 2014-06-09 16:22:11.817557 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.817559 7f868b759700 5 queue_op 0x2e20ae0 seq 3532 osr(0.8c 0x2bd6480) 826 bytes (queue has 42 ops and 2254215 bytes) 2014-06-09 16:22:11.817563 7f868b759700 10 op_submit_finish 3532 2014-06-09 16:22:11.817550 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.5 info 3.4b9( v 214'558 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242), identical to ours 2014-06-09 16:22:11.817570 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817571 7f86a5610700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.817576 7f86a5610700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.817567 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.817581 7f86a5610700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.817575 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 5 notify: (query_epoch:243, epoch_sent:243, info:1.1( empty local-les=206 n=0 ec=1 les/c 206/206 242/242/242)) 2014-06-09 16:22:11.817582 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.817587 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 5, already have info from that osd, ignoring 2014-06-09 16:22:11.817594 7f86a5610700 15 _write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2281472~516096 2014-06-09 16:22:11.817597 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817592 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817600 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817601 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.817613 7f868b759700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.817613 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.817621 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817628 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817631 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.817639 7f868a757700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.817646 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.817651 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.817656 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:185 2014-06-09 16:22:11.817663 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.817672 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817671 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817678 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817676 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.817689 7f868b759700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.817688 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.817696 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817703 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817700 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:179 2014-06-09 16:22:11.817712 7f86a5610700 10 lookup_strip_header done cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.817714 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.817714 7f868a757700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.817726 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.817718 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.817731 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.817730 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.817744 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817739 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817746 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817748 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.817757 7f868b759700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.817759 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.817767 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817767 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.817781 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.817776 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817786 7f868a757700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.817785 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.521( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.521( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.817798 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.817811 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817818 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.817814 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.38d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.817823 7f868b759700 5 queue_transactions existing osr(0.521 0x2bd7570)/0x2bd7570 2014-06-09 16:22:11.817826 7f868b759700 10 op_submit_start 3533 2014-06-09 16:22:11.817827 7f868b759700 5 queue_transactions (trailing journal) 3533 2014-06-09 16:22:11.817828 7f868b759700 5 queue_op 0x2e213e0 seq 3533 osr(0.521 0x2bd7570) 828 bytes (queue has 43 ops and 2255043 bytes) 2014-06-09 16:22:11.817831 7f868b759700 10 op_submit_finish 3533 2014-06-09 16:22:11.817833 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.817851 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817846 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.817855 7f868a757700 5 queue_transactions existing osr(1.38d 0x384d570)/0x384d570 2014-06-09 16:22:11.817858 7f868a757700 10 op_submit_start 3534 2014-06-09 16:22:11.817859 7f868a757700 5 queue_transactions (trailing journal) 3534 2014-06-09 16:22:11.817854 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.817860 7f868a757700 5 queue_op 0xcdab0e0 seq 3534 osr(1.38d 0x384d570) 828 bytes (queue has 44 ops and 2255871 bytes) 2014-06-09 16:22:11.817863 7f868a757700 10 op_submit_finish 3534 2014-06-09 16:22:11.817861 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.817872 7f868b759700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.817865 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.6d3( v 214'632 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.817883 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.817883 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.9 activated and committed 2014-06-09 16:22:11.817897 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.817895 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:11.817905 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:11.817919 7f868a757700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.817919 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15359d20 (19 active) 2014-06-09 16:22:11.817927 7f868b759700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.817929 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1056 2014-06-09 16:22:11.817945 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817949 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.4e0( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.817976 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.817976 7f8690763700 10 op_submit_start 3535 2014-06-09 16:22:11.817979 7f8690763700 5 queue_transactions (trailing journal) 3535 2014-06-09 16:22:11.817981 7f868a757700 5 queue_transactions existing osr(2.4e0 0x2ed16f0)/0x2ed16f0 2014-06-09 16:22:11.817980 7f8690763700 5 queue_op 0x2e20fc0 seq 3535 osr(1.631 0x384dfb0) 828 bytes (queue has 45 ops and 2256699 bytes) 2014-06-09 16:22:11.817983 7f8690763700 10 op_submit_finish 3535 2014-06-09 16:22:11.817992 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.818006 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.3 0.1f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.818018 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.3 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.818022 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.818037 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 0.915522 7 0.000185 2014-06-09 16:22:11.818049 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.818061 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.818067 7f868a757700 10 op_submit_start 3536 2014-06-09 16:22:11.818070 7f868a757700 5 queue_transactions (trailing journal) 3536 2014-06-09 16:22:11.818071 7f868a757700 5 queue_op 0xcdab6e0 seq 3536 osr(2.4e0 0x2ed16f0) 828 bytes (queue has 46 ops and 2257527 bytes) 2014-06-09 16:22:11.818074 7f868a757700 10 op_submit_finish 3536 2014-06-09 16:22:11.818070 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.818080 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.818078 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.a2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.818088 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.818133 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.818137 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.818138 7f868a757700 5 queue_transactions existing osr(1.a2 0x384c790)/0x384c790 2014-06-09 16:22:11.818141 7f868a757700 10 op_submit_start 3537 2014-06-09 16:22:11.818142 7f868a757700 5 queue_transactions (trailing journal) 3537 2014-06-09 16:22:11.818143 7f868a757700 5 queue_op 0xcdaa1e0 seq 3537 osr(1.a2 0x384c790) 826 bytes (queue has 47 ops and 2258353 bytes) 2014-06-09 16:22:11.818146 7f868a757700 10 op_submit_finish 3537 2014-06-09 16:22:11.818150 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=206 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.792( v 213'667 (0'0,213'667] local-les=243 n=135 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.818161 7f8690763700 5 queue_transactions existing osr(0.1f9 0x2bd6880)/0x2bd6880 2014-06-09 16:22:11.818164 7f8690763700 10 op_submit_start 3538 2014-06-09 16:22:11.818165 7f8690763700 5 queue_transactions (trailing journal) 3538 2014-06-09 16:22:11.818167 7f8690763700 5 queue_op 0x2e209c0 seq 3538 osr(0.1f9 0x2bd6880) 2009 bytes (queue has 48 ops and 2260362 bytes) 2014-06-09 16:22:11.818170 7f8690763700 10 op_submit_finish 3538 2014-06-09 16:22:11.818164 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=206 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.5 3.792( v 213'667 (0'0,213'667] local-les=243 n=135 ec=118 les/c 206/206 242/242/205) 2014-06-09 16:22:11.818173 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.818178 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=206 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.024324 7 0.000171 2014-06-09 16:22:11.818189 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=206 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.818200 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=206 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.818210 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=206 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.818220 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=243 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'667 -> 213'667 2014-06-09 16:22:11.818186 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.818264 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.818271 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.818273 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=243 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.818285 7f8690763700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.818300 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.818312 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.818313 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.818316 7f868a757700 5 queue_transactions existing osr(3.792 0x3ed1fc0)/0x3ed1fc0 2014-06-09 16:22:11.818318 7f868a757700 10 op_submit_start 3539 2014-06-09 16:22:11.818319 7f868a757700 5 queue_transactions (trailing journal) 3539 2014-06-09 16:22:11.818320 7f868a757700 5 queue_op 0xcdaa480 seq 3539 osr(3.792 0x3ed1fc0) 1335 bytes (queue has 49 ops and 2261697 bytes) 2014-06-09 16:22:11.818323 7f868a757700 10 op_submit_finish 3539 2014-06-09 16:22:11.818318 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.1bd( v 214'661 (0'0,214'661] local-les=212 n=131 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.818331 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.818339 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (19 active) 2014-06-09 16:22:11.818342 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.818345 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.750( v 214'518 (0'0,214'518] local-les=212 n=98 ec=118 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.818360 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0310 (20 active) 2014-06-09 16:22:11.818358 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.818369 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.818363 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MLogRec from 5 2014-06-09 16:22:11.818373 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.818375 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd5 2014-06-09 16:22:11.818385 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.818386 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.818392 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.818396 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.818400 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.818411 7f8690763700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.818405 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'556 (0'0,214'556] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.5: log((214'556,214'558], crt=0'0) missing(0) 2014-06-09 16:22:11.818416 7f868a757700 10 merge_log log((214'556,214'558], crt=0'0) from osd.5 into log((0'0,214'556], crt=0'0) 2014-06-09 16:22:11.818420 7f868a757700 10 merge_log extending head to 214'558 2014-06-09 16:22:11.818422 7f868a757700 20 ? 214'558 (214'557) modify 975e64b9/rbd_data.11716b8b4567.0000000000001bb1/head//3 by client.4561.0:40170 2014-06-09 10:19:33.713473 2014-06-09 16:22:11.818426 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.818431 7f868a757700 20 ? 214'557 (214'556) modify 975e64b9/rbd_data.11716b8b4567.0000000000001bb1/head//3 by client.4561.0:40169 2014-06-09 10:19:33.179567 2014-06-09 16:22:11.818440 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.818439 7f868a757700 20 merge_log 214'557 (214'556) modify 975e64b9/rbd_data.11716b8b4567.0000000000001bb1/head//3 by client.4561.0:40169 2014-06-09 10:19:33.179567 2014-06-09 16:22:11.818443 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:11.818452 7f868a757700 20 merge_log 214'558 (214'557) modify 975e64b9/rbd_data.11716b8b4567.0000000000001bb1/head//3 by client.4561.0:40170 2014-06-09 10:19:33.713473 2014-06-09 16:22:11.818460 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.818469 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.818477 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.818489 7f8690763700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.818503 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:962 2014-06-09 16:22:11.818517 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.818521 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.818533 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.818545 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.882399 7 0.000180 2014-06-09 16:22:11.818556 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.818567 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.818577 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=191 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.818587 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'629 -> 213'629 2014-06-09 16:22:11.818638 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.818665 7f868a757700 10 merge_log result log((0'0,214'558], crt=214'558) missing(1) changed=1 2014-06-09 16:22:11.818678 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.818683 7f8690763700 5 queue_transactions existing osr(3.712 0x3ed1b90)/0x3ed1b90 2014-06-09 16:22:11.818671 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 peering m=1] peer osd.5 now 3.4b9( v 214'558 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) missing(0) 2014-06-09 16:22:11.818686 7f8690763700 10 op_submit_start 3540 2014-06-09 16:22:11.818687 7f8690763700 5 queue_transactions (trailing journal) 3540 2014-06-09 16:22:11.818688 7f8690763700 5 queue_op 0x2e20b40 seq 3540 osr(3.712 0x3ed1b90) 1429 bytes (queue has 50 ops and 2263126 bytes) 2014-06-09 16:22:11.818691 7f8690763700 10 op_submit_finish 3540 2014-06-09 16:22:11.818691 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.805243 6 0.004379 2014-06-09 16:22:11.818694 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.818704 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.818705 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.818717 7f8690763700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.818721 7f8690763700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.818715 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 peering m=1] state: osd.5 has no missing, identical log 2014-06-09 16:22:11.818724 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.778( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.044512 7 0.000217 2014-06-09 16:22:11.818728 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000024 0 0.000000 2014-06-09 16:22:11.818734 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.778( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.818740 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.818743 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.778( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.818752 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.818750 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 0.926660 0 0.000000 2014-06-09 16:22:11.818761 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.818765 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.818769 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.818775 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.818787 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.818797 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.818801 7f866f5f0700 5 -- op tracker -- , seq: 3276, time: 2014-06-09 16:22:11.817244, event: header_read, op: osd_op(client.4539.0:34390 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.818805 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'556 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.818814 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.818814 7f866f5f0700 5 -- op tracker -- , seq: 3276, time: 2014-06-09 16:22:11.817246, event: throttled, op: osd_op(client.4539.0:34390 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.818819 7f8690763700 5 queue_transactions existing osr(2.778 0x3ed0050)/0x3ed0050 2014-06-09 16:22:11.818821 7f8690763700 2 waiting 51 > 50 ops || 2265135 > 104857600 2014-06-09 16:22:11.818824 7f866f5f0700 5 -- op tracker -- , seq: 3276, time: 2014-06-09 16:22:11.818766, event: all_read, op: osd_op(client.4539.0:34390 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.818828 7f868ef60700 2 waiting 51 > 50 ops || 2265127 > 104857600 2014-06-09 16:22:11.818836 7f866f5f0700 5 -- op tracker -- , seq: 3276, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34390 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.818845 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.818871 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826800 prio 63 cost 516096 latency 0.001627 osd_op(client.4539.0:34390 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.818881 7f8689755700 2 waiting 51 > 50 ops || 2265069 > 104857600 2014-06-09 16:22:11.818894 7f8692767700 2 waiting 51 > 50 ops || 2264461 > 104857600 2014-06-09 16:22:11.818906 7f8689f56700 2 waiting 51 > 50 ops || 2263954 > 104857600 2014-06-09 16:22:11.818915 7f868ff62700 2 waiting 51 > 50 ops || 2264461 > 104857600 2014-06-09 16:22:11.818912 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.818925 7f868f761700 2 waiting 51 > 50 ops || 2264555 > 104857600 2014-06-09 16:22:11.818936 7f868cf5c700 2 waiting 51 > 50 ops || 2263954 > 104857600 2014-06-09 16:22:11.818933 7f868a757700 10 osd.26 243 queue_for_recovery queued pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.818942 7f868bf5a700 2 waiting 51 > 50 ops || 2263954 > 104857600 2014-06-09 16:22:11.818948 7f868df5e700 2 waiting 51 > 50 ops || 2326964 > 104857600 2014-06-09 16:22:11.818947 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 3.4b9( v 214'558 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) 2014-06-09 16:22:11.818954 7f8692f68700 2 waiting 51 > 50 ops || 2264179 > 104857600 2014-06-09 16:22:11.818956 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 is up to date, queueing in pending_activators 2014-06-09 16:22:11.818959 7f868e75f700 2 waiting 51 > 50 ops || 2781506 > 104857600 2014-06-09 16:22:11.818968 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.5 3.4b9( v 214'558 (0'0,214'558] local-les=206 n=108 ec=118 les/c 206/206 242/242/242) uptodate 2014-06-09 16:22:11.818981 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.818994 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 975e64b9/rbd_data.11716b8b4567.0000000000001bb1/head//3 214'558 also missing on osd.26 2014-06-09 16:22:11.819004 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {975e64b9/rbd_data.11716b8b4567.0000000000001bb1/head//3=214'558(214'556)} 2014-06-09 16:22:11.819016 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 5 as a source 2014-06-09 16:22:11.819025 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 975e64b9/rbd_data.11716b8b4567.0000000000001bb1/head//3 214'558 is on osd.5 2014-06-09 16:22:11.819037 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {975e64b9/rbd_data.11716b8b4567.0000000000001bb1/head//3=214'558(214'556)} 2014-06-09 16:22:11.819047 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.819057 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 206 2014-06-09 16:22:11.819067 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 5 2014-06-09 16:22:11.819078 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.819088 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.819122 7f868a757700 10 write_log with: dirty_to: 0'0, dirty_from: 214'556, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.819159 7f868a757700 5 queue_transactions existing osr(3.4b9 0x3ed1460)/0x3ed1460 2014-06-09 16:22:11.819162 7f868a757700 2 waiting 51 > 50 ops || 2265069 > 104857600 2014-06-09 16:22:11.819364 7f86a5610700 10 _generic_write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2281472~516096 = 0 2014-06-09 16:22:11.819391 7f86a5610700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.819400 7f86a5610700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.819404 7f86a5610700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.819409 7f86a5610700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.819465 7f86a5610700 10 cid: oid: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 not skipping op, *spos 3492.0.0 2014-06-09 16:22:11.819467 7f86a5610700 10 > header.spos 3457.0.0 2014-06-09 16:22:11.819521 7f86a5610700 20 set_header setting 30152 cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 parent seq 0 2014-06-09 16:22:11.819556 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3492.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.819560 7f86a5610700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3492.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.819562 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.820163 7f866f5f0700 5 -- op tracker -- , seq: 3277, time: 2014-06-09 16:22:11.818907, event: header_read, op: osd_op(client.4539.0:34391 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.820175 7f866f5f0700 5 -- op tracker -- , seq: 3277, time: 2014-06-09 16:22:11.818915, event: throttled, op: osd_op(client.4539.0:34391 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.820186 7f866f5f0700 5 -- op tracker -- , seq: 3277, time: 2014-06-09 16:22:11.820133, event: all_read, op: osd_op(client.4539.0:34391 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.820196 7f866f5f0700 5 -- op tracker -- , seq: 3277, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34391 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.820203 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.820225 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826700 prio 63 cost 516096 latency 0.001317 osd_op(client.4539.0:34391 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.821187 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.821199 7f86a5610700 10 _do_op 0x2fc8d7a0 seq 3492 r = 0, finisher 0x3050a340 0x2d5b8380 2014-06-09 16:22:11.821205 7f86a5610700 10 _finish_op 0x2fc8d7a0 seq 3492 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.821212 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3479.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.821216 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.821217 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.821218 7f8690763700 10 op_submit_start 3541 2014-06-09 16:22:11.821220 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3390e0 (2 active) 2014-06-09 16:22:11.821222 7f8690763700 5 queue_transactions (trailing journal) 3541 2014-06-09 16:22:11.821224 7f86a5610700 5 _do_op 0x2fcae180 seq 3493 osr(2.4d 0x2ed05d0)/0x2ed05d0 start 2014-06-09 16:22:11.821225 7f8690763700 5 queue_op 0x2e21440 seq 3541 osr(2.778 0x3ed0050) 2009 bytes (queue has 50 ops and 1746755 bytes) 2014-06-09 16:22:11.821227 7f86a5610700 10 _do_transaction on 0xc149680 2014-06-09 16:22:11.821228 7f8690763700 10 op_submit_finish 3541 2014-06-09 16:22:11.821234 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.821237 7f86a5e11700 10 _do_op 0x30d96060 seq 3479 r = 0, finisher 0x30e19040 0 2014-06-09 16:22:11.821241 7f86a5e11700 10 _finish_op 0x30d96060 seq 3479 osr(3.74f 0x3ed1ea0)/0x3ed1ea0 2014-06-09 16:22:11.821244 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f77a0 (1 active) 2014-06-09 16:22:11.821246 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a000 (2 active) 2014-06-09 16:22:11.821248 7f86a5e11700 5 _do_op 0xbe7cf60 seq 3494 osr(2.4a7 0x2ed17d0)/0x2ed17d0 start 2014-06-09 16:22:11.821233 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:11.821250 7f86a5e11700 10 _do_transaction on 0xc227ec0 2014-06-09 16:22:11.821256 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.821275 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.821281 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.821289 7f8690763700 5 queue_transactions existing osr(3.62a 0x3ed1aa0)/0x3ed1aa0 2014-06-09 16:22:11.821292 7f8690763700 10 op_submit_start 3542 2014-06-09 16:22:11.821293 7f8690763700 5 queue_transactions (trailing journal) 3542 2014-06-09 16:22:11.821291 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3493.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.821295 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.821295 7f8690763700 5 queue_op 0x2e20a20 seq 3542 osr(3.62a 0x3ed1aa0) 828 bytes (queue has 50 ops and 1746530 bytes) 2014-06-09 16:22:11.821297 7f8690763700 10 op_submit_finish 3542 2014-06-09 16:22:11.821300 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 3 notify: (query_epoch:242, epoch_sent:243, info:1.4c3( empty local-les=201 n=0 ec=1 les/c 201/201 242/242/242)) 2014-06-09 16:22:11.821311 7f86a5610700 10 _do_op 0x2fcae180 seq 3493 r = 0, finisher 0x18742920 0 2014-06-09 16:22:11.821314 7f86a5610700 10 _finish_op 0x2fcae180 seq 3493 osr(2.4d 0x2ed05d0)/0x2ed05d0 2014-06-09 16:22:11.821317 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3390e0 (1 active) 2014-06-09 16:22:11.821319 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcea0 (2 active) 2014-06-09 16:22:11.821321 7f86a5610700 5 _do_op 0xc2d5b00 seq 3495 osr(1.4c9 0x384d8b0)/0x384d8b0 start 2014-06-09 16:22:11.821314 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 3, already have info from that osd, ignoring 2014-06-09 16:22:11.821324 7f86a5610700 10 _do_transaction on 0xbfcd680 2014-06-09 16:22:11.821325 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.821329 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.821330 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.821339 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.821343 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.821355 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3494.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.821360 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.821353 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.821362 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.821378 7f8690763700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.821382 7f86a5e11700 10 _do_op 0xbe7cf60 seq 3494 r = 0, finisher 0x30e184c0 0 2014-06-09 16:22:11.821386 7f86a5e11700 10 _finish_op 0xbe7cf60 seq 3494 osr(2.4a7 0x2ed17d0)/0x2ed17d0 2014-06-09 16:22:11.821389 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a000 (1 active) 2014-06-09 16:22:11.821391 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0000 (2 active) 2014-06-09 16:22:11.821393 7f86a5e11700 5 _do_op 0xc64a240 seq 3496 osr(1.603 0x384dc90)/0x384dc90 start 2014-06-09 16:22:11.821396 7f86a5e11700 10 _do_transaction on 0xc2c6900 2014-06-09 16:22:11.821401 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.821398 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:987 2014-06-09 16:22:11.821417 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.821421 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.821433 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.821438 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.821440 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.821448 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.821455 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3495.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.821460 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.821460 7f8690763700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.821479 7f86a5610700 10 _do_op 0xc2d5b00 seq 3495 r = 0, finisher 0x3006ce20 0 2014-06-09 16:22:11.821474 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.821483 7f86a5610700 10 _finish_op 0xc2d5b00 seq 3495 osr(1.4c9 0x384d8b0)/0x384d8b0 2014-06-09 16:22:11.821486 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcea0 (1 active) 2014-06-09 16:22:11.821488 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.821488 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e7e0 (2 active) 2014-06-09 16:22:11.821490 7f86a5610700 5 _do_op 0x308ca300 seq 3497 osr(2.552 0x2ed1950)/0x2ed1950 start 2014-06-09 16:22:11.821493 7f86a5610700 10 _do_transaction on 0xbf28c00 2014-06-09 16:22:11.821499 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.821492 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.13a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:0.13a( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.821508 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.821504 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.13a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.821513 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.821522 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3496.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.821526 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.821518 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.41b( v 214'609 (0'0,214'609] local-les=212 n=115 ec=118 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.821530 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.821540 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.821543 7f86a5e11700 10 _do_op 0xc64a240 seq 3496 r = 0, finisher 0x2d738ea0 0 2014-06-09 16:22:11.821547 7f86a5e11700 10 _finish_op 0xc64a240 seq 3496 osr(1.603 0x384dc90)/0x384dc90 2014-06-09 16:22:11.821549 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0000 (1 active) 2014-06-09 16:22:11.821551 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.821544 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.821553 7f86a5e11700 5 _do_op 0x2fcaec00 seq 3498 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.821556 7f86a5e11700 10 _do_transaction on 0x34f1ec0 2014-06-09 16:22:11.821560 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.821555 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.821564 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.821569 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.821570 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.821581 7f8690763700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.821584 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3497.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.821588 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.821595 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.821608 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.821609 7f86a5e11700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.821611 7f86a5610700 10 _do_op 0x308ca300 seq 3497 r = 0, finisher 0x3006d0a0 0 2014-06-09 16:22:11.821615 7f86a5610700 10 _finish_op 0x308ca300 seq 3497 osr(2.552 0x2ed1950)/0x2ed1950 2014-06-09 16:22:11.821617 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e7e0 (1 active) 2014-06-09 16:22:11.821619 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f320 (2 active) 2014-06-09 16:22:11.821612 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:11.821621 7f86a5610700 5 _do_op 0x2c06f1a0 seq 3499 osr(3.50d 0x3ed16c0)/0x3ed16c0 start 2014-06-09 16:22:11.821624 7f86a5610700 10 _do_transaction on 0xc239380 2014-06-09 16:22:11.821625 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.821629 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.821652 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.821652 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.821656 7f8690763700 5 queue_transactions existing osr(3.4fa 0x3ed1720)/0x3ed1720 2014-06-09 16:22:11.821659 7f8690763700 10 op_submit_start 3543 2014-06-09 16:22:11.821660 7f8690763700 5 queue_transactions (trailing journal) 3543 2014-06-09 16:22:11.821661 7f8690763700 5 queue_op 0x2e207e0 seq 3543 osr(3.4fa 0x3ed1720) 828 bytes (queue has 46 ops and 1739841 bytes) 2014-06-09 16:22:11.821664 7f8690763700 10 op_submit_finish 3543 2014-06-09 16:22:11.821666 7f86a5e11700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.821672 7f86a5e11700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.821678 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.821685 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8fe40 (19 active) 2014-06-09 16:22:11.821691 7f86a5e11700 15 _write 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 3026944~1167360 2014-06-09 16:22:11.821693 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.821704 7f868ef60700 10 op_submit_start 3544 2014-06-09 16:22:11.821706 7f868ef60700 5 queue_transactions (trailing journal) 3544 2014-06-09 16:22:11.821707 7f868ef60700 5 queue_op 0x230b8660 seq 3544 osr(0.4bd 0x2bd76f0) 2001 bytes (queue has 47 ops and 1741842 bytes) 2014-06-09 16:22:11.821710 7f868ef60700 10 op_submit_finish 3544 2014-06-09 16:22:11.821713 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.821728 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.821738 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.821746 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.821759 7f868ef60700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.821761 7f86a5e11700 10 lookup_strip_header done cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.821771 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:916 2014-06-09 16:22:11.821785 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.821795 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.821789 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.604( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.821826 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.821831 7f868ef60700 5 queue_transactions existing osr(1.604 0x384dc50)/0x384dc50 2014-06-09 16:22:11.821834 7f868ef60700 10 op_submit_start 3545 2014-06-09 16:22:11.821835 7f868ef60700 5 queue_transactions (trailing journal) 3545 2014-06-09 16:22:11.821836 7f868ef60700 5 queue_op 0x16097da0 seq 3545 osr(1.604 0x384dc50) 828 bytes (queue has 48 ops and 1742670 bytes) 2014-06-09 16:22:11.821838 7f868ef60700 10 op_submit_finish 3545 2014-06-09 16:22:11.821841 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:11.821878 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.821883 7f868ef60700 5 queue_transactions existing osr(3.31c 0x3ed0d40)/0x3ed0d40 2014-06-09 16:22:11.821885 7f868ef60700 10 op_submit_start 3546 2014-06-09 16:22:11.821886 7f868ef60700 5 queue_transactions (trailing journal) 3546 2014-06-09 16:22:11.821887 7f868ef60700 5 queue_op 0x30032c00 seq 3546 osr(3.31c 0x3ed0d40) 828 bytes (queue has 49 ops and 1743498 bytes) 2014-06-09 16:22:11.821889 7f868ef60700 10 op_submit_finish 3546 2014-06-09 16:22:11.821892 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.821904 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.821915 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.821919 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.821925 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.774( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.868624 7 0.000600 2014-06-09 16:22:11.821937 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.774( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.821948 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.774( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.821958 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.821968 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.821976 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.822095 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.822100 7f868ef60700 5 queue_transactions existing osr(1.774 0x2ed0390)/0x2ed0390 2014-06-09 16:22:11.822103 7f868ef60700 10 op_submit_start 3547 2014-06-09 16:22:11.822104 7f868ef60700 5 queue_transactions (trailing journal) 3547 2014-06-09 16:22:11.822105 7f868ef60700 5 queue_op 0x2c06e2a0 seq 3547 osr(1.774 0x2ed0390) 1883 bytes (queue has 50 ops and 1745381 bytes) 2014-06-09 16:22:11.822107 7f868ef60700 10 op_submit_finish 3547 2014-06-09 16:22:11.822110 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:2.5fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.822134 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.822144 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.822149 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:11.822178 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.822182 7f868ef60700 5 queue_transactions existing osr(3.9b 0x3ed0630)/0x3ed0630 2014-06-09 16:22:11.822184 7f868ef60700 2 waiting 51 > 50 ops || 1746207 > 104857600 2014-06-09 16:22:11.822256 7f8689755700 2 waiting 51 > 50 ops || 1747324 > 104857600 2014-06-09 16:22:11.822285 7f8692767700 2 waiting 51 > 50 ops || 1746716 > 104857600 2014-06-09 16:22:11.822292 7f8689f56700 2 waiting 51 > 50 ops || 1746209 > 104857600 2014-06-09 16:22:11.822297 7f868ff62700 2 waiting 51 > 50 ops || 1746716 > 104857600 2014-06-09 16:22:11.822302 7f868f761700 2 waiting 51 > 50 ops || 1746810 > 104857600 2014-06-09 16:22:11.822308 7f868cf5c700 2 waiting 51 > 50 ops || 1746209 > 104857600 2014-06-09 16:22:11.822314 7f868bf5a700 2 waiting 51 > 50 ops || 1746209 > 104857600 2014-06-09 16:22:11.822319 7f868df5e700 2 waiting 51 > 50 ops || 1809219 > 104857600 2014-06-09 16:22:11.822325 7f8692f68700 2 waiting 51 > 50 ops || 1746434 > 104857600 2014-06-09 16:22:11.822331 7f868e75f700 2 waiting 51 > 50 ops || 2263761 > 104857600 2014-06-09 16:22:11.822340 7f868a757700 2 waiting 51 > 50 ops || 1747324 > 104857600 2014-06-09 16:22:11.823439 7f866f5f0700 5 -- op tracker -- , seq: 3278, time: 2014-06-09 16:22:11.820249, event: header_read, op: osd_op(client.4539.0:34392 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1171456] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.823450 7f866f5f0700 5 -- op tracker -- , seq: 3278, time: 2014-06-09 16:22:11.820250, event: throttled, op: osd_op(client.4539.0:34392 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1171456] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.823461 7f866f5f0700 5 -- op tracker -- , seq: 3278, time: 2014-06-09 16:22:11.823396, event: all_read, op: osd_op(client.4539.0:34392 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1171456] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.823469 7f866f5f0700 5 -- op tracker -- , seq: 3278, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34392 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1171456] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.823480 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.823510 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826b00 prio 63 cost 1171456 latency 0.003260 osd_op(client.4539.0:34392 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 3022848~1171456] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.824213 7f866f5f0700 5 -- op tracker -- , seq: 3279, time: 2014-06-09 16:22:11.823554, event: header_read, op: osd_op(client.4539.0:34393 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~376832] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.824226 7f866f5f0700 5 -- op tracker -- , seq: 3279, time: 2014-06-09 16:22:11.823556, event: throttled, op: osd_op(client.4539.0:34393 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~376832] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.824237 7f866f5f0700 5 -- op tracker -- , seq: 3279, time: 2014-06-09 16:22:11.824182, event: all_read, op: osd_op(client.4539.0:34393 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~376832] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.824248 7f866f5f0700 5 -- op tracker -- , seq: 3279, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34393 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~376832] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.824254 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.824277 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826a00 prio 63 cost 376832 latency 0.000723 osd_op(client.4539.0:34393 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~376832] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.824943 7f866f5f0700 5 -- op tracker -- , seq: 3280, time: 2014-06-09 16:22:11.824310, event: header_read, op: osd_op(client.4539.0:34394 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 376832~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.824955 7f866f5f0700 5 -- op tracker -- , seq: 3280, time: 2014-06-09 16:22:11.824311, event: throttled, op: osd_op(client.4539.0:34394 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 376832~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.824969 7f866f5f0700 5 -- op tracker -- , seq: 3280, time: 2014-06-09 16:22:11.824914, event: all_read, op: osd_op(client.4539.0:34394 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 376832~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.824980 7f866f5f0700 5 -- op tracker -- , seq: 3280, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34394 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 376832~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.824986 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.825004 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826900 prio 63 cost 516096 latency 0.000694 osd_op(client.4539.0:34394 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 376832~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.825681 7f866f5f0700 5 -- op tracker -- , seq: 3281, time: 2014-06-09 16:22:11.825028, event: header_read, op: osd_op(client.4539.0:34395 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 892928~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.825692 7f866f5f0700 5 -- op tracker -- , seq: 3281, time: 2014-06-09 16:22:11.825029, event: throttled, op: osd_op(client.4539.0:34395 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 892928~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.825704 7f866f5f0700 5 -- op tracker -- , seq: 3281, time: 2014-06-09 16:22:11.825642, event: all_read, op: osd_op(client.4539.0:34395 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 892928~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.825713 7f866f5f0700 5 -- op tracker -- , seq: 3281, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34395 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 892928~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.825718 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.825734 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826f00 prio 63 cost 516096 latency 0.000706 osd_op(client.4539.0:34395 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 892928~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.826803 7f866f5f0700 5 -- op tracker -- , seq: 3282, time: 2014-06-09 16:22:11.825755, event: header_read, op: osd_op(client.4539.0:34396 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1409024~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.826817 7f866f5f0700 5 -- op tracker -- , seq: 3282, time: 2014-06-09 16:22:11.825756, event: throttled, op: osd_op(client.4539.0:34396 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1409024~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.826828 7f866f5f0700 5 -- op tracker -- , seq: 3282, time: 2014-06-09 16:22:11.826772, event: all_read, op: osd_op(client.4539.0:34396 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1409024~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.826839 7f866f5f0700 5 -- op tracker -- , seq: 3282, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34396 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1409024~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.826846 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.826867 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826e00 prio 63 cost 516096 latency 0.001111 osd_op(client.4539.0:34396 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1409024~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.828701 7f86a5e11700 10 _generic_write 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 3026944~1167360 = 0 2014-06-09 16:22:11.828763 7f86a5e11700 15 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.828780 7f86a5e11700 10 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.828787 7f86a5e11700 15 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 2014-06-09 16:22:11.828795 7f86a5e11700 10 _setattrs 3.60_head/886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 = 0 2014-06-09 16:22:11.828909 7f866f5f0700 5 -- op tracker -- , seq: 3283, time: 2014-06-09 16:22:11.826893, event: header_read, op: osd_op(client.4539.0:34397 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1925120~1032192] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.828924 7f866f5f0700 5 -- op tracker -- , seq: 3283, time: 2014-06-09 16:22:11.826894, event: throttled, op: osd_op(client.4539.0:34397 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1925120~1032192] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.828939 7f866f5f0700 5 -- op tracker -- , seq: 3283, time: 2014-06-09 16:22:11.828874, event: all_read, op: osd_op(client.4539.0:34397 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1925120~1032192] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.828951 7f866f5f0700 5 -- op tracker -- , seq: 3283, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34397 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1925120~1032192] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.828960 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.828980 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826d00 prio 63 cost 1032192 latency 0.002087 osd_op(client.4539.0:34397 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 1925120~1032192] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.828989 7f86a5e11700 10 cid: oid: 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 not skipping op, *spos 3498.0.0 2014-06-09 16:22:11.828994 7f86a5e11700 10 > header.spos 3478.0.0 2014-06-09 16:22:11.829113 7f86a5e11700 20 set_header setting 30151 cid 3.60_head oid 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 parent seq 0 2014-06-09 16:22:11.829222 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3498.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.829232 7f86a5e11700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3498.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.829235 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.829839 7f866f5f0700 5 -- op tracker -- , seq: 3284, time: 2014-06-09 16:22:11.829003, event: header_read, op: osd_op(client.4539.0:34398 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2957312~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.829854 7f866f5f0700 5 -- op tracker -- , seq: 3284, time: 2014-06-09 16:22:11.829005, event: throttled, op: osd_op(client.4539.0:34398 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2957312~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.829866 7f866f5f0700 5 -- op tracker -- , seq: 3284, time: 2014-06-09 16:22:11.829803, event: all_read, op: osd_op(client.4539.0:34398 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2957312~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.829877 7f866f5f0700 5 -- op tracker -- , seq: 3284, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34398 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2957312~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.829884 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.829898 7f866f5f0700 15 osd.26 243 enqueue_op 0x32826c00 prio 63 cost 516096 latency 0.000895 osd_op(client.4539.0:34398 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 2957312~516096] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.831164 7f866f5f0700 5 -- op tracker -- , seq: 3285, time: 2014-06-09 16:22:11.829921, event: header_read, op: osd_op(client.4539.0:34399 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 3473408~720896] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.831178 7f866f5f0700 5 -- op tracker -- , seq: 3285, time: 2014-06-09 16:22:11.829922, event: throttled, op: osd_op(client.4539.0:34399 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 3473408~720896] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.831191 7f866f5f0700 5 -- op tracker -- , seq: 3285, time: 2014-06-09 16:22:11.831131, event: all_read, op: osd_op(client.4539.0:34399 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 3473408~720896] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.831227 7f866f5f0700 5 -- op tracker -- , seq: 3285, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34399 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 3473408~720896] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.831237 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.831260 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827400 prio 63 cost 720896 latency 0.001339 osd_op(client.4539.0:34399 rbd_data.115b6b8b4567.00000000000018aa [set-alloc-hint object_size 4194304 write_size 4194304,write 3473408~720896] 3.c8afe094 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.831665 7f866f5f0700 5 -- op tracker -- , seq: 3286, time: 2014-06-09 16:22:11.831283, event: header_read, op: osd_op(client.4539.0:34408 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.831678 7f866f5f0700 5 -- op tracker -- , seq: 3286, time: 2014-06-09 16:22:11.831284, event: throttled, op: osd_op(client.4539.0:34408 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.831688 7f866f5f0700 5 -- op tracker -- , seq: 3286, time: 2014-06-09 16:22:11.831639, event: all_read, op: osd_op(client.4539.0:34408 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.831696 7f866f5f0700 5 -- op tracker -- , seq: 3286, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34408 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.831715 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.831734 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827300 prio 63 cost 245760 latency 0.000451 osd_op(client.4539.0:34408 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.832407 7f866f5f0700 5 -- op tracker -- , seq: 3287, time: 2014-06-09 16:22:11.831754, event: header_read, op: osd_op(client.4539.0:34409 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 245760~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.832416 7f866f5f0700 5 -- op tracker -- , seq: 3287, time: 2014-06-09 16:22:11.831754, event: throttled, op: osd_op(client.4539.0:34409 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 245760~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.832422 7f866f5f0700 5 -- op tracker -- , seq: 3287, time: 2014-06-09 16:22:11.832385, event: all_read, op: osd_op(client.4539.0:34409 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 245760~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.832428 7f866f5f0700 5 -- op tracker -- , seq: 3287, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34409 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 245760~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.832434 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.832447 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827200 prio 63 cost 516096 latency 0.000693 osd_op(client.4539.0:34409 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 245760~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.833971 7f866f5f0700 5 -- op tracker -- , seq: 3288, time: 2014-06-09 16:22:11.832463, event: header_read, op: osd_op(client.4539.0:34410 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 761856~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.833980 7f866f5f0700 5 -- op tracker -- , seq: 3288, time: 2014-06-09 16:22:11.832464, event: throttled, op: osd_op(client.4539.0:34410 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 761856~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.833991 7f866f5f0700 5 -- op tracker -- , seq: 3288, time: 2014-06-09 16:22:11.833944, event: all_read, op: osd_op(client.4539.0:34410 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 761856~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.833997 7f866f5f0700 5 -- op tracker -- , seq: 3288, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34410 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 761856~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.834001 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.834016 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827100 prio 63 cost 1032192 latency 0.001553 osd_op(client.4539.0:34410 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 761856~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.835367 7f866f5f0700 5 -- op tracker -- , seq: 3289, time: 2014-06-09 16:22:11.834028, event: header_read, op: osd_op(client.4539.0:34411 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.835375 7f866f5f0700 5 -- op tracker -- , seq: 3289, time: 2014-06-09 16:22:11.834030, event: throttled, op: osd_op(client.4539.0:34411 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.835381 7f866f5f0700 5 -- op tracker -- , seq: 3289, time: 2014-06-09 16:22:11.835346, event: all_read, op: osd_op(client.4539.0:34411 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.835386 7f866f5f0700 5 -- op tracker -- , seq: 3289, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34411 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.835393 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.835405 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827000 prio 63 cost 1032192 latency 0.001376 osd_op(client.4539.0:34411 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.836002 7f866f5f0700 5 -- op tracker -- , seq: 3290, time: 2014-06-09 16:22:11.835416, event: header_read, op: osd_op(client.4539.0:34412 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.836010 7f866f5f0700 5 -- op tracker -- , seq: 3290, time: 2014-06-09 16:22:11.835417, event: throttled, op: osd_op(client.4539.0:34412 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.836017 7f866f5f0700 5 -- op tracker -- , seq: 3290, time: 2014-06-09 16:22:11.835983, event: all_read, op: osd_op(client.4539.0:34412 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.836022 7f866f5f0700 5 -- op tracker -- , seq: 3290, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34412 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.836026 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.836036 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827a00 prio 63 cost 516096 latency 0.000620 osd_op(client.4539.0:34412 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.836611 7f866f5f0700 5 -- op tracker -- , seq: 3291, time: 2014-06-09 16:22:11.836048, event: header_read, op: osd_op(client.4539.0:34413 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3342336~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.836619 7f866f5f0700 5 -- op tracker -- , seq: 3291, time: 2014-06-09 16:22:11.836049, event: throttled, op: osd_op(client.4539.0:34413 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3342336~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.836635 7f866f5f0700 5 -- op tracker -- , seq: 3291, time: 2014-06-09 16:22:11.836594, event: all_read, op: osd_op(client.4539.0:34413 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3342336~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.836641 7f866f5f0700 5 -- op tracker -- , seq: 3291, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34413 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3342336~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.836644 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.836656 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827900 prio 63 cost 516096 latency 0.000607 osd_op(client.4539.0:34413 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3342336~516096] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.836849 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.836878 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3499.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.836885 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.836923 7f86a5610700 10 _do_op 0x2c06f1a0 seq 3499 r = 0, finisher 0x4eb92a0 0 2014-06-09 16:22:11.836931 7f86a5610700 10 _finish_op 0x2c06f1a0 seq 3499 osr(3.50d 0x3ed16c0)/0x3ed16c0 2014-06-09 16:22:11.836939 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f320 (1 active) 2014-06-09 16:22:11.836942 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383fe60 (2 active) 2014-06-09 16:22:11.836944 7f86a5610700 5 _do_op 0x16fd5140 seq 3500 osr(1.30 0x384c280)/0x384c280 start 2014-06-09 16:22:11.836943 7f868ef60700 10 op_submit_start 3548 2014-06-09 16:22:11.836947 7f868ef60700 5 queue_transactions (trailing journal) 3548 2014-06-09 16:22:11.836947 7f86a5610700 10 _do_transaction on 0x3a86180 2014-06-09 16:22:11.836948 7f868ef60700 5 queue_op 0x11fce2a0 seq 3548 osr(3.9b 0x3ed0630) 826 bytes (queue has 50 ops and 1744872 bytes) 2014-06-09 16:22:11.836953 7f868ef60700 10 op_submit_finish 3548 2014-06-09 16:22:11.836953 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.836959 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.836986 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.836982 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.836992 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.836998 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3500.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837002 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837001 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.837011 7f86a5610700 10 _do_op 0x16fd5140 seq 3500 r = 0, finisher 0xc6e8680 0 2014-06-09 16:22:11.837014 7f86a5610700 10 _finish_op 0x16fd5140 seq 3500 osr(1.30 0x384c280)/0x384c280 2014-06-09 16:22:11.837017 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383fe60 (1 active) 2014-06-09 16:22:11.837018 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89aea0 (2 active) 2014-06-09 16:22:11.837020 7f86a5610700 5 _do_op 0x2fcafec0 seq 3501 osr(0.f 0x2bd60e0)/0x2bd60e0 start 2014-06-09 16:22:11.837022 7f86a5610700 10 _do_transaction on 0xbe89bc0 2014-06-09 16:22:11.837020 7f868ef60700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.837027 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837040 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.837055 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837059 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837063 7f86a5e11700 10 _do_op 0x2fcaec00 seq 3498 r = 0, finisher 0x2e0ed20 0x2f087080 2014-06-09 16:22:11.837068 7f86a5e11700 10 _finish_op 0x2fcaec00 seq 3498 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.837069 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3501.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837072 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837072 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.837074 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62ed80 (2 active) 2014-06-09 16:22:11.837065 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.7d4( v 214'543 lc 214'534 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.837076 7f86a5e11700 5 _do_op 0x2e21d40 seq 3502 osr(3.366 0x3ed1080)/0x3ed1080 start 2014-06-09 16:22:11.837079 7f86a5e11700 10 _do_transaction on 0xbf58b40 2014-06-09 16:22:11.837084 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837084 7f86a5610700 10 _do_op 0x2fcafec0 seq 3501 r = 0, finisher 0x318c33a0 0 2014-06-09 16:22:11.837087 7f86a5610700 10 _finish_op 0x2fcafec0 seq 3501 osr(0.f 0x2bd60e0)/0x2bd60e0 2014-06-09 16:22:11.837080 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.837089 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89aea0 (1 active) 2014-06-09 16:22:11.837091 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac240 (2 active) 2014-06-09 16:22:11.837093 7f86a5610700 5 _do_op 0x16fd4f60 seq 3503 osr(1.5f8 0x384dcf0)/0x384dcf0 start 2014-06-09 16:22:11.837096 7f86a5610700 10 _do_transaction on 0x3a07680 2014-06-09 16:22:11.837091 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.837101 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837101 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.837114 7f868ef60700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.837124 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837130 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:861 2014-06-09 16:22:11.837143 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3502.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837147 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837147 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.837152 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.27( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.837168 7f86a5e11700 10 _do_op 0x2e21d40 seq 3502 r = 0, finisher 0x318c3c60 0 2014-06-09 16:22:11.837172 7f86a5e11700 10 _finish_op 0x2e21d40 seq 3502 osr(3.366 0x3ed1080)/0x3ed1080 2014-06-09 16:22:11.837175 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62ed80 (1 active) 2014-06-09 16:22:11.837177 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc120 (2 active) 2014-06-09 16:22:11.837179 7f86a5e11700 5 _do_op 0x2fcaeae0 seq 3504 osr(3.25a 0x3ed0c00)/0x3ed0c00 start 2014-06-09 16:22:11.837181 7f86a5e11700 10 _do_transaction on 0xc2bd5c0 2014-06-09 16:22:11.837187 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837188 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837194 7f868ef60700 5 queue_transactions existing osr(2.27 0x2ed0250)/0x2ed0250 2014-06-09 16:22:11.837197 7f868ef60700 10 op_submit_start 3549 2014-06-09 16:22:11.837199 7f868ef60700 5 queue_transactions (trailing journal) 3549 2014-06-09 16:22:11.837199 7f868ef60700 5 queue_op 0x30061320 seq 3549 osr(2.27 0x2ed0250) 826 bytes (queue has 47 ops and 571922 bytes) 2014-06-09 16:22:11.837202 7f868ef60700 10 op_submit_finish 3549 2014-06-09 16:22:11.837209 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837205 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.837226 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3503.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837219 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.837231 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837233 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.837238 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.837249 7f86a5610700 10 _do_op 0x16fd4f60 seq 3503 r = 0, finisher 0x318c2200 0 2014-06-09 16:22:11.837243 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.566( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] exit Started/Stray 1.379753 7 0.000216 2014-06-09 16:22:11.837253 7f86a5610700 10 _finish_op 0x16fd4f60 seq 3503 osr(1.5f8 0x384dcf0)/0x384dcf0 2014-06-09 16:22:11.837256 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac240 (1 active) 2014-06-09 16:22:11.837258 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114d80 (2 active) 2014-06-09 16:22:11.837260 7f86a5610700 5 _do_op 0x2fc8c840 seq 3505 osr(1.300 0x384d050)/0x384d050 start 2014-06-09 16:22:11.837262 7f86a5610700 10 _do_transaction on 0x4801b00 2014-06-09 16:22:11.837257 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.566( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.837268 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837269 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.566( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.837277 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837279 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.837285 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.837290 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.837294 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3504.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837299 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.837318 7f86a5e11700 10 _do_op 0x2fcaeae0 seq 3504 r = 0, finisher 0x2e0e8a0 0 2014-06-09 16:22:11.837322 7f86a5e11700 10 _finish_op 0x2fcaeae0 seq 3504 osr(3.25a 0x3ed0c00)/0x3ed0c00 2014-06-09 16:22:11.837324 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc120 (1 active) 2014-06-09 16:22:11.837326 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4240 (2 active) 2014-06-09 16:22:11.837328 7f86a5e11700 5 _do_op 0x2ea536e0 seq 3506 osr(1.5e 0x384c5d0)/0x384c5d0 start 2014-06-09 16:22:11.837331 7f86a5e11700 10 _do_transaction on 0xbe895c0 2014-06-09 16:22:11.837336 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837338 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837343 7f868ef60700 5 queue_transactions existing osr(2.566 0x2ed18f0)/0x2ed18f0 2014-06-09 16:22:11.837346 7f868ef60700 10 op_submit_start 3550 2014-06-09 16:22:11.837347 7f868ef60700 5 queue_transactions (trailing journal) 3550 2014-06-09 16:22:11.837346 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837362 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3505.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837366 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837380 7f86a5610700 10 _do_op 0x2fc8c840 seq 3505 r = 0, finisher 0x30c54780 0 2014-06-09 16:22:11.837384 7f86a5610700 10 _finish_op 0x2fc8c840 seq 3505 osr(1.300 0x384d050)/0x384d050 2014-06-09 16:22:11.837386 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114d80 (1 active) 2014-06-09 16:22:11.837388 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4065a0 (2 active) 2014-06-09 16:22:11.837391 7f86a5610700 5 _do_op 0x309a22a0 seq 3507 osr(1.99 0x384c7d0)/0x384c7d0 start 2014-06-09 16:22:11.837393 7f86a5610700 10 _do_transaction on 0xc1123c0 2014-06-09 16:22:11.837399 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837406 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837348 7f868ef60700 5 queue_op 0x300614a0 seq 3550 osr(2.566 0x2ed18f0) 1954 bytes (queue has 46 ops and 570826 bytes) 2014-06-09 16:22:11.837412 7f868ef60700 10 op_submit_finish 3550 2014-06-09 16:22:11.837420 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3506.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837424 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.837416 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.837431 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.837441 7f86a5e11700 10 _do_op 0x2ea536e0 seq 3506 r = 0, finisher 0x3006d160 0 2014-06-09 16:22:11.837445 7f86a5e11700 10 _finish_op 0x2ea536e0 seq 3506 osr(1.5e 0x384c5d0)/0x384c5d0 2014-06-09 16:22:11.837447 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4240 (1 active) 2014-06-09 16:22:11.837442 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.837449 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338480 (2 active) 2014-06-09 16:22:11.837452 7f86a5e11700 5 _do_op 0x309a29c0 seq 3508 osr(3.51e 0x3ed16a0)/0x3ed16a0 start 2014-06-09 16:22:11.837455 7f86a5e11700 10 _do_transaction on 0xc2bf080 2014-06-09 16:22:11.837452 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.837460 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837467 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837467 7f868ef60700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.837482 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3507.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837486 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837483 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:842 2014-06-09 16:22:11.837499 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837499 7f86a5610700 10 _do_op 0x309a22a0 seq 3507 r = 0, finisher 0xc5a78a0 0 2014-06-09 16:22:11.837503 7f86a5610700 10 _finish_op 0x309a22a0 seq 3507 osr(1.99 0x384c7d0)/0x384c7d0 2014-06-09 16:22:11.837505 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4065a0 (1 active) 2014-06-09 16:22:11.837507 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383fe60 (2 active) 2014-06-09 16:22:11.837509 7f86a5610700 5 _do_op 0x230b9b00 seq 3509 osr(1.30 0x384c280)/0x384c280 start 2014-06-09 16:22:11.837511 7f86a5610700 10 _do_transaction on 0xc2f4900 2014-06-09 16:22:11.837503 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.76( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.837516 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837525 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837536 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837541 7f868ef60700 5 queue_transactions existing osr(1.76 0x384c530)/0x384c530 2014-06-09 16:22:11.837540 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3508.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837544 7f868ef60700 10 op_submit_start 3551 2014-06-09 16:22:11.837545 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.837545 7f868ef60700 5 queue_transactions (trailing journal) 3551 2014-06-09 16:22:11.837548 7f868ef60700 5 queue_op 0x1e08aa20 seq 3551 osr(1.76 0x384c530) 826 bytes (queue has 44 ops and 569107 bytes) 2014-06-09 16:22:11.837551 7f868ef60700 10 op_submit_finish 3551 2014-06-09 16:22:11.837553 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.837565 7f86a5e11700 10 _do_op 0x309a29c0 seq 3508 r = 0, finisher 0x31791fa0 0 2014-06-09 16:22:11.837569 7f86a5e11700 10 _finish_op 0x309a29c0 seq 3508 osr(3.51e 0x3ed16a0)/0x3ed16a0 2014-06-09 16:22:11.837572 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338480 (1 active) 2014-06-09 16:22:11.837574 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4adc20 (2 active) 2014-06-09 16:22:11.837565 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.837576 7f86a5e11700 5 _do_op 0x2e21200 seq 3510 osr(1.188 0x384cb70)/0x384cb70 start 2014-06-09 16:22:11.837578 7f86a5e11700 10 _do_transaction on 0x33392c0 2014-06-09 16:22:11.837578 7f868ef60700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.837583 7f868ef60700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.837584 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837593 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837586 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.879025 7 0.000178 2014-06-09 16:22:11.837598 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.837607 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3509.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837611 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837609 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.837619 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.837628 7f86a5610700 10 _do_op 0x230b9b00 seq 3509 r = 0, finisher 0xc169460 0 2014-06-09 16:22:11.837632 7f86a5610700 10 _finish_op 0x230b9b00 seq 3509 osr(1.30 0x384c280)/0x384c280 2014-06-09 16:22:11.837634 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383fe60 (1 active) 2014-06-09 16:22:11.837628 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.837636 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f320 (2 active) 2014-06-09 16:22:11.837638 7f86a5610700 5 _do_op 0x2fcaf020 seq 3511 osr(0.4da 0x2bd75f0)/0x2bd75f0 start 2014-06-09 16:22:11.837641 7f86a5610700 10 _do_transaction on 0xc12bec0 2014-06-09 16:22:11.837637 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.837646 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837653 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837670 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3510.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837674 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.837687 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837689 7f86a5e11700 10 _do_op 0x2e21200 seq 3510 r = 0, finisher 0x314124e0 0 2014-06-09 16:22:11.837693 7f86a5e11700 10 _finish_op 0x2e21200 seq 3510 osr(1.188 0x384cb70)/0x384cb70 2014-06-09 16:22:11.837692 7f868ef60700 5 queue_transactions existing osr(2.4af 0x2ed1790)/0x2ed1790 2014-06-09 16:22:11.837695 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4adc20 (1 active) 2014-06-09 16:22:11.837697 7f868ef60700 10 op_submit_start 3552 2014-06-09 16:22:11.837697 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922240 (2 active) 2014-06-09 16:22:11.837698 7f868ef60700 5 queue_transactions (trailing journal) 3552 2014-06-09 16:22:11.837699 7f86a5e11700 5 _do_op 0x309a37a0 seq 3512 osr(2.1e0 0x2ed0850)/0x2ed0850 start 2014-06-09 16:22:11.837702 7f86a5e11700 10 _do_transaction on 0xc27e480 2014-06-09 16:22:11.837700 7f868ef60700 5 queue_op 0xc2d59e0 seq 3552 osr(2.4af 0x2ed1790) 1883 bytes (queue has 42 ops and 567102 bytes) 2014-06-09 16:22:11.837705 7f868ef60700 10 op_submit_finish 3552 2014-06-09 16:22:11.837708 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837707 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:243, info:1.779( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.837716 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837721 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.837733 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837731 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3511.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837735 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837738 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.9b( v 213'568 (0'0,213'568] local-les=243 n=118 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:11.837751 7f86a5610700 10 _do_op 0x2fcaf020 seq 3511 r = 0, finisher 0xcd0fbc0 0 2014-06-09 16:22:11.837755 7f86a5610700 10 _finish_op 0x2fcaf020 seq 3511 osr(0.4da 0x2bd75f0)/0x2bd75f0 2014-06-09 16:22:11.837758 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f320 (1 active) 2014-06-09 16:22:11.837760 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd0e0 (2 active) 2014-06-09 16:22:11.837762 7f86a5610700 5 _do_op 0x2fcaf0e0 seq 3513 osr(2.75 0x2ed04f0)/0x2ed04f0 start 2014-06-09 16:22:11.837764 7f86a5610700 10 _do_transaction on 0xc2efc80 2014-06-09 16:22:11.837769 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837776 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837778 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837782 7f868ef60700 5 queue_transactions existing osr(3.9b 0x3ed0630)/0x3ed0630 2014-06-09 16:22:11.837785 7f868ef60700 10 op_submit_start 3553 2014-06-09 16:22:11.837786 7f868ef60700 5 queue_transactions (trailing journal) 3553 2014-06-09 16:22:11.837787 7f868ef60700 5 queue_op 0xc2d5380 seq 3553 osr(3.9b 0x3ed0630) 826 bytes (queue has 42 ops and 567100 bytes) 2014-06-09 16:22:11.837790 7f868ef60700 10 op_submit_finish 3553 2014-06-09 16:22:11.837795 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3512.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837799 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.837793 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=201 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.837813 7f86a5e11700 10 _do_op 0x309a37a0 seq 3512 r = 0, finisher 0xbd726e0 0 2014-06-09 16:22:11.837817 7f86a5e11700 10 _finish_op 0x309a37a0 seq 3512 osr(2.1e0 0x2ed0850)/0x2ed0850 2014-06-09 16:22:11.837807 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=201 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.3 3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) 2014-06-09 16:22:11.837820 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922240 (1 active) 2014-06-09 16:22:11.837822 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89a6c0 (2 active) 2014-06-09 16:22:11.837824 7f86a5e11700 5 _do_op 0x300336e0 seq 3514 osr(3.582 0x3ed18a0)/0x3ed18a0 start 2014-06-09 16:22:11.837827 7f86a5e11700 10 _do_transaction on 0xc22c300 2014-06-09 16:22:11.837821 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=201 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.858775 7 0.000180 2014-06-09 16:22:11.837833 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837833 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=201 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.837840 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837845 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=201 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.837853 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3513.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837857 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837855 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=201 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.837865 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'560 -> 213'560 2014-06-09 16:22:11.837874 7f86a5610700 10 _do_op 0x2fcaf0e0 seq 3513 r = 0, finisher 0x13965240 0 2014-06-09 16:22:11.837878 7f86a5610700 10 _finish_op 0x2fcaf0e0 seq 3513 osr(2.75 0x2ed04f0)/0x2ed04f0 2014-06-09 16:22:11.837880 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd0e0 (1 active) 2014-06-09 16:22:11.837882 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb6900 (2 active) 2014-06-09 16:22:11.837885 7f86a5610700 5 _do_op 0x2fcaf140 seq 3515 osr(3.793 0x3ed1fd0)/0x3ed1fd0 start 2014-06-09 16:22:11.837887 7f86a5610700 10 _do_transaction on 0xc145380 2014-06-09 16:22:11.837892 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837901 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837902 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.837916 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3514.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837921 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.837940 7f86a5e11700 10 _do_op 0x300336e0 seq 3514 r = 0, finisher 0xbebe740 0 2014-06-09 16:22:11.837944 7f86a5e11700 10 _finish_op 0x300336e0 seq 3514 osr(3.582 0x3ed18a0)/0x3ed18a0 2014-06-09 16:22:11.837947 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89a6c0 (1 active) 2014-06-09 16:22:11.837949 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358a20 (2 active) 2014-06-09 16:22:11.837951 7f86a5e11700 5 _do_op 0x2fcae360 seq 3516 osr(2.36c 0x2ed1170)/0x2ed1170 start 2014-06-09 16:22:11.837951 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837953 7f86a5e11700 10 _do_transaction on 0x31298c0 2014-06-09 16:22:11.837956 7f868ef60700 5 queue_transactions existing osr(3.4c4 0x3ed1420)/0x3ed1420 2014-06-09 16:22:11.837959 7f868ef60700 10 op_submit_start 3554 2014-06-09 16:22:11.837959 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.837960 7f868ef60700 5 queue_transactions (trailing journal) 3554 2014-06-09 16:22:11.837961 7f868ef60700 5 queue_op 0xc2d42a0 seq 3554 osr(3.4c4 0x3ed1420) 1335 bytes (queue has 40 ops and 564602 bytes) 2014-06-09 16:22:11.837964 7f868ef60700 10 op_submit_finish 3554 2014-06-09 16:22:11.837968 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.837967 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.312( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.312( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.837985 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3515.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.837989 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.837994 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.837999 7f868ef60700 5 queue_transactions existing osr(2.312 0x2ed0f30)/0x2ed0f30 2014-06-09 16:22:11.838001 7f868ef60700 10 op_submit_start 3555 2014-06-09 16:22:11.838002 7f868ef60700 5 queue_transactions (trailing journal) 3555 2014-06-09 16:22:11.838004 7f868ef60700 5 queue_op 0xc2d5320 seq 3555 osr(2.312 0x2ed0f30) 828 bytes (queue has 41 ops and 565430 bytes) 2014-06-09 16:22:11.838003 7f86a5610700 10 _do_op 0x2fcaf140 seq 3515 r = 0, finisher 0xcd32720 0 2014-06-09 16:22:11.838007 7f868ef60700 10 op_submit_finish 3555 2014-06-09 16:22:11.838007 7f86a5610700 10 _finish_op 0x2fcaf140 seq 3515 osr(3.793 0x3ed1fd0)/0x3ed1fd0 2014-06-09 16:22:11.838010 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb6900 (1 active) 2014-06-09 16:22:11.838012 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2879e0 (2 active) 2014-06-09 16:22:11.838014 7f86a5610700 5 _do_op 0xc64bda0 seq 3517 osr(1.92 0x384c410)/0x384c410 start 2014-06-09 16:22:11.838017 7f86a5610700 10 _do_transaction on 0xc120240 2014-06-09 16:22:11.838009 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.27( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.27( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.838021 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.838029 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.838045 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3516.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.838049 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838050 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.838054 7f868ef60700 5 queue_transactions existing osr(2.27 0x2ed0250)/0x2ed0250 2014-06-09 16:22:11.838056 7f868ef60700 10 op_submit_start 3556 2014-06-09 16:22:11.838058 7f868ef60700 5 queue_transactions (trailing journal) 3556 2014-06-09 16:22:11.838059 7f868ef60700 5 queue_op 0xc2d4780 seq 3556 osr(2.27 0x2ed0250) 826 bytes (queue has 41 ops and 564827 bytes) 2014-06-09 16:22:11.838061 7f868ef60700 10 op_submit_finish 3556 2014-06-09 16:22:11.838063 7f86a5e11700 10 _do_op 0x2fcae360 seq 3516 r = 0, finisher 0xc5a7000 0 2014-06-09 16:22:11.838067 7f86a5e11700 10 _finish_op 0x2fcae360 seq 3516 osr(2.36c 0x2ed1170)/0x2ed1170 2014-06-09 16:22:11.838069 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358a20 (1 active) 2014-06-09 16:22:11.838071 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcb40 (2 active) 2014-06-09 16:22:11.838073 7f86a5e11700 5 _do_op 0xc64bc20 seq 3518 osr(3.4c9 0x3ed13d0)/0x3ed13d0 start 2014-06-09 16:22:11.838075 7f86a5e11700 10 _do_transaction on 0xc2809c0 2014-06-09 16:22:11.838081 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.838081 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41ef0 (19 active) 2014-06-09 16:22:11.838089 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.838103 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:11.838102 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3517.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.838107 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.838108 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.4fa( v 213'628 (0'0,213'628] local-les=243 n=124 ec=118 les/c 243/243 242/242/205) 2014-06-09 16:22:11.838123 7f86a5610700 10 _do_op 0xc64bda0 seq 3517 r = 0, finisher 0xb9c1fa0 0 2014-06-09 16:22:11.838127 7f86a5610700 10 _finish_op 0xc64bda0 seq 3517 osr(1.92 0x384c410)/0x384c410 2014-06-09 16:22:11.838130 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2879e0 (1 active) 2014-06-09 16:22:11.838132 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd680 (2 active) 2014-06-09 16:22:11.838133 7f86a5610700 5 _do_op 0x14dd6300 seq 3519 osr(0.492 0x2bd7410)/0x2bd7410 start 2014-06-09 16:22:11.838136 7f86a5610700 10 _do_transaction on 0x399ce40 2014-06-09 16:22:11.838141 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.838146 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838149 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.838151 7f868ef60700 5 queue_transactions existing osr(3.4fa 0x3ed1720)/0x3ed1720 2014-06-09 16:22:11.838154 7f868ef60700 10 op_submit_start 3557 2014-06-09 16:22:11.838156 7f868ef60700 5 queue_transactions (trailing journal) 3557 2014-06-09 16:22:11.838157 7f868ef60700 5 queue_op 0xc2d5a40 seq 3557 osr(3.4fa 0x3ed1720) 828 bytes (queue has 40 ops and 563968 bytes) 2014-06-09 16:22:11.838160 7f868ef60700 10 op_submit_finish 3557 2014-06-09 16:22:11.838166 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3518.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.838170 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.838162 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.312( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.312( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.838190 7f86a5e11700 10 _do_op 0xc64bc20 seq 3518 r = 0, finisher 0x30c55bc0 0 2014-06-09 16:22:11.838194 7f86a5e11700 10 _finish_op 0xc64bc20 seq 3518 osr(3.4c9 0x3ed13d0)/0x3ed13d0 2014-06-09 16:22:11.838196 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcb40 (1 active) 2014-06-09 16:22:11.838197 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838198 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c7e0 (2 active) 2014-06-09 16:22:11.838200 7f86a5e11700 5 _do_op 0x308cad20 seq 3520 osr(3.542 0x3ed15e0)/0x3ed15e0 start 2014-06-09 16:22:11.838201 7f868ef60700 5 queue_transactions existing osr(2.312 0x2ed0f30)/0x2ed0f30 2014-06-09 16:22:11.838203 7f86a5e11700 10 _do_transaction on 0x4dccfc0 2014-06-09 16:22:11.838203 7f868ef60700 10 op_submit_start 3558 2014-06-09 16:22:11.838205 7f868ef60700 5 queue_transactions (trailing journal) 3558 2014-06-09 16:22:11.838206 7f868ef60700 5 queue_op 0x2e212c0 seq 3558 osr(2.312 0x2ed0f30) 828 bytes (queue has 40 ops and 563461 bytes) 2014-06-09 16:22:11.838208 7f868ef60700 10 op_submit_finish 3558 2014-06-09 16:22:11.838209 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.838218 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.838211 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.604( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.604( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.838234 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3519.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.838238 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.838244 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838248 7f868ef60700 5 queue_transactions existing osr(1.604 0x384dc50)/0x384dc50 2014-06-09 16:22:11.838250 7f868ef60700 10 op_submit_start 3559 2014-06-09 16:22:11.838252 7f868ef60700 5 queue_transactions (trailing journal) 3559 2014-06-09 16:22:11.838252 7f86a5610700 10 _do_op 0x14dd6300 seq 3519 r = 0, finisher 0x30c55480 0 2014-06-09 16:22:11.838253 7f868ef60700 5 queue_op 0x2e200c0 seq 3559 osr(1.604 0x384dc50) 828 bytes (queue has 41 ops and 564289 bytes) 2014-06-09 16:22:11.838256 7f86a5610700 10 _finish_op 0x14dd6300 seq 3519 osr(0.492 0x2bd7410)/0x2bd7410 2014-06-09 16:22:11.838258 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd680 (1 active) 2014-06-09 16:22:11.838260 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89eea0 (2 active) 2014-06-09 16:22:11.838266 7f86a5610700 5 _do_op 0x1e08b560 seq 3521 osr(1.755 0x2ed0090)/0x2ed0090 start 2014-06-09 16:22:11.838269 7f86a5610700 10 _do_transaction on 0xc2ef980 2014-06-09 16:22:11.838268 7f868ef60700 10 op_submit_finish 3559 2014-06-09 16:22:11.838273 7f8689755700 10 op_submit_start 3560 2014-06-09 16:22:11.838274 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.838275 7f8689755700 5 queue_transactions (trailing journal) 3560 2014-06-09 16:22:11.838276 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.838277 7f8689755700 5 queue_op 0x2ea53860 seq 3560 osr(3.55b 0x3ed1980) 1943 bytes (queue has 42 ops and 565468 bytes) 2014-06-09 16:22:11.838279 7f8689755700 10 op_submit_finish 3560 2014-06-09 16:22:11.838273 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.76( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.76( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.838284 7f8692767700 10 op_submit_start 3561 2014-06-09 16:22:11.838286 7f8692767700 5 queue_transactions (trailing journal) 3561 2014-06-09 16:22:11.838288 7f8692767700 5 queue_op 0xc2ead80 seq 3561 osr(3.70e 0x3ed1ba0) 1335 bytes (queue has 42 ops and 565468 bytes) 2014-06-09 16:22:11.838291 7f8692767700 10 op_submit_finish 3561 2014-06-09 16:22:11.838290 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x998a000 (0 active) 2014-06-09 16:22:11.838292 7f86a5e11700 15 _omap_rmkeyrange meta/1fe155aa/pglog_3.542/0//-1 [0000000214.00000000000000000709,4294967295.18446744073709551615] 2014-06-09 16:22:11.838294 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x98a2000 (1 active) 2014-06-09 16:22:11.838295 7f86a5e11700 15 get_omap_iterator meta/1fe155aa/pglog_3.542/0//-1 2014-06-09 16:22:11.838296 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.838286 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.838299 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x98a2000 (0 active) 2014-06-09 16:22:11.838301 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x8e4c000 (1 active) 2014-06-09 16:22:11.838303 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.838301 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.838311 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f23c0 (19 active) 2014-06-09 16:22:11.838311 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838316 7f868ef60700 5 queue_transactions existing osr(1.76 0x384c530)/0x384c530 2014-06-09 16:22:11.838317 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.838311 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.838318 7f868ef60700 10 op_submit_start 3562 2014-06-09 16:22:11.838320 7f868ef60700 5 queue_transactions (trailing journal) 3562 2014-06-09 16:22:11.838321 7f868ef60700 5 queue_op 0x2e21560 seq 3562 osr(1.76 0x384c530) 826 bytes (queue has 43 ops and 566294 bytes) 2014-06-09 16:22:11.838324 7f868ef60700 10 op_submit_finish 3562 2014-06-09 16:22:11.838319 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.838334 7f8689755700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.838327 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.31c( v 213'500 (0'0,213'500] local-les=243 n=100 ec=118 les/c 243/243 242/242/205) 2014-06-09 16:22:11.838346 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1119 2014-06-09 16:22:11.838356 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.838357 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838366 7f8689f56700 10 op_submit_start 3563 2014-06-09 16:22:11.838368 7f8689f56700 5 queue_transactions (trailing journal) 3563 2014-06-09 16:22:11.838369 7f8689f56700 5 queue_op 0x2ea52a80 seq 3563 osr(1.691 0x384de30) 828 bytes (queue has 44 ops and 567122 bytes) 2014-06-09 16:22:11.838372 7f8689f56700 10 op_submit_finish 3563 2014-06-09 16:22:11.838366 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.838376 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.838381 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.838376 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.838386 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.838393 7f8689755700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.838392 7f86a5e11700 15 _omap_rmkeys meta/1fe155aa/pglog_3.542/0//-1 2014-06-09 16:22:11.838392 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.838402 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:144 2014-06-09 16:22:11.838404 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.838416 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.838417 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe155aa/pglog_3.542/0//-1 2014-06-09 16:22:11.838413 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.838425 7f86a5e11700 10 _omap_rmkeys meta/1fe155aa/pglog_3.542/0//-1 r = 0 2014-06-09 16:22:11.838429 7f86a5e11700 15 _omap_rmkeys meta/1fe155aa/pglog_3.542/0//-1 2014-06-09 16:22:11.838428 7f8689f56700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.838420 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.113( v 213'650 (0'0,213'650] local-les=213 n=128 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.838431 7f86a5e11700 10 _omap_rmkeys meta/1fe155aa/pglog_3.542/0//-1 r = 0 2014-06-09 16:22:11.838435 7f86a5e11700 15 _omap_setkeys meta/1fe155aa/pglog_3.542/0//-1 2014-06-09 16:22:11.838439 7f868ff62700 10 op_submit_start 3564 2014-06-09 16:22:11.838441 7f868ff62700 5 queue_transactions (trailing journal) 3564 2014-06-09 16:22:11.838442 7f868ff62700 5 queue_op 0x2c06ec00 seq 3564 osr(3.516 0x3ed16d0) 1335 bytes (queue has 45 ops and 568457 bytes) 2014-06-09 16:22:11.838443 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3520.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.838433 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.113( v 213'650 (0'0,213'650] local-les=213 n=128 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info+log from osd.9 3.113( v 227'654 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) log((213'650,227'654], crt=0'0) 2014-06-09 16:22:11.838445 7f868ff62700 10 op_submit_finish 3564 2014-06-09 16:22:11.838446 7f86a5e11700 10 cid: oid: 1fe155aa/pglog_3.542/0//-1 skipping op, spos 3520.0.0 <= header.spos 150408.0.0 2014-06-09 16:22:11.838447 7f8689755700 10 merge_log log((213'650,227'654], crt=0'0) from osd.9 into log((0'0,213'650], crt=0'0) 2014-06-09 16:22:11.838451 7f8689755700 10 merge_log extending head to 227'654 2014-06-09 16:22:11.838444 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:963 2014-06-09 16:22:11.838456 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.838462 7f868ef60700 5 queue_transactions existing osr(3.31c 0x3ed0d40)/0x3ed0d40 2014-06-09 16:22:11.838462 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e130 (19 active) 2014-06-09 16:22:11.838464 7f868ef60700 10 op_submit_start 3565 2014-06-09 16:22:11.838465 7f868ef60700 5 queue_transactions (trailing journal) 3565 2014-06-09 16:22:11.838466 7f868ef60700 5 queue_op 0x288f95c0 seq 3565 osr(3.31c 0x3ed0d40) 828 bytes (queue has 46 ops and 569285 bytes) 2014-06-09 16:22:11.838469 7f868ef60700 10 op_submit_finish 3565 2014-06-09 16:22:11.838474 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e130 (20 active) 2014-06-09 16:22:11.838479 7f86a5e11700 10 _do_op 0x308cad20 seq 3520 r = 0, finisher 0x32074580 0 2014-06-09 16:22:11.838472 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.631( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.631( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.838483 7f86a5e11700 10 _finish_op 0x308cad20 seq 3520 osr(3.542 0x3ed15e0)/0x3ed15e0 2014-06-09 16:22:11.838478 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.4f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.4f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.838486 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c7e0 (1 active) 2014-06-09 16:22:11.838488 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.838490 7f86a5e11700 5 _do_op 0xbf7f3e0 seq 3522 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.838492 7f86a5e11700 10 _do_transaction on 0xc233380 2014-06-09 16:22:11.838496 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.838502 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.838505 7f868ff62700 5 queue_transactions existing osr(2.4f4 0x2ed16b0)/0x2ed16b0 2014-06-09 16:22:11.838506 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838507 7f868ff62700 10 op_submit_start 3566 2014-06-09 16:22:11.838508 7f868ff62700 5 queue_transactions (trailing journal) 3566 2014-06-09 16:22:11.838509 7f868ff62700 5 queue_op 0x230b9860 seq 3566 osr(2.4f4 0x2ed16b0) 828 bytes (queue has 46 ops and 568387 bytes) 2014-06-09 16:22:11.838511 7f868ff62700 10 op_submit_finish 3566 2014-06-09 16:22:11.838511 7f868ef60700 5 queue_transactions existing osr(1.631 0x384dfb0)/0x384dfb0 2014-06-09 16:22:11.838510 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.838513 7f868ef60700 10 op_submit_start 3567 2014-06-09 16:22:11.838515 7f868ef60700 5 queue_transactions (trailing journal) 3567 2014-06-09 16:22:11.838516 7f868ef60700 5 queue_op 0x288f9bc0 seq 3567 osr(1.631 0x384dfb0) 828 bytes (queue has 47 ops and 569215 bytes) 2014-06-09 16:22:11.838519 7f868ef60700 10 op_submit_finish 3567 2014-06-09 16:22:11.838512 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 2 notify: (query_epoch:242, epoch_sent:243, info:3.4b3( v 213'517 (0'0,213'517] local-les=191 n=98 ec=118 les/c 191/191 242/242/242)) 2014-06-09 16:22:11.838522 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 2, already have info from that osd, ignoring 2014-06-09 16:22:11.838526 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3521.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.838530 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.838531 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.838521 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.62a( v 213'708 (0'0,213'708] local-les=243 n=130 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:11.838533 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.838540 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.838545 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.838549 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.838553 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838556 7f868ff62700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.838555 7f86a5e11700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.838557 7f868ef60700 5 queue_transactions existing osr(3.62a 0x3ed1aa0)/0x3ed1aa0 2014-06-09 16:22:11.838557 7f86a5610700 10 _do_op 0x1e08b560 seq 3521 r = 0, finisher 0x318c3140 0 2014-06-09 16:22:11.838560 7f868ef60700 10 op_submit_start 3568 2014-06-09 16:22:11.838561 7f868ef60700 5 queue_transactions (trailing journal) 3568 2014-06-09 16:22:11.838561 7f86a5610700 10 _finish_op 0x1e08b560 seq 3521 osr(1.755 0x2ed0090)/0x2ed0090 2014-06-09 16:22:11.838563 7f868ef60700 5 queue_op 0x288f97a0 seq 3568 osr(3.62a 0x3ed1aa0) 828 bytes (queue has 47 ops and 568034 bytes) 2014-06-09 16:22:11.838563 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89eea0 (1 active) 2014-06-09 16:22:11.838565 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2879e0 (2 active) 2014-06-09 16:22:11.838570 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.838570 7f86a5610700 5 _do_op 0xbf7f800 seq 3523 osr(1.92 0x384c410)/0x384c410 start 2014-06-09 16:22:11.838570 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.838572 7f86a5610700 10 _do_transaction on 0xbf883c0 2014-06-09 16:22:11.838567 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.838577 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.838583 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.838574 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.838587 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.838588 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.838598 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.838598 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.838598 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.838606 7f868ff62700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.838608 7f868ff62700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.838613 7f86a5e11700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.838611 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.104921 7 0.000188 2014-06-09 16:22:11.838608 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.838619 7f86a5e11700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.838617 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.838618 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.838625 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.838624 7f868ff62700 5 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.838629 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.838627 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.838635 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.838637 7f86a5e11700 15 _write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2797568~516096 2014-06-09 16:22:11.838640 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.838642 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:164 2014-06-09 16:22:11.838653 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.838661 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.838670 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.838671 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.838673 7f868ff62700 5 queue_transactions existing osr(0.1da 0x2bd6920)/0x2bd6920 2014-06-09 16:22:11.838676 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.838688 7f868bf5a700 2 waiting 51 > 50 ops || 573120 > 104857600 2014-06-09 16:22:11.838688 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.838696 7f868df5e700 2 waiting 51 > 50 ops || 636130 > 104857600 2014-06-09 16:22:11.838702 7f8692f68700 2 waiting 51 > 50 ops || 573345 > 104857600 2014-06-09 16:22:11.838697 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.838708 7f868e75f700 2 waiting 51 > 50 ops || 1090672 > 104857600 2014-06-09 16:22:11.838706 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.838714 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.838719 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.838732 7f86a5e11700 10 lookup_strip_header done cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.838730 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:206 2014-06-09 16:22:11.838740 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.838748 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.838748 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.838757 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.838763 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 1 2014-06-09 16:22:11.838779 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.1: 3.1aa( v 213'493 (0'0,213'493] local-les=191 n=97 ec=118 les/c 191/191 242/242/242) log((0'0,213'493], crt=0'0) missing(0) 2014-06-09 16:22:11.838794 7f868a757700 2 waiting 51 > 50 ops || 574235 > 104857600 2014-06-09 16:22:11.838794 7f8689f56700 10 proc_replica_log for osd.1: 3.1aa( v 213'493 (0'0,213'493] local-les=191 n=97 ec=118 les/c 191/191 242/242/242) log((0'0,213'493], crt=0'0) missing(0) 2014-06-09 16:22:11.838807 7f868ef60700 10 op_submit_finish 3568 2014-06-09 16:22:11.838813 7f868ff62700 10 op_submit_start 3569 2014-06-09 16:22:11.838815 7f868ff62700 5 queue_transactions (trailing journal) 3569 2014-06-09 16:22:11.838816 7f868ff62700 5 queue_op 0xbc2db60 seq 3569 osr(0.1da 0x2bd6920) 2001 bytes (queue has 50 ops and 572292 bytes) 2014-06-09 16:22:11.838818 7f868ff62700 10 op_submit_finish 3569 2014-06-09 16:22:11.838821 7f868f761700 10 op_submit_start 3570 2014-06-09 16:22:11.838812 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.13a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.13a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.838823 7f868f761700 5 queue_transactions (trailing journal) 3570 2014-06-09 16:22:11.838825 7f868f761700 5 queue_op 0xbe7db00 seq 3570 osr(3.1eb 0x3ed09c0) 1429 bytes (queue has 50 ops and 572292 bytes) 2014-06-09 16:22:11.838828 7f868f761700 10 op_submit_finish 3570 2014-06-09 16:22:11.838822 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.2c3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.838826 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.13a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.838833 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:243, info:1.43b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.838835 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.13a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.838844 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.13a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.838854 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.838846 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.838857 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.838856 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.838858 7f868ff62700 5 queue_transactions existing osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 2014-06-09 16:22:11.838860 7f868ff62700 2 waiting 51 > 50 ops || 573120 > 104857600 2014-06-09 16:22:11.838861 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.4db( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.4db( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.838873 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.13a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.838888 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838893 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.838897 7f868cf5c700 10 op_submit_start 3571 2014-06-09 16:22:11.838899 7f868cf5c700 5 queue_transactions (trailing journal) 3571 2014-06-09 16:22:11.838901 7f868cf5c700 5 queue_op 0x308cb980 seq 3571 osr(0.663 0x2bd7f30) 828 bytes (queue has 50 ops and 572292 bytes) 2014-06-09 16:22:11.838904 7f868cf5c700 10 op_submit_finish 3571 2014-06-09 16:22:11.838908 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.838909 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.838923 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.646( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.838934 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.838926 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.838452 7f8689755700 20 ? 227'654 (227'653) modify 2594913/rbd_data.11756b8b4567.00000000000015fb/head//3 by client.4565.0:33656 2014-06-09 10:21:06.790159 2014-06-09 16:22:11.838937 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.838800 7f8689f56700 20 merge_log cut point (usually last shared) is 213'493 (213'492) modify 2df891aa/rbd_data.115f6b8b4567.000000000000165d/head//3 by client.4543.0:33062 2014-06-09 10:18:02.642076 2014-06-09 16:22:11.838943 7f8689755700 20 ? 227'653 (227'652) modify 2594913/rbd_data.11756b8b4567.00000000000015fb/head//3 by client.4565.0:33655 2014-06-09 10:21:06.758058 2014-06-09 16:22:11.838939 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.37e( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.37e( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.838946 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.838950 7f8689755700 20 ? 227'652 (227'651) modify 2594913/rbd_data.11756b8b4567.00000000000015fb/head//3 by client.4565.0:33654 2014-06-09 10:21:06.734498 2014-06-09 16:22:11.838952 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.37e( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.838958 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.838955 7f8689755700 20 ? 227'651 (0'0) modify 2594913/rbd_data.11756b8b4567.00000000000015fb/head//3 by client.4565.0:33653 2014-06-09 10:21:06.723561 2014-06-09 16:22:11.838961 7f8689755700 20 merge_log 227'651 (0'0) modify 2594913/rbd_data.11756b8b4567.00000000000015fb/head//3 by client.4565.0:33653 2014-06-09 10:21:06.723561 2014-06-09 16:22:11.838960 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.37e( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.838968 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.37e( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.838972 7f8689755700 20 merge_log 227'652 (227'651) modify 2594913/rbd_data.11756b8b4567.00000000000015fb/head//3 by client.4565.0:33654 2014-06-09 10:21:06.734498 2014-06-09 16:22:11.838980 7f868cf5c700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.838973 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:962 2014-06-09 16:22:11.838981 7f8689755700 20 merge_log 227'653 (227'652) modify 2594913/rbd_data.11756b8b4567.00000000000015fb/head//3 by client.4565.0:33655 2014-06-09 10:21:06.758058 2014-06-09 16:22:11.838988 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.838989 7f8689755700 20 merge_log 227'654 (227'653) modify 2594913/rbd_data.11756b8b4567.00000000000015fb/head//3 by client.4565.0:33656 2014-06-09 10:21:06.790159 2014-06-09 16:22:11.838996 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.37e( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:145 2014-06-09 16:22:11.839009 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.839013 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.839013 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.4c3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.839024 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.752792 7 0.000347 2014-06-09 16:22:11.839029 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.839034 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.839037 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.839044 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.839046 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.839057 7f868ef60700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.839056 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:11.839066 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.839069 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.4c3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:145 2014-06-09 16:22:11.839075 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.839084 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.839082 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.839088 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7af( v 213'628 (0'0,213'628] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.7af( v 213'628 (0'0,213'628] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.839095 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:11.839101 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7af( v 213'628 (0'0,213'628] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.839106 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:11.839110 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7af( v 213'628 (0'0,213'628] local-les=243 n=120 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.839115 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.839118 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7af( v 213'628 (0'0,213'628] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.839129 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.839130 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.839145 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:181 2014-06-09 16:22:11.839177 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.839180 7f868cf5c700 5 queue_transactions existing osr(0.67e 0x2bd7e30)/0x2bd7e30 2014-06-09 16:22:11.839172 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.7af( v 213'628 (0'0,213'628] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1040 2014-06-09 16:22:11.839183 7f868cf5c700 2 waiting 51 > 50 ops || 573153 > 104857600 2014-06-09 16:22:11.839189 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.839192 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.75b( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.839203 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.839212 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.839215 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:1.76c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.839224 7f868f761700 5 queue_transactions existing osr(1.4db 0x384d850)/0x384d850 2014-06-09 16:22:11.839227 7f868f761700 2 waiting 51 > 50 ops || 573120 > 104857600 2014-06-09 16:22:11.839226 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.839235 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.839236 7f8689755700 10 merge_log result log((0'0,227'654], crt=227'654) missing(1) changed=1 2014-06-09 16:22:11.839238 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.1bc( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.839241 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 213'650 (0'0,227'654] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=227'654 lcod 0'0 inactive NOTIFY m=1] exit Started/Stray 1.071359 8 0.000358 2014-06-09 16:22:11.839250 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.839253 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 213'650 (0'0,227'654] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=227'654 lcod 0'0 inactive NOTIFY m=1] enter Started/ReplicaActive 2014-06-09 16:22:11.839259 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.839262 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 213'650 (0'0,227'654] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=227'654 lcod 0'0 inactive NOTIFY m=1] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.839266 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.839270 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 213'650 (0'0,227'654] local-les=213 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=227'654 lcod 0'0 inactive NOTIFY m=1] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.839278 7f868ef60700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.839278 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 213'650 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=227'654 lcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.839292 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.1bc( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.839307 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.839311 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.6e8( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.839339 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.839344 7f868ef60700 5 queue_transactions existing osr(1.6e8 0x2ed0190)/0x2ed0190 2014-06-09 16:22:11.839346 7f868ef60700 2 waiting 51 > 50 ops || 573120 > 104857600 2014-06-09 16:22:11.839365 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=227'654 lcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.839394 7f8689755700 10 write_log with: dirty_to: 0'0, dirty_from: 213'650, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.839385 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.1 now 3.1aa( v 213'493 (0'0,213'493] local-les=191 n=97 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.839399 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.839407 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.547972 4 0.000256 2014-06-09 16:22:11.839415 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.839421 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.053530 0 0.000000 2014-06-09 16:22:11.839429 7f8689755700 5 queue_transactions existing osr(3.113 0x3ed0800)/0x3ed0800 2014-06-09 16:22:11.839431 7f8689755700 2 waiting 51 > 50 ops || 575041 > 104857600 2014-06-09 16:22:11.839430 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.836995 7f866f5f0700 5 -- op tracker -- , seq: 3292, time: 2014-06-09 16:22:11.836668, event: header_read, op: osd_op(client.4539.0:34414 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3858432~335872] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.839436 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=191 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.839446 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.839445 7f866f5f0700 5 -- op tracker -- , seq: 3292, time: 2014-06-09 16:22:11.836668, event: throttled, op: osd_op(client.4539.0:34414 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3858432~335872] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.839453 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.839456 7f866f5f0700 5 -- op tracker -- , seq: 3292, time: 2014-06-09 16:22:11.836972, event: all_read, op: osd_op(client.4539.0:34414 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3858432~335872] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.839462 7f866f5f0700 5 -- op tracker -- , seq: 3292, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34414 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3858432~335872] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.839458 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'499 -> 214'499 2014-06-09 16:22:11.839468 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.839486 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827800 prio 63 cost 335872 latency 0.002818 osd_op(client.4539.0:34414 rbd_data.115b6b8b4567.00000000000018ac [set-alloc-hint object_size 4194304 write_size 4194304,write 3858432~335872] 3.735e6e5d RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.839490 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.1aa( v 213'493 (0'0,213'493] local-les=191 n=97 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.839505 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 sending log((213'493,214'499], crt=0'0) 2014-06-09 16:22:11.839512 7f8689f56700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.839523 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.1 3.1aa( v 214'499 lc 213'493 (0'0,214'499] local-les=191 n=97 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.839532 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.1 has 1 missing 2014-06-09 16:22:11.839538 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing f0e71aa/rbd_data.11816b8b4567.00000000000018ee/head//3 214'499 is on osd.26 2014-06-09 16:22:11.839545 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {f0e71aa/rbd_data.11816b8b4567.00000000000018ee/head//3=214'499} 2014-06-09 16:22:11.839551 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 1 as a source 2014-06-09 16:22:11.839557 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing f0e71aa/rbd_data.11816b8b4567.00000000000018ee/head//3 214'499 also missing on osd.1 2014-06-09 16:22:11.839562 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {f0e71aa/rbd_data.11816b8b4567.00000000000018ee/head//3=214'499} 2014-06-09 16:22:11.839568 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.839573 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.839579 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 1 2014-06-09 16:22:11.839584 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.839590 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.839682 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.839685 7f8689f56700 5 queue_transactions existing osr(3.1aa 0x3ed0ac0)/0x3ed0ac0 2014-06-09 16:22:11.839687 7f8689f56700 2 waiting 51 > 50 ops || 573345 > 104857600 2014-06-09 16:22:11.840378 7f86a5e11700 10 _generic_write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2797568~516096 = 0 2014-06-09 16:22:11.840418 7f86a5e11700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.840431 7f86a5e11700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.840437 7f86a5e11700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.840444 7f86a5e11700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.840524 7f86a5e11700 10 cid: oid: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 not skipping op, *spos 3522.0.0 2014-06-09 16:22:11.840528 7f86a5e11700 10 > header.spos 3492.0.0 2014-06-09 16:22:11.840607 7f86a5e11700 20 set_header setting 30152 cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 parent seq 0 2014-06-09 16:22:11.840667 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3522.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.840674 7f86a5e11700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3522.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.840677 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.842623 7f866f5f0700 5 -- op tracker -- , seq: 3293, time: 2014-06-09 16:22:11.839506, event: header_read, op: osd_op(client.4539.0:34415 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1728512] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.842636 7f866f5f0700 5 -- op tracker -- , seq: 3293, time: 2014-06-09 16:22:11.839507, event: throttled, op: osd_op(client.4539.0:34415 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1728512] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.842648 7f866f5f0700 5 -- op tracker -- , seq: 3293, time: 2014-06-09 16:22:11.842581, event: all_read, op: osd_op(client.4539.0:34415 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1728512] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.842675 7f866f5f0700 5 -- op tracker -- , seq: 3293, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34415 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1728512] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.842686 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.842712 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827700 prio 63 cost 1728512 latency 0.003206 osd_op(client.4539.0:34415 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1728512] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.843355 7f86a5e11700 10 _do_op 0xbf7f3e0 seq 3522 r = 0, finisher 0x4eb8920 0x309ea7c0 2014-06-09 16:22:11.843363 7f86a5e11700 10 _finish_op 0xbf7f3e0 seq 3522 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.843373 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.843376 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1145a0 (2 active) 2014-06-09 16:22:11.843376 7f868bf5a700 10 op_submit_start 3572 2014-06-09 16:22:11.843378 7f86a5e11700 5 _do_op 0xbf7faa0 seq 3524 osr(0.e8 0x2bd6510)/0x2bd6510 start 2014-06-09 16:22:11.843379 7f868bf5a700 5 queue_transactions (trailing journal) 3572 2014-06-09 16:22:11.843382 7f86a5e11700 10 _do_transaction on 0xc141200 2014-06-09 16:22:11.843382 7f868bf5a700 5 queue_op 0xc64a0c0 seq 3572 osr(1.603 0x384dc90) 828 bytes (queue has 50 ops and 54740 bytes) 2014-06-09 16:22:11.843387 7f868bf5a700 10 op_submit_finish 3572 2014-06-09 16:22:11.843388 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.843393 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.669( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.669( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.843415 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.669( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.843426 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.669( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843434 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.669( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843453 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.843459 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.843474 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.669( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.843492 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.843500 7f868df5e700 2 waiting 51 > 50 ops || 118578 > 104857600 2014-06-09 16:22:11.843508 7f8692f68700 2 waiting 51 > 50 ops || 55793 > 104857600 2014-06-09 16:22:11.843514 7f868e75f700 2 waiting 51 > 50 ops || 573120 > 104857600 2014-06-09 16:22:11.843521 7f868a757700 2 waiting 51 > 50 ops || 56683 > 104857600 2014-06-09 16:22:11.843529 7f868ff62700 2 waiting 51 > 50 ops || 55568 > 104857600 2014-06-09 16:22:11.843527 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3524.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.843522 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.35( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.35( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.843532 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.843535 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.35( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.843546 7f86a5e11700 10 _do_op 0xbf7faa0 seq 3524 r = 0, finisher 0xbce7360 0 2014-06-09 16:22:11.843544 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.35( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843550 7f86a5e11700 10 _finish_op 0xbf7faa0 seq 3524 osr(0.e8 0x2bd6510)/0x2bd6510 2014-06-09 16:22:11.843556 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1145a0 (1 active) 2014-06-09 16:22:11.843552 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.35( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843559 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ed80 (2 active) 2014-06-09 16:22:11.843561 7f86a5e11700 5 _do_op 0xbf7e120 seq 3525 osr(0.8c 0x2bd6480)/0x2bd6480 start 2014-06-09 16:22:11.843564 7f86a5e11700 10 _do_transaction on 0xc31d380 2014-06-09 16:22:11.843564 7f868bf5a700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.843569 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.843580 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.35( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:156 2014-06-09 16:22:11.843595 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.843602 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.843599 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.843634 7f868cf5c700 10 op_submit_start 3573 2014-06-09 16:22:11.843636 7f868cf5c700 5 queue_transactions (trailing journal) 3573 2014-06-09 16:22:11.843637 7f868cf5c700 5 queue_op 0x30d96960 seq 3573 osr(0.67e 0x2bd7e30) 861 bytes (queue has 50 ops and 54775 bytes) 2014-06-09 16:22:11.843641 7f868cf5c700 10 op_submit_finish 3573 2014-06-09 16:22:11.843641 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3525.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.843646 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.843653 7f868f761700 2 waiting 51 > 50 ops || 55603 > 104857600 2014-06-09 16:22:11.843645 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3dc( v 213'619 (0'0,213'619] local-les=243 n=117 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.3dc( v 213'619 (0'0,213'619] local-les=243 n=117 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.843657 7f86a5e11700 10 _do_op 0xbf7e120 seq 3525 r = 0, finisher 0x32480340 0 2014-06-09 16:22:11.843661 7f86a5e11700 10 _finish_op 0xbf7e120 seq 3525 osr(0.8c 0x2bd6480)/0x2bd6480 2014-06-09 16:22:11.843665 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ed80 (1 active) 2014-06-09 16:22:11.843667 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc240 (2 active) 2014-06-09 16:22:11.843670 7f86a5e11700 5 _do_op 0x33a2d20 seq 3526 osr(0.521 0x2bd7570)/0x2bd7570 start 2014-06-09 16:22:11.843661 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3dc( v 213'619 (0'0,213'619] local-les=243 n=117 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.843672 7f86a5e11700 10 _do_transaction on 0x3934cc0 2014-06-09 16:22:11.843677 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.843673 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3dc( v 213'619 (0'0,213'619] local-les=243 n=117 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843682 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3dc( v 213'619 (0'0,213'619] local-les=243 n=117 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843693 7f868f761700 10 op_submit_start 3574 2014-06-09 16:22:11.843696 7f868f761700 5 queue_transactions (trailing journal) 3574 2014-06-09 16:22:11.843696 7f868cf5c700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.843697 7f868f761700 5 queue_op 0xbe7cea0 seq 3574 osr(1.4db 0x384d850) 828 bytes (queue has 50 ops and 54777 bytes) 2014-06-09 16:22:11.843701 7f868f761700 10 op_submit_finish 3574 2014-06-09 16:22:11.843710 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.843705 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.843722 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.843734 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.843712 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.3dc( v 213'619 (0'0,213'619] local-les=243 n=117 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1013 2014-06-09 16:22:11.843745 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.843748 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3526.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.843752 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.843741 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.843750 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.843763 7f86a5e11700 10 _do_op 0x33a2d20 seq 3526 r = 0, finisher 0x3281bd80 0 2014-06-09 16:22:11.843756 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.843767 7f86a5e11700 10 _finish_op 0x33a2d20 seq 3526 osr(0.521 0x2bd7570)/0x2bd7570 2014-06-09 16:22:11.843769 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc240 (1 active) 2014-06-09 16:22:11.843771 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114480 (2 active) 2014-06-09 16:22:11.843763 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.843773 7f86a5e11700 5 _do_op 0x33a3f80 seq 3527 osr(1.67a 0x384de90)/0x384de90 start 2014-06-09 16:22:11.843775 7f86a5e11700 10 _do_transaction on 0xc145680 2014-06-09 16:22:11.843768 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843773 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843780 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.843777 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843782 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843793 7f868f761700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.843795 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.843804 7f8689755700 10 op_submit_start 3575 2014-06-09 16:22:11.843806 7f8689755700 5 queue_transactions (trailing journal) 3575 2014-06-09 16:22:11.843807 7f8689f56700 2 waiting 51 > 50 ops || 57751 > 104857600 2014-06-09 16:22:11.843808 7f8689755700 5 queue_op 0x11fcefc0 seq 3575 osr(3.113 0x3ed0800) 2749 bytes (queue has 50 ops and 56698 bytes) 2014-06-09 16:22:11.843811 7f8689755700 10 op_submit_finish 3575 2014-06-09 16:22:11.843813 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.843814 7f8692f68700 2 waiting 51 > 50 ops || 57751 > 104857600 2014-06-09 16:22:11.843818 7f868e75f700 2 waiting 51 > 50 ops || 575078 > 104857600 2014-06-09 16:22:11.843810 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:932 2014-06-09 16:22:11.843815 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.843826 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3527.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.843828 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.843830 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.843829 7f868df5e700 2 waiting 51 > 50 ops || 120536 > 104857600 2014-06-09 16:22:11.843835 7f868ff62700 2 waiting 51 > 50 ops || 57526 > 104857600 2014-06-09 16:22:11.843829 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.843840 7f86a5e11700 10 _do_op 0x33a3f80 seq 3527 r = 0, finisher 0x3050bce0 0 2014-06-09 16:22:11.843843 7f86a5e11700 10 _finish_op 0x33a3f80 seq 3527 osr(1.67a 0x384de90)/0x384de90 2014-06-09 16:22:11.843833 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.843839 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843612 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.843848 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114480 (1 active) 2014-06-09 16:22:11.843850 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcb40 (2 active) 2014-06-09 16:22:11.843851 7f8689f56700 10 op_submit_start 3576 2014-06-09 16:22:11.843853 7f8689f56700 5 queue_transactions (trailing journal) 3576 2014-06-09 16:22:11.843847 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843853 7f86a5e11700 5 _do_op 0x308caa20 seq 3528 osr(0.71f 0x384c060)/0x384c060 start 2014-06-09 16:22:11.843855 7f86a5e11700 10 _do_transaction on 0xc138c00 2014-06-09 16:22:11.843849 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843854 7f8689f56700 5 queue_op 0x2fc8c0c0 seq 3576 osr(3.1aa 0x3ed0ac0) 1053 bytes (queue has 50 ops and 56923 bytes) 2014-06-09 16:22:11.843849 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.843858 7f8689f56700 10 op_submit_finish 3576 2014-06-09 16:22:11.843860 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.843860 7f8689755700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.843861 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.843857 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843868 7f868bf5a700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.843862 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.843865 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.843875 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.843876 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.843875 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.843884 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.843878 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:149 2014-06-09 16:22:11.843890 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.843891 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.843884 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843893 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.843892 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843894 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.843894 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.843904 7f8689f56700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.843906 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0270 (19 active) 2014-06-09 16:22:11.843909 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3528.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.843914 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.843914 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.843907 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.843907 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.2 3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) 2014-06-09 16:22:11.843917 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.843918 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.843921 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 0.934106 7 0.000164 2014-06-09 16:22:11.843924 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.843934 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.843936 7f868bf5a700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.843934 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.843941 7f86a5e11700 10 _do_op 0x308caa20 seq 3528 r = 0, finisher 0x32073fe0 0 2014-06-09 16:22:11.843944 7f86a5e11700 10 _finish_op 0x308caa20 seq 3528 osr(0.71f 0x384c060)/0x384c060 2014-06-09 16:22:11.843938 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.843947 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcb40 (1 active) 2014-06-09 16:22:11.843949 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3586c0 (2 active) 2014-06-09 16:22:11.843951 7f86a5e11700 5 _do_op 0xcdab860 seq 3529 osr(2.372 0x2ed1150)/0x2ed1150 start 2014-06-09 16:22:11.843954 7f86a5e11700 10 _do_transaction on 0x3339800 2014-06-09 16:22:11.843945 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:979 2014-06-09 16:22:11.843947 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.843950 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.843959 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.843958 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.843958 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=191 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.843959 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.843962 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.843963 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 6 2014-06-09 16:22:11.843970 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.843969 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'564 -> 213'564 2014-06-09 16:22:11.843977 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.843978 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.843977 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd6 2014-06-09 16:22:11.843986 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.843992 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.843988 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.844000 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (19 active) 2014-06-09 16:22:11.844003 7f8692f68700 10 op_submit_start 3577 2014-06-09 16:22:11.844004 7f8692f68700 5 queue_transactions (trailing journal) 3577 2014-06-09 16:22:11.843997 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.844005 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3529.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.844006 7f8692f68700 5 queue_op 0x30d961e0 seq 3577 osr(3.15c 0x3ed0720) 1053 bytes (queue has 50 ops and 56073 bytes) 2014-06-09 16:22:11.844009 7f8692f68700 10 op_submit_finish 3577 2014-06-09 16:22:11.844009 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.844007 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 213'561 (0'0,213'561] local-les=208 n=110 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.6: log((213'561,214'563], crt=0'0) missing(0) 2014-06-09 16:22:11.844019 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (20 active) 2014-06-09 16:22:11.844017 7f868bf5a700 10 merge_log log((213'561,214'563], crt=0'0) from osd.6 into log((0'0,213'561], crt=0'0) 2014-06-09 16:22:11.844014 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.844022 7f868bf5a700 10 merge_log extending head to 214'563 2014-06-09 16:22:11.844014 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:0.4b3( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.844028 7f86a5e11700 10 _do_op 0xcdab860 seq 3529 r = 0, finisher 0x32073a80 0 2014-06-09 16:22:11.844023 7f868bf5a700 20 ? 214'563 (214'562) modify 8880fc28/rbd_data.11636b8b4567.0000000000001813/head//3 by client.4547.0:35323 2014-06-09 10:19:08.514491 2014-06-09 16:22:11.844031 7f86a5e11700 10 _finish_op 0xcdab860 seq 3529 osr(2.372 0x2ed1150)/0x2ed1150 2014-06-09 16:22:11.844024 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.7ce( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.7ce( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 243/243 242/242/220) 2014-06-09 16:22:11.844034 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3586c0 (1 active) 2014-06-09 16:22:11.844028 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.844032 7f868bf5a700 20 ? 214'562 (0'0) modify 8880fc28/rbd_data.11636b8b4567.0000000000001813/head//3 by client.4547.0:35322 2014-06-09 10:19:08.440893 2014-06-09 16:22:11.844036 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114ea0 (2 active) 2014-06-09 16:22:11.844038 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.844039 7f86a5e11700 5 _do_op 0xcdabf20 seq 3530 osr(0.301 0x2bd6e10)/0x2bd6e10 start 2014-06-09 16:22:11.844041 7f86a5e11700 10 _do_transaction on 0x2ef9080 2014-06-09 16:22:11.844038 7f868bf5a700 20 merge_log 214'562 (0'0) modify 8880fc28/rbd_data.11636b8b4567.0000000000001813/head//3 by client.4547.0:35322 2014-06-09 10:19:08.440893 2014-06-09 16:22:11.844046 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.844042 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.844048 7f868bf5a700 20 merge_log 214'563 (214'562) modify 8880fc28/rbd_data.11636b8b4567.0000000000001813/head//3 by client.4547.0:35323 2014-06-09 10:19:08.514491 2014-06-09 16:22:11.844055 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.844060 7f8689755700 5 queue_transactions existing osr(3.1e1 0x3ed09b0)/0x3ed09b0 2014-06-09 16:22:11.844055 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.844063 7f8689755700 10 op_submit_start 3578 2014-06-09 16:22:11.844064 7f8689755700 5 queue_transactions (trailing journal) 3578 2014-06-09 16:22:11.844066 7f8689755700 5 queue_op 0x16097d40 seq 3578 osr(3.1e1 0x3ed09b0) 1429 bytes (queue has 50 ops and 56674 bytes) 2014-06-09 16:22:11.844068 7f8689755700 10 op_submit_finish 3578 2014-06-09 16:22:11.844068 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.844064 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.844073 7f868cf5c700 5 queue_transactions existing osr(3.7ce 0xb7f01a0)/0xb7f01a0 2014-06-09 16:22:11.844075 7f868cf5c700 2 waiting 51 > 50 ops || 57502 > 104857600 2014-06-09 16:22:11.844071 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.844077 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.844071 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 4 notify: (query_epoch:243, epoch_sent:243, info:0.708( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.844083 7f8692f68700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.844084 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.844083 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.844093 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.844091 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3530.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.844096 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.844099 7f868a757700 2 waiting 51 > 50 ops || 58617 > 104857600 2014-06-09 16:22:11.844094 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.844104 7f86a5e11700 10 _do_op 0xcdabf20 seq 3530 r = 0, finisher 0x32073c80 0 2014-06-09 16:22:11.844107 7f86a5e11700 10 _finish_op 0xcdabf20 seq 3530 osr(0.301 0x2bd6e10)/0x2bd6e10 2014-06-09 16:22:11.844102 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.844101 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.844111 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.844114 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.844107 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.844115 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.844119 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.844118 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.844126 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.844127 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.844135 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.844130 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.844134 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.844138 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.844138 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.844146 7f8689755700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.844149 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.844145 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.844149 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.844156 7f8692f68700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.844157 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.844157 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.844164 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3523.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.844168 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.844170 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.844161 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=212 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.844169 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:174 2014-06-09 16:22:11.844182 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.844174 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=212 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:11.844174 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.364( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.364( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.844183 7f86a5610700 10 _do_op 0xbf7f800 seq 3523 r = 0, finisher 0x1a003080 0 2014-06-09 16:22:11.844193 7f868ff62700 10 op_submit_start 3579 2014-06-09 16:22:11.844186 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=212 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.394681 7 0.000153 2014-06-09 16:22:11.844195 7f868ff62700 5 queue_transactions (trailing journal) 3579 2014-06-09 16:22:11.844185 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.844196 7f868ff62700 5 queue_op 0xc2d5ec0 seq 3579 osr(0.2c3 0x2bd6ba0) 828 bytes (queue has 50 ops and 56674 bytes) 2014-06-09 16:22:11.844201 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114ea0 (1 active) 2014-06-09 16:22:11.844197 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=212 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.844203 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd0e0 (2 active) 2014-06-09 16:22:11.844198 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.844207 7f86a5e11700 5 _do_op 0xcdaa3c0 seq 3531 osr(2.75 0x2ed04f0)/0x2ed04f0 start 2014-06-09 16:22:11.844210 7f86a5e11700 10 _do_transaction on 0xc3209c0 2014-06-09 16:22:11.844207 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.844215 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.844217 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.844221 7f8689755700 5 queue_transactions existing osr(2.364 0x2ed11b0)/0x2ed11b0 2014-06-09 16:22:11.844215 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.844216 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=212 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.844226 7f8689755700 2 waiting 51 > 50 ops || 57502 > 104857600 2014-06-09 16:22:11.844228 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.844230 7f868a757700 2 waiting 51 > 50 ops || 58617 > 104857600 2014-06-09 16:22:11.844225 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=212 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.844235 7f868cf5c700 2 waiting 51 > 50 ops || 57502 > 104857600 2014-06-09 16:22:11.844241 7f86a5610700 10 _finish_op 0xbf7f800 seq 3523 osr(1.92 0x384c410)/0x384c410 2014-06-09 16:22:11.844235 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'588 -> 213'588 2014-06-09 16:22:11.844238 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:893 2014-06-09 16:22:11.844246 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2879e0 (1 active) 2014-06-09 16:22:11.844248 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ed80 (2 active) 2014-06-09 16:22:11.844252 7f868ff62700 10 op_submit_finish 3579 2014-06-09 16:22:11.844253 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.844252 7f868bf5a700 10 merge_log result log((0'0,214'563], crt=214'563) missing(1) changed=1 2014-06-09 16:22:11.844257 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.1ff( v 214'683 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.844257 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 3 2014-06-09 16:22:11.844258 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 peering m=1] peer osd.6 now 3.428( v 214'563 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) missing(0) 2014-06-09 16:22:11.844271 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd3 2014-06-09 16:22:11.844272 7f8692f68700 7 osd.26 pg_epoch: 243 pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: handle_pg_notify from osd.8 2014-06-09 16:22:11.844275 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.844283 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.844282 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.608687 3 0.000200 2014-06-09 16:22:11.844282 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] got dup osd.8 info 3.1ff( v 214'683 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242), identical to ours 2014-06-09 16:22:11.844293 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.844302 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.844298 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.844302 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'622 (0'0,214'622] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.3: log((214'622,214'624], crt=0'0) missing(0) 2014-06-09 16:22:11.844312 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.844306 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.1fc( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:11.844313 7f868ff62700 10 merge_log log((214'622,214'624], crt=0'0) from osd.3 into log((0'0,214'622], crt=0'0) 2014-06-09 16:22:11.844316 7f8689f56700 5 queue_transactions existing osr(3.555 0x3ed15b0)/0x3ed15b0 2014-06-09 16:22:11.844317 7f868ff62700 10 merge_log extending head to 214'624 2014-06-09 16:22:11.844319 7f8689f56700 10 op_submit_start 3580 2014-06-09 16:22:11.844309 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 peering m=1] state: osd.6 has no missing, identical log 2014-06-09 16:22:11.844320 7f8689f56700 5 queue_transactions (trailing journal) 3580 2014-06-09 16:22:11.844322 7f8689f56700 5 queue_op 0x30d97d40 seq 3580 osr(3.555 0x3ed15b0) 1335 bytes (queue has 50 ops and 57183 bytes) 2014-06-09 16:22:11.844325 7f8689f56700 10 op_submit_finish 3580 2014-06-09 16:22:11.844318 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.844318 7f868ff62700 20 ? 214'624 (214'623) modify de5f0805/rbd_data.117b6b8b4567.00000000000017ec/head//3 by client.4571.0:35813 2014-06-09 10:19:05.966957 2014-06-09 16:22:11.844327 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.844327 7f868ff62700 20 ? 214'623 (214'622) modify de5f0805/rbd_data.117b6b8b4567.00000000000017ec/head//3 by client.4571.0:35812 2014-06-09 10:19:01.293997 2014-06-09 16:22:11.844325 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000028 0 0.000000 2014-06-09 16:22:11.844333 7f868ff62700 20 merge_log 214'623 (214'622) modify de5f0805/rbd_data.117b6b8b4567.00000000000017ec/head//3 by client.4571.0:35812 2014-06-09 10:19:01.293997 2014-06-09 16:22:11.844339 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f25d0 (19 active) 2014-06-09 16:22:11.844331 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.844344 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.844341 7f868ff62700 20 merge_log 214'624 (214'623) modify de5f0805/rbd_data.117b6b8b4567.00000000000017ec/head//3 by client.4571.0:35813 2014-06-09 10:19:05.966957 2014-06-09 16:22:11.844339 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.844343 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.844351 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.844350 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.103269 0 0.000000 2014-06-09 16:22:11.844358 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.844371 7f8692f68700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.844366 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.844376 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.844388 7f868ef60700 2 waiting 51 > 50 ops || 58011 > 104857600 2014-06-09 16:22:11.844383 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.844394 7f868a757700 2 waiting 51 > 50 ops || 59126 > 104857600 2014-06-09 16:22:11.844397 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.844398 7f868cf5c700 2 waiting 51 > 50 ops || 58011 > 104857600 2014-06-09 16:22:11.844389 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.844401 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.73( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.844401 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.844410 7f869af78700 5 -- op tracker -- , seq: 3274, time: 2014-06-09 16:22:11.844407, event: done, op: pg_info(1 pgs e243:3.15c) 2014-06-09 16:22:11.844411 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 213'561 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.844427 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844429 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844437 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.844441 7f8692f68700 5 queue_transactions existing osr(2.73 0x2ed0510)/0x2ed0510 2014-06-09 16:22:11.844444 7f8692f68700 2 waiting 51 > 50 ops || 58009 > 104857600 2014-06-09 16:22:11.844445 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844447 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844449 7f869af78700 20 osd.26 243 _dispatch 0x30c13a40 pg_info(1 pgs e243:0.2a2) v4 2014-06-09 16:22:11.844454 7f869af78700 5 -- op tracker -- , seq: 3294, time: 2014-06-09 16:22:11.752823, event: header_read, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.844461 7f869af78700 5 -- op tracker -- , seq: 3294, time: 2014-06-09 16:22:11.752824, event: throttled, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.844467 7f869af78700 5 -- op tracker -- , seq: 3294, time: 2014-06-09 16:22:11.752839, event: all_read, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.844471 7f8689755700 2 waiting 51 > 50 ops || 58011 > 104857600 2014-06-09 16:22:11.844474 7f869af78700 5 -- op tracker -- , seq: 3294, time: 2014-06-09 16:22:11.844445, event: dispatched, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.844481 7f869af78700 5 -- op tracker -- , seq: 3294, time: 2014-06-09 16:22:11.844479, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.844486 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2a2) v4 from osd.4 2014-06-09 16:22:11.844490 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c13a40 2014-06-09 16:22:11.844496 7f869af78700 5 -- op tracker -- , seq: 3294, time: 2014-06-09 16:22:11.844494, event: started, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.844507 7f868e75f700 2 waiting 51 > 50 ops || 575563 > 104857600 2014-06-09 16:22:11.844514 7f869af78700 5 -- op tracker -- , seq: 3294, time: 2014-06-09 16:22:11.844511, event: done, op: pg_info(1 pgs e243:0.2a2) 2014-06-09 16:22:11.844522 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844525 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844522 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.844540 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844542 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844535 7f868bf5a700 10 osd.26 243 queue_for_recovery queued pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.844543 7f869af78700 20 osd.26 243 _dispatch 0x30c90700 pg_info(1 pgs e243:2.48f) v4 2014-06-09 16:22:11.844549 7f869af78700 5 -- op tracker -- , seq: 3295, time: 2014-06-09 16:22:11.745243, event: header_read, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.844556 7f869af78700 5 -- op tracker -- , seq: 3295, time: 2014-06-09 16:22:11.745244, event: throttled, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.844548 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.6 3.428( v 214'563 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) 2014-06-09 16:22:11.844564 7f868ff62700 10 merge_log result log((0'0,214'624], crt=214'624) missing(1) changed=1 2014-06-09 16:22:11.844563 7f869af78700 5 -- op tracker -- , seq: 3295, time: 2014-06-09 16:22:11.745256, event: all_read, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.844561 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.6 is up to date, queueing in pending_activators 2014-06-09 16:22:11.844571 7f869af78700 5 -- op tracker -- , seq: 3295, time: 2014-06-09 16:22:11.844540, event: dispatched, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.844568 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 peering m=1] peer osd.3 now 3.5( v 214'624 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) missing(0) 2014-06-09 16:22:11.844579 7f869af78700 5 -- op tracker -- , seq: 3295, time: 2014-06-09 16:22:11.844577, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.844584 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.48f) v4 from osd.8 2014-06-09 16:22:11.844587 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c90700 2014-06-09 16:22:11.844579 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.6 3.428( v 214'563 (0'0,214'563] local-les=208 n=111 ec=118 les/c 208/208 242/242/242) uptodate 2014-06-09 16:22:11.844592 7f869af78700 5 -- op tracker -- , seq: 3295, time: 2014-06-09 16:22:11.844590, event: started, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.844587 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.599827 3 0.000213 2014-06-09 16:22:11.844594 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.844600 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.844613 7f869af78700 5 -- op tracker -- , seq: 3295, time: 2014-06-09 16:22:11.844610, event: done, op: pg_info(1 pgs e243:2.48f) 2014-06-09 16:22:11.844607 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 8880fc28/rbd_data.11636b8b4567.0000000000001813/head//3 214'563 also missing on osd.26 2014-06-09 16:22:11.844610 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 peering m=1] state: osd.3 has no missing, identical log 2014-06-09 16:22:11.844621 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844623 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844619 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {8880fc28/rbd_data.11636b8b4567.0000000000001813/head//3=214'563} 2014-06-09 16:22:11.844623 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000023 0 0.000000 2014-06-09 16:22:11.844635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844637 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844639 7f869af78700 20 osd.26 243 _dispatch 0x30a19dc0 pg_info(1 pgs e243:3.51e) v4 2014-06-09 16:22:11.844631 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 6 as a source 2014-06-09 16:22:11.844634 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.844644 7f869af78700 5 -- op tracker -- , seq: 3296, time: 2014-06-09 16:22:11.744151, event: header_read, op: pg_info(1 pgs e243:3.51e) 2014-06-09 16:22:11.844643 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.664754 0 0.000000 2014-06-09 16:22:11.844642 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 8880fc28/rbd_data.11636b8b4567.0000000000001813/head//3 214'563 is on osd.6 2014-06-09 16:22:11.844651 7f869af78700 5 -- op tracker -- , seq: 3296, time: 2014-06-09 16:22:11.744151, event: throttled, op: pg_info(1 pgs e243:3.51e) 2014-06-09 16:22:11.844659 7f869af78700 5 -- op tracker -- , seq: 3296, time: 2014-06-09 16:22:11.744165, event: all_read, op: pg_info(1 pgs e243:3.51e) 2014-06-09 16:22:11.844655 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.844656 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {8880fc28/rbd_data.11636b8b4567.0000000000001813/head//3=214'563} 2014-06-09 16:22:11.844666 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.844669 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.844677 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.844680 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 208 2014-06-09 16:22:11.844686 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.844696 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'622 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.844693 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 6 2014-06-09 16:22:11.844705 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.844717 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.844666 7f869af78700 5 -- op tracker -- , seq: 3296, time: 2014-06-09 16:22:11.844635, event: dispatched, op: pg_info(1 pgs e243:3.51e) 2014-06-09 16:22:11.844754 7f868bf5a700 10 write_log with: dirty_to: 0'0, dirty_from: 213'561, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.844754 7f869af78700 5 -- op tracker -- , seq: 3296, time: 2014-06-09 16:22:11.844751, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.51e) 2014-06-09 16:22:11.844760 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.51e) v4 from osd.2 2014-06-09 16:22:11.844763 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a19dc0 2014-06-09 16:22:11.844768 7f869af78700 5 -- op tracker -- , seq: 3296, time: 2014-06-09 16:22:11.844766, event: started, op: pg_info(1 pgs e243:3.51e) 2014-06-09 16:22:11.844786 7f869af78700 5 -- op tracker -- , seq: 3296, time: 2014-06-09 16:22:11.844783, event: done, op: pg_info(1 pgs e243:3.51e) 2014-06-09 16:22:11.844797 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844795 7f868bf5a700 5 queue_transactions existing osr(3.428 0x3ed11e0)/0x3ed11e0 2014-06-09 16:22:11.844799 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844799 7f868bf5a700 2 waiting 51 > 50 ops || 59126 > 104857600 2014-06-09 16:22:11.844811 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844804 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.844813 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844815 7f869af78700 20 osd.26 243 _dispatch 0x30c13880 pg_info(1 pgs e243:0.5c3) v4 2014-06-09 16:22:11.844814 7f868ff62700 10 osd.26 243 queue_for_recovery queued pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.844820 7f869af78700 5 -- op tracker -- , seq: 3297, time: 2014-06-09 16:22:11.752922, event: header_read, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.844828 7f869af78700 5 -- op tracker -- , seq: 3297, time: 2014-06-09 16:22:11.752923, event: throttled, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.844826 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 3.5( v 214'624 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) 2014-06-09 16:22:11.844834 7f869af78700 5 -- op tracker -- , seq: 3297, time: 2014-06-09 16:22:11.752933, event: all_read, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.844836 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 is up to date, queueing in pending_activators 2014-06-09 16:22:11.844844 7f869af78700 5 -- op tracker -- , seq: 3297, time: 2014-06-09 16:22:11.844811, event: dispatched, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.844852 7f869af78700 5 -- op tracker -- , seq: 3297, time: 2014-06-09 16:22:11.844849, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.844854 7f868df5e700 2 waiting 51 > 50 ops || 121021 > 104857600 2014-06-09 16:22:11.844850 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.3 3.5( v 214'624 (0'0,214'624] local-les=201 n=123 ec=118 les/c 201/201 242/242/242) uptodate 2014-06-09 16:22:11.844861 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.844870 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing de5f0805/rbd_data.117b6b8b4567.00000000000017ec/head//3 214'624 also missing on osd.26 2014-06-09 16:22:11.844856 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5c3) v4 from osd.4 2014-06-09 16:22:11.844881 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c13880 2014-06-09 16:22:11.844879 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {de5f0805/rbd_data.117b6b8b4567.00000000000017ec/head//3=214'624(214'622)} 2014-06-09 16:22:11.844889 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 3 as a source 2014-06-09 16:22:11.844887 7f869af78700 5 -- op tracker -- , seq: 3297, time: 2014-06-09 16:22:11.844885, event: started, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.844897 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing de5f0805/rbd_data.117b6b8b4567.00000000000017ec/head//3 214'624 is on osd.3 2014-06-09 16:22:11.844912 7f869af78700 5 -- op tracker -- , seq: 3297, time: 2014-06-09 16:22:11.844910, event: done, op: pg_info(1 pgs e243:0.5c3) 2014-06-09 16:22:11.844907 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {de5f0805/rbd_data.117b6b8b4567.00000000000017ec/head//3=214'624(214'622)} 2014-06-09 16:22:11.844920 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844922 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844918 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.844935 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.844936 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.844928 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 201 2014-06-09 16:22:11.844938 7f869af78700 20 osd.26 243 _dispatch 0x30d9ee00 pg_info(1 pgs e243:3.1aa) v4 2014-06-09 16:22:11.844943 7f869af78700 5 -- op tracker -- , seq: 3298, time: 2014-06-09 16:22:11.844700, event: header_read, op: pg_info(1 pgs e243:3.1aa) 2014-06-09 16:22:11.844938 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 3 2014-06-09 16:22:11.844948 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.844950 7f869af78700 5 -- op tracker -- , seq: 3298, time: 2014-06-09 16:22:11.844701, event: throttled, op: pg_info(1 pgs e243:3.1aa) 2014-06-09 16:22:11.844959 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.844965 7f869af78700 5 -- op tracker -- , seq: 3298, time: 2014-06-09 16:22:11.844723, event: all_read, op: pg_info(1 pgs e243:3.1aa) 2014-06-09 16:22:11.844971 7f869af78700 5 -- op tracker -- , seq: 3298, time: 2014-06-09 16:22:11.844934, event: dispatched, op: pg_info(1 pgs e243:3.1aa) 2014-06-09 16:22:11.844981 7f869af78700 5 -- op tracker -- , seq: 3298, time: 2014-06-09 16:22:11.844979, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1aa) 2014-06-09 16:22:11.844986 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1aa) v4 from osd.1 2014-06-09 16:22:11.844989 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9ee00 2014-06-09 16:22:11.844993 7f868ff62700 10 write_log with: dirty_to: 0'0, dirty_from: 214'622, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.844994 7f869af78700 5 -- op tracker -- , seq: 3298, time: 2014-06-09 16:22:11.844992, event: started, op: pg_info(1 pgs e243:3.1aa) 2014-06-09 16:22:11.845011 7f869af78700 5 -- op tracker -- , seq: 3298, time: 2014-06-09 16:22:11.845009, event: done, op: pg_info(1 pgs e243:3.1aa) 2014-06-09 16:22:11.845019 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.845021 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.845027 7f868ff62700 5 queue_transactions existing osr(3.5 0x3ed03f0)/0x3ed03f0 2014-06-09 16:22:11.845030 7f868ff62700 2 waiting 51 > 50 ops || 59114 > 104857600 2014-06-09 16:22:11.845031 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.845032 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.845033 7f869af78700 20 osd.26 243 _dispatch 0x30acc540 pg_info(1 pgs e243:1.3c3) v4 2014-06-09 16:22:11.845035 7f86a5610700 5 _do_op 0x2e20ae0 seq 3532 osr(0.8c 0x2bd6480)/0x2bd6480 start 2014-06-09 16:22:11.845038 7f86a5610700 10 _do_transaction on 0xc28e0c0 2014-06-09 16:22:11.845038 7f869af78700 5 -- op tracker -- , seq: 3299, time: 2014-06-09 16:22:11.745535, event: header_read, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845042 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845044 7f869af78700 5 -- op tracker -- , seq: 3299, time: 2014-06-09 16:22:11.745536, event: throttled, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845050 7f869af78700 5 -- op tracker -- , seq: 3299, time: 2014-06-09 16:22:11.745547, event: all_read, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845055 7f869af78700 5 -- op tracker -- , seq: 3299, time: 2014-06-09 16:22:11.845031, event: dispatched, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845062 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845061 7f869af78700 5 -- op tracker -- , seq: 3299, time: 2014-06-09 16:22:11.845059, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845065 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3c3) v4 from osd.8 2014-06-09 16:22:11.845068 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acc540 2014-06-09 16:22:11.845072 7f869af78700 5 -- op tracker -- , seq: 3299, time: 2014-06-09 16:22:11.845070, event: started, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845076 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3531.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845080 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.845086 7f869af78700 5 -- op tracker -- , seq: 3299, time: 2014-06-09 16:22:11.845084, event: done, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845094 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.845096 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.845101 7f86a5e11700 10 _do_op 0xcdaa3c0 seq 3531 r = 0, finisher 0x320736c0 0 2014-06-09 16:22:11.845106 7f86a5e11700 10 _finish_op 0xcdaa3c0 seq 3531 osr(2.75 0x2ed04f0)/0x2ed04f0 2014-06-09 16:22:11.845110 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.845112 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.845113 7f869af78700 20 osd.26 243 _dispatch 0x30bd88c0 pg_info(1 pgs e243:3.58f) v4 2014-06-09 16:22:11.845116 7f868a757700 10 op_submit_start 3581 2014-06-09 16:22:11.845118 7f868a757700 5 queue_transactions (trailing journal) 3581 2014-06-09 16:22:11.845119 7f868a757700 5 queue_op 0x241040c0 seq 3581 osr(3.4b9 0x3ed1460) 1943 bytes (queue has 50 ops and 58300 bytes) 2014-06-09 16:22:11.845118 7f869af78700 5 -- op tracker -- , seq: 3300, time: 2014-06-09 16:22:11.746440, event: header_read, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.845124 7f868cf5c700 2 waiting 51 > 50 ops || 59128 > 104857600 2014-06-09 16:22:11.845125 7f869af78700 5 -- op tracker -- , seq: 3300, time: 2014-06-09 16:22:11.746441, event: throttled, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.845128 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845130 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd0e0 (1 active) 2014-06-09 16:22:11.845131 7f8692f68700 2 waiting 51 > 50 ops || 59126 > 104857600 2014-06-09 16:22:11.845132 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc240 (2 active) 2014-06-09 16:22:11.845132 7f869af78700 5 -- op tracker -- , seq: 3300, time: 2014-06-09 16:22:11.746452, event: all_read, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.845137 7f868a757700 10 op_submit_finish 3581 2014-06-09 16:22:11.845139 7f869af78700 5 -- op tracker -- , seq: 3300, time: 2014-06-09 16:22:11.845110, event: dispatched, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.845147 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x8e4c000 (0 active) 2014-06-09 16:22:11.845146 7f869af78700 5 -- op tracker -- , seq: 3300, time: 2014-06-09 16:22:11.845144, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.845150 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x8680000 (1 active) 2014-06-09 16:22:11.845151 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.58f) v4 from osd.2 2014-06-09 16:22:11.845152 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.845143 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.a2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.a2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.845154 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30bd88c0 2014-06-09 16:22:11.845158 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3532.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845161 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.845159 7f869af78700 5 -- op tracker -- , seq: 3300, time: 2014-06-09 16:22:11.845157, event: started, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.845163 7f8689755700 2 waiting 51 > 50 ops || 59128 > 104857600 2014-06-09 16:22:11.845171 7f868e75f700 2 waiting 51 > 50 ops || 576680 > 104857600 2014-06-09 16:22:11.845172 7f86a5610700 10 _do_op 0x2e20ae0 seq 3532 r = 0, finisher 0x307f1580 0 2014-06-09 16:22:11.845175 7f86a5610700 10 _finish_op 0x2e20ae0 seq 3532 osr(0.8c 0x2bd6480)/0x2bd6480 2014-06-09 16:22:11.845176 7f869af78700 5 -- op tracker -- , seq: 3300, time: 2014-06-09 16:22:11.845173, event: done, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.845184 7f868df5e700 10 op_submit_start 3582 2014-06-09 16:22:11.845184 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.845185 7f868df5e700 5 queue_transactions (trailing journal) 3582 2014-06-09 16:22:11.845187 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.845186 7f868df5e700 5 queue_op 0xc704b40 seq 3582 osr(3.319 0x3ed0d60) 63838 bytes (queue has 50 ops and 121312 bytes) 2014-06-09 16:22:11.845189 7f868a757700 5 queue_transactions existing osr(1.a2 0x384c790)/0x384c790 2014-06-09 16:22:11.845190 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.845191 7f868a757700 2 waiting 51 > 50 ops || 122138 > 104857600 2014-06-09 16:22:11.845192 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ed80 (1 active) 2014-06-09 16:22:11.845194 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359560 (2 active) 2014-06-09 16:22:11.845196 7f868cf5c700 2 waiting 51 > 50 ops || 122140 > 104857600 2014-06-09 16:22:11.845198 7f868df5e700 10 op_submit_finish 3582 2014-06-09 16:22:11.845200 7f868ff62700 2 waiting 51 > 50 ops || 123243 > 104857600 2014-06-09 16:22:11.845202 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.845204 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.845205 7f868ef60700 2 waiting 51 > 50 ops || 122140 > 104857600 2014-06-09 16:22:11.845205 7f869af78700 20 osd.26 243 _dispatch 0x30c136c0 pg_info(1 pgs e243:1.2a3) v4 2014-06-09 16:22:11.845209 7f8692f68700 2 waiting 51 > 50 ops || 122138 > 104857600 2014-06-09 16:22:11.845209 7f86a5e11700 5 _do_op 0x2e213e0 seq 3533 osr(0.521 0x2bd7570)/0x2bd7570 start 2014-06-09 16:22:11.845212 7f86a5e11700 10 _do_transaction on 0xbe89ec0 2014-06-09 16:22:11.845211 7f869af78700 5 -- op tracker -- , seq: 3301, time: 2014-06-09 16:22:11.753697, event: header_read, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.845217 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845219 7f869af78700 5 -- op tracker -- , seq: 3301, time: 2014-06-09 16:22:11.753697, event: throttled, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.845203 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc14aa80 243'608 rep_tid=12 committed?=0 applied?=0 lock=0 op=osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.845226 7f869af78700 5 -- op tracker -- , seq: 3301, time: 2014-06-09 16:22:11.753711, event: all_read, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.845228 7f868df5e700 10 osd.26 243 dequeue_op 0x1c54bb00 finish 2014-06-09 16:22:11.845232 7f869af78700 5 -- op tracker -- , seq: 3301, time: 2014-06-09 16:22:11.845202, event: dispatched, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.845236 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.845240 7f869af78700 5 -- op tracker -- , seq: 3301, time: 2014-06-09 16:22:11.845238, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.845244 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.2a3) v4 from osd.4 2014-06-09 16:22:11.845247 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c136c0 2014-06-09 16:22:11.845250 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845251 7f869af78700 5 -- op tracker -- , seq: 3301, time: 2014-06-09 16:22:11.845249, event: started, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.845254 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8fc00 (20 active) 2014-06-09 16:22:11.845240 7f868b759700 10 osd.26 243 dequeue_op 0xdfa7200 prio 196 cost 0 latency 0.081768 osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) v2 pg pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.845260 7f868b759700 5 -- op tracker -- , seq: 2958, time: 2014-06-09 16:22:11.845256, event: reached_pg, op: osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.845264 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3533.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845257 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.5e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.845268 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.845277 7f86a5e11700 10 _do_op 0x2e213e0 seq 3533 r = 0, finisher 0xcd32280 0 2014-06-09 16:22:11.845269 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0xdfa7200 2014-06-09 16:22:11.845280 7f86a5e11700 10 _finish_op 0x2e213e0 seq 3533 osr(0.521 0x2bd7570)/0x2bd7570 2014-06-09 16:22:11.845285 7f868b759700 5 -- op tracker -- , seq: 2958, time: 2014-06-09 16:22:11.845281, event: started, op: osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.845291 7f868e75f700 10 op_submit_start 3583 2014-06-09 16:22:11.845291 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.845292 7f868e75f700 5 queue_transactions (trailing journal) 3583 2014-06-09 16:22:11.845293 7f868e75f700 5 queue_op 0x2fc8c8a0 seq 3583 osr(3.60 0x3ed02e0) 518380 bytes (queue has 50 ops and 638864 bytes) 2014-06-09 16:22:11.845296 7f868df5e700 5 queue_transactions existing osr(1.5e 0x384c5d0)/0x384c5d0 2014-06-09 16:22:11.845298 7f868df5e700 2 waiting 51 > 50 ops || 639690 > 104857600 2014-06-09 16:22:11.845298 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc240 (1 active) 2014-06-09 16:22:11.845291 7f868b759700 7 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 12 op ack_type 4 from 5 2014-06-09 16:22:11.845300 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114000 (2 active) 2014-06-09 16:22:11.845302 7f86a5610700 5 _do_op 0xcdab0e0 seq 3534 osr(1.38d 0x384d570)/0x384d570 start 2014-06-09 16:22:11.845305 7f868e75f700 10 op_submit_finish 3583 2014-06-09 16:22:11.845306 7f86a5610700 10 _do_transaction on 0x3338180 2014-06-09 16:22:11.845310 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845309 7f868b759700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.845303, event: sub_op_commit_rec, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.845316 7f868b759700 10 osd.26 243 dequeue_op 0xdfa7200 finish 2014-06-09 16:22:11.845323 7f868b759700 5 -- op tracker -- , seq: 2958, time: 2014-06-09 16:22:11.845320, event: done, op: osd_sub_op_reply(client.4569.0:38637 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.845309 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] eval_repop repgather(0xbff66c0 243'645 rep_tid=13 committed?=0 applied?=0 lock=0 op=osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.845333 7f868e75f700 10 osd.26 243 dequeue_op 0x257c5400 finish 2014-06-09 16:22:11.845334 7f868b759700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.845341 7f86a5e11700 5 _do_op 0x2e20fc0 seq 3535 osr(1.631 0x384dfb0)/0x384dfb0 start 2014-06-09 16:22:11.845344 7f86a5e11700 10 _do_transaction on 0xc2dc9c0 2014-06-09 16:22:11.845346 7f868b759700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.845352 7f868e75f700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.845343 7f8690f64700 10 osd.26 243 dequeue_op 0x257c5300 prio 63 cost 364544 latency 1.087551 osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] 2014-06-09 16:22:11.845370 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40e90 (20 active) 2014-06-09 16:22:11.845353 7f868b759700 10 osd.26 243 dequeue_op 0x19fe0700 prio 63 cost 1548288 latency 1.363717 osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.845367 7f8690f64700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.845362, event: reached_pg, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.845371 7f8689755700 2 waiting 51 > 50 ops || 639692 > 104857600 2014-06-09 16:22:11.845376 7f868bf5a700 2 waiting 51 > 50 ops || 640807 > 104857600 2014-06-09 16:22:11.845374 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.3d1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.845379 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.845388 7f868b759700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.845373, event: reached_pg, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.845392 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] handle_message: 0x257c5300 2014-06-09 16:22:11.845402 7f869af78700 5 -- op tracker -- , seq: 3301, time: 2014-06-09 16:22:11.845263, event: done, op: pg_info(1 pgs e243:1.2a3) 2014-06-09 16:22:11.845396 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.845412 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.845412 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.845414 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.845402 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_op osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.845416 7f868e75f700 5 queue_transactions existing osr(2.3d1 0x2ed13b0)/0x2ed13b0 2014-06-09 16:22:11.845409 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x19fe0700 2014-06-09 16:22:11.845419 7f868e75f700 2 waiting 51 > 50 ops || 639692 > 104857600 2014-06-09 16:22:11.845423 7f868cf5c700 2 waiting 51 > 50 ops || 639692 > 104857600 2014-06-09 16:22:11.845426 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.845427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.845427 7f868ff62700 2 waiting 51 > 50 ops || 640795 > 104857600 2014-06-09 16:22:11.845428 7f869af78700 20 osd.26 243 _dispatch 0x190b76c0 pg_info(1 pgs e243:1.3c3) v4 2014-06-09 16:22:11.845431 7f868ef60700 2 waiting 51 > 50 ops || 639692 > 104857600 2014-06-09 16:22:11.845420 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.845435 7f868a757700 2 waiting 51 > 50 ops || 639690 > 104857600 2014-06-09 16:22:11.845426 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] get_object_context: found obc in cache: 0x305c6b00 2014-06-09 16:22:11.845433 7f869af78700 5 -- op tracker -- , seq: 3302, time: 2014-06-09 16:22:11.745846, event: header_read, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845441 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845440 7f869af78700 5 -- op tracker -- , seq: 3302, time: 2014-06-09 16:22:11.745847, event: throttled, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845440 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x30b5e580 2014-06-09 16:22:11.845446 7f869af78700 5 -- op tracker -- , seq: 3302, time: 2014-06-09 16:22:11.745860, event: all_read, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845437 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] get_object_context: 0x305c6b00 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 rwstate(write n=5 w=0) oi: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'645 client.4572.0:30636 wrlock_by=unknown.0.0:0 dirty s 3829760 uv645) ssc: 0x2e8775e0 snapset: 0=[]:[]+head 2014-06-09 16:22:11.845452 7f869af78700 5 -- op tracker -- , seq: 3302, time: 2014-06-09 16:22:11.845425, event: dispatched, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845453 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] find_object_context 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 @head oi=5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3(243'645 client.4572.0:30636 wrlock_by=unknown.0.0:0 dirty s 3829760 uv645) 2014-06-09 16:22:11.845459 7f869af78700 5 -- op tracker -- , seq: 3302, time: 2014-06-09 16:22:11.845457, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845449 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30b5e580 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 rwstate(write n=3 w=0) oi: 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3(243'595 client.4571.0:35949 wrlock_by=unknown.0.0:0 dirty s 1384448 uv595) ssc: 0x2e8769a0 snapset: 0=[]:[]+head 2014-06-09 16:22:11.845463 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3c3) v4 from osd.8 2014-06-09 16:22:11.845466 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b76c0 2014-06-09 16:22:11.845470 7f869af78700 5 -- op tracker -- , seq: 3302, time: 2014-06-09 16:22:11.845469, event: started, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845464 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 @head oi=78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3(243'595 client.4571.0:35949 wrlock_by=unknown.0.0:0 dirty s 1384448 uv595) 2014-06-09 16:22:11.845481 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845483 7f8692f68700 2 waiting 51 > 50 ops || 639690 > 104857600 2014-06-09 16:22:11.845485 7f8690f64700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.845470, event: started, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.845495 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3534.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845499 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.845496 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] execute_ctx 0x30640000 2014-06-09 16:22:11.845506 7f868b759700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.845480, event: started, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.845515 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x304f6c00 2014-06-09 16:22:11.845535 7f868b759700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.845530, event: started, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.845544 7f86a5610700 10 _do_op 0xcdab0e0 seq 3534 r = 0, finisher 0x32073840 0 2014-06-09 16:22:11.845548 7f86a5610700 10 _finish_op 0xcdab0e0 seq 3534 osr(1.38d 0x384d570)/0x384d570 2014-06-09 16:22:11.845541 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] ov 243'595 av 243'596 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.845557 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359560 (1 active) 2014-06-09 16:22:11.845560 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301200 (2 active) 2014-06-09 16:22:11.845562 7f86a5610700 5 _do_op 0xcdab6e0 seq 3536 osr(2.4e0 0x2ed16f0)/0x2ed16f0 start 2014-06-09 16:22:11.845562 7f8689755700 10 op_submit_start 3584 2014-06-09 16:22:11.845564 7f868bf5a700 2 waiting 51 > 50 ops || 640807 > 104857600 2014-06-09 16:22:11.845565 7f86a5610700 10 _do_transaction on 0xc220840 2014-06-09 16:22:11.845565 7f8689755700 5 queue_transactions (trailing journal) 3584 2014-06-09 16:22:11.845556 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 2014-06-09 16:22:11.845567 7f8689755700 5 queue_op 0x324e7080 seq 3584 osr(2.364 0x2ed11b0) 828 bytes (queue has 50 ops and 638864 bytes) 2014-06-09 16:22:11.845570 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845571 7f8689755700 10 op_submit_finish 3584 2014-06-09 16:22:11.845570 7f868e75f700 2 waiting 51 > 50 ops || 639692 > 104857600 2014-06-09 16:22:11.845576 7f868cf5c700 2 waiting 51 > 50 ops || 639692 > 104857600 2014-06-09 16:22:11.845569 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.845581 7f868ff62700 2 waiting 51 > 50 ops || 640795 > 104857600 2014-06-09 16:22:11.845585 7f868ef60700 2 waiting 51 > 50 ops || 639692 > 104857600 2014-06-09 16:22:11.845575 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.450( v 213'620 (0'0,213'620] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 9 2014-06-09 16:22:11.845589 7f868a757700 2 waiting 51 > 50 ops || 639690 > 104857600 2014-06-09 16:22:11.845594 7f8692f68700 2 waiting 51 > 50 ops || 639690 > 104857600 2014-06-09 16:22:11.845594 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845600 7f868df5e700 2 waiting 51 > 50 ops || 639690 > 104857600 2014-06-09 16:22:11.845590 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.450( v 213'620 (0'0,213'620] local-les=213 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info+log from osd.9 3.450( v 221'627 (0'0,221'627] local-les=243 n=121 ec=118 les/c 213/213 242/242/212) log((213'620,221'627], crt=0'0) 2014-06-09 16:22:11.845591 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 1384448~1548288 2014-06-09 16:22:11.845603 7f8689755700 10 merge_log log((213'620,221'627], crt=0'0) from osd.9 into log((0'0,213'620], crt=0'0) 2014-06-09 16:22:11.845607 7f8689755700 10 merge_log extending head to 221'627 2014-06-09 16:22:11.845608 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3535.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845613 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.845608 7f8689755700 20 ? 221'627 (219'626) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32637 2014-06-09 10:20:59.348214 2014-06-09 16:22:11.845612 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228e00 2014-06-09 16:22:11.845618 7f8689755700 20 ? 219'626 (219'625) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32636 2014-06-09 10:20:57.075463 2014-06-09 16:22:11.845624 7f86a5e11700 10 _do_op 0x2e20fc0 seq 3535 r = 0, finisher 0x314adc80 0 2014-06-09 16:22:11.845627 7f86a5e11700 10 _finish_op 0x2e20fc0 seq 3535 osr(1.631 0x384dfb0)/0x384dfb0 2014-06-09 16:22:11.845632 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114000 (1 active) 2014-06-09 16:22:11.845625 7f8689755700 20 ? 219'625 (218'624) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32635 2014-06-09 10:20:56.670155 2014-06-09 16:22:11.845624 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 snapset=0x2e8769d8 snapc=0=[] 2014-06-09 16:22:11.845635 7f868bf5a700 10 op_submit_start 3585 2014-06-09 16:22:11.845637 7f868bf5a700 5 queue_transactions (trailing journal) 3585 2014-06-09 16:22:11.845637 7f868e75f700 2 waiting 51 > 50 ops || 640807 > 104857600 2014-06-09 16:22:11.845638 7f868bf5a700 5 queue_op 0x30032360 seq 3585 osr(3.428 0x3ed11e0) 1943 bytes (queue has 50 ops and 639979 bytes) 2014-06-09 16:22:11.845636 7f8689755700 20 ? 218'624 (216'623) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32634 2014-06-09 10:20:55.777312 2014-06-09 16:22:11.845642 7f868cf5c700 2 waiting 51 > 50 ops || 640807 > 104857600 2014-06-09 16:22:11.845642 7f8689755700 20 ? 216'623 (215'622) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32633 2014-06-09 10:20:53.055086 2014-06-09 16:22:11.845646 7f868ff62700 2 waiting 51 > 50 ops || 641910 > 104857600 2014-06-09 16:22:11.845638 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.845650 7f868ef60700 2 waiting 51 > 50 ops || 640807 > 104857600 2014-06-09 16:22:11.845647 7f8689755700 20 ? 215'622 (215'621) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32632 2014-06-09 10:20:51.902084 2014-06-09 16:22:11.845652 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845654 7f868a757700 2 waiting 51 > 50 ops || 640805 > 104857600 2014-06-09 16:22:11.845652 7f8689755700 20 ? 215'621 (0'0) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32631 2014-06-09 10:20:51.897531 2014-06-09 16:22:11.845649 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 0x304f6c00 op modify 2014-06-09 16:22:11.845659 7f8692f68700 2 waiting 51 > 50 ops || 640805 > 104857600 2014-06-09 16:22:11.845658 7f8689755700 20 merge_log 215'621 (0'0) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32631 2014-06-09 10:20:51.897531 2014-06-09 16:22:11.845658 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3398c0 (2 active) 2014-06-09 16:22:11.845668 7f868bf5a700 10 op_submit_finish 3585 2014-06-09 16:22:11.845667 7f868df5e700 2 waiting 51 > 50 ops || 640805 > 104857600 2014-06-09 16:22:11.845665 7f8689755700 20 merge_log 215'622 (215'621) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32632 2014-06-09 10:20:51.902084 2014-06-09 16:22:11.845673 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x8680000 (0 active) 2014-06-09 16:22:11.845676 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x3e0e000 (1 active) 2014-06-09 16:22:11.845678 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.845673 7f8689755700 20 merge_log 216'623 (215'622) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32633 2014-06-09 10:20:53.055086 2014-06-09 16:22:11.845673 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.37a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.845666 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:42.788751 2014-06-09 16:22:11.845683 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3536.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845686 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.845684 7f8689755700 20 merge_log 218'624 (216'623) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32634 2014-06-09 10:20:55.777312 2014-06-09 16:22:11.845692 7f8689755700 20 merge_log 219'625 (218'624) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32635 2014-06-09 10:20:56.670155 2014-06-09 16:22:11.845696 7f86a5610700 10 _do_op 0xcdab6e0 seq 3536 r = 0, finisher 0x281b6a00 0 2014-06-09 16:22:11.845699 7f86a5610700 10 _finish_op 0xcdab6e0 seq 3536 osr(2.4e0 0x2ed16f0)/0x2ed16f0 2014-06-09 16:22:11.845699 7f8689755700 20 merge_log 219'626 (219'625) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32636 2014-06-09 10:20:57.075463 2014-06-09 16:22:11.845704 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301200 (1 active) 2014-06-09 16:22:11.845705 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.845694 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.845706 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781c20 (2 active) 2014-06-09 16:22:11.845708 7f86a5610700 5 _do_op 0x2e209c0 seq 3538 osr(0.1f9 0x2bd6880)/0x2bd6880 start 2014-06-09 16:22:11.845710 7f868bf5a700 5 queue_transactions existing osr(2.37a 0x2ed1160)/0x2ed1160 2014-06-09 16:22:11.845710 7f86a5610700 10 _do_transaction on 0xc1fd980 2014-06-09 16:22:11.845707 7f8689755700 20 merge_log 221'627 (219'626) modify 4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3 by client.4575.0:32637 2014-06-09 10:20:59.348214 2014-06-09 16:22:11.845712 7f868bf5a700 10 op_submit_start 3586 2014-06-09 16:22:11.845713 7f868bf5a700 5 queue_transactions (trailing journal) 3586 2014-06-09 16:22:11.845714 7f868bf5a700 5 queue_op 0xc64aa80 seq 3586 osr(2.37a 0x2ed1160) 828 bytes (queue has 50 ops and 639979 bytes) 2014-06-09 16:22:11.845717 7f868bf5a700 10 op_submit_finish 3586 2014-06-09 16:22:11.845716 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845714 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.845719 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.845726 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 14 on osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.845734 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.845748 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845744 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.845744 7f868b759700 7 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 14 o 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.845752 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.845765 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.845764 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3538.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845768 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.845767 7f869af78700 5 -- op tracker -- , seq: 3302, time: 2014-06-09 16:22:11.845485, event: done, op: pg_info(1 pgs e243:1.3c3) 2014-06-09 16:22:11.845780 7f86a5610700 10 _do_op 0x2e209c0 seq 3538 r = 0, finisher 0x3206e980 0 2014-06-09 16:22:11.845783 7f86a5610700 10 _finish_op 0x2e209c0 seq 3538 osr(0.1f9 0x2bd6880)/0x2bd6880 2014-06-09 16:22:11.845782 7f8690f64700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.845510, event: started, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.845786 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781c20 (1 active) 2014-06-09 16:22:11.845788 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd200 (2 active) 2014-06-09 16:22:11.845781 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:991 2014-06-09 16:22:11.845790 7f86a5610700 5 _do_op 0xcdaa480 seq 3539 osr(3.792 0x3ed1fc0)/0x3ed1fc0 start 2014-06-09 16:22:11.845792 7f86a5610700 10 _do_transaction on 0x34f09c0 2014-06-09 16:22:11.845798 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.845797 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845790 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] ov 243'645 av 243'646 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.845802 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.845805 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 2014-06-09 16:22:11.845814 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.845817 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.845828 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845823 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.845831 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.845830 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op write 3829760~364544 2014-06-09 16:22:11.845844 7f868bf5a700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.845843 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3539.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845847 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.845846 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] do_osd_op_effects on session 0xd229400 2014-06-09 16:22:11.845857 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 snapset=0x2e877618 snapc=0=[] 2014-06-09 16:22:11.845858 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.845867 7f86a5610700 10 _do_op 0xcdaa480 seq 3539 r = 0, finisher 0x3206f400 0 2014-06-09 16:22:11.845870 7f86a5610700 10 _finish_op 0xcdaa480 seq 3539 osr(3.792 0x3ed1fc0)/0x3ed1fc0 2014-06-09 16:22:11.845872 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd200 (1 active) 2014-06-09 16:22:11.845874 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1200 (2 active) 2014-06-09 16:22:11.845875 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.845876 7f86a5610700 5 _do_op 0x2e20b40 seq 3540 osr(3.712 0x3ed1b90)/0x3ed1b90 start 2014-06-09 16:22:11.845878 7f86a5610700 10 _do_transaction on 0xc317bc0 2014-06-09 16:22:11.845869 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] make_writeable 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.845884 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845879 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) 2014-06-09 16:22:11.845880 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] finish_ctx 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 0x30640000 op modify 2014-06-09 16:22:11.845890 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.7 activated and committed 2014-06-09 16:22:11.845893 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] set mtime to 2014-06-09 10:20:27.269831 2014-06-09 16:22:11.845899 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.845905 7f8689755700 10 merge_log result log((0'0,221'627], crt=221'627) missing(1) changed=1 2014-06-09 16:22:11.845907 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.845915 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845910 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.450( v 221'627 lc 213'620 (0'0,221'627] local-les=213 n=121 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=221'627 lcod 0'0 inactive NOTIFY m=1] exit Started/Stray 1.001266 8 0.000677 2014-06-09 16:22:11.845919 7f868bf5a700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.845931 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3540.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.845927 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.450( v 221'627 lc 213'620 (0'0,221'627] local-les=213 n=121 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=221'627 lcod 0'0 inactive NOTIFY m=1] enter Started/ReplicaActive 2014-06-09 16:22:11.845935 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.845924 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] final snapset 0=[]:[]+head in 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.845934 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:143 2014-06-09 16:22:11.845939 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.450( v 221'627 lc 213'620 (0'0,221'627] local-les=213 n=121 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=221'627 lcod 0'0 inactive NOTIFY m=1] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.845948 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.845949 7f86a5610700 10 _do_op 0x2e20b40 seq 3540 r = 0, finisher 0x32072480 0 2014-06-09 16:22:11.845953 7f86a5610700 10 _finish_op 0x2e20b40 seq 3540 osr(3.712 0x3ed1b90)/0x3ed1b90 2014-06-09 16:22:11.845944 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] zeroing write result code 0 2014-06-09 16:22:11.845955 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1200 (1 active) 2014-06-09 16:22:11.845957 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77aa20 (2 active) 2014-06-09 16:22:11.845950 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.450( v 221'627 lc 213'620 (0'0,221'627] local-les=213 n=121 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=221'627 lcod 0'0 inactive NOTIFY m=1] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.845959 7f86a5610700 5 _do_op 0x2e21440 seq 3541 osr(2.778 0x3ed0050)/0x3ed0050 start 2014-06-09 16:22:11.845952 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.845961 7f86a5610700 10 _do_transaction on 0xc1f7a40 2014-06-09 16:22:11.845966 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.845961 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.450( v 221'627 lc 213'620 (0'0,221'627] local-les=243 n=121 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=221'627 lcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.845956 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] new_repop rep_tid 15 on osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.845964 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.845972 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.845973 7f8690f64700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] issue_repop rep_tid 15 o 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.845980 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.845991 7f868bf5a700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.845998 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.845996 7f8690f64700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.845990, event: waiting for subops from 7, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.846006 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.846012 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3541.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.846015 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.846017 7f8690f64700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.846019 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846024 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.846035 7f86a5610700 10 _do_op 0x2e21440 seq 3541 r = 0, finisher 0xd9fbbe0 0 2014-06-09 16:22:11.846038 7f86a5610700 10 _finish_op 0x2e21440 seq 3541 osr(2.778 0x3ed0050)/0x3ed0050 2014-06-09 16:22:11.846041 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77aa20 (1 active) 2014-06-09 16:22:11.846035 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.846042 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb320 (2 active) 2014-06-09 16:22:11.846045 7f86a5610700 5 _do_op 0x2e20a20 seq 3542 osr(3.62a 0x3ed1aa0)/0x3ed1aa0 start 2014-06-09 16:22:11.846047 7f86a5610700 10 _do_transaction on 0xc3f1080 2014-06-09 16:22:11.846044 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846051 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.846033 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'645 (0'0,243'645] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] append_log log((0'0,243'645], crt=0'0) [243'646 (243'645) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30637 2014-06-09 10:20:27.269831] 2014-06-09 16:22:11.846050 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846062 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.846060 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] add_log_entry 243'646 (243'645) modify 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 by client.4572.0:30637 2014-06-09 10:20:27.269831 2014-06-09 16:22:11.846083 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.846076 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.846077 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.450( v 221'627 lc 219'626 (0'0,221'627] local-les=243 n=121 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=221'627 lcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.846083 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] append_log adding 1 keys 2014-06-09 16:22:11.846093 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846096 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3542.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.846100 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.846097 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.6cd( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.846109 7f86a5610700 10 _do_op 0x2e20a20 seq 3542 r = 0, finisher 0xd9fb3a0 0 2014-06-09 16:22:11.846112 7f86a5610700 10 _finish_op 0x2e20a20 seq 3542 osr(3.62a 0x3ed1aa0)/0x3ed1aa0 2014-06-09 16:22:11.846115 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb320 (1 active) 2014-06-09 16:22:11.846117 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f440 (2 active) 2014-06-09 16:22:11.846119 7f86a5610700 5 _do_op 0x2e207e0 seq 3543 osr(3.4fa 0x3ed1720)/0x3ed1720 start 2014-06-09 16:22:11.846121 7f86a5610700 10 _do_transaction on 0xc142240 2014-06-09 16:22:11.846120 7f8690f64700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'646, trimmed: 2014-06-09 16:22:11.846125 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.846125 7f8689755700 10 write_log with: dirty_to: 0'0, dirty_from: 213'620, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.846136 7f8690f64700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.846138 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846139 7f8690f64700 10 op_submit_start 3587 2014-06-09 16:22:11.846140 7f8690f64700 5 queue_transactions (trailing journal) 3587 2014-06-09 16:22:11.846141 7f8690f64700 5 queue_op 0x2fcaf560 seq 3587 osr(3.60 0x3ed02e0) 366828 bytes (queue has 46 ops and 999197 bytes) 2014-06-09 16:22:11.846144 7f8690f64700 10 op_submit_finish 3587 2014-06-09 16:22:11.846143 7f868bf5a700 5 queue_transactions existing osr(2.6cd 0x2ed1e70)/0x2ed1e70 2014-06-09 16:22:11.846145 7f868bf5a700 10 op_submit_start 3588 2014-06-09 16:22:11.846147 7f868bf5a700 5 queue_transactions (trailing journal) 3588 2014-06-09 16:22:11.846148 7f868bf5a700 5 queue_op 0x324e71a0 seq 3588 osr(2.6cd 0x2ed1e70) 828 bytes (queue has 47 ops and 1000025 bytes) 2014-06-09 16:22:11.846151 7f868bf5a700 10 op_submit_finish 3588 2014-06-09 16:22:11.846157 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.846145 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] eval_repop repgather(0xc2c6240 243'646 rep_tid=15 committed?=0 applied?=0 lock=0 op=osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.846162 7f8690f64700 10 osd.26 243 dequeue_op 0x257c5300 finish 2014-06-09 16:22:11.846154 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.846169 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.846169 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3543.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.846173 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.846167 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.846174 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.846177 7f8689755700 5 queue_transactions existing osr(3.450 0x3ed1160)/0x3ed1160 2014-06-09 16:22:11.846180 7f8689755700 10 op_submit_start 3589 2014-06-09 16:22:11.846181 7f8689755700 5 queue_transactions (trailing journal) 3589 2014-06-09 16:22:11.846175 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846182 7f86a5610700 10 _do_op 0x2e207e0 seq 3543 r = 0, finisher 0xd9fb1a0 0 2014-06-09 16:22:11.846182 7f8689755700 5 queue_op 0xbe7c540 seq 3589 osr(3.450 0x3ed1160) 3400 bytes (queue has 48 ops and 1003425 bytes) 2014-06-09 16:22:11.846185 7f8689755700 10 op_submit_finish 3589 2014-06-09 16:22:11.846185 7f86a5610700 10 _finish_op 0x2e207e0 seq 3543 osr(3.4fa 0x3ed1720)/0x3ed1720 2014-06-09 16:22:11.846182 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846188 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f440 (1 active) 2014-06-09 16:22:11.846190 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77bb00 (2 active) 2014-06-09 16:22:11.846178 7f8690f64700 10 osd.26 243 dequeue_op 0x10c55c00 prio 63 cost 516096 latency 1.706661 osd_op(client.4573.0:33492 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 311296~516096] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.846193 7f86a5610700 5 _do_op 0x230b8660 seq 3544 osr(0.4bd 0x2bd76f0)/0x2bd76f0 start 2014-06-09 16:22:11.846194 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.846195 7f86a5610700 10 _do_transaction on 0xc33a480 2014-06-09 16:22:11.846188 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.846198 7f8690f64700 5 -- op tracker -- , seq: 116, time: 2014-06-09 16:22:11.846193, event: reached_pg, op: osd_op(client.4573.0:33492 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 311296~516096] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.846201 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.846199 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: got info+log from osd.8 0.4c7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.846209 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.8 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.846212 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.846206 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:192 2014-06-09 16:22:11.846205 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.846221 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846215 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] exit Started/Stray 0.863617 7 0.000179 2014-06-09 16:22:11.846217 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x10c55c00 2014-06-09 16:22:11.846224 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.846233 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.846225 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.846226 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:33492 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 311296~516096] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.846233 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.846238 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.846242 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.846249 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3544.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.846242 7f8690f64700 3 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4573.0:33492 was 214'578 2014-06-09 16:22:11.846252 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.846247 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846250 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.846255 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846257 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.846267 7f8690f64700 10 osd.26 243 dequeue_op 0x10c55c00 finish 2014-06-09 16:22:11.846266 7f868bf5a700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.846270 7f86a5610700 10 _do_op 0x230b8660 seq 3544 r = 0, finisher 0x3006cee0 0 2014-06-09 16:22:11.846273 7f86a5610700 10 _finish_op 0x230b8660 seq 3544 osr(0.4bd 0x2bd76f0)/0x2bd76f0 2014-06-09 16:22:11.846276 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77bb00 (1 active) 2014-06-09 16:22:11.846274 7f8690f64700 5 -- op tracker -- , seq: 116, time: 2014-06-09 16:22:11.846270, event: done, op: osd_op(client.4573.0:33492 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 311296~516096] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.846278 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff59e0 (2 active) 2014-06-09 16:22:11.846280 7f86a5610700 5 _do_op 0x16097da0 seq 3545 osr(1.604 0x384dc50)/0x384dc50 start 2014-06-09 16:22:11.846282 7f86a5610700 10 _do_transaction on 0xbf92e40 2014-06-09 16:22:11.846287 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.846281 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:152 2014-06-09 16:22:11.846294 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846295 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.846303 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.846298 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.676( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.846307 7f8689755700 5 queue_transactions existing osr(0.4c7 0x2bd76b0)/0x2bd76b0 2014-06-09 16:22:11.846309 7f8689755700 10 op_submit_start 3590 2014-06-09 16:22:11.846310 7f8689755700 5 queue_transactions (trailing journal) 3590 2014-06-09 16:22:11.846311 7f8689755700 5 queue_op 0xbe7dc20 seq 3590 osr(0.4c7 0x2bd76b0) 1876 bytes (queue has 47 ops and 1002472 bytes) 2014-06-09 16:22:11.846313 7f8689755700 10 op_submit_finish 3590 2014-06-09 16:22:11.846319 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.846319 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358340 (20 active) 2014-06-09 16:22:11.846316 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.846329 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846331 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3545.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.846323 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.65( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.846335 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.846334 7f868bf5a700 5 queue_transactions existing osr(1.676 0x384dee0)/0x384dee0 2014-06-09 16:22:11.846329 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.846337 7f868bf5a700 10 op_submit_start 3591 2014-06-09 16:22:11.846338 7f868bf5a700 5 queue_transactions (trailing journal) 3591 2014-06-09 16:22:11.846339 7f868bf5a700 5 queue_op 0x324e73e0 seq 3591 osr(1.676 0x384dee0) 828 bytes (queue has 48 ops and 1003300 bytes) 2014-06-09 16:22:11.846342 7f868bf5a700 10 op_submit_finish 3591 2014-06-09 16:22:11.846337 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846344 7f86a5610700 10 _do_op 0x16097da0 seq 3545 r = 0, finisher 0x3006cae0 0 2014-06-09 16:22:11.846347 7f86a5610700 10 _finish_op 0x16097da0 seq 3545 osr(1.604 0x384dc50)/0x384dc50 2014-06-09 16:22:11.846349 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff59e0 (1 active) 2014-06-09 16:22:11.846344 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846351 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358120 (2 active) 2014-06-09 16:22:11.846353 7f86a5610700 5 _do_op 0x30032c00 seq 3546 osr(3.31c 0x3ed0d40)/0x3ed0d40 start 2014-06-09 16:22:11.846355 7f86a5610700 10 _do_transaction on 0x3a06d80 2014-06-09 16:22:11.846345 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.846356 7f8689755700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.846360 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.846365 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.846359 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.846370 7f8690f64700 5 queue_transactions existing osr(2.65 0x2ed0570)/0x2ed0570 2014-06-09 16:22:11.846373 7f8690f64700 10 op_submit_start 3592 2014-06-09 16:22:11.846374 7f8690f64700 5 queue_transactions (trailing journal) 3592 2014-06-09 16:22:11.846369 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.846375 7f8690f64700 5 queue_op 0x2fcaf500 seq 3592 osr(2.65 0x2ed0570) 826 bytes (queue has 48 ops and 1003298 bytes) 2014-06-09 16:22:11.846378 7f8690f64700 10 op_submit_finish 3592 2014-06-09 16:22:11.846367 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846383 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.846383 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846381 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.846392 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.846395 7f868bf5a700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.846387 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.846396 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.846399 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.846405 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3546.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.846409 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.846406 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846408 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846407 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:147 2014-06-09 16:22:11.846420 7f86a5610700 10 _do_op 0x30032c00 seq 3546 r = 0, finisher 0x3006c4e0 0 2014-06-09 16:22:11.846421 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846415 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846415 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846423 7f86a5610700 10 _finish_op 0x30032c00 seq 3546 osr(3.31c 0x3ed0d40)/0x3ed0d40 2014-06-09 16:22:11.846425 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358120 (1 active) 2014-06-09 16:22:11.846426 7f8689755700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.846427 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922000 (2 active) 2014-06-09 16:22:11.846429 7f8690f64700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.846430 7f86a5610700 5 _do_op 0x2c06e2a0 seq 3547 osr(1.774 0x2ed0390)/0x2ed0390 start 2014-06-09 16:22:11.846433 7f86a5610700 10 _do_transaction on 0xbebc480 2014-06-09 16:22:11.846425 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.56( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.846438 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.846502 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.846519 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3547.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.846523 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.846521 7f868b759700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.845763, event: waiting for subops from 9, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.846541 7f868b759700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.846542 7f86a5610700 10 _do_op 0x2c06e2a0 seq 3547 r = 0, finisher 0x32072ac0 0 2014-06-09 16:22:11.846546 7f86a5610700 10 _finish_op 0x2c06e2a0 seq 3547 osr(1.774 0x2ed0390)/0x2ed0390 2014-06-09 16:22:11.846548 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922000 (1 active) 2014-06-09 16:22:11.846550 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4066c0 (2 active) 2014-06-09 16:22:11.846552 7f86a5610700 5 _do_op 0x11fce2a0 seq 3548 osr(3.9b 0x3ed0630)/0x3ed0630 start 2014-06-09 16:22:11.846443 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1061 2014-06-09 16:22:11.846462 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846545 7f868e75f700 10 op_submit_start 3593 2014-06-09 16:22:11.846567 7f868bf5a700 5 queue_transactions existing osr(1.56 0x384c5f0)/0x384c5f0 2014-06-09 16:22:11.846569 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.846569 7f868e75f700 5 queue_transactions (trailing journal) 3593 2014-06-09 16:22:11.846573 7f86a5610700 10 _do_transaction on 0xbebd080 2014-06-09 16:22:11.846572 7f868e75f700 5 queue_op 0x2fc8c9c0 seq 3593 osr(2.3d1 0x2ed13b0) 828 bytes (queue has 48 ops and 1002241 bytes) 2014-06-09 16:22:11.846575 7f868e75f700 10 op_submit_finish 3593 2014-06-09 16:22:11.846579 7f868bf5a700 10 op_submit_start 3594 2014-06-09 16:22:11.846580 7f868bf5a700 5 queue_transactions (trailing journal) 3594 2014-06-09 16:22:11.846565 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'595 (0'0,243'595] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'595], crt=0'0) [243'596 (243'595) modify 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 by client.4571.0:35950 2014-06-09 10:19:42.788751] 2014-06-09 16:22:11.846581 7f868bf5a700 5 queue_op 0x324e7560 seq 3594 osr(1.56 0x384c5f0) 826 bytes (queue has 48 ops and 1002241 bytes) 2014-06-09 16:22:11.846573 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.846583 7f868bf5a700 10 op_submit_finish 3594 2014-06-09 16:22:11.846580 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.846586 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.846586 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:11.846587 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'596 (243'595) modify 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 by client.4571.0:35950 2014-06-09 10:19:42.788751 2014-06-09 16:22:11.846594 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846602 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846593 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.846613 7f8690f64700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.846607 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.846610 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846620 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846618 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846624 7f868bf5a700 5 queue_transactions existing osr(3.7e2 0xb7f0120)/0xb7f0120 2014-06-09 16:22:11.846626 7f868bf5a700 10 op_submit_start 3595 2014-06-09 16:22:11.846627 7f868bf5a700 5 queue_transactions (trailing journal) 3595 2014-06-09 16:22:11.846628 7f868bf5a700 5 queue_op 0x324e7620 seq 3595 osr(3.7e2 0xb7f0120) 828 bytes (queue has 49 ops and 1003069 bytes) 2014-06-09 16:22:11.846630 7f868bf5a700 10 op_submit_finish 3595 2014-06-09 16:22:11.846629 7f868e75f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.846629 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.846633 7f868bf5a700 20 osd.26 243 Sending info 3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) to shard 6 2014-06-09 16:22:11.846644 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.846645 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.846647 7f868b759700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'596, trimmed: 2014-06-09 16:22:11.846644 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:184 2014-06-09 16:22:11.846648 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.846662 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.846665 7f868b759700 5 queue_transactions existing osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.846660 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.846667 7f868b759700 10 op_submit_start 3596 2014-06-09 16:22:11.846669 7f868b759700 5 queue_transactions (trailing journal) 3596 2014-06-09 16:22:11.846670 7f868b759700 5 queue_op 0x2e20de0 seq 3596 osr(3.1e 0x3ed0380) 1550572 bytes (queue has 50 ops and 2553641 bytes) 2014-06-09 16:22:11.846672 7f868b759700 10 op_submit_finish 3596 2014-06-09 16:22:11.846672 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41980 (19 active) 2014-06-09 16:22:11.846668 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846666 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.846679 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.846675 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846678 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.846686 7f8690f64700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.846674 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0x3128000 243'596 rep_tid=14 committed?=0 applied?=0 lock=0 op=osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.846686 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846692 7f868b759700 10 osd.26 243 dequeue_op 0x19fe0700 finish 2014-06-09 16:22:11.846695 7f868b759700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.846694 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846683 7f868bf5a700 10 osd.26 243 dequeue_op 0x32826100 prio 63 cost 4195304 latency 0.054652 MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 pg pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] 2014-06-09 16:22:11.846705 7f868e75f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.846713 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f3fc0 (20 active) 2014-06-09 16:22:11.846712 7f868bf5a700 5 -- op tracker -- , seq: 3239, time: 2014-06-09 16:22:11.846704, event: reached_pg, op: MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:11.846699 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.846715 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.846716 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.37a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.37a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.846725 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.846729 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.846720 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] 1 flushes_in_progress pending waiting for active on 0x32826100 2014-06-09 16:22:11.846731 7f868bf5a700 10 osd.26 243 dequeue_op 0x32826100 finish 2014-06-09 16:22:11.846734 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.846729 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.589( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.846739 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.846732 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.6d2( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.846746 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.846736 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.846751 7f868b759700 5 queue_transactions existing osr(2.37a 0x2ed1160)/0x2ed1160 2014-06-09 16:22:11.846753 7f868b759700 2 waiting 51 > 50 ops || 2554469 > 104857600 2014-06-09 16:22:11.846757 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.846743 7f868bf5a700 10 osd.26 243 dequeue_op 0x257c4200 prio 63 cost 139264 latency 0.125799 osd_op(client.4538.0:36626 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 4055040~139264] 3.923772b RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.846762 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.846761 7f8690f64700 5 queue_transactions existing osr(1.589 0x384da10)/0x384da10 2014-06-09 16:22:11.846764 7f8690f64700 2 waiting 51 > 50 ops || 2554469 > 104857600 2014-06-09 16:22:11.846754 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] exit Started/Primary/Active/Activating 0.779253 6 0.000414 2014-06-09 16:22:11.846765 7f868bf5a700 5 -- op tracker -- , seq: 2750, time: 2014-06-09 16:22:11.846760, event: reached_pg, op: osd_op(client.4538.0:36626 rbd_data.115a6b8b4567.0000000000001a29 [set-alloc-hint object_size 4194304 write_size 4194304,write 4055040~139264] 3.923772b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.846769 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.846777 7f868e75f700 5 queue_transactions existing osr(0.6d2 0x384c1f0)/0x384c1f0 2014-06-09 16:22:11.846780 7f868e75f700 2 waiting 51 > 50 ops || 2554469 > 104857600 2014-06-09 16:22:11.846773 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.846442 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.846785 7f868cf5c700 2 waiting 51 > 50 ops || 2554469 > 104857600 2014-06-09 16:22:11.846781 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] needs_recovery is recovered 2014-06-09 16:22:11.846792 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.846785 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x257c4200 2014-06-09 16:22:11.846794 7f868bf5a700 10 osd.26 243 dequeue_op 0x257c4200 finish 2014-06-09 16:22:11.846798 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.846794 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] exit Started/Primary/Active/Recovered 0.000026 0 0.000000 2014-06-09 16:22:11.846797 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.846809 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.846812 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.846806 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.846814 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) 2014-06-09 16:22:11.846808 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 1.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.846820 7f868ff62700 2 waiting 51 > 50 ops || 2555572 > 104857600 2014-06-09 16:22:11.846820 7f8689755700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.846823 7f8689755700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.846817 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] finish_recovery 2014-06-09 16:22:11.846825 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.846827 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.846828 7f868ef60700 2 waiting 51 > 50 ops || 2554469 > 104857600 2014-06-09 16:22:11.846818 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:11.846829 7f869af78700 20 osd.26 243 _dispatch 0x30c90000 pg_info(1 pgs e243:3.58f) v4 2014-06-09 16:22:11.846832 7f868a757700 2 waiting 51 > 50 ops || 2554467 > 104857600 2014-06-09 16:22:11.846827 7f8689755700 5 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.932067 7 0.000171 2014-06-09 16:22:11.846827 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active] clear_recovery_state 2014-06-09 16:22:11.846838 7f86a5e11700 5 _do_op 0xcdaa1e0 seq 3537 osr(1.a2 0x384c790)/0x384c790 start 2014-06-09 16:22:11.846835 7f869af78700 5 -- op tracker -- , seq: 3303, time: 2014-06-09 16:22:11.746461, event: header_read, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.846840 7f86a5e11700 10 _do_transaction on 0x3472840 2014-06-09 16:22:11.846834 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.846843 7f869af78700 5 -- op tracker -- , seq: 3303, time: 2014-06-09 16:22:11.746461, event: throttled, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.846846 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.846840 7f8689755700 5 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.846845 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.846849 7f869af78700 5 -- op tracker -- , seq: 3303, time: 2014-06-09 16:22:11.746476, event: all_read, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.846841 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.846851 7f8689755700 5 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.846857 7f869af78700 5 -- op tracker -- , seq: 3303, time: 2014-06-09 16:22:11.846824, event: dispatched, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.846854 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.846864 7f869af78700 5 -- op tracker -- , seq: 3303, time: 2014-06-09 16:22:11.846862, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.846856 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.846867 7f868bf5a700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.846861 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.846868 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.58f) v4 from osd.2 2014-06-09 16:22:11.846872 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c90000 2014-06-09 16:22:11.846868 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] share_pg_info 2014-06-09 16:22:11.846871 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.846882 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.846883 7f8691f66700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.846879 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.846883 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1029 2014-06-09 16:22:11.846899 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846899 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] publish_stats_to_osd 243:1020 2014-06-09 16:22:11.846904 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.3b6( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.846877 7f869af78700 5 -- op tracker -- , seq: 3303, time: 2014-06-09 16:22:11.846875, event: started, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.846925 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.846927 7f869af78700 5 -- op tracker -- , seq: 3303, time: 2014-06-09 16:22:11.846925, event: done, op: pg_info(1 pgs e243:3.58f) 2014-06-09 16:22:11.846931 7f8689755700 5 queue_transactions existing osr(1.1fb 0x384cd30)/0x384cd30 2014-06-09 16:22:11.846933 7f8689755700 2 waiting 51 > 50 ops || 2555524 > 104857600 2014-06-09 16:22:11.846935 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.846935 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.846937 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.846937 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.846940 7f8691f66700 5 queue_transactions existing osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.846941 7f868bf5a700 5 queue_transactions existing osr(0.3b6 0x2bd72d0)/0x2bd72d0 2014-06-09 16:22:11.846942 7f8691f66700 2 waiting 51 > 50 ops || 2554499 > 104857600 2014-06-09 16:22:11.846940 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3537.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.846945 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.846944 7f868bf5a700 2 waiting 51 > 50 ops || 2554469 > 104857600 2014-06-09 16:22:11.846948 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.846950 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.846951 7f869af78700 20 osd.26 243 _dispatch 0x30c13500 pg_info(1 pgs e243:2.40c) v4 2014-06-09 16:22:11.846957 7f86a5e11700 10 _do_op 0xcdaa1e0 seq 3537 r = 0, finisher 0x281b6d80 0 2014-06-09 16:22:11.846960 7f86a5e11700 10 _finish_op 0xcdaa1e0 seq 3537 osr(1.a2 0x384c790)/0x384c790 2014-06-09 16:22:11.846955 7f869af78700 5 -- op tracker -- , seq: 3304, time: 2014-06-09 16:22:11.753727, event: header_read, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.846968 7f869af78700 5 -- op tracker -- , seq: 3304, time: 2014-06-09 16:22:11.753727, event: throttled, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.846979 7f868df5e700 2 waiting 51 > 50 ops || 2554469 > 104857600 2014-06-09 16:22:11.846979 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3398c0 (1 active) 2014-06-09 16:22:11.846981 7f868b759700 10 op_submit_start 3597 2014-06-09 16:22:11.846982 7f868b759700 5 queue_transactions (trailing journal) 3597 2014-06-09 16:22:11.846984 7f868e75f700 2 waiting 51 > 50 ops || 2554471 > 104857600 2014-06-09 16:22:11.846977 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.846984 7f868b759700 5 queue_op 0x2e20ba0 seq 3597 osr(2.37a 0x2ed1160) 828 bytes (queue has 50 ops and 2553643 bytes) 2014-06-09 16:22:11.846989 7f8690f64700 2 waiting 51 > 50 ops || 2554471 > 104857600 2014-06-09 16:22:11.846981 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383ec60 (2 active) 2014-06-09 16:22:11.846994 7f868b759700 10 op_submit_finish 3597 2014-06-09 16:22:11.846994 7f868ff62700 2 waiting 51 > 50 ops || 2555574 > 104857600 2014-06-09 16:22:11.846998 7f868ef60700 2 waiting 51 > 50 ops || 2554471 > 104857600 2014-06-09 16:22:11.847002 7f868a757700 2 waiting 51 > 50 ops || 2554469 > 104857600 2014-06-09 16:22:11.846997 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.6cd( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.6cd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:11.847007 7f86a5e11700 5 _do_op 0x30061320 seq 3549 osr(2.27 0x2ed0250)/0x2ed0250 start 2014-06-09 16:22:11.847010 7f86a5e11700 10 _do_transaction on 0xc33a9c0 2014-06-09 16:22:11.847010 7f8689755700 2 waiting 51 > 50 ops || 2555526 > 104857600 2014-06-09 16:22:11.847015 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.847023 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.847024 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.847028 7f868b759700 5 queue_transactions existing osr(2.6cd 0x2ed1e70)/0x2ed1e70 2014-06-09 16:22:11.847032 7f868b759700 2 waiting 51 > 50 ops || 2554471 > 104857600 2014-06-09 16:22:11.846974 7f869af78700 5 -- op tracker -- , seq: 3304, time: 2014-06-09 16:22:11.753739, event: all_read, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.847036 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3548.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.847041 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.847041 7f869af78700 5 -- op tracker -- , seq: 3304, time: 2014-06-09 16:22:11.846948, event: dispatched, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.847047 7f869af78700 5 -- op tracker -- , seq: 3304, time: 2014-06-09 16:22:11.847046, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.847051 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.40c) v4 from osd.4 2014-06-09 16:22:11.847054 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c13500 2014-06-09 16:22:11.847058 7f86a5610700 10 _do_op 0x11fce2a0 seq 3548 r = 0, finisher 0x32072940 0 2014-06-09 16:22:11.847058 7f869af78700 5 -- op tracker -- , seq: 3304, time: 2014-06-09 16:22:11.847056, event: started, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.847062 7f86a5610700 10 _finish_op 0x11fce2a0 seq 3548 osr(3.9b 0x3ed0630)/0x3ed0630 2014-06-09 16:22:11.847072 7f868bf5a700 10 op_submit_start 3598 2014-06-09 16:22:11.847074 7f868bf5a700 5 queue_transactions (trailing journal) 3598 2014-06-09 16:22:11.847071 7f869af78700 5 -- op tracker -- , seq: 3304, time: 2014-06-09 16:22:11.847069, event: done, op: pg_info(1 pgs e243:2.40c) 2014-06-09 16:22:11.847073 7f868cf5c700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847075 7f868bf5a700 5 queue_op 0x324e77a0 seq 3598 osr(0.3b6 0x2bd72d0) 828 bytes (queue has 50 ops and 2553645 bytes) 2014-06-09 16:22:11.847078 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847078 7f868e75f700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847079 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847079 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4066c0 (1 active) 2014-06-09 16:22:11.847082 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc5a0 (2 active) 2014-06-09 16:22:11.847084 7f8690f64700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847086 7f868bf5a700 10 op_submit_finish 3598 2014-06-09 16:22:11.847088 7f868ff62700 2 waiting 51 > 50 ops || 2555576 > 104857600 2014-06-09 16:22:11.847093 7f868ef60700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847097 7f868a757700 2 waiting 51 > 50 ops || 2554471 > 104857600 2014-06-09 16:22:11.847097 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847099 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847089 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:11.847100 7f869af78700 20 osd.26 243 _dispatch 0x309e7c00 pg_info(1 pgs e243:3.2d3) v4 2014-06-09 16:22:11.847101 7f8689755700 2 waiting 51 > 50 ops || 2555528 > 104857600 2014-06-09 16:22:11.847108 7f868b759700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847111 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.847105 7f869af78700 5 -- op tracker -- , seq: 3305, time: 2014-06-09 16:22:11.746496, event: header_read, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847117 7f869af78700 5 -- op tracker -- , seq: 3305, time: 2014-06-09 16:22:11.746496, event: throttled, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847121 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.847123 7f869af78700 5 -- op tracker -- , seq: 3305, time: 2014-06-09 16:22:11.746509, event: all_read, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847124 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3549.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.847126 7f868bf5a700 5 queue_transactions existing osr(3.66a 0x3ed19d0)/0x3ed19d0 2014-06-09 16:22:11.847128 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.847128 7f868bf5a700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847128 7f869af78700 5 -- op tracker -- , seq: 3305, time: 2014-06-09 16:22:11.847097, event: dispatched, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847133 7f868df5e700 2 waiting 51 > 50 ops || 2554471 > 104857600 2014-06-09 16:22:11.847134 7f869af78700 5 -- op tracker -- , seq: 3305, time: 2014-06-09 16:22:11.847132, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847136 7f86a5610700 5 _do_op 0x300614a0 seq 3550 osr(2.566 0x2ed18f0)/0x2ed18f0 start 2014-06-09 16:22:11.847137 7f86a5e11700 10 _do_op 0x30061320 seq 3549 r = 0, finisher 0x3206f840 0 2014-06-09 16:22:11.847139 7f86a5610700 10 _do_transaction on 0xbf51b00 2014-06-09 16:22:11.847138 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2d3) v4 from osd.2 2014-06-09 16:22:11.847140 7f86a5e11700 10 _finish_op 0x30061320 seq 3549 osr(2.27 0x2ed0250)/0x2ed0250 2014-06-09 16:22:11.847141 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e7c00 2014-06-09 16:22:11.847145 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.847145 7f869af78700 5 -- op tracker -- , seq: 3305, time: 2014-06-09 16:22:11.847143, event: started, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847150 7f868e75f700 10 op_submit_start 3599 2014-06-09 16:22:11.847152 7f868e75f700 5 queue_transactions (trailing journal) 3599 2014-06-09 16:22:11.847151 7f8690f64700 2 waiting 51 > 50 ops || 2554475 > 104857600 2014-06-09 16:22:11.847154 7f868e75f700 5 queue_op 0x2fcaef60 seq 3599 osr(0.6d2 0x384c1f0) 828 bytes (queue has 50 ops and 2553647 bytes) 2014-06-09 16:22:11.847158 7f868ff62700 2 waiting 51 > 50 ops || 2555578 > 104857600 2014-06-09 16:22:11.847163 7f868ef60700 2 waiting 51 > 50 ops || 2554475 > 104857600 2014-06-09 16:22:11.847167 7f868a757700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847165 7f869af78700 5 -- op tracker -- , seq: 3305, time: 2014-06-09 16:22:11.847163, event: done, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847172 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847172 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383ec60 (1 active) 2014-06-09 16:22:11.847173 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847174 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339c20 (2 active) 2014-06-09 16:22:11.847179 7f868e75f700 10 op_submit_finish 3599 2014-06-09 16:22:11.847181 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.847182 7f8689755700 2 waiting 51 > 50 ops || 2555530 > 104857600 2014-06-09 16:22:11.847184 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847188 7f868b759700 2 waiting 51 > 50 ops || 2554475 > 104857600 2014-06-09 16:22:11.847186 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847182 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:11.847192 7f869af78700 20 osd.26 243 _dispatch 0x30c13340 pg_info(1 pgs e243:1.14c) v4 2014-06-09 16:22:11.847193 7f868bf5a700 2 waiting 51 > 50 ops || 2554475 > 104857600 2014-06-09 16:22:11.847198 7f868df5e700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847197 7f869af78700 5 -- op tracker -- , seq: 3306, time: 2014-06-09 16:22:11.753761, event: header_read, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.847203 7f869af78700 5 -- op tracker -- , seq: 3306, time: 2014-06-09 16:22:11.753762, event: throttled, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.847209 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3550.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.847209 7f869af78700 5 -- op tracker -- , seq: 3306, time: 2014-06-09 16:22:11.753774, event: all_read, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.847213 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.847214 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.847215 7f869af78700 5 -- op tracker -- , seq: 3306, time: 2014-06-09 16:22:11.847184, event: dispatched, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.847218 7f868e75f700 5 queue_transactions existing osr(3.19 0x3ed03a0)/0x3ed03a0 2014-06-09 16:22:11.847221 7f868e75f700 2 waiting 51 > 50 ops || 2554473 > 104857600 2014-06-09 16:22:11.847221 7f869af78700 5 -- op tracker -- , seq: 3306, time: 2014-06-09 16:22:11.847219, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.847225 7f868cf5c700 2 waiting 51 > 50 ops || 2554475 > 104857600 2014-06-09 16:22:11.847225 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.14c) v4 from osd.4 2014-06-09 16:22:11.847227 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c13340 2014-06-09 16:22:11.847228 7f86a5e11700 5 _do_op 0x1e08aa20 seq 3551 osr(1.76 0x384c530)/0x384c530 start 2014-06-09 16:22:11.847231 7f86a5e11700 10 _do_transaction on 0xc262c00 2014-06-09 16:22:11.847231 7f869af78700 5 -- op tracker -- , seq: 3306, time: 2014-06-09 16:22:11.847230, event: started, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.847233 7f86a5610700 10 _do_op 0x300614a0 seq 3550 r = 0, finisher 0xcd0f480 0 2014-06-09 16:22:11.847236 7f86a5610700 10 _finish_op 0x300614a0 seq 3550 osr(2.566 0x2ed18f0)/0x2ed18f0 2014-06-09 16:22:11.847236 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.847242 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc5a0 (1 active) 2014-06-09 16:22:11.847244 7f8690f64700 10 op_submit_start 3600 2014-06-09 16:22:11.847244 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac5e60 (2 active) 2014-06-09 16:22:11.847245 7f8690f64700 5 queue_transactions (trailing journal) 3600 2014-06-09 16:22:11.847245 7f868ff62700 2 waiting 51 > 50 ops || 2554452 > 104857600 2014-06-09 16:22:11.847246 7f86a5610700 5 _do_op 0xc2d59e0 seq 3552 osr(2.4af 0x2ed1790)/0x2ed1790 start 2014-06-09 16:22:11.847246 7f8690f64700 5 queue_op 0x2fcaf380 seq 3600 osr(1.589 0x384da10) 828 bytes (queue has 50 ops and 2552521 bytes) 2014-06-09 16:22:11.847249 7f8690f64700 10 op_submit_finish 3600 2014-06-09 16:22:11.847249 7f86a5610700 10 _do_transaction on 0x2e62a80 2014-06-09 16:22:11.847250 7f868ef60700 2 waiting 51 > 50 ops || 2553349 > 104857600 2014-06-09 16:22:11.847254 7f868a757700 2 waiting 51 > 50 ops || 2553347 > 104857600 2014-06-09 16:22:11.847255 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.847259 7f8689755700 2 waiting 51 > 50 ops || 2554404 > 104857600 2014-06-09 16:22:11.847251 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 243/201 242/242/200) 2014-06-09 16:22:11.847281 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.847280 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.847284 7f8690f64700 5 queue_transactions existing osr(3.3a9 0x3ed0f60)/0x3ed0f60 2014-06-09 16:22:11.847286 7f8690f64700 2 waiting 51 > 50 ops || 2553349 > 104857600 2014-06-09 16:22:11.847289 7f868b759700 2 waiting 51 > 50 ops || 2553349 > 104857600 2014-06-09 16:22:11.847294 7f868bf5a700 2 waiting 51 > 50 ops || 2553349 > 104857600 2014-06-09 16:22:11.847294 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3551.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.847298 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.847300 7f868df5e700 2 waiting 51 > 50 ops || 2553347 > 104857600 2014-06-09 16:22:11.847298 7f869af78700 5 -- op tracker -- , seq: 3306, time: 2014-06-09 16:22:11.847296, event: done, op: pg_info(1 pgs e243:1.14c) 2014-06-09 16:22:11.847305 7f868e75f700 2 waiting 51 > 50 ops || 2553347 > 104857600 2014-06-09 16:22:11.847306 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847307 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847310 7f868cf5c700 2 waiting 51 > 50 ops || 2553349 > 104857600 2014-06-09 16:22:11.847317 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847319 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847319 7f869af78700 20 osd.26 243 _dispatch 0x30a19c00 pg_info(1 pgs e243:0.3fe) v4 2014-06-09 16:22:11.847324 7f869af78700 5 -- op tracker -- , seq: 3307, time: 2014-06-09 16:22:11.746517, event: header_read, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.847329 7f869af78700 5 -- op tracker -- , seq: 3307, time: 2014-06-09 16:22:11.746518, event: throttled, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.847330 7f86a5e11700 10 _do_op 0x1e08aa20 seq 3551 r = 0, finisher 0xcd0f040 0 2014-06-09 16:22:11.847334 7f86a5e11700 10 _finish_op 0x1e08aa20 seq 3551 osr(1.76 0x384c530)/0x384c530 2014-06-09 16:22:11.847335 7f869af78700 5 -- op tracker -- , seq: 3307, time: 2014-06-09 16:22:11.746528, event: all_read, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.847340 7f869af78700 5 -- op tracker -- , seq: 3307, time: 2014-06-09 16:22:11.847317, event: dispatched, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.847343 7f868ef60700 10 op_submit_start 3601 2014-06-09 16:22:11.847345 7f868ef60700 5 queue_transactions (trailing journal) 3601 2014-06-09 16:22:11.847346 7f869af78700 5 -- op tracker -- , seq: 3307, time: 2014-06-09 16:22:11.847344, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.847346 7f868ef60700 5 queue_op 0x33a2c00 seq 3601 osr(1.6e8 0x2ed0190) 828 bytes (queue has 50 ops and 2552523 bytes) 2014-06-09 16:22:11.847349 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3fe) v4 from osd.2 2014-06-09 16:22:11.847351 7f868a757700 2 waiting 51 > 50 ops || 2553349 > 104857600 2014-06-09 16:22:11.847352 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a19c00 2014-06-09 16:22:11.847356 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339c20 (1 active) 2014-06-09 16:22:11.847356 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.847357 7f8689755700 2 waiting 51 > 50 ops || 2554406 > 104857600 2014-06-09 16:22:11.847356 7f869af78700 5 -- op tracker -- , seq: 3307, time: 2014-06-09 16:22:11.847354, event: started, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.847359 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4066c0 (2 active) 2014-06-09 16:22:11.847363 7f868ef60700 10 op_submit_finish 3601 2014-06-09 16:22:11.847363 7f8690f64700 2 waiting 51 > 50 ops || 2553351 > 104857600 2014-06-09 16:22:11.847367 7f868b759700 2 waiting 51 > 50 ops || 2553351 > 104857600 2014-06-09 16:22:11.847372 7f868bf5a700 2 waiting 51 > 50 ops || 2553351 > 104857600 2014-06-09 16:22:11.847367 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.847377 7f868df5e700 2 waiting 51 > 50 ops || 2553349 > 104857600 2014-06-09 16:22:11.847379 7f869af78700 5 -- op tracker -- , seq: 3307, time: 2014-06-09 16:22:11.847377, event: done, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.847384 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3552.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.847386 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847387 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847388 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.847381 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.847390 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.847397 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847399 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847400 7f869af78700 20 osd.26 243 _dispatch 0x30c13180 pg_info(1 pgs e243:3.527) v4 2014-06-09 16:22:11.847402 7f86a5610700 10 _do_op 0xc2d59e0 seq 3552 r = 0, finisher 0xcd9fe00 0 2014-06-09 16:22:11.847398 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.847404 7f86a5610700 10 _finish_op 0xc2d59e0 seq 3552 osr(2.4af 0x2ed1790)/0x2ed1790 2014-06-09 16:22:11.847404 7f869af78700 5 -- op tracker -- , seq: 3308, time: 2014-06-09 16:22:11.753803, event: header_read, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.847411 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.847410 7f869af78700 5 -- op tracker -- , seq: 3308, time: 2014-06-09 16:22:11.753804, event: throttled, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.847414 7f868cf5c700 10 op_submit_start 3602 2014-06-09 16:22:11.847415 7f868cf5c700 5 queue_transactions (trailing journal) 3602 2014-06-09 16:22:11.847416 7f869af78700 5 -- op tracker -- , seq: 3308, time: 2014-06-09 16:22:11.753820, event: all_read, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.847417 7f868cf5c700 5 queue_op 0x14dd66c0 seq 3602 osr(3.7ce 0xb7f01a0) 828 bytes (queue has 50 ops and 2551468 bytes) 2014-06-09 16:22:11.847421 7f868ff62700 2 waiting 51 > 50 ops || 2553399 > 104857600 2014-06-09 16:22:11.847421 7f869af78700 5 -- op tracker -- , seq: 3308, time: 2014-06-09 16:22:11.847397, event: dispatched, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.847426 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac5e60 (1 active) 2014-06-09 16:22:11.847429 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7b00 (2 active) 2014-06-09 16:22:11.847429 7f8689755700 2 waiting 51 > 50 ops || 2553351 > 104857600 2014-06-09 16:22:11.847433 7f868cf5c700 10 op_submit_finish 3602 2014-06-09 16:22:11.847427 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.847435 7f8690f64700 2 waiting 51 > 50 ops || 2552296 > 104857600 2014-06-09 16:22:11.847439 7f868b759700 2 waiting 51 > 50 ops || 2552296 > 104857600 2014-06-09 16:22:11.847441 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.847443 7f868bf5a700 2 waiting 51 > 50 ops || 2552296 > 104857600 2014-06-09 16:22:11.847436 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.37( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.37( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.847446 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.847449 7f869af78700 5 -- op tracker -- , seq: 3308, time: 2014-06-09 16:22:11.847447, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.847456 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.527) v4 from osd.4 2014-06-09 16:22:11.847459 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c13180 2014-06-09 16:22:11.847457 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.847463 7f869af78700 5 -- op tracker -- , seq: 3308, time: 2014-06-09 16:22:11.847462, event: started, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.847467 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.847466 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.847473 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.847479 7f868cf5c700 5 queue_transactions existing osr(2.37 0x2ed05f0)/0x2ed05f0 2014-06-09 16:22:11.847478 7f869af78700 5 -- op tracker -- , seq: 3308, time: 2014-06-09 16:22:11.847476, event: done, op: pg_info(1 pgs e243:3.527) 2014-06-09 16:22:11.847482 7f868cf5c700 2 waiting 51 > 50 ops || 2552294 > 104857600 2014-06-09 16:22:11.847484 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847486 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847486 7f868df5e700 2 waiting 51 > 50 ops || 2552294 > 104857600 2014-06-09 16:22:11.847485 7f868ef60700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.847490 7f868a757700 2 waiting 51 > 50 ops || 2552294 > 104857600 2014-06-09 16:22:11.847494 7f86a5610700 5 _do_op 0xc2d42a0 seq 3554 osr(3.4c4 0x3ed1420)/0x3ed1420 start 2014-06-09 16:22:11.847496 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847497 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847498 7f869af78700 20 osd.26 243 _dispatch 0x30a19a40 pg_info(1 pgs e243:3.2d3) v4 2014-06-09 16:22:11.847502 7f869af78700 5 -- op tracker -- , seq: 3309, time: 2014-06-09 16:22:11.746533, event: header_read, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847500 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:171 2014-06-09 16:22:11.847508 7f869af78700 5 -- op tracker -- , seq: 3309, time: 2014-06-09 16:22:11.746533, event: throttled, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847515 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.847514 7f869af78700 5 -- op tracker -- , seq: 3309, time: 2014-06-09 16:22:11.746544, event: all_read, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847497 7f86a5610700 10 _do_transaction on 0xc2aa9c0 2014-06-09 16:22:11.847519 7f869af78700 5 -- op tracker -- , seq: 3309, time: 2014-06-09 16:22:11.847495, event: dispatched, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847525 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.847525 7f869af78700 5 -- op tracker -- , seq: 3309, time: 2014-06-09 16:22:11.847523, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847529 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2d3) v4 from osd.2 2014-06-09 16:22:11.847519 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 243/191 242/242/220) 2014-06-09 16:22:11.847531 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a19a40 2014-06-09 16:22:11.847535 7f869af78700 5 -- op tracker -- , seq: 3309, time: 2014-06-09 16:22:11.847533, event: started, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847553 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.847554 7f869af78700 5 -- op tracker -- , seq: 3309, time: 2014-06-09 16:22:11.847552, event: done, op: pg_info(1 pgs e243:3.2d3) 2014-06-09 16:22:11.847557 7f868ef60700 5 queue_transactions existing osr(3.603 0x3ed1af0)/0x3ed1af0 2014-06-09 16:22:11.847560 7f868ef60700 2 waiting 51 > 50 ops || 2552296 > 104857600 2014-06-09 16:22:11.847559 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.847561 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847562 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847564 7f868e75f700 2 waiting 51 > 50 ops || 2552294 > 104857600 2014-06-09 16:22:11.847568 7f86a5e11700 5 _do_op 0xc2d5380 seq 3553 osr(3.9b 0x3ed0630)/0x3ed0630 start 2014-06-09 16:22:11.847571 7f86a5e11700 10 _do_transaction on 0xc291bc0 2014-06-09 16:22:11.847572 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.847576 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3554.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.849066 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.849067 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.847581 7f8692f68700 2 waiting 51 > 50 ops || 2552294 > 104857600 2014-06-09 16:22:11.849071 7f869af78700 20 osd.26 243 _dispatch 0x30a19880 pg_info(1 pgs e243:0.3fe) v4 2014-06-09 16:22:11.849072 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.849097 7f8691f66700 2 waiting 51 > 50 ops || 2552326 > 104857600 2014-06-09 16:22:11.849078 7f869af78700 5 -- op tracker -- , seq: 3310, time: 2014-06-09 16:22:11.746557, event: header_read, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.849103 7f869af78700 5 -- op tracker -- , seq: 3310, time: 2014-06-09 16:22:11.746557, event: throttled, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.849110 7f869af78700 5 -- op tracker -- , seq: 3310, time: 2014-06-09 16:22:11.746569, event: all_read, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.849117 7f869af78700 5 -- op tracker -- , seq: 3310, time: 2014-06-09 16:22:11.847571, event: dispatched, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.849125 7f86a5610700 10 _do_op 0xc2d42a0 seq 3554 r = 0, finisher 0xbd6d340 0 2014-06-09 16:22:11.849125 7f869af78700 5 -- op tracker -- , seq: 3310, time: 2014-06-09 16:22:11.849123, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.849130 7f86a5610700 10 _finish_op 0xc2d42a0 seq 3554 osr(3.4c4 0x3ed1420)/0x3ed1420 2014-06-09 16:22:11.849130 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.3fe) v4 from osd.2 2014-06-09 16:22:11.849133 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a19880 2014-06-09 16:22:11.849139 7f869af78700 5 -- op tracker -- , seq: 3310, time: 2014-06-09 16:22:11.849137, event: started, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.849148 7f8689755700 10 op_submit_start 3603 2014-06-09 16:22:11.849150 7f8689755700 5 queue_transactions (trailing journal) 3603 2014-06-09 16:22:11.849151 7f8689755700 5 queue_op 0xbe7cde0 seq 3603 osr(1.1fb 0x384cd30) 1883 bytes (queue has 50 ops and 2552016 bytes) 2014-06-09 16:22:11.849152 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.849156 7f8690f64700 2 waiting 51 > 50 ops || 2552844 > 104857600 2014-06-09 16:22:11.849158 7f869af78700 5 -- op tracker -- , seq: 3310, time: 2014-06-09 16:22:11.849156, event: done, op: pg_info(1 pgs e243:0.3fe) 2014-06-09 16:22:11.849161 7f868b759700 2 waiting 51 > 50 ops || 2552844 > 104857600 2014-06-09 16:22:11.849166 7f868bf5a700 2 waiting 51 > 50 ops || 2552844 > 104857600 2014-06-09 16:22:11.849167 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.849170 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.849167 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3553.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.849170 7f868cf5c700 2 waiting 51 > 50 ops || 2552842 > 104857600 2014-06-09 16:22:11.849172 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.849175 7f868df5e700 2 waiting 51 > 50 ops || 2552842 > 104857600 2014-06-09 16:22:11.849179 7f868a757700 2 waiting 51 > 50 ops || 2552842 > 104857600 2014-06-09 16:22:11.849183 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.849184 7f868ef60700 2 waiting 51 > 50 ops || 2552844 > 104857600 2014-06-09 16:22:11.849185 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.849184 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7b00 (1 active) 2014-06-09 16:22:11.849186 7f869af78700 20 osd.26 243 _dispatch 0x30a196c0 pg_info(1 pgs e243:3.582) v4 2014-06-09 16:22:11.849187 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc5a0 (2 active) 2014-06-09 16:22:11.849182 7f866f5f0700 5 -- op tracker -- , seq: 3311, time: 2014-06-09 16:22:11.842823, event: header_read, op: osd_op(client.4539.0:34416 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 1728512~1032192] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.849189 7f868e75f700 2 waiting 51 > 50 ops || 2552842 > 104857600 2014-06-09 16:22:11.849192 7f8689755700 10 op_submit_finish 3603 2014-06-09 16:22:11.849195 7f86a5e11700 10 _do_op 0xc2d5380 seq 3553 r = 0, finisher 0xcd9f020 0 2014-06-09 16:22:11.849195 7f8692f68700 2 waiting 51 > 50 ops || 2552842 > 104857600 2014-06-09 16:22:11.849197 7f86a5e11700 10 _finish_op 0xc2d5380 seq 3553 osr(3.9b 0x3ed0630)/0x3ed0630 2014-06-09 16:22:11.849196 7f866f5f0700 5 -- op tracker -- , seq: 3311, time: 2014-06-09 16:22:11.842825, event: throttled, op: osd_op(client.4539.0:34416 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 1728512~1032192] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.849206 7f868ff62700 10 op_submit_start 3604 2014-06-09 16:22:11.849196 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:243, info:0.708( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.849207 7f868ff62700 5 queue_transactions (trailing journal) 3604 2014-06-09 16:22:11.849207 7f8692f68700 2 waiting 51 > 50 ops || 2553947 > 104857600 2014-06-09 16:22:11.849209 7f868ff62700 5 queue_op 0x308ca4e0 seq 3604 osr(3.5 0x3ed03f0) 1931 bytes (queue has 50 ops and 2553121 bytes) 2014-06-09 16:22:11.849213 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4066c0 (1 active) 2014-06-09 16:22:11.849210 7f866f5f0700 5 -- op tracker -- , seq: 3311, time: 2014-06-09 16:22:11.849134, event: all_read, op: osd_op(client.4539.0:34416 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 1728512~1032192] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.849215 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383ec60 (2 active) 2014-06-09 16:22:11.849211 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.849219 7f868ff62700 10 op_submit_finish 3604 2014-06-09 16:22:11.849223 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.849229 7f866f5f0700 5 -- op tracker -- , seq: 3311, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34416 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 1728512~1032192] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.849225 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.3f0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.849228 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.849239 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.849239 7f869af78700 5 -- op tracker -- , seq: 3312, time: 2014-06-09 16:22:11.749483, event: header_read, op: pg_info(1 pgs e243:3.582) 2014-06-09 16:22:11.849239 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.849248 7f869af78700 5 -- op tracker -- , seq: 3312, time: 2014-06-09 16:22:11.749484, event: throttled, op: pg_info(1 pgs e243:3.582) 2014-06-09 16:22:11.849247 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.849258 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.849255 7f869af78700 5 -- op tracker -- , seq: 3312, time: 2014-06-09 16:22:11.749497, event: all_read, op: pg_info(1 pgs e243:3.582) 2014-06-09 16:22:11.849255 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.849262 7f868ff62700 5 queue_transactions existing osr(2.3f0 0x2ed1320)/0x2ed1320 2014-06-09 16:22:11.849264 7f868ff62700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849266 7f8689755700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.849262 7f869af78700 5 -- op tracker -- , seq: 3312, time: 2014-06-09 16:22:11.849183, event: dispatched, op: pg_info(1 pgs e243:3.582) 2014-06-09 16:22:11.849271 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x3e0e000 (0 active) 2014-06-09 16:22:11.849264 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827600 prio 63 cost 1032192 latency 0.006441 osd_op(client.4539.0:34416 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 1728512~1032192] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.849274 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.849274 7f869af78700 5 -- op tracker -- , seq: 3312, time: 2014-06-09 16:22:11.849272, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.582) 2014-06-09 16:22:11.849279 7f86a5e11700 5 _do_op 0xc2d4780 seq 3556 osr(2.27 0x2ed0250)/0x2ed0250 start 2014-06-09 16:22:11.849279 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.582) v4 from osd.2 2014-06-09 16:22:11.849282 7f86a5e11700 10 _do_transaction on 0xc20be00 2014-06-09 16:22:11.849282 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a196c0 2014-06-09 16:22:11.849287 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.849281 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.849287 7f869af78700 5 -- op tracker -- , seq: 3312, time: 2014-06-09 16:22:11.849285, event: started, op: pg_info(1 pgs e243:3.582) 2014-06-09 16:22:11.849296 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.849304 7f869af78700 5 -- op tracker -- , seq: 3312, time: 2014-06-09 16:22:11.849302, event: done, op: pg_info(1 pgs e243:3.582) 2014-06-09 16:22:11.849300 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.369( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.849313 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.849315 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.849320 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.849327 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.849329 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.849330 7f869af78700 20 osd.26 243 _dispatch 0x33441180 pg_info(1 pgs e243:3.428) v4 2014-06-09 16:22:11.849333 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3556.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.849337 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.849337 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.849335 7f869af78700 5 -- op tracker -- , seq: 3313, time: 2014-06-09 16:22:11.848538, event: header_read, op: pg_info(1 pgs e243:3.428) 2014-06-09 16:22:11.849342 7f8689755700 5 queue_transactions existing osr(0.369 0x2bd6ff0)/0x2bd6ff0 2014-06-09 16:22:11.849342 7f869af78700 5 -- op tracker -- , seq: 3313, time: 2014-06-09 16:22:11.848539, event: throttled, op: pg_info(1 pgs e243:3.428) 2014-06-09 16:22:11.849345 7f8689755700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849348 7f86a5e11700 10 _do_op 0xc2d4780 seq 3556 r = 0, finisher 0xbd6b780 0 2014-06-09 16:22:11.849349 7f868b759700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849348 7f869af78700 5 -- op tracker -- , seq: 3313, time: 2014-06-09 16:22:11.848553, event: all_read, op: pg_info(1 pgs e243:3.428) 2014-06-09 16:22:11.849351 7f86a5e11700 10 _finish_op 0xc2d4780 seq 3556 osr(2.27 0x2ed0250)/0x2ed0250 2014-06-09 16:22:11.849358 7f868bf5a700 10 op_submit_start 3605 2014-06-09 16:22:11.849357 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383ec60 (1 active) 2014-06-09 16:22:11.849359 7f868bf5a700 5 queue_transactions (trailing journal) 3605 2014-06-09 16:22:11.849357 7f8692f68700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849359 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f440 (2 active) 2014-06-09 16:22:11.849360 7f868bf5a700 5 queue_op 0x324e6060 seq 3605 osr(3.66a 0x3ed19d0) 828 bytes (queue has 50 ops and 2553123 bytes) 2014-06-09 16:22:11.849361 7f86a5e11700 5 _do_op 0xc2d5a40 seq 3557 osr(3.4fa 0x3ed1720)/0x3ed1720 start 2014-06-09 16:22:11.849363 7f868bf5a700 10 op_submit_finish 3605 2014-06-09 16:22:11.849364 7f86a5e11700 10 _do_transaction on 0xbebb2c0 2014-06-09 16:22:11.849364 7f868a757700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849370 7f868ef60700 2 waiting 51 > 50 ops || 2553951 > 104857600 2014-06-09 16:22:11.849355 7f869af78700 5 -- op tracker -- , seq: 3313, time: 2014-06-09 16:22:11.849326, event: dispatched, op: pg_info(1 pgs e243:3.428) 2014-06-09 16:22:11.849365 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.849375 7f868e75f700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849379 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.849380 7f8691f66700 2 waiting 51 > 50 ops || 2553981 > 104857600 2014-06-09 16:22:11.849378 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.849385 7f868ff62700 2 waiting 51 > 50 ops || 2553951 > 104857600 2014-06-09 16:22:11.849378 7f869af78700 5 -- op tracker -- , seq: 3313, time: 2014-06-09 16:22:11.849375, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.428) 2014-06-09 16:22:11.849390 7f8689755700 2 waiting 51 > 50 ops || 2553951 > 104857600 2014-06-09 16:22:11.849390 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.428) v4 from osd.6 2014-06-09 16:22:11.849387 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.849394 7f868b759700 2 waiting 51 > 50 ops || 2553951 > 104857600 2014-06-09 16:22:11.849394 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x33441180 2014-06-09 16:22:11.849397 7f8690f64700 2 waiting 51 > 50 ops || 2553951 > 104857600 2014-06-09 16:22:11.849399 7f869af78700 5 -- op tracker -- , seq: 3313, time: 2014-06-09 16:22:11.849397, event: started, op: pg_info(1 pgs e243:3.428) 2014-06-09 16:22:11.849396 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.849405 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.849412 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.849417 7f869af78700 5 -- op tracker -- , seq: 3313, time: 2014-06-09 16:22:11.849414, event: done, op: pg_info(1 pgs e243:3.428) 2014-06-09 16:22:11.849424 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.849426 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.849428 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.849426 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.849435 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.849441 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3557.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.849444 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.849443 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.849451 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.849454 7f86a5e11700 10 _do_op 0xc2d5a40 seq 3557 r = 0, finisher 0x307f0200 0 2014-06-09 16:22:11.849457 7f86a5e11700 10 _finish_op 0xc2d5a40 seq 3557 osr(3.4fa 0x3ed1720)/0x3ed1720 2014-06-09 16:22:11.849462 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f440 (1 active) 2014-06-09 16:22:11.849455 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.849464 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc5a0 (2 active) 2014-06-09 16:22:11.849466 7f86a5e11700 5 _do_op 0xc2d5320 seq 3555 osr(2.312 0x2ed0f30)/0x2ed0f30 start 2014-06-09 16:22:11.849466 7f8692f68700 10 op_submit_start 3606 2014-06-09 16:22:11.849468 7f86a5e11700 10 _do_transaction on 0xc1f7980 2014-06-09 16:22:11.849469 7f8692f68700 5 queue_transactions (trailing journal) 3606 2014-06-09 16:22:11.849470 7f8692f68700 5 queue_op 0x324e6a20 seq 3606 osr(2.73 0x2ed0510) 826 bytes (queue has 50 ops and 2553121 bytes) 2014-06-09 16:22:11.849467 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.849473 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.849474 7f8692f68700 10 op_submit_finish 3606 2014-06-09 16:22:11.849474 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.849478 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.82( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.849482 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.849490 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.849497 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.849504 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.849509 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.849516 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3555.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.849519 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.849523 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.849518 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.849528 7f8692f68700 5 queue_transactions existing osr(1.82 0x384c4d0)/0x384c4d0 2014-06-09 16:22:11.849526 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.849531 7f8692f68700 2 waiting 51 > 50 ops || 2553947 > 104857600 2014-06-09 16:22:11.849534 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.849536 7f868a757700 2 waiting 51 > 50 ops || 2553947 > 104857600 2014-06-09 16:22:11.849542 7f868ef60700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849539 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.849548 7f868e75f700 2 waiting 51 > 50 ops || 2553947 > 104857600 2014-06-09 16:22:11.849553 7f8691f66700 2 waiting 51 > 50 ops || 2553979 > 104857600 2014-06-09 16:22:11.849550 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.849559 7f868ff62700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849563 7f8689755700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849558 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.849566 7f868b759700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849570 7f8690f64700 2 waiting 51 > 50 ops || 2553949 > 104857600 2014-06-09 16:22:11.849566 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.849556 7f86a5e11700 10 _do_op 0xc2d5320 seq 3555 r = 0, finisher 0xbdf0260 0 2014-06-09 16:22:11.849575 7f86a5e11700 10 _finish_op 0xc2d5320 seq 3555 osr(2.312 0x2ed0f30)/0x2ed0f30 2014-06-09 16:22:11.849579 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc5a0 (1 active) 2014-06-09 16:22:11.849574 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.849581 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff59e0 (2 active) 2014-06-09 16:22:11.849582 7f8692f68700 10 op_submit_start 3607 2014-06-09 16:22:11.849583 7f86a5e11700 5 _do_op 0x2e200c0 seq 3559 osr(1.604 0x384dc50)/0x384dc50 start 2014-06-09 16:22:11.849584 7f8692f68700 5 queue_transactions (trailing journal) 3607 2014-06-09 16:22:11.849585 7f86a5e11700 10 _do_transaction on 0xbe88e40 2014-06-09 16:22:11.849581 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.849586 7f8692f68700 5 queue_op 0x324e6ae0 seq 3607 osr(1.82 0x384c4d0) 826 bytes (queue has 50 ops and 2553119 bytes) 2014-06-09 16:22:11.849589 7f8692f68700 10 op_submit_finish 3607 2014-06-09 16:22:11.849590 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.849593 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.849592 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.772( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.849602 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.849609 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.849606 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.772( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: got info+log from osd.2 0.772( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.849618 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.849619 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.2 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.849622 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.849624 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.849623 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.849634 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3559.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.849629 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.772( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] exit Started/Stray 0.940716 7 0.000172 2014-06-09 16:22:11.849638 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.849636 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.849641 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.772( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.849647 7f86a5e11700 10 _do_op 0x2e200c0 seq 3559 r = 0, finisher 0xc168d40 0 2014-06-09 16:22:11.849651 7f86a5e11700 10 _finish_op 0x2e200c0 seq 3559 osr(1.604 0x384dc50)/0x384dc50 2014-06-09 16:22:11.849653 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff59e0 (1 active) 2014-06-09 16:22:11.849655 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b200 (2 active) 2014-06-09 16:22:11.849644 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0xe2ac300,0x21966200,0x21966100,0x21966000,0x230daf00,0xe2ac200,0xe2ac900,0xe2ac800,0x1c66a500,0x1c66a400,0x1c66a300,0x1c769f00,0x1c769e00,0x1c769600,0x1c769d00,0x1c769500,0x230da900,0x1c769400,0x22008900,0x22008800,0x22008700 2014-06-09 16:22:11.849657 7f86a5e11700 5 _do_op 0x2ea53860 seq 3560 osr(3.55b 0x3ed1980)/0x3ed1980 start 2014-06-09 16:22:11.849659 7f86a5e11700 10 _do_transaction on 0xc0d1800 2014-06-09 16:22:11.849656 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.772( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.849665 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.849666 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.772( empty local-les=191 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.849676 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.772( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.849685 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.772( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.849695 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.849712 7f86a5e11700 15 _omap_rmkeyrange meta/1fe10d7b/pglog_3.55b/0//-1 [0000000214.00000000000000000714,4294967295.18446744073709551615] 2014-06-09 16:22:11.849716 7f86a5e11700 15 get_omap_iterator meta/1fe10d7b/pglog_3.55b/0//-1 2014-06-09 16:22:11.849718 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.849741 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.849729 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.849746 7f8692f68700 5 queue_transactions existing osr(0.772 0x384c2f0)/0x384c2f0 2014-06-09 16:22:11.849749 7f8692f68700 10 op_submit_start 3608 2014-06-09 16:22:11.849750 7f8692f68700 5 queue_transactions (trailing journal) 3608 2014-06-09 16:22:11.849751 7f8692f68700 5 queue_op 0x324e6ba0 seq 3608 osr(0.772 0x384c2f0) 1883 bytes (queue has 50 ops and 2554174 bytes) 2014-06-09 16:22:11.849754 7f8692f68700 10 op_submit_finish 3608 2014-06-09 16:22:11.849747 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.849757 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.849761 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:976 2014-06-09 16:22:11.849770 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.849769 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 0.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.849781 7f8692f68700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.849784 7f8692f68700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.849778 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.849788 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.056972 7 0.000626 2014-06-09 16:22:11.849797 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.849798 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.849804 7f86a5e11700 15 _omap_rmkeys meta/1fe10d7b/pglog_3.55b/0//-1 2014-06-09 16:22:11.849802 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.6f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.849809 7f8692f68700 5 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.849818 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.849828 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.849838 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe10d7b/pglog_3.55b/0//-1 2014-06-09 16:22:11.849835 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.849847 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.849848 7f86a5e11700 10 _omap_rmkeys meta/1fe10d7b/pglog_3.55b/0//-1 r = 0 2014-06-09 16:22:11.849852 7f868bf5a700 5 queue_transactions existing osr(2.6f 0x2ed0550)/0x2ed0550 2014-06-09 16:22:11.849853 7f86a5e11700 15 _omap_rmkeys meta/1fe10d7b/pglog_3.55b/0//-1 2014-06-09 16:22:11.849854 7f868bf5a700 2 waiting 51 > 50 ops || 2555000 > 104857600 2014-06-09 16:22:11.849857 7f86a5e11700 10 _omap_rmkeys meta/1fe10d7b/pglog_3.55b/0//-1 r = 0 2014-06-09 16:22:11.849859 7f868cf5c700 2 waiting 51 > 50 ops || 2555000 > 104857600 2014-06-09 16:22:11.849864 7f868ef60700 2 waiting 51 > 50 ops || 2555002 > 104857600 2014-06-09 16:22:11.849865 7f86a5610700 5 _do_op 0x2e212c0 seq 3558 osr(2.312 0x2ed0f30)/0x2ed0f30 start 2014-06-09 16:22:11.849868 7f86a5610700 10 _do_transaction on 0xc3f29c0 2014-06-09 16:22:11.849869 7f868e75f700 2 waiting 51 > 50 ops || 2555000 > 104857600 2014-06-09 16:22:11.849873 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.849877 7f86a5e11700 15 _omap_setkeys meta/1fe10d7b/pglog_3.55b/0//-1 2014-06-09 16:22:11.849891 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.849895 7f8692f68700 5 queue_transactions existing osr(0.5fb 0x2bd79a0)/0x2bd79a0 2014-06-09 16:22:11.849898 7f8692f68700 2 waiting 51 > 50 ops || 2556183 > 104857600 2014-06-09 16:22:11.849895 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3560.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.849902 7f868a757700 2 waiting 51 > 50 ops || 2555000 > 104857600 2014-06-09 16:22:11.849902 7f86a5e11700 10 cid: oid: 1fe10d7b/pglog_3.55b/0//-1 skipping op, spos 3560.0.0 <= header.spos 150442.0.0 2014-06-09 16:22:11.849908 7f8691f66700 2 waiting 51 > 50 ops || 2555032 > 104857600 2014-06-09 16:22:11.849905 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.849913 7f8689755700 2 waiting 51 > 50 ops || 2555002 > 104857600 2014-06-09 16:22:11.849918 7f868b759700 2 waiting 51 > 50 ops || 2555002 > 104857600 2014-06-09 16:22:11.849922 7f8690f64700 2 waiting 51 > 50 ops || 2555002 > 104857600 2014-06-09 16:22:11.849926 7f868df5e700 2 waiting 51 > 50 ops || 2555000 > 104857600 2014-06-09 16:22:11.849929 7f868ff62700 2 waiting 51 > 50 ops || 2555002 > 104857600 2014-06-09 16:22:11.849956 7f86a5e11700 10 _do_op 0x2ea53860 seq 3560 r = 0, finisher 0xbd679a0 0 2014-06-09 16:22:11.849960 7f86a5e11700 10 _finish_op 0x2ea53860 seq 3560 osr(3.55b 0x3ed1980)/0x3ed1980 2014-06-09 16:22:11.849996 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850010 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3558.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850014 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850029 7f86a5610700 10 _do_op 0x2e212c0 seq 3558 r = 0, finisher 0xc168920 0 2014-06-09 16:22:11.850054 7f868cf5c700 10 op_submit_start 3609 2014-06-09 16:22:11.850055 7f868cf5c700 5 queue_transactions (trailing journal) 3609 2014-06-09 16:22:11.850056 7f868cf5c700 5 queue_op 0xd470300 seq 3609 osr(2.37 0x2ed05f0) 826 bytes (queue has 50 ops and 2553057 bytes) 2014-06-09 16:22:11.850061 7f868ef60700 2 waiting 51 > 50 ops || 2553885 > 104857600 2014-06-09 16:22:11.850065 7f868e75f700 2 waiting 51 > 50 ops || 2553883 > 104857600 2014-06-09 16:22:11.850069 7f8692f68700 2 waiting 51 > 50 ops || 2555066 > 104857600 2014-06-09 16:22:11.850073 7f868a757700 2 waiting 51 > 50 ops || 2553883 > 104857600 2014-06-09 16:22:11.850077 7f8691f66700 2 waiting 51 > 50 ops || 2553915 > 104857600 2014-06-09 16:22:11.850081 7f8689755700 2 waiting 51 > 50 ops || 2553885 > 104857600 2014-06-09 16:22:11.850085 7f868b759700 2 waiting 51 > 50 ops || 2553885 > 104857600 2014-06-09 16:22:11.850088 7f8690f64700 2 waiting 51 > 50 ops || 2553885 > 104857600 2014-06-09 16:22:11.850092 7f868df5e700 2 waiting 51 > 50 ops || 2553883 > 104857600 2014-06-09 16:22:11.850096 7f868ff62700 2 waiting 51 > 50 ops || 2553885 > 104857600 2014-06-09 16:22:11.850101 7f868bf5a700 2 waiting 51 > 50 ops || 2553883 > 104857600 2014-06-09 16:22:11.850094 7f866f5f0700 5 -- op tracker -- , seq: 3314, time: 2014-06-09 16:22:11.849290, event: header_read, op: osd_op(client.4539.0:34417 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 2760704~516096] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.850110 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b200 (1 active) 2014-06-09 16:22:11.850113 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcada5a0 (2 active) 2014-06-09 16:22:11.850117 7f86a5e11700 5 _do_op 0xc2ead80 seq 3561 osr(3.70e 0x3ed1ba0)/0x3ed1ba0 start 2014-06-09 16:22:11.850118 7f86a5610700 10 _finish_op 0x2e212c0 seq 3558 osr(2.312 0x2ed0f30)/0x2ed0f30 2014-06-09 16:22:11.850120 7f86a5e11700 10 _do_transaction on 0x2d82840 2014-06-09 16:22:11.850122 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc5a0 (1 active) 2014-06-09 16:22:11.850124 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339c20 (2 active) 2014-06-09 16:22:11.850127 7f86a5610700 5 _do_op 0x2e21560 seq 3562 osr(1.76 0x384c530)/0x384c530 start 2014-06-09 16:22:11.850127 7f868e75f700 2 waiting 51 > 50 ops || 2553883 > 104857600 2014-06-09 16:22:11.850130 7f86a5610700 10 _do_transaction on 0xc12e3c0 2014-06-09 16:22:11.850132 7f868cf5c700 10 op_submit_finish 3609 2014-06-09 16:22:11.850134 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850138 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.4db( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.4db( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.850175 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.850178 7f868cf5c700 5 queue_transactions existing osr(1.4db 0x384d850)/0x384d850 2014-06-09 16:22:11.850181 7f868cf5c700 2 waiting 51 > 50 ops || 2553885 > 104857600 2014-06-09 16:22:11.850182 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850185 7f8692f68700 2 waiting 51 > 50 ops || 2555066 > 104857600 2014-06-09 16:22:11.850189 7f868a757700 2 waiting 51 > 50 ops || 2553883 > 104857600 2014-06-09 16:22:11.850194 7f868ef60700 10 op_submit_start 3610 2014-06-09 16:22:11.850196 7f868ef60700 5 queue_transactions (trailing journal) 3610 2014-06-09 16:22:11.850194 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3562.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850197 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850197 7f868ef60700 5 queue_op 0x24104240 seq 3610 osr(3.603 0x3ed1af0) 828 bytes (queue has 50 ops and 2553057 bytes) 2014-06-09 16:22:11.850200 7f868ef60700 10 op_submit_finish 3610 2014-06-09 16:22:11.850205 7f86a5610700 10 _do_op 0x2e21560 seq 3562 r = 0, finisher 0xcc95ea0 0 2014-06-09 16:22:11.850208 7f86a5610700 10 _finish_op 0x2e21560 seq 3562 osr(1.76 0x384c530)/0x384c530 2014-06-09 16:22:11.850212 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339c20 (1 active) 2014-06-09 16:22:11.850204 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.850215 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358c60 (2 active) 2014-06-09 16:22:11.850217 7f86a5610700 5 _do_op 0x2ea52a80 seq 3563 osr(1.691 0x384de30)/0x384de30 start 2014-06-09 16:22:11.850219 7f86a5610700 10 _do_transaction on 0x3540840 2014-06-09 16:22:11.850218 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.850225 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850228 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.850245 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.850255 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850257 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.850271 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.850284 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.850287 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.850295 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3563.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850300 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850313 7f86a5610700 10 _do_op 0x2ea52a80 seq 3563 r = 0, finisher 0x318c3200 0 2014-06-09 16:22:11.850317 7f86a5610700 10 _finish_op 0x2ea52a80 seq 3563 osr(1.691 0x384de30)/0x384de30 2014-06-09 16:22:11.850298 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.850320 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358c60 (1 active) 2014-06-09 16:22:11.850323 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300120 (2 active) 2014-06-09 16:22:11.850324 7f86a5610700 5 _do_op 0x2c06ec00 seq 3564 osr(3.516 0x3ed16d0)/0x3ed16d0 start 2014-06-09 16:22:11.850326 7f86a5610700 10 _do_transaction on 0x48009c0 2014-06-09 16:22:11.850320 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.850331 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850327 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.850337 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.850349 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.850358 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850361 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.850390 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41ef0 (19 active) 2014-06-09 16:22:11.850397 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3564.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850399 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.850401 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850412 7f86a5610700 10 _do_op 0x2c06ec00 seq 3564 r = 0, finisher 0x3042a540 0 2014-06-09 16:22:11.850415 7f86a5610700 10 _finish_op 0x2c06ec00 seq 3564 osr(3.516 0x3ed16d0)/0x3ed16d0 2014-06-09 16:22:11.850418 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300120 (1 active) 2014-06-09 16:22:11.850404 7f868ef60700 10 osd.26 243 dequeue_op 0xdfa7d00 prio 63 cost 516096 latency 0.037095 osd_op(client.4539.0:34389 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.850420 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358120 (2 active) 2014-06-09 16:22:11.850422 7f86a5610700 5 _do_op 0x288f95c0 seq 3565 osr(3.31c 0x3ed0d40)/0x3ed0d40 start 2014-06-09 16:22:11.850425 7f86a5610700 10 _do_transaction on 0xc0ccb40 2014-06-09 16:22:11.850431 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850447 7f868e75f700 10 op_submit_start 3611 2014-06-09 16:22:11.850449 7f868e75f700 5 queue_transactions (trailing journal) 3611 2014-06-09 16:22:11.850451 7f868e75f700 5 queue_op 0x2fcaf920 seq 3611 osr(3.19 0x3ed03a0) 826 bytes (queue has 48 ops and 2550894 bytes) 2014-06-09 16:22:11.850463 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850465 7f868e75f700 10 op_submit_finish 3611 2014-06-09 16:22:11.850469 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.850482 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.850490 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.850498 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3565.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850502 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850497 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.850510 7f868e75f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.850523 7f86a5610700 10 _do_op 0x288f95c0 seq 3565 r = 0, finisher 0xbd69ca0 0 2014-06-09 16:22:11.850527 7f86a5610700 10 _finish_op 0x288f95c0 seq 3565 osr(3.31c 0x3ed0d40)/0x3ed0d40 2014-06-09 16:22:11.850530 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358120 (1 active) 2014-06-09 16:22:11.850532 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f200 (2 active) 2014-06-09 16:22:11.850535 7f86a5610700 5 _do_op 0x230b9860 seq 3566 osr(2.4f4 0x2ed16b0)/0x2ed16b0 start 2014-06-09 16:22:11.850537 7f86a5610700 10 _do_transaction on 0xc33a540 2014-06-09 16:22:11.850542 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850524 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.850555 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.850567 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850560 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.e6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.850592 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.850592 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3566.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850595 7f868e75f700 5 queue_transactions existing osr(2.e6 0x2ed07d0)/0x2ed07d0 2014-06-09 16:22:11.850596 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850597 7f868e75f700 10 op_submit_start 3612 2014-06-09 16:22:11.850598 7f868e75f700 5 queue_transactions (trailing journal) 3612 2014-06-09 16:22:11.850598 7f868e75f700 5 queue_op 0x2fcae420 seq 3612 osr(2.e6 0x2ed07d0) 826 bytes (queue has 48 ops and 2550892 bytes) 2014-06-09 16:22:11.850600 7f868e75f700 10 op_submit_finish 3612 2014-06-09 16:22:11.850602 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.3d1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.3d1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:11.850607 7f86a5610700 10 _do_op 0x230b9860 seq 3566 r = 0, finisher 0x31412240 0 2014-06-09 16:22:11.850611 7f86a5610700 10 _finish_op 0x230b9860 seq 3566 osr(2.4f4 0x2ed16b0)/0x2ed16b0 2014-06-09 16:22:11.850614 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f200 (1 active) 2014-06-09 16:22:11.850616 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114000 (2 active) 2014-06-09 16:22:11.850618 7f86a5610700 5 _do_op 0x288f9bc0 seq 3567 osr(1.631 0x384dfb0)/0x384dfb0 start 2014-06-09 16:22:11.850620 7f86a5610700 10 _do_transaction on 0xbebdb00 2014-06-09 16:22:11.850626 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850640 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.850644 7f868e75f700 5 queue_transactions existing osr(2.3d1 0x2ed13b0)/0x2ed13b0 2014-06-09 16:22:11.850646 7f868e75f700 10 op_submit_start 3613 2014-06-09 16:22:11.850646 7f868e75f700 5 queue_transactions (trailing journal) 3613 2014-06-09 16:22:11.850647 7f868e75f700 5 queue_op 0x2fcae600 seq 3613 osr(2.3d1 0x2ed13b0) 828 bytes (queue has 48 ops and 2550892 bytes) 2014-06-09 16:22:11.850649 7f868e75f700 10 op_submit_finish 3613 2014-06-09 16:22:11.850649 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850651 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.850658 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3567.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850660 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850659 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.850667 7f86a5610700 10 _do_op 0x288f9bc0 seq 3567 r = 0, finisher 0xbd730e0 0 2014-06-09 16:22:11.850664 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.850669 7f86a5610700 10 _finish_op 0x288f9bc0 seq 3567 osr(1.631 0x384dfb0)/0x384dfb0 2014-06-09 16:22:11.850670 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114000 (1 active) 2014-06-09 16:22:11.850668 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.850672 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb320 (2 active) 2014-06-09 16:22:11.850674 7f86a5610700 5 _do_op 0x288f97a0 seq 3568 osr(3.62a 0x3ed1aa0)/0x3ed1aa0 start 2014-06-09 16:22:11.850675 7f86a5610700 10 _do_transaction on 0xbf58c00 2014-06-09 16:22:11.850676 7f868e75f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.850686 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.850701 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850701 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.850708 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:11.850723 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850720 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.850726 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.850732 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3568.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850734 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850731 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.850740 7f868e75f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.850742 7f86a5610700 10 _do_op 0x288f97a0 seq 3568 r = 0, finisher 0xcc82aa0 0 2014-06-09 16:22:11.850744 7f86a5610700 10 _finish_op 0x288f97a0 seq 3568 osr(3.62a 0x3ed1aa0)/0x3ed1aa0 2014-06-09 16:22:11.850745 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb320 (1 active) 2014-06-09 16:22:11.850747 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a360 (2 active) 2014-06-09 16:22:11.850748 7f86a5610700 5 _do_op 0xbc2db60 seq 3569 osr(0.1da 0x2bd6920)/0x2bd6920 start 2014-06-09 16:22:11.850750 7f86a5610700 10 _do_transaction on 0x39dbd40 2014-06-09 16:22:11.850753 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850750 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:948 2014-06-09 16:22:11.850760 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.850772 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850780 7f8689755700 10 op_submit_start 3614 2014-06-09 16:22:11.850782 7f8689755700 5 queue_transactions (trailing journal) 3614 2014-06-09 16:22:11.850782 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3569.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850785 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850784 7f8689755700 5 queue_op 0xbe7c480 seq 3614 osr(0.369 0x2bd6ff0) 828 bytes (queue has 48 ops and 2550892 bytes) 2014-06-09 16:22:11.850787 7f8689755700 10 op_submit_finish 3614 2014-06-09 16:22:11.850788 7f8690f64700 10 op_submit_start 3615 2014-06-09 16:22:11.850790 7f8690f64700 5 queue_transactions (trailing journal) 3615 2014-06-09 16:22:11.850792 7f8690f64700 5 queue_op 0x2fcaf320 seq 3615 osr(3.3a9 0x3ed0f60) 828 bytes (queue has 49 ops and 2551720 bytes) 2014-06-09 16:22:11.850795 7f8690f64700 10 op_submit_finish 3615 2014-06-09 16:22:11.850797 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.850793 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.94( v 214'571 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.850809 7f86a5610700 10 _do_op 0xbc2db60 seq 3569 r = 0, finisher 0x3042a420 0 2014-06-09 16:22:11.850812 7f86a5610700 10 _finish_op 0xbc2db60 seq 3569 osr(0.1da 0x2bd6920)/0x2bd6920 2014-06-09 16:22:11.850813 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a360 (1 active) 2014-06-09 16:22:11.850815 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77ab40 (2 active) 2014-06-09 16:22:11.850809 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.850816 7f86a5610700 5 _do_op 0xbe7db00 seq 3570 osr(3.1eb 0x3ed09c0)/0x3ed09c0 start 2014-06-09 16:22:11.850819 7f86a5610700 10 _do_transaction on 0xc271800 2014-06-09 16:22:11.850809 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] state: peer osd.9 activated and committed 2014-06-09 16:22:11.850823 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850818 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.850821 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] all_activated_and_committed 2014-06-09 16:22:11.850826 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.850830 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] share_pg_info 2014-06-09 16:22:11.850834 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.850844 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850845 7f8689755700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.850841 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.850855 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3570.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850857 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850854 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.850846 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.c1( v 214'609 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.850856 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] publish_stats_to_osd 243:911 2014-06-09 16:22:11.850867 7f86a5610700 10 _do_op 0xbe7db00 seq 3570 r = 0, finisher 0x2e0e060 0 2014-06-09 16:22:11.850862 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.850869 7f86a5610700 10 _finish_op 0xbe7db00 seq 3570 osr(3.1eb 0x3ed09c0)/0x3ed09c0 2014-06-09 16:22:11.850871 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77ab40 (1 active) 2014-06-09 16:22:11.850872 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.850872 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbc20 (2 active) 2014-06-09 16:22:11.850874 7f86a5610700 5 _do_op 0x308cb980 seq 3571 osr(0.663 0x2bd7f30)/0x2bd7f30 start 2014-06-09 16:22:11.850875 7f86a5610700 10 _do_transaction on 0xc23e900 2014-06-09 16:22:11.850870 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.850879 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850879 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.850875 7f8689755700 20 osd.26 243 Sending info 3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) to shard 4 2014-06-09 16:22:11.850868 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] state: peer osd.3 activated and committed 2014-06-09 16:22:11.850882 7f8689755700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.850883 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.850882 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] all_activated_and_committed 2014-06-09 16:22:11.850895 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.850894 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] share_pg_info 2014-06-09 16:22:11.850905 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850903 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.850911 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e770 (19 active) 2014-06-09 16:22:11.850911 7f868e75f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.850914 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3571.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850917 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850910 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.850918 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.850925 7f86a5610700 10 _do_op 0x308cb980 seq 3571 r = 0, finisher 0x318c22c0 0 2014-06-09 16:22:11.850927 7f86a5610700 10 _finish_op 0x308cb980 seq 3571 osr(0.663 0x2bd7f30)/0x2bd7f30 2014-06-09 16:22:11.850928 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbc20 (1 active) 2014-06-09 16:22:11.850930 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0000 (2 active) 2014-06-09 16:22:11.850931 7f86a5610700 5 _do_op 0xc64a0c0 seq 3572 osr(1.603 0x384dc90)/0x384dc90 start 2014-06-09 16:22:11.850925 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.850932 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e770 (20 active) 2014-06-09 16:22:11.850933 7f86a5610700 10 _do_transaction on 0xc295bc0 2014-06-09 16:22:11.850936 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850928 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] publish_stats_to_osd 243:1030 2014-06-09 16:22:11.850938 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.850935 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.364( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.364( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.850946 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.850956 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.850953 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.850961 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.850965 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.850964 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3572.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.850965 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.850967 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.850970 7f8689755700 5 queue_transactions existing osr(2.364 0x2ed11b0)/0x2ed11b0 2014-06-09 16:22:11.850972 7f8689755700 10 op_submit_start 3616 2014-06-09 16:22:11.850973 7f8689755700 5 queue_transactions (trailing journal) 3616 2014-06-09 16:22:11.850965 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.850974 7f8689755700 5 queue_op 0xbe7dc80 seq 3616 osr(2.364 0x2ed11b0) 828 bytes (queue has 47 ops and 2548290 bytes) 2014-06-09 16:22:11.850976 7f8689755700 10 op_submit_finish 3616 2014-06-09 16:22:11.850980 7f86a5610700 10 _do_op 0xc64a0c0 seq 3572 r = 0, finisher 0x30629b00 0 2014-06-09 16:22:11.850982 7f86a5610700 10 _finish_op 0xc64a0c0 seq 3572 osr(1.603 0x384dc90)/0x384dc90 2014-06-09 16:22:11.850976 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.850971 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:11.850984 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0000 (1 active) 2014-06-09 16:22:11.850985 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339560 (2 active) 2014-06-09 16:22:11.850979 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.369( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.369( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.850987 7f86a5610700 5 _do_op 0x30d96960 seq 3573 osr(0.67e 0x2bd7e30)/0x2bd7e30 start 2014-06-09 16:22:11.850988 7f86a5610700 10 _do_transaction on 0xbf88600 2014-06-09 16:22:11.850984 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.850992 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.850991 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851000 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851012 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.851007 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851013 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.851015 7f8689755700 5 queue_transactions existing osr(0.369 0x2bd6ff0)/0x2bd6ff0 2014-06-09 16:22:11.851017 7f8689755700 10 op_submit_start 3617 2014-06-09 16:22:11.851018 7f8689755700 5 queue_transactions (trailing journal) 3617 2014-06-09 16:22:11.851018 7f868e75f700 5 queue_transactions existing osr(3.609 0x3ed1b00)/0x3ed1b00 2014-06-09 16:22:11.851019 7f8689755700 5 queue_op 0xbe7d5c0 seq 3617 osr(0.369 0x2bd6ff0) 828 bytes (queue has 47 ops and 2548290 bytes) 2014-06-09 16:22:11.851022 7f8689755700 10 op_submit_finish 3617 2014-06-09 16:22:11.851019 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:190 2014-06-09 16:22:11.851028 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851027 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.603( v 213'555 (0'0,213'555] local-les=243 n=110 ec=118 les/c 243/243 242/242/220) 2014-06-09 16:22:11.851034 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851039 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.851042 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851046 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851056 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3573.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.851060 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.851062 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.851057 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851065 7f8689755700 5 queue_transactions existing osr(3.603 0x3ed1af0)/0x3ed1af0 2014-06-09 16:22:11.851067 7f8689755700 10 op_submit_start 3618 2014-06-09 16:22:11.851068 7f8689755700 5 queue_transactions (trailing journal) 3618 2014-06-09 16:22:11.851064 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851069 7f8689755700 5 queue_op 0xbe7df80 seq 3618 osr(3.603 0x3ed1af0) 828 bytes (queue has 49 ops and 2549946 bytes) 2014-06-09 16:22:11.851072 7f8689755700 10 op_submit_finish 3618 2014-06-09 16:22:11.851073 7f86a5610700 10 _do_op 0x30d96960 seq 3573 r = 0, finisher 0xbd6c120 0 2014-06-09 16:22:11.851076 7f86a5610700 10 _finish_op 0x30d96960 seq 3573 osr(0.67e 0x2bd7e30)/0x2bd7e30 2014-06-09 16:22:11.851072 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851079 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339560 (1 active) 2014-06-09 16:22:11.851081 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383e6c0 (2 active) 2014-06-09 16:22:11.851074 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.6e8( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.6e8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.851084 7f86a5610700 5 _do_op 0xbe7cea0 seq 3574 osr(1.4db 0x384d850)/0x384d850 start 2014-06-09 16:22:11.851080 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851086 7f86a5610700 10 _do_transaction on 0xbf36780 2014-06-09 16:22:11.851087 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851092 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.851098 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:206 2014-06-09 16:22:11.851106 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.851110 7f8689755700 5 queue_transactions existing osr(1.6e8 0x2ed0190)/0x2ed0190 2014-06-09 16:22:11.851112 7f8689755700 10 op_submit_start 3619 2014-06-09 16:22:11.851107 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851113 7f8689755700 5 queue_transactions (trailing journal) 3619 2014-06-09 16:22:11.851114 7f8689755700 5 queue_op 0xbe7d620 seq 3619 osr(1.6e8 0x2ed0190) 828 bytes (queue has 49 ops and 2549913 bytes) 2014-06-09 16:22:11.851117 7f8689755700 10 op_submit_finish 3619 2014-06-09 16:22:11.851114 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851122 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851119 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.5d9( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.851126 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851136 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851143 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851150 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851156 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.851157 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851164 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851172 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3574.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.851177 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.851175 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.851183 7f8690f64700 20 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851187 7f86a5610700 10 _do_op 0xbe7cea0 seq 3574 r = 0, finisher 0x2e0e120 0 2014-06-09 16:22:11.851191 7f86a5610700 10 _finish_op 0xbe7cea0 seq 3574 osr(1.4db 0x384d850)/0x384d850 2014-06-09 16:22:11.851193 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383e6c0 (1 active) 2014-06-09 16:22:11.851190 7f8690f64700 20 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851196 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca48ea0 (2 active) 2014-06-09 16:22:11.851197 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851198 7f86a5610700 5 _do_op 0x11fcefc0 seq 3575 osr(3.113 0x3ed0800)/0x3ed0800 start 2014-06-09 16:22:11.851200 7f86a5610700 10 _do_transaction on 0x97bb680 2014-06-09 16:22:11.851207 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.851201 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851212 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851219 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851226 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851233 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851240 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851251 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.851251 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.851260 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851270 7f86a5610700 15 _omap_rmkeyrange meta/1fe2f6ea/pglog_3.113/0//-1 [0000000213.00000000000000000650,4294967295.18446744073709551615] 2014-06-09 16:22:11.851267 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851274 7f86a5610700 15 get_omap_iterator meta/1fe2f6ea/pglog_3.113/0//-1 2014-06-09 16:22:11.851275 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851279 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851290 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851297 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851304 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851312 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851319 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851331 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.851339 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851346 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851354 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851358 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851369 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851375 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851382 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851388 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851394 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851404 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:185 2014-06-09 16:22:11.851413 7f86a5610700 15 _omap_rmkeys meta/1fe2f6ea/pglog_3.113/0//-1 2014-06-09 16:22:11.851411 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851417 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851424 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851428 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851439 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851446 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851453 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851460 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851467 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851475 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe2f6ea/pglog_3.113/0//-1 2014-06-09 16:22:11.851478 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:152 2014-06-09 16:22:11.851486 7f86a5610700 10 _omap_rmkeys meta/1fe2f6ea/pglog_3.113/0//-1 r = 0 2014-06-09 16:22:11.851491 7f86a5610700 15 _omap_rmkeys meta/1fe2f6ea/pglog_3.113/0//-1 2014-06-09 16:22:11.851487 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851496 7f86a5610700 10 _omap_rmkeys meta/1fe2f6ea/pglog_3.113/0//-1 r = 0 2014-06-09 16:22:11.851494 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851502 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851507 7f86a5610700 15 _omap_setkeys meta/1fe2f6ea/pglog_3.113/0//-1 2014-06-09 16:22:11.851506 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851516 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851523 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851533 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3575.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.851530 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851540 7f86a5610700 10 cid: oid: 1fe2f6ea/pglog_3.113/0//-1 skipping op, spos 3575.0.0 <= header.spos 145601.0.0 2014-06-09 16:22:11.851537 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851544 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.851544 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851555 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:170 2014-06-09 16:22:11.851563 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851569 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851577 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851580 7f86a5610700 10 _do_op 0x11fcefc0 seq 3575 r = 0, finisher 0x31435740 0 2014-06-09 16:22:11.851585 7f86a5610700 10 _finish_op 0x11fcefc0 seq 3575 osr(3.113 0x3ed0800)/0x3ed0800 2014-06-09 16:22:11.851581 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851588 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca48ea0 (1 active) 2014-06-09 16:22:11.851590 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6480 (2 active) 2014-06-09 16:22:11.851592 7f86a5610700 5 _do_op 0x2fc8c0c0 seq 3576 osr(3.1aa 0x3ed0ac0)/0x3ed0ac0 start 2014-06-09 16:22:11.851595 7f86a5610700 10 _do_transaction on 0xc1618c0 2014-06-09 16:22:11.851591 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851598 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851602 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.851604 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851611 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851623 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851635 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.851633 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.851641 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851647 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851654 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851652 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3576.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.851657 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.851661 7f868e75f700 10 op_submit_start 3620 2014-06-09 16:22:11.851664 7f868e75f700 5 queue_transactions (trailing journal) 3620 2014-06-09 16:22:11.851657 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851666 7f868e75f700 5 queue_op 0x308cba40 seq 3620 osr(3.609 0x3ed1b00) 828 bytes (queue has 47 ops and 2546336 bytes) 2014-06-09 16:22:11.851670 7f868e75f700 10 op_submit_finish 3620 2014-06-09 16:22:11.851670 7f86a5610700 10 _do_op 0x2fc8c0c0 seq 3576 r = 0, finisher 0xcc94060 0 2014-06-09 16:22:11.851668 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851674 7f86a5610700 10 _finish_op 0x2fc8c0c0 seq 3576 osr(3.1aa 0x3ed0ac0)/0x3ed0ac0 2014-06-09 16:22:11.851677 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6480 (1 active) 2014-06-09 16:22:11.851674 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851679 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922c60 (2 active) 2014-06-09 16:22:11.851681 7f86a5610700 5 _do_op 0x30d961e0 seq 3577 osr(3.15c 0x3ed0720)/0x3ed0720 start 2014-06-09 16:22:11.851684 7f86a5610700 10 _do_transaction on 0xc0d4780 2014-06-09 16:22:11.851673 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.2b( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.851681 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851689 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.851689 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851696 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851708 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.851713 7f868e75f700 5 queue_transactions existing osr(2.2b 0x2ed0230)/0x2ed0230 2014-06-09 16:22:11.851708 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.851716 7f868e75f700 10 op_submit_start 3621 2014-06-09 16:22:11.851717 7f868e75f700 5 queue_transactions (trailing journal) 3621 2014-06-09 16:22:11.851719 7f868e75f700 5 queue_op 0x308cbaa0 seq 3621 osr(2.2b 0x2ed0230) 826 bytes (queue has 47 ops and 2546109 bytes) 2014-06-09 16:22:11.851719 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.851722 7f868e75f700 10 op_submit_finish 3621 2014-06-09 16:22:11.851717 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851725 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851733 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851725 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.851735 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3577.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.851740 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.851737 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851740 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.851748 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851750 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.851755 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851760 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.851766 7f86a5610700 10 _do_op 0x30d961e0 seq 3577 r = 0, finisher 0x318c2a80 0 2014-06-09 16:22:11.851763 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851771 7f86a5610700 10 _finish_op 0x30d961e0 seq 3577 osr(3.15c 0x3ed0720)/0x3ed0720 2014-06-09 16:22:11.851774 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922c60 (1 active) 2014-06-09 16:22:11.851774 7f868e75f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.851771 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851776 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922360 (2 active) 2014-06-09 16:22:11.851779 7f86a5610700 5 _do_op 0x16097d40 seq 3578 osr(3.1e1 0x3ed09b0)/0x3ed09b0 start 2014-06-09 16:22:11.851782 7f86a5610700 10 _do_transaction on 0x390c9c0 2014-06-09 16:22:11.851778 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851788 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.851789 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:205 2014-06-09 16:22:11.851790 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:964 2014-06-09 16:22:11.851798 7f8690f64700 20 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851808 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.851805 7f8690f64700 20 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851813 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851812 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.620( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.851821 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.851817 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.851828 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.851835 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.851838 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3578.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.851843 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.851844 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.851842 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.851849 7f868e75f700 5 queue_transactions existing osr(2.620 0x2ed1cb0)/0x2ed1cb0 2014-06-09 16:22:11.851852 7f868e75f700 10 op_submit_start 3622 2014-06-09 16:22:11.851853 7f868e75f700 5 queue_transactions (trailing journal) 3622 2014-06-09 16:22:11.851854 7f868e75f700 5 queue_op 0x30606060 seq 3622 osr(2.620 0x2ed1cb0) 828 bytes (queue has 47 ops and 2545884 bytes) 2014-06-09 16:22:11.851850 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.851857 7f868e75f700 10 op_submit_finish 3622 2014-06-09 16:22:11.851858 7f86a5610700 10 _do_op 0x16097d40 seq 3578 r = 0, finisher 0xc2d74e0 0 2014-06-09 16:22:11.851862 7f86a5610700 10 _finish_op 0x16097d40 seq 3578 osr(3.1e1 0x3ed09b0)/0x3ed09b0 2014-06-09 16:22:11.851857 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.851865 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922360 (1 active) 2014-06-09 16:22:11.851867 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6fc0 (2 active) 2014-06-09 16:22:11.851860 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.851869 7f86a5610700 5 _do_op 0xc2d5ec0 seq 3579 osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 start 2014-06-09 16:22:11.851872 7f86a5610700 10 _do_transaction on 0x398ac00 2014-06-09 16:22:11.851869 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:161 2014-06-09 16:22:11.851877 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.851873 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.851878 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.851883 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.851886 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.851894 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.851891 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.851903 7f868e75f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.851909 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358340 (19 active) 2014-06-09 16:22:11.851912 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.851916 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.851918 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:192 2014-06-09 16:22:11.851927 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3579.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.851931 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.851936 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.851920 7f8690f64700 10 osd.26 243 dequeue_op 0x16059100 prio 63 cost 516096 latency 1.556190 osd_op(client.4567.0:35113 rbd_data.11776b8b4567.00000000000017ad [set-alloc-hint object_size 4194304 write_size 4194304,write 385024~516096] 3.8984db19 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.851942 7f86a5610700 10 _do_op 0xc2d5ec0 seq 3579 r = 0, finisher 0xccf91c0 0 2014-06-09 16:22:11.851943 7f868b759700 10 op_submit_start 3623 2014-06-09 16:22:11.851945 7f868b759700 5 queue_transactions (trailing journal) 3623 2014-06-09 16:22:11.851945 7f86a5610700 10 _finish_op 0xc2d5ec0 seq 3579 osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 2014-06-09 16:22:11.851946 7f868b759700 5 queue_op 0x324f58c0 seq 3623 osr(2.6cd 0x2ed1e70) 828 bytes (queue has 46 ops and 2544455 bytes) 2014-06-09 16:22:11.851940 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.851948 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6fc0 (1 active) 2014-06-09 16:22:11.851951 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6ddd40 (2 active) 2014-06-09 16:22:11.851958 7f868b759700 10 op_submit_finish 3623 2014-06-09 16:22:11.851952 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.851961 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.851969 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.851963 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.676( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.676( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.851978 7f868df5e700 10 op_submit_start 3624 2014-06-09 16:22:11.851980 7f868df5e700 5 queue_transactions (trailing journal) 3624 2014-06-09 16:22:11.851981 7f868e75f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.851982 7f868df5e700 5 queue_op 0xc705080 seq 3624 osr(1.5e 0x384c5d0) 826 bytes (queue has 46 ops and 2544453 bytes) 2014-06-09 16:22:11.851985 7f868df5e700 10 op_submit_finish 3624 2014-06-09 16:22:11.851992 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.851989 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.691( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.691( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.852000 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.852006 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.852006 7f868b759700 5 queue_transactions existing osr(1.676 0x384dee0)/0x384dee0 2014-06-09 16:22:11.852009 7f868b759700 10 op_submit_start 3625 2014-06-09 16:22:11.852011 7f868b759700 5 queue_transactions (trailing journal) 3625 2014-06-09 16:22:11.852012 7f868b759700 5 queue_op 0x324f4ae0 seq 3625 osr(1.676 0x384dee0) 828 bytes (queue has 47 ops and 2545281 bytes) 2014-06-09 16:22:11.852015 7f868b759700 10 op_submit_finish 3625 2014-06-09 16:22:11.852010 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.113( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.852022 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.852026 7f868df5e700 5 queue_transactions existing osr(1.691 0x384de30)/0x384de30 2014-06-09 16:22:11.852018 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.56( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=16-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.56( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.852029 7f868df5e700 10 op_submit_start 3626 2014-06-09 16:22:11.852030 7f868df5e700 5 queue_transactions (trailing journal) 3626 2014-06-09 16:22:11.852031 7f868df5e700 5 queue_op 0xc705620 seq 3626 osr(1.691 0x384de30) 828 bytes (queue has 48 ops and 2546109 bytes) 2014-06-09 16:22:11.852034 7f868df5e700 10 op_submit_finish 3626 2014-06-09 16:22:11.852045 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.852046 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.852036 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.852050 7f868e75f700 5 queue_transactions existing osr(2.113 0x2ed06b0)/0x2ed06b0 2014-06-09 16:22:11.852050 7f868b759700 5 queue_transactions existing osr(1.56 0x384c5f0)/0x384c5f0 2014-06-09 16:22:11.852052 7f868e75f700 10 op_submit_start 3627 2014-06-09 16:22:11.852054 7f868e75f700 5 queue_transactions (trailing journal) 3627 2014-06-09 16:22:11.852055 7f868e75f700 5 queue_op 0x324f5680 seq 3627 osr(2.113 0x2ed06b0) 828 bytes (queue has 50 ops and 2547763 bytes) 2014-06-09 16:22:11.852058 7f868e75f700 10 op_submit_finish 3627 2014-06-09 16:22:11.852064 7f868b759700 10 op_submit_start 3628 2014-06-09 16:22:11.852066 7f868b759700 5 queue_transactions (trailing journal) 3628 2014-06-09 16:22:11.852067 7f868b759700 5 queue_op 0x324f4960 seq 3628 osr(1.56 0x384c5f0) 826 bytes (queue has 50 ops and 2547763 bytes) 2014-06-09 16:22:11.852051 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.852070 7f868b759700 10 op_submit_finish 3628 2014-06-09 16:22:11.852064 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.dd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.852072 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.852074 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.7e2( v 213'680 (0'0,213'680] local-les=243 n=131 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:11.852081 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.852094 7f868df5e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.852099 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.852103 7f868e75f700 5 queue_transactions existing osr(0.dd 0x2bd6550)/0x2bd6550 2014-06-09 16:22:11.852105 7f868e75f700 2 waiting 51 > 50 ops || 2548589 > 104857600 2014-06-09 16:22:11.852110 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.852114 7f868b759700 5 queue_transactions existing osr(3.7e2 0xb7f0120)/0xb7f0120 2014-06-09 16:22:11.852107 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1162 2014-06-09 16:22:11.852117 7f868b759700 2 waiting 51 > 50 ops || 2548591 > 104857600 2014-06-09 16:22:11.850112 7f866f5f0700 5 -- op tracker -- , seq: 3314, time: 2014-06-09 16:22:11.849291, event: throttled, op: osd_op(client.4539.0:34417 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 2760704~516096] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.852122 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.852127 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.382( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.852134 7f868ef60700 5 -- op tracker -- , seq: 3275, time: 2014-06-09 16:22:11.850422, event: reached_pg, op: osd_op(client.4539.0:34389 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1474560~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.852157 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.852162 7f868df5e700 5 queue_transactions existing osr(1.382 0x384d5b0)/0x384d5b0 2014-06-09 16:22:11.852165 7f868df5e700 2 waiting 51 > 50 ops || 2548591 > 104857600 2014-06-09 16:22:11.852163 7f8690f64700 5 -- op tracker -- , seq: 236, time: 2014-06-09 16:22:11.851940, event: reached_pg, op: osd_op(client.4567.0:35113 rbd_data.11776b8b4567.00000000000017ad [set-alloc-hint object_size 4194304 write_size 4194304,write 385024~516096] 3.8984db19 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.852157 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.852170 7f868ff62700 2 waiting 51 > 50 ops || 2548591 > 104857600 2014-06-09 16:22:11.852172 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 1 flushes_in_progress pending waiting for active on 0xdfa7d00 2014-06-09 16:22:11.852172 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.852182 7f868ef60700 10 osd.26 243 dequeue_op 0xdfa7d00 finish 2014-06-09 16:22:11.852185 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.852184 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x16059100 2014-06-09 16:22:11.852205 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:11.852195 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4567.0:35113 rbd_data.11776b8b4567.00000000000017ad [set-alloc-hint object_size 4194304 write_size 4194304,write 385024~516096] 3.8984db19 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.852209 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.3a9( v 213'584 (0'0,213'584] local-les=243 n=115 ec=118 les/c 243/243 242/242/200) 2014-06-09 16:22:11.852215 7f8690f64700 3 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4567.0:35113 was 214'605 2014-06-09 16:22:11.852241 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.852242 7f8690f64700 10 osd.26 243 dequeue_op 0x16059100 finish 2014-06-09 16:22:11.852245 7f868ef60700 5 queue_transactions existing osr(3.3a9 0x3ed0f60)/0x3ed0f60 2014-06-09 16:22:11.852247 7f868ef60700 2 waiting 51 > 50 ops || 2548591 > 104857600 2014-06-09 16:22:11.852251 7f8690f64700 5 -- op tracker -- , seq: 236, time: 2014-06-09 16:22:11.852245, event: done, op: osd_op(client.4567.0:35113 rbd_data.11776b8b4567.00000000000017ad [set-alloc-hint object_size 4194304 write_size 4194304,write 385024~516096] 3.8984db19 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.852255 7f86a5610700 5 _do_op 0x30d97d40 seq 3580 osr(3.555 0x3ed15b0)/0x3ed15b0 start 2014-06-09 16:22:11.852258 7f86a5610700 10 _do_transaction on 0x398b440 2014-06-09 16:22:11.852264 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.852280 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.852288 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.852302 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.852292 7f8690f64700 10 osd.26 243 dequeue_op 0x1c54ba00 prio 63 cost 516096 latency 1.356474 osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.852318 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3580.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.852315 7f8690f64700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.852310, event: reached_pg, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.852322 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.852323 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.852334 7f86a5610700 10 _do_op 0x30d97d40 seq 3580 r = 0, finisher 0xbd606e0 0 2014-06-09 16:22:11.852337 7f86a5610700 10 _finish_op 0x30d97d40 seq 3580 osr(3.555 0x3ed15b0)/0x3ed15b0 2014-06-09 16:22:11.852335 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1c54ba00 2014-06-09 16:22:11.852343 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6ddd40 (1 active) 2014-06-09 16:22:11.852345 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4900 (2 active) 2014-06-09 16:22:11.852347 7f86a5610700 5 _do_op 0x241040c0 seq 3581 osr(3.4b9 0x3ed1460)/0x3ed1460 start 2014-06-09 16:22:11.852349 7f86a5610700 10 _do_transaction on 0x37e4480 2014-06-09 16:22:11.852348 7f868e75f700 10 op_submit_start 3629 2014-06-09 16:22:11.852351 7f868e75f700 5 queue_transactions (trailing journal) 3629 2014-06-09 16:22:11.852352 7f868e75f700 5 queue_op 0x324f5740 seq 3629 osr(0.dd 0x2bd6550) 826 bytes (queue has 50 ops and 2547254 bytes) 2014-06-09 16:22:11.852355 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.852356 7f868e75f700 10 op_submit_finish 3629 2014-06-09 16:22:11.852344 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.852359 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.501( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.852365 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: found obc in cache: 0x30df7b80 2014-06-09 16:22:11.852387 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.852376 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30df7b80 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 rwstate(write n=1 w=0) oi: 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3(243'608 client.4569.0:38637 wrlock_by=unknown.0.0:0 dirty s 61440 uv608) ssc: 0x2e876e00 snapset: 0=[]:[]+head 2014-06-09 16:22:11.852394 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.852399 7f868e75f700 5 queue_transactions existing osr(2.501 0x2ed1690)/0x2ed1690 2014-06-09 16:22:11.852402 7f868e75f700 2 waiting 51 > 50 ops || 2548082 > 104857600 2014-06-09 16:22:11.852392 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 @head oi=8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3(243'608 client.4569.0:38637 wrlock_by=unknown.0.0:0 dirty s 61440 uv608) 2014-06-09 16:22:11.852403 7f86a5610700 15 _omap_rmkeyrange meta/1fe064b5/pglog_3.4b9/0//-1 [0000000214.00000000000000000556,4294967295.18446744073709551615] 2014-06-09 16:22:11.852406 7f86a5610700 15 get_omap_iterator meta/1fe064b5/pglog_3.4b9/0//-1 2014-06-09 16:22:11.852408 7f868cf5c700 2 waiting 51 > 50 ops || 2548082 > 104857600 2014-06-09 16:22:11.852414 7f8692f68700 2 waiting 51 > 50 ops || 2549263 > 104857600 2014-06-09 16:22:11.852419 7f868a757700 2 waiting 51 > 50 ops || 2548080 > 104857600 2014-06-09 16:22:11.852423 7f868b759700 2 waiting 51 > 50 ops || 2548082 > 104857600 2014-06-09 16:22:11.852430 7f868df5e700 2 waiting 51 > 50 ops || 2548082 > 104857600 2014-06-09 16:22:11.852417 7f8690f64700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.852412, event: started, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.852437 7f868ff62700 2 waiting 51 > 50 ops || 2548082 > 104857600 2014-06-09 16:22:11.852441 7f868ef60700 2 waiting 51 > 50 ops || 2548082 > 104857600 2014-06-09 16:22:11.852435 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30641e00 2014-06-09 16:22:11.852453 7f8690f64700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.852448, event: started, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.852459 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] ov 243'608 av 243'609 snapc 0=[] snapset 0=[]:[]+head 2014-06-09 16:22:11.852473 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 2014-06-09 16:22:11.852490 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.852484 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.852493 7f86a5610700 15 _omap_rmkeys meta/1fe064b5/pglog_3.4b9/0//-1 2014-06-09 16:22:11.852497 7f8689755700 5 queue_transactions existing osr(2.5d9 0x2ed1a70)/0x2ed1a70 2014-06-09 16:22:11.852500 7f8689755700 2 waiting 51 > 50 ops || 2548082 > 104857600 2014-06-09 16:22:11.852497 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 61440~516096 2014-06-09 16:22:11.852513 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228c00 2014-06-09 16:22:11.852526 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe064b5/pglog_3.4b9/0//-1 2014-06-09 16:22:11.852525 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 snapset=0x2e876e38 snapc=0=[] 2014-06-09 16:22:11.852534 7f86a5610700 10 _omap_rmkeys meta/1fe064b5/pglog_3.4b9/0//-1 r = 0 2014-06-09 16:22:11.852537 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.852538 7f86a5610700 15 _omap_rmkeys meta/1fe064b5/pglog_3.4b9/0//-1 2014-06-09 16:22:11.852551 7f86a5610700 10 _omap_rmkeys meta/1fe064b5/pglog_3.4b9/0//-1 r = 0 2014-06-09 16:22:11.852547 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 0x30641e00 op modify 2014-06-09 16:22:11.852559 7f86a5610700 15 _omap_setkeys meta/1fe064b5/pglog_3.4b9/0//-1 2014-06-09 16:22:11.852559 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:00.652425 2014-06-09 16:22:11.852576 7f86a5610700 10 cid: oid: 1fe064b5/pglog_3.4b9/0//-1 skipping op, spos 3581.0.0 <= header.spos 150415.0.0 2014-06-09 16:22:11.852581 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3581.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.852584 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.852582 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.852598 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.852614 7f86a5610700 10 _do_op 0x241040c0 seq 3581 r = 0, finisher 0x3206ef00 0 2014-06-09 16:22:11.852617 7f86a5610700 10 _finish_op 0x241040c0 seq 3581 osr(3.4b9 0x3ed1460)/0x3ed1460 2014-06-09 16:22:11.852609 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 16 on osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.852625 7f8690f64700 7 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 16 o 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.852632 7f866f5f0700 5 -- op tracker -- , seq: 3314, time: 2014-06-09 16:22:11.850064, event: all_read, op: osd_op(client.4539.0:34417 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 2760704~516096] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.852643 7f866f5f0700 5 -- op tracker -- , seq: 3314, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34417 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 2760704~516096] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.852653 7f8690f64700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.852643, event: waiting for subops from 5, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.852671 7f8690f64700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.852711 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.852735 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827500 prio 63 cost 516096 latency 0.003444 osd_op(client.4539.0:34417 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 2760704~516096] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.852690 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'608 (0'0,243'608] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,243'608], crt=0'0) [243'609 (243'608) modify 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 by client.4569.0:38638 2014-06-09 10:20:00.652425] 2014-06-09 16:22:11.852753 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'609 (243'608) modify 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 by client.4569.0:38638 2014-06-09 10:20:00.652425 2014-06-09 16:22:11.852778 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.852809 7f8690f64700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'609, trimmed: 2014-06-09 16:22:11.852830 7f8690f64700 5 queue_transactions existing osr(3.319 0x3ed0d60)/0x3ed0d60 2014-06-09 16:22:11.853393 7f8682ae5700 5 -- op tracker -- , seq: 3315, time: 2014-06-09 16:22:11.853355, event: header_read, op: osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.853403 7f8682ae5700 5 -- op tracker -- , seq: 3315, time: 2014-06-09 16:22:11.853357, event: throttled, op: osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.853410 7f8682ae5700 5 -- op tracker -- , seq: 3315, time: 2014-06-09 16:22:11.853367, event: all_read, op: osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.853418 7f8682ae5700 5 -- op tracker -- , seq: 3315, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.853422 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.853427 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:11.853434 7f8682ae5700 15 osd.26 243 enqueue_op 0x32826500 prio 196 cost 0 latency 0.000079 osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.853599 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.853661 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.853680 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3561.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.853684 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.853700 7f86a5e11700 10 _do_op 0xc2ead80 seq 3561 r = 0, finisher 0x318c3f00 0 2014-06-09 16:22:11.854199 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.854202 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.854204 7f869af78700 20 osd.26 243 _dispatch 0x30d9ea80 pg_info(1 pgs e243:3.55b) v4 2014-06-09 16:22:11.854219 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4900 (1 active) 2014-06-09 16:22:11.854222 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338ea0 (2 active) 2014-06-09 16:22:11.854223 7f868cf5c700 10 op_submit_start 3630 2014-06-09 16:22:11.854225 7f868cf5c700 5 queue_transactions (trailing journal) 3630 2014-06-09 16:22:11.854226 7f86a5610700 5 _do_op 0xc704b40 seq 3582 osr(3.319 0x3ed0d60)/0x3ed0d60 start 2014-06-09 16:22:11.854229 7f86a5610700 10 _do_transaction on 0xc31d800 2014-06-09 16:22:11.854227 7f868cf5c700 5 queue_op 0x24105c80 seq 3630 osr(1.4db 0x384d850) 828 bytes (queue has 50 ops and 2546139 bytes) 2014-06-09 16:22:11.854230 7f86a5e11700 10 _finish_op 0xc2ead80 seq 3561 osr(3.70e 0x3ed1ba0)/0x3ed1ba0 2014-06-09 16:22:11.854233 7f8692f68700 2 waiting 51 > 50 ops || 2548148 > 104857600 2014-06-09 16:22:11.854235 7f86a5610700 15 _omap_setkeys meta/1fe22c6a/pglog_3.319/0//-1 2014-06-09 16:22:11.859140 7f868a757700 2 waiting 51 > 50 ops || 2546965 > 104857600 2014-06-09 16:22:11.854210 7f869af78700 5 -- op tracker -- , seq: 3316, time: 2014-06-09 16:22:11.854148, event: header_read, op: pg_info(1 pgs e243:3.55b) 2014-06-09 16:22:11.859149 7f868b759700 2 waiting 51 > 50 ops || 2546967 > 104857600 2014-06-09 16:22:11.859158 7f868df5e700 2 waiting 51 > 50 ops || 2546967 > 104857600 2014-06-09 16:22:11.859154 7f869af78700 5 -- op tracker -- , seq: 3316, time: 2014-06-09 16:22:11.854149, event: throttled, op: pg_info(1 pgs e243:3.55b) 2014-06-09 16:22:11.859165 7f868ff62700 2 waiting 51 > 50 ops || 2546967 > 104857600 2014-06-09 16:22:11.859166 7f869af78700 5 -- op tracker -- , seq: 3316, time: 2014-06-09 16:22:11.854166, event: all_read, op: pg_info(1 pgs e243:3.55b) 2014-06-09 16:22:11.859171 7f868ef60700 2 waiting 51 > 50 ops || 2546967 > 104857600 2014-06-09 16:22:11.859173 7f869af78700 5 -- op tracker -- , seq: 3316, time: 2014-06-09 16:22:11.854198, event: dispatched, op: pg_info(1 pgs e243:3.55b) 2014-06-09 16:22:11.859181 7f869af78700 5 -- op tracker -- , seq: 3316, time: 2014-06-09 16:22:11.859179, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.55b) 2014-06-09 16:22:11.859185 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.55b) v4 from osd.4 2014-06-09 16:22:11.859190 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9ea80 2014-06-09 16:22:11.859196 7f869af78700 5 -- op tracker -- , seq: 3316, time: 2014-06-09 16:22:11.859194, event: started, op: pg_info(1 pgs e243:3.55b) 2014-06-09 16:22:11.859218 7f869af78700 5 -- op tracker -- , seq: 3316, time: 2014-06-09 16:22:11.859216, event: done, op: pg_info(1 pgs e243:3.55b) 2014-06-09 16:22:11.859228 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.859228 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe22c6a/pglog_3.319/0//-1 2014-06-09 16:22:11.859230 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.859249 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.859246 7f866f5f0700 5 -- op tracker -- , seq: 3317, time: 2014-06-09 16:22:11.852759, event: header_read, op: osd_op(client.4539.0:34418 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 3276800~917504] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.859261 7f8689755700 2 waiting 51 > 50 ops || 2546967 > 104857600 2014-06-09 16:22:11.859270 7f868e75f700 2 waiting 51 > 50 ops || 2546967 > 104857600 2014-06-09 16:22:11.859270 7f866f5f0700 5 -- op tracker -- , seq: 3317, time: 2014-06-09 16:22:11.852760, event: throttled, op: osd_op(client.4539.0:34418 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 3276800~917504] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.859278 7f868bf5a700 2 waiting 51 > 50 ops || 2546965 > 104857600 2014-06-09 16:22:11.859280 7f866f5f0700 5 -- op tracker -- , seq: 3317, time: 2014-06-09 16:22:11.854417, event: all_read, op: osd_op(client.4539.0:34418 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 3276800~917504] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.859287 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.859293 7f866f5f0700 5 -- op tracker -- , seq: 3317, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34418 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 3276800~917504] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.859302 7f86a5610700 15 _omap_rmkeys meta/1fe22c6a/pglog_3.319/0//-1 2014-06-09 16:22:11.859299 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.859307 7f86a5610700 10 _omap_rmkeys meta/1fe22c6a/pglog_3.319/0//-1 r = 0 2014-06-09 16:22:11.859313 7f86a5610700 15 _omap_setkeys meta/1fe22c6a/pglog_3.319/0//-1 2014-06-09 16:22:11.859326 7f86a5610700 15 _touch 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.859321 7f866f5f0700 15 osd.26 243 enqueue_op 0xdfa7f00 prio 63 cost 917504 latency 0.006561 osd_op(client.4539.0:34418 rbd_data.115b6b8b4567.00000000000018ad [set-alloc-hint object_size 4194304 write_size 4194304,write 3276800~917504] 3.f02c7300 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.859363 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.319_head oid 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.859389 7f86a5610700 20 write_state seq is 30155 2014-06-09 16:22:11.859397 7f86a5610700 20 set_header setting 30154 cid 3.319_head oid 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 parent seq 0 2014-06-09 16:22:11.859410 7f86a5610700 10 _touch 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 = 0 2014-06-09 16:22:11.859417 7f86a5610700 15 _write 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 0~61440 2014-06-09 16:22:11.859418 7f8690f64700 2 waiting 51 > 50 ops || 3064528 > 104857600 2014-06-09 16:22:11.859423 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.859442 7f8691f66700 2 waiting 51 > 50 ops || 2546997 > 104857600 2014-06-09 16:22:11.859604 7f8682be6700 5 -- op tracker -- , seq: 3318, time: 2014-06-09 16:22:11.859567, event: header_read, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) 2014-06-09 16:22:11.859623 7f8682be6700 5 -- op tracker -- , seq: 3318, time: 2014-06-09 16:22:11.859568, event: throttled, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) 2014-06-09 16:22:11.859631 7f8682be6700 5 -- op tracker -- , seq: 3318, time: 2014-06-09 16:22:11.859580, event: all_read, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) 2014-06-09 16:22:11.859640 7f8682be6700 5 -- op tracker -- , seq: 3318, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) 2014-06-09 16:22:11.859639 7f86a5610700 10 _generic_write 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 0~61440 = 0 2014-06-09 16:22:11.859645 7f8682be6700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) v2 epoch 243 2014-06-09 16:22:11.859651 7f86a5610700 15 _setattrs 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.859650 7f8682be6700 20 osd.26 243 should_share_map osd.5 172.16.96.201:6837/51639 243 2014-06-09 16:22:11.859661 7f86a5610700 10 _setattrs 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 = 0 2014-06-09 16:22:11.859657 7f8682be6700 15 osd.26 243 enqueue_op 0x32826000 prio 196 cost 0 latency 0.000090 osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) v2 2014-06-09 16:22:11.859667 7f86a5610700 15 _setattrs 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.859675 7f86a5610700 10 _setattrs 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 = 0 2014-06-09 16:22:11.859689 7f86a5610700 10 cid: oid: 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 not skipping op, *spos 3582.0.0 2014-06-09 16:22:11.859692 7f86a5610700 10 > header.spos 0.0.0 2014-06-09 16:22:11.859697 7f86a5610700 20 set_header setting 30154 cid 3.319_head oid 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 parent seq 0 2014-06-09 16:22:11.859713 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3582.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.859718 7f86a5610700 10 cid: oid: 1fe22c6a/pglog_3.319/0//-1 skipping op, spos 3582.0.0 <= header.spos 150438.0.0 2014-06-09 16:22:11.859709 7f8682be6700 5 -- op tracker -- , seq: 3319, time: 2014-06-09 16:22:11.859674, event: header_read, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.859721 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.859725 7f8682be6700 5 -- op tracker -- , seq: 3319, time: 2014-06-09 16:22:11.859675, event: throttled, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.859733 7f8682be6700 5 -- op tracker -- , seq: 3319, time: 2014-06-09 16:22:11.859686, event: all_read, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.859741 7f8682be6700 5 -- op tracker -- , seq: 3319, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.859744 7f8682be6700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.859750 7f8682be6700 20 osd.26 243 should_share_map osd.5 172.16.96.201:6837/51639 243 2014-06-09 16:22:11.859755 7f8682be6700 15 osd.26 243 enqueue_op 0x32827b00 prio 196 cost 0 latency 0.000081 osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.859994 7f86a5610700 10 _do_op 0xc704b40 seq 3582 r = 0, finisher 0xbd7c9c0 0x447c5c0 2014-06-09 16:22:11.861649 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcada5a0 (1 active) 2014-06-09 16:22:11.861639 7f866f5f0700 5 -- op tracker -- , seq: 3320, time: 2014-06-09 16:22:11.859351, event: header_read, op: osd_op(client.4539.0:34481 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1912832] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.861651 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.861655 7f86a5e11700 5 _do_op 0x2fc8c8a0 seq 3583 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.861654 7f868b759700 2 waiting 51 > 50 ops || 2547641 > 104857600 2014-06-09 16:22:11.861660 7f86a5e11700 10 _do_transaction on 0xc23fd40 2014-06-09 16:22:11.861661 7f868cf5c700 10 op_submit_finish 3630 2014-06-09 16:22:11.861664 7f868a757700 2 waiting 51 > 50 ops || 2547639 > 104857600 2014-06-09 16:22:11.861666 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.861663 7f866f5f0700 5 -- op tracker -- , seq: 3320, time: 2014-06-09 16:22:11.859353, event: throttled, op: osd_op(client.4539.0:34481 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1912832] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.861671 7f868ff62700 2 waiting 51 > 50 ops || 2547641 > 104857600 2014-06-09 16:22:11.861677 7f868ef60700 2 waiting 51 > 50 ops || 2547641 > 104857600 2014-06-09 16:22:11.861674 7f866f5f0700 5 -- op tracker -- , seq: 3320, time: 2014-06-09 16:22:11.861615, event: all_read, op: osd_op(client.4539.0:34481 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1912832] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.861670 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.861683 7f8692f68700 10 op_submit_start 3631 2014-06-09 16:22:11.861684 7f8692f68700 5 queue_transactions (trailing journal) 3631 2014-06-09 16:22:11.861684 7f8689755700 2 waiting 51 > 50 ops || 2547641 > 104857600 2014-06-09 16:22:11.861687 7f866f5f0700 5 -- op tracker -- , seq: 3320, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34481 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1912832] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.861686 7f8692f68700 5 queue_op 0x324e6d80 seq 3631 osr(0.5fb 0x2bd79a0) 2009 bytes (queue has 50 ops and 2546813 bytes) 2014-06-09 16:22:11.861690 7f8692f68700 10 op_submit_finish 3631 2014-06-09 16:22:11.861690 7f868e75f700 2 waiting 51 > 50 ops || 2547641 > 104857600 2014-06-09 16:22:11.861687 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.861694 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.861696 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.861695 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.861704 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.861721 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.861712 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.861718 7f866f5f0700 15 osd.26 243 enqueue_op 0xdfa7c00 prio 63 cost 1912832 latency 0.002367 osd_op(client.4539.0:34481 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1912832] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.861724 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.861734 7f86a5e11700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.861733 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.861741 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.861750 7f8692f68700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.861751 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.861758 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.861764 7f868df5e700 2 waiting 51 > 50 ops || 2547641 > 104857600 2014-06-09 16:22:11.861772 7f8690f64700 2 waiting 51 > 50 ops || 3065202 > 104857600 2014-06-09 16:22:11.861769 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1034 2014-06-09 16:22:11.861781 7f8691f66700 2 waiting 51 > 50 ops || 2547671 > 104857600 2014-06-09 16:22:11.861787 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.861789 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.861792 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.308( v 214'552 lc 214'547 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.861807 7f86a5e11700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.861813 7f86a5e11700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.861806 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:11.861808 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.663( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.663( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.861819 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.861817 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.861826 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.861833 7f86a5e11700 15 _write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 3313664~516096 2014-06-09 16:22:11.861840 7f8692f68700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.861845 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.861852 7f868cf5c700 5 queue_transactions existing osr(0.663 0x2bd7f30)/0x2bd7f30 2014-06-09 16:22:11.861855 7f868cf5c700 2 waiting 51 > 50 ops || 2547641 > 104857600 2014-06-09 16:22:11.861860 7f86a5610700 10 _finish_op 0xc704b40 seq 3582 osr(3.319 0x3ed0d60)/0x3ed0d60 2014-06-09 16:22:11.861856 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:902 2014-06-09 16:22:11.861871 7f868a757700 10 op_submit_start 3632 2014-06-09 16:22:11.861873 7f868a757700 5 queue_transactions (trailing journal) 3632 2014-06-09 16:22:11.861874 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.861874 7f868a757700 5 queue_op 0x3280b3e0 seq 3632 osr(1.a2 0x384c790) 826 bytes (queue has 50 ops and 2483801 bytes) 2014-06-09 16:22:11.861881 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338ea0 (1 active) 2014-06-09 16:22:11.861884 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62eb40 (2 active) 2014-06-09 16:22:11.861889 7f868a757700 10 op_submit_finish 3632 2014-06-09 16:22:11.861879 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 9 notify: (query_epoch:243, epoch_sent:243, info:0.5ed( empty local-les=213 n=0 ec=1 les/c 213/213 242/242/242)) 2014-06-09 16:22:11.861893 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 9, already have info from that osd, ignoring 2014-06-09 16:22:11.861892 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.372( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.372( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.861905 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.861933 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.861935 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (19 active) 2014-06-09 16:22:11.861937 7f868a757700 5 queue_transactions existing osr(2.372 0x2ed1150)/0x2ed1150 2014-06-09 16:22:11.861940 7f868a757700 2 waiting 51 > 50 ops || 2484629 > 104857600 2014-06-09 16:22:11.861944 7f86a5610700 5 _do_op 0x324e7080 seq 3584 osr(2.364 0x2ed11b0)/0x2ed11b0 start 2014-06-09 16:22:11.861947 7f86a5610700 10 _do_transaction on 0x390c6c0 2014-06-09 16:22:11.861953 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.861959 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (20 active) 2014-06-09 16:22:11.861963 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.73( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.73( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.862006 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.862012 7f8692f68700 5 queue_transactions existing osr(2.73 0x2ed0510)/0x2ed0510 2014-06-09 16:22:11.862014 7f8692f68700 2 waiting 51 > 50 ops || 2484627 > 104857600 2014-06-09 16:22:11.862050 7f868ff62700 2 waiting 51 > 50 ops || 2484629 > 104857600 2014-06-09 16:22:11.862061 7f868ef60700 2 waiting 51 > 50 ops || 2484629 > 104857600 2014-06-09 16:22:11.862067 7f8689755700 2 waiting 51 > 50 ops || 2484629 > 104857600 2014-06-09 16:22:11.862073 7f868e75f700 2 waiting 51 > 50 ops || 2484629 > 104857600 2014-06-09 16:22:11.862080 7f868df5e700 2 waiting 51 > 50 ops || 2484629 > 104857600 2014-06-09 16:22:11.862087 7f8690f64700 2 waiting 51 > 50 ops || 3002190 > 104857600 2014-06-09 16:22:11.862093 7f8691f66700 2 waiting 51 > 50 ops || 2484659 > 104857600 2014-06-09 16:22:11.862100 7f868cf5c700 2 waiting 51 > 50 ops || 2484629 > 104857600 2014-06-09 16:22:11.862106 7f868b759700 2 waiting 51 > 50 ops || 2484629 > 104857600 2014-06-09 16:22:11.862112 7f868bf5a700 2 waiting 51 > 50 ops || 2484627 > 104857600 2014-06-09 16:22:11.862159 7f86a5e11700 10 lookup_strip_header done cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.862177 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.862724 7f866f5f0700 5 -- op tracker -- , seq: 3321, time: 2014-06-09 16:22:11.861744, event: header_read, op: osd_op(client.4539.0:34482 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 1912832~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.862745 7f866f5f0700 5 -- op tracker -- , seq: 3321, time: 2014-06-09 16:22:11.861746, event: throttled, op: osd_op(client.4539.0:34482 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 1912832~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.862756 7f866f5f0700 5 -- op tracker -- , seq: 3321, time: 2014-06-09 16:22:11.862660, event: all_read, op: osd_op(client.4539.0:34482 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 1912832~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.862772 7f866f5f0700 5 -- op tracker -- , seq: 3321, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34482 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 1912832~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.862778 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.862799 7f866f5f0700 15 osd.26 243 enqueue_op 0x2f097c00 prio 63 cost 516096 latency 0.001054 osd_op(client.4539.0:34482 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 1912832~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.864122 7f866f5f0700 5 -- op tracker -- , seq: 3322, time: 2014-06-09 16:22:11.862823, event: header_read, op: osd_op(client.4539.0:34483 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2428928~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.864138 7f866f5f0700 5 -- op tracker -- , seq: 3322, time: 2014-06-09 16:22:11.862824, event: throttled, op: osd_op(client.4539.0:34483 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2428928~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.864148 7f866f5f0700 5 -- op tracker -- , seq: 3322, time: 2014-06-09 16:22:11.864091, event: all_read, op: osd_op(client.4539.0:34483 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2428928~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.864159 7f866f5f0700 5 -- op tracker -- , seq: 3322, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34483 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2428928~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.864166 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.864258 7f866f5f0700 15 osd.26 243 enqueue_op 0x2c06ae00 prio 63 cost 516096 latency 0.001435 osd_op(client.4539.0:34483 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2428928~516096] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.866045 7f866f5f0700 5 -- op tracker -- , seq: 3323, time: 2014-06-09 16:22:11.864283, event: header_read, op: osd_op(client.4539.0:34484 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~1032192] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.866059 7f866f5f0700 5 -- op tracker -- , seq: 3323, time: 2014-06-09 16:22:11.864284, event: throttled, op: osd_op(client.4539.0:34484 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~1032192] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.866073 7f866f5f0700 5 -- op tracker -- , seq: 3323, time: 2014-06-09 16:22:11.866006, event: all_read, op: osd_op(client.4539.0:34484 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~1032192] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.866085 7f866f5f0700 5 -- op tracker -- , seq: 3323, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34484 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~1032192] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.866163 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.866187 7f866f5f0700 15 osd.26 243 enqueue_op 0x2c06af00 prio 63 cost 1032192 latency 0.001903 osd_op(client.4539.0:34484 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 2945024~1032192] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.866729 7f866f5f0700 5 -- op tracker -- , seq: 3324, time: 2014-06-09 16:22:11.866213, event: header_read, op: osd_op(client.4539.0:34485 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.866753 7f866f5f0700 5 -- op tracker -- , seq: 3324, time: 2014-06-09 16:22:11.866215, event: throttled, op: osd_op(client.4539.0:34485 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.866762 7f866f5f0700 5 -- op tracker -- , seq: 3324, time: 2014-06-09 16:22:11.866700, event: all_read, op: osd_op(client.4539.0:34485 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.866771 7f866f5f0700 5 -- op tracker -- , seq: 3324, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34485 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.866776 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.866800 7f866f5f0700 15 osd.26 243 enqueue_op 0x1b0d7e00 prio 63 cost 217088 latency 0.000587 osd_op(client.4539.0:34485 rbd_data.115b6b8b4567.00000000000018ba [set-alloc-hint object_size 4194304 write_size 4194304,write 3977216~217088] 3.9b6d165f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.867527 7f86a5e11700 10 _generic_write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 3313664~516096 = 0 2014-06-09 16:22:11.867568 7f86a5e11700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.867585 7f86a5e11700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.867595 7f86a5e11700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.867604 7f86a5e11700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.867698 7f86a5e11700 10 cid: oid: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 not skipping op, *spos 3583.0.0 2014-06-09 16:22:11.867703 7f86a5e11700 10 > header.spos 3522.0.0 2014-06-09 16:22:11.867785 7f86a5e11700 20 set_header setting 30152 cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 parent seq 0 2014-06-09 16:22:11.867834 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3583.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.867842 7f86a5e11700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3583.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.867845 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.868270 7f8682de8700 5 -- op tracker -- , seq: 3325, time: 2014-06-09 16:22:11.868228, event: header_read, op: osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.868283 7f8682de8700 5 -- op tracker -- , seq: 3325, time: 2014-06-09 16:22:11.868230, event: throttled, op: osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.868291 7f8682de8700 5 -- op tracker -- , seq: 3325, time: 2014-06-09 16:22:11.868241, event: all_read, op: osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.868307 7f8682de8700 5 -- op tracker -- , seq: 3325, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.868313 7f8682de8700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.868318 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:11.868325 7f8682de8700 15 osd.26 243 enqueue_op 0x32827e00 prio 196 cost 0 latency 0.000097 osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.869864 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.869891 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3584.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.869896 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.869909 7f86a5e11700 10 _do_op 0x2fc8c8a0 seq 3583 r = 0, finisher 0x306cf080 0x30c9f840 2014-06-09 16:22:11.869916 7f86a5e11700 10 _finish_op 0x2fc8c8a0 seq 3583 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.869916 7f86a5610700 10 _do_op 0x324e7080 seq 3584 r = 0, finisher 0x324d1720 0 2014-06-09 16:22:11.869926 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.869930 7f868a757700 10 op_submit_start 3633 2014-06-09 16:22:11.869930 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0ea0 (2 active) 2014-06-09 16:22:11.869932 7f868a757700 5 queue_transactions (trailing journal) 3633 2014-06-09 16:22:11.869935 7f86a5e11700 5 _do_op 0x30032360 seq 3585 osr(3.428 0x3ed11e0)/0x3ed11e0 start 2014-06-09 16:22:11.869934 7f868a757700 5 queue_op 0x3280ade0 seq 3633 osr(2.372 0x2ed1150) 828 bytes (queue has 50 ops and 1966249 bytes) 2014-06-09 16:22:11.869933 7f8692f68700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.869938 7f868a757700 10 op_submit_finish 3633 2014-06-09 16:22:11.869938 7f86a5e11700 10 _do_transaction on 0x48015c0 2014-06-09 16:22:11.869944 7f868ff62700 2 waiting 51 > 50 ops || 1967077 > 104857600 2014-06-09 16:22:11.869946 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.869951 7f868ef60700 2 waiting 51 > 50 ops || 1967077 > 104857600 2014-06-09 16:22:11.869943 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.38d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.38d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.869958 7f8689755700 2 waiting 51 > 50 ops || 1967077 > 104857600 2014-06-09 16:22:11.869966 7f868e75f700 2 waiting 51 > 50 ops || 1967077 > 104857600 2014-06-09 16:22:11.869984 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.869996 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.870002 7f86a5e11700 15 _omap_rmkeyrange meta/1fe18b9a/pglog_3.428/0//-1 [0000000213.00000000000000000561,4294967295.18446744073709551615] 2014-06-09 16:22:11.870004 7f868a757700 5 queue_transactions existing osr(1.38d 0x384d570)/0x384d570 2014-06-09 16:22:11.870005 7f86a5e11700 15 get_omap_iterator meta/1fe18b9a/pglog_3.428/0//-1 2014-06-09 16:22:11.870007 7f868a757700 2 waiting 51 > 50 ops || 1967077 > 104857600 2014-06-09 16:22:11.870012 7f868df5e700 2 waiting 51 > 50 ops || 1967077 > 104857600 2014-06-09 16:22:11.870019 7f8690f64700 2 waiting 51 > 50 ops || 2484638 > 104857600 2014-06-09 16:22:11.870024 7f86a5610700 10 _finish_op 0x324e7080 seq 3584 osr(2.364 0x2ed11b0)/0x2ed11b0 2014-06-09 16:22:11.870026 7f8691f66700 2 waiting 51 > 50 ops || 1967107 > 104857600 2014-06-09 16:22:11.870035 7f868cf5c700 2 waiting 51 > 50 ops || 1967077 > 104857600 2014-06-09 16:22:11.870041 7f868b759700 2 waiting 51 > 50 ops || 1967077 > 104857600 2014-06-09 16:22:11.870050 7f868bf5a700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870063 7f8692f68700 10 op_submit_start 3634 2014-06-09 16:22:11.870062 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62eb40 (1 active) 2014-06-09 16:22:11.870065 7f8692f68700 5 queue_transactions (trailing journal) 3634 2014-06-09 16:22:11.870066 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286240 (2 active) 2014-06-09 16:22:11.870069 7f86a5610700 5 _do_op 0xc64aa80 seq 3586 osr(2.37a 0x2ed1160)/0x2ed1160 start 2014-06-09 16:22:11.870067 7f8692f68700 5 queue_op 0x324e7ce0 seq 3634 osr(2.73 0x2ed0510) 826 bytes (queue has 50 ops and 1966247 bytes) 2014-06-09 16:22:11.870071 7f8692f68700 10 op_submit_finish 3634 2014-06-09 16:22:11.870071 7f86a5610700 10 _do_transaction on 0xc120900 2014-06-09 16:22:11.870077 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.870076 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.82( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.82( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:11.870116 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.870115 7f86a5e11700 15 _omap_rmkeys meta/1fe18b9a/pglog_3.428/0//-1 2014-06-09 16:22:11.870123 7f8692f68700 5 queue_transactions existing osr(1.82 0x384c4d0)/0x384c4d0 2014-06-09 16:22:11.870126 7f8692f68700 2 waiting 51 > 50 ops || 1967073 > 104857600 2014-06-09 16:22:11.870130 7f868ef60700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870135 7f868ff62700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870140 7f868e75f700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870142 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe18b9a/pglog_3.428/0//-1 2014-06-09 16:22:11.870147 7f868a757700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870152 7f86a5e11700 10 _omap_rmkeys meta/1fe18b9a/pglog_3.428/0//-1 r = 0 2014-06-09 16:22:11.870153 7f868df5e700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870157 7f8690f64700 2 waiting 51 > 50 ops || 2484636 > 104857600 2014-06-09 16:22:11.870157 7f86a5e11700 15 _omap_rmkeys meta/1fe18b9a/pglog_3.428/0//-1 2014-06-09 16:22:11.870162 7f86a5e11700 10 _omap_rmkeys meta/1fe18b9a/pglog_3.428/0//-1 r = 0 2014-06-09 16:22:11.870162 7f8691f66700 2 waiting 51 > 50 ops || 1967105 > 104857600 2014-06-09 16:22:11.870169 7f868cf5c700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870170 7f86a5e11700 15 _omap_setkeys meta/1fe18b9a/pglog_3.428/0//-1 2014-06-09 16:22:11.870174 7f868b759700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870180 7f868bf5a700 2 waiting 51 > 50 ops || 1967073 > 104857600 2014-06-09 16:22:11.870185 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3585.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.870187 7f8689755700 2 waiting 51 > 50 ops || 1967075 > 104857600 2014-06-09 16:22:11.870189 7f86a5e11700 10 cid: oid: 1fe18b9a/pglog_3.428/0//-1 skipping op, spos 3585.0.0 <= header.spos 149882.0.0 2014-06-09 16:22:11.870191 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.870211 7f86a5e11700 10 _do_op 0x30032360 seq 3585 r = 0, finisher 0xbcc82a0 0 2014-06-09 16:22:11.870215 7f86a5e11700 10 _finish_op 0x30032360 seq 3585 osr(3.428 0x3ed11e0)/0x3ed11e0 2014-06-09 16:22:11.870220 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0ea0 (1 active) 2014-06-09 16:22:11.870222 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.870223 7f8692f68700 10 op_submit_start 3635 2014-06-09 16:22:11.870224 7f86a5e11700 5 _do_op 0x2fcaf560 seq 3587 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.870226 7f8692f68700 5 queue_transactions (trailing journal) 3635 2014-06-09 16:22:11.870226 7f86a5e11700 10 _do_transaction on 0xc24e6c0 2014-06-09 16:22:11.870227 7f8692f68700 5 queue_op 0x324e7b60 seq 3635 osr(1.82 0x384c4d0) 826 bytes (queue has 50 ops and 1965130 bytes) 2014-06-09 16:22:11.870231 7f8692f68700 10 op_submit_finish 3635 2014-06-09 16:22:11.870231 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.870242 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.870234 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.870251 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.870257 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3586.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.870261 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.870261 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.870270 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.870281 7f86a5e11700 10 lookup_strip_header done cid meta oid 51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.870283 7f86a5610700 10 _do_op 0xc64aa80 seq 3586 r = 0, finisher 0x306ce180 0 2014-06-09 16:22:11.870287 7f86a5610700 10 _finish_op 0xc64aa80 seq 3586 osr(2.37a 0x2ed1160)/0x2ed1160 2014-06-09 16:22:11.870287 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.870289 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286240 (1 active) 2014-06-09 16:22:11.870291 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497b00 (2 active) 2014-06-09 16:22:11.870293 7f86a5610700 5 _do_op 0x324e71a0 seq 3588 osr(2.6cd 0x2ed1e70)/0x2ed1e70 start 2014-06-09 16:22:11.870296 7f86a5610700 10 _do_transaction on 0x3144480 2014-06-09 16:22:11.870296 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.870301 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.870308 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.870325 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.870323 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.870330 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.870338 7f86a5e11700 15 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.870344 7f86a5e11700 10 _omap_rmkeys meta/51682b0b/pglog_3.60/0//-1 r = 0 2014-06-09 16:22:11.870343 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.870350 7f86a5e11700 15 _omap_setkeys meta/51682b0b/pglog_3.60/0//-1 2014-06-09 16:22:11.870352 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.870362 7f86a5e11700 15 _write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 3829760~364544 2014-06-09 16:22:11.870360 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.870373 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.870389 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:151 2014-06-09 16:22:11.870404 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.870409 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.870422 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.785356 7 0.028464 2014-06-09 16:22:11.870436 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.870452 7f86a5e11700 10 lookup_strip_header done cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.870448 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.870466 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.870460 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000025 0 0.000000 2014-06-09 16:22:11.870472 7f8692f68700 5 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.870481 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.870489 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.870502 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:11.870514 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:11.870524 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.870536 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.870552 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:11.870591 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.870596 7f8692f68700 5 queue_transactions existing osr(2.342 0x2ed0ee0)/0x2ed0ee0 2014-06-09 16:22:11.870599 7f8692f68700 10 op_submit_start 3636 2014-06-09 16:22:11.870600 7f8692f68700 5 queue_transactions (trailing journal) 3636 2014-06-09 16:22:11.870601 7f8692f68700 5 queue_op 0x308cb260 seq 3636 osr(2.342 0x2ed0ee0) 861 bytes (queue has 50 ops and 1965163 bytes) 2014-06-09 16:22:11.870604 7f8692f68700 10 op_submit_finish 3636 2014-06-09 16:22:11.870608 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.870621 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.870631 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.870640 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.870649 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.870658 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.870672 7f866f5f0700 5 -- op tracker -- , seq: 3326, time: 2014-06-09 16:22:11.866821, event: header_read, op: osd_op(client.4539.0:34606 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2285568] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.870673 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:174 2014-06-09 16:22:11.870683 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.870692 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.870702 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.870703 7f866f5f0700 5 -- op tracker -- , seq: 3326, time: 2014-06-09 16:22:11.866822, event: throttled, op: osd_op(client.4539.0:34606 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2285568] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.870707 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.870717 7f866f5f0700 5 -- op tracker -- , seq: 3326, time: 2014-06-09 16:22:11.870640, event: all_read, op: osd_op(client.4539.0:34606 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2285568] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.870720 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.870728 7f866f5f0700 5 -- op tracker -- , seq: 3326, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34606 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2285568] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.870728 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.870736 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.870735 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.870743 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.870751 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.870758 7f866f5f0700 15 osd.26 243 enqueue_op 0x19fe1e00 prio 63 cost 2285568 latency 0.003937 osd_op(client.4539.0:34606 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2285568] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.870764 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.870774 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.870784 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.870793 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.870798 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.870810 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.870818 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.870826 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.870835 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.870842 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.870857 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.870867 7f8692f68700 20 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.870875 7f8692f68700 20 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.870884 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.870888 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.65( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.65( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:11.870928 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.870933 7f8692f68700 5 queue_transactions existing osr(2.65 0x2ed0570)/0x2ed0570 2014-06-09 16:22:11.870935 7f8692f68700 2 waiting 51 > 50 ops || 1965989 > 104857600 2014-06-09 16:22:11.870941 7f868ef60700 2 waiting 51 > 50 ops || 1965991 > 104857600 2014-06-09 16:22:11.870947 7f868ff62700 2 waiting 51 > 50 ops || 1965991 > 104857600 2014-06-09 16:22:11.870954 7f868e75f700 2 waiting 51 > 50 ops || 1965991 > 104857600 2014-06-09 16:22:11.870963 7f868a757700 2 waiting 51 > 50 ops || 1965991 > 104857600 2014-06-09 16:22:11.870970 7f868df5e700 2 waiting 51 > 50 ops || 1965991 > 104857600 2014-06-09 16:22:11.870975 7f8690f64700 2 waiting 51 > 50 ops || 2483552 > 104857600 2014-06-09 16:22:11.870980 7f8691f66700 2 waiting 51 > 50 ops || 1966021 > 104857600 2014-06-09 16:22:11.870985 7f868cf5c700 2 waiting 51 > 50 ops || 1965991 > 104857600 2014-06-09 16:22:11.870990 7f868b759700 2 waiting 51 > 50 ops || 1965991 > 104857600 2014-06-09 16:22:11.870995 7f868bf5a700 2 waiting 51 > 50 ops || 1965989 > 104857600 2014-06-09 16:22:11.871001 7f8689755700 2 waiting 51 > 50 ops || 1965991 > 104857600 2014-06-09 16:22:11.871169 7f86a5e11700 10 _generic_write 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 3829760~364544 = 0 2014-06-09 16:22:11.871187 7f86a5e11700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.871195 7f86a5e11700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.871198 7f86a5e11700 15 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 2014-06-09 16:22:11.871203 7f86a5e11700 10 _setattrs 3.60_head/5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 = 0 2014-06-09 16:22:11.871234 7f86a5e11700 10 cid: oid: 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 not skipping op, *spos 3587.0.0 2014-06-09 16:22:11.871236 7f86a5e11700 10 > header.spos 3583.0.0 2014-06-09 16:22:11.871315 7f86a5e11700 20 set_header setting 30152 cid 3.60_head oid 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 parent seq 0 2014-06-09 16:22:11.871340 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3587.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.871344 7f86a5e11700 10 cid: oid: 51682b0b/pglog_3.60/0//-1 skipping op, spos 3587.0.0 <= header.spos 150282.0.0 2014-06-09 16:22:11.871346 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.871558 7f866f5f0700 5 -- op tracker -- , seq: 3327, time: 2014-06-09 16:22:11.870785, event: header_read, op: osd_op(client.4539.0:34607 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.871573 7f866f5f0700 5 -- op tracker -- , seq: 3327, time: 2014-06-09 16:22:11.870788, event: throttled, op: osd_op(client.4539.0:34607 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.871589 7f866f5f0700 5 -- op tracker -- , seq: 3327, time: 2014-06-09 16:22:11.871512, event: all_read, op: osd_op(client.4539.0:34607 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.871600 7f866f5f0700 5 -- op tracker -- , seq: 3327, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34607 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.871606 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.871630 7f866f5f0700 15 osd.26 243 enqueue_op 0x32827f00 prio 63 cost 516096 latency 0.000845 osd_op(client.4539.0:34607 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2285568~516096] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.872700 7f86a5e11700 10 _do_op 0x2fcaf560 seq 3587 r = 0, finisher 0xbd5e000 0x2ff1ed40 2014-06-09 16:22:11.872708 7f86a5e11700 10 _finish_op 0x2fcaf560 seq 3587 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.872718 7f8692f68700 10 op_submit_start 3637 2014-06-09 16:22:11.872721 7f8692f68700 5 queue_transactions (trailing journal) 3637 2014-06-09 16:22:11.872723 7f8692f68700 5 queue_op 0x100381e0 seq 3637 osr(2.65 0x2ed0570) 826 bytes (queue has 50 ops and 1599161 bytes) 2014-06-09 16:22:11.872730 7f868ef60700 2 waiting 51 > 50 ops || 1599989 > 104857600 2014-06-09 16:22:11.872737 7f868ff62700 2 waiting 51 > 50 ops || 1599989 > 104857600 2014-06-09 16:22:11.872755 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.872756 7f868e75f700 2 waiting 51 > 50 ops || 1599989 > 104857600 2014-06-09 16:22:11.872760 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad8c0 (2 active) 2014-06-09 16:22:11.872765 7f868a757700 2 waiting 51 > 50 ops || 1599989 > 104857600 2014-06-09 16:22:11.872765 7f86a5e11700 5 _do_op 0xbe7c540 seq 3589 osr(3.450 0x3ed1160)/0x3ed1160 start 2014-06-09 16:22:11.872768 7f86a5e11700 10 _do_transaction on 0xc23f980 2014-06-09 16:22:11.872771 7f868df5e700 2 waiting 51 > 50 ops || 1599989 > 104857600 2014-06-09 16:22:11.872773 7f8692f68700 10 op_submit_finish 3637 2014-06-09 16:22:11.872776 7f8690f64700 2 waiting 51 > 50 ops || 2117550 > 104857600 2014-06-09 16:22:11.872777 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.872780 7f8691f66700 2 waiting 51 > 50 ops || 1600019 > 104857600 2014-06-09 16:22:11.872785 7f868cf5c700 2 waiting 51 > 50 ops || 1599989 > 104857600 2014-06-09 16:22:11.872790 7f868b759700 2 waiting 51 > 50 ops || 1599989 > 104857600 2014-06-09 16:22:11.872780 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.872795 7f868bf5a700 2 waiting 51 > 50 ops || 1599987 > 104857600 2014-06-09 16:22:11.872800 7f8689755700 2 waiting 51 > 50 ops || 1599989 > 104857600 2014-06-09 16:22:11.872800 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.872811 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.872819 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.872838 7f8692f68700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.872861 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.872859 7f8692f68700 15 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.872876 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.872883 7f86a5e11700 15 _omap_rmkeyrange meta/1fe198aa/pglog_3.450/0//-1 [0000000213.00000000000000000620,4294967295.18446744073709551615] 2014-06-09 16:22:11.872887 7f86a5e11700 15 get_omap_iterator meta/1fe198aa/pglog_3.450/0//-1 2014-06-09 16:22:11.872902 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 4 notify: (query_epoch:242, epoch_sent:243, info:1.76c( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/242)) 2014-06-09 16:22:11.872953 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 4, already have info from that osd, ignoring 2014-06-09 16:22:11.872963 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.872968 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.872979 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.873031 7f86a5e11700 15 _omap_rmkeys meta/1fe198aa/pglog_3.450/0//-1 2014-06-09 16:22:11.873068 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe198aa/pglog_3.450/0//-1 2014-06-09 16:22:11.872987 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.873075 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.873098 7f86a5e11700 10 _omap_rmkeys meta/1fe198aa/pglog_3.450/0//-1 r = 0 2014-06-09 16:22:11.873104 7f86a5e11700 15 _omap_rmkeys meta/1fe198aa/pglog_3.450/0//-1 2014-06-09 16:22:11.873108 7f86a5e11700 10 _omap_rmkeys meta/1fe198aa/pglog_3.450/0//-1 r = 0 2014-06-09 16:22:11.873120 7f86a5e11700 15 _omap_setkeys meta/1fe198aa/pglog_3.450/0//-1 2014-06-09 16:22:11.873147 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3589.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.873153 7f86a5e11700 10 cid: oid: 1fe198aa/pglog_3.450/0//-1 skipping op, spos 3589.0.0 <= header.spos 148284.0.0 2014-06-09 16:22:11.873157 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.873203 7f86a5e11700 10 _do_op 0xbe7c540 seq 3589 r = 0, finisher 0xbd5dca0 0 2014-06-09 16:22:11.873207 7f86a5e11700 10 _finish_op 0xbe7c540 seq 3589 osr(3.450 0x3ed1160)/0x3ed1160 2014-06-09 16:22:11.873212 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad8c0 (1 active) 2014-06-09 16:22:11.873204 7f866f5f0700 5 -- op tracker -- , seq: 3328, time: 2014-06-09 16:22:11.871665, event: header_read, op: osd_op(client.4539.0:34608 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~1032192] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.873215 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f78c0 (2 active) 2014-06-09 16:22:11.873217 7f86a5e11700 5 _do_op 0xbe7dc20 seq 3590 osr(0.4c7 0x2bd76b0)/0x2bd76b0 start 2014-06-09 16:22:11.873217 7f868ef60700 10 op_submit_start 3638 2014-06-09 16:22:11.873219 7f86a5e11700 10 _do_transaction on 0x3338fc0 2014-06-09 16:22:11.873219 7f868ef60700 5 queue_transactions (trailing journal) 3638 2014-06-09 16:22:11.873221 7f868ff62700 2 waiting 51 > 50 ops || 1597417 > 104857600 2014-06-09 16:22:11.873221 7f868ef60700 5 queue_op 0x241050e0 seq 3638 osr(3.3a9 0x3ed0f60) 828 bytes (queue has 50 ops and 1596589 bytes) 2014-06-09 16:22:11.873225 7f868ef60700 10 op_submit_finish 3638 2014-06-09 16:22:11.873224 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.873222 7f866f5f0700 5 -- op tracker -- , seq: 3328, time: 2014-06-09 16:22:11.871666, event: throttled, op: osd_op(client.4539.0:34608 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~1032192] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.873236 7f866f5f0700 5 -- op tracker -- , seq: 3328, time: 2014-06-09 16:22:11.873164, event: all_read, op: osd_op(client.4539.0:34608 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~1032192] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.873229 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.589( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.589( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.873246 7f866f5f0700 5 -- op tracker -- , seq: 3328, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34608 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~1032192] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.873256 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.873254 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.873267 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.873271 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3590.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.873273 7f868ef60700 5 queue_transactions existing osr(1.589 0x384da10)/0x384da10 2014-06-09 16:22:11.873274 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.873276 7f868ef60700 2 waiting 51 > 50 ops || 1597417 > 104857600 2014-06-09 16:22:11.873281 7f868e75f700 2 waiting 51 > 50 ops || 1597417 > 104857600 2014-06-09 16:22:11.873276 7f866f5f0700 15 osd.26 243 enqueue_op 0x30dda800 prio 63 cost 1032192 latency 0.001611 osd_op(client.4539.0:34608 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2801664~1032192] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.873287 7f868a757700 2 waiting 51 > 50 ops || 1597417 > 104857600 2014-06-09 16:22:11.873293 7f86a5e11700 10 _do_op 0xbe7dc20 seq 3590 r = 0, finisher 0xbd5daa0 0 2014-06-09 16:22:11.873296 7f86a5e11700 10 _finish_op 0xbe7dc20 seq 3590 osr(0.4c7 0x2bd76b0)/0x2bd76b0 2014-06-09 16:22:11.873301 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f78c0 (1 active) 2014-06-09 16:22:11.873303 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f0e0 (2 active) 2014-06-09 16:22:11.873305 7f86a5e11700 5 _do_op 0x324e73e0 seq 3591 osr(1.676 0x384dee0)/0x384dee0 start 2014-06-09 16:22:11.873307 7f86a5e11700 10 _do_transaction on 0xc160d80 2014-06-09 16:22:11.873311 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.873320 7f868ff62700 10 op_submit_start 3639 2014-06-09 16:22:11.873322 7f868ff62700 5 queue_transactions (trailing journal) 3639 2014-06-09 16:22:11.873324 7f868ff62700 5 queue_op 0xc2d4a80 seq 3639 osr(2.3f0 0x2ed1320) 828 bytes (queue has 50 ops and 1595541 bytes) 2014-06-09 16:22:11.873328 7f868ff62700 10 op_submit_finish 3639 2014-06-09 16:22:11.873339 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.873332 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.3b4( v 214'677 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) 2014-06-09 16:22:11.873351 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3591.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.873354 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.873353 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] state: peer osd.8 activated and committed 2014-06-09 16:22:11.873364 7f86a5e11700 10 _do_op 0x324e73e0 seq 3591 r = 0, finisher 0xcc95720 0 2014-06-09 16:22:11.873367 7f86a5e11700 10 _finish_op 0x324e73e0 seq 3591 osr(1.676 0x384dee0)/0x384dee0 2014-06-09 16:22:11.873369 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f0e0 (1 active) 2014-06-09 16:22:11.873371 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115200 (2 active) 2014-06-09 16:22:11.873373 7f86a5e11700 5 _do_op 0x2fcaf500 seq 3592 osr(2.65 0x2ed0570)/0x2ed0570 start 2014-06-09 16:22:11.873375 7f86a5e11700 10 _do_transaction on 0xc2c5800 2014-06-09 16:22:11.873367 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] all_activated_and_committed 2014-06-09 16:22:11.873379 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.873380 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] share_pg_info 2014-06-09 16:22:11.873397 7f868ff62700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.873406 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.873417 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3592.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.873421 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.873415 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] publish_stats_to_osd 243:1121 2014-06-09 16:22:11.873430 7f86a5e11700 10 _do_op 0x2fcaf500 seq 3592 r = 0, finisher 0x17001e20 0 2014-06-09 16:22:11.873432 7f86a5e11700 10 _finish_op 0x2fcaf500 seq 3592 osr(2.65 0x2ed0570)/0x2ed0570 2014-06-09 16:22:11.873434 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115200 (1 active) 2014-06-09 16:22:11.873435 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.873436 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496c60 (2 active) 2014-06-09 16:22:11.873438 7f86a5e11700 5 _do_op 0x2fc8c9c0 seq 3593 osr(2.3d1 0x2ed13b0)/0x2ed13b0 start 2014-06-09 16:22:11.873440 7f86a5e11700 10 _do_transaction on 0x3414c00 2014-06-09 16:22:11.873444 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.873440 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.873453 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.873462 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.873471 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.873470 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.873483 7f868ff62700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.873484 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3593.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.873488 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.873499 7f86a5e11700 10 _do_op 0x2fc8c9c0 seq 3593 r = 0, finisher 0x314353c0 0 2014-06-09 16:22:11.873502 7f86a5e11700 10 _finish_op 0x2fc8c9c0 seq 3593 osr(2.3d1 0x2ed13b0)/0x2ed13b0 2014-06-09 16:22:11.873505 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496c60 (1 active) 2014-06-09 16:22:11.873498 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.873507 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d440 (2 active) 2014-06-09 16:22:11.873509 7f86a5e11700 5 _do_op 0x324e7560 seq 3594 osr(1.56 0x384c5f0)/0x384c5f0 start 2014-06-09 16:22:11.873512 7f86a5e11700 10 _do_transaction on 0xc33ec00 2014-06-09 16:22:11.873513 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.873517 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.873520 7f8690f64700 10 op_submit_start 3640 2014-06-09 16:22:11.873522 7f8690f64700 5 queue_transactions (trailing journal) 3640 2014-06-09 16:22:11.873523 7f8690f64700 5 queue_op 0xc64a6c0 seq 3640 osr(3.319 0x3ed0d60) 518389 bytes (queue has 48 ops and 2111448 bytes) 2014-06-09 16:22:11.873526 7f8690f64700 10 op_submit_finish 3640 2014-06-09 16:22:11.873533 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.873548 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.873531 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0x3e62240 243'609 rep_tid=16 committed?=0 applied?=0 lock=0 op=osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.873550 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.873559 7f8690f64700 10 osd.26 243 dequeue_op 0x1c54ba00 finish 2014-06-09 16:22:11.873561 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3594.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.873565 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.873564 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.873560 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.873572 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.873568 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.873581 7f86a5e11700 10 _do_op 0x324e7560 seq 3594 r = 0, finisher 0xbd5ce80 0 2014-06-09 16:22:11.873582 7f868ff62700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.873584 7f86a5e11700 10 _finish_op 0x324e7560 seq 3594 osr(1.56 0x384c5f0)/0x384c5f0 2014-06-09 16:22:11.873587 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d440 (1 active) 2014-06-09 16:22:11.873589 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286c60 (2 active) 2014-06-09 16:22:11.873591 7f86a5e11700 5 _do_op 0x324e7620 seq 3595 osr(3.7e2 0xb7f0120)/0xb7f0120 start 2014-06-09 16:22:11.873593 7f86a5e11700 10 _do_transaction on 0xc2f4600 2014-06-09 16:22:11.873598 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.873612 7f868df5e700 10 op_submit_start 3641 2014-06-09 16:22:11.873615 7f868df5e700 5 queue_transactions (trailing journal) 3641 2014-06-09 16:22:11.873616 7f868df5e700 5 queue_op 0xc705860 seq 3641 osr(1.382 0x384d5b0) 828 bytes (queue has 48 ops and 2111450 bytes) 2014-06-09 16:22:11.873620 7f868df5e700 10 op_submit_finish 3641 2014-06-09 16:22:11.873630 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.873623 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.54c( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.873599 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.873643 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3595.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.873646 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.873647 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.873656 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.873655 7f86a5e11700 10 _do_op 0x324e7620 seq 3595 r = 0, finisher 0xbd5cc80 0 2014-06-09 16:22:11.873658 7f86a5e11700 10 _finish_op 0x324e7620 seq 3595 osr(3.7e2 0xb7f0120)/0xb7f0120 2014-06-09 16:22:11.873661 7f868df5e700 5 queue_transactions existing osr(0.54c 0x2bd7850)/0x2bd7850 2014-06-09 16:22:11.873661 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286c60 (1 active) 2014-06-09 16:22:11.873652 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.873663 7f868df5e700 10 op_submit_start 3642 2014-06-09 16:22:11.873663 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4a20 (2 active) 2014-06-09 16:22:11.873664 7f868df5e700 5 queue_transactions (trailing journal) 3642 2014-06-09 16:22:11.873665 7f86a5e11700 5 _do_op 0x2e20de0 seq 3596 osr(3.1e 0x3ed0380)/0x3ed0380 start 2014-06-09 16:22:11.873665 7f868df5e700 5 queue_op 0xc705560 seq 3642 osr(0.54c 0x2bd7850) 828 bytes (queue has 48 ops and 2111450 bytes) 2014-06-09 16:22:11.873667 7f868df5e700 10 op_submit_finish 3642 2014-06-09 16:22:11.873667 7f86a5e11700 10 _do_transaction on 0x2ed4a80 2014-06-09 16:22:11.873672 7f86a5e11700 15 _omap_setkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.873666 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.873670 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.873676 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.873683 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.873684 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.873697 7f868ff62700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.873692 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.873700 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.873711 7f86a5e11700 10 lookup_strip_header done cid meta oid 51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.873713 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.873712 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:992 2014-06-09 16:22:11.873725 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.873727 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.873725 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:179 2014-06-09 16:22:11.873740 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.873744 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.873756 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.873756 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.873764 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.873769 7f86a5e11700 15 _omap_rmkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.873775 7f86a5e11700 10 _omap_rmkeys meta/51682acc/pglog_3.1e/0//-1 r = 0 2014-06-09 16:22:11.873772 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.873780 7f86a5e11700 15 _omap_setkeys meta/51682acc/pglog_3.1e/0//-1 2014-06-09 16:22:11.873784 7f868df5e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.873793 7f86a5e11700 15 _write 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 1384448~1548288 2014-06-09 16:22:11.873825 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.873839 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.873856 7f86a5e11700 10 lookup_strip_header done cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.873848 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.873881 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.873881 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.873894 7f868ff62700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.873907 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.873922 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.873927 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.2c3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.2c3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.873961 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.873965 7f868ff62700 5 queue_transactions existing osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 2014-06-09 16:22:11.873968 7f868ff62700 10 op_submit_start 3643 2014-06-09 16:22:11.873970 7f868ff62700 5 queue_transactions (trailing journal) 3643 2014-06-09 16:22:11.873971 7f868ff62700 5 queue_op 0xc2d4180 seq 3643 osr(0.2c3 0x2bd6ba0) 828 bytes (queue has 49 ops and 2112278 bytes) 2014-06-09 16:22:11.873974 7f868ff62700 10 op_submit_finish 3643 2014-06-09 16:22:11.873977 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.874004 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.874013 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.874021 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.874033 7f868ff62700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.874051 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:148 2014-06-09 16:22:11.874066 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.874129 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3f0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.3f0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.874163 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.874168 7f868ff62700 5 queue_transactions existing osr(2.3f0 0x2ed1320)/0x2ed1320 2014-06-09 16:22:11.874171 7f868ff62700 10 op_submit_start 3644 2014-06-09 16:22:11.874172 7f868ff62700 5 queue_transactions (trailing journal) 3644 2014-06-09 16:22:11.874174 7f868ff62700 5 queue_op 0xc2d4f60 seq 3644 osr(2.3f0 0x2ed1320) 828 bytes (queue has 50 ops and 2113106 bytes) 2014-06-09 16:22:11.874177 7f868ff62700 10 op_submit_finish 3644 2014-06-09 16:22:11.874180 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:2.290( empty local-les=212 n=0 ec=1 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.874193 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.874204 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.874209 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MNotifyRec from 8 notify: (query_epoch:243, epoch_sent:243, info:3.56a( v 214'575 (0'0,214'575] local-les=212 n=108 ec=118 les/c 212/212 242/242/242)) 2014-06-09 16:22:11.874223 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:11.874238 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.874242 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.874254 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.874260 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.874264 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.874273 7f868ff62700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.874288 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.874303 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.874307 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.380( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.874337 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.874342 7f868ff62700 5 queue_transactions existing osr(0.380 0x2bd6f20)/0x2bd6f20 2014-06-09 16:22:11.874344 7f868ff62700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.874604 7f866f5f0700 5 -- op tracker -- , seq: 3329, time: 2014-06-09 16:22:11.873300, event: header_read, op: osd_op(client.4539.0:34609 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~360448] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.874615 7f866f5f0700 5 -- op tracker -- , seq: 3329, time: 2014-06-09 16:22:11.873301, event: throttled, op: osd_op(client.4539.0:34609 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~360448] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.874652 7f866f5f0700 5 -- op tracker -- , seq: 3329, time: 2014-06-09 16:22:11.874574, event: all_read, op: osd_op(client.4539.0:34609 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~360448] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.874663 7f866f5f0700 5 -- op tracker -- , seq: 3329, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34609 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~360448] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.874669 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.874687 7f866f5f0700 15 osd.26 243 enqueue_op 0x30dda700 prio 63 cost 360448 latency 0.001387 osd_op(client.4539.0:34609 rbd_data.115b6b8b4567.00000000000018d4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3833856~360448] 3.e945d006 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.875637 7f866f5f0700 5 -- op tracker -- , seq: 3330, time: 2014-06-09 16:22:11.874710, event: header_read, op: osd_op(client.4539.0:34625 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.875652 7f866f5f0700 5 -- op tracker -- , seq: 3330, time: 2014-06-09 16:22:11.874711, event: throttled, op: osd_op(client.4539.0:34625 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.875663 7f866f5f0700 5 -- op tracker -- , seq: 3330, time: 2014-06-09 16:22:11.875581, event: all_read, op: osd_op(client.4539.0:34625 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.875778 7f866f5f0700 5 -- op tracker -- , seq: 3330, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34625 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.875786 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.875811 7f866f5f0700 15 osd.26 243 enqueue_op 0x30dda600 prio 63 cost 475136 latency 0.001101 osd_op(client.4539.0:34625 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.875945 7f868cf5c700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.875955 7f868b759700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.875982 7f8691f66700 2 waiting 51 > 50 ops || 2113964 > 104857600 2014-06-09 16:22:11.875989 7f868bf5a700 2 waiting 51 > 50 ops || 2113932 > 104857600 2014-06-09 16:22:11.875996 7f868ef60700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.876005 7f868e75f700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.873799 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:156 2014-06-09 16:22:11.876021 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.876029 7f868a757700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.876036 7f8689755700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.876032 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.45d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.876069 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.876074 7f868df5e700 5 queue_transactions existing osr(1.45d 0x384d670)/0x384d670 2014-06-09 16:22:11.876077 7f868df5e700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.877469 7f866f5f0700 5 -- op tracker -- , seq: 3331, time: 2014-06-09 16:22:11.875838, event: header_read, op: osd_op(client.4539.0:34626 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 475136~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.877505 7f866f5f0700 5 -- op tracker -- , seq: 3331, time: 2014-06-09 16:22:11.875839, event: throttled, op: osd_op(client.4539.0:34626 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 475136~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.877522 7f866f5f0700 5 -- op tracker -- , seq: 3331, time: 2014-06-09 16:22:11.877431, event: all_read, op: osd_op(client.4539.0:34626 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 475136~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.877534 7f866f5f0700 5 -- op tracker -- , seq: 3331, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34626 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 475136~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.877561 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.877675 7f866f5f0700 15 osd.26 243 enqueue_op 0x30dda500 prio 63 cost 1032192 latency 0.001836 osd_op(client.4539.0:34626 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 475136~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.873084 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.878781 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.878803 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:152 2014-06-09 16:22:11.878816 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.878825 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.878838 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.878849 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.1a4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.878889 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.878897 7f8692f68700 5 queue_transactions existing osr(0.1a4 0x2bd6710)/0x2bd6710 2014-06-09 16:22:11.878901 7f8692f68700 2 waiting 51 > 50 ops || 2113934 > 104857600 2014-06-09 16:22:11.879633 7f866f5f0700 5 -- op tracker -- , seq: 3332, time: 2014-06-09 16:22:11.877722, event: header_read, op: osd_op(client.4539.0:34627 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1507328~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.879648 7f866f5f0700 5 -- op tracker -- , seq: 3332, time: 2014-06-09 16:22:11.877723, event: throttled, op: osd_op(client.4539.0:34627 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1507328~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.879658 7f866f5f0700 5 -- op tracker -- , seq: 3332, time: 2014-06-09 16:22:11.879585, event: all_read, op: osd_op(client.4539.0:34627 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1507328~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.879669 7f866f5f0700 5 -- op tracker -- , seq: 3332, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34627 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1507328~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.879678 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.879704 7f866f5f0700 15 osd.26 243 enqueue_op 0x30dda400 prio 63 cost 1032192 latency 0.001982 osd_op(client.4539.0:34627 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 1507328~1032192] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.880280 7f866f5f0700 5 -- op tracker -- , seq: 3333, time: 2014-06-09 16:22:11.879729, event: header_read, op: osd_op(client.4539.0:34628 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 2539520~516096] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.880294 7f866f5f0700 5 -- op tracker -- , seq: 3333, time: 2014-06-09 16:22:11.879730, event: throttled, op: osd_op(client.4539.0:34628 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 2539520~516096] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.880305 7f866f5f0700 5 -- op tracker -- , seq: 3333, time: 2014-06-09 16:22:11.880252, event: all_read, op: osd_op(client.4539.0:34628 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 2539520~516096] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.880317 7f866f5f0700 5 -- op tracker -- , seq: 3333, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34628 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 2539520~516096] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.880325 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.880341 7f866f5f0700 15 osd.26 243 enqueue_op 0x30dda300 prio 63 cost 516096 latency 0.000612 osd_op(client.4539.0:34628 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 2539520~516096] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.881579 7f86a5e11700 10 _generic_write 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 1384448~1548288 = 0 2014-06-09 16:22:11.881674 7f86a5e11700 15 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.881693 7f86a5e11700 10 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.881704 7f86a5e11700 15 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 2014-06-09 16:22:11.881713 7f86a5e11700 10 _setattrs 3.1e_head/78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 = 0 2014-06-09 16:22:11.881911 7f86a5e11700 10 cid: oid: 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 not skipping op, *spos 3596.0.0 2014-06-09 16:22:11.881918 7f86a5e11700 10 > header.spos 3486.0.0 2014-06-09 16:22:11.881943 7f866f5f0700 5 -- op tracker -- , seq: 3334, time: 2014-06-09 16:22:11.880363, event: header_read, op: osd_op(client.4539.0:34629 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3055616~1138688] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.881957 7f866f5f0700 5 -- op tracker -- , seq: 3334, time: 2014-06-09 16:22:11.880364, event: throttled, op: osd_op(client.4539.0:34629 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3055616~1138688] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.881968 7f866f5f0700 5 -- op tracker -- , seq: 3334, time: 2014-06-09 16:22:11.881914, event: all_read, op: osd_op(client.4539.0:34629 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3055616~1138688] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.881979 7f866f5f0700 5 -- op tracker -- , seq: 3334, time: 0.000000, event: dispatched, op: osd_op(client.4539.0:34629 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3055616~1138688] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.881987 7f866f5f0700 20 osd.26 243 should_share_map client.4539 172.16.96.220:0/1123664 243 2014-06-09 16:22:11.881999 7f86a5e11700 20 set_header setting 30153 cid 3.1e_head oid 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 parent seq 0 2014-06-09 16:22:11.882010 7f866f5f0700 15 osd.26 243 enqueue_op 0x30dda200 prio 63 cost 1138688 latency 0.001647 osd_op(client.4539.0:34629 rbd_data.115b6b8b4567.00000000000018d8 [set-alloc-hint object_size 4194304 write_size 4194304,write 3055616~1138688] 3.c7aa2d62 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.882122 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3596.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.882130 7f86a5e11700 10 cid: oid: 51682acc/pglog_3.1e/0//-1 skipping op, spos 3596.0.0 <= header.spos 150326.0.0 2014-06-09 16:22:11.882134 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.889783 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.889809 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3588.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.889815 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.890078 7f86a5e11700 10 _do_op 0x2e20de0 seq 3596 r = 0, finisher 0x281b7680 0x2eefec80 2014-06-09 16:22:11.890086 7f86a5e11700 10 _finish_op 0x2e20de0 seq 3596 osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.890095 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4a20 (1 active) 2014-06-09 16:22:11.890099 7f868cf5c700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.890097 7f868ff62700 10 op_submit_start 3645 2014-06-09 16:22:11.890103 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286240 (2 active) 2014-06-09 16:22:11.890104 7f868ff62700 5 queue_transactions (trailing journal) 3645 2014-06-09 16:22:11.890106 7f86a5e11700 5 _do_op 0x2e20ba0 seq 3597 osr(2.37a 0x2ed1160)/0x2ed1160 start 2014-06-09 16:22:11.890107 7f868b759700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.890106 7f868ff62700 5 queue_op 0xc2d41e0 seq 3645 osr(0.380 0x2bd6f20) 828 bytes (queue has 50 ops and 563362 bytes) 2014-06-09 16:22:11.890109 7f86a5e11700 10 _do_transaction on 0xbe89200 2014-06-09 16:22:11.890110 7f868ff62700 10 op_submit_finish 3645 2014-06-09 16:22:11.890112 7f8691f66700 2 waiting 51 > 50 ops || 564220 > 104857600 2014-06-09 16:22:11.890116 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.890117 7f868bf5a700 2 waiting 51 > 50 ops || 564188 > 104857600 2014-06-09 16:22:11.890123 7f868ef60700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.890116 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:11.890131 7f868e75f700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.890141 7f868a757700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.890150 7f8689755700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.890156 7f868df5e700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.890158 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.890165 7f8692f68700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.890167 7f868ff62700 5 queue_transactions existing osr(3.6d5 0x3ed1c30)/0x3ed1c30 2014-06-09 16:22:11.890169 7f868ff62700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.899181 7f866f3ee700 10 osd.26 243 new session 0xd228000 con=0x30ba09a0 addr=172.16.96.220:0/1123692 2014-06-09 16:22:11.899529 7f866f3ee700 5 -- op tracker -- , seq: 3335, time: 2014-06-09 16:22:11.899466, event: header_read, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.899544 7f866f3ee700 5 -- op tracker -- , seq: 3335, time: 2014-06-09 16:22:11.899469, event: throttled, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.899553 7f866f3ee700 5 -- op tracker -- , seq: 3335, time: 2014-06-09 16:22:11.899491, event: all_read, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.899563 7f866f3ee700 5 -- op tracker -- , seq: 3335, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.899571 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.899598 7f866f3ee700 15 osd.26 243 enqueue_op 0x2c06b000 prio 63 cost 8192 latency 0.000132 osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.907726 7f866f3ee700 5 -- op tracker -- , seq: 3336, time: 2014-06-09 16:22:11.899664, event: header_read, op: osd_op(client.4540.0:33716 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2048000] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.907768 7f866f3ee700 5 -- op tracker -- , seq: 3336, time: 2014-06-09 16:22:11.899665, event: throttled, op: osd_op(client.4540.0:33716 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2048000] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.907810 7f866f3ee700 5 -- op tracker -- , seq: 3336, time: 2014-06-09 16:22:11.907636, event: all_read, op: osd_op(client.4540.0:33716 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2048000] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.907842 7f866f3ee700 5 -- op tracker -- , seq: 3336, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33716 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2048000] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.907868 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.907933 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddaa00 prio 63 cost 2048000 latency 0.008269 osd_op(client.4540.0:33716 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2048000] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.908992 7f866f3ee700 5 -- op tracker -- , seq: 3337, time: 2014-06-09 16:22:11.907989, event: header_read, op: osd_op(client.4540.0:33717 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2048000~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.909025 7f866f3ee700 5 -- op tracker -- , seq: 3337, time: 2014-06-09 16:22:11.907991, event: throttled, op: osd_op(client.4540.0:33717 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2048000~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.909054 7f866f3ee700 5 -- op tracker -- , seq: 3337, time: 2014-06-09 16:22:11.908919, event: all_read, op: osd_op(client.4540.0:33717 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2048000~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.909111 7f866f3ee700 5 -- op tracker -- , seq: 3337, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33717 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2048000~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.909138 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.909174 7f866f3ee700 15 osd.26 243 enqueue_op 0x30dda900 prio 63 cost 516096 latency 0.001185 osd_op(client.4540.0:33717 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2048000~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.909822 7f866f3ee700 5 -- op tracker -- , seq: 3338, time: 2014-06-09 16:22:11.909201, event: header_read, op: osd_op(client.4540.0:33718 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2564096~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.909834 7f866f3ee700 5 -- op tracker -- , seq: 3338, time: 2014-06-09 16:22:11.909202, event: throttled, op: osd_op(client.4540.0:33718 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2564096~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.909845 7f866f3ee700 5 -- op tracker -- , seq: 3338, time: 2014-06-09 16:22:11.909782, event: all_read, op: osd_op(client.4540.0:33718 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2564096~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.909855 7f866f3ee700 5 -- op tracker -- , seq: 3338, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33718 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2564096~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.909862 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.909878 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddad00 prio 63 cost 516096 latency 0.000677 osd_op(client.4540.0:33718 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 2564096~516096] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.911127 7f866f3ee700 5 -- op tracker -- , seq: 3339, time: 2014-06-09 16:22:11.909901, event: header_read, op: osd_op(client.4540.0:33719 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3080192~1032192] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.911139 7f866f3ee700 5 -- op tracker -- , seq: 3339, time: 2014-06-09 16:22:11.909902, event: throttled, op: osd_op(client.4540.0:33719 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3080192~1032192] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.911149 7f866f3ee700 5 -- op tracker -- , seq: 3339, time: 2014-06-09 16:22:11.911100, event: all_read, op: osd_op(client.4540.0:33719 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3080192~1032192] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.911159 7f866f3ee700 5 -- op tracker -- , seq: 3339, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33719 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3080192~1032192] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.911165 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.911183 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddac00 prio 63 cost 1032192 latency 0.001282 osd_op(client.4540.0:33719 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 3080192~1032192] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.911322 7f866f3ee700 5 -- op tracker -- , seq: 3340, time: 2014-06-09 16:22:11.911204, event: header_read, op: osd_op(client.4540.0:33720 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 4112384~81920] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.911335 7f866f3ee700 5 -- op tracker -- , seq: 3340, time: 2014-06-09 16:22:11.911205, event: throttled, op: osd_op(client.4540.0:33720 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 4112384~81920] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.911346 7f866f3ee700 5 -- op tracker -- , seq: 3340, time: 2014-06-09 16:22:11.911297, event: all_read, op: osd_op(client.4540.0:33720 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 4112384~81920] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.911356 7f866f3ee700 5 -- op tracker -- , seq: 3340, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33720 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 4112384~81920] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.911361 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.911376 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddab00 prio 63 cost 81920 latency 0.000171 osd_op(client.4540.0:33720 rbd_data.115c6b8b4567.00000000000017c5 [set-alloc-hint object_size 4194304 write_size 4194304,write 4112384~81920] 3.d6afa799 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.912408 7f866f3ee700 5 -- op tracker -- , seq: 3341, time: 2014-06-09 16:22:11.911394, event: header_read, op: osd_op(client.4540.0:33727 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.912419 7f866f3ee700 5 -- op tracker -- , seq: 3341, time: 2014-06-09 16:22:11.911395, event: throttled, op: osd_op(client.4540.0:33727 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.912429 7f866f3ee700 5 -- op tracker -- , seq: 3341, time: 2014-06-09 16:22:11.912381, event: all_read, op: osd_op(client.4540.0:33727 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.912445 7f866f3ee700 5 -- op tracker -- , seq: 3341, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33727 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.912451 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.912468 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb100 prio 63 cost 884736 latency 0.001073 osd_op(client.4540.0:33727 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.913066 7f866f3ee700 5 -- op tracker -- , seq: 3342, time: 2014-06-09 16:22:11.912505, event: header_read, op: osd_op(client.4540.0:33728 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 884736~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.913077 7f866f3ee700 5 -- op tracker -- , seq: 3342, time: 2014-06-09 16:22:11.912506, event: throttled, op: osd_op(client.4540.0:33728 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 884736~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.913090 7f866f3ee700 5 -- op tracker -- , seq: 3342, time: 2014-06-09 16:22:11.913043, event: all_read, op: osd_op(client.4540.0:33728 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 884736~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.913100 7f866f3ee700 5 -- op tracker -- , seq: 3342, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33728 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 884736~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.913106 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.913133 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb000 prio 63 cost 516096 latency 0.000628 osd_op(client.4540.0:33728 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 884736~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.914430 7f866f3ee700 5 -- op tracker -- , seq: 3343, time: 2014-06-09 16:22:11.913156, event: header_read, op: osd_op(client.4540.0:33729 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1400832~1032192] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.914442 7f866f3ee700 5 -- op tracker -- , seq: 3343, time: 2014-06-09 16:22:11.913156, event: throttled, op: osd_op(client.4540.0:33729 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1400832~1032192] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.914452 7f866f3ee700 5 -- op tracker -- , seq: 3343, time: 2014-06-09 16:22:11.914403, event: all_read, op: osd_op(client.4540.0:33729 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1400832~1032192] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.914468 7f866f3ee700 5 -- op tracker -- , seq: 3343, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33729 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1400832~1032192] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.914474 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.914492 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddaf00 prio 63 cost 1032192 latency 0.001336 osd_op(client.4540.0:33729 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 1400832~1032192] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.915129 7f866f3ee700 5 -- op tracker -- , seq: 3344, time: 2014-06-09 16:22:11.914515, event: header_read, op: osd_op(client.4540.0:33730 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2433024~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.915140 7f866f3ee700 5 -- op tracker -- , seq: 3344, time: 2014-06-09 16:22:11.914516, event: throttled, op: osd_op(client.4540.0:33730 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2433024~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.915168 7f866f3ee700 5 -- op tracker -- , seq: 3344, time: 2014-06-09 16:22:11.915102, event: all_read, op: osd_op(client.4540.0:33730 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2433024~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.915178 7f866f3ee700 5 -- op tracker -- , seq: 3344, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33730 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2433024~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.915183 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.915200 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddae00 prio 63 cost 516096 latency 0.000685 osd_op(client.4540.0:33730 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2433024~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.915813 7f866f3ee700 5 -- op tracker -- , seq: 3345, time: 2014-06-09 16:22:11.915217, event: header_read, op: osd_op(client.4540.0:33731 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2949120~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.915824 7f866f3ee700 5 -- op tracker -- , seq: 3345, time: 2014-06-09 16:22:11.915218, event: throttled, op: osd_op(client.4540.0:33731 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2949120~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.915834 7f866f3ee700 5 -- op tracker -- , seq: 3345, time: 2014-06-09 16:22:11.915788, event: all_read, op: osd_op(client.4540.0:33731 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2949120~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.915878 7f866f3ee700 5 -- op tracker -- , seq: 3345, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33731 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2949120~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.915884 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.915917 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb600 prio 63 cost 516096 latency 0.000700 osd_op(client.4540.0:33731 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 2949120~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.916791 7f866f3ee700 5 -- op tracker -- , seq: 3346, time: 2014-06-09 16:22:11.915942, event: header_read, op: osd_op(client.4540.0:33732 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3465216~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.916804 7f866f3ee700 5 -- op tracker -- , seq: 3346, time: 2014-06-09 16:22:11.915943, event: throttled, op: osd_op(client.4540.0:33732 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3465216~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.916814 7f866f3ee700 5 -- op tracker -- , seq: 3346, time: 2014-06-09 16:22:11.916751, event: all_read, op: osd_op(client.4540.0:33732 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3465216~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.916824 7f866f3ee700 5 -- op tracker -- , seq: 3346, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33732 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3465216~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.916830 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.916844 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb500 prio 63 cost 516096 latency 0.000902 osd_op(client.4540.0:33732 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3465216~516096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.917112 7f866f3ee700 5 -- op tracker -- , seq: 3347, time: 2014-06-09 16:22:11.916863, event: header_read, op: osd_op(client.4540.0:33733 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.917123 7f866f3ee700 5 -- op tracker -- , seq: 3347, time: 2014-06-09 16:22:11.916863, event: throttled, op: osd_op(client.4540.0:33733 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.917134 7f866f3ee700 5 -- op tracker -- , seq: 3347, time: 2014-06-09 16:22:11.917090, event: all_read, op: osd_op(client.4540.0:33733 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.917156 7f866f3ee700 5 -- op tracker -- , seq: 3347, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33733 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.917164 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.917177 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb400 prio 63 cost 208896 latency 0.000314 osd_op(client.4540.0:33733 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3985408~208896] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.917233 7f866f3ee700 5 -- op tracker -- , seq: 3348, time: 2014-06-09 16:22:11.917197, event: header_read, op: osd_op(client.4540.0:33864 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3981312~4096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.917244 7f866f3ee700 5 -- op tracker -- , seq: 3348, time: 2014-06-09 16:22:11.917198, event: throttled, op: osd_op(client.4540.0:33864 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3981312~4096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.917254 7f866f3ee700 5 -- op tracker -- , seq: 3348, time: 2014-06-09 16:22:11.917211, event: all_read, op: osd_op(client.4540.0:33864 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3981312~4096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.917264 7f866f3ee700 5 -- op tracker -- , seq: 3348, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33864 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3981312~4096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.917282 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.917295 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb300 prio 63 cost 4096 latency 0.000098 osd_op(client.4540.0:33864 rbd_data.115c6b8b4567.00000000000017c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 3981312~4096] 3.7b49c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.918339 7f866f3ee700 5 -- op tracker -- , seq: 3349, time: 2014-06-09 16:22:11.917313, event: header_read, op: osd_op(client.4540.0:33928 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~667648] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.918351 7f866f3ee700 5 -- op tracker -- , seq: 3349, time: 2014-06-09 16:22:11.917314, event: throttled, op: osd_op(client.4540.0:33928 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~667648] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.918360 7f866f3ee700 5 -- op tracker -- , seq: 3349, time: 2014-06-09 16:22:11.918312, event: all_read, op: osd_op(client.4540.0:33928 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~667648] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.918371 7f866f3ee700 5 -- op tracker -- , seq: 3349, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33928 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~667648] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.918379 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.918412 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb200 prio 63 cost 667648 latency 0.001099 osd_op(client.4540.0:33928 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~667648] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.920550 7f866f3ee700 5 -- op tracker -- , seq: 3350, time: 2014-06-09 16:22:11.918432, event: header_read, op: osd_op(client.4540.0:33929 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~1548288] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.920560 7f866f3ee700 5 -- op tracker -- , seq: 3350, time: 2014-06-09 16:22:11.918433, event: throttled, op: osd_op(client.4540.0:33929 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~1548288] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.920566 7f866f3ee700 5 -- op tracker -- , seq: 3350, time: 2014-06-09 16:22:11.920513, event: all_read, op: osd_op(client.4540.0:33929 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~1548288] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.920571 7f866f3ee700 5 -- op tracker -- , seq: 3350, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33929 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~1548288] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.920575 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.920592 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddbc00 prio 63 cost 1548288 latency 0.002160 osd_op(client.4540.0:33929 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 667648~1548288] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.921735 7f866f3ee700 5 -- op tracker -- , seq: 3351, time: 2014-06-09 16:22:11.920610, event: header_read, op: osd_op(client.4540.0:33930 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~1032192] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.921744 7f866f3ee700 5 -- op tracker -- , seq: 3351, time: 2014-06-09 16:22:11.920611, event: throttled, op: osd_op(client.4540.0:33930 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~1032192] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.921750 7f866f3ee700 5 -- op tracker -- , seq: 3351, time: 2014-06-09 16:22:11.921717, event: all_read, op: osd_op(client.4540.0:33930 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~1032192] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.921755 7f866f3ee700 5 -- op tracker -- , seq: 3351, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33930 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~1032192] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.921758 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.921774 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddbb00 prio 63 cost 1032192 latency 0.001164 osd_op(client.4540.0:33930 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 2215936~1032192] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.923149 7f866f3ee700 5 -- op tracker -- , seq: 3352, time: 2014-06-09 16:22:11.921786, event: header_read, op: osd_op(client.4540.0:33931 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~946176] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923157 7f866f3ee700 5 -- op tracker -- , seq: 3352, time: 2014-06-09 16:22:11.921787, event: throttled, op: osd_op(client.4540.0:33931 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~946176] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923164 7f866f3ee700 5 -- op tracker -- , seq: 3352, time: 2014-06-09 16:22:11.923129, event: all_read, op: osd_op(client.4540.0:33931 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~946176] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923170 7f866f3ee700 5 -- op tracker -- , seq: 3352, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33931 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~946176] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923173 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.923200 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddba00 prio 63 cost 946176 latency 0.001413 osd_op(client.4540.0:33931 rbd_data.115c6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 3248128~946176] 3.569b715c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.923256 7f866f3ee700 5 -- op tracker -- , seq: 3353, time: 2014-06-09 16:22:11.923216, event: header_read, op: osd_op(client.4540.0:33998 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~20480] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923264 7f866f3ee700 5 -- op tracker -- , seq: 3353, time: 2014-06-09 16:22:11.923216, event: throttled, op: osd_op(client.4540.0:33998 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~20480] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923270 7f866f3ee700 5 -- op tracker -- , seq: 3353, time: 2014-06-09 16:22:11.923241, event: all_read, op: osd_op(client.4540.0:33998 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~20480] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923275 7f866f3ee700 5 -- op tracker -- , seq: 3353, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33998 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~20480] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923278 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.923288 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb900 prio 63 cost 20480 latency 0.000071 osd_op(client.4540.0:33998 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~20480] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.923947 7f866f3ee700 5 -- op tracker -- , seq: 3354, time: 2014-06-09 16:22:11.923304, event: header_read, op: osd_op(client.4540.0:33999 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 20480~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923959 7f866f3ee700 5 -- op tracker -- , seq: 3354, time: 2014-06-09 16:22:11.923305, event: throttled, op: osd_op(client.4540.0:33999 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 20480~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923973 7f866f3ee700 5 -- op tracker -- , seq: 3354, time: 2014-06-09 16:22:11.923903, event: all_read, op: osd_op(client.4540.0:33999 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 20480~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923978 7f866f3ee700 5 -- op tracker -- , seq: 3354, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:33999 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 20480~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.923982 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.923992 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb800 prio 63 cost 516096 latency 0.000688 osd_op(client.4540.0:33999 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 20480~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.924577 7f866f3ee700 5 -- op tracker -- , seq: 3355, time: 2014-06-09 16:22:11.924005, event: header_read, op: osd_op(client.4540.0:34000 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 536576~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.924586 7f866f3ee700 5 -- op tracker -- , seq: 3355, time: 2014-06-09 16:22:11.924005, event: throttled, op: osd_op(client.4540.0:34000 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 536576~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.924598 7f866f3ee700 5 -- op tracker -- , seq: 3355, time: 2014-06-09 16:22:11.924541, event: all_read, op: osd_op(client.4540.0:34000 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 536576~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.924604 7f866f3ee700 5 -- op tracker -- , seq: 3355, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:34000 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 536576~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.924607 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.924617 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddb700 prio 63 cost 516096 latency 0.000612 osd_op(client.4540.0:34000 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 536576~516096] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.926112 7f866f3ee700 5 -- op tracker -- , seq: 3356, time: 2014-06-09 16:22:11.924629, event: header_read, op: osd_op(client.4540.0:34001 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1052672~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.926119 7f866f3ee700 5 -- op tracker -- , seq: 3356, time: 2014-06-09 16:22:11.924630, event: throttled, op: osd_op(client.4540.0:34001 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1052672~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.926130 7f866f3ee700 5 -- op tracker -- , seq: 3356, time: 2014-06-09 16:22:11.926085, event: all_read, op: osd_op(client.4540.0:34001 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1052672~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.926138 7f866f3ee700 5 -- op tracker -- , seq: 3356, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:34001 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1052672~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.926142 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.926151 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddbf00 prio 63 cost 1032192 latency 0.001522 osd_op(client.4540.0:34001 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1052672~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.927552 7f866f3ee700 5 -- op tracker -- , seq: 3357, time: 2014-06-09 16:22:11.926166, event: header_read, op: osd_op(client.4540.0:34002 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2084864~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.927559 7f866f3ee700 5 -- op tracker -- , seq: 3357, time: 2014-06-09 16:22:11.926166, event: throttled, op: osd_op(client.4540.0:34002 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2084864~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.927564 7f866f3ee700 5 -- op tracker -- , seq: 3357, time: 2014-06-09 16:22:11.927527, event: all_read, op: osd_op(client.4540.0:34002 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2084864~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.927578 7f866f3ee700 5 -- op tracker -- , seq: 3357, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:34002 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2084864~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.927582 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.927592 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddbe00 prio 63 cost 1032192 latency 0.001425 osd_op(client.4540.0:34002 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 2084864~1032192] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.928902 7f866f3ee700 5 -- op tracker -- , seq: 3358, time: 2014-06-09 16:22:11.927604, event: header_read, op: osd_op(client.4540.0:34003 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3117056~1077248] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.928909 7f866f3ee700 5 -- op tracker -- , seq: 3358, time: 2014-06-09 16:22:11.927605, event: throttled, op: osd_op(client.4540.0:34003 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3117056~1077248] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.928915 7f866f3ee700 5 -- op tracker -- , seq: 3358, time: 2014-06-09 16:22:11.928877, event: all_read, op: osd_op(client.4540.0:34003 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3117056~1077248] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.928920 7f866f3ee700 5 -- op tracker -- , seq: 3358, time: 0.000000, event: dispatched, op: osd_op(client.4540.0:34003 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3117056~1077248] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.928924 7f866f3ee700 20 osd.26 243 should_share_map client.4540 172.16.96.220:0/1123692 243 2014-06-09 16:22:11.928935 7f866f3ee700 15 osd.26 243 enqueue_op 0x30ddbd00 prio 63 cost 1077248 latency 0.001330 osd_op(client.4540.0:34003 rbd_data.115c6b8b4567.00000000000017f4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3117056~1077248] 3.904cc25a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.942035 7f86a5610700 10 _do_op 0x324e71a0 seq 3588 r = 0, finisher 0xbccc260 0 2014-06-09 16:22:11.942047 7f86a5610700 10 _finish_op 0x324e71a0 seq 3588 osr(2.6cd 0x2ed1e70)/0x2ed1e70 2014-06-09 16:22:11.942053 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497b00 (1 active) 2014-06-09 16:22:11.942056 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339440 (2 active) 2014-06-09 16:22:11.942066 7f86a5610700 5 _do_op 0x324e77a0 seq 3598 osr(0.3b6 0x2bd72d0)/0x2bd72d0 start 2014-06-09 16:22:11.942069 7f86a5610700 10 _do_transaction on 0xc128480 2014-06-09 16:22:11.942076 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.942074 7f868cf5c700 10 op_submit_start 3646 2014-06-09 16:22:11.942079 7f868cf5c700 5 queue_transactions (trailing journal) 3646 2014-06-09 16:22:11.942081 7f868cf5c700 5 queue_op 0x288f8420 seq 3646 osr(0.663 0x2bd7f30) 828 bytes (queue has 50 ops and 563362 bytes) 2014-06-09 16:22:11.942086 7f868cf5c700 10 op_submit_finish 3646 2014-06-09 16:22:11.942093 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.942115 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.942127 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.942136 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.942149 7f868b759700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942156 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.942158 7f8691f66700 2 waiting 51 > 50 ops || 564220 > 104857600 2014-06-09 16:22:11.942164 7f868bf5a700 2 waiting 51 > 50 ops || 564188 > 104857600 2014-06-09 16:22:11.942170 7f868ef60700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942176 7f868e75f700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942179 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.942186 7f868a757700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942194 7f8689755700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942198 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.942204 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.942220 7f868df5e700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942217 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.942226 7f8692f68700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942226 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.942234 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.942247 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.942262 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.942279 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.942283 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.942296 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.942304 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.942312 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.942346 7f868cf5c700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.942349 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.942363 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.942371 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3597.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.942376 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.942378 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.942382 7f868ff62700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942382 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.16c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.942407 7f86a5e11700 10 _do_op 0x2e20ba0 seq 3597 r = 0, finisher 0x314128a0 0 2014-06-09 16:22:11.942412 7f86a5e11700 10 _finish_op 0x2e20ba0 seq 3597 osr(2.37a 0x2ed1160)/0x2ed1160 2014-06-09 16:22:11.942417 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286240 (1 active) 2014-06-09 16:22:11.942418 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.942419 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301d40 (2 active) 2014-06-09 16:22:11.942420 7f868b759700 10 op_submit_start 3647 2014-06-09 16:22:11.942422 7f868b759700 5 queue_transactions (trailing journal) 3647 2014-06-09 16:22:11.942422 7f86a5e11700 5 _do_op 0x2fcaef60 seq 3599 osr(0.6d2 0x384c1f0)/0x384c1f0 start 2014-06-09 16:22:11.942425 7f86a5e11700 10 _do_transaction on 0xc252000 2014-06-09 16:22:11.942424 7f868cf5c700 5 queue_transactions existing osr(0.16c 0x2bd67f0)/0x2bd67f0 2014-06-09 16:22:11.942424 7f868b759700 5 queue_op 0x324f47e0 seq 3647 osr(3.7e2 0xb7f0120) 828 bytes (queue has 50 ops and 563362 bytes) 2014-06-09 16:22:11.942428 7f868b759700 10 op_submit_finish 3647 2014-06-09 16:22:11.942427 7f868cf5c700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942430 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.942432 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:11.942443 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.942459 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3598.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.942463 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.942478 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.942485 7f868b759700 5 queue_transactions existing osr(3.31f 0x3ed0d50)/0x3ed0d50 2014-06-09 16:22:11.942487 7f868b759700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942492 7f8691f66700 2 waiting 51 > 50 ops || 564220 > 104857600 2014-06-09 16:22:11.942496 7f868bf5a700 2 waiting 51 > 50 ops || 564188 > 104857600 2014-06-09 16:22:11.942500 7f868ef60700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942509 7f868e75f700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942516 7f868a757700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942530 7f86a5610700 10 _do_op 0x324e77a0 seq 3598 r = 0, finisher 0xbd67280 0 2014-06-09 16:22:11.942531 7f8689755700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942535 7f86a5610700 10 _finish_op 0x324e77a0 seq 3598 osr(0.3b6 0x2bd72d0)/0x2bd72d0 2014-06-09 16:22:11.942541 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339440 (1 active) 2014-06-09 16:22:11.942542 7f8692f68700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942544 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524c60 (2 active) 2014-06-09 16:22:11.942544 7f868cf5c700 10 op_submit_start 3648 2014-06-09 16:22:11.942546 7f868cf5c700 5 queue_transactions (trailing journal) 3648 2014-06-09 16:22:11.942546 7f86a5610700 5 _do_op 0x2fcaf380 seq 3600 osr(1.589 0x384da10)/0x384da10 start 2014-06-09 16:22:11.942547 7f868ff62700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942549 7f86a5610700 10 _do_transaction on 0xc294840 2014-06-09 16:22:11.942548 7f868cf5c700 5 queue_op 0x288f8480 seq 3648 osr(0.16c 0x2bd67f0) 828 bytes (queue has 50 ops and 563362 bytes) 2014-06-09 16:22:11.942551 7f8691f66700 2 waiting 51 > 50 ops || 564220 > 104857600 2014-06-09 16:22:11.942552 7f868cf5c700 10 op_submit_finish 3648 2014-06-09 16:22:11.942554 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.942556 7f868b759700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942560 7f868ef60700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942564 7f868bf5a700 2 waiting 51 > 50 ops || 564188 > 104857600 2014-06-09 16:22:11.942556 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:11.942568 7f868a757700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942573 7f8689755700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942571 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.942581 7f868df5e700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942587 7f868e75f700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942582 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.942590 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.942601 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.942604 7f868cf5c700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.942616 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3599.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.942620 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.942620 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:988 2014-06-09 16:22:11.942635 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.942643 7f86a5e11700 10 _do_op 0x2fcaef60 seq 3599 r = 0, finisher 0x3061fee0 0 2014-06-09 16:22:11.942647 7f86a5e11700 10 _finish_op 0x2fcaef60 seq 3599 osr(0.6d2 0x384c1f0)/0x384c1f0 2014-06-09 16:22:11.942640 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.6ec( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.942657 7f868ff62700 10 op_submit_start 3649 2014-06-09 16:22:11.942658 7f868ff62700 5 queue_transactions (trailing journal) 3649 2014-06-09 16:22:11.942660 7f868ff62700 5 queue_op 0xc2d4660 seq 3649 osr(3.6d5 0x3ed1c30) 828 bytes (queue has 50 ops and 563362 bytes) 2014-06-09 16:22:11.942665 7f8691f66700 2 waiting 51 > 50 ops || 564220 > 104857600 2014-06-09 16:22:11.942670 7f868b759700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942673 7f868ef60700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942676 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:11.942677 7f868bf5a700 2 waiting 51 > 50 ops || 564188 > 104857600 2014-06-09 16:22:11.942680 7f868cf5c700 5 queue_transactions existing osr(2.6ec 0x2ed1e10)/0x2ed1e10 2014-06-09 16:22:11.942681 7f868a757700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942685 7f868cf5c700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942687 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301d40 (1 active) 2014-06-09 16:22:11.942689 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1159e0 (2 active) 2014-06-09 16:22:11.942692 7f868df5e700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942694 7f868ff62700 10 op_submit_finish 3649 2014-06-09 16:22:11.942698 7f868e75f700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942699 7f868ff62700 20 osd.26 243 Sending info 3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) to shard 3 2014-06-09 16:22:11.942705 7f8689755700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942707 7f868ff62700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.942710 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.942745 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e130 (19 active) 2014-06-09 16:22:11.942746 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3600.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.942750 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.942765 7f86a5610700 10 _do_op 0x2fcaf380 seq 3600 r = 0, finisher 0xbfcfea0 0 2014-06-09 16:22:11.942768 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e130 (20 active) 2014-06-09 16:22:11.942768 7f86a5610700 10 _finish_op 0x2fcaf380 seq 3600 osr(1.589 0x384da10)/0x384da10 2014-06-09 16:22:11.942772 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.6d5( v 213'532 (0'0,213'532] local-les=243 n=108 ec=118 les/c 243/243 242/242/205) 2014-06-09 16:22:11.942808 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.942814 7f868ff62700 5 queue_transactions existing osr(3.6d5 0x3ed1c30)/0x3ed1c30 2014-06-09 16:22:11.942820 7f86a5e11700 5 _do_op 0x33a2c00 seq 3601 osr(1.6e8 0x2ed0190)/0x2ed0190 start 2014-06-09 16:22:11.942823 7f86a5e11700 10 _do_transaction on 0x2c80cc0 2014-06-09 16:22:11.942828 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.942859 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524c60 (1 active) 2014-06-09 16:22:11.942862 7f868b759700 10 op_submit_start 3650 2014-06-09 16:22:11.942864 7f868b759700 5 queue_transactions (trailing journal) 3650 2014-06-09 16:22:11.942863 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2870e0 (2 active) 2014-06-09 16:22:11.942865 7f868b759700 5 queue_op 0x324f4660 seq 3650 osr(3.31f 0x3ed0d50) 828 bytes (queue has 50 ops and 563362 bytes) 2014-06-09 16:22:11.942866 7f86a5610700 5 _do_op 0x14dd66c0 seq 3602 osr(3.7ce 0xb7f01a0)/0xb7f01a0 start 2014-06-09 16:22:11.942868 7f868b759700 10 op_submit_finish 3650 2014-06-09 16:22:11.942868 7f86a5610700 10 _do_transaction on 0x471c0c0 2014-06-09 16:22:11.942873 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.942870 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.681( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.942896 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.942900 7f868b759700 5 queue_transactions existing osr(0.681 0x2bd7e10)/0x2bd7e10 2014-06-09 16:22:11.942901 7f868b759700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942904 7f868ef60700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942908 7f868bf5a700 2 waiting 51 > 50 ops || 564188 > 104857600 2014-06-09 16:22:11.942914 7f868a757700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942920 7f868cf5c700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942924 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.942927 7f868df5e700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942933 7f868e75f700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942939 7f8689755700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942939 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3601.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.942943 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.942945 7f8691f66700 2 waiting 51 > 50 ops || 564220 > 104857600 2014-06-09 16:22:11.942950 7f868ff62700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.942955 7f8692f68700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943018 7f86a5e11700 10 _do_op 0x33a2c00 seq 3601 r = 0, finisher 0xbd76240 0 2014-06-09 16:22:11.943022 7f86a5e11700 10 _finish_op 0x33a2c00 seq 3601 osr(1.6e8 0x2ed0190)/0x2ed0190 2014-06-09 16:22:11.943033 7f868ef60700 10 op_submit_start 3651 2014-06-09 16:22:11.943034 7f868ef60700 5 queue_transactions (trailing journal) 3651 2014-06-09 16:22:11.943034 7f868bf5a700 2 waiting 51 > 50 ops || 564188 > 104857600 2014-06-09 16:22:11.943036 7f868ef60700 5 queue_op 0x11fcfaa0 seq 3651 osr(1.589 0x384da10) 828 bytes (queue has 50 ops and 563362 bytes) 2014-06-09 16:22:11.943041 7f868a757700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943045 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1159e0 (1 active) 2014-06-09 16:22:11.943045 7f868cf5c700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943047 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b320 (2 active) 2014-06-09 16:22:11.943052 7f868ef60700 10 op_submit_finish 3651 2014-06-09 16:22:11.943052 7f868df5e700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943058 7f868e75f700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943063 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.943064 7f8689755700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943056 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.943069 7f8691f66700 2 waiting 51 > 50 ops || 564220 > 104857600 2014-06-09 16:22:11.943070 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.943079 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.943084 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3602.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.943088 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.943088 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.943096 7f86a5610700 10 _do_op 0x14dd66c0 seq 3602 r = 0, finisher 0x31412580 0 2014-06-09 16:22:11.943099 7f86a5610700 10 _finish_op 0x14dd66c0 seq 3602 osr(3.7ce 0xb7f01a0)/0xb7f01a0 2014-06-09 16:22:11.943096 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.943109 7f8692f68700 10 op_submit_start 3652 2014-06-09 16:22:11.943110 7f8692f68700 5 queue_transactions (trailing journal) 3652 2014-06-09 16:22:11.943104 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.943112 7f8692f68700 5 queue_op 0x24105f80 seq 3652 osr(0.1a4 0x2bd6710) 828 bytes (queue has 50 ops and 563362 bytes) 2014-06-09 16:22:11.943117 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2870e0 (1 active) 2014-06-09 16:22:11.943119 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca486c0 (2 active) 2014-06-09 16:22:11.943123 7f8692f68700 10 op_submit_finish 3652 2014-06-09 16:22:11.943118 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:162 2014-06-09 16:22:11.943127 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.943127 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.697( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.943135 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.943143 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.943148 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.943157 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.943161 7f8692f68700 5 queue_transactions existing osr(1.697 0x384de10)/0x384de10 2014-06-09 16:22:11.943163 7f8692f68700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943159 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.886902 6 0.000358 2014-06-09 16:22:11.943167 7f86a5610700 5 _do_op 0x308ca4e0 seq 3604 osr(3.5 0x3ed03f0)/0x3ed03f0 start 2014-06-09 16:22:11.943170 7f86a5610700 10 _do_transaction on 0xc000b40 2014-06-09 16:22:11.943175 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.943172 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.943182 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.943193 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:11.943203 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.943211 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.943212 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.943218 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.943227 7f86a5610700 15 _omap_rmkeyrange meta/103a006e/pglog_3.5/0//-1 [0000000214.00000000000000000622,4294967295.18446744073709551615] 2014-06-09 16:22:11.943230 7f86a5610700 15 get_omap_iterator meta/103a006e/pglog_3.5/0//-1 2014-06-09 16:22:11.943230 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.943242 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.943251 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.943263 7f868ef60700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.943278 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:11.943310 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.943314 7f868ef60700 5 queue_transactions existing osr(0.30b 0x2bd6dd0)/0x2bd6dd0 2014-06-09 16:22:11.943317 7f868ef60700 2 waiting 51 > 50 ops || 564223 > 104857600 2014-06-09 16:22:11.943321 7f868b759700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943326 7f868a757700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943329 7f868ff62700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943334 7f86a5e11700 5 _do_op 0xbe7cde0 seq 3603 osr(1.1fb 0x384cd30)/0x384cd30 start 2014-06-09 16:22:11.943334 7f868df5e700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943337 7f86a5e11700 10 _do_transaction on 0xbe89680 2014-06-09 16:22:11.943343 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.943342 7f868e75f700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943349 7f8689755700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943354 7f8691f66700 2 waiting 51 > 50 ops || 564220 > 104857600 2014-06-09 16:22:11.943359 7f868bf5a700 2 waiting 51 > 50 ops || 564188 > 104857600 2014-06-09 16:22:11.943364 7f868cf5c700 2 waiting 51 > 50 ops || 564190 > 104857600 2014-06-09 16:22:11.943383 7f86a5610700 15 _omap_rmkeys meta/103a006e/pglog_3.5/0//-1 2014-06-09 16:22:11.943411 7f86a5610700 10 lookup_strip_header done cid meta oid 103a006e/pglog_3.5/0//-1 2014-06-09 16:22:11.943418 7f86a5610700 10 _omap_rmkeys meta/103a006e/pglog_3.5/0//-1 r = 0 2014-06-09 16:22:11.943422 7f86a5610700 15 _omap_rmkeys meta/103a006e/pglog_3.5/0//-1 2014-06-09 16:22:11.943424 7f86a5610700 10 _omap_rmkeys meta/103a006e/pglog_3.5/0//-1 r = 0 2014-06-09 16:22:11.943429 7f86a5610700 15 _omap_setkeys meta/103a006e/pglog_3.5/0//-1 2014-06-09 16:22:11.943440 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3604.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.943444 7f86a5610700 10 cid: oid: 103a006e/pglog_3.5/0//-1 skipping op, spos 3604.0.0 <= header.spos 150439.0.0 2014-06-09 16:22:11.943446 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.943467 7f86a5610700 10 _do_op 0x308ca4e0 seq 3604 r = 0, finisher 0xbd5ff20 0 2014-06-09 16:22:11.943471 7f86a5610700 10 _finish_op 0x308ca4e0 seq 3604 osr(3.5 0x3ed03f0)/0x3ed03f0 2014-06-09 16:22:11.943480 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca486c0 (1 active) 2014-06-09 16:22:11.943482 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1e60 (2 active) 2014-06-09 16:22:11.943483 7f868ef60700 10 op_submit_start 3653 2014-06-09 16:22:11.943484 7f868ef60700 5 queue_transactions (trailing journal) 3653 2014-06-09 16:22:11.943484 7f86a5610700 5 _do_op 0x324e6060 seq 3605 osr(3.66a 0x3ed19d0)/0x3ed19d0 start 2014-06-09 16:22:11.943487 7f86a5610700 10 _do_transaction on 0xbf50f00 2014-06-09 16:22:11.943486 7f868ef60700 5 queue_op 0x15996d80 seq 3653 osr(0.30b 0x2bd6dd0) 861 bytes (queue has 50 ops and 562292 bytes) 2014-06-09 16:22:11.943488 7f868ef60700 10 op_submit_finish 3653 2014-06-09 16:22:11.943491 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.943491 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.943503 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.907543 8 0.000376 2014-06-09 16:22:11.943509 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.943516 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.943521 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000012 0 0.000000 2014-06-09 16:22:11.943527 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.943532 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.943536 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.943542 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.943549 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.943554 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.943561 7f868ef60700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.943571 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:11.943591 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.943593 7f868ef60700 5 queue_transactions existing osr(2.17c 0x2ed09b0)/0x2ed09b0 2014-06-09 16:22:11.943594 7f868ef60700 2 waiting 51 > 50 ops || 563153 > 104857600 2014-06-09 16:22:11.943597 7f868b759700 2 waiting 51 > 50 ops || 563120 > 104857600 2014-06-09 16:22:11.943602 7f868a757700 2 waiting 51 > 50 ops || 563120 > 104857600 2014-06-09 16:22:11.943608 7f868ff62700 2 waiting 51 > 50 ops || 563120 > 104857600 2014-06-09 16:22:11.943613 7f868df5e700 2 waiting 51 > 50 ops || 563120 > 104857600 2014-06-09 16:22:11.943629 7f868e75f700 2 waiting 51 > 50 ops || 563120 > 104857600 2014-06-09 16:22:11.943633 7f8689755700 2 waiting 51 > 50 ops || 563120 > 104857600 2014-06-09 16:22:11.943637 7f8691f66700 2 waiting 51 > 50 ops || 563150 > 104857600 2014-06-09 16:22:11.943641 7f868bf5a700 2 waiting 51 > 50 ops || 563118 > 104857600 2014-06-09 16:22:11.943647 7f868cf5c700 2 waiting 51 > 50 ops || 563120 > 104857600 2014-06-09 16:22:11.943648 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.943654 7f8692f68700 2 waiting 51 > 50 ops || 563120 > 104857600 2014-06-09 16:22:11.943666 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3603.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.943670 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.943693 7f86a5e11700 10 _do_op 0xbe7cde0 seq 3603 r = 0, finisher 0xbd5c300 0 2014-06-09 16:22:11.943698 7f86a5e11700 10 _finish_op 0xbe7cde0 seq 3603 osr(1.1fb 0x384cd30)/0x384cd30 2014-06-09 16:22:11.943702 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b320 (1 active) 2014-06-09 16:22:11.943705 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52fe60 (2 active) 2014-06-09 16:22:11.943705 7f868ef60700 10 op_submit_start 3654 2014-06-09 16:22:11.943707 7f868ef60700 5 queue_transactions (trailing journal) 3654 2014-06-09 16:22:11.943707 7f86a5e11700 5 _do_op 0x324e6a20 seq 3606 osr(2.73 0x2ed0510)/0x2ed0510 start 2014-06-09 16:22:11.943709 7f86a5e11700 10 _do_transaction on 0xc0e5440 2014-06-09 16:22:11.943708 7f868ef60700 5 queue_op 0x30606780 seq 3654 osr(2.17c 0x2ed09b0) 861 bytes (queue has 50 ops and 561270 bytes) 2014-06-09 16:22:11.943711 7f868ef60700 10 op_submit_finish 3654 2014-06-09 16:22:11.943713 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.943713 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.943728 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.943736 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x19fe0100,0x19fe0000,0x1c54b500,0x1c54b400,0x1c54b300,0x1c54b200,0x1c54b100 2014-06-09 16:22:11.943748 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.943761 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3606.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.943765 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.943773 7f86a5e11700 10 _do_op 0x324e6a20 seq 3606 r = 0, finisher 0x318c2b40 0 2014-06-09 16:22:11.943767 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.943776 7f86a5e11700 10 _finish_op 0x324e6a20 seq 3606 osr(2.73 0x2ed0510)/0x2ed0510 2014-06-09 16:22:11.943778 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52fe60 (1 active) 2014-06-09 16:22:11.943780 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300ea0 (2 active) 2014-06-09 16:22:11.943782 7f86a5e11700 5 _do_op 0x324e6ae0 seq 3607 osr(1.82 0x384c4d0)/0x384c4d0 start 2014-06-09 16:22:11.943776 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.943785 7f86a5e11700 10 _do_transaction on 0xbf29d40 2014-06-09 16:22:11.943789 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.943784 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.943797 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1057 2014-06-09 16:22:11.943807 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.943816 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.943823 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.943826 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.943835 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3607.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.943838 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.943830 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.943846 7f86a5e11700 10 _do_op 0x324e6ae0 seq 3607 r = 0, finisher 0x18742a00 0 2014-06-09 16:22:11.943842 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.943849 7f86a5e11700 10 _finish_op 0x324e6ae0 seq 3607 osr(1.82 0x384c4d0)/0x384c4d0 2014-06-09 16:22:11.943851 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300ea0 (1 active) 2014-06-09 16:22:11.943853 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9225a0 (2 active) 2014-06-09 16:22:11.943849 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.943855 7f86a5e11700 5 _do_op 0x324e6ba0 seq 3608 osr(0.772 0x384c2f0)/0x384c2f0 start 2014-06-09 16:22:11.943857 7f86a5e11700 10 _do_transaction on 0x2ed49c0 2014-06-09 16:22:11.943862 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.943857 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.943864 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.943871 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.943884 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 2014-06-09 16:22:11.943895 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.943892 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.943900 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.943908 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.943910 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3608.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.943914 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.943913 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.943924 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.943931 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.943936 7f86a5e11700 10 _do_op 0x324e6ba0 seq 3608 r = 0, finisher 0x15a3f1e0 0 2014-06-09 16:22:11.943939 7f86a5e11700 10 _finish_op 0x324e6ba0 seq 3608 osr(0.772 0x384c2f0)/0x384c2f0 2014-06-09 16:22:11.943942 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9225a0 (1 active) 2014-06-09 16:22:11.943944 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1150e0 (2 active) 2014-06-09 16:22:11.943938 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.943946 7f86a5e11700 5 _do_op 0xd470300 seq 3609 osr(2.37 0x2ed05f0)/0x2ed05f0 start 2014-06-09 16:22:11.943948 7f86a5e11700 10 _do_transaction on 0xc33e0c0 2014-06-09 16:22:11.943946 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.943953 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.943953 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.943965 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:186 2014-06-09 16:22:11.943973 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.943981 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.943987 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.943989 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.943999 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3609.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944003 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.943994 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944011 7f86a5e11700 10 _do_op 0xd470300 seq 3609 r = 0, finisher 0x3061edc0 0 2014-06-09 16:22:11.944006 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944014 7f86a5e11700 10 _finish_op 0xd470300 seq 3609 osr(2.37 0x2ed05f0)/0x2ed05f0 2014-06-09 16:22:11.944016 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1150e0 (1 active) 2014-06-09 16:22:11.944018 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbd40 (2 active) 2014-06-09 16:22:11.944020 7f86a5e11700 5 _do_op 0x24104240 seq 3610 osr(3.603 0x3ed1af0)/0x3ed1af0 start 2014-06-09 16:22:11.944022 7f86a5e11700 10 _do_transaction on 0xc2aa240 2014-06-09 16:22:11.944015 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x26aa9900,0x26aa9800,0x26aa9700,0x26aa9600,0x26aa9500 2014-06-09 16:22:11.944026 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944036 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944046 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944059 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944054 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944072 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3610.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944075 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944067 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1222 2014-06-09 16:22:11.944083 7f86a5e11700 10 _do_op 0x24104240 seq 3610 r = 0, finisher 0x3050b040 0 2014-06-09 16:22:11.944077 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944086 7f86a5e11700 10 _finish_op 0x24104240 seq 3610 osr(3.603 0x3ed1af0)/0x3ed1af0 2014-06-09 16:22:11.944088 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbd40 (1 active) 2014-06-09 16:22:11.944090 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383ed80 (2 active) 2014-06-09 16:22:11.944092 7f86a5e11700 5 _do_op 0x2fcaf920 seq 3611 osr(3.19 0x3ed03a0)/0x3ed03a0 start 2014-06-09 16:22:11.944086 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944094 7f86a5e11700 10 _do_transaction on 0xc2de840 2014-06-09 16:22:11.944096 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944098 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944100 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944111 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944119 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944131 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944126 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944134 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944144 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3611.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944147 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944142 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944160 7f86a5e11700 10 _do_op 0x2fcaf920 seq 3611 r = 0, finisher 0xbd5c620 0 2014-06-09 16:22:11.944154 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.944163 7f86a5e11700 10 _finish_op 0x2fcaf920 seq 3611 osr(3.19 0x3ed03a0)/0x3ed03a0 2014-06-09 16:22:11.944165 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383ed80 (1 active) 2014-06-09 16:22:11.944167 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f200 (2 active) 2014-06-09 16:22:11.944163 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944169 7f86a5e11700 5 _do_op 0x2fcae420 seq 3612 osr(2.e6 0x2ed07d0)/0x2ed07d0 start 2014-06-09 16:22:11.944171 7f86a5e11700 10 _do_transaction on 0xc2dfe00 2014-06-09 16:22:11.944171 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944176 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944178 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944182 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944194 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944201 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944209 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944209 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944216 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944222 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3612.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944225 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944224 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944233 7f86a5e11700 10 _do_op 0x2fcae420 seq 3612 r = 0, finisher 0xbd5c4c0 0 2014-06-09 16:22:11.944235 7f86a5e11700 10 _finish_op 0x2fcae420 seq 3612 osr(2.e6 0x2ed07d0)/0x2ed07d0 2014-06-09 16:22:11.944238 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f200 (1 active) 2014-06-09 16:22:11.944239 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496c60 (2 active) 2014-06-09 16:22:11.944241 7f86a5e11700 5 _do_op 0x2fcae600 seq 3613 osr(2.3d1 0x2ed13b0)/0x2ed13b0 start 2014-06-09 16:22:11.944235 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.944243 7f86a5e11700 10 _do_transaction on 0xc0d6c00 2014-06-09 16:22:11.944247 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944244 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944250 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944258 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944262 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944273 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944281 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944289 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944296 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944304 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944316 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.944324 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944331 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944340 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.944349 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.944350 7f869af78700 20 osd.26 243 _dispatch 0x12c09c00 pg_info(1 pgs e243:3.5) v4 2014-06-09 16:22:11.944344 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944353 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944356 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944358 7f869af78700 5 -- op tracker -- , seq: 3359, time: 2014-06-09 16:22:11.944299, event: header_read, op: pg_info(1 pgs e243:3.5) 2014-06-09 16:22:11.944367 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3613.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944364 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944367 7f869af78700 5 -- op tracker -- , seq: 3359, time: 2014-06-09 16:22:11.944301, event: throttled, op: pg_info(1 pgs e243:3.5) 2014-06-09 16:22:11.944370 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944371 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944374 7f869af78700 5 -- op tracker -- , seq: 3359, time: 2014-06-09 16:22:11.944317, event: all_read, op: pg_info(1 pgs e243:3.5) 2014-06-09 16:22:11.944380 7f86a5e11700 10 _do_op 0x2fcae600 seq 3613 r = 0, finisher 0xbd5caa0 0 2014-06-09 16:22:11.944382 7f86a5e11700 10 _finish_op 0x2fcae600 seq 3613 osr(2.3d1 0x2ed13b0)/0x2ed13b0 2014-06-09 16:22:11.944385 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496c60 (1 active) 2014-06-09 16:22:11.944379 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944386 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497680 (2 active) 2014-06-09 16:22:11.944388 7f86a5e11700 5 _do_op 0xbe7c480 seq 3614 osr(0.369 0x2bd6ff0)/0x2bd6ff0 start 2014-06-09 16:22:11.944391 7f86a5e11700 10 _do_transaction on 0xc320600 2014-06-09 16:22:11.944387 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944382 7f869af78700 5 -- op tracker -- , seq: 3359, time: 2014-06-09 16:22:11.944345, event: dispatched, op: pg_info(1 pgs e243:3.5) 2014-06-09 16:22:11.944395 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944398 7f869af78700 5 -- op tracker -- , seq: 3359, time: 2014-06-09 16:22:11.944396, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5) 2014-06-09 16:22:11.944399 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.944404 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5) v4 from osd.3 2014-06-09 16:22:11.944408 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12c09c00 2014-06-09 16:22:11.944407 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944415 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944415 7f869af78700 5 -- op tracker -- , seq: 3359, time: 2014-06-09 16:22:11.944412, event: started, op: pg_info(1 pgs e243:3.5) 2014-06-09 16:22:11.944424 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944428 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944428 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944441 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3614.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944444 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944439 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944440 7f869af78700 5 -- op tracker -- , seq: 3359, time: 2014-06-09 16:22:11.944437, event: done, op: pg_info(1 pgs e243:3.5) 2014-06-09 16:22:11.944451 7f86a5e11700 10 _do_op 0xbe7c480 seq 3614 r = 0, finisher 0xbd5c280 0 2014-06-09 16:22:11.944447 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944454 7f86a5e11700 10 _finish_op 0xbe7c480 seq 3614 osr(0.369 0x2bd6ff0)/0x2bd6ff0 2014-06-09 16:22:11.944456 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497680 (1 active) 2014-06-09 16:22:11.944456 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.944458 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e360 (2 active) 2014-06-09 16:22:11.944458 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.944454 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944460 7f86a5e11700 5 _do_op 0x2fcaf320 seq 3615 osr(3.3a9 0x3ed0f60)/0x3ed0f60 start 2014-06-09 16:22:11.944463 7f86a5e11700 10 _do_transaction on 0xc228780 2014-06-09 16:22:11.944467 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944462 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944469 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944481 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.944489 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944499 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944496 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944505 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944511 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3615.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944514 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944509 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944519 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944529 7f86a5e11700 10 _do_op 0x2fcaf320 seq 3615 r = 0, finisher 0xbd5e5e0 0 2014-06-09 16:22:11.944531 7f86a5e11700 10 _finish_op 0x2fcaf320 seq 3615 osr(3.3a9 0x3ed0f60)/0x3ed0f60 2014-06-09 16:22:11.944527 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944534 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e360 (1 active) 2014-06-09 16:22:11.944536 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62eb40 (2 active) 2014-06-09 16:22:11.944538 7f86a5e11700 5 _do_op 0xbe7dc80 seq 3616 osr(2.364 0x2ed11b0)/0x2ed11b0 start 2014-06-09 16:22:11.944540 7f86a5e11700 10 _do_transaction on 0xbfb29c0 2014-06-09 16:22:11.944534 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944544 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944541 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944548 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944560 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:162 2014-06-09 16:22:11.944568 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944578 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944576 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944584 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944589 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3616.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944593 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944588 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944602 7f86a5e11700 10 _do_op 0xbe7dc80 seq 3616 r = 0, finisher 0xbe30300 0 2014-06-09 16:22:11.944599 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944604 7f86a5e11700 10 _finish_op 0xbe7dc80 seq 3616 osr(2.364 0x2ed11b0)/0x2ed11b0 2014-06-09 16:22:11.944607 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62eb40 (1 active) 2014-06-09 16:22:11.944609 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497680 (2 active) 2014-06-09 16:22:11.944610 7f86a5e11700 5 _do_op 0xbe7d5c0 seq 3617 osr(0.369 0x2bd6ff0)/0x2bd6ff0 start 2014-06-09 16:22:11.944606 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944612 7f86a5e11700 10 _do_transaction on 0xbe88000 2014-06-09 16:22:11.944616 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944613 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944620 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944627 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944640 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.944648 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944648 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944655 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944661 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3617.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944664 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944664 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944671 7f86a5e11700 10 _do_op 0xbe7d5c0 seq 3617 r = 0, finisher 0xccf88e0 0 2014-06-09 16:22:11.944674 7f86a5e11700 10 _finish_op 0xbe7d5c0 seq 3617 osr(0.369 0x2bd6ff0)/0x2bd6ff0 2014-06-09 16:22:11.944668 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944676 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497680 (1 active) 2014-06-09 16:22:11.944678 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbd40 (2 active) 2014-06-09 16:22:11.944680 7f86a5e11700 5 _do_op 0xbe7df80 seq 3618 osr(3.603 0x3ed1af0)/0x3ed1af0 start 2014-06-09 16:22:11.944682 7f86a5e11700 10 _do_transaction on 0xbf2f380 2014-06-09 16:22:11.944679 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944686 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944686 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944693 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944701 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944708 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944719 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944719 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.944731 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3618.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944727 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944734 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944735 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944742 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944743 7f86a5e11700 10 _do_op 0xbe7df80 seq 3618 r = 0, finisher 0xc40c680 0 2014-06-09 16:22:11.944746 7f86a5e11700 10 _finish_op 0xbe7df80 seq 3618 osr(3.603 0x3ed1af0)/0x3ed1af0 2014-06-09 16:22:11.944748 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbd40 (1 active) 2014-06-09 16:22:11.944750 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1159e0 (2 active) 2014-06-09 16:22:11.944752 7f86a5e11700 5 _do_op 0xbe7d620 seq 3619 osr(1.6e8 0x2ed0190)/0x2ed0190 start 2014-06-09 16:22:11.944754 7f86a5e11700 10 _do_transaction on 0x2ed5800 2014-06-09 16:22:11.944747 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944758 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944759 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944767 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944776 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944791 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944785 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944794 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944803 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3619.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944806 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944807 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1094 2014-06-09 16:22:11.944814 7f86a5e11700 10 _do_op 0xbe7d620 seq 3619 r = 0, finisher 0xc1695e0 0 2014-06-09 16:22:11.944817 7f86a5e11700 10 _finish_op 0xbe7d620 seq 3619 osr(1.6e8 0x2ed0190)/0x2ed0190 2014-06-09 16:22:11.944819 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1159e0 (1 active) 2014-06-09 16:22:11.944821 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcfc0 (2 active) 2014-06-09 16:22:11.944816 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944823 7f86a5e11700 5 _do_op 0x308cba40 seq 3620 osr(3.609 0x3ed1b00)/0x3ed1b00 start 2014-06-09 16:22:11.944825 7f86a5e11700 10 _do_transaction on 0xc2dc480 2014-06-09 16:22:11.944830 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944824 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944834 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944838 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.944849 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.944856 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.944862 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944864 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.944875 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3620.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944872 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.944878 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944893 7f86a5e11700 10 _do_op 0x308cba40 seq 3620 r = 0, finisher 0xbd5c920 0 2014-06-09 16:22:11.944896 7f86a5e11700 10 _finish_op 0x308cba40 seq 3620 osr(3.609 0x3ed1b00)/0x3ed1b00 2014-06-09 16:22:11.944898 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcfc0 (1 active) 2014-06-09 16:22:11.944900 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4d80 (2 active) 2014-06-09 16:22:11.944902 7f86a5e11700 5 _do_op 0x308cbaa0 seq 3621 osr(2.2b 0x2ed0230)/0x2ed0230 start 2014-06-09 16:22:11.944905 7f86a5e11700 10 _do_transaction on 0xbfec840 2014-06-09 16:22:11.944909 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944879 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.944942 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.944946 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.944954 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3621.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.944958 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.944955 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.944966 7f86a5e11700 10 _do_op 0x308cbaa0 seq 3621 r = 0, finisher 0x324f9ae0 0 2014-06-09 16:22:11.944962 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.944968 7f86a5e11700 10 _finish_op 0x308cbaa0 seq 3621 osr(2.2b 0x2ed0230)/0x2ed0230 2014-06-09 16:22:11.944970 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.944971 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4d80 (1 active) 2014-06-09 16:22:11.944972 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f9e0 (2 active) 2014-06-09 16:22:11.944974 7f86a5e11700 5 _do_op 0x30606060 seq 3622 osr(2.620 0x2ed1cb0)/0x2ed1cb0 start 2014-06-09 16:22:11.944977 7f86a5e11700 10 _do_transaction on 0x390d680 2014-06-09 16:22:11.944981 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.944974 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.944985 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.889254 7 0.000384 2014-06-09 16:22:11.944996 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.945006 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945014 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945015 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:11.945026 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3622.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945029 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945024 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.945032 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.945037 7f86a5e11700 10 _do_op 0x30606060 seq 3622 r = 0, finisher 0x3251cbe0 0 2014-06-09 16:22:11.945040 7f86a5e11700 10 _finish_op 0x30606060 seq 3622 osr(2.620 0x2ed1cb0)/0x2ed1cb0 2014-06-09 16:22:11.945042 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f9e0 (1 active) 2014-06-09 16:22:11.945044 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497b00 (2 active) 2014-06-09 16:22:11.945039 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.945046 7f86a5e11700 5 _do_op 0x324f58c0 seq 3623 osr(2.6cd 0x2ed1e70)/0x2ed1e70 start 2014-06-09 16:22:11.945048 7f86a5e11700 10 _do_transaction on 0xbf82fc0 2014-06-09 16:22:11.945052 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945049 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:11.945059 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:11.945069 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.945081 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.945084 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945096 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3623.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945100 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945096 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:11.945109 7f86a5e11700 10 _do_op 0x324f58c0 seq 3623 r = 0, finisher 0x30628040 0 2014-06-09 16:22:11.945112 7f86a5e11700 10 _finish_op 0x324f58c0 seq 3623 osr(2.6cd 0x2ed1e70)/0x2ed1e70 2014-06-09 16:22:11.945114 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497b00 (1 active) 2014-06-09 16:22:11.945116 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4240 (2 active) 2014-06-09 16:22:11.945118 7f86a5e11700 5 _do_op 0xc705080 seq 3624 osr(1.5e 0x384c5d0)/0x384c5d0 start 2014-06-09 16:22:11.945120 7f86a5e11700 10 _do_transaction on 0xc148300 2014-06-09 16:22:11.945124 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945127 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945131 7f868ef60700 5 queue_transactions existing osr(2.18 0x2ed0290)/0x2ed0290 2014-06-09 16:22:11.945134 7f868ef60700 10 op_submit_start 3655 2014-06-09 16:22:11.945135 7f868ef60700 5 queue_transactions (trailing journal) 3655 2014-06-09 16:22:11.945136 7f868ef60700 5 queue_op 0x14dd6f00 seq 3655 osr(2.18 0x2ed0290) 858 bytes (queue has 33 ops and 546181 bytes) 2014-06-09 16:22:11.945138 7f868ef60700 10 op_submit_finish 3655 2014-06-09 16:22:11.945151 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41ef0 (19 active) 2014-06-09 16:22:11.945158 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945158 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.945172 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3624.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945175 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945185 7f86a5e11700 10 _do_op 0xc705080 seq 3624 r = 0, finisher 0x314136c0 0 2014-06-09 16:22:11.945174 7f868ef60700 10 osd.26 243 dequeue_op 0x32827e00 prio 196 cost 0 latency 0.076946 osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) v2 pg pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.945188 7f86a5e11700 10 _finish_op 0xc705080 seq 3624 osr(1.5e 0x384c5d0)/0x384c5d0 2014-06-09 16:22:11.945190 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4240 (1 active) 2014-06-09 16:22:11.945192 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f0e0 (2 active) 2014-06-09 16:22:11.945194 7f86a5e11700 5 _do_op 0x324f4ae0 seq 3625 osr(1.676 0x384dee0)/0x384dee0 start 2014-06-09 16:22:11.945196 7f86a5e11700 10 _do_transaction on 0xc0d1380 2014-06-09 16:22:11.945193 7f868ef60700 5 -- op tracker -- , seq: 3325, time: 2014-06-09 16:22:11.945190, event: reached_pg, op: osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.945200 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945202 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x32827e00 2014-06-09 16:22:11.945216 7f868ef60700 5 -- op tracker -- , seq: 3325, time: 2014-06-09 16:22:11.945213, event: started, op: osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.945234 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945227 7f868ef60700 7 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 14 op ack_type 4 from 9 2014-06-09 16:22:11.945246 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3625.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945250 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945247 7f868ef60700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.945239, event: sub_op_commit_rec, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.945254 7f868ef60700 10 osd.26 243 dequeue_op 0x32827e00 finish 2014-06-09 16:22:11.945261 7f868ef60700 5 -- op tracker -- , seq: 3325, time: 2014-06-09 16:22:11.945257, event: done, op: osd_sub_op_reply(client.4571.0:35950 3.1e 78ae781e/rbd_data.117b6b8b4567.00000000000017ff/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.945265 7f86a5e11700 10 _do_op 0x324f4ae0 seq 3625 r = 0, finisher 0x17fb76c0 0 2014-06-09 16:22:11.945268 7f86a5e11700 10 _finish_op 0x324f4ae0 seq 3625 osr(1.676 0x384dee0)/0x384dee0 2014-06-09 16:22:11.945270 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f0e0 (1 active) 2014-06-09 16:22:11.945272 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358c60 (2 active) 2014-06-09 16:22:11.945273 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.945273 7f86a5e11700 5 _do_op 0xc705620 seq 3626 osr(1.691 0x384de30)/0x384de30 start 2014-06-09 16:22:11.945276 7f86a5e11700 10 _do_transaction on 0xc253a40 2014-06-09 16:22:11.945280 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945290 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:11.945294 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.945313 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945308 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.945317 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1c769b00,0x1c769a00,0x1c769900,0x1c769800,0x1c769700 2014-06-09 16:22:11.945326 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3626.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945330 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945337 7f86a5e11700 10 _do_op 0xc705620 seq 3626 r = 0, finisher 0x139654c0 0 2014-06-09 16:22:11.945340 7f86a5e11700 10 _finish_op 0xc705620 seq 3626 osr(1.691 0x384de30)/0x384de30 2014-06-09 16:22:11.945342 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358c60 (1 active) 2014-06-09 16:22:11.945344 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f0e0 (2 active) 2014-06-09 16:22:11.945339 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945346 7f86a5e11700 5 _do_op 0x324f5680 seq 3627 osr(2.113 0x2ed06b0)/0x2ed06b0 start 2014-06-09 16:22:11.945349 7f86a5e11700 10 _do_transaction on 0x398be00 2014-06-09 16:22:11.945353 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945348 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.945356 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.945369 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1113 2014-06-09 16:22:11.945379 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.945386 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945387 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.945397 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945399 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3627.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945402 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945402 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.945410 7f86a5e11700 10 _do_op 0x324f5680 seq 3627 r = 0, finisher 0x3251cf60 0 2014-06-09 16:22:11.945412 7f86a5e11700 10 _finish_op 0x324f5680 seq 3627 osr(2.113 0x2ed06b0)/0x2ed06b0 2014-06-09 16:22:11.945415 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f0e0 (1 active) 2014-06-09 16:22:11.945416 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d440 (2 active) 2014-06-09 16:22:11.945413 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.945418 7f86a5e11700 5 _do_op 0x324f4960 seq 3628 osr(1.56 0x384c5f0)/0x384c5f0 start 2014-06-09 16:22:11.945420 7f86a5e11700 10 _do_transaction on 0x3145c80 2014-06-09 16:22:11.945420 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.945425 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945427 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945435 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.945442 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.945458 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945454 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.945463 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.945470 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3628.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945474 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945470 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.945479 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945481 7f86a5e11700 10 _do_op 0x324f4960 seq 3628 r = 0, finisher 0x17fb7580 0 2014-06-09 16:22:11.945484 7f86a5e11700 10 _finish_op 0x324f4960 seq 3628 osr(1.56 0x384c5f0)/0x384c5f0 2014-06-09 16:22:11.945486 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d440 (1 active) 2014-06-09 16:22:11.945488 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287560 (2 active) 2014-06-09 16:22:11.945482 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.945489 7f86a5e11700 5 _do_op 0x324f5740 seq 3629 osr(0.dd 0x2bd6550)/0x2bd6550 start 2014-06-09 16:22:11.945491 7f86a5e11700 10 _do_transaction on 0x344c000 2014-06-09 16:22:11.945496 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945493 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.945499 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.945507 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945514 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.945521 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.945528 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945533 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.945540 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3629.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945543 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945541 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.945549 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.945556 7f86a5e11700 10 _do_op 0x324f5740 seq 3629 r = 0, finisher 0xbd607a0 0 2014-06-09 16:22:11.945557 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945558 7f86a5e11700 10 _finish_op 0x324f5740 seq 3629 osr(0.dd 0x2bd6550)/0x2bd6550 2014-06-09 16:22:11.945561 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287560 (1 active) 2014-06-09 16:22:11.945562 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383e6c0 (2 active) 2014-06-09 16:22:11.945564 7f86a5e11700 5 _do_op 0x24105c80 seq 3630 osr(1.4db 0x384d850)/0x384d850 start 2014-06-09 16:22:11.945566 7f86a5e11700 10 _do_transaction on 0x30d0000 2014-06-09 16:22:11.945561 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.945570 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945572 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.945579 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.945586 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945594 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.945603 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945603 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.945615 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3630.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945619 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945615 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.945626 7f86a5e11700 10 _do_op 0x24105c80 seq 3630 r = 0, finisher 0x189aac20 0 2014-06-09 16:22:11.945624 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.945629 7f86a5e11700 10 _finish_op 0x24105c80 seq 3630 osr(1.4db 0x384d850)/0x384d850 2014-06-09 16:22:11.945632 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383e6c0 (1 active) 2014-06-09 16:22:11.945633 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f8c0 (2 active) 2014-06-09 16:22:11.945635 7f86a5e11700 5 _do_op 0x324e6d80 seq 3631 osr(0.5fb 0x2bd79a0)/0x2bd79a0 start 2014-06-09 16:22:11.945631 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.945637 7f86a5e11700 10 _do_transaction on 0xbf50840 2014-06-09 16:22:11.945639 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945642 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945643 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.945653 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.945660 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.945668 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945675 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945675 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.945682 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.945689 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3631.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945693 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945694 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.945704 7f86a5e11700 10 _do_op 0x324e6d80 seq 3631 r = 0, finisher 0x3251d9e0 0 2014-06-09 16:22:11.945706 7f86a5e11700 10 _finish_op 0x324e6d80 seq 3631 osr(0.5fb 0x2bd79a0)/0x2bd79a0 2014-06-09 16:22:11.945702 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.945709 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f8c0 (1 active) 2014-06-09 16:22:11.945710 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3398c0 (2 active) 2014-06-09 16:22:11.945712 7f86a5e11700 5 _do_op 0x3280b3e0 seq 3632 osr(1.a2 0x384c790)/0x384c790 start 2014-06-09 16:22:11.945714 7f86a5e11700 10 _do_transaction on 0x394e300 2014-06-09 16:22:11.945709 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.945717 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945718 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945721 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.945732 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.945739 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.945751 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945746 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945754 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.945763 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3632.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945761 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.945767 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945772 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.945780 7f86a5e11700 10 _do_op 0x3280b3e0 seq 3632 r = 0, finisher 0x3206e880 0 2014-06-09 16:22:11.945782 7f86a5e11700 10 _finish_op 0x3280b3e0 seq 3632 osr(1.a2 0x384c790)/0x384c790 2014-06-09 16:22:11.945785 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3398c0 (1 active) 2014-06-09 16:22:11.945787 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3586c0 (2 active) 2014-06-09 16:22:11.945781 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.945788 7f86a5e11700 5 _do_op 0x3280ade0 seq 3633 osr(2.372 0x2ed1150)/0x2ed1150 start 2014-06-09 16:22:11.945791 7f86a5e11700 10 _do_transaction on 0xbdbd2c0 2014-06-09 16:22:11.945788 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.945795 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945796 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945800 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.487( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.945828 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945830 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945834 7f868ef60700 5 queue_transactions existing osr(2.487 0x2ed14b0)/0x2ed14b0 2014-06-09 16:22:11.945836 7f868ef60700 10 op_submit_start 3656 2014-06-09 16:22:11.945837 7f868ef60700 5 queue_transactions (trailing journal) 3656 2014-06-09 16:22:11.945838 7f868ef60700 5 queue_op 0xc2d47e0 seq 3656 osr(2.487 0x2ed14b0) 828 bytes (queue has 25 ops and 538384 bytes) 2014-06-09 16:22:11.945841 7f868ef60700 10 op_submit_finish 3656 2014-06-09 16:22:11.945841 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3633.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945844 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945843 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.945852 7f86a5e11700 10 _do_op 0x3280ade0 seq 3633 r = 0, finisher 0xd9fb9c0 0 2014-06-09 16:22:11.945855 7f86a5e11700 10 _finish_op 0x3280ade0 seq 3633 osr(2.372 0x2ed1150)/0x2ed1150 2014-06-09 16:22:11.945857 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3586c0 (1 active) 2014-06-09 16:22:11.945859 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52fe60 (2 active) 2014-06-09 16:22:11.945860 7f86a5e11700 5 _do_op 0x324e7ce0 seq 3634 osr(2.73 0x2ed0510)/0x2ed0510 start 2014-06-09 16:22:11.945855 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.945863 7f86a5e11700 10 _do_transaction on 0xc28bb00 2014-06-09 16:22:11.945867 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945863 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.945870 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945878 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.945885 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.945900 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945896 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.945905 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.945913 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3634.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945916 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945912 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.945920 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.945924 7f86a5e11700 10 _do_op 0x324e7ce0 seq 3634 r = 0, finisher 0x3042b6e0 0 2014-06-09 16:22:11.945927 7f86a5e11700 10 _finish_op 0x324e7ce0 seq 3634 osr(2.73 0x2ed0510)/0x2ed0510 2014-06-09 16:22:11.945929 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52fe60 (1 active) 2014-06-09 16:22:11.945931 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300ea0 (2 active) 2014-06-09 16:22:11.945924 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.945933 7f86a5e11700 5 _do_op 0x324e7b60 seq 3635 osr(1.82 0x384c4d0)/0x384c4d0 start 2014-06-09 16:22:11.945935 7f86a5e11700 10 _do_transaction on 0x34e9440 2014-06-09 16:22:11.945939 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.945935 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.909434 8 0.000363 2014-06-09 16:22:11.945945 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.945954 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.945963 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:11.945972 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.945973 7f868ef60700 5 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.945984 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3635.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.945981 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.945988 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.945988 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.945996 7f86a5e11700 10 _do_op 0x324e7b60 seq 3635 r = 0, finisher 0x4eb9060 0 2014-06-09 16:22:11.945998 7f86a5e11700 10 _finish_op 0x324e7b60 seq 3635 osr(1.82 0x384c4d0)/0x384c4d0 2014-06-09 16:22:11.946001 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300ea0 (1 active) 2014-06-09 16:22:11.946002 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339d40 (2 active) 2014-06-09 16:22:11.946004 7f86a5e11700 5 _do_op 0x308cb260 seq 3636 osr(2.342 0x2ed0ee0)/0x2ed0ee0 start 2014-06-09 16:22:11.945997 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:11.946006 7f86a5e11700 10 _do_transaction on 0x2dce9c0 2014-06-09 16:22:11.946011 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.946007 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:11.946015 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.946026 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.946044 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.946041 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:11.946060 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3636.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.946064 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.946071 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946074 7f86a5e11700 10 _do_op 0x308cb260 seq 3636 r = 0, finisher 0xc168120 0 2014-06-09 16:22:11.946075 7f868ef60700 5 queue_transactions existing osr(1.522 0x384db30)/0x384db30 2014-06-09 16:22:11.946077 7f868ef60700 10 op_submit_start 3657 2014-06-09 16:22:11.946077 7f86a5e11700 10 _finish_op 0x308cb260 seq 3636 osr(2.342 0x2ed0ee0)/0x2ed0ee0 2014-06-09 16:22:11.946078 7f868ef60700 5 queue_transactions (trailing journal) 3657 2014-06-09 16:22:11.946079 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339d40 (1 active) 2014-06-09 16:22:11.946079 7f868ef60700 5 queue_op 0xbe7dd40 seq 3657 osr(1.522 0x384db30) 861 bytes (queue has 22 ops and 535904 bytes) 2014-06-09 16:22:11.946081 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115200 (2 active) 2014-06-09 16:22:11.946089 7f868ef60700 10 op_submit_finish 3657 2014-06-09 16:22:11.946093 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.31b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.31b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.946115 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.31b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.946123 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.946124 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.31b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.946131 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.31b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.946137 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3605.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.946140 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.946143 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.946156 7f86a5610700 10 _do_op 0x324e6060 seq 3605 r = 0, finisher 0xcc82fc0 0 2014-06-09 16:22:11.946159 7f86a5610700 10 _finish_op 0x324e6060 seq 3605 osr(3.66a 0x3ed19d0)/0x3ed19d0 2014-06-09 16:22:11.946161 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1e60 (1 active) 2014-06-09 16:22:11.946163 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e360 (2 active) 2014-06-09 16:22:11.946158 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.31b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:184 2014-06-09 16:22:11.946165 7f86a5610700 5 _do_op 0x241050e0 seq 3638 osr(3.3a9 0x3ed0f60)/0x3ed0f60 start 2014-06-09 16:22:11.946167 7f86a5610700 10 _do_transaction on 0xc257200 2014-06-09 16:22:11.946171 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946172 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.946175 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.946188 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.946206 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.946197 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x10c55900,0x16058300,0x4679600,0x10c55800,0x16058200,0x4679500,0x10c55700,0x16058100,0x4679400,0x10c55600,0x16058000,0x4679300,0x10c55500,0x13022a00 2014-06-09 16:22:11.946218 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3638.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.946222 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.946231 7f86a5610700 10 _do_op 0x241050e0 seq 3638 r = 0, finisher 0xbd73900 0 2014-06-09 16:22:11.946233 7f86a5610700 10 _finish_op 0x241050e0 seq 3638 osr(3.3a9 0x3ed0f60)/0x3ed0f60 2014-06-09 16:22:11.946236 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e360 (1 active) 2014-06-09 16:22:11.946238 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301320 (2 active) 2014-06-09 16:22:11.946239 7f86a5610700 5 _do_op 0xc2d4a80 seq 3639 osr(2.3f0 0x2ed1320)/0x2ed1320 start 2014-06-09 16:22:11.946242 7f86a5610700 10 _do_transaction on 0xbf2e840 2014-06-09 16:22:11.946246 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.946242 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.946251 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.946260 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.946279 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.946275 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:886 2014-06-09 16:22:11.946285 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.946291 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3639.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.946295 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.946293 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.946303 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946303 7f86a5610700 10 _do_op 0xc2d4a80 seq 3639 r = 0, finisher 0xcc82360 0 2014-06-09 16:22:11.946305 7f86a5610700 10 _finish_op 0xc2d4a80 seq 3639 osr(2.3f0 0x2ed1320)/0x2ed1320 2014-06-09 16:22:11.946308 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301320 (1 active) 2014-06-09 16:22:11.946309 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338ea0 (2 active) 2014-06-09 16:22:11.946311 7f86a5610700 5 _do_op 0xc64a6c0 seq 3640 osr(3.319 0x3ed0d60)/0x3ed0d60 start 2014-06-09 16:22:11.946314 7f86a5610700 10 _do_transaction on 0xc24d500 2014-06-09 16:22:11.946307 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.946318 7f86a5610700 15 _omap_setkeys meta/1fe22c6a/pglog_3.319/0//-1 2014-06-09 16:22:11.946318 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.902043 7 0.005742 2014-06-09 16:22:11.946328 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.946337 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.946346 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:11.946355 7f868ef60700 5 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.946362 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe22c6a/pglog_3.319/0//-1 2014-06-09 16:22:11.946363 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.946370 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.946376 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.946379 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:11.946389 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:11.946398 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.946409 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.946410 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.946422 7f86a5610700 15 _omap_rmkeys meta/1fe22c6a/pglog_3.319/0//-1 2014-06-09 16:22:11.946428 7f86a5610700 10 _omap_rmkeys meta/1fe22c6a/pglog_3.319/0//-1 r = 0 2014-06-09 16:22:11.946424 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:11.946433 7f86a5610700 15 _omap_setkeys meta/1fe22c6a/pglog_3.319/0//-1 2014-06-09 16:22:11.946447 7f86a5610700 15 _write 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 61440~516096 2014-06-09 16:22:11.946453 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946457 7f868ef60700 5 queue_transactions existing osr(0.325 0x2bd6d30)/0x2bd6d30 2014-06-09 16:22:11.946460 7f868ef60700 10 op_submit_start 3658 2014-06-09 16:22:11.946461 7f868ef60700 5 queue_transactions (trailing journal) 3658 2014-06-09 16:22:11.946462 7f868ef60700 5 queue_op 0x30d96cc0 seq 3658 osr(0.325 0x2bd6d30) 861 bytes (queue has 22 ops and 535937 bytes) 2014-06-09 16:22:11.946464 7f868ef60700 10 op_submit_finish 3658 2014-06-09 16:22:11.946467 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.946489 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.946497 7f86a5610700 10 lookup_strip_header done cid 3.319_head oid 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.946498 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1c66aa00,0x1c66a900,0x230dba00,0x230db900,0x230db800,0x230db700,0x230db600,0x230db500,0x230db400 2014-06-09 16:22:11.946511 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.946529 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.946538 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.946546 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.946559 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:915 2014-06-09 16:22:11.946568 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.946576 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.946586 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946590 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.946601 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.946608 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.946615 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.946622 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.946629 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.946640 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:165 2014-06-09 16:22:11.946648 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.946656 7f868ef60700 20 osd.26 pg_epoch: 243 pg[2.359( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.946664 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946667 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.946677 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.946684 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.946692 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.946699 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.946706 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.946718 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:174 2014-06-09 16:22:11.946726 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.946734 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.537( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.946742 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946745 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.946755 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.946762 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.946769 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.946777 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.946784 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.946795 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.946860 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.946867 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.428( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.946875 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946879 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.946891 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.946899 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.946908 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.946916 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.946925 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.946937 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:933 2014-06-09 16:22:11.946947 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.946955 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.701( v 213'541 (0'0,213'541] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.946965 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.946969 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.946981 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.946989 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.946998 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.947005 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.947012 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.947024 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.947032 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.947039 7f868ef60700 20 osd.26 pg_epoch: 243 pg[1.1da( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.947048 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.947052 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.947065 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.947073 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x306b300,0x306bf00,0x306be00,0xe033a00,0xe033900,0xe033800,0xe2ad400 2014-06-09 16:22:11.947099 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.947108 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.947116 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.947128 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:917 2014-06-09 16:22:11.947138 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.947148 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.4ac( v 213'565 (0'0,213'565] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.947157 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.947161 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.947173 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.947182 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x19fe1500,0x19fe1400,0x19fe1300,0x19fe1200,0x19fe1100,0x19fe1000,0x19fe0f00 2014-06-09 16:22:11.947208 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.947217 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.947225 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.947237 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:872 2014-06-09 16:22:11.947246 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.947254 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.947264 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.947276 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41ef0 (19 active) 2014-06-09 16:22:11.947282 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.947286 7f868ef60700 10 osd.26 243 dequeue_op 0x32826000 prio 196 cost 0 latency 0.087719 osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) v2 pg pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.947304 7f868ef60700 5 -- op tracker -- , seq: 3318, time: 2014-06-09 16:22:11.947301, event: reached_pg, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) 2014-06-09 16:22:11.947313 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x32826000 2014-06-09 16:22:11.947326 7f868ef60700 5 -- op tracker -- , seq: 3318, time: 2014-06-09 16:22:11.947323, event: started, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) 2014-06-09 16:22:11.947337 7f868ef60700 7 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 16 op ack_type 1 from 5 2014-06-09 16:22:11.947359 7f868ef60700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.947351, event: sub_op_applied_rec, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.947371 7f868ef60700 10 osd.26 243 dequeue_op 0x32826000 finish 2014-06-09 16:22:11.947378 7f868ef60700 5 -- op tracker -- , seq: 3318, time: 2014-06-09 16:22:11.947375, event: done, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ack, result = 0) 2014-06-09 16:22:11.947388 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.947407 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:11.947410 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.947424 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd4 2014-06-09 16:22:11.947435 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.947444 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.947453 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'729 (0'0,214'729] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.4: log((214'729,214'730], crt=0'0) missing(0) 2014-06-09 16:22:11.947463 7f868ef60700 10 merge_log log((214'729,214'730], crt=0'0) from osd.4 into log((0'0,214'729], crt=0'0) 2014-06-09 16:22:11.947467 7f868ef60700 10 merge_log extending head to 214'730 2014-06-09 16:22:11.947469 7f868ef60700 20 ? 214'730 (214'729) modify 4c0ec24/rbd_data.11606b8b4567.0000000000001820/head//3 by client.4544.0:34401 2014-06-09 10:19:20.426568 2014-06-09 16:22:11.947476 7f868ef60700 20 merge_log 214'730 (214'729) modify 4c0ec24/rbd_data.11606b8b4567.0000000000001820/head//3 by client.4544.0:34401 2014-06-09 10:19:20.426568 2014-06-09 16:22:11.947681 7f868ff62700 10 op_submit_start 3659 2014-06-09 16:22:11.947683 7f868ff62700 5 queue_transactions (trailing journal) 3659 2014-06-09 16:22:11.947684 7f868ff62700 5 queue_op 0xc2d4f00 seq 3659 osr(3.6d5 0x3ed1c30) 828 bytes (queue has 24 ops and 537593 bytes) 2014-06-09 16:22:11.947687 7f868ff62700 10 op_submit_finish 3659 2014-06-09 16:22:11.947688 7f868e75f700 10 op_submit_start 3660 2014-06-09 16:22:11.947691 7f868e75f700 5 queue_transactions (trailing journal) 3660 2014-06-09 16:22:11.947692 7f868e75f700 5 queue_op 0x324f4000 seq 3660 osr(2.501 0x2ed1690) 828 bytes (queue has 25 ops and 538421 bytes) 2014-06-09 16:22:11.947695 7f868e75f700 10 op_submit_finish 3660 2014-06-09 16:22:11.947692 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.947705 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.947711 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40e90 (19 active) 2014-06-09 16:22:11.947710 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.947714 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.947717 7f868e75f700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.947724 7f868ff62700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.947721 7f868e75f700 10 osd.26 243 dequeue_op 0x32827b00 prio 196 cost 0 latency 0.088047 osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) v2 pg pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.947736 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:183 2014-06-09 16:22:11.947739 7f868e75f700 5 -- op tracker -- , seq: 3319, time: 2014-06-09 16:22:11.947735, event: reached_pg, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.947747 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.947750 7f868b759700 10 op_submit_start 3661 2014-06-09 16:22:11.947751 7f868b759700 5 queue_transactions (trailing journal) 3661 2014-06-09 16:22:11.947752 7f868b759700 5 queue_op 0x324f44e0 seq 3661 osr(0.681 0x2bd7e10) 828 bytes (queue has 25 ops and 538421 bytes) 2014-06-09 16:22:11.947754 7f868b759700 10 op_submit_finish 3661 2014-06-09 16:22:11.947748 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x32827b00 2014-06-09 16:22:11.947757 7f868a757700 10 op_submit_start 3662 2014-06-09 16:22:11.947760 7f868a757700 5 queue_transactions (trailing journal) 3662 2014-06-09 16:22:11.947761 7f868a757700 5 queue_op 0x3280bbc0 seq 3662 osr(1.38d 0x384d570) 828 bytes (queue has 25 ops and 538421 bytes) 2014-06-09 16:22:11.947764 7f868a757700 10 op_submit_finish 3662 2014-06-09 16:22:11.947757 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.5b0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.947763 7f868e75f700 5 -- op tracker -- , seq: 3319, time: 2014-06-09 16:22:11.947760, event: started, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.947768 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.4e0( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.4e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:11.947769 7f868e75f700 7 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 16 op ack_type 4 from 5 2014-06-09 16:22:11.947788 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.947788 7f868e75f700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.947782, event: sub_op_commit_rec, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.947794 7f868e75f700 10 osd.26 243 dequeue_op 0x32827b00 finish 2014-06-09 16:22:11.947793 7f868ef60700 10 merge_log result log((0'0,214'730], crt=214'730) missing(1) changed=1 2014-06-09 16:22:11.947799 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.947800 7f868e75f700 5 -- op tracker -- , seq: 3319, time: 2014-06-09 16:22:11.947797, event: done, op: osd_sub_op_reply(client.4569.0:38638 3.319 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.947803 7f868a757700 5 queue_transactions existing osr(2.4e0 0x2ed16f0)/0x2ed16f0 2014-06-09 16:22:11.947806 7f868a757700 10 op_submit_start 3663 2014-06-09 16:22:11.947807 7f868a757700 5 queue_transactions (trailing journal) 3663 2014-06-09 16:22:11.947797 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) 2014-06-09 16:22:11.947808 7f868a757700 5 queue_op 0x3280a7e0 seq 3663 osr(2.4e0 0x2ed16f0) 828 bytes (queue has 26 ops and 539249 bytes) 2014-06-09 16:22:11.947810 7f868a757700 10 op_submit_finish 3663 2014-06-09 16:22:11.947810 7f868e75f700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.947798 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 peering m=1] peer osd.4 now 3.424( v 214'730 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) missing(0) 2014-06-09 16:22:11.947811 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.947813 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.67a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.67a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.947817 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.698808 3 0.000158 2014-06-09 16:22:11.947821 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.947827 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40e90 (20 active) 2014-06-09 16:22:11.947829 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.947831 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.947830 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.e6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.e6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.947842 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.947845 7f868a757700 5 queue_transactions existing osr(1.67a 0x384de90)/0x384de90 2014-06-09 16:22:11.947848 7f868a757700 10 op_submit_start 3664 2014-06-09 16:22:11.947841 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 peering m=1] state: osd.4 has no missing, identical log 2014-06-09 16:22:11.947849 7f868a757700 5 queue_transactions (trailing journal) 3664 2014-06-09 16:22:11.947850 7f868a757700 5 queue_op 0x3280b740 seq 3664 osr(1.67a 0x384de90) 828 bytes (queue has 27 ops and 540077 bytes) 2014-06-09 16:22:11.947853 7f868a757700 10 op_submit_finish 3664 2014-06-09 16:22:11.947852 7f868ff62700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.947860 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.947854 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000024 0 0.000000 2014-06-09 16:22:11.947856 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.e8( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.e8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.947868 7f8689755700 10 op_submit_start 3665 2014-06-09 16:22:11.947870 7f8689755700 5 queue_transactions (trailing journal) 3665 2014-06-09 16:22:11.947872 7f8689755700 5 queue_op 0xbe7c780 seq 3665 osr(2.5d9 0x2ed1a70) 828 bytes (queue has 28 ops and 540905 bytes) 2014-06-09 16:22:11.947875 7f8689755700 10 op_submit_finish 3665 2014-06-09 16:22:11.947868 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:961 2014-06-09 16:22:11.947884 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.947885 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.947878 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.947888 7f868a757700 5 queue_transactions existing osr(0.e8 0x2bd6510)/0x2bd6510 2014-06-09 16:22:11.947891 7f868a757700 10 op_submit_start 3666 2014-06-09 16:22:11.947892 7f868a757700 5 queue_transactions (trailing journal) 3666 2014-06-09 16:22:11.947893 7f868a757700 5 queue_op 0x3280ae40 seq 3666 osr(0.e8 0x2bd6510) 826 bytes (queue has 29 ops and 541731 bytes) 2014-06-09 16:22:11.947895 7f868a757700 10 op_submit_finish 3666 2014-06-09 16:22:11.947887 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.380( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.380( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.947892 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.947887 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.947900 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.947897 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4da( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=70-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.4da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:11.947900 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.410157 0 0.000000 2014-06-09 16:22:11.947908 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.947914 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.947918 7f868ff62700 5 queue_transactions existing osr(0.380 0x2bd6f20)/0x2bd6f20 2014-06-09 16:22:11.947920 7f868ff62700 10 op_submit_start 3667 2014-06-09 16:22:11.947921 7f868ff62700 5 queue_transactions (trailing journal) 3667 2014-06-09 16:22:11.947915 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.947915 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.947922 7f868ff62700 5 queue_op 0xc2d5c20 seq 3667 osr(0.380 0x2bd6f20) 828 bytes (queue has 30 ops and 542559 bytes) 2014-06-09 16:22:11.947925 7f868ff62700 10 op_submit_finish 3667 2014-06-09 16:22:11.947925 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.947922 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.947928 7f868a757700 5 queue_transactions existing osr(0.4da 0x2bd75f0)/0x2bd75f0 2014-06-09 16:22:11.947931 7f868a757700 10 op_submit_start 3668 2014-06-09 16:22:11.947931 7f868a757700 5 queue_transactions (trailing journal) 3668 2014-06-09 16:22:11.947924 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.947932 7f868a757700 5 queue_op 0x327fc000 seq 3668 osr(0.4da 0x2bd75f0) 828 bytes (queue has 31 ops and 543387 bytes) 2014-06-09 16:22:11.947935 7f868a757700 10 op_submit_finish 3668 2014-06-09 16:22:11.947927 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.947935 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:152 2014-06-09 16:22:11.947935 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.947939 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.947937 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.947944 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.947946 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.947945 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.947948 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.947952 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.947961 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.947955 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.947954 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.947956 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.947964 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.947963 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.947974 7f868a757700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.947965 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.947970 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.947978 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.947982 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:174 2014-06-09 16:22:11.947989 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.947987 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x21967f00,0x21967e00,0x22009a00,0x22009300,0x22009200,0x26aa9f00,0x26aa9e00 2014-06-09 16:22:11.947992 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.947999 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948008 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.948012 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948020 7f8691f66700 10 op_submit_start 3669 2014-06-09 16:22:11.948021 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.948022 7f8691f66700 5 queue_transactions (trailing journal) 3669 2014-06-09 16:22:11.948023 7f8691f66700 5 queue_op 0x30607680 seq 3669 osr(3.60 0x3ed02e0) 858 bytes (queue has 32 ops and 544245 bytes) 2014-06-09 16:22:11.948027 7f8691f66700 10 op_submit_finish 3669 2014-06-09 16:22:11.948023 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948025 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.35b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.948030 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948032 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) 2014-06-09 16:22:11.948038 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948035 7f868c75b700 10 osd.26 243 dequeue_op 0xdfa6400 prio 196 cost 0 latency 0.220152 osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] 2014-06-09 16:22:11.948045 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948053 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.948056 7f868a757700 5 queue_transactions existing osr(0.35b 0x2bd6fe0)/0x2bd6fe0 2014-06-09 16:22:11.948058 7f868a757700 10 op_submit_start 3670 2014-06-09 16:22:11.948059 7f868a757700 5 queue_transactions (trailing journal) 3670 2014-06-09 16:22:11.948054 7f868c75b700 5 -- op tracker -- , seq: 2779, time: 2014-06-09 16:22:11.948050, event: reached_pg, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.948054 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948060 7f868a757700 5 queue_op 0x327fcc00 seq 3670 osr(0.35b 0x2bd6fe0) 828 bytes (queue has 33 ops and 545073 bytes) 2014-06-09 16:22:11.948063 7f868a757700 10 op_submit_finish 3670 2014-06-09 16:22:11.948045 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.9 activated and committed 2014-06-09 16:22:11.948062 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] handle_message: 0xdfa6400 2014-06-09 16:22:11.948065 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.948066 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.948065 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.553( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.553( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.948073 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.948076 7f868c75b700 5 -- op tracker -- , seq: 2779, time: 2014-06-09 16:22:11.948073, event: started, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.948074 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948077 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.553( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.948086 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.948082 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948090 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.948085 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.553( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.948082 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] sub_op_modify_reply: tid 4 op ack_type 1 from 7 2014-06-09 16:22:11.948091 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.553( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.948094 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948102 7f868a757700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.948099 7f868c75b700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.948093, event: sub_op_applied_rec, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.948106 7f868c75b700 10 osd.26 243 dequeue_op 0xdfa6400 finish 2014-06-09 16:22:11.948100 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:158 2014-06-09 16:22:11.948104 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948114 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948111 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948114 7f868c75b700 5 -- op tracker -- , seq: 2779, time: 2014-06-09 16:22:11.948111, event: done, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.948119 7f868df5e700 10 op_submit_start 3671 2014-06-09 16:22:11.948120 7f868df5e700 5 queue_transactions (trailing journal) 3671 2014-06-09 16:22:11.948115 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.553( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:192 2014-06-09 16:22:11.948122 7f868df5e700 5 queue_op 0x324f5f20 seq 3671 osr(1.45d 0x384d670) 828 bytes (queue has 33 ops and 545073 bytes) 2014-06-09 16:22:11.948125 7f868df5e700 10 op_submit_finish 3671 2014-06-09 16:22:11.948119 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948126 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.948132 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.948127 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948129 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.606( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.948135 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948138 7f868c75b700 10 osd.26 243 dequeue_op 0x1c66a500 prio 63 cost 24576 latency 1.393109 osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.948147 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.948160 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.948156 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948159 7f868c75b700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.948154, event: reached_pg, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.948159 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948163 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948171 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.948169 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948181 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (19 active) 2014-06-09 16:22:11.948178 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948175 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.948166 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.948187 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.893017 6 0.000440 2014-06-09 16:22:11.948189 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1c66a500 2014-06-09 16:22:11.948197 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (20 active) 2014-06-09 16:22:11.948198 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.948203 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.17( v 213'672 (0'0,213'672] local-les=243 n=137 ec=118 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.17( v 213'672 (0'0,213'672] local-les=243 n=137 ec=118 les/c 243/243 242/242/220) 2014-06-09 16:22:11.948208 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948207 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1140 2014-06-09 16:22:11.948199 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.948217 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948218 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:11.948229 7f868c75b700 15 getattr 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 '_' 2014-06-09 16:22:11.948226 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948233 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948235 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.948229 7f868ff62700 5 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.948237 7f8691f66700 5 queue_transactions existing osr(3.17 0x3ed03b0)/0x3ed03b0 2014-06-09 16:22:11.948240 7f8691f66700 10 op_submit_start 3672 2014-06-09 16:22:11.948241 7f8691f66700 5 queue_transactions (trailing journal) 3672 2014-06-09 16:22:11.948242 7f8691f66700 5 queue_op 0x1e08bb60 seq 3672 osr(3.17 0x3ed03b0) 826 bytes (queue has 34 ops and 545899 bytes) 2014-06-09 16:22:11.948244 7f8691f66700 10 op_submit_finish 3672 2014-06-09 16:22:11.948238 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.948239 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948246 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.948249 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948246 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.948256 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948256 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:11.948259 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.948264 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948268 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.948267 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:11.948272 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948275 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.948277 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.948279 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948287 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.948289 7f868ff62700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.948290 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.948299 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948308 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.948304 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.948306 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948316 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.948318 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948310 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 but can_create 2014-06-09 16:22:11.948323 7f868c75b700 15 getattr 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 'snapset' 2014-06-09 16:22:11.948320 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948322 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.948331 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948337 7f868b759700 5 queue_transactions existing osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 2014-06-09 16:22:11.948340 7f868b759700 10 op_submit_start 3673 2014-06-09 16:22:11.948340 7f868b759700 5 queue_transactions (trailing journal) 3673 2014-06-09 16:22:11.948334 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.948342 7f868b759700 5 queue_op 0x324f4360 seq 3673 osr(0.5b0 0x2bd7ab0) 828 bytes (queue has 35 ops and 546727 bytes) 2014-06-09 16:22:11.948344 7f868b759700 10 op_submit_finish 3673 2014-06-09 16:22:11.948339 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948343 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.948346 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948354 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.948347 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.4d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.4d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.948350 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.948356 7f868c75b700 15 getattr 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/snapdir//3 'snapset' 2014-06-09 16:22:11.948354 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948361 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.948358 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.4d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.948361 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948366 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.4d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.948373 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.4d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.948373 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.948373 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:163 2014-06-09 16:22:11.948383 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.948384 7f868b759700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.948381 7f8689755700 20 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948388 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948389 7f8689755700 20 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948397 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.948392 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.700( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.948391 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30e28840 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.948398 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.4d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.948401 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948403 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.948402 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.948411 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948411 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.948411 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948411 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.948418 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.948415 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:11.948420 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948429 7f8691f66700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.948427 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948421 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30e28840 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 rwstate(none n=0 w=0) oi: cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd40e0 snapset: 0=[]:[] 2014-06-09 16:22:11.948428 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.948434 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948437 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.948436 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 @head oi=cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.948442 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948442 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.700( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:168 2014-06-09 16:22:11.948445 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.948455 7f868c75b700 15 getattr 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/snapdir//3 '_' 2014-06-09 16:22:11.948457 7f868b759700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.948458 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948453 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:167 2014-06-09 16:22:11.948462 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948461 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.18d( v 213'668 (0'0,213'668] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.18d( v 213'668 (0'0,213'668] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:11.948469 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.668( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948469 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.6d8( v 213'642 (0'0,213'642] local-les=243 n=123 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1081 2014-06-09 16:22:11.948478 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.948483 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948475 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.18d( v 213'668 (0'0,213'668] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.948485 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.948482 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948485 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.18d( v 213'668 (0'0,213'668] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.948486 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:11.948487 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/snapdir//3 and !can_create 2014-06-09 16:22:11.948492 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948494 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.18d( v 213'668 (0'0,213'668] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.948500 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948506 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.948505 7f868c75b700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.948499, event: started, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.948508 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948515 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948518 7f868b759700 5 queue_transactions existing osr(3.39e 0x3ed0fa0)/0x3ed0fa0 2014-06-09 16:22:11.948512 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30642a00 2014-06-09 16:22:11.948521 7f868b759700 10 op_submit_start 3674 2014-06-09 16:22:11.948522 7f868b759700 5 queue_transactions (trailing journal) 3674 2014-06-09 16:22:11.948516 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948523 7f868b759700 5 queue_op 0x324f56e0 seq 3674 osr(3.39e 0x3ed0fa0) 828 bytes (queue has 36 ops and 547555 bytes) 2014-06-09 16:22:11.948516 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.18d( v 213'668 (0'0,213'668] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1081 2014-06-09 16:22:11.948526 7f868b759700 10 op_submit_finish 3674 2014-06-09 16:22:11.948523 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948531 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948529 7f868c75b700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.948523, event: started, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.948528 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948535 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:190 2014-06-09 16:22:11.948534 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948540 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948535 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] ov 0'0 av 243'592 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.948544 7f8689755700 20 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948545 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948547 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948551 7f8689755700 20 osd.26 pg_epoch: 243 pg[0.370( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948548 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 2014-06-09 16:22:11.948559 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.948553 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948555 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948558 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.948562 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948562 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948563 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948570 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948570 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948573 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~24576 2014-06-09 16:22:11.948576 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948577 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948582 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:176 2014-06-09 16:22:11.948584 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0xe032600,0xe033700,0xe033600,0x29f5b200,0x29f5b100,0x29f5b000,0x29f5af00,0x29f5ae00,0x29f5ad00,0x29f5ac00 2014-06-09 16:22:11.948590 7f868b759700 20 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948590 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.948589 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228a00 2014-06-09 16:22:11.948597 7f868b759700 20 osd.26 pg_epoch: 243 pg[1.71b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948598 7f8691f66700 20 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948605 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948599 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 snapset=0x30dd4118 snapc=0=[] 2014-06-09 16:22:11.948605 7f8691f66700 20 osd.26 pg_epoch: 243 pg[2.154( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948613 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948609 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.948608 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948619 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948618 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948618 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.948617 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948627 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948627 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948629 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 0x30642a00 op modify 2014-06-09 16:22:11.948630 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948634 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948635 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948639 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x11e62f00,0x11e62e00,0x11e62d00,0x11e62c00,0x13022400,0x13022300,0x13022200,0x13022100,0x26aa9d00,0x29f5aa00,0x29f5a900,0x29f5a800 2014-06-09 16:22:11.948642 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948641 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:25.931208 2014-06-09 16:22:11.948647 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:987 2014-06-09 16:22:11.948650 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948657 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948660 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.948660 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=121 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.948665 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948669 7f868b759700 20 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948674 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.948675 7f868b759700 20 osd.26 pg_epoch: 243 pg[0.214( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948684 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948675 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948678 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.948679 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.948685 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948687 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.31f( v 213'583 (0'0,213'583] local-les=243 n=118 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:11.948688 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 17 on osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.948691 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.873243 7 0.012645 2014-06-09 16:22:11.948694 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948702 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.948703 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 17 o cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.948707 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1035 2014-06-09 16:22:11.948716 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948719 7f868b759700 5 queue_transactions existing osr(3.31f 0x3ed0d50)/0x3ed0d50 2014-06-09 16:22:11.948721 7f868b759700 10 op_submit_start 3675 2014-06-09 16:22:11.948713 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948722 7f868b759700 5 queue_transactions (trailing journal) 3675 2014-06-09 16:22:11.948717 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948723 7f868b759700 5 queue_op 0xc64b320 seq 3675 osr(3.31f 0x3ed0d50) 828 bytes (queue has 37 ops and 548383 bytes) 2014-06-09 16:22:11.948725 7f868b759700 10 op_submit_finish 3675 2014-06-09 16:22:11.948725 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.3b8( v 214'622 (0'0,214'622] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948724 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:11.948735 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948728 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948735 7f8689755700 5 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.948739 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948739 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948744 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.948746 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948751 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948752 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.948754 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948760 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948761 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948763 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:11.948768 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948768 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948774 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:11.948777 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948780 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:182 2014-06-09 16:22:11.948783 7f868c75b700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.948778, event: waiting for subops from 8, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.948785 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.948786 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948789 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948797 7f8689755700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.948796 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.3a6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948801 7f868c75b700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.948804 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948799 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1024 2014-06-09 16:22:11.948809 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948812 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:964 2014-06-09 16:22:11.948817 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.0( v 213'622 (0'0,213'622] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948827 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948818 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 214'591 (0'0,214'591] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'591], crt=0'0) [243'592 (0'0) modify cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 by client.4570.0:33987 2014-06-09 10:19:25.931208] 2014-06-09 16:22:11.948830 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948845 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.948841 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948837 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948849 7f8689755700 5 queue_transactions existing osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.948838 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'592 (0'0) modify cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 by client.4570.0:33987 2014-06-09 10:19:25.931208 2014-06-09 16:22:11.948851 7f8689755700 10 op_submit_start 3676 2014-06-09 16:22:11.948852 7f8689755700 5 queue_transactions (trailing journal) 3676 2014-06-09 16:22:11.948853 7f8689755700 5 queue_op 0x309a31a0 seq 3676 osr(3.1e 0x3ed0380) 858 bytes (queue has 38 ops and 549241 bytes) 2014-06-09 16:22:11.948848 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948856 7f8689755700 10 op_submit_finish 3676 2014-06-09 16:22:11.948851 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948857 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948858 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948859 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948865 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948866 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948857 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.948870 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948873 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948873 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948877 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948881 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948885 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948885 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.948892 7f868c75b700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'592, trimmed: 2014-06-09 16:22:11.948892 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948893 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.948894 7f8691f66700 20 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948900 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948900 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948902 7f8691f66700 20 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948910 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948911 7f868c75b700 5 queue_transactions existing osr(3.647 0x3ed1a70)/0x3ed1a70 2014-06-09 16:22:11.948908 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948914 7f868c75b700 10 op_submit_start 3677 2014-06-09 16:22:11.948915 7f868c75b700 5 queue_transactions (trailing journal) 3677 2014-06-09 16:22:11.948916 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948916 7f868c75b700 5 queue_op 0x1e08af60 seq 3677 osr(3.647 0x3ed1a70) 26974 bytes (queue has 39 ops and 576215 bytes) 2014-06-09 16:22:11.948912 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.948919 7f868c75b700 10 op_submit_finish 3677 2014-06-09 16:22:11.948914 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948920 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948920 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 243/191 242/242/220) 2014-06-09 16:22:11.948925 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948928 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948921 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0x2d83ec0 243'592 rep_tid=17 committed?=0 applied?=0 lock=0 op=osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.948936 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.948932 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948938 7f868c75b700 10 osd.26 243 dequeue_op 0x1c66a500 finish 2014-06-09 16:22:11.948942 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.948940 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948947 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.948940 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948948 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.948951 7f868b759700 5 queue_transactions existing osr(3.d1 0x3ed0560)/0x3ed0560 2014-06-09 16:22:11.948954 7f868b759700 10 op_submit_start 3678 2014-06-09 16:22:11.948948 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948955 7f868b759700 5 queue_transactions (trailing journal) 3678 2014-06-09 16:22:11.948951 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948956 7f868b759700 5 queue_op 0x24105da0 seq 3678 osr(3.d1 0x3ed0560) 826 bytes (queue has 40 ops and 577041 bytes) 2014-06-09 16:22:11.948958 7f868b759700 10 op_submit_finish 3678 2014-06-09 16:22:11.948956 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948950 7f868c75b700 10 osd.26 243 dequeue_op 0x1c66aa00 prio 63 cost 16384 latency 1.241855 osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.948958 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948961 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.948967 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948969 7f868c75b700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:11.948964, event: reached_pg, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.948968 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.948971 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.948974 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948976 7f8691f66700 20 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948979 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.948976 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.948981 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.948984 7f8691f66700 20 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.948986 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.948993 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.948988 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1c66aa00 2014-06-09 16:22:11.948994 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.948993 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.948996 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.949001 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949003 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.948997 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.949008 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.949014 7f868c75b700 15 getattr 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 '_' 2014-06-09 16:22:11.949010 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949018 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.949013 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:190 2014-06-09 16:22:11.949015 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.949022 7f868b759700 20 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949023 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.949021 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.949029 7f868b759700 20 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949037 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949031 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.949032 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.949038 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949039 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.949040 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.5b0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.5b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:11.949047 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.949050 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.949054 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.949064 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949058 7f8691f66700 20 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949067 7f868b759700 5 queue_transactions existing osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 2014-06-09 16:22:11.949062 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949069 7f868b759700 10 op_submit_start 3679 2014-06-09 16:22:11.949070 7f868b759700 5 queue_transactions (trailing journal) 3679 2014-06-09 16:22:11.949071 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.949071 7f868b759700 5 queue_op 0x24104300 seq 3679 osr(0.5b0 0x2bd7ab0) 828 bytes (queue has 41 ops and 577869 bytes) 2014-06-09 16:22:11.949066 7f8691f66700 20 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949073 7f868b759700 10 op_submit_finish 3679 2014-06-09 16:22:11.949075 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.949073 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.949073 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 but can_create 2014-06-09 16:22:11.949076 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.53( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.53( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.949084 7f868c75b700 15 getattr 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 'snapset' 2014-06-09 16:22:11.949079 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.949082 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949087 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.53( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.949089 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.641( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949089 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.949097 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.949095 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.53( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.949097 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.949102 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.53( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.949101 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.949111 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.949105 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.949112 7f868b759700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.949113 7f868c75b700 15 getattr 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/snapdir//3 'snapset' 2014-06-09 16:22:11.949111 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.949113 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.949118 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.949121 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949123 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.53( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.949126 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.949135 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949137 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.949133 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.949133 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.949141 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949148 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f3fc0 (19 active) 2014-06-09 16:22:11.949143 7f8691f66700 20 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949142 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30bd2dc0 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.949150 7f8691f66700 20 osd.26 pg_epoch: 243 pg[2.74b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949158 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.949153 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.949166 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f3fc0 (20 active) 2014-06-09 16:22:11.949161 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.949162 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.949172 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.949173 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.681( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.681( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.949170 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30bd2dc0 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 rwstate(none n=0 w=0) oi: 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x26a46b60 snapset: 0=[]:[] 2014-06-09 16:22:11.949179 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.949191 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949187 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.949186 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 @head oi=5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.949194 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.949200 7f868c75b700 15 getattr 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/snapdir//3 '_' 2014-06-09 16:22:11.949203 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949196 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.697( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.697( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.949206 7f868b759700 5 queue_transactions existing osr(0.681 0x2bd7e10)/0x2bd7e10 2014-06-09 16:22:11.949201 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949209 7f868b759700 10 op_submit_start 3680 2014-06-09 16:22:11.949210 7f868b759700 5 queue_transactions (trailing journal) 3680 2014-06-09 16:22:11.949211 7f868b759700 5 queue_op 0x324e7b00 seq 3680 osr(0.681 0x2bd7e10) 828 bytes (queue has 42 ops and 578697 bytes) 2014-06-09 16:22:11.949213 7f868b759700 10 op_submit_finish 3680 2014-06-09 16:22:11.949208 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.697( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.949213 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:176 2014-06-09 16:22:11.949217 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.697( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.949215 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.39e( v 213'667 (0'0,213'667] local-les=243 n=126 ec=118 les/c 243/243 242/242/222) 2014-06-09 16:22:11.949223 7f8691f66700 20 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949224 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.697( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.949235 7f868a757700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.949230 7f8691f66700 20 osd.26 pg_epoch: 243 pg[1.757( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949237 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.949239 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.949241 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949244 7f868b759700 5 queue_transactions existing osr(3.39e 0x3ed0fa0)/0x3ed0fa0 2014-06-09 16:22:11.949247 7f868b759700 10 op_submit_start 3681 2014-06-09 16:22:11.949248 7f868b759700 5 queue_transactions (trailing journal) 3681 2014-06-09 16:22:11.949239 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/snapdir//3 and !can_create 2014-06-09 16:22:11.949249 7f868b759700 5 queue_op 0x324e7bc0 seq 3681 osr(3.39e 0x3ed0fa0) 828 bytes (queue has 43 ops and 579525 bytes) 2014-06-09 16:22:11.949251 7f868b759700 10 op_submit_finish 3681 2014-06-09 16:22:11.949243 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.949249 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.697( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:153 2014-06-09 16:22:11.949255 7f868c75b700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:11.949250, event: started, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.949253 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.d1( v 213'637 (0'0,213'637] local-les=243 n=123 ec=118 les/c 243/243 242/242/220) 2014-06-09 16:22:11.949262 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949256 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.458( v 213'610 (0'0,213'610] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) log((0'0,213'610], crt=0'0) missing(0) 2014-06-09 16:22:11.949262 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x307e1600 2014-06-09 16:22:11.949270 7f8691f66700 10 proc_replica_log for osd.2: 3.458( v 213'610 (0'0,213'610] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) log((0'0,213'610], crt=0'0) missing(0) 2014-06-09 16:22:11.949265 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:11.949279 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949282 7f868b759700 5 queue_transactions existing osr(3.d1 0x3ed0560)/0x3ed0560 2014-06-09 16:22:11.949274 7f8691f66700 20 merge_log cut point (usually last shared) is 213'610 (0'0) modify 94c5cc58/rbd_data.11626b8b4567.0000000000001870/head//3 by client.4546.0:35552 2014-06-09 10:18:16.492894 2014-06-09 16:22:11.949285 7f868b759700 10 op_submit_start 3682 2014-06-09 16:22:11.949286 7f868b759700 5 queue_transactions (trailing journal) 3682 2014-06-09 16:22:11.949287 7f868b759700 5 queue_op 0x324e7c80 seq 3682 osr(3.d1 0x3ed0560) 826 bytes (queue has 44 ops and 580351 bytes) 2014-06-09 16:22:11.949289 7f868b759700 10 op_submit_finish 3682 2014-06-09 16:22:11.949284 7f8691765700 10 osd.26 243 dequeue_op 0xdfa6300 prio 196 cost 0 latency 0.219730 osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] 2014-06-09 16:22:11.949297 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949291 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.3c0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.949301 7f868a757700 5 queue_transactions existing osr(3.e0 0x3ed0500)/0x3ed0500 2014-06-09 16:22:11.949303 7f868a757700 10 op_submit_start 3683 2014-06-09 16:22:11.949304 7f868a757700 5 queue_transactions (trailing journal) 3683 2014-06-09 16:22:11.949305 7f868a757700 5 queue_op 0x327fc060 seq 3683 osr(3.e0 0x3ed0500) 826 bytes (queue has 45 ops and 581177 bytes) 2014-06-09 16:22:11.949307 7f868a757700 10 op_submit_finish 3683 2014-06-09 16:22:11.949318 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949310 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.423( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.423( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:11.949321 7f868b759700 5 queue_transactions existing osr(2.3c0 0x2ed13f0)/0x2ed13f0 2014-06-09 16:22:11.949324 7f868b759700 10 op_submit_start 3684 2014-06-09 16:22:11.949325 7f868b759700 5 queue_transactions (trailing journal) 3684 2014-06-09 16:22:11.949326 7f868b759700 5 queue_op 0x324e7d40 seq 3684 osr(2.3c0 0x2ed13f0) 828 bytes (queue has 46 ops and 582005 bytes) 2014-06-09 16:22:11.949328 7f868b759700 10 op_submit_finish 3684 2014-06-09 16:22:11.949322 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.423( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:11.949330 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.423( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.949330 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.722( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.949337 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.423( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.949341 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.949347 7f868a757700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.949348 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.949355 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.949360 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.423( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:11.949368 7f868b759700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.949375 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949383 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.722( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:184 2014-06-09 16:22:11.949397 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949401 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3e2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.3e2( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.949413 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3e2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.949422 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3e2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.949421 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.91( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.949429 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.3e2( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.949440 7f868b759700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.949450 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949453 7f868a757700 5 queue_transactions existing osr(1.91 0x384c430)/0x384c430 2014-06-09 16:22:11.949456 7f868a757700 10 op_submit_start 3685 2014-06-09 16:22:11.949457 7f868a757700 5 queue_transactions (trailing journal) 3685 2014-06-09 16:22:11.949458 7f868a757700 5 queue_op 0xc88a000 seq 3685 osr(1.91 0x384c430) 826 bytes (queue has 47 ops and 582831 bytes) 2014-06-09 16:22:11.949460 7f868a757700 10 op_submit_finish 3685 2014-06-09 16:22:11.949450 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.3e2( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.949463 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.c9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.949475 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949478 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:11.949491 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949494 7f868a757700 5 queue_transactions existing osr(2.c9 0x2ed04a0)/0x2ed04a0 2014-06-09 16:22:11.949496 7f868a757700 10 op_submit_start 3686 2014-06-09 16:22:11.949497 7f868a757700 5 queue_transactions (trailing journal) 3686 2014-06-09 16:22:11.949498 7f868a757700 5 queue_op 0xc88bf20 seq 3686 osr(2.c9 0x2ed04a0) 826 bytes (queue has 48 ops and 583657 bytes) 2014-06-09 16:22:11.949500 7f868a757700 10 op_submit_finish 3686 2014-06-09 16:22:11.949487 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.949507 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949510 7f868b759700 5 queue_transactions existing osr(3.2fd 0x3ed0de0)/0x3ed0de0 2014-06-09 16:22:11.949503 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.301( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:11.949513 7f868b759700 10 op_submit_start 3687 2014-06-09 16:22:11.949514 7f868b759700 5 queue_transactions (trailing journal) 3687 2014-06-09 16:22:11.949515 7f868b759700 5 queue_op 0x324e7f80 seq 3687 osr(3.2fd 0x3ed0de0) 828 bytes (queue has 49 ops and 584485 bytes) 2014-06-09 16:22:11.949517 7f868b759700 10 op_submit_finish 3687 2014-06-09 16:22:11.949511 7f868ef60700 10 osd.26 243 queue_for_recovery queued pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.949520 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.949530 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949535 7f868a757700 5 queue_transactions existing osr(0.301 0x2bd6e10)/0x2bd6e10 2014-06-09 16:22:11.949525 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 3.424( v 214'730 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) 2014-06-09 16:22:11.949537 7f868a757700 10 op_submit_start 3688 2014-06-09 16:22:11.949538 7f868a757700 5 queue_transactions (trailing journal) 3688 2014-06-09 16:22:11.949539 7f868a757700 5 queue_op 0xc88aea0 seq 3688 osr(0.301 0x2bd6e10) 828 bytes (queue has 50 ops and 585313 bytes) 2014-06-09 16:22:11.949542 7f868a757700 10 op_submit_finish 3688 2014-06-09 16:22:11.949537 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 is up to date, queueing in pending_activators 2014-06-09 16:22:11.949547 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.949551 7f868b759700 5 queue_transactions existing osr(2.f4 0x2ed0790)/0x2ed0790 2014-06-09 16:22:11.949544 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.6bb( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.6bb( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.949553 7f868b759700 2 waiting 51 > 50 ops || 586139 > 104857600 2014-06-09 16:22:11.949552 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.4 3.424( v 214'730 (0'0,214'730] local-les=203 n=142 ec=118 les/c 203/203 242/242/242) uptodate 2014-06-09 16:22:11.949556 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.6bb( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.949565 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.6bb( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.949573 7f868e75f700 5 queue_transactions existing osr(2.e6 0x2ed07d0)/0x2ed07d0 2014-06-09 16:22:11.949566 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.949576 7f868e75f700 2 waiting 51 > 50 ops || 586139 > 104857600 2014-06-09 16:22:11.949574 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.6bb( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.949585 7f868a757700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.949578 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 4c0ec24/rbd_data.11606b8b4567.0000000000001820/head//3 214'730 also missing on osd.26 2014-06-09 16:22:11.949600 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.6bb( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.949614 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949588 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {4c0ec24/rbd_data.11606b8b4567.0000000000001820/head//3=214'730(214'729)} 2014-06-09 16:22:11.949621 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x85b0000 (1 active) 2014-06-09 16:22:11.949624 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.949621 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:973 2014-06-09 16:22:11.949620 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 4 as a source 2014-06-09 16:22:11.949632 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.35b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.35b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.949635 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 4c0ec24/rbd_data.11606b8b4567.0000000000001820/head//3 214'730 is on osd.4 2014-06-09 16:22:11.949654 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.949658 7f868ff62700 5 queue_transactions existing osr(3.300 0x3ed0df0)/0x3ed0df0 2014-06-09 16:22:11.949647 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {4c0ec24/rbd_data.11606b8b4567.0000000000001820/head//3=214'730(214'729)} 2014-06-09 16:22:11.949659 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.949660 7f868ff62700 2 waiting 51 > 50 ops || 586174 > 104857600 2014-06-09 16:22:11.949662 7f868a757700 5 queue_transactions existing osr(0.35b 0x2bd6fe0)/0x2bd6fe0 2014-06-09 16:22:11.949664 7f868a757700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.949660 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.949670 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 203 2014-06-09 16:22:11.949670 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.949681 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949681 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 4 2014-06-09 16:22:11.949690 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.482( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949683 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.458( v 213'610 (0'0,213'610] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.949698 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.949691 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.949699 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.949703 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.949702 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.949710 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.731194 4 0.000267 2014-06-09 16:22:11.949715 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.949721 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.949724 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.949730 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.269063 0 0.000000 2014-06-09 16:22:11.949737 7f868ef60700 10 write_log with: dirty_to: 0'0, dirty_from: 214'729, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.949732 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.949743 7f868df5e700 5 queue_transactions existing osr(2.606 0x2ed1d30)/0x2ed1d30 2014-06-09 16:22:11.949741 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.949746 7f868df5e700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.949743 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.949752 7f868bf5a700 2 waiting 51 > 50 ops || 586139 > 104857600 2014-06-09 16:22:11.949748 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949758 7f868cf5c700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.949753 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.949764 7f8692f68700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.949764 7f868ef60700 5 queue_transactions existing osr(3.424 0x3ed1200)/0x3ed1200 2014-06-09 16:22:11.949767 7f868ef60700 2 waiting 51 > 50 ops || 587039 > 104857600 2014-06-09 16:22:11.949760 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.949765 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.949770 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949775 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.949778 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.2ef( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949788 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.949784 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 213'616 -> 213'616 2014-06-09 16:22:11.949791 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.949798 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.949803 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.949807 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.949811 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.949816 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949823 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.949828 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949832 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.18e( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949824 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.458( v 213'610 (0'0,213'610] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.949837 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.949839 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.949846 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.949850 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.949854 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.949847 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'610,213'616], crt=0'0) 2014-06-09 16:22:11.949859 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.949861 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.949863 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.949870 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:200 2014-06-09 16:22:11.949875 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.949881 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.949889 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.949878 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.458( v 213'616 lc 213'610 (0'0,213'616] local-les=191 n=125 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.949894 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:11.949903 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e770 (19 active) 2014-06-09 16:22:11.949909 7f8689755700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.949904 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 94c5cc58/rbd_data.11626b8b4567.0000000000001870/head//3 213'616 is on osd.26 2014-06-09 16:22:11.949915 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {94c5cc58/rbd_data.11626b8b4567.0000000000001870/head//3=213'616(213'610)} 2014-06-09 16:22:11.949914 7f8689755700 10 osd.26 243 dequeue_op 0x10c55900 prio 63 cost 876544 latency 1.866119 osd_op(client.4572.0:30124 rbd_data.117c6b8b4567.0000000000001421 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~876544] 3.511440ce RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.949926 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.949936 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 94c5cc58/rbd_data.11626b8b4567.0000000000001870/head//3 213'616 also missing on osd.2 2014-06-09 16:22:11.949947 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {94c5cc58/rbd_data.11626b8b4567.0000000000001870/head//3=213'616(213'610)} 2014-06-09 16:22:11.949957 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.949966 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.949976 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.949985 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.949995 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.950095 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.950100 7f8691f66700 5 queue_transactions existing osr(3.458 0x3ed1170)/0x3ed1170 2014-06-09 16:22:11.950103 7f8691f66700 2 waiting 51 > 50 ops || 586366 > 104857600 2014-06-09 16:22:11.949277 7f868c75b700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:11.949273, event: started, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950124 7f8691765700 5 -- op tracker -- , seq: 2781, time: 2014-06-09 16:22:11.949298, event: reached_pg, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.950119 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] ov 0'0 av 243'566 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.950134 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] handle_message: 0xdfa6300 2014-06-09 16:22:11.950135 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 2014-06-09 16:22:11.950140 7f8689755700 5 -- op tracker -- , seq: 100, time: 2014-06-09 16:22:11.949930, event: reached_pg, op: osd_op(client.4572.0:30124 rbd_data.117c6b8b4567.0000000000001421 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~876544] 3.511440ce RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950147 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.950158 7f8691765700 5 -- op tracker -- , seq: 2781, time: 2014-06-09 16:22:11.950146, event: started, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.950154 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.950161 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~16384 2014-06-09 16:22:11.950167 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x10c55900 2014-06-09 16:22:11.950166 7f8691765700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] sub_op_modify_reply: tid 4 op ack_type 4 from 7 2014-06-09 16:22:11.950177 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228e00 2014-06-09 16:22:11.950183 7f8691765700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.950177, event: sub_op_commit_rec, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950176 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4572.0:30124 rbd_data.117c6b8b4567.0000000000001421 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~876544] 3.511440ce RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.950189 7f8691765700 10 osd.26 243 dequeue_op 0xdfa6300 finish 2014-06-09 16:22:11.950188 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 snapset=0x26a46b98 snapc=0=[] 2014-06-09 16:22:11.950194 7f8689755700 3 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4572.0:30124 was 214'542 2014-06-09 16:22:11.950200 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.950209 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.950221 7f8689755700 10 osd.26 243 dequeue_op 0x10c55900 finish 2014-06-09 16:22:11.950211 7f868af58700 10 osd.26 243 dequeue_op 0xdfa6200 prio 196 cost 0 latency 0.206896 osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] 2014-06-09 16:22:11.950220 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 0x307e1600 op modify 2014-06-09 16:22:11.950198 7f8691765700 5 -- op tracker -- , seq: 2781, time: 2014-06-09 16:22:11.950195, event: done, op: osd_sub_op_reply(client.4572.0:30633 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.950239 7f8689755700 5 -- op tracker -- , seq: 100, time: 2014-06-09 16:22:11.950224, event: done, op: osd_op(client.4572.0:30124 rbd_data.117c6b8b4567.0000000000001421 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~876544] 3.511440ce RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950232 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:28.423818 2014-06-09 16:22:11.950259 7f8689755700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.950260 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=106 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.950276 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e770 (20 active) 2014-06-09 16:22:11.950278 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.950279 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5d9( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.5d9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.950287 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 18 on osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.950307 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.950301 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 18 o 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.950312 7f8689755700 5 queue_transactions existing osr(2.5d9 0x2ed1a70)/0x2ed1a70 2014-06-09 16:22:11.950314 7f8689755700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.950321 7f868c75b700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:11.950316, event: waiting for subops from 3, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950333 7f868c75b700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.950345 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 213'565 (0'0,213'565] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'565], crt=0'0) [243'566 (0'0) modify 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 by client.4571.0:36397 2014-06-09 10:20:28.423818] 2014-06-09 16:22:11.950366 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'565 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'566 (0'0) modify 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 by client.4571.0:36397 2014-06-09 10:20:28.423818 2014-06-09 16:22:11.950386 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'565 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.950412 7f868c75b700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'566, trimmed: 2014-06-09 16:22:11.950424 7f868c75b700 5 queue_transactions existing osr(3.3f5 0x3ed1280)/0x3ed1280 2014-06-09 16:22:11.950426 7f868c75b700 2 waiting 51 > 50 ops || 604095 > 104857600 2014-06-09 16:22:11.950432 7f868af58700 5 -- op tracker -- , seq: 2812, time: 2014-06-09 16:22:11.950225, event: reached_pg, op: osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.950445 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.950439 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] handle_message: 0xdfa6200 2014-06-09 16:22:11.950451 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.950453 7f868af58700 5 -- op tracker -- , seq: 2812, time: 2014-06-09 16:22:11.950449, event: started, op: osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.950461 7f868af58700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] sub_op_modify_reply: tid 6 op ack_type 4 from 7 2014-06-09 16:22:11.950455 7f8691765700 10 osd.26 243 dequeue_op 0x10c55b00 prio 63 cost 516096 latency 1.803400 osd_op(client.4573.0:33493 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 827392~516096] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.950477 7f868af58700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.950471, event: sub_op_commit_rec, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950484 7f868af58700 10 osd.26 243 dequeue_op 0xdfa6200 finish 2014-06-09 16:22:11.950489 7f8691765700 5 -- op tracker -- , seq: 135, time: 2014-06-09 16:22:11.950471, event: reached_pg, op: osd_op(client.4573.0:33493 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 827392~516096] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950492 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] op_applied: 2 2014-06-09 16:22:11.950499 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.950510 7f86a4e0f700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.950504, event: op_applied, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950510 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x10c55b00 2014-06-09 16:22:11.950517 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] op_applied on primary on version 243'638 2014-06-09 16:22:11.950526 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] repop_all_applied: repop tid 2 all applied 2014-06-09 16:22:11.950523 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:33493 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 827392~516096] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.950535 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] eval_repop repgather(0xc295200 243'638 rep_tid=2 committed?=1 applied?=1 lock=0 op=osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.950539 7f8691765700 3 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4573.0:33493 was 214'579 2014-06-09 16:22:11.950550 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] log_op_stats osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 217469 outb 0 rlat 0.000000 lat 1.217599 2014-06-09 16:22:11.950562 7f8691765700 10 osd.26 243 dequeue_op 0x10c55b00 finish 2014-06-09 16:22:11.950564 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] publish_stats_to_osd 243:1021 2014-06-09 16:22:11.950570 7f8691765700 5 -- op tracker -- , seq: 135, time: 2014-06-09 16:22:11.950565, event: done, op: osd_op(client.4573.0:33493 rbd_data.117d6b8b4567.0000000000001663 [set-alloc-hint object_size 4194304 write_size 4194304,write 827392~516096] 3.51e7f0de RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950575 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 214'636 active+clean] requeue_ops 2014-06-09 16:22:11.950594 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.950584 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 243'638 active+clean] removing repgather(0xc295200 243'638 rep_tid=2 committed?=1 applied?=1 op=osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.950597 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 243'638 active+clean] q front is repgather(0xc295200 243'638 rep_tid=2 committed?=1 applied?=1 op=osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.950612 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153584b0 (20 active) 2014-06-09 16:22:11.950610 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 243'638 active+clean] remove_repop repgather(0xc295200 243'638 rep_tid=2 committed?=1 applied?=1 op=osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.950616 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.950623 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 243'638 active+clean] requeue_ops 2014-06-09 16:22:11.950627 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.950634 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.950641 7f86a4e0f700 5 -- op tracker -- , seq: 637, time: 2014-06-09 16:22:11.950637, event: done, op: osd_op(client.4572.0:30632 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950641 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.950649 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.950656 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.950667 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.950676 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.950682 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.950690 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.950695 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.950697 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.950710 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.950706 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.950715 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x4679b00,0x4679a00,0x4679900,0x4679800,0x4679700,0x257c5200,0x257c5100,0x257c5000,0x257c4f00,0x257c4e00,0x29f5bd00,0x13022000,0x16058900,0x16058800,0x16058700,0x16058600,0x16058500,0x16058400 2014-06-09 16:22:11.950725 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.950740 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 243'638 active+clean] op_commit: 4 2014-06-09 16:22:11.950756 7f86a460e700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.950750, event: op_commit, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950763 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'639 crt=0'0 lcod 243'638 mlcod 243'638 active+clean] repop_all_committed: repop tid 4 all committed 2014-06-09 16:22:11.950770 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.950779 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.950772 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] eval_repop repgather(0xc139740 243'640 rep_tid=4 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.950787 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.950789 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] log_op_stats osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 2064661 outb 0 rlat 0.000000 lat 1.217393 2014-06-09 16:22:11.950799 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1236 2014-06-09 16:22:11.950803 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] publish_stats_to_osd 243:1022 2014-06-09 16:22:11.950808 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.950816 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.950811 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] sending commit on repgather(0xc139740 243'640 rep_tid=4 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 0x30bbe280 2014-06-09 16:22:11.950826 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.950830 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.950841 7f86a460e700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.950836, event: commit_sent, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950841 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.896597 7 0.009514 2014-06-09 16:22:11.950852 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd653a0] 2014-06-09 16:22:11.950853 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.950864 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.950855 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31413ae0,0x31456b20,0x3042aaa0,0x314784e0,0xcc82fe0,0x30d61da0,0x3042a360,0x3042b920,0xbd6b420,0x3042a080,0xc168c20,0x3042b380,0x30e188c0,0xc4119a0,0x30d61880,0x3042a7c0,0x31479120,0xbd75f80,0x30d61700,0xcd0e2c0,0x318c2f60,0x318c3760,0x18742d40,0x13965380,0xbef0380,0xbd6fe60,0x3061ef00,0x2e0ffa0,0x30e19180,0xbcc8420,0x3042acc0,0xbef02c0,0x30e19080,0xbd69fe0,0x30e18500,0x3006ce60,0xc411940,0x3006d0e0,0x2d7392a0,0x30509460,0x2e0ec20,0x318c33e0,0x318c3ca0,0x318c2240,0x2e0e8e0,0x3042bda0,0x3006d1a0,0xc5a7500,0x31791fe0,0x13964100,0x314124a0,0xbd75b20,0x3050b9a0,0x13965a40,0xc5a6140,0xcd32760,0xc5a6ca0,0xbd7df00,0x30c55ba0,0x30c55460,0x30c54860,0x318c3180,0xbd74220,0xcd0e300,0x324803c0,0x3281be00,0x3281b840,0x32073a00,0x32073ac0,0x32073d00,0xc5a78e0,0x32073740,0x307f1600,0xcd32300,0x281b7580,0x314adcc0,0x281b6900,0x3206e900,0x3206fb80,0x32072400,0xd9fac00,0xd9fb280,0xd9fb140,0x187431a0,0x3006cb20,0x3006c560,0x32072a40,0x182cb880,0x320728c0,0x3206ff40,0xcd0f440,0xcd9e3c0,0xcd9ff00,0xcd9f940,0xbd641c0,0xcc82b60,0xc1699a0,0xbd6a940,0xcc95060,0x30629f00,0xc168fa0,0xbd637e0,0x318c3240,0x3042a520,0xc4110c0,0x2d7394a0,0xbd72740,0xcc826e0,0x3042a400,0x2e0e0a0,0x318c2300,0x19941980,0xbd6a320,0x2e0e160,0x31435700,0xcc94b60,0x318c2ac0,0x324d1020,0xccf9160,0xbd606c0,0x3206ef80,0x318c3f40,0xbd65840,0xbd78de0,0x324d16a0,0xbd70c40,0xcc82ec0,0xbd5f0e0,0xbd5dcc0,0xbd5da40,0x306cfe00,0xbfcef80,0xbd66080,0xbd5ce00,0xbd5cc00,0x281b7080,0xc6e8660,0x30629840,0xcc83a40,0x3050a580,0x1a002560,0xbd77de0,0xbd63ba0,0xbd5f800,0xbd5c2e0,0x318c2b80,0x318c25c0,0x18743560,0x29283ea0,0xc169a20,0xbd5c5c0,0xbd5c440,0xbd5ca20,0xbd5c220,0xbd5e540,0xccf8460,0xccf9060,0xc169e20,0x15a3f120,0xbd5c8c0,0x324f99e0,0x3251cae0,0x281b7000,0x1a0024e0,0x17fb7660,0xbef1fe0,0x3251ce60,0x17fb74e0,0xbd606a0,0xc40c900,0x3251d960,0xcc82860,0xd9fb960,0xcc95c60,0x4eb96e0,0x30e18060,0xbdf1b40,0x3062b800,0x307f03c0] 2014-06-09 16:22:11.950874 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:11.950885 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.950887 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] op_commit: 5 2014-06-09 16:22:11.950894 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.950901 7f86a460e700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.950896, event: op_commit, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950902 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.950907 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=214'592 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] repop_all_committed: repop tid 5 all committed 2014-06-09 16:22:11.950916 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'593 crt=0'0 lcod 214'592 mlcod 0'0 active+clean] eval_repop repgather(0xbf2e6c0 243'593 rep_tid=5 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.950920 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:11.950931 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:11.950930 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'593 crt=0'0 lcod 214'592 mlcod 0'0 active+clean] log_op_stats osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 inb 352637 outb 0 rlat 0.000000 lat 1.530334 2014-06-09 16:22:11.950941 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.950945 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'593 crt=0'0 lcod 214'592 mlcod 0'0 active+clean] publish_stats_to_osd 243:965 2014-06-09 16:22:11.950953 7f8691765700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.950953 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'593 crt=0'0 lcod 214'592 mlcod 0'0 active+clean] sending commit on repgather(0xbf2e6c0 243'593 rep_tid=5 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 0x257c6c80 2014-06-09 16:22:11.950967 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:976 2014-06-09 16:22:11.950984 7f86a460e700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.950979, event: commit_sent, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.950995 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.951004 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.951011 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.349( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:11.951020 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'593 crt=0'0 lcod 214'592 mlcod 0'0 active+clean] op_commit: 7 2014-06-09 16:22:11.951034 7f86a460e700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.951028, event: op_commit, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951031 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.951041 7f868d75d700 10 osd.26 243 dequeue_op 0xdfa7100 prio 196 cost 0 latency 0.195687 osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] 2014-06-09 16:22:11.951055 7f8691765700 5 queue_transactions existing osr(3.319 0x3ed0d60)/0x3ed0d60 2014-06-09 16:22:11.951047 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'593 crt=0'0 lcod 214'592 mlcod 0'0 active+clean] repop_all_committed: repop tid 7 all committed 2014-06-09 16:22:11.951058 7f8691765700 2 waiting 51 > 50 ops || 586174 > 104857600 2014-06-09 16:22:11.951057 7f868d75d700 5 -- op tracker -- , seq: 2877, time: 2014-06-09 16:22:11.951054, event: reached_pg, op: osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951059 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'594 crt=0'0 lcod 243'593 mlcod 0'0 active+clean] eval_repop repgather(0xc049740 243'594 rep_tid=7 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.951064 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] handle_message: 0xdfa7100 2014-06-09 16:22:11.951079 7f868d75d700 5 -- op tracker -- , seq: 2877, time: 2014-06-09 16:22:11.951075, event: started, op: osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951075 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'594 crt=0'0 lcod 243'593 mlcod 0'0 active+clean] log_op_stats osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.496133 2014-06-09 16:22:11.951085 7f868d75d700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] sub_op_modify_reply: tid 9 op ack_type 4 from 7 2014-06-09 16:22:11.951090 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'594 crt=0'0 lcod 243'593 mlcod 0'0 active+clean] publish_stats_to_osd 243:966 2014-06-09 16:22:11.951102 7f86a5610700 10 _generic_write 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 61440~516096 = 0 2014-06-09 16:22:11.951103 7f868d75d700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.951096, event: sub_op_commit_rec, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951109 7f868d75d700 10 osd.26 243 dequeue_op 0xdfa7100 finish 2014-06-09 16:22:11.951099 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'594 crt=0'0 lcod 243'593 mlcod 0'0 active+clean] sending commit on repgather(0xc049740 243'594 rep_tid=7 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 0x30bbf180 2014-06-09 16:22:11.951115 7f868d75d700 5 -- op tracker -- , seq: 2877, time: 2014-06-09 16:22:11.951113, event: done, op: osd_sub_op_reply(client.4572.0:30634 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951123 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.951127 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.951127 7f86a460e700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.951122, event: commit_sent, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951131 7f86a5610700 15 _setattrs 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.951130 7f868d75d700 10 osd.26 243 dequeue_op 0x1c769b00 prio 63 cost 1032192 latency 1.453418 osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.951136 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] op_commit: 6 2014-06-09 16:22:11.951143 7f868d75d700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:11.951140, event: reached_pg, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951146 7f86a5610700 10 _setattrs 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 = 0 2014-06-09 16:22:11.951152 7f86a5610700 15 _setattrs 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 2014-06-09 16:22:11.951147 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.951151 7f86a460e700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.951146, event: op_commit, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951160 7f86a5610700 10 _setattrs 3.319_head/8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 = 0 2014-06-09 16:22:11.951156 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1c769b00 2014-06-09 16:22:11.951157 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'640 crt=0'0 lcod 243'639 mlcod 243'638 active+clean] repop_all_committed: repop tid 6 all committed 2014-06-09 16:22:11.951166 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'641 crt=0'0 lcod 243'640 mlcod 243'638 active+clean] eval_repop repgather(0xc1f0540 243'641 rep_tid=6 committed?=1 applied?=0 lock=0 op=osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.951166 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.951186 7f868d75d700 15 getattr 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 '_' 2014-06-09 16:22:11.951181 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'641 crt=0'0 lcod 243'640 mlcod 243'638 active+clean] log_op_stats osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1548565 outb 0 rlat 0.000000 lat 1.187277 2014-06-09 16:22:11.951194 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'641 crt=0'0 lcod 243'640 mlcod 243'638 active+clean] publish_stats_to_osd 243:1023 2014-06-09 16:22:11.951203 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'641 crt=0'0 lcod 243'640 mlcod 243'638 active+clean] sending commit on repgather(0xc1f0540 243'641 rep_tid=6 committed?=1 applied?=0 lock=0 op=osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 0x30bbe500 2014-06-09 16:22:11.951234 7f86a460e700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.951229, event: commit_sent, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951252 7f868d75d700 10 getattr got.size() is 0 2014-06-09 16:22:11.951253 7f86a5e11700 5 _do_op 0x100381e0 seq 3637 osr(2.65 0x2ed0570)/0x2ed0570 start 2014-06-09 16:22:11.951247 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'594 crt=0'0 lcod 243'593 mlcod 0'0 active+clean] op_commit: 8 2014-06-09 16:22:11.951257 7f86a5e11700 10 _do_transaction on 0xc048900 2014-06-09 16:22:11.951254 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 but can_create 2014-06-09 16:22:11.951263 7f868d75d700 15 getattr 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 'snapset' 2014-06-09 16:22:11.951264 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.951263 7f86a460e700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.951258, event: op_commit, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951270 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'594 crt=0'0 lcod 243'593 mlcod 0'0 active+clean] repop_all_committed: repop tid 8 all committed 2014-06-09 16:22:11.951286 7f868d75d700 10 getattr got.size() is 0 2014-06-09 16:22:11.951287 7f868d75d700 15 getattr 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/snapdir//3 'snapset' 2014-06-09 16:22:11.951279 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] eval_repop repgather(0xc128a80 243'595 rep_tid=8 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.951290 7f86a5610700 10 cid: oid: 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 not skipping op, *spos 3640.0.0 2014-06-09 16:22:11.951293 7f86a5610700 10 > header.spos 3582.0.0 2014-06-09 16:22:11.951292 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] log_op_stats osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.470724 2014-06-09 16:22:11.951305 7f868d75d700 10 getattr got.size() is 0 2014-06-09 16:22:11.951306 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] publish_stats_to_osd 243:967 2014-06-09 16:22:11.951313 7f86a5610700 20 set_header setting 30154 cid 3.319_head oid 8721c319/rbd_data.11796b8b4567.0000000000001aab/head//3 parent seq 0 2014-06-09 16:22:11.951310 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30dc8580 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:11.951318 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:11.951315 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] sending commit on repgather(0xc128a80 243'595 rep_tid=8 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 0x30bbf400 2014-06-09 16:22:11.951324 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:11.951331 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30dc8580 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 rwstate(none n=0 w=0) oi: 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc29a0 snapset: 0=[]:[] 2014-06-09 16:22:11.951343 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 @head oi=942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.951346 7f86a460e700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.951341, event: commit_sent, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951354 7f868d75d700 15 getattr 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/snapdir//3 '_' 2014-06-09 16:22:11.951357 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'641 crt=0'0 lcod 243'640 mlcod 243'638 active+clean] op_commit: 9 2014-06-09 16:22:11.951364 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3640.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.951369 7f86a5610700 10 cid: oid: 1fe22c6a/pglog_3.319/0//-1 skipping op, spos 3640.0.0 <= header.spos 150438.0.0 2014-06-09 16:22:11.951373 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.951374 7f868d75d700 10 getattr got.size() is 0 2014-06-09 16:22:11.951372 7f86a460e700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.951367, event: op_commit, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951376 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 942188f6/rbd_data.117f6b8b4567.000000000000153d/snapdir//3 and !can_create 2014-06-09 16:22:11.951377 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'641 crt=0'0 lcod 243'640 mlcod 243'638 active+clean] repop_all_committed: repop tid 9 all committed 2014-06-09 16:22:11.951387 7f868d75d700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:11.951384, event: started, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951392 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30e0f600 2014-06-09 16:22:11.951386 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] eval_repop repgather(0xc2af740 243'642 rep_tid=9 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.951402 7f868d75d700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:11.951399, event: started, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951401 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] log_op_stats osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.200839 2014-06-09 16:22:11.951406 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] ov 0'0 av 243'684 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.951414 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 2014-06-09 16:22:11.951414 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] publish_stats_to_osd 243:1024 2014-06-09 16:22:11.951421 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.951430 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~1032192 2014-06-09 16:22:11.951424 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] sending commit on repgather(0xc2af740 243'642 rep_tid=9 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 0xdfb0000 2014-06-09 16:22:11.951442 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229e00 2014-06-09 16:22:11.951455 7f86a460e700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.951450, event: commit_sent, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951453 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 snapset=0x14dc29d8 snapc=0=[] 2014-06-09 16:22:11.951464 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.951463 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.951474 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.951475 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.951482 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 0x30e0f600 op modify 2014-06-09 16:22:11.951483 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.4d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:181 2014-06-09 16:22:11.951491 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:43.566871 2014-06-09 16:22:11.951496 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] op_commit: 10 2014-06-09 16:22:11.951505 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=127 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:11.951511 7f86a460e700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.951506, event: op_commit, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951515 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.951521 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 19 on osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.951521 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.951530 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.951530 7f868d75d700 7 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 19 o 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:11.951539 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.300( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:11.951548 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.951551 7f868d75d700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:11.951546, event: waiting for subops from 3, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951555 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.951556 7f86af56b700 10 osd.26 243 new session 0x1078a800 con=0x30ba0c60 addr=172.16.96.220:0/1123720 2014-06-09 16:22:11.951563 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.99( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:11.951568 7f868d75d700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.951575 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.951583 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.951592 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.36c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:194 2014-06-09 16:22:11.951585 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 213'683 (0'0,213'683] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'683], crt=0'0) [243'684 (0'0) modify 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 by client.4575.0:32325 2014-06-09 10:19:43.566871] 2014-06-09 16:22:11.951602 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] op_commit: 11 2014-06-09 16:22:11.951635 7f86a460e700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.951611, event: op_commit, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951663 7f868af58700 5 -- op tracker -- , seq: 2812, time: 2014-06-09 16:22:11.950488, event: done, op: osd_sub_op_reply(client.4570.0:34071 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951663 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.951675 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.951672 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.951680 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.67e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:182 2014-06-09 16:22:11.951692 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (20 active) 2014-06-09 16:22:11.951696 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.951709 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.951703 7f8692767700 10 osd.26 243 dequeue_op 0xdfa7000 prio 196 cost 0 latency 0.188256 osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] 2014-06-09 16:22:11.951717 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.951721 7f8692767700 5 -- op tracker -- , seq: 2957, time: 2014-06-09 16:22:11.951718, event: reached_pg, op: osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951721 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.951728 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] handle_message: 0xdfa7000 2014-06-09 16:22:11.951733 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.951743 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.951741 7f8692767700 5 -- op tracker -- , seq: 2957, time: 2014-06-09 16:22:11.951738, event: started, op: osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951746 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.951747 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] sub_op_modify_reply: tid 10 op ack_type 4 from 7 2014-06-09 16:22:11.951756 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.951764 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.951764 7f8692767700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.951759, event: sub_op_commit_rec, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951767 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.951770 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'642 crt=0'0 lcod 243'641 mlcod 243'638 active+clean] repop_all_committed: repop tid 10 all committed 2014-06-09 16:22:11.951777 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.951785 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.951779 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'643 crt=0'0 lcod 243'642 mlcod 243'638 active+clean] eval_repop repgather(0xc232840 243'643 rep_tid=10 committed?=1 applied?=0 lock=0 op=osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.951795 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.951796 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'643 crt=0'0 lcod 243'642 mlcod 243'638 active+clean] log_op_stats osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1167637 outb 0 rlat 0.000000 lat 1.165851 2014-06-09 16:22:11.951607 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'684 (0'0) modify 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 by client.4575.0:32325 2014-06-09 10:19:43.566871 2014-06-09 16:22:11.951812 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'643 crt=0'0 lcod 243'642 mlcod 243'638 active+clean] publish_stats_to_osd 243:1025 2014-06-09 16:22:11.951822 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.951821 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'643 crt=0'0 lcod 243'642 mlcod 243'638 active+clean] sending commit on repgather(0xc232840 243'643 rep_tid=10 committed?=1 applied?=0 lock=0 op=osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 0xdfb0280 2014-06-09 16:22:11.951848 7f868d75d700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'684, trimmed: 2014-06-09 16:22:11.951853 7f8692767700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.951848, event: commit_sent, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951860 7f8692767700 10 osd.26 243 dequeue_op 0xdfa7000 finish 2014-06-09 16:22:11.951864 7f868d75d700 5 queue_transactions existing osr(3.f6 0x3ed04b0)/0x3ed04b0 2014-06-09 16:22:11.951866 7f868d75d700 2 waiting 51 > 50 ops || 1619893 > 104857600 2014-06-09 16:22:11.951870 7f8692767700 5 -- op tracker -- , seq: 2957, time: 2014-06-09 16:22:11.951866, event: done, op: osd_sub_op_reply(client.4570.0:34072 3.60 886c3060/rbd_data.117a6b8b4567.00000000000016e4/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951881 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.951887 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.951876 7f868f761700 10 osd.26 243 dequeue_op 0xdfa6f00 prio 196 cost 0 latency 0.184213 osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'643 crt=0'0 lcod 243'642 mlcod 243'638 active+clean] 2014-06-09 16:22:11.951894 7f868f761700 5 -- op tracker -- , seq: 2971, time: 2014-06-09 16:22:11.951891, event: reached_pg, op: osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951890 7f8692767700 10 osd.26 243 dequeue_op 0x16058300 prio 63 cost 253952 latency 1.714528 osd_op(client.4576.0:35762 rbd_data.11806b8b4567.0000000000001807 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~253952] 3.122100ce RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.951900 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'643 crt=0'0 lcod 243'642 mlcod 243'638 active+clean] handle_message: 0xdfa6f00 2014-06-09 16:22:11.951910 7f8692767700 5 -- op tracker -- , seq: 190, time: 2014-06-09 16:22:11.951906, event: reached_pg, op: osd_op(client.4576.0:35762 rbd_data.11806b8b4567.0000000000001807 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~253952] 3.122100ce RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951923 7f868f761700 5 -- op tracker -- , seq: 2971, time: 2014-06-09 16:22:11.951910, event: started, op: osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.951919 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.951928 7f868f761700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'643 crt=0'0 lcod 243'642 mlcod 243'638 active+clean] sub_op_modify_reply: tid 11 op ack_type 4 from 7 2014-06-09 16:22:11.951931 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x16058300 2014-06-09 16:22:11.951944 7f868f761700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.951939, event: sub_op_commit_rec, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951940 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4576.0:35762 rbd_data.11806b8b4567.0000000000001807 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~253952] 3.122100ce RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.951950 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'643 crt=0'0 lcod 243'642 mlcod 243'638 active+clean] repop_all_committed: repop tid 11 all committed 2014-06-09 16:22:11.951956 7f8692767700 3 osd.26 pg_epoch: 243 pg[3.ce( v 214'547 (0'0,214'547] local-les=243 n=109 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4576.0:35762 was 213'535 2014-06-09 16:22:11.951959 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] eval_repop repgather(0x30d1740 243'644 rep_tid=11 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.951979 7f8692767700 10 osd.26 243 dequeue_op 0x16058300 finish 2014-06-09 16:22:11.951973 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] log_op_stats osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.196710 2014-06-09 16:22:11.951987 7f8692767700 5 -- op tracker -- , seq: 190, time: 2014-06-09 16:22:11.951982, event: done, op: osd_op(client.4576.0:35762 rbd_data.11806b8b4567.0000000000001807 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~253952] 3.122100ce RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.951987 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] publish_stats_to_osd 243:1026 2014-06-09 16:22:11.952000 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.951997 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] sending commit on repgather(0x30d1740 243'644 rep_tid=11 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 0xdfb1680 2014-06-09 16:22:11.952020 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f23c0 (20 active) 2014-06-09 16:22:11.952028 7f868f761700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.952023, event: commit_sent, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.952024 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=243 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952035 7f868f761700 10 osd.26 243 dequeue_op 0xdfa6f00 finish 2014-06-09 16:22:11.952037 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.688( v 213'675 (0'0,213'675] local-les=243 n=126 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952043 7f868f761700 5 -- op tracker -- , seq: 2971, time: 2014-06-09 16:22:11.952040, event: done, op: osd_sub_op_reply(client.4572.0:30635 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.952048 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.952053 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.952059 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.952063 7f868f761700 10 osd.26 243 dequeue_op 0x29f5b200 prio 63 cost 106496 latency 1.140247 osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.952084 7f868f761700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.952079, event: reached_pg, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.952092 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.952105 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x29f5b200 2014-06-09 16:22:11.952115 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952115 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.952129 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952135 7f868f761700 15 getattr 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 '_' 2014-06-09 16:22:11.952139 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952143 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952154 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.624( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952163 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952167 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952179 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.213( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952189 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952192 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952203 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952196 7f8689f56700 10 osd.26 243 dequeue_op 0xdfa6e00 prio 196 cost 0 latency 0.144475 osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] 2014-06-09 16:22:11.952212 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952216 7f8689f56700 5 -- op tracker -- , seq: 3254, time: 2014-06-09 16:22:11.952213, event: reached_pg, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.952216 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952225 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] handle_message: 0xdfa6e00 2014-06-09 16:22:11.952229 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952239 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952240 7f8689f56700 5 -- op tracker -- , seq: 3254, time: 2014-06-09 16:22:11.952236, event: started, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.952242 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952246 7f8689f56700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] sub_op_modify_reply: tid 13 op ack_type 1 from 7 2014-06-09 16:22:11.952253 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952262 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952265 7f8689f56700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.952259, event: sub_op_applied_rec, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.952272 7f8689f56700 10 osd.26 243 dequeue_op 0xdfa6e00 finish 2014-06-09 16:22:11.952265 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952277 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952286 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952282 7f8689f56700 5 -- op tracker -- , seq: 3254, time: 2014-06-09 16:22:11.952278, event: done, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ack, result = 0) 2014-06-09 16:22:11.952293 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.952290 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952301 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952308 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952311 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f25d0 (20 active) 2014-06-09 16:22:11.952310 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952318 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952324 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952316 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:11.952326 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952334 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952340 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952342 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952348 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952353 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952356 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.952355 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952361 7f8689f56700 5 queue_transactions existing osr(3.77 0x3ed0250)/0x3ed0250 2014-06-09 16:22:11.952363 7f8689f56700 2 waiting 51 > 50 ops || 586139 > 104857600 2014-06-09 16:22:11.952361 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952367 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952369 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952376 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952380 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952382 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952389 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952393 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952382 7f8690f64700 10 osd.26 243 dequeue_op 0x32826600 prio 196 cost 0 latency 0.143361 osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] 2014-06-09 16:22:11.952396 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952402 7f8690f64700 5 -- op tracker -- , seq: 3255, time: 2014-06-09 16:22:11.952398, event: reached_pg, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.952404 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952410 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952412 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952409 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] handle_message: 0x32826600 2014-06-09 16:22:11.952420 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952426 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952424 7f8690f64700 5 -- op tracker -- , seq: 3255, time: 2014-06-09 16:22:11.952421, event: started, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.952435 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (19 active) 2014-06-09 16:22:11.952439 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.952430 7f8690f64700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] sub_op_modify_reply: tid 13 op ack_type 4 from 7 2014-06-09 16:22:11.952442 7f868af58700 10 osd.26 243 dequeue_op 0x1b0d7800 prio 63 cost 1019904 latency 0.231173 osd_op(client.4538.0:36645 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1019904] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.952448 7f8690f64700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.952442, event: sub_op_commit_rec, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.952458 7f8690f64700 10 osd.26 243 dequeue_op 0x32826600 finish 2014-06-09 16:22:11.952458 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.952466 7f8690f64700 5 -- op tracker -- , seq: 3255, time: 2014-06-09 16:22:11.952463, event: done, op: osd_sub_op_reply(client.4572.0:30636 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.952463 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952478 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.952474 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.952478 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952489 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.952496 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358400 (20 active) 2014-06-09 16:22:11.952494 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952501 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.952505 7f868af58700 5 -- op tracker -- , seq: 2756, time: 2014-06-09 16:22:11.952453, event: reached_pg, op: osd_op(client.4538.0:36645 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1019904] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.952505 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.771( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952514 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.952508 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952513 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.952523 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.952518 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952514 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.952522 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.952529 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.952529 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.952537 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.952529 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 1 flushes_in_progress pending waiting for active on 0x1b0d7800 2014-06-09 16:22:11.952539 7f868af58700 10 osd.26 243 dequeue_op 0x1b0d7800 finish 2014-06-09 16:22:11.952540 7f8690f64700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.952542 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.952537 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.952547 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.952556 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:156 2014-06-09 16:22:11.952553 7f868af58700 10 osd.26 243 dequeue_op 0x32826800 prio 63 cost 516096 latency 0.135309 osd_op(client.4539.0:34390 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.952570 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.952576 7f868af58700 5 -- op tracker -- , seq: 3276, time: 2014-06-09 16:22:11.952571, event: reached_pg, op: osd_op(client.4539.0:34390 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 1990656~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.952584 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.952591 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.220( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.952541 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.952599 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952602 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 1 flushes_in_progress pending waiting for active on 0x32826800 2014-06-09 16:22:11.952610 7f868af58700 10 osd.26 243 dequeue_op 0x32826800 finish 2014-06-09 16:22:11.952614 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.952613 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.952610 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.13c( v 213'621 (0'0,213'621] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.952624 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.952623 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.952635 7f8690f64700 5 queue_transactions existing osr(0.220 0x2bd6a70)/0x2bd6a70 2014-06-09 16:22:11.952632 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.952642 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (20 active) 2014-06-09 16:22:11.952631 7f8690763700 10 osd.26 243 dequeue_op 0x32826500 prio 196 cost 0 latency 0.099276 osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) v2 pg pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] 2014-06-09 16:22:11.952649 7f8690763700 5 -- op tracker -- , seq: 3315, time: 2014-06-09 16:22:11.952646, event: reached_pg, op: osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.952646 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:11.952656 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] handle_message: 0x32826500 2014-06-09 16:22:11.952671 7f8690763700 5 -- op tracker -- , seq: 3315, time: 2014-06-09 16:22:11.952667, event: started, op: osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.952638 7f8690f64700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.952679 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.952683 7f868af58700 5 queue_transactions existing osr(3.616 0x3ed1a90)/0x3ed1a90 2014-06-09 16:22:11.952676 7f8690763700 7 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] sub_op_modify_reply: tid 15 op ack_type 4 from 7 2014-06-09 16:22:11.952686 7f868af58700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.952688 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952694 7f8690763700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.952688, event: sub_op_commit_rec, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.952701 7f8690763700 10 osd.26 243 dequeue_op 0x32826500 finish 2014-06-09 16:22:11.952704 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.952708 7f8690763700 5 -- op tracker -- , seq: 3315, time: 2014-06-09 16:22:11.952704, event: done, op: osd_sub_op_reply(client.4572.0:30637 3.60 5a3860/rbd_data.117c6b8b4567.0000000000001472/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.952718 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.952725 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.952718 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.952736 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952729 7f8690763700 10 osd.26 243 dequeue_op 0x2c06b000 prio 63 cost 8192 latency 0.053263 osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:11.952748 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.952752 7f8690763700 5 -- op tracker -- , seq: 3335, time: 2014-06-09 16:22:11.952747, event: reached_pg, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.952759 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] flushed 2014-06-09 16:22:11.952759 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.952771 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x2c06b000 2014-06-09 16:22:11.952778 7f8690763700 10 osd.26 243 dequeue_op 0x2c06b000 finish 2014-06-09 16:22:11.952780 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.952774 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952788 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.952798 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8fe40 (20 active) 2014-06-09 16:22:11.952801 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.952803 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.952818 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952816 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.952827 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.952828 7f868f761700 10 getattr got.size() is 0 2014-06-09 16:22:11.952826 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.952834 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.952838 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.952847 7f8690763700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.952848 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952831 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 but can_create 2014-06-09 16:22:11.952855 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.952856 7f868f761700 15 getattr 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 'snapset' 2014-06-09 16:22:11.952865 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.72b( v 213'642 (0'0,213'642] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1092 2014-06-09 16:22:11.952870 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.952881 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.952885 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952885 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.5d3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.952895 7f868f761700 10 getattr got.size() is 0 2014-06-09 16:22:11.952896 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.952898 7f868f761700 15 getattr 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/snapdir//3 'snapset' 2014-06-09 16:22:11.952897 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.952906 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.952906 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.952914 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.952926 7f8690763700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.952929 7f868f761700 10 getattr got.size() is 0 2014-06-09 16:22:11.952923 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 213 same_interval_since 242 2014-06-09 16:22:11.952936 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.952935 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x307a7340 f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.952939 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.5d3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.952953 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.952948 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.952949 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.952957 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.952957 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:11.952969 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.952978 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.952966 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x307a7340 f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 rwstate(none n=0 w=0) oi: f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x26a462a0 snapset: 0=[]:[] 2014-06-09 16:22:11.952986 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.952980 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 @head oi=f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.952990 7f8690763700 5 queue_transactions existing osr(3.5b3 0x3ed17e0)/0x3ed17e0 2014-06-09 16:22:11.952992 7f8690763700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.952987 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.952996 7f868f761700 15 getattr 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/snapdir//3 '_' 2014-06-09 16:22:11.952999 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953004 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953015 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.953011 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.953021 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953025 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.953027 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953039 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953039 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.953047 7f868f761700 10 getattr got.size() is 0 2014-06-09 16:22:11.953049 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953049 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.953049 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid f1db70ff/rbd_data.11816b8b4567.0000000000001948/snapdir//3 and !can_create 2014-06-09 16:22:11.953052 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953058 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.953063 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.953071 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953066 7f868f761700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.953061, event: started, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.953075 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953073 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x308b5e00 2014-06-09 16:22:11.953084 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.953082 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953093 7f868f761700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.953088, event: started, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.953093 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.953094 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.953102 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953099 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] ov 0'0 av 243'613 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.953106 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953111 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953112 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 2014-06-09 16:22:11.953121 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.953118 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.953128 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953122 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.953136 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.953136 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~106496 2014-06-09 16:22:11.953151 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953152 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0x1078ae00 2014-06-09 16:22:11.953162 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.953156 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953162 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 snapset=0x26a462d8 snapc=0=[] 2014-06-09 16:22:11.953167 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.953169 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.953176 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953172 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.953179 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.953180 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953181 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.953190 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.129( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.953189 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.953190 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 0x308b5e00 op modify 2014-06-09 16:22:11.953199 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953202 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953205 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953201 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:32.053710 2014-06-09 16:22:11.953216 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.953212 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.1cf( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.953221 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953220 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=117 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.953227 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.953237 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953245 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.953237 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.953240 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=243 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.953247 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 20 on osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.953253 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.753( v 213'594 (0'0,213'594] local-les=243 n=110 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.953253 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.953264 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953261 7f868f761700 7 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 20 o f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.953268 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.69d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.953270 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 201 same_interval_since 242 2014-06-09 16:22:11.953281 7f868f761700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.953276, event: waiting for subops from 5, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.953284 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.953291 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.953297 7f868f761700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.953304 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.953302 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953308 7f8692767700 5 queue_transactions existing osr(2.69d 0x2ed1ef0)/0x2ed1ef0 2014-06-09 16:22:11.953316 7f8692767700 2 waiting 51 > 50 ops || 586141 > 104857600 2014-06-09 16:22:11.953316 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 213'612 (0'0,213'612] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'612], crt=0'0) [243'613 (0'0) modify f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 by client.4577.0:37476 2014-06-09 10:19:32.053710] 2014-06-09 16:22:11.953339 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=213'612 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'613 (0'0) modify f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 by client.4577.0:37476 2014-06-09 10:19:32.053710 2014-06-09 16:22:11.953360 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=213'612 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.953389 7f868f761700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'613, trimmed: 2014-06-09 16:22:11.953407 7f868f761700 5 queue_transactions existing osr(3.ff 0x3ed0870)/0x3ed0870 2014-06-09 16:22:11.953410 7f868f761700 2 waiting 51 > 50 ops || 694197 > 104857600 2014-06-09 16:22:11.953417 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.953442 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.953444 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.953446 7f869af78700 20 osd.26 243 _dispatch 0x31388700 pg_info(1 pgs e243:0.725) v4 2014-06-09 16:22:11.953459 7f869af78700 5 -- op tracker -- , seq: 3360, time: 2014-06-09 16:22:11.953374, event: header_read, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.953468 7f869af78700 5 -- op tracker -- , seq: 3360, time: 2014-06-09 16:22:11.953376, event: throttled, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.953475 7f869af78700 5 -- op tracker -- , seq: 3360, time: 2014-06-09 16:22:11.953392, event: all_read, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.953481 7f869af78700 5 -- op tracker -- , seq: 3360, time: 2014-06-09 16:22:11.953441, event: dispatched, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.953488 7f869af78700 5 -- op tracker -- , seq: 3360, time: 2014-06-09 16:22:11.953486, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.953493 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.725) v4 from osd.7 2014-06-09 16:22:11.953496 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31388700 2014-06-09 16:22:11.953502 7f869af78700 5 -- op tracker -- , seq: 3360, time: 2014-06-09 16:22:11.953500, event: started, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.953515 7f869af78700 5 -- op tracker -- , seq: 3360, time: 2014-06-09 16:22:11.953514, event: done, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.953520 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.953521 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.953561 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.953579 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953593 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.953605 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.953619 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953632 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.953641 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.953657 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953666 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.953676 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.953692 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953705 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.953720 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.953735 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953746 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.953753 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.953767 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] op_applied: 3 2014-06-09 16:22:11.953785 7f86a4e0f700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.953778, event: op_applied, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.953794 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] op_applied on primary on version 243'639 2014-06-09 16:22:11.953800 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] repop_all_applied: repop tid 3 all applied 2014-06-09 16:22:11.953805 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] eval_repop repgather(0xbfca900 243'639 rep_tid=3 committed?=1 applied?=1 lock=0 op=osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.953815 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] log_op_stats osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.190876 2014-06-09 16:22:11.953824 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] publish_stats_to_osd 243:1027 2014-06-09 16:22:11.953831 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'638 active+clean] requeue_ops 2014-06-09 16:22:11.953840 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] removing repgather(0xbfca900 243'639 rep_tid=3 committed?=1 applied?=1 op=osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.953855 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] q front is repgather(0xbfca900 243'639 rep_tid=3 committed?=1 applied?=1 op=osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.953864 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] remove_repop repgather(0xbfca900 243'639 rep_tid=3 committed?=1 applied?=1 op=osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.953872 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] requeue_ops 2014-06-09 16:22:11.953886 7f86a4e0f700 5 -- op tracker -- , seq: 692, time: 2014-06-09 16:22:11.953882, event: done, op: osd_op(client.4570.0:34070 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 962560~516096] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.953924 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.953938 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.953952 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.953968 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.953982 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.953999 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954012 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.954022 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.954036 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954047 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.954056 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.954071 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954085 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.954100 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.954114 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954124 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.954134 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.954149 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 206 same_interval_since 242 2014-06-09 16:22:11.954164 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.954187 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.954200 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954210 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.954220 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.954243 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954252 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.954262 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.954277 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954288 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.954298 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.954314 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954323 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.954330 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.954339 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954351 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.954368 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.954385 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954396 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.954405 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.954425 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954436 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.954447 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.954464 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954475 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.954485 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.954501 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954515 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.954524 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.954538 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954553 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.954567 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.954588 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954598 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.954608 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.954617 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954625 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.954634 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] flushed 2014-06-09 16:22:11.954648 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954662 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.954675 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.954691 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954701 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.954710 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.954724 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954739 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.954752 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.954770 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954779 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.954788 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.954804 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954813 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.954823 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.954836 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.954849 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.954864 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954874 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.954884 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.954899 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954909 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.954919 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.954935 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954948 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.954961 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.954983 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.954997 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.955012 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.955037 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955050 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.955064 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.955080 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.955094 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.955108 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189abde0,0xc5a70e0,0xbd5a120,0xbebffc0,0xc5a7540,0xc6e99a0,0xbd78440,0xbd71cc0,0xbe31980,0x3062bde0,0xbd6d6e0,0xbccd640,0x13965d60,0xc2d63c0,0xbe306e0,0xbd5a920,0xbd667a0,0xbd680a0,0xbebf4c0,0x17001380,0x305083e0,0xbcc9f60,0xbd6eaa0,0xbd6a960,0x3062bd20,0x189aa660,0xbd76d40,0x3050bb00,0xc411880,0xbd6bf80,0xbd7ca20,0x33a7380,0xc2d6c40,0x314ac380,0x31479d80,0x307f07e0,0x18742ec0,0x3061e2c0,0x306cef60,0x3050bec0,0xcc838e0,0xbd6ddc0,0xcdb2820,0x292820a0,0x19941120,0xbd6b880,0xcc95360,0xbdbb760,0xbfce100,0x314acfc0,0xc2d7760,0x3062a640,0x30c543a0,0xbef19e0,0x31456c80,0x307f0d20,0x307f1f00,0xcd325a0,0x314acce0,0x307f1360,0xcd32f60,0x314ac100,0x314ac9c0,0x314adf20,0x314adca0,0x314127e0,0x307f1060,0x33a7a60,0xc2d7840,0x30c54e80,0xbccc060,0x189aa860,0xcd33460,0x314ac580,0xc6e97e0,0x314ad360,0x314121a0,0xcd33220,0x314567e0,0x307f1760,0x31435400,0x31456f20,0x31434b20,0x314571a0,0xcd33f20,0x31435620,0x31435320,0x3050a3a0,0x31456820,0xbfcea60,0x314ad080,0x17001540,0xbd69cc0,0xc6e9080,0x30c54ac0,0x31413320,0xcdb3de0,0x31457ea0,0xc5a75a0,0x31478820,0x31413ce0] 2014-06-09 16:22:11.955118 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6ef20,0x31413aa0,0x31456ae0,0x3042aac0,0x31478520,0x2d738fe0,0x30d61dc0,0x3042a380,0x3042b8a0,0xbd7d8a0,0x3042a0a0,0x4eb8aa0,0x3042b3a0,0x30e18880,0x30629020,0x30d618a0,0xc410a60,0x314790e0,0xc6e9fc0,0x30e19680,0xbccde40,0x318c2fa0,0x318c3720,0xc6e80e0,0xc1681c0,0xbd71340,0x30508660,0xbd7ce00,0x2e0ffc0,0x30e19940,0xbd5b6a0,0x3042ace0,0x3050a340,0x30e19040,0x18742920,0x30e184c0,0x3006ce20,0x2d738ea0,0x3006d0a0,0x4eb92a0,0xc6e8680,0x2e0ed20,0x318c33a0,0x318c3c60,0x318c2200,0x2e0e8a0,0x30c54780,0x3006d160,0xc5a78a0,0x31791fa0,0xc169460,0x314124e0,0xcd0fbc0,0xbd726e0,0x13965240,0xbebe740,0xcd32720,0xc5a7000,0xb9c1fa0,0x30c55bc0,0x30c55480,0x32074580,0x318c3140,0x4eb8920,0xbce7360,0x32480340,0x3281bd80,0x3050bce0,0x32073fe0,0x32073a80,0x32073c80,0x1a003080,0x320736c0,0x307f1580,0xcd32280,0x32073840,0x314adc80,0x281b6a00,0x3206e980,0x3206f400,0x32072480,0xd9fbbe0,0xd9fb3a0,0xd9fb1a0,0x3006cee0,0x3006cae0,0x3006c4e0,0x32072ac0,0x281b6d80,0x32072940,0x3206f840,0xcd0f480,0xcd0f040,0xcd9fe00,0xbd6d340,0xcd9f020,0xbd6b780,0x307f0200,0xbdf0260,0xc168d40,0xbd679a0,0xc168920,0xcc95ea0,0x318c3200,0x3042a540,0xbd69ca0,0x31412240,0xbd730e0,0xcc82aa0,0x3042a420,0x2e0e060,0x318c22c0,0x30629b00,0xbd6c120,0x2e0e120,0x31435740,0xcc94060,0x318c2a80,0xc2d74e0,0xccf91c0,0xbd606e0,0x3206ef00,0x318c3f00,0xbd7c9c0,0x306cf080,0x324d1720,0xbcc82a0,0x306ce180,0xbd5e000,0xbd5dca0,0xbd5daa0,0xcc95720,0x17001e20,0x314353c0,0xbd5ce80,0xbd5cc80,0x281b7680,0xbccc260,0x314128a0,0xbd67280,0x3061fee0,0xbfcfea0,0xbd76240,0x31412580,0xbd5ff20,0xbd5c300,0x318c2b40,0x18742a00,0x15a3f1e0,0x3061edc0,0x3050b040,0xbd5c620,0xbd5c4c0,0xbd5caa0,0xbd5c280,0xbd5e5e0,0xbe30300,0xccf88e0,0xc40c680,0xc1695e0,0xbd5c920,0x324f9ae0,0x3251cbe0,0x30628040,0x314136c0,0x17fb76c0,0x139654c0,0x3251cf60,0x17fb7580,0xbd607a0,0x189aac20,0x3251d9e0,0x3206e880,0xd9fb9c0,0x3042b6e0,0x4eb9060,0xc168120,0xcc82fc0,0xbd73900,0xcc82360] 2014-06-09 16:22:11.955139 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] op_applied: 4 2014-06-09 16:22:11.955151 7f86a4e0f700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.955146, event: op_applied, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.955158 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] op_applied on primary on version 243'640 2014-06-09 16:22:11.955165 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] repop_all_applied: repop tid 4 all applied 2014-06-09 16:22:11.955173 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] eval_repop repgather(0xc139740 243'640 rep_tid=4 committed?=1 applied?=1 lock=0 op=osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.955184 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] log_op_stats osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 2064661 outb 0 rlat 0.000000 lat 1.221787 2014-06-09 16:22:11.955193 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] publish_stats_to_osd 243:1028 2014-06-09 16:22:11.955200 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] requeue_ops 2014-06-09 16:22:11.955205 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] removing repgather(0xc139740 243'640 rep_tid=4 committed?=1 applied?=1 op=osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.955214 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] q front is repgather(0xc139740 243'640 rep_tid=4 committed?=1 applied?=1 op=osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.955224 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] remove_repop repgather(0xc139740 243'640 rep_tid=4 committed?=1 applied?=1 op=osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.955236 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] requeue_ops 2014-06-09 16:22:11.955249 7f86a4e0f700 5 -- op tracker -- , seq: 662, time: 2014-06-09 16:22:11.955246, event: done, op: osd_op(client.4572.0:30633 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 217088~2064384] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.955289 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955298 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.955312 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.955329 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.955337 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.955345 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955352 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.955364 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.955362 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.955376 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 210 same_interval_since 242 2014-06-09 16:22:11.955387 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3637.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.955391 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.955390 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.955399 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.955411 7f86a5e11700 10 _do_op 0x100381e0 seq 3637 r = 0, finisher 0x4eb9240 0 2014-06-09 16:22:11.955416 7f86a5e11700 10 _finish_op 0x100381e0 seq 3637 osr(2.65 0x2ed0570)/0x2ed0570 2014-06-09 16:22:11.955412 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 201 same_interval_since 242 2014-06-09 16:22:11.955426 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115200 (1 active) 2014-06-09 16:22:11.955429 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496120 (2 active) 2014-06-09 16:22:11.955432 7f86a5e11700 5 _do_op 0xc705860 seq 3641 osr(1.382 0x384d5b0)/0x384d5b0 start 2014-06-09 16:22:11.955436 7f86a5e11700 10 _do_transaction on 0xc1fcfc0 2014-06-09 16:22:11.955427 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.955442 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.955440 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.955451 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955460 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.955470 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.955480 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.955485 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955495 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.955496 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3641.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.955500 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.955502 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.955510 7f86a5e11700 10 _do_op 0xc705860 seq 3641 r = 0, finisher 0xbd762c0 0 2014-06-09 16:22:11.955514 7f86a5e11700 10 _finish_op 0xc705860 seq 3641 osr(1.382 0x384d5b0)/0x384d5b0 2014-06-09 16:22:11.955511 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] op_applied: 5 2014-06-09 16:22:11.955518 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496120 (1 active) 2014-06-09 16:22:11.955521 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ea20 (2 active) 2014-06-09 16:22:11.955523 7f86a5e11700 5 _do_op 0xc705560 seq 3642 osr(0.54c 0x2bd7850)/0x2bd7850 start 2014-06-09 16:22:11.955525 7f86a5e11700 10 _do_transaction on 0xc2dae40 2014-06-09 16:22:11.955531 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.955522 7f86a4e0f700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.955518, event: op_applied, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.955535 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] op_applied on primary on version 243'593 2014-06-09 16:22:11.955547 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] repop_all_applied: repop tid 5 all applied 2014-06-09 16:22:11.955566 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.955556 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] eval_repop repgather(0xbf2e6c0 243'593 rep_tid=5 committed?=1 applied?=1 lock=0 op=osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.955579 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3642.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.955583 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.955573 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] log_op_stats osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 inb 352637 outb 0 rlat 0.000000 lat 1.534977 2014-06-09 16:22:11.955592 7f86a5e11700 10 _do_op 0xc705560 seq 3642 r = 0, finisher 0xbd6a460 0 2014-06-09 16:22:11.955594 7f868b759700 10 op_submit_start 3689 2014-06-09 16:22:11.955595 7f86a5e11700 10 _finish_op 0xc705560 seq 3642 osr(0.54c 0x2bd7850)/0x2bd7850 2014-06-09 16:22:11.955588 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] publish_stats_to_osd 243:968 2014-06-09 16:22:11.955597 7f868b759700 5 queue_transactions (trailing journal) 3689 2014-06-09 16:22:11.955598 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ea20 (1 active) 2014-06-09 16:22:11.955600 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6fc0 (2 active) 2014-06-09 16:22:11.955599 7f868b759700 5 queue_op 0x324e6c60 seq 3689 osr(2.f4 0x2ed0790) 826 bytes (queue has 48 ops and 583657 bytes) 2014-06-09 16:22:11.955602 7f86a5e11700 5 _do_op 0xc2d4180 seq 3643 osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 start 2014-06-09 16:22:11.955603 7f868b759700 10 op_submit_finish 3689 2014-06-09 16:22:11.955604 7f86a5e11700 10 _do_transaction on 0xc2efbc0 2014-06-09 16:22:11.955599 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 0'0 active+clean] requeue_ops 2014-06-09 16:22:11.955608 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.955607 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.955628 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.817219 7 0.000389 2014-06-09 16:22:11.955622 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] removing repgather(0xbf2e6c0 243'593 rep_tid=5 committed?=1 applied?=1 op=osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.955638 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.955650 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.955639 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] q front is repgather(0xbf2e6c0 243'593 rep_tid=5 committed?=1 applied?=1 op=osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.955650 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.955663 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3643.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.955653 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] remove_repop repgather(0xbf2e6c0 243'593 rep_tid=5 committed?=1 applied?=1 op=osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.955667 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.955662 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 2014-06-09 16:22:11.955667 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] requeue_ops 2014-06-09 16:22:11.955675 7f86a5e11700 10 _do_op 0xc2d4180 seq 3643 r = 0, finisher 0xbd60260 0 2014-06-09 16:22:11.955672 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.955678 7f86a5e11700 10 _finish_op 0xc2d4180 seq 3643 osr(0.2c3 0x2bd6ba0)/0x2bd6ba0 2014-06-09 16:22:11.955680 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6fc0 (1 active) 2014-06-09 16:22:11.955682 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301320 (2 active) 2014-06-09 16:22:11.955680 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.955684 7f86a5e11700 5 _do_op 0xc2d4f60 seq 3644 osr(2.3f0 0x2ed1320)/0x2ed1320 start 2014-06-09 16:22:11.955687 7f86a5e11700 10 _do_transaction on 0x2c19d40 2014-06-09 16:22:11.955686 7f86a4e0f700 5 -- op tracker -- , seq: 376, time: 2014-06-09 16:22:11.955681, event: done, op: osd_op(client.4571.0:35947 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~352256] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.955692 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.955687 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.955698 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.955708 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.955716 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.955712 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955726 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.955727 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.955728 7f868b759700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.955739 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3644.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.955744 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.955743 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:11.955754 7f86a5e11700 10 _do_op 0xc2d4f60 seq 3644 r = 0, finisher 0x2d739380 0 2014-06-09 16:22:11.955758 7f86a5e11700 10 _finish_op 0xc2d4f60 seq 3644 osr(2.3f0 0x2ed1320)/0x2ed1320 2014-06-09 16:22:11.955760 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301320 (1 active) 2014-06-09 16:22:11.955762 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d9e0 (2 active) 2014-06-09 16:22:11.955755 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.955764 7f86a5e11700 5 _do_op 0xc2d41e0 seq 3645 osr(0.380 0x2bd6f20)/0x2bd6f20 start 2014-06-09 16:22:11.955767 7f86a5e11700 10 _do_transaction on 0xc1f7ec0 2014-06-09 16:22:11.955771 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.955777 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.955770 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955782 7f868b759700 5 queue_transactions existing osr(1.2b0 0x384d1b0)/0x384d1b0 2014-06-09 16:22:11.955783 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.955785 7f868b759700 10 op_submit_start 3690 2014-06-09 16:22:11.955786 7f868b759700 5 queue_transactions (trailing journal) 3690 2014-06-09 16:22:11.955787 7f868b759700 5 queue_op 0x16fd4c60 seq 3690 osr(1.2b0 0x384d1b0) 861 bytes (queue has 47 ops and 582862 bytes) 2014-06-09 16:22:11.955789 7f868b759700 10 op_submit_finish 3690 2014-06-09 16:22:11.955792 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) 2014-06-09 16:22:11.955798 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.955806 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.955819 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3645.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.955822 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.955815 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.955817 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955824 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.955833 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.955831 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.955841 7f868b759700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.955843 7f86a5e11700 10 _do_op 0xc2d41e0 seq 3645 r = 0, finisher 0xbfcf500 0 2014-06-09 16:22:11.955847 7f86a5e11700 10 _finish_op 0xc2d41e0 seq 3645 osr(0.380 0x2bd6f20)/0x2bd6f20 2014-06-09 16:22:11.955849 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d9e0 (1 active) 2014-06-09 16:22:11.955851 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbc20 (2 active) 2014-06-09 16:22:11.955853 7f86a5e11700 5 _do_op 0x288f8420 seq 3646 osr(0.663 0x2bd7f30)/0x2bd7f30 start 2014-06-09 16:22:11.955849 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.955855 7f86a5e11700 10 _do_transaction on 0xc29d680 2014-06-09 16:22:11.955861 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.955854 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:150 2014-06-09 16:22:11.955863 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.955872 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.955891 7f868e75f700 10 op_submit_start 3691 2014-06-09 16:22:11.955893 7f868e75f700 5 queue_transactions (trailing journal) 3691 2014-06-09 16:22:11.955895 7f868e75f700 5 queue_op 0x324f40c0 seq 3691 osr(2.e6 0x2ed07d0) 826 bytes (queue has 47 ops and 582860 bytes) 2014-06-09 16:22:11.955898 7f868e75f700 10 op_submit_finish 3691 2014-06-09 16:22:11.955897 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.955902 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.620( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.620( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:11.955912 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3646.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.955916 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.955926 7f86a5e11700 10 _do_op 0x288f8420 seq 3646 r = 0, finisher 0xcc95dc0 0 2014-06-09 16:22:11.955930 7f86a5e11700 10 _finish_op 0x288f8420 seq 3646 osr(0.663 0x2bd7f30)/0x2bd7f30 2014-06-09 16:22:11.955934 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.955933 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbc20 (1 active) 2014-06-09 16:22:11.955935 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286c60 (2 active) 2014-06-09 16:22:11.955937 7f86a5e11700 5 _do_op 0x324f47e0 seq 3647 osr(3.7e2 0xb7f0120)/0xb7f0120 start 2014-06-09 16:22:11.955940 7f86a5e11700 10 _do_transaction on 0xc263500 2014-06-09 16:22:11.955942 7f868ff62700 10 op_submit_start 3692 2014-06-09 16:22:11.955943 7f868ff62700 5 queue_transactions (trailing journal) 3692 2014-06-09 16:22:11.955946 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.955944 7f868ff62700 5 queue_op 0x30d96420 seq 3692 osr(3.300 0x3ed0df0) 861 bytes (queue has 48 ops and 583721 bytes) 2014-06-09 16:22:11.955953 7f868ff62700 10 op_submit_finish 3692 2014-06-09 16:22:11.955956 7f868a757700 10 op_submit_start 3693 2014-06-09 16:22:11.955959 7f868a757700 5 queue_transactions (trailing journal) 3693 2014-06-09 16:22:11.955960 7f868a757700 5 queue_op 0xc88a840 seq 3693 osr(0.35b 0x2bd6fe0) 828 bytes (queue has 48 ops and 583721 bytes) 2014-06-09 16:22:11.955963 7f868a757700 10 op_submit_finish 3693 2014-06-09 16:22:11.955958 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.955967 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) 2014-06-09 16:22:11.955970 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.955977 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.955982 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.955985 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.955996 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3647.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.955993 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.956000 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.955979 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.5 activated and committed 2014-06-09 16:22:11.956001 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.956011 7f86a5e11700 10 _do_op 0x324f47e0 seq 3647 r = 0, finisher 0x17fb7400 0 2014-06-09 16:22:11.956008 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.958663 7f86a5e11700 10 _finish_op 0x324f47e0 seq 3647 osr(3.7e2 0xb7f0120)/0xb7f0120 2014-06-09 16:22:11.958664 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.958664 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.958674 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.958680 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.958682 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.958691 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.958694 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:128 2014-06-09 16:22:11.958696 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.958707 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.958707 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.958711 7f868a757700 20 osd.26 243 Sending info 3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) to shard 5 2014-06-09 16:22:11.958717 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.958714 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.958722 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.958729 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.958736 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.956015 7f868df5e700 10 op_submit_start 3694 2014-06-09 16:22:11.958749 7f868df5e700 5 queue_transactions (trailing journal) 3694 2014-06-09 16:22:11.958749 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0310 (19 active) 2014-06-09 16:22:11.958751 7f868df5e700 5 queue_op 0x324f4c00 seq 3694 osr(2.606 0x2ed1d30) 828 bytes (queue has 50 ops and 585375 bytes) 2014-06-09 16:22:11.956022 7f868cf5c700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.956033 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.958758 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.958760 7f8692f68700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.958771 7f868ef60700 2 waiting 51 > 50 ops || 587101 > 104857600 2014-06-09 16:22:11.958768 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.958779 7f8691f66700 2 waiting 51 > 50 ops || 586428 > 104857600 2014-06-09 16:22:11.958765 7f868a757700 10 osd.26 243 dequeue_op 0x19fe1500 prio 63 cost 16384 latency 1.598090 osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.958778 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.958784 7f8689755700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.958785 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.958787 7f868a757700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:11.958782, event: reached_pg, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.958793 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.958776 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.958795 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.958798 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.958808 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x19fe1500 2014-06-09 16:22:11.958809 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.873195 7 0.000412 2014-06-09 16:22:11.958804 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] op_applied: 7 2014-06-09 16:22:11.958819 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.958824 7f86a4e0f700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.958819, event: op_applied, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.958817 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.958828 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.958829 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] op_applied on primary on version 243'594 2014-06-09 16:22:11.958838 7f868a757700 15 getattr 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 '_' 2014-06-09 16:22:11.958836 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] repop_all_applied: repop tid 7 all applied 2014-06-09 16:22:11.958837 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:11.958847 7f868ff62700 5 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.958841 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] eval_repop repgather(0xc049740 243'594 rep_tid=7 committed?=1 applied?=1 lock=0 op=osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.958855 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.958861 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.958862 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] log_op_stats osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.503920 2014-06-09 16:22:11.958872 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.958878 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] publish_stats_to_osd 243:969 2014-06-09 16:22:11.958881 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.958885 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] requeue_ops 2014-06-09 16:22:11.958890 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.958890 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] removing repgather(0xc049740 243'594 rep_tid=7 committed?=1 applied?=1 op=osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.958902 7f868ff62700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.958899 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] q front is repgather(0xc049740 243'594 rep_tid=7 committed?=1 applied?=1 op=osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.958907 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] remove_repop repgather(0xc049740 243'594 rep_tid=7 committed?=1 applied?=1 op=osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.958914 7f868a757700 10 getattr got.size() is 0 2014-06-09 16:22:11.958916 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] requeue_ops 2014-06-09 16:22:11.958917 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 but can_create 2014-06-09 16:22:11.958924 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:11.958930 7f868a757700 15 getattr 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 'snapset' 2014-06-09 16:22:11.958928 7f86a4e0f700 5 -- op tracker -- , seq: 392, time: 2014-06-09 16:22:11.958925, event: done, op: osd_op(client.4571.0:35948 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 352256~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.958955 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.958959 7f868ff62700 5 queue_transactions existing osr(2.e4 0x2ed0410)/0x2ed0410 2014-06-09 16:22:11.958961 7f868ff62700 2 waiting 51 > 50 ops || 586233 > 104857600 2014-06-09 16:22:11.958962 7f868a757700 10 getattr got.size() is 0 2014-06-09 16:22:11.958965 7f868a757700 15 getattr 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/snapdir//3 'snapset' 2014-06-09 16:22:11.958965 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.958976 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.958987 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.958997 7f868a757700 10 getattr got.size() is 0 2014-06-09 16:22:11.959005 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959006 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30df6840 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:11.956030 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.25b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.959020 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.959020 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:11.959030 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.959030 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:11.959042 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] op_applied: 6 2014-06-09 16:22:11.959053 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.959040 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30df6840 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 rwstate(none n=0 w=0) oi: 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc27e0 snapset: 0=[]:[] 2014-06-09 16:22:11.959053 7f86a4e0f700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.959049, event: op_applied, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.959057 7f868b759700 5 queue_transactions existing osr(0.25b 0x2bd69f0)/0x2bd69f0 2014-06-09 16:22:11.959061 7f868b759700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.959057 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] op_applied on primary on version 243'641 2014-06-09 16:22:11.959056 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 @head oi=22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.959063 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] repop_all_applied: repop tid 6 all applied 2014-06-09 16:22:11.959073 7f868a757700 15 getattr 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/snapdir//3 '_' 2014-06-09 16:22:11.959068 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] eval_repop repgather(0xc1f0540 243'641 rep_tid=6 committed?=1 applied?=1 lock=0 op=osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.959078 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] log_op_stats osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1548565 outb 0 rlat 0.000000 lat 1.195175 2014-06-09 16:22:11.959087 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] publish_stats_to_osd 243:1029 2014-06-09 16:22:11.959093 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] requeue_ops 2014-06-09 16:22:11.959098 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] removing repgather(0xc1f0540 243'641 rep_tid=6 committed?=1 applied?=1 op=osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.959106 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] q front is repgather(0xc1f0540 243'641 rep_tid=6 committed?=1 applied?=1 op=osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.959108 7f868a757700 10 getattr got.size() is 0 2014-06-09 16:22:11.959114 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] remove_repop repgather(0xc1f0540 243'641 rep_tid=6 committed?=1 applied?=1 op=osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.959122 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] requeue_ops 2014-06-09 16:22:11.959118 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 22b8b78a/rbd_data.11756b8b4567.0000000000001593/snapdir//3 and !can_create 2014-06-09 16:22:11.959133 7f86a4e0f700 5 -- op tracker -- , seq: 699, time: 2014-06-09 16:22:11.959130, event: done, op: osd_op(client.4570.0:34071 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1478656~1548288] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.956043 7f868e75f700 5 queue_transactions existing osr(2.620 0x2ed1cb0)/0x2ed1cb0 2014-06-09 16:22:11.959142 7f868e75f700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.959155 7f868c75b700 2 waiting 51 > 50 ops || 604157 > 104857600 2014-06-09 16:22:11.959157 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 213 same_interval_since 242 2014-06-09 16:22:11.959168 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.959176 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.959182 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.959185 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.959187 7f869af78700 20 osd.26 243 _dispatch 0x30c77880 pg_info(1 pgs e243:0.725) v4 2014-06-09 16:22:11.959189 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959191 7f8691765700 2 waiting 51 > 50 ops || 586236 > 104857600 2014-06-09 16:22:11.959197 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.959194 7f869af78700 5 -- op tracker -- , seq: 3361, time: 2014-06-09 16:22:11.953535, event: header_read, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.959203 7f869af78700 5 -- op tracker -- , seq: 3361, time: 2014-06-09 16:22:11.953535, event: throttled, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.959207 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.959217 7f869af78700 5 -- op tracker -- , seq: 3361, time: 2014-06-09 16:22:11.953544, event: all_read, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.959220 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959223 7f869af78700 5 -- op tracker -- , seq: 3361, time: 2014-06-09 16:22:11.959181, event: dispatched, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.959228 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.959231 7f868d75d700 2 waiting 51 > 50 ops || 1619955 > 104857600 2014-06-09 16:22:11.959231 7f869af78700 5 -- op tracker -- , seq: 3361, time: 2014-06-09 16:22:11.959229, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.959236 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.725) v4 from osd.7 2014-06-09 16:22:11.959239 7f8689f56700 2 waiting 51 > 50 ops || 586201 > 104857600 2014-06-09 16:22:11.959240 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c77880 2014-06-09 16:22:11.959236 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.959247 7f8690f64700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.956141 7f86a5610700 10 _do_op 0xc64a6c0 seq 3640 r = 0, finisher 0x324f8260 0x190b9740 2014-06-09 16:22:11.959247 7f869af78700 5 -- op tracker -- , seq: 3361, time: 2014-06-09 16:22:11.959245, event: started, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.959255 7f868af58700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.959252 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959263 7f8690763700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.959265 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.959268 7f868a757700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:11.959132, event: started, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.959277 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.959280 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30e0f000 2014-06-09 16:22:11.959290 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] op_applied: 8 2014-06-09 16:22:11.959299 7f8692767700 2 waiting 51 > 50 ops || 586203 > 104857600 2014-06-09 16:22:11.959299 7f868a757700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:11.959293, event: started, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.959304 7f868f761700 2 waiting 51 > 50 ops || 694259 > 104857600 2014-06-09 16:22:11.959315 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286c60 (1 active) 2014-06-09 16:22:11.959319 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0a20 (2 active) 2014-06-09 16:22:11.959309 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] ov 0'0 av 243'546 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.959324 7f86a5e11700 5 _do_op 0x288f8480 seq 3648 osr(0.16c 0x2bd67f0)/0x2bd67f0 start 2014-06-09 16:22:11.959328 7f86a5e11700 10 _do_transaction on 0x3418600 2014-06-09 16:22:11.959324 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 2014-06-09 16:22:11.959334 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.959336 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.959350 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~16384 2014-06-09 16:22:11.959351 7f86827e2700 5 -- op tracker -- , seq: 3362, time: 2014-06-09 16:22:11.957343, event: header_read, op: osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.959371 7f86a4e0f700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.959296, event: op_applied, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.959367 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229000 2014-06-09 16:22:11.959379 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 snapset=0x14dc2818 snapc=0=[] 2014-06-09 16:22:11.959380 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] op_applied on primary on version 243'595 2014-06-09 16:22:11.959388 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.959391 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.959392 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] repop_all_applied: repop tid 8 all applied 2014-06-09 16:22:11.959404 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3648.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.959408 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.959402 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.959401 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] eval_repop repgather(0xc128a80 243'595 rep_tid=8 committed?=1 applied?=1 lock=0 op=osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.959418 7f86a5e11700 10 _do_op 0x288f8480 seq 3648 r = 0, finisher 0x31791f40 0 2014-06-09 16:22:11.959421 7f86a5e11700 10 _finish_op 0x288f8480 seq 3648 osr(0.16c 0x2bd67f0)/0x2bd67f0 2014-06-09 16:22:11.959413 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 0x30e0f000 op modify 2014-06-09 16:22:11.959424 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0a20 (1 active) 2014-06-09 16:22:11.959426 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301c20 (2 active) 2014-06-09 16:22:11.959428 7f86a5e11700 5 _do_op 0xc2d4660 seq 3649 osr(3.6d5 0x3ed1c30)/0x3ed1c30 start 2014-06-09 16:22:11.959430 7f86a5e11700 10 _do_transaction on 0x3415200 2014-06-09 16:22:11.959418 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] log_op_stats osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.478849 2014-06-09 16:22:11.959435 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.959425 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:18:50.860316 2014-06-09 16:22:11.959432 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] publish_stats_to_osd 243:970 2014-06-09 16:22:11.959442 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] requeue_ops 2014-06-09 16:22:11.959450 7f86827e2700 5 -- op tracker -- , seq: 3362, time: 2014-06-09 16:22:11.957344, event: throttled, op: osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.959447 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:11.959451 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] removing repgather(0xc128a80 243'595 rep_tid=8 committed?=1 applied?=1 op=osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.959460 7f86827e2700 5 -- op tracker -- , seq: 3362, time: 2014-06-09 16:22:11.957351, event: all_read, op: osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.959469 7f86827e2700 5 -- op tracker -- , seq: 3362, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.959471 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.959465 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.959474 7f86827e2700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.959465 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] q front is repgather(0xc128a80 243'595 rep_tid=8 committed?=1 applied?=1 op=osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.959478 7f86827e2700 20 osd.26 243 should_share_map osd.3 172.16.96.201:6823/50569 243 2014-06-09 16:22:11.959484 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3649.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.959476 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 21 on osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.959488 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.959485 7f86827e2700 15 osd.26 243 enqueue_op 0x257c4000 prio 196 cost 0 latency 0.002142 osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.959479 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] remove_repop repgather(0xc128a80 243'595 rep_tid=8 committed?=1 applied?=1 op=osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.959490 7f868a757700 7 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 21 o 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:11.959494 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] requeue_ops 2014-06-09 16:22:11.959503 7f868df5e700 10 op_submit_finish 3694 2014-06-09 16:22:11.959507 7f86a5e11700 10 _do_op 0xc2d4660 seq 3649 r = 0, finisher 0xbd798a0 0 2014-06-09 16:22:11.959510 7f86a5e11700 10 _finish_op 0xc2d4660 seq 3649 osr(3.6d5 0x3ed1c30)/0x3ed1c30 2014-06-09 16:22:11.959512 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301c20 (1 active) 2014-06-09 16:22:11.959514 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358360 (2 active) 2014-06-09 16:22:11.959511 7f86a4e0f700 5 -- op tracker -- , seq: 403, time: 2014-06-09 16:22:11.959506, event: done, op: osd_op(client.4571.0:35949 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 868352~516096] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.959516 7f86a5e11700 5 _do_op 0x324f4660 seq 3650 osr(3.31f 0x3ed0d50)/0x3ed0d50 start 2014-06-09 16:22:11.959519 7f86a5e11700 10 _do_transaction on 0xc280540 2014-06-09 16:22:11.959510 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.560( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.959524 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.959527 7f868a757700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:11.959507, event: waiting for subops from 1, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.959543 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959545 7f868a757700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:11.959548 7f868df5e700 5 queue_transactions existing osr(1.560 0x384da90)/0x384da90 2014-06-09 16:22:11.959550 7f868df5e700 10 op_submit_start 3695 2014-06-09 16:22:11.959551 7f868df5e700 5 queue_transactions (trailing journal) 3695 2014-06-09 16:22:11.959552 7f868df5e700 5 queue_op 0x324f4cc0 seq 3695 osr(1.560 0x384da90) 828 bytes (queue has 48 ops and 583719 bytes) 2014-06-09 16:22:11.959555 7f868df5e700 10 op_submit_finish 3695 2014-06-09 16:22:11.959560 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.959555 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959558 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.27a( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.959571 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.959573 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3650.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.959577 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.959587 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959586 7f86a5e11700 10 _do_op 0x324f4660 seq 3650 r = 0, finisher 0x17fb7a80 0 2014-06-09 16:22:11.959589 7f86a5e11700 10 _finish_op 0x324f4660 seq 3650 osr(3.31f 0x3ed0d50)/0x3ed0d50 2014-06-09 16:22:11.959591 7f868df5e700 5 queue_transactions existing osr(2.27a 0x2ed0d90)/0x2ed0d90 2014-06-09 16:22:11.959592 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358360 (1 active) 2014-06-09 16:22:11.959586 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.959594 7f868df5e700 10 op_submit_start 3696 2014-06-09 16:22:11.959593 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524c60 (2 active) 2014-06-09 16:22:11.959595 7f868df5e700 5 queue_transactions (trailing journal) 3696 2014-06-09 16:22:11.959595 7f86a5e11700 5 _do_op 0x11fcfaa0 seq 3651 osr(1.589 0x384da10)/0x384da10 start 2014-06-09 16:22:11.959596 7f868df5e700 5 queue_op 0x324f4d80 seq 3696 osr(2.27a 0x2ed0d90) 828 bytes (queue has 48 ops and 583719 bytes) 2014-06-09 16:22:11.959598 7f86a5e11700 10 _do_transaction on 0xbe8b200 2014-06-09 16:22:11.959599 7f868df5e700 10 op_submit_finish 3696 2014-06-09 16:22:11.959602 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.959601 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959601 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:11.959619 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.959618 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.959628 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.959636 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.959643 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.959648 7f868df5e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.959640 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.959656 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3651.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.959660 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.959660 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959662 7f868df5e700 15 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:135 2014-06-09 16:22:11.959670 7f86a5e11700 10 _do_op 0x11fcfaa0 seq 3651 r = 0, finisher 0x3062bc20 0 2014-06-09 16:22:11.959673 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.959673 7f86a5e11700 10 _finish_op 0x11fcfaa0 seq 3651 osr(1.589 0x384da10)/0x384da10 2014-06-09 16:22:11.959675 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524c60 (1 active) 2014-06-09 16:22:11.959677 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959677 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9237a0 (2 active) 2014-06-09 16:22:11.959680 7f86a5e11700 5 _do_op 0x24105f80 seq 3652 osr(0.1a4 0x2bd6710)/0x2bd6710 start 2014-06-09 16:22:11.959682 7f86a5e11700 10 _do_transaction on 0xc2ef080 2014-06-09 16:22:11.959687 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.959681 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.382( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.382( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.959686 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.959706 7f868bf5a700 10 op_submit_start 3697 2014-06-09 16:22:11.959709 7f868bf5a700 5 queue_transactions (trailing journal) 3697 2014-06-09 16:22:11.959700 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959711 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959710 7f868bf5a700 5 queue_op 0x308cb320 seq 3697 osr(2.6f 0x2ed0550) 826 bytes (queue has 47 ops and 582891 bytes) 2014-06-09 16:22:11.959713 7f868bf5a700 10 op_submit_finish 3697 2014-06-09 16:22:11.959715 7f868df5e700 5 queue_transactions existing osr(1.382 0x384d5b0)/0x384d5b0 2014-06-09 16:22:11.959715 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.959718 7f868df5e700 10 op_submit_start 3698 2014-06-09 16:22:11.959719 7f868df5e700 5 queue_transactions (trailing journal) 3698 2014-06-09 16:22:11.959720 7f868df5e700 5 queue_op 0x324f4f00 seq 3698 osr(1.382 0x384d5b0) 828 bytes (queue has 48 ops and 583719 bytes) 2014-06-09 16:22:11.959722 7f868df5e700 10 op_submit_finish 3698 2014-06-09 16:22:11.959721 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.959717 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.959724 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.54c( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:11.959734 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3652.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.959738 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.959730 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.959732 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.959740 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.959746 7f86a5e11700 10 _do_op 0x24105f80 seq 3652 r = 0, finisher 0xbd5efc0 0 2014-06-09 16:22:11.959749 7f86a5e11700 10 _finish_op 0x24105f80 seq 3652 osr(0.1a4 0x2bd6710)/0x2bd6710 2014-06-09 16:22:11.959751 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959751 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9237a0 (1 active) 2014-06-09 16:22:11.959753 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc900 (2 active) 2014-06-09 16:22:11.959747 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.959748 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.959755 7f86a5e11700 5 _do_op 0x15996d80 seq 3653 osr(0.30b 0x2bd6dd0)/0x2bd6dd0 start 2014-06-09 16:22:11.959755 7f868df5e700 5 queue_transactions existing osr(0.54c 0x2bd7850)/0x2bd7850 2014-06-09 16:22:11.959757 7f86a5e11700 10 _do_transaction on 0x36b2900 2014-06-09 16:22:11.959757 7f868df5e700 10 op_submit_start 3699 2014-06-09 16:22:11.959758 7f868df5e700 5 queue_transactions (trailing journal) 3699 2014-06-09 16:22:11.959759 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.959759 7f868df5e700 5 queue_op 0x324f4fc0 seq 3699 osr(0.54c 0x2bd7850) 828 bytes (queue has 48 ops and 583719 bytes) 2014-06-09 16:22:11.959762 7f868df5e700 10 op_submit_finish 3699 2014-06-09 16:22:11.959756 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.959762 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.959764 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.959764 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.959770 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.959777 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.959783 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.959786 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.959786 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] op_applied: 9 2014-06-09 16:22:11.959792 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.959792 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.959797 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.959799 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.959808 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.959812 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3653.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.959817 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.959818 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.959829 7f86a5e11700 10 _do_op 0x15996d80 seq 3653 r = 0, finisher 0xcd0e3c0 0 2014-06-09 16:22:11.959832 7f86a5e11700 10 _finish_op 0x15996d80 seq 3653 osr(0.30b 0x2bd6dd0)/0x2bd6dd0 2014-06-09 16:22:11.959835 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc900 (1 active) 2014-06-09 16:22:11.959837 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1440 (2 active) 2014-06-09 16:22:11.959839 7f86a5e11700 5 _do_op 0x30606780 seq 3654 osr(2.17c 0x2ed09b0)/0x2ed09b0 start 2014-06-09 16:22:11.959841 7f86a5e11700 10 _do_transaction on 0xc2ae6c0 2014-06-09 16:22:11.959833 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:179 2014-06-09 16:22:11.959847 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.959848 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959848 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.959852 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.959860 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.959864 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.959868 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.959872 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.959875 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.959878 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.959884 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.959889 7f868df5e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.959883 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.959901 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3654.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.959899 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.959906 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.959904 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.959912 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.959923 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959919 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.959929 7f86a5e11700 10 _do_op 0x30606780 seq 3654 r = 0, finisher 0x2d7381e0 0 2014-06-09 16:22:11.959926 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.959934 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.959926 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.25f( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.959934 7f86a5e11700 10 _finish_op 0x30606780 seq 3654 osr(2.17c 0x2ed09b0)/0x2ed09b0 2014-06-09 16:22:11.959937 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1440 (1 active) 2014-06-09 16:22:11.959939 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383fc20 (2 active) 2014-06-09 16:22:11.959941 7f86a5e11700 5 _do_op 0x14dd6f00 seq 3655 osr(2.18 0x2ed0290)/0x2ed0290 start 2014-06-09 16:22:11.959944 7f86a5e11700 10 _do_transaction on 0xc3f0fc0 2014-06-09 16:22:11.959940 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.959950 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.959955 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959951 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.959958 7f868df5e700 5 queue_transactions existing osr(0.25f 0x2bd6d00)/0x2bd6d00 2014-06-09 16:22:11.959960 7f868df5e700 10 op_submit_start 3700 2014-06-09 16:22:11.959961 7f868df5e700 5 queue_transactions (trailing journal) 3700 2014-06-09 16:22:11.959962 7f868df5e700 5 queue_op 0x324f5200 seq 3700 osr(0.25f 0x2bd6d00) 828 bytes (queue has 47 ops and 582825 bytes) 2014-06-09 16:22:11.959958 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.959964 7f868df5e700 10 op_submit_finish 3700 2014-06-09 16:22:11.959965 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.959967 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 243/201 242/242/200) 2014-06-09 16:22:11.959972 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.959979 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.959986 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.959994 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.959990 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.959997 7f868df5e700 5 queue_transactions existing osr(3.359 0x3ed10f0)/0x3ed10f0 2014-06-09 16:22:11.959999 7f868df5e700 10 op_submit_start 3701 2014-06-09 16:22:11.960000 7f868df5e700 5 queue_transactions (trailing journal) 3701 2014-06-09 16:22:11.960001 7f868df5e700 5 queue_op 0x324f52c0 seq 3701 osr(3.359 0x3ed10f0) 828 bytes (queue has 48 ops and 583653 bytes) 2014-06-09 16:22:11.960003 7f868df5e700 10 op_submit_finish 3701 2014-06-09 16:22:11.959998 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.960002 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3655.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960007 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960005 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.960013 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.960005 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.960018 7f86a5e11700 10 _do_op 0x14dd6f00 seq 3655 r = 0, finisher 0xbd789c0 0 2014-06-09 16:22:11.960022 7f86a5e11700 10 _finish_op 0x14dd6f00 seq 3655 osr(2.18 0x2ed0290)/0x2ed0290 2014-06-09 16:22:11.960016 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.960024 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383fc20 (1 active) 2014-06-09 16:22:11.960026 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7560 (2 active) 2014-06-09 16:22:11.960028 7f86a5e11700 5 _do_op 0xc2d47e0 seq 3656 osr(2.487 0x2ed14b0)/0x2ed14b0 start 2014-06-09 16:22:11.960030 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960031 7f86a5e11700 10 _do_transaction on 0x390d140 2014-06-09 16:22:11.960027 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.960033 7f868df5e700 5 queue_transactions existing osr(2.d3 0x2ed0450)/0x2ed0450 2014-06-09 16:22:11.960035 7f868df5e700 10 op_submit_start 3702 2014-06-09 16:22:11.960036 7f868df5e700 5 queue_transactions (trailing journal) 3702 2014-06-09 16:22:11.960036 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960037 7f868df5e700 5 queue_op 0x324f5380 seq 3702 osr(2.d3 0x2ed0450) 826 bytes (queue has 48 ops and 583621 bytes) 2014-06-09 16:22:11.960034 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.960040 7f868df5e700 10 op_submit_finish 3702 2014-06-09 16:22:11.960041 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.960042 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:11.960049 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.960055 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.960054 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:11.960062 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.960066 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.960071 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960070 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.960074 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.960082 7f868df5e700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.960080 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.960085 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3656.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960089 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960095 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.960095 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1007 2014-06-09 16:22:11.960100 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.2d0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.960110 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960101 7f86a5e11700 10 _do_op 0xc2d47e0 seq 3656 r = 0, finisher 0xc2d6e20 0 2014-06-09 16:22:11.960115 7f86a5e11700 10 _finish_op 0xc2d47e0 seq 3656 osr(2.487 0x2ed14b0)/0x2ed14b0 2014-06-09 16:22:11.960120 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7560 (1 active) 2014-06-09 16:22:11.960123 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525440 (2 active) 2014-06-09 16:22:11.960125 7f86a5e11700 5 _do_op 0xbe7dd40 seq 3657 osr(1.522 0x384db30)/0x384db30 start 2014-06-09 16:22:11.960126 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8fc00 (19 active) 2014-06-09 16:22:11.960127 7f86a5e11700 10 _do_transaction on 0xc0dad80 2014-06-09 16:22:11.960129 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.960133 7f868bf5a700 5 queue_transactions existing osr(1.2d0 0x384d180)/0x384d180 2014-06-09 16:22:11.960133 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960135 7f868bf5a700 10 op_submit_start 3703 2014-06-09 16:22:11.960136 7f868bf5a700 5 queue_transactions (trailing journal) 3703 2014-06-09 16:22:11.960137 7f868bf5a700 5 queue_op 0xc704900 seq 3703 osr(1.2d0 0x384d180) 828 bytes (queue has 48 ops and 583621 bytes) 2014-06-09 16:22:11.960144 7f868bf5a700 10 op_submit_finish 3703 2014-06-09 16:22:11.960152 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8fc00 (20 active) 2014-06-09 16:22:11.960147 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.960159 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.960156 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.e0( v 213'736 (0'0,213'736] local-les=243 n=138 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:11.960165 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.960169 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960173 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.960184 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960180 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.960188 7f868df5e700 5 queue_transactions existing osr(3.e0 0x3ed0500)/0x3ed0500 2014-06-09 16:22:11.960190 7f868df5e700 10 op_submit_start 3704 2014-06-09 16:22:11.960191 7f868df5e700 5 queue_transactions (trailing journal) 3704 2014-06-09 16:22:11.960193 7f868df5e700 5 queue_op 0x2fc8df20 seq 3704 osr(3.e0 0x3ed0500) 826 bytes (queue has 49 ops and 584447 bytes) 2014-06-09 16:22:11.960195 7f868df5e700 10 op_submit_finish 3704 2014-06-09 16:22:11.960192 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3657.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960197 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960188 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.960198 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.91( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.91( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.960209 7f86a5e11700 10 _do_op 0xbe7dd40 seq 3657 r = 0, finisher 0x31456a60 0 2014-06-09 16:22:11.960205 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.960213 7f86a5e11700 10 _finish_op 0xbe7dd40 seq 3657 osr(1.522 0x384db30)/0x384db30 2014-06-09 16:22:11.960215 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525440 (1 active) 2014-06-09 16:22:11.960217 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d0e0 (2 active) 2014-06-09 16:22:11.960214 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.960219 7f86a5e11700 5 _do_op 0x30d96cc0 seq 3658 osr(0.325 0x2bd6d30)/0x2bd6d30 start 2014-06-09 16:22:11.960221 7f86a5e11700 10 _do_transaction on 0xc28b200 2014-06-09 16:22:11.960225 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960221 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.960227 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960229 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.960229 7f868df5e700 5 queue_transactions existing osr(1.91 0x384c430)/0x384c430 2014-06-09 16:22:11.960231 7f868df5e700 10 op_submit_start 3705 2014-06-09 16:22:11.960232 7f868df5e700 5 queue_transactions (trailing journal) 3705 2014-06-09 16:22:11.960233 7f868df5e700 5 queue_op 0x30d97320 seq 3705 osr(1.91 0x384c430) 826 bytes (queue has 49 ops and 584412 bytes) 2014-06-09 16:22:11.960235 7f868df5e700 10 op_submit_finish 3705 2014-06-09 16:22:11.960237 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.c9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.c9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.960233 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.960249 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.960260 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960256 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.960264 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960267 7f868df5e700 5 queue_transactions existing osr(2.c9 0x2ed04a0)/0x2ed04a0 2014-06-09 16:22:11.960269 7f868df5e700 10 op_submit_start 3706 2014-06-09 16:22:11.960270 7f868df5e700 5 queue_transactions (trailing journal) 3706 2014-06-09 16:22:11.960264 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.960271 7f868df5e700 5 queue_op 0x16fd56e0 seq 3706 osr(2.c9 0x2ed04a0) 826 bytes (queue has 50 ops and 585238 bytes) 2014-06-09 16:22:11.960273 7f868df5e700 10 op_submit_finish 3706 2014-06-09 16:22:11.960271 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.960276 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3658.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960280 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960275 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.45d( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.45d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.960278 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.960298 7f86a5e11700 10 _do_op 0x30d96cc0 seq 3658 r = 0, finisher 0x1e08efa0 0 2014-06-09 16:22:11.960302 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960290 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.960302 7f86a5e11700 10 _finish_op 0x30d96cc0 seq 3658 osr(0.325 0x2bd6d30)/0x2bd6d30 2014-06-09 16:22:11.960305 7f868df5e700 5 queue_transactions existing osr(1.45d 0x384d670)/0x384d670 2014-06-09 16:22:11.960305 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d0e0 (1 active) 2014-06-09 16:22:11.960308 7f868df5e700 10 op_submit_start 3707 2014-06-09 16:22:11.960309 7f868df5e700 5 queue_transactions (trailing journal) 3707 2014-06-09 16:22:11.960308 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301c20 (2 active) 2014-06-09 16:22:11.960304 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.960310 7f868df5e700 5 queue_op 0x16fd4660 seq 3707 osr(1.45d 0x384d670) 828 bytes (queue has 50 ops and 585205 bytes) 2014-06-09 16:22:11.960310 7f86a5e11700 5 _do_op 0xc2d4f00 seq 3659 osr(3.6d5 0x3ed1c30)/0x3ed1c30 start 2014-06-09 16:22:11.960312 7f868df5e700 10 op_submit_finish 3707 2014-06-09 16:22:11.960313 7f86a5e11700 10 _do_transaction on 0xbf88480 2014-06-09 16:22:11.960312 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.960318 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960320 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.960314 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.606( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.606( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:11.960323 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.960340 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960335 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.960343 7f868df5e700 5 queue_transactions existing osr(2.606 0x2ed1d30)/0x2ed1d30 2014-06-09 16:22:11.960345 7f868df5e700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960354 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960343 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.960357 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.960368 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3659.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960369 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.960372 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960383 7f86a5e11700 10 _do_op 0xc2d4f00 seq 3659 r = 0, finisher 0x3042b280 0 2014-06-09 16:22:11.960386 7f86a5e11700 10 _finish_op 0xc2d4f00 seq 3659 osr(3.6d5 0x3ed1c30)/0x3ed1c30 2014-06-09 16:22:11.960391 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301c20 (1 active) 2014-06-09 16:22:11.960386 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.960393 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f7a0 (2 active) 2014-06-09 16:22:11.960404 7f86a5e11700 5 _do_op 0x324f4000 seq 3660 osr(2.501 0x2ed1690)/0x2ed1690 start 2014-06-09 16:22:11.960406 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.960407 7f86a5e11700 10 _do_transaction on 0x2c81c80 2014-06-09 16:22:11.960412 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960410 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.960422 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.960431 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.960439 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.960450 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960447 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.960454 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.960464 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3660.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960468 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960467 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:11.960478 7f86a5e11700 10 _do_op 0x324f4000 seq 3660 r = 0, finisher 0xbd605a0 0 2014-06-09 16:22:11.960482 7f86a5e11700 10 _finish_op 0x324f4000 seq 3660 osr(2.501 0x2ed1690)/0x2ed1690 2014-06-09 16:22:11.960476 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.960484 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f7a0 (1 active) 2014-06-09 16:22:11.960487 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0a20 (2 active) 2014-06-09 16:22:11.960489 7f86a5e11700 5 _do_op 0x324f44e0 seq 3661 osr(0.681 0x2bd7e10)/0x2bd7e10 start 2014-06-09 16:22:11.960485 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.960491 7f86a5e11700 10 _do_transaction on 0xc2718c0 2014-06-09 16:22:11.960494 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.960498 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960498 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:11.960510 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: got info+log from osd.4 1.76e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.960522 7f868bf5a700 10 merge_log log((0'0,0'0], crt=0'0) from osd.4 into log((0'0,0'0], crt=0'0) 2014-06-09 16:22:11.960526 7f868bf5a700 10 merge_log result log((0'0,0'0], crt=0'0) missing(0) changed=0 2014-06-09 16:22:11.960533 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960531 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] exit Started/Stray 1.095223 7 0.000166 2014-06-09 16:22:11.960546 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3661.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960543 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:11.960550 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960550 7f868df5e700 10 op_submit_start 3708 2014-06-09 16:22:11.960555 7f868df5e700 5 queue_transactions (trailing journal) 3708 2014-06-09 16:22:11.960556 7f868df5e700 5 queue_op 0xc2ead80 seq 3708 osr(2.606 0x2ed1d30) 828 bytes (queue has 49 ops and 584377 bytes) 2014-06-09 16:22:11.960558 7f868df5e700 10 op_submit_finish 3708 2014-06-09 16:22:11.960555 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:11.960560 7f86a5e11700 10 _do_op 0x324f44e0 seq 3661 r = 0, finisher 0x17fb7900 0 2014-06-09 16:22:11.960564 7f86a5e11700 10 _finish_op 0x324f44e0 seq 3661 osr(0.681 0x2bd7e10)/0x2bd7e10 2014-06-09 16:22:11.960566 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0a20 (1 active) 2014-06-09 16:22:11.960561 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.25f( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.25f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.960569 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359560 (2 active) 2014-06-09 16:22:11.960564 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=203 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:11.960572 7f86a5e11700 5 _do_op 0x3280bbc0 seq 3662 osr(1.38d 0x384d570)/0x384d570 start 2014-06-09 16:22:11.960575 7f86a5e11700 10 _do_transaction on 0x2e62900 2014-06-09 16:22:11.960573 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] activate - no missing, moving last_complete 0'0 -> 0'0 2014-06-09 16:22:11.960580 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960581 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.960588 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960592 7f868df5e700 5 queue_transactions existing osr(0.25f 0x2bd6d00)/0x2bd6d00 2014-06-09 16:22:11.960594 7f868df5e700 10 op_submit_start 3709 2014-06-09 16:22:11.960595 7f868df5e700 5 queue_transactions (trailing journal) 3709 2014-06-09 16:22:11.960596 7f868df5e700 5 queue_op 0x30061320 seq 3709 osr(0.25f 0x2bd6d00) 828 bytes (queue has 49 ops and 584377 bytes) 2014-06-09 16:22:11.960598 7f868df5e700 10 op_submit_finish 3709 2014-06-09 16:22:11.960600 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.560( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.560( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.960614 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960624 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960627 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.960627 7f868df5e700 5 queue_transactions existing osr(1.560 0x384da90)/0x384da90 2014-06-09 16:22:11.960629 7f868df5e700 10 op_submit_start 3710 2014-06-09 16:22:11.960630 7f868df5e700 5 queue_transactions (trailing journal) 3710 2014-06-09 16:22:11.960627 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3662.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960630 7f868bf5a700 5 queue_transactions existing osr(1.76e 0x2ed03b0)/0x2ed03b0 2014-06-09 16:22:11.960632 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960631 7f868df5e700 5 queue_op 0x33a2d20 seq 3710 osr(1.560 0x384da90) 828 bytes (queue has 50 ops and 585205 bytes) 2014-06-09 16:22:11.960632 7f868bf5a700 2 waiting 51 > 50 ops || 587206 > 104857600 2014-06-09 16:22:11.960634 7f868df5e700 10 op_submit_finish 3710 2014-06-09 16:22:11.959562 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 213'545 (0'0,213'545] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'545], crt=0'0) [243'546 (0'0) modify 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 by client.4565.0:32907 2014-06-09 10:18:50.860316] 2014-06-09 16:22:11.960636 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.359( v 213'669 (0'0,213'669] local-les=243 n=126 ec=118 les/c 243/243 242/242/200) 2014-06-09 16:22:11.960647 7f86a5e11700 10 _do_op 0x3280bbc0 seq 3662 r = 0, finisher 0xd9fb860 0 2014-06-09 16:22:11.960651 7f86a5e11700 10 _finish_op 0x3280bbc0 seq 3662 osr(1.38d 0x384d570)/0x384d570 2014-06-09 16:22:11.960657 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359560 (1 active) 2014-06-09 16:22:11.960659 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301200 (2 active) 2014-06-09 16:22:11.960662 7f86a5e11700 5 _do_op 0x3280a7e0 seq 3663 osr(2.4e0 0x2ed16f0)/0x2ed16f0 start 2014-06-09 16:22:11.960663 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960649 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 luod=213'545 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'546 (0'0) modify 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 by client.4565.0:32907 2014-06-09 10:18:50.860316 2014-06-09 16:22:11.960664 7f86a5e11700 10 _do_transaction on 0x344dc80 2014-06-09 16:22:11.960667 7f868df5e700 5 queue_transactions existing osr(3.359 0x3ed10f0)/0x3ed10f0 2014-06-09 16:22:11.960669 7f868df5e700 10 op_submit_start 3711 2014-06-09 16:22:11.960670 7f868df5e700 5 queue_transactions (trailing journal) 3711 2014-06-09 16:22:11.960670 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960671 7f868df5e700 5 queue_op 0x241048a0 seq 3711 osr(3.359 0x3ed10f0) 828 bytes (queue has 50 ops and 585205 bytes) 2014-06-09 16:22:11.960673 7f868df5e700 10 op_submit_finish 3711 2014-06-09 16:22:11.960675 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.27a( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.27a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.960669 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 luod=213'545 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.960701 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960703 7f868a757700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'546, trimmed: 2014-06-09 16:22:11.960705 7f868df5e700 5 queue_transactions existing osr(2.27a 0x2ed0d90)/0x2ed0d90 2014-06-09 16:22:11.960705 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960708 7f868df5e700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960713 7f868ef60700 2 waiting 51 > 50 ops || 586931 > 104857600 2014-06-09 16:22:11.960718 7f8691f66700 2 waiting 51 > 50 ops || 586258 > 104857600 2014-06-09 16:22:11.960719 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3663.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960722 7f8689755700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960722 7f868a757700 5 queue_transactions existing osr(3.78a 0x3ed1d80)/0x3ed1d80 2014-06-09 16:22:11.960723 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960725 7f868a757700 2 waiting 51 > 50 ops || 603987 > 104857600 2014-06-09 16:22:11.960730 7f868b759700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960733 7f86a5e11700 10 _do_op 0x3280a7e0 seq 3663 r = 0, finisher 0xd9fb6a0 0 2014-06-09 16:22:11.960735 7f868e75f700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960736 7f86a5e11700 10 _finish_op 0x3280a7e0 seq 3663 osr(2.4e0 0x2ed16f0)/0x2ed16f0 2014-06-09 16:22:11.959647 7f86af56b700 5 -- op tracker -- , seq: 3363, time: 2014-06-09 16:22:11.953471, event: header_read, op: osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.960745 7f868df5e700 10 op_submit_start 3712 2014-06-09 16:22:11.960745 7f8691765700 2 waiting 51 > 50 ops || 586066 > 104857600 2014-06-09 16:22:11.960744 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301200 (1 active) 2014-06-09 16:22:11.960746 7f868df5e700 5 queue_transactions (trailing journal) 3712 2014-06-09 16:22:11.960747 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114480 (2 active) 2014-06-09 16:22:11.960748 7f868df5e700 5 queue_op 0x15997ce0 seq 3712 osr(2.27a 0x2ed0d90) 828 bytes (queue has 50 ops and 585205 bytes) 2014-06-09 16:22:11.960750 7f8689f56700 2 waiting 51 > 50 ops || 586031 > 104857600 2014-06-09 16:22:11.960749 7f86a5e11700 5 _do_op 0x3280b740 seq 3664 osr(1.67a 0x384de90)/0x384de90 start 2014-06-09 16:22:11.960751 7f868df5e700 10 op_submit_finish 3712 2014-06-09 16:22:11.960752 7f86a5e11700 10 _do_transaction on 0xc3363c0 2014-06-09 16:22:11.960753 7f8690f64700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960757 7f868af58700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960757 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960761 7f8690763700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960753 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.d3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.960766 7f868d75d700 2 waiting 51 > 50 ops || 1619785 > 104857600 2014-06-09 16:22:11.960770 7f868f761700 2 waiting 51 > 50 ops || 694089 > 104857600 2014-06-09 16:22:11.960782 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.960780 7f86af56b700 5 -- op tracker -- , seq: 3363, time: 2014-06-09 16:22:11.953473, event: throttled, op: osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.960787 7f868df5e700 5 queue_transactions existing osr(2.d3 0x2ed0450)/0x2ed0450 2014-06-09 16:22:11.960789 7f868df5e700 2 waiting 51 > 50 ops || 586031 > 104857600 2014-06-09 16:22:11.960793 7f868ff62700 2 waiting 51 > 50 ops || 586063 > 104857600 2014-06-09 16:22:11.960793 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960798 7f868bf5a700 2 waiting 51 > 50 ops || 587206 > 104857600 2014-06-09 16:22:11.960802 7f868ef60700 2 waiting 51 > 50 ops || 586931 > 104857600 2014-06-09 16:22:11.960806 7f8691f66700 2 waiting 51 > 50 ops || 586258 > 104857600 2014-06-09 16:22:11.960794 7f86af56b700 5 -- op tracker -- , seq: 3363, time: 2014-06-09 16:22:11.956122, event: all_read, op: osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.960810 7f8689755700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960807 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3664.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960812 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960814 7f868a757700 2 waiting 51 > 50 ops || 603987 > 104857600 2014-06-09 16:22:11.960819 7f868b759700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960820 7f86af56b700 5 -- op tracker -- , seq: 3363, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.960823 7f868e75f700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960822 7f86a5e11700 10 _do_op 0x3280b740 seq 3664 r = 0, finisher 0xd9fb560 0 2014-06-09 16:22:11.960826 7f86a5e11700 10 _finish_op 0x3280b740 seq 3664 osr(1.67a 0x384de90)/0x384de90 2014-06-09 16:22:11.960827 7f868c75b700 2 waiting 51 > 50 ops || 603987 > 104857600 2014-06-09 16:22:11.960828 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:11.960832 7f8692f68700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960838 7f868cf5c700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960834 7f869af78700 5 -- op tracker -- , seq: 3361, time: 2014-06-09 16:22:11.959265, event: done, op: pg_info(1 pgs e243:0.725) 2014-06-09 16:22:11.960843 7f8692767700 2 waiting 51 > 50 ops || 586033 > 104857600 2014-06-09 16:22:11.960848 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.960850 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.960854 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114480 (1 active) 2014-06-09 16:22:11.960855 7f8691765700 10 op_submit_start 3713 2014-06-09 16:22:11.960856 7f8691765700 5 queue_transactions (trailing journal) 3713 2014-06-09 16:22:11.960856 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653560 (2 active) 2014-06-09 16:22:11.960858 7f8691765700 5 queue_op 0x2fcaf9e0 seq 3713 osr(3.319 0x3ed0d60) 861 bytes (queue has 50 ops and 585238 bytes) 2014-06-09 16:22:11.960861 7f8691765700 10 op_submit_finish 3713 2014-06-09 16:22:11.960860 7f86a5e11700 5 _do_op 0xbe7c780 seq 3665 osr(2.5d9 0x2ed1a70)/0x2ed1a70 start 2014-06-09 16:22:11.960863 7f86a5e11700 10 _do_transaction on 0x39db500 2014-06-09 16:22:11.960857 7f86a4e0f700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.959797, event: op_applied, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.960867 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.960868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.960868 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960870 7f869af78700 20 osd.26 243 _dispatch 0x76e3a40 pg_info(1 pgs e243:0.5f5) v4 2014-06-09 16:22:11.960867 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.960869 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] op_applied on primary on version 243'642 2014-06-09 16:22:11.960875 7f869af78700 5 -- op tracker -- , seq: 3364, time: 2014-06-09 16:22:11.955002, event: header_read, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.960883 7f869af78700 5 -- op tracker -- , seq: 3364, time: 2014-06-09 16:22:11.955003, event: throttled, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.960880 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.960880 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] repop_all_applied: repop tid 9 all applied 2014-06-09 16:22:11.960888 7f869af78700 5 -- op tracker -- , seq: 3364, time: 2014-06-09 16:22:11.955016, event: all_read, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.960888 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.960894 7f869af78700 5 -- op tracker -- , seq: 3364, time: 2014-06-09 16:22:11.960867, event: dispatched, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.960896 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.960901 7f869af78700 5 -- op tracker -- , seq: 3364, time: 2014-06-09 16:22:11.960899, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.960903 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960889 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] eval_repop repgather(0xc2af740 243'642 rep_tid=9 committed?=1 applied?=1 lock=0 op=osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.960906 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5f5) v4 from osd.1 2014-06-09 16:22:11.960908 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e3a40 2014-06-09 16:22:11.960904 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.960913 7f869af78700 5 -- op tracker -- , seq: 3364, time: 2014-06-09 16:22:11.960911, event: started, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.960911 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.960916 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3665.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960920 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960907 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] log_op_stats osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.210345 2014-06-09 16:22:11.960929 7f86a5e11700 10 _do_op 0xbe7c780 seq 3665 r = 0, finisher 0x324f86e0 0 2014-06-09 16:22:11.960927 7f869af78700 5 -- op tracker -- , seq: 3364, time: 2014-06-09 16:22:11.960925, event: done, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.960924 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.960923 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] publish_stats_to_osd 243:1030 2014-06-09 16:22:11.960932 7f86a5e11700 10 _finish_op 0xbe7c780 seq 3665 osr(2.5d9 0x2ed1a70)/0x2ed1a70 2014-06-09 16:22:11.960934 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653560 (1 active) 2014-06-09 16:22:11.960936 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1145a0 (2 active) 2014-06-09 16:22:11.960937 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.960933 7f8691765700 20 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.960938 7f86a5e11700 5 _do_op 0x3280ae40 seq 3666 osr(0.e8 0x2bd6510)/0x2bd6510 start 2014-06-09 16:22:11.960939 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.960941 7f86a5e11700 10 _do_transaction on 0x331eb40 2014-06-09 16:22:11.960934 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] requeue_ops 2014-06-09 16:22:11.960946 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.960940 7f8691765700 20 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.960949 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.960951 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.960952 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.960953 7f869af78700 20 osd.26 243 _dispatch 0x76e2540 pg_info(1 pgs e243:2.55b) v4 2014-06-09 16:22:11.960943 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] removing repgather(0xc2af740 243'642 rep_tid=9 committed?=1 applied?=1 op=osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.960958 7f869af78700 5 -- op tracker -- , seq: 3365, time: 2014-06-09 16:22:11.957270, event: header_read, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.960953 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.960964 7f869af78700 5 -- op tracker -- , seq: 3365, time: 2014-06-09 16:22:11.957271, event: throttled, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.960957 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] q front is repgather(0xc2af740 243'642 rep_tid=9 committed?=1 applied?=1 op=osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.960964 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.960970 7f869af78700 5 -- op tracker -- , seq: 3365, time: 2014-06-09 16:22:11.957285, event: all_read, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.960976 7f869af78700 5 -- op tracker -- , seq: 3365, time: 2014-06-09 16:22:11.960951, event: dispatched, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.960972 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.960979 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.960982 7f869af78700 5 -- op tracker -- , seq: 3365, time: 2014-06-09 16:22:11.960980, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.960972 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] remove_repop repgather(0xc2af740 243'642 rep_tid=9 committed?=1 applied?=1 op=osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.960986 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.55b) v4 from osd.9 2014-06-09 16:22:11.960981 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.960989 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e2540 2014-06-09 16:22:11.960991 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3666.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.960987 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] requeue_ops 2014-06-09 16:22:11.960995 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.960989 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.960993 7f869af78700 5 -- op tracker -- , seq: 3365, time: 2014-06-09 16:22:11.960992, event: started, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.960997 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.961005 7f86a5e11700 10 _do_op 0x3280ae40 seq 3666 r = 0, finisher 0xbccd8a0 0 2014-06-09 16:22:11.961007 7f86a5e11700 10 _finish_op 0x3280ae40 seq 3666 osr(0.e8 0x2bd6510)/0x2bd6510 2014-06-09 16:22:11.961009 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1145a0 (1 active) 2014-06-09 16:22:11.961011 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d9e0 (2 active) 2014-06-09 16:22:11.961007 7f86a4e0f700 5 -- op tracker -- , seq: 669, time: 2014-06-09 16:22:11.961001, event: done, op: osd_op(client.4572.0:30634 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2281472~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961013 7f86a5e11700 5 _do_op 0xc2d5c20 seq 3667 osr(0.380 0x2bd6f20)/0x2bd6f20 start 2014-06-09 16:22:11.961015 7f86a5e11700 10 _do_transaction on 0x344c600 2014-06-09 16:22:11.961008 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.961020 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.961018 7f8691765700 20 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.961024 7f8691765700 20 osd.26 pg_epoch: 243 pg[2.1fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.961027 7f869af78700 5 -- op tracker -- , seq: 3365, time: 2014-06-09 16:22:11.961005, event: done, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.961033 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.961035 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.961037 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.961037 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.961034 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.961047 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.961049 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.961050 7f869af78700 20 osd.26 243 _dispatch 0x31745dc0 pg_info(1 pgs e243:0.493) v4 2014-06-09 16:22:11.961047 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.961054 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.961048 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.961054 7f869af78700 5 -- op tracker -- , seq: 3366, time: 2014-06-09 16:22:11.955199, event: header_read, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.961055 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.961061 7f869af78700 5 -- op tracker -- , seq: 3366, time: 2014-06-09 16:22:11.955200, event: throttled, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.961066 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3667.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.961062 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.961067 7f869af78700 5 -- op tracker -- , seq: 3366, time: 2014-06-09 16:22:11.955215, event: all_read, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.961070 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.961065 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.961073 7f869af78700 5 -- op tracker -- , seq: 3366, time: 2014-06-09 16:22:11.961047, event: dispatched, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.961070 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.961079 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.961079 7f869af78700 5 -- op tracker -- , seq: 3366, time: 2014-06-09 16:22:11.961077, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.961078 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.961083 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.493) v4 from osd.6 2014-06-09 16:22:11.961085 7f86a5e11700 10 _do_op 0xc2d5c20 seq 3667 r = 0, finisher 0xcd334c0 0 2014-06-09 16:22:11.961086 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31745dc0 2014-06-09 16:22:11.961088 7f86a5e11700 10 _finish_op 0xc2d5c20 seq 3667 osr(0.380 0x2bd6f20)/0x2bd6f20 2014-06-09 16:22:11.961090 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d9e0 (1 active) 2014-06-09 16:22:11.961092 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f320 (2 active) 2014-06-09 16:22:11.961090 7f869af78700 5 -- op tracker -- , seq: 3366, time: 2014-06-09 16:22:11.961088, event: started, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.961095 7f86a5e11700 5 _do_op 0x327fc000 seq 3668 osr(0.4da 0x2bd75f0)/0x2bd75f0 start 2014-06-09 16:22:11.961089 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.961097 7f86a5e11700 10 _do_transaction on 0xc2afa40 2014-06-09 16:22:11.961094 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4a7( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.961102 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.961098 7f8691765700 20 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.961105 7f869af78700 5 -- op tracker -- , seq: 3366, time: 2014-06-09 16:22:11.961103, event: done, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.961105 7f8691765700 20 osd.26 pg_epoch: 243 pg[1.4c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.961113 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.961114 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.961116 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.961109 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.961122 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.961117 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.961128 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.961135 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.961136 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.961137 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.4c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.961143 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.961149 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3668.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.961152 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.961152 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.961153 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.552( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.961161 7f86a5e11700 10 _do_op 0x327fc000 seq 3668 r = 0, finisher 0xbd71b40 0 2014-06-09 16:22:11.961164 7f86a5e11700 10 _finish_op 0x327fc000 seq 3668 osr(0.4da 0x2bd75f0)/0x2bd75f0 2014-06-09 16:22:11.961165 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.961160 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.961166 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f320 (1 active) 2014-06-09 16:22:11.961168 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4120 (2 active) 2014-06-09 16:22:11.961170 7f86a5e11700 5 _do_op 0x30607680 seq 3669 osr(3.60 0x3ed02e0)/0x3ed02e0 start 2014-06-09 16:22:11.961172 7f86a5e11700 10 _do_transaction on 0x3339bc0 2014-06-09 16:22:11.961177 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.961173 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:147 2014-06-09 16:22:11.961177 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] op_commit: 12 2014-06-09 16:22:11.961182 7f8691765700 20 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.961190 7f8691765700 20 osd.26 pg_epoch: 243 pg[1.a3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.961199 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.961196 7f86a460e700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.961190, event: op_commit, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961205 7f8689f56700 10 op_submit_start 3714 2014-06-09 16:22:11.961207 7f8689f56700 5 queue_transactions (trailing journal) 3714 2014-06-09 16:22:11.961208 7f8689f56700 5 queue_op 0x14dd6d80 seq 3714 osr(3.77 0x3ed0250) 826 bytes (queue has 47 ops and 582754 bytes) 2014-06-09 16:22:11.961211 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.961212 7f8689f56700 10 op_submit_finish 3714 2014-06-09 16:22:11.961205 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=214'607 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] repop_all_committed: repop tid 12 all committed 2014-06-09 16:22:11.961226 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3669.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.961216 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=243'608 crt=0'0 lcod 214'607 mlcod 0'0 active+clean] eval_repop repgather(0xc14aa80 243'608 rep_tid=12 committed?=1 applied?=0 lock=0 op=osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.961229 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.961216 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.961241 7f86a5e11700 10 _do_op 0x30607680 seq 3669 r = 0, finisher 0x3006dfc0 0 2014-06-09 16:22:11.961231 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=243'608 crt=0'0 lcod 214'607 mlcod 0'0 active+clean] log_op_stats osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 inb 61824 outb 0 rlat 0.000000 lat 1.479466 2014-06-09 16:22:11.961244 7f86a5e11700 10 _finish_op 0x30607680 seq 3669 osr(3.60 0x3ed02e0)/0x3ed02e0 2014-06-09 16:22:11.961246 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4120 (1 active) 2014-06-09 16:22:11.961237 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.961248 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287b00 (2 active) 2014-06-09 16:22:11.961250 7f86a5e11700 5 _do_op 0x327fcc00 seq 3670 osr(0.35b 0x2bd6fe0)/0x2bd6fe0 start 2014-06-09 16:22:11.961252 7f86a5e11700 10 _do_transaction on 0xc0ce9c0 2014-06-09 16:22:11.961246 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=243'608 crt=0'0 lcod 214'607 mlcod 0'0 active+clean] publish_stats_to_osd 243:977 2014-06-09 16:22:11.961256 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.961249 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.961259 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.961255 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=243'608 crt=0'0 lcod 214'607 mlcod 0'0 active+clean] sending commit on repgather(0xc14aa80 243'608 rep_tid=12 committed?=1 applied?=0 lock=0 op=osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) 0x257c7680 2014-06-09 16:22:11.961277 7f8690f64700 10 op_submit_start 3715 2014-06-09 16:22:11.961279 7f8690f64700 5 queue_transactions (trailing journal) 3715 2014-06-09 16:22:11.961280 7f8690f64700 5 queue_op 0x2fcaf200 seq 3715 osr(0.220 0x2bd6a70) 828 bytes (queue has 47 ops and 582724 bytes) 2014-06-09 16:22:11.961283 7f8690f64700 10 op_submit_finish 3715 2014-06-09 16:22:11.961285 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.961287 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.961289 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.961290 7f869af78700 20 osd.26 243 _dispatch 0x313881c0 pg_info(1 pgs e243:2.409) v4 2014-06-09 16:22:11.961286 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.348( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.961292 7f86a460e700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.961286, event: commit_sent, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961302 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3670.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.961306 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.961305 7f869af78700 5 -- op tracker -- , seq: 3367, time: 2014-06-09 16:22:11.955538, event: header_read, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.961302 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] op_commit: 13 2014-06-09 16:22:11.961315 7f86a5e11700 10 _do_op 0x327fcc00 seq 3670 r = 0, finisher 0xbd75bc0 0 2014-06-09 16:22:11.961313 7f869af78700 5 -- op tracker -- , seq: 3367, time: 2014-06-09 16:22:11.955538, event: throttled, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.961318 7f86a5e11700 10 _finish_op 0x327fcc00 seq 3670 osr(0.35b 0x2bd6fe0)/0x2bd6fe0 2014-06-09 16:22:11.961318 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.961320 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287b00 (1 active) 2014-06-09 16:22:11.961322 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c19e0 (2 active) 2014-06-09 16:22:11.961324 7f86a5e11700 5 _do_op 0x324f5f20 seq 3671 osr(1.45d 0x384d670)/0x384d670 start 2014-06-09 16:22:11.961326 7f86a5e11700 10 _do_transaction on 0x331e3c0 2014-06-09 16:22:11.961330 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.961327 7f86a460e700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.961313, event: op_commit, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961334 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'644 crt=0'0 lcod 243'643 mlcod 243'643 active+clean] repop_all_committed: repop tid 13 all committed 2014-06-09 16:22:11.961346 7f8690763700 10 op_submit_start 3716 2014-06-09 16:22:11.961348 7f8690763700 5 queue_transactions (trailing journal) 3716 2014-06-09 16:22:11.961349 7f8690763700 5 queue_op 0x2e21620 seq 3716 osr(3.5b3 0x3ed17e0) 828 bytes (queue has 47 ops and 582724 bytes) 2014-06-09 16:22:11.961352 7f8690763700 10 op_submit_finish 3716 2014-06-09 16:22:11.961346 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'645 crt=0'0 lcod 243'644 mlcod 243'643 active+clean] eval_repop repgather(0xbff66c0 243'645 rep_tid=13 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.961355 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 243/201 242/242/200) 2014-06-09 16:22:11.961358 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.552( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.961366 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.961360 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'645 crt=0'0 lcod 243'644 mlcod 243'643 active+clean] log_op_stats osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.204901 2014-06-09 16:22:11.961379 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3671.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.961383 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.961375 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'645 crt=0'0 lcod 243'644 mlcod 243'643 active+clean] publish_stats_to_osd 243:1031 2014-06-09 16:22:11.961375 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=243 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.961386 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.961390 7f8690763700 5 queue_transactions existing osr(3.5c6 0x3ed17f0)/0x3ed17f0 2014-06-09 16:22:11.961390 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.961393 7f8690763700 10 op_submit_start 3717 2014-06-09 16:22:11.961394 7f8690763700 5 queue_transactions (trailing journal) 3717 2014-06-09 16:22:11.961395 7f8690763700 5 queue_op 0x2e21ce0 seq 3717 osr(3.5c6 0x3ed17f0) 828 bytes (queue has 48 ops and 583552 bytes) 2014-06-09 16:22:11.961397 7f8690763700 10 op_submit_finish 3717 2014-06-09 16:22:11.961385 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'645 crt=0'0 lcod 243'644 mlcod 243'643 active+clean] sending commit on repgather(0xbff66c0 243'645 rep_tid=13 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 0x32070f00 2014-06-09 16:22:11.961398 7f86a5e11700 10 _do_op 0x324f5f20 seq 3671 r = 0, finisher 0x15a3fa80 0 2014-06-09 16:22:11.961401 7f86a5e11700 10 _finish_op 0x324f5f20 seq 3671 osr(1.45d 0x384d670)/0x384d670 2014-06-09 16:22:11.961404 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c19e0 (1 active) 2014-06-09 16:22:11.961406 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383fb00 (2 active) 2014-06-09 16:22:11.961400 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.458( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.961408 7f86a5e11700 5 _do_op 0x1e08bb60 seq 3672 osr(3.17 0x3ed03b0)/0x3ed03b0 start 2014-06-09 16:22:11.961410 7f86a5e11700 10 _do_transaction on 0x3338300 2014-06-09 16:22:11.961405 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.50d( v 213'657 (0'0,213'657] local-les=243 n=126 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.961415 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.961417 7f86a460e700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.961412, event: commit_sent, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961428 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.961433 7f8690763700 5 queue_transactions existing osr(0.458 0x2bd7510)/0x2bd7510 2014-06-09 16:22:11.961435 7f8690763700 10 op_submit_start 3718 2014-06-09 16:22:11.961436 7f8690763700 5 queue_transactions (trailing journal) 3718 2014-06-09 16:22:11.961429 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'645 crt=0'0 lcod 243'644 mlcod 243'643 active+clean] op_commit: 15 2014-06-09 16:22:11.961437 7f8690763700 5 queue_op 0x2e205a0 seq 3718 osr(0.458 0x2bd7510) 828 bytes (queue has 48 ops and 583552 bytes) 2014-06-09 16:22:11.961430 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.961440 7f8690763700 10 op_submit_finish 3718 2014-06-09 16:22:11.961446 7f86a460e700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.961439, event: op_commit, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961443 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.961449 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.961443 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:11.961451 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.961452 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 luod=243'645 crt=0'0 lcod 243'644 mlcod 243'643 active+clean] repop_all_committed: repop tid 15 all committed 2014-06-09 16:22:11.961462 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3672.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.961465 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.961460 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.961457 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.1c7( v 213'647 (0'0,213'647] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) log((0'0,213'647], crt=0'0) missing(0) 2014-06-09 16:22:11.961460 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] eval_repop repgather(0xc2c6240 243'646 rep_tid=15 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.961470 7f8690763700 10 proc_replica_log for osd.2: 3.1c7( v 213'647 (0'0,213'647] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) log((0'0,213'647], crt=0'0) missing(0) 2014-06-09 16:22:11.961468 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.961475 7f86a5e11700 10 _do_op 0x1e08bb60 seq 3672 r = 0, finisher 0x318c2720 0 2014-06-09 16:22:11.961478 7f86a5e11700 10 _finish_op 0x1e08bb60 seq 3672 osr(3.17 0x3ed03b0)/0x3ed03b0 2014-06-09 16:22:11.961475 7f8690763700 20 merge_log cut point (usually last shared) is 213'647 (213'646) modify fcd8d9c7/rbd_data.117b6b8b4567.0000000000001731/head//3 by client.4571.0:34742 2014-06-09 10:16:36.412340 2014-06-09 16:22:11.961475 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.961481 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383fb00 (1 active) 2014-06-09 16:22:11.961484 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6ddb00 (2 active) 2014-06-09 16:22:11.961474 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] log_op_stats osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 364821 outb 0 rlat 0.000000 lat 1.203682 2014-06-09 16:22:11.961485 7f86a5e11700 5 _do_op 0x324f4360 seq 3673 osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 start 2014-06-09 16:22:11.961488 7f86a5e11700 10 _do_transaction on 0xc127200 2014-06-09 16:22:11.961493 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.961488 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:176 2014-06-09 16:22:11.961488 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] publish_stats_to_osd 243:1032 2014-06-09 16:22:11.961496 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.961496 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] sending commit on repgather(0xc2c6240 243'646 rep_tid=15 committed?=1 applied?=0 lock=0 op=osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 0x30bbf900 2014-06-09 16:22:11.961503 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.161( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.961512 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.961516 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.961525 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.961524 7f86a460e700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.961520, event: commit_sent, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961528 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.861435 7 0.000385 2014-06-09 16:22:11.961538 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3673.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.961542 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.961534 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] op_commit: 14 2014-06-09 16:22:11.961539 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.961551 7f86a5e11700 10 _do_op 0x324f4360 seq 3673 r = 0, finisher 0x3281bdc0 0 2014-06-09 16:22:11.961549 7f86a460e700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.961544, event: op_commit, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961554 7f86a5e11700 10 _finish_op 0x324f4360 seq 3673 osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 2014-06-09 16:22:11.961548 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.961556 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6ddb00 (1 active) 2014-06-09 16:22:11.961558 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb9e0 (2 active) 2014-06-09 16:22:11.961560 7f86a5e11700 5 _do_op 0x324f56e0 seq 3674 osr(3.39e 0x3ed0fa0)/0x3ed0fa0 start 2014-06-09 16:22:11.961555 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 luod=243'595 crt=0'0 lcod 243'594 mlcod 243'594 active+clean] repop_all_committed: repop tid 14 all committed 2014-06-09 16:22:11.961562 7f86a5e11700 10 _do_transaction on 0xc0e2a80 2014-06-09 16:22:11.961558 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:11.961567 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.961567 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.961563 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] eval_repop repgather(0x3128000 243'596 rep_tid=14 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.961576 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.961576 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] log_op_stats osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 inb 1548565 outb 0 rlat 0.000000 lat 1.479940 2014-06-09 16:22:11.961583 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.961589 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] publish_stats_to_osd 243:971 2014-06-09 16:22:11.961599 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.961593 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:11.961597 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] sending commit on repgather(0x3128000 243'596 rep_tid=14 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 0x32070000 2014-06-09 16:22:11.961605 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:11.961612 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3674.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.961615 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.961614 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:11.961624 7f86a5e11700 10 _do_op 0x324f56e0 seq 3674 r = 0, finisher 0xcd9ea40 0 2014-06-09 16:22:11.961627 7f86a5e11700 10 _finish_op 0x324f56e0 seq 3674 osr(3.39e 0x3ed0fa0)/0x3ed0fa0 2014-06-09 16:22:11.961629 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb9e0 (1 active) 2014-06-09 16:22:11.961623 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:11.961631 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358360 (2 active) 2014-06-09 16:22:11.961628 7f86a460e700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.961622, event: commit_sent, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.961633 7f86a5e11700 5 _do_op 0xc64b320 seq 3675 osr(3.31f 0x3ed0d50)/0x3ed0d50 start 2014-06-09 16:22:11.961635 7f86a5e11700 10 _do_transaction on 0xc31e000 2014-06-09 16:22:11.961632 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.966227 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.966231 7f8691765700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.966249 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 2014-06-09 16:22:11.966281 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.966283 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.966295 7f8691765700 5 queue_transactions existing osr(0.ef 0x2bd64f0)/0x2bd64f0 2014-06-09 16:22:11.966304 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3675.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.966309 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.966320 7f86a5e11700 10 _do_op 0xc64b320 seq 3675 r = 0, finisher 0x324f8e20 0 2014-06-09 16:22:11.961640 7f86a5610700 10 _finish_op 0xc64a6c0 seq 3640 osr(3.319 0x3ed0d60)/0x3ed0d60 2014-06-09 16:22:11.961641 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] op_applied: 10 2014-06-09 16:22:11.961643 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.966378 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.966395 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.342( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 2014-06-09 16:22:11.966407 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31413ae0,0x31456b20,0x3042aaa0,0x314784e0,0xcc82fe0,0x30d61da0,0x3042a360,0x3042b920,0xbd6b420,0x3042a080,0xc168c20,0x3042b380,0x30e188c0,0xc4119a0,0x30d61880,0x3042a7c0,0x31479120,0xbd75f80,0x30d61700,0xcd0e2c0,0x318c2f60,0x318c3760,0x18742d40,0x13965380,0xbef0380,0xbd6fe60,0x3061ef00,0x2e0ffa0,0x30e19180,0xbcc8420,0x3042acc0,0xbef02c0,0x30e19080,0xbd69fe0,0x30e18500,0x3006ce60,0xc411940,0x3006d0e0,0x2d7392a0,0x30509460,0x2e0ec20,0x318c33e0,0x318c3ca0,0x318c2240,0x2e0e8e0,0x3042bda0,0x3006d1a0,0xc5a7500,0x31791fe0,0x13964100,0x314124a0,0xbd75b20,0x3050b9a0,0x13965a40,0xc5a6140,0xcd32760,0xc5a6ca0,0xbd7df00,0x30c55ba0,0x30c55460,0x30c54860,0x318c3180,0xbd74220,0xcd0e300,0x324803c0,0x3281be00,0x3281b840,0x32073a00,0x32073ac0,0x32073d00,0xc5a78e0,0x32073740,0x307f1600,0xcd32300,0x281b7580,0x314adcc0,0x281b6900,0x3206e900,0x3206fb80,0x32072400,0xd9fac00,0xd9fb280,0xd9fb140,0x187431a0,0x3006cb20,0x3006c560,0x32072a40,0x182cb880,0x320728c0,0x3206ff40,0xcd0f440,0xcd9e3c0,0xcd9ff00,0xcd9f940,0xbd641c0,0xcc82b60,0xc1699a0,0xbd6a940,0xcc95060,0x30629f00,0xc168fa0,0xbd637e0,0x318c3240,0x3042a520,0xc4110c0,0x2d7394a0,0xbd72740,0xcc826e0,0x3042a400,0x2e0e0a0,0x318c2300,0x19941980,0xbd6a320,0x2e0e160,0x31435700,0xcc94b60,0x318c2ac0,0x324d1020,0xccf9160,0xbd606c0,0x3206ef80,0x318c3f40,0xbd65840,0xbd78de0,0x324d16a0,0xbd70c40,0xcc82ec0,0xbd5f0e0,0xbd5dcc0,0xbd5da40,0x306cfe00,0xbfcef80,0xbd66080,0xbd5ce00,0xbd5cc00,0x281b7080,0xc6e8660,0x30629840,0xcc83a40,0x3050a580,0x1a002560,0xbd77de0,0xbd63ba0,0xbd5f800,0xbd5c2e0,0x318c2b80,0x318c25c0,0x18743560,0x29283ea0,0xc169a20,0xbd5c5c0,0xbd5c440,0xbd5ca20,0xbd5c220,0xbd5e540,0xccf8460,0xccf9060,0xc169e20,0x15a3f120,0xbd5c8c0,0x324f99e0,0x3251cae0,0x281b7000,0x1a0024e0,0x17fb7660,0xbef1fe0,0x3251ce60,0x17fb74e0,0xbd606a0,0xc40c900,0x3251d960,0xcc82860,0xd9fb960,0xcc95c60,0x4eb96e0,0x30e18060,0xbdf1b40,0x3062b800,0x307f03c0] 2014-06-09 16:22:11.966436 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x325342e0,0x30628860,0x18742080,0xbd60200,0x189aa180,0xbd7dbe0,0xbd74180,0x17fb7ba0,0x324d1c60,0xbd5efe0,0x3061e5a0,0x17fb7a00,0x306298c0,0xbd5f760,0xccf89a0,0x3062a400,0x305080a0,0x31bff420,0xc2d6100,0x31434ee0,0xcd330c0,0xbd605c0,0x17fb78a0,0xd9fbfe0,0xd9fb640,0xd9fb4c0,0x324f85e0,0xc6e9ee0,0xcd33b40,0xc5a7f60,0x3006dfa0,0x189ab5e0,0x18742120,0x3061e020,0xcd0fba0,0xbd5d0e0,0x324f9220] 2014-06-09 16:22:11.961693 7f869af78700 5 -- op tracker -- , seq: 3367, time: 2014-06-09 16:22:11.955547, event: all_read, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.966449 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=243'608 crt=0'0 lcod 214'607 mlcod 0'0 active+clean] op_commit: 16 2014-06-09 16:22:11.961840 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.1c7( v 213'647 (0'0,213'647] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:11.961856 7f868af58700 10 op_submit_start 3719 2014-06-09 16:22:11.966473 7f868af58700 5 queue_transactions (trailing journal) 3719 2014-06-09 16:22:11.966475 7f868af58700 5 queue_op 0x30d97bc0 seq 3719 osr(3.616 0x3ed1a90) 828 bytes (queue has 50 ops and 694467 bytes) 2014-06-09 16:22:11.961861 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.966472 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:11.966488 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.675137 4 0.000259 2014-06-09 16:22:11.966496 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.4b3( v 213'517 (0'0,213'517] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:902 2014-06-09 16:22:11.966500 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:11.966512 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.966508 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.173125 0 0.000000 2014-06-09 16:22:11.966517 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.966521 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:11.966530 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.966530 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=191 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:11.966538 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.966542 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.966545 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.966551 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:11.966557 7f8689f56700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.966559 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'650 -> 214'650 2014-06-09 16:22:11.966570 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.966583 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.966587 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.211( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.966602 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.1c7( v 213'647 (0'0,213'647] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:11.966617 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.966621 7f8689f56700 5 queue_transactions existing osr(1.211 0x384cc90)/0x384cc90 2014-06-09 16:22:11.961867 7f8690f64700 5 queue_transactions existing osr(1.348 0x384d310)/0x384d310 2014-06-09 16:22:11.966622 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'647,214'650], crt=0'0) 2014-06-09 16:22:11.966634 7f8690763700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.966463 7f869af78700 5 -- op tracker -- , seq: 3367, time: 2014-06-09 16:22:11.961284, event: dispatched, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.966648 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.1c7( v 214'650 lc 213'647 (0'0,214'650] local-les=191 n=131 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:11.961868 7f868ef60700 2 waiting 51 > 50 ops || 696193 > 104857600 2014-06-09 16:22:11.966662 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:11.966658 7f869af78700 5 -- op tracker -- , seq: 3367, time: 2014-06-09 16:22:11.966656, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.966676 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.409) v4 from osd.0 2014-06-09 16:22:11.966679 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x313881c0 2014-06-09 16:22:11.966672 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing dcda11c7/rbd_data.11656b8b4567.0000000000001840/head//3 214'650 is on osd.26 2014-06-09 16:22:11.966684 7f869af78700 5 -- op tracker -- , seq: 3367, time: 2014-06-09 16:22:11.966682, event: started, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.966683 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {dcda11c7/rbd_data.11656b8b4567.0000000000001840/head//3=214'650} 2014-06-09 16:22:11.966693 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:11.966700 7f869af78700 5 -- op tracker -- , seq: 3367, time: 2014-06-09 16:22:11.966699, event: done, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.966701 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing dcda11c7/rbd_data.11656b8b4567.0000000000001840/head//3 214'650 also missing on osd.2 2014-06-09 16:22:11.966710 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.966712 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.966711 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {dcda11c7/rbd_data.11656b8b4567.0000000000001840/head//3=214'650} 2014-06-09 16:22:11.966725 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.966726 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.966720 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:11.966728 7f869af78700 20 osd.26 243 _dispatch 0x30c001c0 pg_info(1 pgs e243:3.52b) v4 2014-06-09 16:22:11.966728 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:11.966733 7f869af78700 5 -- op tracker -- , seq: 3368, time: 2014-06-09 16:22:11.957067, event: header_read, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.966741 7f869af78700 5 -- op tracker -- , seq: 3368, time: 2014-06-09 16:22:11.957067, event: throttled, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.966738 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:11.966747 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:11.966747 7f869af78700 5 -- op tracker -- , seq: 3368, time: 2014-06-09 16:22:11.957076, event: all_read, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.966756 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.966764 7f869af78700 5 -- op tracker -- , seq: 3368, time: 2014-06-09 16:22:11.966725, event: dispatched, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.966771 7f869af78700 5 -- op tracker -- , seq: 3368, time: 2014-06-09 16:22:11.966770, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.966775 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.52b) v4 from osd.4 2014-06-09 16:22:11.966778 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c001c0 2014-06-09 16:22:11.966782 7f869af78700 5 -- op tracker -- , seq: 3368, time: 2014-06-09 16:22:11.966781, event: started, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.966797 7f869af78700 5 -- op tracker -- , seq: 3368, time: 2014-06-09 16:22:11.966795, event: done, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.966804 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.966806 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.966816 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.966818 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.966829 7f869af78700 20 osd.26 243 _dispatch 0x1a038c40 pg_info(1 pgs e243:3.2df) v4 2014-06-09 16:22:11.966835 7f869af78700 5 -- op tracker -- , seq: 3369, time: 2014-06-09 16:22:11.958984, event: header_read, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.966842 7f869af78700 5 -- op tracker -- , seq: 3369, time: 2014-06-09 16:22:11.958985, event: throttled, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.960849 7f86af56b700 15 osd.26 243 enqueue_op 0x2c06ad00 prio 63 cost 401408 latency 0.007378 osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.966850 7f869af78700 5 -- op tracker -- , seq: 3369, time: 2014-06-09 16:22:11.958999, event: all_read, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.966868 7f8690763700 10 log is not dirty 2014-06-09 16:22:11.966874 7f8691f66700 2 waiting 51 > 50 ops || 695520 > 104857600 2014-06-09 16:22:11.966874 7f86a4e0f700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.966362, event: op_applied, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.966886 7f86828e3700 5 -- op tracker -- , seq: 3370, time: 2014-06-09 16:22:11.962445, event: header_read, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) 2014-06-09 16:22:11.966882 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] op_applied on primary on version 243'643 2014-06-09 16:22:11.966892 7f8690763700 5 queue_transactions existing osr(3.1c7 0x3ed0a10)/0x3ed0a10 2014-06-09 16:22:11.966896 7f8690763700 2 waiting 51 > 50 ops || 695520 > 104857600 2014-06-09 16:22:11.966897 7f86828e3700 5 -- op tracker -- , seq: 3370, time: 2014-06-09 16:22:11.962445, event: throttled, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) 2014-06-09 16:22:11.966896 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] repop_all_applied: repop tid 10 all applied 2014-06-09 16:22:11.966905 7f86828e3700 5 -- op tracker -- , seq: 3370, time: 2014-06-09 16:22:11.962450, event: all_read, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) 2014-06-09 16:22:11.966912 7f86828e3700 5 -- op tracker -- , seq: 3370, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) 2014-06-09 16:22:11.966904 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] eval_repop repgather(0xc232840 243'643 rep_tid=10 committed?=1 applied?=1 lock=0 op=osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.966917 7f86828e3700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) v2 epoch 243 2014-06-09 16:22:11.966921 7f86828e3700 20 osd.26 243 should_share_map osd.8 172.16.96.201:6859/53035 243 2014-06-09 16:22:11.966919 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] log_op_stats osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1167637 outb 0 rlat 0.000000 lat 1.180974 2014-06-09 16:22:11.966928 7f86828e3700 15 osd.26 243 enqueue_op 0x230da200 prio 196 cost 0 latency 0.004483 osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) v2 2014-06-09 16:22:11.966934 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] publish_stats_to_osd 243:1033 2014-06-09 16:22:11.966943 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'643 active+clean] requeue_ops 2014-06-09 16:22:11.966953 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] removing repgather(0xc232840 243'643 rep_tid=10 committed?=1 applied?=1 op=osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.966967 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] q front is repgather(0xc232840 243'643 rep_tid=10 committed?=1 applied?=1 op=osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.966977 7f86828e3700 5 -- op tracker -- , seq: 3371, time: 2014-06-09 16:22:11.966948, event: header_read, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.966987 7f86828e3700 5 -- op tracker -- , seq: 3371, time: 2014-06-09 16:22:11.966948, event: throttled, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.966980 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] remove_repop repgather(0xc232840 243'643 rep_tid=10 committed?=1 applied?=1 op=osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.966995 7f86828e3700 5 -- op tracker -- , seq: 3371, time: 2014-06-09 16:22:11.966959, event: all_read, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.966993 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] requeue_ops 2014-06-09 16:22:11.967002 7f86828e3700 5 -- op tracker -- , seq: 3371, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967005 7f86828e3700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.967009 7f86828e3700 20 osd.26 243 should_share_map osd.8 172.16.96.201:6859/53035 243 2014-06-09 16:22:11.967014 7f86828e3700 15 osd.26 243 enqueue_op 0x1b0d6f00 prio 196 cost 0 latency 0.000066 osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.967017 7f86a4e0f700 5 -- op tracker -- , seq: 727, time: 2014-06-09 16:22:11.967012, event: done, op: osd_op(client.4570.0:34072 rbd_data.117a6b8b4567.00000000000016e4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3026944~1167360] 3.886c3060 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.967060 7f86a460e700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.966462, event: op_commit, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.967062 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967075 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.967073 7f869af78700 5 -- op tracker -- , seq: 3369, time: 2014-06-09 16:22:11.966816, event: dispatched, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.967070 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 luod=243'608 crt=0'0 lcod 214'607 mlcod 0'0 active+clean] repop_all_committed: repop tid 16 all committed 2014-06-09 16:22:11.967086 7f86823be700 5 -- op tracker -- , seq: 3372, time: 2014-06-09 16:22:11.962385, event: header_read, op: osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967086 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.967081 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] eval_repop repgather(0x3e62240 243'609 rep_tid=16 committed?=1 applied?=0 lock=0 op=osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.967094 7f86823be700 5 -- op tracker -- , seq: 3372, time: 2014-06-09 16:22:11.962386, event: throttled, op: osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967102 7f86823be700 5 -- op tracker -- , seq: 3372, time: 2014-06-09 16:22:11.962392, event: all_read, op: osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967096 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] log_op_stats osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516375 outb 0 rlat 0.000000 lat 1.471278 2014-06-09 16:22:11.967109 7f86823be700 5 -- op tracker -- , seq: 3372, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967113 7f86823be700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.967111 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] publish_stats_to_osd 243:978 2014-06-09 16:22:11.967117 7f86823be700 20 osd.26 243 should_share_map osd.1 172.16.96.201:6811/57862 243 2014-06-09 16:22:11.967116 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967123 7f86823be700 15 osd.26 243 enqueue_op 0x1b0b6900 prio 196 cost 0 latency 0.004738 osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.967127 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.967120 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] sending commit on repgather(0x3e62240 243'609 rep_tid=16 committed?=1 applied?=0 lock=0 op=osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) 0xbf40f00 2014-06-09 16:22:11.967138 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.967154 7f86a460e700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.967148, event: commit_sent, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.967155 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967168 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.967163 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.967172 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.967178 7f869af78700 5 -- op tracker -- , seq: 3369, time: 2014-06-09 16:22:11.967176, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.967184 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2df) v4 from osd.7 2014-06-09 16:22:11.967181 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.30b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:11.967187 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1a038c40 2014-06-09 16:22:11.967190 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.967193 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.967198 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.967206 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.967206 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.17c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:11.967214 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.967216 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.967225 7f869af78700 5 -- op tracker -- , seq: 3369, time: 2014-06-09 16:22:11.967190, event: started, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.967224 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.967230 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.967232 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.18( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 2014-06-09 16:22:11.967238 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.967243 7f869af78700 5 -- op tracker -- , seq: 3369, time: 2014-06-09 16:22:11.967240, event: done, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.967241 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.967251 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967252 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967248 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.967256 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.522( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 2014-06-09 16:22:11.967264 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967266 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967261 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967267 7f869af78700 20 osd.26 243 _dispatch 0x31388a80 pg_info(1 pgs e243:3.f7) v4 2014-06-09 16:22:11.967270 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.967265 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.967272 7f869af78700 5 -- op tracker -- , seq: 3373, time: 2014-06-09 16:22:11.955024, event: header_read, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.967272 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.967280 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.967280 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.325( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 2014-06-09 16:22:11.967291 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967290 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] _finish_recovery 2014-06-09 16:22:11.967299 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.967299 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] purge_strays 2014-06-09 16:22:11.967309 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] publish_stats_to_osd 243:1034 2014-06-09 16:22:11.967311 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.1e0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.967278 7f869af78700 5 -- op tracker -- , seq: 3373, time: 2014-06-09 16:22:11.955025, event: throttled, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.967323 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x325342e0,0x30628860,0x18742080,0xbd60200,0x189aa180,0xbd7dbe0,0xbd74180,0x17fb7ba0,0x324d1c60,0xbd5efe0,0x3061e5a0,0x17fb7a00,0x306298c0,0xbd5f760,0xccf89a0,0x3062a400,0x305080a0,0x31bff420,0xc2d6100,0x31434ee0,0xcd330c0,0xbd605c0,0x17fb78a0,0xd9fbfe0,0xd9fb640,0xd9fb4c0,0x324f85e0,0xc6e9ee0,0xcd33b40,0xc5a7f60,0x3006dfa0,0x189ab5e0,0x18742120,0x3061e020,0xcd0fba0,0xbd5d0e0,0x324f9220] 2014-06-09 16:22:11.967330 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.967332 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.967332 7f869af78700 5 -- op tracker -- , seq: 3373, time: 2014-06-09 16:22:11.955036, event: all_read, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.967339 7f869af78700 5 -- op tracker -- , seq: 3373, time: 2014-06-09 16:22:11.967264, event: dispatched, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.967346 7f869af78700 5 -- op tracker -- , seq: 3373, time: 2014-06-09 16:22:11.967344, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.967341 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.967350 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.f7) v4 from osd.1 2014-06-09 16:22:11.967353 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31388a80 2014-06-09 16:22:11.967351 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.967386 7f869af78700 5 -- op tracker -- , seq: 3373, time: 2014-06-09 16:22:11.967383, event: started, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.967390 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967403 7f8689755700 2 waiting 51 > 50 ops || 695295 > 104857600 2014-06-09 16:22:11.967407 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.967414 7f869af78700 5 -- op tracker -- , seq: 3373, time: 2014-06-09 16:22:11.967412, event: done, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.967423 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967425 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967423 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.793( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.967437 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967439 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967440 7f869af78700 20 osd.26 243 _dispatch 0x317456c0 pg_info(1 pgs e243:2.1c6) v4 2014-06-09 16:22:11.967445 7f869af78700 5 -- op tracker -- , seq: 3374, time: 2014-06-09 16:22:11.957309, event: header_read, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.967452 7f868a757700 2 waiting 51 > 50 ops || 713249 > 104857600 2014-06-09 16:22:11.967446 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=243 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967455 7f869af78700 5 -- op tracker -- , seq: 3374, time: 2014-06-09 16:22:11.957310, event: throttled, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.967458 7f868b759700 2 waiting 51 > 50 ops || 695295 > 104857600 2014-06-09 16:22:11.967461 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.967463 7f868e75f700 2 waiting 51 > 50 ops || 695295 > 104857600 2014-06-09 16:22:11.967461 7f869af78700 5 -- op tracker -- , seq: 3374, time: 2014-06-09 16:22:11.957323, event: all_read, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.967469 7f868c75b700 2 waiting 51 > 50 ops || 713249 > 104857600 2014-06-09 16:22:11.967468 7f869af78700 5 -- op tracker -- , seq: 3374, time: 2014-06-09 16:22:11.967437, event: dispatched, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.967475 7f8692f68700 2 waiting 51 > 50 ops || 695295 > 104857600 2014-06-09 16:22:11.967482 7f868cf5c700 2 waiting 51 > 50 ops || 695295 > 104857600 2014-06-09 16:22:11.967481 7f869af78700 5 -- op tracker -- , seq: 3374, time: 2014-06-09 16:22:11.967479, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.967475 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c9( v 213'646 (0'0,213'646] local-les=243 n=124 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.967484 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1c6) v4 from osd.9 2014-06-09 16:22:11.967486 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x317456c0 2014-06-09 16:22:11.967490 7f869af78700 5 -- op tracker -- , seq: 3374, time: 2014-06-09 16:22:11.967488, event: started, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.967499 7f869af78700 5 -- op tracker -- , seq: 3374, time: 2014-06-09 16:22:11.967498, event: done, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.967494 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.492( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967504 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967505 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967508 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.967513 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967514 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967514 7f869af78700 20 osd.26 243 _dispatch 0x76e2c40 pg_info(1 pgs e243:2.3aa) v4 2014-06-09 16:22:11.967517 7f869af78700 5 -- op tracker -- , seq: 3375, time: 2014-06-09 16:22:11.961156, event: header_read, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.967521 7f869af78700 5 -- op tracker -- , seq: 3375, time: 2014-06-09 16:22:11.961156, event: throttled, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.967527 7f869af78700 5 -- op tracker -- , seq: 3375, time: 2014-06-09 16:22:11.961169, event: all_read, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.967523 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.492( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] flushed 2014-06-09 16:22:11.967533 7f869af78700 5 -- op tracker -- , seq: 3375, time: 2014-06-09 16:22:11.967513, event: dispatched, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.967539 7f869af78700 5 -- op tracker -- , seq: 3375, time: 2014-06-09 16:22:11.967537, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.967544 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3aa) v4 from osd.8 2014-06-09 16:22:11.967547 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e2c40 2014-06-09 16:22:11.967542 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 206 same_interval_since 242 2014-06-09 16:22:11.967553 7f869af78700 5 -- op tracker -- , seq: 3375, time: 2014-06-09 16:22:11.967551, event: started, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.967564 7f869af78700 5 -- op tracker -- , seq: 3375, time: 2014-06-09 16:22:11.967563, event: done, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.967558 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.967569 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967570 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967578 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967578 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967579 7f869af78700 20 osd.26 243 _dispatch 0x2176c380 pg_info(1 pgs e243:0.493) v4 2014-06-09 16:22:11.967573 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.967582 7f869af78700 5 -- op tracker -- , seq: 3376, time: 2014-06-09 16:22:11.955282, event: header_read, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.967586 7f869af78700 5 -- op tracker -- , seq: 3376, time: 2014-06-09 16:22:11.955283, event: throttled, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.967589 7f869af78700 5 -- op tracker -- , seq: 3376, time: 2014-06-09 16:22:11.955295, event: all_read, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.967592 7f869af78700 5 -- op tracker -- , seq: 3376, time: 2014-06-09 16:22:11.967577, event: dispatched, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.967596 7f869af78700 5 -- op tracker -- , seq: 3376, time: 2014-06-09 16:22:11.967595, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.967598 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.493) v4 from osd.6 2014-06-09 16:22:11.967600 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176c380 2014-06-09 16:22:11.967592 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.755( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967602 7f869af78700 5 -- op tracker -- , seq: 3376, time: 2014-06-09 16:22:11.967601, event: started, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.967605 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.967611 7f869af78700 5 -- op tracker -- , seq: 3376, time: 2014-06-09 16:22:11.967610, event: done, op: pg_info(1 pgs e243:0.493) 2014-06-09 16:22:11.967629 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967630 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967624 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.755( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.967639 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.967639 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.967640 7f869af78700 20 osd.26 243 _dispatch 0x31388380 pg_info(1 pgs e243:2.409) v4 2014-06-09 16:22:11.967643 7f869af78700 5 -- op tracker -- , seq: 3377, time: 2014-06-09 16:22:11.955560, event: header_read, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.967648 7f869af78700 5 -- op tracker -- , seq: 3377, time: 2014-06-09 16:22:11.955561, event: throttled, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.967642 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] op_applied: 11 2014-06-09 16:22:11.967651 7f869af78700 5 -- op tracker -- , seq: 3377, time: 2014-06-09 16:22:11.955573, event: all_read, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.967654 7f869af78700 5 -- op tracker -- , seq: 3377, time: 2014-06-09 16:22:11.967638, event: dispatched, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.967657 7f8692767700 2 waiting 51 > 50 ops || 695295 > 104857600 2014-06-09 16:22:11.967664 7f868d75d700 2 waiting 51 > 50 ops || 1729047 > 104857600 2014-06-09 16:22:11.967661 7f869af78700 5 -- op tracker -- , seq: 3377, time: 2014-06-09 16:22:11.967659, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.967675 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.409) v4 from osd.0 2014-06-09 16:22:11.967678 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31388380 2014-06-09 16:22:11.967684 7f8691765700 2 waiting 51 > 50 ops || 695325 > 104857600 2014-06-09 16:22:11.967684 7f86a4e0f700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.967654, event: op_applied, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.967701 7f8690f64700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.967699 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338ea0 (1 active) 2014-06-09 16:22:11.967704 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4a20 (2 active) 2014-06-09 16:22:11.967706 7f8689f56700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.967708 7f86a5610700 5 _do_op 0x309a31a0 seq 3676 osr(3.1e 0x3ed0380)/0x3ed0380 start 2014-06-09 16:22:11.967702 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] op_applied on primary on version 243'644 2014-06-09 16:22:11.967712 7f86a5610700 10 _do_transaction on 0xbf8bec0 2014-06-09 16:22:11.967713 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] repop_all_applied: repop tid 11 all applied 2014-06-09 16:22:11.967719 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.967724 7f8690763700 2 waiting 51 > 50 ops || 178857 > 104857600 2014-06-09 16:22:11.967732 7f8689755700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.967721 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] eval_repop repgather(0x30d1740 243'644 rep_tid=11 committed?=1 applied?=1 lock=0 op=osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.967736 7f868a757700 2 waiting 51 > 50 ops || 196586 > 104857600 2014-06-09 16:22:11.967740 7f868b759700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.967744 7f868e75f700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.967736 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] log_op_stats osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.212474 2014-06-09 16:22:11.967748 7f868c75b700 2 waiting 51 > 50 ops || 196586 > 104857600 2014-06-09 16:22:11.967752 7f8692f68700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.967756 7f868cf5c700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.967750 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] publish_stats_to_osd 243:1035 2014-06-09 16:22:11.967760 7f8692767700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.967765 7f868d75d700 2 waiting 51 > 50 ops || 1212384 > 104857600 2014-06-09 16:22:11.967759 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] requeue_ops 2014-06-09 16:22:11.967770 7f8691765700 2 waiting 51 > 50 ops || 178662 > 104857600 2014-06-09 16:22:11.967771 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.967768 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] removing repgather(0x30d1740 243'644 rep_tid=11 committed?=1 applied?=1 op=osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.967786 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3676.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.967789 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.967781 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] q front is repgather(0x30d1740 243'644 rep_tid=11 committed?=1 applied?=1 op=osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.967794 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] remove_repop repgather(0x30d1740 243'644 rep_tid=11 committed?=1 applied?=1 op=osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.967806 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] requeue_ops 2014-06-09 16:22:11.967823 7f86a5610700 10 _do_op 0x309a31a0 seq 3676 r = 0, finisher 0x324f8be0 0 2014-06-09 16:22:11.967828 7f86a5610700 10 _finish_op 0x309a31a0 seq 3676 osr(3.1e 0x3ed0380)/0x3ed0380 2014-06-09 16:22:11.967824 7f86a4e0f700 5 -- op tracker -- , seq: 677, time: 2014-06-09 16:22:11.967819, event: done, op: osd_op(client.4572.0:30635 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 2797568~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.967832 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4a20 (1 active) 2014-06-09 16:22:11.967834 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286000 (2 active) 2014-06-09 16:22:11.967835 7f86a5610700 5 _do_op 0x1e08af60 seq 3677 osr(3.647 0x3ed1a70)/0x3ed1a70 start 2014-06-09 16:22:11.967837 7f86a5610700 10 _do_transaction on 0xbf36a80 2014-06-09 16:22:11.967838 7f8689f56700 2 waiting 51 > 50 ops || 178602 > 104857600 2014-06-09 16:22:11.967840 7f86a5610700 15 _omap_setkeys meta/1fe13f4a/pglog_3.647/0//-1 2014-06-09 16:22:11.967844 7f8690763700 2 waiting 51 > 50 ops || 178827 > 104857600 2014-06-09 16:22:11.967850 7f8689755700 2 waiting 51 > 50 ops || 178602 > 104857600 2014-06-09 16:22:11.967854 7f868a757700 2 waiting 51 > 50 ops || 196556 > 104857600 2014-06-09 16:22:11.967858 7f868b759700 2 waiting 51 > 50 ops || 178602 > 104857600 2014-06-09 16:22:11.967862 7f868e75f700 2 waiting 51 > 50 ops || 178602 > 104857600 2014-06-09 16:22:11.967863 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967871 7f8682be6700 5 -- op tracker -- , seq: 3378, time: 2014-06-09 16:22:11.967840, event: header_read, op: osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967878 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.967882 7f8682be6700 5 -- op tracker -- , seq: 3378, time: 2014-06-09 16:22:11.967841, event: throttled, op: osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967886 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe13f4a/pglog_3.647/0//-1 2014-06-09 16:22:11.967890 7f8682be6700 5 -- op tracker -- , seq: 3378, time: 2014-06-09 16:22:11.967849, event: all_read, op: osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967898 7f8682be6700 5 -- op tracker -- , seq: 3378, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.967894 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.71f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.967901 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.967902 7f8682be6700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.967907 7f8682be6700 20 osd.26 243 should_share_map osd.5 172.16.96.201:6837/51639 243 2014-06-09 16:22:11.967913 7f8682be6700 15 osd.26 243 enqueue_op 0xdfa7e00 prio 196 cost 0 latency 0.000073 osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.967924 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967936 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.967936 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.967951 7f86a5610700 15 _omap_rmkeys meta/1fe13f4a/pglog_3.647/0//-1 2014-06-09 16:22:11.967948 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1f9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.967957 7f86a5610700 10 _omap_rmkeys meta/1fe13f4a/pglog_3.647/0//-1 r = 0 2014-06-09 16:22:11.967963 7f86a5610700 15 _omap_setkeys meta/1fe13f4a/pglog_3.647/0//-1 2014-06-09 16:22:11.967960 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=243 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.967973 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:11.967977 7f86a5610700 15 _touch 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.967984 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.792( v 213'667 (0'0,213'667] local-les=243 n=135 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.968000 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968013 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.968018 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.647_head oid cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.968024 7f868c75b700 2 waiting 51 > 50 ops || 196556 > 104857600 2014-06-09 16:22:11.968030 7f8692f68700 2 waiting 51 > 50 ops || 178602 > 104857600 2014-06-09 16:22:11.968026 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.968042 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968050 7f86a5610700 20 write_state seq is 30156 2014-06-09 16:22:11.968051 7f868cf5c700 2 waiting 51 > 50 ops || 178602 > 104857600 2014-06-09 16:22:11.968055 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.968055 7f8692767700 2 waiting 51 > 50 ops || 178602 > 104857600 2014-06-09 16:22:11.968057 7f86a5610700 20 set_header setting 30155 cid 3.647_head oid cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 parent seq 0 2014-06-09 16:22:11.968071 7f86a5610700 10 _touch 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 = 0 2014-06-09 16:22:11.968072 7f868d75d700 2 waiting 51 > 50 ops || 1212354 > 104857600 2014-06-09 16:22:11.968068 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.968078 7f86a5610700 15 _write 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 0~24576 2014-06-09 16:22:11.968084 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.968086 7f8691765700 2 waiting 51 > 50 ops || 178632 > 104857600 2014-06-09 16:22:11.968087 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968098 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.968110 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.968129 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968141 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.968149 7f86a5610700 10 _generic_write 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 0~24576 = 0 2014-06-09 16:22:11.968159 7f86a5610700 15 _setattrs 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.968153 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.968169 7f86a5610700 10 _setattrs 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 = 0 2014-06-09 16:22:11.968175 7f86a5610700 15 _setattrs 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 2014-06-09 16:22:11.968172 7f86af56b700 5 -- op tracker -- , seq: 3379, time: 2014-06-09 16:22:11.967060, event: header_read, op: osd_op(client.4541.0:32044 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~348160] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.968175 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968182 7f86a5610700 10 _setattrs 3.647_head/cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 = 0 2014-06-09 16:22:11.968183 7f86af56b700 5 -- op tracker -- , seq: 3379, time: 2014-06-09 16:22:11.967062, event: throttled, op: osd_op(client.4541.0:32044 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~348160] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.968188 7f86a5610700 10 cid: oid: cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 not skipping op, *spos 3677.0.0 2014-06-09 16:22:11.968189 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.968190 7f86a5610700 10 > header.spos 0.0.0 2014-06-09 16:22:11.968193 7f86a5610700 20 set_header setting 30155 cid 3.647_head oid cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 parent seq 0 2014-06-09 16:22:11.968192 7f86af56b700 5 -- op tracker -- , seq: 3379, time: 2014-06-09 16:22:11.968148, event: all_read, op: osd_op(client.4541.0:32044 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~348160] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.968201 7f86af56b700 5 -- op tracker -- , seq: 3379, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32044 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~348160] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.968202 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3677.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.968205 7f86a5610700 10 cid: oid: 1fe13f4a/pglog_3.647/0//-1 skipping op, spos 3677.0.0 <= header.spos 150316.0.0 2014-06-09 16:22:11.968199 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] flushed 2014-06-09 16:22:11.968207 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.968207 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:11.968216 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968228 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.968226 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7700 prio 63 cost 348160 latency 0.001166 osd_op(client.4541.0:32044 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~348160] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.968240 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.968256 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968268 7f86a4e0f700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:11.968280 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.968305 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.968322 7f86a5610700 10 _do_op 0x1e08af60 seq 3677 r = 0, finisher 0x1322d960 0x30e2f3c0 2014-06-09 16:22:11.968326 7f86a5610700 10 _finish_op 0x1e08af60 seq 3677 osr(3.647 0x3ed1a70)/0x3ed1a70 2014-06-09 16:22:11.968320 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.968331 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286000 (1 active) 2014-06-09 16:22:11.968333 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbb00 (2 active) 2014-06-09 16:22:11.968335 7f86a5610700 5 _do_op 0x24105da0 seq 3678 osr(3.d1 0x3ed0560)/0x3ed0560 start 2014-06-09 16:22:11.968337 7f86a5610700 10 _do_transaction on 0xc120cc0 2014-06-09 16:22:11.968341 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.968333 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.968342 7f8690763700 2 waiting 51 > 50 ops || 152681 > 104857600 2014-06-09 16:22:11.968348 7f8689755700 2 waiting 51 > 50 ops || 152456 > 104857600 2014-06-09 16:22:11.968352 7f868a757700 2 waiting 51 > 50 ops || 170410 > 104857600 2014-06-09 16:22:11.968356 7f868b759700 2 waiting 51 > 50 ops || 152456 > 104857600 2014-06-09 16:22:11.968360 7f868e75f700 2 waiting 51 > 50 ops || 152456 > 104857600 2014-06-09 16:22:11.968364 7f868c75b700 2 waiting 51 > 50 ops || 170410 > 104857600 2014-06-09 16:22:11.968368 7f8692f68700 2 waiting 51 > 50 ops || 152456 > 104857600 2014-06-09 16:22:11.968366 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968382 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.968391 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.968396 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.968407 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3678.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.968411 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.968421 7f868cf5c700 2 waiting 51 > 50 ops || 152456 > 104857600 2014-06-09 16:22:11.968421 7f86a5610700 10 _do_op 0x24105da0 seq 3678 r = 0, finisher 0x3251ca20 0 2014-06-09 16:22:11.968425 7f86a5610700 10 _finish_op 0x24105da0 seq 3678 osr(3.d1 0x3ed0560)/0x3ed0560 2014-06-09 16:22:11.968427 7f8692767700 2 waiting 51 > 50 ops || 152456 > 104857600 2014-06-09 16:22:11.968420 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968433 7f868d75d700 2 waiting 51 > 50 ops || 1186208 > 104857600 2014-06-09 16:22:11.968453 7f8691765700 2 waiting 51 > 50 ops || 152486 > 104857600 2014-06-09 16:22:11.968463 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.968467 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbb00 (1 active) 2014-06-09 16:22:11.968469 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6ddb00 (2 active) 2014-06-09 16:22:11.968473 7f86a5610700 5 _do_op 0x24104300 seq 3679 osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 start 2014-06-09 16:22:11.968476 7f86a5610700 10 _do_transaction on 0xc005a40 2014-06-09 16:22:11.968477 7f868af58700 10 op_submit_finish 3719 2014-06-09 16:22:11.968477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f8ae0,0x3062aba0,0x3251ce20] 2014-06-09 16:22:11.968482 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.968481 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.968481 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] _finish_recovery 2014-06-09 16:22:11.968484 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.5a2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.968491 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] purge_strays 2014-06-09 16:22:11.968497 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968500 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] publish_stats_to_osd 243:972 2014-06-09 16:22:11.968512 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.968519 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.968522 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.968510 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 17 2014-06-09 16:22:11.968527 7f868af58700 5 queue_transactions existing osr(0.5a2 0x2bd7af0)/0x2bd7af0 2014-06-09 16:22:11.968530 7f868af58700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968533 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3679.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.968526 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.968536 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.968534 7f86a460e700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.968528, event: op_commit, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.968544 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f8ae0,0x3062aba0,0x3251ce20] 2014-06-09 16:22:11.968546 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.968547 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.968549 7f86a5610700 10 _do_op 0x24104300 seq 3679 r = 0, finisher 0x33a7f60 0 2014-06-09 16:22:11.968552 7f868f761700 10 op_submit_start 3720 2014-06-09 16:22:11.968553 7f868f761700 5 queue_transactions (trailing journal) 3720 2014-06-09 16:22:11.968555 7f86a5610700 10 _finish_op 0x24104300 seq 3679 osr(0.5b0 0x2bd7ab0)/0x2bd7ab0 2014-06-09 16:22:11.968555 7f868f761700 5 queue_op 0xbdc8600 seq 3720 osr(3.ff 0x3ed0870) 108884 bytes (queue has 50 ops and 151855 bytes) 2014-06-09 16:22:11.968560 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6ddb00 (1 active) 2014-06-09 16:22:11.968561 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x18742ca0] 2014-06-09 16:22:11.968562 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0a20 (2 active) 2014-06-09 16:22:11.968563 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x18742ca0] 2014-06-09 16:22:11.968553 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=227'654 lcod 0'0 inactive m=1] _activate_committed 243 telling primary 2014-06-09 16:22:11.968564 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.968564 7f868b759700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968565 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.968565 7f86a5610700 5 _do_op 0x324e7b00 seq 3680 osr(0.681 0x2bd7e10)/0x2bd7e10 start 2014-06-09 16:22:11.968568 7f86a5610700 10 _do_transaction on 0x344d8c0 2014-06-09 16:22:11.968569 7f868f761700 10 op_submit_finish 3720 2014-06-09 16:22:11.968569 7f868e75f700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968569 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.968573 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.968573 7f868c75b700 2 waiting 51 > 50 ops || 170637 > 104857600 2014-06-09 16:22:11.968577 7f8692f68700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968581 7f868cf5c700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968586 7f8692767700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968589 7f868d75d700 2 waiting 51 > 50 ops || 1186435 > 104857600 2014-06-09 16:22:11.968583 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] flushed 2014-06-09 16:22:11.968575 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=213'612 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc0d1440 243'613 rep_tid=20 committed?=0 applied?=0 lock=0 op=osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.968596 7f8691765700 2 waiting 51 > 50 ops || 152713 > 104857600 2014-06-09 16:22:11.968598 7f868f761700 10 osd.26 243 dequeue_op 0x29f5b200 finish 2014-06-09 16:22:11.968602 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.968608 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.968610 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.968616 7f868df5e700 10 op_submit_start 3721 2014-06-09 16:22:11.968617 7f868df5e700 5 queue_transactions (trailing journal) 3721 2014-06-09 16:22:11.968608 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.968619 7f868df5e700 5 queue_op 0x159966c0 seq 3721 osr(2.d3 0x2ed0450) 826 bytes (queue has 50 ops and 151855 bytes) 2014-06-09 16:22:11.968623 7f868df5e700 10 op_submit_finish 3721 2014-06-09 16:22:11.968621 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3680.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.968625 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.968621 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.968635 7f86a5610700 10 _do_op 0x324e7b00 seq 3680 r = 0, finisher 0x3006c760 0 2014-06-09 16:22:11.968628 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.36d( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.968641 7f86a5610700 10 _finish_op 0x324e7b00 seq 3680 osr(0.681 0x2bd7e10)/0x2bd7e10 2014-06-09 16:22:11.968636 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.968646 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0a20 (1 active) 2014-06-09 16:22:11.968648 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb9e0 (2 active) 2014-06-09 16:22:11.968649 7f868b759700 10 op_submit_start 3722 2014-06-09 16:22:11.968651 7f868b759700 5 queue_transactions (trailing journal) 3722 2014-06-09 16:22:11.968650 7f86a5610700 5 _do_op 0x324e7bc0 seq 3681 osr(3.39e 0x3ed0fa0)/0x3ed0fa0 start 2014-06-09 16:22:11.968653 7f86a5610700 10 _do_transaction on 0x32fe840 2014-06-09 16:22:11.968652 7f868b759700 5 queue_op 0xbf7f440 seq 3722 osr(0.25b 0x2bd69f0) 828 bytes (queue has 50 ops and 151855 bytes) 2014-06-09 16:22:11.968648 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.968656 7f868b759700 10 op_submit_finish 3722 2014-06-09 16:22:11.968658 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.968663 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.968659 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.52e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.968668 7f868df5e700 5 queue_transactions existing osr(0.36d 0x2bd6fc0)/0x2bd6fc0 2014-06-09 16:22:11.968670 7f868df5e700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968663 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968674 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314136e0] 2014-06-09 16:22:11.968676 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314136e0] 2014-06-09 16:22:11.968677 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.968678 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.968678 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.968682 7f868ff62700 10 op_submit_start 3723 2014-06-09 16:22:11.968683 7f868ff62700 5 queue_transactions (trailing journal) 3723 2014-06-09 16:22:11.968684 7f868ff62700 5 queue_op 0x30d968a0 seq 3723 osr(2.e4 0x2ed0410) 858 bytes (queue has 50 ops and 151855 bytes) 2014-06-09 16:22:11.968687 7f868ff62700 10 op_submit_finish 3723 2014-06-09 16:22:11.968690 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.968692 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.968695 7f868b759700 5 queue_transactions existing osr(2.52e 0x2ed1600)/0x2ed1600 2014-06-09 16:22:11.968697 7f868b759700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968689 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.968700 7f868e75f700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968692 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.968704 7f868c75b700 2 waiting 51 > 50 ops || 170637 > 104857600 2014-06-09 16:22:11.968705 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3681.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.968709 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.968710 7f8692f68700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968705 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.968715 7f868cf5c700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968710 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968720 7f8692767700 2 waiting 51 > 50 ops || 152683 > 104857600 2014-06-09 16:22:11.968714 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.968724 7f868d75d700 2 waiting 51 > 50 ops || 1186435 > 104857600 2014-06-09 16:22:11.968724 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.968725 7f86a5610700 10 _do_op 0x324e7bc0 seq 3681 r = 0, finisher 0x31412e60 0 2014-06-09 16:22:11.968727 7f8691765700 2 waiting 51 > 50 ops || 152713 > 104857600 2014-06-09 16:22:11.968723 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.968730 7f86a5610700 10 _finish_op 0x324e7bc0 seq 3681 osr(3.39e 0x3ed0fa0)/0x3ed0fa0 2014-06-09 16:22:11.968735 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb9e0 (1 active) 2014-06-09 16:22:11.968738 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbb00 (2 active) 2014-06-09 16:22:11.968731 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.968740 7f86a5610700 5 _do_op 0x324e7c80 seq 3682 osr(3.d1 0x3ed0560)/0x3ed0560 start 2014-06-09 16:22:11.968742 7f86a5610700 10 _do_transaction on 0xc227e00 2014-06-09 16:22:11.968738 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.968740 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.968747 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.968755 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:184 2014-06-09 16:22:11.968765 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.968773 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.968757 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 206 same_interval_since 242 2014-06-09 16:22:11.968782 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.968782 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.968785 7f868bf5a700 10 op_submit_start 3724 2014-06-09 16:22:11.968787 7f868bf5a700 5 queue_transactions (trailing journal) 3724 2014-06-09 16:22:11.968788 7f868bf5a700 5 queue_op 0x11fce360 seq 3724 osr(1.76e 0x2ed03b0) 2001 bytes (queue has 49 ops and 151027 bytes) 2014-06-09 16:22:11.968791 7f868bf5a700 10 op_submit_finish 3724 2014-06-09 16:22:11.968784 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.968786 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.968795 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3682.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.968798 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.968797 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.968797 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.968807 7f86a5610700 10 _do_op 0x324e7c80 seq 3682 r = 0, finisher 0x31435e40 0 2014-06-09 16:22:11.968805 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.968811 7f86a5610700 10 _finish_op 0x324e7c80 seq 3682 osr(3.d1 0x3ed0560)/0x3ed0560 2014-06-09 16:22:11.968814 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbb00 (1 active) 2014-06-09 16:22:11.968798 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.968816 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f440 (2 active) 2014-06-09 16:22:11.968810 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.968818 7f86a5610700 5 _do_op 0x327fc060 seq 3683 osr(3.e0 0x3ed0500)/0x3ed0500 start 2014-06-09 16:22:11.968813 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.968820 7f86a5610700 10 _do_transaction on 0x97bb200 2014-06-09 16:22:11.968820 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.968825 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.968819 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x11e62200,0x16bd7b00,0x11e62100,0x16bd7a00,0x11e62000,0x19fe0e00,0x14052700,0x19fe0d00,0x14052600,0x19fe0c00,0x14052500,0x19fe0b00 2014-06-09 16:22:11.968827 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.968826 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.968841 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.968839 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:190 2014-06-09 16:22:11.968847 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.968854 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.968862 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.968853 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:11.968861 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.968866 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.968874 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3683.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.968878 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.968859 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.968872 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] op_applied: 12 2014-06-09 16:22:11.968879 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.968879 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.968887 7f86a5610700 10 _do_op 0x327fc060 seq 3683 r = 0, finisher 0xbd75880 0 2014-06-09 16:22:11.968890 7f86a5610700 10 _finish_op 0x327fc060 seq 3683 osr(3.e0 0x3ed0500)/0x3ed0500 2014-06-09 16:22:11.968893 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f440 (1 active) 2014-06-09 16:22:11.968887 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.968887 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.968889 7f86a4e0f700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.968883, event: op_applied, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.968895 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407e60 (2 active) 2014-06-09 16:22:11.968897 7f86a5610700 5 _do_op 0x324e7d40 seq 3684 osr(2.3c0 0x2ed13f0)/0x2ed13f0 start 2014-06-09 16:22:11.968900 7f86a5610700 10 _do_transaction on 0x3128cc0 2014-06-09 16:22:11.968896 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.968904 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.968897 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] op_applied on primary on version 243'608 2014-06-09 16:22:11.968901 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1060 2014-06-09 16:22:11.968905 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.968907 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] repop_all_applied: repop tid 12 all applied 2014-06-09 16:22:11.968910 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.968913 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.968918 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.968928 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.968916 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] eval_repop repgather(0xc14aa80 243'608 rep_tid=12 committed?=1 applied?=1 lock=0 op=osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.968925 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1090 2014-06-09 16:22:11.968939 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.968932 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.968934 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.968933 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] log_op_stats osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 inb 61824 outb 0 rlat 0.000000 lat 1.487168 2014-06-09 16:22:11.968943 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.968942 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.968950 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.968951 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3684.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.968955 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.968949 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.968948 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] publish_stats_to_osd 243:979 2014-06-09 16:22:11.968954 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.968958 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 0'0 active+clean] requeue_ops 2014-06-09 16:22:11.968965 7f86a5610700 10 _do_op 0x324e7d40 seq 3684 r = 0, finisher 0xbd6cb60 0 2014-06-09 16:22:11.968968 7f86a5610700 10 _finish_op 0x324e7d40 seq 3684 osr(2.3c0 0x2ed13f0)/0x2ed13f0 2014-06-09 16:22:11.968971 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407e60 (1 active) 2014-06-09 16:22:11.968972 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f8c0 (2 active) 2014-06-09 16:22:11.968974 7f86a5610700 5 _do_op 0xc88a000 seq 3685 osr(1.91 0x384c430)/0x384c430 start 2014-06-09 16:22:11.968977 7f86a5610700 10 _do_transaction on 0xbeba900 2014-06-09 16:22:11.968957 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.968965 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.968967 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] removing repgather(0xc14aa80 243'608 rep_tid=12 committed?=1 applied?=1 op=osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.968981 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.968981 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.968980 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.968981 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] q front is repgather(0xc14aa80 243'608 rep_tid=12 committed?=1 applied?=1 op=osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.968988 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.968988 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.968997 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.969001 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:152 2014-06-09 16:22:11.968995 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] remove_repop repgather(0xc14aa80 243'608 rep_tid=12 committed?=1 applied?=1 op=osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.969004 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.969009 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.969015 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969009 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] requeue_ops 2014-06-09 16:22:11.969015 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:168 2014-06-09 16:22:11.969016 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.969024 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969023 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.969028 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3685.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969032 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969029 7f86a4e0f700 5 -- op tracker -- , seq: 410, time: 2014-06-09 16:22:11.969023, event: done, op: osd_op(client.4569.0:38637 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 0~61440] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.969031 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.39f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.969038 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.969041 7f86a5610700 10 _do_op 0xc88a000 seq 3685 r = 0, finisher 0x327f3700 0 2014-06-09 16:22:11.969045 7f86a5610700 10 _finish_op 0xc88a000 seq 3685 osr(1.91 0x384c430)/0x384c430 2014-06-09 16:22:11.969047 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f8c0 (1 active) 2014-06-09 16:22:11.969042 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.969049 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff50e0 (2 active) 2014-06-09 16:22:11.969049 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41980 (19 active) 2014-06-09 16:22:11.969051 7f86a5610700 5 _do_op 0xc88bf20 seq 3686 osr(2.c9 0x2ed04a0)/0x2ed04a0 start 2014-06-09 16:22:11.969054 7f86a5610700 10 _do_transaction on 0xc28a300 2014-06-09 16:22:11.969052 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.969058 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969058 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.969058 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] op_applied: 13 2014-06-09 16:22:11.969066 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.969077 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) 2014-06-09 16:22:11.969073 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.969075 7f86a4e0f700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.969069, event: op_applied, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.969080 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.969080 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.3b6( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.3b6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:11.969082 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] op_applied on primary on version 243'645 2014-06-09 16:22:11.969092 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969090 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:206 2014-06-09 16:22:11.969092 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] repop_all_applied: repop tid 13 all applied 2014-06-09 16:22:11.969099 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.969104 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3686.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969108 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969109 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969106 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.70f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.969100 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] eval_repop repgather(0xbff66c0 243'645 rep_tid=13 committed?=1 applied?=1 lock=0 op=osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.969113 7f868bf5a700 5 queue_transactions existing osr(0.3b6 0x2bd72d0)/0x2bd72d0 2014-06-09 16:22:11.969114 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.969116 7f868bf5a700 10 op_submit_start 3725 2014-06-09 16:22:11.969117 7f868bf5a700 5 queue_transactions (trailing journal) 3725 2014-06-09 16:22:11.969119 7f868bf5a700 5 queue_op 0x14dd6de0 seq 3725 osr(0.3b6 0x2bd72d0) 828 bytes (queue has 46 ops and 148549 bytes) 2014-06-09 16:22:11.969121 7f868bf5a700 10 op_submit_finish 3725 2014-06-09 16:22:11.969123 7f86a5610700 10 _do_op 0xc88bf20 seq 3686 r = 0, finisher 0x32481600 0 2014-06-09 16:22:11.969119 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.969126 7f86a5610700 10 _finish_op 0xc88bf20 seq 3686 osr(2.c9 0x2ed04a0)/0x2ed04a0 2014-06-09 16:22:11.969116 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] log_op_stats osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516373 outb 0 rlat 0.000000 lat 1.212656 2014-06-09 16:22:11.969129 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff50e0 (1 active) 2014-06-09 16:22:11.969130 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc7e0 (2 active) 2014-06-09 16:22:11.969123 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.6f( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.6f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:11.969132 7f86a5610700 5 _do_op 0x324e7f80 seq 3687 osr(3.2fd 0x3ed0de0)/0x3ed0de0 start 2014-06-09 16:22:11.969135 7f86a5610700 10 _do_transaction on 0xc264b40 2014-06-09 16:22:11.969129 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.969131 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] publish_stats_to_osd 243:1036 2014-06-09 16:22:11.969139 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969136 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.969140 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] requeue_ops 2014-06-09 16:22:11.969144 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.969151 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969155 7f868bf5a700 5 queue_transactions existing osr(2.6f 0x2ed0550)/0x2ed0550 2014-06-09 16:22:11.969157 7f868bf5a700 10 op_submit_start 3726 2014-06-09 16:22:11.969158 7f868bf5a700 5 queue_transactions (trailing journal) 3726 2014-06-09 16:22:11.969152 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.969159 7f868bf5a700 5 queue_op 0x324e64e0 seq 3726 osr(2.6f 0x2ed0550) 826 bytes (queue has 46 ops and 148549 bytes) 2014-06-09 16:22:11.969161 7f868bf5a700 10 op_submit_finish 3726 2014-06-09 16:22:11.969149 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] removing repgather(0xbff66c0 243'645 rep_tid=13 committed?=1 applied?=1 op=osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.969159 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.969164 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.2d0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.2d0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:11.969173 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969163 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] q front is repgather(0xbff66c0 243'645 rep_tid=13 committed?=1 applied?=1 op=osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.969170 7f868ff62700 15 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.969179 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.969186 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3687.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969177 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] remove_repop repgather(0xbff66c0 243'645 rep_tid=13 committed?=1 applied?=1 op=osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.969190 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969190 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969186 7f868ff62700 20 osd.26 pg_epoch: 243 pg[0.627( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.969194 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.969193 7f868bf5a700 5 queue_transactions existing osr(1.2d0 0x384d180)/0x384d180 2014-06-09 16:22:11.969196 7f868bf5a700 10 op_submit_start 3727 2014-06-09 16:22:11.969197 7f868bf5a700 5 queue_transactions (trailing journal) 3727 2014-06-09 16:22:11.969190 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] requeue_ops 2014-06-09 16:22:11.969198 7f868bf5a700 5 queue_op 0x324e65a0 seq 3727 osr(1.2d0 0x384d180) 828 bytes (queue has 47 ops and 149377 bytes) 2014-06-09 16:22:11.969198 7f86a5610700 10 _do_op 0x324e7f80 seq 3687 r = 0, finisher 0x31412fa0 0 2014-06-09 16:22:11.969200 7f868bf5a700 10 op_submit_finish 3727 2014-06-09 16:22:11.969201 7f86a5610700 10 _finish_op 0x324e7f80 seq 3687 osr(3.2fd 0x3ed0de0)/0x3ed0de0 2014-06-09 16:22:11.969204 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc7e0 (1 active) 2014-06-09 16:22:11.969198 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.969206 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114ea0 (2 active) 2014-06-09 16:22:11.969207 7f86a5610700 5 _do_op 0xc88aea0 seq 3688 osr(0.301 0x2bd6e10)/0x2bd6e10 start 2014-06-09 16:22:11.969210 7f86a5610700 10 _do_transaction on 0x3414300 2014-06-09 16:22:11.969202 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.66a( v 213'544 (0'0,213'544] local-les=243 n=107 ec=118 les/c 243/243 242/242/222) 2014-06-09 16:22:11.969209 7f86a4e0f700 5 -- op tracker -- , seq: 680, time: 2014-06-09 16:22:11.969203, event: done, op: osd_op(client.4572.0:30636 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3313664~516096] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.969214 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969209 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.893200 7 0.000376 2014-06-09 16:22:11.969221 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.969230 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969234 7f868bf5a700 5 queue_transactions existing osr(3.66a 0x3ed19d0)/0x3ed19d0 2014-06-09 16:22:11.969236 7f868bf5a700 10 op_submit_start 3728 2014-06-09 16:22:11.969237 7f868bf5a700 5 queue_transactions (trailing journal) 3728 2014-06-09 16:22:11.969231 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.969238 7f868bf5a700 5 queue_op 0x324e6660 seq 3728 osr(3.66a 0x3ed19d0) 828 bytes (queue has 47 ops and 149377 bytes) 2014-06-09 16:22:11.969240 7f868bf5a700 10 op_submit_finish 3728 2014-06-09 16:22:11.969242 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:11.969247 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969235 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 208 same_interval_since 242 2014-06-09 16:22:11.969242 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.74( v 214'550 lc 214'549 (0'0,214'550] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.969252 7f868ff62700 5 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.969252 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.969254 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.969261 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3688.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969265 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969260 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.969263 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.969267 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.969274 7f86a5610700 10 _do_op 0xc88aea0 seq 3688 r = 0, finisher 0x32480880 0 2014-06-09 16:22:11.969266 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.969277 7f86a5610700 10 _finish_op 0xc88aea0 seq 3688 osr(0.301 0x2bd6e10)/0x2bd6e10 2014-06-09 16:22:11.969272 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.969280 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114ea0 (1 active) 2014-06-09 16:22:11.969282 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa120 (2 active) 2014-06-09 16:22:11.969284 7f86a5610700 5 _do_op 0x324e6c60 seq 3689 osr(2.f4 0x2ed0790)/0x2ed0790 start 2014-06-09 16:22:11.969285 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.969286 7f86a5610700 10 _do_transaction on 0xc0d8c00 2014-06-09 16:22:11.969278 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:11.969290 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969289 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:11.969291 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] op_applied: 15 2014-06-09 16:22:11.969298 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.969296 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.74( v 214'550 (0'0,214'550] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:981 2014-06-09 16:22:11.969310 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969309 7f868ff62700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.969308 7f86a4e0f700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.969301, event: op_applied, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.969313 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.708( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.969314 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] op_applied on primary on version 243'646 2014-06-09 16:22:11.969324 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969319 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:190 2014-06-09 16:22:11.969325 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.969325 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] repop_all_applied: repop tid 15 all applied 2014-06-09 16:22:11.969336 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3689.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969333 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.969340 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969340 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.969348 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.969334 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] eval_repop repgather(0xc2c6240 243'646 rep_tid=15 committed?=1 applied?=1 lock=0 op=osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.969349 7f86a5610700 10 _do_op 0x324e6c60 seq 3689 r = 0, finisher 0x33a7400 0 2014-06-09 16:22:11.969352 7f868ff62700 5 queue_transactions existing osr(1.37e 0x384d5d0)/0x384d5d0 2014-06-09 16:22:11.969352 7f868bf5a700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.969353 7f86a5610700 10 _finish_op 0x324e6c60 seq 3689 osr(2.f4 0x2ed0790)/0x2ed0790 2014-06-09 16:22:11.969354 7f868ff62700 10 op_submit_start 3729 2014-06-09 16:22:11.969355 7f868ff62700 5 queue_transactions (trailing journal) 3729 2014-06-09 16:22:11.969355 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa120 (1 active) 2014-06-09 16:22:11.969357 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0240 (2 active) 2014-06-09 16:22:11.969357 7f868ff62700 5 queue_op 0x24105aa0 seq 3729 osr(1.37e 0x384d5d0) 861 bytes (queue has 46 ops and 148584 bytes) 2014-06-09 16:22:11.969359 7f868ff62700 10 op_submit_finish 3729 2014-06-09 16:22:11.969359 7f86a5610700 5 _do_op 0x16fd4c60 seq 3690 osr(1.2b0 0x384d1b0)/0x384d1b0 start 2014-06-09 16:22:11.969362 7f86a5610700 10 _do_transaction on 0xbeba6c0 2014-06-09 16:22:11.969350 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] log_op_stats osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4 inb 364821 outb 0 rlat 0.000000 lat 1.211557 2014-06-09 16:22:11.969367 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969362 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.708( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:155 2014-06-09 16:22:11.969362 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.969364 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] publish_stats_to_osd 243:1037 2014-06-09 16:22:11.969375 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969372 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.969373 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] requeue_ops 2014-06-09 16:22:11.969380 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.969387 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.969382 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] removing repgather(0xc2c6240 243'646 rep_tid=15 committed?=1 applied?=1 op=osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.969394 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.969400 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969378 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:11.969401 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.969396 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] q front is repgather(0xc2c6240 243'646 rep_tid=15 committed?=1 applied?=1 op=osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.969414 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3690.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969418 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969411 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.969409 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] remove_repop repgather(0xc2c6240 243'646 rep_tid=15 committed?=1 applied?=1 op=osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.969424 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969420 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.969428 7f868bf5a700 5 queue_transactions existing osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 2014-06-09 16:22:11.969423 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.60( v 243'646 (0'0,243'646] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 243'645 mlcod 243'645 active+clean] requeue_ops 2014-06-09 16:22:11.969427 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.73a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.969433 7f868bf5a700 10 op_submit_start 3730 2014-06-09 16:22:11.969434 7f868bf5a700 5 queue_transactions (trailing journal) 3730 2014-06-09 16:22:11.969435 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.969434 7f86a5610700 10 _do_op 0x16fd4c60 seq 3690 r = 0, finisher 0x12051740 0 2014-06-09 16:22:11.969435 7f868bf5a700 5 queue_op 0x2fcaf980 seq 3730 osr(3.2b6 0x3ed0ee0) 828 bytes (queue has 47 ops and 149412 bytes) 2014-06-09 16:22:11.969437 7f86a5610700 10 _finish_op 0x16fd4c60 seq 3690 osr(1.2b0 0x384d1b0)/0x384d1b0 2014-06-09 16:22:11.969440 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0240 (1 active) 2014-06-09 16:22:11.969442 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f200 (2 active) 2014-06-09 16:22:11.969447 7f86a5610700 5 _do_op 0x324f40c0 seq 3691 osr(2.e6 0x2ed07d0)/0x2ed07d0 start 2014-06-09 16:22:11.969449 7f868bf5a700 10 op_submit_finish 3730 2014-06-09 16:22:11.969438 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.969450 7f86a5610700 10 _do_transaction on 0xc22da40 2014-06-09 16:22:11.969446 7f86a4e0f700 5 -- op tracker -- , seq: 704, time: 2014-06-09 16:22:11.969441, event: done, op: osd_op(client.4572.0:30637 rbd_data.117c6b8b4567.0000000000001472 [set-alloc-hint object_size 4194304 write_size 4194304,write 3829760~364544] 3.5a3860 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.969455 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969452 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.969453 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.699( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.969461 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x21967000,0x21966f00,0x21966d00,0x26aa9c00,0x26aa9b00,0x26aa9a00 2014-06-09 16:22:11.969479 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969483 7f868bf5a700 5 queue_transactions existing osr(2.699 0x2ed1f10)/0x2ed1f10 2014-06-09 16:22:11.969485 7f868bf5a700 10 op_submit_start 3731 2014-06-09 16:22:11.969486 7f868bf5a700 5 queue_transactions (trailing journal) 3731 2014-06-09 16:22:11.969475 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.450( v 221'627 lc 219'626 (0'0,221'627] local-les=243 n=121 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 crt=221'627 lcod 0'0 inactive m=1] _activate_committed 243 telling primary 2014-06-09 16:22:11.969487 7f868bf5a700 5 queue_op 0x2fcaeea0 seq 3731 osr(2.699 0x2ed1f10) 828 bytes (queue has 47 ops and 149379 bytes) 2014-06-09 16:22:11.969490 7f868bf5a700 10 op_submit_finish 3731 2014-06-09 16:22:11.969490 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969484 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.969492 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.969494 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.969492 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.7a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.969503 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3691.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969507 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969501 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.969503 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.450( v 221'627 lc 219'626 (0'0,221'627] local-les=243 n=121 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=221'627 lcod 0'0 active m=1] flushed 2014-06-09 16:22:11.969516 7f86a5610700 10 _do_op 0x324f40c0 seq 3691 r = 0, finisher 0xbd698e0 0 2014-06-09 16:22:11.969519 7f86a5610700 10 _finish_op 0x324f40c0 seq 3691 osr(2.e6 0x2ed07d0)/0x2ed07d0 2014-06-09 16:22:11.969520 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969513 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1022 2014-06-09 16:22:11.969522 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f200 (1 active) 2014-06-09 16:22:11.969523 7f868bf5a700 5 queue_transactions existing osr(1.7a 0x384c510)/0x384c510 2014-06-09 16:22:11.969523 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287320 (2 active) 2014-06-09 16:22:11.969526 7f868bf5a700 10 op_submit_start 3732 2014-06-09 16:22:11.969527 7f868bf5a700 5 queue_transactions (trailing journal) 3732 2014-06-09 16:22:11.969526 7f86a5610700 5 _do_op 0x30d96420 seq 3692 osr(3.300 0x3ed0df0)/0x3ed0df0 start 2014-06-09 16:22:11.969528 7f86a5610700 10 _do_transaction on 0xc0d9200 2014-06-09 16:22:11.969528 7f868bf5a700 5 queue_op 0x2fcafe00 seq 3732 osr(1.7a 0x384c510) 826 bytes (queue has 47 ops and 149379 bytes) 2014-06-09 16:22:11.969530 7f868bf5a700 10 op_submit_finish 3732 2014-06-09 16:22:11.969523 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.969533 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969530 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.969532 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.969533 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.122( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.969541 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.969543 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.969557 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e130 (19 active) 2014-06-09 16:22:11.969559 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969555 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.4c7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] flushed 2014-06-09 16:22:11.969562 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.969562 7f868bf5a700 5 queue_transactions existing osr(1.122 0x384c890)/0x384c890 2014-06-09 16:22:11.969565 7f868bf5a700 10 op_submit_start 3733 2014-06-09 16:22:11.969566 7f868bf5a700 5 queue_transactions (trailing journal) 3733 2014-06-09 16:22:11.969567 7f868bf5a700 5 queue_op 0x2fcae660 seq 3733 osr(1.122 0x384c890) 828 bytes (queue has 48 ops and 150207 bytes) 2014-06-09 16:22:11.969567 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969569 7f868bf5a700 10 op_submit_finish 3733 2014-06-09 16:22:11.969566 7f868ff62700 10 osd.26 243 dequeue_op 0xdfa7e00 prio 196 cost 0 latency 0.001726 osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) v2 pg pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=213'612 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.969572 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.117( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.969582 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3692.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969586 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969578 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] op_applied: 14 2014-06-09 16:22:11.969583 7f868ff62700 5 -- op tracker -- , seq: 3378, time: 2014-06-09 16:22:11.969579, event: reached_pg, op: osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.969595 7f86a5610700 10 _do_op 0x30d96420 seq 3692 r = 0, finisher 0xc1697a0 0 2014-06-09 16:22:11.969599 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969598 7f86a5610700 10 _finish_op 0x30d96420 seq 3692 osr(3.300 0x3ed0df0)/0x3ed0df0 2014-06-09 16:22:11.969591 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=213'612 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0xdfa7e00 2014-06-09 16:22:11.969595 7f86a4e0f700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.969589, event: op_applied, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.969601 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287320 (1 active) 2014-06-09 16:22:11.969602 7f868bf5a700 5 queue_transactions existing osr(0.117 0x2bd66a0)/0x2bd66a0 2014-06-09 16:22:11.969603 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287b00 (2 active) 2014-06-09 16:22:11.969604 7f868bf5a700 10 op_submit_start 3734 2014-06-09 16:22:11.969605 7f868bf5a700 5 queue_transactions (trailing journal) 3734 2014-06-09 16:22:11.969605 7f86a5610700 5 _do_op 0xc88a840 seq 3693 osr(0.35b 0x2bd6fe0)/0x2bd6fe0 start 2014-06-09 16:22:11.969607 7f86a5610700 10 _do_transaction on 0xc31fe00 2014-06-09 16:22:11.969606 7f868bf5a700 5 queue_op 0x2fcafe60 seq 3734 osr(0.117 0x2bd66a0) 828 bytes (queue has 48 ops and 150174 bytes) 2014-06-09 16:22:11.969609 7f868bf5a700 10 op_submit_finish 3734 2014-06-09 16:22:11.969606 7f868ff62700 5 -- op tracker -- , seq: 3378, time: 2014-06-09 16:22:11.969602, event: started, op: osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.969602 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] op_applied on primary on version 243'596 2014-06-09 16:22:11.969611 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969611 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] repop_all_applied: repop tid 14 all applied 2014-06-09 16:22:11.969611 7f868ff62700 7 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=213'612 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 20 op ack_type 4 from 5 2014-06-09 16:22:11.969611 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:11.969628 7f868ff62700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.969623, event: sub_op_commit_rec, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.969620 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] eval_repop repgather(0x3128000 243'596 rep_tid=14 committed?=1 applied?=1 lock=0 op=osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.969635 7f868ff62700 10 osd.26 243 dequeue_op 0xdfa7e00 finish 2014-06-09 16:22:11.969639 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.969643 7f868bf5a700 5 queue_transactions existing osr(3.139 0x3ed0780)/0x3ed0780 2014-06-09 16:22:11.969642 7f868ff62700 5 -- op tracker -- , seq: 3378, time: 2014-06-09 16:22:11.969638, event: done, op: osd_sub_op_reply(client.4577.0:37476 3.ff f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.969646 7f868bf5a700 10 op_submit_start 3735 2014-06-09 16:22:11.969646 7f868bf5a700 5 queue_transactions (trailing journal) 3735 2014-06-09 16:22:11.969645 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969635 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] log_op_stats osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4 inb 1548565 outb 0 rlat 0.000000 lat 1.487999 2014-06-09 16:22:11.969647 7f868bf5a700 5 queue_op 0x2fcae900 seq 3735 osr(3.139 0x3ed0780) 828 bytes (queue has 49 ops and 151002 bytes) 2014-06-09 16:22:11.969650 7f868bf5a700 10 op_submit_finish 3735 2014-06-09 16:22:11.969652 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.969650 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] publish_stats_to_osd 243:973 2014-06-09 16:22:11.969658 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3693.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.969661 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.969652 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.2b( v 214'623 lc 214'618 (0'0,214'623] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.969660 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'594 active+clean] requeue_ops 2014-06-09 16:22:11.969670 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e130 (20 active) 2014-06-09 16:22:11.969670 7f86a5610700 10 _do_op 0xc88a840 seq 3693 r = 0, finisher 0x324f87c0 0 2014-06-09 16:22:11.969673 7f86a5610700 10 _finish_op 0xc88a840 seq 3693 osr(0.35b 0x2bd6fe0)/0x2bd6fe0 2014-06-09 16:22:11.969666 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.969675 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287b00 (1 active) 2014-06-09 16:22:11.969677 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52fc20 (2 active) 2014-06-09 16:22:11.969679 7f86a5610700 5 _do_op 0x324f4c00 seq 3694 osr(2.606 0x2ed1d30)/0x2ed1d30 start 2014-06-09 16:22:11.969681 7f86a5610700 10 _do_transaction on 0x34f03c0 2014-06-09 16:22:11.969669 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'595 active+clean] removing repgather(0x3128000 243'596 rep_tid=14 committed?=1 applied?=1 op=osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.969675 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.969673 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.2f2( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.969686 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.969684 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.974470 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.969696 7f8690f64700 10 op_submit_start 3736 2014-06-09 16:22:11.969684 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'595 active+clean] q front is repgather(0x3128000 243'596 rep_tid=14 committed?=1 applied?=1 op=osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.974477 7f8690f64700 5 queue_transactions (trailing journal) 3736 2014-06-09 16:22:11.974481 7f8690f64700 5 queue_op 0x2fcaf1a0 seq 3736 osr(1.348 0x384d310) 828 bytes (queue has 50 ops and 151830 bytes) 2014-06-09 16:22:11.974492 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.2b( v 214'623 (0'0,214'623] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1077 2014-06-09 16:22:11.974508 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.974483 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'595 active+clean] remove_repop repgather(0x3128000 243'596 rep_tid=14 committed?=1 applied?=1 op=osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.974510 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.1e( v 243'596 (0'0,243'596] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 243'595 mlcod 243'595 active+clean] requeue_ops 2014-06-09 16:22:11.974513 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.759( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.974524 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.974530 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.974535 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.974542 7f868bf5a700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.974551 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.759( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.974559 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.974562 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:11.974584 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.974588 7f868bf5a700 5 queue_transactions existing osr(3.136 0x3ed07a0)/0x3ed07a0 2014-06-09 16:22:11.969705 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.969705 7f869af78700 5 -- op tracker -- , seq: 3377, time: 2014-06-09 16:22:11.967681, event: started, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.974597 7f868ff62700 5 queue_transactions existing osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 2014-06-09 16:22:11.969720 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.969770 7f86a5e11700 10 _finish_op 0xc64b320 seq 3675 osr(3.31f 0x3ed0d50)/0x3ed0d50 2014-06-09 16:22:11.969788 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314134a0,0x31435cc0,0xc411820,0x30628080,0x327f23c0,0x32481400,0xc5a7120,0x32481fc0,0xc168ac0,0x12051f40,0x31434d40,0x4eb9b00,0x324f85c0] 2014-06-09 16:22:11.974618 7f869af78700 5 -- op tracker -- , seq: 3377, time: 2014-06-09 16:22:11.974616, event: done, op: pg_info(1 pgs e243:2.409) 2014-06-09 16:22:11.974630 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974633 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974624 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.974634 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.974636 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3694.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.974642 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.974646 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974648 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974643 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.2b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:11.974649 7f869af78700 20 osd.26 243 _dispatch 0x33441500 pg_info(1 pgs e243:2.492) v4 2014-06-09 16:22:11.974654 7f869af78700 5 -- op tracker -- , seq: 3380, time: 2014-06-09 16:22:11.963195, event: header_read, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.974652 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.974662 7f86a5610700 10 _do_op 0x324f4c00 seq 3694 r = 0, finisher 0x3050ae20 0 2014-06-09 16:22:11.974661 7f869af78700 5 -- op tracker -- , seq: 3380, time: 2014-06-09 16:22:11.963195, event: throttled, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.974661 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.974668 7f869af78700 5 -- op tracker -- , seq: 3380, time: 2014-06-09 16:22:11.963203, event: all_read, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.974677 7f869af78700 5 -- op tracker -- , seq: 3380, time: 2014-06-09 16:22:11.974646, event: dispatched, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.974673 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.300( v 213'576 (0'0,213'576] local-les=243 n=119 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:974 2014-06-09 16:22:11.974682 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314134a0,0x31435cc0,0xc411820,0x30628080,0x327f23c0,0x32481400,0xc5a7120,0x32481fc0,0xc168ac0,0x12051f40,0x31434d40,0x4eb9b00,0x324f85c0] 2014-06-09 16:22:11.974686 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x29283fe0] 2014-06-09 16:22:11.974687 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x29283fe0] 2014-06-09 16:22:11.974689 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.974690 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.974683 7f869af78700 5 -- op tracker -- , seq: 3380, time: 2014-06-09 16:22:11.974681, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.969802 7f868e75f700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.974695 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.492) v4 from osd.2 2014-06-09 16:22:11.974698 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x33441500 2014-06-09 16:22:11.974705 7f868c75b700 2 waiting 51 > 50 ops || 170612 > 104857600 2014-06-09 16:22:11.974704 7f86a4e0f700 5 -- op tracker -- , seq: 434, time: 2014-06-09 16:22:11.974535, event: done, op: osd_op(client.4571.0:35950 rbd_data.117b6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 1384448~1548288] 3.78ae781e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.974711 7f8692f68700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.974717 7f868cf5c700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.974725 7f8692767700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.974730 7f868d75d700 2 waiting 51 > 50 ops || 1186410 > 104857600 2014-06-09 16:22:11.974737 7f8691765700 2 waiting 51 > 50 ops || 152688 > 104857600 2014-06-09 16:22:11.974740 7f869af78700 5 -- op tracker -- , seq: 3380, time: 2014-06-09 16:22:11.974702, event: started, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.974755 7f869af78700 5 -- op tracker -- , seq: 3380, time: 2014-06-09 16:22:11.974754, event: done, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.974761 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974762 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974754 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 201 same_interval_since 242 2014-06-09 16:22:11.974770 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974771 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974772 7f869af78700 20 osd.26 243 _dispatch 0x150e1880 pg_info(1 pgs e243:1.26c) v4 2014-06-09 16:22:11.974775 7f869af78700 5 -- op tracker -- , seq: 3382, time: 2014-06-09 16:22:11.957178, event: header_read, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:11.974779 7f869af78700 5 -- op tracker -- , seq: 3382, time: 2014-06-09 16:22:11.957179, event: throttled, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:11.974773 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.974783 7f869af78700 5 -- op tracker -- , seq: 3382, time: 2014-06-09 16:22:11.957186, event: all_read, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:11.974786 7f869af78700 5 -- op tracker -- , seq: 3382, time: 2014-06-09 16:22:11.974770, event: dispatched, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:11.974791 7f869af78700 5 -- op tracker -- , seq: 3382, time: 2014-06-09 16:22:11.974790, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:11.974794 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.26c) v4 from osd.4 2014-06-09 16:22:11.974795 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e1880 2014-06-09 16:22:11.974789 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.974798 7f869af78700 5 -- op tracker -- , seq: 3382, time: 2014-06-09 16:22:11.974797, event: started, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:11.974807 7f869af78700 5 -- op tracker -- , seq: 3382, time: 2014-06-09 16:22:11.974806, event: done, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:11.974811 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974812 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974820 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974820 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974821 7f869af78700 20 osd.26 243 _dispatch 0x76e2700 pg_info(1 pgs e243:3.4c5) v4 2014-06-09 16:22:11.974813 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.974824 7f869af78700 5 -- op tracker -- , seq: 3383, time: 2014-06-09 16:22:11.959004, event: header_read, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.974829 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.974827 7f869af78700 5 -- op tracker -- , seq: 3383, time: 2014-06-09 16:22:11.959004, event: throttled, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.974833 7f869af78700 5 -- op tracker -- , seq: 3383, time: 2014-06-09 16:22:11.959015, event: all_read, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.974837 7f869af78700 5 -- op tracker -- , seq: 3383, time: 2014-06-09 16:22:11.974819, event: dispatched, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.974841 7f869af78700 5 -- op tracker -- , seq: 3383, time: 2014-06-09 16:22:11.974840, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.974845 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c5) v4 from osd.7 2014-06-09 16:22:11.974846 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e2700 2014-06-09 16:22:11.974849 7f869af78700 5 -- op tracker -- , seq: 3383, time: 2014-06-09 16:22:11.974848, event: started, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.974845 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.1fb( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.974858 7f869af78700 5 -- op tracker -- , seq: 3383, time: 2014-06-09 16:22:11.974857, event: done, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.974863 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974863 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974870 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974871 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974863 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.772( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.974872 7f869af78700 20 osd.26 243 _dispatch 0x333c3a40 pg_info(1 pgs e243:0.6b6) v4 2014-06-09 16:22:11.974874 7f869af78700 5 -- op tracker -- , seq: 3384, time: 2014-06-09 16:22:11.972239, event: header_read, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.974877 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.974878 7f869af78700 5 -- op tracker -- , seq: 3384, time: 2014-06-09 16:22:11.972240, event: throttled, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.974882 7f869af78700 5 -- op tracker -- , seq: 3384, time: 2014-06-09 16:22:11.972249, event: all_read, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.974885 7f869af78700 5 -- op tracker -- , seq: 3384, time: 2014-06-09 16:22:11.974870, event: dispatched, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.974888 7f869af78700 5 -- op tracker -- , seq: 3384, time: 2014-06-09 16:22:11.974887, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.974891 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6b6) v4 from osd.3 2014-06-09 16:22:11.974892 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x333c3a40 2014-06-09 16:22:11.974895 7f869af78700 5 -- op tracker -- , seq: 3384, time: 2014-06-09 16:22:11.974894, event: started, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.974888 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.772( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] flushed 2014-06-09 16:22:11.974903 7f869af78700 5 -- op tracker -- , seq: 3384, time: 2014-06-09 16:22:11.974902, event: done, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.974907 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974908 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974918 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974919 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974920 7f869af78700 20 osd.26 243 _dispatch 0x30c77180 pg_info(1 pgs e243:0.70e) v4 2014-06-09 16:22:11.974922 7f869af78700 5 -- op tracker -- , seq: 3385, time: 2014-06-09 16:22:11.959324, event: header_read, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.974928 7f869af78700 5 -- op tracker -- , seq: 3385, time: 2014-06-09 16:22:11.959324, event: throttled, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.974932 7f869af78700 5 -- op tracker -- , seq: 3385, time: 2014-06-09 16:22:11.959336, event: all_read, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.974935 7f869af78700 5 -- op tracker -- , seq: 3385, time: 2014-06-09 16:22:11.974918, event: dispatched, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.974940 7f869af78700 5 -- op tracker -- , seq: 3385, time: 2014-06-09 16:22:11.974939, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.974942 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.70e) v4 from osd.1 2014-06-09 16:22:11.974944 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c77180 2014-06-09 16:22:11.974946 7f869af78700 5 -- op tracker -- , seq: 3385, time: 2014-06-09 16:22:11.974945, event: started, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.974956 7f869af78700 5 -- op tracker -- , seq: 3385, time: 2014-06-09 16:22:11.974955, event: done, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.974965 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974966 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974958 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.974973 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.974974 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.974974 7f869af78700 20 osd.26 243 _dispatch 0x31745c00 pg_info(1 pgs e243:2.55b) v4 2014-06-09 16:22:11.974977 7f869af78700 5 -- op tracker -- , seq: 3386, time: 2014-06-09 16:22:11.957358, event: header_read, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.974973 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.974982 7f869af78700 5 -- op tracker -- , seq: 3386, time: 2014-06-09 16:22:11.957358, event: throttled, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.974986 7f869af78700 5 -- op tracker -- , seq: 3386, time: 2014-06-09 16:22:11.957370, event: all_read, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.974989 7f869af78700 5 -- op tracker -- , seq: 3386, time: 2014-06-09 16:22:11.974973, event: dispatched, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.974993 7f869af78700 5 -- op tracker -- , seq: 3386, time: 2014-06-09 16:22:11.974992, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.974995 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.55b) v4 from osd.9 2014-06-09 16:22:11.974997 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31745c00 2014-06-09 16:22:11.974999 7f869af78700 5 -- op tracker -- , seq: 3386, time: 2014-06-09 16:22:11.974998, event: started, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.974996 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[0.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.975009 7f869af78700 5 -- op tracker -- , seq: 3386, time: 2014-06-09 16:22:11.975008, event: done, op: pg_info(1 pgs e243:2.55b) 2014-06-09 16:22:11.975014 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975015 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975021 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975022 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975023 7f869af78700 20 osd.26 243 _dispatch 0x31388540 pg_info(1 pgs e243:3.4c8) v4 2014-06-09 16:22:11.975025 7f869af78700 5 -- op tracker -- , seq: 3387, time: 2014-06-09 16:22:11.961211, event: header_read, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.975029 7f869af78700 5 -- op tracker -- , seq: 3387, time: 2014-06-09 16:22:11.961212, event: throttled, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.975034 7f869af78700 5 -- op tracker -- , seq: 3387, time: 2014-06-09 16:22:11.961223, event: all_read, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.975038 7f869af78700 5 -- op tracker -- , seq: 3387, time: 2014-06-09 16:22:11.975021, event: dispatched, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.975041 7f869af78700 5 -- op tracker -- , seq: 3387, time: 2014-06-09 16:22:11.975040, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.975054 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c8) v4 from osd.8 2014-06-09 16:22:11.975057 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31388540 2014-06-09 16:22:11.975062 7f86af56b700 5 -- op tracker -- , seq: 3381, time: 2014-06-09 16:22:11.968246, event: header_read, op: osd_op(client.4541.0:32045 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 348160~2580480] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.975035 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6ef20,0x31413aa0,0x31456ae0,0x3042aac0,0x31478520,0x2d738fe0,0x30d61dc0,0x3042a380,0x3042b8a0,0xbd7d8a0,0x3042a0a0,0x4eb8aa0,0x3042b3a0,0x30e18880,0x30629020,0x30d618a0,0xc410a60,0x314790e0,0xc6e9fc0,0x30e19680,0xbccde40,0x318c2fa0,0x318c3720,0xc6e80e0,0xc1681c0,0xbd71340,0x30508660,0xbd7ce00,0x2e0ffc0,0x30e19940,0xbd5b6a0,0x3042ace0,0x3050a340,0x30e19040,0x18742920,0x30e184c0,0x3006ce20,0x2d738ea0,0x3006d0a0,0x4eb92a0,0xc6e8680,0x2e0ed20,0x318c33a0,0x318c3c60,0x318c2200,0x2e0e8a0,0x30c54780,0x3006d160,0xc5a78a0,0x31791fa0,0xc169460,0x314124e0,0xcd0fbc0,0xbd726e0,0x13965240,0xbebe740,0xcd32720,0xc5a7000,0xb9c1fa0,0x30c55bc0,0x30c55480,0x32074580,0x318c3140,0x4eb8920,0xbce7360,0x32480340,0x3281bd80,0x3050bce0,0x32073fe0,0x32073a80,0x32073c80,0x1a003080,0x320736c0,0x307f1580,0xcd32280,0x32073840,0x314adc80,0x281b6a00,0x3206e980,0x3206f400,0x32072480,0xd9fbbe0,0xd9fb3a0,0xd9fb1a0,0x3006cee0,0x3006cae0,0x3006c4e0,0x32072ac0,0x281b6d80,0x32072940,0x3206f840,0xcd0f480,0xcd0f040,0xcd9fe00,0xbd6d340,0xcd9f020,0xbd6b780,0x307f0200,0xbdf0260,0xc168d40,0xbd679a0,0xc168920,0xcc95ea0,0x318c3200,0x3042a540,0xbd69ca0,0x31412240,0xbd730e0,0xcc82aa0,0x3042a420,0x2e0e060,0x318c22c0,0x30629b00,0xbd6c120,0x2e0e120,0x31435740,0xcc94060,0x318c2a80,0xc2d74e0,0xccf91c0,0xbd606e0,0x3206ef00,0x318c3f00,0xbd7c9c0,0x306cf080,0x324d1720,0xbcc82a0,0x306ce180,0xbd5e000,0xbd5dca0,0xbd5daa0,0xcc95720,0x17001e20,0x314353c0,0xbd5ce80,0xbd5cc80,0x281b7680,0xbccc260,0x314128a0,0xbd67280,0x3061fee0,0xbfcfea0,0xbd76240,0x31412580,0xbd5ff20,0xbd5c300,0x318c2b40,0x18742a00,0x15a3f1e0,0x3061edc0,0x3050b040,0xbd5c620,0xbd5c4c0,0xbd5caa0,0xbd5c280,0xbd5e5e0,0xbe30300,0xccf88e0,0xc40c680,0xc1695e0,0xbd5c920,0x324f9ae0,0x3251cbe0,0x30628040,0x314136c0,0x17fb76c0,0x139654c0,0x3251cf60,0x17fb7580,0xbd607a0,0x189aac20,0x3251d9e0,0x3206e880,0xd9fb9c0,0x3042b6e0,0x4eb9060,0xc168120,0xcc82fc0,0xbd73900,0xcc82360] 2014-06-09 16:22:11.975074 7f86af56b700 5 -- op tracker -- , seq: 3381, time: 2014-06-09 16:22:11.968247, event: throttled, op: osd_op(client.4541.0:32045 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 348160~2580480] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.975071 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x4eb9240,0xbd762c0,0xbd6a460,0xbd60260,0x2d739380,0xbfcf500,0xcc95dc0,0x17fb7400,0x31791f40,0xbd798a0,0x17fb7a80,0x3062bc20,0xbd5efc0,0xcd0e3c0,0x2d7381e0,0xbd789c0,0xc2d6e20,0x31456a60,0x1e08efa0,0x3042b280,0xbd605a0,0x17fb7900,0xd9fb860,0xd9fb6a0,0xd9fb560,0x324f86e0,0xbccd8a0,0xcd334c0,0xbd71b40,0x3006dfc0,0xbd75bc0,0x15a3fa80,0x318c2720,0x3281bdc0,0xcd9ea40,0x324f8260,0x324f8be0,0x1322d960,0x3251ca20,0x33a7f60,0x3006c760,0x31412e60,0x31435e40,0xbd75880,0xbd6cb60,0x327f3700,0x32481600,0x31412fa0,0x32480880,0x33a7400,0x12051740,0xbd698e0,0xc1697a0,0x324f87c0] 2014-06-09 16:22:11.975085 7f86af56b700 5 -- op tracker -- , seq: 3381, time: 2014-06-09 16:22:11.974325, event: all_read, op: osd_op(client.4541.0:32045 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 348160~2580480] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.975096 7f86af56b700 5 -- op tracker -- , seq: 3381, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32045 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 348160~2580480] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.975103 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:11.975104 7f869af78700 5 -- op tracker -- , seq: 3387, time: 2014-06-09 16:22:11.975060, event: started, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.975121 7f869af78700 5 -- op tracker -- , seq: 3387, time: 2014-06-09 16:22:11.975119, event: done, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.975129 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975130 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975125 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7b00 prio 63 cost 2580480 latency 0.006879 osd_op(client.4541.0:32045 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 348160~2580480] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.975142 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975143 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975144 7f869af78700 20 osd.26 243 _dispatch 0x76e2380 pg_info(1 pgs e243:1.384) v4 2014-06-09 16:22:11.975149 7f869af78700 5 -- op tracker -- , seq: 3388, time: 2014-06-09 16:22:11.957243, event: header_read, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.975157 7f869af78700 5 -- op tracker -- , seq: 3388, time: 2014-06-09 16:22:11.957244, event: throttled, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.975163 7f869af78700 5 -- op tracker -- , seq: 3388, time: 2014-06-09 16:22:11.957257, event: all_read, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.975161 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] op_applied: 16 2014-06-09 16:22:11.975175 7f869af78700 5 -- op tracker -- , seq: 3388, time: 2014-06-09 16:22:11.975141, event: dispatched, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.975184 7f869af78700 5 -- op tracker -- , seq: 3388, time: 2014-06-09 16:22:11.975182, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.975189 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.384) v4 from osd.6 2014-06-09 16:22:11.975192 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e2380 2014-06-09 16:22:11.975196 7f869af78700 5 -- op tracker -- , seq: 3388, time: 2014-06-09 16:22:11.975195, event: started, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.975211 7f869af78700 5 -- op tracker -- , seq: 3388, time: 2014-06-09 16:22:11.975209, event: done, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.975218 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975220 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975231 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975232 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975233 7f869af78700 20 osd.26 243 _dispatch 0x76e36c0 pg_info(1 pgs e243:0.47d) v4 2014-06-09 16:22:11.975238 7f869af78700 5 -- op tracker -- , seq: 3389, time: 2014-06-09 16:22:11.955583, event: header_read, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.975244 7f869af78700 5 -- op tracker -- , seq: 3389, time: 2014-06-09 16:22:11.955583, event: throttled, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.975250 7f869af78700 5 -- op tracker -- , seq: 3389, time: 2014-06-09 16:22:11.955593, event: all_read, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.975273 7f869af78700 5 -- op tracker -- , seq: 3389, time: 2014-06-09 16:22:11.975231, event: dispatched, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.975279 7f869af78700 5 -- op tracker -- , seq: 3389, time: 2014-06-09 16:22:11.975278, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.975285 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.47d) v4 from osd.0 2014-06-09 16:22:11.975287 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e36c0 2014-06-09 16:22:11.975292 7f869af78700 5 -- op tracker -- , seq: 3389, time: 2014-06-09 16:22:11.975290, event: started, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.975306 7f869af78700 5 -- op tracker -- , seq: 3389, time: 2014-06-09 16:22:11.975304, event: done, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.975314 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975316 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975328 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975329 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975331 7f869af78700 20 osd.26 243 _dispatch 0x30fcc700 pg_info(1 pgs e243:1.77c) v4 2014-06-09 16:22:11.975335 7f869af78700 5 -- op tracker -- , seq: 3390, time: 2014-06-09 16:22:11.968691, event: header_read, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.975344 7f869af78700 5 -- op tracker -- , seq: 3390, time: 2014-06-09 16:22:11.968692, event: throttled, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.975348 7f869af78700 5 -- op tracker -- , seq: 3390, time: 2014-06-09 16:22:11.968705, event: all_read, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.975351 7f869af78700 5 -- op tracker -- , seq: 3390, time: 2014-06-09 16:22:11.975328, event: dispatched, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.975354 7f869af78700 5 -- op tracker -- , seq: 3390, time: 2014-06-09 16:22:11.975353, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.975373 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.77c) v4 from osd.5 2014-06-09 16:22:11.975377 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcc700 2014-06-09 16:22:11.975382 7f869af78700 5 -- op tracker -- , seq: 3390, time: 2014-06-09 16:22:11.975380, event: started, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.975397 7f869af78700 5 -- op tracker -- , seq: 3390, time: 2014-06-09 16:22:11.975395, event: done, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.975406 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975408 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975417 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975418 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975418 7f869af78700 20 osd.26 243 _dispatch 0x12c08fc0 pg_info(1 pgs e243:2.492) v4 2014-06-09 16:22:11.975421 7f869af78700 5 -- op tracker -- , seq: 3391, time: 2014-06-09 16:22:11.963216, event: header_read, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.975425 7f869af78700 5 -- op tracker -- , seq: 3391, time: 2014-06-09 16:22:11.963216, event: throttled, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.975429 7f869af78700 5 -- op tracker -- , seq: 3391, time: 2014-06-09 16:22:11.963228, event: all_read, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.975432 7f869af78700 5 -- op tracker -- , seq: 3391, time: 2014-06-09 16:22:11.975417, event: dispatched, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.975436 7f869af78700 5 -- op tracker -- , seq: 3391, time: 2014-06-09 16:22:11.975435, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.975438 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.492) v4 from osd.2 2014-06-09 16:22:11.975440 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12c08fc0 2014-06-09 16:22:11.975443 7f869af78700 5 -- op tracker -- , seq: 3391, time: 2014-06-09 16:22:11.975442, event: started, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.975456 7f8689755700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.975466 7f8690763700 2 waiting 51 > 50 ops || 152883 > 104857600 2014-06-09 16:22:11.975469 7f869af78700 5 -- op tracker -- , seq: 3391, time: 2014-06-09 16:22:11.975466, event: done, op: pg_info(1 pgs e243:2.492) 2014-06-09 16:22:11.975472 7f868a757700 2 waiting 51 > 50 ops || 170612 > 104857600 2014-06-09 16:22:11.975477 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975479 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975481 7f868bf5a700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.975490 7f868ff62700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.975491 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975492 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975493 7f869af78700 20 osd.26 243 _dispatch 0x30ef6e00 pg_info(1 pgs e243:1.6b6) v4 2014-06-09 16:22:11.975499 7f869af78700 5 -- op tracker -- , seq: 3392, time: 2014-06-09 16:22:11.957194, event: header_read, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:11.975504 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358360 (1 active) 2014-06-09 16:22:11.975506 7f869af78700 5 -- op tracker -- , seq: 3392, time: 2014-06-09 16:22:11.957195, event: throttled, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:11.975509 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc7e0 (2 active) 2014-06-09 16:22:11.975512 7f869af78700 5 -- op tracker -- , seq: 3392, time: 2014-06-09 16:22:11.957203, event: all_read, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:11.975514 7f86a5e11700 5 _do_op 0x324f4cc0 seq 3695 osr(1.560 0x384da90)/0x384da90 start 2014-06-09 16:22:11.975518 7f86a5e11700 10 _do_transaction on 0x3419a40 2014-06-09 16:22:11.975522 7f868c75b700 2 waiting 51 > 50 ops || 170612 > 104857600 2014-06-09 16:22:11.975525 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.975518 7f869af78700 5 -- op tracker -- , seq: 3392, time: 2014-06-09 16:22:11.975491, event: dispatched, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:11.975527 7f868cf5c700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.975533 7f8692767700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.975536 7f869af78700 5 -- op tracker -- , seq: 3392, time: 2014-06-09 16:22:11.975533, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:11.975542 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6b6) v4 from osd.4 2014-06-09 16:22:11.975545 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef6e00 2014-06-09 16:22:11.975550 7f869af78700 5 -- op tracker -- , seq: 3392, time: 2014-06-09 16:22:11.975548, event: started, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:11.975537 7f868d75d700 2 waiting 51 > 50 ops || 1186410 > 104857600 2014-06-09 16:22:11.975566 7f86a4e0f700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.975175, event: op_applied, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.975577 7f869af78700 5 -- op tracker -- , seq: 3392, time: 2014-06-09 16:22:11.975565, event: done, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:11.975574 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] op_applied on primary on version 243'609 2014-06-09 16:22:11.975582 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] repop_all_applied: repop tid 16 all applied 2014-06-09 16:22:11.975586 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975588 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975587 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] eval_repop repgather(0x3e62240 243'609 rep_tid=16 committed?=1 applied?=1 lock=0 op=osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.975600 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975601 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975602 7f869af78700 20 osd.26 243 _dispatch 0x76e2fc0 pg_info(1 pgs e243:3.148) v4 2014-06-09 16:22:11.975600 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] log_op_stats osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4 inb 516375 outb 0 rlat 0.000000 lat 1.479781 2014-06-09 16:22:11.975607 7f869af78700 5 -- op tracker -- , seq: 3393, time: 2014-06-09 16:22:11.959093, event: header_read, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.975629 7f8691765700 2 waiting 51 > 50 ops || 152688 > 104857600 2014-06-09 16:22:11.975632 7f8690f64700 10 op_submit_finish 3736 2014-06-09 16:22:11.975632 7f869af78700 5 -- op tracker -- , seq: 3393, time: 2014-06-09 16:22:11.959093, event: throttled, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.975639 7f8689f56700 10 op_submit_start 3737 2014-06-09 16:22:11.975641 7f8689f56700 5 queue_transactions (trailing journal) 3737 2014-06-09 16:22:11.975639 7f869af78700 5 -- op tracker -- , seq: 3393, time: 2014-06-09 16:22:11.959105, event: all_read, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.975643 7f8689f56700 5 queue_op 0x16096180 seq 3737 osr(1.211 0x384cc90) 828 bytes (queue has 50 ops and 151830 bytes) 2014-06-09 16:22:11.975647 7f8689f56700 10 op_submit_finish 3737 2014-06-09 16:22:11.975645 7f869af78700 5 -- op tracker -- , seq: 3393, time: 2014-06-09 16:22:11.975599, event: dispatched, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.975636 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] publish_stats_to_osd 243:980 2014-06-09 16:22:11.975639 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.11d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.975651 7f869af78700 5 -- op tracker -- , seq: 3393, time: 2014-06-09 16:22:11.975649, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.975655 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.148) v4 from osd.7 2014-06-09 16:22:11.975658 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e2fc0 2014-06-09 16:22:11.975651 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] requeue_ops 2014-06-09 16:22:11.975663 7f869af78700 5 -- op tracker -- , seq: 3393, time: 2014-06-09 16:22:11.975661, event: started, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.975658 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.77( v 213'477 (0'0,213'477] local-les=243 n=92 ec=118 les/c 243/243 242/242/212) 2014-06-09 16:22:11.975661 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] removing repgather(0x3e62240 243'609 rep_tid=16 committed?=1 applied?=1 op=osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.975674 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.975679 7f8690f64700 5 queue_transactions existing osr(1.11d 0x384c8b0)/0x384c8b0 2014-06-09 16:22:11.975678 7f869af78700 5 -- op tracker -- , seq: 3393, time: 2014-06-09 16:22:11.975676, event: done, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.975684 7f8690f64700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.975686 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975674 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] q front is repgather(0x3e62240 243'609 rep_tid=16 committed?=1 applied?=1 op=osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.975687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975691 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.975696 7f8689f56700 5 queue_transactions existing osr(3.77 0x3ed0250)/0x3ed0250 2014-06-09 16:22:11.975699 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975699 7f8689f56700 2 waiting 51 > 50 ops || 152656 > 104857600 2014-06-09 16:22:11.975688 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] remove_repop repgather(0x3e62240 243'609 rep_tid=16 committed?=1 applied?=1 op=osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.975701 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975702 7f869af78700 20 osd.26 243 _dispatch 0x30fcc8c0 pg_info(1 pgs e243:0.1cd) v4 2014-06-09 16:22:11.975704 7f8691f66700 10 op_submit_start 3738 2014-06-09 16:22:11.975705 7f8691f66700 5 queue_transactions (trailing journal) 3738 2014-06-09 16:22:11.975701 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] requeue_ops 2014-06-09 16:22:11.975707 7f8691f66700 5 queue_op 0xcabd560 seq 3738 osr(3.458 0x3ed1170) 1053 bytes (queue has 50 ops and 151830 bytes) 2014-06-09 16:22:11.975710 7f8691f66700 10 op_submit_finish 3738 2014-06-09 16:22:11.975707 7f869af78700 5 -- op tracker -- , seq: 3394, time: 2014-06-09 16:22:11.972255, event: header_read, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:11.975714 7f869af78700 5 -- op tracker -- , seq: 3394, time: 2014-06-09 16:22:11.972255, event: throttled, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:11.975719 7f869af78700 5 -- op tracker -- , seq: 3394, time: 2014-06-09 16:22:11.972267, event: all_read, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:11.975724 7f868af58700 10 op_submit_start 3739 2014-06-09 16:22:11.975726 7f868af58700 5 queue_transactions (trailing journal) 3739 2014-06-09 16:22:11.975728 7f868af58700 5 queue_op 0x30d97b60 seq 3739 osr(0.5a2 0x2bd7af0) 828 bytes (queue has 50 ops and 151830 bytes) 2014-06-09 16:22:11.975731 7f868af58700 10 op_submit_finish 3739 2014-06-09 16:22:11.975728 7f869af78700 5 -- op tracker -- , seq: 3394, time: 2014-06-09 16:22:11.975698, event: dispatched, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:11.975722 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.975735 7f869af78700 5 -- op tracker -- , seq: 3394, time: 2014-06-09 16:22:11.975733, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:11.975737 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.975739 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1cd) v4 from osd.3 2014-06-09 16:22:11.975742 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcc8c0 2014-06-09 16:22:11.975736 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.975736 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.113( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.975747 7f869af78700 5 -- op tracker -- , seq: 3394, time: 2014-06-09 16:22:11.975745, event: started, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:11.975745 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.975754 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3695.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.975758 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.975754 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.975763 7f869af78700 5 -- op tracker -- , seq: 3394, time: 2014-06-09 16:22:11.975761, event: done, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:11.975767 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.975762 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.975771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975771 7f868af58700 5 queue_transactions existing osr(1.113 0x384c910)/0x384c910 2014-06-09 16:22:11.975773 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975773 7f868af58700 2 waiting 51 > 50 ops || 152658 > 104857600 2014-06-09 16:22:11.975770 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.975778 7f868ef60700 10 op_submit_start 3740 2014-06-09 16:22:11.975780 7f868ef60700 5 queue_transactions (trailing journal) 3740 2014-06-09 16:22:11.975781 7f868ef60700 5 queue_op 0x309a2780 seq 3740 osr(3.424 0x3ed1200) 1726 bytes (queue has 50 ops and 151830 bytes) 2014-06-09 16:22:11.975784 7f868ef60700 10 op_submit_finish 3740 2014-06-09 16:22:11.975784 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975786 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975787 7f869af78700 20 osd.26 243 _dispatch 0x31745880 pg_info(1 pgs e243:0.5f5) v4 2014-06-09 16:22:11.975791 7f869af78700 5 -- op tracker -- , seq: 3395, time: 2014-06-09 16:22:11.959382, event: header_read, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.975794 7f86a5e11700 10 _do_op 0x324f4cc0 seq 3695 r = 0, finisher 0x305089c0 0 2014-06-09 16:22:11.975792 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.975800 7f86a5e11700 10 _finish_op 0x324f4cc0 seq 3695 osr(1.560 0x384da90)/0x384da90 2014-06-09 16:22:11.975799 7f869af78700 5 -- op tracker -- , seq: 3395, time: 2014-06-09 16:22:11.959383, event: throttled, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.975796 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:174 2014-06-09 16:22:11.975806 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc7e0 (1 active) 2014-06-09 16:22:11.975805 7f869af78700 5 -- op tracker -- , seq: 3395, time: 2014-06-09 16:22:11.959394, event: all_read, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.975808 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300fc0 (2 active) 2014-06-09 16:22:11.975810 7f86a5e11700 5 _do_op 0x324f4d80 seq 3696 osr(2.27a 0x2ed0d90)/0x2ed0d90 start 2014-06-09 16:22:11.975805 7f8691f66700 20 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.975804 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.975812 7f86a5e11700 10 _do_transaction on 0xc2cacc0 2014-06-09 16:22:11.975811 7f869af78700 5 -- op tracker -- , seq: 3395, time: 2014-06-09 16:22:11.975784, event: dispatched, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.975817 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.975813 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.975813 7f8691f66700 20 osd.26 pg_epoch: 243 pg[0.403( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.975817 7f869af78700 5 -- op tracker -- , seq: 3395, time: 2014-06-09 16:22:11.975816, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.975822 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.975822 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5f5) v4 from osd.1 2014-06-09 16:22:11.975825 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31745880 2014-06-09 16:22:11.975821 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.975830 7f869af78700 5 -- op tracker -- , seq: 3395, time: 2014-06-09 16:22:11.975827, event: started, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.975830 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.975827 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.975836 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.975846 7f869af78700 5 -- op tracker -- , seq: 3395, time: 2014-06-09 16:22:11.975843, event: done, op: pg_info(1 pgs e243:0.5f5) 2014-06-09 16:22:11.975853 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975855 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975848 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.975855 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.975858 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.975839 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.829964 7 0.000368 2014-06-09 16:22:11.975867 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975868 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975870 7f869af78700 20 osd.26 243 _dispatch 0x33440a80 pg_info(1 pgs e243:2.1c6) v4 2014-06-09 16:22:11.975866 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.975874 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.975871 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.975875 7f869af78700 5 -- op tracker -- , seq: 3396, time: 2014-06-09 16:22:11.957398, event: header_read, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.975882 7f869af78700 5 -- op tracker -- , seq: 3396, time: 2014-06-09 16:22:11.957399, event: throttled, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.975878 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.975881 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.975889 7f869af78700 5 -- op tracker -- , seq: 3396, time: 2014-06-09 16:22:11.957409, event: all_read, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.975891 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3696.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.975895 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.975889 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.975895 7f869af78700 5 -- op tracker -- , seq: 3396, time: 2014-06-09 16:22:11.975866, event: dispatched, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.975891 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:11.975896 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.975901 7f869af78700 5 -- op tracker -- , seq: 3396, time: 2014-06-09 16:22:11.975899, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.975906 7f86a5e11700 10 _do_op 0x324f4d80 seq 3696 r = 0, finisher 0x306ce9a0 0 2014-06-09 16:22:11.975901 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.975906 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1c6) v4 from osd.9 2014-06-09 16:22:11.975909 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x33440a80 2014-06-09 16:22:11.975909 7f86a5e11700 10 _finish_op 0x324f4d80 seq 3696 osr(2.27a 0x2ed0d90)/0x2ed0d90 2014-06-09 16:22:11.975904 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.975912 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300fc0 (1 active) 2014-06-09 16:22:11.975914 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbe60 (2 active) 2014-06-09 16:22:11.975909 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.975915 7f86a5e11700 5 _do_op 0x308cb320 seq 3697 osr(2.6f 0x2ed0550)/0x2ed0550 start 2014-06-09 16:22:11.975914 7f869af78700 5 -- op tracker -- , seq: 3396, time: 2014-06-09 16:22:11.975912, event: started, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.975918 7f86a5e11700 10 _do_transaction on 0xc3f3140 2014-06-09 16:22:11.975912 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.975923 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.975919 7f868ef60700 10 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.975916 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.975930 7f869af78700 5 -- op tracker -- , seq: 3396, time: 2014-06-09 16:22:11.975927, event: done, op: pg_info(1 pgs e243:2.1c6) 2014-06-09 16:22:11.975937 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975932 7f868ef60700 15 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.975939 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975936 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.975940 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.975951 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.975947 7f868ef60700 20 osd.26 pg_epoch: 243 pg[0.105( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.975946 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.975953 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.975954 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.975955 7f869af78700 20 osd.26 243 _dispatch 0x12c096c0 pg_info(1 pgs e243:2.3aa) v4 2014-06-09 16:22:11.975954 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.975962 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.975961 7f869af78700 5 -- op tracker -- , seq: 3397, time: 2014-06-09 16:22:11.961247, event: header_read, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.975967 7f8691f66700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.975968 7f869af78700 5 -- op tracker -- , seq: 3397, time: 2014-06-09 16:22:11.961248, event: throttled, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.975958 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.5f8( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.975974 7f869af78700 5 -- op tracker -- , seq: 3397, time: 2014-06-09 16:22:11.961258, event: all_read, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.975977 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3697.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.975980 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.975980 7f869af78700 5 -- op tracker -- , seq: 3397, time: 2014-06-09 16:22:11.975951, event: dispatched, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.975986 7f869af78700 5 -- op tracker -- , seq: 3397, time: 2014-06-09 16:22:11.975984, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.975982 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:11.975991 7f86a5e11700 10 _do_op 0x308cb320 seq 3697 r = 0, finisher 0x3061e640 0 2014-06-09 16:22:11.975991 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.3aa) v4 from osd.8 2014-06-09 16:22:11.975993 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12c096c0 2014-06-09 16:22:11.975994 7f86a5e11700 10 _finish_op 0x308cb320 seq 3697 osr(2.6f 0x2ed0550)/0x2ed0550 2014-06-09 16:22:11.975996 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.975996 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbe60 (1 active) 2014-06-09 16:22:11.975998 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496120 (2 active) 2014-06-09 16:22:11.976000 7f86a5e11700 5 _do_op 0x324f4f00 seq 3698 osr(1.382 0x384d5b0)/0x384d5b0 start 2014-06-09 16:22:11.976000 7f868ef60700 5 queue_transactions existing osr(2.5f8 0x2ed1db0)/0x2ed1db0 2014-06-09 16:22:11.975999 7f869af78700 5 -- op tracker -- , seq: 3397, time: 2014-06-09 16:22:11.975997, event: started, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.976003 7f868ef60700 10 op_submit_start 3741 2014-06-09 16:22:11.976002 7f86a5e11700 10 _do_transaction on 0xc220300 2014-06-09 16:22:11.976005 7f868ef60700 5 queue_transactions (trailing journal) 3741 2014-06-09 16:22:11.976006 7f868ef60700 5 queue_op 0x15996de0 seq 3741 osr(2.5f8 0x2ed1db0) 828 bytes (queue has 48 ops and 150176 bytes) 2014-06-09 16:22:11.976008 7f868ef60700 10 op_submit_finish 3741 2014-06-09 16:22:11.976009 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976012 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.976016 7f8691f66700 5 queue_transactions existing osr(1.2fb 0x384d090)/0x384d090 2014-06-09 16:22:11.976013 7f869af78700 5 -- op tracker -- , seq: 3397, time: 2014-06-09 16:22:11.976011, event: done, op: pg_info(1 pgs e243:2.3aa) 2014-06-09 16:22:11.976018 7f8691f66700 10 op_submit_start 3742 2014-06-09 16:22:11.976019 7f8691f66700 5 queue_transactions (trailing journal) 3742 2014-06-09 16:22:11.976020 7f8691f66700 5 queue_op 0x16fd4ae0 seq 3742 osr(1.2fb 0x384d090) 861 bytes (queue has 49 ops and 151037 bytes) 2014-06-09 16:22:11.976011 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:11.976021 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976022 7f8691f66700 10 op_submit_finish 3742 2014-06-09 16:22:11.976023 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976025 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.976036 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976038 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976039 7f869af78700 20 osd.26 243 _dispatch 0x76e3dc0 pg_info(1 pgs e243:1.233) v4 2014-06-09 16:22:11.976040 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976037 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.976044 7f868ef60700 5 queue_transactions existing osr(3.7fc 0xb7f00e0)/0xb7f00e0 2014-06-09 16:22:11.976044 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976046 7f868ef60700 10 op_submit_start 3743 2014-06-09 16:22:11.976047 7f868ef60700 5 queue_transactions (trailing journal) 3743 2014-06-09 16:22:11.976044 7f869af78700 5 -- op tracker -- , seq: 3398, time: 2014-06-09 16:22:11.957284, event: header_read, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.976048 7f868ef60700 5 queue_op 0x3250ed80 seq 3743 osr(3.7fc 0xb7f00e0) 828 bytes (queue has 50 ops and 151865 bytes) 2014-06-09 16:22:11.976050 7f868ef60700 10 op_submit_finish 3743 2014-06-09 16:22:11.976046 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.976051 7f869af78700 5 -- op tracker -- , seq: 3398, time: 2014-06-09 16:22:11.957284, event: throttled, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.976053 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.976058 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3698.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976057 7f869af78700 5 -- op tracker -- , seq: 3398, time: 2014-06-09 16:22:11.957297, event: all_read, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.976061 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976064 7f8691f66700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.976063 7f869af78700 5 -- op tracker -- , seq: 3398, time: 2014-06-09 16:22:11.976036, event: dispatched, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.976053 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.220( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.976071 7f86a5e11700 10 _do_op 0x324f4f00 seq 3698 r = 0, finisher 0x17001b40 0 2014-06-09 16:22:11.976070 7f869af78700 5 -- op tracker -- , seq: 3398, time: 2014-06-09 16:22:11.976068, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.976073 7f86a5e11700 10 _finish_op 0x324f4f00 seq 3698 osr(1.382 0x384d5b0)/0x384d5b0 2014-06-09 16:22:11.976074 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.233) v4 from osd.6 2014-06-09 16:22:11.976076 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496120 (1 active) 2014-06-09 16:22:11.976077 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e3dc0 2014-06-09 16:22:11.976078 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ea20 (2 active) 2014-06-09 16:22:11.976080 7f86a5e11700 5 _do_op 0x324f4fc0 seq 3699 osr(0.54c 0x2bd7850)/0x2bd7850 start 2014-06-09 16:22:11.976082 7f86a5e11700 10 _do_transaction on 0x4800540 2014-06-09 16:22:11.976082 7f869af78700 5 -- op tracker -- , seq: 3398, time: 2014-06-09 16:22:11.976080, event: started, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.976079 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.976087 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976091 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976092 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.976095 7f868ef60700 5 queue_transactions existing osr(2.220 0x2ed0b10)/0x2ed0b10 2014-06-09 16:22:11.976097 7f868ef60700 10 op_submit_start 3744 2014-06-09 16:22:11.976098 7f868ef60700 5 queue_transactions (trailing journal) 3744 2014-06-09 16:22:11.976099 7f868ef60700 5 queue_op 0x3250ec00 seq 3744 osr(2.220 0x2ed0b10) 828 bytes (queue has 50 ops and 151865 bytes) 2014-06-09 16:22:11.976097 7f869af78700 5 -- op tracker -- , seq: 3398, time: 2014-06-09 16:22:11.976095, event: done, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.976102 7f868ef60700 10 op_submit_finish 3744 2014-06-09 16:22:11.976096 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.714( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.976105 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976107 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976104 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) 2014-06-09 16:22:11.976118 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976120 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976121 7f869af78700 20 osd.26 243 _dispatch 0x76e3880 pg_info(1 pgs e243:0.47d) v4 2014-06-09 16:22:11.976121 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976123 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.976117 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.976126 7f8691f66700 5 queue_transactions existing osr(0.714 0x384c0d0)/0x384c0d0 2014-06-09 16:22:11.976129 7f8691f66700 2 waiting 51 > 50 ops || 152693 > 104857600 2014-06-09 16:22:11.976126 7f869af78700 5 -- op tracker -- , seq: 3399, time: 2014-06-09 16:22:11.955597, event: header_read, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.976126 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.976133 7f869af78700 5 -- op tracker -- , seq: 3399, time: 2014-06-09 16:22:11.955598, event: throttled, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.976134 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3699.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976139 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976135 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.976140 7f869af78700 5 -- op tracker -- , seq: 3399, time: 2014-06-09 16:22:11.955608, event: all_read, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.976146 7f869af78700 5 -- op tracker -- , seq: 3399, time: 2014-06-09 16:22:11.976118, event: dispatched, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.976148 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.976153 7f869af78700 5 -- op tracker -- , seq: 3399, time: 2014-06-09 16:22:11.976151, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.976156 7f86a5e11700 10 _do_op 0x324f4fc0 seq 3699 r = 0, finisher 0x318c2fa0 0 2014-06-09 16:22:11.976157 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.47d) v4 from osd.0 2014-06-09 16:22:11.976159 7f86a5e11700 10 _finish_op 0x324f4fc0 seq 3699 osr(0.54c 0x2bd7850)/0x2bd7850 2014-06-09 16:22:11.976160 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e3880 2014-06-09 16:22:11.976164 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ea20 (1 active) 2014-06-09 16:22:11.976167 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287e60 (2 active) 2014-06-09 16:22:11.976165 7f869af78700 5 -- op tracker -- , seq: 3399, time: 2014-06-09 16:22:11.976163, event: started, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.976169 7f86a5e11700 5 _do_op 0x324f5200 seq 3700 osr(0.25f 0x2bd6d00)/0x2bd6d00 start 2014-06-09 16:22:11.976171 7f86a5e11700 10 _do_transaction on 0xc125800 2014-06-09 16:22:11.976175 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976178 7f869af78700 5 -- op tracker -- , seq: 3399, time: 2014-06-09 16:22:11.976176, event: done, op: pg_info(1 pgs e243:0.47d) 2014-06-09 16:22:11.976172 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1080 2014-06-09 16:22:11.976183 7f8691f66700 10 op_submit_start 3745 2014-06-09 16:22:11.976185 7f8691f66700 5 queue_transactions (trailing journal) 3745 2014-06-09 16:22:11.976186 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976186 7f8691f66700 5 queue_op 0xcabcde0 seq 3745 osr(0.714 0x384c0d0) 828 bytes (queue has 50 ops and 151865 bytes) 2014-06-09 16:22:11.976188 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976188 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976189 7f8691f66700 10 op_submit_finish 3745 2014-06-09 16:22:11.976192 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.f2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.976200 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976192 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.976202 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976203 7f869af78700 20 osd.26 243 _dispatch 0x230ba1c0 pg_info(1 pgs e243:1.77c) v4 2014-06-09 16:22:11.976211 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976208 7f869af78700 5 -- op tracker -- , seq: 3400, time: 2014-06-09 16:22:11.968722, event: header_read, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.976205 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.976216 7f869af78700 5 -- op tracker -- , seq: 3400, time: 2014-06-09 16:22:11.968723, event: throttled, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.976220 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976214 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.976223 7f868ef60700 5 queue_transactions existing osr(2.f2 0x2ed07e0)/0x2ed07e0 2014-06-09 16:22:11.976222 7f869af78700 5 -- op tracker -- , seq: 3400, time: 2014-06-09 16:22:11.968739, event: all_read, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.976223 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3700.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976226 7f868ef60700 2 waiting 51 > 50 ops || 152691 > 104857600 2014-06-09 16:22:11.976227 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976222 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.976228 7f869af78700 5 -- op tracker -- , seq: 3400, time: 2014-06-09 16:22:11.976200, event: dispatched, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.976233 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x85b0000 (0 active) 2014-06-09 16:22:11.976236 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.976236 7f86a5e11700 10 _do_op 0x324f5200 seq 3700 r = 0, finisher 0x30e18a40 0 2014-06-09 16:22:11.976235 7f869af78700 5 -- op tracker -- , seq: 3400, time: 2014-06-09 16:22:11.976233, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.976239 7f86a5e11700 10 _finish_op 0x324f5200 seq 3700 osr(0.25f 0x2bd6d00)/0x2bd6d00 2014-06-09 16:22:11.976240 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.77c) v4 from osd.5 2014-06-09 16:22:11.976242 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230ba1c0 2014-06-09 16:22:11.976242 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.976244 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287e60 (1 active) 2014-06-09 16:22:11.976246 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f560 (2 active) 2014-06-09 16:22:11.976248 7f86a5e11700 5 _do_op 0x324f52c0 seq 3701 osr(3.359 0x3ed10f0)/0x3ed10f0 start 2014-06-09 16:22:11.976250 7f86a5e11700 10 _do_transaction on 0xc23e240 2014-06-09 16:22:11.976247 7f869af78700 5 -- op tracker -- , seq: 3400, time: 2014-06-09 16:22:11.976245, event: started, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.976255 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976255 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062abe0,0x30508680,0xbd5d280,0xc168be0,0x30e18d00,0x318c24a0] 2014-06-09 16:22:11.976260 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062abe0,0x30508680,0xbd5d280,0xc168be0,0x30e18d00,0x318c24a0] 2014-06-09 16:22:11.976264 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.976265 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.976261 7f869af78700 5 -- op tracker -- , seq: 3400, time: 2014-06-09 16:22:11.976259, event: done, op: pg_info(1 pgs e243:1.77c) 2014-06-09 16:22:11.976266 7f868c75b700 10 op_submit_start 3746 2014-06-09 16:22:11.976269 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976270 7f868c75b700 5 queue_transactions (trailing journal) 3746 2014-06-09 16:22:11.976271 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976272 7f868c75b700 5 queue_op 0x2ea53260 seq 3746 osr(3.3f5 0x3ed1280) 18782 bytes (queue has 50 ops and 169819 bytes) 2014-06-09 16:22:11.976275 7f868c75b700 10 op_submit_finish 3746 2014-06-09 16:22:11.976272 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:881 2014-06-09 16:22:11.976285 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976286 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976287 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.976287 7f869af78700 20 osd.26 243 _dispatch 0x3130bdc0 pg_info(1 pgs e243:0.2c6) v4 2014-06-09 16:22:11.976288 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976292 7f869af78700 5 -- op tracker -- , seq: 3401, time: 2014-06-09 16:22:11.963236, event: header_read, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:11.976279 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'565 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc144fc0 243'566 rep_tid=18 committed?=0 applied?=0 lock=0 op=osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.976291 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.55d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.976302 7f868c75b700 10 osd.26 243 dequeue_op 0x1c66aa00 finish 2014-06-09 16:22:11.976301 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3701.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976300 7f869af78700 5 -- op tracker -- , seq: 3401, time: 2014-06-09 16:22:11.963237, event: throttled, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:11.976305 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976307 7f869af78700 5 -- op tracker -- , seq: 3401, time: 2014-06-09 16:22:11.963243, event: all_read, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:11.976310 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.976313 7f869af78700 5 -- op tracker -- , seq: 3401, time: 2014-06-09 16:22:11.976284, event: dispatched, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:11.976316 7f86a5e11700 10 _do_op 0x324f52c0 seq 3701 r = 0, finisher 0x318c2a20 0 2014-06-09 16:22:11.976317 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.976322 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.976322 7f86a5e11700 10 _finish_op 0x324f52c0 seq 3701 osr(3.359 0x3ed10f0)/0x3ed10f0 2014-06-09 16:22:11.976320 7f869af78700 5 -- op tracker -- , seq: 3401, time: 2014-06-09 16:22:11.976318, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:11.976324 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f560 (1 active) 2014-06-09 16:22:11.976324 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2c6) v4 from osd.2 2014-06-09 16:22:11.976325 7f8691f66700 5 queue_transactions existing osr(0.55d 0x2bd77f0)/0x2bd77f0 2014-06-09 16:22:11.976326 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3007e0 (2 active) 2014-06-09 16:22:11.976327 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130bdc0 2014-06-09 16:22:11.976328 7f8691f66700 10 op_submit_start 3747 2014-06-09 16:22:11.976329 7f8691f66700 5 queue_transactions (trailing journal) 3747 2014-06-09 16:22:11.976328 7f86a5e11700 5 _do_op 0x324f5380 seq 3702 osr(2.d3 0x2ed0450)/0x2ed0450 start 2014-06-09 16:22:11.976330 7f86a5e11700 10 _do_transaction on 0xbd8ecc0 2014-06-09 16:22:11.976330 7f8691f66700 5 queue_op 0xcabd320 seq 3747 osr(0.55d 0x2bd77f0) 828 bytes (queue has 50 ops and 169819 bytes) 2014-06-09 16:22:11.976332 7f8691f66700 10 op_submit_finish 3747 2014-06-09 16:22:11.976321 7f868c75b700 10 osd.26 243 dequeue_op 0x230da200 prio 196 cost 0 latency 0.013876 osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) v2 pg pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.976332 7f869af78700 5 -- op tracker -- , seq: 3401, time: 2014-06-09 16:22:11.976330, event: started, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:11.976335 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976339 7f868c75b700 5 -- op tracker -- , seq: 3370, time: 2014-06-09 16:22:11.976336, event: reached_pg, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) 2014-06-09 16:22:11.976347 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x230da200 2014-06-09 16:22:11.976355 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (19 active) 2014-06-09 16:22:11.976360 7f868c75b700 5 -- op tracker -- , seq: 3370, time: 2014-06-09 16:22:11.976357, event: started, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) 2014-06-09 16:22:11.976365 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 17 op ack_type 1 from 8 2014-06-09 16:22:11.976372 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976383 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (20 active) 2014-06-09 16:22:11.976381 7f868c75b700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.976375, event: sub_op_applied_rec, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.976388 7f868c75b700 10 osd.26 243 dequeue_op 0x230da200 finish 2014-06-09 16:22:11.976387 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3702.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976391 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976389 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.714( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.714( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.976394 7f868c75b700 5 -- op tracker -- , seq: 3370, time: 2014-06-09 16:22:11.976391, event: done, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ack, result = 0) 2014-06-09 16:22:11.976403 7f86a5e11700 10 _do_op 0x324f5380 seq 3702 r = 0, finisher 0x3050bb40 0 2014-06-09 16:22:11.976406 7f86a5e11700 10 _finish_op 0x324f5380 seq 3702 osr(2.d3 0x2ed0450)/0x2ed0450 2014-06-09 16:22:11.976409 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.976409 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3007e0 (1 active) 2014-06-09 16:22:11.976411 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f67e0 (2 active) 2014-06-09 16:22:11.976413 7f86a5e11700 5 _do_op 0xc704900 seq 3703 osr(1.2d0 0x384d180)/0x384d180 start 2014-06-09 16:22:11.976412 7f869af78700 5 -- op tracker -- , seq: 3401, time: 2014-06-09 16:22:11.976391, event: done, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:11.976415 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.976416 7f86a5e11700 10 _do_transaction on 0xc1f2fc0 2014-06-09 16:22:11.976418 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.976420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976420 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976422 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976422 7f8691f66700 5 queue_transactions existing osr(0.714 0x384c0d0)/0x384c0d0 2014-06-09 16:22:11.976425 7f8691f66700 10 op_submit_start 3748 2014-06-09 16:22:11.976426 7f8691f66700 5 queue_transactions (trailing journal) 3748 2014-06-09 16:22:11.976427 7f8691f66700 5 queue_op 0xcabd3e0 seq 3748 osr(0.714 0x384c0d0) 828 bytes (queue has 50 ops and 169821 bytes) 2014-06-09 16:22:11.976429 7f8691f66700 10 op_submit_finish 3748 2014-06-09 16:22:11.976434 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976436 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976437 7f869af78700 20 osd.26 243 _dispatch 0x309e6000 pg_info(1 pgs e243:1.582) v4 2014-06-09 16:22:11.976432 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.55d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.55d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:11.976442 7f869af78700 5 -- op tracker -- , seq: 3402, time: 2014-06-09 16:22:11.957208, event: header_read, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:11.976450 7f869af78700 5 -- op tracker -- , seq: 3402, time: 2014-06-09 16:22:11.957209, event: throttled, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:11.976455 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976456 7f869af78700 5 -- op tracker -- , seq: 3402, time: 2014-06-09 16:22:11.957217, event: all_read, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:11.976460 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.976464 7f8691f66700 5 queue_transactions existing osr(0.55d 0x2bd77f0)/0x2bd77f0 2014-06-09 16:22:11.976462 7f869af78700 5 -- op tracker -- , seq: 3402, time: 2014-06-09 16:22:11.976434, event: dispatched, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:11.976467 7f8691f66700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.976468 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3703.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976472 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976469 7f869af78700 5 -- op tracker -- , seq: 3402, time: 2014-06-09 16:22:11.976467, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:11.976474 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.582) v4 from osd.4 2014-06-09 16:22:11.976476 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e6000 2014-06-09 16:22:11.976481 7f86a5e11700 10 _do_op 0xc704900 seq 3703 r = 0, finisher 0xbd5b7a0 0 2014-06-09 16:22:11.976481 7f869af78700 5 -- op tracker -- , seq: 3402, time: 2014-06-09 16:22:11.976479, event: started, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:11.976484 7f86a5e11700 10 _finish_op 0xc704900 seq 3703 osr(1.2d0 0x384d180)/0x384d180 2014-06-09 16:22:11.976474 7f868f761700 10 osd.26 243 dequeue_op 0x257c4000 prio 196 cost 0 latency 0.019131 osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) v2 pg pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'565 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.976489 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f67e0 (1 active) 2014-06-09 16:22:11.976491 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f440 (2 active) 2014-06-09 16:22:11.976493 7f86a5e11700 5 _do_op 0x2fc8df20 seq 3704 osr(3.e0 0x3ed0500)/0x3ed0500 start 2014-06-09 16:22:11.976496 7f86a5e11700 10 _do_transaction on 0xbf36240 2014-06-09 16:22:11.976494 7f868f761700 5 -- op tracker -- , seq: 3362, time: 2014-06-09 16:22:11.976491, event: reached_pg, op: osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.976500 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976507 7f869af78700 5 -- op tracker -- , seq: 3402, time: 2014-06-09 16:22:11.976494, event: done, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:11.976506 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'565 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x257c4000 2014-06-09 16:22:11.976516 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976518 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976520 7f868ef60700 10 op_submit_start 3749 2014-06-09 16:22:11.976522 7f868ef60700 5 queue_transactions (trailing journal) 3749 2014-06-09 16:22:11.976523 7f868ef60700 5 queue_op 0x3250fce0 seq 3749 osr(2.f2 0x2ed07e0) 826 bytes (queue has 50 ops and 169819 bytes) 2014-06-09 16:22:11.976521 7f868f761700 5 -- op tracker -- , seq: 3362, time: 2014-06-09 16:22:11.976518, event: started, op: osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.976526 7f868ef60700 10 op_submit_finish 3749 2014-06-09 16:22:11.976530 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976532 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976533 7f869af78700 20 osd.26 243 _dispatch 0x15a288c0 pg_info(1 pgs e243:3.2df) v4 2014-06-09 16:22:11.976527 7f868f761700 7 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'565 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 18 op ack_type 4 from 3 2014-06-09 16:22:11.976536 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976529 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.fd( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.976538 7f869af78700 5 -- op tracker -- , seq: 3403, time: 2014-06-09 16:22:11.959110, event: header_read, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.976548 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3704.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976548 7f869af78700 5 -- op tracker -- , seq: 3403, time: 2014-06-09 16:22:11.959110, event: throttled, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.976552 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976554 7f869af78700 5 -- op tracker -- , seq: 3403, time: 2014-06-09 16:22:11.959119, event: all_read, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.976558 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976560 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c29a0,0xbd77bc0,0x3061fd60] 2014-06-09 16:22:11.976561 7f869af78700 5 -- op tracker -- , seq: 3403, time: 2014-06-09 16:22:11.976530, event: dispatched, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.976563 7f868ef60700 5 queue_transactions existing osr(2.fd 0x2ed0770)/0x2ed0770 2014-06-09 16:22:11.976565 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c29a0,0xbd77bc0,0x3061fd60] 2014-06-09 16:22:11.976566 7f868ef60700 2 waiting 51 > 50 ops || 170645 > 104857600 2014-06-09 16:22:11.976567 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.976568 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.976570 7f868bf5a700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.976569 7f86a5e11700 10 _do_op 0x2fc8df20 seq 3704 r = 0, finisher 0xbebf3e0 0 2014-06-09 16:22:11.976568 7f869af78700 5 -- op tracker -- , seq: 3403, time: 2014-06-09 16:22:11.976566, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.976572 7f868df5e700 10 op_submit_start 3750 2014-06-09 16:22:11.976574 7f868df5e700 5 queue_transactions (trailing journal) 3750 2014-06-09 16:22:11.976573 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2df) v4 from osd.7 2014-06-09 16:22:11.976575 7f86a5e11700 10 _finish_op 0x2fc8df20 seq 3704 osr(3.e0 0x3ed0500)/0x3ed0500 2014-06-09 16:22:11.976576 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a288c0 2014-06-09 16:22:11.976575 7f868df5e700 5 queue_op 0xbf7e540 seq 3750 osr(0.36d 0x2bd6fc0) 828 bytes (queue has 50 ops and 169819 bytes) 2014-06-09 16:22:11.976581 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31413bc0] 2014-06-09 16:22:11.976580 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f440 (1 active) 2014-06-09 16:22:11.976582 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31413bc0] 2014-06-09 16:22:11.976583 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.976583 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f8c0 (2 active) 2014-06-09 16:22:11.976584 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.976580 7f86a4e0f700 5 -- op tracker -- , seq: 440, time: 2014-06-09 16:22:11.975721, event: done, op: osd_op(client.4569.0:38638 rbd_data.11796b8b4567.0000000000001aab [set-alloc-hint object_size 4194304 write_size 4194304,write 61440~516096] 3.8721c319 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.976588 7f868df5e700 10 op_submit_finish 3750 2014-06-09 16:22:11.976587 7f86a5e11700 5 _do_op 0x30d97320 seq 3705 osr(1.91 0x384c430)/0x384c430 start 2014-06-09 16:22:11.976591 7f86a5e11700 10 _do_transaction on 0x2c18600 2014-06-09 16:22:11.976595 7f868ef60700 10 op_submit_start 3751 2014-06-09 16:22:11.976596 7f868ef60700 5 queue_transactions (trailing journal) 3751 2014-06-09 16:22:11.976596 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976594 7f869af78700 5 -- op tracker -- , seq: 3403, time: 2014-06-09 16:22:11.976579, event: started, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.976598 7f868ef60700 5 queue_op 0x3250fb60 seq 3751 osr(2.fd 0x2ed0770) 826 bytes (queue has 50 ops and 169819 bytes) 2014-06-09 16:22:11.976601 7f868ef60700 10 op_submit_finish 3751 2014-06-09 16:22:11.976594 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.654( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.976603 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.976613 7f869af78700 5 -- op tracker -- , seq: 3403, time: 2014-06-09 16:22:11.976611, event: done, op: pg_info(1 pgs e243:3.2df) 2014-06-09 16:22:11.976615 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.976622 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976623 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976626 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.976623 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.976631 7f868df5e700 5 queue_transactions existing osr(2.654 0x2ed1fd0)/0x2ed1fd0 2014-06-09 16:22:11.976631 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976633 7f868df5e700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.976635 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976630 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.976637 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976637 7f8690763700 2 waiting 51 > 50 ops || 170872 > 104857600 2014-06-09 16:22:11.976638 7f869af78700 20 osd.26 243 _dispatch 0x30fccfc0 pg_info(1 pgs e243:1.4bb) v4 2014-06-09 16:22:11.976642 7f868cf5c700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.976642 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.976644 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3705.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976643 7f869af78700 5 -- op tracker -- , seq: 3404, time: 2014-06-09 16:22:11.972283, event: header_read, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:11.976647 7f868b759700 10 op_submit_start 3752 2014-06-09 16:22:11.976648 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976648 7f868b759700 5 queue_transactions (trailing journal) 3752 2014-06-09 16:22:11.976650 7f869af78700 5 -- op tracker -- , seq: 3404, time: 2014-06-09 16:22:11.972284, event: throttled, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:11.976656 7f869af78700 5 -- op tracker -- , seq: 3404, time: 2014-06-09 16:22:11.972290, event: all_read, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:11.976659 7f86a5e11700 10 _do_op 0x30d97320 seq 3705 r = 0, finisher 0x314128c0 0 2014-06-09 16:22:11.976657 7f868ef60700 15 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:179 2014-06-09 16:22:11.976662 7f869af78700 5 -- op tracker -- , seq: 3404, time: 2014-06-09 16:22:11.976635, event: dispatched, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:11.976666 7f86a5e11700 10 _finish_op 0x30d97320 seq 3705 osr(1.91 0x384c430)/0x384c430 2014-06-09 16:22:11.976670 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f8c0 (1 active) 2014-06-09 16:22:11.976671 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976670 7f869af78700 5 -- op tracker -- , seq: 3404, time: 2014-06-09 16:22:11.976667, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:11.976672 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff50e0 (2 active) 2014-06-09 16:22:11.976675 7f86a5e11700 5 _do_op 0x16fd56e0 seq 3706 osr(2.c9 0x2ed04a0)/0x2ed04a0 start 2014-06-09 16:22:11.976675 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4bb) v4 from osd.3 2014-06-09 16:22:11.976677 7f86a5e11700 10 _do_transaction on 0xbe1b5c0 2014-06-09 16:22:11.976677 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fccfc0 2014-06-09 16:22:11.976682 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976682 7f869af78700 5 -- op tracker -- , seq: 3404, time: 2014-06-09 16:22:11.976680, event: started, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:11.976676 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:11.976690 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.976698 7f869af78700 5 -- op tracker -- , seq: 3404, time: 2014-06-09 16:22:11.976695, event: done, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:11.976700 7f868b759700 5 queue_op 0xbf7e060 seq 3752 osr(2.52e 0x2ed1600) 828 bytes (queue has 49 ops and 168993 bytes) 2014-06-09 16:22:11.976704 7f868b759700 10 op_submit_finish 3752 2014-06-09 16:22:11.976701 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.976712 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976715 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976715 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976708 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.976723 7f868ef60700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.976726 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976728 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976727 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3706.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976729 7f869af78700 20 osd.26 243 _dispatch 0x30ef7340 pg_info(1 pgs e243:3.f7) v4 2014-06-09 16:22:11.976731 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976734 7f869af78700 5 -- op tracker -- , seq: 3405, time: 2014-06-09 16:22:11.959399, event: header_read, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.976740 7f86a5e11700 10 _do_op 0x16fd56e0 seq 3706 r = 0, finisher 0x31413380 0 2014-06-09 16:22:11.976743 7f86a5e11700 10 _finish_op 0x16fd56e0 seq 3706 osr(2.c9 0x2ed04a0)/0x2ed04a0 2014-06-09 16:22:11.976741 7f869af78700 5 -- op tracker -- , seq: 3405, time: 2014-06-09 16:22:11.959400, event: throttled, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.976735 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1099 2014-06-09 16:22:11.976746 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff50e0 (1 active) 2014-06-09 16:22:11.976747 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c19e0 (2 active) 2014-06-09 16:22:11.976747 7f869af78700 5 -- op tracker -- , seq: 3405, time: 2014-06-09 16:22:11.959409, event: all_read, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.976749 7f86a5e11700 5 _do_op 0x16fd4660 seq 3707 osr(1.45d 0x384d670)/0x384d670 start 2014-06-09 16:22:11.976751 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976752 7f86a5e11700 10 _do_transaction on 0xc2dc300 2014-06-09 16:22:11.976753 7f869af78700 5 -- op tracker -- , seq: 3405, time: 2014-06-09 16:22:11.976726, event: dispatched, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.976756 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976759 7f869af78700 5 -- op tracker -- , seq: 3405, time: 2014-06-09 16:22:11.976758, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.976755 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.69c( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.976764 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.f7) v4 from osd.1 2014-06-09 16:22:11.976767 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef7340 2014-06-09 16:22:11.976769 7f868e75f700 10 op_submit_start 3753 2014-06-09 16:22:11.976771 7f868e75f700 5 queue_transactions (trailing journal) 3753 2014-06-09 16:22:11.976773 7f868e75f700 5 queue_op 0x324f4180 seq 3753 osr(2.620 0x2ed1cb0) 828 bytes (queue has 48 ops and 168167 bytes) 2014-06-09 16:22:11.976772 7f869af78700 5 -- op tracker -- , seq: 3405, time: 2014-06-09 16:22:11.976770, event: started, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.976776 7f868e75f700 10 op_submit_finish 3753 2014-06-09 16:22:11.976784 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976788 7f868ef60700 5 queue_transactions existing osr(2.69c 0x2ed1f20)/0x2ed1f20 2014-06-09 16:22:11.976779 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.dd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:11.976791 7f868ef60700 10 op_submit_start 3754 2014-06-09 16:22:11.976792 7f868ef60700 5 queue_transactions (trailing journal) 3754 2014-06-09 16:22:11.976791 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976789 7f869af78700 5 -- op tracker -- , seq: 3405, time: 2014-06-09 16:22:11.976786, event: done, op: pg_info(1 pgs e243:3.f7) 2014-06-09 16:22:11.976793 7f868ef60700 5 queue_op 0x32503320 seq 3754 osr(2.69c 0x2ed1f20) 828 bytes (queue has 49 ops and 168995 bytes) 2014-06-09 16:22:11.976795 7f868ef60700 10 op_submit_finish 3754 2014-06-09 16:22:11.976797 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976799 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976804 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3707.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976797 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.7fc( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 243/243 242/242/222) 2014-06-09 16:22:11.976808 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.976810 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.976810 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976812 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976813 7f869af78700 20 osd.26 243 _dispatch 0x3115b500 pg_info(1 pgs e243:0.6f0) v4 2014-06-09 16:22:11.976814 7f868e75f700 5 queue_transactions existing osr(0.dd 0x2bd6550)/0x2bd6550 2014-06-09 16:22:11.976817 7f868e75f700 10 op_submit_start 3755 2014-06-09 16:22:11.976817 7f868e75f700 5 queue_transactions (trailing journal) 3755 2014-06-09 16:22:11.976818 7f86a5e11700 10 _do_op 0x16fd4660 seq 3707 r = 0, finisher 0xccf8ba0 0 2014-06-09 16:22:11.976819 7f868e75f700 5 queue_op 0x324f4240 seq 3755 osr(0.dd 0x2bd6550) 826 bytes (queue has 50 ops and 169821 bytes) 2014-06-09 16:22:11.976818 7f869af78700 5 -- op tracker -- , seq: 3406, time: 2014-06-09 16:22:11.958713, event: header_read, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.976821 7f86a5e11700 10 _finish_op 0x16fd4660 seq 3707 osr(1.45d 0x384d670)/0x384d670 2014-06-09 16:22:11.976823 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c19e0 (1 active) 2014-06-09 16:22:11.976824 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.976825 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52fc20 (2 active) 2014-06-09 16:22:11.976825 7f869af78700 5 -- op tracker -- , seq: 3406, time: 2014-06-09 16:22:11.958714, event: throttled, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.976828 7f868ef60700 5 queue_transactions existing osr(3.7fc 0xb7f00e0)/0xb7f00e0 2014-06-09 16:22:11.976709 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.976833 7f8692f68700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.976833 7f86a5610700 10 _finish_op 0x324f4c00 seq 3694 osr(2.606 0x2ed1d30)/0x2ed1d30 2014-06-09 16:22:11.976832 7f869af78700 5 -- op tracker -- , seq: 3406, time: 2014-06-09 16:22:11.958727, event: all_read, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.976839 7f869af78700 5 -- op tracker -- , seq: 3406, time: 2014-06-09 16:22:11.976810, event: dispatched, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.976836 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.921086 7 0.000419 2014-06-09 16:22:11.976845 7f869af78700 5 -- op tracker -- , seq: 3406, time: 2014-06-09 16:22:11.976843, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.976850 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6f0) v4 from osd.9 2014-06-09 16:22:11.976852 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3115b500 2014-06-09 16:22:11.976847 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.976857 7f869af78700 5 -- op tracker -- , seq: 3406, time: 2014-06-09 16:22:11.976855, event: started, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.976857 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.976866 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52fc20 (1 active) 2014-06-09 16:22:11.976869 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287e60 (2 active) 2014-06-09 16:22:11.976872 7f86a5610700 5 _do_op 0x30061320 seq 3709 osr(0.25f 0x2bd6d00)/0x2bd6d00 start 2014-06-09 16:22:11.976875 7f86a5610700 10 _do_transaction on 0xc320240 2014-06-09 16:22:11.976868 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:11.976874 7f869af78700 5 -- op tracker -- , seq: 3406, time: 2014-06-09 16:22:11.976871, event: done, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.976876 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31412f40,0x31413200,0xbd5b080] 2014-06-09 16:22:11.976879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31412f40,0x31413200,0xbd5b080] 2014-06-09 16:22:11.976880 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976881 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.976882 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.976882 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976884 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976878 7f868b759700 5 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.976889 7f8690f64700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.976886 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.976894 7f8689755700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.976896 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976897 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976898 7f868a757700 2 waiting 51 > 50 ops || 188603 > 104857600 2014-06-09 16:22:11.976893 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.976899 7f869af78700 20 osd.26 243 _dispatch 0x12c09880 pg_info(1 pgs e243:3.4c8) v4 2014-06-09 16:22:11.976902 7f8691f66700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.976906 7f868bf5a700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.976904 7f869af78700 5 -- op tracker -- , seq: 3407, time: 2014-06-09 16:22:11.961263, event: header_read, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.976910 7f8690763700 2 waiting 51 > 50 ops || 170874 > 104857600 2014-06-09 16:22:11.976903 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:11.976911 7f869af78700 5 -- op tracker -- , seq: 3407, time: 2014-06-09 16:22:11.961263, event: throttled, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.976914 7f868cf5c700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.976918 7f8692767700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.976913 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:11.976918 7f869af78700 5 -- op tracker -- , seq: 3407, time: 2014-06-09 16:22:11.961273, event: all_read, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.976927 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.976925 7f869af78700 5 -- op tracker -- , seq: 3407, time: 2014-06-09 16:22:11.976895, event: dispatched, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.976922 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:11.976934 7f869af78700 5 -- op tracker -- , seq: 3407, time: 2014-06-09 16:22:11.976932, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.976932 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:11.976939 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c8) v4 from osd.8 2014-06-09 16:22:11.976942 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12c09880 2014-06-09 16:22:11.976941 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.976947 7f869af78700 5 -- op tracker -- , seq: 3407, time: 2014-06-09 16:22:11.976945, event: started, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.976949 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3709.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.976953 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.976953 7f868b759700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.976954 7f86a5e11700 5 _do_op 0xc2ead80 seq 3708 osr(2.606 0x2ed1d30)/0x2ed1d30 start 2014-06-09 16:22:11.976958 7f86a5e11700 10 _do_transaction on 0xc129b00 2014-06-09 16:22:11.976963 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.976963 7f869af78700 5 -- op tracker -- , seq: 3407, time: 2014-06-09 16:22:11.976961, event: done, op: pg_info(1 pgs e243:3.4c8) 2014-06-09 16:22:11.976971 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976973 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976969 7f868b759700 15 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 2014-06-09 16:22:11.976985 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.976986 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.976988 7f869af78700 20 osd.26 243 _dispatch 0x30c12a80 pg_info(1 pgs e243:1.384) v4 2014-06-09 16:22:11.976992 7f869af78700 5 -- op tracker -- , seq: 3408, time: 2014-06-09 16:22:11.957355, event: header_read, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.976995 7f8692f68700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.977000 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.977000 7f8691765700 2 waiting 51 > 50 ops || 170679 > 104857600 2014-06-09 16:22:11.977000 7f869af78700 5 -- op tracker -- , seq: 3408, time: 2014-06-09 16:22:11.957356, event: throttled, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.977004 7f868b759700 5 queue_transactions existing osr(0.3e 0x2bd6270)/0x2bd6270 2014-06-09 16:22:11.977005 7f868af58700 2 waiting 51 > 50 ops || 170649 > 104857600 2014-06-09 16:22:11.977007 7f868b759700 2 waiting 51 > 50 ops || 170679 > 104857600 2014-06-09 16:22:11.977006 7f869af78700 5 -- op tracker -- , seq: 3408, time: 2014-06-09 16:22:11.957370, event: all_read, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.977011 7f868d75d700 2 waiting 51 > 50 ops || 1204401 > 104857600 2014-06-09 16:22:11.977015 7f868e75f700 10 op_submit_finish 3755 2014-06-09 16:22:11.977013 7f869af78700 5 -- op tracker -- , seq: 3408, time: 2014-06-09 16:22:11.976984, event: dispatched, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.977017 7f86a5610700 10 _do_op 0x30061320 seq 3709 r = 0, finisher 0xbd6bd60 0 2014-06-09 16:22:11.977020 7f869af78700 5 -- op tracker -- , seq: 3408, time: 2014-06-09 16:22:11.977018, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.977023 7f86a5610700 10 _finish_op 0x30061320 seq 3709 osr(0.25f 0x2bd6d00)/0x2bd6d00 2014-06-09 16:22:11.977025 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.384) v4 from osd.6 2014-06-09 16:22:11.977027 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c12a80 2014-06-09 16:22:11.977027 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287e60 (1 active) 2014-06-09 16:22:11.977030 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc7e0 (2 active) 2014-06-09 16:22:11.977032 7f86a5610700 5 _do_op 0x33a2d20 seq 3710 osr(1.560 0x384da90)/0x384da90 start 2014-06-09 16:22:11.977034 7f86a5610700 10 _do_transaction on 0x338b2c0 2014-06-09 16:22:11.977033 7f869af78700 5 -- op tracker -- , seq: 3408, time: 2014-06-09 16:22:11.977031, event: started, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.977020 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.19( v 213'743 (0'0,213'743] local-les=243 n=141 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:11.977039 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.977044 7f868ef60700 10 op_submit_start 3756 2014-06-09 16:22:11.977045 7f868ef60700 5 queue_transactions (trailing journal) 3756 2014-06-09 16:22:11.977046 7f868ef60700 5 queue_op 0x325031a0 seq 3756 osr(3.7fc 0xb7f00e0) 828 bytes (queue has 49 ops and 168993 bytes) 2014-06-09 16:22:11.977049 7f868ef60700 10 op_submit_finish 3756 2014-06-09 16:22:11.977050 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.977048 7f869af78700 5 -- op tracker -- , seq: 3408, time: 2014-06-09 16:22:11.977046, event: done, op: pg_info(1 pgs e243:1.384) 2014-06-09 16:22:11.977057 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.977058 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.977059 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.977053 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.289( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.977062 7f868e75f700 5 queue_transactions existing osr(3.19 0x3ed03a0)/0x3ed03a0 2014-06-09 16:22:11.977065 7f868e75f700 10 op_submit_start 3757 2014-06-09 16:22:11.977065 7f868e75f700 5 queue_transactions (trailing journal) 3757 2014-06-09 16:22:11.977064 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3708.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.977067 7f868e75f700 5 queue_op 0x324f4300 seq 3757 osr(3.19 0x3ed03a0) 826 bytes (queue has 50 ops and 169819 bytes) 2014-06-09 16:22:11.977068 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.977069 7f868e75f700 10 op_submit_finish 3757 2014-06-09 16:22:11.977070 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.977072 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.977073 7f869af78700 20 osd.26 243 _dispatch 0x30c30000 pg_info(1 pgs e243:0.448) v4 2014-06-09 16:22:11.977078 7f86a5e11700 10 _do_op 0xc2ead80 seq 3708 r = 0, finisher 0x314135c0 0 2014-06-09 16:22:11.977072 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.113( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.113( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:11.977078 7f869af78700 5 -- op tracker -- , seq: 3409, time: 2014-06-09 16:22:11.956817, event: header_read, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.977081 7f86a5e11700 10 _finish_op 0xc2ead80 seq 3708 osr(2.606 0x2ed1d30)/0x2ed1d30 2014-06-09 16:22:11.977082 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.977084 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52fc20 (1 active) 2014-06-09 16:22:11.977086 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f560 (2 active) 2014-06-09 16:22:11.977086 7f868ef60700 5 queue_transactions existing osr(1.289 0x384ceb0)/0x384ceb0 2014-06-09 16:22:11.977085 7f869af78700 5 -- op tracker -- , seq: 3409, time: 2014-06-09 16:22:11.956817, event: throttled, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.977089 7f868ef60700 10 op_submit_start 3758 2014-06-09 16:22:11.977088 7f86a5e11700 5 _do_op 0x241048a0 seq 3711 osr(3.359 0x3ed10f0)/0x3ed10f0 start 2014-06-09 16:22:11.977090 7f868ef60700 5 queue_transactions (trailing journal) 3758 2014-06-09 16:22:11.977090 7f86a5e11700 10 _do_transaction on 0x3a86e40 2014-06-09 16:22:11.977091 7f868ef60700 5 queue_op 0x32503020 seq 3758 osr(1.289 0x384ceb0) 828 bytes (queue has 50 ops and 169819 bytes) 2014-06-09 16:22:11.977093 7f868ef60700 10 op_submit_finish 3758 2014-06-09 16:22:11.977091 7f869af78700 5 -- op tracker -- , seq: 3409, time: 2014-06-09 16:22:11.956830, event: all_read, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.977095 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.977097 7f869af78700 5 -- op tracker -- , seq: 3409, time: 2014-06-09 16:22:11.977070, event: dispatched, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.977100 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.977104 7f8689f56700 2 waiting 51 > 50 ops || 170645 > 104857600 2014-06-09 16:22:11.977104 7f869af78700 5 -- op tracker -- , seq: 3409, time: 2014-06-09 16:22:11.977102, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.981415 7f868e75f700 5 queue_transactions existing osr(2.113 0x2ed06b0)/0x2ed06b0 2014-06-09 16:22:11.981419 7f868e75f700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.977096 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 8 2014-06-09 16:22:11.981424 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.448) v4 from osd.0 2014-06-09 16:22:11.981430 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c30000 2014-06-09 16:22:11.981429 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: GetLog: received master log from osd8 2014-06-09 16:22:11.981437 7f869af78700 5 -- op tracker -- , seq: 3409, time: 2014-06-09 16:22:11.981434, event: started, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.981446 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: leaving GetLog 2014-06-09 16:22:11.981452 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: processing master log 2014-06-09 16:22:11.981458 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'681 (0'0,214'681] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_master_log for osd.8: log((214'681,214'683], crt=0'0) missing(0) 2014-06-09 16:22:11.981458 7f868f761700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:11.976538, event: sub_op_commit_rec, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.981464 7f8689755700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.981464 7f868ef60700 10 merge_log log((214'681,214'683], crt=0'0) from osd.8 into log((0'0,214'681], crt=0'0) 2014-06-09 16:22:11.981468 7f868ef60700 10 merge_log extending head to 214'683 2014-06-09 16:22:11.981470 7f868a757700 2 waiting 51 > 50 ops || 188601 > 104857600 2014-06-09 16:22:11.981470 7f868f761700 10 osd.26 243 dequeue_op 0x257c4000 finish 2014-06-09 16:22:11.981474 7f8691f66700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.981469 7f868ef60700 20 ? 214'683 (214'682) modify 547499ff/rbd_data.115f6b8b4567.00000000000016a4/head//3 by client.4543.0:33506 2014-06-09 10:19:46.736685 2014-06-09 16:22:11.981479 7f868bf5a700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.981478 7f868ef60700 20 ? 214'682 (214'681) modify 547499ff/rbd_data.115f6b8b4567.00000000000016a4/head//3 by client.4543.0:33505 2014-06-09 10:19:46.120692 2014-06-09 16:22:11.981485 7f8690763700 2 waiting 51 > 50 ops || 170872 > 104857600 2014-06-09 16:22:11.981483 7f868ef60700 20 merge_log 214'682 (214'681) modify 547499ff/rbd_data.115f6b8b4567.00000000000016a4/head//3 by client.4543.0:33505 2014-06-09 10:19:46.120692 2014-06-09 16:22:11.981490 7f868cf5c700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.981496 7f8692767700 2 waiting 51 > 50 ops || 170647 > 104857600 2014-06-09 16:22:11.981493 7f868ef60700 20 merge_log 214'683 (214'682) modify 547499ff/rbd_data.115f6b8b4567.00000000000016a4/head//3 by client.4543.0:33506 2014-06-09 10:19:46.736685 2014-06-09 16:22:11.977136 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.981503 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 17 2014-06-09 16:22:11.981520 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3711.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.981521 7f86a4e0f700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.981515, event: op_applied, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.981525 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.981528 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'592 2014-06-09 16:22:11.981537 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_applied: repop tid 17 all applied 2014-06-09 16:22:11.981544 7f86a5e11700 10 _do_op 0x241048a0 seq 3711 r = 0, finisher 0x30e19880 0 2014-06-09 16:22:11.981548 7f86a5e11700 10 _finish_op 0x241048a0 seq 3711 osr(3.359 0x3ed10f0)/0x3ed10f0 2014-06-09 16:22:11.981554 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f560 (1 active) 2014-06-09 16:22:11.981557 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300fc0 (2 active) 2014-06-09 16:22:11.981545 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0x2d83ec0 243'592 rep_tid=17 committed?=0 applied?=1 lock=0 op=osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.981560 7f86a5e11700 5 _do_op 0x15997ce0 seq 3712 osr(2.27a 0x2ed0d90)/0x2ed0d90 start 2014-06-09 16:22:11.981563 7f86a5e11700 10 _do_transaction on 0xc210240 2014-06-09 16:22:11.981568 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.981561 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] sending ack on repgather(0x2d83ec0 243'592 rep_tid=17 committed?=0 applied?=1 lock=0 op=osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4) 0x30bbfb80 2014-06-09 16:22:11.981627 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.981630 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x4eb9240,0xbd762c0,0xbd6a460,0xbd60260,0x2d739380,0xbfcf500,0xcc95dc0,0x17fb7400,0x31791f40,0xbd798a0,0x17fb7a80,0x3062bc20,0xbd5efc0,0xcd0e3c0,0x2d7381e0,0xbd789c0,0xc2d6e20,0x31456a60,0x1e08efa0,0x3042b280,0xbd605a0,0x17fb7900,0xd9fb860,0xd9fb6a0,0xd9fb560,0x324f86e0,0xbccd8a0,0xcd334c0,0xbd71b40,0x3006dfc0,0xbd75bc0,0x15a3fa80,0x318c2720,0x3281bdc0,0xcd9ea40,0x324f8260,0x324f8be0,0x1322d960,0x3251ca20,0x33a7f60,0x3006c760,0x31412e60,0x31435e40,0xbd75880,0xbd6cb60,0x327f3700,0x32481600,0x31412fa0,0x32480880,0x33a7400,0x12051740,0xbd698e0,0xc1697a0,0x324f87c0] 2014-06-09 16:22:11.981640 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f8e20,0x305089c0,0x306ce9a0,0x3061e640,0x17001b40,0x318c2fa0,0x30e18a40,0x318c2a20,0x3050bb40,0xbd5b7a0,0xbebf3e0,0x314128c0,0x31413380,0xccf8ba0,0x3050ae20,0xbd6bd60,0x314135c0,0x30e19880] 2014-06-09 16:22:11.981642 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3712.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.981647 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.981658 7f86a5e11700 10 _do_op 0x15997ce0 seq 3712 r = 0, finisher 0x3061f960 0 2014-06-09 16:22:11.981662 7f86a5e11700 10 _finish_op 0x15997ce0 seq 3712 osr(2.27a 0x2ed0d90)/0x2ed0d90 2014-06-09 16:22:11.981665 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300fc0 (1 active) 2014-06-09 16:22:11.981667 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338ea0 (2 active) 2014-06-09 16:22:11.981671 7f86a5e11700 5 _do_op 0x2fcaf9e0 seq 3713 osr(3.319 0x3ed0d60)/0x3ed0d60 start 2014-06-09 16:22:11.981673 7f86a5e11700 10 _do_transaction on 0xc129d40 2014-06-09 16:22:11.981680 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.981677 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f8e20,0x305089c0,0x306ce9a0,0x3061e640,0x17001b40,0x318c2fa0,0x30e18a40,0x318c2a20,0x3050bb40,0xbd5b7a0,0xbebf3e0,0x314128c0,0x31413380,0xccf8ba0,0x3050ae20,0xbd6bd60,0x314135c0,0x30e19880] 2014-06-09 16:22:11.981686 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061f960] 2014-06-09 16:22:11.981690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061f960] 2014-06-09 16:22:11.981691 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.981692 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.981699 7f869af78700 5 -- op tracker -- , seq: 3409, time: 2014-06-09 16:22:11.981460, event: done, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.981712 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.981714 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.981714 7f868ef60700 10 merge_log result log((0'0,214'683], crt=214'683) missing(1) changed=1 2014-06-09 16:22:11.981720 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.981718 7f868f761700 5 -- op tracker -- , seq: 3362, time: 2014-06-09 16:22:11.981474, event: done, op: osd_sub_op_reply(client.4571.0:36397 3.3f5 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.981728 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.981729 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.981731 7f869af78700 20 osd.26 243 _dispatch 0x30de5880 pg_info(1 pgs e243:3.6bb) v4 2014-06-09 16:22:11.981719 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 peering m=1] peer osd.8 now 3.1ff( v 214'683 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) missing(0) 2014-06-09 16:22:11.981731 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.981735 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3713.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.981739 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.981735 7f869af78700 5 -- op tracker -- , seq: 3410, time: 2014-06-09 16:22:11.969052, event: header_read, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.981742 7f869af78700 5 -- op tracker -- , seq: 3410, time: 2014-06-09 16:22:11.969053, event: throttled, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.981739 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetLog 0.732141 2 0.000165 2014-06-09 16:22:11.981748 7f869af78700 5 -- op tracker -- , seq: 3410, time: 2014-06-09 16:22:11.969066, event: all_read, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.981751 7f86a5e11700 10 _do_op 0x2fcaf9e0 seq 3713 r = 0, finisher 0xc6e99e0 0 2014-06-09 16:22:11.981754 7f86a5e11700 10 _finish_op 0x2fcaf9e0 seq 3713 osr(3.319 0x3ed0d60)/0x3ed0d60 2014-06-09 16:22:11.981754 7f869af78700 5 -- op tracker -- , seq: 3410, time: 2014-06-09 16:22:11.981727, event: dispatched, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.981759 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338ea0 (1 active) 2014-06-09 16:22:11.981752 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 peering m=1] enter Started/Primary/Peering/GetMissing 2014-06-09 16:22:11.981761 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff58c0 (2 active) 2014-06-09 16:22:11.981760 7f869af78700 5 -- op tracker -- , seq: 3410, time: 2014-06-09 16:22:11.981758, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.981763 7f86a5e11700 5 _do_op 0x14dd6d80 seq 3714 osr(3.77 0x3ed0250)/0x3ed0250 start 2014-06-09 16:22:11.981765 7f86a5e11700 10 _do_transaction on 0xbf36f00 2014-06-09 16:22:11.981764 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6bb) v4 from osd.5 2014-06-09 16:22:11.981766 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153599f0 (20 active) 2014-06-09 16:22:11.981767 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5880 2014-06-09 16:22:11.981770 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.981763 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 peering m=1] state: osd.8 has no missing, identical log 2014-06-09 16:22:11.981772 7f869af78700 5 -- op tracker -- , seq: 3410, time: 2014-06-09 16:22:11.981770, event: started, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.981770 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.799( v 214'665 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:11.981777 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering/GetMissing 0.000025 0 0.000000 2014-06-09 16:22:11.981787 7f869af78700 5 -- op tracker -- , seq: 3410, time: 2014-06-09 16:22:11.981785, event: done, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.981794 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.981796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.981789 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 peering m=1] state: Leaving Peering 2014-06-09 16:22:11.981788 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.9 activated and committed 2014-06-09 16:22:11.981805 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.981799 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 peering m=1] exit Started/Primary/Peering 1.022956 0 0.000000 2014-06-09 16:22:11.981808 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.981801 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:11.981809 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.981811 7f869af78700 20 osd.26 243 _dispatch 0x16064c40 pg_info(1 pgs e243:3.5ce) v4 2014-06-09 16:22:11.981815 7f869af78700 5 -- op tracker -- , seq: 3411, time: 2014-06-09 16:22:11.963251, event: header_read, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:11.981817 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3714.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.981811 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:11.981821 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.981815 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active 2014-06-09 16:22:11.981822 7f869af78700 5 -- op tracker -- , seq: 3411, time: 2014-06-09 16:22:11.963251, event: throttled, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:11.981828 7f868f761700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:11.981828 7f869af78700 5 -- op tracker -- , seq: 3411, time: 2014-06-09 16:22:11.963266, event: all_read, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:11.981825 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] state: In Active, about to call activate 2014-06-09 16:22:11.981834 7f869af78700 5 -- op tracker -- , seq: 3411, time: 2014-06-09 16:22:11.981808, event: dispatched, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:11.981840 7f869af78700 5 -- op tracker -- , seq: 3411, time: 2014-06-09 16:22:11.981839, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:11.981844 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5ce) v4 from osd.2 2014-06-09 16:22:11.981838 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:11.981847 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16064c40 2014-06-09 16:22:11.981851 7f86a5e11700 10 _do_op 0x14dd6d80 seq 3714 r = 0, finisher 0xbd7d0c0 0 2014-06-09 16:22:11.981851 7f869af78700 5 -- op tracker -- , seq: 3411, time: 2014-06-09 16:22:11.981850, event: started, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:11.981854 7f86a5e11700 10 _finish_op 0x14dd6d80 seq 3714 osr(3.77 0x3ed0250)/0x3ed0250 2014-06-09 16:22:11.981848 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] activate - snap_trimq [] 2014-06-09 16:22:11.981845 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1103 2014-06-09 16:22:11.981857 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff58c0 (1 active) 2014-06-09 16:22:11.981859 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115b00 (2 active) 2014-06-09 16:22:11.981861 7f86a5e11700 5 _do_op 0x2fcaf200 seq 3715 osr(0.220 0x2bd6a70)/0x2bd6a70 start 2014-06-09 16:22:11.981863 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.981863 7f86a5e11700 10 _do_transaction on 0xc31cc00 2014-06-09 16:22:11.981856 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'681 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:11.981868 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.981866 7f869af78700 5 -- op tracker -- , seq: 3411, time: 2014-06-09 16:22:11.981864, event: done, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:11.981873 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.981874 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.981867 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.74f( v 213'540 lc 213'533 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.981887 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.981888 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.981880 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:11.981889 7f869af78700 20 osd.26 243 _dispatch 0x309e68c0 pg_info(1 pgs e243:1.504) v4 2014-06-09 16:22:11.981894 7f869af78700 5 -- op tracker -- , seq: 3412, time: 2014-06-09 16:22:11.957286, event: header_read, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:11.981890 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.981900 7f869af78700 5 -- op tracker -- , seq: 3412, time: 2014-06-09 16:22:11.957287, event: throttled, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:11.981902 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.981899 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.981906 7f869af78700 5 -- op tracker -- , seq: 3412, time: 2014-06-09 16:22:11.957299, event: all_read, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:11.981912 7f868f761700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:11.981912 7f869af78700 5 -- op tracker -- , seq: 3412, time: 2014-06-09 16:22:11.981886, event: dispatched, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:11.981915 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3715.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.981919 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.981918 7f869af78700 5 -- op tracker -- , seq: 3412, time: 2014-06-09 16:22:11.981916, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:11.981922 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.504) v4 from osd.4 2014-06-09 16:22:11.981924 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e68c0 2014-06-09 16:22:11.981928 7f86a5e11700 10 _do_op 0x2fcaf200 seq 3715 r = 0, finisher 0xbdf1960 0 2014-06-09 16:22:11.981931 7f86a5e11700 10 _finish_op 0x2fcaf200 seq 3715 osr(0.220 0x2bd6a70)/0x2bd6a70 2014-06-09 16:22:11.981929 7f869af78700 5 -- op tracker -- , seq: 3412, time: 2014-06-09 16:22:11.981927, event: started, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:11.981933 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115b00 (1 active) 2014-06-09 16:22:11.981935 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6ddc20 (2 active) 2014-06-09 16:22:11.981927 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:879 2014-06-09 16:22:11.981937 7f86a5e11700 5 _do_op 0x2e21620 seq 3716 osr(3.5b3 0x3ed17e0)/0x3ed17e0 start 2014-06-09 16:22:11.981940 7f86a5e11700 10 _do_transaction on 0x32fa6c0 2014-06-09 16:22:11.981942 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.981944 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.981946 7f869af78700 5 -- op tracker -- , seq: 3412, time: 2014-06-09 16:22:11.981944, event: done, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:11.981953 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.981954 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.981946 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.48e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.981965 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.981966 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.981967 7f869af78700 20 osd.26 243 _dispatch 0x30c77a40 pg_info(1 pgs e243:3.4c5) v4 2014-06-09 16:22:11.981972 7f869af78700 5 -- op tracker -- , seq: 3413, time: 2014-06-09 16:22:11.959123, event: header_read, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.981978 7f869af78700 5 -- op tracker -- , seq: 3413, time: 2014-06-09 16:22:11.959124, event: throttled, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.981979 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.981981 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.981983 7f869af78700 5 -- op tracker -- , seq: 3413, time: 2014-06-09 16:22:11.959134, event: all_read, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.981985 7f868f761700 5 queue_transactions existing osr(1.48e 0x384d9b0)/0x384d9b0 2014-06-09 16:22:11.981989 7f869af78700 5 -- op tracker -- , seq: 3413, time: 2014-06-09 16:22:11.981965, event: dispatched, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.981993 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3716.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.981996 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.981988 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] activate - starting recovery 2014-06-09 16:22:11.981995 7f869af78700 5 -- op tracker -- , seq: 3413, time: 2014-06-09 16:22:11.981993, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.981999 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c5) v4 from osd.7 2014-06-09 16:22:11.982002 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c77a40 2014-06-09 16:22:11.982002 7f868ef60700 10 osd.26 243 queue_for_recovery queued pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:11.982007 7f86a5e11700 10 _do_op 0x2e21620 seq 3716 r = 0, finisher 0xbcc9080 0 2014-06-09 16:22:11.982007 7f869af78700 5 -- op tracker -- , seq: 3413, time: 2014-06-09 16:22:11.982005, event: started, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.982011 7f86a5e11700 10 _finish_op 0x2e21620 seq 3716 osr(3.5b3 0x3ed17e0)/0x3ed17e0 2014-06-09 16:22:11.982014 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6ddc20 (1 active) 2014-06-09 16:22:11.982016 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6527e0 (2 active) 2014-06-09 16:22:11.982018 7f86a5e11700 5 _do_op 0x2e21ce0 seq 3717 osr(3.5c6 0x3ed17f0)/0x3ed17f0 start 2014-06-09 16:22:11.982021 7f86a5e11700 10 _do_transaction on 0x480d500 2014-06-09 16:22:11.982015 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.8 3.1ff( v 214'683 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) 2014-06-09 16:22:11.982026 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.982021 7f869af78700 5 -- op tracker -- , seq: 3413, time: 2014-06-09 16:22:11.982019, event: done, op: pg_info(1 pgs e243:3.4c5) 2014-06-09 16:22:11.982031 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982032 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982027 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.8 is up to date, queueing in pending_activators 2014-06-09 16:22:11.982043 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982044 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982045 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a000 pg_info(1 pgs e243:3.1a0) v4 2014-06-09 16:22:11.982049 7f869af78700 5 -- op tracker -- , seq: 3414, time: 2014-06-09 16:22:11.972357, event: header_read, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:11.982042 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] activate peer osd.8 3.1ff( v 214'683 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242) uptodate 2014-06-09 16:22:11.982061 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e99e0,0xbd7d0c0,0xbdf1960,0xbcc9080] 2014-06-09 16:22:11.982055 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery primary has 1 missing 2014-06-09 16:22:11.982066 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.982070 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e99e0,0xbd7d0c0,0xbdf1960,0xbcc9080] 2014-06-09 16:22:11.982073 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.982074 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.982066 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 547499ff/rbd_data.115f6b8b4567.00000000000016a4/head//3 214'683 also missing on osd.26 2014-06-09 16:22:11.982055 7f869af78700 5 -- op tracker -- , seq: 3414, time: 2014-06-09 16:22:11.972358, event: throttled, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:11.982081 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3717.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.982085 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.982084 7f869af78700 5 -- op tracker -- , seq: 3414, time: 2014-06-09 16:22:11.972371, event: all_read, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:11.982077 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1 u=1] needs_recovery_map missing {547499ff/rbd_data.115f6b8b4567.00000000000016a4/head//3=214'683(214'681)} 2014-06-09 16:22:11.982091 7f869af78700 5 -- op tracker -- , seq: 3414, time: 2014-06-09 16:22:11.982042, event: dispatched, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:11.982089 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1 u=1] activate: adding 8 as a source 2014-06-09 16:22:11.982097 7f869af78700 5 -- op tracker -- , seq: 3414, time: 2014-06-09 16:22:11.982095, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:11.982098 7f86a5e11700 10 _do_op 0x2e21ce0 seq 3717 r = 0, finisher 0x30e15e20 0 2014-06-09 16:22:11.982101 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1a0) v4 from osd.3 2014-06-09 16:22:11.982102 7f86a5e11700 10 _finish_op 0x2e21ce0 seq 3717 osr(3.5c6 0x3ed17f0)/0x3ed17f0 2014-06-09 16:22:11.982104 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5a000 2014-06-09 16:22:11.982107 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6527e0 (1 active) 2014-06-09 16:22:11.982099 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1 u=1] search_for_missing 547499ff/rbd_data.115f6b8b4567.00000000000016a4/head//3 214'683 is on osd.8 2014-06-09 16:22:11.982109 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c1e60 (2 active) 2014-06-09 16:22:11.982112 7f86a5e11700 5 _do_op 0x2e205a0 seq 3718 osr(0.458 0x2bd7510)/0x2bd7510 start 2014-06-09 16:22:11.982109 7f869af78700 5 -- op tracker -- , seq: 3414, time: 2014-06-09 16:22:11.982107, event: started, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:11.982114 7f86a5e11700 10 _do_transaction on 0xbf38fc0 2014-06-09 16:22:11.982111 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] needs_recovery_map missing {547499ff/rbd_data.115f6b8b4567.00000000000016a4/head//3=214'683(214'681)} 2014-06-09 16:22:11.982121 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.982126 7f869af78700 5 -- op tracker -- , seq: 3414, time: 2014-06-09 16:22:11.982124, event: done, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:11.982123 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound 2014-06-09 16:22:11.982133 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982135 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982134 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] _calc_past_interval_range: already have past intervals back to 213 2014-06-09 16:22:11.982146 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982148 7f869af78700 20 osd.26 243 _dispatch 0x309e61c0 pg_info(1 pgs e243:0.70e) v4 2014-06-09 16:22:11.982145 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] build_might_have_unfound: built 8 2014-06-09 16:22:11.982152 7f869af78700 5 -- op tracker -- , seq: 3415, time: 2014-06-09 16:22:11.959413, event: header_read, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.982158 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.982159 7f869af78700 5 -- op tracker -- , seq: 3415, time: 2014-06-09 16:22:11.959414, event: throttled, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.982155 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:11.982164 7f869af78700 5 -- op tracker -- , seq: 3415, time: 2014-06-09 16:22:11.959426, event: all_read, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.982172 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3718.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.982166 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] enter Started/Primary/Active/Activating 2014-06-09 16:22:11.982177 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.982170 7f869af78700 5 -- op tracker -- , seq: 3415, time: 2014-06-09 16:22:11.982145, event: dispatched, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.982182 7f869af78700 5 -- op tracker -- , seq: 3415, time: 2014-06-09 16:22:11.982180, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.982188 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.70e) v4 from osd.1 2014-06-09 16:22:11.982191 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e61c0 2014-06-09 16:22:11.982195 7f86a5e11700 10 _do_op 0x2e205a0 seq 3718 r = 0, finisher 0xbd72560 0 2014-06-09 16:22:11.982199 7f86a5e11700 10 _finish_op 0x2e205a0 seq 3718 osr(0.458 0x2bd7510)/0x2bd7510 2014-06-09 16:22:11.982195 7f869af78700 5 -- op tracker -- , seq: 3415, time: 2014-06-09 16:22:11.982193, event: started, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.982202 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c1e60 (1 active) 2014-06-09 16:22:11.982202 7f868ef60700 10 write_log with: dirty_to: 0'0, dirty_from: 214'681, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:11.982204 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4b40 (2 active) 2014-06-09 16:22:11.982207 7f86a5e11700 5 _do_op 0x30d97bc0 seq 3719 osr(3.616 0x3ed1a90)/0x3ed1a90 start 2014-06-09 16:22:11.982210 7f86a5e11700 10 _do_transaction on 0xc10ba40 2014-06-09 16:22:11.982211 7f869af78700 5 -- op tracker -- , seq: 3415, time: 2014-06-09 16:22:11.982209, event: done, op: pg_info(1 pgs e243:0.70e) 2014-06-09 16:22:11.982215 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.982221 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982223 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982234 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982236 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982237 7f869af78700 20 osd.26 243 _dispatch 0x15a28a80 pg_info(1 pgs e243:1.5cb) v4 2014-06-09 16:22:11.982238 7f868ef60700 5 queue_transactions existing osr(3.1ff 0x3ed0940)/0x3ed0940 2014-06-09 16:22:11.982241 7f869af78700 5 -- op tracker -- , seq: 3416, time: 2014-06-09 16:22:11.958750, event: header_read, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.982250 7f869af78700 5 -- op tracker -- , seq: 3416, time: 2014-06-09 16:22:11.958750, event: throttled, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.982252 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.982256 7f869af78700 5 -- op tracker -- , seq: 3416, time: 2014-06-09 16:22:11.958762, event: all_read, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.982263 7f869af78700 5 -- op tracker -- , seq: 3416, time: 2014-06-09 16:22:11.982234, event: dispatched, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.982266 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3719.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.982271 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.982273 7f869af78700 5 -- op tracker -- , seq: 3416, time: 2014-06-09 16:22:11.982270, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.982275 7f8688f54700 12 OSD::recovery_tp worker wq OSD::RecoveryWQ start processing 0x69d0000 (1 active) 2014-06-09 16:22:11.982277 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5cb) v4 from osd.9 2014-06-09 16:22:11.982279 7f8688f54700 10 osd.26 243 do_recovery can start 5 (0/15 rops) 2014-06-09 16:22:11.982280 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15a28a80 2014-06-09 16:22:11.982283 7f86a5e11700 10 _do_op 0x30d97bc0 seq 3719 r = 0, finisher 0x189aa540 0 2014-06-09 16:22:11.982287 7f86a5e11700 10 _finish_op 0x30d97bc0 seq 3719 osr(3.616 0x3ed1a90)/0x3ed1a90 2014-06-09 16:22:11.982284 7f869af78700 5 -- op tracker -- , seq: 3416, time: 2014-06-09 16:22:11.982282, event: started, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.982289 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4b40 (1 active) 2014-06-09 16:22:11.982292 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358ea0 (2 active) 2014-06-09 16:22:11.982294 7f86a5e11700 5 _do_op 0xbdc8600 seq 3720 osr(3.ff 0x3ed0870)/0x3ed0870 start 2014-06-09 16:22:11.982297 7f86a5e11700 10 _do_transaction on 0xbf275c0 2014-06-09 16:22:11.982301 7f86a5e11700 15 _omap_setkeys meta/516eb9d7/pglog_3.ff/0//-1 2014-06-09 16:22:11.982302 7f869af78700 5 -- op tracker -- , seq: 3416, time: 2014-06-09 16:22:11.982300, event: done, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.982305 7f868e75f700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.982309 7f8689755700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.982310 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982311 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982313 7f868a757700 2 waiting 51 > 50 ops || 1223465 > 104857600 2014-06-09 16:22:11.982317 7f8691f66700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.982321 7f868bf5a700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.982325 7f8690763700 2 waiting 51 > 50 ops || 1205736 > 104857600 2014-06-09 16:22:11.982326 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982327 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982328 7f869af78700 20 osd.26 243 _dispatch 0x30fcdc00 pg_info(1 pgs e243:1.35c) v4 2014-06-09 16:22:11.982329 7f868cf5c700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.982333 7f8692767700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.982333 7f869af78700 5 -- op tracker -- , seq: 3417, time: 2014-06-09 16:22:11.967032, event: header_read, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.982340 7f869af78700 5 -- op tracker -- , seq: 3417, time: 2014-06-09 16:22:11.967032, event: throttled, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.982344 7f868b759700 2 waiting 51 > 50 ops || 1205541 > 104857600 2014-06-09 16:22:11.982343 7f86a5e11700 10 lookup_strip_header done cid meta oid 516eb9d7/pglog_3.ff/0//-1 2014-06-09 16:22:11.982358 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.977143 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x29282d00,0x13964c20] 2014-06-09 16:22:11.982366 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x29282d00,0x13964c20] 2014-06-09 16:22:11.982368 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e19800,0x31457a20,0x1a002c60,0xbdf10e0,0x2d7397a0,0x30e18380,0x30e15f00,0xbd660c0,0xc5a70a0] 2014-06-09 16:22:11.982373 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] _finish_recovery 2014-06-09 16:22:11.982385 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] purge_strays 2014-06-09 16:22:11.982394 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.982396 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.319( v 243'609 (0'0,243'609] local-les=243 n=115 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 243'608 mlcod 243'608 active+clean] publish_stats_to_osd 243:981 2014-06-09 16:22:11.982407 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e19800,0x31457a20,0x1a002c60,0xbdf10e0,0x2d7397a0,0x30e18380,0x30e15f00,0xbd660c0,0xc5a70a0] 2014-06-09 16:22:11.982410 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.982411 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.977147 7f868df5e700 10 op_submit_start 3759 2014-06-09 16:22:11.982411 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3710.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.982415 7f868df5e700 5 queue_transactions (trailing journal) 3759 2014-06-09 16:22:11.982416 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.982417 7f868df5e700 5 queue_op 0xbf7f8c0 seq 3759 osr(2.654 0x2ed1fd0) 828 bytes (queue has 50 ops and 1204683 bytes) 2014-06-09 16:22:11.982421 7f868df5e700 10 op_submit_finish 3759 2014-06-09 16:22:11.982426 7f868f761700 10 op_submit_start 3760 2014-06-09 16:22:11.982428 7f868f761700 5 queue_transactions (trailing journal) 3760 2014-06-09 16:22:11.982429 7f86a5610700 10 _do_op 0x33a2d20 seq 3710 r = 0, finisher 0xbd5a480 0 2014-06-09 16:22:11.982429 7f868f761700 5 queue_op 0xc2d5500 seq 3760 osr(1.48e 0x384d9b0) 828 bytes (queue has 50 ops and 1204683 bytes) 2014-06-09 16:22:11.982432 7f868f761700 10 op_submit_finish 3760 2014-06-09 16:22:11.982436 7f86a5610700 10 _finish_op 0x33a2d20 seq 3710 osr(1.560 0x384da90)/0x384da90 2014-06-09 16:22:11.982427 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.67d( v 214'684 lc 213'683 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.982442 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc7e0 (1 active) 2014-06-09 16:22:11.982444 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3007e0 (2 active) 2014-06-09 16:22:11.982437 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.616( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:11.982447 7f86a5610700 5 _do_op 0x159966c0 seq 3721 osr(2.d3 0x2ed0450)/0x2ed0450 start 2014-06-09 16:22:11.982450 7f86a5610700 10 _do_transaction on 0x34735c0 2014-06-09 16:22:11.982442 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.982455 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.982452 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.982460 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.982468 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.982472 7f868f761700 5 queue_transactions existing osr(0.616 0x2bd7950)/0x2bd7950 2014-06-09 16:22:11.982472 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.982475 7f868f761700 10 op_submit_start 3761 2014-06-09 16:22:11.982476 7f868f761700 5 queue_transactions (trailing journal) 3761 2014-06-09 16:22:11.982477 7f868f761700 5 queue_op 0xc2d45a0 seq 3761 osr(0.616 0x2bd7950) 828 bytes (queue has 50 ops and 1204683 bytes) 2014-06-09 16:22:11.982479 7f868f761700 10 op_submit_finish 3761 2014-06-09 16:22:11.982482 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.4fb( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:11.982487 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1198 2014-06-09 16:22:11.982493 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.982500 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.982507 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3721.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.982511 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.982512 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.982504 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.982515 7f868f761700 5 queue_transactions existing osr(1.4fb 0x384d7f0)/0x384d7f0 2014-06-09 16:22:11.982517 7f868f761700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.982517 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.982524 7f86a5610700 10 _do_op 0x159966c0 seq 3721 r = 0, finisher 0xbebe780 0 2014-06-09 16:22:11.982528 7f86a5610700 10 _finish_op 0x159966c0 seq 3721 osr(2.d3 0x2ed0450)/0x2ed0450 2014-06-09 16:22:11.982528 7f868ef60700 10 op_submit_start 3762 2014-06-09 16:22:11.982530 7f868ef60700 5 queue_transactions (trailing journal) 3762 2014-06-09 16:22:11.982525 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.982533 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3007e0 (1 active) 2014-06-09 16:22:11.982532 7f868ef60700 5 queue_op 0x308ca420 seq 3762 osr(3.1ff 0x3ed0940) 1943 bytes (queue has 49 ops and 1203857 bytes) 2014-06-09 16:22:11.982535 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525200 (2 active) 2014-06-09 16:22:11.982538 7f8689755700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982533 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.982539 7f86a5610700 5 _do_op 0xbf7f440 seq 3722 osr(0.25b 0x2bd69f0)/0x2bd69f0 start 2014-06-09 16:22:11.982542 7f86a5610700 10 _do_transaction on 0x3488900 2014-06-09 16:22:11.982543 7f868a757700 2 waiting 51 > 50 ops || 1223467 > 104857600 2014-06-09 16:22:11.982545 7f868df5e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.982547 7f8691f66700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982547 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.982551 7f868bf5a700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982555 7f8690763700 2 waiting 51 > 50 ops || 1205738 > 104857600 2014-06-09 16:22:11.982559 7f868cf5c700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982563 7f8692767700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982567 7f868b759700 2 waiting 51 > 50 ops || 1205543 > 104857600 2014-06-09 16:22:11.982560 7f868df5e700 15 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.982574 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.982585 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.982578 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:11.982600 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3722.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.982604 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.982614 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.982615 7f86a5610700 10 _do_op 0xbf7f440 seq 3722 r = 0, finisher 0x19940b60 0 2014-06-09 16:22:11.982618 7f868df5e700 5 queue_transactions existing osr(3.98 0x3ed0620)/0x3ed0620 2014-06-09 16:22:11.982619 7f86a5610700 10 _finish_op 0xbf7f440 seq 3722 osr(0.25b 0x2bd69f0)/0x2bd69f0 2014-06-09 16:22:11.982621 7f868df5e700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.982647 7f868e75f700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982653 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc40c020,0xbcc9380,0x327f2d00] 2014-06-09 16:22:11.982658 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc40c020,0xbcc9380,0x327f2d00] 2014-06-09 16:22:11.982660 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.982661 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.982660 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525200 (1 active) 2014-06-09 16:22:11.982663 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62efc0 (2 active) 2014-06-09 16:22:11.982663 7f8691f66700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982666 7f86a5610700 5 _do_op 0x30d968a0 seq 3723 osr(2.e4 0x2ed0410)/0x2ed0410 start 2014-06-09 16:22:11.982668 7f86a5610700 10 _do_transaction on 0xbf8aa80 2014-06-09 16:22:11.982669 7f868bf5a700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982673 7f8690763700 2 waiting 51 > 50 ops || 1205738 > 104857600 2014-06-09 16:22:11.982674 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.982677 7f868cf5c700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982681 7f8692767700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982346 7f869af78700 5 -- op tracker -- , seq: 3417, time: 2014-06-09 16:22:11.967044, event: all_read, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.982685 7f868a757700 2 waiting 51 > 50 ops || 1223467 > 104857600 2014-06-09 16:22:11.982689 7f868b759700 2 waiting 51 > 50 ops || 1205543 > 104857600 2014-06-09 16:22:11.982687 7f869af78700 5 -- op tracker -- , seq: 3417, time: 2014-06-09 16:22:11.982325, event: dispatched, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.982694 7f869af78700 5 -- op tracker -- , seq: 3417, time: 2014-06-09 16:22:11.982692, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.982698 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.35c) v4 from osd.8 2014-06-09 16:22:11.982702 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcdc00 2014-06-09 16:22:11.982703 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.982707 7f869af78700 5 -- op tracker -- , seq: 3417, time: 2014-06-09 16:22:11.982705, event: started, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.982721 7f86a5e11700 15 _omap_rmkeys meta/516eb9d7/pglog_3.ff/0//-1 2014-06-09 16:22:11.982725 7f869af78700 5 -- op tracker -- , seq: 3417, time: 2014-06-09 16:22:11.982722, event: done, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.982727 7f86a5e11700 10 _omap_rmkeys meta/516eb9d7/pglog_3.ff/0//-1 r = 0 2014-06-09 16:22:11.982733 7f86a5e11700 15 _omap_setkeys meta/516eb9d7/pglog_3.ff/0//-1 2014-06-09 16:22:11.982738 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982740 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982747 7f86a5e11700 15 _touch 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.982753 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982754 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982756 7f869af78700 20 osd.26 243 _dispatch 0x26000c40 pg_info(1 pgs e243:1.233) v4 2014-06-09 16:22:11.982761 7f869af78700 5 -- op tracker -- , seq: 3418, time: 2014-06-09 16:22:11.957381, event: header_read, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.982767 7f869af78700 5 -- op tracker -- , seq: 3418, time: 2014-06-09 16:22:11.957381, event: throttled, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.982774 7f869af78700 5 -- op tracker -- , seq: 3418, time: 2014-06-09 16:22:11.957393, event: all_read, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.982784 7f868e75f700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982780 7f869af78700 5 -- op tracker -- , seq: 3418, time: 2014-06-09 16:22:11.982752, event: dispatched, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.982790 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e15e20,0xbd72560,0x189aa540,0xbd5a480,0xbebe780,0x19940b60] 2014-06-09 16:22:11.982792 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.ff_head oid f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.982795 7f869af78700 5 -- op tracker -- , seq: 3418, time: 2014-06-09 16:22:11.982793, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.982800 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.233) v4 from osd.6 2014-06-09 16:22:11.982801 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e15e20,0xbd72560,0x189aa540,0xbd5a480,0xbebe780,0x19940b60] 2014-06-09 16:22:11.982804 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.982804 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26000c40 2014-06-09 16:22:11.982805 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.982809 7f869af78700 5 -- op tracker -- , seq: 3418, time: 2014-06-09 16:22:11.982807, event: started, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.982824 7f869af78700 5 -- op tracker -- , seq: 3418, time: 2014-06-09 16:22:11.982822, event: done, op: pg_info(1 pgs e243:1.233) 2014-06-09 16:22:11.982833 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982835 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982835 7f86a5e11700 20 write_state seq is 30157 2014-06-09 16:22:11.982842 7f86a5e11700 20 set_header setting 30156 cid 3.ff_head oid f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 parent seq 0 2014-06-09 16:22:11.982847 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982849 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982850 7f869af78700 20 osd.26 243 _dispatch 0x309e7dc0 pg_info(1 pgs e243:1.3bc) v4 2014-06-09 16:22:11.982855 7f869af78700 5 -- op tracker -- , seq: 3419, time: 2014-06-09 16:22:11.956870, event: header_read, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.982858 7f86a5e11700 10 _touch 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 = 0 2014-06-09 16:22:11.982862 7f869af78700 5 -- op tracker -- , seq: 3419, time: 2014-06-09 16:22:11.956870, event: throttled, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.982864 7f86a5e11700 15 _write 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 0~106496 2014-06-09 16:22:11.982868 7f869af78700 5 -- op tracker -- , seq: 3419, time: 2014-06-09 16:22:11.956879, event: all_read, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.982872 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:11.982873 7f869af78700 5 -- op tracker -- , seq: 3419, time: 2014-06-09 16:22:11.982847, event: dispatched, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.982879 7f868ef60700 10 op_submit_finish 3762 2014-06-09 16:22:11.982879 7f869af78700 5 -- op tracker -- , seq: 3419, time: 2014-06-09 16:22:11.982877, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.982883 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3bc) v4 from osd.0 2014-06-09 16:22:11.982886 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e7dc0 2014-06-09 16:22:11.982891 7f869af78700 5 -- op tracker -- , seq: 3419, time: 2014-06-09 16:22:11.982889, event: started, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.982888 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.34d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.982899 7f8692f68700 10 op_submit_start 3763 2014-06-09 16:22:11.982901 7f8692f68700 5 queue_transactions (trailing journal) 3763 2014-06-09 16:22:11.982903 7f8692f68700 5 queue_op 0x324e6600 seq 3763 osr(1.697 0x384de10) 828 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.982906 7f8692f68700 10 op_submit_finish 3763 2014-06-09 16:22:11.982906 7f869af78700 5 -- op tracker -- , seq: 3419, time: 2014-06-09 16:22:11.982903, event: done, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.982913 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982914 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982920 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.982911 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.982924 7f868ef60700 5 queue_transactions existing osr(2.34d 0x2ed0e70)/0x2ed0e70 2014-06-09 16:22:11.982926 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982926 7f868ef60700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.982928 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982929 7f869af78700 20 osd.26 243 _dispatch 0x2fc941c0 pg_info(1 pgs e243:3.6bb) v4 2014-06-09 16:22:11.982924 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.982933 7f869af78700 5 -- op tracker -- , seq: 3420, time: 2014-06-09 16:22:11.969110, event: header_read, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.982933 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.982940 7f869af78700 5 -- op tracker -- , seq: 3420, time: 2014-06-09 16:22:11.969111, event: throttled, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.982946 7f869af78700 5 -- op tracker -- , seq: 3420, time: 2014-06-09 16:22:11.969129, event: all_read, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.982942 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.982952 7f869af78700 5 -- op tracker -- , seq: 3420, time: 2014-06-09 16:22:11.982925, event: dispatched, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.982951 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.982959 7f869af78700 5 -- op tracker -- , seq: 3420, time: 2014-06-09 16:22:11.982957, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.982963 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.6bb) v4 from osd.5 2014-06-09 16:22:11.982966 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc941c0 2014-06-09 16:22:11.982961 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.982971 7f869af78700 5 -- op tracker -- , seq: 3420, time: 2014-06-09 16:22:11.982969, event: started, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.982976 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:183 2014-06-09 16:22:11.982986 7f869af78700 5 -- op tracker -- , seq: 3420, time: 2014-06-09 16:22:11.982984, event: done, op: pg_info(1 pgs e243:3.6bb) 2014-06-09 16:22:11.982985 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.982995 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.982997 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.982994 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.360( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.983004 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.983009 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983010 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983011 7f869af78700 20 osd.26 243 _dispatch 0x30ef7180 pg_info(1 pgs e243:0.4d5) v4 2014-06-09 16:22:11.983016 7f869af78700 5 -- op tracker -- , seq: 3421, time: 2014-06-09 16:22:11.963274, event: header_read, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:11.983009 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.983023 7f869af78700 5 -- op tracker -- , seq: 3421, time: 2014-06-09 16:22:11.963275, event: throttled, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:11.983029 7f869af78700 5 -- op tracker -- , seq: 3421, time: 2014-06-09 16:22:11.963282, event: all_read, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:11.983024 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.983036 7f869af78700 5 -- op tracker -- , seq: 3421, time: 2014-06-09 16:22:11.983008, event: dispatched, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:11.983034 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x11e63300,0x14052c00,0x14052b00,0x14052a00,0x14052900,0x14052800,0x16059a00,0x16059900 2014-06-09 16:22:11.983042 7f869af78700 5 -- op tracker -- , seq: 3421, time: 2014-06-09 16:22:11.983040, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:11.983047 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4d5) v4 from osd.2 2014-06-09 16:22:11.983049 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef7180 2014-06-09 16:22:11.983054 7f869af78700 5 -- op tracker -- , seq: 3421, time: 2014-06-09 16:22:11.983052, event: started, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:11.983069 7f869af78700 5 -- op tracker -- , seq: 3421, time: 2014-06-09 16:22:11.983067, event: done, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:11.983066 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.983076 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983076 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.983091 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983093 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983086 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.983094 7f869af78700 20 osd.26 243 _dispatch 0x2a0708c0 pg_info(1 pgs e243:3.52b) v4 2014-06-09 16:22:11.983099 7f869af78700 5 -- op tracker -- , seq: 3422, time: 2014-06-09 16:22:11.960379, event: header_read, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.983106 7f869af78700 5 -- op tracker -- , seq: 3422, time: 2014-06-09 16:22:11.960380, event: throttled, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.983100 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1011 2014-06-09 16:22:11.983112 7f869af78700 5 -- op tracker -- , seq: 3422, time: 2014-06-09 16:22:11.960392, event: all_read, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.983111 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.983118 7f869af78700 5 -- op tracker -- , seq: 3422, time: 2014-06-09 16:22:11.983090, event: dispatched, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.983124 7f869af78700 5 -- op tracker -- , seq: 3422, time: 2014-06-09 16:22:11.983122, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.983121 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.983129 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.52b) v4 from osd.4 2014-06-09 16:22:11.983131 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.983132 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2a0708c0 2014-06-09 16:22:11.983136 7f869af78700 5 -- op tracker -- , seq: 3422, time: 2014-06-09 16:22:11.983135, event: started, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.983136 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.983148 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.983152 7f869af78700 5 -- op tracker -- , seq: 3422, time: 2014-06-09 16:22:11.983150, event: done, op: pg_info(1 pgs e243:3.52b) 2014-06-09 16:22:11.983161 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983162 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983156 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.983165 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.983177 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983178 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983180 7f869af78700 20 osd.26 243 _dispatch 0x30adb6c0 pg_info(1 pgs e243:3.148) v4 2014-06-09 16:22:11.983174 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.983184 7f869af78700 5 -- op tracker -- , seq: 3423, time: 2014-06-09 16:22:11.959164, event: header_read, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.983183 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.983191 7f869af78700 5 -- op tracker -- , seq: 3423, time: 2014-06-09 16:22:11.959165, event: throttled, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.983197 7f869af78700 5 -- op tracker -- , seq: 3423, time: 2014-06-09 16:22:11.959177, event: all_read, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.983196 7f8692f68700 15 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:11.983203 7f869af78700 5 -- op tracker -- , seq: 3423, time: 2014-06-09 16:22:11.983177, event: dispatched, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.983209 7f8691765700 10 op_submit_start 3764 2014-06-09 16:22:11.983211 7f8691765700 5 queue_transactions (trailing journal) 3764 2014-06-09 16:22:11.983206 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.983209 7f869af78700 5 -- op tracker -- , seq: 3423, time: 2014-06-09 16:22:11.983207, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.983213 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.148) v4 from osd.7 2014-06-09 16:22:11.983213 7f8691765700 5 queue_op 0xc64b0e0 seq 3764 osr(0.ef 0x2bd64f0) 858 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.983217 7f8691765700 10 op_submit_finish 3764 2014-06-09 16:22:11.983216 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adb6c0 2014-06-09 16:22:11.983213 7f8692f68700 20 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.983222 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.983220 7f869af78700 5 -- op tracker -- , seq: 3423, time: 2014-06-09 16:22:11.983219, event: started, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.983222 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.983225 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) 2014-06-09 16:22:11.983236 7f869af78700 5 -- op tracker -- , seq: 3423, time: 2014-06-09 16:22:11.983234, event: done, op: pg_info(1 pgs e243:3.148) 2014-06-09 16:22:11.983235 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.983243 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983237 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.8 activated and committed 2014-06-09 16:22:11.983245 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983244 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.983246 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 212/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.983256 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983257 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983259 7f869af78700 20 osd.26 243 _dispatch 0x76e3180 pg_info(1 pgs e243:3.4c9) v4 2014-06-09 16:22:11.983253 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.983254 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.983263 7f869af78700 5 -- op tracker -- , seq: 3424, time: 2014-06-09 16:22:11.972378, event: header_read, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:11.983266 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.983262 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.983270 7f869af78700 5 -- op tracker -- , seq: 3424, time: 2014-06-09 16:22:11.972379, event: throttled, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:11.983271 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.983276 7f869af78700 5 -- op tracker -- , seq: 3424, time: 2014-06-09 16:22:11.972394, event: all_read, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:11.983282 7f869af78700 5 -- op tracker -- , seq: 3424, time: 2014-06-09 16:22:11.983256, event: dispatched, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:11.983281 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:968 2014-06-09 16:22:11.983289 7f869af78700 5 -- op tracker -- , seq: 3424, time: 2014-06-09 16:22:11.983287, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:11.983285 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1104 2014-06-09 16:22:11.983293 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c9) v4 from osd.3 2014-06-09 16:22:11.983295 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.983296 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e3180 2014-06-09 16:22:11.983294 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.983301 7f869af78700 5 -- op tracker -- , seq: 3424, time: 2014-06-09 16:22:11.983299, event: started, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:11.983299 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:11.983303 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.983311 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.983309 7f86a5e11700 10 _generic_write 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 0~106496 = 0 2014-06-09 16:22:11.983316 7f869af78700 5 -- op tracker -- , seq: 3424, time: 2014-06-09 16:22:11.983314, event: done, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:11.983323 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983315 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.983323 7f86a5e11700 15 _setattrs 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.983325 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983328 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.983331 7f8692f68700 5 queue_transactions existing osr(3.6f 0x3ed0280)/0x3ed0280 2014-06-09 16:22:11.983333 7f8692f68700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.983333 7f86a5e11700 10 _setattrs 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 = 0 2014-06-09 16:22:11.983327 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.842867 7 0.000340 2014-06-09 16:22:11.983336 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983338 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983339 7f8688f54700 15 OSD::recovery_tp worker wq OSD::RecoveryWQ done processing 0x69d0000 (0 active) 2014-06-09 16:22:11.983339 7f869af78700 20 osd.26 243 _dispatch 0xddc41c0 pg_info(1 pgs e243:1.10d) v4 2014-06-09 16:22:11.983339 7f86a5e11700 15 _setattrs 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 2014-06-09 16:22:11.983341 7f8688f54700 20 OSD::recovery_tp worker waiting 2014-06-09 16:22:11.983338 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.983344 7f869af78700 5 -- op tracker -- , seq: 3425, time: 2014-06-09 16:22:11.959430, event: header_read, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.983347 7f86a5e11700 10 _setattrs 3.ff_head/f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 = 0 2014-06-09 16:22:11.983350 7f869af78700 5 -- op tracker -- , seq: 3425, time: 2014-06-09 16:22:11.959430, event: throttled, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.983350 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.983356 7f869af78700 5 -- op tracker -- , seq: 3425, time: 2014-06-09 16:22:11.959441, event: all_read, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.983362 7f869af78700 5 -- op tracker -- , seq: 3425, time: 2014-06-09 16:22:11.983336, event: dispatched, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.983360 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:11.983369 7f86a5e11700 10 cid: oid: f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 not skipping op, *spos 3720.0.0 2014-06-09 16:22:11.983368 7f869af78700 5 -- op tracker -- , seq: 3425, time: 2014-06-09 16:22:11.983366, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.983371 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:11.983368 7f868af58700 10 op_submit_start 3765 2014-06-09 16:22:11.983371 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.10d) v4 from osd.1 2014-06-09 16:22:11.983373 7f868af58700 5 queue_transactions (trailing journal) 3765 2014-06-09 16:22:11.983374 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xddc41c0 2014-06-09 16:22:11.983374 7f868af58700 5 queue_op 0x30d97b00 seq 3765 osr(1.113 0x384c910) 828 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.983378 7f868af58700 10 op_submit_finish 3765 2014-06-09 16:22:11.983372 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.983377 7f86a5e11700 20 set_header setting 30156 cid 3.ff_head oid f1db70ff/rbd_data.11816b8b4567.0000000000001948/head//3 parent seq 0 2014-06-09 16:22:11.983379 7f869af78700 5 -- op tracker -- , seq: 3425, time: 2014-06-09 16:22:11.983377, event: started, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.983380 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.983383 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.3cb( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.983387 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.983393 7f869af78700 5 -- op tracker -- , seq: 3425, time: 2014-06-09 16:22:11.983391, event: done, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.983398 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3720.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.983401 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983403 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983402 7f86a5e11700 10 cid: oid: 516eb9d7/pglog_3.ff/0//-1 skipping op, spos 3720.0.0 <= header.spos 149772.0.0 2014-06-09 16:22:11.983405 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.983399 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:11.983413 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.983414 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983415 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983409 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:11.983416 7f869af78700 20 osd.26 243 _dispatch 0x334416c0 pg_info(1 pgs e243:0.6f0) v4 2014-06-09 16:22:11.983417 7f868af58700 5 queue_transactions existing osr(1.3cb 0x384d4f0)/0x384d4f0 2014-06-09 16:22:11.983420 7f868af58700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.983418 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.983424 7f8689f56700 10 op_submit_start 3766 2014-06-09 16:22:11.983421 7f869af78700 5 -- op tracker -- , seq: 3426, time: 2014-06-09 16:22:11.958796, event: header_read, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.983425 7f8689f56700 5 queue_transactions (trailing journal) 3766 2014-06-09 16:22:11.983426 7f8689f56700 5 queue_op 0xd04f080 seq 3766 osr(3.77 0x3ed0250) 826 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.983429 7f8689f56700 10 op_submit_finish 3766 2014-06-09 16:22:11.983427 7f869af78700 5 -- op tracker -- , seq: 3426, time: 2014-06-09 16:22:11.958796, event: throttled, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.983429 7f8691765700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.983433 7f869af78700 5 -- op tracker -- , seq: 3426, time: 2014-06-09 16:22:11.958808, event: all_read, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.983440 7f869af78700 5 -- op tracker -- , seq: 3426, time: 2014-06-09 16:22:11.983413, event: dispatched, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.983434 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:11.983444 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 2014-06-09 16:22:11.983447 7f869af78700 5 -- op tracker -- , seq: 3426, time: 2014-06-09 16:22:11.983445, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.983452 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6f0) v4 from osd.9 2014-06-09 16:22:11.983454 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x334416c0 2014-06-09 16:22:11.983459 7f869af78700 5 -- op tracker -- , seq: 3426, time: 2014-06-09 16:22:11.983457, event: started, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.983465 7f8689f56700 10 log is not dirty 2014-06-09 16:22:11.983469 7f8689f56700 5 queue_transactions existing osr(3.76 0x3ed0240)/0x3ed0240 2014-06-09 16:22:11.983471 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.983471 7f8689f56700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.983475 7f868d75d700 10 op_submit_start 3767 2014-06-09 16:22:11.983474 7f8691765700 5 queue_transactions existing osr(2.2b5 0x2ed0c80)/0x2ed0c80 2014-06-09 16:22:11.983473 7f869af78700 5 -- op tracker -- , seq: 3426, time: 2014-06-09 16:22:11.983470, event: done, op: pg_info(1 pgs e243:0.6f0) 2014-06-09 16:22:11.983476 7f868d75d700 5 queue_transactions (trailing journal) 3767 2014-06-09 16:22:11.983477 7f8691765700 2 waiting 51 > 50 ops || 1205546 > 104857600 2014-06-09 16:22:11.983478 7f868d75d700 5 queue_op 0xd326ba0 seq 3767 osr(3.f6 0x3ed04b0) 1034580 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.983480 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983481 7f868d75d700 10 op_submit_finish 3767 2014-06-09 16:22:11.983482 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983494 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983495 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983496 7f869af78700 20 osd.26 243 _dispatch 0x3130a1c0 pg_info(1 pgs e243:1.35c) v4 2014-06-09 16:22:11.983501 7f869af78700 5 -- op tracker -- , seq: 3427, time: 2014-06-09 16:22:11.967338, event: header_read, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.983507 7f869af78700 5 -- op tracker -- , seq: 3427, time: 2014-06-09 16:22:11.967339, event: throttled, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.983488 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0x390c840 243'684 rep_tid=19 committed?=0 applied?=0 lock=0 op=osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.983514 7f868d75d700 10 osd.26 243 dequeue_op 0x1c769b00 finish 2014-06-09 16:22:11.983513 7f869af78700 5 -- op tracker -- , seq: 3427, time: 2014-06-09 16:22:11.967352, event: all_read, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.983518 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.983518 7f869af78700 5 -- op tracker -- , seq: 3427, time: 2014-06-09 16:22:11.983493, event: dispatched, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.983526 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.983524 7f869af78700 5 -- op tracker -- , seq: 3427, time: 2014-06-09 16:22:11.983522, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.983528 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.35c) v4 from osd.8 2014-06-09 16:22:11.983531 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130a1c0 2014-06-09 16:22:11.983536 7f869af78700 5 -- op tracker -- , seq: 3427, time: 2014-06-09 16:22:11.983534, event: started, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.983530 7f868d75d700 10 osd.26 243 dequeue_op 0x1b0d6f00 prio 196 cost 0 latency 0.016582 osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) v2 pg pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.983549 7f868d75d700 5 -- op tracker -- , seq: 3371, time: 2014-06-09 16:22:11.983545, event: reached_pg, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.983562 7f869af78700 5 -- op tracker -- , seq: 3427, time: 2014-06-09 16:22:11.983549, event: done, op: pg_info(1 pgs e243:1.35c) 2014-06-09 16:22:11.983559 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1b0d6f00 2014-06-09 16:22:11.983570 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983572 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983575 7f868d75d700 5 -- op tracker -- , seq: 3371, time: 2014-06-09 16:22:11.983571, event: started, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.983583 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983584 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983586 7f869af78700 20 osd.26 243 _dispatch 0x16064a80 pg_info(1 pgs e243:3.544) v4 2014-06-09 16:22:11.983581 7f868d75d700 7 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 17 op ack_type 4 from 8 2014-06-09 16:22:11.983591 7f869af78700 5 -- op tracker -- , seq: 3428, time: 2014-06-09 16:22:11.960480, event: header_read, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.983601 7f869af78700 5 -- op tracker -- , seq: 3428, time: 2014-06-09 16:22:11.960481, event: throttled, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.983603 7f868ff62700 10 op_submit_start 3768 2014-06-09 16:22:11.983605 7f868ff62700 5 queue_transactions (trailing journal) 3768 2014-06-09 16:22:11.983607 7f868ff62700 5 queue_op 0xc2d5740 seq 3768 osr(2.2f2 0x2ed0fb0) 828 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.983607 7f869af78700 5 -- op tracker -- , seq: 3428, time: 2014-06-09 16:22:11.960493, event: all_read, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.983610 7f868ff62700 10 op_submit_finish 3768 2014-06-09 16:22:11.983617 7f869af78700 5 -- op tracker -- , seq: 3428, time: 2014-06-09 16:22:11.983582, event: dispatched, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.983639 7f8690f64700 10 op_submit_start 3769 2014-06-09 16:22:11.983642 7f8690f64700 5 queue_transactions (trailing journal) 3769 2014-06-09 16:22:11.983643 7f8690f64700 5 queue_op 0x30a2a80 seq 3769 osr(1.11d 0x384c8b0) 828 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.983647 7f8690f64700 10 op_submit_finish 3769 2014-06-09 16:22:11.983630 7f868d75d700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.983594, event: sub_op_commit_rec, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.983652 7f868f761700 10 op_submit_start 3770 2014-06-09 16:22:11.983654 7f868f761700 5 queue_transactions (trailing journal) 3770 2014-06-09 16:22:11.983655 7f868f761700 5 queue_op 0xc2d4c00 seq 3770 osr(1.4fb 0x384d7f0) 828 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.983658 7f868f761700 10 op_submit_finish 3770 2014-06-09 16:22:11.983653 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'591 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 17 all committed 2014-06-09 16:22:11.983652 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.71e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.983662 7f86827e2700 5 -- op tracker -- , seq: 3429, time: 2014-06-09 16:22:11.983630, event: header_read, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) 2014-06-09 16:22:11.983663 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.5b6( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.983663 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 0'0 active] eval_repop repgather(0x2d83ec0 243'592 rep_tid=17 committed?=1 applied?=1 lock=0 op=osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.983673 7f86827e2700 5 -- op tracker -- , seq: 3429, time: 2014-06-09 16:22:11.983631, event: throttled, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) 2014-06-09 16:22:11.983684 7f8690f64700 10 log is not dirty 2014-06-09 16:22:11.983688 7f8690f64700 5 queue_transactions existing osr(1.71e 0x2ed00b0)/0x2ed00b0 2014-06-09 16:22:11.983687 7f86827e2700 5 -- op tracker -- , seq: 3429, time: 2014-06-09 16:22:11.983641, event: all_read, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) 2014-06-09 16:22:11.983691 7f8690f64700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.983679 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 0'0 active] log_op_stats osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4 inb 24960 outb 0 rlat 1.426560 lat 1.428650 2014-06-09 16:22:11.983695 7f86827e2700 5 -- op tracker -- , seq: 3429, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) 2014-06-09 16:22:11.983701 7f868f761700 10 log is not dirty 2014-06-09 16:22:11.983695 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 0'0 active] publish_stats_to_osd 243:977 2014-06-09 16:22:11.983701 7f86827e2700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) v2 epoch 243 2014-06-09 16:22:11.983705 7f868f761700 5 queue_transactions existing osr(2.5b6 0x2ed1b90)/0x2ed1b90 2014-06-09 16:22:11.983697 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:11.983708 7f868f761700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.983705 7f86827e2700 20 osd.26 243 should_share_map osd.3 172.16.96.201:6823/50569 243 2014-06-09 16:22:11.983712 7f8689755700 10 op_submit_start 3771 2014-06-09 16:22:11.983713 7f8689755700 5 queue_transactions (trailing journal) 3771 2014-06-09 16:22:11.983714 7f8689755700 5 queue_op 0xc2eae40 seq 3771 osr(2.5d9 0x2ed1a70) 828 bytes (queue has 50 ops and 1204685 bytes) 2014-06-09 16:22:11.983712 7f86827e2700 15 osd.26 243 enqueue_op 0x230da100 prio 196 cost 0 latency 0.000082 osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) v2 2014-06-09 16:22:11.983717 7f8689755700 10 op_submit_finish 3771 2014-06-09 16:22:11.983707 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 0'0 active] sending commit on repgather(0x2d83ec0 243'592 rep_tid=17 committed?=1 applied?=1 lock=0 op=osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4) 0x30bbf680 2014-06-09 16:22:11.983727 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.983721 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.983731 7f868ff62700 5 queue_transactions existing osr(3.37a 0x3ed1040)/0x3ed1040 2014-06-09 16:22:11.983735 7f868ff62700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.983734 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.983739 7f868d75d700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.983734, event: commit_sent, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.983743 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.983750 7f86827e2700 5 -- op tracker -- , seq: 3430, time: 2014-06-09 16:22:11.983728, event: header_read, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.983746 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.983751 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.983758 7f86827e2700 5 -- op tracker -- , seq: 3430, time: 2014-06-09 16:22:11.983729, event: throttled, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.983759 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.983765 7f86827e2700 5 -- op tracker -- , seq: 3430, time: 2014-06-09 16:22:11.983736, event: all_read, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.983758 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] removing repgather(0x2d83ec0 243'592 rep_tid=17 committed?=1 applied?=1 op=osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.983767 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.983773 7f86827e2700 5 -- op tracker -- , seq: 3430, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.983777 7f86827e2700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:11.983781 7f86827e2700 20 osd.26 243 should_share_map osd.3 172.16.96.201:6823/50569 243 2014-06-09 16:22:11.983780 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:191 2014-06-09 16:22:11.983786 7f86827e2700 15 osd.26 243 enqueue_op 0x270bff00 prio 196 cost 0 latency 0.000058 osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:11.983774 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] q front is repgather(0x2d83ec0 243'592 rep_tid=17 committed?=1 applied?=1 op=osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.983789 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.983796 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.983806 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.983805 7f868df5e700 2 waiting 51 > 50 ops || 1205511 > 104857600 2014-06-09 16:22:11.983792 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] remove_repop repgather(0x2d83ec0 243'592 rep_tid=17 committed?=1 applied?=1 op=osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.983808 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] requeue_ops 2014-06-09 16:22:11.983810 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.983824 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.983835 7f868d75d700 5 -- op tracker -- , seq: 501, time: 2014-06-09 16:22:11.983829, event: done, op: osd_op(client.4570.0:33987 rbd_data.117a6b8b4567.00000000000016d7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~24576] 3.cb9e4e47 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.983832 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0xe2acf00,0xe2ace00,0xe2adf00,0xe2ade00,0xe2add00,0xe2adc00 2014-06-09 16:22:11.983850 7f868d75d700 10 osd.26 243 dequeue_op 0x1b0d6f00 finish 2014-06-09 16:22:11.983857 7f868d75d700 5 -- op tracker -- , seq: 3371, time: 2014-06-09 16:22:11.983853, event: done, op: osd_sub_op_reply(client.4570.0:33987 3.647 cb9e4e47/rbd_data.117a6b8b4567.00000000000016d7/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.983858 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.983867 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.983867 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.983875 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.983890 7f869af78700 5 -- op tracker -- , seq: 3428, time: 2014-06-09 16:22:11.983623, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.983895 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f35f0 (20 active) 2014-06-09 16:22:11.983897 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.544) v4 from osd.6 2014-06-09 16:22:11.983900 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16064a80 2014-06-09 16:22:11.983905 7f869af78700 5 -- op tracker -- , seq: 3428, time: 2014-06-09 16:22:11.983903, event: started, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.983902 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.18a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.983920 7f869af78700 5 -- op tracker -- , seq: 3428, time: 2014-06-09 16:22:11.983918, event: done, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.983928 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983930 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983938 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.983942 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.983943 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.983943 7f868d75d700 5 queue_transactions existing osr(1.18a 0x384cb50)/0x384cb50 2014-06-09 16:22:11.983944 7f869af78700 20 osd.26 243 _dispatch 0x309e7a40 pg_info(1 pgs e243:0.448) v4 2014-06-09 16:22:11.983945 7f868d75d700 2 waiting 51 > 50 ops || 1205513 > 104857600 2014-06-09 16:22:11.983949 7f869af78700 5 -- op tracker -- , seq: 3431, time: 2014-06-09 16:22:11.956890, event: header_read, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.983955 7f869af78700 5 -- op tracker -- , seq: 3431, time: 2014-06-09 16:22:11.956890, event: throttled, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.983961 7f869af78700 5 -- op tracker -- , seq: 3431, time: 2014-06-09 16:22:11.956898, event: all_read, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.983966 7f869af78700 5 -- op tracker -- , seq: 3431, time: 2014-06-09 16:22:11.983941, event: dispatched, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.983972 7f869af78700 5 -- op tracker -- , seq: 3431, time: 2014-06-09 16:22:11.983970, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.983976 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.448) v4 from osd.0 2014-06-09 16:22:11.983979 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e7a40 2014-06-09 16:22:11.983983 7f869af78700 5 -- op tracker -- , seq: 3431, time: 2014-06-09 16:22:11.983981, event: started, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.983997 7f869af78700 5 -- op tracker -- , seq: 3431, time: 2014-06-09 16:22:11.983995, event: done, op: pg_info(1 pgs e243:0.448) 2014-06-09 16:22:11.984003 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984005 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984015 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984017 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984018 7f869af78700 20 osd.26 243 _dispatch 0x2fc95340 pg_info(1 pgs e243:3.46f) v4 2014-06-09 16:22:11.984022 7f869af78700 5 -- op tracker -- , seq: 3432, time: 2014-06-09 16:22:11.969134, event: header_read, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.984029 7f869af78700 5 -- op tracker -- , seq: 3432, time: 2014-06-09 16:22:11.969134, event: throttled, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.984035 7f869af78700 5 -- op tracker -- , seq: 3432, time: 2014-06-09 16:22:11.969145, event: all_read, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.984041 7f869af78700 5 -- op tracker -- , seq: 3432, time: 2014-06-09 16:22:11.984015, event: dispatched, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.984046 7f869af78700 5 -- op tracker -- , seq: 3432, time: 2014-06-09 16:22:11.984045, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.984050 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.46f) v4 from osd.5 2014-06-09 16:22:11.984052 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fc95340 2014-06-09 16:22:11.984056 7f869af78700 5 -- op tracker -- , seq: 3432, time: 2014-06-09 16:22:11.984055, event: started, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.984069 7f869af78700 5 -- op tracker -- , seq: 3432, time: 2014-06-09 16:22:11.984068, event: done, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.984070 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.984076 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984077 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984087 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984088 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984089 7f869af78700 20 osd.26 243 _dispatch 0x30c776c0 pg_info(1 pgs e243:2.4b9) v4 2014-06-09 16:22:11.984090 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3723.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.984095 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.984093 7f869af78700 5 -- op tracker -- , seq: 3433, time: 2014-06-09 16:22:11.963289, event: header_read, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:11.984098 7f869af78700 5 -- op tracker -- , seq: 3433, time: 2014-06-09 16:22:11.963289, event: throttled, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:11.984103 7f869af78700 5 -- op tracker -- , seq: 3433, time: 2014-06-09 16:22:11.963299, event: all_read, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:11.984108 7f869af78700 5 -- op tracker -- , seq: 3433, time: 2014-06-09 16:22:11.984087, event: dispatched, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:11.984111 7f86a5610700 10 _do_op 0x30d968a0 seq 3723 r = 0, finisher 0xbd751a0 0 2014-06-09 16:22:11.984114 7f869af78700 5 -- op tracker -- , seq: 3433, time: 2014-06-09 16:22:11.984112, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:11.984118 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4b9) v4 from osd.2 2014-06-09 16:22:11.984118 7f86a5610700 10 _finish_op 0x30d968a0 seq 3723 osr(2.e4 0x2ed0410)/0x2ed0410 2014-06-09 16:22:11.984120 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c776c0 2014-06-09 16:22:11.984124 7f869af78700 5 -- op tracker -- , seq: 3433, time: 2014-06-09 16:22:11.984122, event: started, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:11.984125 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62efc0 (1 active) 2014-06-09 16:22:11.984128 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89bd40 (2 active) 2014-06-09 16:22:11.984127 7f8691f66700 10 op_submit_start 3772 2014-06-09 16:22:11.984130 7f8691f66700 5 queue_transactions (trailing journal) 3772 2014-06-09 16:22:11.984130 7f86a5610700 5 _do_op 0x11fce360 seq 3724 osr(1.76e 0x2ed03b0)/0x2ed03b0 start 2014-06-09 16:22:11.984132 7f86a5610700 10 _do_transaction on 0x2ed5e00 2014-06-09 16:22:11.984131 7f8691f66700 5 queue_op 0xcabdec0 seq 3772 osr(0.55d 0x2bd77f0) 828 bytes (queue has 50 ops and 1204655 bytes) 2014-06-09 16:22:11.984134 7f8691f66700 10 op_submit_finish 3772 2014-06-09 16:22:11.984136 7f869af78700 5 -- op tracker -- , seq: 3433, time: 2014-06-09 16:22:11.984134, event: done, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:11.984138 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.984142 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984143 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984138 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.198( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.984153 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984155 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984155 7f869af78700 20 osd.26 243 _dispatch 0x30c00700 pg_info(1 pgs e243:0.33a) v4 2014-06-09 16:22:11.984160 7f869af78700 5 -- op tracker -- , seq: 3434, time: 2014-06-09 16:22:11.960398, event: header_read, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:11.984165 7f869af78700 5 -- op tracker -- , seq: 3434, time: 2014-06-09 16:22:11.960398, event: throttled, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:11.984171 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.984170 7f869af78700 5 -- op tracker -- , seq: 3434, time: 2014-06-09 16:22:11.960408, event: all_read, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:11.984175 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.984175 7f8691f66700 5 queue_transactions existing osr(1.198 0x384cb30)/0x384cb30 2014-06-09 16:22:11.984175 7f869af78700 5 -- op tracker -- , seq: 3434, time: 2014-06-09 16:22:11.984153, event: dispatched, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:11.984178 7f8691f66700 2 waiting 51 > 50 ops || 1205483 > 104857600 2014-06-09 16:22:11.984180 7f869af78700 5 -- op tracker -- , seq: 3434, time: 2014-06-09 16:22:11.984179, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:11.984184 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.33a) v4 from osd.4 2014-06-09 16:22:11.984186 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c00700 2014-06-09 16:22:11.984190 7f869af78700 5 -- op tracker -- , seq: 3434, time: 2014-06-09 16:22:11.984188, event: started, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:11.984192 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3724.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.984197 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.984203 7f869af78700 5 -- op tracker -- , seq: 3434, time: 2014-06-09 16:22:11.984201, event: done, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:11.984209 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984210 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984220 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984221 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984222 7f869af78700 20 osd.26 243 _dispatch 0x30caf6c0 pg_info(1 pgs e243:0.1f9) v4 2014-06-09 16:22:11.984224 7f86a5610700 10 _do_op 0x11fce360 seq 3724 r = 0, finisher 0x3006cda0 0 2014-06-09 16:22:11.984226 7f869af78700 5 -- op tracker -- , seq: 3435, time: 2014-06-09 16:22:11.972432, event: header_read, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:11.984229 7f86a5610700 10 _finish_op 0x11fce360 seq 3724 osr(1.76e 0x2ed03b0)/0x2ed03b0 2014-06-09 16:22:11.984232 7f869af78700 5 -- op tracker -- , seq: 3435, time: 2014-06-09 16:22:11.972432, event: throttled, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:11.984235 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89bd40 (1 active) 2014-06-09 16:22:11.984238 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339440 (2 active) 2014-06-09 16:22:11.984237 7f869af78700 5 -- op tracker -- , seq: 3435, time: 2014-06-09 16:22:11.972446, event: all_read, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:11.984239 7f8691f66700 10 op_submit_start 3773 2014-06-09 16:22:11.984240 7f86a5610700 5 _do_op 0x14dd6de0 seq 3725 osr(0.3b6 0x2bd72d0)/0x2bd72d0 start 2014-06-09 16:22:11.984240 7f8691f66700 5 queue_transactions (trailing journal) 3773 2014-06-09 16:22:11.984242 7f86a5610700 10 _do_transaction on 0x2ed58c0 2014-06-09 16:22:11.984242 7f8691f66700 5 queue_op 0xcabd1a0 seq 3773 osr(1.198 0x384cb30) 828 bytes (queue has 50 ops and 1203482 bytes) 2014-06-09 16:22:11.984244 7f8691f66700 10 op_submit_finish 3773 2014-06-09 16:22:11.984242 7f869af78700 5 -- op tracker -- , seq: 3435, time: 2014-06-09 16:22:11.984220, event: dispatched, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:11.984246 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.984248 7f869af78700 5 -- op tracker -- , seq: 3435, time: 2014-06-09 16:22:11.984246, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:11.984252 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1f9) v4 from osd.3 2014-06-09 16:22:11.984254 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30caf6c0 2014-06-09 16:22:11.984247 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.5b7( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.984258 7f869af78700 5 -- op tracker -- , seq: 3435, time: 2014-06-09 16:22:11.984257, event: started, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:11.984270 7f869af78700 5 -- op tracker -- , seq: 3435, time: 2014-06-09 16:22:11.984268, event: done, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:11.984276 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984277 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984278 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.984283 7f8691f66700 5 queue_transactions existing osr(2.5b7 0x2ed1b70)/0x2ed1b70 2014-06-09 16:22:11.984283 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.984286 7f8691f66700 2 waiting 51 > 50 ops || 1204310 > 104857600 2014-06-09 16:22:11.984288 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984289 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984290 7f869af78700 20 osd.26 243 _dispatch 0x2f0396c0 pg_info(1 pgs e243:1.10d) v4 2014-06-09 16:22:11.984294 7f869af78700 5 -- op tracker -- , seq: 3436, time: 2014-06-09 16:22:11.967148, event: header_read, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.984297 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3725.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.984299 7f869af78700 5 -- op tracker -- , seq: 3436, time: 2014-06-09 16:22:11.967149, event: throttled, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.984302 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.984304 7f869af78700 5 -- op tracker -- , seq: 3436, time: 2014-06-09 16:22:11.967162, event: all_read, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.984309 7f869af78700 5 -- op tracker -- , seq: 3436, time: 2014-06-09 16:22:11.984288, event: dispatched, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.984313 7f86a5610700 10 _do_op 0x14dd6de0 seq 3725 r = 0, finisher 0x32075000 0 2014-06-09 16:22:11.984314 7f86a5e11700 10 _do_op 0xbdc8600 seq 3720 r = 0, finisher 0xbdf19c0 0x30fd6140 2014-06-09 16:22:11.984317 7f86a5610700 10 _finish_op 0x14dd6de0 seq 3725 osr(0.3b6 0x2bd72d0)/0x2bd72d0 2014-06-09 16:22:11.984315 7f869af78700 5 -- op tracker -- , seq: 3436, time: 2014-06-09 16:22:11.984314, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.984320 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.10d) v4 from osd.1 2014-06-09 16:22:11.984321 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7b860,0xbd5a040,0x314340e0,0x314ac1a0] 2014-06-09 16:22:11.984323 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2f0396c0 2014-06-09 16:22:11.984322 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339440 (1 active) 2014-06-09 16:22:11.984325 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fbe60 (2 active) 2014-06-09 16:22:11.984327 7f869af78700 5 -- op tracker -- , seq: 3436, time: 2014-06-09 16:22:11.984325, event: started, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.984329 7f86a5610700 5 _do_op 0x324e64e0 seq 3726 osr(2.6f 0x2ed0550)/0x2ed0550 start 2014-06-09 16:22:11.984324 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.984333 7f86a5610700 10 _do_transaction on 0x471d440 2014-06-09 16:22:11.984333 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.984338 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.984341 7f869af78700 5 -- op tracker -- , seq: 3436, time: 2014-06-09 16:22:11.984339, event: done, op: pg_info(1 pgs e243:1.10d) 2014-06-09 16:22:11.984342 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:11.984352 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984354 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984354 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=213'612 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 20 2014-06-09 16:22:11.984365 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.984366 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.984367 7f869af78700 20 osd.26 243 _dispatch 0x33440e00 pg_info(1 pgs e243:1.5cb) v4 2014-06-09 16:22:11.984371 7f8691f66700 10 op_submit_start 3774 2014-06-09 16:22:11.984373 7f868bf5a700 2 waiting 51 > 50 ops || 1204310 > 104857600 2014-06-09 16:22:11.984373 7f8691f66700 5 queue_transactions (trailing journal) 3774 2014-06-09 16:22:11.984375 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.984375 7f8691f66700 5 queue_op 0xcabd200 seq 3774 osr(2.5b7 0x2ed1b70) 828 bytes (queue has 50 ops and 1203482 bytes) 2014-06-09 16:22:11.984372 7f86a460e700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.984366, event: op_commit, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.984378 7f8691f66700 10 op_submit_finish 3774 2014-06-09 16:22:11.984379 7f868cf5c700 2 waiting 51 > 50 ops || 1204310 > 104857600 2014-06-09 16:22:11.984383 7f8692767700 2 waiting 51 > 50 ops || 1204310 > 104857600 2014-06-09 16:22:11.984387 7f8690763700 2 waiting 51 > 50 ops || 1204535 > 104857600 2014-06-09 16:22:11.984388 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3726.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.984381 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.646( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.984382 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 luod=213'612 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 20 all committed 2014-06-09 16:22:11.984392 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.984392 7f868a757700 2 waiting 51 > 50 ops || 1222264 > 104857600 2014-06-09 16:22:11.984402 7f86a5610700 10 _do_op 0x324e64e0 seq 3726 r = 0, finisher 0x30629060 0 2014-06-09 16:22:11.984405 7f86a5610700 10 _finish_op 0x324e64e0 seq 3726 osr(2.6f 0x2ed0550)/0x2ed0550 2014-06-09 16:22:11.984393 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] eval_repop repgather(0xc0d1440 243'613 rep_tid=20 committed?=1 applied?=0 lock=0 op=osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.984399 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:969 2014-06-09 16:22:11.984411 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fbe60 (1 active) 2014-06-09 16:22:11.984413 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f67e0 (2 active) 2014-06-09 16:22:11.984415 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.984415 7f86a5610700 5 _do_op 0x324e65a0 seq 3727 osr(1.2d0 0x384d180)/0x384d180 start 2014-06-09 16:22:11.984410 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.984417 7f86a5610700 10 _do_transaction on 0xc12fd40 2014-06-09 16:22:11.984419 7f8691f66700 5 queue_transactions existing osr(2.646 0x2ed1c30)/0x2ed1c30 2014-06-09 16:22:11.984408 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] log_op_stats osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4 inb 106877 outb 0 rlat 0.000000 lat 1.172593 2014-06-09 16:22:11.984422 7f8691f66700 10 op_submit_start 3775 2014-06-09 16:22:11.984422 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.984423 7f8691f66700 5 queue_transactions (trailing journal) 3775 2014-06-09 16:22:11.984424 7f8691f66700 5 queue_op 0xcabc060 seq 3775 osr(2.646 0x2ed1c30) 828 bytes (queue has 50 ops and 1203484 bytes) 2014-06-09 16:22:11.984418 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.2c0( v 214'605 (0'0,214'605] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.984427 7f8691f66700 10 op_submit_finish 3775 2014-06-09 16:22:11.984428 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.984424 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] publish_stats_to_osd 243:988 2014-06-09 16:22:11.984429 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.1fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.984433 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.984433 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] sending commit on repgather(0xc0d1440 243'613 rep_tid=20 committed?=1 applied?=0 lock=0 op=osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4) 0x32070c80 2014-06-09 16:22:11.984444 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.984452 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.984456 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.989251 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3727.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.989257 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.989274 7f86a5610700 10 _do_op 0x324e65a0 seq 3727 r = 0, finisher 0x314345e0 0 2014-06-09 16:22:11.984460 7f8691f66700 10 log is not dirty 2014-06-09 16:22:11.989291 7f8691f66700 5 queue_transactions existing osr(1.1fa 0x384cd50)/0x384cd50 2014-06-09 16:22:11.984462 7f868e75f700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.984459 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.989303 7f8692f68700 2 waiting 51 > 50 ops || 1204310 > 104857600 2014-06-09 16:22:11.989310 7f868af58700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989316 7f8689f56700 2 waiting 51 > 50 ops || 1204310 > 104857600 2014-06-09 16:22:11.989305 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.984467 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd751a0,0x3006cda0,0x32075000,0x30629060] 2014-06-09 16:22:11.989321 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.989331 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:11.989339 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:182 2014-06-09 16:22:11.989346 7f8691765700 2 waiting 51 > 50 ops || 1204345 > 104857600 2014-06-09 16:22:11.989350 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.989354 7f8690f64700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989350 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.989360 7f868f761700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989359 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.401( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.989370 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.989368 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[1.76e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] flushed 2014-06-09 16:22:11.989376 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.989390 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.989397 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd751a0,0x3006cda0,0x32075000,0x30629060] 2014-06-09 16:22:11.989400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.989401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.989398 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.989407 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.989416 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.989423 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.989436 7f8689755700 15 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.989445 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.989453 7f8689755700 20 osd.26 pg_epoch: 243 pg[1.282( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.989462 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.989466 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.989478 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.989486 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.989494 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.989502 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.989511 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.989523 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.989532 7f8689755700 20 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.989540 7f8689755700 20 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.989549 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.989553 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.989567 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.989576 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.989585 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.989594 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.989606 7f868ff62700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989604 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.989613 7f868df5e700 2 waiting 51 > 50 ops || 1204310 > 104857600 2014-06-09 16:22:11.989617 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1112 2014-06-09 16:22:11.989627 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.989636 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.989647 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.989651 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.989668 7f868d75d700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989664 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.989676 7f868cf5c700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989673 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.989683 7f8692767700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989680 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.989688 7f8690763700 2 waiting 51 > 50 ops || 1204537 > 104857600 2014-06-09 16:22:11.989693 7f868a757700 2 waiting 51 > 50 ops || 1222266 > 104857600 2014-06-09 16:22:11.989694 7f8689755700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.989700 7f868ef60700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989708 7f868bf5a700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989714 7f868b759700 2 waiting 51 > 50 ops || 1204342 > 104857600 2014-06-09 16:22:11.989710 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:179 2014-06-09 16:22:11.989726 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.989731 7f8691f66700 2 waiting 51 > 50 ops || 1204312 > 104857600 2014-06-09 16:22:11.989732 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.989746 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.990604 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1c66ba00,0x1c66b900,0x1c66b800,0x1c66b700,0x1c66b600,0x1c66b500,0x1c66b400,0x29f5ab00,0x4678800,0x4678700,0x4678600,0x4678500,0x4678400 2014-06-09 16:22:11.990652 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.990663 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.990671 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.990685 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:925 2014-06-09 16:22:11.990695 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.990704 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.990715 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.990720 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.990732 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.990740 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.990748 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.990756 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.990764 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.990776 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.990785 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.990793 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.990802 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.990806 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:11.990820 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.990830 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.990839 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.990852 7f8689755700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.990866 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:966 2014-06-09 16:22:11.990893 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.990897 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.990967 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.855539 7 0.000309 2014-06-09 16:22:11.990981 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.990992 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.991003 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:11.991013 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.991022 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.991030 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.991042 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.991053 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.991062 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.991075 7f8689755700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.991090 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:11.991124 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.991128 7f8689755700 5 queue_transactions existing osr(2.28b 0x2ed0cd0)/0x2ed0cd0 2014-06-09 16:22:11.991131 7f8689755700 2 waiting 51 > 50 ops || 1204345 > 104857600 2014-06-09 16:22:11.984465 7f86a460e700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.984460, event: commit_sent, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.991145 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7b860,0xbd5a040,0x314340e0,0x314ac1a0] 2014-06-09 16:22:11.991148 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd79980,0xc2d6a20] 2014-06-09 16:22:11.991150 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd79980,0xc2d6a20] 2014-06-09 16:22:11.991152 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.991153 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.984473 7f86a5e11700 10 _finish_op 0xbdc8600 seq 3720 osr(3.ff 0x3ed0870)/0x3ed0870 2014-06-09 16:22:11.991166 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358ea0 (1 active) 2014-06-09 16:22:11.991167 7f868e75f700 10 op_submit_start 3776 2014-06-09 16:22:11.991170 7f868e75f700 5 queue_transactions (trailing journal) 3776 2014-06-09 16:22:11.991169 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1e60 (2 active) 2014-06-09 16:22:11.991172 7f868e75f700 5 queue_op 0x324f43c0 seq 3776 osr(2.113 0x2ed06b0) 828 bytes (queue has 50 ops and 1095428 bytes) 2014-06-09 16:22:11.991174 7f86a5e11700 5 _do_op 0x324e6660 seq 3728 osr(3.66a 0x3ed19d0)/0x3ed19d0 start 2014-06-09 16:22:11.991178 7f868e75f700 10 op_submit_finish 3776 2014-06-09 16:22:11.991178 7f86a5e11700 10 _do_transaction on 0xc33a3c0 2014-06-09 16:22:11.991184 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.991182 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.6d2( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.6d2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.991220 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.991225 7f868e75f700 5 queue_transactions existing osr(0.6d2 0x384c1f0)/0x384c1f0 2014-06-09 16:22:11.991228 7f868e75f700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.991233 7f8692f68700 2 waiting 51 > 50 ops || 1096254 > 104857600 2014-06-09 16:22:11.991238 7f868af58700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.991243 7f8689f56700 2 waiting 51 > 50 ops || 1096254 > 104857600 2014-06-09 16:22:11.991249 7f86a5610700 10 _finish_op 0x324e65a0 seq 3727 osr(1.2d0 0x384d180)/0x384d180 2014-06-09 16:22:11.991254 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f67e0 (1 active) 2014-06-09 16:22:11.991257 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407320 (2 active) 2014-06-09 16:22:11.991256 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.991260 7f86a5610700 5 _do_op 0x24105aa0 seq 3729 osr(1.37e 0x384d5d0)/0x384d5d0 start 2014-06-09 16:22:11.991263 7f86a5610700 10 _do_transaction on 0xbf51800 2014-06-09 16:22:11.991270 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.991274 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3728.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.991279 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.991303 7f8691765700 10 op_submit_start 3777 2014-06-09 16:22:11.991306 7f8691765700 5 queue_transactions (trailing journal) 3777 2014-06-09 16:22:11.991300 7f869af78700 5 -- op tracker -- , seq: 3437, time: 2014-06-09 16:22:11.958853, event: header_read, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.991308 7f8691765700 5 queue_op 0x30d975c0 seq 3777 osr(2.2b5 0x2ed0c80) 861 bytes (queue has 50 ops and 1095461 bytes) 2014-06-09 16:22:11.991311 7f8691765700 10 op_submit_finish 3777 2014-06-09 16:22:11.991311 7f869af78700 5 -- op tracker -- , seq: 3437, time: 2014-06-09 16:22:11.958853, event: throttled, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.991320 7f868e75f700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991319 7f869af78700 5 -- op tracker -- , seq: 3437, time: 2014-06-09 16:22:11.958867, event: all_read, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.991326 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdf19c0,0x314345e0] 2014-06-09 16:22:11.991327 7f868f761700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991325 7f869af78700 5 -- op tracker -- , seq: 3437, time: 2014-06-09 16:22:11.984364, event: dispatched, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.991315 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.991335 7f868ff62700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991335 7f869af78700 5 -- op tracker -- , seq: 3437, time: 2014-06-09 16:22:11.991333, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.991329 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] op_applied: 20 2014-06-09 16:22:11.991339 7f868df5e700 2 waiting 51 > 50 ops || 1096287 > 104857600 2014-06-09 16:22:11.991335 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.991344 7f8690f64700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991347 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991346 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5cb) v4 from osd.9 2014-06-09 16:22:11.991350 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x33440e00 2014-06-09 16:22:11.991351 7f868cf5c700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991348 7f86a4e0f700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.991341, event: op_applied, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.991356 7f8692767700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991352 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.991360 7f8690763700 2 waiting 51 > 50 ops || 1096514 > 104857600 2014-06-09 16:22:11.991360 7f86a5e11700 10 _do_op 0x324e6660 seq 3728 r = 0, finisher 0x31412ae0 0 2014-06-09 16:22:11.991364 7f868a757700 2 waiting 51 > 50 ops || 1114243 > 104857600 2014-06-09 16:22:11.991365 7f86a5e11700 10 _finish_op 0x324e6660 seq 3728 osr(3.66a 0x3ed19d0)/0x3ed19d0 2014-06-09 16:22:11.991358 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] op_applied on primary on version 243'613 2014-06-09 16:22:11.991363 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.991369 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc83480] 2014-06-09 16:22:11.991371 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc83480] 2014-06-09 16:22:11.991373 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.991374 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.991368 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] repop_all_applied: repop tid 20 all applied 2014-06-09 16:22:11.991371 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.991369 7f8678d96700 5 -- op tracker -- , seq: 3438, time: 2014-06-09 16:22:11.987085, event: header_read, op: osd_op(client.4577.0:38076 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.73fa29ff ack+ondisk+write e243) 2014-06-09 16:22:11.991379 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1e60 (1 active) 2014-06-09 16:22:11.991382 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0000 (2 active) 2014-06-09 16:22:11.991378 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.991384 7f86a5e11700 5 _do_op 0x2fcaf980 seq 3730 osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 start 2014-06-09 16:22:11.991387 7f86a5e11700 10 _do_transaction on 0xbd8e9c0 2014-06-09 16:22:11.991383 7f8678d96700 5 -- op tracker -- , seq: 3438, time: 2014-06-09 16:22:11.987087, event: throttled, op: osd_op(client.4577.0:38076 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.73fa29ff ack+ondisk+write e243) 2014-06-09 16:22:11.991377 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] eval_repop repgather(0xc0d1440 243'613 rep_tid=20 committed?=1 applied?=1 lock=0 op=osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.991386 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.991393 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.991393 7f8678d96700 5 -- op tracker -- , seq: 3438, time: 2014-06-09 16:22:11.988281, event: all_read, op: osd_op(client.4577.0:38076 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.73fa29ff ack+ondisk+write e243) 2014-06-09 16:22:11.991393 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.991403 7f8678d96700 5 -- op tracker -- , seq: 3438, time: 0.000000, event: dispatched, op: osd_op(client.4577.0:38076 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.73fa29ff ack+ondisk+write e243) 2014-06-09 16:22:11.991393 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] log_op_stats osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4 inb 106877 outb 0 rlat 0.000000 lat 1.179577 2014-06-09 16:22:11.991406 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:11.991410 7f8678d96700 20 osd.26 243 should_share_map client.4577 172.16.96.220:0/1127735 243 2014-06-09 16:22:11.991408 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] publish_stats_to_osd 243:989 2014-06-09 16:22:11.991414 7f8691765700 20 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.991418 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.991421 7f8691765700 20 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.991429 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991433 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.991427 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 213'612 active] removing repgather(0xc0d1440 243'613 rep_tid=20 committed?=1 applied?=1 op=osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.991433 7f8678d96700 15 osd.26 243 enqueue_op 0x2f097100 prio 63 cost 516096 latency 0.004348 osd_op(client.4577.0:38076 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2207744~516096] 3.73fa29ff ack+ondisk+write e243) v4 2014-06-09 16:22:11.991433 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.991444 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.991448 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3730.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.991441 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 213'612 active] q front is repgather(0xc0d1440 243'613 rep_tid=20 committed?=1 applied?=1 op=osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.991452 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.991450 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.991456 7f869af78700 5 -- op tracker -- , seq: 3437, time: 2014-06-09 16:22:11.991354, event: started, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.991458 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.991454 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 213'612 active] remove_repop repgather(0xc0d1440 243'613 rep_tid=20 committed?=1 applied?=1 op=osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:11.991465 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.991467 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.ff( v 243'613 (0'0,243'613] local-les=243 n=118 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 213'612 mlcod 213'612 active] requeue_ops 2014-06-09 16:22:11.991473 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.991477 7f869af78700 5 -- op tracker -- , seq: 3437, time: 2014-06-09 16:22:11.991475, event: done, op: pg_info(1 pgs e243:1.5cb) 2014-06-09 16:22:11.991482 7f86a5e11700 10 _do_op 0x2fcaf980 seq 3730 r = 0, finisher 0xbccd160 0 2014-06-09 16:22:11.991483 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:11.991489 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.991489 7f86a5e11700 10 _finish_op 0x2fcaf980 seq 3730 osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 2014-06-09 16:22:11.991492 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.991493 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0000 (1 active) 2014-06-09 16:22:11.991496 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496d80 (2 active) 2014-06-09 16:22:11.991491 7f8691765700 20 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.991493 7f86a4e0f700 5 -- op tracker -- , seq: 752, time: 2014-06-09 16:22:11.991486, event: done, op: osd_op(client.4577.0:37476 rbd_data.11816b8b4567.0000000000001948 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~106496] 3.f1db70ff RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.991498 7f86a5e11700 5 _do_op 0x2fcaeea0 seq 3731 osr(2.699 0x2ed1f10)/0x2ed1f10 start 2014-06-09 16:22:11.991501 7f86a5e11700 10 _do_transaction on 0x3339ec0 2014-06-09 16:22:11.991499 7f8691765700 20 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.991506 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.991506 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.991508 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991508 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.991509 7f869af78700 20 osd.26 243 _dispatch 0x1d4c41c0 pg_info(1 pgs e243:0.298) v4 2014-06-09 16:22:11.991515 7f869af78700 5 -- op tracker -- , seq: 3439, time: 2014-06-09 16:22:11.967362, event: header_read, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:11.991520 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdf19c0,0x314345e0] 2014-06-09 16:22:11.991522 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412ae0,0xbccd160] 2014-06-09 16:22:11.991512 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.991523 7f869af78700 5 -- op tracker -- , seq: 3439, time: 2014-06-09 16:22:11.967362, event: throttled, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:11.991526 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412ae0,0xbccd160] 2014-06-09 16:22:11.991527 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.991528 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.991529 7f869af78700 5 -- op tracker -- , seq: 3439, time: 2014-06-09 16:22:11.967374, event: all_read, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:11.991534 7f868d75d700 10 op_submit_start 3778 2014-06-09 16:22:11.991527 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] requeue_ops 0x32826100 2014-06-09 16:22:11.991536 7f868d75d700 5 queue_transactions (trailing journal) 3778 2014-06-09 16:22:11.991537 7f868d75d700 5 queue_op 0x288f9aa0 seq 3778 osr(1.18a 0x384cb50) 828 bytes (queue has 50 ops and 1095461 bytes) 2014-06-09 16:22:11.991536 7f869af78700 5 -- op tracker -- , seq: 3439, time: 2014-06-09 16:22:11.991505, event: dispatched, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:11.991540 7f868d75d700 10 op_submit_finish 3778 2014-06-09 16:22:11.991541 7f868b759700 2 waiting 51 > 50 ops || 1096319 > 104857600 2014-06-09 16:22:11.991543 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991544 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.991543 7f869af78700 5 -- op tracker -- , seq: 3439, time: 2014-06-09 16:22:11.991541, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:11.991547 7f868bf5a700 10 op_submit_start 3779 2014-06-09 16:22:11.991549 7f868bf5a700 5 queue_transactions (trailing journal) 3779 2014-06-09 16:22:11.991548 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.298) v4 from osd.8 2014-06-09 16:22:11.991551 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1d4c41c0 2014-06-09 16:22:11.991551 7f868bf5a700 5 queue_op 0x15996c00 seq 3779 osr(3.136 0x3ed07a0) 828 bytes (queue has 50 ops and 1095461 bytes) 2014-06-09 16:22:11.991554 7f868bf5a700 10 op_submit_finish 3779 2014-06-09 16:22:11.991545 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.3cd( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.991547 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.991557 7f869af78700 5 -- op tracker -- , seq: 3439, time: 2014-06-09 16:22:11.991555, event: started, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:11.991559 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3731.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.991562 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.991558 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.991566 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991558 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.699( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.699( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:11.991574 7f86a5e11700 10 _do_op 0x2fcaeea0 seq 3731 r = 0, finisher 0x30c559a0 0 2014-06-09 16:22:11.991573 7f869af78700 5 -- op tracker -- , seq: 3439, time: 2014-06-09 16:22:11.991571, event: done, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:11.991578 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.991578 7f86a5e11700 10 _finish_op 0x2fcaeea0 seq 3731 osr(2.699 0x2ed1f10)/0x2ed1f10 2014-06-09 16:22:11.991570 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.991581 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.991582 7f868d75d700 5 queue_transactions existing osr(1.3cd 0x384d4b0)/0x384d4b0 2014-06-09 16:22:11.991583 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.991584 7f868d75d700 10 op_submit_start 3780 2014-06-09 16:22:11.991586 7f868d75d700 5 queue_transactions (trailing journal) 3780 2014-06-09 16:22:11.991585 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496d80 (1 active) 2014-06-09 16:22:11.991587 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359d40 (2 active) 2014-06-09 16:22:11.991582 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.529( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.991587 7f868d75d700 5 queue_op 0xd327500 seq 3780 osr(1.3cd 0x384d4b0) 828 bytes (queue has 50 ops and 1095461 bytes) 2014-06-09 16:22:11.991590 7f868d75d700 10 op_submit_finish 3780 2014-06-09 16:22:11.991589 7f86a5e11700 5 _do_op 0x2fcafe00 seq 3732 osr(1.7a 0x384c510)/0x384c510 start 2014-06-09 16:22:11.991591 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991591 7f86a5e11700 10 _do_transaction on 0xc221b00 2014-06-09 16:22:11.991595 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.991596 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.991596 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.991598 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.991599 7f869af78700 20 osd.26 243 _dispatch 0x12c09dc0 pg_info(1 pgs e243:3.544) v4 2014-06-09 16:22:11.991600 7f868bf5a700 5 queue_transactions existing osr(2.699 0x2ed1f10)/0x2ed1f10 2014-06-09 16:22:11.991594 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.991592 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:11.991603 7f868bf5a700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991607 7f868b759700 2 waiting 51 > 50 ops || 1096319 > 104857600 2014-06-09 16:22:11.991605 7f869af78700 5 -- op tracker -- , seq: 3440, time: 2014-06-09 16:22:11.960502, event: header_read, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.991605 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:11.991612 7f8689755700 2 waiting 51 > 50 ops || 1096322 > 104857600 2014-06-09 16:22:11.991619 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991620 7f8692f68700 2 waiting 51 > 50 ops || 1096287 > 104857600 2014-06-09 16:22:11.991624 7f868af58700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991625 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.991612 7f869af78700 5 -- op tracker -- , seq: 3440, time: 2014-06-09 16:22:11.960503, event: throttled, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.991628 7f8689f56700 2 waiting 51 > 50 ops || 1096287 > 104857600 2014-06-09 16:22:11.991629 7f868d75d700 5 queue_transactions existing osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 2014-06-09 16:22:11.991631 7f868d75d700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991623 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.991632 7f869af78700 5 -- op tracker -- , seq: 3440, time: 2014-06-09 16:22:11.960514, event: all_read, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.991633 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x275cfe80,0x30c559c0] 2014-06-09 16:22:11.991636 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x275cfe80,0x30c559c0] 2014-06-09 16:22:11.991637 7f868ff62700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991639 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.991640 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.991638 7f869af78700 5 -- op tracker -- , seq: 3440, time: 2014-06-09 16:22:11.991596, event: dispatched, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.991641 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30c559a0] 2014-06-09 16:22:11.991641 7f868df5e700 2 waiting 51 > 50 ops || 1096287 > 104857600 2014-06-09 16:22:11.991637 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.991647 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30c559a0] 2014-06-09 16:22:11.991647 7f8690f64700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991648 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991649 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.991650 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.991653 7f868cf5c700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991654 7f869af78700 5 -- op tracker -- , seq: 3440, time: 2014-06-09 16:22:11.991643, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.991658 7f8692767700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.991652 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:11.991659 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.544) v4 from osd.6 2014-06-09 16:22:11.991662 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12c09dc0 2014-06-09 16:22:11.991663 7f8690763700 2 waiting 51 > 50 ops || 1096514 > 104857600 2014-06-09 16:22:11.991667 7f868a757700 2 waiting 51 > 50 ops || 1114243 > 104857600 2014-06-09 16:22:11.991663 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:11.991668 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.991667 7f869af78700 5 -- op tracker -- , seq: 3440, time: 2014-06-09 16:22:11.991665, event: started, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.991671 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.991682 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3732.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.991686 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.991689 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x153584b0 (19 active) 2014-06-09 16:22:11.991695 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.991700 7f86a5e11700 10 _do_op 0x2fcafe00 seq 3732 r = 0, finisher 0x3045a180 0 2014-06-09 16:22:11.991705 7f86a5e11700 10 _finish_op 0x2fcafe00 seq 3732 osr(1.7a 0x384c510)/0x384c510 2014-06-09 16:22:11.991705 7f869af78700 5 -- op tracker -- , seq: 3440, time: 2014-06-09 16:22:11.991703, event: done, op: pg_info(1 pgs e243:3.544) 2014-06-09 16:22:11.991712 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359d40 (1 active) 2014-06-09 16:22:11.991713 7f868bf5a700 10 op_submit_start 3781 2014-06-09 16:22:11.991701 7f8691765700 10 osd.26 243 dequeue_op 0x230da100 prio 196 cost 0 latency 0.008070 osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) v2 pg pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.991713 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.991715 7f868bf5a700 5 queue_transactions (trailing journal) 3781 2014-06-09 16:22:11.991715 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7806c0 (2 active) 2014-06-09 16:22:11.991716 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.991717 7f86a5e11700 5 _do_op 0x2fcae660 seq 3733 osr(1.122 0x384c890)/0x384c890 start 2014-06-09 16:22:11.991716 7f868bf5a700 5 queue_op 0x15996c60 seq 3781 osr(2.699 0x2ed1f10) 828 bytes (queue has 50 ops and 1095463 bytes) 2014-06-09 16:22:11.991719 7f868bf5a700 10 op_submit_finish 3781 2014-06-09 16:22:11.991719 7f86a5e11700 10 _do_transaction on 0x35be000 2014-06-09 16:22:11.991724 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.991719 7f8691765700 5 -- op tracker -- , seq: 3429, time: 2014-06-09 16:22:11.991716, event: reached_pg, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) 2014-06-09 16:22:11.991728 7f868e75f700 2 waiting 51 > 50 ops || 1096291 > 104857600 2014-06-09 16:22:11.991728 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.991730 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.991722 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.644( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.991731 7f869af78700 20 osd.26 243 _dispatch 0x150e0e00 pg_info(1 pgs e243:1.3bc) v4 2014-06-09 16:22:11.991733 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3045a2a0] 2014-06-09 16:22:11.991735 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3045a2a0] 2014-06-09 16:22:11.991737 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.991728 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x230da100 2014-06-09 16:22:11.991738 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.991736 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.991736 7f869af78700 5 -- op tracker -- , seq: 3441, time: 2014-06-09 16:22:11.956907, event: header_read, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.991746 7f869af78700 5 -- op tracker -- , seq: 3441, time: 2014-06-09 16:22:11.956907, event: throttled, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.991754 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.991753 7f869af78700 5 -- op tracker -- , seq: 3441, time: 2014-06-09 16:22:11.956917, event: all_read, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.991755 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3729.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.991759 7f868bf5a700 5 queue_transactions existing osr(0.644 0x2bd7bf0)/0x2bd7bf0 2014-06-09 16:22:11.991760 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.991759 7f869af78700 5 -- op tracker -- , seq: 3441, time: 2014-06-09 16:22:11.991728, event: dispatched, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.991761 7f868bf5a700 2 waiting 51 > 50 ops || 1096291 > 104857600 2014-06-09 16:22:11.991766 7f868f761700 2 waiting 51 > 50 ops || 1096291 > 104857600 2014-06-09 16:22:11.991766 7f869af78700 5 -- op tracker -- , seq: 3441, time: 2014-06-09 16:22:11.991764, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.991770 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.3bc) v4 from osd.0 2014-06-09 16:22:11.991773 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x150e0e00 2014-06-09 16:22:11.991778 7f869af78700 5 -- op tracker -- , seq: 3441, time: 2014-06-09 16:22:11.991776, event: started, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.991801 7f86a5610700 10 _do_op 0x24105aa0 seq 3729 r = 0, finisher 0x3062a120 0 2014-06-09 16:22:11.991808 7f86a5610700 10 _finish_op 0x24105aa0 seq 3729 osr(1.37e 0x384d5d0)/0x384d5d0 2014-06-09 16:22:11.991813 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407320 (1 active) 2014-06-09 16:22:11.991816 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653200 (2 active) 2014-06-09 16:22:11.991817 7f868e75f700 10 op_submit_start 3782 2014-06-09 16:22:11.991818 7f86a5610700 5 _do_op 0x2fcafe60 seq 3734 osr(0.117 0x2bd66a0)/0x2bd66a0 start 2014-06-09 16:22:11.991820 7f868e75f700 5 queue_transactions (trailing journal) 3782 2014-06-09 16:22:11.991821 7f86a5610700 10 _do_transaction on 0x3338f00 2014-06-09 16:22:11.991821 7f868e75f700 5 queue_op 0x324f4480 seq 3782 osr(0.6d2 0x384c1f0) 828 bytes (queue has 50 ops and 1095430 bytes) 2014-06-09 16:22:11.991824 7f868e75f700 10 op_submit_finish 3782 2014-06-09 16:22:11.991826 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.991832 7f8691f66700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991827 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.2b( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=7-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.2b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:11.991836 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.991838 7f868ef60700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991850 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3733.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.991854 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.991859 7f868e75f700 10 log is not dirty 2014-06-09 16:22:11.991863 7f868e75f700 5 queue_transactions existing osr(2.2b 0x2ed0230)/0x2ed0230 2014-06-09 16:22:11.991866 7f868e75f700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.991870 7f868b759700 2 waiting 51 > 50 ops || 1096288 > 104857600 2014-06-09 16:22:11.991875 7f868af58700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991876 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x12050f80] 2014-06-09 16:22:11.991879 7f8689f56700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.991883 7f8692f68700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.991886 7f8689755700 2 waiting 51 > 50 ops || 1096291 > 104857600 2014-06-09 16:22:11.991878 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.991886 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3045a180,0x3062a120] 2014-06-09 16:22:11.991890 7f868df5e700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.991894 7f868ff62700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991888 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.991897 7f868cf5c700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991899 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3045a180,0x3062a120] 2014-06-09 16:22:11.991901 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.991901 7f8692767700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991902 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.991897 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.37e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 2014-06-09 16:22:11.991905 7f8690763700 2 waiting 51 > 50 ops || 1096483 > 104857600 2014-06-09 16:22:11.991909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x12050f80] 2014-06-09 16:22:11.991911 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.991912 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.991910 7f868a757700 2 waiting 51 > 50 ops || 1114212 > 104857600 2014-06-09 16:22:11.991918 7f868bf5a700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991918 7f86a5e11700 10 _do_op 0x2fcae660 seq 3733 r = 0, finisher 0xbd7c800 0 2014-06-09 16:22:11.991923 7f868f761700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991923 7f86a5e11700 10 _finish_op 0x2fcae660 seq 3733 osr(1.122 0x384c890)/0x384c890 2014-06-09 16:22:11.991927 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcdb2440] 2014-06-09 16:22:11.991928 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcdb2440] 2014-06-09 16:22:11.991931 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.991932 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.991936 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7806c0 (1 active) 2014-06-09 16:22:11.991938 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f440 (2 active) 2014-06-09 16:22:11.991939 7f86a5e11700 5 _do_op 0x2fcae900 seq 3735 osr(3.139 0x3ed0780)/0x3ed0780 start 2014-06-09 16:22:11.991941 7f86a5e11700 10 _do_transaction on 0xc058cc0 2014-06-09 16:22:11.991945 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.991954 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.991970 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3734.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.991974 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.991988 7f868d75d700 10 op_submit_start 3783 2014-06-09 16:22:11.991990 7f868d75d700 5 queue_transactions (trailing journal) 3783 2014-06-09 16:22:11.991990 7f8690f64700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.991798 7f86af56b700 5 -- op tracker -- , seq: 3442, time: 2014-06-09 16:22:11.975148, event: header_read, op: osd_op(client.4541.0:32046 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 2928640~1265664] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.991992 7f868d75d700 5 queue_op 0x11fce9c0 seq 3783 osr(3.6ba 0x3ed1ce0) 828 bytes (queue has 50 ops and 1095430 bytes) 2014-06-09 16:22:11.991995 7f868d75d700 10 op_submit_finish 3783 2014-06-09 16:22:11.991997 7f868ef60700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992002 7f868e75f700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992008 7f868b759700 2 waiting 51 > 50 ops || 1096288 > 104857600 2014-06-09 16:22:11.991998 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.4c0( v 214'586 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) 2014-06-09 16:22:11.992013 7f868af58700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992018 7f8689f56700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992005 7f86af56b700 5 -- op tracker -- , seq: 3442, time: 2014-06-09 16:22:11.975149, event: throttled, op: osd_op(client.4541.0:32046 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 2928640~1265664] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.992023 7f8692f68700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992026 7f8689755700 2 waiting 51 > 50 ops || 1096291 > 104857600 2014-06-09 16:22:11.992017 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.7 activated and committed 2014-06-09 16:22:11.992030 7f868df5e700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992034 7f868ff62700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992031 7f86af56b700 5 -- op tracker -- , seq: 3442, time: 2014-06-09 16:22:11.978170, event: all_read, op: osd_op(client.4541.0:32046 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 2928640~1265664] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.992038 7f868cf5c700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992029 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 210/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:11.992041 7f8692767700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992043 7f86af56b700 5 -- op tracker -- , seq: 3442, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32046 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 2928640~1265664] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.992046 7f8690763700 2 waiting 51 > 50 ops || 1096483 > 104857600 2014-06-09 16:22:11.992041 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:11.992050 7f868a757700 2 waiting 51 > 50 ops || 1114212 > 104857600 2014-06-09 16:22:11.992051 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:11.992056 7f868bf5a700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992057 7f868d75d700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.992061 7f868f761700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992060 7f8691765700 5 -- op tracker -- , seq: 3429, time: 2014-06-09 16:22:11.991740, event: started, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) 2014-06-09 16:22:11.992069 7f86af56b700 15 osd.26 243 enqueue_op 0x257c5600 prio 63 cost 1265664 latency 0.016921 osd_op(client.4541.0:32046 rbd_data.115d6b8b4567.0000000000001623 [set-alloc-hint object_size 4194304 write_size 4194304,write 2928640~1265664] 3.2aba3d05 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.992069 7f8691765700 7 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 19 op ack_type 1 from 3 2014-06-09 16:22:11.992073 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:981 2014-06-09 16:22:11.992091 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.992089 7f8691765700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:11.992082, event: sub_op_applied_rec, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.992097 7f8691765700 10 osd.26 243 dequeue_op 0x230da100 finish 2014-06-09 16:22:11.992096 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.1ef( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.992104 7f8691765700 5 -- op tracker -- , seq: 3429, time: 2014-06-09 16:22:11.992101, event: done, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ack, result = 0) 2014-06-09 16:22:11.992115 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.992129 7f868d75d700 10 log is not dirty 2014-06-09 16:22:11.992134 7f868d75d700 5 queue_transactions existing osr(1.1ef 0x384cdb0)/0x384cdb0 2014-06-09 16:22:11.992137 7f868d75d700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992152 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153584b0 (20 active) 2014-06-09 16:22:11.992161 7f8691f66700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992156 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.48b( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.992174 7f86a5610700 10 _do_op 0x2fcafe60 seq 3734 r = 0, finisher 0x30509140 0 2014-06-09 16:22:11.992179 7f86a5610700 10 _finish_op 0x2fcafe60 seq 3734 osr(0.117 0x2bd66a0)/0x2bd66a0 2014-06-09 16:22:11.992185 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653200 (1 active) 2014-06-09 16:22:11.992187 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.992188 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ade60 (2 active) 2014-06-09 16:22:11.992190 7f86a5610700 5 _do_op 0x2fcaf1a0 seq 3736 osr(1.348 0x384d310)/0x384d310 start 2014-06-09 16:22:11.992192 7f8691765700 5 queue_transactions existing osr(0.48b 0x2bd7450)/0x2bd7450 2014-06-09 16:22:11.992193 7f86a5610700 10 _do_transaction on 0xc253380 2014-06-09 16:22:11.992194 7f8691765700 10 op_submit_start 3784 2014-06-09 16:22:11.992195 7f8691765700 5 queue_transactions (trailing journal) 3784 2014-06-09 16:22:11.992196 7f8691765700 5 queue_op 0x16fd4300 seq 3784 osr(0.48b 0x2bd7450) 828 bytes (queue has 50 ops and 1095430 bytes) 2014-06-09 16:22:11.992199 7f8691765700 10 op_submit_finish 3784 2014-06-09 16:22:11.992199 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.992201 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.48b( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.48b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:11.992209 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.992225 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3735.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.992229 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.992229 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.992234 7f8691765700 5 queue_transactions existing osr(0.48b 0x2bd7450)/0x2bd7450 2014-06-09 16:22:11.992236 7f8691765700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992240 7f8690f64700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992245 7f868ef60700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992247 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbef0be0] 2014-06-09 16:22:11.992250 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbef0be0] 2014-06-09 16:22:11.992251 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.992250 7f868e75f700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992252 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.992250 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7c800,0x30509140] 2014-06-09 16:22:11.992256 7f868b759700 2 waiting 51 > 50 ops || 1096288 > 104857600 2014-06-09 16:22:11.992257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7c800,0x30509140] 2014-06-09 16:22:11.992259 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.992260 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.992261 7f868af58700 2 waiting 51 > 50 ops || 1096258 > 104857600 2014-06-09 16:22:11.992259 7f86a5e11700 10 _do_op 0x2fcae900 seq 3735 r = 0, finisher 0xbef1400 0 2014-06-09 16:22:11.992266 7f8689f56700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992253 7f869af78700 5 -- op tracker -- , seq: 3441, time: 2014-06-09 16:22:11.991794, event: done, op: pg_info(1 pgs e243:1.3bc) 2014-06-09 16:22:11.992270 7f86a5e11700 10 _finish_op 0x2fcae900 seq 3735 osr(3.139 0x3ed0780)/0x3ed0780 2014-06-09 16:22:11.992270 7f8692f68700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992274 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd63ec0] 2014-06-09 16:22:11.992275 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.992276 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd63ec0] 2014-06-09 16:22:11.992277 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.992277 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.992278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.992287 7f868df5e700 10 op_submit_start 3785 2014-06-09 16:22:11.992287 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f440 (1 active) 2014-06-09 16:22:11.992288 7f868df5e700 5 queue_transactions (trailing journal) 3785 2014-06-09 16:22:11.992289 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb200 (2 active) 2014-06-09 16:22:11.992291 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.992290 7f868df5e700 5 queue_op 0xbf7f500 seq 3785 osr(3.98 0x3ed0620) 826 bytes (queue has 50 ops and 1095428 bytes) 2014-06-09 16:22:11.992291 7f86a5e11700 5 _do_op 0x16096180 seq 3737 osr(1.211 0x384cc90)/0x384cc90 start 2014-06-09 16:22:11.992293 7f868df5e700 10 op_submit_finish 3785 2014-06-09 16:22:11.992292 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.992294 7f86a5e11700 10 _do_transaction on 0xc1f29c0 2014-06-09 16:22:11.992294 7f869af78700 20 osd.26 243 _dispatch 0x30c77500 pg_info(1 pgs e243:0.648) v4 2014-06-09 16:22:11.992294 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.992299 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.992299 7f869af78700 5 -- op tracker -- , seq: 3443, time: 2014-06-09 16:22:11.969261, event: header_read, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:11.992296 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.4c9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.992306 7f869af78700 5 -- op tracker -- , seq: 3443, time: 2014-06-09 16:22:11.969262, event: throttled, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:11.992310 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3736.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.992314 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.992312 7f869af78700 5 -- op tracker -- , seq: 3443, time: 2014-06-09 16:22:11.969281, event: all_read, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:11.992318 7f869af78700 5 -- op tracker -- , seq: 3443, time: 2014-06-09 16:22:11.992290, event: dispatched, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:11.992325 7f869af78700 5 -- op tracker -- , seq: 3443, time: 2014-06-09 16:22:11.992323, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:11.992328 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.992328 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef1400] 2014-06-09 16:22:11.992330 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.648) v4 from osd.5 2014-06-09 16:22:11.992333 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c77500 2014-06-09 16:22:11.992333 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef1400] 2014-06-09 16:22:11.992333 7f868df5e700 5 queue_transactions existing osr(0.4c9 0x2bd7650)/0x2bd7650 2014-06-09 16:22:11.992335 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.992336 7f868df5e700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992337 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.992340 7f868ff62700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992338 7f869af78700 5 -- op tracker -- , seq: 3443, time: 2014-06-09 16:22:11.992336, event: started, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:11.992345 7f868cf5c700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992344 7f86a5610700 10 _do_op 0x2fcaf1a0 seq 3736 r = 0, finisher 0x324f8360 0 2014-06-09 16:22:11.992349 7f8692767700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992350 7f86a5610700 10 _finish_op 0x2fcaf1a0 seq 3736 osr(1.348 0x384d310)/0x384d310 2014-06-09 16:22:11.992357 7f868df5e700 10 op_submit_start 3786 2014-06-09 16:22:11.992357 7f868a757700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992358 7f868df5e700 5 queue_transactions (trailing journal) 3786 2014-06-09 16:22:11.992357 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ade60 (1 active) 2014-06-09 16:22:11.992361 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c1b00 (2 active) 2014-06-09 16:22:11.992360 7f868df5e700 5 queue_op 0xbf7f2c0 seq 3786 osr(0.4c9 0x2bd7650) 828 bytes (queue has 50 ops and 1095428 bytes) 2014-06-09 16:22:11.992363 7f868df5e700 10 op_submit_finish 3786 2014-06-09 16:22:11.992363 7f8691765700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992363 7f86a5610700 5 _do_op 0xcabd560 seq 3738 osr(3.458 0x3ed1170)/0x3ed1170 start 2014-06-09 16:22:11.992366 7f86a5610700 10 _do_transaction on 0xc058540 2014-06-09 16:22:11.992368 7f868d75d700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992372 7f8691f66700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992373 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.992366 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.15b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.992377 7f8689755700 2 waiting 51 > 50 ops || 1096289 > 104857600 2014-06-09 16:22:11.992378 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f81e0] 2014-06-09 16:22:11.992380 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f81e0] 2014-06-09 16:22:11.992381 7f8690f64700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992380 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.992382 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.992384 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.992385 7f868ef60700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992390 7f868e75f700 2 waiting 51 > 50 ops || 1096254 > 104857600 2014-06-09 16:22:11.992394 7f868b759700 2 waiting 51 > 50 ops || 1096286 > 104857600 2014-06-09 16:22:11.992397 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.992399 7f868af58700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992354 7f869af78700 5 -- op tracker -- , seq: 3443, time: 2014-06-09 16:22:11.992351, event: done, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:11.992402 7f868df5e700 5 queue_transactions existing osr(1.15b 0x384cbf0)/0x384cbf0 2014-06-09 16:22:11.992394 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3737.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.992404 7f868df5e700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992406 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.992405 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.992408 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.992408 7f868bf5a700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992413 7f868f761700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992417 7f8689f56700 2 waiting 51 > 50 ops || 1096254 > 104857600 2014-06-09 16:22:11.992420 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.992420 7f868cf5c700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992421 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.992423 7f869af78700 20 osd.26 243 _dispatch 0x309e6540 pg_info(1 pgs e243:3.408) v4 2014-06-09 16:22:11.992424 7f8692767700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f8360] 2014-06-09 16:22:11.992428 7f8690763700 2 waiting 51 > 50 ops || 1096481 > 104857600 2014-06-09 16:22:11.992427 7f86a5e11700 10 _do_op 0x16096180 seq 3737 r = 0, finisher 0xbd7c5c0 0 2014-06-09 16:22:11.992430 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f8360] 2014-06-09 16:22:11.992431 7f8692f68700 2 waiting 51 > 50 ops || 1096254 > 104857600 2014-06-09 16:22:11.992428 7f869af78700 5 -- op tracker -- , seq: 3444, time: 2014-06-09 16:22:11.963304, event: header_read, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:11.992431 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.992433 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.992432 7f86a5e11700 10 _finish_op 0x16096180 seq 3737 osr(1.211 0x384cc90)/0x384cc90 2014-06-09 16:22:11.992434 7f868ff62700 2 waiting 51 > 50 ops || 1096256 > 104857600 2014-06-09 16:22:11.992436 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd786c0] 2014-06-09 16:22:11.992438 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd786c0] 2014-06-09 16:22:11.992439 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.992440 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.992444 7f868a757700 10 op_submit_start 3787 2014-06-09 16:22:11.992445 7f868a757700 5 queue_transactions (trailing journal) 3787 2014-06-09 16:22:11.992444 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb200 (1 active) 2014-06-09 16:22:11.992447 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52dc20 (2 active) 2014-06-09 16:22:11.992446 7f868a757700 5 queue_op 0xc88b020 seq 3787 osr(3.78a 0x3ed1d80) 18782 bytes (queue has 50 ops and 1113382 bytes) 2014-06-09 16:22:11.992450 7f868a757700 10 op_submit_finish 3787 2014-06-09 16:22:11.992449 7f86a5e11700 5 _do_op 0x30d97b60 seq 3739 osr(0.5a2 0x2bd7af0)/0x2bd7af0 start 2014-06-09 16:22:11.992452 7f86a5e11700 10 _do_transaction on 0xc29acc0 2014-06-09 16:22:11.992457 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.992454 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 luod=213'545 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc20e300 243'546 rep_tid=21 committed?=0 applied?=0 lock=0 op=osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:11.992476 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.992477 7f868a757700 10 osd.26 243 dequeue_op 0x19fe1500 finish 2014-06-09 16:22:11.992484 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.992491 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.992494 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3738.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.992499 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.992495 7f868a757700 10 osd.26 243 dequeue_op 0x270bff00 prio 196 cost 0 latency 0.008767 osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) v2 pg pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.992512 7f868d75d700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992517 7f8691f66700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992521 7f8689755700 2 waiting 51 > 50 ops || 1114243 > 104857600 2014-06-09 16:22:11.992524 7f8690f64700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992524 7f86a5610700 10 _do_op 0xcabd560 seq 3738 r = 0, finisher 0x30e3f480 0 2014-06-09 16:22:11.992528 7f868ef60700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992530 7f86a5610700 10 _finish_op 0xcabd560 seq 3738 osr(3.458 0x3ed1170)/0x3ed1170 2014-06-09 16:22:11.992532 7f868e75f700 2 waiting 51 > 50 ops || 1114208 > 104857600 2014-06-09 16:22:11.992535 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3fa80] 2014-06-09 16:22:11.992536 7f868b759700 2 waiting 51 > 50 ops || 1114240 > 104857600 2014-06-09 16:22:11.992538 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3fa80] 2014-06-09 16:22:11.992539 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.992540 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.992541 7f8691765700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992544 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7c5c0] 2014-06-09 16:22:11.992436 7f869af78700 5 -- op tracker -- , seq: 3444, time: 2014-06-09 16:22:11.963305, event: throttled, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:11.992546 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7c5c0] 2014-06-09 16:22:11.992546 7f868df5e700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992548 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.992550 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.992553 7f868bf5a700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992556 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.992557 7f868f761700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992552 7f868a757700 5 -- op tracker -- , seq: 3430, time: 2014-06-09 16:22:11.992509, event: reached_pg, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.992562 7f8689f56700 2 waiting 51 > 50 ops || 1114208 > 104857600 2014-06-09 16:22:11.992566 7f868cf5c700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992570 7f8692767700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992561 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x270bff00 2014-06-09 16:22:11.992571 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3739.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.992575 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.992574 7f8690763700 2 waiting 51 > 50 ops || 1114435 > 104857600 2014-06-09 16:22:11.992579 7f8692f68700 2 waiting 51 > 50 ops || 1114208 > 104857600 2014-06-09 16:22:11.992576 7f868a757700 5 -- op tracker -- , seq: 3430, time: 2014-06-09 16:22:11.992573, event: started, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.992582 7f868ff62700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992586 7f86a5e11700 10 _do_op 0x30d97b60 seq 3739 r = 0, finisher 0x200500c0 0 2014-06-09 16:22:11.992582 7f868a757700 7 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 19 op ack_type 4 from 3 2014-06-09 16:22:11.992595 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c1b00 (1 active) 2014-06-09 16:22:11.992598 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922d80 (2 active) 2014-06-09 16:22:11.992599 7f868a757700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:11.992593, event: sub_op_commit_rec, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.992602 7f86a5610700 5 _do_op 0x309a2780 seq 3740 osr(3.424 0x3ed1200)/0x3ed1200 start 2014-06-09 16:22:11.992604 7f86a5e11700 10 _finish_op 0x30d97b60 seq 3739 osr(0.5a2 0x2bd7af0)/0x2bd7af0 2014-06-09 16:22:11.992606 7f86a5610700 10 _do_transaction on 0xc320b40 2014-06-09 16:22:11.992605 7f868a757700 10 osd.26 243 dequeue_op 0x270bff00 finish 2014-06-09 16:22:11.992607 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52dc20 (1 active) 2014-06-09 16:22:11.992610 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac000 (2 active) 2014-06-09 16:22:11.992612 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.992613 7f86a5e11700 5 _do_op 0x15996de0 seq 3741 osr(2.5f8 0x2ed1db0)/0x2ed1db0 start 2014-06-09 16:22:11.992612 7f868a757700 5 -- op tracker -- , seq: 3430, time: 2014-06-09 16:22:11.992608, event: done, op: osd_sub_op_reply(client.4575.0:32325 3.f6 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.992615 7f86a5e11700 10 _do_transaction on 0xc138180 2014-06-09 16:22:11.992622 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.992622 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.992647 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e340 (20 active) 2014-06-09 16:22:11.992638 7f868c75b700 10 osd.26 243 dequeue_op 0x1b0b6900 prio 196 cost 0 latency 0.030252 osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) v2 pg pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 luod=213'545 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.992652 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.992651 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.216( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.992658 7f868c75b700 5 -- op tracker -- , seq: 3372, time: 2014-06-09 16:22:11.992655, event: reached_pg, op: osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.992669 7f86a5610700 15 _omap_rmkeyrange meta/1fe1f4da/pglog_3.424/0//-1 [0000000214.00000000000000000729,4294967295.18446744073709551615] 2014-06-09 16:22:11.992673 7f86a5610700 15 get_omap_iterator meta/1fe1f4da/pglog_3.424/0//-1 2014-06-09 16:22:11.992680 7f868af58700 10 op_submit_start 3788 2014-06-09 16:22:11.992681 7f868af58700 5 queue_transactions (trailing journal) 3788 2014-06-09 16:22:11.992683 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.992682 7f868af58700 5 queue_op 0x30d97aa0 seq 3788 osr(1.3cb 0x384d4f0) 828 bytes (queue has 49 ops and 1112329 bytes) 2014-06-09 16:22:11.992685 7f868af58700 10 op_submit_finish 3788 2014-06-09 16:22:11.992688 7f868a757700 5 queue_transactions existing osr(2.216 0x2ed0b50)/0x2ed0b50 2014-06-09 16:22:11.992691 7f868a757700 10 op_submit_start 3789 2014-06-09 16:22:11.992692 7f868a757700 5 queue_transactions (trailing journal) 3789 2014-06-09 16:22:11.992684 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 luod=213'545 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1b0b6900 2014-06-09 16:22:11.992693 7f868a757700 5 queue_op 0xc88a3c0 seq 3789 osr(2.216 0x2ed0b50) 828 bytes (queue has 50 ops and 1113157 bytes) 2014-06-09 16:22:11.992696 7f868a757700 10 op_submit_finish 3789 2014-06-09 16:22:11.992689 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:11.992699 7f868c75b700 5 -- op tracker -- , seq: 3372, time: 2014-06-09 16:22:11.992696, event: started, op: osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.992701 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.992699 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.5c6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:11.992706 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 luod=213'545 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 21 op ack_type 4 from 1 2014-06-09 16:22:11.992710 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.992717 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.992730 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.992729 7f868af58700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.992734 7f868a757700 5 queue_transactions existing osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 2014-06-09 16:22:11.992737 7f868a757700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992741 7f868d75d700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992722 7f868c75b700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:11.992716, event: sub_op_commit_rec, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.992745 7f868c75b700 10 osd.26 243 dequeue_op 0x1b0b6900 finish 2014-06-09 16:22:11.992746 7f8689755700 2 waiting 51 > 50 ops || 1114018 > 104857600 2014-06-09 16:22:11.992750 7f8690f64700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992745 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:11.992754 7f8691f66700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992752 7f868c75b700 5 -- op tracker -- , seq: 3372, time: 2014-06-09 16:22:11.992748, event: done, op: osd_sub_op_reply(client.4565.0:32907 3.78a 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 [] ondisk, result = 0) 2014-06-09 16:22:11.992758 7f868e75f700 2 waiting 51 > 50 ops || 1113983 > 104857600 2014-06-09 16:22:11.992760 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.992762 7f868b759700 2 waiting 51 > 50 ops || 1114015 > 104857600 2014-06-09 16:22:11.992762 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:11.992766 7f868ef60700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992768 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:11.992771 7f8691765700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992764 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.992776 7f868bf5a700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992776 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.992782 7f868f761700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992779 7f869af78700 5 -- op tracker -- , seq: 3444, time: 2014-06-09 16:22:11.963321, event: all_read, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:11.992786 7f8689f56700 2 waiting 51 > 50 ops || 1113983 > 104857600 2014-06-09 16:22:11.992787 7f869af78700 5 -- op tracker -- , seq: 3444, time: 2014-06-09 16:22:11.992419, event: dispatched, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:11.992784 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.992790 7f868cf5c700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992794 7f8692767700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992791 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.992772 7f868c75b700 10 osd.26 243 dequeue_op 0x4679b00 prio 63 cost 458752 latency 1.165142 osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:11.992798 7f8690763700 2 waiting 51 > 50 ops || 1114210 > 104857600 2014-06-09 16:22:11.992797 7f869af78700 5 -- op tracker -- , seq: 3444, time: 2014-06-09 16:22:11.992795, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:11.992803 7f8692f68700 2 waiting 51 > 50 ops || 1113983 > 104857600 2014-06-09 16:22:11.992804 7f868af58700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.992803 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.408) v4 from osd.2 2014-06-09 16:22:11.992807 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e6540 2014-06-09 16:22:11.992808 7f868ff62700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992812 7f868df5e700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992820 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:11.992835 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.992817 7f868c75b700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:11.992800, event: reached_pg, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.992840 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.4a7( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:11.992848 7f869af78700 5 -- op tracker -- , seq: 3444, time: 2014-06-09 16:22:11.992810, event: started, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:11.992845 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:11.992865 7f869af78700 5 -- op tracker -- , seq: 3444, time: 2014-06-09 16:22:11.992863, event: done, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:11.992861 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x4679b00 2014-06-09 16:22:11.992873 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.992875 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.992875 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.992880 7f868af58700 5 queue_transactions existing osr(0.4a7 0x2bd7370)/0x2bd7370 2014-06-09 16:22:11.992883 7f868af58700 2 waiting 51 > 50 ops || 1113985 > 104857600 2014-06-09 16:22:11.992872 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:11.992887 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.992889 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.992890 7f869af78700 20 osd.26 243 _dispatch 0x31388c40 pg_info(1 pgs e243:2.719) v4 2014-06-09 16:22:11.992896 7f868c75b700 15 getattr 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 '_' 2014-06-09 16:22:11.992895 7f869af78700 5 -- op tracker -- , seq: 3446, time: 2014-06-09 16:22:11.960412, event: header_read, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:11.992902 7f869af78700 5 -- op tracker -- , seq: 3446, time: 2014-06-09 16:22:11.960413, event: throttled, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:11.992908 7f869af78700 5 -- op tracker -- , seq: 3446, time: 2014-06-09 16:22:11.960422, event: all_read, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:11.992914 7f869af78700 5 -- op tracker -- , seq: 3446, time: 2014-06-09 16:22:11.992887, event: dispatched, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:11.992921 7f869af78700 5 -- op tracker -- , seq: 3446, time: 2014-06-09 16:22:11.992919, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:11.992928 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1322de00] 2014-06-09 16:22:11.992928 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.719) v4 from osd.4 2014-06-09 16:22:11.992931 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1322de00] 2014-06-09 16:22:11.992931 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31388c40 2014-06-09 16:22:11.992932 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.992934 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.992938 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e3f480,0x200500c0] 2014-06-09 16:22:11.992936 7f869af78700 5 -- op tracker -- , seq: 3446, time: 2014-06-09 16:22:11.992934, event: started, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:11.992952 7f869af78700 5 -- op tracker -- , seq: 3446, time: 2014-06-09 16:22:11.992950, event: done, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:11.992941 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:11.992958 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.992960 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.992966 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.992959 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.458( v 213'616 (0'0,213'616] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:11.992973 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.992974 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.992976 7f869af78700 20 osd.26 243 _dispatch 0x31745a40 pg_info(1 pgs e243:0.6b6) v4 2014-06-09 16:22:11.992978 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e3f480,0x200500c0] 2014-06-09 16:22:11.992980 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.992982 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.992970 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 but can_create 2014-06-09 16:22:11.992981 7f869af78700 5 -- op tracker -- , seq: 3447, time: 2014-06-09 16:22:11.972453, event: header_read, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.992985 7f868c75b700 15 getattr 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 'snapset' 2014-06-09 16:22:11.992988 7f869af78700 5 -- op tracker -- , seq: 3447, time: 2014-06-09 16:22:11.972453, event: throttled, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.993003 7f869af78700 5 -- op tracker -- , seq: 3447, time: 2014-06-09 16:22:11.972464, event: all_read, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.993011 7f869af78700 5 -- op tracker -- , seq: 3447, time: 2014-06-09 16:22:11.992972, event: dispatched, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.993019 7f869af78700 5 -- op tracker -- , seq: 3447, time: 2014-06-09 16:22:11.993017, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.993027 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.6b6) v4 from osd.3 2014-06-09 16:22:11.993030 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31745a40 2014-06-09 16:22:11.993065 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.993068 7f868c75b700 15 getattr 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/snapdir//3 'snapset' 2014-06-09 16:22:11.993096 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.993105 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30be2b00 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:11.993118 7f86a5610700 15 _omap_rmkeys meta/1fe1f4da/pglog_3.424/0//-1 2014-06-09 16:22:11.993118 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:11.993129 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:11.993139 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30be2b00 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 rwstate(none n=0 w=0) oi: bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc3c00 snapset: 0=[]:[] 2014-06-09 16:22:11.993156 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe1f4da/pglog_3.424/0//-1 2014-06-09 16:22:11.993155 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 @head oi=bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:11.993168 7f86a5610700 10 _omap_rmkeys meta/1fe1f4da/pglog_3.424/0//-1 r = 0 2014-06-09 16:22:11.993173 7f86a5610700 15 _omap_rmkeys meta/1fe1f4da/pglog_3.424/0//-1 2014-06-09 16:22:11.993175 7f868c75b700 15 getattr 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/snapdir//3 '_' 2014-06-09 16:22:11.993178 7f86a5610700 10 _omap_rmkeys meta/1fe1f4da/pglog_3.424/0//-1 r = 0 2014-06-09 16:22:11.993186 7f86a5610700 15 _omap_setkeys meta/1fe1f4da/pglog_3.424/0//-1 2014-06-09 16:22:11.993200 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3740.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.993205 7f86a5610700 10 cid: oid: 1fe1f4da/pglog_3.424/0//-1 skipping op, spos 3740.0.0 <= header.spos 150431.0.0 2014-06-09 16:22:11.993207 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:11.993209 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.993210 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid bc749224/rbd_data.11776b8b4567.00000000000017fb/snapdir//3 and !can_create 2014-06-09 16:22:11.993229 7f86af56b700 5 -- op tracker -- , seq: 3445, time: 2014-06-09 16:22:11.992093, event: header_read, op: osd_op(client.4541.0:32083 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~208896] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.993233 7f86a5610700 10 _do_op 0x309a2780 seq 3740 r = 0, finisher 0x29283e80 0 2014-06-09 16:22:11.993240 7f86a5610700 10 _finish_op 0x309a2780 seq 3740 osr(3.424 0x3ed1200)/0x3ed1200 2014-06-09 16:22:11.993243 7f86af56b700 5 -- op tracker -- , seq: 3445, time: 2014-06-09 16:22:11.992094, event: throttled, op: osd_op(client.4541.0:32083 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~208896] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.993248 7f868a757700 10 op_submit_start 3790 2014-06-09 16:22:11.993248 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922d80 (1 active) 2014-06-09 16:22:11.993250 7f868a757700 5 queue_transactions (trailing journal) 3790 2014-06-09 16:22:11.993251 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652b40 (2 active) 2014-06-09 16:22:11.993250 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x29283e80] 2014-06-09 16:22:11.993253 7f86a5610700 5 _do_op 0x16fd4ae0 seq 3742 osr(1.2fb 0x384d090)/0x384d090 start 2014-06-09 16:22:11.993252 7f868a757700 5 queue_op 0xc88a720 seq 3790 osr(2.5c6 0x2ed1ad0) 828 bytes (queue has 50 ops and 1112259 bytes) 2014-06-09 16:22:11.993255 7f868a757700 10 op_submit_finish 3790 2014-06-09 16:22:11.993255 7f86a5610700 10 _do_transaction on 0x3339d40 2014-06-09 16:22:11.993257 7f868c75b700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:11.993223, event: started, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.993261 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.993262 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x29283e80] 2014-06-09 16:22:11.993264 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.993265 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.993258 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.2c0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:11.993265 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30643000 2014-06-09 16:22:11.993284 7f868c75b700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:11.993279, event: started, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.993291 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.993296 7f868a757700 5 queue_transactions existing osr(1.2c0 0x384d170)/0x384d170 2014-06-09 16:22:11.993299 7f868a757700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.993300 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.993290 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] ov 0'0 av 243'740 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:11.993303 7f868d75d700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.993308 7f8689755700 2 waiting 51 > 50 ops || 1113120 > 104857600 2014-06-09 16:22:11.993312 7f8690f64700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.993305 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 2014-06-09 16:22:11.993316 7f8691f66700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.993320 7f868e75f700 2 waiting 51 > 50 ops || 1113085 > 104857600 2014-06-09 16:22:11.993317 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3742.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.993322 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.993324 7f868b759700 2 waiting 51 > 50 ops || 1113117 > 104857600 2014-06-09 16:22:11.993317 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:11.993327 7f868ef60700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.993331 7f8691765700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.993337 7f868bf5a700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.993332 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~458752 2014-06-09 16:22:11.993338 7f869af78700 5 -- op tracker -- , seq: 3447, time: 2014-06-09 16:22:11.993033, event: started, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.993343 7f868f761700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.993342 7f86a5610700 10 _do_op 0x16fd4ae0 seq 3742 r = 0, finisher 0x31bfe900 0 2014-06-09 16:22:11.993346 7f86a5610700 10 _finish_op 0x16fd4ae0 seq 3742 osr(1.2fb 0x384d090)/0x384d090 2014-06-09 16:22:11.993347 7f8689f56700 2 waiting 51 > 50 ops || 1113085 > 104857600 2014-06-09 16:22:11.993352 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b640,0x31bfe820] 2014-06-09 16:22:11.993349 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228400 2014-06-09 16:22:11.993356 7f869af78700 5 -- op tracker -- , seq: 3447, time: 2014-06-09 16:22:11.993353, event: done, op: pg_info(1 pgs e243:0.6b6) 2014-06-09 16:22:11.993364 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.993367 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.993360 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 snapset=0x14dc3c38 snapc=0=[] 2014-06-09 16:22:11.993355 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 203 same_interval_since 242 2014-06-09 16:22:11.993379 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.993372 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:11.993381 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.993382 7f869af78700 20 osd.26 243 _dispatch 0x30fcd340 pg_info(1 pgs e243:2.1e0) v4 2014-06-09 16:22:11.993374 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.993388 7f869af78700 5 -- op tracker -- , seq: 3448, time: 2014-06-09 16:22:11.968398, event: header_read, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:11.993383 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:11.993395 7f869af78700 5 -- op tracker -- , seq: 3448, time: 2014-06-09 16:22:11.968399, event: throttled, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:11.993389 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'730 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:11.993394 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 0x30643000 op modify 2014-06-09 16:22:11.993401 7f869af78700 5 -- op tracker -- , seq: 3448, time: 2014-06-09 16:22:11.968412, event: all_read, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:11.993408 7f869af78700 5 -- op tracker -- , seq: 3448, time: 2014-06-09 16:22:11.993379, event: dispatched, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:11.993404 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:11.993414 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:11.993407 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:43.356837 2014-06-09 16:22:11.993416 7f869af78700 5 -- op tracker -- , seq: 3448, time: 2014-06-09 16:22:11.993414, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:11.993426 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1e0) v4 from osd.9 2014-06-09 16:22:11.993429 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcd340 2014-06-09 16:22:11.993424 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.2fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:11.993433 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b640,0x31bfe820] 2014-06-09 16:22:11.993435 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.993437 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.993431 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=135 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:11.993447 7f8692767700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.993448 7f868a757700 10 op_submit_start 3791 2014-06-09 16:22:11.993447 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652b40 (1 active) 2014-06-09 16:22:11.993449 7f868a757700 5 queue_transactions (trailing journal) 3791 2014-06-09 16:22:11.993450 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300480 (2 active) 2014-06-09 16:22:11.993451 7f868a757700 5 queue_op 0xcabcba0 seq 3791 osr(1.2c0 0x384d170) 828 bytes (queue has 50 ops and 1112226 bytes) 2014-06-09 16:22:11.993453 7f8690763700 2 waiting 51 > 50 ops || 1113279 > 104857600 2014-06-09 16:22:11.993454 7f868a757700 10 op_submit_finish 3791 2014-06-09 16:22:11.993453 7f86a5610700 5 _do_op 0x3250ed80 seq 3743 osr(3.7fc 0xb7f00e0)/0xb7f00e0 start 2014-06-09 16:22:11.993456 7f86a5610700 10 _do_transaction on 0xc262600 2014-06-09 16:22:11.993457 7f868ff62700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.993450 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:11.993461 7f868df5e700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.993461 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.993457 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/200) 2014-06-09 16:22:11.993462 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 22 on osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:11.993478 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 22 o bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:11.993490 7f868a757700 10 log is not dirty 2014-06-09 16:22:11.993494 7f868a757700 5 queue_transactions existing osr(3.52a 0x3ed1670)/0x3ed1670 2014-06-09 16:22:11.993497 7f868a757700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.993500 7f8692f68700 2 waiting 51 > 50 ops || 1113052 > 104857600 2014-06-09 16:22:11.993435 7f869af78700 5 -- op tracker -- , seq: 3448, time: 2014-06-09 16:22:11.993432, event: started, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:11.993501 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.993505 7f868cf5c700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.993515 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3743.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.993512 7f868c75b700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:11.993496, event: waiting for subops from 6, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:11.993520 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.993531 7f86a5610700 10 _do_op 0x3250ed80 seq 3743 r = 0, finisher 0xbd5d000 0 2014-06-09 16:22:11.993529 7f86af56b700 5 -- op tracker -- , seq: 3445, time: 2014-06-09 16:22:11.992772, event: all_read, op: osd_op(client.4541.0:32083 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~208896] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.993533 7f868c75b700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:11.993537 7f86a5610700 10 _finish_op 0x3250ed80 seq 3743 osr(3.7fc 0xb7f00e0)/0xb7f00e0 2014-06-09 16:22:11.993543 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300480 (1 active) 2014-06-09 16:22:11.993542 7f86af56b700 5 -- op tracker -- , seq: 3445, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32083 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~208896] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.993545 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497d40 (2 active) 2014-06-09 16:22:11.993546 7f8692767700 10 op_submit_start 3792 2014-06-09 16:22:11.993548 7f8692767700 5 queue_transactions (trailing journal) 3792 2014-06-09 16:22:11.993548 7f86a5610700 5 _do_op 0x3250ec00 seq 3744 osr(2.220 0x2ed0b10)/0x2ed0b10 start 2014-06-09 16:22:11.993550 7f86a5610700 10 _do_transaction on 0xc058f00 2014-06-09 16:22:11.993549 7f8692767700 5 queue_op 0xc2eade0 seq 3792 osr(2.69d 0x2ed1ef0) 828 bytes (queue has 50 ops and 1112226 bytes) 2014-06-09 16:22:11.993548 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:11.993552 7f8692767700 10 op_submit_finish 3792 2014-06-09 16:22:11.993556 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.993556 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.33f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:11.993552 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 214'739 (0'0,214'739] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'739], crt=0'0) [243'740 (0'0) modify bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 by client.4567.0:35562 2014-06-09 10:20:43.356837] 2014-06-09 16:22:11.993564 7f86af56b700 15 osd.26 243 enqueue_op 0x257c4000 prio 63 cost 208896 latency 0.001471 osd_op(client.4541.0:32083 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~208896] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.993575 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'740 (0'0) modify bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 by client.4567.0:35562 2014-06-09 10:20:43.356837 2014-06-09 16:22:11.993592 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.993593 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.993598 7f8692767700 5 queue_transactions existing osr(0.33f 0x2bd7060)/0x2bd7060 2014-06-09 16:22:11.993600 7f8692767700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.993595 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:11.993607 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3744.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.993611 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.993622 7f86a5610700 10 _do_op 0x3250ec00 seq 3744 r = 0, finisher 0x30e146e0 0 2014-06-09 16:22:11.993625 7f86a5610700 10 _finish_op 0x3250ec00 seq 3744 osr(2.220 0x2ed0b10)/0x2ed0b10 2014-06-09 16:22:11.993625 7f868c75b700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'740, trimmed: 2014-06-09 16:22:11.993631 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497d40 (1 active) 2014-06-09 16:22:11.993633 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e5a0 (2 active) 2014-06-09 16:22:11.993634 7f8692767700 10 op_submit_start 3793 2014-06-09 16:22:11.993636 7f8692767700 5 queue_transactions (trailing journal) 3793 2014-06-09 16:22:11.993636 7f86a5610700 5 _do_op 0xcabcde0 seq 3745 osr(0.714 0x384c0d0)/0x384c0d0 start 2014-06-09 16:22:11.993638 7f86a5610700 10 _do_transaction on 0xc20a780 2014-06-09 16:22:11.993638 7f8692767700 5 queue_op 0x16fd4360 seq 3793 osr(0.33f 0x2bd7060) 828 bytes (queue has 50 ops and 1112226 bytes) 2014-06-09 16:22:11.993641 7f8692767700 10 op_submit_finish 3793 2014-06-09 16:22:11.993643 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.993645 7f868c75b700 5 queue_transactions existing osr(3.224 0x3ed0c80)/0x3ed0c80 2014-06-09 16:22:11.993648 7f868c75b700 2 waiting 51 > 50 ops || 1573376 > 104857600 2014-06-09 16:22:11.993644 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.a7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.993676 7f8692767700 10 log is not dirty 2014-06-09 16:22:11.993680 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.993682 7f8690f64700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.993695 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3745.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.993699 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.993709 7f86a5610700 10 _do_op 0xcabcde0 seq 3745 r = 0, finisher 0x2d738cc0 0 2014-06-09 16:22:11.993713 7f86a5610700 10 _finish_op 0xcabcde0 seq 3745 osr(0.714 0x384c0d0)/0x384c0d0 2014-06-09 16:22:11.993718 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e5a0 (1 active) 2014-06-09 16:22:11.993721 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406ea0 (2 active) 2014-06-09 16:22:11.993723 7f86a5610700 5 _do_op 0x2ea53260 seq 3746 osr(3.3f5 0x3ed1280)/0x3ed1280 start 2014-06-09 16:22:11.993726 7f86a5610700 10 _do_transaction on 0xbf2e0c0 2014-06-09 16:22:11.998053 7f86a5610700 15 _omap_setkeys meta/1fe0bce5/pglog_3.3f5/0//-1 2014-06-09 16:22:11.998135 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe0bce5/pglog_3.3f5/0//-1 2014-06-09 16:22:11.998155 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.993732 7f868af58700 10 op_submit_start 3794 2014-06-09 16:22:11.998199 7f868af58700 5 queue_transactions (trailing journal) 3794 2014-06-09 16:22:11.998202 7f868af58700 5 queue_op 0x30d97980 seq 3794 osr(0.4a7 0x2bd7370) 828 bytes (queue has 50 ops and 1112226 bytes) 2014-06-09 16:22:11.998206 7f868af58700 10 op_submit_finish 3794 2014-06-09 16:22:11.998212 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.27c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:11.998257 7f868af58700 10 log is not dirty 2014-06-09 16:22:11.998265 7f868af58700 5 queue_transactions existing osr(1.27c 0x384cf30)/0x384cf30 2014-06-09 16:22:11.993735 7f868e75f700 2 waiting 51 > 50 ops || 1113052 > 104857600 2014-06-09 16:22:11.993739 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e14680,0x30629240,0xd9faa80] 2014-06-09 16:22:11.998283 7f868b759700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998285 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e14680,0x30629240,0xd9faa80] 2014-06-09 16:22:11.998288 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.998289 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.993743 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31bfe900,0xbd5d000,0x30e146e0,0x2d738cc0] 2014-06-09 16:22:11.998292 7f868ef60700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998298 7f8691765700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998306 7f868bf5a700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998309 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31bfe900,0xbd5d000,0x30e146e0,0x2d738cc0] 2014-06-09 16:22:11.998313 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:11.998314 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:11.993781 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.998316 7f868f761700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.993910 7f8692767700 5 queue_transactions existing osr(1.a7 0x384c730)/0x384c730 2014-06-09 16:22:11.993981 7f869af78700 5 -- op tracker -- , seq: 3448, time: 2014-06-09 16:22:11.993979, event: done, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:11.998323 7f8689f56700 2 waiting 51 > 50 ops || 1113052 > 104857600 2014-06-09 16:22:11.998330 7f8690763700 2 waiting 51 > 50 ops || 1113279 > 104857600 2014-06-09 16:22:11.998338 7f868ff62700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998339 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.998343 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.998345 7f868df5e700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998344 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3741.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.998349 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:11.998352 7f868a757700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998360 7f8692f68700 2 waiting 51 > 50 ops || 1113052 > 104857600 2014-06-09 16:22:11.998366 7f868cf5c700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998351 7f86af56b700 5 -- op tracker -- , seq: 3449, time: 2014-06-09 16:22:11.993584, event: header_read, op: osd_op(client.4541.0:32084 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 208896~2580480] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.998372 7f8690f64700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998373 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.998375 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.998374 7f86af56b700 5 -- op tracker -- , seq: 3449, time: 2014-06-09 16:22:11.993585, event: throttled, op: osd_op(client.4541.0:32084 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 208896~2580480] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.998379 7f8689755700 2 waiting 51 > 50 ops || 1113087 > 104857600 2014-06-09 16:22:11.998385 7f868d75d700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998385 7f86af56b700 5 -- op tracker -- , seq: 3449, time: 2014-06-09 16:22:11.998041, event: all_read, op: osd_op(client.4541.0:32084 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 208896~2580480] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.998392 7f8691f66700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998395 7f86af56b700 5 -- op tracker -- , seq: 3449, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32084 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 208896~2580480] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:11.998394 7f86a5e11700 10 _do_op 0x15996de0 seq 3741 r = 0, finisher 0xc6e8820 0 2014-06-09 16:22:11.998398 7f868c75b700 2 waiting 51 > 50 ops || 1573376 > 104857600 2014-06-09 16:22:11.998404 7f868af58700 2 waiting 51 > 50 ops || 1113054 > 104857600 2014-06-09 16:22:11.998403 7f869af78700 20 osd.26 243 _dispatch 0x31ac28c0 pg_info(1 pgs e243:2.552) v4 2014-06-09 16:22:11.998402 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:11.998409 7f8692767700 2 waiting 51 > 50 ops || 1113052 > 104857600 2014-06-09 16:22:11.998413 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x170009c0] 2014-06-09 16:22:11.998410 7f869af78700 5 -- op tracker -- , seq: 3450, time: 2014-06-09 16:22:11.967381, event: header_read, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:11.998415 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x170009c0] 2014-06-09 16:22:11.998417 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:11.998418 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:11.998421 7f86a5e11700 10 _finish_op 0x15996de0 seq 3741 osr(2.5f8 0x2ed1db0)/0x2ed1db0 2014-06-09 16:22:11.998419 7f869af78700 5 -- op tracker -- , seq: 3450, time: 2014-06-09 16:22:11.967382, event: throttled, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:11.998428 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.998431 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac000 (1 active) 2014-06-09 16:22:11.998433 7f868b759700 10 op_submit_start 3795 2014-06-09 16:22:11.998425 7f86af56b700 15 osd.26 243 enqueue_op 0x230da000 prio 63 cost 2580480 latency 0.004841 osd_op(client.4541.0:32084 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 208896~2580480] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:11.998435 7f868b759700 5 queue_transactions (trailing journal) 3795 2014-06-09 16:22:11.998435 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653b00 (2 active) 2014-06-09 16:22:11.998436 7f868b759700 5 queue_op 0x16fd5b60 seq 3795 osr(0.3e 0x2bd6270) 858 bytes (queue has 50 ops and 1112256 bytes) 2014-06-09 16:22:11.998437 7f86a5e11700 5 _do_op 0xcabd320 seq 3747 osr(0.55d 0x2bd77f0)/0x2bd77f0 start 2014-06-09 16:22:11.998439 7f868b759700 10 op_submit_finish 3795 2014-06-09 16:22:11.998440 7f86a5e11700 10 _do_transaction on 0x3415200 2014-06-09 16:22:11.998445 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.998445 7f86a5610700 15 _omap_rmkeys meta/1fe0bce5/pglog_3.3f5/0//-1 2014-06-09 16:22:11.998450 7f86a5610700 10 _omap_rmkeys meta/1fe0bce5/pglog_3.3f5/0//-1 r = 0 2014-06-09 16:22:11.998444 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.998457 7f86a5610700 15 _omap_setkeys meta/1fe0bce5/pglog_3.3f5/0//-1 2014-06-09 16:22:11.998459 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.860950 7 0.000402 2014-06-09 16:22:11.998469 7f86a5610700 15 _touch 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.998473 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.998485 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.998497 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000025 0 0.000000 2014-06-09 16:22:11.998508 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.998517 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.998528 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.3f5_head oid 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.998524 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.998537 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:11.998549 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:11.998559 7f86a5610700 20 write_state seq is 30158 2014-06-09 16:22:11.998558 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.998567 7f86a5610700 20 set_header setting 30157 cid 3.3f5_head oid 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 parent seq 0 2014-06-09 16:22:11.998574 7f868b759700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:11.998581 7f86a5610700 10 _touch 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 = 0 2014-06-09 16:22:11.998588 7f86a5610700 15 _write 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 0~16384 2014-06-09 16:22:11.998594 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:11.998592 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:186 2014-06-09 16:22:11.998626 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.998630 7f868b759700 5 queue_transactions existing osr(1.62a 0x384dfd0)/0x384dfd0 2014-06-09 16:22:11.998633 7f868b759700 2 waiting 51 > 50 ops || 1113117 > 104857600 2014-06-09 16:22:11.998637 7f868ef60700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998642 7f8691765700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998426 7f869af78700 5 -- op tracker -- , seq: 3450, time: 2014-06-09 16:22:11.967392, event: all_read, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:11.998648 7f868bf5a700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998651 7f869af78700 5 -- op tracker -- , seq: 3450, time: 2014-06-09 16:22:11.998372, event: dispatched, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:11.998655 7f868f761700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998660 7f86a5610700 10 _generic_write 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 0~16384 = 0 2014-06-09 16:22:11.998662 7f8689f56700 2 waiting 51 > 50 ops || 1113082 > 104857600 2014-06-09 16:22:11.998660 7f869af78700 5 -- op tracker -- , seq: 3450, time: 2014-06-09 16:22:11.998658, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:11.998665 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.552) v4 from osd.8 2014-06-09 16:22:11.998667 7f8690763700 2 waiting 51 > 50 ops || 1113309 > 104857600 2014-06-09 16:22:11.998668 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31ac28c0 2014-06-09 16:22:11.998669 7f86a5610700 15 _setattrs 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.998674 7f868ff62700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998674 7f869af78700 5 -- op tracker -- , seq: 3450, time: 2014-06-09 16:22:11.998672, event: started, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:11.998680 7f86a5610700 10 _setattrs 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 = 0 2014-06-09 16:22:11.998681 7f868df5e700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998686 7f86a5610700 15 _setattrs 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 2014-06-09 16:22:11.998687 7f868a757700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998694 7f86a5610700 10 _setattrs 3.3f5_head/5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 = 0 2014-06-09 16:22:11.998701 7f86a5610700 10 cid: oid: 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 not skipping op, *spos 3746.0.0 2014-06-09 16:22:11.998704 7f86a5610700 10 > header.spos 0.0.0 2014-06-09 16:22:11.998706 7f8692f68700 2 waiting 51 > 50 ops || 1113082 > 104857600 2014-06-09 16:22:11.998709 7f86a5610700 20 set_header setting 30157 cid 3.3f5_head oid 5b1b53f5/rbd_data.117b6b8b4567.0000000000001843/head//3 parent seq 0 2014-06-09 16:22:11.998708 7f869af78700 5 -- op tracker -- , seq: 3450, time: 2014-06-09 16:22:11.998706, event: done, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:11.998712 7f868cf5c700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998717 7f8690f64700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998717 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.998723 7f8689755700 2 waiting 51 > 50 ops || 1113117 > 104857600 2014-06-09 16:22:11.998723 7f86a5610700 10 cid: oid: 1fe0bce5/pglog_3.3f5/0//-1 skipping op, spos 3746.0.0 <= header.spos 148178.0.0 2014-06-09 16:22:11.998728 7f868d75d700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998728 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3746.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.998731 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.998732 7f8691f66700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998736 7f868c75b700 2 waiting 51 > 50 ops || 1573406 > 104857600 2014-06-09 16:22:11.998740 7f868af58700 2 waiting 51 > 50 ops || 1113084 > 104857600 2014-06-09 16:22:11.998744 7f8692767700 2 waiting 51 > 50 ops || 1113082 > 104857600 2014-06-09 16:22:11.998748 7f868e75f700 2 waiting 51 > 50 ops || 1113082 > 104857600 2014-06-09 16:22:11.998719 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.998765 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.998767 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.998768 7f869af78700 20 osd.26 243 _dispatch 0x30ada700 pg_info(1 pgs e243:3.46f) v4 2014-06-09 16:22:11.998774 7f869af78700 5 -- op tracker -- , seq: 3451, time: 2014-06-09 16:22:11.969289, event: header_read, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.998781 7f869af78700 5 -- op tracker -- , seq: 3451, time: 2014-06-09 16:22:11.969289, event: throttled, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.998790 7f869af78700 5 -- op tracker -- , seq: 3451, time: 2014-06-09 16:22:11.969299, event: all_read, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.998796 7f869af78700 5 -- op tracker -- , seq: 3451, time: 2014-06-09 16:22:11.998765, event: dispatched, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.998812 7f869af78700 5 -- op tracker -- , seq: 3451, time: 2014-06-09 16:22:11.998809, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.998816 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.46f) v4 from osd.5 2014-06-09 16:22:11.998820 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ada700 2014-06-09 16:22:11.998825 7f869af78700 5 -- op tracker -- , seq: 3451, time: 2014-06-09 16:22:11.998823, event: started, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.998837 7f86a5610700 10 _do_op 0x2ea53260 seq 3746 r = 0, finisher 0x3205ce40 0x30bca380 2014-06-09 16:22:11.998842 7f869af78700 5 -- op tracker -- , seq: 3451, time: 2014-06-09 16:22:11.998839, event: done, op: pg_info(1 pgs e243:3.46f) 2014-06-09 16:22:11.998844 7f86a5610700 10 _finish_op 0x2ea53260 seq 3746 osr(3.3f5 0x3ed1280)/0x3ed1280 2014-06-09 16:22:11.998853 7f868b759700 10 op_submit_start 3796 2014-06-09 16:22:11.998852 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406ea0 (1 active) 2014-06-09 16:22:11.998855 7f868b759700 5 queue_transactions (trailing journal) 3796 2014-06-09 16:22:11.998856 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e5a0 (2 active) 2014-06-09 16:22:11.998856 7f868b759700 5 queue_op 0x30607800 seq 3796 osr(1.62a 0x384dfd0) 861 bytes (queue has 50 ops and 1094335 bytes) 2014-06-09 16:22:11.998859 7f868b759700 10 op_submit_finish 3796 2014-06-09 16:22:11.998858 7f86a5610700 5 _do_op 0xcabd3e0 seq 3748 osr(0.714 0x384c0d0)/0x384c0d0 start 2014-06-09 16:22:11.998862 7f86a5610700 10 _do_transaction on 0xc1f7ec0 2014-06-09 16:22:11.998867 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.998863 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.998879 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.998889 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x306b700,0x306b200,0x306b100,0xe032f00,0xe032e00 2014-06-09 16:22:11.998905 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.998921 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3748.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.998925 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.998924 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.998935 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.998942 7f86a5610700 10 _do_op 0xcabd3e0 seq 3748 r = 0, finisher 0x32534c20 0 2014-06-09 16:22:11.998946 7f86a5610700 10 _finish_op 0xcabd3e0 seq 3748 osr(0.714 0x384c0d0)/0x384c0d0 2014-06-09 16:22:11.998949 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e5a0 (1 active) 2014-06-09 16:22:11.998944 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.998951 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0900 (2 active) 2014-06-09 16:22:11.998954 7f86a5610700 5 _do_op 0x3250fce0 seq 3749 osr(2.f2 0x2ed07e0)/0x2ed07e0 start 2014-06-09 16:22:11.998957 7f86a5610700 10 _do_transaction on 0xc227440 2014-06-09 16:22:11.998963 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.998960 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:935 2014-06-09 16:22:11.998970 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.998980 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.998990 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.998999 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.998995 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.999007 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.999015 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3749.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999019 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999015 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.999023 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.999031 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.999035 7f86a5610700 10 _do_op 0x3250fce0 seq 3749 r = 0, finisher 0x2e0f320 0 2014-06-09 16:22:11.999040 7f86a5610700 10 _finish_op 0x3250fce0 seq 3749 osr(2.f2 0x2ed07e0)/0x2ed07e0 2014-06-09 16:22:11.999043 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0900 (1 active) 2014-06-09 16:22:11.999038 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.999045 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3397a0 (2 active) 2014-06-09 16:22:11.999047 7f86a5610700 5 _do_op 0xbf7e540 seq 3750 osr(0.36d 0x2bd6fc0)/0x2bd6fc0 start 2014-06-09 16:22:11.999050 7f86a5610700 10 _do_transaction on 0xbf4e9c0 2014-06-09 16:22:11.999055 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999051 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.999060 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.999068 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.82( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.999076 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.999081 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.999091 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999094 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.999102 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.999105 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3750.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999109 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999109 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.999119 7f86a5610700 10 _do_op 0xbf7e540 seq 3750 r = 0, finisher 0xc5a6bc0 0 2014-06-09 16:22:11.999122 7f86a5610700 10 _finish_op 0xbf7e540 seq 3750 osr(0.36d 0x2bd6fc0)/0x2bd6fc0 2014-06-09 16:22:11.999117 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.999125 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3397a0 (1 active) 2014-06-09 16:22:11.999126 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358b40 (2 active) 2014-06-09 16:22:11.999128 7f86a5610700 5 _do_op 0x3250fb60 seq 3751 osr(2.fd 0x2ed0770)/0x2ed0770 start 2014-06-09 16:22:11.999129 7f86a5610700 10 _do_transaction on 0xbdebd40 2014-06-09 16:22:11.999125 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.999134 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999137 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:11.999146 7f868b759700 20 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.999153 7f868b759700 20 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.999162 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.999170 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999166 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:11.999178 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.830896 7 0.000401 2014-06-09 16:22:11.999184 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3751.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999189 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999189 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:11.999200 7f86a5610700 10 _do_op 0x3250fb60 seq 3751 r = 0, finisher 0x275cffc0 0 2014-06-09 16:22:11.999204 7f86a5610700 10 _finish_op 0x3250fb60 seq 3751 osr(2.fd 0x2ed0770)/0x2ed0770 2014-06-09 16:22:11.999198 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.999207 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358b40 (1 active) 2014-06-09 16:22:11.999209 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fac60 (2 active) 2014-06-09 16:22:11.999212 7f86a5610700 5 _do_op 0xbf7e060 seq 3752 osr(2.52e 0x2ed1600)/0x2ed1600 start 2014-06-09 16:22:11.999214 7f86a5610700 10 _do_transaction on 0x37e4300 2014-06-09 16:22:11.999208 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:11.999219 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999219 7f868b759700 5 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:11.999227 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:11.999234 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:11.999245 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:11.999253 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999255 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:11.999264 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:11.999267 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3752.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999272 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999275 7f868b759700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:11.999288 7f86a5610700 10 _do_op 0xbf7e060 seq 3752 r = 0, finisher 0x12051e80 0 2014-06-09 16:22:11.999292 7f86a5610700 10 _finish_op 0xbf7e060 seq 3752 osr(2.52e 0x2ed1600)/0x2ed1600 2014-06-09 16:22:11.999294 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fac60 (1 active) 2014-06-09 16:22:11.999297 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f9e0 (2 active) 2014-06-09 16:22:11.999290 7f868b759700 15 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:180 2014-06-09 16:22:11.999299 7f86a5610700 5 _do_op 0x324f4180 seq 3753 osr(2.620 0x2ed1cb0)/0x2ed1cb0 start 2014-06-09 16:22:11.999301 7f86a5610700 10 _do_transaction on 0xc327380 2014-06-09 16:22:11.999306 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999324 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.999328 7f868b759700 5 queue_transactions existing osr(1.70c 0x2ed00f0)/0x2ed00f0 2014-06-09 16:22:11.999330 7f868b759700 10 op_submit_start 3797 2014-06-09 16:22:11.999331 7f868b759700 5 queue_transactions (trailing journal) 3797 2014-06-09 16:22:11.999333 7f868b759700 5 queue_op 0x30d96c00 seq 3797 osr(1.70c 0x2ed00f0) 861 bytes (queue has 46 ops and 1091060 bytes) 2014-06-09 16:22:11.999335 7f868b759700 10 op_submit_finish 3797 2014-06-09 16:22:11.999343 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999338 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.999350 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.999357 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3753.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999362 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999357 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.999366 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.999372 7f86a5610700 10 _do_op 0x324f4180 seq 3753 r = 0, finisher 0x31435180 0 2014-06-09 16:22:11.999375 7f86a5610700 10 _finish_op 0x324f4180 seq 3753 osr(2.620 0x2ed1cb0)/0x2ed1cb0 2014-06-09 16:22:11.999378 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f9e0 (1 active) 2014-06-09 16:22:11.999374 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.999380 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496b40 (2 active) 2014-06-09 16:22:11.999382 7f86a5610700 5 _do_op 0x32503320 seq 3754 osr(2.69c 0x2ed1f20)/0x2ed1f20 start 2014-06-09 16:22:11.999385 7f86a5610700 10 _do_transaction on 0xc103200 2014-06-09 16:22:11.999382 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.999390 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999394 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:11.999403 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.999411 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.999419 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.999426 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999423 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:11.999435 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:11.999440 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3754.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999444 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999443 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:11.999450 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:11.999457 7f86a5610700 10 _do_op 0x32503320 seq 3754 r = 0, finisher 0x3061fa20 0 2014-06-09 16:22:11.999460 7f86a5610700 10 _finish_op 0x32503320 seq 3754 osr(2.69c 0x2ed1f20)/0x2ed1f20 2014-06-09 16:22:11.999463 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496b40 (1 active) 2014-06-09 16:22:11.999458 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:11.999465 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287560 (2 active) 2014-06-09 16:22:11.999467 7f86a5610700 5 _do_op 0x324f4240 seq 3755 osr(0.dd 0x2bd6550)/0x2bd6550 start 2014-06-09 16:22:11.999470 7f86a5610700 10 _do_transaction on 0x34f1980 2014-06-09 16:22:11.999466 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:11.999475 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999478 7f868b759700 15 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:11.999487 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:11.999495 7f868b759700 20 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:11.999503 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.999509 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999520 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3755.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999524 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999530 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f3fc0 (19 active) 2014-06-09 16:22:11.999534 7f86a5610700 10 _do_op 0x324f4240 seq 3755 r = 0, finisher 0x31413940 0 2014-06-09 16:22:11.999538 7f86a5610700 10 _finish_op 0x324f4240 seq 3755 osr(0.dd 0x2bd6550)/0x2bd6550 2014-06-09 16:22:11.999541 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287560 (1 active) 2014-06-09 16:22:11.999543 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300480 (2 active) 2014-06-09 16:22:11.999545 7f86a5610700 5 _do_op 0x325031a0 seq 3756 osr(3.7fc 0xb7f00e0)/0xb7f00e0 start 2014-06-09 16:22:11.999548 7f86a5610700 10 _do_transaction on 0xbf43680 2014-06-09 16:22:11.999553 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999568 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f3fc0 (20 active) 2014-06-09 16:22:11.999572 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2fd( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:11.999588 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999601 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3756.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999603 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999606 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.999610 7f868b759700 5 queue_transactions existing osr(3.2fd 0x3ed0de0)/0x3ed0de0 2014-06-09 16:22:11.999611 7f86a5610700 10 _do_op 0x325031a0 seq 3756 r = 0, finisher 0x4eb82e0 0 2014-06-09 16:22:11.999651 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x325350a0,0xcdb2500,0x275ceb40,0xcd326e0,0x275ceaa0,0x32481dc0,0x31434f80,0x4eb92a0,0x31413fc0] 2014-06-09 16:22:11.999664 7f86a5610700 10 _finish_op 0x325031a0 seq 3756 osr(3.7fc 0xb7f00e0)/0xb7f00e0 2014-06-09 16:22:11.999657 7f868ef60700 10 op_submit_start 3798 2014-06-09 16:22:11.999667 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300480 (1 active) 2014-06-09 16:22:11.999669 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383ed80 (2 active) 2014-06-09 16:22:11.999670 7f868ef60700 5 queue_transactions (trailing journal) 3798 2014-06-09 16:22:11.999671 7f86a5610700 5 _do_op 0x324f4300 seq 3757 osr(3.19 0x3ed03a0)/0x3ed03a0 start 2014-06-09 16:22:11.999674 7f86a5610700 10 _do_transaction on 0xc33d440 2014-06-09 16:22:11.999673 7f868ef60700 5 queue_op 0x32502d20 seq 3798 osr(2.34d 0x2ed0e70) 828 bytes (queue has 48 ops and 1092716 bytes) 2014-06-09 16:22:11.999676 7f868ef60700 10 op_submit_finish 3798 2014-06-09 16:22:11.999678 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999678 7f868ff62700 10 op_submit_start 3799 2014-06-09 16:22:11.999679 7f868df5e700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999681 7f868ff62700 5 queue_transactions (trailing journal) 3799 2014-06-09 16:22:11.999669 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'565 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 18 2014-06-09 16:22:11.999681 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.3f0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.999707 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999714 7f868ef60700 10 log is not dirty 2014-06-09 16:22:11.999682 7f868ff62700 5 queue_op 0x30a2780 seq 3799 osr(3.37a 0x3ed1040) 828 bytes (queue has 50 ops and 1094597 bytes) 2014-06-09 16:22:11.999717 7f868ff62700 10 op_submit_finish 3799 2014-06-09 16:22:11.999718 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3757.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999720 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999720 7f868ef60700 5 queue_transactions existing osr(1.3f0 0x384d400)/0x384d400 2014-06-09 16:22:11.999722 7f868ef60700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999726 7f868a757700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999731 7f8691765700 10 op_submit_start 3800 2014-06-09 16:22:11.999733 7f8691765700 5 queue_transactions (trailing journal) 3800 2014-06-09 16:22:11.999722 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.999734 7f8691765700 5 queue_op 0xc34afc0 seq 3800 osr(0.48b 0x2bd7450) 828 bytes (queue has 50 ops and 1094597 bytes) 2014-06-09 16:22:11.999736 7f86a5610700 10 _do_op 0x324f4300 seq 3757 r = 0, finisher 0x31413cc0 0 2014-06-09 16:22:11.999738 7f8691765700 10 op_submit_finish 3800 2014-06-09 16:22:11.999739 7f86a5610700 10 _finish_op 0x324f4300 seq 3757 osr(3.19 0x3ed03a0)/0x3ed03a0 2014-06-09 16:22:11.999744 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383ed80 (1 active) 2014-06-09 16:22:11.999746 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652480 (2 active) 2014-06-09 16:22:11.999737 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.999747 7f86a5610700 5 _do_op 0x32503020 seq 3758 osr(1.289 0x384ceb0)/0x384ceb0 start 2014-06-09 16:22:11.999747 7f868df5e700 10 op_submit_start 3801 2014-06-09 16:22:11.999749 7f86a5610700 10 _do_transaction on 0xc2326c0 2014-06-09 16:22:11.999749 7f868df5e700 5 queue_transactions (trailing journal) 3801 2014-06-09 16:22:11.999752 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999750 7f868df5e700 5 queue_op 0xbf7e4e0 seq 3801 osr(1.15b 0x384cbf0) 828 bytes (queue has 50 ops and 1094599 bytes) 2014-06-09 16:22:11.999754 7f868df5e700 10 op_submit_finish 3801 2014-06-09 16:22:11.999747 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.999743 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.66a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:11.999755 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.999757 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.562( v 214'607 lc 214'606 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:11.999768 7f868ff62700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.999775 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999777 7f8691765700 10 log is not dirty 2014-06-09 16:22:11.999772 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:11.999782 7f868b759700 10 op_submit_start 3802 2014-06-09 16:22:11.999784 7f868b759700 5 queue_transactions (trailing journal) 3802 2014-06-09 16:22:11.999785 7f868b759700 5 queue_op 0xc2ea9c0 seq 3802 osr(3.2fd 0x3ed0de0) 828 bytes (queue has 50 ops and 1094599 bytes) 2014-06-09 16:22:11.999784 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3758.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999788 7f868b759700 10 op_submit_finish 3802 2014-06-09 16:22:11.999789 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999782 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.999786 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:155 2014-06-09 16:22:11.999791 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.999798 7f86a5610700 10 _do_op 0x32503020 seq 3758 r = 0, finisher 0x307f19a0 0 2014-06-09 16:22:11.999802 7f86a5610700 10 _finish_op 0x32503020 seq 3758 osr(1.289 0x384ceb0)/0x384ceb0 2014-06-09 16:22:11.999792 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.3c0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.3c0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:11.999804 7f868df5e700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:11.999805 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652480 (1 active) 2014-06-09 16:22:11.999807 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359440 (2 active) 2014-06-09 16:22:11.999809 7f86a5610700 5 _do_op 0xbf7f8c0 seq 3759 osr(2.654 0x2ed1fd0)/0x2ed1fd0 start 2014-06-09 16:22:11.999812 7f86a5610700 10 _do_transaction on 0xc05f500 2014-06-09 16:22:11.999800 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.999817 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999820 7f8692f68700 10 op_submit_start 3803 2014-06-09 16:22:11.999821 7f8692f68700 5 queue_transactions (trailing journal) 3803 2014-06-09 16:22:11.999824 7f868b759700 10 log is not dirty 2014-06-09 16:22:11.999823 7f8692f68700 5 queue_op 0x324e6300 seq 3803 osr(3.6f 0x3ed0280) 826 bytes (queue has 50 ops and 1094597 bytes) 2014-06-09 16:22:11.999826 7f8692f68700 10 op_submit_finish 3803 2014-06-09 16:22:11.999828 7f868b759700 5 queue_transactions existing osr(2.3c0 0x2ed13f0)/0x2ed13f0 2014-06-09 16:22:11.999821 7f868df5e700 15 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1059 2014-06-09 16:22:11.999830 7f868b759700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999836 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.999837 7f8691765700 5 queue_transactions existing osr(0.66a 0x2bd7ef0)/0x2bd7ef0 2014-06-09 16:22:11.999830 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.6b0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:11.999840 7f8691765700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999846 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.999847 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.999840 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:11.999851 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999852 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:11.999861 7f8692f68700 10 log is not dirty 2014-06-09 16:22:11.999863 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:11.999864 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:11.999864 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3759.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999865 7f8692f68700 5 queue_transactions existing osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 2014-06-09 16:22:11.999866 7f869af78700 20 osd.26 243 _dispatch 0x313888c0 pg_info(1 pgs e243:0.179) v4 2014-06-09 16:22:11.999860 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:11.999867 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999868 7f8692f68700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999872 7f868bf5a700 10 op_submit_start 3804 2014-06-09 16:22:11.999873 7f868bf5a700 5 queue_transactions (trailing journal) 3804 2014-06-09 16:22:11.999868 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:11.999875 7f86a5610700 10 _do_op 0xbf7f8c0 seq 3759 r = 0, finisher 0x3061fd00 0 2014-06-09 16:22:11.999875 7f868bf5a700 5 queue_op 0x159969c0 seq 3804 osr(0.644 0x2bd7bf0) 828 bytes (queue has 50 ops and 1094597 bytes) 2014-06-09 16:22:11.999877 7f86a5610700 10 _finish_op 0xbf7f8c0 seq 3759 osr(2.654 0x2ed1fd0)/0x2ed1fd0 2014-06-09 16:22:11.999879 7f868df5e700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:11.999881 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359440 (1 active) 2014-06-09 16:22:11.999883 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c05a0 (2 active) 2014-06-09 16:22:11.999886 7f86a5610700 5 _do_op 0xc2d5500 seq 3760 osr(1.48e 0x384d9b0)/0x384d9b0 start 2014-06-09 16:22:11.999887 7f86a5610700 10 _do_transaction on 0xc3f2e40 2014-06-09 16:22:11.999887 7f8689755700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:11.999891 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999883 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.206( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:11.999893 7f868d75d700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999897 7f8691f66700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999893 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:151 2014-06-09 16:22:11.999901 7f868c75b700 2 waiting 51 > 50 ops || 1555747 > 104857600 2014-06-09 16:22:11.999906 7f868af58700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999906 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.999909 7f8692767700 2 waiting 51 > 50 ops || 1095423 > 104857600 2014-06-09 16:22:11.999911 7f868ff62700 10 log is not dirty 2014-06-09 16:22:11.999914 7f868e75f700 2 waiting 51 > 50 ops || 1095423 > 104857600 2014-06-09 16:22:11.999915 7f868ff62700 5 queue_transactions existing osr(1.206 0x384cd10)/0x384cd10 2014-06-09 16:22:11.999917 7f868ef60700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999911 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:11.999921 7f868a757700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999924 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999925 7f8691765700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999930 7f8692f68700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999932 7f868ff62700 2 waiting 51 > 50 ops || 1095425 > 104857600 2014-06-09 16:22:11.999934 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3760.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999937 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999937 7f868bf5a700 10 op_submit_finish 3804 2014-06-09 16:22:11.999942 7f868df5e700 10 log is not dirty 2014-06-09 16:22:11.999943 7f86a5610700 10 _do_op 0xc2d5500 seq 3760 r = 0, finisher 0x3205d8e0 0 2014-06-09 16:22:11.999946 7f86a5610700 10 _finish_op 0xc2d5500 seq 3760 osr(1.48e 0x384d9b0)/0x384d9b0 2014-06-09 16:22:11.999946 7f868df5e700 5 queue_transactions existing osr(3.579 0x3ed18d0)/0x3ed18d0 2014-06-09 16:22:11.999942 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.67c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:11.999952 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c05a0 (1 active) 2014-06-09 16:22:11.999952 7f868cf5c700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:11.999953 7f8689755700 10 op_submit_start 3805 2014-06-09 16:22:11.999954 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300c60 (2 active) 2014-06-09 16:22:11.999955 7f8689755700 5 queue_transactions (trailing journal) 3805 2014-06-09 16:22:11.999956 7f86a5610700 5 _do_op 0xc2d45a0 seq 3761 osr(0.616 0x2bd7950)/0x2bd7950 start 2014-06-09 16:22:11.999957 7f86a5610700 10 _do_transaction on 0x4734e40 2014-06-09 16:22:11.999957 7f8690f64700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:11.999957 7f8689755700 5 queue_op 0x30d97560 seq 3805 osr(2.28b 0x2ed0cd0) 861 bytes (queue has 50 ops and 1094630 bytes) 2014-06-09 16:22:11.999960 7f8689755700 10 op_submit_finish 3805 2014-06-09 16:22:11.999960 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:11.999962 7f868af58700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:11.999966 7f8692767700 2 waiting 51 > 50 ops || 1095456 > 104857600 2014-06-09 16:22:11.999970 7f868e75f700 2 waiting 51 > 50 ops || 1095456 > 104857600 2014-06-09 16:22:11.999974 7f868ef60700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:11.999965 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 243/201 242/242/200) 2014-06-09 16:22:11.999978 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:11.999978 7f868a757700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:11.999982 7f8691765700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:11.999983 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:11.999983 7f868bf5a700 5 queue_transactions existing osr(2.67c 0x2ed1f90)/0x2ed1f90 2014-06-09 16:22:11.999985 7f868bf5a700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:11.999990 7f868f761700 10 op_submit_start 3806 2014-06-09 16:22:11.999992 7f868f761700 5 queue_transactions (trailing journal) 3806 2014-06-09 16:22:11.999992 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3761.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:11.999995 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:11.999993 7f868f761700 5 queue_op 0xc2d5f80 seq 3806 osr(2.5b6 0x2ed1b90) 828 bytes (queue has 50 ops and 1094630 bytes) 2014-06-09 16:22:11.999997 7f868f761700 10 op_submit_finish 3806 2014-06-09 16:22:11.999999 7f8689755700 10 log is not dirty 2014-06-09 16:22:11.999856 7f86a460e700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:11.999685, event: op_commit, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.000003 7f8689755700 5 queue_transactions existing osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 2014-06-09 16:22:12.000006 7f8689755700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:12.000010 7f868d75d700 2 waiting 51 > 50 ops || 1095458 > 104857600 2014-06-09 16:22:12.000010 7f86a5610700 10 _do_op 0xc2d45a0 seq 3761 r = 0, finisher 0x17000a60 0 2014-06-09 16:22:12.000013 7f86a5610700 10 _finish_op 0xc2d45a0 seq 3761 osr(0.616 0x2bd7950)/0x2bd7950 2014-06-09 16:22:12.000004 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'565 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 18 all committed 2014-06-09 16:22:12.000002 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.2a5( v 214'635 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) 2014-06-09 16:22:12.000020 7f868cf5c700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000020 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300c60 (1 active) 2014-06-09 16:22:12.000021 7f868c75b700 10 op_submit_start 3807 2014-06-09 16:22:12.000023 7f868c75b700 5 queue_transactions (trailing journal) 3807 2014-06-09 16:22:12.000023 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3ad80 (2 active) 2014-06-09 16:22:12.000025 7f8692f68700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000025 7f86a5610700 5 _do_op 0x308ca420 seq 3762 osr(3.1ff 0x3ed0940)/0x3ed0940 start 2014-06-09 16:22:12.000025 7f868c75b700 5 queue_op 0x1e08af00 seq 3807 osr(3.224 0x3ed0c80) 461150 bytes (queue has 50 ops and 1554952 bytes) 2014-06-09 16:22:12.000028 7f86a5610700 10 _do_transaction on 0xbf36b40 2014-06-09 16:22:12.000028 7f868c75b700 10 op_submit_finish 3807 2014-06-09 16:22:12.000029 7f8692767700 2 waiting 51 > 50 ops || 1555778 > 104857600 2014-06-09 16:22:12.000021 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.9 activated and committed 2014-06-09 16:22:12.000033 7f868e75f700 2 waiting 51 > 50 ops || 1555778 > 104857600 2014-06-09 16:22:12.000034 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.000037 7f868af58700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000041 7f868a757700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000017 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] eval_repop repgather(0xc144fc0 243'566 rep_tid=18 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.000034 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 213/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.000044 7f868ef60700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000049 7f8691765700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000033 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc139080 243'740 rep_tid=22 committed?=0 applied?=0 lock=0 op=osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.000045 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.000044 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] log_op_stats osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4 inb 16768 outb 0 rlat 0.000000 lat 1.292948 2014-06-09 16:22:12.000057 7f868c75b700 10 osd.26 243 dequeue_op 0x4679b00 finish 2014-06-09 16:22:12.000063 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.000064 7f868f761700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.000068 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.000063 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] publish_stats_to_osd 243:916 2014-06-09 16:22:12.000070 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.000085 7f86a5610700 15 _omap_rmkeyrange meta/1fe13b76/pglog_3.1ff/0//-1 [0000000214.00000000000000000681,4294967295.18446744073709551615] 2014-06-09 16:22:12.000074 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] sending commit on repgather(0xc144fc0 243'566 rep_tid=18 committed?=1 applied?=0 lock=0 op=osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4) 0xdfb0f00 2014-06-09 16:22:12.000088 7f86a5610700 15 get_omap_iterator meta/1fe13b76/pglog_3.1ff/0//-1 2014-06-09 16:22:12.000076 7f868c75b700 10 osd.26 243 dequeue_op 0x21967000 prio 63 cost 1081344 latency 1.376260 osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.000082 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:996 2014-06-09 16:22:12.000100 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.000099 7f868c75b700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.000094, event: reached_pg, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.000105 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.6e7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/211) 2014-06-09 16:22:12.000110 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.000124 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x21967000 2014-06-09 16:22:12.000140 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.000136 7f86a460e700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:12.000104, event: commit_sent, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.000144 7f868f761700 5 queue_transactions existing osr(0.6e7 0x384c1b0)/0x384c1b0 2014-06-09 16:22:12.000134 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.000147 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x325350a0,0xcdb2500,0x275ceb40,0xcd326e0,0x275ceaa0,0x32481dc0,0x31434f80,0x4eb92a0,0x31413fc0] 2014-06-09 16:22:12.000150 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509560,0x31413100,0x13964080,0xc5a6140,0xbcc89a0,0xcd32ee0] 2014-06-09 16:22:12.000153 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509560,0x31413100,0x13964080,0xc5a6140,0xbcc89a0,0xcd32ee0] 2014-06-09 16:22:12.000155 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.000156 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.000150 7f868f761700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000156 7f868c75b700 15 getattr 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 '_' 2014-06-09 16:22:12.000161 7f8689f56700 10 op_submit_start 3808 2014-06-09 16:22:12.000163 7f8689f56700 5 queue_transactions (trailing journal) 3808 2014-06-09 16:22:12.000162 7f869af78700 5 -- op tracker -- , seq: 3452, time: 2014-06-09 16:22:11.963329, event: header_read, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.000164 7f8689f56700 5 queue_op 0xd04ed20 seq 3808 osr(3.76 0x3ed0240) 826 bytes (queue has 50 ops and 1554952 bytes) 2014-06-09 16:22:12.000168 7f8689f56700 10 op_submit_finish 3808 2014-06-09 16:22:12.000170 7f869af78700 5 -- op tracker -- , seq: 3452, time: 2014-06-09 16:22:11.963330, event: throttled, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.000173 7f8690763700 10 op_submit_start 3809 2014-06-09 16:22:12.000175 7f8690763700 5 queue_transactions (trailing journal) 3809 2014-06-09 16:22:12.000177 7f8690763700 5 queue_op 0x2e21bc0 seq 3809 osr(3.1c7 0x3ed0a10) 1053 bytes (queue has 50 ops and 1554952 bytes) 2014-06-09 16:22:12.000176 7f869af78700 5 -- op tracker -- , seq: 3452, time: 2014-06-09 16:22:11.963337, event: all_read, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.000180 7f8690763700 10 op_submit_finish 3809 2014-06-09 16:22:12.000183 7f869af78700 5 -- op tracker -- , seq: 3452, time: 2014-06-09 16:22:11.999862, event: dispatched, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.000174 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.000190 7f869af78700 5 -- op tracker -- , seq: 3452, time: 2014-06-09 16:22:12.000188, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:11.999659 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8820,0x3205ce40,0x32534c20,0x2e0f320,0xc5a6bc0,0x275cffc0,0x12051e80,0x31435180,0x3061fa20,0x31413940] 2014-06-09 16:22:12.000185 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:12.000195 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.179) v4 from osd.2 2014-06-09 16:22:12.000198 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x313888c0 2014-06-09 16:22:12.000205 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.000203 7f869af78700 5 -- op tracker -- , seq: 3452, time: 2014-06-09 16:22:12.000201, event: started, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.000200 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.000209 7f8689f56700 5 queue_transactions existing osr(3.7a 0x3ed0220)/0x3ed0220 2014-06-09 16:22:12.000199 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] op_applied: 18 2014-06-09 16:22:12.000212 7f8689f56700 2 waiting 51 > 50 ops || 1555778 > 104857600 2014-06-09 16:22:12.000216 7f868bf5a700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000211 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.000218 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:12.000221 7f868ff62700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000219 7f86a4e0f700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:12.000213, event: op_applied, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.000226 7f8691f66700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000219 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.000221 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 but can_create 2014-06-09 16:22:12.000231 7f869af78700 5 -- op tracker -- , seq: 3452, time: 2014-06-09 16:22:12.000217, event: done, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.000234 7f8690763700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.000235 7f868c75b700 15 getattr 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 'snapset' 2014-06-09 16:22:12.000228 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] op_applied on primary on version 243'566 2014-06-09 16:22:12.000240 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.000242 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.000239 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] repop_all_applied: repop tid 18 all applied 2014-06-09 16:22:12.000248 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] eval_repop repgather(0xc144fc0 243'566 rep_tid=18 committed?=1 applied?=1 lock=0 op=osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.000255 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1022 2014-06-09 16:22:12.000265 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:12.000267 7f868c75b700 15 getattr 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/snapdir//3 'snapset' 2014-06-09 16:22:12.000272 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.000263 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] log_op_stats osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4 inb 16768 outb 0 rlat 0.000000 lat 1.293167 2014-06-09 16:22:12.000279 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.000281 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.000282 7f869af78700 20 osd.26 243 _dispatch 0x31388e00 pg_info(1 pgs e243:0.57b) v4 2014-06-09 16:22:12.000277 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.5b3( v 213'676 (0'0,213'676] local-les=243 n=127 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:12.000279 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] publish_stats_to_osd 243:917 2014-06-09 16:22:12.000287 7f869af78700 5 -- op tracker -- , seq: 3453, time: 2014-06-09 16:22:11.960426, event: header_read, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.000296 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:12.000295 7f869af78700 5 -- op tracker -- , seq: 3453, time: 2014-06-09 16:22:11.960426, event: throttled, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.000290 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.000301 7f869af78700 5 -- op tracker -- , seq: 3453, time: 2014-06-09 16:22:11.960439, event: all_read, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.000306 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.000307 7f869af78700 5 -- op tracker -- , seq: 3453, time: 2014-06-09 16:22:12.000279, event: dispatched, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.000310 7f8690763700 5 queue_transactions existing osr(3.5b3 0x3ed17e0)/0x3ed17e0 2014-06-09 16:22:12.000312 7f8690763700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000300 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 213'565 active] removing repgather(0xc144fc0 243'566 rep_tid=18 committed?=1 applied?=1 op=osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.000304 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x2c42b00 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.000316 7f868df5e700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000314 7f869af78700 5 -- op tracker -- , seq: 3453, time: 2014-06-09 16:22:12.000312, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.000318 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.57b) v4 from osd.4 2014-06-09 16:22:12.000321 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31388e00 2014-06-09 16:22:12.000317 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.000314 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 213'565 active] q front is repgather(0xc144fc0 243'566 rep_tid=18 committed?=1 applied?=1 op=osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.000326 7f869af78700 5 -- op tracker -- , seq: 3453, time: 2014-06-09 16:22:12.000324, event: started, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.000328 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.000328 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 213'565 active] remove_repop repgather(0xc144fc0 243'566 rep_tid=18 committed?=1 applied?=1 op=osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.000341 7f869af78700 5 -- op tracker -- , seq: 3453, time: 2014-06-09 16:22:12.000339, event: done, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.000349 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.000342 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.3f5( v 243'566 (0'0,243'566] local-les=243 n=107 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'565 mlcod 213'565 active] requeue_ops 2014-06-09 16:22:12.000350 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.000338 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x2c42b00 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 rwstate(none n=0 w=0) oi: bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd5180 snapset: 0=[]:[] 2014-06-09 16:22:12.000362 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.000363 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.000364 7f869af78700 20 osd.26 243 _dispatch 0x2176cfc0 pg_info(1 pgs e243:3.4c4) v4 2014-06-09 16:22:12.000354 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 @head oi=bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.000362 7f8683620700 5 -- op tracker -- , seq: 3454, time: 2014-06-09 16:22:12.000334, event: header_read, op: osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.000372 7f86a5610700 15 _omap_rmkeys meta/1fe13b76/pglog_3.1ff/0//-1 2014-06-09 16:22:12.000373 7f868c75b700 15 getattr 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/snapdir//3 '_' 2014-06-09 16:22:12.000373 7f8683620700 5 -- op tracker -- , seq: 3454, time: 2014-06-09 16:22:12.000335, event: throttled, op: osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.000380 7f8690f64700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000381 7f8683620700 5 -- op tracker -- , seq: 3454, time: 2014-06-09 16:22:12.000343, event: all_read, op: osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.000390 7f8683620700 5 -- op tracker -- , seq: 3454, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.000396 7f8683620700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:12.000400 7f8683620700 20 osd.26 243 should_share_map osd.6 172.16.96.201:6849/52045 243 2014-06-09 16:22:12.000401 7f869af78700 5 -- op tracker -- , seq: 3455, time: 2014-06-09 16:22:11.972509, event: header_read, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.000406 7f8683620700 15 osd.26 243 enqueue_op 0x257c5300 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:12.000409 7f8689755700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000409 7f869af78700 5 -- op tracker -- , seq: 3455, time: 2014-06-09 16:22:11.972510, event: throttled, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.000413 7f868d75d700 2 waiting 51 > 50 ops || 1555780 > 104857600 2014-06-09 16:22:12.000415 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe13b76/pglog_3.1ff/0//-1 2014-06-09 16:22:12.000416 7f869af78700 5 -- op tracker -- , seq: 3455, time: 2014-06-09 16:22:11.972524, event: all_read, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.000422 7f869af78700 5 -- op tracker -- , seq: 3455, time: 2014-06-09 16:22:12.000361, event: dispatched, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.000425 7f86a5610700 10 _omap_rmkeys meta/1fe13b76/pglog_3.1ff/0//-1 r = 0 2014-06-09 16:22:12.000428 7f868b759700 10 op_submit_start 3810 2014-06-09 16:22:12.000429 7f868b759700 5 queue_transactions (trailing journal) 3810 2014-06-09 16:22:12.000429 7f86a5610700 15 _omap_rmkeys meta/1fe13b76/pglog_3.1ff/0//-1 2014-06-09 16:22:12.000428 7f869af78700 5 -- op tracker -- , seq: 3455, time: 2014-06-09 16:22:12.000427, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.000431 7f868b759700 5 queue_op 0xbf7f380 seq 3810 osr(2.3c0 0x2ed13f0) 828 bytes (queue has 50 ops and 1554952 bytes) 2014-06-09 16:22:12.000433 7f868b759700 10 op_submit_finish 3810 2014-06-09 16:22:12.000433 7f86a5610700 10 _omap_rmkeys meta/1fe13b76/pglog_3.1ff/0//-1 r = 0 2014-06-09 16:22:12.000433 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c4) v4 from osd.3 2014-06-09 16:22:12.000436 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2176cfc0 2014-06-09 16:22:12.000441 7f86a5610700 15 _omap_setkeys meta/1fe13b76/pglog_3.1ff/0//-1 2014-06-09 16:22:12.000441 7f869af78700 5 -- op tracker -- , seq: 3455, time: 2014-06-09 16:22:12.000439, event: started, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.000436 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.000446 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:12.000456 7f86a5610700 10 cid: oid: 1fe13b76/pglog_3.1ff/0//-1 skipping op, spos 3762.0.0 <= header.spos 150421.0.0 2014-06-09 16:22:12.000448 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid bd747be6/rbd_data.11786b8b4567.0000000000001732/snapdir//3 and !can_create 2014-06-09 16:22:12.000457 7f869af78700 5 -- op tracker -- , seq: 3455, time: 2014-06-09 16:22:12.000455, event: done, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.000461 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3762.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.000464 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.000464 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.000468 7f868b759700 5 queue_transactions existing osr(2.f4 0x2ed0790)/0x2ed0790 2014-06-09 16:22:12.000470 7f868b759700 2 waiting 51 > 50 ops || 1555778 > 104857600 2014-06-09 16:22:12.000468 7f868c75b700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.000462, event: started, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.000482 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.000485 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.000477 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30640c00 2014-06-09 16:22:12.000488 7f86a5610700 10 _do_op 0x308ca420 seq 3762 r = 0, finisher 0x327f2740 0 2014-06-09 16:22:12.000494 7f86a5610700 10 _finish_op 0x308ca420 seq 3762 osr(3.1ff 0x3ed0940)/0x3ed0940 2014-06-09 16:22:12.000495 7f868c75b700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.000490, event: started, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.000499 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3ad80 (1 active) 2014-06-09 16:22:12.000501 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1c20 (2 active) 2014-06-09 16:22:12.000501 7f868cf5c700 10 op_submit_start 3811 2014-06-09 16:22:12.000504 7f868cf5c700 5 queue_transactions (trailing journal) 3811 2014-06-09 16:22:12.000504 7f86a5610700 5 _do_op 0x324e6600 seq 3763 osr(1.697 0x384de10)/0x384de10 start 2014-06-09 16:22:12.000506 7f86a5610700 10 _do_transaction on 0xc3269c0 2014-06-09 16:22:12.000505 7f868cf5c700 5 queue_op 0x230b8120 seq 3811 osr(2.6ec 0x2ed1e10) 828 bytes (queue has 50 ops and 1553837 bytes) 2014-06-09 16:22:12.000508 7f868cf5c700 10 op_submit_finish 3811 2014-06-09 16:22:12.000511 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.000501 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] ov 0'0 av 243'624 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.000514 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.000516 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.000517 7f869af78700 20 osd.26 243 _dispatch 0x280c5180 pg_info(1 pgs e243:3.113) v4 2014-06-09 16:22:12.000512 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.5b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:12.000516 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 2014-06-09 16:22:12.000523 7f869af78700 5 -- op tracker -- , seq: 3456, time: 2014-06-09 16:22:11.972149, event: header_read, op: pg_info(1 pgs e243:3.113) 2014-06-09 16:22:12.000531 7f869af78700 5 -- op tracker -- , seq: 3456, time: 2014-06-09 16:22:11.972150, event: throttled, op: pg_info(1 pgs e243:3.113) 2014-06-09 16:22:12.000528 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.000537 7f869af78700 5 -- op tracker -- , seq: 3456, time: 2014-06-09 16:22:11.972163, event: all_read, op: pg_info(1 pgs e243:3.113) 2014-06-09 16:22:12.000542 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.000543 7f869af78700 5 -- op tracker -- , seq: 3456, time: 2014-06-09 16:22:12.000514, event: dispatched, op: pg_info(1 pgs e243:3.113) 2014-06-09 16:22:12.000546 7f868cf5c700 5 queue_transactions existing osr(0.5b7 0x2bd7a90)/0x2bd7a90 2014-06-09 16:22:12.000548 7f868cf5c700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000549 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.000543 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~1081344 2014-06-09 16:22:12.000552 7f8692f68700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000550 7f869af78700 5 -- op tracker -- , seq: 3456, time: 2014-06-09 16:22:12.000548, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.113) 2014-06-09 16:22:12.000556 7f8692767700 2 waiting 51 > 50 ops || 1554663 > 104857600 2014-06-09 16:22:12.000555 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.113) v4 from osd.9 2014-06-09 16:22:12.000558 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c5180 2014-06-09 16:22:12.000559 7f868e75f700 2 waiting 51 > 50 ops || 1554663 > 104857600 2014-06-09 16:22:12.000563 7f868af58700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000563 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3763.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.000566 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.000563 7f869af78700 5 -- op tracker -- , seq: 3456, time: 2014-06-09 16:22:12.000562, event: started, op: pg_info(1 pgs e243:3.113) 2014-06-09 16:22:12.000566 7f868a757700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000559 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228600 2014-06-09 16:22:12.000570 7f868ef60700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000573 7f8691765700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000578 7f868f761700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000579 7f86a5610700 10 _do_op 0x324e6600 seq 3763 r = 0, finisher 0x31435ea0 0 2014-06-09 16:22:12.000571 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 snapset=0x30dd51b8 snapc=0=[] 2014-06-09 16:22:12.000582 7f86a5610700 10 _finish_op 0x324e6600 seq 3763 osr(1.697 0x384de10)/0x384de10 2014-06-09 16:22:12.000584 7f8689f56700 2 waiting 51 > 50 ops || 1554663 > 104857600 2014-06-09 16:22:12.000587 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x327f3100,0xbd5f560] 2014-06-09 16:22:12.000588 7f868bf5a700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000590 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x327f3100,0xbd5f560] 2014-06-09 16:22:12.000582 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.000591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.000592 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.000593 7f868ff62700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000598 7f8691f66700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000593 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.000602 7f8690763700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000607 7f868df5e700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000604 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 0x30640c00 op modify 2014-06-09 16:22:12.000618 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.000617 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:09.539606 2014-06-09 16:22:12.000630 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3747.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.000632 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.000640 7f86a5e11700 10 _do_op 0xcabd320 seq 3747 r = 0, finisher 0xd9fa3a0 0 2014-06-09 16:22:12.000637 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=121 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.000647 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x120503c0] 2014-06-09 16:22:12.000580 7f869af78700 5 -- op tracker -- , seq: 3456, time: 2014-06-09 16:22:12.000578, event: done, op: pg_info(1 pgs e243:3.113) 2014-06-09 16:22:12.000650 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x120503c0] 2014-06-09 16:22:12.000652 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.000653 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.000659 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.000662 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.000655 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.000670 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.000671 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.000672 7f869af78700 20 osd.26 243 _dispatch 0x12c09a40 pg_info(1 pgs e243:0.298) v4 2014-06-09 16:22:12.000675 7f869af78700 5 -- op tracker -- , seq: 3457, time: 2014-06-09 16:22:11.967410, event: header_read, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:12.000666 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 23 on osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.000680 7f869af78700 5 -- op tracker -- , seq: 3457, time: 2014-06-09 16:22:11.967410, event: throttled, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:12.000684 7f869af78700 5 -- op tracker -- , seq: 3457, time: 2014-06-09 16:22:11.967421, event: all_read, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:12.000688 7f869af78700 5 -- op tracker -- , seq: 3457, time: 2014-06-09 16:22:12.000670, event: dispatched, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:12.000681 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 23 o bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.000692 7f869af78700 5 -- op tracker -- , seq: 3457, time: 2014-06-09 16:22:12.000690, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:12.000703 7f8690f64700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000704 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.298) v4 from osd.8 2014-06-09 16:22:12.000707 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12c09a40 2014-06-09 16:22:12.000708 7f8689755700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000713 7f868d75d700 2 waiting 51 > 50 ops || 1554665 > 104857600 2014-06-09 16:22:12.000711 7f86a4e0f700 5 -- op tracker -- , seq: 604, time: 2014-06-09 16:22:12.000362, event: done, op: osd_op(client.4571.0:36397 rbd_data.117b6b8b4567.0000000000001843 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.5b1b53f5 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.000718 7f868b759700 2 waiting 51 > 50 ops || 1554663 > 104857600 2014-06-09 16:22:12.000730 7f868cf5c700 10 op_submit_start 3812 2014-06-09 16:22:12.000732 7f868cf5c700 5 queue_transactions (trailing journal) 3812 2014-06-09 16:22:12.000730 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1c20 (1 active) 2014-06-09 16:22:12.000733 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0b40 (2 active) 2014-06-09 16:22:12.000733 7f868cf5c700 5 queue_op 0x230b8600 seq 3812 osr(0.5b7 0x2bd7a90) 828 bytes (queue has 50 ops and 1553837 bytes) 2014-06-09 16:22:12.000736 7f868cf5c700 10 op_submit_finish 3812 2014-06-09 16:22:12.000737 7f86a5610700 5 _do_op 0xc64b0e0 seq 3764 osr(0.ef 0x2bd64f0)/0x2bd64f0 start 2014-06-09 16:22:12.000740 7f86a5610700 10 _do_transaction on 0xbfa4480 2014-06-09 16:22:12.000745 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.000738 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=212 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:12.000752 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=212 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info from osd.8 3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) 2014-06-09 16:22:12.000764 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=212 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] exit Started/Stray 1.545744 7 0.000178 2014-06-09 16:22:12.000779 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.000775 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=212 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive 2014-06-09 16:22:12.000783 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8820,0x3205ce40,0x32534c20,0x2e0f320,0xc5a6bc0,0x275cffc0,0x12051e80,0x31435180,0x3061fa20,0x31413940] 2014-06-09 16:22:12.000787 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x4eb82e0,0x31413cc0,0x307f19a0,0x3061fd00,0x3205d8e0,0x17000a60,0x327f2740,0x31435ea0] 2014-06-09 16:22:12.000786 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=212 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:12.000795 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3764.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.000799 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.000798 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=212 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: In ReplicaActive, about to call activate 2014-06-09 16:22:12.000808 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] activate - no missing, moving last_complete 213'634 -> 213'634 2014-06-09 16:22:12.000801 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] _activate_committed 243 peer_activated now 26 last_epoch_started 212 same_interval_since 242 2014-06-09 16:22:12.000817 7f86a5610700 10 _do_op 0xc64b0e0 seq 3764 r = 0, finisher 0xbd67760 0 2014-06-09 16:22:12.000823 7f86a5610700 10 _finish_op 0xc64b0e0 seq 3764 osr(0.ef 0x2bd64f0)/0x2bd64f0 2014-06-09 16:22:12.000826 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0b40 (1 active) 2014-06-09 16:22:12.000827 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653440 (2 active) 2014-06-09 16:22:12.000821 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:12.000829 7f86a5610700 5 _do_op 0x30d97b00 seq 3765 osr(1.113 0x384c910)/0x384c910 start 2014-06-09 16:22:12.000832 7f86a5610700 10 _do_transaction on 0x37e5440 2014-06-09 16:22:12.000836 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.000836 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] flushed 2014-06-09 16:22:12.000845 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:12.000855 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x4eb82e0,0x31413cc0,0x307f19a0,0x3061fd00,0x3205d8e0,0x17000a60,0x327f2740,0x31435ea0] 2014-06-09 16:22:12.000859 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd67760] 2014-06-09 16:22:12.000864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd67760] 2014-06-09 16:22:12.000865 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.000866 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.000866 7f86af56b700 5 -- op tracker -- , seq: 3458, time: 2014-06-09 16:22:11.998454, event: header_read, op: osd_op(client.4541.0:32085 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 2789376~1032192] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.000870 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.000871 7f86a5e11700 10 _finish_op 0xcabd320 seq 3747 osr(0.55d 0x2bd77f0)/0x2bd77f0 2014-06-09 16:22:12.000876 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653b00 (1 active) 2014-06-09 16:22:12.000879 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff58c0 (2 active) 2014-06-09 16:22:12.000877 7f86af56b700 5 -- op tracker -- , seq: 3458, time: 2014-06-09 16:22:11.998455, event: throttled, op: osd_op(client.4541.0:32085 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 2789376~1032192] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.000881 7f86a5e11700 5 _do_op 0xd04f080 seq 3766 osr(3.77 0x3ed0250)/0x3ed0250 start 2014-06-09 16:22:12.000884 7f86a5e11700 10 _do_transaction on 0xc326d80 2014-06-09 16:22:12.000886 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.000884 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3765.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.000888 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.000886 7f86af56b700 5 -- op tracker -- , seq: 3458, time: 2014-06-09 16:22:12.000832, event: all_read, op: osd_op(client.4541.0:32085 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 2789376~1032192] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.000888 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.000890 7f868cf5c700 5 queue_transactions existing osr(3.292 0x3ed0ba0)/0x3ed0ba0 2014-06-09 16:22:12.000893 7f868cf5c700 10 op_submit_start 3813 2014-06-09 16:22:12.000894 7f868cf5c700 5 queue_transactions (trailing journal) 3813 2014-06-09 16:22:12.000895 7f868cf5c700 5 queue_op 0x100388a0 seq 3813 osr(3.292 0x3ed0ba0) 1335 bytes (queue has 49 ops and 1553486 bytes) 2014-06-09 16:22:12.000895 7f86af56b700 5 -- op tracker -- , seq: 3458, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32085 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 2789376~1032192] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.000897 7f868cf5c700 10 op_submit_finish 3813 2014-06-09 16:22:12.000898 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5b4e0] 2014-06-09 16:22:12.000900 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.000900 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) 2014-06-09 16:22:12.000901 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.000911 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.000912 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.6 activated and committed 2014-06-09 16:22:12.000918 7f86af56b700 15 osd.26 243 enqueue_op 0xdfa6400 prio 63 cost 1032192 latency 0.002464 osd_op(client.4541.0:32085 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 2789376~1032192] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.000920 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.000921 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.000927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xd9fa3a0] 2014-06-09 16:22:12.000931 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5b4e0] 2014-06-09 16:22:12.000933 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.000934 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.000928 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.000936 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xd9fa3a0] 2014-06-09 16:22:12.000938 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.000939 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.000940 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.000942 7f86a5610700 10 _do_op 0x30d97b00 seq 3765 r = 0, finisher 0x3050a480 0 2014-06-09 16:22:12.000947 7f86a5610700 10 _finish_op 0x30d97b00 seq 3765 osr(1.113 0x384c910)/0x384c910 2014-06-09 16:22:12.000951 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653440 (1 active) 2014-06-09 16:22:12.000953 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34965a0 (2 active) 2014-06-09 16:22:12.000955 7f86a5610700 5 _do_op 0xd326ba0 seq 3767 osr(3.f6 0x3ed04b0)/0x3ed04b0 start 2014-06-09 16:22:12.000957 7f86a5610700 10 _do_transaction on 0xc0d72c0 2014-06-09 16:22:12.000962 7f86a5610700 15 _omap_setkeys meta/516e98b6/pglog_3.f6/0//-1 2014-06-09 16:22:12.000956 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:134 2014-06-09 16:22:12.000967 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd66fe0] 2014-06-09 16:22:12.000970 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.000971 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd66fe0] 2014-06-09 16:22:12.000973 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.000974 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.000973 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.000974 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.57a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:12.000986 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3766.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.000991 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.001003 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.001004 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a480] 2014-06-09 16:22:12.001007 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a480] 2014-06-09 16:22:12.001007 7f868cf5c700 5 queue_transactions existing osr(2.57a 0x2ed1870)/0x2ed1870 2014-06-09 16:22:12.001009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.001010 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.001010 7f868cf5c700 10 op_submit_start 3814 2014-06-09 16:22:12.001012 7f868cf5c700 5 queue_transactions (trailing journal) 3814 2014-06-09 16:22:12.001013 7f868cf5c700 5 queue_op 0x2c06f740 seq 3814 osr(2.57a 0x2ed1870) 828 bytes (queue has 49 ops and 1553486 bytes) 2014-06-09 16:22:12.001015 7f868cf5c700 10 op_submit_finish 3814 2014-06-09 16:22:12.001018 7f86a5610700 10 lookup_strip_header done cid meta oid 516e98b6/pglog_3.f6/0//-1 2014-06-09 16:22:12.001020 7f86a5e11700 10 _do_op 0xd04f080 seq 3766 r = 0, finisher 0xbd6d960 0 2014-06-09 16:22:12.001026 7f86a5e11700 10 _finish_op 0xd04f080 seq 3766 osr(3.77 0x3ed0250)/0x3ed0250 2014-06-09 16:22:12.001018 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:12.001030 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff58c0 (1 active) 2014-06-09 16:22:12.001032 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd440 (2 active) 2014-06-09 16:22:12.001033 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.001034 7f86a5e11700 5 _do_op 0xc2d5740 seq 3768 osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 start 2014-06-09 16:22:12.001030 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.001037 7f86a5e11700 10 _do_transaction on 0xc2ef2c0 2014-06-09 16:22:12.001042 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.001038 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.001047 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc93e0] 2014-06-09 16:22:12.001049 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc93e0] 2014-06-09 16:22:12.001050 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.001045 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.001052 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.001055 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6d960] 2014-06-09 16:22:12.001057 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6d960] 2014-06-09 16:22:12.001056 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.001058 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.001059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.001070 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.001070 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:193 2014-06-09 16:22:12.001083 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.001085 7f86a5610700 15 _omap_rmkeys meta/516e98b6/pglog_3.f6/0//-1 2014-06-09 16:22:12.001091 7f86a5610700 10 _omap_rmkeys meta/516e98b6/pglog_3.f6/0//-1 r = 0 2014-06-09 16:22:12.001087 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:12.001097 7f86a5610700 15 _omap_setkeys meta/516e98b6/pglog_3.f6/0//-1 2014-06-09 16:22:12.001099 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.001109 7f86a5610700 15 _touch 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:12.001107 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.001114 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.001117 7f869af78700 5 -- op tracker -- , seq: 3457, time: 2014-06-09 16:22:12.000711, event: started, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:12.001125 7f868cf5c700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.001135 7f868c75b700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.000700, event: waiting for subops from 7, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.001137 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1053 2014-06-09 16:22:12.001153 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.001156 7f868c75b700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:12.001156 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.16c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.16c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.001171 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.f6_head oid 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:12.001181 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.001185 7f868cf5c700 5 queue_transactions existing osr(0.16c 0x2bd67f0)/0x2bd67f0 2014-06-09 16:22:12.001187 7f868cf5c700 10 op_submit_start 3815 2014-06-09 16:22:12.001188 7f868cf5c700 5 queue_transactions (trailing journal) 3815 2014-06-09 16:22:12.001189 7f868cf5c700 5 queue_op 0xcabdf80 seq 3815 osr(0.16c 0x2bd67f0) 828 bytes (queue has 49 ops and 1553488 bytes) 2014-06-09 16:22:12.001192 7f868cf5c700 10 op_submit_finish 3815 2014-06-09 16:22:12.001192 7f86a5610700 20 write_state seq is 30159 2014-06-09 16:22:12.001175 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 213'623 (0'0,213'623] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'623], crt=0'0) [243'624 (0'0) modify bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 by client.4568.0:35343 2014-06-09 10:20:09.539606] 2014-06-09 16:22:12.001197 7f86a5610700 20 set_header setting 30158 cid 3.f6_head oid 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 parent seq 0 2014-06-09 16:22:12.001194 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) 2014-06-09 16:22:12.001205 7f86a5610700 10 _touch 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 = 0 2014-06-09 16:22:12.001206 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.3 activated and committed 2014-06-09 16:22:12.001211 7f86a5610700 15 _write 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 0~1032192 2014-06-09 16:22:12.001199 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=213'623 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'624 (0'0) modify bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 by client.4568.0:35343 2014-06-09 10:20:09.539606 2014-06-09 16:22:12.001214 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.001224 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:12.001220 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.001220 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=213'623 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.001230 7f868cf5c700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.001244 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:110 2014-06-09 16:22:12.001249 7f868c75b700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'624, trimmed: 2014-06-09 16:22:12.001257 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.001261 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.530( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.001269 7f868c75b700 5 queue_transactions existing osr(3.3e6 0x3ed12a0)/0x3ed12a0 2014-06-09 16:22:12.001272 7f868c75b700 10 op_submit_start 3816 2014-06-09 16:22:12.001273 7f868c75b700 5 queue_transactions (trailing journal) 3816 2014-06-09 16:22:12.001274 7f868c75b700 5 queue_op 0x100389c0 seq 3816 osr(3.3e6 0x3ed12a0) 1083742 bytes (queue has 50 ops and 2637230 bytes) 2014-06-09 16:22:12.001277 7f868c75b700 10 op_submit_finish 3816 2014-06-09 16:22:12.001287 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.001290 7f868cf5c700 5 queue_transactions existing osr(2.530 0x2ed19b0)/0x2ed19b0 2014-06-09 16:22:12.001293 7f868cf5c700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001279 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=213'623 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc1f2300 243'624 rep_tid=23 committed?=0 applied?=0 lock=0 op=osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.001296 7f8692f68700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001296 7f868c75b700 10 osd.26 243 dequeue_op 0x21967000 finish 2014-06-09 16:22:12.001300 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.001301 7f8692767700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.001305 7f868e75f700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.001309 7f868af58700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001312 7f868a757700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001331 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41050 (20 active) 2014-06-09 16:22:12.001333 7f868ef60700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001342 7f868f761700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001335 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.3ab( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.001349 7f8689f56700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.001353 7f868bf5a700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001357 7f868ff62700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001362 7f8691f66700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001366 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.001367 7f8690763700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001371 7f868c75b700 5 queue_transactions existing osr(0.3ab 0x2bd72f0)/0x2bd72f0 2014-06-09 16:22:12.001372 7f868df5e700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001373 7f868c75b700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001378 7f8690f64700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.001412 7f86af56b700 5 -- op tracker -- , seq: 3459, time: 2014-06-09 16:22:12.000938, event: header_read, op: osd_op(client.4541.0:32086 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 3821568~372736] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.001423 7f86af56b700 5 -- op tracker -- , seq: 3459, time: 2014-06-09 16:22:12.000939, event: throttled, op: osd_op(client.4541.0:32086 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 3821568~372736] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.001435 7f86af56b700 5 -- op tracker -- , seq: 3459, time: 2014-06-09 16:22:12.001379, event: all_read, op: osd_op(client.4541.0:32086 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 3821568~372736] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.001446 7f86af56b700 5 -- op tracker -- , seq: 3459, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32086 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 3821568~372736] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.001459 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.001475 7f86af56b700 15 osd.26 243 enqueue_op 0xdfa6f00 prio 63 cost 372736 latency 0.000537 osd_op(client.4541.0:32086 rbd_data.115d6b8b4567.000000000000162d [set-alloc-hint object_size 4194304 write_size 4194304,write 3821568~372736] 3.b845587f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.001709 7f86af56b700 5 -- op tracker -- , seq: 3460, time: 2014-06-09 16:22:12.001498, event: header_read, op: osd_op(client.4541.0:32198 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~135168] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.001721 7f86af56b700 5 -- op tracker -- , seq: 3460, time: 2014-06-09 16:22:12.001499, event: throttled, op: osd_op(client.4541.0:32198 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~135168] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.001731 7f86af56b700 5 -- op tracker -- , seq: 3460, time: 2014-06-09 16:22:12.001661, event: all_read, op: osd_op(client.4541.0:32198 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~135168] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.001741 7f86af56b700 5 -- op tracker -- , seq: 3460, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32198 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~135168] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.001747 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.001762 7f86af56b700 15 osd.26 243 enqueue_op 0xdfa7b00 prio 63 cost 135168 latency 0.000264 osd_op(client.4541.0:32198 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~135168] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.002411 7f8689755700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.002419 7f868d75d700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.002427 7f868b759700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.002435 7f8691765700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.002580 7f86af56b700 5 -- op tracker -- , seq: 3461, time: 2014-06-09 16:22:12.001782, event: header_read, op: osd_op(client.4541.0:32199 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 135168~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.002592 7f86af56b700 5 -- op tracker -- , seq: 3461, time: 2014-06-09 16:22:12.001783, event: throttled, op: osd_op(client.4541.0:32199 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 135168~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.002603 7f86af56b700 5 -- op tracker -- , seq: 3461, time: 2014-06-09 16:22:12.002521, event: all_read, op: osd_op(client.4541.0:32199 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 135168~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.002613 7f86af56b700 5 -- op tracker -- , seq: 3461, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32199 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 135168~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.002620 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.002644 7f86af56b700 15 osd.26 243 enqueue_op 0x257c5400 prio 63 cost 516096 latency 0.000862 osd_op(client.4541.0:32199 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 135168~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.003350 7f86af56b700 5 -- op tracker -- , seq: 3462, time: 2014-06-09 16:22:12.002666, event: header_read, op: osd_op(client.4541.0:32200 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 651264~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.003366 7f86af56b700 5 -- op tracker -- , seq: 3462, time: 2014-06-09 16:22:12.002668, event: throttled, op: osd_op(client.4541.0:32200 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 651264~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.003378 7f86af56b700 5 -- op tracker -- , seq: 3462, time: 2014-06-09 16:22:12.003280, event: all_read, op: osd_op(client.4541.0:32200 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 651264~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.003397 7f86af56b700 5 -- op tracker -- , seq: 3462, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32200 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 651264~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.003405 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.003409 7f869af78700 5 -- op tracker -- , seq: 3457, time: 2014-06-09 16:22:12.001136, event: done, op: pg_info(1 pgs e243:0.298) 2014-06-09 16:22:12.003423 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7d00 prio 63 cost 516096 latency 0.000757 osd_op(client.4541.0:32200 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 651264~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.004458 7f86a5610700 10 _generic_write 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 0~1032192 = 0 2014-06-09 16:22:12.005834 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.005837 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.005855 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.005857 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.005858 7f869af78700 20 osd.26 243 _dispatch 0x76e2000 pg_info(1 pgs e243:0.648) v4 2014-06-09 16:22:12.005863 7f869af78700 5 -- op tracker -- , seq: 3463, time: 2014-06-09 16:22:11.969304, event: header_read, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:12.005871 7f869af78700 5 -- op tracker -- , seq: 3463, time: 2014-06-09 16:22:11.969305, event: throttled, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:12.005877 7f869af78700 5 -- op tracker -- , seq: 3463, time: 2014-06-09 16:22:11.969321, event: all_read, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:12.005882 7f869af78700 5 -- op tracker -- , seq: 3463, time: 2014-06-09 16:22:12.005854, event: dispatched, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:12.005886 7f86a5610700 15 _setattrs 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:12.005889 7f869af78700 5 -- op tracker -- , seq: 3463, time: 2014-06-09 16:22:12.005887, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:12.005901 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.648) v4 from osd.5 2014-06-09 16:22:12.005903 7f86a5610700 10 _setattrs 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 = 0 2014-06-09 16:22:12.005905 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e2000 2014-06-09 16:22:12.005908 7f86a5610700 15 _setattrs 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 2014-06-09 16:22:12.005911 7f869af78700 5 -- op tracker -- , seq: 3463, time: 2014-06-09 16:22:12.005910, event: started, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:12.005916 7f86a5610700 10 _setattrs 3.f6_head/942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 = 0 2014-06-09 16:22:12.005932 7f869af78700 5 -- op tracker -- , seq: 3463, time: 2014-06-09 16:22:12.005930, event: done, op: pg_info(1 pgs e243:0.648) 2014-06-09 16:22:12.005940 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.005942 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.005954 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.005955 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.005956 7f869af78700 20 osd.26 243 _dispatch 0x76e28c0 pg_info(1 pgs e243:3.105) v4 2014-06-09 16:22:12.005960 7f869af78700 5 -- op tracker -- , seq: 3464, time: 2014-06-09 16:22:11.963342, event: header_read, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.005966 7f869af78700 5 -- op tracker -- , seq: 3464, time: 2014-06-09 16:22:11.963342, event: throttled, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.005972 7f869af78700 5 -- op tracker -- , seq: 3464, time: 2014-06-09 16:22:11.963351, event: all_read, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.005982 7f869af78700 5 -- op tracker -- , seq: 3464, time: 2014-06-09 16:22:12.005953, event: dispatched, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.005989 7f869af78700 5 -- op tracker -- , seq: 3464, time: 2014-06-09 16:22:12.005987, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.006019 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.105) v4 from osd.2 2014-06-09 16:22:12.006024 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e28c0 2014-06-09 16:22:12.006030 7f869af78700 5 -- op tracker -- , seq: 3464, time: 2014-06-09 16:22:12.006028, event: started, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.006046 7f869af78700 5 -- op tracker -- , seq: 3464, time: 2014-06-09 16:22:12.006045, event: done, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.006054 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006056 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006069 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006070 7f869af78700 20 osd.26 243 _dispatch 0x30de5dc0 pg_info(1 pgs e243:0.512) v4 2014-06-09 16:22:12.006074 7f869af78700 5 -- op tracker -- , seq: 3465, time: 2014-06-09 16:22:11.960443, event: header_read, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.006080 7f869af78700 5 -- op tracker -- , seq: 3465, time: 2014-06-09 16:22:11.960443, event: throttled, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.006085 7f869af78700 5 -- op tracker -- , seq: 3465, time: 2014-06-09 16:22:11.960453, event: all_read, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.006090 7f869af78700 5 -- op tracker -- , seq: 3465, time: 2014-06-09 16:22:12.006068, event: dispatched, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.006090 7f86a5610700 10 cid: oid: 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 not skipping op, *spos 3767.0.0 2014-06-09 16:22:12.006095 7f86a5610700 10 > header.spos 0.0.0 2014-06-09 16:22:12.006097 7f869af78700 5 -- op tracker -- , seq: 3465, time: 2014-06-09 16:22:12.006095, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.006100 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.512) v4 from osd.4 2014-06-09 16:22:12.006103 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5dc0 2014-06-09 16:22:12.006107 7f869af78700 5 -- op tracker -- , seq: 3465, time: 2014-06-09 16:22:12.006106, event: started, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.006121 7f869af78700 5 -- op tracker -- , seq: 3465, time: 2014-06-09 16:22:12.006119, event: done, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.006129 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006128 7f86a5610700 20 set_header setting 30158 cid 3.f6_head oid 942188f6/rbd_data.117f6b8b4567.000000000000153d/head//3 parent seq 0 2014-06-09 16:22:12.006131 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006145 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006146 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006147 7f869af78700 20 osd.26 243 _dispatch 0x30a19500 pg_info(1 pgs e243:0.1cd) v4 2014-06-09 16:22:12.006152 7f869af78700 5 -- op tracker -- , seq: 3466, time: 2014-06-09 16:22:11.972593, event: header_read, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:12.006158 7f869af78700 5 -- op tracker -- , seq: 3466, time: 2014-06-09 16:22:11.972594, event: throttled, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:12.006163 7f869af78700 5 -- op tracker -- , seq: 3466, time: 2014-06-09 16:22:11.972609, event: all_read, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:12.006168 7f869af78700 5 -- op tracker -- , seq: 3466, time: 2014-06-09 16:22:12.006145, event: dispatched, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:12.006173 7f869af78700 5 -- op tracker -- , seq: 3466, time: 2014-06-09 16:22:12.006172, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:12.006177 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1cd) v4 from osd.3 2014-06-09 16:22:12.006179 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30a19500 2014-06-09 16:22:12.006183 7f869af78700 5 -- op tracker -- , seq: 3466, time: 2014-06-09 16:22:12.006182, event: started, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:12.006196 7f869af78700 5 -- op tracker -- , seq: 3466, time: 2014-06-09 16:22:12.006194, event: done, op: pg_info(1 pgs e243:0.1cd) 2014-06-09 16:22:12.006202 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006204 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006215 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006216 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006217 7f869af78700 20 osd.26 243 _dispatch 0x3411ddc0 pg_info(1 pgs e243:2.1e0) v4 2014-06-09 16:22:12.006221 7f869af78700 5 -- op tracker -- , seq: 3467, time: 2014-06-09 16:22:11.972213, event: header_read, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:12.006224 7f86a5610700 10 cid: oid: 516e98b6/pglog_3.f6/0//-1 skipping op, spos 3767.0.0 <= header.spos 149506.0.0 2014-06-09 16:22:12.006227 7f869af78700 5 -- op tracker -- , seq: 3467, time: 2014-06-09 16:22:11.972214, event: throttled, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:12.006231 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3767.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.006232 7f869af78700 5 -- op tracker -- , seq: 3467, time: 2014-06-09 16:22:11.972229, event: all_read, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:12.006235 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.006238 7f869af78700 5 -- op tracker -- , seq: 3467, time: 2014-06-09 16:22:12.006215, event: dispatched, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:12.006245 7f869af78700 5 -- op tracker -- , seq: 3467, time: 2014-06-09 16:22:12.006243, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:12.006249 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.1e0) v4 from osd.9 2014-06-09 16:22:12.006251 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3411ddc0 2014-06-09 16:22:12.006256 7f869af78700 5 -- op tracker -- , seq: 3467, time: 2014-06-09 16:22:12.006254, event: started, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:12.006269 7f869af78700 5 -- op tracker -- , seq: 3467, time: 2014-06-09 16:22:12.006267, event: done, op: pg_info(1 pgs e243:2.1e0) 2014-06-09 16:22:12.006276 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006277 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006288 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006289 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006290 7f869af78700 20 osd.26 243 _dispatch 0x30adafc0 pg_info(1 pgs e243:2.552) v4 2014-06-09 16:22:12.006295 7f869af78700 5 -- op tracker -- , seq: 3468, time: 2014-06-09 16:22:11.967428, event: header_read, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:12.006301 7f869af78700 5 -- op tracker -- , seq: 3468, time: 2014-06-09 16:22:11.967428, event: throttled, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:12.006307 7f869af78700 5 -- op tracker -- , seq: 3468, time: 2014-06-09 16:22:11.967439, event: all_read, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:12.006312 7f869af78700 5 -- op tracker -- , seq: 3468, time: 2014-06-09 16:22:12.006287, event: dispatched, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:12.006317 7f869af78700 5 -- op tracker -- , seq: 3468, time: 2014-06-09 16:22:12.006315, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:12.006321 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.552) v4 from osd.8 2014-06-09 16:22:12.006323 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30adafc0 2014-06-09 16:22:12.006327 7f869af78700 5 -- op tracker -- , seq: 3468, time: 2014-06-09 16:22:12.006325, event: started, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:12.006340 7f869af78700 5 -- op tracker -- , seq: 3468, time: 2014-06-09 16:22:12.006338, event: done, op: pg_info(1 pgs e243:2.552) 2014-06-09 16:22:12.006346 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006347 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006357 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006358 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006359 7f869af78700 20 osd.26 243 _dispatch 0x76e21c0 pg_info(1 pgs e243:1.286) v4 2014-06-09 16:22:12.006363 7f869af78700 5 -- op tracker -- , seq: 3469, time: 2014-06-09 16:22:11.969328, event: header_read, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006369 7f869af78700 5 -- op tracker -- , seq: 3469, time: 2014-06-09 16:22:11.969329, event: throttled, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006374 7f869af78700 5 -- op tracker -- , seq: 3469, time: 2014-06-09 16:22:11.969339, event: all_read, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006379 7f869af78700 5 -- op tracker -- , seq: 3469, time: 2014-06-09 16:22:12.006357, event: dispatched, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006384 7f869af78700 5 -- op tracker -- , seq: 3469, time: 2014-06-09 16:22:12.006383, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006388 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.286) v4 from osd.5 2014-06-09 16:22:12.006390 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e21c0 2014-06-09 16:22:12.006394 7f869af78700 5 -- op tracker -- , seq: 3469, time: 2014-06-09 16:22:12.006392, event: started, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006406 7f869af78700 5 -- op tracker -- , seq: 3469, time: 2014-06-09 16:22:12.006404, event: done, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006412 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006414 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006423 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006424 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006425 7f869af78700 20 osd.26 243 _dispatch 0x76e2a80 pg_info(1 pgs e243:0.330) v4 2014-06-09 16:22:12.006430 7f869af78700 5 -- op tracker -- , seq: 3470, time: 2014-06-09 16:22:11.963355, event: header_read, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.006435 7f869af78700 5 -- op tracker -- , seq: 3470, time: 2014-06-09 16:22:11.963355, event: throttled, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.006440 7f869af78700 5 -- op tracker -- , seq: 3470, time: 2014-06-09 16:22:11.963361, event: all_read, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.006445 7f869af78700 5 -- op tracker -- , seq: 3470, time: 2014-06-09 16:22:12.006423, event: dispatched, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.006451 7f869af78700 5 -- op tracker -- , seq: 3470, time: 2014-06-09 16:22:12.006449, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.006454 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.330) v4 from osd.2 2014-06-09 16:22:12.006457 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e2a80 2014-06-09 16:22:12.006460 7f869af78700 5 -- op tracker -- , seq: 3470, time: 2014-06-09 16:22:12.006459, event: started, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.006473 7f869af78700 5 -- op tracker -- , seq: 3470, time: 2014-06-09 16:22:12.006471, event: done, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.006479 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006480 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006491 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006493 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006494 7f869af78700 20 osd.26 243 _dispatch 0x31475dc0 pg_info(1 pgs e243:0.33a) v4 2014-06-09 16:22:12.006499 7f869af78700 5 -- op tracker -- , seq: 3471, time: 2014-06-09 16:22:11.960457, event: header_read, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:12.006506 7f869af78700 5 -- op tracker -- , seq: 3471, time: 2014-06-09 16:22:11.960458, event: throttled, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:12.006512 7f869af78700 5 -- op tracker -- , seq: 3471, time: 2014-06-09 16:22:11.960471, event: all_read, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:12.006518 7f869af78700 5 -- op tracker -- , seq: 3471, time: 2014-06-09 16:22:12.006491, event: dispatched, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:12.006527 7f869af78700 5 -- op tracker -- , seq: 3471, time: 2014-06-09 16:22:12.006525, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:12.006533 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.33a) v4 from osd.4 2014-06-09 16:22:12.006536 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31475dc0 2014-06-09 16:22:12.006540 7f86af56b700 5 -- op tracker -- , seq: 3472, time: 2014-06-09 16:22:12.005850, event: header_read, op: osd_op(client.4541.0:32201 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1167360~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.006554 7f86af56b700 5 -- op tracker -- , seq: 3472, time: 2014-06-09 16:22:12.005852, event: throttled, op: osd_op(client.4541.0:32201 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1167360~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.006569 7f86af56b700 5 -- op tracker -- , seq: 3472, time: 2014-06-09 16:22:12.006480, event: all_read, op: osd_op(client.4541.0:32201 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1167360~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.006582 7f86af56b700 5 -- op tracker -- , seq: 3472, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32201 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1167360~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.006589 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.006589 7f869af78700 5 -- op tracker -- , seq: 3471, time: 2014-06-09 16:22:12.006539, event: started, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:12.006612 7f869af78700 5 -- op tracker -- , seq: 3471, time: 2014-06-09 16:22:12.006610, event: done, op: pg_info(1 pgs e243:0.33a) 2014-06-09 16:22:12.006623 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006625 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006637 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006639 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006608 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7400 prio 63 cost 516096 latency 0.000758 osd_op(client.4541.0:32201 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1167360~516096] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.006640 7f869af78700 20 osd.26 243 _dispatch 0x30ef76c0 pg_info(1 pgs e243:1.4bb) v4 2014-06-09 16:22:12.006645 7f869af78700 5 -- op tracker -- , seq: 3473, time: 2014-06-09 16:22:11.972622, event: header_read, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:12.006652 7f869af78700 5 -- op tracker -- , seq: 3473, time: 2014-06-09 16:22:11.972622, event: throttled, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:12.006658 7f869af78700 5 -- op tracker -- , seq: 3473, time: 2014-06-09 16:22:11.972634, event: all_read, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:12.006664 7f869af78700 5 -- op tracker -- , seq: 3473, time: 2014-06-09 16:22:12.006637, event: dispatched, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:12.006669 7f869af78700 5 -- op tracker -- , seq: 3473, time: 2014-06-09 16:22:12.006668, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:12.006671 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4bb) v4 from osd.3 2014-06-09 16:22:12.006673 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef76c0 2014-06-09 16:22:12.006676 7f869af78700 5 -- op tracker -- , seq: 3473, time: 2014-06-09 16:22:12.006675, event: started, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:12.006686 7f869af78700 5 -- op tracker -- , seq: 3473, time: 2014-06-09 16:22:12.006685, event: done, op: pg_info(1 pgs e243:1.4bb) 2014-06-09 16:22:12.006691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006692 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006699 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006700 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006700 7f869af78700 20 osd.26 243 _dispatch 0x30e10380 pg_info(1 pgs e243:3.450) v4 2014-06-09 16:22:12.006703 7f869af78700 5 -- op tracker -- , seq: 3474, time: 2014-06-09 16:22:11.972283, event: header_read, op: pg_info(1 pgs e243:3.450) 2014-06-09 16:22:12.006707 7f869af78700 5 -- op tracker -- , seq: 3474, time: 2014-06-09 16:22:11.972284, event: throttled, op: pg_info(1 pgs e243:3.450) 2014-06-09 16:22:12.006710 7f869af78700 5 -- op tracker -- , seq: 3474, time: 2014-06-09 16:22:11.972296, event: all_read, op: pg_info(1 pgs e243:3.450) 2014-06-09 16:22:12.006713 7f869af78700 5 -- op tracker -- , seq: 3474, time: 2014-06-09 16:22:12.006699, event: dispatched, op: pg_info(1 pgs e243:3.450) 2014-06-09 16:22:12.006717 7f869af78700 5 -- op tracker -- , seq: 3474, time: 2014-06-09 16:22:12.006716, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.450) 2014-06-09 16:22:12.006719 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.450) v4 from osd.9 2014-06-09 16:22:12.006721 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e10380 2014-06-09 16:22:12.006723 7f869af78700 5 -- op tracker -- , seq: 3474, time: 2014-06-09 16:22:12.006722, event: started, op: pg_info(1 pgs e243:3.450) 2014-06-09 16:22:12.006732 7f869af78700 5 -- op tracker -- , seq: 3474, time: 2014-06-09 16:22:12.006731, event: done, op: pg_info(1 pgs e243:3.450) 2014-06-09 16:22:12.006736 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006737 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006748 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006750 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006751 7f869af78700 20 osd.26 243 _dispatch 0x333c3dc0 pg_info(1 pgs e243:0.492) v4 2014-06-09 16:22:12.006755 7f869af78700 5 -- op tracker -- , seq: 3475, time: 2014-06-09 16:22:11.969763, event: header_read, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.006760 7f869af78700 5 -- op tracker -- , seq: 3475, time: 2014-06-09 16:22:11.969763, event: throttled, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.006763 7f869af78700 5 -- op tracker -- , seq: 3475, time: 2014-06-09 16:22:11.969774, event: all_read, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.006766 7f869af78700 5 -- op tracker -- , seq: 3475, time: 2014-06-09 16:22:12.006747, event: dispatched, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.006770 7f869af78700 5 -- op tracker -- , seq: 3475, time: 2014-06-09 16:22:12.006769, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.006772 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.492) v4 from osd.8 2014-06-09 16:22:12.006774 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x333c3dc0 2014-06-09 16:22:12.006777 7f869af78700 5 -- op tracker -- , seq: 3475, time: 2014-06-09 16:22:12.006776, event: started, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.006789 7f869af78700 5 -- op tracker -- , seq: 3475, time: 2014-06-09 16:22:12.006788, event: done, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.006793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006794 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006802 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006802 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006803 7f869af78700 20 osd.26 243 _dispatch 0x76e3500 pg_info(1 pgs e243:1.286) v4 2014-06-09 16:22:12.006805 7f869af78700 5 -- op tracker -- , seq: 3476, time: 2014-06-09 16:22:11.969344, event: header_read, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006809 7f869af78700 5 -- op tracker -- , seq: 3476, time: 2014-06-09 16:22:11.969344, event: throttled, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006812 7f869af78700 5 -- op tracker -- , seq: 3476, time: 2014-06-09 16:22:11.969362, event: all_read, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006816 7f869af78700 5 -- op tracker -- , seq: 3476, time: 2014-06-09 16:22:12.006801, event: dispatched, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006819 7f869af78700 5 -- op tracker -- , seq: 3476, time: 2014-06-09 16:22:12.006818, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006821 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.286) v4 from osd.5 2014-06-09 16:22:12.006823 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e3500 2014-06-09 16:22:12.006825 7f869af78700 5 -- op tracker -- , seq: 3476, time: 2014-06-09 16:22:12.006824, event: started, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006833 7f869af78700 5 -- op tracker -- , seq: 3476, time: 2014-06-09 16:22:12.006832, event: done, op: pg_info(1 pgs e243:1.286) 2014-06-09 16:22:12.006840 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006841 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006847 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006848 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006848 7f869af78700 20 osd.26 243 _dispatch 0x76e2e00 pg_info(1 pgs e243:3.251) v4 2014-06-09 16:22:12.006851 7f869af78700 5 -- op tracker -- , seq: 3477, time: 2014-06-09 16:22:11.963364, event: header_read, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.006855 7f869af78700 5 -- op tracker -- , seq: 3477, time: 2014-06-09 16:22:11.963364, event: throttled, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.006858 7f869af78700 5 -- op tracker -- , seq: 3477, time: 2014-06-09 16:22:11.963371, event: all_read, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.006861 7f869af78700 5 -- op tracker -- , seq: 3477, time: 2014-06-09 16:22:12.006847, event: dispatched, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.006866 7f869af78700 5 -- op tracker -- , seq: 3477, time: 2014-06-09 16:22:12.006864, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.006871 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.251) v4 from osd.2 2014-06-09 16:22:12.006873 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e2e00 2014-06-09 16:22:12.006878 7f869af78700 5 -- op tracker -- , seq: 3477, time: 2014-06-09 16:22:12.006876, event: started, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.006893 7f869af78700 5 -- op tracker -- , seq: 3477, time: 2014-06-09 16:22:12.006891, event: done, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.006900 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006902 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006914 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006916 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006916 7f869af78700 20 osd.26 243 _dispatch 0x31475c00 pg_info(1 pgs e243:2.719) v4 2014-06-09 16:22:12.006921 7f869af78700 5 -- op tracker -- , seq: 3478, time: 2014-06-09 16:22:11.960476, event: header_read, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:12.006928 7f869af78700 5 -- op tracker -- , seq: 3478, time: 2014-06-09 16:22:11.960476, event: throttled, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:12.006934 7f869af78700 5 -- op tracker -- , seq: 3478, time: 2014-06-09 16:22:11.960486, event: all_read, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:12.006940 7f869af78700 5 -- op tracker -- , seq: 3478, time: 2014-06-09 16:22:12.006914, event: dispatched, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:12.006946 7f869af78700 5 -- op tracker -- , seq: 3478, time: 2014-06-09 16:22:12.006944, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:12.006950 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.719) v4 from osd.4 2014-06-09 16:22:12.006953 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31475c00 2014-06-09 16:22:12.006957 7f869af78700 5 -- op tracker -- , seq: 3478, time: 2014-06-09 16:22:12.006956, event: started, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:12.006972 7f869af78700 5 -- op tracker -- , seq: 3478, time: 2014-06-09 16:22:12.006970, event: done, op: pg_info(1 pgs e243:2.719) 2014-06-09 16:22:12.006979 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006980 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006994 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.006996 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.006997 7f869af78700 20 osd.26 243 _dispatch 0x30ef7500 pg_info(1 pgs e243:3.1a0) v4 2014-06-09 16:22:12.007002 7f869af78700 5 -- op tracker -- , seq: 3479, time: 2014-06-09 16:22:11.972648, event: header_read, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:12.007009 7f869af78700 5 -- op tracker -- , seq: 3479, time: 2014-06-09 16:22:11.972648, event: throttled, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:12.007014 7f869af78700 5 -- op tracker -- , seq: 3479, time: 2014-06-09 16:22:11.972659, event: all_read, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:12.007020 7f869af78700 5 -- op tracker -- , seq: 3479, time: 2014-06-09 16:22:12.006994, event: dispatched, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:12.007027 7f869af78700 5 -- op tracker -- , seq: 3479, time: 2014-06-09 16:22:12.007025, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:12.007031 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1a0) v4 from osd.3 2014-06-09 16:22:12.007033 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef7500 2014-06-09 16:22:12.007038 7f869af78700 5 -- op tracker -- , seq: 3479, time: 2014-06-09 16:22:12.007036, event: started, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:12.007053 7f869af78700 5 -- op tracker -- , seq: 3479, time: 2014-06-09 16:22:12.007051, event: done, op: pg_info(1 pgs e243:3.1a0) 2014-06-09 16:22:12.007060 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007062 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007073 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007074 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007075 7f869af78700 20 osd.26 243 _dispatch 0x76e3c00 pg_info(1 pgs e243:0.4bd) v4 2014-06-09 16:22:12.007080 7f869af78700 5 -- op tracker -- , seq: 3480, time: 2014-06-09 16:22:11.971099, event: header_read, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.007086 7f869af78700 5 -- op tracker -- , seq: 3480, time: 2014-06-09 16:22:11.971099, event: throttled, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.007092 7f869af78700 5 -- op tracker -- , seq: 3480, time: 2014-06-09 16:22:11.971108, event: all_read, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.007098 7f869af78700 5 -- op tracker -- , seq: 3480, time: 2014-06-09 16:22:12.007072, event: dispatched, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.007104 7f869af78700 5 -- op tracker -- , seq: 3480, time: 2014-06-09 16:22:12.007102, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.007108 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4bd) v4 from osd.8 2014-06-09 16:22:12.007111 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e3c00 2014-06-09 16:22:12.007115 7f869af78700 5 -- op tracker -- , seq: 3480, time: 2014-06-09 16:22:12.007114, event: started, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.007129 7f869af78700 5 -- op tracker -- , seq: 3480, time: 2014-06-09 16:22:12.007128, event: done, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.007136 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007137 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007149 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007150 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007151 7f869af78700 20 osd.26 243 _dispatch 0x3343ddc0 pg_info(1 pgs e243:3.9f) v4 2014-06-09 16:22:12.007156 7f869af78700 5 -- op tracker -- , seq: 3481, time: 2014-06-09 16:22:11.969367, event: header_read, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007163 7f869af78700 5 -- op tracker -- , seq: 3481, time: 2014-06-09 16:22:11.969368, event: throttled, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007169 7f869af78700 5 -- op tracker -- , seq: 3481, time: 2014-06-09 16:22:11.969378, event: all_read, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007175 7f869af78700 5 -- op tracker -- , seq: 3481, time: 2014-06-09 16:22:12.007148, event: dispatched, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007181 7f869af78700 5 -- op tracker -- , seq: 3481, time: 2014-06-09 16:22:12.007179, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007185 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.9f) v4 from osd.5 2014-06-09 16:22:12.007188 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343ddc0 2014-06-09 16:22:12.007192 7f869af78700 5 -- op tracker -- , seq: 3481, time: 2014-06-09 16:22:12.007190, event: started, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007207 7f869af78700 5 -- op tracker -- , seq: 3481, time: 2014-06-09 16:22:12.007205, event: done, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007215 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007217 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007229 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007230 7f869af78700 20 osd.26 243 _dispatch 0x76e3340 pg_info(1 pgs e243:1.465) v4 2014-06-09 16:22:12.007236 7f869af78700 5 -- op tracker -- , seq: 3482, time: 2014-06-09 16:22:11.963376, event: header_read, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.007242 7f869af78700 5 -- op tracker -- , seq: 3482, time: 2014-06-09 16:22:11.963376, event: throttled, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.007248 7f869af78700 5 -- op tracker -- , seq: 3482, time: 2014-06-09 16:22:11.963382, event: all_read, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.007254 7f869af78700 5 -- op tracker -- , seq: 3482, time: 2014-06-09 16:22:12.007227, event: dispatched, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.007261 7f869af78700 5 -- op tracker -- , seq: 3482, time: 2014-06-09 16:22:12.007259, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.007265 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.465) v4 from osd.2 2014-06-09 16:22:12.007267 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x76e3340 2014-06-09 16:22:12.007272 7f869af78700 5 -- op tracker -- , seq: 3482, time: 2014-06-09 16:22:12.007270, event: started, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.007286 7f869af78700 5 -- op tracker -- , seq: 3482, time: 2014-06-09 16:22:12.007283, event: done, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.007292 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007294 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007305 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007306 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007307 7f869af78700 20 osd.26 243 _dispatch 0x31475a40 pg_info(1 pgs e243:0.57b) v4 2014-06-09 16:22:12.007312 7f869af78700 5 -- op tracker -- , seq: 3483, time: 2014-06-09 16:22:11.960490, event: header_read, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.007318 7f869af78700 5 -- op tracker -- , seq: 3483, time: 2014-06-09 16:22:11.960490, event: throttled, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.007324 7f869af78700 5 -- op tracker -- , seq: 3483, time: 2014-06-09 16:22:11.960499, event: all_read, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.007330 7f869af78700 5 -- op tracker -- , seq: 3483, time: 2014-06-09 16:22:12.007304, event: dispatched, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.007336 7f869af78700 5 -- op tracker -- , seq: 3483, time: 2014-06-09 16:22:12.007334, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.007341 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.57b) v4 from osd.4 2014-06-09 16:22:12.007343 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31475a40 2014-06-09 16:22:12.007348 7f869af78700 5 -- op tracker -- , seq: 3483, time: 2014-06-09 16:22:12.007346, event: started, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.007364 7f869af78700 5 -- op tracker -- , seq: 3483, time: 2014-06-09 16:22:12.007362, event: done, op: pg_info(1 pgs e243:0.57b) 2014-06-09 16:22:12.007370 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007372 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007382 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007384 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007385 7f869af78700 20 osd.26 243 _dispatch 0x1c03ec40 pg_info(1 pgs e243:3.4c9) v4 2014-06-09 16:22:12.007389 7f869af78700 5 -- op tracker -- , seq: 3484, time: 2014-06-09 16:22:11.972668, event: header_read, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:12.007396 7f869af78700 5 -- op tracker -- , seq: 3484, time: 2014-06-09 16:22:11.972668, event: throttled, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:12.007402 7f869af78700 5 -- op tracker -- , seq: 3484, time: 2014-06-09 16:22:11.972686, event: all_read, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:12.007408 7f869af78700 5 -- op tracker -- , seq: 3484, time: 2014-06-09 16:22:12.007382, event: dispatched, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:12.007414 7f869af78700 5 -- op tracker -- , seq: 3484, time: 2014-06-09 16:22:12.007412, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:12.007419 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c9) v4 from osd.3 2014-06-09 16:22:12.007422 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1c03ec40 2014-06-09 16:22:12.007426 7f869af78700 5 -- op tracker -- , seq: 3484, time: 2014-06-09 16:22:12.007424, event: started, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:12.007441 7f869af78700 5 -- op tracker -- , seq: 3484, time: 2014-06-09 16:22:12.007439, event: done, op: pg_info(1 pgs e243:3.4c9) 2014-06-09 16:22:12.007448 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007450 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007460 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007462 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007463 7f869af78700 20 osd.26 243 _dispatch 0x30fcc1c0 pg_info(1 pgs e243:2.566) v4 2014-06-09 16:22:12.007467 7f869af78700 5 -- op tracker -- , seq: 3485, time: 2014-06-09 16:22:11.971153, event: header_read, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.007474 7f869af78700 5 -- op tracker -- , seq: 3485, time: 2014-06-09 16:22:11.971153, event: throttled, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.007480 7f869af78700 5 -- op tracker -- , seq: 3485, time: 2014-06-09 16:22:11.971161, event: all_read, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.007486 7f869af78700 5 -- op tracker -- , seq: 3485, time: 2014-06-09 16:22:12.007460, event: dispatched, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.007492 7f869af78700 5 -- op tracker -- , seq: 3485, time: 2014-06-09 16:22:12.007490, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.007496 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.566) v4 from osd.8 2014-06-09 16:22:12.007499 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcc1c0 2014-06-09 16:22:12.007503 7f869af78700 5 -- op tracker -- , seq: 3485, time: 2014-06-09 16:22:12.007501, event: started, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.007517 7f869af78700 5 -- op tracker -- , seq: 3485, time: 2014-06-09 16:22:12.007515, event: done, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.007524 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007526 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007537 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007539 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007539 7f869af78700 20 osd.26 243 _dispatch 0x3343dc00 pg_info(1 pgs e243:3.9f) v4 2014-06-09 16:22:12.007544 7f869af78700 5 -- op tracker -- , seq: 3486, time: 2014-06-09 16:22:11.969383, event: header_read, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007550 7f869af78700 5 -- op tracker -- , seq: 3486, time: 2014-06-09 16:22:11.969383, event: throttled, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007557 7f869af78700 5 -- op tracker -- , seq: 3486, time: 2014-06-09 16:22:11.969393, event: all_read, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007563 7f869af78700 5 -- op tracker -- , seq: 3486, time: 2014-06-09 16:22:12.007537, event: dispatched, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007570 7f869af78700 5 -- op tracker -- , seq: 3486, time: 2014-06-09 16:22:12.007568, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007574 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.9f) v4 from osd.5 2014-06-09 16:22:12.007576 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343dc00 2014-06-09 16:22:12.007581 7f869af78700 5 -- op tracker -- , seq: 3486, time: 2014-06-09 16:22:12.007579, event: started, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007595 7f869af78700 5 -- op tracker -- , seq: 3486, time: 2014-06-09 16:22:12.007594, event: done, op: pg_info(1 pgs e243:3.9f) 2014-06-09 16:22:12.007602 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007604 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007621 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007622 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007623 7f869af78700 20 osd.26 243 _dispatch 0x15998380 pg_info(1 pgs e243:0.2c6) v4 2014-06-09 16:22:12.007628 7f869af78700 5 -- op tracker -- , seq: 3487, time: 2014-06-09 16:22:11.963386, event: header_read, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:12.007635 7f869af78700 5 -- op tracker -- , seq: 3487, time: 2014-06-09 16:22:11.963386, event: throttled, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:12.007641 7f869af78700 5 -- op tracker -- , seq: 3487, time: 2014-06-09 16:22:11.963392, event: all_read, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:12.007647 7f869af78700 5 -- op tracker -- , seq: 3487, time: 2014-06-09 16:22:12.007620, event: dispatched, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:12.007653 7f869af78700 5 -- op tracker -- , seq: 3487, time: 2014-06-09 16:22:12.007651, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:12.007658 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.2c6) v4 from osd.2 2014-06-09 16:22:12.007660 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x15998380 2014-06-09 16:22:12.007665 7f869af78700 5 -- op tracker -- , seq: 3487, time: 2014-06-09 16:22:12.007663, event: started, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:12.007679 7f869af78700 5 -- op tracker -- , seq: 3487, time: 2014-06-09 16:22:12.007677, event: done, op: pg_info(1 pgs e243:0.2c6) 2014-06-09 16:22:12.007686 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007688 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007698 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007700 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007701 7f869af78700 20 osd.26 243 _dispatch 0x31475880 pg_info(1 pgs e243:0.512) v4 2014-06-09 16:22:12.007705 7f869af78700 5 -- op tracker -- , seq: 3488, time: 2014-06-09 16:22:11.960503, event: header_read, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.007712 7f869af78700 5 -- op tracker -- , seq: 3488, time: 2014-06-09 16:22:11.960504, event: throttled, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.007718 7f869af78700 5 -- op tracker -- , seq: 3488, time: 2014-06-09 16:22:11.960514, event: all_read, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.007724 7f869af78700 5 -- op tracker -- , seq: 3488, time: 2014-06-09 16:22:12.007698, event: dispatched, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.007731 7f869af78700 5 -- op tracker -- , seq: 3488, time: 2014-06-09 16:22:12.007729, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.007735 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.512) v4 from osd.4 2014-06-09 16:22:12.007738 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31475880 2014-06-09 16:22:12.007743 7f869af78700 5 -- op tracker -- , seq: 3488, time: 2014-06-09 16:22:12.007741, event: started, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.007756 7f869af78700 5 -- op tracker -- , seq: 3488, time: 2014-06-09 16:22:12.007755, event: done, op: pg_info(1 pgs e243:0.512) 2014-06-09 16:22:12.007763 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007765 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007777 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007779 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007780 7f869af78700 20 osd.26 243 _dispatch 0x309e6a80 pg_info(1 pgs e243:0.1f9) v4 2014-06-09 16:22:12.007785 7f869af78700 5 -- op tracker -- , seq: 3489, time: 2014-06-09 16:22:11.972696, event: header_read, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:12.007792 7f869af78700 5 -- op tracker -- , seq: 3489, time: 2014-06-09 16:22:11.972696, event: throttled, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:12.007798 7f869af78700 5 -- op tracker -- , seq: 3489, time: 2014-06-09 16:22:11.972709, event: all_read, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:12.007804 7f869af78700 5 -- op tracker -- , seq: 3489, time: 2014-06-09 16:22:12.007777, event: dispatched, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:12.007811 7f869af78700 5 -- op tracker -- , seq: 3489, time: 2014-06-09 16:22:12.007809, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:12.007815 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1f9) v4 from osd.3 2014-06-09 16:22:12.007817 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e6a80 2014-06-09 16:22:12.007822 7f869af78700 5 -- op tracker -- , seq: 3489, time: 2014-06-09 16:22:12.007820, event: started, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:12.007840 7f869af78700 5 -- op tracker -- , seq: 3489, time: 2014-06-09 16:22:12.007838, event: done, op: pg_info(1 pgs e243:0.1f9) 2014-06-09 16:22:12.007847 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007849 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007860 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007862 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007863 7f869af78700 20 osd.26 243 _dispatch 0x30fcce00 pg_info(1 pgs e243:0.492) v4 2014-06-09 16:22:12.007868 7f869af78700 5 -- op tracker -- , seq: 3490, time: 2014-06-09 16:22:11.971210, event: header_read, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.007875 7f869af78700 5 -- op tracker -- , seq: 3490, time: 2014-06-09 16:22:11.971210, event: throttled, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.007880 7f869af78700 5 -- op tracker -- , seq: 3490, time: 2014-06-09 16:22:11.971221, event: all_read, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.007886 7f869af78700 5 -- op tracker -- , seq: 3490, time: 2014-06-09 16:22:12.007860, event: dispatched, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.007892 7f869af78700 5 -- op tracker -- , seq: 3490, time: 2014-06-09 16:22:12.007890, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.007896 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.492) v4 from osd.8 2014-06-09 16:22:12.007899 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcce00 2014-06-09 16:22:12.007904 7f869af78700 5 -- op tracker -- , seq: 3490, time: 2014-06-09 16:22:12.007902, event: started, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.007919 7f869af78700 5 -- op tracker -- , seq: 3490, time: 2014-06-09 16:22:12.007916, event: done, op: pg_info(1 pgs e243:0.492) 2014-06-09 16:22:12.007926 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007928 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007938 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.007939 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.007941 7f869af78700 20 osd.26 243 _dispatch 0x3343da40 pg_info(1 pgs e243:2.5a4) v4 2014-06-09 16:22:12.007945 7f869af78700 5 -- op tracker -- , seq: 3491, time: 2014-06-09 16:22:11.969397, event: header_read, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.007952 7f869af78700 5 -- op tracker -- , seq: 3491, time: 2014-06-09 16:22:11.969397, event: throttled, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.007957 7f869af78700 5 -- op tracker -- , seq: 3491, time: 2014-06-09 16:22:11.969414, event: all_read, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.007965 7f869af78700 5 -- op tracker -- , seq: 3491, time: 2014-06-09 16:22:12.007938, event: dispatched, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.007971 7f869af78700 5 -- op tracker -- , seq: 3491, time: 2014-06-09 16:22:12.007969, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.007976 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5a4) v4 from osd.5 2014-06-09 16:22:12.007979 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343da40 2014-06-09 16:22:12.007983 7f869af78700 5 -- op tracker -- , seq: 3491, time: 2014-06-09 16:22:12.007982, event: started, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.007997 7f869af78700 5 -- op tracker -- , seq: 3491, time: 2014-06-09 16:22:12.007995, event: done, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.008004 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008006 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008018 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008019 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008020 7f869af78700 20 osd.26 243 _dispatch 0x1abd5c00 pg_info(1 pgs e243:2.40a) v4 2014-06-09 16:22:12.008025 7f869af78700 5 -- op tracker -- , seq: 3492, time: 2014-06-09 16:22:11.965958, event: header_read, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.008032 7f869af78700 5 -- op tracker -- , seq: 3492, time: 2014-06-09 16:22:11.965959, event: throttled, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.008038 7f869af78700 5 -- op tracker -- , seq: 3492, time: 2014-06-09 16:22:11.965967, event: all_read, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.008045 7f869af78700 5 -- op tracker -- , seq: 3492, time: 2014-06-09 16:22:12.008017, event: dispatched, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.008051 7f869af78700 5 -- op tracker -- , seq: 3492, time: 2014-06-09 16:22:12.008049, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.008056 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.40a) v4 from osd.2 2014-06-09 16:22:12.008059 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1abd5c00 2014-06-09 16:22:12.008064 7f869af78700 5 -- op tracker -- , seq: 3492, time: 2014-06-09 16:22:12.008062, event: started, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.008080 7f869af78700 5 -- op tracker -- , seq: 3492, time: 2014-06-09 16:22:12.008078, event: done, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.008088 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008090 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008103 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008104 7f869af78700 20 osd.26 243 _dispatch 0x314756c0 pg_info(1 pgs e243:1.26c) v4 2014-06-09 16:22:12.008109 7f869af78700 5 -- op tracker -- , seq: 3493, time: 2014-06-09 16:22:11.962747, event: header_read, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:12.008120 7f869af78700 5 -- op tracker -- , seq: 3493, time: 2014-06-09 16:22:11.962748, event: throttled, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:12.008126 7f869af78700 5 -- op tracker -- , seq: 3493, time: 2014-06-09 16:22:11.962758, event: all_read, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:12.008132 7f869af78700 5 -- op tracker -- , seq: 3493, time: 2014-06-09 16:22:12.008101, event: dispatched, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:12.008139 7f869af78700 5 -- op tracker -- , seq: 3493, time: 2014-06-09 16:22:12.008137, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:12.008143 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.26c) v4 from osd.4 2014-06-09 16:22:12.008146 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x314756c0 2014-06-09 16:22:12.008150 7f869af78700 5 -- op tracker -- , seq: 3493, time: 2014-06-09 16:22:12.008149, event: started, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:12.008165 7f869af78700 5 -- op tracker -- , seq: 3493, time: 2014-06-09 16:22:12.008163, event: done, op: pg_info(1 pgs e243:1.26c) 2014-06-09 16:22:12.008172 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008173 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008191 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008193 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008194 7f869af78700 20 osd.26 243 _dispatch 0x33441a40 pg_info(1 pgs e243:3.4c4) v4 2014-06-09 16:22:12.008199 7f869af78700 5 -- op tracker -- , seq: 3494, time: 2014-06-09 16:22:11.972744, event: header_read, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.008205 7f869af78700 5 -- op tracker -- , seq: 3494, time: 2014-06-09 16:22:11.972744, event: throttled, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.008210 7f869af78700 5 -- op tracker -- , seq: 3494, time: 2014-06-09 16:22:11.972757, event: all_read, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.008216 7f869af78700 5 -- op tracker -- , seq: 3494, time: 2014-06-09 16:22:12.008191, event: dispatched, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.008223 7f869af78700 5 -- op tracker -- , seq: 3494, time: 2014-06-09 16:22:12.008221, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.008228 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4c4) v4 from osd.3 2014-06-09 16:22:12.008231 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x33441a40 2014-06-09 16:22:12.008235 7f869af78700 5 -- op tracker -- , seq: 3494, time: 2014-06-09 16:22:12.008233, event: started, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.008253 7f869af78700 5 -- op tracker -- , seq: 3494, time: 2014-06-09 16:22:12.008251, event: done, op: pg_info(1 pgs e243:3.4c4) 2014-06-09 16:22:12.008266 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008268 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008279 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008281 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008282 7f869af78700 20 osd.26 243 _dispatch 0x30caec40 pg_info(1 pgs e243:0.4bd) v4 2014-06-09 16:22:12.008286 7f869af78700 5 -- op tracker -- , seq: 3495, time: 2014-06-09 16:22:11.971265, event: header_read, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.008294 7f869af78700 5 -- op tracker -- , seq: 3495, time: 2014-06-09 16:22:11.971265, event: throttled, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.008300 7f869af78700 5 -- op tracker -- , seq: 3495, time: 2014-06-09 16:22:11.971274, event: all_read, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.008306 7f869af78700 5 -- op tracker -- , seq: 3495, time: 2014-06-09 16:22:12.008279, event: dispatched, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.008313 7f869af78700 5 -- op tracker -- , seq: 3495, time: 2014-06-09 16:22:12.008311, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.008318 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4bd) v4 from osd.8 2014-06-09 16:22:12.008321 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30caec40 2014-06-09 16:22:12.008326 7f869af78700 5 -- op tracker -- , seq: 3495, time: 2014-06-09 16:22:12.008324, event: started, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.008340 7f869af78700 5 -- op tracker -- , seq: 3495, time: 2014-06-09 16:22:12.008338, event: done, op: pg_info(1 pgs e243:0.4bd) 2014-06-09 16:22:12.008347 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008349 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008360 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008362 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008363 7f869af78700 20 osd.26 243 _dispatch 0x3343d880 pg_info(1 pgs e243:2.5a4) v4 2014-06-09 16:22:12.008367 7f869af78700 5 -- op tracker -- , seq: 3496, time: 2014-06-09 16:22:11.969419, event: header_read, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.008374 7f869af78700 5 -- op tracker -- , seq: 3496, time: 2014-06-09 16:22:11.969419, event: throttled, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.008380 7f869af78700 5 -- op tracker -- , seq: 3496, time: 2014-06-09 16:22:11.969430, event: all_read, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.008386 7f869af78700 5 -- op tracker -- , seq: 3496, time: 2014-06-09 16:22:12.008360, event: dispatched, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.008393 7f869af78700 5 -- op tracker -- , seq: 3496, time: 2014-06-09 16:22:12.008391, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.008397 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.5a4) v4 from osd.5 2014-06-09 16:22:12.008400 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343d880 2014-06-09 16:22:12.008404 7f869af78700 5 -- op tracker -- , seq: 3496, time: 2014-06-09 16:22:12.008403, event: started, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.008418 7f869af78700 5 -- op tracker -- , seq: 3496, time: 2014-06-09 16:22:12.008416, event: done, op: pg_info(1 pgs e243:2.5a4) 2014-06-09 16:22:12.008426 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008427 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008439 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008440 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008441 7f869af78700 20 osd.26 243 _dispatch 0x29ec5a40 pg_info(1 pgs e243:3.5ce) v4 2014-06-09 16:22:12.008446 7f869af78700 5 -- op tracker -- , seq: 3497, time: 2014-06-09 16:22:11.966073, event: header_read, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:12.008453 7f869af78700 5 -- op tracker -- , seq: 3497, time: 2014-06-09 16:22:11.966074, event: throttled, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:12.008459 7f869af78700 5 -- op tracker -- , seq: 3497, time: 2014-06-09 16:22:11.966090, event: all_read, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:12.008465 7f869af78700 5 -- op tracker -- , seq: 3497, time: 2014-06-09 16:22:12.008438, event: dispatched, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:12.008473 7f869af78700 5 -- op tracker -- , seq: 3497, time: 2014-06-09 16:22:12.008471, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:12.008478 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.5ce) v4 from osd.2 2014-06-09 16:22:12.008481 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x29ec5a40 2014-06-09 16:22:12.008486 7f869af78700 5 -- op tracker -- , seq: 3497, time: 2014-06-09 16:22:12.008484, event: started, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:12.008509 7f869af78700 5 -- op tracker -- , seq: 3497, time: 2014-06-09 16:22:12.008506, event: done, op: pg_info(1 pgs e243:3.5ce) 2014-06-09 16:22:12.008517 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008519 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008531 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008532 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008534 7f869af78700 20 osd.26 243 _dispatch 0x31475500 pg_info(1 pgs e243:1.6b6) v4 2014-06-09 16:22:12.008539 7f869af78700 5 -- op tracker -- , seq: 3498, time: 2014-06-09 16:22:11.962800, event: header_read, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:12.008546 7f869af78700 5 -- op tracker -- , seq: 3498, time: 2014-06-09 16:22:11.962801, event: throttled, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:12.008552 7f869af78700 5 -- op tracker -- , seq: 3498, time: 2014-06-09 16:22:11.962809, event: all_read, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:12.008559 7f869af78700 5 -- op tracker -- , seq: 3498, time: 2014-06-09 16:22:12.008530, event: dispatched, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:12.008567 7f869af78700 5 -- op tracker -- , seq: 3498, time: 2014-06-09 16:22:12.008564, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:12.008576 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.6b6) v4 from osd.4 2014-06-09 16:22:12.008579 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31475500 2014-06-09 16:22:12.008585 7f869af78700 5 -- op tracker -- , seq: 3498, time: 2014-06-09 16:22:12.008583, event: started, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:12.008604 7f869af78700 5 -- op tracker -- , seq: 3498, time: 2014-06-09 16:22:12.008602, event: done, op: pg_info(1 pgs e243:1.6b6) 2014-06-09 16:22:12.008612 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008614 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008626 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008628 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008630 7f869af78700 20 osd.26 243 _dispatch 0x30ef6fc0 pg_info(1 pgs e243:2.566) v4 2014-06-09 16:22:12.008635 7f869af78700 5 -- op tracker -- , seq: 3499, time: 2014-06-09 16:22:11.971279, event: header_read, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.008647 7f869af78700 5 -- op tracker -- , seq: 3499, time: 2014-06-09 16:22:11.971279, event: throttled, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.008654 7f869af78700 5 -- op tracker -- , seq: 3499, time: 2014-06-09 16:22:11.971290, event: all_read, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.008661 7f869af78700 5 -- op tracker -- , seq: 3499, time: 2014-06-09 16:22:12.008626, event: dispatched, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.008671 7f869af78700 5 -- op tracker -- , seq: 3499, time: 2014-06-09 16:22:12.008669, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.008676 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.566) v4 from osd.8 2014-06-09 16:22:12.008679 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef6fc0 2014-06-09 16:22:12.008684 7f869af78700 5 -- op tracker -- , seq: 3499, time: 2014-06-09 16:22:12.008682, event: started, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.008699 7f869af78700 5 -- op tracker -- , seq: 3499, time: 2014-06-09 16:22:12.008697, event: done, op: pg_info(1 pgs e243:2.566) 2014-06-09 16:22:12.008707 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008709 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008722 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008724 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008726 7f869af78700 20 osd.26 243 _dispatch 0x3343d6c0 pg_info(1 pgs e243:3.437) v4 2014-06-09 16:22:12.008731 7f869af78700 5 -- op tracker -- , seq: 3500, time: 2014-06-09 16:22:11.969615, event: header_read, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.008738 7f869af78700 5 -- op tracker -- , seq: 3500, time: 2014-06-09 16:22:11.969617, event: throttled, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.008744 7f869af78700 5 -- op tracker -- , seq: 3500, time: 2014-06-09 16:22:11.969628, event: all_read, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.008753 7f869af78700 5 -- op tracker -- , seq: 3500, time: 2014-06-09 16:22:12.008722, event: dispatched, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.008760 7f869af78700 5 -- op tracker -- , seq: 3500, time: 2014-06-09 16:22:12.008759, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.008765 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.437) v4 from osd.5 2014-06-09 16:22:12.008768 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343d6c0 2014-06-09 16:22:12.008772 7f869af78700 5 -- op tracker -- , seq: 3500, time: 2014-06-09 16:22:12.008771, event: started, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.008788 7f869af78700 5 -- op tracker -- , seq: 3500, time: 2014-06-09 16:22:12.008786, event: done, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.008795 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008797 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008808 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008809 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008811 7f869af78700 20 osd.26 243 _dispatch 0x1ac956c0 pg_info(1 pgs e243:3.342) v4 2014-06-09 16:22:12.008815 7f869af78700 5 -- op tracker -- , seq: 3501, time: 2014-06-09 16:22:11.966104, event: header_read, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.008823 7f869af78700 5 -- op tracker -- , seq: 3501, time: 2014-06-09 16:22:11.966105, event: throttled, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.008829 7f869af78700 5 -- op tracker -- , seq: 3501, time: 2014-06-09 16:22:11.966113, event: all_read, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.008835 7f869af78700 5 -- op tracker -- , seq: 3501, time: 2014-06-09 16:22:12.008808, event: dispatched, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.008841 7f869af78700 5 -- op tracker -- , seq: 3501, time: 2014-06-09 16:22:12.008840, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.008846 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.342) v4 from osd.2 2014-06-09 16:22:12.008849 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac956c0 2014-06-09 16:22:12.008853 7f869af78700 5 -- op tracker -- , seq: 3501, time: 2014-06-09 16:22:12.008851, event: started, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.008868 7f869af78700 5 -- op tracker -- , seq: 3501, time: 2014-06-09 16:22:12.008866, event: done, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.008875 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008877 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008889 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008891 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008892 7f869af78700 20 osd.26 243 _dispatch 0x31475340 pg_info(1 pgs e243:1.582) v4 2014-06-09 16:22:12.008897 7f869af78700 5 -- op tracker -- , seq: 3502, time: 2014-06-09 16:22:11.962851, event: header_read, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:12.008904 7f869af78700 5 -- op tracker -- , seq: 3502, time: 2014-06-09 16:22:11.962852, event: throttled, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:12.008910 7f869af78700 5 -- op tracker -- , seq: 3502, time: 2014-06-09 16:22:11.962860, event: all_read, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:12.008916 7f869af78700 5 -- op tracker -- , seq: 3502, time: 2014-06-09 16:22:12.008889, event: dispatched, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:12.008923 7f869af78700 5 -- op tracker -- , seq: 3502, time: 2014-06-09 16:22:12.008921, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:12.008927 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.582) v4 from osd.4 2014-06-09 16:22:12.008930 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31475340 2014-06-09 16:22:12.008934 7f869af78700 5 -- op tracker -- , seq: 3502, time: 2014-06-09 16:22:12.008932, event: started, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:12.008950 7f869af78700 5 -- op tracker -- , seq: 3502, time: 2014-06-09 16:22:12.008948, event: done, op: pg_info(1 pgs e243:1.582) 2014-06-09 16:22:12.008958 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008960 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008971 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.008973 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.008974 7f869af78700 20 osd.26 243 _dispatch 0x26001880 pg_info(1 pgs e243:3.555) v4 2014-06-09 16:22:12.008980 7f869af78700 5 -- op tracker -- , seq: 3503, time: 2014-06-09 16:22:11.972447, event: header_read, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.008987 7f869af78700 5 -- op tracker -- , seq: 3503, time: 2014-06-09 16:22:11.972447, event: throttled, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.008993 7f869af78700 5 -- op tracker -- , seq: 3503, time: 2014-06-09 16:22:11.972458, event: all_read, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.009000 7f869af78700 5 -- op tracker -- , seq: 3503, time: 2014-06-09 16:22:12.008971, event: dispatched, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.009008 7f869af78700 5 -- op tracker -- , seq: 3503, time: 2014-06-09 16:22:12.009006, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.009012 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.555) v4 from osd.8 2014-06-09 16:22:12.009015 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26001880 2014-06-09 16:22:12.009020 7f869af78700 5 -- op tracker -- , seq: 3503, time: 2014-06-09 16:22:12.009018, event: started, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.009036 7f869af78700 5 -- op tracker -- , seq: 3503, time: 2014-06-09 16:22:12.009034, event: done, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.009043 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009045 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009056 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009058 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009059 7f869af78700 20 osd.26 243 _dispatch 0x3343d500 pg_info(1 pgs e243:3.437) v4 2014-06-09 16:22:12.009064 7f869af78700 5 -- op tracker -- , seq: 3504, time: 2014-06-09 16:22:11.969636, event: header_read, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.009077 7f869af78700 5 -- op tracker -- , seq: 3504, time: 2014-06-09 16:22:11.969637, event: throttled, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.009086 7f869af78700 5 -- op tracker -- , seq: 3504, time: 2014-06-09 16:22:11.969656, event: all_read, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.009096 7f869af78700 5 -- op tracker -- , seq: 3504, time: 2014-06-09 16:22:12.009056, event: dispatched, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.009103 7f869af78700 5 -- op tracker -- , seq: 3504, time: 2014-06-09 16:22:12.009101, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.009107 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.437) v4 from osd.5 2014-06-09 16:22:12.009110 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343d500 2014-06-09 16:22:12.009115 7f869af78700 5 -- op tracker -- , seq: 3504, time: 2014-06-09 16:22:12.009113, event: started, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.009133 7f869af78700 5 -- op tracker -- , seq: 3504, time: 2014-06-09 16:22:12.009131, event: done, op: pg_info(1 pgs e243:3.437) 2014-06-09 16:22:12.009141 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009143 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009155 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009157 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009159 7f869af78700 20 osd.26 243 _dispatch 0x1ac94a80 pg_info(1 pgs e243:3.105) v4 2014-06-09 16:22:12.009164 7f869af78700 5 -- op tracker -- , seq: 3505, time: 2014-06-09 16:22:11.966117, event: header_read, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.009176 7f869af78700 5 -- op tracker -- , seq: 3505, time: 2014-06-09 16:22:11.966117, event: throttled, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.009183 7f869af78700 5 -- op tracker -- , seq: 3505, time: 2014-06-09 16:22:11.966123, event: all_read, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.009189 7f869af78700 5 -- op tracker -- , seq: 3505, time: 2014-06-09 16:22:12.009155, event: dispatched, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.009196 7f869af78700 5 -- op tracker -- , seq: 3505, time: 2014-06-09 16:22:12.009194, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.009201 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.105) v4 from osd.2 2014-06-09 16:22:12.009204 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac94a80 2014-06-09 16:22:12.009209 7f869af78700 5 -- op tracker -- , seq: 3505, time: 2014-06-09 16:22:12.009207, event: started, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.009225 7f869af78700 5 -- op tracker -- , seq: 3505, time: 2014-06-09 16:22:12.009223, event: done, op: pg_info(1 pgs e243:3.105) 2014-06-09 16:22:12.009233 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009235 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009246 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009248 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009249 7f869af78700 20 osd.26 243 _dispatch 0x31475180 pg_info(1 pgs e243:1.504) v4 2014-06-09 16:22:12.009254 7f869af78700 5 -- op tracker -- , seq: 3506, time: 2014-06-09 16:22:11.962905, event: header_read, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:12.009262 7f869af78700 5 -- op tracker -- , seq: 3506, time: 2014-06-09 16:22:11.962906, event: throttled, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:12.009268 7f869af78700 5 -- op tracker -- , seq: 3506, time: 2014-06-09 16:22:11.962914, event: all_read, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:12.009275 7f869af78700 5 -- op tracker -- , seq: 3506, time: 2014-06-09 16:22:12.009246, event: dispatched, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:12.009283 7f869af78700 5 -- op tracker -- , seq: 3506, time: 2014-06-09 16:22:12.009281, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:12.009290 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.504) v4 from osd.4 2014-06-09 16:22:12.009294 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31475180 2014-06-09 16:22:12.009298 7f869af78700 5 -- op tracker -- , seq: 3506, time: 2014-06-09 16:22:12.009296, event: started, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:12.009313 7f869af78700 5 -- op tracker -- , seq: 3506, time: 2014-06-09 16:22:12.009311, event: done, op: pg_info(1 pgs e243:1.504) 2014-06-09 16:22:12.009321 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009323 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009333 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009335 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009336 7f869af78700 20 osd.26 243 _dispatch 0x230bbdc0 pg_info(1 pgs e243:3.70e) v4 2014-06-09 16:22:12.009340 7f869af78700 5 -- op tracker -- , seq: 3507, time: 2014-06-09 16:22:11.972506, event: header_read, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.009347 7f869af78700 5 -- op tracker -- , seq: 3507, time: 2014-06-09 16:22:11.972506, event: throttled, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.009353 7f869af78700 5 -- op tracker -- , seq: 3507, time: 2014-06-09 16:22:11.972514, event: all_read, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.009358 7f869af78700 5 -- op tracker -- , seq: 3507, time: 2014-06-09 16:22:12.009333, event: dispatched, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.009365 7f869af78700 5 -- op tracker -- , seq: 3507, time: 2014-06-09 16:22:12.009363, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.009369 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.70e) v4 from osd.8 2014-06-09 16:22:12.009372 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230bbdc0 2014-06-09 16:22:12.009377 7f869af78700 5 -- op tracker -- , seq: 3507, time: 2014-06-09 16:22:12.009375, event: started, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.009392 7f869af78700 5 -- op tracker -- , seq: 3507, time: 2014-06-09 16:22:12.009390, event: done, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.009405 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009407 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009419 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009421 7f869af78700 20 osd.26 243 _dispatch 0x3343d340 pg_info(1 pgs e243:3.ad) v4 2014-06-09 16:22:12.009425 7f869af78700 5 -- op tracker -- , seq: 3508, time: 2014-06-09 16:22:11.969661, event: header_read, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009432 7f869af78700 5 -- op tracker -- , seq: 3508, time: 2014-06-09 16:22:11.969662, event: throttled, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009439 7f869af78700 5 -- op tracker -- , seq: 3508, time: 2014-06-09 16:22:11.969672, event: all_read, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009445 7f869af78700 5 -- op tracker -- , seq: 3508, time: 2014-06-09 16:22:12.009418, event: dispatched, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009452 7f869af78700 5 -- op tracker -- , seq: 3508, time: 2014-06-09 16:22:12.009450, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009456 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.ad) v4 from osd.5 2014-06-09 16:22:12.009459 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343d340 2014-06-09 16:22:12.009463 7f869af78700 5 -- op tracker -- , seq: 3508, time: 2014-06-09 16:22:12.009462, event: started, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009478 7f869af78700 5 -- op tracker -- , seq: 3508, time: 2014-06-09 16:22:12.009477, event: done, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009486 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009487 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009499 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009501 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009503 7f869af78700 20 osd.26 243 _dispatch 0x2ea5a1c0 pg_info(1 pgs e243:0.4d5) v4 2014-06-09 16:22:12.009508 7f869af78700 5 -- op tracker -- , seq: 3509, time: 2014-06-09 16:22:11.966127, event: header_read, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:12.009515 7f869af78700 5 -- op tracker -- , seq: 3509, time: 2014-06-09 16:22:11.966127, event: throttled, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:12.009526 7f869af78700 5 -- op tracker -- , seq: 3509, time: 2014-06-09 16:22:11.966138, event: all_read, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:12.009536 7f869af78700 5 -- op tracker -- , seq: 3509, time: 2014-06-09 16:22:12.009499, event: dispatched, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:12.009543 7f869af78700 5 -- op tracker -- , seq: 3509, time: 2014-06-09 16:22:12.009541, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:12.009548 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4d5) v4 from osd.2 2014-06-09 16:22:12.009551 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2ea5a1c0 2014-06-09 16:22:12.009556 7f869af78700 5 -- op tracker -- , seq: 3509, time: 2014-06-09 16:22:12.009554, event: started, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:12.009572 7f869af78700 5 -- op tracker -- , seq: 3509, time: 2014-06-09 16:22:12.009569, event: done, op: pg_info(1 pgs e243:0.4d5) 2014-06-09 16:22:12.009584 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009586 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009600 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009602 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009603 7f869af78700 20 osd.26 243 _dispatch 0x31474fc0 pg_info(1 pgs e243:1.54a) v4 2014-06-09 16:22:12.009608 7f869af78700 5 -- op tracker -- , seq: 3510, time: 2014-06-09 16:22:11.962956, event: header_read, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.009620 7f869af78700 5 -- op tracker -- , seq: 3510, time: 2014-06-09 16:22:11.962956, event: throttled, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.009630 7f869af78700 5 -- op tracker -- , seq: 3510, time: 2014-06-09 16:22:11.962964, event: all_read, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.009641 7f869af78700 5 -- op tracker -- , seq: 3510, time: 2014-06-09 16:22:12.009600, event: dispatched, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.009651 7f869af78700 5 -- op tracker -- , seq: 3510, time: 2014-06-09 16:22:12.009649, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.009659 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.54a) v4 from osd.4 2014-06-09 16:22:12.009662 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31474fc0 2014-06-09 16:22:12.009667 7f869af78700 5 -- op tracker -- , seq: 3510, time: 2014-06-09 16:22:12.009665, event: started, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.009683 7f869af78700 5 -- op tracker -- , seq: 3510, time: 2014-06-09 16:22:12.009681, event: done, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.009691 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009693 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009706 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009708 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009709 7f869af78700 20 osd.26 243 _dispatch 0x30c128c0 pg_info(1 pgs e243:0.4c7) v4 2014-06-09 16:22:12.009714 7f869af78700 5 -- op tracker -- , seq: 3511, time: 2014-06-09 16:22:11.972575, event: header_read, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.009721 7f869af78700 5 -- op tracker -- , seq: 3511, time: 2014-06-09 16:22:11.972576, event: throttled, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.009728 7f869af78700 5 -- op tracker -- , seq: 3511, time: 2014-06-09 16:22:11.972588, event: all_read, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.009735 7f869af78700 5 -- op tracker -- , seq: 3511, time: 2014-06-09 16:22:12.009706, event: dispatched, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.009746 7f869af78700 5 -- op tracker -- , seq: 3511, time: 2014-06-09 16:22:12.009744, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.009754 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4c7) v4 from osd.8 2014-06-09 16:22:12.009757 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c128c0 2014-06-09 16:22:12.009763 7f869af78700 5 -- op tracker -- , seq: 3511, time: 2014-06-09 16:22:12.009760, event: started, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.009779 7f869af78700 5 -- op tracker -- , seq: 3511, time: 2014-06-09 16:22:12.009776, event: done, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.009786 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009788 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009800 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009802 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009803 7f869af78700 20 osd.26 243 _dispatch 0x3343d180 pg_info(1 pgs e243:3.ad) v4 2014-06-09 16:22:12.009808 7f869af78700 5 -- op tracker -- , seq: 3512, time: 2014-06-09 16:22:11.969676, event: header_read, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009815 7f869af78700 5 -- op tracker -- , seq: 3512, time: 2014-06-09 16:22:11.969676, event: throttled, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009822 7f869af78700 5 -- op tracker -- , seq: 3512, time: 2014-06-09 16:22:11.969687, event: all_read, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009829 7f869af78700 5 -- op tracker -- , seq: 3512, time: 2014-06-09 16:22:12.009800, event: dispatched, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009836 7f869af78700 5 -- op tracker -- , seq: 3512, time: 2014-06-09 16:22:12.009834, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009846 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.ad) v4 from osd.5 2014-06-09 16:22:12.009849 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343d180 2014-06-09 16:22:12.009854 7f869af78700 5 -- op tracker -- , seq: 3512, time: 2014-06-09 16:22:12.009852, event: started, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009870 7f869af78700 5 -- op tracker -- , seq: 3512, time: 2014-06-09 16:22:12.009868, event: done, op: pg_info(1 pgs e243:3.ad) 2014-06-09 16:22:12.009879 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009880 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009893 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009894 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009895 7f869af78700 20 osd.26 243 _dispatch 0x3130a000 pg_info(1 pgs e243:0.330) v4 2014-06-09 16:22:12.009900 7f869af78700 5 -- op tracker -- , seq: 3513, time: 2014-06-09 16:22:11.966149, event: header_read, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.009908 7f869af78700 5 -- op tracker -- , seq: 3513, time: 2014-06-09 16:22:11.966150, event: throttled, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.009914 7f869af78700 5 -- op tracker -- , seq: 3513, time: 2014-06-09 16:22:11.966156, event: all_read, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.009921 7f869af78700 5 -- op tracker -- , seq: 3513, time: 2014-06-09 16:22:12.009892, event: dispatched, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.009928 7f869af78700 5 -- op tracker -- , seq: 3513, time: 2014-06-09 16:22:12.009926, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.009937 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.330) v4 from osd.2 2014-06-09 16:22:12.009941 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130a000 2014-06-09 16:22:12.009946 7f869af78700 5 -- op tracker -- , seq: 3513, time: 2014-06-09 16:22:12.009943, event: started, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.009961 7f869af78700 5 -- op tracker -- , seq: 3513, time: 2014-06-09 16:22:12.009959, event: done, op: pg_info(1 pgs e243:0.330) 2014-06-09 16:22:12.009969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009971 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009982 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.009984 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.009986 7f869af78700 20 osd.26 243 _dispatch 0x31474e00 pg_info(1 pgs e243:1.54a) v4 2014-06-09 16:22:12.009991 7f869af78700 5 -- op tracker -- , seq: 3514, time: 2014-06-09 16:22:11.963007, event: header_read, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.009998 7f869af78700 5 -- op tracker -- , seq: 3514, time: 2014-06-09 16:22:11.963008, event: throttled, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.010005 7f869af78700 5 -- op tracker -- , seq: 3514, time: 2014-06-09 16:22:11.963016, event: all_read, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.010011 7f869af78700 5 -- op tracker -- , seq: 3514, time: 2014-06-09 16:22:12.009982, event: dispatched, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.010018 7f869af78700 5 -- op tracker -- , seq: 3514, time: 2014-06-09 16:22:12.010016, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.010024 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.54a) v4 from osd.4 2014-06-09 16:22:12.010027 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31474e00 2014-06-09 16:22:12.010031 7f869af78700 5 -- op tracker -- , seq: 3514, time: 2014-06-09 16:22:12.010030, event: started, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.010046 7f869af78700 5 -- op tracker -- , seq: 3514, time: 2014-06-09 16:22:12.010044, event: done, op: pg_info(1 pgs e243:1.54a) 2014-06-09 16:22:12.010054 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010056 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010069 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010071 7f869af78700 20 osd.26 243 _dispatch 0x30c69500 pg_info(1 pgs e243:3.70e) v4 2014-06-09 16:22:12.010076 7f869af78700 5 -- op tracker -- , seq: 3515, time: 2014-06-09 16:22:11.972716, event: header_read, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.010083 7f869af78700 5 -- op tracker -- , seq: 3515, time: 2014-06-09 16:22:11.972717, event: throttled, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.010089 7f869af78700 5 -- op tracker -- , seq: 3515, time: 2014-06-09 16:22:11.972727, event: all_read, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.010095 7f869af78700 5 -- op tracker -- , seq: 3515, time: 2014-06-09 16:22:12.010067, event: dispatched, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.010100 7f869af78700 5 -- op tracker -- , seq: 3515, time: 2014-06-09 16:22:12.010099, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.010105 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.70e) v4 from osd.8 2014-06-09 16:22:12.010107 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c69500 2014-06-09 16:22:12.010111 7f869af78700 5 -- op tracker -- , seq: 3515, time: 2014-06-09 16:22:12.010110, event: started, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.010127 7f869af78700 5 -- op tracker -- , seq: 3515, time: 2014-06-09 16:22:12.010125, event: done, op: pg_info(1 pgs e243:3.70e) 2014-06-09 16:22:12.010133 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010135 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010146 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010148 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010149 7f869af78700 20 osd.26 243 _dispatch 0x3343cfc0 pg_info(1 pgs e243:3.48d) v4 2014-06-09 16:22:12.010155 7f869af78700 5 -- op tracker -- , seq: 3516, time: 2014-06-09 16:22:11.969699, event: header_read, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010162 7f869af78700 5 -- op tracker -- , seq: 3516, time: 2014-06-09 16:22:11.969700, event: throttled, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010168 7f869af78700 5 -- op tracker -- , seq: 3516, time: 2014-06-09 16:22:11.969710, event: all_read, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010178 7f869af78700 5 -- op tracker -- , seq: 3516, time: 2014-06-09 16:22:12.010146, event: dispatched, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010186 7f869af78700 5 -- op tracker -- , seq: 3516, time: 2014-06-09 16:22:12.010184, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010190 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.48d) v4 from osd.5 2014-06-09 16:22:12.010193 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343cfc0 2014-06-09 16:22:12.010198 7f869af78700 5 -- op tracker -- , seq: 3516, time: 2014-06-09 16:22:12.010196, event: started, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010213 7f869af78700 5 -- op tracker -- , seq: 3516, time: 2014-06-09 16:22:12.010211, event: done, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010221 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010223 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010234 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010236 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010237 7f869af78700 20 osd.26 243 _dispatch 0x30ef7880 pg_info(1 pgs e243:2.4b9) v4 2014-06-09 16:22:12.010242 7f869af78700 5 -- op tracker -- , seq: 3517, time: 2014-06-09 16:22:11.966160, event: header_read, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:12.010248 7f869af78700 5 -- op tracker -- , seq: 3517, time: 2014-06-09 16:22:11.966160, event: throttled, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:12.010254 7f869af78700 5 -- op tracker -- , seq: 3517, time: 2014-06-09 16:22:11.966166, event: all_read, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:12.010260 7f869af78700 5 -- op tracker -- , seq: 3517, time: 2014-06-09 16:22:12.010234, event: dispatched, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:12.010265 7f869af78700 5 -- op tracker -- , seq: 3517, time: 2014-06-09 16:22:12.010264, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:12.010269 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4b9) v4 from osd.2 2014-06-09 16:22:12.010271 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef7880 2014-06-09 16:22:12.010275 7f869af78700 5 -- op tracker -- , seq: 3517, time: 2014-06-09 16:22:12.010274, event: started, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:12.010288 7f869af78700 5 -- op tracker -- , seq: 3517, time: 2014-06-09 16:22:12.010286, event: done, op: pg_info(1 pgs e243:2.4b9) 2014-06-09 16:22:12.010294 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010296 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010307 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010308 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010309 7f869af78700 20 osd.26 243 _dispatch 0x31474c40 pg_info(1 pgs e243:2.482) v4 2014-06-09 16:22:12.010313 7f869af78700 5 -- op tracker -- , seq: 3518, time: 2014-06-09 16:22:11.963058, event: header_read, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010319 7f869af78700 5 -- op tracker -- , seq: 3518, time: 2014-06-09 16:22:11.963058, event: throttled, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010324 7f869af78700 5 -- op tracker -- , seq: 3518, time: 2014-06-09 16:22:11.963066, event: all_read, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010329 7f869af78700 5 -- op tracker -- , seq: 3518, time: 2014-06-09 16:22:12.010306, event: dispatched, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010335 7f869af78700 5 -- op tracker -- , seq: 3518, time: 2014-06-09 16:22:12.010333, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010338 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.482) v4 from osd.4 2014-06-09 16:22:12.010341 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31474c40 2014-06-09 16:22:12.010344 7f869af78700 5 -- op tracker -- , seq: 3518, time: 2014-06-09 16:22:12.010343, event: started, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010357 7f869af78700 5 -- op tracker -- , seq: 3518, time: 2014-06-09 16:22:12.010355, event: done, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010363 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010364 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010374 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010376 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010377 7f869af78700 20 osd.26 243 _dispatch 0x30acd340 pg_info(1 pgs e243:3.555) v4 2014-06-09 16:22:12.010381 7f869af78700 5 -- op tracker -- , seq: 3519, time: 2014-06-09 16:22:11.976160, event: header_read, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.010386 7f869af78700 5 -- op tracker -- , seq: 3519, time: 2014-06-09 16:22:11.976160, event: throttled, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.010391 7f869af78700 5 -- op tracker -- , seq: 3519, time: 2014-06-09 16:22:11.976174, event: all_read, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.010396 7f869af78700 5 -- op tracker -- , seq: 3519, time: 2014-06-09 16:22:12.010374, event: dispatched, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.010402 7f869af78700 5 -- op tracker -- , seq: 3519, time: 2014-06-09 16:22:12.010400, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.010406 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.555) v4 from osd.8 2014-06-09 16:22:12.010408 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30acd340 2014-06-09 16:22:12.010412 7f869af78700 5 -- op tracker -- , seq: 3519, time: 2014-06-09 16:22:12.010410, event: started, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.010425 7f869af78700 5 -- op tracker -- , seq: 3519, time: 2014-06-09 16:22:12.010423, event: done, op: pg_info(1 pgs e243:3.555) 2014-06-09 16:22:12.010432 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010434 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010446 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010447 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010449 7f869af78700 20 osd.26 243 _dispatch 0x3343ce00 pg_info(1 pgs e243:3.48d) v4 2014-06-09 16:22:12.010454 7f869af78700 5 -- op tracker -- , seq: 3520, time: 2014-06-09 16:22:11.969726, event: header_read, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010460 7f869af78700 5 -- op tracker -- , seq: 3520, time: 2014-06-09 16:22:11.969727, event: throttled, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010465 7f869af78700 5 -- op tracker -- , seq: 3520, time: 2014-06-09 16:22:11.969741, event: all_read, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010470 7f869af78700 5 -- op tracker -- , seq: 3520, time: 2014-06-09 16:22:12.010446, event: dispatched, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010476 7f869af78700 5 -- op tracker -- , seq: 3520, time: 2014-06-09 16:22:12.010474, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010479 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.48d) v4 from osd.5 2014-06-09 16:22:12.010482 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343ce00 2014-06-09 16:22:12.010486 7f869af78700 5 -- op tracker -- , seq: 3520, time: 2014-06-09 16:22:12.010484, event: started, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010498 7f869af78700 5 -- op tracker -- , seq: 3520, time: 2014-06-09 16:22:12.010497, event: done, op: pg_info(1 pgs e243:3.48d) 2014-06-09 16:22:12.010505 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010506 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010516 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010517 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010518 7f869af78700 20 osd.26 243 _dispatch 0x30ef7c00 pg_info(1 pgs e243:3.251) v4 2014-06-09 16:22:12.010522 7f869af78700 5 -- op tracker -- , seq: 3521, time: 2014-06-09 16:22:11.966170, event: header_read, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.010528 7f869af78700 5 -- op tracker -- , seq: 3521, time: 2014-06-09 16:22:11.966170, event: throttled, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.010533 7f869af78700 5 -- op tracker -- , seq: 3521, time: 2014-06-09 16:22:11.966176, event: all_read, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.010538 7f869af78700 5 -- op tracker -- , seq: 3521, time: 2014-06-09 16:22:12.010516, event: dispatched, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.010544 7f869af78700 5 -- op tracker -- , seq: 3521, time: 2014-06-09 16:22:12.010542, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.010548 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.251) v4 from osd.2 2014-06-09 16:22:12.010551 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef7c00 2014-06-09 16:22:12.010555 7f869af78700 5 -- op tracker -- , seq: 3521, time: 2014-06-09 16:22:12.010553, event: started, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.010572 7f869af78700 5 -- op tracker -- , seq: 3521, time: 2014-06-09 16:22:12.010570, event: done, op: pg_info(1 pgs e243:3.251) 2014-06-09 16:22:12.010578 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010579 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010588 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010590 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010591 7f869af78700 20 osd.26 243 _dispatch 0x31474a80 pg_info(1 pgs e243:2.482) v4 2014-06-09 16:22:12.010594 7f869af78700 5 -- op tracker -- , seq: 3522, time: 2014-06-09 16:22:11.963110, event: header_read, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010600 7f869af78700 5 -- op tracker -- , seq: 3522, time: 2014-06-09 16:22:11.963110, event: throttled, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010605 7f869af78700 5 -- op tracker -- , seq: 3522, time: 2014-06-09 16:22:11.963118, event: all_read, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010610 7f869af78700 5 -- op tracker -- , seq: 3522, time: 2014-06-09 16:22:12.010588, event: dispatched, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010616 7f869af78700 5 -- op tracker -- , seq: 3522, time: 2014-06-09 16:22:12.010614, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010620 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.482) v4 from osd.4 2014-06-09 16:22:12.010622 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31474a80 2014-06-09 16:22:12.010626 7f869af78700 5 -- op tracker -- , seq: 3522, time: 2014-06-09 16:22:12.010625, event: started, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010638 7f869af78700 5 -- op tracker -- , seq: 3522, time: 2014-06-09 16:22:12.010636, event: done, op: pg_info(1 pgs e243:2.482) 2014-06-09 16:22:12.010644 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010646 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010656 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010657 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010658 7f869af78700 20 osd.26 243 _dispatch 0x310db6c0 pg_info(1 pgs e243:0.4c7) v4 2014-06-09 16:22:12.010662 7f869af78700 5 -- op tracker -- , seq: 3523, time: 2014-06-09 16:22:11.979040, event: header_read, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.010668 7f869af78700 5 -- op tracker -- , seq: 3523, time: 2014-06-09 16:22:11.979040, event: throttled, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.010673 7f869af78700 5 -- op tracker -- , seq: 3523, time: 2014-06-09 16:22:11.979048, event: all_read, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.010678 7f869af78700 5 -- op tracker -- , seq: 3523, time: 2014-06-09 16:22:12.010656, event: dispatched, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.010684 7f869af78700 5 -- op tracker -- , seq: 3523, time: 2014-06-09 16:22:12.010683, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.010688 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.4c7) v4 from osd.8 2014-06-09 16:22:12.010691 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x310db6c0 2014-06-09 16:22:12.010695 7f869af78700 5 -- op tracker -- , seq: 3523, time: 2014-06-09 16:22:12.010693, event: started, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.010707 7f869af78700 5 -- op tracker -- , seq: 3523, time: 2014-06-09 16:22:12.010705, event: done, op: pg_info(1 pgs e243:0.4c7) 2014-06-09 16:22:12.010713 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010714 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010724 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010725 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010726 7f869af78700 20 osd.26 243 _dispatch 0x3343cc40 pg_info(1 pgs e243:2.4a7) v4 2014-06-09 16:22:12.010730 7f869af78700 5 -- op tracker -- , seq: 3524, time: 2014-06-09 16:22:11.969749, event: header_read, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.010736 7f869af78700 5 -- op tracker -- , seq: 3524, time: 2014-06-09 16:22:11.969750, event: throttled, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.010741 7f869af78700 5 -- op tracker -- , seq: 3524, time: 2014-06-09 16:22:11.969760, event: all_read, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.010746 7f869af78700 5 -- op tracker -- , seq: 3524, time: 2014-06-09 16:22:12.010724, event: dispatched, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.010751 7f869af78700 5 -- op tracker -- , seq: 3524, time: 2014-06-09 16:22:12.010750, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.010755 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4a7) v4 from osd.5 2014-06-09 16:22:12.010757 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343cc40 2014-06-09 16:22:12.010759 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.010761 7f869af78700 5 -- op tracker -- , seq: 3524, time: 2014-06-09 16:22:12.010760, event: started, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.010774 7f869af78700 5 -- op tracker -- , seq: 3524, time: 2014-06-09 16:22:12.010772, event: done, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.010780 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010781 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010783 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3768.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.010788 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.010791 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.010792 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.010793 7f869af78700 20 osd.26 243 _dispatch 0x30ef6c40 pg_info(1 pgs e243:1.465) v4 2014-06-09 16:22:12.010797 7f869af78700 5 -- op tracker -- , seq: 3525, time: 2014-06-09 16:22:11.966179, event: header_read, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.010803 7f869af78700 5 -- op tracker -- , seq: 3525, time: 2014-06-09 16:22:11.966179, event: throttled, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.010809 7f869af78700 5 -- op tracker -- , seq: 3525, time: 2014-06-09 16:22:11.966187, event: all_read, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.010816 7f86a5e11700 10 _do_op 0xc2d5740 seq 3768 r = 0, finisher 0x3251c640 0 2014-06-09 16:22:12.010816 7f869af78700 5 -- op tracker -- , seq: 3525, time: 2014-06-09 16:22:12.010790, event: dispatched, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.010822 7f869af78700 5 -- op tracker -- , seq: 3525, time: 2014-06-09 16:22:12.010820, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.010826 7f86a5e11700 10 _finish_op 0xc2d5740 seq 3768 osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 2014-06-09 16:22:12.010826 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.465) v4 from osd.2 2014-06-09 16:22:12.010830 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30ef6c40 2014-06-09 16:22:12.010840 7f868cf5c700 10 op_submit_start 3817 2014-06-09 16:22:12.010840 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd440 (1 active) 2014-06-09 16:22:12.010842 7f868cf5c700 5 queue_transactions (trailing journal) 3817 2014-06-09 16:22:12.010844 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7807e0 (2 active) 2014-06-09 16:22:12.010844 7f868cf5c700 5 queue_op 0xcabdaa0 seq 3817 osr(2.530 0x2ed19b0) 828 bytes (queue has 50 ops and 2637230 bytes) 2014-06-09 16:22:12.010847 7f868cf5c700 10 op_submit_finish 3817 2014-06-09 16:22:12.010847 7f86a5e11700 5 _do_op 0x30a2a80 seq 3769 osr(1.11d 0x384c8b0)/0x384c8b0 start 2014-06-09 16:22:12.010850 7f86a5e11700 10 _do_transaction on 0xc12d740 2014-06-09 16:22:12.010857 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.010869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3ef80] 2014-06-09 16:22:12.010873 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3ef80] 2014-06-09 16:22:12.010875 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.010875 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (19 active) 2014-06-09 16:22:12.010876 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.010883 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.010896 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.010888 7f868cf5c700 10 osd.26 243 dequeue_op 0x257c5300 prio 196 cost 0 latency 0.010553 osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) v2 pg pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.010908 7f8692f68700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.010911 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3769.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.010914 7f8692767700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.010916 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.010919 7f868e75f700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.010925 7f868af58700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.010930 7f868a757700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.010930 7f86a5e11700 10 _do_op 0x30a2a80 seq 3769 r = 0, finisher 0xbd5c8c0 0 2014-06-09 16:22:12.010936 7f868ef60700 2 waiting 51 > 50 ops || 2638058 > 104857600 2014-06-09 16:22:12.010936 7f86a5e11700 10 _finish_op 0x30a2a80 seq 3769 osr(1.11d 0x384c8b0)/0x384c8b0 2014-06-09 16:22:12.010942 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251c640] 2014-06-09 16:22:12.010947 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251c640] 2014-06-09 16:22:12.010948 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.010950 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.010946 7f869af78700 5 -- op tracker -- , seq: 3525, time: 2014-06-09 16:22:12.010833, event: started, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.010960 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7807e0 (1 active) 2014-06-09 16:22:12.010958 7f868cf5c700 5 -- op tracker -- , seq: 3454, time: 2014-06-09 16:22:12.010904, event: reached_pg, op: osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.010963 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5680 (2 active) 2014-06-09 16:22:12.010963 7f8689f56700 10 op_submit_start 3818 2014-06-09 16:22:12.010965 7f8689f56700 5 queue_transactions (trailing journal) 3818 2014-06-09 16:22:12.010965 7f86a5e11700 5 _do_op 0xc2d4c00 seq 3770 osr(1.4fb 0x384d7f0)/0x384d7f0 start 2014-06-09 16:22:12.010968 7f86a5e11700 10 _do_transaction on 0x34e9e00 2014-06-09 16:22:12.010967 7f8689f56700 5 queue_op 0xd04f2c0 seq 3818 osr(3.7a 0x3ed0220) 826 bytes (queue has 50 ops and 2637228 bytes) 2014-06-09 16:22:12.010970 7f8689f56700 10 op_submit_finish 3818 2014-06-09 16:22:12.010971 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc169e20] 2014-06-09 16:22:12.010974 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc169e20] 2014-06-09 16:22:12.010974 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.010975 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.010966 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x257c5300 2014-06-09 16:22:12.010976 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.010974 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.5be( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:12.010981 7f868cf5c700 5 -- op tracker -- , seq: 3454, time: 2014-06-09 16:22:12.010978, event: started, op: osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.016541 7f868cf5c700 7 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 22 op ack_type 4 from 6 2014-06-09 16:22:12.016568 7f868cf5c700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:12.016561, event: sub_op_commit_rec, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.016577 7f868cf5c700 10 osd.26 243 dequeue_op 0x257c5300 finish 2014-06-09 16:22:12.016585 7f868cf5c700 5 -- op tracker -- , seq: 3454, time: 2014-06-09 16:22:12.016581, event: done, op: osd_sub_op_reply(client.4567.0:35562 3.224 bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.016599 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.016643 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f040 (20 active) 2014-06-09 16:22:12.016649 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.5b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.5b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.016686 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.016694 7f868cf5c700 5 queue_transactions existing osr(0.5b7 0x2bd7a90)/0x2bd7a90 2014-06-09 16:22:12.010988 7f8692f68700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.011014 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.011012 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.016710 7f8689f56700 5 queue_transactions existing osr(2.5be 0x2ed1b30)/0x2ed1b30 2014-06-09 16:22:12.016710 7f869af78700 5 -- op tracker -- , seq: 3525, time: 2014-06-09 16:22:12.010999, event: done, op: pg_info(1 pgs e243:1.465) 2014-06-09 16:22:12.016714 7f8689f56700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016720 7f868ff62700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016723 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.016725 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.016727 7f8691f66700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016730 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3770.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.016733 7f8690763700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016738 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.016743 7f868df5e700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016752 7f868c75b700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016757 7f86a5e11700 10 _do_op 0xc2d4c00 seq 3770 r = 0, finisher 0xcd32de0 0 2014-06-09 16:22:12.016761 7f8690f64700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016765 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.016765 7f86a5e11700 10 _finish_op 0xc2d4c00 seq 3770 osr(1.4fb 0x384d7f0)/0x384d7f0 2014-06-09 16:22:12.016767 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.016767 7f8689755700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016769 7f869af78700 20 osd.26 243 _dispatch 0x30fcda40 pg_info(1 pgs e243:3.465) v4 2014-06-09 16:22:12.016773 7f868d75d700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.011019 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5c8c0] 2014-06-09 16:22:12.016778 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5c8c0] 2014-06-09 16:22:12.016778 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd333a0] 2014-06-09 16:22:12.016780 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.016780 7f868b759700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.016781 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd333a0] 2014-06-09 16:22:12.016781 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.016782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.016783 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.011275 7f86a5610700 10 _do_op 0xd326ba0 seq 3767 r = 0, finisher 0x281b6900 0x9b0cdc0 2014-06-09 16:22:12.016787 7f8691765700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016775 7f869af78700 5 -- op tracker -- , seq: 3526, time: 2014-06-09 16:22:11.966383, event: header_read, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.016791 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314adf00] 2014-06-09 16:22:12.016793 7f868f761700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016798 7f8692767700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.016803 7f868e75f700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.016794 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 19 2014-06-09 16:22:12.016809 7f868af58700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016814 7f868a757700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016819 7f868ef60700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016824 7f868bf5a700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016795 7f869af78700 5 -- op tracker -- , seq: 3526, time: 2014-06-09 16:22:11.966384, event: throttled, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.016830 7f868cf5c700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016843 7f8692f68700 10 op_submit_start 3819 2014-06-09 16:22:12.016843 7f8689f56700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016845 7f8692f68700 5 queue_transactions (trailing journal) 3819 2014-06-09 16:22:12.016844 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5680 (1 active) 2014-06-09 16:22:12.016846 7f8692f68700 5 queue_op 0x324e6180 seq 3819 osr(2.6b0 0x2ed1ed0) 828 bytes (queue has 50 ops and 2637228 bytes) 2014-06-09 16:22:12.016848 7f868ff62700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016848 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653560 (2 active) 2014-06-09 16:22:12.016852 7f8690763700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016853 7f86a5e11700 5 _do_op 0xc2eae40 seq 3771 osr(2.5d9 0x2ed1a70)/0x2ed1a70 start 2014-06-09 16:22:12.016856 7f86a5e11700 10 _do_transaction on 0xc2498c0 2014-06-09 16:22:12.016857 7f868df5e700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016862 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.016863 7f868c75b700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016870 7f8690f64700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016873 7f8689755700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016868 7f86af56b700 5 -- op tracker -- , seq: 3528, time: 2014-06-09 16:22:12.006651, event: header_read, op: osd_op(client.4541.0:32202 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1683456~2510848] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.016877 7f868d75d700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016883 7f868b759700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.016888 7f8691765700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016889 7f86af56b700 5 -- op tracker -- , seq: 3528, time: 2014-06-09 16:22:12.006651, event: throttled, op: osd_op(client.4541.0:32202 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1683456~2510848] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.016893 7f868f761700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016897 7f8692767700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.016902 7f868e75f700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.016907 7f868af58700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016911 7f868a757700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016916 7f868ef60700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016920 7f868bf5a700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016920 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.016925 7f868cf5c700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016930 7f8691f66700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016931 7f86af56b700 5 -- op tracker -- , seq: 3528, time: 2014-06-09 16:22:12.012586, event: all_read, op: osd_op(client.4541.0:32202 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1683456~2510848] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.016936 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3771.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.016940 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.016942 7f86af56b700 5 -- op tracker -- , seq: 3528, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32202 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1683456~2510848] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.016952 7f86a5e11700 10 _do_op 0xc2eae40 seq 3771 r = 0, finisher 0x31791cc0 0 2014-06-09 16:22:12.016952 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.016955 7f86a5e11700 10 _finish_op 0xc2eae40 seq 3771 osr(2.5d9 0x2ed1a70)/0x2ed1a70 2014-06-09 16:22:12.016963 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653560 (1 active) 2014-06-09 16:22:12.016965 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653b00 (2 active) 2014-06-09 16:22:12.016966 7f868ff62700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016968 7f86a5e11700 5 _do_op 0xcabdec0 seq 3772 osr(0.55d 0x2bd77f0)/0x2bd77f0 start 2014-06-09 16:22:12.016970 7f86a5e11700 10 _do_transaction on 0x2c19d40 2014-06-09 16:22:12.016972 7f8690763700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016975 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.016977 7f868df5e700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016984 7f868c75b700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016990 7f8690f64700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016994 7f8689755700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.016988 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7900 prio 63 cost 2510848 latency 0.010337 osd_op(client.4541.0:32202 rbd_data.115d6b8b4567.000000000000163e [set-alloc-hint object_size 4194304 write_size 4194304,write 1683456~2510848] 3.a557c24 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.016998 7f868d75d700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017003 7f868b759700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017009 7f8691765700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017014 7f868f761700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017013 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.017019 7f8692767700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017024 7f868e75f700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017028 7f868af58700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017029 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3772.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.017033 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.017033 7f868a757700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017038 7f868ef60700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017043 7f868bf5a700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017046 7f86a5e11700 10 _do_op 0xcabdec0 seq 3772 r = 0, finisher 0xbd76820 0 2014-06-09 16:22:12.017049 7f86a5e11700 10 _finish_op 0xcabdec0 seq 3772 osr(0.55d 0x2bd77f0)/0x2bd77f0 2014-06-09 16:22:12.017062 7f868ff62700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017062 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653b00 (1 active) 2014-06-09 16:22:12.017065 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b680 (2 active) 2014-06-09 16:22:12.017067 7f868df5e700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017067 7f86a5e11700 5 _do_op 0xcabd1a0 seq 3773 osr(1.198 0x384cb30)/0x384cb30 start 2014-06-09 16:22:12.017070 7f86a5e11700 10 _do_transaction on 0xc2efbc0 2014-06-09 16:22:12.017072 7f8690763700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017075 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.017078 7f8690f64700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017083 7f8689755700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017087 7f868d75d700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017092 7f868b759700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017098 7f8691765700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017102 7f868f761700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017106 7f8692767700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017111 7f868e75f700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017112 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.017115 7f868af58700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017120 7f868a757700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017124 7f868ef60700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017126 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3773.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.017128 7f868bf5a700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017131 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.017133 7f8691f66700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017139 7f868c75b700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017164 7f86a5e11700 10 _do_op 0xcabd1a0 seq 3773 r = 0, finisher 0x30d61be0 0 2014-06-09 16:22:12.017169 7f86a5e11700 10 _finish_op 0xcabd1a0 seq 3773 osr(1.198 0x384cb30)/0x384cb30 2014-06-09 16:22:12.017176 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b680 (1 active) 2014-06-09 16:22:12.017178 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd8c0 (2 active) 2014-06-09 16:22:12.017179 7f868df5e700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017180 7f86a5e11700 5 _do_op 0xcabd200 seq 3774 osr(2.5b7 0x2ed1b70)/0x2ed1b70 start 2014-06-09 16:22:12.017183 7f86a5e11700 10 _do_transaction on 0xc2dae40 2014-06-09 16:22:12.017185 7f8690763700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017188 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.017191 7f8690f64700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017196 7f8689755700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017201 7f868d75d700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017207 7f868b759700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017212 7f8691765700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017217 7f868f761700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017222 7f8692767700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017226 7f868e75f700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017225 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.017231 7f868af58700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017235 7f868a757700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017240 7f868ef60700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017240 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3774.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.017243 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.017245 7f868bf5a700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017250 7f8691f66700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017250 7f86a5e11700 10 _do_op 0xcabd200 seq 3774 r = 0, finisher 0xbdbae80 0 2014-06-09 16:22:12.017253 7f86a5e11700 10 _finish_op 0xcabd200 seq 3774 osr(2.5b7 0x2ed1b70)/0x2ed1b70 2014-06-09 16:22:12.017256 7f868c75b700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017267 7f8690763700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017272 7f8690f64700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017266 7f86a460e700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:12.016810, event: op_commit, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.017276 7f868d75d700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017281 7f868b759700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017284 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd8c0 (1 active) 2014-06-09 16:22:12.017287 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa7e0 (2 active) 2014-06-09 16:22:12.017287 7f8691765700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017289 7f86a5e11700 5 _do_op 0xcabc060 seq 3775 osr(2.646 0x2ed1c30)/0x2ed1c30 start 2014-06-09 16:22:12.017278 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=213'683 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 19 all committed 2014-06-09 16:22:12.017292 7f86a5e11700 10 _do_transaction on 0xc1fcfc0 2014-06-09 16:22:12.017291 7f868f761700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017296 7f8692767700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017298 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.017301 7f868e75f700 2 waiting 51 > 50 ops || 2638054 > 104857600 2014-06-09 16:22:12.017305 7f868af58700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017292 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] eval_repop repgather(0x390c840 243'684 rep_tid=19 committed?=1 applied?=0 lock=0 op=osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.017309 7f868a757700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017314 7f868ef60700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017311 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] log_op_stats osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1032573 outb 0 rlat 0.000000 lat 1.519599 2014-06-09 16:22:12.017330 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] publish_stats_to_osd 243:1114 2014-06-09 16:22:12.017344 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.017346 7f868bf5a700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017352 7f8691f66700 2 waiting 51 > 50 ops || 2638056 > 104857600 2014-06-09 16:22:12.017342 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] sending commit on repgather(0x390c840 243'684 rep_tid=19 committed?=1 applied?=0 lock=0 op=osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4) 0xdfb1b80 2014-06-09 16:22:12.017359 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3775.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.017364 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.017377 7f86a5e11700 10 _do_op 0xcabc060 seq 3775 r = 0, finisher 0xbd5d280 0 2014-06-09 16:22:12.017381 7f86a5e11700 10 _finish_op 0xcabc060 seq 3775 osr(2.646 0x2ed1c30)/0x2ed1c30 2014-06-09 16:22:12.017383 7f86a460e700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:12.017377, event: commit_sent, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.017392 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314adf00] 2014-06-09 16:22:12.017391 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa7e0 (1 active) 2014-06-09 16:22:12.017395 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f0e0 (2 active) 2014-06-09 16:22:12.017394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x13964000,0x3006df60,0xbd646c0,0xbd6c280,0x3062abe0] 2014-06-09 16:22:12.017397 7f86a5e11700 5 _do_op 0x324f43c0 seq 3776 osr(2.113 0x2ed06b0)/0x2ed06b0 start 2014-06-09 16:22:12.017397 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x13964000,0x3006df60,0xbd646c0,0xbd6c280,0x3062abe0] 2014-06-09 16:22:12.017399 7f86a5e11700 10 _do_transaction on 0x390db00 2014-06-09 16:22:12.017400 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.017401 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.017404 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.017409 7f8682ae5700 5 -- op tracker -- , seq: 3527, time: 2014-06-09 16:22:12.011575, event: header_read, op: osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.017419 7f8682ae5700 5 -- op tracker -- , seq: 3527, time: 2014-06-09 16:22:12.011577, event: throttled, op: osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.017427 7f8682ae5700 5 -- op tracker -- , seq: 3527, time: 2014-06-09 16:22:12.011587, event: all_read, op: osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.017436 7f8682ae5700 5 -- op tracker -- , seq: 3527, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.017441 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.017441 7f8682ae5700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:12.017445 7f8682ae5700 20 osd.26 243 should_share_map osd.7 172.16.96.201:6851/52636 243 2014-06-09 16:22:12.017452 7f8682ae5700 15 osd.26 243 enqueue_op 0x32826400 prio 196 cost 0 latency 0.005876 osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:12.017455 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3776.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.017459 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.017470 7f86a5e11700 10 _do_op 0x324f43c0 seq 3776 r = 0, finisher 0xbd607e0 0 2014-06-09 16:22:12.017470 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd32de0,0x31791cc0,0xbd76820,0x30d61be0,0xbdbae80,0xbd5d280] 2014-06-09 16:22:12.017476 7f86a5e11700 10 _finish_op 0x324f43c0 seq 3776 osr(2.113 0x2ed06b0)/0x2ed06b0 2014-06-09 16:22:12.017478 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f0e0 (1 active) 2014-06-09 16:22:12.017481 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89a7e0 (2 active) 2014-06-09 16:22:12.017483 7f86a5e11700 5 _do_op 0x30d975c0 seq 3777 osr(2.2b5 0x2ed0c80)/0x2ed0c80 start 2014-06-09 16:22:12.017485 7f86a5e11700 10 _do_transaction on 0xc2ad800 2014-06-09 16:22:12.017486 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd32de0,0x31791cc0,0xbd76820,0x30d61be0,0xbdbae80,0xbd5d280] 2014-06-09 16:22:12.017490 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd607e0] 2014-06-09 16:22:12.017490 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.017494 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd607e0] 2014-06-09 16:22:12.017496 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.017497 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.017500 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd60bc0] 2014-06-09 16:22:12.017502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd60bc0] 2014-06-09 16:22:12.017503 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.017504 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.017508 7f86a5610700 10 _finish_op 0xd326ba0 seq 3767 osr(3.f6 0x3ed04b0)/0x3ed04b0 2014-06-09 16:22:12.017518 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x281b6900] 2014-06-09 16:22:12.017526 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.017521 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] op_applied: 19 2014-06-09 16:22:12.017542 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3777.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.017541 7f86a4e0f700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:12.017535, event: op_applied, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.017546 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.017549 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] op_applied on primary on version 243'684 2014-06-09 16:22:12.017556 7f86af56b700 5 -- op tracker -- , seq: 3529, time: 2014-06-09 16:22:12.017080, event: header_read, op: osd_op(client.4541.0:32272 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~319488] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.017559 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] repop_all_applied: repop tid 19 all applied 2014-06-09 16:22:12.017567 7f86a5e11700 10 _do_op 0x30d975c0 seq 3777 r = 0, finisher 0xcd9ee80 0 2014-06-09 16:22:12.017569 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] eval_repop repgather(0x390c840 243'684 rep_tid=19 committed?=1 applied?=1 lock=0 op=osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.017586 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] log_op_stats osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1032573 outb 0 rlat 0.000000 lat 1.519874 2014-06-09 16:22:12.017602 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] publish_stats_to_osd 243:1115 2014-06-09 16:22:12.017614 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.017626 7f868b759700 2 waiting 51 > 50 ops || 1604302 > 104857600 2014-06-09 16:22:12.017634 7f8691765700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017569 7f86af56b700 5 -- op tracker -- , seq: 3529, time: 2014-06-09 16:22:12.017082, event: throttled, op: osd_op(client.4541.0:32272 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~319488] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.017625 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] removing repgather(0x390c840 243'684 rep_tid=19 committed?=1 applied?=1 op=osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.017639 7f868f761700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017644 7f8692767700 2 waiting 51 > 50 ops || 1604302 > 104857600 2014-06-09 16:22:12.017649 7f868e75f700 2 waiting 51 > 50 ops || 1604302 > 104857600 2014-06-09 16:22:12.017640 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] q front is repgather(0x390c840 243'684 rep_tid=19 committed?=1 applied?=1 op=osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.017653 7f868af58700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017658 7f868a757700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017653 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] remove_repop repgather(0x390c840 243'684 rep_tid=19 committed?=1 applied?=1 op=osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.017662 7f868ef60700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017663 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] requeue_ops 2014-06-09 16:22:12.017667 7f868bf5a700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017672 7f8691f66700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017671 7f86af56b700 5 -- op tracker -- , seq: 3529, time: 2014-06-09 16:22:12.017519, event: all_read, op: osd_op(client.4541.0:32272 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~319488] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.017678 7f8690763700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017682 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x281b6900] 2014-06-09 16:22:12.017683 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.017684 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.017682 7f86af56b700 5 -- op tracker -- , seq: 3529, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32272 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~319488] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.017684 7f868c75b700 2 waiting 51 > 50 ops || 1604304 > 104857600 2014-06-09 16:22:12.017687 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7c940] 2014-06-09 16:22:12.017688 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.017688 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.017694 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.017700 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.2b5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.017705 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7c940] 2014-06-09 16:22:12.017705 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.017706 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.017704 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7500 prio 63 cost 319488 latency 0.000624 osd_op(client.4541.0:32272 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~319488] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.017712 7f86a5610700 5 -- op tracker -- , seq: 447, time: 2014-06-09 16:22:12.017708, event: done, op: osd_op(client.4575.0:32325 rbd_data.117f6b8b4567.000000000000153d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1032192] 3.942188f6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.017729 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34965a0 (1 active) 2014-06-09 16:22:12.017731 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcfc0 (2 active) 2014-06-09 16:22:12.017734 7f86a5610700 5 _do_op 0x288f9aa0 seq 3778 osr(1.18a 0x384cb50)/0x384cb50 start 2014-06-09 16:22:12.017736 7f86a5610700 10 _do_transaction on 0x2d83ec0 2014-06-09 16:22:12.017739 7f8692f68700 10 op_submit_finish 3819 2014-06-09 16:22:12.017741 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.017741 7f86a5e11700 10 _finish_op 0x30d975c0 seq 3777 osr(2.2b5 0x2ed0c80)/0x2ed0c80 2014-06-09 16:22:12.017744 7f8689f56700 10 op_submit_start 3820 2014-06-09 16:22:12.017746 7f8689f56700 5 queue_transactions (trailing journal) 3820 2014-06-09 16:22:12.017747 7f8689f56700 5 queue_op 0xd04ed80 seq 3820 osr(2.5be 0x2ed1b30) 828 bytes (queue has 49 ops and 1602615 bytes) 2014-06-09 16:22:12.017754 7f868f761700 2 waiting 51 > 50 ops || 1604271 > 104857600 2014-06-09 16:22:12.017756 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89a7e0 (1 active) 2014-06-09 16:22:12.017759 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f7a0 (2 active) 2014-06-09 16:22:12.017760 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd9ee80] 2014-06-09 16:22:12.017762 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (19 active) 2014-06-09 16:22:12.017763 7f86a5e11700 5 _do_op 0x15996c00 seq 3779 osr(3.136 0x3ed07a0)/0x3ed07a0 start 2014-06-09 16:22:12.017764 7f8689f56700 10 op_submit_finish 3820 2014-06-09 16:22:12.017765 7f86a5e11700 10 _do_transaction on 0x2eaa0c0 2014-06-09 16:22:12.017765 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd9ee80] 2014-06-09 16:22:12.017767 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.017768 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.017770 7f8692f68700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.017771 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.017769 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.186( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.017779 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.017790 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3778.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.017793 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.017774 7f8692f68700 10 osd.26 243 dequeue_op 0x32826400 prio 196 cost 0 latency 0.006199 osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) v2 pg pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=213'623 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.017802 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.017800 7f8692f68700 5 -- op tracker -- , seq: 3527, time: 2014-06-09 16:22:12.017796, event: reached_pg, op: osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.017807 7f8689f56700 5 queue_transactions existing osr(0.186 0x2bd67d0)/0x2bd67d0 2014-06-09 16:22:12.017808 7f868cf5c700 10 op_submit_start 3821 2014-06-09 16:22:12.017809 7f868cf5c700 5 queue_transactions (trailing journal) 3821 2014-06-09 16:22:12.017810 7f8689f56700 2 waiting 51 > 50 ops || 1604271 > 104857600 2014-06-09 16:22:12.017810 7f868cf5c700 5 queue_op 0xcabc240 seq 3821 osr(0.5b7 0x2bd7a90) 828 bytes (queue has 50 ops and 1603443 bytes) 2014-06-09 16:22:12.017813 7f868cf5c700 10 op_submit_finish 3821 2014-06-09 16:22:12.017816 7f868ff62700 10 op_submit_start 3822 2014-06-09 16:22:12.017807 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=213'623 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x32826400 2014-06-09 16:22:12.017817 7f868ff62700 5 queue_transactions (trailing journal) 3822 2014-06-09 16:22:12.017822 7f86a5610700 10 _do_op 0x288f9aa0 seq 3778 r = 0, finisher 0x3196c480 0 2014-06-09 16:22:12.017822 7f8692f68700 5 -- op tracker -- , seq: 3527, time: 2014-06-09 16:22:12.017819, event: started, op: osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.017827 7f86a5610700 10 _finish_op 0x288f9aa0 seq 3778 osr(1.18a 0x384cb50)/0x384cb50 2014-06-09 16:22:12.017817 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6ec( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.6ec( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.017828 7f8692f68700 7 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=213'623 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 23 op ack_type 4 from 7 2014-06-09 16:22:12.017838 7f868af58700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017848 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3196c480] 2014-06-09 16:22:12.017846 7f8692f68700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.017840, event: sub_op_commit_rec, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.017849 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.017851 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3196c480] 2014-06-09 16:22:12.017852 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.017853 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.017852 7f8692f68700 10 osd.26 243 dequeue_op 0x32826400 finish 2014-06-09 16:22:12.017854 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.017856 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcfc0 (1 active) 2014-06-09 16:22:12.017857 7f868cf5c700 5 queue_transactions existing osr(2.6ec 0x2ed1e10)/0x2ed1e10 2014-06-09 16:22:12.017858 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1d40 (2 active) 2014-06-09 16:22:12.017860 7f868cf5c700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017858 7f868ff62700 5 queue_op 0x30a2ea0 seq 3822 osr(1.206 0x384cd10) 828 bytes (queue has 50 ops and 1603441 bytes) 2014-06-09 16:22:12.017860 7f86a5610700 5 _do_op 0xd327500 seq 3780 osr(1.3cd 0x384d4b0)/0x384d4b0 start 2014-06-09 16:22:12.017859 7f8692f68700 5 -- op tracker -- , seq: 3527, time: 2014-06-09 16:22:12.017855, event: done, op: osd_sub_op_reply(client.4568.0:35343 3.3e6 bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.017862 7f868ff62700 10 op_submit_finish 3822 2014-06-09 16:22:12.017863 7f86a5610700 10 _do_transaction on 0xc145a40 2014-06-09 16:22:12.017864 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2320] 2014-06-09 16:22:12.017865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2320] 2014-06-09 16:22:12.017864 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3779.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.017867 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.017868 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.017868 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.017868 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.017869 7f8692f68700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.017868 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.26a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.017893 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40520 (20 active) 2014-06-09 16:22:12.017899 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.017903 7f868ff62700 5 queue_transactions existing osr(2.26a 0x2ed0dd0)/0x2ed0dd0 2014-06-09 16:22:12.017897 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.6b0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.6b0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.017905 7f868ff62700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017905 7f86a5e11700 10 _do_op 0x15996c00 seq 3779 r = 0, finisher 0x15a3f100 0 2014-06-09 16:22:12.017909 7f868df5e700 10 op_submit_start 3823 2014-06-09 16:22:12.017911 7f868df5e700 5 queue_transactions (trailing journal) 3823 2014-06-09 16:22:12.017910 7f86a5e11700 10 _finish_op 0x15996c00 seq 3779 osr(3.136 0x3ed07a0)/0x3ed07a0 2014-06-09 16:22:12.017912 7f868df5e700 5 queue_op 0xbf7f680 seq 3823 osr(3.579 0x3ed18d0) 828 bytes (queue has 49 ops and 1602613 bytes) 2014-06-09 16:22:12.017914 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a7720] 2014-06-09 16:22:12.017917 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a7720] 2014-06-09 16:22:12.017919 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.017920 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.017922 7f868bf5a700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017924 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f7a0 (1 active) 2014-06-09 16:22:12.017927 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496d80 (2 active) 2014-06-09 16:22:12.017927 7f8691f66700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017931 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.017933 7f868df5e700 10 op_submit_finish 3823 2014-06-09 16:22:12.017932 7f8690763700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017935 7f8692f68700 5 queue_transactions existing osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 2014-06-09 16:22:12.017937 7f8689755700 10 op_submit_start 3824 2014-06-09 16:22:12.017938 7f8692f68700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017938 7f8689755700 5 queue_transactions (trailing journal) 3824 2014-06-09 16:22:12.017940 7f8689755700 5 queue_op 0xbe7cf00 seq 3824 osr(3.2c9 0x3ed0ea0) 828 bytes (queue has 50 ops and 1603441 bytes) 2014-06-09 16:22:12.017942 7f868a757700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017943 7f8689755700 10 op_submit_finish 3824 2014-06-09 16:22:12.017948 7f8692767700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.017948 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.017963 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.858825 7 0.000342 2014-06-09 16:22:12.017976 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.017989 7f868df5e700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8fc00 (19 active) 2014-06-09 16:22:12.017989 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.017997 7f868df5e700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.018001 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000025 0 0.000000 2014-06-09 16:22:12.018001 7f868df5e700 10 osd.26 243 dequeue_op 0x11e62200 prio 63 cost 245760 latency 1.888659 osd_op(client.4569.0:38498 rbd_data.11796b8b4567.0000000000001a91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.2ea43ab3 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.018011 7f8689755700 5 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.018020 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.018023 7f868df5e700 5 -- op tracker -- , seq: 104, time: 2014-06-09 16:22:12.018019, event: reached_pg, op: osd_op(client.4569.0:38498 rbd_data.11796b8b4567.0000000000001a91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.2ea43ab3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.018027 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.018032 7f868df5e700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.018041 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.018046 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x11e62200 2014-06-09 16:22:12.018052 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.018062 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.018079 7f8689755700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:12.018056 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4569.0:38498 rbd_data.11796b8b4567.0000000000001a91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.2ea43ab3 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.018094 7f868df5e700 3 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4569.0:38498 was 213'631 2014-06-09 16:22:12.018096 7f8689755700 15 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.018124 7f868df5e700 10 osd.26 243 dequeue_op 0x11e62200 finish 2014-06-09 16:22:12.018129 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.018134 7f8689755700 5 queue_transactions existing osr(0.290 0x2bd6c80)/0x2bd6c80 2014-06-09 16:22:12.018136 7f8689755700 2 waiting 51 > 50 ops || 1604302 > 104857600 2014-06-09 16:22:12.018133 7f868df5e700 5 -- op tracker -- , seq: 104, time: 2014-06-09 16:22:12.018128, event: done, op: osd_op(client.4569.0:38498 rbd_data.11796b8b4567.0000000000001a91 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~245760] 3.2ea43ab3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.018141 7f8690f64700 10 op_submit_start 3825 2014-06-09 16:22:12.018142 7f8690f64700 5 queue_transactions (trailing journal) 3825 2014-06-09 16:22:12.018143 7f8690f64700 5 queue_op 0x30a2960 seq 3825 osr(1.71e 0x2ed00b0) 828 bytes (queue has 50 ops and 1603441 bytes) 2014-06-09 16:22:12.018147 7f8690f64700 10 op_submit_finish 3825 2014-06-09 16:22:12.018148 7f868df5e700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.018152 7f868d75d700 10 op_submit_start 3826 2014-06-09 16:22:12.018155 7f868d75d700 5 queue_transactions (trailing journal) 3826 2014-06-09 16:22:12.018157 7f868d75d700 5 queue_op 0x11fcfda0 seq 3826 osr(1.1ef 0x384cdb0) 828 bytes (queue has 50 ops and 1603441 bytes) 2014-06-09 16:22:12.018160 7f868d75d700 10 op_submit_finish 3826 2014-06-09 16:22:12.018153 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.151( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.018171 7f868df5e700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8fc00 (20 active) 2014-06-09 16:22:12.018167 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.65e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.018174 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.36d( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.36d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.018184 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.018189 7f8690f64700 5 queue_transactions existing osr(0.151 0x2bd65e0)/0x2bd65e0 2014-06-09 16:22:12.018192 7f8690f64700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018196 7f868b759700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018202 7f868f761700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018203 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x15a3f100] 2014-06-09 16:22:12.018204 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.018207 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x15a3f100] 2014-06-09 16:22:12.018208 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.018209 7f868cf5c700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018209 7f868d75d700 5 queue_transactions existing osr(1.65e 0x384df30)/0x384df30 2014-06-09 16:22:12.018209 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.018211 7f868d75d700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.017930 7f86a5e11700 5 _do_op 0x15996c60 seq 3781 osr(2.699 0x2ed1f10)/0x2ed1f10 start 2014-06-09 16:22:12.018213 7f868ff62700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018214 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.018215 7f8691765700 10 op_submit_start 3827 2014-06-09 16:22:12.018216 7f86a5e11700 10 _do_transaction on 0xc316c00 2014-06-09 16:22:12.018216 7f8691765700 5 queue_transactions (trailing journal) 3827 2014-06-09 16:22:12.018219 7f868af58700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018219 7f868df5e700 5 queue_transactions existing osr(0.36d 0x2bd6fc0)/0x2bd6fc0 2014-06-09 16:22:12.018218 7f8691765700 5 queue_op 0xc34a240 seq 3827 osr(0.66a 0x2bd7ef0) 828 bytes (queue has 50 ops and 1603441 bytes) 2014-06-09 16:22:12.018221 7f8691765700 10 op_submit_finish 3827 2014-06-09 16:22:12.018221 7f868df5e700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018224 7f868c75b700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018225 7f868e75f700 10 op_submit_start 3828 2014-06-09 16:22:12.018227 7f868e75f700 5 queue_transactions (trailing journal) 3828 2014-06-09 16:22:12.018229 7f868e75f700 5 queue_op 0x324f4540 seq 3828 osr(2.2b 0x2ed0230) 826 bytes (queue has 50 ops and 1603441 bytes) 2014-06-09 16:22:12.018231 7f868e75f700 10 op_submit_finish 3828 2014-06-09 16:22:12.018231 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.018236 7f868ef60700 10 op_submit_start 3829 2014-06-09 16:22:12.018226 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.d1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.018237 7f868ef60700 5 queue_transactions (trailing journal) 3829 2014-06-09 16:22:12.018239 7f868ef60700 5 queue_op 0x32503f80 seq 3829 osr(1.3f0 0x384d400) 828 bytes (queue has 50 ops and 1603441 bytes) 2014-06-09 16:22:12.018242 7f868ef60700 10 op_submit_finish 3829 2014-06-09 16:22:12.018246 7f8689f56700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018237 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.609( v 213'658 (0'0,213'658] local-les=243 n=129 ec=118 les/c 243/243 242/242/212) 2014-06-09 16:22:12.018246 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.487( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.487( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.018258 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.018262 7f8691765700 5 queue_transactions existing osr(1.d1 0x384c9b0)/0x384c9b0 2014-06-09 16:22:12.018264 7f8691765700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018268 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.018273 7f868e75f700 5 queue_transactions existing osr(3.609 0x3ed1b00)/0x3ed1b00 2014-06-09 16:22:12.018275 7f868e75f700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018280 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.018284 7f868ef60700 5 queue_transactions existing osr(2.487 0x2ed14b0)/0x2ed14b0 2014-06-09 16:22:12.018286 7f868ef60700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018325 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.018340 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3781.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.018344 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.018356 7f86a5e11700 10 _do_op 0x15996c60 seq 3781 r = 0, finisher 0x3042a480 0 2014-06-09 16:22:12.018362 7f86a5e11700 10 _finish_op 0x15996c60 seq 3781 osr(2.699 0x2ed1f10)/0x2ed1f10 2014-06-09 16:22:12.018365 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5d600] 2014-06-09 16:22:12.018367 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5d600] 2014-06-09 16:22:12.018368 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.018369 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.018370 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496d80 (1 active) 2014-06-09 16:22:12.018371 7f868bf5a700 10 op_submit_start 3830 2014-06-09 16:22:12.018372 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301d40 (2 active) 2014-06-09 16:22:12.018373 7f868bf5a700 5 queue_transactions (trailing journal) 3830 2014-06-09 16:22:12.018375 7f86a5e11700 5 _do_op 0x324f4480 seq 3782 osr(0.6d2 0x384c1f0)/0x384c1f0 start 2014-06-09 16:22:12.018375 7f868bf5a700 5 queue_op 0x15997b00 seq 3830 osr(2.67c 0x2ed1f90) 828 bytes (queue has 50 ops and 1603441 bytes) 2014-06-09 16:22:12.018377 7f86a5e11700 10 _do_transaction on 0x97bbd40 2014-06-09 16:22:12.018378 7f868bf5a700 10 op_submit_finish 3830 2014-06-09 16:22:12.018382 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.018381 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.7a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=4-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.7a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.018410 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.018414 7f868bf5a700 5 queue_transactions existing osr(1.7a 0x384c510)/0x384c510 2014-06-09 16:22:12.018416 7f868bf5a700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018418 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.018420 7f8691f66700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018426 7f8690763700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018433 7f8692f68700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018432 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3782.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.018436 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.018440 7f868a757700 2 waiting 51 > 50 ops || 1604269 > 104857600 2014-06-09 16:22:12.018446 7f8692767700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018447 7f86a5e11700 10 _do_op 0x324f4480 seq 3782 r = 0, finisher 0x324d1360 0 2014-06-09 16:22:12.018452 7f8689755700 2 waiting 51 > 50 ops || 1604302 > 104857600 2014-06-09 16:22:12.018453 7f86a5e11700 10 _finish_op 0x324f4480 seq 3782 osr(0.6d2 0x384c1f0)/0x384c1f0 2014-06-09 16:22:12.018458 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d1260] 2014-06-09 16:22:12.018460 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d1260] 2014-06-09 16:22:12.018460 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042a480] 2014-06-09 16:22:12.018461 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.018462 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.018463 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042a480] 2014-06-09 16:22:12.018465 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.018466 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.018474 7f868b759700 2 waiting 51 > 50 ops || 1604265 > 104857600 2014-06-09 16:22:12.018475 7f868bf5a700 10 op_submit_start 3831 2014-06-09 16:22:12.018477 7f868bf5a700 5 queue_transactions (trailing journal) 3831 2014-06-09 16:22:12.018478 7f868bf5a700 5 queue_op 0x159976e0 seq 3831 osr(1.7a 0x384c510) 826 bytes (queue has 50 ops and 1603439 bytes) 2014-06-09 16:22:12.018480 7f868cf5c700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018482 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301d40 (1 active) 2014-06-09 16:22:12.018484 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301560 (2 active) 2014-06-09 16:22:12.018485 7f868d75d700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018488 7f86a5e11700 5 _do_op 0x11fce9c0 seq 3783 osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 start 2014-06-09 16:22:12.018490 7f868bf5a700 10 op_submit_finish 3831 2014-06-09 16:22:12.018490 7f868ff62700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018491 7f86a5e11700 10 _do_transaction on 0xc0d9680 2014-06-09 16:22:12.018495 7f868af58700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018496 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.018493 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.122( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.122( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.018525 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.018530 7f868bf5a700 5 queue_transactions existing osr(1.122 0x384c890)/0x384c890 2014-06-09 16:22:12.018532 7f868bf5a700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018532 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.018536 7f868df5e700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018545 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3783.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.018517 7f86af56b700 5 -- op tracker -- , seq: 3530, time: 2014-06-09 16:22:12.017723, event: header_read, op: osd_op(client.4541.0:32273 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 319488~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.018550 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.018559 7f86af56b700 5 -- op tracker -- , seq: 3530, time: 2014-06-09 16:22:12.017727, event: throttled, op: osd_op(client.4541.0:32273 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 319488~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.018571 7f86a5e11700 10 _do_op 0x11fce9c0 seq 3783 r = 0, finisher 0xbd76880 0 2014-06-09 16:22:12.018572 7f86af56b700 5 -- op tracker -- , seq: 3530, time: 2014-06-09 16:22:12.018487, event: all_read, op: osd_op(client.4541.0:32273 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 319488~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.018577 7f86a5e11700 10 _finish_op 0x11fce9c0 seq 3783 osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 2014-06-09 16:22:12.018580 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c3180] 2014-06-09 16:22:12.018583 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c3180] 2014-06-09 16:22:12.018584 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.018586 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.018586 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d1360] 2014-06-09 16:22:12.018585 7f86af56b700 5 -- op tracker -- , seq: 3530, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32273 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 319488~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.018591 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d1360] 2014-06-09 16:22:12.018592 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.018593 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.018595 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.018600 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301560 (1 active) 2014-06-09 16:22:12.018602 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89efc0 (2 active) 2014-06-09 16:22:12.018602 7f8689f56700 10 op_submit_start 3832 2014-06-09 16:22:12.018604 7f8689f56700 5 queue_transactions (trailing journal) 3832 2014-06-09 16:22:12.018604 7f8691765700 2 waiting 51 > 50 ops || 1604265 > 104857600 2014-06-09 16:22:12.018605 7f86a5e11700 5 _do_op 0x16fd4300 seq 3784 osr(0.48b 0x2bd7450)/0x2bd7450 start 2014-06-09 16:22:12.018606 7f8689f56700 5 queue_op 0xcd91140 seq 3832 osr(0.186 0x2bd67d0) 828 bytes (queue has 50 ops and 1603439 bytes) 2014-06-09 16:22:12.018607 7f86a5e11700 10 _do_transaction on 0xc33fc80 2014-06-09 16:22:12.018609 7f8689f56700 10 op_submit_finish 3832 2014-06-09 16:22:12.018598 7f869af78700 5 -- op tracker -- , seq: 3526, time: 2014-06-09 16:22:11.966396, event: all_read, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.018611 7f868e75f700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018613 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.018611 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7200 prio 63 cost 516096 latency 0.000888 osd_op(client.4541.0:32273 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 319488~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.018618 7f868ef60700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018615 7f869af78700 5 -- op tracker -- , seq: 3526, time: 2014-06-09 16:22:12.016765, event: dispatched, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.018612 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:12.018623 7f8690f64700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018623 7f869af78700 5 -- op tracker -- , seq: 3526, time: 2014-06-09 16:22:12.018621, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.018627 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76880] 2014-06-09 16:22:12.018628 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.465) v4 from osd.4 2014-06-09 16:22:12.018632 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76880] 2014-06-09 16:22:12.018632 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcda40 2014-06-09 16:22:12.018625 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.018634 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.018635 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.018635 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.018638 7f869af78700 5 -- op tracker -- , seq: 3526, time: 2014-06-09 16:22:12.018636, event: started, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.018643 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.018658 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.018660 7f869af78700 5 -- op tracker -- , seq: 3526, time: 2014-06-09 16:22:12.018657, event: done, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.018672 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.018678 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.018681 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.018675 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:192 2014-06-09 16:22:12.018686 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3784.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.018690 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.018691 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.018695 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.018696 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.018698 7f869af78700 20 osd.26 243 _dispatch 0x3343ca80 pg_info(1 pgs e243:3.4b9) v4 2014-06-09 16:22:12.018703 7f86a5e11700 10 _do_op 0x16fd4300 seq 3784 r = 0, finisher 0x324764c0 0 2014-06-09 16:22:12.018703 7f869af78700 5 -- op tracker -- , seq: 3531, time: 2014-06-09 16:22:11.969764, event: header_read, op: pg_info(1 pgs e243:3.4b9) 2014-06-09 16:22:12.018697 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.21e( v 213'615 (0'0,213'615] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 4 2014-06-09 16:22:12.018709 7f86a5e11700 10 _finish_op 0x16fd4300 seq 3784 osr(0.48b 0x2bd7450)/0x2bd7450 2014-06-09 16:22:12.018711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a79e0] 2014-06-09 16:22:12.018713 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a79e0] 2014-06-09 16:22:12.018714 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.018715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.018715 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89efc0 (1 active) 2014-06-09 16:22:12.018718 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525c20 (2 active) 2014-06-09 16:22:12.018720 7f86a5e11700 5 _do_op 0xbf7f500 seq 3785 osr(3.98 0x3ed0620)/0x3ed0620 start 2014-06-09 16:22:12.018721 7f8691765700 10 op_submit_start 3833 2014-06-09 16:22:12.018711 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.21e( v 213'615 (0'0,213'615] local-les=203 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive NOTIFY] state: got info+log from osd.4 3.21e( v 221'617 (0'0,221'617] local-les=243 n=121 ec=118 les/c 203/203 242/242/202) log((213'615,221'617], crt=0'0) 2014-06-09 16:22:12.018722 7f86a5e11700 10 _do_transaction on 0xc3ed080 2014-06-09 16:22:12.018722 7f8691765700 5 queue_transactions (trailing journal) 3833 2014-06-09 16:22:12.018724 7f8691765700 5 queue_op 0xc34af60 seq 3833 osr(1.d1 0x384c9b0) 826 bytes (queue has 50 ops and 1603437 bytes) 2014-06-09 16:22:12.018724 7f8689f56700 10 merge_log log((213'615,221'617], crt=0'0) from osd.4 into log((0'0,213'615], crt=0'0) 2014-06-09 16:22:12.018727 7f8691765700 10 op_submit_finish 3833 2014-06-09 16:22:12.018728 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.018729 7f8689f56700 10 merge_log extending head to 221'617 2014-06-09 16:22:12.018730 7f8689f56700 20 ? 221'617 (221'616) modify 90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3 by client.4569.0:38869 2014-06-09 10:20:59.295020 2014-06-09 16:22:12.018730 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.112( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.018738 7f8689f56700 20 ? 221'616 (0'0) modify 90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3 by client.4569.0:38868 2014-06-09 10:20:59.267026 2014-06-09 16:22:12.018744 7f8689f56700 20 merge_log 221'616 (0'0) modify 90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3 by client.4569.0:38868 2014-06-09 10:20:59.267026 2014-06-09 16:22:12.018753 7f8689f56700 20 merge_log 221'617 (221'616) modify 90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3 by client.4569.0:38869 2014-06-09 10:20:59.295020 2014-06-09 16:22:12.018764 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.018767 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.018773 7f8691765700 5 queue_transactions existing osr(2.112 0x2ed06d0)/0x2ed06d0 2014-06-09 16:22:12.018776 7f8691765700 2 waiting 51 > 50 ops || 1604265 > 104857600 2014-06-09 16:22:12.018780 7f8690763700 2 waiting 51 > 50 ops || 1604265 > 104857600 2014-06-09 16:22:12.018778 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3785.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.018783 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.018788 7f8692f68700 2 waiting 51 > 50 ops || 1604265 > 104857600 2014-06-09 16:22:12.018789 7f869af78700 5 -- op tracker -- , seq: 3531, time: 2014-06-09 16:22:11.969765, event: throttled, op: pg_info(1 pgs e243:3.4b9) 2014-06-09 16:22:12.018794 7f868a757700 2 waiting 51 > 50 ops || 1604265 > 104857600 2014-06-09 16:22:12.018793 7f86a5e11700 10 _do_op 0xbf7f500 seq 3785 r = 0, finisher 0xcd33920 0 2014-06-09 16:22:12.018799 7f8692767700 2 waiting 51 > 50 ops || 1604263 > 104857600 2014-06-09 16:22:12.018800 7f86a5e11700 10 _finish_op 0xbf7f500 seq 3785 osr(3.98 0x3ed0620)/0x3ed0620 2014-06-09 16:22:12.018805 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324764c0] 2014-06-09 16:22:12.018808 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324764c0] 2014-06-09 16:22:12.018807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd33a20] 2014-06-09 16:22:12.018810 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.018810 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd33a20] 2014-06-09 16:22:12.018811 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.018812 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.018813 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.018819 7f868f761700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018820 7f8691765700 10 op_submit_start 3834 2014-06-09 16:22:12.018821 7f8691765700 5 queue_transactions (trailing journal) 3834 2014-06-09 16:22:12.018823 7f8691765700 5 queue_op 0xc34b260 seq 3834 osr(2.112 0x2ed06d0) 828 bytes (queue has 50 ops and 1603439 bytes) 2014-06-09 16:22:12.018825 7f868d75d700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018826 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525c20 (1 active) 2014-06-09 16:22:12.018829 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcc60 (2 active) 2014-06-09 16:22:12.018831 7f868ff62700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018832 7f86a5e11700 5 _do_op 0xbf7f2c0 seq 3786 osr(0.4c9 0x2bd7650)/0x2bd7650 start 2014-06-09 16:22:12.018835 7f86a5e11700 10 _do_transaction on 0xc0d9200 2014-06-09 16:22:12.018835 7f868af58700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018841 7f8691765700 10 op_submit_finish 3834 2014-06-09 16:22:12.018840 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.018845 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.578( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.018877 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.018877 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.018882 7f8691765700 5 queue_transactions existing osr(0.578 0x2bd7800)/0x2bd7800 2014-06-09 16:22:12.018885 7f8691765700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018890 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3786.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.018816 7f869af78700 5 -- op tracker -- , seq: 3531, time: 2014-06-09 16:22:11.969774, event: all_read, op: pg_info(1 pgs e243:3.4b9) 2014-06-09 16:22:12.018895 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.018898 7f869af78700 5 -- op tracker -- , seq: 3531, time: 2014-06-09 16:22:12.018694, event: dispatched, op: pg_info(1 pgs e243:3.4b9) 2014-06-09 16:22:12.018905 7f86a5e11700 10 _do_op 0xbf7f2c0 seq 3786 r = 0, finisher 0x3062b980 0 2014-06-09 16:22:12.018905 7f869af78700 5 -- op tracker -- , seq: 3531, time: 2014-06-09 16:22:12.018903, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.4b9) 2014-06-09 16:22:12.018911 7f86a5e11700 10 _finish_op 0xbf7f2c0 seq 3786 osr(0.4c9 0x2bd7650)/0x2bd7650 2014-06-09 16:22:12.018913 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd62180] 2014-06-09 16:22:12.018916 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd62180] 2014-06-09 16:22:12.018917 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.018918 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.018919 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd33920] 2014-06-09 16:22:12.018910 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.4b9) v4 from osd.5 2014-06-09 16:22:12.018921 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343ca80 2014-06-09 16:22:12.018923 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd33920] 2014-06-09 16:22:12.018926 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.018927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.018927 7f869af78700 5 -- op tracker -- , seq: 3531, time: 2014-06-09 16:22:12.018925, event: started, op: pg_info(1 pgs e243:3.4b9) 2014-06-09 16:22:12.018934 7f868df5e700 10 op_submit_start 3835 2014-06-09 16:22:12.018936 7f868df5e700 5 queue_transactions (trailing journal) 3835 2014-06-09 16:22:12.018937 7f868df5e700 5 queue_op 0xbf7e480 seq 3835 osr(0.36d 0x2bd6fc0) 828 bytes (queue has 50 ops and 1603439 bytes) 2014-06-09 16:22:12.018941 7f868b759700 2 waiting 51 > 50 ops || 1604265 > 104857600 2014-06-09 16:22:12.018946 7f868c75b700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018946 7f869af78700 5 -- op tracker -- , seq: 3531, time: 2014-06-09 16:22:12.018944, event: done, op: pg_info(1 pgs e243:3.4b9) 2014-06-09 16:22:12.018951 7f868e75f700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018952 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcc60 (1 active) 2014-06-09 16:22:12.018954 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.018955 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b8c0 (2 active) 2014-06-09 16:22:12.018956 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.018958 7f868ef60700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.018959 7f868df5e700 10 op_submit_finish 3835 2014-06-09 16:22:12.018969 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.018971 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.018963 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.4c9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.4c9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.018972 7f869af78700 20 osd.26 243 _dispatch 0x19fdea80 pg_info(1 pgs e243:2.40a) v4 2014-06-09 16:22:12.018978 7f869af78700 5 -- op tracker -- , seq: 3532, time: 2014-06-09 16:22:11.966190, event: header_read, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.018985 7f869af78700 5 -- op tracker -- , seq: 3532, time: 2014-06-09 16:22:11.966190, event: throttled, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.018992 7f869af78700 5 -- op tracker -- , seq: 3532, time: 2014-06-09 16:22:11.966196, event: all_read, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.018995 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.018997 7f869af78700 5 -- op tracker -- , seq: 3532, time: 2014-06-09 16:22:12.018969, event: dispatched, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.019000 7f868df5e700 5 queue_transactions existing osr(0.4c9 0x2bd7650)/0x2bd7650 2014-06-09 16:22:12.019003 7f868df5e700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019007 7f8690f64700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019005 7f869af78700 5 -- op tracker -- , seq: 3532, time: 2014-06-09 16:22:12.019003, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.019010 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.40a) v4 from osd.2 2014-06-09 16:22:12.019010 7f8689f56700 10 merge_log result log((0'0,221'617], crt=221'617) missing(1) changed=1 2014-06-09 16:22:12.019013 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062b980] 2014-06-09 16:22:12.019013 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x19fdea80 2014-06-09 16:22:12.019017 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062b980] 2014-06-09 16:22:12.019018 7f8689755700 2 waiting 51 > 50 ops || 1604300 > 104857600 2014-06-09 16:22:12.019019 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.019020 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.019023 7f86a5e11700 5 _do_op 0xc88b020 seq 3787 osr(3.78a 0x3ed1d80)/0x3ed1d80 start 2014-06-09 16:22:12.019024 7f8690763700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019026 7f86a5e11700 10 _do_transaction on 0xbf39b00 2014-06-09 16:22:12.019016 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 213'615 (0'0,221'617] local-les=203 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=221'617 lcod 0'0 inactive NOTIFY m=1] exit Started/Stray 1.399952 8 0.000293 2014-06-09 16:22:12.019030 7f8692f68700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019019 7f869af78700 5 -- op tracker -- , seq: 3532, time: 2014-06-09 16:22:12.019017, event: started, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.019031 7f86a5e11700 15 _omap_setkeys meta/1fe04edb/pglog_3.78a/0//-1 2014-06-09 16:22:12.019035 7f868a757700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019031 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 213'615 (0'0,221'617] local-les=203 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=221'617 lcod 0'0 inactive NOTIFY m=1] enter Started/ReplicaActive 2014-06-09 16:22:12.019041 7f8692767700 2 waiting 51 > 50 ops || 1604265 > 104857600 2014-06-09 16:22:12.019047 7f868cf5c700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019044 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 213'615 (0'0,221'617] local-les=203 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=221'617 lcod 0'0 inactive NOTIFY m=1] enter Started/ReplicaActive/RepNotRecovering 2014-06-09 16:22:12.019052 7f868d75d700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019056 7f868ff62700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019054 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 213'615 (0'0,221'617] local-les=203 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=221'617 lcod 0'0 inactive NOTIFY m=1] state: In ReplicaActive, about to call activate 2014-06-09 16:22:12.019061 7f868af58700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019066 7f8691765700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019071 7f868f761700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019045 7f869af78700 5 -- op tracker -- , seq: 3532, time: 2014-06-09 16:22:12.019043, event: done, op: pg_info(1 pgs e243:2.40a) 2014-06-09 16:22:12.019078 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019080 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019064 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 213'615 (0'0,221'617] local-les=243 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=221'617 lcod 0'0 inactive m=1] activate - not complete, missing(1) 2014-06-09 16:22:12.019086 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe04edb/pglog_3.78a/0//-1 2014-06-09 16:22:12.019092 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019094 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019095 7f869af78700 20 osd.26 243 _dispatch 0x30fcddc0 pg_info(1 pgs e243:1.438) v4 2014-06-09 16:22:12.019102 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.019100 7f869af78700 5 -- op tracker -- , seq: 3533, time: 2014-06-09 16:22:11.966405, event: header_read, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.019107 7f869af78700 5 -- op tracker -- , seq: 3533, time: 2014-06-09 16:22:11.966405, event: throttled, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.019113 7f869af78700 5 -- op tracker -- , seq: 3533, time: 2014-06-09 16:22:11.966418, event: all_read, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.019120 7f869af78700 5 -- op tracker -- , seq: 3533, time: 2014-06-09 16:22:12.019092, event: dispatched, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.019125 7f869af78700 5 -- op tracker -- , seq: 3533, time: 2014-06-09 16:22:12.019124, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.019128 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.438) v4 from osd.4 2014-06-09 16:22:12.019130 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcddc0 2014-06-09 16:22:12.019135 7f869af78700 5 -- op tracker -- , seq: 3533, time: 2014-06-09 16:22:12.019133, event: started, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.019138 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.019153 7f86a5e11700 15 _omap_rmkeys meta/1fe04edb/pglog_3.78a/0//-1 2014-06-09 16:22:12.019152 7f869af78700 5 -- op tracker -- , seq: 3533, time: 2014-06-09 16:22:12.019150, event: done, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.019159 7f86a5e11700 10 _omap_rmkeys meta/1fe04edb/pglog_3.78a/0//-1 r = 0 2014-06-09 16:22:12.019160 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019163 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019165 7f86a5e11700 15 _omap_setkeys meta/1fe04edb/pglog_3.78a/0//-1 2014-06-09 16:22:12.019174 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019176 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019177 7f86a5e11700 15 _touch 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:12.019177 7f869af78700 20 osd.26 243 _dispatch 0x3343c8c0 pg_info(1 pgs e243:3.792) v4 2014-06-09 16:22:12.019182 7f869af78700 5 -- op tracker -- , seq: 3534, time: 2014-06-09 16:22:11.969778, event: header_read, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.019188 7f869af78700 5 -- op tracker -- , seq: 3534, time: 2014-06-09 16:22:11.969778, event: throttled, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.019195 7f869af78700 5 -- op tracker -- , seq: 3534, time: 2014-06-09 16:22:11.969795, event: all_read, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.019201 7f869af78700 5 -- op tracker -- , seq: 3534, time: 2014-06-09 16:22:12.019174, event: dispatched, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.019201 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 221'616 (0'0,221'617] local-les=243 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=221'617 lcod 0'0 inactive m=1] state: Activate Finished 2014-06-09 16:22:12.019208 7f869af78700 5 -- op tracker -- , seq: 3534, time: 2014-06-09 16:22:12.019206, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.019217 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.792) v4 from osd.5 2014-06-09 16:22:12.019218 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.78a_head oid 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:12.019221 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343c8c0 2014-06-09 16:22:12.019225 7f86af56b700 5 -- op tracker -- , seq: 3535, time: 2014-06-09 16:22:12.018631, event: header_read, op: osd_op(client.4541.0:32274 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 835584~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.019238 7f86af56b700 5 -- op tracker -- , seq: 3535, time: 2014-06-09 16:22:12.018632, event: throttled, op: osd_op(client.4541.0:32274 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 835584~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.019242 7f8689f56700 10 write_log with: dirty_to: 0'0, dirty_from: 213'615, dirty_divergent_priors: 0, writeout_from: 4294967295'18446744073709551615, trimmed: 2014-06-09 16:22:12.019249 7f86a5e11700 20 write_state seq is 30160 2014-06-09 16:22:12.019249 7f86af56b700 5 -- op tracker -- , seq: 3535, time: 2014-06-09 16:22:12.019183, event: all_read, op: osd_op(client.4541.0:32274 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 835584~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.019256 7f86a5e11700 20 set_header setting 30159 cid 3.78a_head oid 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 parent seq 0 2014-06-09 16:22:12.019260 7f86af56b700 5 -- op tracker -- , seq: 3535, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32274 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 835584~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.019267 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.019270 7f86a5e11700 10 _touch 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 = 0 2014-06-09 16:22:12.019269 7f869af78700 5 -- op tracker -- , seq: 3534, time: 2014-06-09 16:22:12.019224, event: started, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.019276 7f86a5e11700 15 _write 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 0~16384 2014-06-09 16:22:12.019280 7f8689f56700 5 queue_transactions existing osr(3.21e 0x3ed0c90)/0x3ed0c90 2014-06-09 16:22:12.019282 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.019283 7f8689f56700 2 waiting 51 > 50 ops || 1605754 > 104857600 2014-06-09 16:22:12.019281 7f86af56b700 15 osd.26 243 enqueue_op 0x32827e00 prio 63 cost 516096 latency 0.000650 osd_op(client.4541.0:32274 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 835584~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.019288 7f8691f66700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019288 7f869af78700 5 -- op tracker -- , seq: 3534, time: 2014-06-09 16:22:12.019286, event: done, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.019291 7f868bf5a700 2 waiting 51 > 50 ops || 1604267 > 104857600 2014-06-09 16:22:12.019296 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019298 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019310 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019311 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019313 7f869af78700 20 osd.26 243 _dispatch 0x30c77340 pg_info(1 pgs e243:3.1b5) v4 2014-06-09 16:22:12.019318 7f869af78700 5 -- op tracker -- , seq: 3536, time: 2014-06-09 16:22:11.966198, event: header_read, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.019329 7f869af78700 5 -- op tracker -- , seq: 3536, time: 2014-06-09 16:22:11.966198, event: throttled, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.019336 7f869af78700 5 -- op tracker -- , seq: 3536, time: 2014-06-09 16:22:11.966204, event: all_read, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.019343 7f86a5e11700 10 _generic_write 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 0~16384 = 0 2014-06-09 16:22:12.019348 7f869af78700 5 -- op tracker -- , seq: 3536, time: 2014-06-09 16:22:12.019309, event: dispatched, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.019353 7f86a5e11700 15 _setattrs 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:12.019356 7f869af78700 5 -- op tracker -- , seq: 3536, time: 2014-06-09 16:22:12.019354, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.019361 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1b5) v4 from osd.2 2014-06-09 16:22:12.019362 7f86a5e11700 10 _setattrs 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 = 0 2014-06-09 16:22:12.019364 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c77340 2014-06-09 16:22:12.019369 7f86a5e11700 15 _setattrs 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 2014-06-09 16:22:12.019369 7f869af78700 5 -- op tracker -- , seq: 3536, time: 2014-06-09 16:22:12.019367, event: started, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.019377 7f86a5e11700 10 _setattrs 3.78a_head/22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 = 0 2014-06-09 16:22:12.019384 7f86a5e11700 10 cid: oid: 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 not skipping op, *spos 3787.0.0 2014-06-09 16:22:12.019387 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.019391 7f86a5e11700 20 set_header setting 30159 cid 3.78a_head oid 22b8b78a/rbd_data.11756b8b4567.0000000000001593/head//3 parent seq 0 2014-06-09 16:22:12.019387 7f869af78700 5 -- op tracker -- , seq: 3536, time: 2014-06-09 16:22:12.019385, event: done, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.019400 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019402 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019404 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3787.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.019409 7f86a5e11700 10 cid: oid: 1fe04edb/pglog_3.78a/0//-1 skipping op, spos 3787.0.0 <= header.spos 149453.0.0 2014-06-09 16:22:12.019412 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.019418 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019420 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019421 7f869af78700 20 osd.26 243 _dispatch 0x12c08e00 pg_info(1 pgs e243:3.1b1) v4 2014-06-09 16:22:12.019425 7f869af78700 5 -- op tracker -- , seq: 3537, time: 2014-06-09 16:22:11.966427, event: header_read, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.019432 7f869af78700 5 -- op tracker -- , seq: 3537, time: 2014-06-09 16:22:11.966428, event: throttled, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.019439 7f869af78700 5 -- op tracker -- , seq: 3537, time: 2014-06-09 16:22:11.966438, event: all_read, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.019446 7f869af78700 5 -- op tracker -- , seq: 3537, time: 2014-06-09 16:22:12.019418, event: dispatched, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.019454 7f869af78700 5 -- op tracker -- , seq: 3537, time: 2014-06-09 16:22:12.019451, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.019458 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1b1) v4 from osd.4 2014-06-09 16:22:12.019460 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12c08e00 2014-06-09 16:22:12.019465 7f869af78700 5 -- op tracker -- , seq: 3537, time: 2014-06-09 16:22:12.019463, event: started, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.019483 7f869af78700 5 -- op tracker -- , seq: 3537, time: 2014-06-09 16:22:12.019480, event: done, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.019492 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019494 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019506 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019508 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019509 7f869af78700 20 osd.26 243 _dispatch 0x3343c700 pg_info(1 pgs e243:2.4a7) v4 2014-06-09 16:22:12.019515 7f869af78700 5 -- op tracker -- , seq: 3538, time: 2014-06-09 16:22:11.969802, event: header_read, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.019518 7f86a5e11700 10 _do_op 0xc88b020 seq 3787 r = 0, finisher 0x31791a80 0x30875200 2014-06-09 16:22:12.019521 7f869af78700 5 -- op tracker -- , seq: 3538, time: 2014-06-09 16:22:11.969802, event: throttled, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.019525 7f86a5e11700 10 _finish_op 0xc88b020 seq 3787 osr(3.78a 0x3ed1d80)/0x3ed1d80 2014-06-09 16:22:12.019526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31791f60] 2014-06-09 16:22:12.019527 7f869af78700 5 -- op tracker -- , seq: 3538, time: 2014-06-09 16:22:11.969817, event: all_read, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.019532 7f869af78700 5 -- op tracker -- , seq: 3538, time: 2014-06-09 16:22:12.019506, event: dispatched, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.019538 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b8c0 (1 active) 2014-06-09 16:22:12.019538 7f869af78700 5 -- op tracker -- , seq: 3538, time: 2014-06-09 16:22:12.019537, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.019541 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1e60 (2 active) 2014-06-09 16:22:12.019530 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 luod=213'545 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 21 2014-06-09 16:22:12.019542 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4a7) v4 from osd.5 2014-06-09 16:22:12.019543 7f86a5e11700 5 _do_op 0x30d97aa0 seq 3788 osr(1.3cb 0x384d4f0)/0x384d4f0 start 2014-06-09 16:22:12.019545 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343c700 2014-06-09 16:22:12.019546 7f86a5e11700 10 _do_transaction on 0xbffe840 2014-06-09 16:22:12.019552 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.019552 7f86a460e700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:12.019545, event: op_commit, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.019561 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 luod=213'545 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 21 all committed 2014-06-09 16:22:12.019563 7f869af78700 5 -- op tracker -- , seq: 3538, time: 2014-06-09 16:22:12.019547, event: started, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.019572 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] eval_repop repgather(0xc20e300 243'546 rep_tid=21 committed?=1 applied?=0 lock=0 op=osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.019585 7f869af78700 5 -- op tracker -- , seq: 3538, time: 2014-06-09 16:22:12.019583, event: done, op: pg_info(1 pgs e243:2.4a7) 2014-06-09 16:22:12.019593 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019594 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019593 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.019589 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] log_op_stats osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4 inb 16768 outb 0 rlat 0.000000 lat 1.658914 2014-06-09 16:22:12.019605 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019607 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019608 7f869af78700 20 osd.26 243 _dispatch 0x30e108c0 pg_info(1 pgs e243:0.763) v4 2014-06-09 16:22:12.019609 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3788.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.019636 7f868e75f700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019607 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] publish_stats_to_osd 243:873 2014-06-09 16:22:12.019637 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31791a80] 2014-06-09 16:22:12.019644 7f868ef60700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019613 7f869af78700 5 -- op tracker -- , seq: 3539, time: 2014-06-09 16:22:11.966209, event: header_read, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.019634 7f868c75b700 10 op_submit_start 3836 2014-06-09 16:22:12.019652 7f868c75b700 5 queue_transactions (trailing journal) 3836 2014-06-09 16:22:12.019654 7f868c75b700 5 queue_op 0x30032960 seq 3836 osr(0.3ab 0x2bd72f0) 828 bytes (queue has 50 ops and 1585485 bytes) 2014-06-09 16:22:12.019657 7f868c75b700 10 op_submit_finish 3836 2014-06-09 16:22:12.019642 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] sending commit on repgather(0xc20e300 243'546 rep_tid=21 committed?=1 applied?=0 lock=0 op=osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4) 0x257c7900 2014-06-09 16:22:12.019661 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.6e( v 214'612 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:12.019681 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.3 activated and committed 2014-06-09 16:22:12.019694 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.019705 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.019719 7f868df5e700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019723 7f868c75b700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.019725 7f8690f64700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019730 7f8689755700 2 waiting 51 > 50 ops || 1586346 > 104857600 2014-06-09 16:22:12.019735 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.019739 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1018 2014-06-09 16:22:12.019753 7f86a5e11700 10 _do_op 0x30d97aa0 seq 3788 r = 0, finisher 0x3062aa80 0 2014-06-09 16:22:12.019757 7f86a5e11700 10 _finish_op 0x30d97aa0 seq 3788 osr(1.3cb 0x384d4f0)/0x384d4f0 2014-06-09 16:22:12.019758 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.019765 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1e60 (1 active) 2014-06-09 16:22:12.019768 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3faea0 (2 active) 2014-06-09 16:22:12.019770 7f86a5e11700 5 _do_op 0xc88a3c0 seq 3789 osr(2.216 0x2ed0b50)/0x2ed0b50 start 2014-06-09 16:22:12.019769 7f8692f68700 10 op_submit_start 3837 2014-06-09 16:22:12.019772 7f86a5e11700 10 _do_transaction on 0xc1392c0 2014-06-09 16:22:12.019772 7f8692f68700 5 queue_transactions (trailing journal) 3837 2014-06-09 16:22:12.019763 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.350( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.019775 7f8692f68700 5 queue_op 0x324e6000 seq 3837 osr(2.6b0 0x2ed1ed0) 828 bytes (queue has 50 ops and 1585485 bytes) 2014-06-09 16:22:12.019778 7f8692f68700 10 op_submit_finish 3837 2014-06-09 16:22:12.019778 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.019782 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1a4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.1a4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.019799 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.019798 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.019804 7f868c75b700 5 queue_transactions existing osr(2.350 0x2ed0e50)/0x2ed0e50 2014-06-09 16:22:12.019807 7f868c75b700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019812 7f8690763700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019815 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3780.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.019818 7f8692767700 2 waiting 51 > 50 ops || 1586311 > 104857600 2014-06-09 16:22:12.019818 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.019820 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.019823 7f868cf5c700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019824 7f8692f68700 5 queue_transactions existing osr(0.1a4 0x2bd6710)/0x2bd6710 2014-06-09 16:22:12.019827 7f8692f68700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019834 7f868ff62700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019839 7f868af58700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019840 7f86a460e700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:12.019678, event: commit_sent, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.019846 7f8691765700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019854 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31791f60] 2014-06-09 16:22:12.019856 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe67840] 2014-06-09 16:22:12.019858 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe67840] 2014-06-09 16:22:12.019854 7f869af78700 5 -- op tracker -- , seq: 3539, time: 2014-06-09 16:22:11.966210, event: throttled, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.019859 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.019860 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.019864 7f868f761700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019863 7f869af78700 5 -- op tracker -- , seq: 3539, time: 2014-06-09 16:22:11.966217, event: all_read, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.019870 7f8689f56700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.019873 7f869af78700 5 -- op tracker -- , seq: 3539, time: 2014-06-09 16:22:12.019605, event: dispatched, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.019876 7f8691f66700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019880 7f868bf5a700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019870 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] op_applied: 21 2014-06-09 16:22:12.019880 7f869af78700 5 -- op tracker -- , seq: 3539, time: 2014-06-09 16:22:12.019878, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.019886 7f868b759700 2 waiting 51 > 50 ops || 1586311 > 104857600 2014-06-09 16:22:12.019891 7f868ef60700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.019887 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.763) v4 from osd.2 2014-06-09 16:22:12.019898 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e108c0 2014-06-09 16:22:12.019898 7f86a4e0f700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:12.019884, event: op_applied, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.019910 7f869af78700 5 -- op tracker -- , seq: 3539, time: 2014-06-09 16:22:12.019901, event: started, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.019907 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] op_applied on primary on version 243'546 2014-06-09 16:22:12.019918 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] repop_all_applied: repop tid 21 all applied 2014-06-09 16:22:12.019926 7f869af78700 5 -- op tracker -- , seq: 3539, time: 2014-06-09 16:22:12.019925, event: done, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.019939 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019941 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019928 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] eval_repop repgather(0xc20e300 243'546 rep_tid=21 committed?=1 applied?=1 lock=0 op=osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.019954 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.019956 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.019944 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] log_op_stats osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4 inb 16768 outb 0 rlat 0.000000 lat 1.659269 2014-06-09 16:22:12.019957 7f869af78700 20 osd.26 243 _dispatch 0x30d9e380 pg_info(1 pgs e243:3.50d) v4 2014-06-09 16:22:12.019963 7f869af78700 5 -- op tracker -- , seq: 3540, time: 2014-06-09 16:22:11.974203, event: header_read, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.019960 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] publish_stats_to_osd 243:874 2014-06-09 16:22:12.019971 7f869af78700 5 -- op tracker -- , seq: 3540, time: 2014-06-09 16:22:11.974204, event: throttled, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.019972 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.019978 7f869af78700 5 -- op tracker -- , seq: 3540, time: 2014-06-09 16:22:11.974216, event: all_read, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.019985 7f869af78700 5 -- op tracker -- , seq: 3540, time: 2014-06-09 16:22:12.019954, event: dispatched, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.019982 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] removing repgather(0xc20e300 243'546 rep_tid=21 committed?=1 applied?=1 op=osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.019993 7f869af78700 5 -- op tracker -- , seq: 3540, time: 2014-06-09 16:22:12.019990, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.019998 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.50d) v4 from osd.4 2014-06-09 16:22:12.020001 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9e380 2014-06-09 16:22:12.020006 7f869af78700 5 -- op tracker -- , seq: 3540, time: 2014-06-09 16:22:12.020004, event: started, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.019997 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] q front is repgather(0xc20e300 243'546 rep_tid=21 committed?=1 applied?=1 op=osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.020012 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] remove_repop repgather(0xc20e300 243'546 rep_tid=21 committed?=1 applied?=1 op=osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.020023 7f869af78700 5 -- op tracker -- , seq: 3540, time: 2014-06-09 16:22:12.020021, event: done, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.020031 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.020033 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.020026 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] requeue_ops 2014-06-09 16:22:12.020045 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.020046 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.020047 7f869af78700 20 osd.26 243 _dispatch 0x3343c540 pg_info(1 pgs e243:3.792) v4 2014-06-09 16:22:12.020053 7f86a4e0f700 5 -- op tracker -- , seq: 284, time: 2014-06-09 16:22:12.020048, event: done, op: osd_op(client.4565.0:32907 rbd_data.11756b8b4567.0000000000001593 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.22b8b78a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.020080 7f869af78700 5 -- op tracker -- , seq: 3541, time: 2014-06-09 16:22:11.969882, event: header_read, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.020085 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31791a80] 2014-06-09 16:22:12.020087 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062aa80] 2014-06-09 16:22:12.020086 7f869af78700 5 -- op tracker -- , seq: 3541, time: 2014-06-09 16:22:11.969883, event: throttled, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.020090 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062aa80] 2014-06-09 16:22:12.020092 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.020093 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.020092 7f869af78700 5 -- op tracker -- , seq: 3541, time: 2014-06-09 16:22:11.969895, event: all_read, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.020096 7f868df5e700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.020101 7f8690f64700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.020100 7f869af78700 5 -- op tracker -- , seq: 3541, time: 2014-06-09 16:22:12.020045, event: dispatched, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.020105 7f8689755700 2 waiting 51 > 50 ops || 1586346 > 104857600 2014-06-09 16:22:12.020110 7f868e75f700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.020107 7f869af78700 5 -- op tracker -- , seq: 3541, time: 2014-06-09 16:22:12.020105, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.020115 7f868d75d700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.020112 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.792) v4 from osd.5 2014-06-09 16:22:12.020121 7f868a757700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.020122 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3343c540 2014-06-09 16:22:12.020128 7f869af78700 5 -- op tracker -- , seq: 3541, time: 2014-06-09 16:22:12.020126, event: started, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.020135 7f86a5610700 10 _do_op 0xd327500 seq 3780 r = 0, finisher 0xc2d7f20 0 2014-06-09 16:22:12.020143 7f86a5610700 10 _finish_op 0xd327500 seq 3780 osr(1.3cd 0x384d4b0)/0x384d4b0 2014-06-09 16:22:12.020143 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251db20] 2014-06-09 16:22:12.020146 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251db20] 2014-06-09 16:22:12.020147 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.020148 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.020151 7f868c75b700 10 op_submit_start 3838 2014-06-09 16:22:12.024725 7f868c75b700 5 queue_transactions (trailing journal) 3838 2014-06-09 16:22:12.024729 7f868c75b700 5 queue_op 0x24104120 seq 3838 osr(2.350 0x2ed0e50) 828 bytes (queue has 50 ops and 1585485 bytes) 2014-06-09 16:22:12.020150 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1d40 (1 active) 2014-06-09 16:22:12.024737 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652900 (2 active) 2014-06-09 16:22:12.020152 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d7f20] 2014-06-09 16:22:12.024744 7f86a5610700 5 _do_op 0xc88a720 seq 3790 osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 start 2014-06-09 16:22:12.024748 7f86a5610700 10 _do_transaction on 0x35bfbc0 2014-06-09 16:22:12.024752 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d7f20] 2014-06-09 16:22:12.024756 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.024755 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.024760 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.020152 7f8690763700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024771 7f8692767700 2 waiting 51 > 50 ops || 1586311 > 104857600 2014-06-09 16:22:12.024776 7f868cf5c700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024783 7f8692f68700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024788 7f868ff62700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024788 7f868c75b700 10 op_submit_finish 3838 2014-06-09 16:22:12.020181 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.024795 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.024830 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3789.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.024835 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.024835 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.024843 7f868c75b700 5 queue_transactions existing osr(3.1f0 0x3ed0990)/0x3ed0990 2014-06-09 16:22:12.024846 7f868c75b700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024851 7f868af58700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024856 7f86a5e11700 10 _do_op 0xc88a3c0 seq 3789 r = 0, finisher 0x3045a400 0 2014-06-09 16:22:12.024868 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074780] 2014-06-09 16:22:12.024870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074780] 2014-06-09 16:22:12.024871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.024872 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.024876 7f86a5e11700 10 _finish_op 0xc88a3c0 seq 3789 osr(2.216 0x2ed0b50)/0x2ed0b50 2014-06-09 16:22:12.024888 7f868f761700 10 op_submit_start 3839 2014-06-09 16:22:12.024890 7f868f761700 5 queue_transactions (trailing journal) 3839 2014-06-09 16:22:12.024892 7f868f761700 5 queue_op 0x33a3860 seq 3839 osr(0.6e7 0x384c1b0) 828 bytes (queue has 50 ops and 1585485 bytes) 2014-06-09 16:22:12.024891 7f8689f56700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.024899 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3faea0 (1 active) 2014-06-09 16:22:12.024901 7f8691f66700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024902 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524b40 (2 active) 2014-06-09 16:22:12.024907 7f868f761700 10 op_submit_finish 3839 2014-06-09 16:22:12.024907 7f868bf5a700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024911 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.6f5( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:12.024948 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.024954 7f868f761700 5 queue_transactions existing osr(1.6f5 0x2ed0170)/0x2ed0170 2014-06-09 16:22:12.024956 7f868f761700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024961 7f868b759700 2 waiting 51 > 50 ops || 1586311 > 104857600 2014-06-09 16:22:12.024967 7f868ef60700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024973 7f868df5e700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024978 7f8690f64700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024983 7f8689755700 2 waiting 51 > 50 ops || 1586346 > 104857600 2014-06-09 16:22:12.024989 7f868e75f700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024994 7f868d75d700 2 waiting 51 > 50 ops || 1586313 > 104857600 2014-06-09 16:22:12.024999 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3045a400] 2014-06-09 16:22:12.025006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3045a400] 2014-06-09 16:22:12.025008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.025009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.025012 7f86a5e11700 5 _do_op 0xcabcba0 seq 3791 osr(1.2c0 0x384d170)/0x384d170 start 2014-06-09 16:22:12.025015 7f86a5e11700 10 _do_transaction on 0x331f8c0 2014-06-09 16:22:12.025021 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.025066 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.025082 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3791.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.025086 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.025099 7f86a5e11700 10 _do_op 0xcabcba0 seq 3791 r = 0, finisher 0xc5a7500 0 2014-06-09 16:22:12.025109 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006d1a0] 2014-06-09 16:22:12.025111 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006d1a0] 2014-06-09 16:22:12.025113 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.025114 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.025117 7f86a5e11700 10 _finish_op 0xcabcba0 seq 3791 osr(1.2c0 0x384d170)/0x384d170 2014-06-09 16:22:12.025127 7f8689f56700 10 op_submit_start 3840 2014-06-09 16:22:12.025129 7f8692767700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025130 7f8689f56700 5 queue_transactions (trailing journal) 3840 2014-06-09 16:22:12.025133 7f868cf5c700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025132 7f8689f56700 5 queue_op 0x16fd5e00 seq 3840 osr(3.21e 0x3ed0c90) 2315 bytes (queue has 50 ops and 1586972 bytes) 2014-06-09 16:22:12.025137 7f8692f68700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025142 7f868ff62700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025146 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a7500] 2014-06-09 16:22:12.025147 7f868c75b700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025153 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a7500] 2014-06-09 16:22:12.025154 7f868af58700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025155 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.025156 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.025159 7f8690763700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025125 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524b40 (1 active) 2014-06-09 16:22:12.025165 7f8691f66700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025165 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338b40 (2 active) 2014-06-09 16:22:12.025171 7f86a5e11700 5 _do_op 0xc2eade0 seq 3792 osr(2.69d 0x2ed1ef0)/0x2ed1ef0 start 2014-06-09 16:22:12.025173 7f8689f56700 10 op_submit_finish 3840 2014-06-09 16:22:12.025172 7f868bf5a700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025174 7f86a5e11700 10 _do_transaction on 0xc291980 2014-06-09 16:22:12.025179 7f868f761700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025183 7f868b759700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025178 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:12.025188 7f868ef60700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025191 7f868df5e700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025195 7f8690f64700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025200 7f8689755700 2 waiting 51 > 50 ops || 1587833 > 104857600 2014-06-09 16:22:12.025203 7f868e75f700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025207 7f868d75d700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025208 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.025214 7f8689f56700 5 queue_transactions existing osr(3.461 0x3ed1120)/0x3ed1120 2014-06-09 16:22:12.025215 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.025216 7f868a757700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025222 7f8689f56700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025253 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.025267 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3792.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.025271 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.025283 7f86a5e11700 10 _do_op 0xc2eade0 seq 3792 r = 0, finisher 0x3062a020 0 2014-06-09 16:22:12.025293 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d6e60] 2014-06-09 16:22:12.025295 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d6e60] 2014-06-09 16:22:12.025297 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.025298 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.025301 7f86a5e11700 10 _finish_op 0xc2eade0 seq 3792 osr(2.69d 0x2ed1ef0)/0x2ed1ef0 2014-06-09 16:22:12.025311 7f868cf5c700 10 op_submit_start 3841 2014-06-09 16:22:12.025313 7f868cf5c700 5 queue_transactions (trailing journal) 3841 2014-06-09 16:22:12.025314 7f868cf5c700 5 queue_op 0xcabc9c0 seq 3841 osr(2.6ec 0x2ed1e10) 828 bytes (queue has 50 ops and 1586972 bytes) 2014-06-09 16:22:12.025319 7f8692f68700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025323 7f868ff62700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025330 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338b40 (1 active) 2014-06-09 16:22:12.025333 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2865a0 (2 active) 2014-06-09 16:22:12.025337 7f868cf5c700 10 op_submit_finish 3841 2014-06-09 16:22:12.025338 7f868c75b700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025345 7f868af58700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025341 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.530( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.530( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.025375 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.025381 7f868cf5c700 5 queue_transactions existing osr(2.530 0x2ed19b0)/0x2ed19b0 2014-06-09 16:22:12.025383 7f868cf5c700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025387 7f8690763700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025393 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062a020] 2014-06-09 16:22:12.025394 7f8691f66700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062a020] 2014-06-09 16:22:12.025401 7f868bf5a700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.025403 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.025406 7f868f761700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025411 7f868b759700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025415 7f868ef60700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025419 7f868df5e700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025423 7f8690f64700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025427 7f8689755700 2 waiting 51 > 50 ops || 1587833 > 104857600 2014-06-09 16:22:12.025431 7f868e75f700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025435 7f868d75d700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025407 7f86a5e11700 5 _do_op 0x16fd4360 seq 3793 osr(0.33f 0x2bd7060)/0x2bd7060 start 2014-06-09 16:22:12.025440 7f868a757700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025441 7f86a5e11700 10 _do_transaction on 0xbf0e300 2014-06-09 16:22:12.025446 7f8689f56700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025447 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.025452 7f8692767700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025492 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.025506 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3793.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.025510 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.025534 7f86a5e11700 10 _do_op 0x16fd4360 seq 3793 r = 0, finisher 0x33a6ec0 0 2014-06-09 16:22:12.025545 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6e960] 2014-06-09 16:22:12.025547 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6e960] 2014-06-09 16:22:12.025548 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.025549 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.025552 7f86a5e11700 10 _finish_op 0x16fd4360 seq 3793 osr(0.33f 0x2bd7060)/0x2bd7060 2014-06-09 16:22:12.025563 7f868ff62700 10 op_submit_start 3842 2014-06-09 16:22:12.025564 7f868ff62700 5 queue_transactions (trailing journal) 3842 2014-06-09 16:22:12.025566 7f868ff62700 5 queue_op 0x30a30e0 seq 3842 osr(2.26a 0x2ed0dd0) 828 bytes (queue has 50 ops and 1586972 bytes) 2014-06-09 16:22:12.025573 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2865a0 (1 active) 2014-06-09 16:22:12.025572 7f868c75b700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025575 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9227e0 (2 active) 2014-06-09 16:22:12.025579 7f868ff62700 10 op_submit_finish 3842 2014-06-09 16:22:12.025579 7f868af58700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025583 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.c6( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.025616 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.025621 7f868ff62700 5 queue_transactions existing osr(1.c6 0x384c610)/0x384c610 2014-06-09 16:22:12.025624 7f868ff62700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025628 7f868cf5c700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025632 7f8690763700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025637 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a6ec0] 2014-06-09 16:22:12.025638 7f8691f66700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025640 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a6ec0] 2014-06-09 16:22:12.025641 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.025642 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.025644 7f868bf5a700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025646 7f86a5e11700 5 _do_op 0x30d97980 seq 3794 osr(0.4a7 0x2bd7370)/0x2bd7370 start 2014-06-09 16:22:12.025649 7f86a5e11700 10 _do_transaction on 0x3145380 2014-06-09 16:22:12.025649 7f868f761700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025654 7f868b759700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025659 7f868ef60700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025663 7f868df5e700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025667 7f8690f64700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025671 7f8689755700 2 waiting 51 > 50 ops || 1587833 > 104857600 2014-06-09 16:22:12.025675 7f868e75f700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025679 7f868d75d700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025684 7f868a757700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025686 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.025690 7f8689f56700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025695 7f8692767700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025701 7f8692f68700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025735 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.025750 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3794.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.025754 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.025767 7f86a5e11700 10 _do_op 0x30d97980 seq 3794 r = 0, finisher 0xbd5af00 0 2014-06-09 16:22:12.025777 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6fba0] 2014-06-09 16:22:12.025779 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6fba0] 2014-06-09 16:22:12.025780 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.025781 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.025785 7f86a5e11700 10 _finish_op 0x30d97980 seq 3794 osr(0.4a7 0x2bd7370)/0x2bd7370 2014-06-09 16:22:12.025795 7f868af58700 10 op_submit_start 3843 2014-06-09 16:22:12.025797 7f868af58700 5 queue_transactions (trailing journal) 3843 2014-06-09 16:22:12.025798 7f868af58700 5 queue_op 0x30d97920 seq 3843 osr(1.27c 0x384cf30) 828 bytes (queue has 50 ops and 1586972 bytes) 2014-06-09 16:22:12.025803 7f868ff62700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025807 7f868cf5c700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025811 7f8690763700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025817 7f8691f66700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025818 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9227e0 (1 active) 2014-06-09 16:22:12.025820 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286fc0 (2 active) 2014-06-09 16:22:12.025822 7f868bf5a700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025824 7f868af58700 10 op_submit_finish 3843 2014-06-09 16:22:12.025827 7f868f761700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025832 7f868b759700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025836 7f868ef60700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025828 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:12.025841 7f868df5e700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025846 7f8690f64700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025850 7f8689755700 2 waiting 51 > 50 ops || 1587833 > 104857600 2014-06-09 16:22:12.025855 7f868e75f700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025860 7f868d75d700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025864 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.025864 7f868a757700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025869 7f868af58700 5 queue_transactions existing osr(3.53d 0x3ed1600)/0x3ed1600 2014-06-09 16:22:12.025869 7f8689f56700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025873 7f868af58700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025879 7f8692767700 2 waiting 51 > 50 ops || 1587798 > 104857600 2014-06-09 16:22:12.025883 7f8692f68700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025886 7f868c75b700 2 waiting 51 > 50 ops || 1587800 > 104857600 2014-06-09 16:22:12.025890 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5af00] 2014-06-09 16:22:12.025893 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5af00] 2014-06-09 16:22:12.025894 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.025895 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.025898 7f86a5e11700 5 _do_op 0x16fd5b60 seq 3795 osr(0.3e 0x2bd6270)/0x2bd6270 start 2014-06-09 16:22:12.025902 7f86a5e11700 10 _do_transaction on 0xbfe40c0 2014-06-09 16:22:12.025908 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.025945 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.025961 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3795.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.025965 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.025977 7f86a5e11700 10 _do_op 0x16fd5b60 seq 3795 r = 0, finisher 0x327f2cc0 0 2014-06-09 16:22:12.025987 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x327f2880] 2014-06-09 16:22:12.025990 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.026002 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.026013 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.3e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.026023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x327f2880] 2014-06-09 16:22:12.026024 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.026025 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.026029 7f86a5e11700 10 _finish_op 0x16fd5b60 seq 3795 osr(0.3e 0x2bd6270)/0x2bd6270 2014-06-09 16:22:12.026039 7f868cf5c700 10 op_submit_start 3844 2014-06-09 16:22:12.026041 7f868cf5c700 5 queue_transactions (trailing journal) 3844 2014-06-09 16:22:12.026042 7f868cf5c700 5 queue_op 0xc931680 seq 3844 osr(2.530 0x2ed19b0) 828 bytes (queue has 50 ops and 1586942 bytes) 2014-06-09 16:22:12.026047 7f8690763700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026052 7f8691f66700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026053 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286fc0 (1 active) 2014-06-09 16:22:12.026056 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525d40 (2 active) 2014-06-09 16:22:12.026058 7f868bf5a700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026060 7f868cf5c700 10 op_submit_finish 3844 2014-06-09 16:22:12.026062 7f868f761700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026067 7f868b759700 2 waiting 51 > 50 ops || 1587768 > 104857600 2014-06-09 16:22:12.026072 7f868ef60700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026064 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.57a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.57a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.026076 7f868df5e700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026081 7f8690f64700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026085 7f8689755700 2 waiting 51 > 50 ops || 1587803 > 104857600 2014-06-09 16:22:12.026090 7f868e75f700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026094 7f868d75d700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026097 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.026098 7f868a757700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026102 7f868cf5c700 5 queue_transactions existing osr(2.57a 0x2ed1870)/0x2ed1870 2014-06-09 16:22:12.026104 7f8689f56700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026107 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x327f2cc0] 2014-06-09 16:22:12.026110 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x327f2cc0] 2014-06-09 16:22:12.026112 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.026113 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.026116 7f868af58700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026120 7f8692767700 2 waiting 51 > 50 ops || 1587768 > 104857600 2014-06-09 16:22:12.026124 7f8692f68700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026128 7f868c75b700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026132 7f868ff62700 2 waiting 51 > 50 ops || 1587768 > 104857600 2014-06-09 16:22:12.026136 7f868cf5c700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026140 7f86a5e11700 5 _do_op 0x30607800 seq 3796 osr(1.62a 0x384dfd0)/0x384dfd0 start 2014-06-09 16:22:12.026143 7f86a5e11700 10 _do_transaction on 0xc05e480 2014-06-09 16:22:12.026149 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.026186 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.026188 7f8691765700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.020145 7f869af78700 5 -- op tracker -- , seq: 3541, time: 2014-06-09 16:22:12.020142, event: done, op: pg_info(1 pgs e243:3.792) 2014-06-09 16:22:12.026205 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026209 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026207 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3796.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.026212 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.026227 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026227 7f86a5e11700 10 _do_op 0x30607800 seq 3796 r = 0, finisher 0xc411540 0 2014-06-09 16:22:12.026229 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026231 7f869af78700 20 osd.26 243 _dispatch 0x30e10000 pg_info(1 pgs e243:3.1b5) v4 2014-06-09 16:22:12.026237 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2e0ff40] 2014-06-09 16:22:12.026237 7f869af78700 5 -- op tracker -- , seq: 3542, time: 2014-06-09 16:22:11.966221, event: header_read, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.026240 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.026246 7f869af78700 5 -- op tracker -- , seq: 3542, time: 2014-06-09 16:22:11.966221, event: throttled, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.026253 7f869af78700 5 -- op tracker -- , seq: 3542, time: 2014-06-09 16:22:11.966227, event: all_read, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.026249 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.026259 7f869af78700 5 -- op tracker -- , seq: 3542, time: 2014-06-09 16:22:12.026226, event: dispatched, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.026267 7f869af78700 5 -- op tracker -- , seq: 3542, time: 2014-06-09 16:22:12.026265, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.026271 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1b5) v4 from osd.2 2014-06-09 16:22:12.026273 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e10000 2014-06-09 16:22:12.026278 7f869af78700 5 -- op tracker -- , seq: 3542, time: 2014-06-09 16:22:12.026276, event: started, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.026259 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.62a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:187 2014-06-09 16:22:12.026291 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2e0ff40] 2014-06-09 16:22:12.026293 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.026294 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.026297 7f86a5e11700 10 _finish_op 0x30607800 seq 3796 osr(1.62a 0x384dfd0)/0x384dfd0 2014-06-09 16:22:12.026298 7f869af78700 5 -- op tracker -- , seq: 3542, time: 2014-06-09 16:22:12.026296, event: done, op: pg_info(1 pgs e243:3.1b5) 2014-06-09 16:22:12.026308 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525d40 (1 active) 2014-06-09 16:22:12.026310 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026311 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d17a0 (2 active) 2014-06-09 16:22:12.026312 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026313 7f86a5e11700 5 _do_op 0x30d96c00 seq 3797 osr(1.70c 0x2ed00f0)/0x2ed00f0 start 2014-06-09 16:22:12.026315 7f86a5e11700 10 _do_transaction on 0xc2c78c0 2014-06-09 16:22:12.026321 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.026321 7f8691f66700 10 op_submit_start 3845 2014-06-09 16:22:12.026324 7f8691f66700 5 queue_transactions (trailing journal) 3845 2014-06-09 16:22:12.026325 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026327 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026326 7f8691f66700 5 queue_op 0xcabdbc0 seq 3845 osr(1.1fa 0x384cd50) 828 bytes (queue has 50 ops and 1586909 bytes) 2014-06-09 16:22:12.026328 7f869af78700 20 osd.26 243 _dispatch 0x31777880 pg_info(1 pgs e243:3.11f) v4 2014-06-09 16:22:12.026330 7f8691f66700 10 op_submit_finish 3845 2014-06-09 16:22:12.026333 7f869af78700 5 -- op tracker -- , seq: 3543, time: 2014-06-09 16:22:11.974325, event: header_read, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.026340 7f869af78700 5 -- op tracker -- , seq: 3543, time: 2014-06-09 16:22:11.974326, event: throttled, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.026334 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.198( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.198( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.026347 7f869af78700 5 -- op tracker -- , seq: 3543, time: 2014-06-09 16:22:11.974339, event: all_read, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.026353 7f869af78700 5 -- op tracker -- , seq: 3543, time: 2014-06-09 16:22:12.026325, event: dispatched, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.026358 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.026361 7f869af78700 5 -- op tracker -- , seq: 3543, time: 2014-06-09 16:22:12.026360, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.026366 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.11f) v4 from osd.4 2014-06-09 16:22:12.026370 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31777880 2014-06-09 16:22:12.026372 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.026373 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3797.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.026377 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.026375 7f869af78700 5 -- op tracker -- , seq: 3543, time: 2014-06-09 16:22:12.026373, event: started, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.026377 7f8691f66700 5 queue_transactions existing osr(1.198 0x384cb30)/0x384cb30 2014-06-09 16:22:12.026380 7f8691f66700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026385 7f868bf5a700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026389 7f86a5e11700 10 _do_op 0x30d96c00 seq 3797 r = 0, finisher 0xc6e8f40 0 2014-06-09 16:22:12.026390 7f868f761700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026394 7f868b759700 2 waiting 51 > 50 ops || 1587735 > 104857600 2014-06-09 16:22:12.026398 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc411540] 2014-06-09 16:22:12.026398 7f868ef60700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc411540] 2014-06-09 16:22:12.026403 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.026403 7f868df5e700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026404 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.026407 7f8690f64700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026409 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c37e0] 2014-06-09 16:22:12.026411 7f8689755700 2 waiting 51 > 50 ops || 1587770 > 104857600 2014-06-09 16:22:12.026416 7f868e75f700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026411 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.026420 7f868d75d700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026424 7f868a757700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026421 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.026430 7f8689f56700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026430 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.70c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:181 2014-06-09 16:22:12.026391 7f869af78700 5 -- op tracker -- , seq: 3543, time: 2014-06-09 16:22:12.026389, event: done, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.026442 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c37e0] 2014-06-09 16:22:12.026443 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.026444 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.026447 7f868af58700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026447 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026450 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026452 7f8692767700 2 waiting 51 > 50 ops || 1587735 > 104857600 2014-06-09 16:22:12.026456 7f8692f68700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026464 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026466 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026467 7f869af78700 20 osd.26 243 _dispatch 0x309e7180 pg_info(1 pgs e243:0.763) v4 2014-06-09 16:22:12.026470 7f867bb63700 5 -- op tracker -- , seq: 3544, time: 2014-06-09 16:22:12.022013, event: header_read, op: osd_op(client.4575.0:32702 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1032192] 3.c0524973 ack+ondisk+write e242) 2014-06-09 16:22:12.026487 7f868c75b700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026492 7f868ff62700 2 waiting 51 > 50 ops || 1587735 > 104857600 2014-06-09 16:22:12.026484 7f867bb63700 5 -- op tracker -- , seq: 3544, time: 2014-06-09 16:22:12.022015, event: throttled, op: osd_op(client.4575.0:32702 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1032192] 3.c0524973 ack+ondisk+write e242) 2014-06-09 16:22:12.026496 7f868cf5c700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026497 7f86a5e11700 10 _finish_op 0x30d96c00 seq 3797 osr(1.70c 0x2ed00f0)/0x2ed00f0 2014-06-09 16:22:12.026510 7f8690763700 10 op_submit_start 3846 2014-06-09 16:22:12.026507 7f867bb63700 5 -- op tracker -- , seq: 3544, time: 2014-06-09 16:22:12.024635, event: all_read, op: osd_op(client.4575.0:32702 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1032192] 3.c0524973 ack+ondisk+write e242) 2014-06-09 16:22:12.026512 7f8690763700 5 queue_transactions (trailing journal) 3846 2014-06-09 16:22:12.026512 7f8691765700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026513 7f8690763700 5 queue_op 0x2e21b00 seq 3846 osr(3.5b3 0x3ed17e0) 828 bytes (queue has 50 ops and 1586876 bytes) 2014-06-09 16:22:12.026518 7f867bb63700 5 -- op tracker -- , seq: 3544, time: 0.000000, event: dispatched, op: osd_op(client.4575.0:32702 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1032192] 3.c0524973 ack+ondisk+write e242) 2014-06-09 16:22:12.026525 7f867bb63700 20 osd.26 243 should_share_map client.4575 172.16.96.220:0/1127679 242 2014-06-09 16:22:12.026530 7f867bb63700 20 osd.26 243 client session last_sent_epoch: 0 versus osdmap epoch 243 2014-06-09 16:22:12.026535 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d17a0 (1 active) 2014-06-09 16:22:12.026536 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8f40] 2014-06-09 16:22:12.026538 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd440 (2 active) 2014-06-09 16:22:12.026542 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8f40] 2014-06-09 16:22:12.026543 7f8690763700 10 op_submit_finish 3846 2014-06-09 16:22:12.026544 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.026546 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.026547 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.182( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.026552 7f867bb63700 15 osd.26 243 enqueue_op 0x2c06a900 prio 63 cost 1032192 latency 0.004539 osd_op(client.4575.0:32702 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 1904640~1032192] 3.c0524973 ack+ondisk+write e242) v4 2014-06-09 16:22:12.026578 7f868bf5a700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026580 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.026584 7f8691f66700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026586 7f8690763700 5 queue_transactions existing osr(2.182 0x2ed0990)/0x2ed0990 2014-06-09 16:22:12.026589 7f868b759700 2 waiting 51 > 50 ops || 1587702 > 104857600 2014-06-09 16:22:12.026594 7f86a5e11700 5 _do_op 0x32502d20 seq 3798 osr(2.34d 0x2ed0e70)/0x2ed0e70 start 2014-06-09 16:22:12.026596 7f868ef60700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026598 7f86a5e11700 10 _do_transaction on 0xc14ba40 2014-06-09 16:22:12.026601 7f868df5e700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026603 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.026606 7f8690f64700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026610 7f8689755700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.026614 7f868e75f700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026619 7f868d75d700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026624 7f868a757700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026629 7f8689f56700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026635 7f868af58700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026638 7f869af78700 5 -- op tracker -- , seq: 3546, time: 2014-06-09 16:22:11.966229, event: header_read, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.026641 7f8692767700 2 waiting 51 > 50 ops || 1587702 > 104857600 2014-06-09 16:22:12.026643 7f869af78700 5 -- op tracker -- , seq: 3546, time: 2014-06-09 16:22:11.966230, event: throttled, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.026646 7f8692f68700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026646 7f869af78700 5 -- op tracker -- , seq: 3546, time: 2014-06-09 16:22:11.966236, event: all_read, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.026650 7f868c75b700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026650 7f869af78700 5 -- op tracker -- , seq: 3546, time: 2014-06-09 16:22:12.026464, event: dispatched, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.026654 7f868ff62700 2 waiting 51 > 50 ops || 1587702 > 104857600 2014-06-09 16:22:12.026654 7f869af78700 5 -- op tracker -- , seq: 3546, time: 2014-06-09 16:22:12.026653, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.026657 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.763) v4 from osd.2 2014-06-09 16:22:12.026659 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x309e7180 2014-06-09 16:22:12.026660 7f868cf5c700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026667 7f8690763700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026673 7f868f761700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.026677 7f86af56b700 5 -- op tracker -- , seq: 3545, time: 2014-06-09 16:22:12.019299, event: header_read, op: osd_op(client.4541.0:32275 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 1351680~1032192] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.026691 7f86af56b700 5 -- op tracker -- , seq: 3545, time: 2014-06-09 16:22:12.019300, event: throttled, op: osd_op(client.4541.0:32275 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 1351680~1032192] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.026704 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.026704 7f86af56b700 5 -- op tracker -- , seq: 3545, time: 2014-06-09 16:22:12.020421, event: all_read, op: osd_op(client.4541.0:32275 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 1351680~1032192] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.026716 7f86af56b700 5 -- op tracker -- , seq: 3545, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32275 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 1351680~1032192] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.026721 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3790.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.026725 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.026724 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.026725 7f869af78700 5 -- op tracker -- , seq: 3546, time: 2014-06-09 16:22:12.026661, event: started, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.026745 7f869af78700 5 -- op tracker -- , seq: 3546, time: 2014-06-09 16:22:12.026742, event: done, op: pg_info(1 pgs e243:0.763) 2014-06-09 16:22:12.026743 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7f00 prio 63 cost 1032192 latency 0.007444 osd_op(client.4541.0:32275 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 1351680~1032192] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.026753 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026756 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026769 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026770 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026772 7f869af78700 20 osd.26 243 _dispatch 0x3130b880 pg_info(1 pgs e243:3.50d) v4 2014-06-09 16:22:12.026778 7f869af78700 5 -- op tracker -- , seq: 3547, time: 2014-06-09 16:22:11.974349, event: header_read, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.026785 7f869af78700 5 -- op tracker -- , seq: 3547, time: 2014-06-09 16:22:11.974350, event: throttled, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.026791 7f869af78700 5 -- op tracker -- , seq: 3547, time: 2014-06-09 16:22:11.974361, event: all_read, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.026797 7f869af78700 5 -- op tracker -- , seq: 3547, time: 2014-06-09 16:22:12.026768, event: dispatched, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.026805 7f869af78700 5 -- op tracker -- , seq: 3547, time: 2014-06-09 16:22:12.026803, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.026815 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.50d) v4 from osd.4 2014-06-09 16:22:12.026818 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3130b880 2014-06-09 16:22:12.026824 7f869af78700 5 -- op tracker -- , seq: 3547, time: 2014-06-09 16:22:12.026821, event: started, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.026848 7f869af78700 5 -- op tracker -- , seq: 3547, time: 2014-06-09 16:22:12.026846, event: done, op: pg_info(1 pgs e243:3.50d) 2014-06-09 16:22:12.026858 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026860 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026874 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026876 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026878 7f869af78700 20 osd.26 243 _dispatch 0x30d9e8c0 pg_info(1 pgs e243:3.408) v4 2014-06-09 16:22:12.026884 7f869af78700 5 -- op tracker -- , seq: 3548, time: 2014-06-09 16:22:11.966761, event: header_read, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:12.026891 7f869af78700 5 -- op tracker -- , seq: 3548, time: 2014-06-09 16:22:11.966762, event: throttled, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:12.026904 7f869af78700 5 -- op tracker -- , seq: 3548, time: 2014-06-09 16:22:11.966778, event: all_read, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:12.026911 7f869af78700 5 -- op tracker -- , seq: 3548, time: 2014-06-09 16:22:12.026874, event: dispatched, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:12.026921 7f869af78700 5 -- op tracker -- , seq: 3548, time: 2014-06-09 16:22:12.026918, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:12.026925 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.408) v4 from osd.2 2014-06-09 16:22:12.026928 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30d9e8c0 2014-06-09 16:22:12.026933 7f869af78700 5 -- op tracker -- , seq: 3548, time: 2014-06-09 16:22:12.026931, event: started, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:12.026952 7f869af78700 5 -- op tracker -- , seq: 3548, time: 2014-06-09 16:22:12.026949, event: done, op: pg_info(1 pgs e243:3.408) 2014-06-09 16:22:12.026959 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026961 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026976 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.026978 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.026979 7f869af78700 20 osd.26 243 _dispatch 0x26001a40 pg_info(1 pgs e243:3.465) v4 2014-06-09 16:22:12.026984 7f869af78700 5 -- op tracker -- , seq: 3549, time: 2014-06-09 16:22:11.974433, event: header_read, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.026991 7f869af78700 5 -- op tracker -- , seq: 3549, time: 2014-06-09 16:22:11.974434, event: throttled, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.027003 7f869af78700 5 -- op tracker -- , seq: 3549, time: 2014-06-09 16:22:11.974442, event: all_read, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.027010 7f869af78700 5 -- op tracker -- , seq: 3549, time: 2014-06-09 16:22:12.026976, event: dispatched, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.027021 7f869af78700 5 -- op tracker -- , seq: 3549, time: 2014-06-09 16:22:12.027018, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.027029 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.465) v4 from osd.4 2014-06-09 16:22:12.027032 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x26001a40 2014-06-09 16:22:12.027037 7f869af78700 5 -- op tracker -- , seq: 3549, time: 2014-06-09 16:22:12.027035, event: started, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.027057 7f869af78700 5 -- op tracker -- , seq: 3549, time: 2014-06-09 16:22:12.027055, event: done, op: pg_info(1 pgs e243:3.465) 2014-06-09 16:22:12.027065 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.027067 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.027079 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.027081 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.027082 7f869af78700 20 osd.26 243 _dispatch 0x30c12000 pg_info(1 pgs e243:0.179) v4 2014-06-09 16:22:12.027087 7f869af78700 5 -- op tracker -- , seq: 3550, time: 2014-06-09 16:22:11.966831, event: header_read, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.027100 7f869af78700 5 -- op tracker -- , seq: 3550, time: 2014-06-09 16:22:11.966832, event: throttled, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.027106 7f869af78700 5 -- op tracker -- , seq: 3550, time: 2014-06-09 16:22:11.966843, event: all_read, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.027117 7f869af78700 5 -- op tracker -- , seq: 3550, time: 2014-06-09 16:22:12.027079, event: dispatched, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.027128 7f869af78700 5 -- op tracker -- , seq: 3550, time: 2014-06-09 16:22:12.027125, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.027133 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.179) v4 from osd.2 2014-06-09 16:22:12.027136 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c12000 2014-06-09 16:22:12.027141 7f869af78700 5 -- op tracker -- , seq: 3550, time: 2014-06-09 16:22:12.027139, event: started, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.027157 7f869af78700 5 -- op tracker -- , seq: 3550, time: 2014-06-09 16:22:12.027155, event: done, op: pg_info(1 pgs e243:0.179) 2014-06-09 16:22:12.027164 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.027166 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.027177 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.027178 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.027179 7f869af78700 20 osd.26 243 _dispatch 0x1ac748c0 pg_info(1 pgs e243:3.2a9) v4 2014-06-09 16:22:12.027184 7f869af78700 5 -- op tracker -- , seq: 3551, time: 2014-06-09 16:22:11.974447, event: header_read, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.027190 7f869af78700 5 -- op tracker -- , seq: 3551, time: 2014-06-09 16:22:11.974447, event: throttled, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.027196 7f869af78700 5 -- op tracker -- , seq: 3551, time: 2014-06-09 16:22:11.974454, event: all_read, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.027201 7f869af78700 5 -- op tracker -- , seq: 3551, time: 2014-06-09 16:22:12.027176, event: dispatched, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.027208 7f869af78700 5 -- op tracker -- , seq: 3551, time: 2014-06-09 16:22:12.027206, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.027213 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2a9) v4 from osd.4 2014-06-09 16:22:12.027215 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x1ac748c0 2014-06-09 16:22:12.027220 7f869af78700 5 -- op tracker -- , seq: 3551, time: 2014-06-09 16:22:12.027218, event: started, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.027232 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.027248 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3798.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.027253 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.027250 7f86af56b700 5 -- op tracker -- , seq: 3552, time: 2014-06-09 16:22:12.026764, event: header_read, op: osd_op(client.4541.0:32276 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2383872~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.027259 7f86af56b700 5 -- op tracker -- , seq: 3552, time: 2014-06-09 16:22:12.026765, event: throttled, op: osd_op(client.4541.0:32276 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2383872~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.027267 7f86a5e11700 10 _do_op 0x32502d20 seq 3798 r = 0, finisher 0x12051740 0 2014-06-09 16:22:12.027266 7f86af56b700 5 -- op tracker -- , seq: 3552, time: 2014-06-09 16:22:12.027228, event: all_read, op: osd_op(client.4541.0:32276 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2383872~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.027273 7f86af56b700 5 -- op tracker -- , seq: 3552, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32276 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2383872~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.027278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31bff880] 2014-06-09 16:22:12.027278 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.027280 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31bff880] 2014-06-09 16:22:12.027281 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.027282 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.027281 7f869af78700 5 -- op tracker -- , seq: 3551, time: 2014-06-09 16:22:12.027247, event: done, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.027285 7f86a5e11700 10 _finish_op 0x32502d20 seq 3798 osr(2.34d 0x2ed0e70)/0x2ed0e70 2014-06-09 16:22:12.027290 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.027292 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.027294 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd440 (1 active) 2014-06-09 16:22:12.027296 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x12051740] 2014-06-09 16:22:12.027296 7f8691765700 10 op_submit_start 3847 2014-06-09 16:22:12.027297 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286120 (2 active) 2014-06-09 16:22:12.027298 7f8691765700 5 queue_transactions (trailing journal) 3847 2014-06-09 16:22:12.027298 7f868bf5a700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027300 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x12051740] 2014-06-09 16:22:12.027299 7f86a5e11700 5 _do_op 0x30a2780 seq 3799 osr(3.37a 0x3ed1040)/0x3ed1040 start 2014-06-09 16:22:12.027302 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.027300 7f8691765700 5 queue_op 0xc34b8c0 seq 3847 osr(0.578 0x2bd7800) 828 bytes (queue has 50 ops and 1586876 bytes) 2014-06-09 16:22:12.027302 7f86a5e11700 10 _do_transaction on 0xc05acc0 2014-06-09 16:22:12.027303 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.027303 7f8691765700 10 op_submit_finish 3847 2014-06-09 16:22:12.027304 7f8691f66700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027308 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.027309 7f868b759700 2 waiting 51 > 50 ops || 1587702 > 104857600 2014-06-09 16:22:12.027288 7f86af56b700 15 osd.26 243 enqueue_op 0xdfa7a00 prio 63 cost 516096 latency 0.000524 osd_op(client.4541.0:32276 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2383872~516096] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.027314 7f868ef60700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027307 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.71b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.027319 7f868df5e700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027324 7f8690f64700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027329 7f8689755700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.027334 7f868e75f700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027339 7f868d75d700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027344 7f868a757700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027344 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.027346 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.027351 7f8689f56700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027353 7f8691765700 5 queue_transactions existing osr(0.71b 0x384c050)/0x384c050 2014-06-09 16:22:12.027356 7f8691765700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027356 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.027358 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.027360 7f869af78700 20 osd.26 243 _dispatch 0x30c12540 pg_info(1 pgs e243:3.342) v4 2014-06-09 16:22:12.027363 7f8692767700 2 waiting 51 > 50 ops || 1587702 > 104857600 2014-06-09 16:22:12.027369 7f8692f68700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027365 7f869af78700 5 -- op tracker -- , seq: 3553, time: 2014-06-09 16:22:11.966855, event: header_read, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.027374 7f868c75b700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027378 7f868ff62700 2 waiting 51 > 50 ops || 1587702 > 104857600 2014-06-09 16:22:12.027376 7f869af78700 5 -- op tracker -- , seq: 3553, time: 2014-06-09 16:22:11.966855, event: throttled, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.027358 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3799.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.027384 7f868cf5c700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027383 7f869af78700 5 -- op tracker -- , seq: 3553, time: 2014-06-09 16:22:11.966868, event: all_read, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.027386 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.027392 7f8690763700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027390 7f869af78700 5 -- op tracker -- , seq: 3553, time: 2014-06-09 16:22:12.027356, event: dispatched, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.027398 7f868f761700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027400 7f86a5e11700 10 _do_op 0x30a2780 seq 3799 r = 0, finisher 0x189ab420 0 2014-06-09 16:22:12.027404 7f868af58700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027406 7f869af78700 5 -- op tracker -- , seq: 3553, time: 2014-06-09 16:22:12.027404, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.027410 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31457ea0] 2014-06-09 16:22:12.027412 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31457ea0] 2014-06-09 16:22:12.027411 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.342) v4 from osd.2 2014-06-09 16:22:12.027414 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.027415 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.027414 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c12540 2014-06-09 16:22:12.027418 7f86a5e11700 10 _finish_op 0x30a2780 seq 3799 osr(3.37a 0x3ed1040)/0x3ed1040 2014-06-09 16:22:12.027429 7f868bf5a700 10 op_submit_start 3848 2014-06-09 16:22:12.027428 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286120 (1 active) 2014-06-09 16:22:12.027430 7f868bf5a700 5 queue_transactions (trailing journal) 3848 2014-06-09 16:22:12.027431 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89efc0 (2 active) 2014-06-09 16:22:12.027431 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189ab420] 2014-06-09 16:22:12.027432 7f868bf5a700 5 queue_op 0x159978c0 seq 3848 osr(1.122 0x384c890) 828 bytes (queue has 50 ops and 1586876 bytes) 2014-06-09 16:22:12.027434 7f86a5e11700 5 _do_op 0xc34afc0 seq 3800 osr(0.48b 0x2bd7450)/0x2bd7450 start 2014-06-09 16:22:12.027436 7f868bf5a700 10 op_submit_finish 3848 2014-06-09 16:22:12.027437 7f86a5e11700 10 _do_transaction on 0xbfcd080 2014-06-09 16:22:12.027437 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189ab420] 2014-06-09 16:22:12.027439 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.027440 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.027442 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.027472 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41980 (19 active) 2014-06-09 16:22:12.027482 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.027484 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.027496 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3800.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.027500 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.027490 7f868bf5a700 10 osd.26 243 dequeue_op 0x13022000 prio 63 cost 950272 latency 1.791404 osd_op(client.4570.0:33836 rbd_data.117a6b8b4567.00000000000016c1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~950272] 3.ccce9a24 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.027513 7f86a5e11700 10 _do_op 0xc34afc0 seq 3800 r = 0, finisher 0xbd78aa0 0 2014-06-09 16:22:12.027515 7f8691f66700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027523 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x19941960] 2014-06-09 16:22:12.027522 7f868b759700 2 waiting 51 > 50 ops || 1587702 > 104857600 2014-06-09 16:22:12.027420 7f869af78700 5 -- op tracker -- , seq: 3553, time: 2014-06-09 16:22:12.027418, event: started, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.027525 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x19941960] 2014-06-09 16:22:12.027527 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.027527 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.027530 7f868ef60700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027531 7f86a5e11700 10 _finish_op 0xc34afc0 seq 3800 osr(0.48b 0x2bd7450)/0x2bd7450 2014-06-09 16:22:12.027542 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89efc0 (1 active) 2014-06-09 16:22:12.027545 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922ea0 (2 active) 2014-06-09 16:22:12.027545 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd78aa0] 2014-06-09 16:22:12.027543 7f868bf5a700 5 -- op tracker -- , seq: 179, time: 2014-06-09 16:22:12.027512, event: reached_pg, op: osd_op(client.4570.0:33836 rbd_data.117a6b8b4567.00000000000016c1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~950272] 3.ccce9a24 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.027547 7f86a5e11700 5 _do_op 0xbf7e4e0 seq 3801 osr(1.15b 0x384cbf0)/0x384cbf0 start 2014-06-09 16:22:12.027550 7f86a5e11700 10 _do_transaction on 0x2ed5800 2014-06-09 16:22:12.027553 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd78aa0] 2014-06-09 16:22:12.027555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.027555 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.027557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.027552 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.027568 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x13022000 2014-06-09 16:22:12.027591 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.027579 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4570.0:33836 rbd_data.117a6b8b4567.00000000000016c1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~950272] 3.ccce9a24 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.027604 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3801.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.027608 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.027600 7f868bf5a700 3 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4570.0:33836 was 214'735 2014-06-09 16:22:12.027636 7f8691f66700 10 op_submit_start 3849 2014-06-09 16:22:12.027638 7f8691f66700 5 queue_transactions (trailing journal) 3849 2014-06-09 16:22:12.027640 7f86a5e11700 10 _do_op 0xbf7e4e0 seq 3801 r = 0, finisher 0x2d739840 0 2014-06-09 16:22:12.027640 7f8691f66700 5 queue_op 0xcabd260 seq 3849 osr(1.198 0x384cb30) 828 bytes (queue has 50 ops and 1586876 bytes) 2014-06-09 16:22:12.027643 7f8691f66700 10 op_submit_finish 3849 2014-06-09 16:22:12.027646 7f86a5e11700 10 _finish_op 0xbf7e4e0 seq 3801 osr(1.15b 0x384cbf0)/0x384cbf0 2014-06-09 16:22:12.027648 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x306cf000] 2014-06-09 16:22:12.027651 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x306cf000] 2014-06-09 16:22:12.027651 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922ea0 (1 active) 2014-06-09 16:22:12.027653 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.027654 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.027654 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc7e0 (2 active) 2014-06-09 16:22:12.027653 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d739840] 2014-06-09 16:22:12.027656 7f86a5e11700 5 _do_op 0xc2ea9c0 seq 3802 osr(3.2fd 0x3ed0de0)/0x3ed0de0 start 2014-06-09 16:22:12.027658 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d739840] 2014-06-09 16:22:12.027658 7f86a5e11700 10 _do_transaction on 0xbfcc0c0 2014-06-09 16:22:12.027647 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:12.027660 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.027660 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.027664 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.027685 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.027691 7f8691f66700 5 queue_transactions existing osr(3.27b 0x3ed0bd0)/0x3ed0bd0 2014-06-09 16:22:12.027694 7f8691f66700 10 op_submit_start 3850 2014-06-09 16:22:12.027695 7f8691f66700 5 queue_transactions (trailing journal) 3850 2014-06-09 16:22:12.027696 7f8691f66700 5 queue_op 0xc930840 seq 3850 osr(3.27b 0x3ed0bd0) 828 bytes (queue has 50 ops and 1586876 bytes) 2014-06-09 16:22:12.027699 7f8691f66700 10 op_submit_finish 3850 2014-06-09 16:22:12.027701 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.027702 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.5b7( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.5b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.027715 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3802.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.027719 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.027729 7f86a5e11700 10 _do_op 0xc2ea9c0 seq 3802 r = 0, finisher 0x3281ad40 0 2014-06-09 16:22:12.027734 7f86a5e11700 10 _finish_op 0xc2ea9c0 seq 3802 osr(3.2fd 0x3ed0de0)/0x3ed0de0 2014-06-09 16:22:12.027736 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdbb820] 2014-06-09 16:22:12.027738 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdbb820] 2014-06-09 16:22:12.027739 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.027739 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc7e0 (1 active) 2014-06-09 16:22:12.027740 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.027741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.027741 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281ad40] 2014-06-09 16:22:12.027742 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3005a0 (2 active) 2014-06-09 16:22:12.027744 7f86a5e11700 5 _do_op 0x324e6300 seq 3803 osr(3.6f 0x3ed0280)/0x3ed0280 start 2014-06-09 16:22:12.027744 7f8691f66700 5 queue_transactions existing osr(2.5b7 0x2ed1b70)/0x2ed1b70 2014-06-09 16:22:12.027747 7f86a5e11700 10 _do_transaction on 0xc134cc0 2014-06-09 16:22:12.027747 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281ad40] 2014-06-09 16:22:12.027747 7f8691f66700 10 op_submit_start 3851 2014-06-09 16:22:12.027748 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.027749 7f8691f66700 5 queue_transactions (trailing journal) 3851 2014-06-09 16:22:12.027749 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.027750 7f8691f66700 5 queue_op 0xc931620 seq 3851 osr(2.5b7 0x2ed1b70) 828 bytes (queue has 50 ops and 1586876 bytes) 2014-06-09 16:22:12.027751 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.027753 7f8691f66700 10 op_submit_finish 3851 2014-06-09 16:22:12.027756 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.646( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.646( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.027787 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.027786 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.027791 7f8691f66700 5 queue_transactions existing osr(2.646 0x2ed1c30)/0x2ed1c30 2014-06-09 16:22:12.027794 7f8691f66700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027798 7f8690f64700 2 waiting 51 > 50 ops || 1587704 > 104857600 2014-06-09 16:22:12.027799 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3803.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.027803 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.027803 7f8689755700 2 waiting 51 > 50 ops || 1587737 > 104857600 2014-06-09 16:22:12.027815 7f86a5e11700 10 _do_op 0x324e6300 seq 3803 r = 0, finisher 0xbd5f480 0 2014-06-09 16:22:12.027821 7f86a5e11700 10 _finish_op 0x324e6300 seq 3803 osr(3.6f 0x3ed0280)/0x3ed0280 2014-06-09 16:22:12.027823 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5f420] 2014-06-09 16:22:12.027824 7f868bf5a700 10 osd.26 243 dequeue_op 0x13022000 finish 2014-06-09 16:22:12.027826 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5f420] 2014-06-09 16:22:12.027827 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.027829 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.027828 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3005a0 (1 active) 2014-06-09 16:22:12.027830 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa5a0 (2 active) 2014-06-09 16:22:12.027832 7f86a5e11700 5 _do_op 0x159969c0 seq 3804 osr(0.644 0x2bd7bf0)/0x2bd7bf0 start 2014-06-09 16:22:12.027835 7f86a5e11700 10 _do_transaction on 0xc3fd200 2014-06-09 16:22:12.027834 7f868bf5a700 5 -- op tracker -- , seq: 179, time: 2014-06-09 16:22:12.027828, event: done, op: osd_op(client.4570.0:33836 rbd_data.117a6b8b4567.00000000000016c1 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~950272] 3.ccce9a24 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.027840 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.027854 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.027860 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5f480] 2014-06-09 16:22:12.027863 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5f480] 2014-06-09 16:22:12.027864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.027864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.027874 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.027887 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3804.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.027891 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.027894 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) 2014-06-09 16:22:12.027900 7f86a5e11700 10 _do_op 0x159969c0 seq 3804 r = 0, finisher 0xbd5d6e0 0 2014-06-09 16:22:12.027906 7f86a5e11700 10 _finish_op 0x159969c0 seq 3804 osr(0.644 0x2bd7bf0)/0x2bd7bf0 2014-06-09 16:22:12.027907 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5d7c0] 2014-06-09 16:22:12.027909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5d7c0] 2014-06-09 16:22:12.027910 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.027910 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.027898 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.117( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.117( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.027911 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa5a0 (1 active) 2014-06-09 16:22:12.027913 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5d6e0] 2014-06-09 16:22:12.027914 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652360 (2 active) 2014-06-09 16:22:12.027917 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5d6e0] 2014-06-09 16:22:12.027916 7f86a5e11700 5 _do_op 0x30d97560 seq 3805 osr(2.28b 0x2ed0cd0)/0x2ed0cd0 start 2014-06-09 16:22:12.027918 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.027919 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.027919 7f86a5e11700 10 _do_transaction on 0x344c6c0 2014-06-09 16:22:12.027925 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.027934 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.027939 7f868bf5a700 5 queue_transactions existing osr(0.117 0x2bd66a0)/0x2bd66a0 2014-06-09 16:22:12.027942 7f868bf5a700 10 op_submit_start 3852 2014-06-09 16:22:12.027943 7f868bf5a700 5 queue_transactions (trailing journal) 3852 2014-06-09 16:22:12.027945 7f868bf5a700 5 queue_op 0x15996780 seq 3852 osr(0.117 0x2bd66a0) 828 bytes (queue has 49 ops and 1586050 bytes) 2014-06-09 16:22:12.027947 7f868bf5a700 10 op_submit_finish 3852 2014-06-09 16:22:12.027950 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.67c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.67c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.027960 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.027975 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3805.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.027980 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.027982 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.027986 7f868bf5a700 5 queue_transactions existing osr(2.67c 0x2ed1f90)/0x2ed1f90 2014-06-09 16:22:12.027988 7f868bf5a700 10 op_submit_start 3853 2014-06-09 16:22:12.027989 7f868bf5a700 5 queue_transactions (trailing journal) 3853 2014-06-09 16:22:12.027991 7f868bf5a700 5 queue_op 0x15996660 seq 3853 osr(2.67c 0x2ed1f90) 828 bytes (queue has 50 ops and 1586878 bytes) 2014-06-09 16:22:12.027993 7f868bf5a700 10 op_submit_finish 3853 2014-06-09 16:22:12.027993 7f86a5e11700 10 _do_op 0x30d97560 seq 3805 r = 0, finisher 0xbd5ae60 0 2014-06-09 16:22:12.027999 7f86a5e11700 10 _finish_op 0x30d97560 seq 3805 osr(2.28b 0x2ed0cd0)/0x2ed0cd0 2014-06-09 16:22:12.028000 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3fc20] 2014-06-09 16:22:12.028004 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652360 (1 active) 2014-06-09 16:22:12.028006 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd9e0 (2 active) 2014-06-09 16:22:12.027996 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.139( v 213'571 (0'0,213'571] local-les=243 n=113 ec=118 les/c 243/243 242/242/205) 2014-06-09 16:22:12.028008 7f86a5e11700 5 _do_op 0xc2d5f80 seq 3806 osr(2.5b6 0x2ed1b90)/0x2ed1b90 start 2014-06-09 16:22:12.028011 7f86a5e11700 10 _do_transaction on 0xbf42540 2014-06-09 16:22:12.028004 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.028015 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.028015 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.028006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ae60] 2014-06-09 16:22:12.028026 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ae60] 2014-06-09 16:22:12.028028 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.028029 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.028030 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.028025 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.28b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:12.028034 7f868bf5a700 5 queue_transactions existing osr(3.139 0x3ed0780)/0x3ed0780 2014-06-09 16:22:12.028035 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3fc20] 2014-06-09 16:22:12.028037 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.028037 7f868bf5a700 10 op_submit_start 3854 2014-06-09 16:22:12.028038 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.028039 7f868bf5a700 5 queue_transactions (trailing journal) 3854 2014-06-09 16:22:12.028040 7f868bf5a700 5 queue_op 0x15996a80 seq 3854 osr(3.139 0x3ed0780) 828 bytes (queue has 50 ops and 1586845 bytes) 2014-06-09 16:22:12.028043 7f868bf5a700 10 op_submit_finish 3854 2014-06-09 16:22:12.028051 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.028046 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2b6( v 213'590 (0'0,213'590] local-les=243 n=115 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:12.028064 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3806.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.028068 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.028079 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.028083 7f868bf5a700 5 queue_transactions existing osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 2014-06-09 16:22:12.028084 7f86a5e11700 10 _do_op 0xc2d5f80 seq 3806 r = 0, finisher 0xc2d7380 0 2014-06-09 16:22:12.028086 7f868bf5a700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028089 7f86a5e11700 10 _finish_op 0xc2d5f80 seq 3806 osr(2.5b6 0x2ed1b90)/0x2ed1b90 2014-06-09 16:22:12.028091 7f868e75f700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028092 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f12a0] 2014-06-09 16:22:12.028095 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f12a0] 2014-06-09 16:22:12.028097 7f8691f66700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028099 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.028101 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.028101 7f868a757700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.027535 7f86a5610700 10 _do_op 0xc88a720 seq 3790 r = 0, finisher 0xcd0fb00 0 2014-06-09 16:22:12.028107 7f868d75d700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028113 7f8691765700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028121 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074740] 2014-06-09 16:22:12.028123 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074740] 2014-06-09 16:22:12.028125 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.028126 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.028125 7f8692767700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028131 7f8692f68700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028128 7f869af78700 5 -- op tracker -- , seq: 3553, time: 2014-06-09 16:22:12.028126, event: done, op: pg_info(1 pgs e243:3.342) 2014-06-09 16:22:12.028137 7f868c75b700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028138 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028142 7f868ff62700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028148 7f868cf5c700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028154 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028155 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028157 7f869af78700 20 osd.26 243 _dispatch 0x314748c0 pg_info(1 pgs e243:1.4c9) v4 2014-06-09 16:22:12.028162 7f869af78700 5 -- op tracker -- , seq: 3554, time: 2014-06-09 16:22:11.974458, event: header_read, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028169 7f869af78700 5 -- op tracker -- , seq: 3554, time: 2014-06-09 16:22:11.974459, event: throttled, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028182 7f869af78700 5 -- op tracker -- , seq: 3554, time: 2014-06-09 16:22:11.974468, event: all_read, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028191 7f8690763700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028197 7f868f761700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028188 7f869af78700 5 -- op tracker -- , seq: 3554, time: 2014-06-09 16:22:12.028153, event: dispatched, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028204 7f868af58700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028207 7f869af78700 5 -- op tracker -- , seq: 3554, time: 2014-06-09 16:22:12.028205, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028211 7f868b759700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028217 7f868ef60700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028223 7f868df5e700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028227 7f8690f64700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028212 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4c9) v4 from osd.4 2014-06-09 16:22:12.028233 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x314748c0 2014-06-09 16:22:12.028233 7f8689755700 2 waiting 51 > 50 ops || 1587706 > 104857600 2014-06-09 16:22:12.028245 7f868bf5a700 10 op_submit_start 3855 2014-06-09 16:22:12.028246 7f8689f56700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028247 7f868bf5a700 5 queue_transactions (trailing journal) 3855 2014-06-09 16:22:12.028246 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd9e0 (1 active) 2014-06-09 16:22:12.028248 7f868bf5a700 5 queue_op 0x159979e0 seq 3855 osr(3.2b6 0x3ed0ee0) 828 bytes (queue has 50 ops and 1586845 bytes) 2014-06-09 16:22:12.028250 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338240 (2 active) 2014-06-09 16:22:12.028250 7f8691f66700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028255 7f868a757700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028255 7f86a5e11700 5 _do_op 0x1e08af00 seq 3807 osr(3.224 0x3ed0c80)/0x3ed0c80 start 2014-06-09 16:22:12.028258 7f86a5e11700 10 _do_transaction on 0xbf42480 2014-06-09 16:22:12.028260 7f868e75f700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028265 7f8691765700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028239 7f869af78700 5 -- op tracker -- , seq: 3554, time: 2014-06-09 16:22:12.028237, event: started, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028271 7f8692767700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028276 7f8692f68700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028280 7f868c75b700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028282 7f869af78700 5 -- op tracker -- , seq: 3554, time: 2014-06-09 16:22:12.028279, event: done, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028285 7f868ff62700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028290 7f868cf5c700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028290 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028292 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028294 7f8690763700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028299 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d7380] 2014-06-09 16:22:12.028300 7f868f761700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028303 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d7380] 2014-06-09 16:22:12.028305 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.028306 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.028306 7f868af58700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028312 7f868b759700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028313 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028317 7f868ef60700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028309 7f86a5e11700 15 _omap_setkeys meta/1fe25d7a/pglog_3.224/0//-1 2014-06-09 16:22:12.028322 7f868df5e700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028327 7f8690f64700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028315 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028332 7f8689755700 2 waiting 51 > 50 ops || 1587706 > 104857600 2014-06-09 16:22:12.028333 7f869af78700 20 osd.26 243 _dispatch 0x160648c0 pg_info(1 pgs e243:3.458) v4 2014-06-09 16:22:12.028338 7f868d75d700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028338 7f869af78700 5 -- op tracker -- , seq: 3555, time: 2014-06-09 16:22:11.967311, event: header_read, op: pg_info(1 pgs e243:3.458) 2014-06-09 16:22:12.028346 7f869af78700 5 -- op tracker -- , seq: 3555, time: 2014-06-09 16:22:11.967311, event: throttled, op: pg_info(1 pgs e243:3.458) 2014-06-09 16:22:12.028352 7f869af78700 5 -- op tracker -- , seq: 3555, time: 2014-06-09 16:22:11.967323, event: all_read, op: pg_info(1 pgs e243:3.458) 2014-06-09 16:22:12.028361 7f869af78700 5 -- op tracker -- , seq: 3555, time: 2014-06-09 16:22:12.028313, event: dispatched, op: pg_info(1 pgs e243:3.458) 2014-06-09 16:22:12.028368 7f869af78700 5 -- op tracker -- , seq: 3555, time: 2014-06-09 16:22:12.028366, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.458) 2014-06-09 16:22:12.028373 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.458) v4 from osd.2 2014-06-09 16:22:12.028376 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x160648c0 2014-06-09 16:22:12.028376 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe25d7a/pglog_3.224/0//-1 2014-06-09 16:22:12.028382 7f869af78700 5 -- op tracker -- , seq: 3555, time: 2014-06-09 16:22:12.028380, event: started, op: pg_info(1 pgs e243:3.458) 2014-06-09 16:22:12.028394 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.028399 7f869af78700 5 -- op tracker -- , seq: 3555, time: 2014-06-09 16:22:12.028397, event: done, op: pg_info(1 pgs e243:3.458) 2014-06-09 16:22:12.028407 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028409 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028421 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028423 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028424 7f869af78700 20 osd.26 243 _dispatch 0x3411d500 pg_info(1 pgs e243:1.438) v4 2014-06-09 16:22:12.028429 7f869af78700 5 -- op tracker -- , seq: 3556, time: 2014-06-09 16:22:11.976441, event: header_read, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.028434 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.028436 7f869af78700 5 -- op tracker -- , seq: 3556, time: 2014-06-09 16:22:11.976442, event: throttled, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.028448 7f869af78700 5 -- op tracker -- , seq: 3556, time: 2014-06-09 16:22:11.976454, event: all_read, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.028451 7f86a5e11700 15 _omap_rmkeys meta/1fe25d7a/pglog_3.224/0//-1 2014-06-09 16:22:12.028457 7f86a5e11700 10 _omap_rmkeys meta/1fe25d7a/pglog_3.224/0//-1 r = 0 2014-06-09 16:22:12.028454 7f869af78700 5 -- op tracker -- , seq: 3556, time: 2014-06-09 16:22:12.028421, event: dispatched, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.028463 7f86a5e11700 15 _omap_setkeys meta/1fe25d7a/pglog_3.224/0//-1 2014-06-09 16:22:12.028464 7f869af78700 5 -- op tracker -- , seq: 3556, time: 2014-06-09 16:22:12.028462, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.028469 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.438) v4 from osd.4 2014-06-09 16:22:12.028472 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3411d500 2014-06-09 16:22:12.028476 7f86a5e11700 15 _touch 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:12.028478 7f869af78700 5 -- op tracker -- , seq: 3556, time: 2014-06-09 16:22:12.028476, event: started, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.028494 7f869af78700 5 -- op tracker -- , seq: 3556, time: 2014-06-09 16:22:12.028492, event: done, op: pg_info(1 pgs e243:1.438) 2014-06-09 16:22:12.028502 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028504 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028507 7f86a5610700 10 _finish_op 0xc88a720 seq 3790 osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 2014-06-09 16:22:12.028516 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028517 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028516 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652900 (1 active) 2014-06-09 16:22:12.028519 7f869af78700 20 osd.26 243 _dispatch 0x30c00000 pg_info(1 pgs e243:0.f) v4 2014-06-09 16:22:12.028517 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.224_head oid bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:12.028519 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2867e0 (2 active) 2014-06-09 16:22:12.028519 7f8691f66700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028527 7f86a5610700 5 _do_op 0xd04ed20 seq 3808 osr(3.76 0x3ed0240)/0x3ed0240 start 2014-06-09 16:22:12.028530 7f86a5610700 10 _do_transaction on 0x2ed4b40 2014-06-09 16:22:12.028530 7f868bf5a700 10 op_submit_finish 3855 2014-06-09 16:22:12.028534 7f8689f56700 10 op_submit_start 3856 2014-06-09 16:22:12.028536 7f8689f56700 5 queue_transactions (trailing journal) 3856 2014-06-09 16:22:12.028536 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.028538 7f8689f56700 5 queue_op 0x10039e00 seq 3856 osr(3.461 0x3ed1120) 828 bytes (queue has 50 ops and 1586845 bytes) 2014-06-09 16:22:12.028541 7f8689f56700 10 op_submit_finish 3856 2014-06-09 16:22:12.028535 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.136( v 213'630 (0'0,213'630] local-les=243 n=119 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:12.028552 7f86a5e11700 20 write_state seq is 30161 2014-06-09 16:22:12.028545 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.211( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.211( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.028560 7f86a5e11700 20 set_header setting 30160 cid 3.224_head oid bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 parent seq 0 2014-06-09 16:22:12.028568 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.028573 7f868bf5a700 5 queue_transactions existing osr(3.136 0x3ed07a0)/0x3ed07a0 2014-06-09 16:22:12.028576 7f868bf5a700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028577 7f86a5e11700 10 _touch 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 = 0 2014-06-09 16:22:12.028580 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.028580 7f868a757700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028585 7f868e75f700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028584 7f86a5e11700 15 _write 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 0~458752 2014-06-09 16:22:12.028585 7f8689f56700 5 queue_transactions existing osr(1.211 0x384cc90)/0x384cc90 2014-06-09 16:22:12.028588 7f8689f56700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028590 7f8691765700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028524 7f869af78700 5 -- op tracker -- , seq: 3557, time: 2014-06-09 16:22:11.967878, event: header_read, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028595 7f8692767700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028594 7f869af78700 5 -- op tracker -- , seq: 3557, time: 2014-06-09 16:22:11.967879, event: throttled, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028599 7f8692f68700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028604 7f868c75b700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028608 7f868ff62700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028602 7f869af78700 5 -- op tracker -- , seq: 3557, time: 2014-06-09 16:22:11.967957, event: all_read, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028612 7f868cf5c700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028617 7f8690763700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028614 7f869af78700 5 -- op tracker -- , seq: 3557, time: 2014-06-09 16:22:12.028515, event: dispatched, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028622 7f868f761700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028622 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd0fb00] 2014-06-09 16:22:12.028627 7f868af58700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028628 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd0fb00] 2014-06-09 16:22:12.028631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.028632 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.028629 7f869af78700 5 -- op tracker -- , seq: 3557, time: 2014-06-09 16:22:12.028620, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028635 7f868b759700 2 waiting 51 > 50 ops || 1587671 > 104857600 2014-06-09 16:22:12.028635 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.f) v4 from osd.2 2014-06-09 16:22:12.028641 7f868ef60700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028646 7f868df5e700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028650 7f8690f64700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028638 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c00000 2014-06-09 16:22:12.028655 7f8689755700 2 waiting 51 > 50 ops || 1587706 > 104857600 2014-06-09 16:22:12.028652 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.028660 7f868d75d700 2 waiting 51 > 50 ops || 1587673 > 104857600 2014-06-09 16:22:12.028660 7f869af78700 5 -- op tracker -- , seq: 3557, time: 2014-06-09 16:22:12.028658, event: started, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028677 7f869af78700 5 -- op tracker -- , seq: 3557, time: 2014-06-09 16:22:12.028674, event: done, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028685 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028687 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028700 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028701 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028703 7f869af78700 20 osd.26 243 _dispatch 0x333c36c0 pg_info(1 pgs e243:1.4c9) v4 2014-06-09 16:22:12.028708 7f869af78700 5 -- op tracker -- , seq: 3558, time: 2014-06-09 16:22:11.976459, event: header_read, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028715 7f869af78700 5 -- op tracker -- , seq: 3558, time: 2014-06-09 16:22:11.976459, event: throttled, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028722 7f869af78700 5 -- op tracker -- , seq: 3558, time: 2014-06-09 16:22:11.976469, event: all_read, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028731 7f869af78700 5 -- op tracker -- , seq: 3558, time: 2014-06-09 16:22:12.028699, event: dispatched, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028739 7f869af78700 5 -- op tracker -- , seq: 3558, time: 2014-06-09 16:22:12.028736, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028743 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.4c9) v4 from osd.4 2014-06-09 16:22:12.028746 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x333c36c0 2014-06-09 16:22:12.028751 7f869af78700 5 -- op tracker -- , seq: 3558, time: 2014-06-09 16:22:12.028750, event: started, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028769 7f869af78700 5 -- op tracker -- , seq: 3558, time: 2014-06-09 16:22:12.028766, event: done, op: pg_info(1 pgs e243:1.4c9) 2014-06-09 16:22:12.028777 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028779 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028790 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028792 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028793 7f869af78700 20 osd.26 243 _dispatch 0x3005a380 pg_info(1 pgs e243:0.f) v4 2014-06-09 16:22:12.028798 7f869af78700 5 -- op tracker -- , seq: 3559, time: 2014-06-09 16:22:11.967968, event: header_read, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028805 7f869af78700 5 -- op tracker -- , seq: 3559, time: 2014-06-09 16:22:11.967969, event: throttled, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028812 7f869af78700 5 -- op tracker -- , seq: 3559, time: 2014-06-09 16:22:11.967980, event: all_read, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028824 7f869af78700 5 -- op tracker -- , seq: 3559, time: 2014-06-09 16:22:12.028790, event: dispatched, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028831 7f869af78700 5 -- op tracker -- , seq: 3559, time: 2014-06-09 16:22:12.028829, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028839 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.f) v4 from osd.2 2014-06-09 16:22:12.028843 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3005a380 2014-06-09 16:22:12.028848 7f869af78700 5 -- op tracker -- , seq: 3559, time: 2014-06-09 16:22:12.028846, event: started, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028863 7f869af78700 5 -- op tracker -- , seq: 3559, time: 2014-06-09 16:22:12.028861, event: done, op: pg_info(1 pgs e243:0.f) 2014-06-09 16:22:12.028871 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028873 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028885 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.028886 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.028888 7f869af78700 20 osd.26 243 _dispatch 0x333c3880 pg_info(1 pgs e243:3.1b1) v4 2014-06-09 16:22:12.028893 7f869af78700 5 -- op tracker -- , seq: 3560, time: 2014-06-09 16:22:11.976473, event: header_read, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.028899 7f869af78700 5 -- op tracker -- , seq: 3560, time: 2014-06-09 16:22:11.976473, event: throttled, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.028911 7f869af78700 5 -- op tracker -- , seq: 3560, time: 2014-06-09 16:22:11.976484, event: all_read, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.028921 7f869af78700 5 -- op tracker -- , seq: 3560, time: 2014-06-09 16:22:12.028884, event: dispatched, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.028932 7f869af78700 5 -- op tracker -- , seq: 3560, time: 2014-06-09 16:22:12.028930, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.028941 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1b1) v4 from osd.4 2014-06-09 16:22:12.028944 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x333c3880 2014-06-09 16:22:12.028949 7f869af78700 5 -- op tracker -- , seq: 3560, time: 2014-06-09 16:22:12.028947, event: started, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.029953 7f86a5e11700 10 _generic_write 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 0~458752 = 0 2014-06-09 16:22:12.031758 7f86af56b700 5 -- op tracker -- , seq: 3561, time: 2014-06-09 16:22:12.027327, event: header_read, op: osd_op(client.4541.0:32277 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~1294336] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.031772 7f86af56b700 5 -- op tracker -- , seq: 3561, time: 2014-06-09 16:22:12.027328, event: throttled, op: osd_op(client.4541.0:32277 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~1294336] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.031780 7f86a5e11700 15 _setattrs 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:12.031781 7f86af56b700 5 -- op tracker -- , seq: 3561, time: 2014-06-09 16:22:12.029699, event: all_read, op: osd_op(client.4541.0:32277 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~1294336] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.031790 7f86af56b700 5 -- op tracker -- , seq: 3561, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32277 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~1294336] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.031792 7f86a5e11700 10 _setattrs 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 = 0 2014-06-09 16:22:12.031798 7f86a5e11700 15 _setattrs 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 2014-06-09 16:22:12.031797 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.031806 7f86a5e11700 10 _setattrs 3.224_head/bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 = 0 2014-06-09 16:22:12.031820 7f86af56b700 15 osd.26 243 enqueue_op 0x2f097d00 prio 63 cost 1294336 latency 0.004492 osd_op(client.4541.0:32277 rbd_data.115d6b8b4567.000000000000164b [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~1294336] 3.7e9cd750 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.031837 7f869af78700 5 -- op tracker -- , seq: 3560, time: 2014-06-09 16:22:12.031833, event: done, op: pg_info(1 pgs e243:3.1b1) 2014-06-09 16:22:12.031848 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.031850 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.031875 7f86a5e11700 10 cid: oid: bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 not skipping op, *spos 3807.0.0 2014-06-09 16:22:12.031880 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.031895 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.031897 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.031896 7f86a5e11700 20 set_header setting 30160 cid 3.224_head oid bc749224/rbd_data.11776b8b4567.00000000000017fb/head//3 parent seq 0 2014-06-09 16:22:12.031899 7f869af78700 20 osd.26 243 _dispatch 0x33440fc0 pg_info(1 pgs e243:3.1c7) v4 2014-06-09 16:22:12.031904 7f869af78700 5 -- op tracker -- , seq: 3562, time: 2014-06-09 16:22:11.969786, event: header_read, op: pg_info(1 pgs e243:3.1c7) 2014-06-09 16:22:12.031912 7f869af78700 5 -- op tracker -- , seq: 3562, time: 2014-06-09 16:22:11.969787, event: throttled, op: pg_info(1 pgs e243:3.1c7) 2014-06-09 16:22:12.031919 7f869af78700 5 -- op tracker -- , seq: 3562, time: 2014-06-09 16:22:11.969802, event: all_read, op: pg_info(1 pgs e243:3.1c7) 2014-06-09 16:22:12.031926 7f869af78700 5 -- op tracker -- , seq: 3562, time: 2014-06-09 16:22:12.031894, event: dispatched, op: pg_info(1 pgs e243:3.1c7) 2014-06-09 16:22:12.031936 7f869af78700 5 -- op tracker -- , seq: 3562, time: 2014-06-09 16:22:12.031933, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1c7) 2014-06-09 16:22:12.031941 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1c7) v4 from osd.2 2014-06-09 16:22:12.031944 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x33440fc0 2014-06-09 16:22:12.031945 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3807.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.031951 7f86a5e11700 10 cid: oid: 1fe25d7a/pglog_3.224/0//-1 skipping op, spos 3807.0.0 <= header.spos 150417.0.0 2014-06-09 16:22:12.031954 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.032015 7f869af78700 5 -- op tracker -- , seq: 3562, time: 2014-06-09 16:22:12.031948, event: started, op: pg_info(1 pgs e243:3.1c7) 2014-06-09 16:22:12.032047 7f869af78700 5 -- op tracker -- , seq: 3562, time: 2014-06-09 16:22:12.032044, event: done, op: pg_info(1 pgs e243:3.1c7) 2014-06-09 16:22:12.032076 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032078 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032101 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032103 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032104 7f869af78700 20 osd.26 243 _dispatch 0x30fcc380 pg_info(1 pgs e243:3.11f) v4 2014-06-09 16:22:12.032110 7f869af78700 5 -- op tracker -- , seq: 3563, time: 2014-06-09 16:22:11.976488, event: header_read, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.032124 7f869af78700 5 -- op tracker -- , seq: 3563, time: 2014-06-09 16:22:11.976488, event: throttled, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.032130 7f869af78700 5 -- op tracker -- , seq: 3563, time: 2014-06-09 16:22:11.976498, event: all_read, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.032141 7f869af78700 5 -- op tracker -- , seq: 3563, time: 2014-06-09 16:22:12.032100, event: dispatched, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.032157 7f86af56b700 5 -- op tracker -- , seq: 3564, time: 2014-06-09 16:22:12.031839, event: header_read, op: osd_op(client.4541.0:32282 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~188416] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.032169 7f86af56b700 5 -- op tracker -- , seq: 3564, time: 2014-06-09 16:22:12.031840, event: throttled, op: osd_op(client.4541.0:32282 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~188416] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.032178 7f86af56b700 5 -- op tracker -- , seq: 3564, time: 2014-06-09 16:22:12.032085, event: all_read, op: osd_op(client.4541.0:32282 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~188416] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.032187 7f86af56b700 5 -- op tracker -- , seq: 3564, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32282 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~188416] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.032194 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.032196 7f869af78700 5 -- op tracker -- , seq: 3563, time: 2014-06-09 16:22:12.032186, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.032204 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.11f) v4 from osd.4 2014-06-09 16:22:12.032207 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcc380 2014-06-09 16:22:12.032213 7f869af78700 5 -- op tracker -- , seq: 3563, time: 2014-06-09 16:22:12.032211, event: started, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.032208 7f86af56b700 15 osd.26 243 enqueue_op 0x257c5700 prio 63 cost 188416 latency 0.000369 osd_op(client.4541.0:32282 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~188416] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.032230 7f869af78700 5 -- op tracker -- , seq: 3563, time: 2014-06-09 16:22:12.032228, event: done, op: pg_info(1 pgs e243:3.11f) 2014-06-09 16:22:12.032238 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032240 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032321 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032324 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032325 7f869af78700 20 osd.26 243 _dispatch 0x3411d6c0 pg_info(1 pgs e243:3.366) v4 2014-06-09 16:22:12.032331 7f869af78700 5 -- op tracker -- , seq: 3565, time: 2014-06-09 16:22:11.975233, event: header_read, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.032339 7f869af78700 5 -- op tracker -- , seq: 3565, time: 2014-06-09 16:22:11.975234, event: throttled, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.032346 7f869af78700 5 -- op tracker -- , seq: 3565, time: 2014-06-09 16:22:11.975248, event: all_read, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.032386 7f869af78700 5 -- op tracker -- , seq: 3565, time: 2014-06-09 16:22:12.032321, event: dispatched, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.032396 7f869af78700 5 -- op tracker -- , seq: 3565, time: 2014-06-09 16:22:12.032393, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.032400 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.366) v4 from osd.2 2014-06-09 16:22:12.032403 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3411d6c0 2014-06-09 16:22:12.032408 7f869af78700 5 -- op tracker -- , seq: 3565, time: 2014-06-09 16:22:12.032406, event: started, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.032427 7f869af78700 5 -- op tracker -- , seq: 3565, time: 2014-06-09 16:22:12.032424, event: done, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.032437 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032439 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032452 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032454 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032455 7f869af78700 20 osd.26 243 _dispatch 0x30fcca80 pg_info(1 pgs e243:3.2a9) v4 2014-06-09 16:22:12.032460 7f869af78700 5 -- op tracker -- , seq: 3566, time: 2014-06-09 16:22:11.976502, event: header_read, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.032482 7f869af78700 5 -- op tracker -- , seq: 3566, time: 2014-06-09 16:22:11.976502, event: throttled, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.032488 7f869af78700 5 -- op tracker -- , seq: 3566, time: 2014-06-09 16:22:11.976513, event: all_read, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.032494 7f869af78700 5 -- op tracker -- , seq: 3566, time: 2014-06-09 16:22:12.032452, event: dispatched, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.032501 7f869af78700 5 -- op tracker -- , seq: 3566, time: 2014-06-09 16:22:12.032499, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.032505 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.2a9) v4 from osd.4 2014-06-09 16:22:12.032508 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcca80 2014-06-09 16:22:12.032513 7f869af78700 5 -- op tracker -- , seq: 3566, time: 2014-06-09 16:22:12.032511, event: started, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.032530 7f869af78700 5 -- op tracker -- , seq: 3566, time: 2014-06-09 16:22:12.032528, event: done, op: pg_info(1 pgs e243:3.2a9) 2014-06-09 16:22:12.032538 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032540 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032552 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032553 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032555 7f869af78700 20 osd.26 243 _dispatch 0x30e11880 pg_info(1 pgs e243:3.712) v4 2014-06-09 16:22:12.032560 7f869af78700 5 -- op tracker -- , seq: 3567, time: 2014-06-09 16:22:11.975259, event: header_read, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.032567 7f869af78700 5 -- op tracker -- , seq: 3567, time: 2014-06-09 16:22:11.975260, event: throttled, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.032614 7f869af78700 5 -- op tracker -- , seq: 3567, time: 2014-06-09 16:22:11.975271, event: all_read, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.032621 7f869af78700 5 -- op tracker -- , seq: 3567, time: 2014-06-09 16:22:12.032551, event: dispatched, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.032628 7f869af78700 5 -- op tracker -- , seq: 3567, time: 2014-06-09 16:22:12.032626, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.032635 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.712) v4 from osd.2 2014-06-09 16:22:12.032638 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30e11880 2014-06-09 16:22:12.032643 7f869af78700 5 -- op tracker -- , seq: 3567, time: 2014-06-09 16:22:12.032641, event: started, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.032660 7f869af78700 5 -- op tracker -- , seq: 3567, time: 2014-06-09 16:22:12.032658, event: done, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.032668 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032670 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032743 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032745 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032746 7f869af78700 20 osd.26 243 _dispatch 0x30fccc40 pg_info(1 pgs e243:1.5f8) v4 2014-06-09 16:22:12.032752 7f869af78700 5 -- op tracker -- , seq: 3568, time: 2014-06-09 16:22:11.977468, event: header_read, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.032759 7f869af78700 5 -- op tracker -- , seq: 3568, time: 2014-06-09 16:22:11.977469, event: throttled, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.032766 7f869af78700 5 -- op tracker -- , seq: 3568, time: 2014-06-09 16:22:11.977481, event: all_read, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.032772 7f869af78700 5 -- op tracker -- , seq: 3568, time: 2014-06-09 16:22:12.032742, event: dispatched, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.032781 7f869af78700 5 -- op tracker -- , seq: 3568, time: 2014-06-09 16:22:12.032779, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.032786 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5f8) v4 from osd.4 2014-06-09 16:22:12.032789 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fccc40 2014-06-09 16:22:12.032833 7f869af78700 5 -- op tracker -- , seq: 3568, time: 2014-06-09 16:22:12.032830, event: started, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.032851 7f869af78700 5 -- op tracker -- , seq: 3568, time: 2014-06-09 16:22:12.032849, event: done, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.032880 7f86af56b700 5 -- op tracker -- , seq: 3569, time: 2014-06-09 16:22:12.032242, event: header_read, op: osd_op(client.4541.0:32283 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 188416~516096] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.032893 7f86af56b700 5 -- op tracker -- , seq: 3569, time: 2014-06-09 16:22:12.032243, event: throttled, op: osd_op(client.4541.0:32283 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 188416~516096] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.032903 7f86af56b700 5 -- op tracker -- , seq: 3569, time: 2014-06-09 16:22:12.032855, event: all_read, op: osd_op(client.4541.0:32283 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 188416~516096] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.032914 7f86af56b700 5 -- op tracker -- , seq: 3569, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32283 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 188416~516096] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.032919 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.032934 7f86af56b700 15 osd.26 243 enqueue_op 0x257c4100 prio 63 cost 516096 latency 0.000692 osd_op(client.4541.0:32283 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 188416~516096] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.032955 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032959 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032972 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.032974 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.032975 7f869af78700 20 osd.26 243 _dispatch 0x16064700 pg_info(1 pgs e243:3.793) v4 2014-06-09 16:22:12.032980 7f869af78700 5 -- op tracker -- , seq: 3570, time: 2014-06-09 16:22:11.975278, event: header_read, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.032993 7f869af78700 5 -- op tracker -- , seq: 3570, time: 2014-06-09 16:22:11.975278, event: throttled, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.032999 7f869af78700 5 -- op tracker -- , seq: 3570, time: 2014-06-09 16:22:11.975289, event: all_read, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.033008 7f869af78700 5 -- op tracker -- , seq: 3570, time: 2014-06-09 16:22:12.032972, event: dispatched, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.033015 7f869af78700 5 -- op tracker -- , seq: 3570, time: 2014-06-09 16:22:12.033013, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.033034 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.793) v4 from osd.2 2014-06-09 16:22:12.033038 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x16064700 2014-06-09 16:22:12.033043 7f869af78700 5 -- op tracker -- , seq: 3570, time: 2014-06-09 16:22:12.033041, event: started, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.033059 7f869af78700 5 -- op tracker -- , seq: 3570, time: 2014-06-09 16:22:12.033057, event: done, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.033068 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033070 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033083 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033085 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033086 7f869af78700 20 osd.26 243 _dispatch 0x30fcd500 pg_info(1 pgs e243:1.188) v4 2014-06-09 16:22:12.033091 7f869af78700 5 -- op tracker -- , seq: 3571, time: 2014-06-09 16:22:11.977533, event: header_read, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.033099 7f869af78700 5 -- op tracker -- , seq: 3571, time: 2014-06-09 16:22:11.977534, event: throttled, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.033106 7f869af78700 5 -- op tracker -- , seq: 3571, time: 2014-06-09 16:22:11.977549, event: all_read, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.033112 7f869af78700 5 -- op tracker -- , seq: 3571, time: 2014-06-09 16:22:12.033083, event: dispatched, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.033119 7f869af78700 5 -- op tracker -- , seq: 3571, time: 2014-06-09 16:22:12.033117, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.033124 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.188) v4 from osd.4 2014-06-09 16:22:12.033127 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcd500 2014-06-09 16:22:12.033201 7f869af78700 5 -- op tracker -- , seq: 3571, time: 2014-06-09 16:22:12.033130, event: started, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.033220 7f869af78700 5 -- op tracker -- , seq: 3571, time: 2014-06-09 16:22:12.033218, event: done, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.033231 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033233 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033246 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033301 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033303 7f869af78700 20 osd.26 243 _dispatch 0x333c3c00 pg_info(1 pgs e243:1.774) v4 2014-06-09 16:22:12.033308 7f869af78700 5 -- op tracker -- , seq: 3572, time: 2014-06-09 16:22:11.975294, event: header_read, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.033318 7f869af78700 5 -- op tracker -- , seq: 3572, time: 2014-06-09 16:22:11.975294, event: throttled, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.033324 7f869af78700 5 -- op tracker -- , seq: 3572, time: 2014-06-09 16:22:11.975304, event: all_read, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.033332 7f869af78700 5 -- op tracker -- , seq: 3572, time: 2014-06-09 16:22:12.033245, event: dispatched, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.033340 7f869af78700 5 -- op tracker -- , seq: 3572, time: 2014-06-09 16:22:12.033338, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.033345 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.774) v4 from osd.2 2014-06-09 16:22:12.033349 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x333c3c00 2014-06-09 16:22:12.033354 7f869af78700 5 -- op tracker -- , seq: 3572, time: 2014-06-09 16:22:12.033352, event: started, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.033407 7f869af78700 5 -- op tracker -- , seq: 3572, time: 2014-06-09 16:22:12.033404, event: done, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.033415 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033417 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033429 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033431 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033432 7f869af78700 20 osd.26 243 _dispatch 0x30fcd6c0 pg_info(1 pgs e243:1.755) v4 2014-06-09 16:22:12.033437 7f869af78700 5 -- op tracker -- , seq: 3573, time: 2014-06-09 16:22:11.977555, event: header_read, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.033444 7f869af78700 5 -- op tracker -- , seq: 3573, time: 2014-06-09 16:22:11.977555, event: throttled, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.033451 7f869af78700 5 -- op tracker -- , seq: 3573, time: 2014-06-09 16:22:11.977564, event: all_read, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.033459 7f869af78700 5 -- op tracker -- , seq: 3573, time: 2014-06-09 16:22:12.033428, event: dispatched, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.033467 7f869af78700 5 -- op tracker -- , seq: 3573, time: 2014-06-09 16:22:12.033465, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.033471 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.755) v4 from osd.4 2014-06-09 16:22:12.033475 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcd6c0 2014-06-09 16:22:12.033536 7f869af78700 5 -- op tracker -- , seq: 3573, time: 2014-06-09 16:22:12.033478, event: started, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.033559 7f869af78700 5 -- op tracker -- , seq: 3573, time: 2014-06-09 16:22:12.033557, event: done, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.033570 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033572 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033584 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033586 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033587 7f869af78700 20 osd.26 243 _dispatch 0x30fcc000 pg_info(1 pgs e243:3.366) v4 2014-06-09 16:22:12.033593 7f869af78700 5 -- op tracker -- , seq: 3574, time: 2014-06-09 16:22:11.975308, event: header_read, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.033649 7f869af78700 5 -- op tracker -- , seq: 3574, time: 2014-06-09 16:22:11.975308, event: throttled, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.033657 7f869af78700 5 -- op tracker -- , seq: 3574, time: 2014-06-09 16:22:11.975320, event: all_read, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.033666 7f869af78700 5 -- op tracker -- , seq: 3574, time: 2014-06-09 16:22:12.033584, event: dispatched, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.033674 7f869af78700 5 -- op tracker -- , seq: 3574, time: 2014-06-09 16:22:12.033672, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.033678 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.366) v4 from osd.2 2014-06-09 16:22:12.033682 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcc000 2014-06-09 16:22:12.033687 7f869af78700 5 -- op tracker -- , seq: 3574, time: 2014-06-09 16:22:12.033685, event: started, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.033706 7f869af78700 5 -- op tracker -- , seq: 3574, time: 2014-06-09 16:22:12.033704, event: done, op: pg_info(1 pgs e243:3.366) 2014-06-09 16:22:12.033714 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033727 7f86a5e11700 10 _do_op 0x1e08af00 seq 3807 r = 0, finisher 0x1a0023a0 0x12c0b640 2014-06-09 16:22:12.033741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b040] 2014-06-09 16:22:12.033745 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 22 2014-06-09 16:22:12.033716 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033772 7f86a460e700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:12.033765, event: op_commit, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.033783 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.033785 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.033786 7f869af78700 20 osd.26 243 _dispatch 0x333c3500 pg_info(1 pgs e243:0.71f) v4 2014-06-09 16:22:12.033780 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'739 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 22 all committed 2014-06-09 16:22:12.033792 7f869af78700 5 -- op tracker -- , seq: 3575, time: 2014-06-09 16:22:11.977604, event: header_read, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.033790 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] eval_repop repgather(0xc139080 243'740 rep_tid=22 committed?=1 applied?=0 lock=0 op=osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.033800 7f869af78700 5 -- op tracker -- , seq: 3575, time: 2014-06-09 16:22:11.977604, event: throttled, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.033809 7f869af78700 5 -- op tracker -- , seq: 3575, time: 2014-06-09 16:22:11.977615, event: all_read, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.033806 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] log_op_stats osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4 inb 459136 outb 0 rlat 0.000000 lat 1.206176 2014-06-09 16:22:12.033816 7f869af78700 5 -- op tracker -- , seq: 3575, time: 2014-06-09 16:22:12.033783, event: dispatched, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.033823 7f869af78700 5 -- op tracker -- , seq: 3575, time: 2014-06-09 16:22:12.033821, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.033828 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.71f) v4 from osd.4 2014-06-09 16:22:12.033824 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] publish_stats_to_osd 243:1237 2014-06-09 16:22:12.033831 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x333c3500 2014-06-09 16:22:12.033834 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] sending commit on repgather(0xc139080 243'740 rep_tid=22 committed?=1 applied?=0 lock=0 op=osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4) 0xdfb1680 2014-06-09 16:22:12.033869 7f86a5e11700 10 _finish_op 0x1e08af00 seq 3807 osr(3.224 0x3ed0c80)/0x3ed0c80 2014-06-09 16:22:12.033886 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1a0023a0] 2014-06-09 16:22:12.033889 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338240 (1 active) 2014-06-09 16:22:12.033892 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89ed80 (2 active) 2014-06-09 16:22:12.033894 7f86a5e11700 5 _do_op 0x2e21bc0 seq 3809 osr(3.1c7 0x3ed0a10)/0x3ed0a10 start 2014-06-09 16:22:12.033897 7f86a5e11700 10 _do_transaction on 0xbf4fb00 2014-06-09 16:22:12.033899 7f8691f66700 10 op_submit_start 3857 2014-06-09 16:22:12.033902 7f8691f66700 5 queue_transactions (trailing journal) 3857 2014-06-09 16:22:12.033902 7f868bf5a700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033905 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.033904 7f8691f66700 5 queue_op 0xc9308a0 seq 3857 osr(2.646 0x2ed1c30) 828 bytes (queue has 50 ops and 1126523 bytes) 2014-06-09 16:22:12.033908 7f8691f66700 10 op_submit_finish 3857 2014-06-09 16:22:12.033908 7f868a757700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033914 7f868e75f700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033913 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.033948 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.033954 7f8691f66700 5 queue_transactions existing osr(3.7ae 0xb7f01d0)/0xb7f01d0 2014-06-09 16:22:12.033956 7f8691f66700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033961 7f8689f56700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033970 7f8691765700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033977 7f8692767700 2 waiting 51 > 50 ops || 1127349 > 104857600 2014-06-09 16:22:12.033983 7f8692f68700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033988 7f868c75b700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033994 7f868ff62700 2 waiting 51 > 50 ops || 1127349 > 104857600 2014-06-09 16:22:12.033999 7f868cf5c700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.034004 7f8690763700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.034014 7f868f761700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.034020 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.034023 7f868af58700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.033887 7f869af78700 5 -- op tracker -- , seq: 3575, time: 2014-06-09 16:22:12.033835, event: started, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.034030 7f868b759700 2 waiting 51 > 50 ops || 1127349 > 104857600 2014-06-09 16:22:12.034038 7f86a460e700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:12.033865, event: commit_sent, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.034047 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3809.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.034051 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b040] 2014-06-09 16:22:12.034054 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.034055 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.034060 7f868ef60700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.034053 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] op_applied: 22 2014-06-09 16:22:12.034066 7f868df5e700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.034073 7f8690f64700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.034070 7f86a4e0f700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:12.034065, event: op_applied, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.034079 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] op_applied on primary on version 243'740 2014-06-09 16:22:12.034089 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] repop_all_applied: repop tid 22 all applied 2014-06-09 16:22:12.034098 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] eval_repop repgather(0xc139080 243'740 rep_tid=22 committed?=1 applied?=1 lock=0 op=osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.034117 7f869af78700 5 -- op tracker -- , seq: 3575, time: 2014-06-09 16:22:12.034115, event: done, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.034112 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] log_op_stats osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4 inb 459136 outb 0 rlat 0.000000 lat 1.206482 2014-06-09 16:22:12.034127 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.034129 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.034126 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] publish_stats_to_osd 243:1238 2014-06-09 16:22:12.034142 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.034136 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.034143 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.034145 7f869af78700 20 osd.26 243 _dispatch 0x30fcc540 pg_info(1 pgs e243:3.1eb) v4 2014-06-09 16:22:12.034150 7f869af78700 5 -- op tracker -- , seq: 3576, time: 2014-06-09 16:22:11.975324, event: header_read, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.034144 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] removing repgather(0xc139080 243'740 rep_tid=22 committed?=1 applied?=1 op=osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.034157 7f869af78700 5 -- op tracker -- , seq: 3576, time: 2014-06-09 16:22:11.975324, event: throttled, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.034157 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] q front is repgather(0xc139080 243'740 rep_tid=22 committed?=1 applied?=1 op=osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.034170 7f869af78700 5 -- op tracker -- , seq: 3576, time: 2014-06-09 16:22:11.975335, event: all_read, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.034170 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] remove_repop repgather(0xc139080 243'740 rep_tid=22 committed?=1 applied?=1 op=osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.034177 7f869af78700 5 -- op tracker -- , seq: 3576, time: 2014-06-09 16:22:12.034141, event: dispatched, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.034182 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] requeue_ops 2014-06-09 16:22:12.034187 7f869af78700 5 -- op tracker -- , seq: 3576, time: 2014-06-09 16:22:12.034185, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.034192 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1eb) v4 from osd.2 2014-06-09 16:22:12.034195 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcc540 2014-06-09 16:22:12.034206 7f86a4e0f700 5 -- op tracker -- , seq: 820, time: 2014-06-09 16:22:12.034201, event: done, op: osd_op(client.4567.0:35562 rbd_data.11776b8b4567.00000000000017fb [set-alloc-hint object_size 4194304 write_size 4194304,write 0~458752] 3.bc749224 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.034234 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1a0023a0] 2014-06-09 16:22:12.034237 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.034238 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.034242 7f8689755700 2 waiting 51 > 50 ops || 1127384 > 104857600 2014-06-09 16:22:12.034250 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.034250 7f868d75d700 2 waiting 51 > 50 ops || 1127351 > 104857600 2014-06-09 16:22:12.034262 7f869af78700 5 -- op tracker -- , seq: 3576, time: 2014-06-09 16:22:12.034252, event: started, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.034270 7f86a5e11700 10 _do_op 0x2e21bc0 seq 3809 r = 0, finisher 0x31412ea0 0 2014-06-09 16:22:12.034289 7f86a5e11700 10 _finish_op 0x2e21bc0 seq 3809 osr(3.1c7 0x3ed0a10)/0x3ed0a10 2014-06-09 16:22:12.034290 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d0340] 2014-06-09 16:22:12.034294 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d0340] 2014-06-09 16:22:12.034296 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.034297 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.034300 7f868a757700 10 op_submit_start 3858 2014-06-09 16:22:12.034301 7f868a757700 5 queue_transactions (trailing journal) 3858 2014-06-09 16:22:12.034301 7f868e75f700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034303 7f868a757700 5 queue_op 0xcabcae0 seq 3858 osr(3.52a 0x3ed1670) 828 bytes (queue has 50 ops and 1126298 bytes) 2014-06-09 16:22:12.034307 7f8691f66700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034312 7f8689f56700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034313 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412ea0] 2014-06-09 16:22:12.034316 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:12.034328 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.034333 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:12.034347 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412ea0] 2014-06-09 16:22:12.034349 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.034350 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.034354 7f8691765700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034355 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3808.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.034359 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.034359 7f8692767700 2 waiting 51 > 50 ops || 1127124 > 104857600 2014-06-09 16:22:12.034363 7f8692f68700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034367 7f868c75b700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034371 7f868ff62700 2 waiting 51 > 50 ops || 1127124 > 104857600 2014-06-09 16:22:12.034318 7f869af78700 5 -- op tracker -- , seq: 3576, time: 2014-06-09 16:22:12.034316, event: done, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.034374 7f868cf5c700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034378 7f8690763700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034378 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.034383 7f868f761700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034382 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89ed80 (1 active) 2014-06-09 16:22:12.034383 7f86a5610700 10 _do_op 0xd04ed20 seq 3808 r = 0, finisher 0xbd60640 0 2014-06-09 16:22:12.034385 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407e60 (2 active) 2014-06-09 16:22:12.034388 7f868af58700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034389 7f86a5610700 10 _finish_op 0xd04ed20 seq 3808 osr(3.76 0x3ed0240)/0x3ed0240 2014-06-09 16:22:12.034390 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd60560] 2014-06-09 16:22:12.034392 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd60560] 2014-06-09 16:22:12.034393 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.034394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.034391 7f86a5e11700 5 _do_op 0xbf7f380 seq 3810 osr(2.3c0 0x2ed13f0)/0x2ed13f0 start 2014-06-09 16:22:12.034396 7f86a5e11700 10 _do_transaction on 0x4734180 2014-06-09 16:22:12.034380 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.034402 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.034410 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2867e0 (1 active) 2014-06-09 16:22:12.034412 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd60640] 2014-06-09 16:22:12.034412 7f868ef60700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034413 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338360 (2 active) 2014-06-09 16:22:12.034416 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd60640] 2014-06-09 16:22:12.034418 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.034417 7f86a5610700 5 _do_op 0x230b8120 seq 3811 osr(2.6ec 0x2ed1e10)/0x2ed1e10 start 2014-06-09 16:22:12.034417 7f868df5e700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034419 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.034419 7f86a5610700 10 _do_transaction on 0xc2abb00 2014-06-09 16:22:12.034423 7f868a757700 10 op_submit_finish 3858 2014-06-09 16:22:12.034423 7f8690f64700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034424 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.034427 7f868e75f700 10 op_submit_start 3859 2014-06-09 16:22:12.034427 7f868d75d700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034429 7f868e75f700 5 queue_transactions (trailing journal) 3859 2014-06-09 16:22:12.034430 7f868e75f700 5 queue_op 0x324f4600 seq 3859 osr(3.609 0x3ed1b00) 828 bytes (queue has 50 ops and 1126300 bytes) 2014-06-09 16:22:12.034401 7f86af56b700 5 -- op tracker -- , seq: 3577, time: 2014-06-09 16:22:12.032953, event: header_read, op: osd_op(client.4541.0:32284 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 704512~1032192] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.034434 7f868e75f700 10 op_submit_finish 3859 2014-06-09 16:22:12.034428 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.034439 7f86af56b700 5 -- op tracker -- , seq: 3577, time: 2014-06-09 16:22:12.032954, event: throttled, op: osd_op(client.4541.0:32284 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 704512~1032192] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.034444 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.894742 7 0.000363 2014-06-09 16:22:12.034449 7f86af56b700 5 -- op tracker -- , seq: 3577, time: 2014-06-09 16:22:12.034247, event: all_read, op: osd_op(client.4541.0:32284 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 704512~1032192] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.034460 7f86af56b700 5 -- op tracker -- , seq: 3577, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32284 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 704512~1032192] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.034457 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.034438 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.501( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.501( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.034466 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.034468 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.034479 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.034493 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.034490 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.034490 7f86af56b700 15 osd.26 243 enqueue_op 0x1c66a500 prio 63 cost 1032192 latency 0.001537 osd_op(client.4541.0:32284 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 704512~1032192] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.034498 7f868e75f700 5 queue_transactions existing osr(2.501 0x2ed1690)/0x2ed1690 2014-06-09 16:22:12.034501 7f868e75f700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034498 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.034505 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.034518 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.034530 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.034539 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.034554 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.034557 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.034557 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.034558 7f869af78700 20 osd.26 243 _dispatch 0x31474700 pg_info(1 pgs e243:0.1da) v4 2014-06-09 16:22:12.034574 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 2014-06-09 16:22:12.034607 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.034611 7f868a757700 5 queue_transactions existing osr(2.5a7 0x2ed1bf0)/0x2ed1bf0 2014-06-09 16:22:12.034613 7f868a757700 2 waiting 51 > 50 ops || 1127161 > 104857600 2014-06-09 16:22:12.034564 7f869af78700 5 -- op tracker -- , seq: 3578, time: 2014-06-09 16:22:11.977625, event: header_read, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.034621 7f869af78700 5 -- op tracker -- , seq: 3578, time: 2014-06-09 16:22:11.977625, event: throttled, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.034628 7f869af78700 5 -- op tracker -- , seq: 3578, time: 2014-06-09 16:22:11.977635, event: all_read, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.034635 7f869af78700 5 -- op tracker -- , seq: 3578, time: 2014-06-09 16:22:12.034553, event: dispatched, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.034641 7f868bf5a700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034642 7f869af78700 5 -- op tracker -- , seq: 3578, time: 2014-06-09 16:22:12.034640, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.034650 7f868b759700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034652 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1da) v4 from osd.4 2014-06-09 16:22:12.034655 7f8689755700 2 waiting 51 > 50 ops || 1127161 > 104857600 2014-06-09 16:22:12.034655 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31474700 2014-06-09 16:22:12.034660 7f8689f56700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034666 7f8691765700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034671 7f8692767700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034661 7f869af78700 5 -- op tracker -- , seq: 3578, time: 2014-06-09 16:22:12.034658, event: started, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.034675 7f8692f68700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034678 7f868c75b700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034682 7f868ff62700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034686 7f868cf5c700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034690 7f8690763700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034700 7f868f761700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034706 7f868af58700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034711 7f8691f66700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034728 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.034686 7f869af78700 5 -- op tracker -- , seq: 3578, time: 2014-06-09 16:22:12.034684, event: done, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.034738 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.034740 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.034743 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3810.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.034748 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.034754 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.034756 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.034757 7f869af78700 20 osd.26 243 _dispatch 0x30fcd180 pg_info(1 pgs e243:2.4af) v4 2014-06-09 16:22:12.034761 7f86a5e11700 10 _do_op 0xbf7f380 seq 3810 r = 0, finisher 0x170010e0 0 2014-06-09 16:22:12.034768 7f86a5e11700 10 _finish_op 0xbf7f380 seq 3810 osr(2.3c0 0x2ed13f0)/0x2ed13f0 2014-06-09 16:22:12.034770 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c3f60] 2014-06-09 16:22:12.034772 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c3f60] 2014-06-09 16:22:12.034774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.034775 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.034763 7f869af78700 5 -- op tracker -- , seq: 3579, time: 2014-06-09 16:22:11.975339, event: header_read, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.034780 7f868df5e700 10 op_submit_start 3860 2014-06-09 16:22:12.034781 7f868df5e700 5 queue_transactions (trailing journal) 3860 2014-06-09 16:22:12.034781 7f8690f64700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034783 7f868df5e700 5 queue_op 0xbf7ef00 seq 3860 osr(0.4c9 0x2bd7650) 828 bytes (queue has 50 ops and 1126300 bytes) 2014-06-09 16:22:12.034787 7f868d75d700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034792 7f868e75f700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034793 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x170010e0] 2014-06-09 16:22:12.034797 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x170010e0] 2014-06-09 16:22:12.034799 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.034800 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.034789 7f869af78700 5 -- op tracker -- , seq: 3579, time: 2014-06-09 16:22:11.975340, event: throttled, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.034803 7f868a757700 2 waiting 51 > 50 ops || 1127161 > 104857600 2014-06-09 16:22:12.034808 7f868bf5a700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034805 7f869af78700 5 -- op tracker -- , seq: 3579, time: 2014-06-09 16:22:11.975350, event: all_read, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.034811 7f868b759700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034815 7f8689755700 2 waiting 51 > 50 ops || 1127161 > 104857600 2014-06-09 16:22:12.034820 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407e60 (1 active) 2014-06-09 16:22:12.034822 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd560 (2 active) 2014-06-09 16:22:12.034827 7f868df5e700 10 op_submit_finish 3860 2014-06-09 16:22:12.034831 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.15b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.15b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.034867 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.034867 7f869af78700 5 -- op tracker -- , seq: 3579, time: 2014-06-09 16:22:12.034754, event: dispatched, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.034874 7f868df5e700 5 queue_transactions existing osr(1.15b 0x384cbf0)/0x384cbf0 2014-06-09 16:22:12.034877 7f868df5e700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034875 7f869af78700 5 -- op tracker -- , seq: 3579, time: 2014-06-09 16:22:12.034874, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.034881 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4af) v4 from osd.2 2014-06-09 16:22:12.034882 7f86a5e11700 5 _do_op 0x230b8600 seq 3812 osr(0.5b7 0x2bd7a90)/0x2bd7a90 start 2014-06-09 16:22:12.034884 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcd180 2014-06-09 16:22:12.034884 7f8689f56700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034885 7f86a5e11700 10 _do_transaction on 0xbf52600 2014-06-09 16:22:12.034891 7f8691765700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034889 7f869af78700 5 -- op tracker -- , seq: 3579, time: 2014-06-09 16:22:12.034887, event: started, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.034895 7f8692767700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034899 7f8692f68700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034903 7f868c75b700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034907 7f868ff62700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.034906 7f869af78700 5 -- op tracker -- , seq: 3579, time: 2014-06-09 16:22:12.034904, event: done, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.034912 7f868cf5c700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034915 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.034916 7f8690763700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034916 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.034921 7f868f761700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034923 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.034927 7f868af58700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034932 7f8691f66700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034938 7f868ef60700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.034974 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.034988 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3812.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.034990 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.034992 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.034992 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.034995 7f869af78700 20 osd.26 243 _dispatch 0x31474540 pg_info(1 pgs e243:0.1da) v4 2014-06-09 16:22:12.035001 7f869af78700 5 -- op tracker -- , seq: 3580, time: 2014-06-09 16:22:11.977673, event: header_read, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.035005 7f86a5e11700 10 _do_op 0x230b8600 seq 3812 r = 0, finisher 0xbd602a0 0 2014-06-09 16:22:12.035011 7f86a5e11700 10 _finish_op 0x230b8600 seq 3812 osr(0.5b7 0x2bd7a90)/0x2bd7a90 2014-06-09 16:22:12.035013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd603c0] 2014-06-09 16:22:12.035015 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd603c0] 2014-06-09 16:22:12.035017 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.035018 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.035024 7f868d75d700 10 op_submit_start 3861 2014-06-09 16:22:12.035025 7f868e75f700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035025 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd602a0] 2014-06-09 16:22:12.035026 7f868d75d700 5 queue_transactions (trailing journal) 3861 2014-06-09 16:22:12.035028 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd602a0] 2014-06-09 16:22:12.035030 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.035009 7f869af78700 5 -- op tracker -- , seq: 3580, time: 2014-06-09 16:22:11.977673, event: throttled, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.035028 7f868d75d700 5 queue_op 0x11fce1e0 seq 3861 osr(1.65e 0x384df30) 828 bytes (queue has 50 ops and 1126300 bytes) 2014-06-09 16:22:12.035031 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.035034 7f868a757700 2 waiting 51 > 50 ops || 1127161 > 104857600 2014-06-09 16:22:12.035034 7f869af78700 5 -- op tracker -- , seq: 3580, time: 2014-06-09 16:22:11.977683, event: all_read, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.035039 7f868bf5a700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035043 7f868b759700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.035040 7f869af78700 5 -- op tracker -- , seq: 3580, time: 2014-06-09 16:22:12.034989, event: dispatched, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.035048 7f8689755700 2 waiting 51 > 50 ops || 1127161 > 104857600 2014-06-09 16:22:12.035048 7f869af78700 5 -- op tracker -- , seq: 3580, time: 2014-06-09 16:22:12.035046, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.035051 7f868df5e700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035053 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.1da) v4 from osd.4 2014-06-09 16:22:12.035056 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd560 (1 active) 2014-06-09 16:22:12.035056 7f8689f56700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035058 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653e60 (2 active) 2014-06-09 16:22:12.035064 7f8691765700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035065 7f868d75d700 10 op_submit_finish 3861 2014-06-09 16:22:12.035069 7f8692767700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.035072 7f8692f68700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035076 7f868c75b700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035080 7f868ff62700 2 waiting 51 > 50 ops || 1127126 > 104857600 2014-06-09 16:22:12.035070 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.2cb( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:12.035084 7f868cf5c700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035088 7f8690763700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035093 7f86a5e11700 5 _do_op 0x100388a0 seq 3813 osr(3.292 0x3ed0ba0)/0x3ed0ba0 start 2014-06-09 16:22:12.035096 7f86a5e11700 10 _do_transaction on 0xbfcba40 2014-06-09 16:22:12.035099 7f868f761700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035102 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.035056 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31474540 2014-06-09 16:22:12.035107 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.035112 7f868d75d700 5 queue_transactions existing osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 2014-06-09 16:22:12.035115 7f868d75d700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035111 7f869af78700 5 -- op tracker -- , seq: 3580, time: 2014-06-09 16:22:12.035109, event: started, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.035119 7f868af58700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035123 7f8691f66700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035129 7f868ef60700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035128 7f869af78700 5 -- op tracker -- , seq: 3580, time: 2014-06-09 16:22:12.035126, event: done, op: pg_info(1 pgs e243:0.1da) 2014-06-09 16:22:12.035135 7f8690f64700 2 waiting 51 > 50 ops || 1127128 > 104857600 2014-06-09 16:22:12.035137 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035139 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035152 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035152 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.035154 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035155 7f869af78700 20 osd.26 243 _dispatch 0x30fcd880 pg_info(1 pgs e243:3.1e1) v4 2014-06-09 16:22:12.035160 7f869af78700 5 -- op tracker -- , seq: 3581, time: 2014-06-09 16:22:11.975355, event: header_read, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.035170 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3813.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.035169 7f869af78700 5 -- op tracker -- , seq: 3581, time: 2014-06-09 16:22:11.975355, event: throttled, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.035174 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.035188 7f86a5e11700 10 _do_op 0x100388a0 seq 3813 r = 0, finisher 0xbd7d220 0 2014-06-09 16:22:12.035194 7f86a5e11700 10 _finish_op 0x100388a0 seq 3813 osr(3.292 0x3ed0ba0)/0x3ed0ba0 2014-06-09 16:22:12.035198 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcf0a0] 2014-06-09 16:22:12.035202 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcf0a0] 2014-06-09 16:22:12.035203 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.035204 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.035204 7f868a757700 10 op_submit_start 3862 2014-06-09 16:22:12.035206 7f868a757700 5 queue_transactions (trailing journal) 3862 2014-06-09 16:22:12.035207 7f868a757700 5 queue_op 0x30033800 seq 3862 osr(2.5a7 0x2ed1bf0) 861 bytes (queue has 50 ops and 1125826 bytes) 2014-06-09 16:22:12.035176 7f869af78700 5 -- op tracker -- , seq: 3581, time: 2014-06-09 16:22:11.975366, event: all_read, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.035211 7f868bf5a700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035216 7f868b759700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035214 7f869af78700 5 -- op tracker -- , seq: 3581, time: 2014-06-09 16:22:12.035151, event: dispatched, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.035219 7f8689755700 2 waiting 51 > 50 ops || 1126687 > 104857600 2014-06-09 16:22:12.035223 7f868df5e700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035222 7f869af78700 5 -- op tracker -- , seq: 3581, time: 2014-06-09 16:22:12.035220, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.035227 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1e1) v4 from osd.2 2014-06-09 16:22:12.035229 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30fcd880 2014-06-09 16:22:12.035232 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7d220] 2014-06-09 16:22:12.035234 7f869af78700 5 -- op tracker -- , seq: 3581, time: 2014-06-09 16:22:12.035232, event: started, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.035236 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 crt=0'0 lcod 0'0 inactive] _activate_committed 243 telling primary 2014-06-09 16:22:12.035249 7f8689f56700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035251 7f869af78700 5 -- op tracker -- , seq: 3581, time: 2014-06-09 16:22:12.035249, event: done, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.035255 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.035272 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035274 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035272 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] flushed 2014-06-09 16:22:12.035288 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035289 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7d220] 2014-06-09 16:22:12.035290 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035291 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.035292 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.035291 7f869af78700 20 osd.26 243 _dispatch 0x31474380 pg_info(1 pgs e243:2.778) v4 2014-06-09 16:22:12.035295 7f8691765700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035300 7f8692767700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035297 7f869af78700 5 -- op tracker -- , seq: 3582, time: 2014-06-09 16:22:11.986653, event: header_read, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.035305 7f8692f68700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035309 7f868c75b700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035313 7f868ff62700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035316 7f868cf5c700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035321 7f8690763700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035325 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653e60 (1 active) 2014-06-09 16:22:12.035328 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d06c0 (2 active) 2014-06-09 16:22:12.035332 7f868a757700 10 op_submit_finish 3862 2014-06-09 16:22:12.035340 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.035346 7f868f761700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035336 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:12.035352 7f868d75d700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035353 7f869af78700 5 -- op tracker -- , seq: 3582, time: 2014-06-09 16:22:11.986654, event: throttled, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.035356 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3811.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.035358 7f868af58700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035360 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.035362 7f8691f66700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035362 7f869af78700 5 -- op tracker -- , seq: 3582, time: 2014-06-09 16:22:11.986671, event: all_read, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.035370 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.035369 7f869af78700 5 -- op tracker -- , seq: 3582, time: 2014-06-09 16:22:12.035288, event: dispatched, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.035374 7f868a757700 5 queue_transactions existing osr(3.761 0x3ed1e10)/0x3ed1e10 2014-06-09 16:22:12.035377 7f868a757700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035376 7f869af78700 5 -- op tracker -- , seq: 3582, time: 2014-06-09 16:22:12.035374, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.035381 7f868ef60700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035381 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.778) v4 from osd.4 2014-06-09 16:22:12.035382 7f86a5610700 10 _do_op 0x230b8120 seq 3811 r = 0, finisher 0xbd5fc80 0 2014-06-09 16:22:12.035385 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31474380 2014-06-09 16:22:12.035386 7f8690f64700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035389 7f86a5610700 10 _finish_op 0x230b8120 seq 3811 osr(2.6ec 0x2ed1e10)/0x2ed1e10 2014-06-09 16:22:12.035390 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5fc20] 2014-06-09 16:22:12.035390 7f868e75f700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.035392 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5fc20] 2014-06-09 16:22:12.035394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.035395 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.035395 7f86a5e11700 5 _do_op 0x2c06f740 seq 3814 osr(2.57a 0x2ed1870)/0x2ed1870 start 2014-06-09 16:22:12.035399 7f86a5e11700 10 _do_transaction on 0xbff6a80 2014-06-09 16:22:12.035401 7f869af78700 5 -- op tracker -- , seq: 3582, time: 2014-06-09 16:22:12.035399, event: started, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.035404 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.035418 7f869af78700 5 -- op tracker -- , seq: 3582, time: 2014-06-09 16:22:12.035415, event: done, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.035442 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.035456 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3814.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.035460 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.035471 7f86a5e11700 10 _do_op 0x2c06f740 seq 3814 r = 0, finisher 0x3042b540 0 2014-06-09 16:22:12.035481 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5d000] 2014-06-09 16:22:12.035483 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5d000] 2014-06-09 16:22:12.035485 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.035486 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.035428 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035498 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035511 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035513 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035515 7f869af78700 20 osd.26 243 _dispatch 0xca43dc0 pg_info(1 pgs e243:3.793) v4 2014-06-09 16:22:12.035520 7f869af78700 5 -- op tracker -- , seq: 3583, time: 2014-06-09 16:22:11.975371, event: header_read, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.035528 7f869af78700 5 -- op tracker -- , seq: 3583, time: 2014-06-09 16:22:11.975371, event: throttled, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.035535 7f869af78700 5 -- op tracker -- , seq: 3583, time: 2014-06-09 16:22:11.975382, event: all_read, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.035542 7f869af78700 5 -- op tracker -- , seq: 3583, time: 2014-06-09 16:22:12.035511, event: dispatched, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.035551 7f869af78700 5 -- op tracker -- , seq: 3583, time: 2014-06-09 16:22:12.035549, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.035559 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.793) v4 from osd.2 2014-06-09 16:22:12.035562 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xca43dc0 2014-06-09 16:22:12.035568 7f869af78700 5 -- op tracker -- , seq: 3583, time: 2014-06-09 16:22:12.035566, event: started, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.035586 7f869af78700 5 -- op tracker -- , seq: 3583, time: 2014-06-09 16:22:12.035583, event: done, op: pg_info(1 pgs e243:3.793) 2014-06-09 16:22:12.035594 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035596 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035608 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035610 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035612 7f869af78700 20 osd.26 243 _dispatch 0x12c088c0 pg_info(1 pgs e243:3.516) v4 2014-06-09 16:22:12.035637 7f869af78700 5 -- op tracker -- , seq: 3584, time: 2014-06-09 16:22:11.987373, event: header_read, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.035643 7f868b759700 10 op_submit_start 3863 2014-06-09 16:22:12.035642 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338360 (1 active) 2014-06-09 16:22:12.035645 7f868b759700 5 queue_transactions (trailing journal) 3863 2014-06-09 16:22:12.035646 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0a20 (2 active) 2014-06-09 16:22:12.035644 7f869af78700 5 -- op tracker -- , seq: 3584, time: 2014-06-09 16:22:11.987374, event: throttled, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.035647 7f868b759700 5 queue_op 0xbf7ec60 seq 3863 osr(2.f4 0x2ed0790) 826 bytes (queue has 50 ops and 1125824 bytes) 2014-06-09 16:22:12.035650 7f868b759700 10 op_submit_finish 3863 2014-06-09 16:22:12.035650 7f86a5610700 5 _do_op 0xcabdf80 seq 3815 osr(0.16c 0x2bd67f0)/0x2bd67f0 start 2014-06-09 16:22:12.035653 7f86a5610700 10 _do_transaction on 0xc05d800 2014-06-09 16:22:12.035658 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.035651 7f869af78700 5 -- op tracker -- , seq: 3584, time: 2014-06-09 16:22:11.987390, event: all_read, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.035663 7f869af78700 5 -- op tracker -- , seq: 3584, time: 2014-06-09 16:22:12.035608, event: dispatched, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.035654 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.25b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.25b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.035670 7f869af78700 5 -- op tracker -- , seq: 3584, time: 2014-06-09 16:22:12.035668, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.035690 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.035695 7f868b759700 5 queue_transactions existing osr(0.25b 0x2bd69f0)/0x2bd69f0 2014-06-09 16:22:12.035696 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.035698 7f868b759700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035702 7f8689755700 2 waiting 51 > 50 ops || 1126685 > 104857600 2014-06-09 16:22:12.035707 7f868df5e700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035710 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3815.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.035712 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5fc80] 2014-06-09 16:22:12.035714 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.035713 7f8689f56700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035716 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5fc80] 2014-06-09 16:22:12.035718 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.035719 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.035721 7f8691765700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035726 7f8692767700 2 waiting 51 > 50 ops || 1126650 > 104857600 2014-06-09 16:22:12.035730 7f8692f68700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035734 7f868c75b700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035738 7f868ff62700 2 waiting 51 > 50 ops || 1126650 > 104857600 2014-06-09 16:22:12.035741 7f868cf5c700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035746 7f8690763700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035751 7f86a5e11700 10 _finish_op 0x2c06f740 seq 3814 osr(2.57a 0x2ed1870)/0x2ed1870 2014-06-09 16:22:12.035761 7f868f761700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035763 7f868d75d700 10 op_submit_start 3864 2014-06-09 16:22:12.035765 7f868d75d700 5 queue_transactions (trailing journal) 3864 2014-06-09 16:22:12.035764 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b540] 2014-06-09 16:22:12.035766 7f868d75d700 5 queue_op 0x11fce660 seq 3864 osr(0.2cb 0x2bd6ed0) 828 bytes (queue has 50 ops and 1125824 bytes) 2014-06-09 16:22:12.035677 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.516) v4 from osd.4 2014-06-09 16:22:12.035769 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b540] 2014-06-09 16:22:12.035770 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x12c088c0 2014-06-09 16:22:12.035771 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.035771 7f868af58700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035772 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.035776 7f8691f66700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035776 7f869af78700 5 -- op tracker -- , seq: 3584, time: 2014-06-09 16:22:12.035773, event: started, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.035780 7f868ef60700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035785 7f8690f64700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035789 7f868e75f700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035777 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d06c0 (1 active) 2014-06-09 16:22:12.035794 7f868bf5a700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035795 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5259e0 (2 active) 2014-06-09 16:22:12.035795 7f869af78700 5 -- op tracker -- , seq: 3584, time: 2014-06-09 16:22:12.035792, event: done, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.035799 7f8689755700 2 waiting 51 > 50 ops || 1126685 > 104857600 2014-06-09 16:22:12.035792 7f86a5610700 10 _do_op 0xcabdf80 seq 3815 r = 0, finisher 0x3061e720 0 2014-06-09 16:22:12.035800 7f868d75d700 10 op_submit_finish 3864 2014-06-09 16:22:12.035804 7f868df5e700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.035803 7f86a5610700 10 _finish_op 0xcabdf80 seq 3815 osr(0.16c 0x2bd67f0)/0x2bd67f0 2014-06-09 16:22:12.035809 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xd9fa5a0] 2014-06-09 16:22:12.035813 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xd9fa5a0] 2014-06-09 16:22:12.035814 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.035804 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.6a8( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.035815 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.035818 7f86a5e11700 5 _do_op 0x100389c0 seq 3816 osr(3.3e6 0x3ed12a0)/0x3ed12a0 start 2014-06-09 16:22:12.035821 7f86a5e11700 10 _do_transaction on 0xc1f7800 2014-06-09 16:22:12.035803 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035825 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035827 7f86a5e11700 15 _omap_setkeys meta/1fe0a505/pglog_3.3e6/0//-1 2014-06-09 16:22:12.035838 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.035838 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035840 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035841 7f869af78700 20 osd.26 243 _dispatch 0xca43c00 pg_info(1 pgs e243:1.774) v4 2014-06-09 16:22:12.035843 7f868d75d700 5 queue_transactions existing osr(0.6a8 0x2bd7d90)/0x2bd7d90 2014-06-09 16:22:12.035846 7f869af78700 5 -- op tracker -- , seq: 3585, time: 2014-06-09 16:22:11.975387, event: header_read, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.035854 7f869af78700 5 -- op tracker -- , seq: 3585, time: 2014-06-09 16:22:11.975387, event: throttled, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.035865 7f869af78700 5 -- op tracker -- , seq: 3585, time: 2014-06-09 16:22:11.975398, event: all_read, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.035874 7f869af78700 5 -- op tracker -- , seq: 3585, time: 2014-06-09 16:22:12.035838, event: dispatched, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.035879 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe0a505/pglog_3.3e6/0//-1 2014-06-09 16:22:12.035882 7f869af78700 5 -- op tracker -- , seq: 3585, time: 2014-06-09 16:22:12.035880, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.035887 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.774) v4 from osd.2 2014-06-09 16:22:12.035890 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xca43c00 2014-06-09 16:22:12.035896 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.035895 7f869af78700 5 -- op tracker -- , seq: 3585, time: 2014-06-09 16:22:12.035893, event: started, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.035912 7f869af78700 5 -- op tracker -- , seq: 3585, time: 2014-06-09 16:22:12.035909, event: done, op: pg_info(1 pgs e243:1.774) 2014-06-09 16:22:12.035922 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.035925 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.035930 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.035945 7f86a5e11700 15 _omap_rmkeys meta/1fe0a505/pglog_3.3e6/0//-1 2014-06-09 16:22:12.035951 7f86a5e11700 10 _omap_rmkeys meta/1fe0a505/pglog_3.3e6/0//-1 r = 0 2014-06-09 16:22:12.035958 7f86a5e11700 15 _omap_setkeys meta/1fe0a505/pglog_3.3e6/0//-1 2014-06-09 16:22:12.035971 7f86a5e11700 15 _touch 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.036012 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.3e6_head oid bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.036035 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036036 7f86a5e11700 20 write_state seq is 30162 2014-06-09 16:22:12.036037 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036039 7f869af78700 20 osd.26 243 _dispatch 0x30c76c40 pg_info(1 pgs e243:1.5f8) v4 2014-06-09 16:22:12.036043 7f86a5e11700 20 set_header setting 30161 cid 3.3e6_head oid bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 parent seq 0 2014-06-09 16:22:12.036044 7f869af78700 5 -- op tracker -- , seq: 3586, time: 2014-06-09 16:22:11.987451, event: header_read, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.036055 7f86a5e11700 10 _touch 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 = 0 2014-06-09 16:22:12.036060 7f86a5e11700 15 _write 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 0~1081344 2014-06-09 16:22:12.036052 7f869af78700 5 -- op tracker -- , seq: 3586, time: 2014-06-09 16:22:11.987452, event: throttled, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.036076 7f869af78700 5 -- op tracker -- , seq: 3586, time: 2014-06-09 16:22:11.987464, event: all_read, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.036082 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.036083 7f869af78700 5 -- op tracker -- , seq: 3586, time: 2014-06-09 16:22:12.036035, event: dispatched, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.036090 7f869af78700 5 -- op tracker -- , seq: 3586, time: 2014-06-09 16:22:12.036088, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.036094 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.5f8) v4 from osd.4 2014-06-09 16:22:12.036097 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c76c40 2014-06-09 16:22:12.036102 7f869af78700 5 -- op tracker -- , seq: 3586, time: 2014-06-09 16:22:12.036100, event: started, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.036113 7f8691765700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036113 7f868f761700 10 op_submit_start 3865 2014-06-09 16:22:12.036116 7f868f761700 5 queue_transactions (trailing journal) 3865 2014-06-09 16:22:12.036117 7f868f761700 5 queue_op 0x33a33e0 seq 3865 osr(1.6f5 0x2ed0170) 828 bytes (queue has 50 ops and 1125824 bytes) 2014-06-09 16:22:12.036119 7f8692f68700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036124 7f8692767700 2 waiting 51 > 50 ops || 1126650 > 104857600 2014-06-09 16:22:12.036128 7f868ff62700 2 waiting 51 > 50 ops || 1126650 > 104857600 2014-06-09 16:22:12.036127 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0a20 (1 active) 2014-06-09 16:22:12.036131 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d10e0 (2 active) 2014-06-09 16:22:12.036132 7f868cf5c700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036136 7f8690763700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036135 7f86a5610700 5 _do_op 0xcabdaa0 seq 3817 osr(2.530 0x2ed19b0)/0x2ed19b0 start 2014-06-09 16:22:12.036137 7f868f761700 10 op_submit_finish 3865 2014-06-09 16:22:12.036139 7f86a5610700 10 _do_transaction on 0xc0d1440 2014-06-09 16:22:12.036140 7f868b759700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036144 7f868a757700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036144 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.036148 7f868c75b700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036141 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.4a5( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:12.036153 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061e720] 2014-06-09 16:22:12.036153 7f8689f56700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036159 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061e720] 2014-06-09 16:22:12.036159 7f868af58700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036161 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.036162 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.036163 7f8691f66700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036168 7f868ef60700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036173 7f8690f64700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036176 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.036177 7f868e75f700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036183 7f868bf5a700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036189 7f8689755700 2 waiting 51 > 50 ops || 1126685 > 104857600 2014-06-09 16:22:12.036193 7f868df5e700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036193 7f868f761700 5 queue_transactions existing osr(2.4a5 0x2ed17f0)/0x2ed17f0 2014-06-09 16:22:12.036197 7f868f761700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036205 7f868d75d700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.036133 7f869af78700 5 -- op tracker -- , seq: 3586, time: 2014-06-09 16:22:12.036130, event: done, op: pg_info(1 pgs e243:1.5f8) 2014-06-09 16:22:12.036211 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036213 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036229 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036230 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036232 7f869af78700 20 osd.26 243 _dispatch 0x270ce000 pg_info(1 pgs e243:3.292) v4 2014-06-09 16:22:12.036237 7f869af78700 5 -- op tracker -- , seq: 3587, time: 2014-06-09 16:22:12.035755, event: header_read, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036250 7f869af78700 5 -- op tracker -- , seq: 3587, time: 2014-06-09 16:22:12.035756, event: throttled, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036257 7f869af78700 5 -- op tracker -- , seq: 3587, time: 2014-06-09 16:22:12.035773, event: all_read, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036265 7f869af78700 5 -- op tracker -- , seq: 3587, time: 2014-06-09 16:22:12.036228, event: dispatched, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036272 7f869af78700 5 -- op tracker -- , seq: 3587, time: 2014-06-09 16:22:12.036270, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036278 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.292) v4 from osd.8 2014-06-09 16:22:12.036281 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x270ce000 2014-06-09 16:22:12.036286 7f869af78700 5 -- op tracker -- , seq: 3587, time: 2014-06-09 16:22:12.036284, event: started, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036303 7f869af78700 5 -- op tracker -- , seq: 3587, time: 2014-06-09 16:22:12.036300, event: done, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036367 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036369 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036382 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036384 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036385 7f869af78700 20 osd.26 243 _dispatch 0xca43a40 pg_info(1 pgs e243:3.1eb) v4 2014-06-09 16:22:12.036390 7f869af78700 5 -- op tracker -- , seq: 3588, time: 2014-06-09 16:22:11.975402, event: header_read, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.036400 7f869af78700 5 -- op tracker -- , seq: 3588, time: 2014-06-09 16:22:11.975403, event: throttled, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.036407 7f869af78700 5 -- op tracker -- , seq: 3588, time: 2014-06-09 16:22:11.975413, event: all_read, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.036414 7f869af78700 5 -- op tracker -- , seq: 3588, time: 2014-06-09 16:22:12.036382, event: dispatched, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.036423 7f869af78700 5 -- op tracker -- , seq: 3588, time: 2014-06-09 16:22:12.036421, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.036428 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1eb) v4 from osd.2 2014-06-09 16:22:12.036488 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xca43a40 2014-06-09 16:22:12.036494 7f869af78700 5 -- op tracker -- , seq: 3588, time: 2014-06-09 16:22:12.036492, event: started, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.036516 7f869af78700 5 -- op tracker -- , seq: 3588, time: 2014-06-09 16:22:12.036514, event: done, op: pg_info(1 pgs e243:3.1eb) 2014-06-09 16:22:12.036527 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036529 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036541 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036543 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036544 7f869af78700 20 osd.26 243 _dispatch 0x2fef41c0 pg_info(1 pgs e243:1.188) v4 2014-06-09 16:22:12.036549 7f869af78700 5 -- op tracker -- , seq: 3589, time: 2014-06-09 16:22:11.987502, event: header_read, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.036632 7f869af78700 5 -- op tracker -- , seq: 3589, time: 2014-06-09 16:22:11.987503, event: throttled, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.036640 7f869af78700 5 -- op tracker -- , seq: 3589, time: 2014-06-09 16:22:11.987516, event: all_read, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.036647 7f869af78700 5 -- op tracker -- , seq: 3589, time: 2014-06-09 16:22:12.036541, event: dispatched, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.036654 7f869af78700 5 -- op tracker -- , seq: 3589, time: 2014-06-09 16:22:12.036652, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.036659 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.188) v4 from osd.4 2014-06-09 16:22:12.036662 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x2fef41c0 2014-06-09 16:22:12.036667 7f869af78700 5 -- op tracker -- , seq: 3589, time: 2014-06-09 16:22:12.036665, event: started, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.036741 7f869af78700 5 -- op tracker -- , seq: 3589, time: 2014-06-09 16:22:12.036738, event: done, op: pg_info(1 pgs e243:1.188) 2014-06-09 16:22:12.036749 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036751 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036762 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036763 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.036765 7f869af78700 20 osd.26 243 _dispatch 0x30c00380 pg_info(1 pgs e243:3.292) v4 2014-06-09 16:22:12.036769 7f869af78700 5 -- op tracker -- , seq: 3590, time: 2014-06-09 16:22:12.036081, event: header_read, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036775 7f869af78700 5 -- op tracker -- , seq: 3590, time: 2014-06-09 16:22:12.036082, event: throttled, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036781 7f869af78700 5 -- op tracker -- , seq: 3590, time: 2014-06-09 16:22:12.036095, event: all_read, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036787 7f869af78700 5 -- op tracker -- , seq: 3590, time: 2014-06-09 16:22:12.036761, event: dispatched, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036797 7f869af78700 5 -- op tracker -- , seq: 3590, time: 2014-06-09 16:22:12.036794, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036801 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.292) v4 from osd.8 2014-06-09 16:22:12.036897 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c00380 2014-06-09 16:22:12.036903 7f869af78700 5 -- op tracker -- , seq: 3590, time: 2014-06-09 16:22:12.036901, event: started, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036920 7f869af78700 5 -- op tracker -- , seq: 3590, time: 2014-06-09 16:22:12.036918, event: done, op: pg_info(1 pgs e243:3.292) 2014-06-09 16:22:12.036987 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.036990 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037003 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037005 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037006 7f869af78700 20 osd.26 243 _dispatch 0xca43880 pg_info(1 pgs e243:2.4af) v4 2014-06-09 16:22:12.037012 7f869af78700 5 -- op tracker -- , seq: 3591, time: 2014-06-09 16:22:11.975417, event: header_read, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.037024 7f869af78700 5 -- op tracker -- , seq: 3591, time: 2014-06-09 16:22:11.975418, event: throttled, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.037031 7f869af78700 5 -- op tracker -- , seq: 3591, time: 2014-06-09 16:22:11.975428, event: all_read, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.037042 7f869af78700 5 -- op tracker -- , seq: 3591, time: 2014-06-09 16:22:12.037002, event: dispatched, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.037049 7f869af78700 5 -- op tracker -- , seq: 3591, time: 2014-06-09 16:22:12.037046, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.037111 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.4af) v4 from osd.2 2014-06-09 16:22:12.037115 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xca43880 2014-06-09 16:22:12.037121 7f869af78700 5 -- op tracker -- , seq: 3591, time: 2014-06-09 16:22:12.037119, event: started, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.037137 7f869af78700 5 -- op tracker -- , seq: 3591, time: 2014-06-09 16:22:12.037135, event: done, op: pg_info(1 pgs e243:2.4af) 2014-06-09 16:22:12.037146 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037147 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037160 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037162 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037163 7f869af78700 20 osd.26 243 _dispatch 0x230bae00 pg_info(1 pgs e243:1.755) v4 2014-06-09 16:22:12.037168 7f869af78700 5 -- op tracker -- , seq: 3592, time: 2014-06-09 16:22:11.987548, event: header_read, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.037233 7f869af78700 5 -- op tracker -- , seq: 3592, time: 2014-06-09 16:22:11.987549, event: throttled, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.037241 7f869af78700 5 -- op tracker -- , seq: 3592, time: 2014-06-09 16:22:11.987561, event: all_read, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.037248 7f869af78700 5 -- op tracker -- , seq: 3592, time: 2014-06-09 16:22:12.037159, event: dispatched, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.037256 7f869af78700 5 -- op tracker -- , seq: 3592, time: 2014-06-09 16:22:12.037253, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.037261 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.755) v4 from osd.4 2014-06-09 16:22:12.037264 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x230bae00 2014-06-09 16:22:12.037269 7f869af78700 5 -- op tracker -- , seq: 3592, time: 2014-06-09 16:22:12.037267, event: started, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.037286 7f869af78700 5 -- op tracker -- , seq: 3592, time: 2014-06-09 16:22:12.037284, event: done, op: pg_info(1 pgs e243:1.755) 2014-06-09 16:22:12.037294 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037297 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037400 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037402 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037404 7f869af78700 20 osd.26 243 _dispatch 0xca436c0 pg_info(1 pgs e243:3.712) v4 2014-06-09 16:22:12.037409 7f869af78700 5 -- op tracker -- , seq: 3593, time: 2014-06-09 16:22:11.975434, event: header_read, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.037416 7f869af78700 5 -- op tracker -- , seq: 3593, time: 2014-06-09 16:22:11.975434, event: throttled, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.037424 7f869af78700 5 -- op tracker -- , seq: 3593, time: 2014-06-09 16:22:11.975445, event: all_read, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.037469 7f869af78700 5 -- op tracker -- , seq: 3593, time: 2014-06-09 16:22:12.037399, event: dispatched, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.037479 7f869af78700 5 -- op tracker -- , seq: 3593, time: 2014-06-09 16:22:12.037477, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.037484 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.712) v4 from osd.2 2014-06-09 16:22:12.037487 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xca436c0 2014-06-09 16:22:12.037493 7f869af78700 5 -- op tracker -- , seq: 3593, time: 2014-06-09 16:22:12.037491, event: started, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.037515 7f869af78700 5 -- op tracker -- , seq: 3593, time: 2014-06-09 16:22:12.037513, event: done, op: pg_info(1 pgs e243:3.712) 2014-06-09 16:22:12.037528 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037530 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037543 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037544 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037546 7f869af78700 20 osd.26 243 _dispatch 0x3411d180 pg_info(1 pgs e243:0.71f) v4 2014-06-09 16:22:12.037551 7f869af78700 5 -- op tracker -- , seq: 3594, time: 2014-06-09 16:22:11.987599, event: header_read, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.037645 7f869af78700 5 -- op tracker -- , seq: 3594, time: 2014-06-09 16:22:11.987600, event: throttled, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.037652 7f869af78700 5 -- op tracker -- , seq: 3594, time: 2014-06-09 16:22:11.987612, event: all_read, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.037658 7f869af78700 5 -- op tracker -- , seq: 3594, time: 2014-06-09 16:22:12.037542, event: dispatched, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.037668 7f869af78700 5 -- op tracker -- , seq: 3594, time: 2014-06-09 16:22:12.037665, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.037673 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.71f) v4 from osd.4 2014-06-09 16:22:12.037676 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3411d180 2014-06-09 16:22:12.037685 7f869af78700 5 -- op tracker -- , seq: 3594, time: 2014-06-09 16:22:12.037683, event: started, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.037713 7f869af78700 5 -- op tracker -- , seq: 3594, time: 2014-06-09 16:22:12.037710, event: done, op: pg_info(1 pgs e243:0.71f) 2014-06-09 16:22:12.037730 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037733 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037746 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037738 7f86af56b700 5 -- op tracker -- , seq: 3595, time: 2014-06-09 16:22:12.034515, event: header_read, op: osd_op(client.4541.0:32285 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 1736704~2064384] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.037747 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037749 7f869af78700 20 osd.26 243 _dispatch 0xca43500 pg_info(1 pgs e243:3.1e1) v4 2014-06-09 16:22:12.037754 7f86af56b700 5 -- op tracker -- , seq: 3595, time: 2014-06-09 16:22:12.034516, event: throttled, op: osd_op(client.4541.0:32285 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 1736704~2064384] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.037766 7f86af56b700 5 -- op tracker -- , seq: 3595, time: 2014-06-09 16:22:12.037693, event: all_read, op: osd_op(client.4541.0:32285 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 1736704~2064384] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.037786 7f86af56b700 5 -- op tracker -- , seq: 3595, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32285 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 1736704~2064384] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.037795 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.037816 7f86af56b700 15 osd.26 243 enqueue_op 0x1b0d7600 prio 63 cost 2064384 latency 0.003301 osd_op(client.4541.0:32285 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 1736704~2064384] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.037795 7f869af78700 5 -- op tracker -- , seq: 3596, time: 2014-06-09 16:22:11.976928, event: header_read, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.037876 7f869af78700 5 -- op tracker -- , seq: 3596, time: 2014-06-09 16:22:11.976929, event: throttled, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.037884 7f869af78700 5 -- op tracker -- , seq: 3596, time: 2014-06-09 16:22:11.976942, event: all_read, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.037891 7f869af78700 5 -- op tracker -- , seq: 3596, time: 2014-06-09 16:22:12.037745, event: dispatched, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.037899 7f869af78700 5 -- op tracker -- , seq: 3596, time: 2014-06-09 16:22:12.037897, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.037904 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1e1) v4 from osd.2 2014-06-09 16:22:12.037907 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xca43500 2014-06-09 16:22:12.037912 7f869af78700 5 -- op tracker -- , seq: 3596, time: 2014-06-09 16:22:12.037910, event: started, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.037933 7f869af78700 5 -- op tracker -- , seq: 3596, time: 2014-06-09 16:22:12.037930, event: done, op: pg_info(1 pgs e243:3.1e1) 2014-06-09 16:22:12.037942 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037944 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037956 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.037958 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.037959 7f869af78700 20 osd.26 243 _dispatch 0x314741c0 pg_info(1 pgs e243:2.778) v4 2014-06-09 16:22:12.037964 7f869af78700 5 -- op tracker -- , seq: 3597, time: 2014-06-09 16:22:11.987648, event: header_read, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.037972 7f869af78700 5 -- op tracker -- , seq: 3597, time: 2014-06-09 16:22:11.987649, event: throttled, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.037979 7f869af78700 5 -- op tracker -- , seq: 3597, time: 2014-06-09 16:22:11.987662, event: all_read, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.037986 7f869af78700 5 -- op tracker -- , seq: 3597, time: 2014-06-09 16:22:12.037956, event: dispatched, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.037994 7f869af78700 5 -- op tracker -- , seq: 3597, time: 2014-06-09 16:22:12.037992, event: waiting_for_osdmap, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.038001 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:2.778) v4 from osd.4 2014-06-09 16:22:12.038004 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x314741c0 2014-06-09 16:22:12.038010 7f869af78700 5 -- op tracker -- , seq: 3597, time: 2014-06-09 16:22:12.038008, event: started, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.038025 7f869af78700 5 -- op tracker -- , seq: 3597, time: 2014-06-09 16:22:12.038023, event: done, op: pg_info(1 pgs e243:2.778) 2014-06-09 16:22:12.038033 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.038035 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.038049 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.038051 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.038052 7f869af78700 20 osd.26 243 _dispatch 0x30c76e00 pg_info(1 pgs e243:1.1fb) v4 2014-06-09 16:22:12.038129 7f869af78700 5 -- op tracker -- , seq: 3598, time: 2014-06-09 16:22:11.978283, event: header_read, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.038137 7f869af78700 5 -- op tracker -- , seq: 3598, time: 2014-06-09 16:22:11.978284, event: throttled, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.038144 7f869af78700 5 -- op tracker -- , seq: 3598, time: 2014-06-09 16:22:11.978298, event: all_read, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.038150 7f869af78700 5 -- op tracker -- , seq: 3598, time: 2014-06-09 16:22:12.038049, event: dispatched, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.038158 7f869af78700 5 -- op tracker -- , seq: 3598, time: 2014-06-09 16:22:12.038156, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.038165 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1fb) v4 from osd.2 2014-06-09 16:22:12.038168 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c76e00 2014-06-09 16:22:12.038173 7f869af78700 5 -- op tracker -- , seq: 3598, time: 2014-06-09 16:22:12.038171, event: started, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.038196 7f869af78700 5 -- op tracker -- , seq: 3598, time: 2014-06-09 16:22:12.038194, event: done, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.038209 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.038211 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.038223 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.038225 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.038227 7f869af78700 20 osd.26 243 _dispatch 0x31474000 pg_info(1 pgs e243:3.516) v4 2014-06-09 16:22:12.038232 7f869af78700 5 -- op tracker -- , seq: 3599, time: 2014-06-09 16:22:11.987757, event: header_read, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.038242 7f869af78700 5 -- op tracker -- , seq: 3599, time: 2014-06-09 16:22:11.987757, event: throttled, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.038249 7f869af78700 5 -- op tracker -- , seq: 3599, time: 2014-06-09 16:22:11.987772, event: all_read, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.038256 7f869af78700 5 -- op tracker -- , seq: 3599, time: 2014-06-09 16:22:12.038223, event: dispatched, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.038265 7f869af78700 5 -- op tracker -- , seq: 3599, time: 2014-06-09 16:22:12.038263, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.038270 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.516) v4 from osd.4 2014-06-09 16:22:12.038273 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x31474000 2014-06-09 16:22:12.038279 7f869af78700 5 -- op tracker -- , seq: 3599, time: 2014-06-09 16:22:12.038277, event: started, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.038298 7f869af78700 5 -- op tracker -- , seq: 3599, time: 2014-06-09 16:22:12.038295, event: done, op: pg_info(1 pgs e243:3.516) 2014-06-09 16:22:12.038382 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.038385 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.038397 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.038392 7f86af56b700 5 -- op tracker -- , seq: 3600, time: 2014-06-09 16:22:12.037842, event: header_read, op: osd_op(client.4541.0:32286 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 3801088~393216] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.038400 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.038402 7f869af78700 20 osd.26 243 _dispatch 0x280c4fc0 pg_info(1 pgs e243:0.772) v4 2014-06-09 16:22:12.038406 7f86af56b700 5 -- op tracker -- , seq: 3600, time: 2014-06-09 16:22:12.037843, event: throttled, op: osd_op(client.4541.0:32286 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 3801088~393216] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.038417 7f86af56b700 5 -- op tracker -- , seq: 3600, time: 2014-06-09 16:22:12.038308, event: all_read, op: osd_op(client.4541.0:32286 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 3801088~393216] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.038428 7f86af56b700 5 -- op tracker -- , seq: 3600, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32286 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 3801088~393216] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.038435 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.038438 7f869af78700 5 -- op tracker -- , seq: 3601, time: 2014-06-09 16:22:11.978324, event: header_read, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.038450 7f869af78700 5 -- op tracker -- , seq: 3601, time: 2014-06-09 16:22:11.978325, event: throttled, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.038451 7f86af56b700 15 osd.26 243 enqueue_op 0x30c6c700 prio 63 cost 393216 latency 0.000608 osd_op(client.4541.0:32286 rbd_data.115d6b8b4567.000000000000164d [set-alloc-hint object_size 4194304 write_size 4194304,write 3801088~393216] 3.cd13df99 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.038457 7f869af78700 5 -- op tracker -- , seq: 3601, time: 2014-06-09 16:22:11.978338, event: all_read, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.040241 7f86a5e11700 10 _generic_write 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 0~1081344 = 0 2014-06-09 16:22:12.040681 7f869af78700 5 -- op tracker -- , seq: 3601, time: 2014-06-09 16:22:12.038397, event: dispatched, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.040694 7f869af78700 5 -- op tracker -- , seq: 3601, time: 2014-06-09 16:22:12.040692, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.040702 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.772) v4 from osd.2 2014-06-09 16:22:12.040706 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c4fc0 2014-06-09 16:22:12.040710 7f86af56b700 5 -- op tracker -- , seq: 3602, time: 2014-06-09 16:22:12.038470, event: header_read, op: osd_op(client.4541.0:32306 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~507904] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.040724 7f86af56b700 5 -- op tracker -- , seq: 3602, time: 2014-06-09 16:22:12.038477, event: throttled, op: osd_op(client.4541.0:32306 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~507904] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.040729 7f86a5e11700 15 _setattrs 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.040735 7f86af56b700 5 -- op tracker -- , seq: 3602, time: 2014-06-09 16:22:12.039083, event: all_read, op: osd_op(client.4541.0:32306 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~507904] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.040743 7f86a5e11700 10 _setattrs 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 = 0 2014-06-09 16:22:12.040748 7f86af56b700 5 -- op tracker -- , seq: 3602, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32306 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~507904] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.040749 7f86a5e11700 15 _setattrs 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 2014-06-09 16:22:12.040757 7f86a5e11700 10 _setattrs 3.3e6_head/bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 = 0 2014-06-09 16:22:12.040756 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.040757 7f869af78700 5 -- op tracker -- , seq: 3601, time: 2014-06-09 16:22:12.040710, event: started, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.040774 7f86af56b700 15 osd.26 243 enqueue_op 0x30c6c600 prio 63 cost 507904 latency 0.002304 osd_op(client.4541.0:32306 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~507904] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.040778 7f869af78700 5 -- op tracker -- , seq: 3601, time: 2014-06-09 16:22:12.040776, event: done, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.040793 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.040796 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.040878 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.040880 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.040882 7f869af78700 20 osd.26 243 _dispatch 0xc2f3dc0 pg_info(1 pgs e243:0.5fb) v4 2014-06-09 16:22:12.040888 7f869af78700 5 -- op tracker -- , seq: 3603, time: 2014-06-09 16:22:11.988617, event: header_read, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.040895 7f869af78700 5 -- op tracker -- , seq: 3603, time: 2014-06-09 16:22:11.988618, event: throttled, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.040901 7f869af78700 5 -- op tracker -- , seq: 3603, time: 2014-06-09 16:22:11.988631, event: all_read, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.040907 7f869af78700 5 -- op tracker -- , seq: 3603, time: 2014-06-09 16:22:12.040878, event: dispatched, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.040918 7f869af78700 5 -- op tracker -- , seq: 3603, time: 2014-06-09 16:22:12.040915, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.040923 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5fb) v4 from osd.4 2014-06-09 16:22:12.040923 7f86a5e11700 10 cid: oid: bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 not skipping op, *spos 3816.0.0 2014-06-09 16:22:12.040928 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.040960 7f86a5e11700 20 set_header setting 30161 cid 3.3e6_head oid bd747be6/rbd_data.11786b8b4567.0000000000001732/head//3 parent seq 0 2014-06-09 16:22:12.040992 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xc2f3dc0 2014-06-09 16:22:12.041000 7f869af78700 5 -- op tracker -- , seq: 3603, time: 2014-06-09 16:22:12.040997, event: started, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041017 7f869af78700 5 -- op tracker -- , seq: 3603, time: 2014-06-09 16:22:12.041015, event: done, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041028 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041030 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041042 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041044 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041045 7f869af78700 20 osd.26 243 _dispatch 0x257cd880 pg_info(1 pgs e243:1.1fb) v4 2014-06-09 16:22:12.041054 7f86a5e11700 10 cid: oid: 1fe0a505/pglog_3.3e6/0//-1 skipping op, spos 3816.0.0 <= header.spos 147272.0.0 2014-06-09 16:22:12.041061 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3816.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.041064 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.041087 7f869af78700 5 -- op tracker -- , seq: 3604, time: 2014-06-09 16:22:11.978349, event: header_read, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.041096 7f869af78700 5 -- op tracker -- , seq: 3604, time: 2014-06-09 16:22:11.978349, event: throttled, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.041107 7f869af78700 5 -- op tracker -- , seq: 3604, time: 2014-06-09 16:22:11.978360, event: all_read, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.041118 7f869af78700 5 -- op tracker -- , seq: 3604, time: 2014-06-09 16:22:12.041042, event: dispatched, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.041126 7f869af78700 5 -- op tracker -- , seq: 3604, time: 2014-06-09 16:22:12.041123, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.041132 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.1fb) v4 from osd.2 2014-06-09 16:22:12.041135 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x257cd880 2014-06-09 16:22:12.041140 7f869af78700 5 -- op tracker -- , seq: 3604, time: 2014-06-09 16:22:12.041138, event: started, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.041157 7f869af78700 5 -- op tracker -- , seq: 3604, time: 2014-06-09 16:22:12.041155, event: done, op: pg_info(1 pgs e243:1.1fb) 2014-06-09 16:22:12.041165 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041167 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041243 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041245 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041247 7f869af78700 20 osd.26 243 _dispatch 0xc2f3c00 pg_info(1 pgs e243:0.5fb) v4 2014-06-09 16:22:12.041252 7f869af78700 5 -- op tracker -- , seq: 3605, time: 2014-06-09 16:22:11.988664, event: header_read, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041259 7f869af78700 5 -- op tracker -- , seq: 3605, time: 2014-06-09 16:22:11.988665, event: throttled, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041266 7f869af78700 5 -- op tracker -- , seq: 3605, time: 2014-06-09 16:22:11.988677, event: all_read, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041272 7f869af78700 5 -- op tracker -- , seq: 3605, time: 2014-06-09 16:22:12.041243, event: dispatched, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041284 7f869af78700 5 -- op tracker -- , seq: 3605, time: 2014-06-09 16:22:12.041282, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041289 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.5fb) v4 from osd.4 2014-06-09 16:22:12.041292 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0xc2f3c00 2014-06-09 16:22:12.041297 7f869af78700 5 -- op tracker -- , seq: 3605, time: 2014-06-09 16:22:12.041295, event: started, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041367 7f869af78700 5 -- op tracker -- , seq: 3605, time: 2014-06-09 16:22:12.041364, event: done, op: pg_info(1 pgs e243:0.5fb) 2014-06-09 16:22:12.041380 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041382 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041395 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041397 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041399 7f869af78700 20 osd.26 243 _dispatch 0x3411d340 pg_info(1 pgs e243:0.772) v4 2014-06-09 16:22:12.041404 7f869af78700 5 -- op tracker -- , seq: 3606, time: 2014-06-09 16:22:11.978368, event: header_read, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.041410 7f869af78700 5 -- op tracker -- , seq: 3606, time: 2014-06-09 16:22:11.978369, event: throttled, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.041417 7f869af78700 5 -- op tracker -- , seq: 3606, time: 2014-06-09 16:22:11.978381, event: all_read, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.041423 7f869af78700 5 -- op tracker -- , seq: 3606, time: 2014-06-09 16:22:12.041395, event: dispatched, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.041450 7f869af78700 5 -- op tracker -- , seq: 3606, time: 2014-06-09 16:22:12.041447, event: waiting_for_osdmap, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.041455 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:0.772) v4 from osd.2 2014-06-09 16:22:12.041458 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3411d340 2014-06-09 16:22:12.041463 7f869af78700 5 -- op tracker -- , seq: 3606, time: 2014-06-09 16:22:12.041461, event: started, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.041477 7f869af78700 5 -- op tracker -- , seq: 3606, time: 2014-06-09 16:22:12.041475, event: done, op: pg_info(1 pgs e243:0.772) 2014-06-09 16:22:12.041486 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041487 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041499 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041500 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041501 7f869af78700 20 osd.26 243 _dispatch 0x270cfdc0 pg_info(1 pgs e243:1.76e) v4 2014-06-09 16:22:12.041506 7f869af78700 5 -- op tracker -- , seq: 3607, time: 2014-06-09 16:22:11.990155, event: header_read, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041513 7f869af78700 5 -- op tracker -- , seq: 3607, time: 2014-06-09 16:22:11.990156, event: throttled, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041525 7f869af78700 5 -- op tracker -- , seq: 3607, time: 2014-06-09 16:22:11.990169, event: all_read, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041531 7f869af78700 5 -- op tracker -- , seq: 3607, time: 2014-06-09 16:22:12.041499, event: dispatched, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041538 7f869af78700 5 -- op tracker -- , seq: 3607, time: 2014-06-09 16:22:12.041536, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041543 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.76e) v4 from osd.4 2014-06-09 16:22:12.041546 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x270cfdc0 2014-06-09 16:22:12.041609 7f869af78700 5 -- op tracker -- , seq: 3607, time: 2014-06-09 16:22:12.041549, event: started, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041631 7f869af78700 5 -- op tracker -- , seq: 3607, time: 2014-06-09 16:22:12.041629, event: done, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041639 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041641 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041654 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041655 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.041656 7f869af78700 20 osd.26 243 _dispatch 0x3145ddc0 pg_info(1 pgs e243:1.76e) v4 2014-06-09 16:22:12.041662 7f869af78700 5 -- op tracker -- , seq: 3608, time: 2014-06-09 16:22:11.990192, event: header_read, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041669 7f869af78700 5 -- op tracker -- , seq: 3608, time: 2014-06-09 16:22:11.990193, event: throttled, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041719 7f869af78700 5 -- op tracker -- , seq: 3608, time: 2014-06-09 16:22:11.990209, event: all_read, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041727 7f869af78700 5 -- op tracker -- , seq: 3608, time: 2014-06-09 16:22:12.041653, event: dispatched, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041735 7f869af78700 5 -- op tracker -- , seq: 3608, time: 2014-06-09 16:22:12.041733, event: waiting_for_osdmap, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041739 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:1.76e) v4 from osd.4 2014-06-09 16:22:12.041742 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3145ddc0 2014-06-09 16:22:12.041748 7f869af78700 5 -- op tracker -- , seq: 3608, time: 2014-06-09 16:22:12.041746, event: started, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041763 7f869af78700 5 -- op tracker -- , seq: 3608, time: 2014-06-09 16:22:12.041761, event: done, op: pg_info(1 pgs e243:1.76e) 2014-06-09 16:22:12.041771 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.041773 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.044102 7f86af56b700 5 -- op tracker -- , seq: 3609, time: 2014-06-09 16:22:12.040798, event: header_read, op: osd_op(client.4541.0:32307 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 507904~2064384] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.044115 7f86af56b700 5 -- op tracker -- , seq: 3609, time: 2014-06-09 16:22:12.040800, event: throttled, op: osd_op(client.4541.0:32307 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 507904~2064384] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.044126 7f86af56b700 5 -- op tracker -- , seq: 3609, time: 2014-06-09 16:22:12.044061, event: all_read, op: osd_op(client.4541.0:32307 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 507904~2064384] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.044134 7f86af56b700 5 -- op tracker -- , seq: 3609, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32307 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 507904~2064384] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.044142 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.044167 7f86af56b700 15 osd.26 243 enqueue_op 0x30c6c500 prio 63 cost 2064384 latency 0.003369 osd_op(client.4541.0:32307 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 507904~2064384] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.045380 7f86af56b700 5 -- op tracker -- , seq: 3610, time: 2014-06-09 16:22:12.044195, event: header_read, op: osd_op(client.4541.0:32308 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 2572288~1032192] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.045390 7f86af56b700 5 -- op tracker -- , seq: 3610, time: 2014-06-09 16:22:12.044197, event: throttled, op: osd_op(client.4541.0:32308 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 2572288~1032192] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.045400 7f86af56b700 5 -- op tracker -- , seq: 3610, time: 2014-06-09 16:22:12.045350, event: all_read, op: osd_op(client.4541.0:32308 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 2572288~1032192] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.045410 7f86af56b700 5 -- op tracker -- , seq: 3610, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32308 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 2572288~1032192] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.045417 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.045438 7f86af56b700 15 osd.26 243 enqueue_op 0x30c6c400 prio 63 cost 1032192 latency 0.001243 osd_op(client.4541.0:32308 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 2572288~1032192] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.045727 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.045750 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3817.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.045755 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.045769 7f86a5610700 10 _do_op 0xcabdaa0 seq 3817 r = 0, finisher 0xbd71d80 0 2014-06-09 16:22:12.045778 7f86a5610700 10 _finish_op 0xcabdaa0 seq 3817 osr(2.530 0x2ed19b0)/0x2ed19b0 2014-06-09 16:22:12.045781 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x275cec20] 2014-06-09 16:22:12.045784 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x275cec20] 2014-06-09 16:22:12.045786 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.045787 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.045790 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d10e0 (1 active) 2014-06-09 16:22:12.045793 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcd80 (2 active) 2014-06-09 16:22:12.045795 7f8691765700 10 op_submit_start 3866 2014-06-09 16:22:12.045795 7f86a5610700 5 _do_op 0xd04f2c0 seq 3818 osr(3.7a 0x3ed0220)/0x3ed0220 start 2014-06-09 16:22:12.045796 7f8691765700 5 queue_transactions (trailing journal) 3866 2014-06-09 16:22:12.045798 7f86a5610700 10 _do_transaction on 0x3419200 2014-06-09 16:22:12.045798 7f8691765700 5 queue_op 0xc34b560 seq 3866 osr(0.71b 0x384c050) 828 bytes (queue has 50 ops and 1125824 bytes) 2014-06-09 16:22:12.045802 7f8691765700 10 op_submit_finish 3866 2014-06-09 16:22:12.045803 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.045807 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.d1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.d1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.045840 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.045854 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3818.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.045857 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.045857 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.045865 7f8691765700 5 queue_transactions existing osr(1.d1 0x384c9b0)/0x384c9b0 2014-06-09 16:22:12.045868 7f8691765700 2 waiting 51 > 50 ops || 1126650 > 104857600 2014-06-09 16:22:12.045874 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd71d80] 2014-06-09 16:22:12.045879 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd71d80] 2014-06-09 16:22:12.045884 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.045886 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.045890 7f8692f68700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.045896 7f8692767700 2 waiting 51 > 50 ops || 1126650 > 104857600 2014-06-09 16:22:12.045896 7f86a5610700 10 _do_op 0xd04f2c0 seq 3818 r = 0, finisher 0xbd60480 0 2014-06-09 16:22:12.045902 7f868ff62700 2 waiting 51 > 50 ops || 1126650 > 104857600 2014-06-09 16:22:12.045903 7f86a5610700 10 _finish_op 0xd04f2c0 seq 3818 osr(3.7a 0x3ed0220)/0x3ed0220 2014-06-09 16:22:12.045912 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd60420] 2014-06-09 16:22:12.045914 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd60420] 2014-06-09 16:22:12.045914 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcd80 (1 active) 2014-06-09 16:22:12.045916 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.045916 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.045916 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5258c0 (2 active) 2014-06-09 16:22:12.045918 7f86a5610700 5 _do_op 0x324e6180 seq 3819 osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 start 2014-06-09 16:22:12.045921 7f86a5610700 10 _do_transaction on 0x344d8c0 2014-06-09 16:22:12.045922 7f8690763700 10 op_submit_start 3867 2014-06-09 16:22:12.045923 7f8690763700 5 queue_transactions (trailing journal) 3867 2014-06-09 16:22:12.045924 7f8690763700 5 queue_op 0x30d96300 seq 3867 osr(2.182 0x2ed0990) 828 bytes (queue has 50 ops and 1125826 bytes) 2014-06-09 16:22:12.045926 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.045928 7f8690763700 10 op_submit_finish 3867 2014-06-09 16:22:12.045931 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.254( v 213'590 (0'0,213'590] local-les=243 n=114 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:12.045961 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.045966 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.045972 7f8690763700 5 queue_transactions existing osr(3.254 0x3ed0c30)/0x3ed0c30 2014-06-09 16:22:12.045974 7f8690763700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.045974 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3819.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.045978 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.045979 7f868b759700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.045985 7f868a757700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.045990 7f868c75b700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.045991 7f86a5610700 10 _do_op 0x324e6180 seq 3819 r = 0, finisher 0x33a6a00 0 2014-06-09 16:22:12.045996 7f86a5610700 10 _finish_op 0x324e6180 seq 3819 osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 2014-06-09 16:22:12.045997 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd75500] 2014-06-09 16:22:12.045999 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd75500] 2014-06-09 16:22:12.046000 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.046001 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd60480] 2014-06-09 16:22:12.046001 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.045999 7f8689f56700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046003 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd60480] 2014-06-09 16:22:12.046005 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.046006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.046008 7f8691765700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.046014 7f8691f66700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046020 7f868ef60700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046025 7f8690f64700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046030 7f868e75f700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046035 7f868bf5a700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046040 7f8689755700 2 waiting 51 > 50 ops || 1126687 > 104857600 2014-06-09 16:22:12.046045 7f868df5e700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046143 7f8682de8700 5 -- op tracker -- , seq: 3611, time: 2014-06-09 16:22:12.031697, event: header_read, op: MOSDPGPush(3.113 243 [PushOp(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3, version: 227'654, data_included: [0~3776512], data_size: 3776512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3@227'654, copy_subset: [0~3776512], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3776512, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.046158 7f8682de8700 5 -- op tracker -- , seq: 3611, time: 2014-06-09 16:22:12.031698, event: throttled, op: MOSDPGPush(3.113 243 [PushOp(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3, version: 227'654, data_included: [0~3776512], data_size: 3776512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3@227'654, copy_subset: [0~3776512], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3776512, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.046170 7f8682de8700 5 -- op tracker -- , seq: 3611, time: 2014-06-09 16:22:12.046106, event: all_read, op: MOSDPGPush(3.113 243 [PushOp(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3, version: 227'654, data_included: [0~3776512], data_size: 3776512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3@227'654, copy_subset: [0~3776512], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3776512, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.046182 7f8682de8700 5 -- op tracker -- , seq: 3611, time: 0.000000, event: dispatched, op: MOSDPGPush(3.113 243 [PushOp(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3, version: 227'654, data_included: [0~3776512], data_size: 3776512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3@227'654, copy_subset: [0~3776512], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3776512, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.046188 7f8682de8700 10 osd.26 243 handle_replica_op MOSDPGPush(3.113 243 [PushOp(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3, version: 227'654, data_included: [0~3776512], data_size: 3776512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3@227'654, copy_subset: [0~3776512], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3776512, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 epoch 243 2014-06-09 16:22:12.046197 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:12.046204 7f8682de8700 15 osd.26 243 enqueue_op 0x32827d00 prio 63 cost 3777512 latency 0.014507 MOSDPGPush(3.113 243 [PushOp(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3, version: 227'654, data_included: [0~3776512], data_size: 3776512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3@227'654, copy_subset: [0~3776512], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3776512, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 2014-06-09 16:22:12.046251 7f86a5e11700 10 _do_op 0x100389c0 seq 3816 r = 0, finisher 0x3251cee0 0x2f086c80 2014-06-09 16:22:12.046260 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f8b60] 2014-06-09 16:22:12.046262 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=213'623 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 23 2014-06-09 16:22:12.046279 7f868f761700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046284 7f86a460e700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.046277, event: op_commit, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.046288 7f868d75d700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046292 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 luod=213'623 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 23 all committed 2014-06-09 16:22:12.046300 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.046303 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.046305 7f869af78700 20 osd.26 243 _dispatch 0x25fff500 MRecoveryReserve REQUEST pgid: 3.113, query_epoch: 243 v2 2014-06-09 16:22:12.046311 7f869af78700 5 -- op tracker -- , seq: 3612, time: 2014-06-09 16:22:12.046227, event: header_read, op: MRecoveryReserve REQUEST pgid: 3.113, query_epoch: 243 2014-06-09 16:22:12.046302 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] eval_repop repgather(0xc1f2300 243'624 rep_tid=23 committed?=1 applied?=0 lock=0 op=osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.046318 7f869af78700 5 -- op tracker -- , seq: 3612, time: 2014-06-09 16:22:12.046228, event: throttled, op: MRecoveryReserve REQUEST pgid: 3.113, query_epoch: 243 2014-06-09 16:22:12.046324 7f869af78700 5 -- op tracker -- , seq: 3612, time: 2014-06-09 16:22:12.046234, event: all_read, op: MRecoveryReserve REQUEST pgid: 3.113, query_epoch: 243 2014-06-09 16:22:12.046318 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] log_op_stats osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1081728 outb 0 rlat 0.000000 lat 1.422502 2014-06-09 16:22:12.046336 7f869af78700 5 -- op tracker -- , seq: 3612, time: 2014-06-09 16:22:12.046299, event: dispatched, op: MRecoveryReserve REQUEST pgid: 3.113, query_epoch: 243 2014-06-09 16:22:12.046336 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] publish_stats_to_osd 243:1023 2014-06-09 16:22:12.046343 7f869af78700 5 -- op tracker -- , seq: 3612, time: 2014-06-09 16:22:12.046341, event: waiting_for_osdmap, op: MRecoveryReserve REQUEST pgid: 3.113, query_epoch: 243 2014-06-09 16:22:12.046357 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x25fff500 2014-06-09 16:22:12.046347 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] sending commit on repgather(0xc1f2300 243'624 rep_tid=23 committed?=1 applied?=0 lock=0 op=osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4) 0xdfb0280 2014-06-09 16:22:12.046364 7f86af56b700 5 -- op tracker -- , seq: 3613, time: 2014-06-09 16:22:12.045462, event: header_read, op: osd_op(client.4541.0:32309 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 3604480~516096] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.046384 7f86a460e700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.046378, event: commit_sent, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.046394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f8b60] 2014-06-09 16:22:12.046396 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.046397 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.046405 7f8692f68700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046400 7f86af56b700 5 -- op tracker -- , seq: 3613, time: 2014-06-09 16:22:12.045463, event: throttled, op: osd_op(client.4541.0:32309 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 3604480~516096] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.046414 7f86af56b700 5 -- op tracker -- , seq: 3613, time: 2014-06-09 16:22:12.046326, event: all_read, op: osd_op(client.4541.0:32309 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 3604480~516096] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.046425 7f86af56b700 5 -- op tracker -- , seq: 3613, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32309 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 3604480~516096] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.046431 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.046434 7f869af78700 5 -- op tracker -- , seq: 3612, time: 2014-06-09 16:22:12.046368, event: done, op: MRecoveryReserve REQUEST pgid: 3.113, query_epoch: 243 2014-06-09 16:22:12.046450 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.046452 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.046448 7f86af56b700 15 osd.26 243 enqueue_op 0x30c6c300 prio 63 cost 516096 latency 0.000986 osd_op(client.4541.0:32309 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 3604480~516096] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.046454 7f8692767700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.046460 7f868ff62700 2 waiting 51 > 50 ops || 1126652 > 104857600 2014-06-09 16:22:12.046464 7f868cf5c700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046465 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.046466 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.046468 7f869af78700 20 osd.26 243 _dispatch 0x190b6fc0 MRecoveryReserve REQUEST pgid: 3.450, query_epoch: 243 v2 2014-06-09 16:22:12.046473 7f869af78700 5 -- op tracker -- , seq: 3614, time: 2014-06-09 16:22:12.046248, event: header_read, op: MRecoveryReserve REQUEST pgid: 3.450, query_epoch: 243 2014-06-09 16:22:12.046475 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5258c0 (1 active) 2014-06-09 16:22:12.046478 7f868af58700 10 op_submit_start 3868 2014-06-09 16:22:12.046479 7f868b759700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046479 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd320 (2 active) 2014-06-09 16:22:12.046480 7f868af58700 5 queue_transactions (trailing journal) 3868 2014-06-09 16:22:12.046480 7f869af78700 5 -- op tracker -- , seq: 3614, time: 2014-06-09 16:22:12.046249, event: throttled, op: MRecoveryReserve REQUEST pgid: 3.450, query_epoch: 243 2014-06-09 16:22:12.046484 7f868a757700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046482 7f868af58700 5 queue_op 0x30d978c0 seq 3868 osr(3.53d 0x3ed1600) 828 bytes (queue has 50 ops and 1125826 bytes) 2014-06-09 16:22:12.046483 7f86a5610700 5 _do_op 0xd04ed80 seq 3820 osr(2.5be 0x2ed1b30)/0x2ed1b30 start 2014-06-09 16:22:12.046486 7f868af58700 10 op_submit_finish 3868 2014-06-09 16:22:12.046487 7f86a5610700 10 _do_transaction on 0x3562900 2014-06-09 16:22:12.046488 7f868c75b700 2 waiting 51 > 50 ops || 1126654 > 104857600 2014-06-09 16:22:12.046486 7f869af78700 5 -- op tracker -- , seq: 3614, time: 2014-06-09 16:22:12.046258, event: all_read, op: MRecoveryReserve REQUEST pgid: 3.450, query_epoch: 243 2014-06-09 16:22:12.046492 7f86a5e11700 10 _finish_op 0x100389c0 seq 3816 osr(3.3e6 0x3ed12a0)/0x3ed12a0 2014-06-09 16:22:12.046493 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.046492 7f869af78700 5 -- op tracker -- , seq: 3614, time: 2014-06-09 16:22:12.046464, event: dispatched, op: MRecoveryReserve REQUEST pgid: 3.450, query_epoch: 243 2014-06-09 16:22:12.046490 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.5a2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.5a2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.046499 7f869af78700 5 -- op tracker -- , seq: 3614, time: 2014-06-09 16:22:12.046497, event: waiting_for_osdmap, op: MRecoveryReserve REQUEST pgid: 3.450, query_epoch: 243 2014-06-09 16:22:12.046501 7f8691765700 10 op_submit_start 3869 2014-06-09 16:22:12.046503 7f8691765700 5 queue_transactions (trailing journal) 3869 2014-06-09 16:22:12.046503 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x190b6fc0 2014-06-09 16:22:12.046504 7f8691765700 5 queue_op 0xc34b800 seq 3869 osr(1.d1 0x384c9b0) 826 bytes (queue has 50 ops and 42910 bytes) 2014-06-09 16:22:12.046510 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5259e0 (1 active) 2014-06-09 16:22:12.046514 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd560 (2 active) 2014-06-09 16:22:12.046512 7f869af78700 5 -- op tracker -- , seq: 3614, time: 2014-06-09 16:22:12.046510, event: done, op: MRecoveryReserve REQUEST pgid: 3.450, query_epoch: 243 2014-06-09 16:22:12.046518 7f8691765700 10 op_submit_finish 3869 2014-06-09 16:22:12.046519 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.046521 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.046523 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.046529 7f868af58700 5 queue_transactions existing osr(0.5a2 0x2bd7af0)/0x2bd7af0 2014-06-09 16:22:12.046531 7f868af58700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046521 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.6f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.046533 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.046534 7f8689f56700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046538 7f8691f66700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046546 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3820.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.046550 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.046553 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.046558 7f8691765700 5 queue_transactions existing osr(1.6f 0x384c570)/0x384c570 2014-06-09 16:22:12.046560 7f8691765700 2 waiting 51 > 50 ops || 43736 > 104857600 2014-06-09 16:22:12.046564 7f868ef60700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046569 7f8690f64700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046574 7f868e75f700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046578 7f8689755700 2 waiting 51 > 50 ops || 43771 > 104857600 2014-06-09 16:22:12.046576 7f86a5610700 10 _do_op 0xd04ed80 seq 3820 r = 0, finisher 0xbd60b00 0 2014-06-09 16:22:12.046581 7f868df5e700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046583 7f86a5610700 10 _finish_op 0xd04ed80 seq 3820 osr(2.5be 0x2ed1b30)/0x2ed1b30 2014-06-09 16:22:12.046586 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd60a80] 2014-06-09 16:22:12.046588 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd60a80] 2014-06-09 16:22:12.046589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.046590 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.046589 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd320 (1 active) 2014-06-09 16:22:12.046590 7f868af58700 10 op_submit_start 3870 2014-06-09 16:22:12.046592 7f868af58700 5 queue_transactions (trailing journal) 3870 2014-06-09 16:22:12.046591 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301e60 (2 active) 2014-06-09 16:22:12.046593 7f8692f68700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046593 7f868af58700 5 queue_op 0x30d97860 seq 3870 osr(0.5a2 0x2bd7af0) 828 bytes (queue has 50 ops and 42910 bytes) 2014-06-09 16:22:12.046594 7f86a5610700 5 _do_op 0x30a2ea0 seq 3822 osr(1.206 0x384cd10)/0x384cd10 start 2014-06-09 16:22:12.046595 7f868af58700 10 op_submit_finish 3870 2014-06-09 16:22:12.046596 7f86a5610700 10 _do_transaction on 0xc10b5c0 2014-06-09 16:22:12.046597 7f8692767700 2 waiting 51 > 50 ops || 43736 > 104857600 2014-06-09 16:22:12.046601 7f868ff62700 2 waiting 51 > 50 ops || 43736 > 104857600 2014-06-09 16:22:12.046601 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.046605 7f868cf5c700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046598 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 243/191 242/242/220) 2014-06-09 16:22:12.046609 7f8690763700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046613 7f868a757700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046617 7f868c75b700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046620 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.046621 7f868b759700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046624 7f868af58700 5 queue_transactions existing osr(3.778 0x3ed1df0)/0x3ed1df0 2014-06-09 16:22:12.046625 7f86a5e11700 5 _do_op 0xcabc240 seq 3821 osr(0.5b7 0x2bd7a90)/0x2bd7a90 start 2014-06-09 16:22:12.046626 7f868af58700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046628 7f86a5e11700 10 _do_transaction on 0x399dd40 2014-06-09 16:22:12.046632 7f8691f66700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046633 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.046639 7f8691765700 2 waiting 51 > 50 ops || 43736 > 104857600 2014-06-09 16:22:12.046641 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.046643 7f868ef60700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046647 7f8690f64700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046651 7f868e75f700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046655 7f8689755700 2 waiting 51 > 50 ops || 43771 > 104857600 2014-06-09 16:22:12.046656 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3822.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.046659 7f868df5e700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046660 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.046664 7f868f761700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046671 7f8689f56700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046671 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a6a00,0x3251cee0,0xbd60b00] 2014-06-09 16:22:12.046675 7f868d75d700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.046675 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] op_applied: 23 2014-06-09 16:22:12.046694 7f86a4e0f700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.046688, event: op_applied, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.046702 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] op_applied on primary on version 243'624 2014-06-09 16:22:12.046716 7f86a5610700 10 _do_op 0x30a2ea0 seq 3822 r = 0, finisher 0xcc942c0 0 2014-06-09 16:22:12.046713 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] repop_all_applied: repop tid 23 all applied 2014-06-09 16:22:12.046721 7f86a5610700 10 _finish_op 0x30a2ea0 seq 3822 osr(1.206 0x384cd10)/0x384cd10 2014-06-09 16:22:12.046726 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301e60 (1 active) 2014-06-09 16:22:12.046728 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0360 (2 active) 2014-06-09 16:22:12.046730 7f86a5610700 5 _do_op 0xbf7f680 seq 3823 osr(3.579 0x3ed18d0)/0x3ed18d0 start 2014-06-09 16:22:12.046732 7f86a5610700 10 _do_transaction on 0xbf2f380 2014-06-09 16:22:12.046722 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] eval_repop repgather(0xc1f2300 243'624 rep_tid=23 committed?=1 applied?=1 lock=0 op=osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.046737 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.046739 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] log_op_stats osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1081728 outb 0 rlat 0.000000 lat 1.422923 2014-06-09 16:22:12.046755 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] publish_stats_to_osd 243:1024 2014-06-09 16:22:12.046772 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.046766 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.046786 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3823.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.046776 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 213'623 active] removing repgather(0xc1f2300 243'624 rep_tid=23 committed?=1 applied?=1 op=osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.046789 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.046800 7f86a5610700 10 _do_op 0xbf7f680 seq 3823 r = 0, finisher 0xcdb27e0 0 2014-06-09 16:22:12.046790 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 213'623 active] q front is repgather(0xc1f2300 243'624 rep_tid=23 committed?=1 applied?=1 op=osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.046803 7f86a5610700 10 _finish_op 0xbf7f680 seq 3823 osr(3.579 0x3ed18d0)/0x3ed18d0 2014-06-09 16:22:12.046805 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0360 (1 active) 2014-06-09 16:22:12.046807 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6c60 (2 active) 2014-06-09 16:22:12.046809 7f86a5610700 5 _do_op 0xbe7cf00 seq 3824 osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 start 2014-06-09 16:22:12.046811 7f86a5610700 10 _do_transaction on 0xc3f3140 2014-06-09 16:22:12.046803 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 213'623 active] remove_repop repgather(0xc1f2300 243'624 rep_tid=23 committed?=1 applied?=1 op=osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.046816 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.046816 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.3e6( v 243'624 (0'0,243'624] local-les=243 n=122 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 213'623 mlcod 213'623 active] requeue_ops 2014-06-09 16:22:12.046843 7f86a4e0f700 5 -- op tracker -- , seq: 597, time: 2014-06-09 16:22:12.046838, event: done, op: osd_op(client.4568.0:35343 rbd_data.11786b8b4567.0000000000001732 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1081344] 3.bd747be6 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.046848 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.046861 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3824.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.046865 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.046870 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a6a00,0x3251cee0,0xbd60b00] 2014-06-09 16:22:12.046873 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc942c0,0xcdb27e0] 2014-06-09 16:22:12.046879 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc942c0,0xcdb27e0] 2014-06-09 16:22:12.046881 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.046882 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.046882 7f86a5610700 10 _do_op 0xbe7cf00 seq 3824 r = 0, finisher 0x15a3ee40 0 2014-06-09 16:22:12.046885 7f86a5610700 10 _finish_op 0xbe7cf00 seq 3824 osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 2014-06-09 16:22:12.046888 7f868bf5a700 10 op_submit_start 3871 2014-06-09 16:22:12.046890 7f868bf5a700 5 queue_transactions (trailing journal) 3871 2014-06-09 16:22:12.046890 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6c60 (1 active) 2014-06-09 16:22:12.046892 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fca20 (2 active) 2014-06-09 16:22:12.046891 7f868bf5a700 5 queue_op 0x15996180 seq 3871 osr(3.136 0x3ed07a0) 828 bytes (queue has 48 ops and 41254 bytes) 2014-06-09 16:22:12.046894 7f868bf5a700 10 op_submit_finish 3871 2014-06-09 16:22:12.046894 7f86a5610700 5 _do_op 0x30a2960 seq 3825 osr(1.71e 0x2ed00b0)/0x2ed00b0 start 2014-06-09 16:22:12.046897 7f86a5610700 10 _do_transaction on 0xc337200 2014-06-09 16:22:12.046902 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.046897 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.644( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.644( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.046931 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.046936 7f868bf5a700 5 queue_transactions existing osr(0.644 0x2bd7bf0)/0x2bd7bf0 2014-06-09 16:22:12.046936 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.046939 7f868bf5a700 10 op_submit_start 3872 2014-06-09 16:22:12.046940 7f868bf5a700 5 queue_transactions (trailing journal) 3872 2014-06-09 16:22:12.046941 7f868bf5a700 5 queue_op 0x15996f60 seq 3872 osr(0.644 0x2bd7bf0) 828 bytes (queue has 49 ops and 42082 bytes) 2014-06-09 16:22:12.046944 7f868bf5a700 10 op_submit_finish 3872 2014-06-09 16:22:12.046949 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3825.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.046952 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.046946 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.2f8( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.046961 7f86a5610700 10 _do_op 0x30a2960 seq 3825 r = 0, finisher 0xbd5c220 0 2014-06-09 16:22:12.046964 7f86a5610700 10 _finish_op 0x30a2960 seq 3825 osr(1.71e 0x2ed00b0)/0x2ed00b0 2014-06-09 16:22:12.046967 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fca20 (1 active) 2014-06-09 16:22:12.046969 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781440 (2 active) 2014-06-09 16:22:12.046971 7f86a5610700 5 _do_op 0x11fcfda0 seq 3826 osr(1.1ef 0x384cdb0)/0x384cdb0 start 2014-06-09 16:22:12.046973 7f86a5610700 10 _do_transaction on 0xc05f5c0 2014-06-09 16:22:12.046977 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.046978 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.046981 7f868bf5a700 5 queue_transactions existing osr(1.2f8 0x384d0d0)/0x384d0d0 2014-06-09 16:22:12.046983 7f868bf5a700 10 op_submit_start 3873 2014-06-09 16:22:12.046984 7f868bf5a700 5 queue_transactions (trailing journal) 3873 2014-06-09 16:22:12.046985 7f868bf5a700 5 queue_op 0x15997320 seq 3873 osr(1.2f8 0x384d0d0) 828 bytes (queue has 49 ops and 42082 bytes) 2014-06-09 16:22:12.046988 7f868bf5a700 10 op_submit_finish 3873 2014-06-09 16:22:12.046991 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.542( v 214'710 (0'0,214'710] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:12.047010 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.047009 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.5 activated and committed 2014-06-09 16:22:12.047023 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3826.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.047026 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.047021 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.047031 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.047039 7f86a5610700 10 _do_op 0x11fcfda0 seq 3826 r = 0, finisher 0x1322c3c0 0 2014-06-09 16:22:12.047042 7f86a5610700 10 _finish_op 0x11fcfda0 seq 3826 osr(1.1ef 0x384cdb0)/0x384cdb0 2014-06-09 16:22:12.047045 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781440 (1 active) 2014-06-09 16:22:12.047047 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1b00 (2 active) 2014-06-09 16:22:12.047049 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.047049 7f86a5610700 5 _do_op 0xc34a240 seq 3827 osr(0.66a 0x2bd7ef0)/0x2bd7ef0 start 2014-06-09 16:22:12.047051 7f86a5610700 10 _do_transaction on 0xbfcb380 2014-06-09 16:22:12.047056 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.047065 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.542( v 214'710 lc 214'709 (0'0,214'710] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'710 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1132 2014-06-09 16:22:12.047083 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.047092 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.047087 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.431( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.047104 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3827.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.047108 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.047117 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.047119 7f86a5610700 10 _do_op 0xc34a240 seq 3827 r = 0, finisher 0x3006cce0 0 2014-06-09 16:22:12.047121 7f86a5610700 10 _finish_op 0xc34a240 seq 3827 osr(0.66a 0x2bd7ef0)/0x2bd7ef0 2014-06-09 16:22:12.047122 7f868bf5a700 5 queue_transactions existing osr(2.431 0x2ed1590)/0x2ed1590 2014-06-09 16:22:12.047124 7f868bf5a700 10 op_submit_start 3874 2014-06-09 16:22:12.047124 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1b00 (1 active) 2014-06-09 16:22:12.047125 7f868bf5a700 5 queue_transactions (trailing journal) 3874 2014-06-09 16:22:12.047125 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4d80 (2 active) 2014-06-09 16:22:12.047126 7f868bf5a700 5 queue_op 0x15997e60 seq 3874 osr(2.431 0x2ed1590) 828 bytes (queue has 48 ops and 41254 bytes) 2014-06-09 16:22:12.047127 7f86a5610700 5 _do_op 0x324f4540 seq 3828 osr(2.2b 0x2ed0230)/0x2ed0230 start 2014-06-09 16:22:12.047128 7f868bf5a700 10 op_submit_finish 3874 2014-06-09 16:22:12.047130 7f86a5610700 10 _do_transaction on 0x3a87ec0 2014-06-09 16:22:12.047134 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.047131 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.2f8( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.2f8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.047158 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.047162 7f868bf5a700 5 queue_transactions existing osr(1.2f8 0x384d0d0)/0x384d0d0 2014-06-09 16:22:12.047165 7f868bf5a700 10 op_submit_start 3875 2014-06-09 16:22:12.047165 7f868bf5a700 5 queue_transactions (trailing journal) 3875 2014-06-09 16:22:12.047167 7f868bf5a700 5 queue_op 0x159970e0 seq 3875 osr(1.2f8 0x384d0d0) 828 bytes (queue has 49 ops and 42082 bytes) 2014-06-09 16:22:12.047169 7f868bf5a700 10 op_submit_finish 3875 2014-06-09 16:22:12.047168 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.047172 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.3e8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.047181 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3828.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.047184 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.047195 7f86a5610700 10 _do_op 0x324f4540 seq 3828 r = 0, finisher 0x324d1060 0 2014-06-09 16:22:12.047198 7f86a5610700 10 _finish_op 0x324f4540 seq 3828 osr(2.2b 0x2ed0230)/0x2ed0230 2014-06-09 16:22:12.047200 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.047201 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4d80 (1 active) 2014-06-09 16:22:12.047202 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301440 (2 active) 2014-06-09 16:22:12.047203 7f868bf5a700 5 queue_transactions existing osr(2.3e8 0x2ed1310)/0x2ed1310 2014-06-09 16:22:12.047204 7f86a5610700 5 _do_op 0x32503f80 seq 3829 osr(1.3f0 0x384d400)/0x384d400 start 2014-06-09 16:22:12.047205 7f868bf5a700 10 op_submit_start 3876 2014-06-09 16:22:12.047206 7f868bf5a700 5 queue_transactions (trailing journal) 3876 2014-06-09 16:22:12.047206 7f86a5610700 10 _do_transaction on 0xc2389c0 2014-06-09 16:22:12.047207 7f868bf5a700 5 queue_op 0x15997020 seq 3876 osr(2.3e8 0x2ed1310) 828 bytes (queue has 49 ops and 42084 bytes) 2014-06-09 16:22:12.047210 7f868bf5a700 10 op_submit_finish 3876 2014-06-09 16:22:12.047211 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.047212 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.2a2( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.047241 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.047244 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.047245 7f868bf5a700 5 queue_transactions existing osr(0.2a2 0x2bd6c20)/0x2bd6c20 2014-06-09 16:22:12.047248 7f868bf5a700 10 op_submit_start 3877 2014-06-09 16:22:12.047249 7f868bf5a700 5 queue_transactions (trailing journal) 3877 2014-06-09 16:22:12.047251 7f868bf5a700 5 queue_op 0x159963c0 seq 3877 osr(0.2a2 0x2bd6c20) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.047253 7f868bf5a700 10 op_submit_finish 3877 2014-06-09 16:22:12.047258 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3829.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.047262 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.047255 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.431( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.431( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.047283 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.047282 7f86a5610700 10 _do_op 0x32503f80 seq 3829 r = 0, finisher 0x318c2060 0 2014-06-09 16:22:12.047286 7f86a5610700 10 _finish_op 0x32503f80 seq 3829 osr(1.3f0 0x384d400)/0x384d400 2014-06-09 16:22:12.047287 7f868bf5a700 5 queue_transactions existing osr(2.431 0x2ed1590)/0x2ed1590 2014-06-09 16:22:12.047288 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301440 (1 active) 2014-06-09 16:22:12.047290 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339680 (2 active) 2014-06-09 16:22:12.047292 7f868bf5a700 10 op_submit_start 3878 2014-06-09 16:22:12.047293 7f868bf5a700 5 queue_transactions (trailing journal) 3878 2014-06-09 16:22:12.047292 7f86a5610700 5 _do_op 0x15997b00 seq 3830 osr(2.67c 0x2ed1f90)/0x2ed1f90 start 2014-06-09 16:22:12.047294 7f86a5610700 10 _do_transaction on 0xbfb3680 2014-06-09 16:22:12.047294 7f868bf5a700 5 queue_op 0xc704180 seq 3878 osr(2.431 0x2ed1590) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.047296 7f868bf5a700 10 op_submit_finish 3878 2014-06-09 16:22:12.047299 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.047299 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.5c3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.047328 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.047333 7f868bf5a700 5 queue_transactions existing osr(0.5c3 0x2bd7a50)/0x2bd7a50 2014-06-09 16:22:12.047333 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.047335 7f868bf5a700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047340 7f8692767700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.047345 7f868ff62700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.047347 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3830.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.047350 7f868cf5c700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047351 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.047354 7f8690763700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047357 7f868a757700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047362 7f868c75b700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047362 7f86a5610700 10 _do_op 0x15997b00 seq 3830 r = 0, finisher 0xbd5e0c0 0 2014-06-09 16:22:12.047365 7f868b759700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047366 7f86a5610700 10 _finish_op 0x15997b00 seq 3830 osr(2.67c 0x2ed1f90)/0x2ed1f90 2014-06-09 16:22:12.047372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x15a3ee40,0xbd5c220,0x1322c3c0,0x3006cce0,0x324d1060,0x318c2060] 2014-06-09 16:22:12.047382 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x15a3ee40,0xbd5c220,0x1322c3c0,0x3006cce0,0x324d1060,0x318c2060] 2014-06-09 16:22:12.047385 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.047386 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.047393 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a61e0,0xbd6c980,0xbd77380,0xbd5c440,0x275cfc40,0x2d738e60,0xbef0520,0x318c2020,0xbd5e120] 2014-06-09 16:22:12.047401 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a61e0,0xbd6c980,0xbd77380,0xbd5c440,0x275cfc40,0x2d738e60,0xbef0520,0x318c2020,0xbd5e120] 2014-06-09 16:22:12.047404 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.047405 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.047413 7f868bf5a700 10 op_submit_start 3879 2014-06-09 16:22:12.047415 7f868bf5a700 5 queue_transactions (trailing journal) 3879 2014-06-09 16:22:12.047414 7f8691f66700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047416 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339680 (1 active) 2014-06-09 16:22:12.047417 7f868bf5a700 5 queue_op 0xc704660 seq 3879 osr(0.5c3 0x2bd7a50) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.047420 7f868ef60700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047421 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5e0c0] 2014-06-09 16:22:12.047424 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5e0c0] 2014-06-09 16:22:12.047425 7f8690f64700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047426 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.047427 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.047429 7f868e75f700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047444 7f8689755700 2 waiting 51 > 50 ops || 43773 > 104857600 2014-06-09 16:22:12.047449 7f868df5e700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047418 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359d40 (2 active) 2014-06-09 16:22:12.047455 7f868f761700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047457 7f86a5610700 5 _do_op 0x159976e0 seq 3831 osr(1.7a 0x384c510)/0x384c510 start 2014-06-09 16:22:12.047458 7f868bf5a700 10 op_submit_finish 3879 2014-06-09 16:22:12.047460 7f86a5610700 10 _do_transaction on 0xc2a0b40 2014-06-09 16:22:12.047461 7f8689f56700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047465 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.047467 7f868d75d700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047462 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:12.047473 7f8692f68700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047479 7f868af58700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047495 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.047500 7f868bf5a700 5 queue_transactions existing osr(3.587 0x3ed1840)/0x3ed1840 2014-06-09 16:22:12.047502 7f868bf5a700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047506 7f8692767700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.047509 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.047511 7f868ff62700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.047516 7f868cf5c700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047519 7f8690763700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047523 7f868a757700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047524 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3831.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.047527 7f868c75b700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047528 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.047531 7f868b759700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047536 7f8691765700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.047544 7f86a5610700 10 _do_op 0x159976e0 seq 3831 r = 0, finisher 0xbd5e220 0 2014-06-09 16:22:12.047552 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5e2a0] 2014-06-09 16:22:12.047553 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5e2a0] 2014-06-09 16:22:12.047554 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.047555 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.047558 7f86a5610700 10 _finish_op 0x159976e0 seq 3831 osr(1.7a 0x384c510)/0x384c510 2014-06-09 16:22:12.047565 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5e220] 2014-06-09 16:22:12.047566 7f8691f66700 10 op_submit_start 3880 2014-06-09 16:22:12.047567 7f868ef60700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047568 7f8691f66700 5 queue_transactions (trailing journal) 3880 2014-06-09 16:22:12.047568 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5e220] 2014-06-09 16:22:12.047570 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.047571 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.047571 7f8690f64700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047570 7f8691f66700 5 queue_op 0xc931860 seq 3880 osr(3.7ae 0xb7f01d0) 828 bytes (queue has 50 ops and 42914 bytes) 2014-06-09 16:22:12.047574 7f868e75f700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047578 7f8689755700 2 waiting 51 > 50 ops || 43775 > 104857600 2014-06-09 16:22:12.047581 7f868df5e700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047564 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359d40 (1 active) 2014-06-09 16:22:12.047585 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e240 (2 active) 2014-06-09 16:22:12.047585 7f868f761700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047588 7f86a5610700 5 _do_op 0xcd91140 seq 3832 osr(0.186 0x2bd67d0)/0x2bd67d0 start 2014-06-09 16:22:12.047591 7f86a5610700 10 _do_transaction on 0xbf89140 2014-06-09 16:22:12.047591 7f8689f56700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047596 7f8691f66700 10 op_submit_finish 3880 2014-06-09 16:22:12.047596 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.047596 7f868d75d700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047602 7f8692f68700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047600 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.5e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.047635 7f868af58700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047641 7f868bf5a700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047647 7f8692767700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047651 7f868ff62700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047654 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.047655 7f868cf5c700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047659 7f8690763700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047663 7f868a757700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047666 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.047667 7f868c75b700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047668 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3832.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.047671 7f868b759700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047671 7f8691f66700 5 queue_transactions existing osr(0.5e 0x2bd63b0)/0x2bd63b0 2014-06-09 16:22:12.047675 7f8691f66700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047672 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.047680 7f8691765700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047677 7f86af56b700 5 -- op tracker -- , seq: 3615, time: 2014-06-09 16:22:12.046466, event: header_read, op: osd_op(client.4541.0:32310 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 4120576~73728] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.047692 7f86a5610700 10 _do_op 0xcd91140 seq 3832 r = 0, finisher 0x324d1320 0 2014-06-09 16:22:12.047689 7f86af56b700 5 -- op tracker -- , seq: 3615, time: 2014-06-09 16:22:12.047390, event: throttled, op: osd_op(client.4541.0:32310 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 4120576~73728] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.047701 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d1220] 2014-06-09 16:22:12.047703 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d1220] 2014-06-09 16:22:12.047704 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.047705 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.047702 7f86af56b700 5 -- op tracker -- , seq: 3615, time: 2014-06-09 16:22:12.047599, event: all_read, op: osd_op(client.4541.0:32310 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 4120576~73728] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.047708 7f86a5610700 10 _finish_op 0xcd91140 seq 3832 osr(0.186 0x2bd67d0)/0x2bd67d0 2014-06-09 16:22:12.047712 7f86af56b700 5 -- op tracker -- , seq: 3615, time: 0.000000, event: dispatched, op: osd_op(client.4541.0:32310 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 4120576~73728] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.047718 7f8690f64700 10 op_submit_start 3881 2014-06-09 16:22:12.047719 7f8690f64700 5 queue_transactions (trailing journal) 3881 2014-06-09 16:22:12.047718 7f86af56b700 20 osd.26 243 should_share_map client.4541 172.16.96.220:0/1123720 243 2014-06-09 16:22:12.047720 7f8690f64700 5 queue_op 0x30a3b00 seq 3881 osr(0.151 0x2bd65e0) 828 bytes (queue has 50 ops and 42914 bytes) 2014-06-09 16:22:12.047725 7f868e75f700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047729 7f8689755700 2 waiting 51 > 50 ops || 43775 > 104857600 2014-06-09 16:22:12.047733 7f868df5e700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047740 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e240 (1 active) 2014-06-09 16:22:12.047739 7f868f761700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047741 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d1320] 2014-06-09 16:22:12.047742 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300900 (2 active) 2014-06-09 16:22:12.047744 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d1320] 2014-06-09 16:22:12.047746 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.047745 7f8689f56700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047746 7f8690f64700 10 op_submit_finish 3881 2014-06-09 16:22:12.047747 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.047733 7f86af56b700 15 osd.26 243 enqueue_op 0x30c6c200 prio 63 cost 73728 latency 0.001267 osd_op(client.4541.0:32310 rbd_data.115d6b8b4567.0000000000001650 [set-alloc-hint object_size 4194304 write_size 4194304,write 4120576~73728] 3.7150a5a RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.047751 7f868d75d700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047750 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.755( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:12.047764 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.047771 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.047776 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.047784 7f8690f64700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.047795 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.755( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:12.047810 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.047814 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.5fb( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:12.047827 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.047832 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.047837 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.047844 7f8690f64700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.047855 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.5fb( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:12.047868 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.047872 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:12.047897 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.047900 7f8690f64700 5 queue_transactions existing osr(3.468 0x3ed1590)/0x3ed1590 2014-06-09 16:22:12.047901 7f8690f64700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047904 7f8692f68700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047908 7f86a5610700 5 _do_op 0xc34af60 seq 3833 osr(1.d1 0x384c9b0)/0x384c9b0 start 2014-06-09 16:22:12.047910 7f86a5610700 10 _do_transaction on 0xbe8be00 2014-06-09 16:22:12.047909 7f868af58700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047913 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.047917 7f868bf5a700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047924 7f8692767700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047928 7f868ff62700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.047934 7f868cf5c700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047939 7f8690763700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047943 7f868a757700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047962 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.047971 7f868c75b700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047977 7f868b759700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.047977 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3833.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.047981 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.048002 7f86a5610700 10 _do_op 0xc34af60 seq 3833 r = 0, finisher 0xbd775c0 0 2014-06-09 16:22:12.048007 7f86a5610700 10 _finish_op 0xc34af60 seq 3833 osr(1.d1 0x384c9b0)/0x384c9b0 2014-06-09 16:22:12.048010 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd76fc0] 2014-06-09 16:22:12.048013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd76fc0] 2014-06-09 16:22:12.048014 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.048015 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.048018 7f8691765700 10 op_submit_start 3882 2014-06-09 16:22:12.048020 7f8691765700 5 queue_transactions (trailing journal) 3882 2014-06-09 16:22:12.048021 7f8691765700 5 queue_op 0xc34b440 seq 3882 osr(1.6f 0x384c570) 826 bytes (queue has 50 ops and 42914 bytes) 2014-06-09 16:22:12.048021 7f8691f66700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048028 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd775c0] 2014-06-09 16:22:12.048028 7f8689755700 2 waiting 51 > 50 ops || 43775 > 104857600 2014-06-09 16:22:12.048031 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd775c0] 2014-06-09 16:22:12.048032 7f868df5e700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.048033 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.048034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.048026 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300900 (1 active) 2014-06-09 16:22:12.048038 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f200 (2 active) 2014-06-09 16:22:12.048037 7f868f761700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.048041 7f86a5610700 5 _do_op 0xc34b260 seq 3834 osr(2.112 0x2ed06d0)/0x2ed06d0 start 2014-06-09 16:22:12.048043 7f8691765700 10 op_submit_finish 3882 2014-06-09 16:22:12.048044 7f86a5610700 10 _do_transaction on 0xc1346c0 2014-06-09 16:22:12.048043 7f8689f56700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.048049 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.048048 7f868d75d700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.048054 7f8690f64700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.048046 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.048059 7f8692f68700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.048064 7f868af58700 2 waiting 51 > 50 ops || 43742 > 104857600 2014-06-09 16:22:12.048060 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.048068 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.048077 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.048085 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.048094 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.048093 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.048107 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3834.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.048111 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.048107 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:172 2014-06-09 16:22:12.048120 7f86a5610700 10 _do_op 0xc34b260 seq 3834 r = 0, finisher 0xbd70e20 0 2014-06-09 16:22:12.048116 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.048126 7f86a5610700 10 _finish_op 0xc34b260 seq 3834 osr(2.112 0x2ed06d0)/0x2ed06d0 2014-06-09 16:22:12.048124 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.532( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.048132 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f200 (1 active) 2014-06-09 16:22:12.048133 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.048133 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3397a0 (2 active) 2014-06-09 16:22:12.048134 7f8691f66700 10 op_submit_start 3883 2014-06-09 16:22:12.048136 7f8691f66700 5 queue_transactions (trailing journal) 3883 2014-06-09 16:22:12.048135 7f86a5610700 5 _do_op 0xbf7e480 seq 3835 osr(0.36d 0x2bd6fc0)/0x2bd6fc0 start 2014-06-09 16:22:12.048138 7f86a5610700 10 _do_transaction on 0xc2aa240 2014-06-09 16:22:12.048137 7f8691f66700 5 queue_op 0xc930360 seq 3883 osr(0.5e 0x2bd63b0) 826 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.048140 7f8691f66700 10 op_submit_finish 3883 2014-06-09 16:22:12.048142 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.048137 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.048143 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.1fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.1fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.048151 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.004206 7 0.000434 2014-06-09 16:22:12.048164 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.048176 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.048178 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.048176 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.048183 7f8691f66700 5 queue_transactions existing osr(1.1fa 0x384cd50)/0x384cd50 2014-06-09 16:22:12.048185 7f8691f66700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048188 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3835.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.048190 7f8692767700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048192 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.048195 7f868ff62700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048188 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 2014-06-09 16:22:12.048200 7f868cf5c700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048201 7f86a5610700 10 _do_op 0xbf7e480 seq 3835 r = 0, finisher 0x32477de0 0 2014-06-09 16:22:12.048204 7f86a5610700 10 _finish_op 0xbf7e480 seq 3835 osr(0.36d 0x2bd6fc0)/0x2bd6fc0 2014-06-09 16:22:12.048204 7f8690763700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048199 7f8691765700 5 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.048210 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.048217 7f8691f66700 10 op_submit_start 3884 2014-06-09 16:22:12.048217 7f868c75b700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048219 7f8691f66700 5 queue_transactions (trailing journal) 3884 2014-06-09 16:22:12.048220 7f8691f66700 5 queue_op 0xc930c00 seq 3884 osr(1.1fa 0x384cd50) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.048222 7f868e75f700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048225 7f868b759700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048218 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.048230 7f8689755700 2 waiting 51 > 50 ops || 43773 > 104857600 2014-06-09 16:22:12.048233 7f868df5e700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048228 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3397a0 (1 active) 2014-06-09 16:22:12.048238 7f868f761700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048238 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e000 (2 active) 2014-06-09 16:22:12.048231 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.048242 7f86a5610700 5 _do_op 0x30032960 seq 3836 osr(0.3ab 0x2bd72f0)/0x2bd72f0 start 2014-06-09 16:22:12.048243 7f8689f56700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048245 7f86a5610700 10 _do_transaction on 0xc143500 2014-06-09 16:22:12.048248 7f8691f66700 10 op_submit_finish 3884 2014-06-09 16:22:12.048249 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.048243 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.048250 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.456( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.048253 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.048266 7f8691765700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.048273 7f868d75d700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048280 7f8690f64700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048283 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.048289 7f8691f66700 5 queue_transactions existing osr(0.456 0x2bd7530)/0x2bd7530 2014-06-09 16:22:12.048282 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:856 2014-06-09 16:22:12.048291 7f8691f66700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048305 7f8692f68700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048313 7f868af58700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048315 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.048319 7f8692767700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048319 7f8691765700 5 queue_transactions existing osr(3.4d 0x3ed0300)/0x3ed0300 2014-06-09 16:22:12.048322 7f8691765700 2 waiting 51 > 50 ops || 43770 > 104857600 2014-06-09 16:22:12.048324 7f868ff62700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048326 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd70e20,0x32477de0] 2014-06-09 16:22:12.048329 7f868cf5c700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048333 7f8690763700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048334 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd70e20,0x32477de0] 2014-06-09 16:22:12.048336 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.048337 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.048337 7f868a757700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048340 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x18742c60,0x20051100] 2014-06-09 16:22:12.048342 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x18742c60,0x20051100] 2014-06-09 16:22:12.048343 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.048344 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.048343 7f868ef60700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048346 7f868bf5a700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048371 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.048383 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3821.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.048386 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.048396 7f86a5e11700 10 _do_op 0xcabc240 seq 3821 r = 0, finisher 0x306cf9e0 0 2014-06-09 16:22:12.048402 7f86a5e11700 10 _finish_op 0xcabc240 seq 3821 osr(0.5b7 0x2bd7a90)/0x2bd7a90 2014-06-09 16:22:12.048403 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042bae0] 2014-06-09 16:22:12.048406 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042bae0] 2014-06-09 16:22:12.048407 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.048408 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.048412 7f868e75f700 10 op_submit_start 3885 2014-06-09 16:22:12.048414 7f868e75f700 5 queue_transactions (trailing journal) 3885 2014-06-09 16:22:12.048415 7f868b759700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048415 7f868e75f700 5 queue_op 0x324f46c0 seq 3885 osr(2.501 0x2ed1690) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.048421 7f8689755700 2 waiting 51 > 50 ops || 43773 > 104857600 2014-06-09 16:22:12.048425 7f868df5e700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048430 7f868f761700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048431 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd560 (1 active) 2014-06-09 16:22:12.048433 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306cf9e0] 2014-06-09 16:22:12.048434 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5258c0 (2 active) 2014-06-09 16:22:12.048435 7f8689f56700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048437 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306cf9e0] 2014-06-09 16:22:12.048438 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.048438 7f868e75f700 10 op_submit_finish 3885 2014-06-09 16:22:12.048439 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.048440 7f868d75d700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048442 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.24c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:12.048452 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.048466 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3836.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.048470 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.048480 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.048480 7f86a5610700 10 _do_op 0x30032960 seq 3836 r = 0, finisher 0x32480ac0 0 2014-06-09 16:22:12.048485 7f868e75f700 5 queue_transactions existing osr(1.24c 0x384cfd0)/0x384cfd0 2014-06-09 16:22:12.048487 7f868e75f700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048490 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30c554c0] 2014-06-09 16:22:12.048492 7f8690f64700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048492 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30c554c0] 2014-06-09 16:22:12.048493 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.048494 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.048496 7f86a5e11700 5 _do_op 0x324e6000 seq 3837 osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 start 2014-06-09 16:22:12.048497 7f86a5610700 10 _finish_op 0x30032960 seq 3836 osr(0.3ab 0x2bd72f0)/0x2bd72f0 2014-06-09 16:22:12.048497 7f8691f66700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048499 7f86a5e11700 10 _do_transaction on 0x344c600 2014-06-09 16:22:12.048503 7f8692f68700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048504 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.048509 7f868af58700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048514 7f8692767700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048520 7f8691765700 2 waiting 51 > 50 ops || 43770 > 104857600 2014-06-09 16:22:12.048526 7f868ff62700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048531 7f868cf5c700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048536 7f8690763700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048536 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.048540 7f868a757700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048546 7f868ef60700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048547 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3837.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.048551 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.048551 7f868bf5a700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048556 7f868c75b700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048567 7f868b759700 10 op_submit_start 3886 2014-06-09 16:22:12.048568 7f8689755700 2 waiting 51 > 50 ops || 43773 > 104857600 2014-06-09 16:22:12.048569 7f868b759700 5 queue_transactions (trailing journal) 3886 2014-06-09 16:22:12.048571 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32480ac0] 2014-06-09 16:22:12.048571 7f868b759700 5 queue_op 0xbf7fb00 seq 3886 osr(0.25b 0x2bd69f0) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.048573 7f868df5e700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048575 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32480ac0] 2014-06-09 16:22:12.048576 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.048577 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.048578 7f8689f56700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048580 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e000 (1 active) 2014-06-09 16:22:12.048582 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52efc0 (2 active) 2014-06-09 16:22:12.048584 7f868f761700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048586 7f86a5610700 5 _do_op 0x24104120 seq 3838 osr(2.350 0x2ed0e50)/0x2ed0e50 start 2014-06-09 16:22:12.048588 7f86a5610700 10 _do_transaction on 0xbebbe00 2014-06-09 16:22:12.048589 7f868e75f700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048590 7f868b759700 10 op_submit_finish 3886 2014-06-09 16:22:12.048594 7f8690f64700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048594 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.048593 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.52e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.52e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.048629 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.048628 7f86a5e11700 10 _do_op 0x324e6000 seq 3837 r = 0, finisher 0x324a8da0 0 2014-06-09 16:22:12.048634 7f868b759700 5 queue_transactions existing osr(2.52e 0x2ed1600)/0x2ed1600 2014-06-09 16:22:12.048634 7f86a5e11700 10 _finish_op 0x324e6000 seq 3837 osr(2.6b0 0x2ed1ed0)/0x2ed1ed0 2014-06-09 16:22:12.048635 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7a340] 2014-06-09 16:22:12.048637 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7a340] 2014-06-09 16:22:12.048639 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.048640 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.048645 7f8692f68700 10 op_submit_start 3887 2014-06-09 16:22:12.048644 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5258c0 (1 active) 2014-06-09 16:22:12.048646 7f8692f68700 5 queue_transactions (trailing journal) 3887 2014-06-09 16:22:12.048646 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c6c0 (2 active) 2014-06-09 16:22:12.048647 7f868af58700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048647 7f8692f68700 5 queue_op 0x324e7680 seq 3887 osr(0.1a4 0x2bd6710) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.048648 7f86a5e11700 5 _do_op 0x33a3860 seq 3839 osr(0.6e7 0x384c1b0)/0x384c1b0 start 2014-06-09 16:22:12.048650 7f8692f68700 10 op_submit_finish 3887 2014-06-09 16:22:12.048651 7f86a5e11700 10 _do_transaction on 0xc129440 2014-06-09 16:22:12.048653 7f8692767700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048656 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.048657 7f8691765700 2 waiting 51 > 50 ops || 43770 > 104857600 2014-06-09 16:22:12.048662 7f868ff62700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048653 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.697( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.697( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.048666 7f868cf5c700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048671 7f8690763700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048675 7f868a757700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048679 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.048680 7f868ef60700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048682 7f8692f68700 5 queue_transactions existing osr(1.697 0x384de10)/0x384de10 2014-06-09 16:22:12.048684 7f8692f68700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048688 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324a8da0] 2014-06-09 16:22:12.048688 7f868c75b700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048692 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324a8da0] 2014-06-09 16:22:12.048693 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.048694 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.048694 7f868d75d700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048700 7f868df5e700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048705 7f8689f56700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048710 7f868f761700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048715 7f868e75f700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048720 7f8690f64700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048723 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.048725 7f868b759700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048731 7f8689755700 2 waiting 51 > 50 ops || 43773 > 104857600 2014-06-09 16:22:12.048737 7f8691f66700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048736 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3838.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.048740 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.048742 7f868bf5a700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048768 7f86a5610700 10 _do_op 0x24104120 seq 3838 r = 0, finisher 0x189aace0 0 2014-06-09 16:22:12.048778 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc1695c0] 2014-06-09 16:22:12.048780 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc1695c0] 2014-06-09 16:22:12.048782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.048783 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.048785 7f86a5610700 10 _finish_op 0x24104120 seq 3838 osr(2.350 0x2ed0e50)/0x2ed0e50 2014-06-09 16:22:12.048794 7f868af58700 10 op_submit_start 3888 2014-06-09 16:22:12.048795 7f868af58700 5 queue_transactions (trailing journal) 3888 2014-06-09 16:22:12.048796 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aace0] 2014-06-09 16:22:12.048796 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.048797 7f868af58700 5 queue_op 0x30d97800 seq 3888 osr(3.778 0x3ed1df0) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.048800 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aace0] 2014-06-09 16:22:12.048801 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.048802 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.048802 7f8692767700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048805 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52efc0 (1 active) 2014-06-09 16:22:12.048806 7f8691765700 2 waiting 51 > 50 ops || 43770 > 104857600 2014-06-09 16:22:12.048808 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407680 (2 active) 2014-06-09 16:22:12.048811 7f868ff62700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.048810 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3839.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.048812 7f86a5610700 5 _do_op 0x16fd5e00 seq 3840 osr(3.21e 0x3ed0c90)/0x3ed0c90 start 2014-06-09 16:22:12.048815 7f86a5610700 10 _do_transaction on 0xc263bc0 2014-06-09 16:22:12.048814 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.048815 7f868cf5c700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048820 7f8690763700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048820 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.048824 7f868a757700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048828 7f868ef60700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048833 7f868af58700 10 op_submit_finish 3888 2014-06-09 16:22:12.048834 7f8692f68700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048840 7f868c75b700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.048837 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.652( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.048858 7f86a5e11700 10 _do_op 0x33a3860 seq 3839 r = 0, finisher 0x324f83e0 0 2014-06-09 16:22:12.048864 7f86a5e11700 10 _finish_op 0x33a3860 seq 3839 osr(0.6e7 0x384c1b0)/0x384c1b0 2014-06-09 16:22:12.048865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x200502e0] 2014-06-09 16:22:12.048868 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x200502e0] 2014-06-09 16:22:12.048869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.048869 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.048870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.048870 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c6c0 (1 active) 2014-06-09 16:22:12.048873 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338360 (2 active) 2014-06-09 16:22:12.048873 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f83e0] 2014-06-09 16:22:12.048874 7f868af58700 5 queue_transactions existing osr(0.652 0x2bd7b90)/0x2bd7b90 2014-06-09 16:22:12.048875 7f86a5e11700 5 _do_op 0xcabc9c0 seq 3841 osr(2.6ec 0x2ed1e10)/0x2ed1e10 start 2014-06-09 16:22:12.048876 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f83e0] 2014-06-09 16:22:12.048877 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.048877 7f868af58700 10 op_submit_start 3889 2014-06-09 16:22:12.048877 7f86a5e11700 10 _do_transaction on 0x3e63140 2014-06-09 16:22:12.048878 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.048878 7f868af58700 5 queue_transactions (trailing journal) 3889 2014-06-09 16:22:12.048879 7f868af58700 5 queue_op 0x30d977a0 seq 3889 osr(0.652 0x2bd7b90) 828 bytes (queue has 50 ops and 42912 bytes) 2014-06-09 16:22:12.048882 7f868af58700 10 op_submit_finish 3889 2014-06-09 16:22:12.048882 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.048885 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.048902 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.048900 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.048911 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.048921 7f86a5610700 15 _omap_rmkeyrange meta/1fe278bb/pglog_3.21e/0//-1 [0000000213.00000000000000000615,4294967295.18446744073709551615] 2014-06-09 16:22:12.048925 7f86a5610700 15 get_omap_iterator meta/1fe278bb/pglog_3.21e/0//-1 2014-06-09 16:22:12.048920 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.048934 7f868af58700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.048951 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.71f( v 213'532 (0'0,213'532] local-les=243 n=101 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:872 2014-06-09 16:22:12.048967 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.048971 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.4a7( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.4a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.049001 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.049006 7f868af58700 5 queue_transactions existing osr(0.4a7 0x2bd7370)/0x2bd7370 2014-06-09 16:22:12.049008 7f868af58700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049012 7f868d75d700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049017 7f8692767700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.049021 7f8689f56700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049025 7f868f761700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049027 7f86a5610700 15 _omap_rmkeys meta/1fe278bb/pglog_3.21e/0//-1 2014-06-09 16:22:12.049030 7f868e75f700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049036 7f8690f64700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049042 7f868b759700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049047 7f8689755700 2 waiting 51 > 50 ops || 43773 > 104857600 2014-06-09 16:22:12.049053 7f8691f66700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049059 7f868bf5a700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049061 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe278bb/pglog_3.21e/0//-1 2014-06-09 16:22:12.049063 7f8691765700 2 waiting 51 > 50 ops || 43770 > 104857600 2014-06-09 16:22:12.049067 7f868ff62700 2 waiting 51 > 50 ops || 43738 > 104857600 2014-06-09 16:22:12.049069 7f86a5610700 10 _omap_rmkeys meta/1fe278bb/pglog_3.21e/0//-1 r = 0 2014-06-09 16:22:12.049072 7f868cf5c700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049072 7f86a5610700 15 _omap_rmkeys meta/1fe278bb/pglog_3.21e/0//-1 2014-06-09 16:22:12.049075 7f86a5610700 10 _omap_rmkeys meta/1fe278bb/pglog_3.21e/0//-1 r = 0 2014-06-09 16:22:12.049076 7f8690763700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049080 7f86a5610700 15 _omap_setkeys meta/1fe278bb/pglog_3.21e/0//-1 2014-06-09 16:22:12.049081 7f868a757700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049085 7f868ef60700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049090 7f8692f68700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049093 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3840.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.049097 7f868c75b700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049097 7f86a5610700 10 cid: oid: 1fe278bb/pglog_3.21e/0//-1 skipping op, spos 3840.0.0 <= header.spos 148648.0.0 2014-06-09 16:22:12.049101 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.049101 7f868df5e700 2 waiting 51 > 50 ops || 43740 > 104857600 2014-06-09 16:22:12.049130 7f86a5610700 10 _do_op 0x16fd5e00 seq 3840 r = 0, finisher 0xbd60a40 0 2014-06-09 16:22:12.049140 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3e360] 2014-06-09 16:22:12.049143 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3e360] 2014-06-09 16:22:12.049144 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.049145 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.049148 7f86a5610700 10 _finish_op 0x16fd5e00 seq 3840 osr(3.21e 0x3ed0c90)/0x3ed0c90 2014-06-09 16:22:12.049156 7f868af58700 10 op_submit_start 3890 2014-06-09 16:22:12.049158 7f868af58700 5 queue_transactions (trailing journal) 3890 2014-06-09 16:22:12.049158 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd60a40] 2014-06-09 16:22:12.049160 7f868af58700 5 queue_op 0x30d976e0 seq 3890 osr(0.4a7 0x2bd7370) 828 bytes (queue has 50 ops and 41425 bytes) 2014-06-09 16:22:12.049163 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.049165 7f868d75d700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049169 7f8692767700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049161 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 221'616 (0'0,221'617] local-les=243 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 crt=221'617 lcod 0'0 inactive m=1] _activate_committed 243 telling primary 2014-06-09 16:22:12.049174 7f8689f56700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049178 7f868f761700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049178 7f86a4e0f700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.049177 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3841.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.049183 7f868e75f700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049189 7f8690f64700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049181 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.049194 7f868b759700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049200 7f8689755700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049193 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 221'616 (0'0,221'617] local-les=243 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=221'617 lcod 0'0 active m=1] flushed 2014-06-09 16:22:12.049205 7f8691f66700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049210 7f868bf5a700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049210 7f86a5e11700 10 _do_op 0xcabc9c0 seq 3841 r = 0, finisher 0x31412400 0 2014-06-09 16:22:12.049212 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd60a40] 2014-06-09 16:22:12.049213 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.049214 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.049215 7f8691765700 2 waiting 51 > 50 ops || 42283 > 104857600 2014-06-09 16:22:12.049219 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042ba80] 2014-06-09 16:22:12.049219 7f868ff62700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049221 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042ba80] 2014-06-09 16:22:12.049223 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.049224 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.049225 7f868cf5c700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049229 7f8690763700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049234 7f868a757700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049236 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407680 (1 active) 2014-06-09 16:22:12.049239 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1200 (2 active) 2014-06-09 16:22:12.049238 7f868ef60700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049244 7f8692f68700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049244 7f86a5610700 5 _do_op 0x30a30e0 seq 3842 osr(2.26a 0x2ed0dd0)/0x2ed0dd0 start 2014-06-09 16:22:12.049247 7f86a5610700 10 _do_transaction on 0xc11b140 2014-06-09 16:22:12.049247 7f868af58700 10 op_submit_finish 3890 2014-06-09 16:22:12.049249 7f868c75b700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049252 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.049253 7f868df5e700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049257 7f86a5e11700 10 _finish_op 0xcabc9c0 seq 3841 osr(2.6ec 0x2ed1e10)/0x2ed1e10 2014-06-09 16:22:12.049252 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.463( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.049264 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338360 (1 active) 2014-06-09 16:22:12.049266 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412400] 2014-06-09 16:22:12.049266 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fde60 (2 active) 2014-06-09 16:22:12.049269 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412400] 2014-06-09 16:22:12.049269 7f86a5e11700 5 _do_op 0x30d97920 seq 3843 osr(1.27c 0x384cf30)/0x384cf30 start 2014-06-09 16:22:12.049270 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.049271 7f86a5e11700 10 _do_transaction on 0xc2175c0 2014-06-09 16:22:12.049271 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.049285 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.049290 7f868af58700 5 queue_transactions existing osr(2.463 0x2ed1510)/0x2ed1510 2014-06-09 16:22:12.049293 7f868af58700 10 op_submit_start 3891 2014-06-09 16:22:12.049294 7f868af58700 5 queue_transactions (trailing journal) 3891 2014-06-09 16:22:12.049294 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.049295 7f868af58700 5 queue_op 0x30d97680 seq 3891 osr(2.463 0x2ed1510) 828 bytes (queue has 50 ops and 41425 bytes) 2014-06-09 16:22:12.049298 7f868af58700 10 op_submit_finish 3891 2014-06-09 16:22:12.049303 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.049301 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.759( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.049317 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3842.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.049321 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.049331 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.049335 7f868af58700 5 queue_transactions existing osr(2.759 0x3ed00b0)/0x3ed00b0 2014-06-09 16:22:12.049338 7f868af58700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049342 7f868d75d700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049342 7f86a5610700 10 _do_op 0x30a30e0 seq 3842 r = 0, finisher 0x314129c0 0 2014-06-09 16:22:12.049347 7f8692767700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049349 7f86a5610700 10 _finish_op 0x30a30e0 seq 3842 osr(2.26a 0x2ed0dd0)/0x2ed0dd0 2014-06-09 16:22:12.049350 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b080] 2014-06-09 16:22:12.049353 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b080] 2014-06-09 16:22:12.049355 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.049356 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.049355 7f868f761700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049357 7f868af58700 10 op_submit_start 3892 2014-06-09 16:22:12.049358 7f868af58700 5 queue_transactions (trailing journal) 3892 2014-06-09 16:22:12.049360 7f8690f64700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049359 7f868af58700 5 queue_op 0x30d97620 seq 3892 osr(2.759 0x3ed00b0) 828 bytes (queue has 50 ops and 41425 bytes) 2014-06-09 16:22:12.049364 7f868b759700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049366 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x314129c0] 2014-06-09 16:22:12.049370 7f8689755700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049370 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x314129c0] 2014-06-09 16:22:12.049371 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.049373 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.049375 7f8691f66700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049375 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1200 (1 active) 2014-06-09 16:22:12.049378 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d10e0 (2 active) 2014-06-09 16:22:12.049380 7f868bf5a700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049382 7f86a5610700 5 _do_op 0xc931680 seq 3844 osr(2.530 0x2ed19b0)/0x2ed19b0 start 2014-06-09 16:22:12.049384 7f86a5610700 10 _do_transaction on 0x398af00 2014-06-09 16:22:12.049385 7f8691765700 2 waiting 51 > 50 ops || 42283 > 104857600 2014-06-09 16:22:12.049389 7f868ff62700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049389 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.049394 7f868cf5c700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049397 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.049399 7f8690763700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049403 7f868a757700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049408 7f868ef60700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049413 7f868af58700 10 op_submit_finish 3892 2014-06-09 16:22:12.049413 7f8692f68700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049419 7f868c75b700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049423 7f868df5e700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049416 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.53d( v 213'594 (0'0,213'594] local-les=243 n=113 ec=118 les/c 243/243 242/242/212) 2014-06-09 16:22:12.049411 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3843.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.049431 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.049442 7f86a5e11700 10 _do_op 0x30d97920 seq 3843 r = 0, finisher 0xbd6a9e0 0 2014-06-09 16:22:12.049450 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.049450 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a7e80] 2014-06-09 16:22:12.049452 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a7e80] 2014-06-09 16:22:12.049453 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.049454 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.049454 7f868af58700 5 queue_transactions existing osr(3.53d 0x3ed1600)/0x3ed1600 2014-06-09 16:22:12.049458 7f86a5e11700 10 _finish_op 0x30d97920 seq 3843 osr(1.27c 0x384cf30)/0x384cf30 2014-06-09 16:22:12.049459 7f868af58700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049465 7f868d75d700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049470 7f8692767700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049470 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.049474 7f8689f56700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049485 7f868f761700 10 op_submit_start 3893 2014-06-09 16:22:12.049484 7f868e75f700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049486 7f868f761700 5 queue_transactions (trailing journal) 3893 2014-06-09 16:22:12.049488 7f868f761700 5 queue_op 0x33a2d20 seq 3893 osr(2.4a5 0x2ed17f0) 828 bytes (queue has 50 ops and 41425 bytes) 2014-06-09 16:22:12.049490 7f8690f64700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049493 7f8689755700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049498 7f8691f66700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049501 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6a9e0] 2014-06-09 16:22:12.049500 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fde60 (1 active) 2014-06-09 16:22:12.049503 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3aea0 (2 active) 2014-06-09 16:22:12.049504 7f868bf5a700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049505 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6a9e0] 2014-06-09 16:22:12.049506 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.049506 7f86a5e11700 5 _do_op 0xcabdbc0 seq 3845 osr(1.1fa 0x384cd50)/0x384cd50 start 2014-06-09 16:22:12.049507 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.049508 7f86a5e11700 10 _do_transaction on 0xc048900 2014-06-09 16:22:12.049508 7f8691765700 2 waiting 51 > 50 ops || 42283 > 104857600 2014-06-09 16:22:12.049510 7f868f761700 10 op_submit_finish 3893 2014-06-09 16:22:12.049513 7f868ff62700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049514 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.049517 7f868cf5c700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049522 7f8690763700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049514 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:12.049526 7f868a757700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049531 7f868ef60700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049536 7f8692f68700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049541 7f868c75b700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049545 7f868df5e700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049548 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.049550 7f868af58700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049553 7f868f761700 5 queue_transactions existing osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 2014-06-09 16:22:12.049555 7f868f761700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049561 7f8692767700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049498 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3844.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.049564 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.049566 7f8689f56700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049570 7f868b759700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049574 7f868d75d700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049580 7f86a5610700 10 _do_op 0xc931680 seq 3844 r = 0, finisher 0xcd33d40 0 2014-06-09 16:22:12.049586 7f86a5610700 10 _finish_op 0xc931680 seq 3844 osr(2.530 0x2ed19b0)/0x2ed19b0 2014-06-09 16:22:12.049587 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314123c0] 2014-06-09 16:22:12.049589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314123c0] 2014-06-09 16:22:12.049590 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.049591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.049596 7f868e75f700 10 op_submit_start 3894 2014-06-09 16:22:12.049596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd33d40] 2014-06-09 16:22:12.049597 7f868e75f700 5 queue_transactions (trailing journal) 3894 2014-06-09 16:22:12.049598 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd33d40] 2014-06-09 16:22:12.049599 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.049598 7f868e75f700 5 queue_op 0x324f4780 seq 3894 osr(1.24c 0x384cfd0) 828 bytes (queue has 50 ops and 41425 bytes) 2014-06-09 16:22:12.049600 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.049602 7f8690f64700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049603 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d10e0 (1 active) 2014-06-09 16:22:12.049605 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6ddc20 (2 active) 2014-06-09 16:22:12.049606 7f8689755700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049609 7f86a5610700 5 _do_op 0x2e21b00 seq 3846 osr(3.5b3 0x3ed17e0)/0x3ed17e0 start 2014-06-09 16:22:12.049611 7f868e75f700 10 op_submit_finish 3894 2014-06-09 16:22:12.049612 7f86a5610700 10 _do_transaction on 0x2dcfc80 2014-06-09 16:22:12.049611 7f8691f66700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049616 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.049617 7f868bf5a700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049621 7f8691765700 2 waiting 51 > 50 ops || 42283 > 104857600 2014-06-09 16:22:12.049614 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.3b7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.049626 7f868ff62700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049630 7f868cf5c700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049635 7f8690763700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049639 7f868a757700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049640 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.049643 7f868e75f700 5 queue_transactions existing osr(0.3b7 0x2bd7290)/0x2bd7290 2014-06-09 16:22:12.049643 7f868ef60700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049650 7f8692f68700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049655 7f868c75b700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049661 7f868df5e700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049667 7f868af58700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049673 7f868f761700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049679 7f8692767700 2 waiting 51 > 50 ops || 42251 > 104857600 2014-06-09 16:22:12.049682 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.049684 7f8689f56700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049688 7f868b759700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049693 7f868d75d700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049698 7f868e75f700 2 waiting 51 > 50 ops || 42253 > 104857600 2014-06-09 16:22:12.049703 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3845.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.049706 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.049721 7f86a5e11700 10 _do_op 0xcabdbc0 seq 3845 r = 0, finisher 0xc169fc0 0 2014-06-09 16:22:12.049730 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x12051f40] 2014-06-09 16:22:12.049733 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x12051f40] 2014-06-09 16:22:12.049734 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.049735 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.049738 7f86a5e11700 10 _finish_op 0xcabdbc0 seq 3845 osr(1.1fa 0x384cd50)/0x384cd50 2014-06-09 16:22:12.049748 7f8689755700 10 op_submit_start 3895 2014-06-09 16:22:12.049749 7f8689755700 5 queue_transactions (trailing journal) 3895 2014-06-09 16:22:12.049750 7f8691f66700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049751 7f8689755700 5 queue_op 0x230b8060 seq 3895 osr(0.290 0x2bd6c80) 861 bytes (queue has 50 ops and 41458 bytes) 2014-06-09 16:22:12.049751 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.049755 7f868bf5a700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049758 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3aea0 (1 active) 2014-06-09 16:22:12.049759 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc169fc0] 2014-06-09 16:22:12.049759 7f8691765700 2 waiting 51 > 50 ops || 42316 > 104857600 2014-06-09 16:22:12.049760 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d05a0 (2 active) 2014-06-09 16:22:12.049762 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc169fc0] 2014-06-09 16:22:12.049764 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.049764 7f8689755700 10 op_submit_finish 3895 2014-06-09 16:22:12.049764 7f868ff62700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.049765 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.049768 7f868cf5c700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049773 7f8690763700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049777 7f868a757700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049776 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3846.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.049768 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.049780 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.049782 7f868ef60700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049787 7f8692f68700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049782 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.049798 7f86a5610700 10 _do_op 0x2e21b00 seq 3846 r = 0, finisher 0x324d08c0 0 2014-06-09 16:22:12.049792 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x14053800,0x14053700,0x14053600,0x14053500,0x1b0b7800,0x1b0b7700,0x1b0b7600,0x1b0b7500,0x1b0b7400,0x1b0b7300 2014-06-09 16:22:12.049805 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd0fe00] 2014-06-09 16:22:12.049807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd0fe00] 2014-06-09 16:22:12.049808 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.049809 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.049812 7f86a5610700 10 _finish_op 0x2e21b00 seq 3846 osr(3.5b3 0x3ed17e0)/0x3ed17e0 2014-06-09 16:22:12.049820 7f8691f66700 10 op_submit_start 3896 2014-06-09 16:22:12.049821 7f868df5e700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049822 7f8691f66700 5 queue_transactions (trailing journal) 3896 2014-06-09 16:22:12.049824 7f8691f66700 5 queue_op 0xc9301e0 seq 3896 osr(0.456 0x2bd7530) 828 bytes (queue has 50 ops and 41458 bytes) 2014-06-09 16:22:12.049827 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d08c0] 2014-06-09 16:22:12.049829 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d08c0] 2014-06-09 16:22:12.049828 7f868af58700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049830 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.049831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.049834 7f868f761700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049838 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6ddc20 (1 active) 2014-06-09 16:22:12.049839 7f8692767700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.049841 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7806c0 (2 active) 2014-06-09 16:22:12.049844 7f8689f56700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049828 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.049845 7f86a5610700 5 _do_op 0x159978c0 seq 3848 osr(1.122 0x384c890)/0x384c890 start 2014-06-09 16:22:12.049848 7f86a5610700 10 _do_transaction on 0xc33dbc0 2014-06-09 16:22:12.049849 7f868b759700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049853 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.049853 7f868d75d700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049847 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.049858 7f868e75f700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049856 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.049863 7f8690f64700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049867 7f8691f66700 10 op_submit_finish 3896 2014-06-09 16:22:12.049869 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1005 2014-06-09 16:22:12.049871 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.27b( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 243/243 242/242/212) 2014-06-09 16:22:12.049880 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.049888 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.049889 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.2e7( v 214'619 (0'0,214'619] local-les=243 n=127 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.049899 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.049900 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3848.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.049904 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.049906 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.049910 7f8691f66700 5 queue_transactions existing osr(3.27b 0x3ed0bd0)/0x3ed0bd0 2014-06-09 16:22:12.049903 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.049915 7f86a5610700 10 _do_op 0x159978c0 seq 3848 r = 0, finisher 0x3050b1e0 0 2014-06-09 16:22:12.049922 7f8691f66700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049916 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.049924 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f83a0] 2014-06-09 16:22:12.049926 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f83a0] 2014-06-09 16:22:12.049927 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.049927 7f868bf5a700 2 waiting 51 > 50 ops || 42286 > 104857600 2014-06-09 16:22:12.049928 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.049932 7f8691765700 2 waiting 51 > 50 ops || 42316 > 104857600 2014-06-09 16:22:12.049931 7f86a5610700 10 _finish_op 0x159978c0 seq 3848 osr(1.122 0x384c890)/0x384c890 2014-06-09 16:22:12.049926 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.049935 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.049949 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050b1e0] 2014-06-09 16:22:12.049952 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050b1e0] 2014-06-09 16:22:12.049945 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.049953 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.049954 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.049957 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7806c0 (1 active) 2014-06-09 16:22:12.049959 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b680 (2 active) 2014-06-09 16:22:12.049953 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.049961 7f86a5610700 5 _do_op 0xcabd260 seq 3849 osr(1.198 0x384cb30)/0x384cb30 start 2014-06-09 16:22:12.049963 7f86a5610700 10 _do_transaction on 0xbf2e840 2014-06-09 16:22:12.049968 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.049966 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:905 2014-06-09 16:22:12.049976 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.049985 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.2d5( v 213'531 (0'0,213'531] local-les=243 n=105 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.049994 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.050003 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.049999 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.050015 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3849.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.050019 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.050012 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.050020 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x306a200,0x22008d00,0xe032200,0x22008c00,0x22008b00,0x22008a00 2014-06-09 16:22:12.050029 7f86a5610700 10 _do_op 0xcabd260 seq 3849 r = 0, finisher 0xc5a7120 0 2014-06-09 16:22:12.050038 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x327f23c0] 2014-06-09 16:22:12.050040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x327f23c0] 2014-06-09 16:22:12.050042 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.050043 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.050046 7f86a5610700 10 _finish_op 0xcabd260 seq 3849 osr(1.198 0x384cb30)/0x384cb30 2014-06-09 16:22:12.050045 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.050054 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a7120] 2014-06-09 16:22:12.050058 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a7120] 2014-06-09 16:22:12.050059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.050060 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.050055 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.050062 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b680 (1 active) 2014-06-09 16:22:12.050065 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3010e0 (2 active) 2014-06-09 16:22:12.050064 7f868cf5c700 10 op_submit_start 3897 2014-06-09 16:22:12.050067 7f868cf5c700 5 queue_transactions (trailing journal) 3897 2014-06-09 16:22:12.050067 7f86a5610700 5 _do_op 0xc930840 seq 3850 osr(3.27b 0x3ed0bd0)/0x3ed0bd0 start 2014-06-09 16:22:12.050069 7f86a5610700 10 _do_transaction on 0xc257200 2014-06-09 16:22:12.050063 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.050069 7f868cf5c700 5 queue_op 0xc930960 seq 3897 osr(2.57a 0x2ed1870) 828 bytes (queue has 49 ops and 40630 bytes) 2014-06-09 16:22:12.050073 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.050075 7f868cf5c700 10 op_submit_finish 3897 2014-06-09 16:22:12.050076 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:974 2014-06-09 16:22:12.050079 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:12.050086 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.050095 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.050105 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.050108 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.050115 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.050110 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.050120 7f868cf5c700 5 queue_transactions existing osr(3.3a8 0x3ed0f80)/0x3ed0f80 2014-06-09 16:22:12.050120 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3850.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.050123 7f868cf5c700 10 op_submit_start 3898 2014-06-09 16:22:12.050123 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.050124 7f868cf5c700 5 queue_transactions (trailing journal) 3898 2014-06-09 16:22:12.050125 7f868cf5c700 5 queue_op 0xc9309c0 seq 3898 osr(3.3a8 0x3ed0f80) 828 bytes (queue has 50 ops and 41458 bytes) 2014-06-09 16:22:12.050121 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.050128 7f868cf5c700 10 op_submit_finish 3898 2014-06-09 16:22:12.050128 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.050131 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.050139 7f86a5610700 10 _do_op 0xc930840 seq 3850 r = 0, finisher 0x314134a0 0 2014-06-09 16:22:12.050137 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.050149 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x18742ca0] 2014-06-09 16:22:12.050151 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x18742ca0] 2014-06-09 16:22:12.050145 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.050152 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.050153 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.050156 7f86a5610700 10 _finish_op 0xc930840 seq 3850 osr(3.27b 0x3ed0bd0)/0x3ed0bd0 2014-06-09 16:22:12.050150 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.050153 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.050159 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.050164 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x314134a0] 2014-06-09 16:22:12.050167 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x314134a0] 2014-06-09 16:22:12.050168 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.050169 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.050172 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3010e0 (1 active) 2014-06-09 16:22:12.050166 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.050166 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:12.050174 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd8c0 (2 active) 2014-06-09 16:22:12.050176 7f86a5610700 5 _do_op 0xc931620 seq 3851 osr(2.5b7 0x2ed1b70)/0x2ed1b70 start 2014-06-09 16:22:12.050178 7f86a5610700 10 _do_transaction on 0xbf50f00 2014-06-09 16:22:12.050174 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.050175 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.050183 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.050183 7f8689755700 20 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.050182 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.050191 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.050195 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.050197 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:12.050209 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.050218 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.050216 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.050218 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x13023500,0x13023400,0x1804df00,0x13023300,0x1804de00,0x13023200,0x19fe0500,0x16bd6c00,0x19fe0400,0x306a300,0x19fe0300,0xe032800,0x19fe0200 2014-06-09 16:22:12.050225 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.050230 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3851.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.050234 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.050234 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.050244 7f86a5610700 10 _do_op 0xc931620 seq 3851 r = 0, finisher 0x4eb98c0 0 2014-06-09 16:22:12.050239 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.050252 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6e820] 2014-06-09 16:22:12.050254 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6e820] 2014-06-09 16:22:12.050256 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.050257 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.050251 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.050260 7f86a5610700 10 _finish_op 0xc931620 seq 3851 osr(2.5b7 0x2ed1b70)/0x2ed1b70 2014-06-09 16:22:12.050259 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.050268 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x4eb98c0] 2014-06-09 16:22:12.050259 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.050272 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x4eb98c0] 2014-06-09 16:22:12.050273 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.050274 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.050267 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.050270 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.050276 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd8c0 (1 active) 2014-06-09 16:22:12.050279 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653200 (2 active) 2014-06-09 16:22:12.050281 7f86a5610700 5 _do_op 0x15996780 seq 3852 osr(0.117 0x2bd66a0)/0x2bd66a0 start 2014-06-09 16:22:12.050283 7f86a5610700 10 _do_transaction on 0xc1392c0 2014-06-09 16:22:12.050276 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.050278 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.050287 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.050285 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.050291 7f8689755700 15 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1049 2014-06-09 16:22:12.050298 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.050301 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.050307 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.050310 7f8689755700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.050320 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.050321 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.050315 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.050325 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.050325 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.050333 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3852.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.050337 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.050329 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.050337 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.913381 7 0.000459 2014-06-09 16:22:12.050346 7f86a5610700 10 _do_op 0x15996780 seq 3852 r = 0, finisher 0x32074a20 0 2014-06-09 16:22:12.050347 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.050355 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1e08ffa0] 2014-06-09 16:22:12.050357 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1e08ffa0] 2014-06-09 16:22:12.050358 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.050359 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.050362 7f86a5610700 10 _finish_op 0x15996780 seq 3852 osr(0.117 0x2bd66a0)/0x2bd66a0 2014-06-09 16:22:12.050363 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.050357 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.050367 7f868cf5c700 5 queue_transactions existing osr(3.e8 0x3ed0510)/0x3ed0510 2014-06-09 16:22:12.050370 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32074a20] 2014-06-09 16:22:12.050370 7f868cf5c700 10 op_submit_start 3899 2014-06-09 16:22:12.050371 7f868cf5c700 5 queue_transactions (trailing journal) 3899 2014-06-09 16:22:12.050372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32074a20] 2014-06-09 16:22:12.050374 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.050372 7f868cf5c700 5 queue_op 0x33a3140 seq 3899 osr(3.e8 0x3ed0510) 826 bytes (queue has 48 ops and 39800 bytes) 2014-06-09 16:22:12.050366 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.050375 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.050377 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653200 (1 active) 2014-06-09 16:22:12.050379 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339680 (2 active) 2014-06-09 16:22:12.050376 7f8689755700 5 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.050383 7f86a5610700 5 _do_op 0x15996660 seq 3853 osr(2.67c 0x2ed1f90)/0x2ed1f90 start 2014-06-09 16:22:12.050385 7f86a5610700 10 _do_transaction on 0xc145a40 2014-06-09 16:22:12.050385 7f868ef60700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050384 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.050391 7f8692f68700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050392 7f868cf5c700 10 op_submit_finish 3899 2014-06-09 16:22:12.050391 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.050397 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.050396 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.624( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.050402 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.050413 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.050422 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.050430 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.050431 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.050434 7f868cf5c700 5 queue_transactions existing osr(2.624 0x2ed1c50)/0x2ed1c50 2014-06-09 16:22:12.050437 7f868cf5c700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050430 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.050441 7f868ff62700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.050439 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.050443 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3853.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.050446 7f8690763700 10 op_submit_start 3900 2014-06-09 16:22:12.050447 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.050447 7f8690763700 5 queue_transactions (trailing journal) 3900 2014-06-09 16:22:12.050449 7f8690763700 5 queue_op 0x30d962a0 seq 3900 osr(3.254 0x3ed0c30) 828 bytes (queue has 50 ops and 41456 bytes) 2014-06-09 16:22:12.050451 7f8689755700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.050452 7f8690763700 10 op_submit_finish 3900 2014-06-09 16:22:12.050457 7f86a5610700 10 _do_op 0x15996660 seq 3853 r = 0, finisher 0xbcc9120 0 2014-06-09 16:22:12.050456 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.458( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.458( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.050466 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc168b20] 2014-06-09 16:22:12.050468 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc168b20] 2014-06-09 16:22:12.050470 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.050471 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.050466 7f8689755700 15 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:197 2014-06-09 16:22:12.050473 7f86a5610700 10 _finish_op 0x15996660 seq 3853 osr(2.67c 0x2ed1f90)/0x2ed1f90 2014-06-09 16:22:12.050483 7f868f761700 10 op_submit_start 3901 2014-06-09 16:22:12.050484 7f868f761700 5 queue_transactions (trailing journal) 3901 2014-06-09 16:22:12.050485 7f868f761700 5 queue_op 0xc64bc80 seq 3901 osr(3.3a5 0x3ed0fb0) 828 bytes (queue has 50 ops and 41456 bytes) 2014-06-09 16:22:12.050489 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.050489 7f8692767700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.050494 7f8689f56700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050494 7f8690763700 5 queue_transactions existing osr(0.458 0x2bd7510)/0x2bd7510 2014-06-09 16:22:12.050496 7f8690763700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050498 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.050498 7f868b759700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050502 7f8689755700 5 queue_transactions existing osr(2.288 0x2ed0d10)/0x2ed0d10 2014-06-09 16:22:12.050505 7f8689755700 2 waiting 51 > 50 ops || 42317 > 104857600 2014-06-09 16:22:12.050507 7f868e75f700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050511 7f8690f64700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050517 7f8691f66700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050521 7f868bf5a700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050525 7f8691765700 2 waiting 51 > 50 ops || 42314 > 104857600 2014-06-09 16:22:12.050528 7f868d75d700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050532 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbcc9120] 2014-06-09 16:22:12.050536 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbcc9120] 2014-06-09 16:22:12.050537 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.050538 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.050538 7f868af58700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050545 7f8692f68700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050542 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339680 (1 active) 2014-06-09 16:22:12.050556 7f86a5e11700 5 _do_op 0xc34b8c0 seq 3847 osr(0.578 0x2bd7800)/0x2bd7800 start 2014-06-09 16:22:12.050560 7f86a5e11700 10 _do_transaction on 0xc33a900 2014-06-09 16:22:12.050565 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.050553 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f440 (2 active) 2014-06-09 16:22:12.050591 7f868f761700 10 op_submit_finish 3901 2014-06-09 16:22:12.050596 7f868a757700 10 op_submit_start 3902 2014-06-09 16:22:12.050600 7f868a757700 5 queue_transactions (trailing journal) 3902 2014-06-09 16:22:12.050602 7f868a757700 5 queue_op 0xcabc600 seq 3902 osr(3.761 0x3ed1e10) 828 bytes (queue has 50 ops and 41456 bytes) 2014-06-09 16:22:12.050604 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.050605 7f868a757700 10 op_submit_finish 3902 2014-06-09 16:22:12.050596 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.ab( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.050617 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3847.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.051485 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.051500 7f86a5e11700 10 _do_op 0xc34b8c0 seq 3847 r = 0, finisher 0xbe67d80 0 2014-06-09 16:22:12.051505 7f86a5e11700 10 _finish_op 0xc34b8c0 seq 3847 osr(0.578 0x2bd7800)/0x2bd7800 2014-06-09 16:22:12.051507 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189ab2c0] 2014-06-09 16:22:12.051510 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189ab2c0] 2014-06-09 16:22:12.051512 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.051512 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.050610 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.050624 7f868ff62700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.050628 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.051529 7f868ef60700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.050631 7f86a5610700 5 _do_op 0x15996a80 seq 3854 osr(3.139 0x3ed0780)/0x3ed0780 start 2014-06-09 16:22:12.051532 7f868f761700 5 queue_transactions existing osr(1.ab 0x384c6f0)/0x384c6f0 2014-06-09 16:22:12.051534 7f868f761700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.051536 7f86a5610700 10 _do_transaction on 0xbffe840 2014-06-09 16:22:12.051538 7f868c75b700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.051525 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.905029 7 0.000369 2014-06-09 16:22:12.051542 7f868df5e700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.051548 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.051541 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.051552 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.051577 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.051576 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000035 0 0.000000 2014-06-09 16:22:12.051586 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3854.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.051589 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.051588 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.051597 7f86a5610700 10 _do_op 0x15996a80 seq 3854 r = 0, finisher 0xc5a71c0 0 2014-06-09 16:22:12.051597 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.051603 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042be80] 2014-06-09 16:22:12.051607 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042be80] 2014-06-09 16:22:12.051608 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.051609 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.051605 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.051620 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.051631 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.051641 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.051655 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.051683 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 2014-06-09 16:22:12.051718 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.051722 7f868a757700 5 queue_transactions existing osr(0.5a8 0x2bd7b00)/0x2bd7b00 2014-06-09 16:22:12.051725 7f868a757700 2 waiting 51 > 50 ops || 42317 > 104857600 2014-06-09 16:22:12.051730 7f868cf5c700 2 waiting 51 > 50 ops || 42284 > 104857600 2014-06-09 16:22:12.051741 7f8692767700 10 op_submit_start 3903 2014-06-09 16:22:12.051743 7f8692767700 5 queue_transactions (trailing journal) 3903 2014-06-09 16:22:12.051741 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d05a0 (1 active) 2014-06-09 16:22:12.051745 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0000 (2 active) 2014-06-09 16:22:12.051745 7f8692767700 5 queue_op 0xd326840 seq 3903 osr(1.a7 0x384c730) 826 bytes (queue has 50 ops and 41454 bytes) 2014-06-09 16:22:12.051748 7f8692767700 10 op_submit_finish 3903 2014-06-09 16:22:12.051749 7f86a5610700 10 _finish_op 0x15996a80 seq 3854 osr(3.139 0x3ed0780)/0x3ed0780 2014-06-09 16:22:12.051752 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f440 (1 active) 2014-06-09 16:22:12.051753 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b7a0 (2 active) 2014-06-09 16:22:12.051755 7f86a5610700 5 _do_op 0x10039e00 seq 3856 osr(3.461 0x3ed1120)/0x3ed1120 start 2014-06-09 16:22:12.051758 7f86a5610700 10 _do_transaction on 0xbf89a40 2014-06-09 16:22:12.051751 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.051763 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.051766 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.051776 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.051785 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.051800 7f8692767700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.051816 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1116 2014-06-09 16:22:12.051832 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.051835 7f8690763700 10 op_submit_start 3904 2014-06-09 16:22:12.051837 7f8690763700 5 queue_transactions (trailing journal) 3904 2014-06-09 16:22:12.051838 7f8690763700 5 queue_op 0x30d96240 seq 3904 osr(0.458 0x2bd7510) 828 bytes (queue has 50 ops and 41454 bytes) 2014-06-09 16:22:12.051840 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.051843 7f8690763700 10 op_submit_finish 3904 2014-06-09 16:22:12.051836 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:12.051849 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.051846 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.a6( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.051857 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3856.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.051860 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.051858 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.051865 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.051871 7f86a5610700 10 _do_op 0x10039e00 seq 3856 r = 0, finisher 0x30c54900 0 2014-06-09 16:22:12.051876 7f8692767700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.051880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6ab60] 2014-06-09 16:22:12.051882 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6ab60] 2014-06-09 16:22:12.051884 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.051883 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.051885 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.051888 7f86a5610700 10 _finish_op 0x10039e00 seq 3856 osr(3.461 0x3ed1120)/0x3ed1120 2014-06-09 16:22:12.051891 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b7a0 (1 active) 2014-06-09 16:22:12.051893 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa7e0 (2 active) 2014-06-09 16:22:12.051895 7f86a5610700 5 _do_op 0xc9308a0 seq 3857 osr(2.646 0x2ed1c30)/0x2ed1c30 start 2014-06-09 16:22:12.051895 7f8690763700 5 queue_transactions existing osr(1.a6 0x384c750)/0x384c750 2014-06-09 16:22:12.051897 7f86a5610700 10 _do_transaction on 0x2dce9c0 2014-06-09 16:22:12.051893 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:156 2014-06-09 16:22:12.051901 7f8690763700 10 op_submit_start 3905 2014-06-09 16:22:12.051902 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.051903 7f8690763700 5 queue_transactions (trailing journal) 3905 2014-06-09 16:22:12.051904 7f8690763700 5 queue_op 0x309a3680 seq 3905 osr(1.a6 0x384c750) 826 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.051906 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.051907 7f8690763700 10 op_submit_finish 3905 2014-06-09 16:22:12.051911 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.298( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:12.051910 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.2fd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:12.051937 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.051940 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.051943 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.051945 7f8692767700 5 queue_transactions existing osr(2.298 0x2ed0c70)/0x2ed0c70 2014-06-09 16:22:12.051947 7f8692767700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.051947 7f8690763700 5 queue_transactions existing osr(0.2fd 0x2bd6e50)/0x2bd6e50 2014-06-09 16:22:12.051950 7f8690763700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.051951 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3857.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.051891 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.051954 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.051956 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.051957 7f869af78700 20 osd.26 243 _dispatch 0x280c4fc0 pg_info(1 pgs e243:3.21e) v4 2014-06-09 16:22:12.051967 7f86a5610700 10 _do_op 0xc9308a0 seq 3857 r = 0, finisher 0x2e0ec20 0 2014-06-09 16:22:12.051964 7f869af78700 5 -- op tracker -- , seq: 3616, time: 2014-06-09 16:22:12.050527, event: header_read, op: pg_info(1 pgs e243:3.21e) 2014-06-09 16:22:12.051970 7f8689f56700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.051977 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x13965600] 2014-06-09 16:22:12.051979 7f868b759700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.051980 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x13965600] 2014-06-09 16:22:12.051981 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.051982 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.051984 7f868e75f700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.051986 7f86a5610700 10 _finish_op 0xc9308a0 seq 3857 osr(2.646 0x2ed1c30)/0x2ed1c30 2014-06-09 16:22:12.051988 7f8690f64700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.051990 7f86a5e11700 5 _do_op 0x159979e0 seq 3855 osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 start 2014-06-09 16:22:12.051993 7f86a5e11700 10 _do_transaction on 0xbf39b00 2014-06-09 16:22:12.051993 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbe67d80,0xc5a71c0,0x30c54900] 2014-06-09 16:22:12.051998 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.052000 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbe67d80,0xc5a71c0,0x30c54900] 2014-06-09 16:22:12.052002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.052002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.052005 7f8691f66700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052010 7f868bf5a700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052014 7f8691765700 2 waiting 51 > 50 ops || 42310 > 104857600 2014-06-09 16:22:12.052033 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.052041 7f868d75d700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052043 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3855.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.052045 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.051973 7f869af78700 5 -- op tracker -- , seq: 3616, time: 2014-06-09 16:22:12.050529, event: throttled, op: pg_info(1 pgs e243:3.21e) 2014-06-09 16:22:12.052053 7f86a5e11700 10 _do_op 0x159979e0 seq 3855 r = 0, finisher 0x1e08ff80 0 2014-06-09 16:22:12.052057 7f869af78700 5 -- op tracker -- , seq: 3616, time: 2014-06-09 16:22:12.050544, event: all_read, op: pg_info(1 pgs e243:3.21e) 2014-06-09 16:22:12.052059 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x320752a0] 2014-06-09 16:22:12.052062 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x320752a0] 2014-06-09 16:22:12.052063 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.052064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.052064 7f869af78700 5 -- op tracker -- , seq: 3616, time: 2014-06-09 16:22:12.051890, event: dispatched, op: pg_info(1 pgs e243:3.21e) 2014-06-09 16:22:12.052071 7f869af78700 5 -- op tracker -- , seq: 3616, time: 2014-06-09 16:22:12.052069, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.21e) 2014-06-09 16:22:12.052075 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.21e) v4 from osd.4 2014-06-09 16:22:12.052078 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x280c4fc0 2014-06-09 16:22:12.052083 7f869af78700 5 -- op tracker -- , seq: 3616, time: 2014-06-09 16:22:12.052082, event: started, op: pg_info(1 pgs e243:3.21e) 2014-06-09 16:22:12.052103 7f869af78700 5 -- op tracker -- , seq: 3616, time: 2014-06-09 16:22:12.052101, event: done, op: pg_info(1 pgs e243:3.21e) 2014-06-09 16:22:12.052047 7f868af58700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052112 7f8692f68700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052118 7f868ff62700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052119 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.052122 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.052122 7f868ef60700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052127 7f868f761700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052132 7f868c75b700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052136 7f868df5e700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052142 7f868a757700 2 waiting 51 > 50 ops || 42313 > 104857600 2014-06-09 16:22:12.052148 7f868cf5c700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052153 7f8689755700 2 waiting 51 > 50 ops || 42313 > 104857600 2014-06-09 16:22:12.052162 7f8692767700 10 op_submit_start 3906 2014-06-09 16:22:12.052163 7f8692767700 5 queue_transactions (trailing journal) 3906 2014-06-09 16:22:12.052163 7f8690763700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052165 7f8692767700 5 queue_op 0x11fcf020 seq 3906 osr(2.298 0x2ed0c70) 828 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.052169 7f868b759700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052174 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0ec20] 2014-06-09 16:22:12.052174 7f868e75f700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052177 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0ec20] 2014-06-09 16:22:12.052179 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.052180 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.052182 7f8690f64700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052185 7f8691f66700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052185 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa7e0 (1 active) 2014-06-09 16:22:12.052190 7f868bf5a700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052188 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287d40 (2 active) 2014-06-09 16:22:12.052194 7f8691765700 2 waiting 51 > 50 ops || 42310 > 104857600 2014-06-09 16:22:12.052197 7f86a5610700 5 _do_op 0xcabcae0 seq 3858 osr(3.52a 0x3ed1670)/0x3ed1670 start 2014-06-09 16:22:12.052200 7f86a5610700 10 _do_transaction on 0xc20bc80 2014-06-09 16:22:12.052199 7f86a5e11700 10 _finish_op 0x159979e0 seq 3855 osr(3.2b6 0x3ed0ee0)/0x3ed0ee0 2014-06-09 16:22:12.052205 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0000 (1 active) 2014-06-09 16:22:12.052207 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1e08ff80] 2014-06-09 16:22:12.052208 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcfc0 (2 active) 2014-06-09 16:22:12.052209 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1e08ff80] 2014-06-09 16:22:12.052211 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.052212 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.052212 7f8692767700 10 op_submit_finish 3906 2014-06-09 16:22:12.052214 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.052214 7f868d75d700 10 op_submit_start 3907 2014-06-09 16:22:12.052215 7f868af58700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052216 7f868d75d700 5 queue_transactions (trailing journal) 3907 2014-06-09 16:22:12.052218 7f868d75d700 5 queue_op 0x11fce900 seq 3907 osr(0.6a8 0x2bd7d90) 828 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.052221 7f8692f68700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052222 7f868d75d700 10 op_submit_finish 3907 2014-06-09 16:22:12.052226 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.052238 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f23c0 (19 active) 2014-06-09 16:22:12.052246 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.052256 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.052262 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.052268 7f868d75d700 5 queue_transactions existing osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 2014-06-09 16:22:12.052251 7f8692767700 10 osd.26 243 dequeue_op 0x16bd7b00 prio 63 cost 884736 latency 1.750284 osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.052269 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3858.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.052270 7f868d75d700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052272 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.052275 7f8690763700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052281 7f868ef60700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052276 7f8692767700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.052271, event: reached_pg, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.052287 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.052305 7f86a5610700 10 _do_op 0xcabcae0 seq 3858 r = 0, finisher 0x2e0e860 0 2014-06-09 16:22:12.052301 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x16bd7b00 2014-06-09 16:22:12.052315 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe30840] 2014-06-09 16:22:12.052317 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe30840] 2014-06-09 16:22:12.052318 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.052319 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.052322 7f86a5610700 10 _finish_op 0xcabcae0 seq 3858 osr(3.52a 0x3ed1670)/0x3ed1670 2014-06-09 16:22:12.052311 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.052331 7f868af58700 10 op_submit_start 3908 2014-06-09 16:22:12.052332 7f868c75b700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052334 7f868af58700 5 queue_transactions (trailing journal) 3908 2014-06-09 16:22:12.052337 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0e860] 2014-06-09 16:22:12.052335 7f868af58700 5 queue_op 0x324c0120 seq 3908 osr(3.53d 0x3ed1600) 828 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.052337 7f8692767700 15 getattr 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 '_' 2014-06-09 16:22:12.052340 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0e860] 2014-06-09 16:22:12.052342 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.052343 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.052345 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287d40 (1 active) 2014-06-09 16:22:12.052347 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcc60 (2 active) 2014-06-09 16:22:12.052351 7f86a5610700 5 _do_op 0xbf7ef00 seq 3860 osr(0.4c9 0x2bd7650)/0x2bd7650 start 2014-06-09 16:22:12.052353 7f868af58700 10 op_submit_finish 3908 2014-06-09 16:22:12.052354 7f86a5610700 10 _do_transaction on 0xc33e0c0 2014-06-09 16:22:12.052359 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.052358 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.3a6( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:12.052389 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.052387 7f86a6b95700 10 osd.26 243 new session 0xc054400 con=0x30d67600 addr=172.16.96.220:0/1123748 2014-06-09 16:22:12.052394 7f868af58700 5 queue_transactions existing osr(0.3a6 0x2bd7330)/0x2bd7330 2014-06-09 16:22:12.052396 7f868af58700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052399 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.052402 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 but can_create 2014-06-09 16:22:12.052416 7f8692767700 15 getattr 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 'snapset' 2014-06-09 16:22:12.052421 7f868df5e700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052427 7f868ff62700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052431 7f868f761700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052435 7f8689755700 2 waiting 51 > 50 ops || 42313 > 104857600 2014-06-09 16:22:12.052440 7f8689f56700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052445 7f868b759700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052450 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.052464 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3860.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.052465 7f868a757700 2 waiting 51 > 50 ops || 42313 > 104857600 2014-06-09 16:22:12.052468 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.052469 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.052472 7f8690f64700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052472 7f8692767700 15 getattr 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/snapdir//3 'snapset' 2014-06-09 16:22:12.052501 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.052505 7f86a5610700 10 _do_op 0xbf7ef00 seq 3860 r = 0, finisher 0xbd7df20 0 2014-06-09 16:22:12.052514 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d6ca0] 2014-06-09 16:22:12.052516 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d6ca0] 2014-06-09 16:22:12.052517 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.052518 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.052521 7f86a5610700 10 _finish_op 0xbf7ef00 seq 3860 osr(0.4c9 0x2bd7650)/0x2bd7650 2014-06-09 16:22:12.052511 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30e07600 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.052530 7f868bf5a700 10 op_submit_start 3909 2014-06-09 16:22:12.052531 7f868bf5a700 5 queue_transactions (trailing journal) 3909 2014-06-09 16:22:12.052524 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.052533 7f868bf5a700 5 queue_op 0xc704ae0 seq 3909 osr(3.587 0x3ed1840) 828 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.052537 7f8691765700 2 waiting 51 > 50 ops || 42310 > 104857600 2014-06-09 16:22:12.052541 7f8692f68700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052534 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.052550 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7df20] 2014-06-09 16:22:12.052554 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7df20] 2014-06-09 16:22:12.052555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.052556 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.052558 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcc60 (1 active) 2014-06-09 16:22:12.052545 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30e07600 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 rwstate(none n=0 w=0) oi: 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd50a0 snapset: 0=[]:[] 2014-06-09 16:22:12.052560 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115680 (2 active) 2014-06-09 16:22:12.052561 7f868d75d700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052564 7f868bf5a700 10 op_submit_finish 3909 2014-06-09 16:22:12.052567 7f8690763700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052561 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 @head oi=8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.052567 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.3e8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.3e8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.052580 7f8692767700 15 getattr 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/snapdir//3 '_' 2014-06-09 16:22:12.052597 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.052601 7f868bf5a700 5 queue_transactions existing osr(2.3e8 0x2ed1310)/0x2ed1310 2014-06-09 16:22:12.052604 7f868bf5a700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052608 7f868ef60700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052611 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.052612 7f868cf5c700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052617 7f86a5610700 5 _do_op 0x11fce1e0 seq 3861 osr(1.65e 0x384df30)/0x384df30 start 2014-06-09 16:22:12.052620 7f86a5610700 10 _do_transaction on 0x34f1500 2014-06-09 16:22:12.052618 7f868af58700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052613 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/snapdir//3 and !can_create 2014-06-09 16:22:12.052624 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.052625 7f868df5e700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052630 7f868ff62700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052634 7f868f761700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052631 7f8692767700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.052626, event: started, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.052639 7f8689755700 2 waiting 51 > 50 ops || 42313 > 104857600 2014-06-09 16:22:12.052644 7f8689f56700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052639 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x306dc400 2014-06-09 16:22:12.052649 7f868b759700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052657 7f8692767700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.052652, event: started, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.052672 7f868a757700 2 waiting 51 > 50 ops || 42313 > 104857600 2014-06-09 16:22:12.052673 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.052664 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] ov 0'0 av 243'637 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.052679 7f8690f64700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052684 7f868c75b700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052677 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 2014-06-09 16:22:12.052689 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.052697 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3861.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.052700 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.052703 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~884736 2014-06-09 16:22:12.052714 7f86a5610700 10 _do_op 0x11fce1e0 seq 3861 r = 0, finisher 0xcd32300 0 2014-06-09 16:22:12.052723 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32073740] 2014-06-09 16:22:12.052725 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32073740] 2014-06-09 16:22:12.052726 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.052718 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228e00 2014-06-09 16:22:12.052727 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.052730 7f86a5610700 10 _finish_op 0x11fce1e0 seq 3861 osr(1.65e 0x384df30)/0x384df30 2014-06-09 16:22:12.052730 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 snapset=0x30dd50d8 snapc=0=[] 2014-06-09 16:22:12.052739 7f8692f68700 10 op_submit_start 3910 2014-06-09 16:22:12.052741 7f8692f68700 5 queue_transactions (trailing journal) 3910 2014-06-09 16:22:12.052742 7f8692f68700 5 queue_op 0x324e7500 seq 3910 osr(1.697 0x384de10) 828 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.052740 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.052751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd32300] 2014-06-09 16:22:12.052752 7f868d75d700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052754 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd32300] 2014-06-09 16:22:12.052755 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.052756 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.052758 7f8690763700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052759 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115680 (1 active) 2014-06-09 16:22:12.052751 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.052763 7f868bf5a700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052767 7f868ef60700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052761 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 0x306dc400 op modify 2014-06-09 16:22:12.052771 7f868cf5c700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052760 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad560 (2 active) 2014-06-09 16:22:12.052778 7f8692f68700 10 op_submit_finish 3910 2014-06-09 16:22:12.052777 7f868af58700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052784 7f868df5e700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052773 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:15.856351 2014-06-09 16:22:12.052789 7f868ff62700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052781 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:243, info:2.1fc( empty local-les=212 n=0 ec=1 les/c 212/213 242/242/242)) 2014-06-09 16:22:12.052794 7f868f761700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052799 7f8689755700 2 waiting 51 > 50 ops || 42313 > 104857600 2014-06-09 16:22:12.052795 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:12.052793 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.052805 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.052810 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.052810 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6f( v 213'510 (0'0,213'510] local-les=243 n=95 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.052820 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 24 on osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.052838 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.052842 7f8692f68700 5 queue_transactions existing osr(3.6f 0x3ed0280)/0x3ed0280 2014-06-09 16:22:12.052835 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 24 o 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.052844 7f8692f68700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.052848 7f8689f56700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052853 7f868b759700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052857 7f86a5610700 5 _do_op 0x30033800 seq 3862 osr(2.5a7 0x2ed1bf0)/0x2ed1bf0 start 2014-06-09 16:22:12.052860 7f86a5610700 10 _do_transaction on 0x3562300 2014-06-09 16:22:12.052857 7f8692767700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.052852, event: waiting for subops from 9, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.052866 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.052875 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.052879 7f868a757700 2 waiting 51 > 50 ops || 42313 > 104857600 2014-06-09 16:22:12.052886 7f8690f64700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052891 7f868c75b700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.052894 7f8691765700 2 waiting 51 > 50 ops || 42310 > 104857600 2014-06-09 16:22:12.052894 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 214'636 (0'0,214'636] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'636], crt=0'0) [243'637 (0'0) modify 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 by client.4571.0:35826 2014-06-09 10:19:15.856351] 2014-06-09 16:22:12.052917 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.052914 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'637 (0'0) modify 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 by client.4571.0:35826 2014-06-09 10:19:15.856351 2014-06-09 16:22:12.052933 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3862.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.052937 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.052934 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.052950 7f86a5610700 10 _do_op 0x30033800 seq 3862 r = 0, finisher 0x318c2240 0 2014-06-09 16:22:12.052960 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2040] 2014-06-09 16:22:12.052963 7f8692767700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'637, trimmed: 2014-06-09 16:22:12.052963 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.052972 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.052982 7f8692767700 5 queue_transactions existing osr(3.2b3 0x3ed0f00)/0x3ed0f00 2014-06-09 16:22:12.052985 7f8692767700 2 waiting 51 > 50 ops || 928586 > 104857600 2014-06-09 16:22:12.052980 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.5a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 2014-06-09 16:22:12.052988 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2040] 2014-06-09 16:22:12.052990 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.052991 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.052994 7f86a5610700 10 _finish_op 0x30033800 seq 3862 osr(2.5a7 0x2ed1bf0)/0x2ed1bf0 2014-06-09 16:22:12.053003 7f868d75d700 10 op_submit_start 3911 2014-06-09 16:22:12.053004 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c2240] 2014-06-09 16:22:12.053005 7f868d75d700 5 queue_transactions (trailing journal) 3911 2014-06-09 16:22:12.053007 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c2240] 2014-06-09 16:22:12.053008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.053009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.053007 7f868d75d700 5 queue_op 0x11fcfc80 seq 3911 osr(3.6cb 0x3ed1cb0) 828 bytes (queue has 50 ops and 41419 bytes) 2014-06-09 16:22:12.053012 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad560 (1 active) 2014-06-09 16:22:12.053012 7f8690763700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053014 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa120 (2 active) 2014-06-09 16:22:12.053018 7f868bf5a700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053019 7f868d75d700 10 op_submit_finish 3911 2014-06-09 16:22:12.053022 7f868ef60700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053025 7f868cf5c700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053029 7f86a5610700 5 _do_op 0xbf7ec60 seq 3863 osr(2.f4 0x2ed0790)/0x2ed0790 start 2014-06-09 16:22:12.053030 7f868af58700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053032 7f86a5610700 10 _do_transaction on 0xc2ae900 2014-06-09 16:22:12.053022 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.38a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.053036 7f868df5e700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053036 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.053069 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.053075 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.053080 7f868d75d700 5 queue_transactions existing osr(2.38a 0x2ed10f0)/0x2ed10f0 2014-06-09 16:22:12.053082 7f868d75d700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053081 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3863.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.053085 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.053087 7f868ff62700 2 waiting 51 > 50 ops || 42245 > 104857600 2014-06-09 16:22:12.053093 7f868f761700 2 waiting 51 > 50 ops || 42245 > 104857600 2014-06-09 16:22:12.053101 7f86a5610700 10 _do_op 0xbf7ec60 seq 3863 r = 0, finisher 0x318c23a0 0 2014-06-09 16:22:12.053109 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2160] 2014-06-09 16:22:12.053111 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2160] 2014-06-09 16:22:12.053112 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.053113 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.053116 7f86a5610700 10 _finish_op 0xbf7ec60 seq 3863 osr(2.f4 0x2ed0790)/0x2ed0790 2014-06-09 16:22:12.053124 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c23a0] 2014-06-09 16:22:12.053128 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c23a0] 2014-06-09 16:22:12.053129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.053130 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.053132 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa120 (1 active) 2014-06-09 16:22:12.053135 7f8690763700 10 op_submit_start 3912 2014-06-09 16:22:12.053136 7f8692f68700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053137 7f8690763700 5 queue_transactions (trailing journal) 3912 2014-06-09 16:22:12.053139 7f8689f56700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053138 7f8690763700 5 queue_op 0x309a30e0 seq 3912 osr(0.2fd 0x2bd6e50) 828 bytes (queue has 50 ops and 41421 bytes) 2014-06-09 16:22:12.053143 7f868b759700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053134 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6b40 (2 active) 2014-06-09 16:22:12.053149 7f86a5610700 5 _do_op 0x11fce660 seq 3864 osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 start 2014-06-09 16:22:12.053152 7f86a5610700 10 _do_transaction on 0xc2e06c0 2014-06-09 16:22:12.053156 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.053163 7f868a757700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053169 7f8690f64700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053170 7f8690763700 10 op_submit_finish 3912 2014-06-09 16:22:12.053174 7f868c75b700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053177 7f8691765700 2 waiting 51 > 50 ops || 42279 > 104857600 2014-06-09 16:22:12.053180 7f8692767700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.053184 7f868bf5a700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053173 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.408( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.053187 7f868ef60700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053190 7f868cf5c700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053195 7f868af58700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053200 7f868df5e700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053210 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.053215 7f8690763700 5 queue_transactions existing osr(1.408 0x384d770)/0x384d770 2014-06-09 16:22:12.053214 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.053217 7f8690763700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053221 7f868d75d700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053226 7f868ff62700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053226 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3864.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.053230 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.053231 7f868f761700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053239 7f86a5610700 10 _do_op 0x11fce660 seq 3864 r = 0, finisher 0x32073a00 0 2014-06-09 16:22:12.053247 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd32320] 2014-06-09 16:22:12.053249 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd32320] 2014-06-09 16:22:12.053250 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.053251 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.053254 7f86a5610700 10 _finish_op 0x11fce660 seq 3864 osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 2014-06-09 16:22:12.053262 7f8689f56700 10 op_submit_start 3913 2014-06-09 16:22:12.053263 7f8689f56700 5 queue_transactions (trailing journal) 3913 2014-06-09 16:22:12.053264 7f8689f56700 5 queue_op 0x10038d20 seq 3913 osr(1.211 0x384cc90) 828 bytes (queue has 50 ops and 41421 bytes) 2014-06-09 16:22:12.053268 7f868b759700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053275 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32073a00] 2014-06-09 16:22:12.053279 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32073a00] 2014-06-09 16:22:12.053280 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.053281 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.053283 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6b40 (1 active) 2014-06-09 16:22:12.053285 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338000 (2 active) 2014-06-09 16:22:12.053288 7f8689f56700 10 op_submit_finish 3913 2014-06-09 16:22:12.053291 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.126( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.053323 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.053327 7f8689f56700 5 queue_transactions existing osr(2.126 0x2ed0670)/0x2ed0670 2014-06-09 16:22:12.053329 7f8689f56700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053333 7f86a5610700 5 _do_op 0x33a33e0 seq 3865 osr(1.6f5 0x2ed0170)/0x2ed0170 start 2014-06-09 16:22:12.053333 7f868a757700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053335 7f86a5610700 10 _do_transaction on 0x35be900 2014-06-09 16:22:12.053339 7f8690f64700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053343 7f868c75b700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053346 7f8691765700 2 waiting 51 > 50 ops || 42279 > 104857600 2014-06-09 16:22:12.053350 7f8692767700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.053353 7f868bf5a700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053356 7f8689755700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053362 7f868cf5c700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053359 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.053367 7f868af58700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053374 7f868df5e700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053381 7f8690763700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053386 7f868d75d700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053391 7f868ff62700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053395 7f868f761700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053397 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.053401 7f8692f68700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053405 7f868ef60700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053417 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3865.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.053420 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.053429 7f86a5610700 10 _do_op 0x33a33e0 seq 3865 r = 0, finisher 0xbd76640 0 2014-06-09 16:22:12.053436 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd67f60] 2014-06-09 16:22:12.053438 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd67f60] 2014-06-09 16:22:12.053439 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.053440 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.053443 7f86a5610700 10 _finish_op 0x33a33e0 seq 3865 osr(1.6f5 0x2ed0170)/0x2ed0170 2014-06-09 16:22:12.053452 7f8689f56700 10 op_submit_start 3914 2014-06-09 16:22:12.053453 7f8689f56700 5 queue_transactions (trailing journal) 3914 2014-06-09 16:22:12.053455 7f8689f56700 5 queue_op 0x30060240 seq 3914 osr(2.126 0x2ed0670) 828 bytes (queue has 50 ops and 41421 bytes) 2014-06-09 16:22:12.053455 7f868a757700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053462 7f8690f64700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053466 7f868c75b700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053470 7f8691765700 2 waiting 51 > 50 ops || 42279 > 104857600 2014-06-09 16:22:12.053475 7f8692767700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.053479 7f868bf5a700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053485 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76640] 2014-06-09 16:22:12.053488 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76640] 2014-06-09 16:22:12.053490 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.053491 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.053494 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338000 (1 active) 2014-06-09 16:22:12.053496 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4adb00 (2 active) 2014-06-09 16:22:12.053484 7f8689755700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053501 7f8689f56700 10 op_submit_finish 3914 2014-06-09 16:22:12.053504 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.219( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:12.053536 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.053541 7f8689f56700 5 queue_transactions existing osr(0.219 0x2bd6ad0)/0x2bd6ad0 2014-06-09 16:22:12.053543 7f8689f56700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053548 7f868cf5c700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053552 7f86a5610700 5 _do_op 0xc34b560 seq 3866 osr(0.71b 0x384c050)/0x384c050 start 2014-06-09 16:22:12.053553 7f868af58700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053555 7f86a5610700 10 _do_transaction on 0xc0cb080 2014-06-09 16:22:12.053559 7f868df5e700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053560 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.053565 7f8690763700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053570 7f868d75d700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053574 7f868ff62700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053579 7f868f761700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053585 7f8692f68700 2 waiting 51 > 50 ops || 42247 > 104857600 2014-06-09 16:22:12.053591 7f868ef60700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053595 7f868b759700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053613 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.053616 7f868e75f700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053625 7f8691f66700 2 waiting 51 > 50 ops || 42249 > 104857600 2014-06-09 16:22:12.053628 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3866.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.053630 7f86a5e11700 5 _do_op 0x324f4600 seq 3859 osr(3.609 0x3ed1b00)/0x3ed1b00 start 2014-06-09 16:22:12.053631 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.053633 7f86a5e11700 10 _do_transaction on 0xc2aa6c0 2014-06-09 16:22:12.053638 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.053673 7f86a5610700 10 _do_op 0xc34b560 seq 3866 r = 0, finisher 0xc2d6860 0 2014-06-09 16:22:12.053684 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd75920] 2014-06-09 16:22:12.053687 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd75920] 2014-06-09 16:22:12.053688 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.053689 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.053693 7f86a5610700 10 _finish_op 0xc34b560 seq 3866 osr(0.71b 0x384c050)/0x384c050 2014-06-09 16:22:12.053702 7f868a757700 10 op_submit_start 3915 2014-06-09 16:22:12.053702 7f8690f64700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053704 7f868a757700 5 queue_transactions (trailing journal) 3915 2014-06-09 16:22:12.053707 7f868c75b700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053706 7f868a757700 5 queue_op 0x2fcae300 seq 3915 osr(0.5a8 0x2bd7b00) 861 bytes (queue has 50 ops and 41454 bytes) 2014-06-09 16:22:12.053711 7f8691765700 2 waiting 51 > 50 ops || 42312 > 104857600 2014-06-09 16:22:12.053715 7f8692767700 2 waiting 51 > 50 ops || 928588 > 104857600 2014-06-09 16:22:12.053719 7f868bf5a700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053724 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d6860] 2014-06-09 16:22:12.053728 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d6860] 2014-06-09 16:22:12.053731 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.053733 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.053735 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4adb00 (1 active) 2014-06-09 16:22:12.053737 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e6c0 (2 active) 2014-06-09 16:22:12.053741 7f86a5610700 5 _do_op 0x30d96300 seq 3867 osr(2.182 0x2ed0990)/0x2ed0990 start 2014-06-09 16:22:12.053744 7f86a5610700 10 _do_transaction on 0xc2f49c0 2014-06-09 16:22:12.053743 7f868a757700 10 op_submit_finish 3915 2014-06-09 16:22:12.053749 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.053748 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 243/213 242/242/211) 2014-06-09 16:22:12.053764 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.053778 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3859.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.053783 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.053784 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.053789 7f868a757700 5 queue_transactions existing osr(3.218 0x3ed0cd0)/0x3ed0cd0 2014-06-09 16:22:12.053791 7f868a757700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053795 7f8689755700 2 waiting 51 > 50 ops || 42315 > 104857600 2014-06-09 16:22:12.053802 7f8689f56700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053808 7f868cf5c700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053825 7f868af58700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053832 7f868df5e700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053838 7f8690763700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053817 7f86a5e11700 10 _do_op 0x324f4600 seq 3859 r = 0, finisher 0x33a78e0 0 2014-06-09 16:22:12.053843 7f868d75d700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053847 7f86a5e11700 10 _finish_op 0x324f4600 seq 3859 osr(3.609 0x3ed1b00)/0x3ed1b00 2014-06-09 16:22:12.053848 7f868ff62700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.053848 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32535880] 2014-06-09 16:22:12.053851 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32535880] 2014-06-09 16:22:12.053852 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.053853 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.053858 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.053872 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3867.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.053877 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.053894 7f86a5610700 10 _do_op 0x30d96300 seq 3867 r = 0, finisher 0x324d0a00 0 2014-06-09 16:22:12.053904 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d0940] 2014-06-09 16:22:12.053907 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d0940] 2014-06-09 16:22:12.053908 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.053909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.053967 7f8690f64700 10 op_submit_start 3916 2014-06-09 16:22:12.053970 7f8690f64700 5 queue_transactions (trailing journal) 3916 2014-06-09 16:22:12.053971 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a78e0] 2014-06-09 16:22:12.053972 7f8690f64700 5 queue_op 0x30a3a40 seq 3916 osr(3.468 0x3ed1590) 828 bytes (queue has 50 ops and 41454 bytes) 2014-06-09 16:22:12.053975 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a78e0] 2014-06-09 16:22:12.053977 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.053976 7f868ef60700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053977 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.053980 7f8692f68700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.053980 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcfc0 (1 active) 2014-06-09 16:22:12.053985 7f868e75f700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053982 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ca20 (2 active) 2014-06-09 16:22:12.053989 7f8691f66700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.053995 7f868f761700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.054002 7f868c75b700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054001 7f86a5e11700 5 _do_op 0x30d978c0 seq 3868 osr(3.53d 0x3ed1600)/0x3ed1600 start 2014-06-09 16:22:12.054005 7f86a5e11700 10 _do_transaction on 0xbf88000 2014-06-09 16:22:12.054007 7f8691765700 2 waiting 51 > 50 ops || 42312 > 104857600 2014-06-09 16:22:12.054011 7f8692767700 2 waiting 51 > 50 ops || 928588 > 104857600 2014-06-09 16:22:12.054011 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.054015 7f868bf5a700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054019 7f86a5610700 10 _finish_op 0x30d96300 seq 3867 osr(2.182 0x2ed0990)/0x2ed0990 2014-06-09 16:22:12.054020 7f868a757700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054026 7f8689755700 2 waiting 51 > 50 ops || 42315 > 104857600 2014-06-09 16:22:12.054031 7f8689f56700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054036 7f868cf5c700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054041 7f868af58700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054047 7f868df5e700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054047 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.054053 7f8690763700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054057 7f868d75d700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054058 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3868.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.054061 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.054062 7f868ff62700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.054068 7f868b759700 2 waiting 51 > 50 ops || 42282 > 104857600 2014-06-09 16:22:12.054088 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d0a00] 2014-06-09 16:22:12.054094 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d0a00] 2014-06-09 16:22:12.054096 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.054097 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.054100 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e6c0 (1 active) 2014-06-09 16:22:12.054103 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300900 (2 active) 2014-06-09 16:22:12.054107 7f8690f64700 10 op_submit_finish 3916 2014-06-09 16:22:12.054120 7f868e75f700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.054113 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.220( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.220( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.054152 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.054158 7f8690f64700 5 queue_transactions existing osr(0.220 0x2bd6a70)/0x2bd6a70 2014-06-09 16:22:12.054161 7f8690f64700 2 waiting 51 > 50 ops || 42280 > 104857600 2014-06-09 16:22:12.054165 7f8692f68700 10 op_submit_start 3917 2014-06-09 16:22:12.054166 7f8692f68700 5 queue_transactions (trailing journal) 3917 2014-06-09 16:22:12.054167 7f8692f68700 5 queue_op 0x324e7380 seq 3917 osr(3.6f 0x3ed0280) 826 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.054170 7f8692f68700 10 op_submit_finish 3917 2014-06-09 16:22:12.054174 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:12.054188 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.054196 7f86a5e11700 10 _do_op 0x30d978c0 seq 3868 r = 0, finisher 0xcc832c0 0 2014-06-09 16:22:12.054197 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.054205 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfce860] 2014-06-09 16:22:12.054208 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfce860] 2014-06-09 16:22:12.054209 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.054210 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.054204 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.054213 7f86a5e11700 10 _finish_op 0x30d978c0 seq 3868 osr(3.53d 0x3ed1600)/0x3ed1600 2014-06-09 16:22:12.054218 7f8692f68700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.054225 7f868f761700 10 op_submit_start 3918 2014-06-09 16:22:12.054227 7f868f761700 5 queue_transactions (trailing journal) 3918 2014-06-09 16:22:12.054229 7f868f761700 5 queue_op 0xc64a240 seq 3918 osr(1.ab 0x384c6f0) 826 bytes (queue has 50 ops and 41450 bytes) 2014-06-09 16:22:12.054234 7f868c75b700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054239 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.054244 7f8692767700 2 waiting 51 > 50 ops || 928584 > 104857600 2014-06-09 16:22:12.054235 7f8692f68700 15 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:119 2014-06-09 16:22:12.054248 7f868bf5a700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054253 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.054255 7f868a757700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054260 7f8689755700 2 waiting 51 > 50 ops || 42311 > 104857600 2014-06-09 16:22:12.054264 7f8689f56700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054258 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.25d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.054269 7f868cf5c700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054273 7f868af58700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054279 7f868df5e700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054282 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc832c0] 2014-06-09 16:22:12.054284 7f8690763700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054286 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc832c0] 2014-06-09 16:22:12.054288 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.054289 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.054288 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.054289 7f868d75d700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054291 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ca20 (1 active) 2014-06-09 16:22:12.054293 7f868ff62700 2 waiting 51 > 50 ops || 42276 > 104857600 2014-06-09 16:22:12.054294 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52dc20 (2 active) 2014-06-09 16:22:12.054293 7f8692f68700 5 queue_transactions existing osr(0.25d 0x2bd6a00)/0x2bd6a00 2014-06-09 16:22:12.054297 7f86a5e11700 5 _do_op 0x30d97860 seq 3870 osr(0.5a2 0x2bd7af0)/0x2bd7af0 start 2014-06-09 16:22:12.054298 7f868b759700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054300 7f86a5e11700 10 _do_transaction on 0xc12c0c0 2014-06-09 16:22:12.054304 7f868f761700 10 op_submit_finish 3918 2014-06-09 16:22:12.054308 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.b9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.054321 7f868ef60700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054327 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.054330 7f8690f64700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054335 7f8691f66700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054339 7f8692f68700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054343 7f868e75f700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054344 7f86a5610700 5 _do_op 0xc34b800 seq 3869 osr(1.d1 0x384c9b0)/0x384c9b0 start 2014-06-09 16:22:12.054345 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.054347 7f86a5610700 10 _do_transaction on 0xc0d9140 2014-06-09 16:22:12.054350 7f868f761700 5 queue_transactions existing osr(0.b9 0x2bd6400)/0x2bd6400 2014-06-09 16:22:12.054352 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.054353 7f868f761700 2 waiting 51 > 50 ops || 42276 > 104857600 2014-06-09 16:22:12.054372 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.054387 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3870.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.054432 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.054443 7f86a5e11700 10 _do_op 0x30d97860 seq 3870 r = 0, finisher 0xbd6ca80 0 2014-06-09 16:22:12.054451 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd78bc0] 2014-06-09 16:22:12.054454 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd78bc0] 2014-06-09 16:22:12.054455 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.054456 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.054459 7f86a5e11700 10 _finish_op 0x30d97860 seq 3870 osr(0.5a2 0x2bd7af0)/0x2bd7af0 2014-06-09 16:22:12.054468 7f868c75b700 10 op_submit_start 3919 2014-06-09 16:22:12.054471 7f868c75b700 5 queue_transactions (trailing journal) 3919 2014-06-09 16:22:12.054470 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6ca80] 2014-06-09 16:22:12.054476 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.054473 7f868c75b700 5 queue_op 0x30032ae0 seq 3919 osr(3.1f0 0x3ed0990) 828 bytes (queue has 50 ops and 41450 bytes) 2014-06-09 16:22:12.054475 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6ca80] 2014-06-09 16:22:12.054491 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.054493 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.054492 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3869.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.054496 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.054506 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.054507 7f86a5610700 10 _do_op 0xc34b800 seq 3869 r = 0, finisher 0xbd723a0 0 2014-06-09 16:22:12.054512 7f8692767700 2 waiting 51 > 50 ops || 928584 > 104857600 2014-06-09 16:22:12.054517 7f868bf5a700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054521 7f868a757700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054521 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a71a0] 2014-06-09 16:22:12.054524 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a71a0] 2014-06-09 16:22:12.054525 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.054526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.054526 7f8689755700 2 waiting 51 > 50 ops || 42311 > 104857600 2014-06-09 16:22:12.054527 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52dc20 (1 active) 2014-06-09 16:22:12.054529 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f7a0 (2 active) 2014-06-09 16:22:12.054530 7f8689f56700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054532 7f86a5e11700 5 _do_op 0x15996180 seq 3871 osr(3.136 0x3ed07a0)/0x3ed07a0 start 2014-06-09 16:22:12.054534 7f86a5e11700 10 _do_transaction on 0xc048cc0 2014-06-09 16:22:12.054536 7f868c75b700 10 op_submit_finish 3919 2014-06-09 16:22:12.054538 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.054539 7f86a5610700 10 _finish_op 0xc34b800 seq 3869 osr(1.d1 0x384c9b0)/0x384c9b0 2014-06-09 16:22:12.054547 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300900 (1 active) 2014-06-09 16:22:12.054548 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd723a0] 2014-06-09 16:22:12.054549 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa5a0 (2 active) 2014-06-09 16:22:12.054551 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd723a0] 2014-06-09 16:22:12.054551 7f86a5610700 5 _do_op 0x15996f60 seq 3872 osr(0.644 0x2bd7bf0)/0x2bd7bf0 start 2014-06-09 16:22:12.054552 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.054553 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.054553 7f86a5610700 10 _do_transaction on 0xc20e300 2014-06-09 16:22:12.054540 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.730( v 214'648 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:12.054558 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.054560 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.5 activated and committed 2014-06-09 16:22:12.054573 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.054584 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.054604 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.054612 7f868c75b700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.054620 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3871.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.054625 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.054628 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1036 2014-06-09 16:22:12.054647 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.054651 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3ab( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.3ab( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.054679 7f86a5e11700 10 _do_op 0x15996180 seq 3871 r = 0, finisher 0x320753c0 0 2014-06-09 16:22:12.054686 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.054689 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x281b6540] 2014-06-09 16:22:12.054691 7f868c75b700 5 queue_transactions existing osr(0.3ab 0x2bd72f0)/0x2bd72f0 2014-06-09 16:22:12.054692 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x281b6540] 2014-06-09 16:22:12.054694 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.054694 7f868c75b700 10 op_submit_start 3920 2014-06-09 16:22:12.054695 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.054695 7f868c75b700 5 queue_transactions (trailing journal) 3920 2014-06-09 16:22:12.054697 7f868c75b700 5 queue_op 0x2fc8d680 seq 3920 osr(0.3ab 0x2bd72f0) 828 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.054698 7f86a5e11700 10 _finish_op 0x15996180 seq 3871 osr(3.136 0x3ed07a0)/0x3ed07a0 2014-06-09 16:22:12.054706 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x320753c0] 2014-06-09 16:22:12.054709 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x320753c0] 2014-06-09 16:22:12.054710 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.054711 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.054714 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f7a0 (1 active) 2014-06-09 16:22:12.054716 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652c60 (2 active) 2014-06-09 16:22:12.054719 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.054721 7f868c75b700 10 op_submit_finish 3920 2014-06-09 16:22:12.054727 7f868cf5c700 10 op_submit_start 3921 2014-06-09 16:22:12.054729 7f868cf5c700 5 queue_transactions (trailing journal) 3921 2014-06-09 16:22:12.054730 7f868cf5c700 5 queue_op 0xc931f80 seq 3921 osr(2.624 0x2ed1c50) 828 bytes (queue has 50 ops and 41452 bytes) 2014-06-09 16:22:12.054734 7f868cf5c700 10 op_submit_finish 3921 2014-06-09 16:22:12.054733 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3872.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.054725 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.30b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.054737 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.054737 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.c6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.054753 7f86a5610700 10 _do_op 0x15996f60 seq 3872 r = 0, finisher 0x31435aa0 0 2014-06-09 16:22:12.054759 7f86a5610700 10 _finish_op 0x15996f60 seq 3872 osr(0.644 0x2bd7bf0)/0x2bd7bf0 2014-06-09 16:22:12.054771 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.054776 7f868cf5c700 5 queue_transactions existing osr(2.c6 0x2ed0490)/0x2ed0490 2014-06-09 16:22:12.054778 7f868cf5c700 10 op_submit_start 3922 2014-06-09 16:22:12.054778 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.054779 7f868cf5c700 5 queue_transactions (trailing journal) 3922 2014-06-09 16:22:12.054781 7f868cf5c700 5 queue_op 0xc930660 seq 3922 osr(2.c6 0x2ed0490) 826 bytes (queue has 50 ops and 41450 bytes) 2014-06-09 16:22:12.054784 7f868c75b700 5 queue_transactions existing osr(2.30b 0x2ed0f50)/0x2ed0f50 2014-06-09 16:22:12.054786 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.054790 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31435d60] 2014-06-09 16:22:12.054792 7f868af58700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054792 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31435d60] 2014-06-09 16:22:12.054794 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.054795 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.054796 7f8690763700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054801 7f868d75d700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054805 7f868ff62700 2 waiting 51 > 50 ops || 42276 > 104857600 2014-06-09 16:22:12.054808 7f86a5e11700 5 _do_op 0x15997320 seq 3873 osr(1.2f8 0x384d0d0)/0x384d0d0 start 2014-06-09 16:22:12.054809 7f868b759700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054812 7f86a5e11700 10 _do_transaction on 0x2eaa0c0 2014-06-09 16:22:12.054817 7f868ef60700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054817 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.054822 7f8690f64700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054827 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31435aa0] 2014-06-09 16:22:12.054830 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31435aa0] 2014-06-09 16:22:12.054832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.054833 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.054835 7f8691f66700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054839 7f8692f68700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054844 7f868e75f700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.054844 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa5a0 (1 active) 2014-06-09 16:22:12.054846 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0c60 (2 active) 2014-06-09 16:22:12.054850 7f86a5610700 5 _do_op 0x15997e60 seq 3874 osr(2.431 0x2ed1590)/0x2ed1590 start 2014-06-09 16:22:12.054850 7f868f761700 2 waiting 51 > 50 ops || 42276 > 104857600 2014-06-09 16:22:12.054853 7f86a5610700 10 _do_transaction on 0x2d83ec0 2014-06-09 16:22:12.054857 7f868cf5c700 10 op_submit_finish 3922 2014-06-09 16:22:12.054858 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.054860 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.55d( v 214'593 lc 214'589 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.054877 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:12.054888 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.054900 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.054897 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.054911 7f868cf5c700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.054914 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3873.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.054919 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.054928 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:935 2014-06-09 16:22:12.054952 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.054957 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:12.054990 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.054994 7f868cf5c700 5 queue_transactions existing osr(3.381 0x3ed0fe0)/0x3ed0fe0 2014-06-09 16:22:12.054997 7f868cf5c700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.055009 7f8692767700 2 waiting 51 > 50 ops || 928584 > 104857600 2014-06-09 16:22:12.055015 7f868bf5a700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.055019 7f868a757700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.055024 7f8689755700 2 waiting 51 > 50 ops || 42311 > 104857600 2014-06-09 16:22:12.055028 7f8689f56700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.055034 7f868c75b700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.055042 7f868df5e700 2 waiting 51 > 50 ops || 42278 > 104857600 2014-06-09 16:22:12.055075 7f86a5e11700 10 _do_op 0x15997320 seq 3873 r = 0, finisher 0x324d1a20 0 2014-06-09 16:22:12.055086 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x281b7700] 2014-06-09 16:22:12.055090 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x281b7700] 2014-06-09 16:22:12.055092 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.055093 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.055096 7f86a5e11700 10 _finish_op 0x15997320 seq 3873 osr(1.2f8 0x384d0d0)/0x384d0d0 2014-06-09 16:22:12.055102 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652c60 (1 active) 2014-06-09 16:22:12.055104 7f8691765700 10 op_submit_start 3923 2014-06-09 16:22:12.055105 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652c60 (2 active) 2014-06-09 16:22:12.055106 7f8691765700 5 queue_transactions (trailing journal) 3923 2014-06-09 16:22:12.055107 7f86a5e11700 5 _do_op 0x159970e0 seq 3875 osr(1.2f8 0x384d0d0)/0x384d0d0 start 2014-06-09 16:22:12.055110 7f86a5e11700 10 _do_transaction on 0x390c840 2014-06-09 16:22:12.055108 7f8691765700 5 queue_op 0xc64a8a0 seq 3923 osr(3.4d 0x3ed0300) 858 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.055112 7f8691765700 10 op_submit_finish 3923 2014-06-09 16:22:12.055115 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.055113 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d1a20] 2014-06-09 16:22:12.055121 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d1a20] 2014-06-09 16:22:12.055123 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.055124 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.055115 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.055129 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.055138 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.055149 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.055147 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.055157 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.055169 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3875.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.055173 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.055165 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.055185 7f86a5e11700 10 _do_op 0x159970e0 seq 3875 r = 0, finisher 0x32534760 0 2014-06-09 16:22:12.055195 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314344e0] 2014-06-09 16:22:12.055197 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314344e0] 2014-06-09 16:22:12.055198 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.055199 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.055193 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:163 2014-06-09 16:22:12.055202 7f86a5e11700 10 _finish_op 0x159970e0 seq 3875 osr(1.2f8 0x384d0d0)/0x384d0d0 2014-06-09 16:22:12.055207 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652c60 (1 active) 2014-06-09 16:22:12.055209 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32534760] 2014-06-09 16:22:12.055203 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.055209 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5257a0 (2 active) 2014-06-09 16:22:12.055212 7f86a5e11700 5 _do_op 0x15997020 seq 3876 osr(2.3e8 0x2ed1310)/0x2ed1310 start 2014-06-09 16:22:12.055214 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32534760] 2014-06-09 16:22:12.055215 7f86a5e11700 10 _do_transaction on 0x34e9e00 2014-06-09 16:22:12.055215 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.055216 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.055212 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.055220 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.055222 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.055232 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.055227 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.765( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.055247 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3874.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.055251 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.055259 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.055264 7f8691765700 5 queue_transactions existing osr(0.765 0x384c360)/0x384c360 2014-06-09 16:22:12.055266 7f8691765700 10 op_submit_start 3924 2014-06-09 16:22:12.055267 7f8691765700 5 queue_transactions (trailing journal) 3924 2014-06-09 16:22:12.055269 7f8691765700 5 queue_op 0x309a3620 seq 3924 osr(0.765 0x384c360) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.055271 7f8691765700 10 op_submit_finish 3924 2014-06-09 16:22:12.055274 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.48d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.055305 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.055309 7f8691765700 5 queue_transactions existing osr(2.48d 0x2ed1470)/0x2ed1470 2014-06-09 16:22:12.055312 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055320 7f86a5610700 10 _do_op 0x15997e60 seq 3874 r = 0, finisher 0x31413c80 0 2014-06-09 16:22:12.055328 7f86a5610700 10 _finish_op 0x15997e60 seq 3874 osr(2.431 0x2ed1590)/0x2ed1590 2014-06-09 16:22:12.055329 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32535aa0] 2014-06-09 16:22:12.055332 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32535aa0] 2014-06-09 16:22:12.055333 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.055334 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.055337 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31413c80] 2014-06-09 16:22:12.055338 7f8691765700 10 op_submit_start 3925 2014-06-09 16:22:12.055340 7f8691765700 5 queue_transactions (trailing journal) 3925 2014-06-09 16:22:12.055341 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31413c80] 2014-06-09 16:22:12.055342 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.055341 7f8691765700 5 queue_op 0x288f9980 seq 3925 osr(2.48d 0x2ed1470) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.055343 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.055346 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055346 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0c60 (1 active) 2014-06-09 16:22:12.055349 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc120 (2 active) 2014-06-09 16:22:12.055351 7f868d75d700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055352 7f86a5610700 5 _do_op 0x159963c0 seq 3877 osr(0.2a2 0x2bd6c20)/0x2bd6c20 start 2014-06-09 16:22:12.055355 7f86a5610700 10 _do_transaction on 0xc12d740 2014-06-09 16:22:12.055356 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.055360 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.055360 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055363 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.055366 7f8691765700 10 op_submit_finish 3925 2014-06-09 16:22:12.055366 7f868ef60700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055371 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055376 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055370 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.578( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.578( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.055380 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055391 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055399 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3876.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.055403 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.055403 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.055408 7f8691765700 5 queue_transactions existing osr(0.578 0x2bd7800)/0x2bd7800 2014-06-09 16:22:12.055411 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055415 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.055420 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055425 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055429 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055433 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055438 7f8689755700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.055442 7f8689f56700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055447 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.055451 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055457 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.055465 7f86a5e11700 10 _do_op 0x15997020 seq 3876 r = 0, finisher 0x4eb9780 0 2014-06-09 16:22:12.055477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd76480] 2014-06-09 16:22:12.055480 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd76480] 2014-06-09 16:22:12.055481 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.055482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.055499 7f86a5e11700 10 _finish_op 0x15997020 seq 3876 osr(2.3e8 0x2ed1310)/0x2ed1310 2014-06-09 16:22:12.055508 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5257a0 (1 active) 2014-06-09 16:22:12.055509 7f8690763700 10 op_submit_start 3926 2014-06-09 16:22:12.055509 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x4eb9780] 2014-06-09 16:22:12.055511 7f8690763700 5 queue_transactions (trailing journal) 3926 2014-06-09 16:22:12.055511 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0c60 (2 active) 2014-06-09 16:22:12.055513 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x4eb9780] 2014-06-09 16:22:12.055513 7f86a5e11700 5 _do_op 0xc704180 seq 3878 osr(2.431 0x2ed1590)/0x2ed1590 start 2014-06-09 16:22:12.055515 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.055513 7f8690763700 5 queue_op 0x324b3740 seq 3926 osr(1.408 0x384d770) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.055516 7f86a5e11700 10 _do_transaction on 0xc2ef2c0 2014-06-09 16:22:12.055516 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.055516 7f8690763700 10 op_submit_finish 3926 2014-06-09 16:22:12.055521 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.055521 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.055519 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.411( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:12.055534 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3877.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.055538 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.055534 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.055544 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.055552 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.055562 7f86a5610700 10 _do_op 0x159963c0 seq 3877 r = 0, finisher 0x33a6a00 0 2014-06-09 16:22:12.055565 7f8690763700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.055568 7f86a5610700 10 _finish_op 0x159963c0 seq 3877 osr(0.2a2 0x2bd6c20)/0x2bd6c20 2014-06-09 16:22:12.055569 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd75500] 2014-06-09 16:22:12.055572 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd75500] 2014-06-09 16:22:12.055573 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.055574 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.055576 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a6a00] 2014-06-09 16:22:12.055579 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a6a00] 2014-06-09 16:22:12.055580 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.055581 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.055584 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc120 (1 active) 2014-06-09 16:22:12.055586 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652a20 (2 active) 2014-06-09 16:22:12.055588 7f86a5610700 5 _do_op 0xc704660 seq 3879 osr(0.5c3 0x2bd7a50)/0x2bd7a50 start 2014-06-09 16:22:12.055590 7f86a5610700 10 _do_transaction on 0xc326d80 2014-06-09 16:22:12.055585 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.411( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:12.055595 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.055602 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.058813 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.1ae( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:12.058835 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.058845 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.058853 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.058871 7f8690763700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.058892 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.1ae( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:12.058910 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.058915 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:12.058955 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.058961 7f8690763700 5 queue_transactions existing osr(3.271 0x3ed0bc0)/0x3ed0bc0 2014-06-09 16:22:12.055608 7f868d75d700 10 op_submit_start 3927 2014-06-09 16:22:12.058971 7f868d75d700 5 queue_transactions (trailing journal) 3927 2014-06-09 16:22:12.058974 7f868d75d700 5 queue_op 0x30d97440 seq 3927 osr(2.38a 0x2ed10f0) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.058978 7f868d75d700 10 op_submit_finish 3927 2014-06-09 16:22:12.058982 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.67( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.059020 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.059025 7f868d75d700 5 queue_transactions existing osr(1.67 0x384c5b0)/0x384c5b0 2014-06-09 16:22:12.055611 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.055649 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.059038 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.059059 7f86a6b95700 5 -- op tracker -- , seq: 3617, time: 2014-06-09 16:22:12.052735, event: header_read, op: osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.059065 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3878.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.059071 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.059075 7f86a6b95700 5 -- op tracker -- , seq: 3617, time: 2014-06-09 16:22:12.052738, event: throttled, op: osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.059088 7f86a6b95700 5 -- op tracker -- , seq: 3617, time: 2014-06-09 16:22:12.056729, event: all_read, op: osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.059093 7f86a5e11700 10 _do_op 0xc704180 seq 3878 r = 0, finisher 0xc40c320 0 2014-06-09 16:22:12.059101 7f86a6b95700 5 -- op tracker -- , seq: 3617, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.059105 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30d60160] 2014-06-09 16:22:12.059108 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30d60160] 2014-06-09 16:22:12.059110 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.059111 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.059110 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.059114 7f86a5e11700 10 _finish_op 0xc704180 seq 3878 osr(2.431 0x2ed1590)/0x2ed1590 2014-06-09 16:22:12.059126 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc40c320] 2014-06-09 16:22:12.059130 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc40c320] 2014-06-09 16:22:12.059132 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.059133 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.059135 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0c60 (1 active) 2014-06-09 16:22:12.059138 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f8c0 (2 active) 2014-06-09 16:22:12.059141 7f86a5e11700 5 _do_op 0xc931860 seq 3880 osr(3.7ae 0xb7f01d0)/0xb7f01d0 start 2014-06-09 16:22:12.059135 7f86a6b95700 15 osd.26 243 enqueue_op 0xdfa6300 prio 63 cost 516096 latency 0.006400 osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.059144 7f86a5e11700 10 _do_transaction on 0x34e9440 2014-06-09 16:22:12.059150 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.059318 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.059329 7f868ff62700 10 op_submit_start 3928 2014-06-09 16:22:12.059332 7f868ff62700 5 queue_transactions (trailing journal) 3928 2014-06-09 16:22:12.059333 7f868ff62700 5 queue_op 0x30a3e00 seq 3928 osr(1.c6 0x384c610) 826 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.059333 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3880.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.059337 7f868ff62700 10 op_submit_finish 3928 2014-06-09 16:22:12.059337 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.059341 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.5f2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:12.059352 7f86a5e11700 10 _do_op 0xc931860 seq 3880 r = 0, finisher 0x318c31e0 0 2014-06-09 16:22:12.059362 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c33e0] 2014-06-09 16:22:12.059365 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c33e0] 2014-06-09 16:22:12.059366 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.059367 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.059370 7f86a5e11700 10 _finish_op 0xc931860 seq 3880 osr(3.7ae 0xb7f01d0)/0xb7f01d0 2014-06-09 16:22:12.059378 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.059378 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c31e0] 2014-06-09 16:22:12.059385 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c31e0] 2014-06-09 16:22:12.059385 7f868ff62700 5 queue_transactions existing osr(2.5f2 0x2ed1df0)/0x2ed1df0 2014-06-09 16:22:12.059387 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.059388 7f868ff62700 10 op_submit_start 3929 2014-06-09 16:22:12.059388 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.059389 7f868ff62700 5 queue_transactions (trailing journal) 3929 2014-06-09 16:22:12.059391 7f868ff62700 5 queue_op 0x30a2d80 seq 3929 osr(2.5f2 0x2ed1df0) 828 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.059391 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f8c0 (1 active) 2014-06-09 16:22:12.059394 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fdb00 (2 active) 2014-06-09 16:22:12.059397 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.059398 7f86a5e11700 5 _do_op 0x30a3b00 seq 3881 osr(0.151 0x2bd65e0)/0x2bd65e0 start 2014-06-09 16:22:12.059401 7f86a5e11700 10 _do_transaction on 0x2facd80 2014-06-09 16:22:12.059404 7f868ff62700 10 op_submit_finish 3929 2014-06-09 16:22:12.059406 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.059407 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.3a9( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:12.059483 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.059488 7f868ff62700 5 queue_transactions existing osr(0.3a9 0x2bd7310)/0x2bd7310 2014-06-09 16:22:12.059491 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.059492 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.059507 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3881.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.059511 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.059521 7f86a5e11700 10 _do_op 0x30a3b00 seq 3881 r = 0, finisher 0x18743560 0 2014-06-09 16:22:12.059531 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205cfc0] 2014-06-09 16:22:12.059533 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205cfc0] 2014-06-09 16:22:12.059538 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.059539 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.059542 7f86a5e11700 10 _finish_op 0x30a3b00 seq 3881 osr(0.151 0x2bd65e0)/0x2bd65e0 2014-06-09 16:22:12.059632 7f868ef60700 10 op_submit_start 3930 2014-06-09 16:22:12.059636 7f868ef60700 5 queue_transactions (trailing journal) 3930 2014-06-09 16:22:12.059638 7f868ef60700 5 queue_op 0x32503e00 seq 3930 osr(2.487 0x2ed14b0) 828 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.059640 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x18743560] 2014-06-09 16:22:12.059644 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.059646 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x18743560] 2014-06-09 16:22:12.059648 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.059649 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.059650 7f8692f68700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.059654 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fdb00 (1 active) 2014-06-09 16:22:12.059656 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0360 (2 active) 2014-06-09 16:22:12.059660 7f86a5e11700 5 _do_op 0xc34b440 seq 3882 osr(1.6f 0x384c570)/0x384c570 start 2014-06-09 16:22:12.059663 7f86a5e11700 10 _do_transaction on 0xc1f6780 2014-06-09 16:22:12.059668 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.059750 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.059764 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3882.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.059768 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.059792 7f86a5e11700 10 _do_op 0xc34b440 seq 3882 r = 0, finisher 0xcc83300 0 2014-06-09 16:22:12.059860 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a0025a0] 2014-06-09 16:22:12.059863 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a0025a0] 2014-06-09 16:22:12.059865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.059866 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.059870 7f86a5e11700 10 _finish_op 0xc34b440 seq 3882 osr(1.6f 0x384c570)/0x384c570 2014-06-09 16:22:12.059882 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc83300] 2014-06-09 16:22:12.059885 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc83300] 2014-06-09 16:22:12.059887 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.059888 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.059891 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0360 (1 active) 2014-06-09 16:22:12.059893 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cea0 (2 active) 2014-06-09 16:22:12.059895 7f86a5e11700 5 _do_op 0xc930360 seq 3883 osr(0.5e 0x2bd63b0)/0x2bd63b0 start 2014-06-09 16:22:12.059898 7f86a5e11700 10 _do_transaction on 0xc28bb00 2014-06-09 16:22:12.059903 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.060003 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.060004 7f86a6b95700 5 -- op tracker -- , seq: 3618, time: 2014-06-09 16:22:12.059164, event: header_read, op: osd_op(client.4542.0:34422 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3334144~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.060015 7f86a6b95700 5 -- op tracker -- , seq: 3618, time: 2014-06-09 16:22:12.059166, event: throttled, op: osd_op(client.4542.0:34422 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3334144~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.060019 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3883.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.060023 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.060026 7f86a6b95700 5 -- op tracker -- , seq: 3618, time: 2014-06-09 16:22:12.059976, event: all_read, op: osd_op(client.4542.0:34422 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3334144~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.060034 7f86a5e11700 10 _do_op 0xc930360 seq 3883 r = 0, finisher 0xbd692a0 0 2014-06-09 16:22:12.060036 7f86a6b95700 5 -- op tracker -- , seq: 3618, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34422 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3334144~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.060042 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.060059 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c3ca0] 2014-06-09 16:22:12.060062 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c3ca0] 2014-06-09 16:22:12.060063 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.060065 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.060060 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6c800 prio 63 cost 516096 latency 0.000896 osd_op(client.4542.0:34422 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3334144~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.060112 7f86a5e11700 10 _finish_op 0xc930360 seq 3883 osr(0.5e 0x2bd63b0)/0x2bd63b0 2014-06-09 16:22:12.060121 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd692a0] 2014-06-09 16:22:12.060128 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd692a0] 2014-06-09 16:22:12.060130 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.060131 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.060133 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cea0 (1 active) 2014-06-09 16:22:12.060136 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3aea0 (2 active) 2014-06-09 16:22:12.060138 7f86a5e11700 5 _do_op 0xc930c00 seq 3884 osr(1.1fa 0x384cd50)/0x384cd50 start 2014-06-09 16:22:12.060141 7f86a5e11700 10 _do_transaction on 0xbdbd2c0 2014-06-09 16:22:12.060146 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.060241 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.060256 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3884.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.060259 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.060271 7f86a5e11700 10 _do_op 0xc930c00 seq 3884 r = 0, finisher 0xbd7aa40 0 2014-06-09 16:22:12.060281 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x20050340] 2014-06-09 16:22:12.060283 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x20050340] 2014-06-09 16:22:12.060285 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.060285 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.060288 7f86a5e11700 10 _finish_op 0xc930c00 seq 3884 osr(1.1fa 0x384cd50)/0x384cd50 2014-06-09 16:22:12.060340 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7aa40] 2014-06-09 16:22:12.060346 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7aa40] 2014-06-09 16:22:12.060348 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.060349 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.060352 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3aea0 (1 active) 2014-06-09 16:22:12.060354 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f7a0 (2 active) 2014-06-09 16:22:12.060356 7f86a5e11700 5 _do_op 0x324f46c0 seq 3885 osr(2.501 0x2ed1690)/0x2ed1690 start 2014-06-09 16:22:12.060359 7f86a5e11700 10 _do_transaction on 0xc327800 2014-06-09 16:22:12.060364 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.060399 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.060413 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3885.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.060417 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.060472 7f86a5e11700 10 _do_op 0x324f46c0 seq 3885 r = 0, finisher 0xbd5ce60 0 2014-06-09 16:22:12.060482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050a560] 2014-06-09 16:22:12.060485 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050a560] 2014-06-09 16:22:12.060486 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.060487 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.060491 7f86a5e11700 10 _finish_op 0x324f46c0 seq 3885 osr(2.501 0x2ed1690)/0x2ed1690 2014-06-09 16:22:12.060491 7f86a6b95700 5 -- op tracker -- , seq: 3619, time: 2014-06-09 16:22:12.060081, event: header_read, op: osd_op(client.4542.0:34423 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3850240~344064] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.060498 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ce60] 2014-06-09 16:22:12.060503 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ce60] 2014-06-09 16:22:12.060504 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.060505 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.060502 7f86a6b95700 5 -- op tracker -- , seq: 3619, time: 2014-06-09 16:22:12.060082, event: throttled, op: osd_op(client.4542.0:34423 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3850240~344064] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.060508 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f7a0 (1 active) 2014-06-09 16:22:12.060510 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525200 (2 active) 2014-06-09 16:22:12.060512 7f86a5e11700 5 _do_op 0xbf7fb00 seq 3886 osr(0.25b 0x2bd69f0)/0x2bd69f0 start 2014-06-09 16:22:12.060514 7f86a5e11700 10 _do_transaction on 0xc291380 2014-06-09 16:22:12.060514 7f86a6b95700 5 -- op tracker -- , seq: 3619, time: 2014-06-09 16:22:12.060450, event: all_read, op: osd_op(client.4542.0:34423 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3850240~344064] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.060520 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.060524 7f86a6b95700 5 -- op tracker -- , seq: 3619, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34423 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3850240~344064] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.060530 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.060546 7f86a6b95700 15 osd.26 243 enqueue_op 0x306af00 prio 63 cost 344064 latency 0.000465 osd_op(client.4542.0:34423 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 3850240~344064] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.060621 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.060635 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3886.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.060639 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.060655 7f86a5e11700 10 _do_op 0xbf7fb00 seq 3886 r = 0, finisher 0x31434320 0 2014-06-09 16:22:12.060665 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31412fe0] 2014-06-09 16:22:12.060668 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31412fe0] 2014-06-09 16:22:12.060680 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.060681 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.060695 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060703 7f8689755700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.060706 7f86a5e11700 10 _finish_op 0xbf7fb00 seq 3886 osr(0.25b 0x2bd69f0)/0x2bd69f0 2014-06-09 16:22:12.060733 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.060741 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060753 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31434320] 2014-06-09 16:22:12.060757 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31434320] 2014-06-09 16:22:12.060759 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.060760 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.060763 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525200 (1 active) 2014-06-09 16:22:12.060765 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9237a0 (2 active) 2014-06-09 16:22:12.060767 7f86a5e11700 5 _do_op 0x324e7680 seq 3887 osr(0.1a4 0x2bd6710)/0x2bd6710 start 2014-06-09 16:22:12.060770 7f86a5e11700 10 _do_transaction on 0x331eb40 2014-06-09 16:22:12.060769 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060775 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.060777 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.060782 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060787 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060792 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060799 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060806 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060813 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060816 7f868ef60700 10 op_submit_finish 3930 2014-06-09 16:22:12.060819 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060822 7f868f761700 10 op_submit_start 3931 2014-06-09 16:22:12.060824 7f868f761700 5 queue_transactions (trailing journal) 3931 2014-06-09 16:22:12.060826 7f868f761700 5 queue_op 0x2fcae2a0 seq 3931 osr(0.b9 0x2bd6400) 826 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.060830 7f868f761700 10 op_submit_finish 3931 2014-06-09 16:22:12.060824 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MLogRec from 2 2014-06-09 16:22:12.060844 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] proc_replica_log for osd.2: 3.18b( v 213'575 (0'0,213'575] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) log((0'0,213'575], crt=0'0) missing(0) 2014-06-09 16:22:12.060859 7f868ef60700 10 proc_replica_log for osd.2: 3.18b( v 213'575 (0'0,213'575] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) log((0'0,213'575], crt=0'0) missing(0) 2014-06-09 16:22:12.060850 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.5b6( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.5b6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.060865 7f868ef60700 20 merge_log cut point (usually last shared) is 213'575 (213'574) modify a1b8218b/rbd_data.115b6b8b4567.00000000000018a7/head//3 by client.4539.0:34382 2014-06-09 10:17:58.939966 2014-06-09 16:22:12.060894 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.060893 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.060901 7f868f761700 5 queue_transactions existing osr(2.5b6 0x2ed1b90)/0x2ed1b90 2014-06-09 16:22:12.060904 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060906 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3887.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.060910 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.060909 7f8689f56700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060915 7f868cf5c700 10 op_submit_start 3932 2014-06-09 16:22:12.060915 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060917 7f868cf5c700 5 queue_transactions (trailing journal) 3932 2014-06-09 16:22:12.060918 7f868cf5c700 5 queue_op 0xbf7f440 seq 3932 osr(3.381 0x3ed0fe0) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.060922 7f868cf5c700 10 op_submit_finish 3932 2014-06-09 16:22:12.060928 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.624( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.624( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.060959 7f86a5e11700 10 _do_op 0x324e7680 seq 3887 r = 0, finisher 0x3042b7a0 0 2014-06-09 16:22:12.060968 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x19941fe0] 2014-06-09 16:22:12.060970 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x19941fe0] 2014-06-09 16:22:12.060971 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.060972 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.060975 7f86a5e11700 10 _finish_op 0x324e7680 seq 3887 osr(0.1a4 0x2bd6710)/0x2bd6710 2014-06-09 16:22:12.060984 7f868c75b700 10 op_submit_start 3933 2014-06-09 16:22:12.060985 7f868c75b700 5 queue_transactions (trailing journal) 3933 2014-06-09 16:22:12.060984 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.060986 7f868c75b700 5 queue_op 0x2ea52780 seq 3933 osr(2.30b 0x2ed0f50) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.060990 7f868cf5c700 5 queue_transactions existing osr(2.624 0x2ed1c50)/0x2ed1c50 2014-06-09 16:22:12.060991 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060993 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.060995 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.060999 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061003 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b7a0] 2014-06-09 16:22:12.061004 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061007 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b7a0] 2014-06-09 16:22:12.061008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.061009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.061009 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061011 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9237a0 (1 active) 2014-06-09 16:22:12.061014 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a7e0 (2 active) 2014-06-09 16:22:12.061013 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061018 7f868c75b700 10 op_submit_finish 3933 2014-06-09 16:22:12.061019 7f8689755700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.061024 7f868bf5a700 10 op_submit_start 3934 2014-06-09 16:22:12.061026 7f868bf5a700 5 queue_transactions (trailing journal) 3934 2014-06-09 16:22:12.061028 7f868bf5a700 5 queue_op 0xc704d80 seq 3934 osr(2.3e8 0x2ed1310) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.061031 7f868bf5a700 10 op_submit_finish 3934 2014-06-09 16:22:12.061024 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.5f9( v 214'611 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:12.061036 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.2a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.061043 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.3 activated and committed 2014-06-09 16:22:12.061069 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.061074 7f868bf5a700 5 queue_transactions existing osr(1.2a3 0x384d1f0)/0x384d1f0 2014-06-09 16:22:12.061076 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061070 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.061080 7f8691765700 10 op_submit_start 3935 2014-06-09 16:22:12.061082 7f8691765700 5 queue_transactions (trailing journal) 3935 2014-06-09 16:22:12.061080 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.061084 7f8691765700 5 queue_op 0x30d96c60 seq 3935 osr(0.578 0x2bd7800) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.061092 7f8691765700 10 op_submit_finish 3935 2014-06-09 16:22:12.061097 7f868c75b700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.061097 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.71b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.71b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.061116 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1008 2014-06-09 16:22:12.061130 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.061133 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.061135 7f8691765700 5 queue_transactions existing osr(0.71b 0x384c050)/0x384c050 2014-06-09 16:22:12.061139 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061143 7f868af58700 10 op_submit_start 3936 2014-06-09 16:22:12.061144 7f868af58700 5 queue_transactions (trailing journal) 3936 2014-06-09 16:22:12.061137 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.709( v 214'577 lc 214'576 (0'0,214'577] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.061146 7f868af58700 5 queue_op 0x324c1320 seq 3936 osr(0.3a6 0x2bd7330) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.061149 7f868af58700 10 op_submit_finish 3936 2014-06-09 16:22:12.061150 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:12.061153 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 243/191 242/242/220) 2014-06-09 16:22:12.061160 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.061167 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.061188 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.061189 7f868c75b700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.061193 7f868af58700 5 queue_transactions existing osr(3.1fc 0x3ed0930)/0x3ed0930 2014-06-09 16:22:12.061195 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061203 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:915 2014-06-09 16:22:12.061217 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.061221 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.350( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.350( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.061257 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.061262 7f868c75b700 5 queue_transactions existing osr(2.350 0x2ed0e50)/0x2ed0e50 2014-06-09 16:22:12.061265 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061270 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061266 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] peer osd.2 now 3.18b( v 213'575 (0'0,213'575] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) missing(0) 2014-06-09 16:22:12.061284 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Got last missing, don't need missing posting CheckRepops 2014-06-09 16:22:12.061295 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061304 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061298 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering/GetMissing 0.851170 4 0.000347 2014-06-09 16:22:12.061312 7f8689f56700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061311 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] state: Leaving Peering 2014-06-09 16:22:12.061322 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 peering] exit Started/Primary/Peering 1.381152 0 0.000000 2014-06-09 16:22:12.061337 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active 2014-06-09 16:22:12.061360 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=191 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: In Active, about to call activate 2014-06-09 16:22:12.061374 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - purged_snaps [] cached_removed_snaps [] 2014-06-09 16:22:12.061387 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061384 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - snap_trimq [] 2014-06-09 16:22:12.061393 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.061396 7f86a5e11700 5 _do_op 0x30d97800 seq 3888 osr(3.778 0x3ed1df0)/0x3ed1df0 start 2014-06-09 16:22:12.061399 7f86a5e11700 10 _do_transaction on 0xbf39980 2014-06-09 16:22:12.061392 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate - no missing, moving last_complete 214'581 -> 214'581 2014-06-09 16:22:12.061404 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.061433 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.18b( v 213'575 (0'0,213'575] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) 2014-06-09 16:22:12.061449 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.061462 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3888.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.061466 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.061457 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 sending log((213'575,214'581], crt=0'0) 2014-06-09 16:22:12.061469 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.061480 7f86a5e11700 10 _do_op 0x30d97800 seq 3888 r = 0, finisher 0x13965700 0 2014-06-09 16:22:12.061485 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate peer osd.2 3.18b( v 214'581 lc 213'575 (0'0,214'581] local-les=191 n=113 ec=118 les/c 191/191 242/242/242) missing missing(1) 2014-06-09 16:22:12.061498 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:12.061508 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 96a3018b/rbd_data.11776b8b4567.0000000000001767/head//3 214'581 is on osd.26 2014-06-09 16:22:12.061527 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061fa40] 2014-06-09 16:22:12.061520 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {96a3018b/rbd_data.11776b8b4567.0000000000001767/head//3=214'581} 2014-06-09 16:22:12.061530 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061fa40] 2014-06-09 16:22:12.061531 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.061532 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.061535 7f86a5e11700 10 _finish_op 0x30d97800 seq 3888 osr(3.778 0x3ed1df0)/0x3ed1df0 2014-06-09 16:22:12.061531 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] activate: adding 2 as a source 2014-06-09 16:22:12.061540 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] search_for_missing 96a3018b/rbd_data.11776b8b4567.0000000000001767/head//3 214'581 also missing on osd.2 2014-06-09 16:22:12.061552 7f868cf5c700 10 op_submit_start 3937 2014-06-09 16:22:12.061555 7f868cf5c700 5 queue_transactions (trailing journal) 3937 2014-06-09 16:22:12.061556 7f868cf5c700 5 queue_op 0x30033a40 seq 3937 osr(2.624 0x2ed1c50) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.061551 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] needs_recovery_map missing {96a3018b/rbd_data.11776b8b4567.0000000000001767/head//3=214'581} 2014-06-09 16:22:12.061561 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound 2014-06-09 16:22:12.061574 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13965700] 2014-06-09 16:22:12.061577 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13965700] 2014-06-09 16:22:12.061570 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _calc_past_interval_range: already have past intervals back to 191 2014-06-09 16:22:12.061579 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.061580 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.061583 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.061581 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] build_might_have_unfound: built 2 2014-06-09 16:22:12.061590 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061596 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061590 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: Activate Finished 2014-06-09 16:22:12.061601 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061606 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061600 7f868ef60700 5 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] enter Started/Primary/Active/Activating 2014-06-09 16:22:12.061611 7f8689755700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.061615 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061619 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061623 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061642 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061647 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061651 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061656 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a7e0 (1 active) 2014-06-09 16:22:12.061658 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3019e0 (2 active) 2014-06-09 16:22:12.061662 7f86a5e11700 5 _do_op 0x30d977a0 seq 3889 osr(0.652 0x2bd7b90)/0x2bd7b90 start 2014-06-09 16:22:12.061665 7f86a5e11700 10 _do_transaction on 0xc05e3c0 2014-06-09 16:22:12.061670 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.061672 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.061657 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061676 7f868ef60700 5 queue_transactions existing osr(3.18b 0x3ed06a0)/0x3ed06a0 2014-06-09 16:22:12.061679 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.061682 7f868cf5c700 10 op_submit_finish 3937 2014-06-09 16:22:12.061685 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061691 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.061695 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061686 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.2d9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.061717 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.061722 7f868cf5c700 5 queue_transactions existing osr(2.2d9 0x2ed0c30)/0x2ed0c30 2014-06-09 16:22:12.061724 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061728 7f8689f56700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061728 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.061734 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.061743 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3889.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.061747 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.061758 7f86a5e11700 10 _do_op 0x30d977a0 seq 3889 r = 0, finisher 0xbebf9e0 0 2014-06-09 16:22:12.061768 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x19940560] 2014-06-09 16:22:12.061770 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x19940560] 2014-06-09 16:22:12.061772 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.061773 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.061775 7f86a5e11700 10 _finish_op 0x30d977a0 seq 3889 osr(0.652 0x2bd7b90)/0x2bd7b90 2014-06-09 16:22:12.061788 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbebf9e0] 2014-06-09 16:22:12.061791 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbebf9e0] 2014-06-09 16:22:12.061793 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.061794 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.061796 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3019e0 (1 active) 2014-06-09 16:22:12.061862 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9227e0 (2 active) 2014-06-09 16:22:12.061865 7f86a5e11700 5 _do_op 0x30d976e0 seq 3890 osr(0.4a7 0x2bd7370)/0x2bd7370 start 2014-06-09 16:22:12.061868 7f86a5e11700 10 _do_transaction on 0xc2c52c0 2014-06-09 16:22:12.061873 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.061909 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.061951 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3890.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.061955 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.061971 7f86a5e11700 10 _do_op 0x30d976e0 seq 3890 r = 0, finisher 0x189aafe0 0 2014-06-09 16:22:12.061981 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509de0] 2014-06-09 16:22:12.061983 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509de0] 2014-06-09 16:22:12.061984 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.061986 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.061989 7f86a5e11700 10 _finish_op 0x30d976e0 seq 3890 osr(0.4a7 0x2bd7370)/0x2bd7370 2014-06-09 16:22:12.061997 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aafe0] 2014-06-09 16:22:12.062000 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aafe0] 2014-06-09 16:22:12.062002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.062003 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.062005 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9227e0 (1 active) 2014-06-09 16:22:12.062007 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e7e0 (2 active) 2014-06-09 16:22:12.062009 7f86a5e11700 5 _do_op 0x30d97680 seq 3891 osr(2.463 0x2ed1510)/0x2ed1510 start 2014-06-09 16:22:12.062012 7f86a5e11700 10 _do_transaction on 0xc14dd40 2014-06-09 16:22:12.062016 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.062103 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.062117 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3891.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.062121 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.062139 7f86a5e11700 10 _do_op 0x30d97680 seq 3891 r = 0, finisher 0x30629320 0 2014-06-09 16:22:12.062149 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd64700] 2014-06-09 16:22:12.062151 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd64700] 2014-06-09 16:22:12.062152 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.062153 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.062156 7f86a5e11700 10 _finish_op 0x30d97680 seq 3891 osr(2.463 0x2ed1510)/0x2ed1510 2014-06-09 16:22:12.062164 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629320] 2014-06-09 16:22:12.062168 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629320] 2014-06-09 16:22:12.062169 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.062170 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.062318 7f86a6b95700 5 -- op tracker -- , seq: 3620, time: 2014-06-09 16:22:12.060566, event: header_read, op: osd_op(client.4542.0:34474 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~745472] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.062330 7f86a6b95700 5 -- op tracker -- , seq: 3620, time: 2014-06-09 16:22:12.060567, event: throttled, op: osd_op(client.4542.0:34474 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~745472] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.062340 7f86a6b95700 5 -- op tracker -- , seq: 3620, time: 2014-06-09 16:22:12.062290, event: all_read, op: osd_op(client.4542.0:34474 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~745472] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.062351 7f86a6b95700 5 -- op tracker -- , seq: 3620, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34474 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~745472] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.062358 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.062381 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6cb00 prio 63 cost 745472 latency 0.001815 osd_op(client.4542.0:34474 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 0~745472] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.062586 7f868b759700 10 op_submit_start 3938 2014-06-09 16:22:12.062589 7f868b759700 5 queue_transactions (trailing journal) 3938 2014-06-09 16:22:12.062591 7f868b759700 5 queue_op 0xbf7e600 seq 3938 osr(2.52e 0x2ed1600) 828 bytes (queue has 48 ops and 39824 bytes) 2014-06-09 16:22:12.062605 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.062610 7f8689755700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.062615 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.062621 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.062626 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.063734 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.063736 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.063738 7f869af78700 20 osd.26 243 _dispatch 0x30de5a40 pg_info(1 pgs e243:3.18b) v4 2014-06-09 16:22:12.063746 7f869af78700 5 -- op tracker -- , seq: 3621, time: 2014-06-09 16:22:12.063682, event: header_read, op: pg_info(1 pgs e243:3.18b) 2014-06-09 16:22:12.063755 7f869af78700 5 -- op tracker -- , seq: 3621, time: 2014-06-09 16:22:12.063683, event: throttled, op: pg_info(1 pgs e243:3.18b) 2014-06-09 16:22:12.063783 7f869af78700 5 -- op tracker -- , seq: 3621, time: 2014-06-09 16:22:12.063700, event: all_read, op: pg_info(1 pgs e243:3.18b) 2014-06-09 16:22:12.063790 7f869af78700 5 -- op tracker -- , seq: 3621, time: 2014-06-09 16:22:12.063732, event: dispatched, op: pg_info(1 pgs e243:3.18b) 2014-06-09 16:22:12.063799 7f869af78700 5 -- op tracker -- , seq: 3621, time: 2014-06-09 16:22:12.063797, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.18b) 2014-06-09 16:22:12.063805 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.18b) v4 from osd.2 2014-06-09 16:22:12.063808 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30de5a40 2014-06-09 16:22:12.063815 7f869af78700 5 -- op tracker -- , seq: 3621, time: 2014-06-09 16:22:12.063813, event: started, op: pg_info(1 pgs e243:3.18b) 2014-06-09 16:22:12.065101 7f8682de8700 5 -- op tracker -- , seq: 3622, time: 2014-06-09 16:22:12.049944, event: header_read, op: MOSDPGPush(3.450 243 [PushOp(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3, version: 221'627, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3@221'627, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.065116 7f8682de8700 5 -- op tracker -- , seq: 3622, time: 2014-06-09 16:22:12.049945, event: throttled, op: MOSDPGPush(3.450 243 [PushOp(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3, version: 221'627, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3@221'627, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.065129 7f8682de8700 5 -- op tracker -- , seq: 3622, time: 2014-06-09 16:22:12.065062, event: all_read, op: MOSDPGPush(3.450 243 [PushOp(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3, version: 221'627, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3@221'627, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.065140 7f8682de8700 5 -- op tracker -- , seq: 3622, time: 0.000000, event: dispatched, op: MOSDPGPush(3.450 243 [PushOp(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3, version: 221'627, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3@221'627, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.065146 7f8682de8700 10 osd.26 243 handle_replica_op MOSDPGPush(3.450 243 [PushOp(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3, version: 221'627, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3@221'627, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 epoch 243 2014-06-09 16:22:12.065154 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:12.065162 7f8682de8700 15 osd.26 243 enqueue_op 0x32827c00 prio 63 cost 4195304 latency 0.015218 MOSDPGPush(3.450 243 [PushOp(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3, version: 221'627, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4c732450/rbd_data.117f6b8b4567.0000000000001567/head//3@221'627, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 2014-06-09 16:22:12.065204 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065225 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065231 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065239 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e7e0 (1 active) 2014-06-09 16:22:12.065239 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065243 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd7a0 (2 active) 2014-06-09 16:22:12.065249 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.065249 7f86a5e11700 5 _do_op 0x30d97620 seq 3892 osr(2.759 0x3ed00b0)/0x3ed00b0 start 2014-06-09 16:22:12.065252 7f868b759700 10 op_submit_finish 3938 2014-06-09 16:22:12.065256 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065263 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.065269 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065254 7f86a5e11700 10 _do_transaction on 0xc317500 2014-06-09 16:22:12.065280 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.065278 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.9d( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.065323 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.065328 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.065337 7f868b759700 5 queue_transactions existing osr(0.9d 0x2bd6490)/0x2bd6490 2014-06-09 16:22:12.065340 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.065345 7f868ff62700 10 op_submit_start 3939 2014-06-09 16:22:12.065344 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3879.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.065347 7f868ff62700 5 queue_transactions (trailing journal) 3939 2014-06-09 16:22:12.065349 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.065349 7f868ff62700 5 queue_op 0x30a2f60 seq 3939 osr(0.3a9 0x2bd7310) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.065353 7f868ff62700 10 op_submit_finish 3939 2014-06-09 16:22:12.065368 7f86a5610700 10 _do_op 0xc704660 seq 3879 r = 0, finisher 0xbd7d740 0 2014-06-09 16:22:12.065359 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.37a( v 213'559 (0'0,213'559] local-les=243 n=111 ec=118 les/c 243/243 242/242/223) 2014-06-09 16:22:12.065376 7f86a5610700 10 _finish_op 0xc704660 seq 3879 osr(0.5c3 0x2bd7a50)/0x2bd7a50 2014-06-09 16:22:12.065378 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5faa0] 2014-06-09 16:22:12.065382 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5faa0] 2014-06-09 16:22:12.065383 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.065385 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.065390 7f8689f56700 10 op_submit_start 3940 2014-06-09 16:22:12.065392 7f8689f56700 5 queue_transactions (trailing journal) 3940 2014-06-09 16:22:12.065393 7f8689f56700 5 queue_op 0xcca7e60 seq 3940 osr(0.219 0x2bd6ad0) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.065399 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065405 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.065407 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652a20 (1 active) 2014-06-09 16:22:12.065409 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922900 (2 active) 2014-06-09 16:22:12.065413 7f8689f56700 10 op_submit_finish 3940 2014-06-09 16:22:12.065413 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7d740] 2014-06-09 16:22:12.065420 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7d740] 2014-06-09 16:22:12.065421 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.065421 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.065422 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.065427 7f868ff62700 5 queue_transactions existing osr(3.37a 0x3ed1040)/0x3ed1040 2014-06-09 16:22:12.065417 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.76( v 213'536 (0'0,213'536] local-les=243 n=102 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:12.065429 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065434 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.065435 7f8690f64700 10 op_submit_start 3941 2014-06-09 16:22:12.065437 7f8690f64700 5 queue_transactions (trailing journal) 3941 2014-06-09 16:22:12.065439 7f8690f64700 5 queue_op 0x30a2a20 seq 3941 osr(0.220 0x2bd6a70) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.065443 7f8690f64700 10 op_submit_finish 3941 2014-06-09 16:22:12.065448 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3892.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.065452 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.065452 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.065447 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.73b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.065459 7f8689f56700 5 queue_transactions existing osr(3.76 0x3ed0240)/0x3ed0240 2014-06-09 16:22:12.065461 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.065466 7f86a5610700 5 _do_op 0x33a2d20 seq 3893 osr(2.4a5 0x2ed17f0)/0x2ed17f0 start 2014-06-09 16:22:12.065465 7f86a5e11700 10 _do_op 0x30d97620 seq 3892 r = 0, finisher 0x306ce360 0 2014-06-09 16:22:12.065469 7f86a5610700 10 _do_transaction on 0xbf0fec0 2014-06-09 16:22:12.065475 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.065475 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd676c0] 2014-06-09 16:22:12.065477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd676c0] 2014-06-09 16:22:12.065478 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.065479 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.065480 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.065483 7f86a5e11700 10 _finish_op 0x30d97620 seq 3892 osr(2.759 0x3ed00b0)/0x3ed00b0 2014-06-09 16:22:12.065484 7f8690f64700 5 queue_transactions existing osr(1.73b 0x2ed00c0)/0x2ed00c0 2014-06-09 16:22:12.065490 7f8689755700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065491 7f8692f68700 10 op_submit_start 3942 2014-06-09 16:22:12.065493 7f8692f68700 5 queue_transactions (trailing journal) 3942 2014-06-09 16:22:12.065495 7f8692f68700 5 queue_op 0xc88a840 seq 3942 osr(0.25d 0x2bd6a00) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.065497 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065498 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306ce360] 2014-06-09 16:22:12.065501 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306ce360] 2014-06-09 16:22:12.065503 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.065502 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065504 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.065507 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065525 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.065530 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065539 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065535 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd7a0 (1 active) 2014-06-09 16:22:12.065544 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.065545 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300000 (2 active) 2014-06-09 16:22:12.065539 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3893.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.065549 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065549 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.065553 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.065557 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065562 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.065565 7f8692f68700 10 op_submit_finish 3942 2014-06-09 16:22:12.065563 7f86a5e11700 5 _do_op 0x324f4780 seq 3894 osr(1.24c 0x384cfd0)/0x384cfd0 start 2014-06-09 16:22:12.065567 7f86a5e11700 10 _do_transaction on 0xc2e1d40 2014-06-09 16:22:12.065568 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065573 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.065578 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065569 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.24d( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.065580 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.065583 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.065587 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065593 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065599 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065603 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065605 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.065610 7f8692f68700 5 queue_transactions existing osr(2.24d 0x2ed0a50)/0x2ed0a50 2014-06-09 16:22:12.065611 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.065616 7f86a5610700 10 _do_op 0x33a2d20 seq 3893 r = 0, finisher 0xbd6df60 0 2014-06-09 16:22:12.065621 7f86a5610700 10 _finish_op 0x33a2d20 seq 3893 osr(2.4a5 0x2ed17f0)/0x2ed17f0 2014-06-09 16:22:12.065622 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509940] 2014-06-09 16:22:12.065625 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509940] 2014-06-09 16:22:12.065626 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.065627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.065628 7f8689755700 10 op_submit_start 3943 2014-06-09 16:22:12.065628 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922900 (1 active) 2014-06-09 16:22:12.065630 7f8689755700 5 queue_transactions (trailing journal) 3943 2014-06-09 16:22:12.065630 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6df60] 2014-06-09 16:22:12.065630 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f8c0 (2 active) 2014-06-09 16:22:12.065632 7f86a5610700 5 _do_op 0x230b8060 seq 3895 osr(0.290 0x2bd6c80)/0x2bd6c80 start 2014-06-09 16:22:12.065633 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6df60] 2014-06-09 16:22:12.065635 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.065636 7f86a5610700 10 _do_transaction on 0xc2cacc0 2014-06-09 16:22:12.065636 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.065641 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.065632 7f8689755700 5 queue_op 0x1e08bd40 seq 3943 osr(2.288 0x2ed0d10) 861 bytes (queue has 50 ops and 41513 bytes) 2014-06-09 16:22:12.065668 7f8689755700 10 op_submit_finish 3943 2014-06-09 16:22:12.065672 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.065687 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3894.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.065691 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.065693 7f8689755700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e770 (19 active) 2014-06-09 16:22:12.065701 7f86a5e11700 10 _do_op 0x324f4780 seq 3894 r = 0, finisher 0x32535980 0 2014-06-09 16:22:12.065711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x325357e0] 2014-06-09 16:22:12.065713 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x325357e0] 2014-06-09 16:22:12.065715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.065716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.065719 7f86a5e11700 10 _finish_op 0x324f4780 seq 3894 osr(1.24c 0x384cfd0)/0x384cfd0 2014-06-09 16:22:12.065726 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32535980] 2014-06-09 16:22:12.065730 7f8689755700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e770 (20 active) 2014-06-09 16:22:12.065732 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32535980] 2014-06-09 16:22:12.065733 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.065734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.065737 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300000 (1 active) 2014-06-09 16:22:12.065739 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcc60 (2 active) 2014-06-09 16:22:12.065741 7f86a5e11700 5 _do_op 0xc9301e0 seq 3896 osr(0.456 0x2bd7530)/0x2bd7530 start 2014-06-09 16:22:12.065743 7f86a5e11700 10 _do_transaction on 0x394e300 2014-06-09 16:22:12.065734 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.2c9( v 213'596 (0'0,213'596] local-les=243 n=116 ec=118 les/c 243/243 242/242/200) 2014-06-09 16:22:12.065748 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.065783 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.065788 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.065793 7f8689755700 5 queue_transactions existing osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 2014-06-09 16:22:12.065796 7f8689755700 10 op_submit_start 3944 2014-06-09 16:22:12.065797 7f8689755700 5 queue_transactions (trailing journal) 3944 2014-06-09 16:22:12.065797 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3896.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.065799 7f8689755700 5 queue_op 0xc2d4900 seq 3944 osr(3.2c9 0x3ed0ea0) 828 bytes (queue has 50 ops and 41513 bytes) 2014-06-09 16:22:12.065801 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.065802 7f8689755700 10 op_submit_finish 3944 2014-06-09 16:22:12.065804 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.33f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.33f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.065829 7f86a5e11700 10 _do_op 0xc9301e0 seq 3896 r = 0, finisher 0x306cfce0 0 2014-06-09 16:22:12.065837 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.065839 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd71f80] 2014-06-09 16:22:12.065841 7f8689755700 5 queue_transactions existing osr(0.33f 0x2bd7060)/0x2bd7060 2014-06-09 16:22:12.065842 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd71f80] 2014-06-09 16:22:12.065843 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.065844 7f8689755700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065844 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.065847 7f86a5e11700 10 _finish_op 0xc9301e0 seq 3896 osr(0.456 0x2bd7530)/0x2bd7530 2014-06-09 16:22:12.065848 7f868cf5c700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065853 7f868af58700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065859 7f868c75b700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065865 7f8691765700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065870 7f8691f66700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065876 7f868ef60700 2 waiting 51 > 50 ops || 42566 > 104857600 2014-06-09 16:22:12.065881 7f868a757700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065887 7f8692767700 2 waiting 51 > 50 ops || 928647 > 104857600 2014-06-09 16:22:12.065891 7f868df5e700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065895 7f868b759700 2 waiting 51 > 50 ops || 42339 > 104857600 2014-06-09 16:22:12.065901 7f8690763700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065906 7f868d75d700 2 waiting 51 > 50 ops || 42339 > 104857600 2014-06-09 16:22:12.065911 7f868ff62700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065915 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.065916 7f8689f56700 2 waiting 51 > 50 ops || 42339 > 104857600 2014-06-09 16:22:12.065921 7f8690f64700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065926 7f868bf5a700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065929 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3895.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.065932 7f868e75f700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065933 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.065936 7f868f761700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065942 7f8692f68700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.065945 7f86a5610700 10 _do_op 0x230b8060 seq 3895 r = 0, finisher 0xbd5ac20 0 2014-06-09 16:22:12.065952 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc8a00] 2014-06-09 16:22:12.065955 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.065966 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.065977 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.290( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:12.065986 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc8a00] 2014-06-09 16:22:12.065988 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.065989 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.065999 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306cfce0] 2014-06-09 16:22:12.066000 7f868cf5c700 10 op_submit_start 3945 2014-06-09 16:22:12.066002 7f868cf5c700 5 queue_transactions (trailing journal) 3945 2014-06-09 16:22:12.066004 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306cfce0] 2014-06-09 16:22:12.066003 7f868cf5c700 5 queue_op 0x300324e0 seq 3945 osr(2.2d9 0x2ed0c30) 828 bytes (queue has 50 ops and 41513 bytes) 2014-06-09 16:22:12.066006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.066007 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.066009 7f868af58700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.066010 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcc60 (1 active) 2014-06-09 16:22:12.066014 7f868c75b700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.066012 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d06c0 (2 active) 2014-06-09 16:22:12.066019 7f8691765700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.066024 7f8691f66700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.066022 7f86a5e11700 5 _do_op 0xc930960 seq 3897 osr(2.57a 0x2ed1870)/0x2ed1870 start 2014-06-09 16:22:12.066029 7f868ef60700 2 waiting 51 > 50 ops || 42566 > 104857600 2014-06-09 16:22:12.066031 7f86a5e11700 10 _do_transaction on 0xc320a80 2014-06-09 16:22:12.066033 7f868a757700 2 waiting 51 > 50 ops || 42341 > 104857600 2014-06-09 16:22:12.066034 7f86a5610700 10 _finish_op 0x230b8060 seq 3895 osr(0.290 0x2bd6c80)/0x2bd6c80 2014-06-09 16:22:12.066044 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.066048 7f86a6b95700 5 -- op tracker -- , seq: 3623, time: 2014-06-09 16:22:12.062404, event: header_read, op: osd_op(client.4542.0:34475 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 745472~1548288] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.066059 7f86a6b95700 5 -- op tracker -- , seq: 3623, time: 2014-06-09 16:22:12.062406, event: throttled, op: osd_op(client.4542.0:34475 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 745472~1548288] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.066070 7f86a6b95700 5 -- op tracker -- , seq: 3623, time: 2014-06-09 16:22:12.065993, event: all_read, op: osd_op(client.4542.0:34475 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 745472~1548288] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.066080 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.066085 7f86a6b95700 5 -- op tracker -- , seq: 3623, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34475 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 745472~1548288] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.066091 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.066094 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3897.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.066099 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.066111 7f86a5e11700 10 _do_op 0xc930960 seq 3897 r = 0, finisher 0xc169800 0 2014-06-09 16:22:12.066120 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e15d60] 2014-06-09 16:22:12.066122 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e15d60] 2014-06-09 16:22:12.066123 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.066117 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6ca00 prio 63 cost 1548288 latency 0.003713 osd_op(client.4542.0:34475 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 745472~1548288] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.066124 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.066754 7f86a6b95700 5 -- op tracker -- , seq: 3624, time: 2014-06-09 16:22:12.066140, event: header_read, op: osd_op(client.4542.0:34476 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.066766 7f86a6b95700 5 -- op tracker -- , seq: 3624, time: 2014-06-09 16:22:12.066141, event: throttled, op: osd_op(client.4542.0:34476 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.066779 7f86a6b95700 5 -- op tracker -- , seq: 3624, time: 2014-06-09 16:22:12.066731, event: all_read, op: osd_op(client.4542.0:34476 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.066789 7f86a6b95700 5 -- op tracker -- , seq: 3624, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34476 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.066794 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.066811 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6c900 prio 63 cost 516096 latency 0.000671 osd_op(client.4542.0:34476 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.066952 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f8c0 (1 active) 2014-06-09 16:22:12.066953 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.066956 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e240 (2 active) 2014-06-09 16:22:12.066960 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.066963 7f868cf5c700 10 op_submit_finish 3945 2014-06-09 16:22:12.066966 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ac20] 2014-06-09 16:22:12.066967 7f868af58700 10 op_submit_start 3946 2014-06-09 16:22:12.066969 7f868af58700 5 queue_transactions (trailing journal) 3946 2014-06-09 16:22:12.066971 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ac20] 2014-06-09 16:22:12.066973 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.066971 7f868af58700 5 queue_op 0x324cc900 seq 3946 osr(3.1fc 0x3ed0930) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.066974 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.066975 7f868af58700 10 op_submit_finish 3946 2014-06-09 16:22:12.066976 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.066968 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.75f( v 213'605 lc 213'595 (0'0,213'605] local-les=243 n=115 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.066979 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3cb( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.3cb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.066986 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:12.066997 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.067006 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.067020 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.067023 7f868cf5c700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.067025 7f868af58700 5 queue_transactions existing osr(1.3cb 0x384d4f0)/0x384d4f0 2014-06-09 16:22:12.067028 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067033 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067039 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067043 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067042 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:989 2014-06-09 16:22:12.067058 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.067063 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.3a8( v 213'701 (0'0,213'701] local-les=243 n=131 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:12.067082 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067089 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067096 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.067096 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067100 7f868cf5c700 5 queue_transactions existing osr(3.3a8 0x3ed0f80)/0x3ed0f80 2014-06-09 16:22:12.067103 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067109 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067109 7f86a5610700 5 _do_op 0xc9309c0 seq 3898 osr(3.3a8 0x3ed0f80)/0x3ed0f80 start 2014-06-09 16:22:12.067115 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.067121 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067128 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067134 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067135 7f86a5610700 10 _do_transaction on 0xbf4fbc0 2014-06-09 16:22:12.067140 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.067146 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067149 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.067152 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067158 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067204 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.067221 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3898.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.067225 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.067241 7f86a5610700 10 _do_op 0xc9309c0 seq 3898 r = 0, finisher 0xc40dd00 0 2014-06-09 16:22:12.067254 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e18a00] 2014-06-09 16:22:12.067256 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e18a00] 2014-06-09 16:22:12.067258 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.067259 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.067262 7f86a5610700 10 _finish_op 0xc9309c0 seq 3898 osr(3.3a8 0x3ed0f80)/0x3ed0f80 2014-06-09 16:22:12.067271 7f868b759700 10 op_submit_start 3947 2014-06-09 16:22:12.067273 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc40dd00] 2014-06-09 16:22:12.067273 7f868b759700 5 queue_transactions (trailing journal) 3947 2014-06-09 16:22:12.067276 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc40dd00] 2014-06-09 16:22:12.067275 7f868b759700 5 queue_op 0xbf7fc80 seq 3947 osr(0.9d 0x2bd6490) 826 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.067280 7f868df5e700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067276 7f8682de8700 5 -- op tracker -- , seq: 3625, time: 2014-06-09 16:22:12.067243, event: header_read, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) 2014-06-09 16:22:12.067283 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.067284 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.067285 7f868d75d700 2 waiting 51 > 50 ops || 42304 > 104857600 2014-06-09 16:22:12.067287 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e240 (1 active) 2014-06-09 16:22:12.067289 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497440 (2 active) 2014-06-09 16:22:12.067289 7f868af58700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067286 7f8682de8700 5 -- op tracker -- , seq: 3625, time: 2014-06-09 16:22:12.067245, event: throttled, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) 2014-06-09 16:22:12.067293 7f86a5610700 5 _do_op 0x33a3140 seq 3899 osr(3.e8 0x3ed0510)/0x3ed0510 start 2014-06-09 16:22:12.067296 7f86a5610700 10 _do_transaction on 0xc303500 2014-06-09 16:22:12.067295 7f8682de8700 5 -- op tracker -- , seq: 3625, time: 2014-06-09 16:22:12.067255, event: all_read, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) 2014-06-09 16:22:12.067295 7f868b759700 10 op_submit_finish 3947 2014-06-09 16:22:12.067302 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.067302 7f8682de8700 5 -- op tracker -- , seq: 3625, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) 2014-06-09 16:22:12.067306 7f8682de8700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) v2 epoch 243 2014-06-09 16:22:12.067311 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:12.067305 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:12.067317 7f8682de8700 15 osd.26 243 enqueue_op 0x30c6cf00 prio 196 cost 0 latency 0.000074 osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) v2 2014-06-09 16:22:12.067338 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.067341 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.067346 7f868b759700 5 queue_transactions existing osr(3.124 0x3ed07c0)/0x3ed07c0 2014-06-09 16:22:12.067349 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067353 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067354 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3899.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.067358 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.067359 7f8689f56700 2 waiting 51 > 50 ops || 42304 > 104857600 2014-06-09 16:22:12.067363 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067368 7f868bf5a700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067373 7f868e75f700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067377 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067382 7f868cf5c700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067389 7f8689755700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067388 7f86a5610700 10 _do_op 0x33a3140 seq 3899 r = 0, finisher 0x30e3f080 0 2014-06-09 16:22:12.067393 7f8692767700 2 waiting 51 > 50 ops || 928612 > 104857600 2014-06-09 16:22:12.067397 7f868c75b700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067397 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3281aa80] 2014-06-09 16:22:12.067393 7f8682de8700 5 -- op tracker -- , seq: 3626, time: 2014-06-09 16:22:12.067366, event: header_read, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.067399 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3281aa80] 2014-06-09 16:22:12.067400 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.067400 7f8691765700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067401 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.067404 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067404 7f86a5610700 10 _finish_op 0x33a3140 seq 3899 osr(3.e8 0x3ed0510)/0x3ed0510 2014-06-09 16:22:12.067403 7f8682de8700 5 -- op tracker -- , seq: 3626, time: 2014-06-09 16:22:12.067367, event: throttled, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.067410 7f8682de8700 5 -- op tracker -- , seq: 3626, time: 2014-06-09 16:22:12.067374, event: all_read, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.067416 7f868df5e700 10 op_submit_start 3948 2014-06-09 16:22:12.067416 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067417 7f868df5e700 5 queue_transactions (trailing journal) 3948 2014-06-09 16:22:12.067418 7f868df5e700 5 queue_op 0xbf7e780 seq 3948 osr(1.15b 0x384cbf0) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.067420 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067419 7f8682de8700 5 -- op tracker -- , seq: 3626, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.067424 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.067424 7f8682de8700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:12.067428 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:12.067433 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e3f080] 2014-06-09 16:22:12.067436 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e3f080] 2014-06-09 16:22:12.067434 7f8682de8700 15 osd.26 243 enqueue_op 0x30c6ce00 prio 196 cost 0 latency 0.000067 osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:12.067438 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.067439 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.067439 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067441 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497440 (1 active) 2014-06-09 16:22:12.067444 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc480 (2 active) 2014-06-09 16:22:12.067445 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067449 7f868df5e700 10 op_submit_finish 3948 2014-06-09 16:22:12.067449 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067453 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067458 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067462 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067453 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.654( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=5-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.654( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.067467 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067472 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067476 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067481 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067486 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067486 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.067490 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.067491 7f868df5e700 5 queue_transactions existing osr(2.654 0x2ed1fd0)/0x2ed1fd0 2014-06-09 16:22:12.067494 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067500 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067501 7f86a5610700 5 _do_op 0x30d962a0 seq 3900 osr(3.254 0x3ed0c30)/0x3ed0c30 start 2014-06-09 16:22:12.067504 7f86a5610700 10 _do_transaction on 0xc23f500 2014-06-09 16:22:12.067505 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067509 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067509 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.067514 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067553 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.067566 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3900.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.067571 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.067581 7f86a5610700 10 _do_op 0x30d962a0 seq 3900 r = 0, finisher 0x324d0a40 0 2014-06-09 16:22:12.067591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d0ac0] 2014-06-09 16:22:12.067593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d0ac0] 2014-06-09 16:22:12.067595 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.067596 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.067599 7f86a5610700 10 _finish_op 0x30d962a0 seq 3900 osr(3.254 0x3ed0c30)/0x3ed0c30 2014-06-09 16:22:12.067608 7f8692f68700 10 op_submit_start 3949 2014-06-09 16:22:12.067609 7f8692f68700 5 queue_transactions (trailing journal) 3949 2014-06-09 16:22:12.067611 7f8692f68700 5 queue_op 0xc88a000 seq 3949 osr(2.24d 0x2ed0a50) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.067621 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.067627 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d0a40] 2014-06-09 16:22:12.067633 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067637 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067640 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d0a40] 2014-06-09 16:22:12.067641 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.067642 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.067642 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067645 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc480 (1 active) 2014-06-09 16:22:12.067647 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067647 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2877a0 (2 active) 2014-06-09 16:22:12.067651 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067652 7f8692f68700 10 op_submit_finish 3949 2014-06-09 16:22:12.067655 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067660 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067664 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067656 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.25d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.25d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.067669 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067673 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067677 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.067688 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.067692 7f8692f68700 5 queue_transactions existing osr(0.25d 0x2bd6a00)/0x2bd6a00 2014-06-09 16:22:12.067695 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067699 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067704 7f86a5610700 5 _do_op 0xc64bc80 seq 3901 osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 start 2014-06-09 16:22:12.067707 7f86a5610700 10 _do_transaction on 0xc2ae480 2014-06-09 16:22:12.067712 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.067715 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067720 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067725 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067731 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067736 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.067761 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.067775 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3901.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.067779 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.067789 7f86a5610700 10 _do_op 0xc64bc80 seq 3901 r = 0, finisher 0x30629960 0 2014-06-09 16:22:12.067798 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062a1e0] 2014-06-09 16:22:12.067801 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062a1e0] 2014-06-09 16:22:12.067802 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.067803 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.067806 7f86a5610700 10 _finish_op 0xc64bc80 seq 3901 osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 2014-06-09 16:22:12.067817 7f868d75d700 10 op_submit_start 3950 2014-06-09 16:22:12.067818 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629960] 2014-06-09 16:22:12.067819 7f868d75d700 5 queue_transactions (trailing journal) 3950 2014-06-09 16:22:12.067822 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629960] 2014-06-09 16:22:12.067821 7f868d75d700 5 queue_op 0x30d97ec0 seq 3950 osr(1.67 0x384c5b0) 826 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.067824 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.067825 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.067827 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2877a0 (1 active) 2014-06-09 16:22:12.067829 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b9e0 (2 active) 2014-06-09 16:22:12.067834 7f868d75d700 10 op_submit_finish 3950 2014-06-09 16:22:12.067846 7f8690763700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067838 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.18a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.18a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.067851 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067856 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067860 7f8689f56700 2 waiting 51 > 50 ops || 42304 > 104857600 2014-06-09 16:22:12.067864 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067868 7f868bf5a700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067872 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.067873 7f868e75f700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067876 7f868d75d700 5 queue_transactions existing osr(1.18a 0x384cb50)/0x384cb50 2014-06-09 16:22:12.067877 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067879 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067882 7f868af58700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067883 7f86a5610700 5 _do_op 0xcabc600 seq 3902 osr(3.761 0x3ed1e10)/0x3ed1e10 start 2014-06-09 16:22:12.067886 7f86a5610700 10 _do_transaction on 0xbffea80 2014-06-09 16:22:12.067886 7f8689755700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067890 7f8692767700 2 waiting 51 > 50 ops || 928612 > 104857600 2014-06-09 16:22:12.067891 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.067895 7f8692f68700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067900 7f868df5e700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067905 7f8691765700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067910 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067914 7f868c75b700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067920 7f868cf5c700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067925 7f868a757700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.067929 7f868ef60700 2 waiting 51 > 50 ops || 42531 > 104857600 2014-06-09 16:22:12.067953 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.067967 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3902.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.067970 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.067982 7f86a5610700 10 _do_op 0xcabc600 seq 3902 r = 0, finisher 0xbd75440 0 2014-06-09 16:22:12.067991 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30d60780] 2014-06-09 16:22:12.067993 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30d60780] 2014-06-09 16:22:12.067995 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.067996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.067998 7f86a5610700 10 _finish_op 0xcabc600 seq 3902 osr(3.761 0x3ed1e10)/0x3ed1e10 2014-06-09 16:22:12.068006 7f8690763700 10 op_submit_start 3951 2014-06-09 16:22:12.068008 7f8690763700 5 queue_transactions (trailing journal) 3951 2014-06-09 16:22:12.068009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd75440] 2014-06-09 16:22:12.068010 7f8690763700 5 queue_op 0x324b2b40 seq 3951 osr(3.271 0x3ed0bc0) 828 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.068013 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd75440] 2014-06-09 16:22:12.068014 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.068015 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.068015 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068018 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b9e0 (1 active) 2014-06-09 16:22:12.068020 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068020 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3587e0 (2 active) 2014-06-09 16:22:12.068024 7f8689f56700 2 waiting 51 > 50 ops || 42304 > 104857600 2014-06-09 16:22:12.068025 7f8690763700 10 op_submit_finish 3951 2014-06-09 16:22:12.068028 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068032 7f868bf5a700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068037 7f868e75f700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068029 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.182( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/19 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.182( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.068041 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068062 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.068066 7f8690763700 5 queue_transactions existing osr(2.182 0x2ed0990)/0x2ed0990 2014-06-09 16:22:12.068069 7f8690763700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068073 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068078 7f868af58700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068078 7f86a5610700 5 _do_op 0xd326840 seq 3903 osr(1.a7 0x384c730)/0x384c730 start 2014-06-09 16:22:12.068081 7f86a5610700 10 _do_transaction on 0x2f28a80 2014-06-09 16:22:12.068083 7f8689755700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068086 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.068087 7f8692767700 2 waiting 51 > 50 ops || 928612 > 104857600 2014-06-09 16:22:12.068093 7f8692f68700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068097 7f868df5e700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068102 7f8691765700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068107 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068112 7f868c75b700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068117 7f868cf5c700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068122 7f868a757700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068126 7f868ef60700 2 waiting 51 > 50 ops || 42531 > 104857600 2014-06-09 16:22:12.068149 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.068162 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3903.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.068166 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.068175 7f86a5610700 10 _do_op 0xd326840 seq 3903 r = 0, finisher 0x30509f00 0 2014-06-09 16:22:12.068185 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6b1c0] 2014-06-09 16:22:12.068187 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6b1c0] 2014-06-09 16:22:12.068189 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.068189 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.068193 7f86a5610700 10 _finish_op 0xd326840 seq 3903 osr(1.a7 0x384c730)/0x384c730 2014-06-09 16:22:12.068201 7f868b759700 10 op_submit_start 3952 2014-06-09 16:22:12.068202 7f868b759700 5 queue_transactions (trailing journal) 3952 2014-06-09 16:22:12.068203 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30509f00] 2014-06-09 16:22:12.068204 7f868b759700 5 queue_op 0xbf7f3e0 seq 3952 osr(3.124 0x3ed07c0) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.068206 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30509f00] 2014-06-09 16:22:12.068208 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.068208 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.068208 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068211 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3587e0 (1 active) 2014-06-09 16:22:12.068213 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068213 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c1e60 (2 active) 2014-06-09 16:22:12.068217 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068218 7f868b759700 10 op_submit_finish 3952 2014-06-09 16:22:12.068222 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068227 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068231 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068221 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.068253 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.068258 7f868b759700 5 queue_transactions existing osr(3.577 0x3ed18c0)/0x3ed18c0 2014-06-09 16:22:12.068260 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068265 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068269 7f868d75d700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068273 7f86a5610700 5 _do_op 0x30d96240 seq 3904 osr(0.458 0x2bd7510)/0x2bd7510 start 2014-06-09 16:22:12.068274 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068276 7f86a5610700 10 _do_transaction on 0xc22c3c0 2014-06-09 16:22:12.068279 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068281 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.068284 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.068289 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068294 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068299 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068304 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068308 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068313 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068318 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068322 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.068333 7f86a6b95700 5 -- op tracker -- , seq: 3627, time: 2014-06-09 16:22:12.066832, event: header_read, op: osd_op(client.4542.0:34477 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2809856~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.068345 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.068344 7f86a6b95700 5 -- op tracker -- , seq: 3627, time: 2014-06-09 16:22:12.066833, event: throttled, op: osd_op(client.4542.0:34477 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2809856~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.068354 7f86a6b95700 5 -- op tracker -- , seq: 3627, time: 2014-06-09 16:22:12.068291, event: all_read, op: osd_op(client.4542.0:34477 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2809856~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.068359 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3904.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.068363 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.068367 7f86a6b95700 5 -- op tracker -- , seq: 3627, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34477 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2809856~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.068373 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.068384 7f86a5610700 10 _do_op 0x30d96240 seq 3904 r = 0, finisher 0xcc82620 0 2014-06-09 16:22:12.068394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc83c80] 2014-06-09 16:22:12.068399 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc83c80] 2014-06-09 16:22:12.068401 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.068393 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6d300 prio 63 cost 516096 latency 0.001561 osd_op(client.4542.0:34477 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 2809856~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.068402 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.068405 7f86a5610700 10 _finish_op 0x30d96240 seq 3904 osr(0.458 0x2bd7510)/0x2bd7510 2014-06-09 16:22:12.068414 7f868ff62700 10 op_submit_start 3953 2014-06-09 16:22:12.068415 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc82620] 2014-06-09 16:22:12.068416 7f868ff62700 5 queue_transactions (trailing journal) 3953 2014-06-09 16:22:12.068418 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc82620] 2014-06-09 16:22:12.068419 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.068418 7f868ff62700 5 queue_op 0x30a2c00 seq 3953 osr(3.37a 0x3ed1040) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.068420 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.068423 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068423 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c1e60 (1 active) 2014-06-09 16:22:12.068425 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3585a0 (2 active) 2014-06-09 16:22:12.068428 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068430 7f868ff62700 10 op_submit_finish 3953 2014-06-09 16:22:12.068432 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068437 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068441 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068434 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.75b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:12.068447 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.068456 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.068463 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.068476 7f868ff62700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.068493 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.75b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:120 2014-06-09 16:22:12.068507 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.068511 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.13c( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.068544 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.068548 7f868ff62700 5 queue_transactions existing osr(2.13c 0x2ed0610)/0x2ed0610 2014-06-09 16:22:12.068550 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068555 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068560 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068564 7f868d75d700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068568 7f86a5610700 5 _do_op 0x309a3680 seq 3905 osr(1.a6 0x384c750)/0x384c750 start 2014-06-09 16:22:12.068569 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068571 7f86a5610700 10 _do_transaction on 0x37e5380 2014-06-09 16:22:12.068576 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068576 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.068580 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.068605 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068611 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068617 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068622 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068627 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068627 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.068632 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068637 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068640 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.068655 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3905.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.068659 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.068669 7f86a5610700 10 _do_op 0x309a3680 seq 3905 r = 0, finisher 0x3045a420 0 2014-06-09 16:22:12.068678 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd33ee0] 2014-06-09 16:22:12.068680 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd33ee0] 2014-06-09 16:22:12.068682 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.068683 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.068685 7f86a5610700 10 _finish_op 0x309a3680 seq 3905 osr(1.a6 0x384c750)/0x384c750 2014-06-09 16:22:12.068694 7f8689f56700 10 op_submit_start 3954 2014-06-09 16:22:12.068696 7f8689f56700 5 queue_transactions (trailing journal) 3954 2014-06-09 16:22:12.068696 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3045a420] 2014-06-09 16:22:12.068698 7f8689f56700 5 queue_op 0xcca70e0 seq 3954 osr(3.76 0x3ed0240) 826 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.068700 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3045a420] 2014-06-09 16:22:12.068701 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.068702 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.068703 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068705 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3585a0 (1 active) 2014-06-09 16:22:12.068707 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc6c0 (2 active) 2014-06-09 16:22:12.068712 7f8689f56700 10 op_submit_finish 3954 2014-06-09 16:22:12.068715 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.7a( v 213'617 (0'0,213'617] local-les=243 n=119 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.068726 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068731 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068736 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068748 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.068753 7f8689f56700 5 queue_transactions existing osr(3.7a 0x3ed0220)/0x3ed0220 2014-06-09 16:22:12.068755 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068760 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068764 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068768 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068772 7f868d75d700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068776 7f86a5610700 5 _do_op 0x11fcf020 seq 3906 osr(2.298 0x2ed0c70)/0x2ed0c70 start 2014-06-09 16:22:12.068778 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068779 7f86a5610700 10 _do_transaction on 0xc29be00 2014-06-09 16:22:12.068783 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068784 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.068788 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.068793 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068798 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068803 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068808 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068813 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068817 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068822 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068826 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.068846 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.068860 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3906.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.068865 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.068877 7f86a5610700 10 _do_op 0x11fcf020 seq 3906 r = 0, finisher 0xbd72bc0 0 2014-06-09 16:22:12.068886 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a6a00] 2014-06-09 16:22:12.068889 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a6a00] 2014-06-09 16:22:12.068890 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.068891 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.068894 7f86a5610700 10 _finish_op 0x11fcf020 seq 3906 osr(2.298 0x2ed0c70)/0x2ed0c70 2014-06-09 16:22:12.068902 7f8690f64700 10 op_submit_start 3955 2014-06-09 16:22:12.068904 7f8690f64700 5 queue_transactions (trailing journal) 3955 2014-06-09 16:22:12.068904 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd72bc0] 2014-06-09 16:22:12.068905 7f8690f64700 5 queue_op 0x30a2660 seq 3955 osr(1.73b 0x2ed00c0) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.068908 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd72bc0] 2014-06-09 16:22:12.068910 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.068911 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.068910 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068913 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc6c0 (1 active) 2014-06-09 16:22:12.068915 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068916 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406240 (2 active) 2014-06-09 16:22:12.068920 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068921 7f8690f64700 10 op_submit_finish 3955 2014-06-09 16:22:12.068925 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:12.068958 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.068963 7f8690f64700 5 queue_transactions existing osr(3.537 0x3ed1620)/0x3ed1620 2014-06-09 16:22:12.068965 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068969 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.068974 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068978 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068981 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068985 7f868d75d700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068989 7f86a5610700 5 _do_op 0x11fce900 seq 3907 osr(0.6a8 0x2bd7d90)/0x2bd7d90 start 2014-06-09 16:22:12.068990 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068992 7f86a5610700 10 _do_transaction on 0x336ac00 2014-06-09 16:22:12.068996 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.068997 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.069000 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.069005 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069010 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069015 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069020 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069024 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069030 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069034 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069038 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.069059 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.069073 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3907.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.069077 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.069087 7f86a5610700 10 _do_op 0x11fce900 seq 3907 r = 0, finisher 0xbd71f20 0 2014-06-09 16:22:12.069096 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd60500] 2014-06-09 16:22:12.069098 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd60500] 2014-06-09 16:22:12.069100 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.069101 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.069104 7f86a5610700 10 _finish_op 0x11fce900 seq 3907 osr(0.6a8 0x2bd7d90)/0x2bd7d90 2014-06-09 16:22:12.069111 7f868bf5a700 10 op_submit_start 3956 2014-06-09 16:22:12.069113 7f868bf5a700 5 queue_transactions (trailing journal) 3956 2014-06-09 16:22:12.069114 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd71f20] 2014-06-09 16:22:12.069115 7f868bf5a700 5 queue_op 0xc705260 seq 3956 osr(1.2a3 0x384d1f0) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.069117 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd71f20] 2014-06-09 16:22:12.069119 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.069120 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.069120 7f868e75f700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069122 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406240 (1 active) 2014-06-09 16:22:12.069125 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069125 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52ca20 (2 active) 2014-06-09 16:22:12.069131 7f868bf5a700 10 op_submit_finish 3956 2014-06-09 16:22:12.069131 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069135 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069139 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069142 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069146 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069135 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.587( v 213'571 (0'0,213'571] local-les=243 n=109 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:12.069150 7f868d75d700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069154 7f86a5610700 5 _do_op 0x324c0120 seq 3908 osr(3.53d 0x3ed1600)/0x3ed1600 start 2014-06-09 16:22:12.069157 7f86a5610700 10 _do_transaction on 0xc253980 2014-06-09 16:22:12.069162 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.069174 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.069180 7f868bf5a700 5 queue_transactions existing osr(3.587 0x3ed1840)/0x3ed1840 2014-06-09 16:22:12.069182 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069187 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069192 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069197 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.069197 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.069202 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069207 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069212 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069217 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069221 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069226 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069231 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069235 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.069222 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3908.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.069239 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.069256 7f86a5610700 10 _do_op 0x324c0120 seq 3908 r = 0, finisher 0xbd76340 0 2014-06-09 16:22:12.069266 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7c6c0] 2014-06-09 16:22:12.069268 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7c6c0] 2014-06-09 16:22:12.069270 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.069271 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.069273 7f86a5610700 10 _finish_op 0x324c0120 seq 3908 osr(3.53d 0x3ed1600)/0x3ed1600 2014-06-09 16:22:12.069281 7f868e75f700 10 op_submit_start 3957 2014-06-09 16:22:12.069283 7f868e75f700 5 queue_transactions (trailing journal) 3957 2014-06-09 16:22:12.069283 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76340] 2014-06-09 16:22:12.069286 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76340] 2014-06-09 16:22:12.069285 7f868e75f700 5 queue_op 0x324f4840 seq 3957 osr(0.3b7 0x2bd7290) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.069288 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.069289 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.069290 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069291 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52ca20 (1 active) 2014-06-09 16:22:12.069294 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069302 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069306 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069309 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069313 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069317 7f868d75d700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069321 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069321 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89a480 (2 active) 2014-06-09 16:22:12.069325 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069326 7f868e75f700 10 op_submit_finish 3957 2014-06-09 16:22:12.069330 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069333 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.069330 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.069344 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.975556 7 0.002973 2014-06-09 16:22:12.069358 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.069369 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.069381 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.069391 7f868e75f700 5 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.069399 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.069406 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.069418 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.069429 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.069438 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.069447 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.069456 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.069469 7f868e75f700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.069484 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 2014-06-09 16:22:12.069516 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.069519 7f868e75f700 5 queue_transactions existing osr(1.f2 0x384c990)/0x384c990 2014-06-09 16:22:12.069522 7f868e75f700 2 waiting 51 > 50 ops || 42338 > 104857600 2014-06-09 16:22:12.069526 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069531 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069535 7f86a5610700 5 _do_op 0xc704ae0 seq 3909 osr(3.587 0x3ed1840)/0x3ed1840 start 2014-06-09 16:22:12.069535 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069539 7f86a5610700 10 _do_transaction on 0x37e5440 2014-06-09 16:22:12.069542 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069544 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.069546 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069562 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069577 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069582 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.069617 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.069631 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3909.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.069635 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.069647 7f86a5610700 10 _do_op 0xc704ae0 seq 3909 r = 0, finisher 0x3042b7e0 0 2014-06-09 16:22:12.069642 7f86a6b95700 5 -- op tracker -- , seq: 3628, time: 2014-06-09 16:22:12.068453, event: header_read, op: osd_op(client.4542.0:34478 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3325952~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.069657 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd73920] 2014-06-09 16:22:12.069659 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd73920] 2014-06-09 16:22:12.069660 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.069661 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.069664 7f86a5610700 10 _finish_op 0xc704ae0 seq 3909 osr(3.587 0x3ed1840)/0x3ed1840 2014-06-09 16:22:12.069660 7f86a6b95700 5 -- op tracker -- , seq: 3628, time: 2014-06-09 16:22:12.068455, event: throttled, op: osd_op(client.4542.0:34478 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3325952~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.069673 7f868f761700 10 op_submit_start 3958 2014-06-09 16:22:12.069674 7f868f761700 5 queue_transactions (trailing journal) 3958 2014-06-09 16:22:12.069673 7f8690f64700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069676 7f868f761700 5 queue_op 0x11fcf200 seq 3958 osr(2.5b6 0x2ed1b90) 828 bytes (queue has 50 ops and 41480 bytes) 2014-06-09 16:22:12.069678 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069682 7f868ff62700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069686 7f868b759700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069689 7f8690763700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069685 7f86a6b95700 5 -- op tracker -- , seq: 3628, time: 2014-06-09 16:22:12.069619, event: all_read, op: osd_op(client.4542.0:34478 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3325952~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.069693 7f868d75d700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069698 7f868bf5a700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069699 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b7e0] 2014-06-09 16:22:12.069702 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b7e0] 2014-06-09 16:22:12.069703 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.069703 7f868af58700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069704 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.069707 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89a480 (1 active) 2014-06-09 16:22:12.069707 7f8689755700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069709 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1c20 (2 active) 2014-06-09 16:22:12.069712 7f8692767700 2 waiting 51 > 50 ops || 928614 > 104857600 2014-06-09 16:22:12.069714 7f868f761700 10 op_submit_finish 3958 2014-06-09 16:22:12.069721 7f86a6b95700 5 -- op tracker -- , seq: 3628, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34478 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3325952~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.069718 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:12.069727 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.069751 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.069746 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6d200 prio 63 cost 516096 latency 0.001292 osd_op(client.4542.0:34478 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3325952~516096] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.069755 7f868f761700 5 queue_transactions existing osr(3.18e 0x3ed0af0)/0x3ed0af0 2014-06-09 16:22:12.069758 7f868f761700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069762 7f868e75f700 2 waiting 51 > 50 ops || 42338 > 104857600 2014-06-09 16:22:12.069766 7f8692f68700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069770 7f868df5e700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069775 7f86a5610700 5 _do_op 0x324e7500 seq 3910 osr(1.697 0x384de10)/0x384de10 start 2014-06-09 16:22:12.069778 7f86a5610700 10 _do_transaction on 0x39db500 2014-06-09 16:22:12.069777 7f8691765700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069783 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.069783 7f8691f66700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069788 7f868c75b700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069794 7f868cf5c700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069799 7f868a757700 2 waiting 51 > 50 ops || 42308 > 104857600 2014-06-09 16:22:12.069803 7f868ef60700 2 waiting 51 > 50 ops || 42533 > 104857600 2014-06-09 16:22:12.069834 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.069848 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3910.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.069852 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.069866 7f86a5610700 10 _do_op 0x324e7500 seq 3910 r = 0, finisher 0x31435fa0 0 2014-06-09 16:22:12.069875 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a7a40] 2014-06-09 16:22:12.069878 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a7a40] 2014-06-09 16:22:12.069879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.069880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.069883 7f86a5610700 10 _finish_op 0x324e7500 seq 3910 osr(1.697 0x384de10)/0x384de10 2014-06-09 16:22:12.069892 7f8689f56700 10 op_submit_start 3959 2014-06-09 16:22:12.069894 7f8689f56700 5 queue_transactions (trailing journal) 3959 2014-06-09 16:22:12.069895 7f8689f56700 5 queue_op 0xd470840 seq 3959 osr(3.7a 0x3ed0220) 826 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.069899 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069903 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069907 7f8690763700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069911 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069917 7f868bf5a700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069920 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31435fa0] 2014-06-09 16:22:12.069923 7f868af58700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069924 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31435fa0] 2014-06-09 16:22:12.069925 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.069926 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.069928 7f8689755700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069929 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1c20 (1 active) 2014-06-09 16:22:12.069931 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524000 (2 active) 2014-06-09 16:22:12.069932 7f8692767700 2 waiting 51 > 50 ops || 928612 > 104857600 2014-06-09 16:22:12.069935 7f8689f56700 10 op_submit_finish 3959 2014-06-09 16:22:12.069939 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.5be( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.5be( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.069972 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.069977 7f8689f56700 5 queue_transactions existing osr(2.5be 0x2ed1b30)/0x2ed1b30 2014-06-09 16:22:12.069979 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069983 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069987 7f868e75f700 2 waiting 51 > 50 ops || 42336 > 104857600 2014-06-09 16:22:12.069991 7f8692f68700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.069995 7f868df5e700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070000 7f86a5610700 5 _do_op 0x11fcfc80 seq 3911 osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 start 2014-06-09 16:22:12.070000 7f8691765700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070003 7f86a5610700 10 _do_transaction on 0xc2e0b40 2014-06-09 16:22:12.070006 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070008 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.070011 7f868c75b700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070016 7f868cf5c700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070021 7f868a757700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070025 7f868ef60700 2 waiting 51 > 50 ops || 42531 > 104857600 2014-06-09 16:22:12.070029 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070062 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.070075 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3911.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.070080 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.070092 7f86a5610700 10 _do_op 0x11fcfc80 seq 3911 r = 0, finisher 0xbd791e0 0 2014-06-09 16:22:12.070101 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32072800] 2014-06-09 16:22:12.070103 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32072800] 2014-06-09 16:22:12.070105 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.070106 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.070109 7f86a5610700 10 _finish_op 0x11fcfc80 seq 3911 osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 2014-06-09 16:22:12.070118 7f868b759700 10 op_submit_start 3960 2014-06-09 16:22:12.070120 7f868b759700 5 queue_transactions (trailing journal) 3960 2014-06-09 16:22:12.070121 7f868b759700 5 queue_op 0xbf7e120 seq 3960 osr(3.577 0x3ed18c0) 828 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.070125 7f8690763700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070129 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070133 7f868bf5a700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070138 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd791e0] 2014-06-09 16:22:12.070140 7f868af58700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070142 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd791e0] 2014-06-09 16:22:12.070143 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.070144 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.070144 7f8689755700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070147 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524000 (1 active) 2014-06-09 16:22:12.070148 7f8692767700 2 waiting 51 > 50 ops || 928612 > 104857600 2014-06-09 16:22:12.070149 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6526c0 (2 active) 2014-06-09 16:22:12.070154 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070159 7f868b759700 10 op_submit_finish 3960 2014-06-09 16:22:12.070162 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.6c0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.070197 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.070201 7f868b759700 5 queue_transactions existing osr(0.6c0 0x2bd7f50)/0x2bd7f50 2014-06-09 16:22:12.070203 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070207 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070212 7f868e75f700 2 waiting 51 > 50 ops || 42336 > 104857600 2014-06-09 16:22:12.070216 7f8692f68700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070220 7f868df5e700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070224 7f86a5610700 5 _do_op 0x309a30e0 seq 3912 osr(0.2fd 0x2bd6e50)/0x2bd6e50 start 2014-06-09 16:22:12.070225 7f8691765700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070228 7f86a5610700 10 _do_transaction on 0xbeba780 2014-06-09 16:22:12.070230 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070232 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.070235 7f868c75b700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070240 7f868cf5c700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070245 7f868a757700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070248 7f868ef60700 2 waiting 51 > 50 ops || 42531 > 104857600 2014-06-09 16:22:12.070252 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070256 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070289 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.070302 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3912.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.070300 7f86a6b95700 5 -- op tracker -- , seq: 3629, time: 2014-06-09 16:22:12.069769, event: header_read, op: osd_op(client.4542.0:34479 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3842048~352256] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.070306 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.070311 7f86a6b95700 5 -- op tracker -- , seq: 3629, time: 2014-06-09 16:22:12.069770, event: throttled, op: osd_op(client.4542.0:34479 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3842048~352256] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.070315 7f86a5610700 10 _do_op 0x309a30e0 seq 3912 r = 0, finisher 0x30508d40 0 2014-06-09 16:22:12.070325 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9f2e0] 2014-06-09 16:22:12.070328 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9f2e0] 2014-06-09 16:22:12.070322 7f86a6b95700 5 -- op tracker -- , seq: 3629, time: 2014-06-09 16:22:12.070279, event: all_read, op: osd_op(client.4542.0:34479 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3842048~352256] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.070332 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.070333 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.070336 7f86a5610700 10 _finish_op 0x309a30e0 seq 3912 osr(0.2fd 0x2bd6e50)/0x2bd6e50 2014-06-09 16:22:12.070346 7f868d75d700 10 op_submit_start 3961 2014-06-09 16:22:12.070348 7f868d75d700 5 queue_transactions (trailing journal) 3961 2014-06-09 16:22:12.070349 7f868d75d700 5 queue_op 0x230b8c60 seq 3961 osr(1.18a 0x384cb50) 828 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.070358 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30508d40] 2014-06-09 16:22:12.070362 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30508d40] 2014-06-09 16:22:12.070363 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.070364 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.070367 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6526c0 (1 active) 2014-06-09 16:22:12.070369 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb200 (2 active) 2014-06-09 16:22:12.070371 7f868bf5a700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070373 7f868d75d700 10 op_submit_finish 3961 2014-06-09 16:22:12.070377 7f868af58700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070382 7f8689755700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070377 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.65e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.65e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.070386 7f8692767700 2 waiting 51 > 50 ops || 928612 > 104857600 2014-06-09 16:22:12.070394 7f86a6b95700 5 -- op tracker -- , seq: 3629, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34479 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3842048~352256] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.070400 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.070409 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.070413 7f868d75d700 5 queue_transactions existing osr(1.65e 0x384df30)/0x384df30 2014-06-09 16:22:12.070415 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070419 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070424 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070418 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6d100 prio 63 cost 352256 latency 0.000649 osd_op(client.4542.0:34479 rbd_data.115e6b8b4567.00000000000017fa [set-alloc-hint object_size 4194304 write_size 4194304,write 3842048~352256] 3.f6bfff4f RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.070428 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070431 7f868e75f700 2 waiting 51 > 50 ops || 42336 > 104857600 2014-06-09 16:22:12.070435 7f8692f68700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070439 7f868df5e700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070443 7f86a5610700 5 _do_op 0x10038d20 seq 3913 osr(1.211 0x384cc90)/0x384cc90 start 2014-06-09 16:22:12.070444 7f8691765700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070446 7f86a5610700 10 _do_transaction on 0xc1f6cc0 2014-06-09 16:22:12.070450 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070451 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.070455 7f868c75b700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070460 7f868cf5c700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070465 7f868a757700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070469 7f868ef60700 2 waiting 51 > 50 ops || 42531 > 104857600 2014-06-09 16:22:12.070473 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070477 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070481 7f8690763700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070513 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.070526 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3913.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.070531 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.070547 7f86a5610700 10 _do_op 0x10038d20 seq 3913 r = 0, finisher 0x2d738100 0 2014-06-09 16:22:12.070557 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd76820] 2014-06-09 16:22:12.070560 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd76820] 2014-06-09 16:22:12.070569 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.070570 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.070573 7f86a5610700 10 _finish_op 0x10038d20 seq 3913 osr(1.211 0x384cc90)/0x384cc90 2014-06-09 16:22:12.070583 7f868bf5a700 10 op_submit_start 3962 2014-06-09 16:22:12.070584 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d738100] 2014-06-09 16:22:12.070588 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d738100] 2014-06-09 16:22:12.070589 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.070590 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.070593 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb200 (1 active) 2014-06-09 16:22:12.070595 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7805a0 (2 active) 2014-06-09 16:22:12.070597 7f86a5610700 5 _do_op 0x30060240 seq 3914 osr(2.126 0x2ed0670)/0x2ed0670 start 2014-06-09 16:22:12.070585 7f868bf5a700 5 queue_transactions (trailing journal) 3962 2014-06-09 16:22:12.070600 7f86a5610700 10 _do_transaction on 0xbfe72c0 2014-06-09 16:22:12.070600 7f868bf5a700 5 queue_op 0xc705b60 seq 3962 osr(3.587 0x3ed1840) 828 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.070603 7f868bf5a700 10 op_submit_finish 3962 2014-06-09 16:22:12.070604 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.070606 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.52d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.070641 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.070639 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.070646 7f868bf5a700 5 queue_transactions existing osr(2.52d 0x2ed19f0)/0x2ed19f0 2014-06-09 16:22:12.070648 7f868bf5a700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070653 7f868af58700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070653 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3914.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.070657 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.070658 7f8689755700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070662 7f8692767700 2 waiting 51 > 50 ops || 928612 > 104857600 2014-06-09 16:22:12.070668 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070674 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070678 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070682 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070686 7f868e75f700 2 waiting 51 > 50 ops || 42336 > 104857600 2014-06-09 16:22:12.070690 7f8692f68700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070693 7f868df5e700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070699 7f8691765700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070700 7f86a5610700 10 _do_op 0x30060240 seq 3914 r = 0, finisher 0x30d61ac0 0 2014-06-09 16:22:12.070705 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070708 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31791cc0] 2014-06-09 16:22:12.070710 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31791cc0] 2014-06-09 16:22:12.070712 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.070713 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.070716 7f86a5610700 10 _finish_op 0x30060240 seq 3914 osr(2.126 0x2ed0670)/0x2ed0670 2014-06-09 16:22:12.070710 7f868c75b700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070730 7f868cf5c700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070736 7f868a757700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070740 7f868ef60700 2 waiting 51 > 50 ops || 42531 > 104857600 2014-06-09 16:22:12.070743 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070747 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070751 7f8690763700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070759 7f868af58700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070760 7f868bf5a700 10 op_submit_start 3963 2014-06-09 16:22:12.070762 7f868bf5a700 5 queue_transactions (trailing journal) 3963 2014-06-09 16:22:12.070764 7f8692767700 2 waiting 51 > 50 ops || 928612 > 104857600 2014-06-09 16:22:12.070764 7f868bf5a700 5 queue_op 0xc34bda0 seq 3963 osr(2.52d 0x2ed19f0) 828 bytes (queue has 50 ops and 41478 bytes) 2014-06-09 16:22:12.070768 7f868d75d700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070772 7f8689f56700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070775 7f868b759700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070779 7f868f761700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070783 7f868e75f700 2 waiting 51 > 50 ops || 42336 > 104857600 2014-06-09 16:22:12.070786 7f8692f68700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070790 7f868df5e700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070795 7f8691765700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070799 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30d61ac0] 2014-06-09 16:22:12.070800 7f8691f66700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070803 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30d61ac0] 2014-06-09 16:22:12.070804 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.070804 7f868c75b700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070806 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.070809 7f868cf5c700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070813 7f868a757700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070817 7f868ef60700 2 waiting 51 > 50 ops || 42531 > 104857600 2014-06-09 16:22:12.070820 7f8690f64700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070824 7f868ff62700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070828 7f8690763700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070833 7f8689755700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070833 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7805a0 (1 active) 2014-06-09 16:22:12.070835 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad320 (2 active) 2014-06-09 16:22:12.070839 7f86a5610700 5 _do_op 0x2fcae300 seq 3915 osr(0.5a8 0x2bd7b00)/0x2bd7b00 start 2014-06-09 16:22:12.070842 7f86a5610700 10 _do_transaction on 0xc26e9c0 2014-06-09 16:22:12.070848 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.070858 7f868bf5a700 10 op_submit_finish 3963 2014-06-09 16:22:12.070862 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.529( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.070883 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.070896 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.070898 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3915.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.070901 7f868bf5a700 5 queue_transactions existing osr(0.529 0x2bd7930)/0x2bd7930 2014-06-09 16:22:12.070902 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.070905 7f868bf5a700 2 waiting 51 > 50 ops || 42306 > 104857600 2014-06-09 16:22:12.070914 7f86a5610700 10 _do_op 0x2fcae300 seq 3915 r = 0, finisher 0x33a70c0 0 2014-06-09 16:22:12.070924 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x29283b60] 2014-06-09 16:22:12.070927 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.070937 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.070946 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.5a8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 2014-06-09 16:22:12.070955 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x29283b60] 2014-06-09 16:22:12.070957 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.070958 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.070961 7f86a5610700 10 _finish_op 0x2fcae300 seq 3915 osr(0.5a8 0x2bd7b00)/0x2bd7b00 2014-06-09 16:22:12.070970 7f8692767700 10 op_submit_start 3964 2014-06-09 16:22:12.070972 7f8692767700 5 queue_transactions (trailing journal) 3964 2014-06-09 16:22:12.070973 7f8692767700 5 queue_op 0xd04f800 seq 3964 osr(3.2b3 0x3ed0f00) 887134 bytes (queue has 50 ops and 927751 bytes) 2014-06-09 16:22:12.070977 7f868d75d700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.070981 7f8689f56700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.070985 7f868b759700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.070988 7f868f761700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.070992 7f868e75f700 2 waiting 51 > 50 ops || 928609 > 104857600 2014-06-09 16:22:12.070996 7f8692f68700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.070999 7f868df5e700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071007 7f8691765700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a70c0] 2014-06-09 16:22:12.071013 7f8691f66700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071014 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a70c0] 2014-06-09 16:22:12.071016 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.071017 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.071018 7f868c75b700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071023 7f868cf5c700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071027 7f868a757700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071032 7f868ef60700 2 waiting 51 > 50 ops || 928804 > 104857600 2014-06-09 16:22:12.071035 7f8690f64700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071039 7f868ff62700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071043 7f8690763700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071020 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad320 (1 active) 2014-06-09 16:22:12.071047 7f8689755700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071048 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc923b00 (2 active) 2014-06-09 16:22:12.071053 7f8692767700 10 op_submit_finish 3964 2014-06-09 16:22:12.071053 7f868bf5a700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071058 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc218d80 243'637 rep_tid=24 committed?=0 applied?=0 lock=0 op=osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.071082 7f8692767700 10 osd.26 243 dequeue_op 0x16bd7b00 finish 2014-06-09 16:22:12.071087 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.071094 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.071098 7f8692767700 10 osd.26 243 dequeue_op 0x30c6cf00 prio 196 cost 0 latency 0.003855 osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) v2 pg pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.071114 7f8692767700 5 -- op tracker -- , seq: 3625, time: 2014-06-09 16:22:12.071111, event: reached_pg, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) 2014-06-09 16:22:12.071122 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x30c6cf00 2014-06-09 16:22:12.071139 7f8692767700 5 -- op tracker -- , seq: 3625, time: 2014-06-09 16:22:12.071135, event: started, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) 2014-06-09 16:22:12.071145 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 24 op ack_type 1 from 9 2014-06-09 16:22:12.071165 7f868af58700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071169 7f86a5610700 5 _do_op 0x30a3a40 seq 3916 osr(3.468 0x3ed1590)/0x3ed1590 start 2014-06-09 16:22:12.071161 7f8692767700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.071156, event: sub_op_applied_rec, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.071172 7f86a5610700 10 _do_transaction on 0xbfed2c0 2014-06-09 16:22:12.071176 7f8692767700 10 osd.26 243 dequeue_op 0x30c6cf00 finish 2014-06-09 16:22:12.071177 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.071184 7f8692767700 5 -- op tracker -- , seq: 3625, time: 2014-06-09 16:22:12.071181, event: done, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ack, result = 0) 2014-06-09 16:22:12.071197 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.071213 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.071227 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3916.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.071230 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.071240 7f86a5610700 10 _do_op 0x30a3a40 seq 3916 r = 0, finisher 0x325352e0 0 2014-06-09 16:22:12.071249 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2b80] 2014-06-09 16:22:12.071252 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2b80] 2014-06-09 16:22:12.071253 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.071254 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.071257 7f86a5610700 10 _finish_op 0x30a3a40 seq 3916 osr(3.468 0x3ed1590)/0x3ed1590 2014-06-09 16:22:12.071266 7f8689f56700 10 op_submit_start 3965 2014-06-09 16:22:12.071267 7f8689f56700 5 queue_transactions (trailing journal) 3965 2014-06-09 16:22:12.071268 7f8689f56700 5 queue_op 0xd471c20 seq 3965 osr(2.5be 0x2ed1b30) 828 bytes (queue has 50 ops and 927751 bytes) 2014-06-09 16:22:12.071272 7f868b759700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071277 7f868f761700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071281 7f868e75f700 2 waiting 51 > 50 ops || 928609 > 104857600 2014-06-09 16:22:12.071286 7f8692f68700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071290 7f868df5e700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071296 7f8691765700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071301 7f8691f66700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071301 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x325352e0] 2014-06-09 16:22:12.071305 7f868c75b700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071304 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f3eb0 (20 active) 2014-06-09 16:22:12.071306 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x325352e0] 2014-06-09 16:22:12.071308 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.071309 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.071311 7f868cf5c700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071310 7f86a6b95700 5 -- op tracker -- , seq: 3630, time: 2014-06-09 16:22:12.070436, event: header_read, op: osd_op(client.4542.0:34496 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~417792] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.071315 7f868a757700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071319 7f868ef60700 2 waiting 51 > 50 ops || 928804 > 104857600 2014-06-09 16:22:12.071310 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.771( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.071322 7f8690f64700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071321 7f86a6b95700 5 -- op tracker -- , seq: 3630, time: 2014-06-09 16:22:12.070436, event: throttled, op: osd_op(client.4542.0:34496 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~417792] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.071326 7f868ff62700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071329 7f8690763700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071334 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc923b00 (1 active) 2014-06-09 16:22:12.071336 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3005a0 (2 active) 2014-06-09 16:22:12.071340 7f8689f56700 10 op_submit_finish 3965 2014-06-09 16:22:12.071342 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.071348 7f8692767700 5 queue_transactions existing osr(0.771 0x384c310)/0x384c310 2014-06-09 16:22:12.071343 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.186( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.186( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.071334 7f8689755700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071363 7f868bf5a700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071331 7f86a6b95700 5 -- op tracker -- , seq: 3630, time: 2014-06-09 16:22:12.071272, event: all_read, op: osd_op(client.4542.0:34496 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~417792] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.071375 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.071379 7f8689f56700 5 queue_transactions existing osr(0.186 0x2bd67d0)/0x2bd67d0 2014-06-09 16:22:12.071382 7f8689f56700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071381 7f86a5e11700 10 _finish_op 0xc930960 seq 3897 osr(2.57a 0x2ed1870)/0x2ed1870 2014-06-09 16:22:12.071382 7f86a6b95700 5 -- op tracker -- , seq: 3630, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34496 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~417792] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.071390 7f868b759700 10 op_submit_start 3966 2014-06-09 16:22:12.071388 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.071392 7f868b759700 5 queue_transactions (trailing journal) 3966 2014-06-09 16:22:12.071391 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d06c0 (1 active) 2014-06-09 16:22:12.071392 7f8692767700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071393 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e360 (2 active) 2014-06-09 16:22:12.071393 7f868b759700 5 queue_op 0xc88ac00 seq 3966 osr(0.6c0 0x2bd7f50) 828 bytes (queue has 50 ops and 927751 bytes) 2014-06-09 16:22:12.071396 7f868b759700 10 op_submit_finish 3966 2014-06-09 16:22:12.071396 7f86a5e11700 5 _do_op 0xc64a240 seq 3918 osr(1.ab 0x384c6f0)/0x384c6f0 start 2014-06-09 16:22:12.071398 7f86a5e11700 10 _do_transaction on 0xc239d40 2014-06-09 16:22:12.071398 7f868f761700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071403 7f868e75f700 2 waiting 51 > 50 ops || 928609 > 104857600 2014-06-09 16:22:12.071404 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.071407 7f8692f68700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071398 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.071411 7f868df5e700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071415 7f8691765700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071420 7f8691f66700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071425 7f868c75b700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071429 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.071430 7f868cf5c700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072281 7f868b759700 5 queue_transactions existing osr(3.7bc 0xb7f0190)/0xb7f0190 2014-06-09 16:22:12.072284 7f868b759700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071407 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6d000 prio 63 cost 417792 latency 0.000970 osd_op(client.4542.0:34496 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 0~417792] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.071434 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc169800] 2014-06-09 16:22:12.072290 7f868a757700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072297 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc169800] 2014-06-09 16:22:12.072299 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.072300 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.072302 7f868ef60700 2 waiting 51 > 50 ops || 928804 > 104857600 2014-06-09 16:22:12.071438 7f86a5610700 5 _do_op 0x324e7380 seq 3917 osr(3.6f 0x3ed0280)/0x3ed0280 start 2014-06-09 16:22:12.072307 7f8690f64700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.071447 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.072312 7f868ff62700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072315 7f8690763700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072309 7f86a5610700 10 _do_transaction on 0xc3363c0 2014-06-09 16:22:12.072322 7f8689755700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072325 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.072326 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3918.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.072329 7f868bf5a700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072330 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.072334 7f8689f56700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072340 7f868af58700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072345 7f868d75d700 2 waiting 51 > 50 ops || 928579 > 104857600 2014-06-09 16:22:12.072383 7f86a5e11700 10 _do_op 0xc64a240 seq 3918 r = 0, finisher 0x3061f960 0 2014-06-09 16:22:12.072390 7f86a5e11700 10 _finish_op 0xc64a240 seq 3918 osr(1.ab 0x384c6f0)/0x384c6f0 2014-06-09 16:22:12.072391 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30c55dc0] 2014-06-09 16:22:12.072393 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30c55dc0] 2014-06-09 16:22:12.072395 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.072396 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.072397 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e360 (1 active) 2014-06-09 16:22:12.072399 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061f960] 2014-06-09 16:22:12.072399 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77bc20 (2 active) 2014-06-09 16:22:12.072399 7f8692767700 10 op_submit_start 3967 2014-06-09 16:22:12.072402 7f8692767700 5 queue_transactions (trailing journal) 3967 2014-06-09 16:22:12.072401 7f86a5e11700 5 _do_op 0x30032ae0 seq 3919 osr(3.1f0 0x3ed0990)/0x3ed0990 start 2014-06-09 16:22:12.072402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061f960] 2014-06-09 16:22:12.072404 7f86a5e11700 10 _do_transaction on 0x394fbc0 2014-06-09 16:22:12.072404 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.072405 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.072403 7f8692767700 5 queue_op 0xcc87c20 seq 3967 osr(0.771 0x384c310) 828 bytes (queue has 50 ops and 927753 bytes) 2014-06-09 16:22:12.072406 7f8692767700 10 op_submit_finish 3967 2014-06-09 16:22:12.072409 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.072409 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.771( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.771( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.072429 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.072443 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.072443 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3917.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.072447 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.072448 7f8692767700 5 queue_transactions existing osr(0.771 0x384c310)/0x384c310 2014-06-09 16:22:12.072451 7f8692767700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072455 7f868f761700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072459 7f868e75f700 2 waiting 51 > 50 ops || 928611 > 104857600 2014-06-09 16:22:12.072463 7f8692f68700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072467 7f868df5e700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072467 7f86a5610700 10 _do_op 0x324e7380 seq 3917 r = 0, finisher 0x31412da0 0 2014-06-09 16:22:12.072472 7f8691765700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072476 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc169760] 2014-06-09 16:22:12.072479 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc169760] 2014-06-09 16:22:12.072478 7f8691f66700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072480 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.072481 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.072483 7f868c75b700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072484 7f86a5610700 10 _finish_op 0x324e7380 seq 3917 osr(3.6f 0x3ed0280)/0x3ed0280 2014-06-09 16:22:12.072492 7f868b759700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072493 7f8692767700 10 op_submit_start 3968 2014-06-09 16:22:12.072495 7f8692767700 5 queue_transactions (trailing journal) 3968 2014-06-09 16:22:12.072496 7f8692767700 5 queue_op 0xcc86780 seq 3968 osr(0.771 0x384c310) 828 bytes (queue has 50 ops and 927755 bytes) 2014-06-09 16:22:12.072497 7f868a757700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072503 7f8690f64700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072503 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412da0] 2014-06-09 16:22:12.072506 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412da0] 2014-06-09 16:22:12.072507 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.072508 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.072509 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.072510 7f868ff62700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072514 7f8690763700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072519 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3005a0 (1 active) 2014-06-09 16:22:12.072521 7f8689755700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072522 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e000 (2 active) 2014-06-09 16:22:12.072524 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3919.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.072526 7f86a5610700 5 _do_op 0x2fc8d680 seq 3920 osr(0.3ab 0x2bd72f0)/0x2bd72f0 start 2014-06-09 16:22:12.072527 7f868bf5a700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072528 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.072529 7f86a5610700 10 _do_transaction on 0x3858240 2014-06-09 16:22:12.072528 7f8692767700 10 op_submit_finish 3968 2014-06-09 16:22:12.072532 7f8689f56700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072537 7f868af58700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072541 7f868d75d700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072532 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.40c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.072540 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.072545 7f86a5e11700 10 _do_op 0x30032ae0 seq 3919 r = 0, finisher 0x17fb7680 0 2014-06-09 16:22:12.072550 7f86a5e11700 10 _finish_op 0x30032ae0 seq 3919 osr(3.1f0 0x3ed0990)/0x3ed0990 2014-06-09 16:22:12.072551 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e181a0] 2014-06-09 16:22:12.072553 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e181a0] 2014-06-09 16:22:12.072555 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.072556 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.072560 7f868b759700 10 op_submit_start 3969 2014-06-09 16:22:12.072560 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17fb7680] 2014-06-09 16:22:12.072561 7f868b759700 5 queue_transactions (trailing journal) 3969 2014-06-09 16:22:12.072564 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17fb7680] 2014-06-09 16:22:12.072563 7f868b759700 5 queue_op 0xc88aba0 seq 3969 osr(3.7bc 0xb7f0190) 828 bytes (queue has 50 ops and 927755 bytes) 2014-06-09 16:22:12.072564 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.072565 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.072566 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.072568 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77bc20 (1 active) 2014-06-09 16:22:12.072570 7f8692767700 5 queue_transactions existing osr(2.40c 0x2ed1250)/0x2ed1250 2014-06-09 16:22:12.072571 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2878c0 (2 active) 2014-06-09 16:22:12.072572 7f8692767700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072575 7f86a5e11700 5 _do_op 0xc931f80 seq 3921 osr(2.624 0x2ed1c50)/0x2ed1c50 start 2014-06-09 16:22:12.072576 7f868b759700 10 op_submit_finish 3969 2014-06-09 16:22:12.072577 7f86a5e11700 10 _do_transaction on 0xc28f740 2014-06-09 16:22:12.072577 7f868f761700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072582 7f868e75f700 2 waiting 51 > 50 ops || 928613 > 104857600 2014-06-09 16:22:12.072582 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.072586 7f8692f68700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072579 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.072590 7f868df5e700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072595 7f8691f66700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072600 7f868c75b700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072605 7f868cf5c700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072609 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.072612 7f868ef60700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.072614 7f868b759700 5 queue_transactions existing osr(1.c9 0x384c9f0)/0x384c9f0 2014-06-09 16:22:12.072616 7f868b759700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072621 7f8690f64700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072625 7f868ff62700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072630 7f8690763700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072637 7f8689755700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072642 7f868bf5a700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072647 7f8689f56700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072653 7f868af58700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072657 7f868d75d700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072655 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.072663 7f8691765700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072669 7f868a757700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072676 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3920.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.072680 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.072702 7f86a5610700 10 _do_op 0x2fc8d680 seq 3920 r = 0, finisher 0x31434640 0 2014-06-09 16:22:12.072712 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31434400] 2014-06-09 16:22:12.072715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31434400] 2014-06-09 16:22:12.072716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.072717 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.072720 7f86a5610700 10 _finish_op 0x2fc8d680 seq 3920 osr(0.3ab 0x2bd72f0)/0x2bd72f0 2014-06-09 16:22:12.072727 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e000 (1 active) 2014-06-09 16:22:12.072728 7f8692767700 10 op_submit_start 3970 2014-06-09 16:22:12.072729 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31434640] 2014-06-09 16:22:12.072730 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359320 (2 active) 2014-06-09 16:22:12.072731 7f8692767700 5 queue_transactions (trailing journal) 3970 2014-06-09 16:22:12.072732 7f86a5610700 5 _do_op 0xc930660 seq 3922 osr(2.c6 0x2ed0490)/0x2ed0490 start 2014-06-09 16:22:12.072733 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31434640] 2014-06-09 16:22:12.072732 7f8692767700 5 queue_op 0xcca68a0 seq 3970 osr(2.40c 0x2ed1250) 828 bytes (queue has 50 ops and 927755 bytes) 2014-06-09 16:22:12.072734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.072734 7f86a5610700 10 _do_transaction on 0x471dec0 2014-06-09 16:22:12.072735 7f8692767700 10 op_submit_finish 3970 2014-06-09 16:22:12.072736 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.072741 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.072743 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.072738 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.14c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.072757 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3921.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.072761 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.072770 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.072774 7f8692767700 5 queue_transactions existing osr(1.14c 0x384c810)/0x384c810 2014-06-09 16:22:12.072777 7f8692767700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072778 7f86a5e11700 10 _do_op 0xc931f80 seq 3921 r = 0, finisher 0x189abee0 0 2014-06-09 16:22:12.072781 7f868f761700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072784 7f86a5e11700 10 _finish_op 0xc931f80 seq 3921 osr(2.624 0x2ed1c50)/0x2ed1c50 2014-06-09 16:22:12.072785 7f868e75f700 2 waiting 51 > 50 ops || 928613 > 104857600 2014-06-09 16:22:12.072785 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d1ca0] 2014-06-09 16:22:12.072787 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d1ca0] 2014-06-09 16:22:12.072789 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.072789 7f8692f68700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072790 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.072793 7f868df5e700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072798 7f8691f66700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072803 7f868c75b700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072809 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.072808 7f868cf5c700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072819 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3922.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.072821 7f868ef60700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.072823 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.072827 7f868b759700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.072833 7f8690f64700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072838 7f868ff62700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072842 7f8690763700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072848 7f8689755700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072853 7f86a5610700 10 _do_op 0xc930660 seq 3922 r = 0, finisher 0xbccc520 0 2014-06-09 16:22:12.072854 7f868bf5a700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072859 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd793e0] 2014-06-09 16:22:12.072862 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd793e0] 2014-06-09 16:22:12.072863 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.072863 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.072866 7f8689f56700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072870 7f868af58700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072875 7f868d75d700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072881 7f8691765700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072886 7f868a757700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072895 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2878c0 (1 active) 2014-06-09 16:22:12.072896 7f8692767700 10 op_submit_start 3971 2014-06-09 16:22:12.072896 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189abee0] 2014-06-09 16:22:12.072898 7f8692767700 5 queue_transactions (trailing journal) 3971 2014-06-09 16:22:12.072898 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286d80 (2 active) 2014-06-09 16:22:12.072901 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189abee0] 2014-06-09 16:22:12.072899 7f8692767700 5 queue_op 0xcca7260 seq 3971 osr(1.14c 0x384c810) 828 bytes (queue has 50 ops and 927755 bytes) 2014-06-09 16:22:12.072901 7f86a5e11700 5 _do_op 0xc64a8a0 seq 3923 osr(3.4d 0x3ed0300)/0x3ed0300 start 2014-06-09 16:22:12.072903 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.072903 7f8692767700 10 op_submit_finish 3971 2014-06-09 16:22:12.072904 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.072904 7f86a5e11700 10 _do_transaction on 0x338bc80 2014-06-09 16:22:12.072910 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.072906 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.48f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/211) 2014-06-09 16:22:12.072935 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.072937 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.072942 7f8692767700 5 queue_transactions existing osr(2.48f 0x2ed1450)/0x2ed1450 2014-06-09 16:22:12.072944 7f8692767700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072946 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3923.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.072949 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.072948 7f868f761700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072953 7f868e75f700 2 waiting 51 > 50 ops || 928613 > 104857600 2014-06-09 16:22:12.072957 7f8692f68700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072958 7f86a5e11700 10 _do_op 0xc64a8a0 seq 3923 r = 0, finisher 0xcdb3ec0 0 2014-06-09 16:22:12.072961 7f868df5e700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.072962 7f86a5e11700 10 _finish_op 0xc64a8a0 seq 3923 osr(3.4d 0x3ed0300)/0x3ed0300 2014-06-09 16:22:12.072965 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x306cedc0] 2014-06-09 16:22:12.072970 7f8692767700 10 op_submit_start 3972 2014-06-09 16:22:12.072971 7f868cf5c700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.072972 7f8692767700 5 queue_transactions (trailing journal) 3972 2014-06-09 16:22:12.072971 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286d80 (1 active) 2014-06-09 16:22:12.072973 7f8692767700 5 queue_op 0xcca6a80 seq 3972 osr(2.48f 0x2ed1450) 828 bytes (queue has 50 ops and 927725 bytes) 2014-06-09 16:22:12.072976 7f868b759700 2 waiting 51 > 50 ops || 928551 > 104857600 2014-06-09 16:22:12.072968 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.072973 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b7a0 (2 active) 2014-06-09 16:22:12.072985 7f86a5e11700 5 _do_op 0x309a3620 seq 3924 osr(0.765 0x384c360)/0x384c360 start 2014-06-09 16:22:12.072987 7f86a5e11700 10 _do_transaction on 0xc143200 2014-06-09 16:22:12.072981 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.072988 7f8692767700 10 op_submit_finish 3972 2014-06-09 16:22:12.072992 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.072991 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.4d( v 213'504 (0'0,213'504] local-les=243 n=102 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:857 2014-06-09 16:22:12.073001 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x306cedc0] 2014-06-09 16:22:12.073002 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.073003 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.072992 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.25a( v 214'581 lc 214'578 (0'0,214'581] local-les=243 n=113 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.073007 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcdb3ec0] 2014-06-09 16:22:12.073011 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcdb3ec0] 2014-06-09 16:22:12.073012 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.073013 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.073016 7f8690f64700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073008 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.073021 7f868ef60700 2 waiting 51 > 50 ops || 928778 > 104857600 2014-06-09 16:22:12.073023 7f86a5610700 10 _finish_op 0xc930660 seq 3922 osr(2.c6 0x2ed0490)/0x2ed0490 2014-06-09 16:22:12.073025 7f8690763700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073018 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.073031 7f8689755700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073028 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.073035 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.073037 7f868bf5a700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073042 7f8689f56700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073045 7f8692767700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.073048 7f868af58700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073054 7f868d75d700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073055 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3924.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.073059 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.073063 7f8691765700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073070 7f86a5e11700 10 _do_op 0x309a3620 seq 3924 r = 0, finisher 0xccf8ec0 0 2014-06-09 16:22:12.073063 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:973 2014-06-09 16:22:12.073079 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.073086 7f868a757700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073092 7f8691f66700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073083 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 243/203 242/242/202) 2014-06-09 16:22:12.073094 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006d500] 2014-06-09 16:22:12.073095 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006d500] 2014-06-09 16:22:12.073096 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.073097 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.073114 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.073118 7f8692767700 5 queue_transactions existing osr(3.527 0x3ed1660)/0x3ed1660 2014-06-09 16:22:12.073120 7f8692767700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073184 7f868f761700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073189 7f868e75f700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.073194 7f8692f68700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073199 7f868df5e700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073203 7f868c75b700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073209 7f868ff62700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073217 7f868b759700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073219 7f868cf5c700 10 op_submit_start 3973 2014-06-09 16:22:12.073221 7f868cf5c700 5 queue_transactions (trailing journal) 3973 2014-06-09 16:22:12.073221 7f8690f64700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073223 7f868cf5c700 5 queue_op 0x30060a20 seq 3973 osr(3.3a8 0x3ed0f80) 828 bytes (queue has 50 ops and 927727 bytes) 2014-06-09 16:22:12.073225 7f8690763700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073227 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbccc520] 2014-06-09 16:22:12.073230 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbccc520] 2014-06-09 16:22:12.073232 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.073233 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.073232 7f8689755700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073235 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359320 (1 active) 2014-06-09 16:22:12.073236 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c06c0 (2 active) 2014-06-09 16:22:12.073237 7f868bf5a700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073239 7f86a5610700 5 _do_op 0x288f9980 seq 3925 osr(2.48d 0x2ed1470)/0x2ed1470 start 2014-06-09 16:22:12.073240 7f86a5e11700 10 _finish_op 0x309a3620 seq 3924 osr(0.765 0x384c360)/0x384c360 2014-06-09 16:22:12.073244 7f86a5610700 10 _do_transaction on 0xc1f69c0 2014-06-09 16:22:12.073250 7f868d75d700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073252 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b7a0 (1 active) 2014-06-09 16:22:12.073254 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf8ec0] 2014-06-09 16:22:12.073255 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4acea0 (2 active) 2014-06-09 16:22:12.073255 7f8691765700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf8ec0] 2014-06-09 16:22:12.073259 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.073258 7f86a5e11700 5 _do_op 0x324b3740 seq 3926 osr(1.408 0x384d770)/0x384d770 start 2014-06-09 16:22:12.073260 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.073260 7f86a5e11700 10 _do_transaction on 0x3128900 2014-06-09 16:22:12.073261 7f868cf5c700 10 op_submit_finish 3973 2014-06-09 16:22:12.073263 7f868a757700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073266 7f868af58700 10 op_submit_start 3974 2014-06-09 16:22:12.073267 7f868af58700 5 queue_transactions (trailing journal) 3974 2014-06-09 16:22:12.073269 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.073269 7f868af58700 5 queue_op 0x324cc300 seq 3974 osr(1.3cb 0x384d4f0) 828 bytes (queue has 50 ops and 927727 bytes) 2014-06-09 16:22:12.073272 7f868af58700 10 op_submit_finish 3974 2014-06-09 16:22:12.073267 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.c6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.c6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.073293 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (19 active) 2014-06-09 16:22:12.073300 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.073305 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.073306 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.073311 7f868cf5c700 5 queue_transactions existing osr(2.c6 0x2ed0490)/0x2ed0490 2014-06-09 16:22:12.073313 7f868cf5c700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073305 7f868af58700 10 osd.26 243 dequeue_op 0x30c6ce00 prio 196 cost 0 latency 0.005938 osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) v2 pg pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.073318 7f8691f66700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073318 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3925.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.073321 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.073324 7f8692767700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073329 7f868f761700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073330 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.073333 7f86a5610700 10 _do_op 0x288f9980 seq 3925 r = 0, finisher 0x3006d640 0 2014-06-09 16:22:12.073335 7f868e75f700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073324 7f868af58700 5 -- op tracker -- , seq: 3626, time: 2014-06-09 16:22:12.073320, event: reached_pg, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.073342 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006d6c0] 2014-06-09 16:22:12.073344 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006d6c0] 2014-06-09 16:22:12.073345 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.073346 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.073349 7f86a5610700 10 _finish_op 0x288f9980 seq 3925 osr(2.48d 0x2ed1470)/0x2ed1470 2014-06-09 16:22:12.073343 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x30c6ce00 2014-06-09 16:22:12.073360 7f868df5e700 10 op_submit_start 3975 2014-06-09 16:22:12.073362 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006d640] 2014-06-09 16:22:12.073362 7f868df5e700 5 queue_transactions (trailing journal) 3975 2014-06-09 16:22:12.073362 7f868c75b700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073360 7f868af58700 5 -- op tracker -- , seq: 3626, time: 2014-06-09 16:22:12.073356, event: started, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.073364 7f868df5e700 5 queue_op 0xbf7ff20 seq 3975 osr(2.654 0x2ed1fd0) 828 bytes (queue has 50 ops and 927727 bytes) 2014-06-09 16:22:12.073366 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006d640] 2014-06-09 16:22:12.073367 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.073368 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.073369 7f868ff62700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073371 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c06c0 (1 active) 2014-06-09 16:22:12.073373 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301b00 (2 active) 2014-06-09 16:22:12.073373 7f868ef60700 2 waiting 51 > 50 ops || 928780 > 104857600 2014-06-09 16:22:12.073366 7f868af58700 7 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 24 op ack_type 4 from 9 2014-06-09 16:22:12.073378 7f868df5e700 10 op_submit_finish 3975 2014-06-09 16:22:12.073377 7f8689f56700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073381 7f8690f64700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073385 7f86a5610700 5 _do_op 0x30d97440 seq 3927 osr(2.38a 0x2ed10f0)/0x2ed10f0 start 2014-06-09 16:22:12.073388 7f86a5610700 10 _do_transaction on 0xc262300 2014-06-09 16:22:12.073386 7f868af58700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.073379, event: sub_op_commit_rec, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.073381 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.98( v 213'611 (0'0,213'611] local-les=243 n=118 ec=118 les/c 243/243 242/242/212) 2014-06-09 16:22:12.073392 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.073393 7f868af58700 10 osd.26 243 dequeue_op 0x30c6ce00 finish 2014-06-09 16:22:12.073400 7f868af58700 5 -- op tracker -- , seq: 3626, time: 2014-06-09 16:22:12.073397, event: done, op: osd_sub_op_reply(client.4571.0:35826 3.2b3 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.073411 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.073412 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.073415 7f868df5e700 5 queue_transactions existing osr(3.98 0x3ed0620)/0x3ed0620 2014-06-09 16:22:12.073417 7f868df5e700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073420 7f8690763700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073424 7f8689755700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073429 7f868bf5a700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073435 7f868b759700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073441 7f8691765700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073442 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.073444 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xc711a90 (20 active) 2014-06-09 16:22:12.073446 7f868a757700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073452 7f868cf5c700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073456 7f8691f66700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073448 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.27c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.27c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.073456 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3926.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.073461 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.073481 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.073485 7f868af58700 5 queue_transactions existing osr(1.27c 0x384cf30)/0x384cf30 2014-06-09 16:22:12.073487 7f868af58700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073486 7f86a5e11700 10 _do_op 0x324b3740 seq 3926 r = 0, finisher 0xcd0f6c0 0 2014-06-09 16:22:12.073492 7f8692f68700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073493 7f86a5e11700 10 _finish_op 0x324b3740 seq 3926 osr(1.408 0x384d770)/0x384d770 2014-06-09 16:22:12.073494 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3045b100] 2014-06-09 16:22:12.073497 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3045b100] 2014-06-09 16:22:12.073498 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.073500 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.073504 7f868e75f700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073505 7f868c75b700 10 op_submit_start 3976 2014-06-09 16:22:12.073507 7f868c75b700 5 queue_transactions (trailing journal) 3976 2014-06-09 16:22:12.073509 7f868f761700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073508 7f868c75b700 5 queue_op 0x16fd5020 seq 3976 osr(2.350 0x2ed0e50) 828 bytes (queue has 50 ops and 927727 bytes) 2014-06-09 16:22:12.073512 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd0f6c0] 2014-06-09 16:22:12.073514 7f868ff62700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073516 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd0f6c0] 2014-06-09 16:22:12.073518 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.073518 7f868ef60700 2 waiting 51 > 50 ops || 928780 > 104857600 2014-06-09 16:22:12.073519 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.073511 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4acea0 (1 active) 2014-06-09 16:22:12.073522 7f8689f56700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073523 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62fe60 (2 active) 2014-06-09 16:22:12.073526 7f8690f64700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073530 7f868df5e700 2 waiting 51 > 50 ops || 928553 > 104857600 2014-06-09 16:22:12.073530 7f868c75b700 10 op_submit_finish 3976 2014-06-09 16:22:12.073534 7f8690763700 2 waiting 51 > 50 ops || 928555 > 104857600 2014-06-09 16:22:12.073527 7f86a5e11700 5 _do_op 0x30a3e00 seq 3928 osr(1.c6 0x384c610)/0x384c610 start 2014-06-09 16:22:12.073540 7f86a5e11700 10 _do_transaction on 0xc015500 2014-06-09 16:22:12.073534 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.073545 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.073553 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.073549 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.073559 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.073566 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3927.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.073570 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.073568 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.073577 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.073590 7f86a5610700 10 _do_op 0x30d97440 seq 3927 r = 0, finisher 0x324d03c0 0 2014-06-09 16:22:12.073586 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.073600 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d02c0] 2014-06-09 16:22:12.073602 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d02c0] 2014-06-09 16:22:12.073603 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.073604 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.073607 7f86a5610700 10 _finish_op 0x30d97440 seq 3927 osr(2.38a 0x2ed10f0)/0x2ed10f0 2014-06-09 16:22:12.073602 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.073615 7f868e75f700 10 op_submit_start 3977 2014-06-09 16:22:12.073617 7f868e75f700 5 queue_transactions (trailing journal) 3977 2014-06-09 16:22:12.073616 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d03c0] 2014-06-09 16:22:12.073612 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.073618 7f868e75f700 5 queue_op 0xc34b980 seq 3977 osr(1.f2 0x384c990) 858 bytes (queue has 50 ops and 927757 bytes) 2014-06-09 16:22:12.073620 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d03c0] 2014-06-09 16:22:12.073621 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.073622 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.073625 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301b00 (1 active) 2014-06-09 16:22:12.073627 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780120 (2 active) 2014-06-09 16:22:12.073621 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.420( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.073631 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.073631 7f86a5610700 5 _do_op 0x30a2d80 seq 3929 osr(2.5f2 0x2ed1df0)/0x2ed1df0 start 2014-06-09 16:22:12.073632 7f868e75f700 10 op_submit_finish 3977 2014-06-09 16:22:12.073634 7f86a5610700 10 _do_transaction on 0xbf46a80 2014-06-09 16:22:12.073638 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.073636 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.073637 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.27( v 215'668 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:12.073651 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.038069 7 0.000391 2014-06-09 16:22:12.073672 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.073666 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.073677 7f868c75b700 5 queue_transactions existing osr(3.27 0x3ed0340)/0x3ed0340 2014-06-09 16:22:12.073679 7f868c75b700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.073678 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.073684 7f868bf5a700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073689 7f8689755700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073686 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.073694 7f8691765700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073699 7f868a757700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073696 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.073705 7f868cf5c700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.073705 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.073710 7f8691f66700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073712 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.073716 7f868af58700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073722 7f8692f68700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073725 7f8692767700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073721 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.073730 7f868d75d700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073736 7f868f761700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073728 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.073741 7f868ff62700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073739 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.073752 7f868e75f700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.073764 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1076 2014-06-09 16:22:12.073796 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.073799 7f868e75f700 5 queue_transactions existing osr(3.310 0x3ed0d80)/0x3ed0d80 2014-06-09 16:22:12.073801 7f868e75f700 2 waiting 51 > 50 ops || 928618 > 104857600 2014-06-09 16:22:12.073804 7f868ef60700 2 waiting 51 > 50 ops || 928810 > 104857600 2014-06-09 16:22:12.073808 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.073808 7f8689f56700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073814 7f8690f64700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073818 7f868df5e700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.073824 7f8690763700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073829 7f868b759700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.073830 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3928.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.073834 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.073844 7f86a5e11700 10 _do_op 0x30a3e00 seq 3928 r = 0, finisher 0x3042b2e0 0 2014-06-09 16:22:12.073849 7f86a5e11700 10 _finish_op 0x30a3e00 seq 3928 osr(1.c6 0x384c610)/0x384c610 2014-06-09 16:22:12.073859 7f868c75b700 10 op_submit_start 3978 2014-06-09 16:22:12.073860 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b2e0] 2014-06-09 16:22:12.073862 7f868c75b700 5 queue_transactions (trailing journal) 3978 2014-06-09 16:22:12.073864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b2e0] 2014-06-09 16:22:12.073863 7f868c75b700 5 queue_op 0xbf7eb40 seq 3978 osr(3.27 0x3ed0340) 826 bytes (queue has 50 ops and 927757 bytes) 2014-06-09 16:22:12.073865 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.073867 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.073868 7f868bf5a700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073868 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b2a0] 2014-06-09 16:22:12.073869 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62fe60 (1 active) 2014-06-09 16:22:12.073871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b2a0] 2014-06-09 16:22:12.073872 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7560 (2 active) 2014-06-09 16:22:12.073872 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.073873 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.073873 7f8689755700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.073875 7f86a5e11700 5 _do_op 0x32503e00 seq 3930 osr(2.487 0x2ed14b0)/0x2ed14b0 start 2014-06-09 16:22:12.073878 7f86a5e11700 10 _do_transaction on 0xc1258c0 2014-06-09 16:22:12.073879 7f868c75b700 10 op_submit_finish 3978 2014-06-09 16:22:12.073883 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.073882 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.073896 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.073905 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.073905 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.073913 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.073919 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3929.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.073923 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.073921 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.073930 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.073943 7f86a5610700 10 _do_op 0x30a2d80 seq 3929 r = 0, finisher 0x3206ed00 0 2014-06-09 16:22:12.073943 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.073952 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd0e2e0] 2014-06-09 16:22:12.073955 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd0e2e0] 2014-06-09 16:22:12.073956 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.073957 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.073953 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.073960 7f86a5610700 10 _finish_op 0x30a2d80 seq 3929 osr(2.5f2 0x2ed1df0)/0x2ed1df0 2014-06-09 16:22:12.073961 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.073970 7f868a757700 10 op_submit_start 3979 2014-06-09 16:22:12.073971 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.073971 7f868a757700 5 queue_transactions (trailing journal) 3979 2014-06-09 16:22:12.073971 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.073972 7f868a757700 5 queue_op 0xcabcb40 seq 3979 osr(3.218 0x3ed0cd0) 828 bytes (queue has 50 ops and 927757 bytes) 2014-06-09 16:22:12.073979 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780120 (1 active) 2014-06-09 16:22:12.073981 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3206ed00] 2014-06-09 16:22:12.073982 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114360 (2 active) 2014-06-09 16:22:12.073984 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3206ed00] 2014-06-09 16:22:12.073986 7f868a757700 10 op_submit_finish 3979 2014-06-09 16:22:12.073987 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.073975 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.b5( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.073988 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.073997 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3930.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.073990 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.074001 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.074008 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.074003 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.019837 6 0.009935 2014-06-09 16:22:12.074011 7f86a5e11700 10 _do_op 0x32503e00 seq 3930 r = 0, finisher 0x32481a40 0 2014-06-09 16:22:12.074013 7f868c75b700 5 queue_transactions existing osr(1.b5 0x384c690)/0x384c690 2014-06-09 16:22:12.074015 7f868c75b700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074016 7f86a5e11700 10 _finish_op 0x32503e00 seq 3930 osr(2.487 0x2ed14b0)/0x2ed14b0 2014-06-09 16:22:12.074015 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.074030 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32481a40] 2014-06-09 16:22:12.074030 7f8691f66700 10 op_submit_start 3980 2014-06-09 16:22:12.074032 7f8691f66700 5 queue_transactions (trailing journal) 3980 2014-06-09 16:22:12.074025 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.074031 7f868af58700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074033 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32481a40] 2014-06-09 16:22:12.074035 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074036 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074034 7f8691f66700 5 queue_op 0x3250e300 seq 3980 osr(3.27b 0x3ed0bd0) 828 bytes (queue has 50 ops and 927757 bytes) 2014-06-09 16:22:12.074037 7f8692f68700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074038 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7560 (1 active) 2014-06-09 16:22:12.074040 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d560 (2 active) 2014-06-09 16:22:12.074041 7f8692767700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074035 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.074044 7f86a5e11700 5 _do_op 0xbf7f440 seq 3932 osr(3.381 0x3ed0fe0)/0x3ed0fe0 start 2014-06-09 16:22:12.074045 7f868d75d700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074046 7f8691f66700 10 op_submit_finish 3980 2014-06-09 16:22:12.074047 7f86a5e11700 10 _do_transaction on 0xc146000 2014-06-09 16:22:12.074049 7f868f761700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074051 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074045 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.074053 7f868ff62700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074058 7f868e75f700 2 waiting 51 > 50 ops || 928618 > 104857600 2014-06-09 16:22:12.074054 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.074049 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.517( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.074063 7f868ef60700 2 waiting 51 > 50 ops || 928810 > 104857600 2014-06-09 16:22:12.074061 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.074071 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.074086 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.074082 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.074091 7f8691f66700 5 queue_transactions existing osr(1.517 0x384db70)/0x384db70 2014-06-09 16:22:12.074093 7f8691f66700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074097 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074091 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.074100 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.074110 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3932.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074114 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.074109 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.074121 7f868a757700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.074132 7f86a5e11700 10 _do_op 0xbf7f440 seq 3932 r = 0, finisher 0x1b0c06c0 0 2014-06-09 16:22:12.074135 7f86a5e11700 10 _finish_op 0xbf7f440 seq 3932 osr(3.381 0x3ed0fe0)/0x3ed0fe0 2014-06-09 16:22:12.074144 7f868af58700 10 op_submit_start 3981 2014-06-09 16:22:12.074138 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.074146 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1b0c06c0] 2014-06-09 16:22:12.074147 7f868af58700 5 queue_transactions (trailing journal) 3981 2014-06-09 16:22:12.074148 7f868af58700 5 queue_op 0x32816180 seq 3981 osr(1.27c 0x384cf30) 828 bytes (queue has 50 ops and 927757 bytes) 2014-06-09 16:22:12.074151 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1b0c06c0] 2014-06-09 16:22:12.074153 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074153 7f8690f64700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074154 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074158 7f868df5e700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074162 7f8690763700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074166 7f868b759700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074169 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.074170 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d560 (1 active) 2014-06-09 16:22:12.074172 7f8689755700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074173 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcb40 (2 active) 2014-06-09 16:22:12.074174 7f868a757700 5 queue_transactions existing osr(1.327 0x384d370)/0x384d370 2014-06-09 16:22:12.074176 7f868a757700 2 waiting 51 > 50 ops || 928618 > 104857600 2014-06-09 16:22:12.074176 7f86a5e11700 5 _do_op 0x2ea52780 seq 3933 osr(2.30b 0x2ed0f50)/0x2ed0f50 start 2014-06-09 16:22:12.074178 7f868af58700 10 op_submit_finish 3981 2014-06-09 16:22:12.074179 7f86a5e11700 10 _do_transaction on 0x97bb8c0 2014-06-09 16:22:12.074181 7f8691765700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074184 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074186 7f868cf5c700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074191 7f868c75b700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074182 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.616( v 213'629 (0'0,213'629] local-les=243 n=125 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:12.074203 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31bfe820,0xc168ac0] 2014-06-09 16:22:12.074209 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31bfe820,0xc168ac0] 2014-06-09 16:22:12.074211 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.074212 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.074215 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.074215 7f8689f56700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074219 7f868af58700 5 queue_transactions existing osr(3.616 0x3ed1a90)/0x3ed1a90 2014-06-09 16:22:12.074220 7f86a5610700 5 _do_op 0x2fcae2a0 seq 3931 osr(0.b9 0x2bd6400)/0x2bd6400 start 2014-06-09 16:22:12.074222 7f868af58700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074223 7f86a5610700 10 _do_transaction on 0xc2efe00 2014-06-09 16:22:12.074226 7f8692f68700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074228 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074231 7f8692767700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074235 7f868f761700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074239 7f868d75d700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074243 7f868ff62700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074247 7f868ef60700 2 waiting 51 > 50 ops || 928810 > 104857600 2014-06-09 16:22:12.074251 7f8691f66700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074258 7f868bf5a700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074228 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074263 7f868e75f700 2 waiting 51 > 50 ops || 928618 > 104857600 2014-06-09 16:22:12.074276 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3933.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074280 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.074290 7f86a5e11700 10 _do_op 0x2ea52780 seq 3933 r = 0, finisher 0xc169ee0 0 2014-06-09 16:22:12.074296 7f86a5e11700 10 _finish_op 0x2ea52780 seq 3933 osr(2.30b 0x2ed0f50)/0x2ed0f50 2014-06-09 16:22:12.074297 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xccf8840] 2014-06-09 16:22:12.074300 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xccf8840] 2014-06-09 16:22:12.074301 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.074303 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.074304 7f8690f64700 10 op_submit_start 3982 2014-06-09 16:22:12.074305 7f868df5e700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074307 7f8690f64700 5 queue_transactions (trailing journal) 3982 2014-06-09 16:22:12.074308 7f8690f64700 5 queue_op 0x30a25a0 seq 3982 osr(3.537 0x3ed1620) 828 bytes (queue has 50 ops and 927757 bytes) 2014-06-09 16:22:12.074310 7f8690763700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074314 7f868b759700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074319 7f8689755700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074320 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc169ee0] 2014-06-09 16:22:12.074324 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc169ee0] 2014-06-09 16:22:12.074326 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074325 7f868a757700 2 waiting 51 > 50 ops || 928618 > 104857600 2014-06-09 16:22:12.074327 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074329 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcb40 (1 active) 2014-06-09 16:22:12.074331 7f8691765700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074332 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5257a0 (2 active) 2014-06-09 16:22:12.074336 7f868cf5c700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074336 7f8690f64700 10 op_submit_finish 3982 2014-06-09 16:22:12.074335 7f86a5e11700 5 _do_op 0xc704d80 seq 3934 osr(2.3e8 0x2ed1310)/0x2ed1310 start 2014-06-09 16:22:12.074341 7f868c75b700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074341 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074339 7f86a5e11700 10 _do_transaction on 0x3415200 2014-06-09 16:22:12.074346 7f8689f56700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074340 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.2ef( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.074351 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074361 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3931.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074366 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.074374 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.074378 7f8690f64700 5 queue_transactions existing osr(1.2ef 0x384d0f0)/0x384d0f0 2014-06-09 16:22:12.074380 7f8690f64700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074385 7f868af58700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074389 7f8692f68700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074390 7f86a5610700 10 _do_op 0x2fcae2a0 seq 3931 r = 0, finisher 0x15a3e360 0 2014-06-09 16:22:12.074393 7f8692767700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074397 7f868f761700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074400 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x320743c0] 2014-06-09 16:22:12.074401 7f868d75d700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074402 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x320743c0] 2014-06-09 16:22:12.074404 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.074405 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.074405 7f868ff62700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074408 7f86a5610700 10 _finish_op 0x2fcae2a0 seq 3931 osr(0.b9 0x2bd6400)/0x2bd6400 2014-06-09 16:22:12.074409 7f868ef60700 2 waiting 51 > 50 ops || 928810 > 104857600 2014-06-09 16:22:12.074414 7f8691f66700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074417 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074419 7f868bf5a700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074425 7f868e75f700 2 waiting 51 > 50 ops || 928618 > 104857600 2014-06-09 16:22:12.074436 7f868df5e700 10 op_submit_start 3983 2014-06-09 16:22:12.074437 7f868df5e700 5 queue_transactions (trailing journal) 3983 2014-06-09 16:22:12.074439 7f868df5e700 5 queue_op 0xbf7fe60 seq 3983 osr(3.98 0x3ed0620) 826 bytes (queue has 50 ops and 927757 bytes) 2014-06-09 16:22:12.074440 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x15a3e360] 2014-06-09 16:22:12.074443 7f8690763700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074444 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x15a3e360] 2014-06-09 16:22:12.074445 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074446 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074448 7f868b759700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074449 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114360 (1 active) 2014-06-09 16:22:12.074452 7f868a757700 2 waiting 51 > 50 ops || 928618 > 104857600 2014-06-09 16:22:12.074430 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3934.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074457 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.074457 7f8691765700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074451 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d05a0 (2 active) 2014-06-09 16:22:12.074462 7f86a5610700 5 _do_op 0x30d96c60 seq 3935 osr(0.578 0x2bd7800)/0x2bd7800 start 2014-06-09 16:22:12.074463 7f868df5e700 10 op_submit_finish 3983 2014-06-09 16:22:12.074463 7f8689755700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074465 7f86a5610700 10 _do_transaction on 0x471d5c0 2014-06-09 16:22:12.074468 7f868cf5c700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074473 7f8689f56700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074470 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074467 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.579( v 213'720 (0'0,213'720] local-les=243 n=144 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:12.074479 7f8690f64700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074484 7f868af58700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074488 7f8692f68700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074492 7f8692767700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074495 7f868f761700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074499 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.074499 7f868d75d700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074504 7f868ff62700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074505 7f868df5e700 5 queue_transactions existing osr(3.579 0x3ed18d0)/0x3ed18d0 2014-06-09 16:22:12.074508 7f868ef60700 2 waiting 51 > 50 ops || 928810 > 104857600 2014-06-09 16:22:12.074511 7f868c75b700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074517 7f868bf5a700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074523 7f868e75f700 2 waiting 51 > 50 ops || 928618 > 104857600 2014-06-09 16:22:12.074525 7f86a5e11700 10 _do_op 0xc704d80 seq 3934 r = 0, finisher 0x31479e40 0 2014-06-09 16:22:12.074531 7f868df5e700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074536 7f8691f66700 2 waiting 51 > 50 ops || 928585 > 104857600 2014-06-09 16:22:12.074537 7f86a5e11700 10 _finish_op 0xc704d80 seq 3934 osr(2.3e8 0x2ed1310)/0x2ed1310 2014-06-09 16:22:12.074538 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd73720] 2014-06-09 16:22:12.074542 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd73720] 2014-06-09 16:22:12.074544 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.074545 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.074546 7f868b759700 10 op_submit_start 3984 2014-06-09 16:22:12.074548 7f868b759700 5 queue_transactions (trailing journal) 3984 2014-06-09 16:22:12.074547 7f868a757700 2 waiting 51 > 50 ops || 928616 > 104857600 2014-06-09 16:22:12.074549 7f868b759700 5 queue_op 0xc88ae40 seq 3984 osr(1.c9 0x384c9f0) 826 bytes (queue has 50 ops and 927755 bytes) 2014-06-09 16:22:12.074552 7f8691765700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074556 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5257a0 (1 active) 2014-06-09 16:22:12.074557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31479e40] 2014-06-09 16:22:12.074557 7f8689755700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074558 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd9e0 (2 active) 2014-06-09 16:22:12.074560 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31479e40] 2014-06-09 16:22:12.074561 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074562 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074562 7f868b759700 10 op_submit_finish 3984 2014-06-09 16:22:12.074562 7f868cf5c700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.074567 7f8689f56700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074572 7f8690f64700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074566 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.124( v 213'648 (0'0,213'648] local-les=243 n=125 ec=118 les/c 243/243 242/242/222) 2014-06-09 16:22:12.074577 7f868af58700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074581 7f8692f68700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074585 7f8692767700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074589 7f868f761700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074592 7f868d75d700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074594 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074596 7f868ff62700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074599 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.074600 7f868ef60700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.074603 7f868b759700 5 queue_transactions existing osr(3.124 0x3ed07c0)/0x3ed07c0 2014-06-09 16:22:12.074606 7f868b759700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074608 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3935.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074610 7f868c75b700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.074612 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.074616 7f86a5e11700 5 _do_op 0x324c1320 seq 3936 osr(0.3a6 0x2bd7330)/0x2bd7330 start 2014-06-09 16:22:12.074616 7f868bf5a700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074618 7f86a5e11700 10 _do_transaction on 0xc143140 2014-06-09 16:22:12.074622 7f868e75f700 2 waiting 51 > 50 ops || 928616 > 104857600 2014-06-09 16:22:12.074623 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074627 7f868df5e700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074630 7f86a5610700 10 _do_op 0x30d96c60 seq 3935 r = 0, finisher 0x3006d7c0 0 2014-06-09 16:22:12.074632 7f8691f66700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074639 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a7360] 2014-06-09 16:22:12.074641 7f8690763700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074644 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a7360] 2014-06-09 16:22:12.074645 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.074646 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.074649 7f86a5610700 10 _finish_op 0x30d96c60 seq 3935 osr(0.578 0x2bd7800)/0x2bd7800 2014-06-09 16:22:12.074660 7f8691765700 10 op_submit_start 3985 2014-06-09 16:22:12.074662 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006d7c0] 2014-06-09 16:22:12.074662 7f8691765700 5 queue_transactions (trailing journal) 3985 2014-06-09 16:22:12.074665 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006d7c0] 2014-06-09 16:22:12.074666 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074664 7f8691765700 5 queue_op 0x15997a40 seq 3985 osr(0.71b 0x384c050) 828 bytes (queue has 50 ops and 927755 bytes) 2014-06-09 16:22:12.074667 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074670 7f8689755700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074670 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d05a0 (1 active) 2014-06-09 16:22:12.074673 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2878c0 (2 active) 2014-06-09 16:22:12.074673 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074675 7f868cf5c700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.074680 7f8689f56700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074680 7f8691765700 10 op_submit_finish 3985 2014-06-09 16:22:12.074676 7f86a5610700 5 _do_op 0x30033a40 seq 3937 osr(2.624 0x2ed1c50)/0x2ed1c50 start 2014-06-09 16:22:12.074685 7f8690f64700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074686 7f86a5610700 10 _do_transaction on 0xc140f00 2014-06-09 16:22:12.074686 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3936.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074690 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.074691 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074685 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.654( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.074719 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.074723 7f8691765700 5 queue_transactions existing osr(0.654 0x2bd7b50)/0x2bd7b50 2014-06-09 16:22:12.074725 7f8691765700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074728 7f86a5e11700 10 _do_op 0x324c1320 seq 3936 r = 0, finisher 0xccf95c0 0 2014-06-09 16:22:12.074735 7f86a5e11700 10 _finish_op 0x324c1320 seq 3936 osr(0.3a6 0x2bd7330)/0x2bd7330 2014-06-09 16:22:12.074736 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002320] 2014-06-09 16:22:12.074739 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002320] 2014-06-09 16:22:12.074741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.074742 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.074742 7f8689755700 10 op_submit_start 3986 2014-06-09 16:22:12.074743 7f8689755700 5 queue_transactions (trailing journal) 3986 2014-06-09 16:22:12.074745 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf95c0] 2014-06-09 16:22:12.074745 7f8689755700 5 queue_op 0x309a35c0 seq 3986 osr(0.33f 0x2bd7060) 828 bytes (queue has 50 ops and 927755 bytes) 2014-06-09 16:22:12.074749 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf95c0] 2014-06-09 16:22:12.074750 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074754 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd9e0 (1 active) 2014-06-09 16:22:12.074756 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fac60 (2 active) 2014-06-09 16:22:12.074757 7f868af58700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074760 7f86a5e11700 5 _do_op 0xbf7e600 seq 3938 osr(2.52e 0x2ed1600)/0x2ed1600 start 2014-06-09 16:22:12.074762 7f8689755700 10 op_submit_finish 3986 2014-06-09 16:22:12.074762 7f8692f68700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074763 7f86a5e11700 10 _do_transaction on 0xc0d5e00 2014-06-09 16:22:12.074766 7f8692767700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074767 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074768 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074770 7f868f761700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074774 7f868d75d700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074765 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.a7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.a7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.074777 7f868ff62700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074782 7f868ef60700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.074780 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3937.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074785 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.074786 7f868c75b700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.074800 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.074805 7f8689755700 5 queue_transactions existing osr(1.a7 0x384c730)/0x384c730 2014-06-09 16:22:12.074807 7f8689755700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.074812 7f868bf5a700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074818 7f868e75f700 2 waiting 51 > 50 ops || 928616 > 104857600 2014-06-09 16:22:12.074823 7f868df5e700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074828 7f8691f66700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074833 7f8690763700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074832 7f86a5610700 10 _do_op 0x30033a40 seq 3937 r = 0, finisher 0x31412300 0 2014-06-09 16:22:12.074839 7f868a757700 2 waiting 51 > 50 ops || 928616 > 104857600 2014-06-09 16:22:12.074844 7f868cf5c700 2 waiting 51 > 50 ops || 928581 > 104857600 2014-06-09 16:22:12.074847 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6a9e0] 2014-06-09 16:22:12.074849 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6a9e0] 2014-06-09 16:22:12.074851 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.074852 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.074854 7f8689f56700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074860 7f8690f64700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074859 7f86a5610700 10 _finish_op 0x30033a40 seq 3937 osr(2.624 0x2ed1c50)/0x2ed1c50 2014-06-09 16:22:12.074872 7f868af58700 10 op_submit_start 3987 2014-06-09 16:22:12.074872 7f868b759700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074873 7f868af58700 5 queue_transactions (trailing journal) 3987 2014-06-09 16:22:12.074873 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2878c0 (1 active) 2014-06-09 16:22:12.074875 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e480 (2 active) 2014-06-09 16:22:12.074875 7f868af58700 5 queue_op 0x324a5560 seq 3987 osr(3.616 0x3ed1a90) 828 bytes (queue has 50 ops and 927755 bytes) 2014-06-09 16:22:12.074878 7f868af58700 10 op_submit_finish 3987 2014-06-09 16:22:12.074877 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412300] 2014-06-09 16:22:12.074878 7f86a5610700 5 _do_op 0x30a2f60 seq 3939 osr(0.3a9 0x2bd7310)/0x2bd7310 start 2014-06-09 16:22:12.074880 7f86a5610700 10 _do_transaction on 0xbf91bc0 2014-06-09 16:22:12.074882 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412300] 2014-06-09 16:22:12.074883 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074884 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074885 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074869 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074880 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.3a6( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.3a6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.074898 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3938.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074903 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.074911 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.074915 7f868af58700 5 queue_transactions existing osr(0.3a6 0x2bd7330)/0x2bd7330 2014-06-09 16:22:12.074917 7f868af58700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074921 7f8692f68700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074924 7f8692767700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074925 7f86a5e11700 10 _do_op 0xbf7e600 seq 3938 r = 0, finisher 0xbd5bca0 0 2014-06-09 16:22:12.074928 7f868f761700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074931 7f868d75d700 2 waiting 51 > 50 ops || 928583 > 104857600 2014-06-09 16:22:12.074932 7f86a5e11700 10 _finish_op 0xbf7e600 seq 3938 osr(2.52e 0x2ed1600)/0x2ed1600 2014-06-09 16:22:12.074933 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d74a0] 2014-06-09 16:22:12.074935 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d74a0] 2014-06-09 16:22:12.074937 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.074938 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.074943 7f868ef60700 10 op_submit_start 3988 2014-06-09 16:22:12.074944 7f868ef60700 5 queue_transactions (trailing journal) 3988 2014-06-09 16:22:12.074944 7f868c75b700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.074946 7f868ef60700 5 queue_op 0x32503b00 seq 3988 osr(3.18b 0x3ed06a0) 1053 bytes (queue has 50 ops and 927980 bytes) 2014-06-09 16:22:12.074950 7f8689755700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.074951 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5bca0] 2014-06-09 16:22:12.074954 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5bca0] 2014-06-09 16:22:12.074956 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.074955 7f868bf5a700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.074957 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.074960 7f868e75f700 2 waiting 51 > 50 ops || 928841 > 104857600 2014-06-09 16:22:12.074965 7f868df5e700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.074967 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.074969 7f8691f66700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.074969 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fac60 (1 active) 2014-06-09 16:22:12.074975 7f8690763700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.074972 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4079e0 (2 active) 2014-06-09 16:22:12.074980 7f868a757700 2 waiting 51 > 50 ops || 928841 > 104857600 2014-06-09 16:22:12.074982 7f86a5e11700 5 _do_op 0xcca7e60 seq 3940 osr(0.219 0x2bd6ad0)/0x2bd6ad0 start 2014-06-09 16:22:12.074983 7f868ef60700 10 op_submit_finish 3988 2014-06-09 16:22:12.074985 7f868cf5c700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.074985 7f86a5e11700 10 _do_transaction on 0xc28a480 2014-06-09 16:22:12.074990 7f8689f56700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.074991 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.074995 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3939.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.074998 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.074990 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d( v 214'604 (0'0,214'604] local-les=243 n=114 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.6d( v 214'604 (0'0,214'604] local-les=243 n=114 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:12.074998 7f869af78700 5 -- op tracker -- , seq: 3621, time: 2014-06-09 16:22:12.074995, event: done, op: pg_info(1 pgs e243:3.18b) 2014-06-09 16:22:12.075005 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d( v 214'604 (0'0,214'604] local-les=243 n=114 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.075015 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d( v 214'604 (0'0,214'604] local-les=243 n=114 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.075026 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.075029 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.075028 7f86a5610700 10 _do_op 0x30a2f60 seq 3939 r = 0, finisher 0xd9fbb80 0 2014-06-09 16:22:12.075024 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.6d( v 214'604 (0'0,214'604] local-les=243 n=114 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.075037 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xd9fbc00] 2014-06-09 16:22:12.075040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xd9fbc00] 2014-06-09 16:22:12.075041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.075042 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.075045 7f86a5610700 10 _finish_op 0x30a2f60 seq 3939 osr(0.3a9 0x2bd7310)/0x2bd7310 2014-06-09 16:22:12.075051 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e480 (1 active) 2014-06-09 16:22:12.075052 7f868c75b700 10 op_submit_start 3989 2014-06-09 16:22:12.075054 7f868c75b700 5 queue_transactions (trailing journal) 3989 2014-06-09 16:22:12.075054 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115b00 (2 active) 2014-06-09 16:22:12.075054 7f8691765700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.075055 7f868c75b700 5 queue_op 0xbf7e2a0 seq 3989 osr(1.b5 0x384c690) 826 bytes (queue has 50 ops and 927978 bytes) 2014-06-09 16:22:12.075056 7f86a5610700 5 _do_op 0x30a2a20 seq 3941 osr(0.220 0x2bd6a70)/0x2bd6a70 start 2014-06-09 16:22:12.075058 7f868c75b700 10 op_submit_finish 3989 2014-06-09 16:22:12.075059 7f86a5610700 10 _do_transaction on 0xc000900 2014-06-09 16:22:12.075064 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.075061 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.075073 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.075079 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.075084 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.075088 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.075092 7f868ef60700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.075092 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.075101 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:157 2014-06-09 16:22:12.075107 7f868c75b700 20 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.075108 7f868ef60700 15 osd.26 pg_epoch: 243 pg[3.6d( v 214'604 (0'0,214'604] local-les=243 n=114 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1042 2014-06-09 16:22:12.075114 7f868c75b700 20 osd.26 pg_epoch: 243 pg[2.368( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.075123 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.075123 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.075129 7f868ef60700 20 osd.26 243 Sending info 3.424( v 214'730 lc 214'729 (0'0,214'730] local-les=243 n=142 ec=118 les/c 203/203 242/242/242) to shard 4 2014-06-09 16:22:12.075135 7f868ef60700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.075128 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.30b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.30b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.075148 7f868ef60700 20 osd.26 243 Sending info 3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) to shard 8 2014-06-09 16:22:12.075154 7f868ef60700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.075159 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.075164 7f868c75b700 5 queue_transactions existing osr(2.30b 0x2ed0f50)/0x2ed0f50 2014-06-09 16:22:12.075166 7f868c75b700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.075183 7f868ef60700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41ef0 (19 active) 2014-06-09 16:22:12.075188 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xd9fbb80] 2014-06-09 16:22:12.075190 7f868ef60700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.075193 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xd9fbb80] 2014-06-09 16:22:12.075195 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.075195 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.075205 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.075194 7f868ef60700 10 osd.26 243 dequeue_op 0x306a200 prio 63 cost 516096 latency 2.151344 osd_op(client.4572.0:29961 rbd_data.117c6b8b4567.000000000000140a [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~516096] 3.5ac10206 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.075215 7f868ef60700 5 -- op tracker -- , seq: 7, time: 2014-06-09 16:22:12.075211, event: reached_pg, op: osd_op(client.4572.0:29961 rbd_data.117c6b8b4567.000000000000140a [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~516096] 3.5ac10206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.075219 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3940.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.075223 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.075223 7f868ef60700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.075235 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x306a200 2014-06-09 16:22:12.075252 7f86a5e11700 10 _do_op 0xcca7e60 seq 3940 r = 0, finisher 0xc2d72a0 0 2014-06-09 16:22:12.075244 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4572.0:29961 rbd_data.117c6b8b4567.000000000000140a [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~516096] 3.5ac10206 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.075258 7f86a5e11700 10 _finish_op 0xcca7e60 seq 3940 osr(0.219 0x2bd6ad0)/0x2bd6ad0 2014-06-09 16:22:12.075259 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd32de0] 2014-06-09 16:22:12.075263 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd32de0] 2014-06-09 16:22:12.075264 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.075266 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.075268 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d72a0] 2014-06-09 16:22:12.075260 7f868ef60700 3 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4572.0:29961 was 213'591 2014-06-09 16:22:12.075271 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d72a0] 2014-06-09 16:22:12.075273 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.075274 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.075276 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4079e0 (1 active) 2014-06-09 16:22:12.075278 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524fc0 (2 active) 2014-06-09 16:22:12.075281 7f86a5e11700 5 _do_op 0xc88a840 seq 3942 osr(0.25d 0x2bd6a00)/0x2bd6a00 start 2014-06-09 16:22:12.075283 7f86a5e11700 10 _do_transaction on 0xc059800 2014-06-09 16:22:12.075287 7f868ef60700 10 osd.26 243 dequeue_op 0x306a200 finish 2014-06-09 16:22:12.075288 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.075292 7f8691765700 10 op_submit_start 3990 2014-06-09 16:22:12.075294 7f8691765700 5 queue_transactions (trailing journal) 3990 2014-06-09 16:22:12.075295 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.075295 7f8691765700 5 queue_op 0x15997aa0 seq 3990 osr(0.654 0x2bd7b50) 828 bytes (queue has 50 ops and 927978 bytes) 2014-06-09 16:22:12.075298 7f8691765700 10 op_submit_finish 3990 2014-06-09 16:22:12.075296 7f868ef60700 5 -- op tracker -- , seq: 7, time: 2014-06-09 16:22:12.075291, event: done, op: osd_op(client.4572.0:29961 rbd_data.117c6b8b4567.000000000000140a [set-alloc-hint object_size 4194304 write_size 4194304,write 2899968~516096] 3.5ac10206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.075309 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3941.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.075311 7f868ef60700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.075301 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 243/203 242/242/202) 2014-06-09 16:22:12.075313 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.075333 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.075332 7f86a5610700 10 _do_op 0x30a2a20 seq 3941 r = 0, finisher 0xcc83120 0 2014-06-09 16:22:12.075338 7f868ef60700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41ef0 (20 active) 2014-06-09 16:22:12.075338 7f8691765700 5 queue_transactions existing osr(3.f0 0x3ed04a0)/0x3ed04a0 2014-06-09 16:22:12.075341 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2e0ee60] 2014-06-09 16:22:12.075342 7f8692f68700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.079937 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2e0ee60] 2014-06-09 16:22:12.079941 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.079928 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.289( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.289( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.079942 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.075346 7f86a5610700 10 _finish_op 0x30a2a20 seq 3941 osr(0.220 0x2bd6a70)/0x2bd6a70 2014-06-09 16:22:12.079943 7f8692767700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.075358 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.079965 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.079966 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.079968 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.079970 7f869af78700 20 osd.26 243 _dispatch 0x30c76700 pg_info(1 pgs e243:3.1ff) v4 2014-06-09 16:22:12.079972 7f868ef60700 5 queue_transactions existing osr(1.289 0x384ceb0)/0x384ceb0 2014-06-09 16:22:12.079975 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3942.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.079980 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.079977 7f869af78700 5 -- op tracker -- , seq: 3631, time: 2014-06-09 16:22:12.076164, event: header_read, op: pg_info(1 pgs e243:3.1ff) 2014-06-09 16:22:12.079987 7f869af78700 5 -- op tracker -- , seq: 3631, time: 2014-06-09 16:22:12.076165, event: throttled, op: pg_info(1 pgs e243:3.1ff) 2014-06-09 16:22:12.079993 7f86a5e11700 10 _do_op 0xc88a840 seq 3942 r = 0, finisher 0xc1696a0 0 2014-06-09 16:22:12.079995 7f869af78700 5 -- op tracker -- , seq: 3631, time: 2014-06-09 16:22:12.076181, event: all_read, op: pg_info(1 pgs e243:3.1ff) 2014-06-09 16:22:12.080000 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5f980] 2014-06-09 16:22:12.080004 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5f980] 2014-06-09 16:22:12.080002 7f869af78700 5 -- op tracker -- , seq: 3631, time: 2014-06-09 16:22:12.079964, event: dispatched, op: pg_info(1 pgs e243:3.1ff) 2014-06-09 16:22:12.080005 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.080006 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.080008 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115b00 (1 active) 2014-06-09 16:22:12.080009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc83120] 2014-06-09 16:22:12.080010 7f868d75d700 10 op_submit_start 3991 2014-06-09 16:22:12.080010 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652240 (2 active) 2014-06-09 16:22:12.080012 7f868d75d700 5 queue_transactions (trailing journal) 3991 2014-06-09 16:22:12.080009 7f869af78700 5 -- op tracker -- , seq: 3631, time: 2014-06-09 16:22:12.080007, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.1ff) 2014-06-09 16:22:12.080013 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc83120] 2014-06-09 16:22:12.080014 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080013 7f86a5610700 5 _do_op 0x1e08bd40 seq 3943 osr(2.288 0x2ed0d10)/0x2ed0d10 start 2014-06-09 16:22:12.080015 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080013 7f868d75d700 5 queue_op 0x16096a20 seq 3991 osr(1.65e 0x384df30) 828 bytes (queue has 50 ops and 927978 bytes) 2014-06-09 16:22:12.080016 7f86a5610700 10 _do_transaction on 0xc2ee180 2014-06-09 16:22:12.080014 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.1ff) v4 from osd.8 2014-06-09 16:22:12.080017 7f868d75d700 10 op_submit_finish 3991 2014-06-09 16:22:12.080018 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c76700 2014-06-09 16:22:12.080023 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080023 7f869af78700 5 -- op tracker -- , seq: 3631, time: 2014-06-09 16:22:12.080021, event: started, op: pg_info(1 pgs e243:3.1ff) 2014-06-09 16:22:12.080020 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6a8( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.6a8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.080041 7f869af78700 5 -- op tracker -- , seq: 3631, time: 2014-06-09 16:22:12.080039, event: done, op: pg_info(1 pgs e243:3.1ff) 2014-06-09 16:22:12.080049 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.080051 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.080053 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.080053 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080058 7f868d75d700 5 queue_transactions existing osr(0.6a8 0x2bd7d90)/0x2bd7d90 2014-06-09 16:22:12.080061 7f868d75d700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080063 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.080064 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.080065 7f868ff62700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080064 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3943.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080067 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080065 7f869af78700 20 osd.26 243 _dispatch 0x30c768c0 pg_info(1 pgs e243:3.424) v4 2014-06-09 16:22:12.080071 7f868b759700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080071 7f869af78700 5 -- op tracker -- , seq: 3632, time: 2014-06-09 16:22:12.076184, event: header_read, op: pg_info(1 pgs e243:3.424) 2014-06-09 16:22:12.080080 7f86a5610700 10 _do_op 0x1e08bd40 seq 3943 r = 0, finisher 0xcc83ea0 0 2014-06-09 16:22:12.080079 7f869af78700 5 -- op tracker -- , seq: 3632, time: 2014-06-09 16:22:12.076185, event: throttled, op: pg_info(1 pgs e243:3.424) 2014-06-09 16:22:12.080086 7f86a5610700 10 _finish_op 0x1e08bd40 seq 3943 osr(2.288 0x2ed0d10)/0x2ed0d10 2014-06-09 16:22:12.080086 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a6260] 2014-06-09 16:22:12.080092 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652240 (1 active) 2014-06-09 16:22:12.080086 7f869af78700 5 -- op tracker -- , seq: 3632, time: 2014-06-09 16:22:12.076198, event: all_read, op: pg_info(1 pgs e243:3.424) 2014-06-09 16:22:12.080094 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6c60 (2 active) 2014-06-09 16:22:12.080094 7f868d75d700 10 op_submit_start 3992 2014-06-09 16:22:12.080096 7f868d75d700 5 queue_transactions (trailing journal) 3992 2014-06-09 16:22:12.080096 7f86a5610700 5 _do_op 0xc2d4900 seq 3944 osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 start 2014-06-09 16:22:12.080098 7f86a5610700 10 _do_transaction on 0xc291440 2014-06-09 16:22:12.080089 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.080097 7f868d75d700 5 queue_op 0x160977a0 seq 3992 osr(0.6a8 0x2bd7d90) 828 bytes (queue has 50 ops and 927945 bytes) 2014-06-09 16:22:12.080100 7f868d75d700 10 op_submit_finish 3992 2014-06-09 16:22:12.080097 7f869af78700 5 -- op tracker -- , seq: 3632, time: 2014-06-09 16:22:12.080062, event: dispatched, op: pg_info(1 pgs e243:3.424) 2014-06-09 16:22:12.080103 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080100 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.080105 7f869af78700 5 -- op tracker -- , seq: 3632, time: 2014-06-09 16:22:12.080102, event: waiting_for_osdmap, op: pg_info(1 pgs e243:3.424) 2014-06-09 16:22:12.080109 7f869af78700 7 osd.26 243 handle_pg_info pg_info(1 pgs e243:3.424) v4 from osd.4 2014-06-09 16:22:12.080112 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x30c768c0 2014-06-09 16:22:12.080103 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.080109 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.288( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:198 2014-06-09 16:22:12.080118 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a6260] 2014-06-09 16:22:12.080117 7f869af78700 5 -- op tracker -- , seq: 3632, time: 2014-06-09 16:22:12.080115, event: started, op: pg_info(1 pgs e243:3.424) 2014-06-09 16:22:12.080120 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.080121 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.080125 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc83ea0] 2014-06-09 16:22:12.080129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc83ea0] 2014-06-09 16:22:12.080130 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080131 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080132 7f869af78700 5 -- op tracker -- , seq: 3632, time: 2014-06-09 16:22:12.080129, event: done, op: pg_info(1 pgs e243:3.424) 2014-06-09 16:22:12.080135 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.080134 7f8689755700 2 waiting 51 > 50 ops || 928771 > 104857600 2014-06-09 16:22:12.080138 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080139 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.080139 7f868d75d700 5 queue_transactions existing osr(3.2d6 0x3ed0e60)/0x3ed0e60 2014-06-09 16:22:12.080140 7f868e75f700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080140 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.080143 7f868d75d700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080148 7f8691f66700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080151 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3944.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080154 7f868bf5a700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080155 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080160 7f868a757700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080166 7f868cf5c700 2 waiting 51 > 50 ops || 928771 > 104857600 2014-06-09 16:22:12.080172 7f8689f56700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080172 7f86a5610700 10 _do_op 0xc2d4900 seq 3944 r = 0, finisher 0xbce6400 0 2014-06-09 16:22:12.080177 7f86a5610700 10 _finish_op 0xc2d4900 seq 3944 osr(3.2c9 0x3ed0ea0)/0x3ed0ea0 2014-06-09 16:22:12.080177 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006c9e0] 2014-06-09 16:22:12.080179 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006c9e0] 2014-06-09 16:22:12.080181 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.080182 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.080182 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6c60 (1 active) 2014-06-09 16:22:12.080185 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbce6400] 2014-06-09 16:22:12.080185 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad7a0 (2 active) 2014-06-09 16:22:12.080187 7f86a5610700 5 _do_op 0x300324e0 seq 3945 osr(2.2d9 0x2ed0c30)/0x2ed0c30 start 2014-06-09 16:22:12.080176 7f86a6b95700 5 -- op tracker -- , seq: 3633, time: 2014-06-09 16:22:12.072306, event: header_read, op: osd_op(client.4542.0:34497 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 417792~2064384] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.080188 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbce6400] 2014-06-09 16:22:12.080189 7f86a5610700 10 _do_transaction on 0xbf46180 2014-06-09 16:22:12.080189 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080190 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080194 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080195 7f868c75b700 10 op_submit_start 3993 2014-06-09 16:22:12.080196 7f868c75b700 5 queue_transactions (trailing journal) 3993 2014-06-09 16:22:12.080197 7f8690f64700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080195 7f86a6b95700 5 -- op tracker -- , seq: 3633, time: 2014-06-09 16:22:12.072307, event: throttled, op: osd_op(client.4542.0:34497 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 417792~2064384] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.080198 7f868c75b700 5 queue_op 0xbf7f740 seq 3993 osr(2.30b 0x2ed0f50) 828 bytes (queue has 50 ops and 927945 bytes) 2014-06-09 16:22:12.080200 7f868c75b700 10 op_submit_finish 3993 2014-06-09 16:22:12.080203 7f868af58700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080207 7f8691765700 2 waiting 51 > 50 ops || 928771 > 104857600 2014-06-09 16:22:12.080203 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.080212 7f868f761700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080205 7f86a6b95700 5 -- op tracker -- , seq: 3633, time: 2014-06-09 16:22:12.076598, event: all_read, op: osd_op(client.4542.0:34497 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 417792~2064384] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.080217 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.080228 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080228 7f86a6b95700 5 -- op tracker -- , seq: 3633, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34497 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 417792~2064384] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.080225 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.080234 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.080233 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.080240 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3945.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080243 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080241 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.080248 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.080260 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6d600 prio 63 cost 2064384 latency 0.007954 osd_op(client.4542.0:34497 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 417792~2064384] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.080261 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.080270 7f86a5610700 10 _do_op 0x300324e0 seq 3945 r = 0, finisher 0xbd746a0 0 2014-06-09 16:22:12.080270 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.080276 7f86a5610700 10 _finish_op 0x300324e0 seq 3945 osr(2.2d9 0x2ed0c30)/0x2ed0c30 2014-06-09 16:22:12.080278 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.3cf( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.080285 7f8690f64700 10 op_submit_start 3994 2014-06-09 16:22:12.080287 7f8690f64700 5 queue_transactions (trailing journal) 3994 2014-06-09 16:22:12.080287 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd746a0] 2014-06-09 16:22:12.080288 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.080290 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd746a0] 2014-06-09 16:22:12.080288 7f8690f64700 5 queue_op 0x30a2060 seq 3994 osr(1.2ef 0x384d0f0) 828 bytes (queue has 50 ops and 927945 bytes) 2014-06-09 16:22:12.080291 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080293 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080294 7f8692767700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080295 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad7a0 (1 active) 2014-06-09 16:22:12.080299 7f8692f68700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080293 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.1e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.080297 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd0e0 (2 active) 2014-06-09 16:22:12.080304 7f868ff62700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080308 7f86a5610700 5 _do_op 0x324cc900 seq 3946 osr(3.1fc 0x3ed0930)/0x3ed0930 start 2014-06-09 16:22:12.080310 7f8690f64700 10 op_submit_finish 3994 2014-06-09 16:22:12.080311 7f86a5610700 10 _do_transaction on 0xc2c43c0 2014-06-09 16:22:12.080316 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080325 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.080314 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.11d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.11d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.080329 7f868c75b700 5 queue_transactions existing osr(2.1e3 0x2ed0810)/0x2ed0810 2014-06-09 16:22:12.080331 7f868c75b700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080336 7f868ef60700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080341 7f868df5e700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080346 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314129c0] 2014-06-09 16:22:12.080347 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.080348 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314129c0] 2014-06-09 16:22:12.080349 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.080350 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.080352 7f8690f64700 5 queue_transactions existing osr(1.11d 0x384c8b0)/0x384c8b0 2014-06-09 16:22:12.080353 7f8690763700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080352 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080357 7f868b759700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080362 7f868d75d700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080367 7f8691f66700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080367 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3946.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080370 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080374 7f868bf5a700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080378 7f86a5610700 10 _do_op 0x324cc900 seq 3946 r = 0, finisher 0xc2d6f00 0 2014-06-09 16:22:12.080379 7f868a757700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080383 7f86a5610700 10 _finish_op 0x324cc900 seq 3946 osr(3.1fc 0x3ed0930)/0x3ed0930 2014-06-09 16:22:12.080385 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd67660] 2014-06-09 16:22:12.080385 7f868cf5c700 2 waiting 51 > 50 ops || 928771 > 104857600 2014-06-09 16:22:12.080386 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd67660] 2014-06-09 16:22:12.080390 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.080391 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.080394 7f8689f56700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080399 7f868af58700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080405 7f8691765700 2 waiting 51 > 50 ops || 928771 > 104857600 2014-06-09 16:22:12.080410 7f868f761700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080431 7f8690f64700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080436 7f868e75f700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080441 7f8689755700 2 waiting 51 > 50 ops || 928771 > 104857600 2014-06-09 16:22:12.080451 7f8692f68700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080452 7f8692767700 10 op_submit_start 3995 2014-06-09 16:22:12.080455 7f8692767700 5 queue_transactions (trailing journal) 3995 2014-06-09 16:22:12.080455 7f868c75b700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080455 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d6f00] 2014-06-09 16:22:12.080456 7f8692767700 5 queue_op 0x16097200 seq 3995 osr(3.527 0x3ed1660) 828 bytes (queue has 50 ops and 927945 bytes) 2014-06-09 16:22:12.080458 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d6f00] 2014-06-09 16:22:12.080459 7f868ef60700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080460 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080461 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080464 7f868df5e700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080464 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd0e0 (1 active) 2014-06-09 16:22:12.080466 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114120 (2 active) 2014-06-09 16:22:12.080468 7f8690763700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080470 7f86a5610700 5 _do_op 0xbf7fc80 seq 3947 osr(0.9d 0x2bd6490)/0x2bd6490 start 2014-06-09 16:22:12.080472 7f86a5610700 10 _do_transaction on 0xc1f66c0 2014-06-09 16:22:12.080473 7f868b759700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080477 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080478 7f868d75d700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080484 7f8691f66700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080489 7f868bf5a700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080494 7f868a757700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080501 7f868cf5c700 2 waiting 51 > 50 ops || 928771 > 104857600 2014-06-09 16:22:12.080506 7f8689f56700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080507 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080512 7f868af58700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080517 7f8691765700 2 waiting 51 > 50 ops || 928771 > 104857600 2014-06-09 16:22:12.080520 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3947.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080524 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080525 7f868f761700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080531 7f8690f64700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080532 7f86a5610700 10 _do_op 0xbf7fc80 seq 3947 r = 0, finisher 0xcd33900 0 2014-06-09 16:22:12.080536 7f868e75f700 2 waiting 51 > 50 ops || 928806 > 104857600 2014-06-09 16:22:12.080543 7f86a5610700 10 _finish_op 0xbf7fc80 seq 3947 osr(0.9d 0x2bd6490)/0x2bd6490 2014-06-09 16:22:12.080544 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32477de0] 2014-06-09 16:22:12.080546 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32477de0] 2014-06-09 16:22:12.080548 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.080549 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.080553 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd33900] 2014-06-09 16:22:12.080555 7f868ff62700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080560 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd33900] 2014-06-09 16:22:12.080562 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080561 7f868c75b700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080562 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080565 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080565 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114120 (1 active) 2014-06-09 16:22:12.080567 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922ea0 (2 active) 2014-06-09 16:22:12.080569 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080569 7f86a5610700 5 _do_op 0xbf7e780 seq 3948 osr(1.15b 0x384cbf0)/0x384cbf0 start 2014-06-09 16:22:12.080572 7f86a5610700 10 _do_transaction on 0x2ed49c0 2014-06-09 16:22:12.080573 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080577 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080577 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080582 7f868d75d700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080587 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080592 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080597 7f868a757700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.080603 7f868cf5c700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080605 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080608 7f8689f56700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080613 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080618 7f8691765700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080618 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3948.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080622 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080623 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080628 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080633 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.080634 7f86a5610700 10 _do_op 0xbf7e780 seq 3948 r = 0, finisher 0x189aa5a0 0 2014-06-09 16:22:12.080637 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080640 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061f500] 2014-06-09 16:22:12.080640 7f86a5610700 10 _finish_op 0xbf7e780 seq 3948 osr(1.15b 0x384cbf0)/0x384cbf0 2014-06-09 16:22:12.080642 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061f500] 2014-06-09 16:22:12.080643 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.080644 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.080648 7f868c75b700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080650 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aa5a0] 2014-06-09 16:22:12.080652 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080654 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aa5a0] 2014-06-09 16:22:12.080655 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080656 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080656 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080659 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080659 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922ea0 (1 active) 2014-06-09 16:22:12.080661 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3386c0 (2 active) 2014-06-09 16:22:12.080663 7f86a5610700 5 _do_op 0xc88a000 seq 3949 osr(2.24d 0x2ed0a50)/0x2ed0a50 start 2014-06-09 16:22:12.080664 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080666 7f86a5610700 10 _do_transaction on 0x2c81c80 2014-06-09 16:22:12.080670 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080669 7f868d75d700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080673 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080678 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080683 7f868a757700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.080689 7f868cf5c700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080694 7f8689f56700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080700 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080700 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080705 7f8691765700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080710 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080715 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080714 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3949.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080717 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080719 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.080723 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080733 7f86a5610700 10 _do_op 0xc88a000 seq 3949 r = 0, finisher 0xcc832c0 0 2014-06-09 16:22:12.080737 7f86a5610700 10 _finish_op 0xc88a000 seq 3949 osr(2.24d 0x2ed0a50)/0x2ed0a50 2014-06-09 16:22:12.080739 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd74f00] 2014-06-09 16:22:12.080741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd74f00] 2014-06-09 16:22:12.080742 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.080744 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.080748 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc832c0] 2014-06-09 16:22:12.080749 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080752 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc832c0] 2014-06-09 16:22:12.080753 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080753 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080754 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080757 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080758 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3386c0 (1 active) 2014-06-09 16:22:12.080760 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115320 (2 active) 2014-06-09 16:22:12.080761 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080764 7f86a5610700 5 _do_op 0x30d97ec0 seq 3950 osr(1.67 0x384c5b0)/0x384c5b0 start 2014-06-09 16:22:12.080766 7f8692767700 10 op_submit_finish 3995 2014-06-09 16:22:12.080767 7f86a5610700 10 _do_transaction on 0xbfcc900 2014-06-09 16:22:12.080766 7f868d75d700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080770 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080771 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080775 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080771 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.080784 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.080792 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.080806 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080800 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.080808 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.080819 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3950.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080815 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.080822 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080828 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.080835 7f86a5610700 10 _do_op 0x30d97ec0 seq 3950 r = 0, finisher 0x32072e80 0 2014-06-09 16:22:12.080840 7f86a5610700 10 _finish_op 0x30d97ec0 seq 3950 osr(1.67 0x384c5b0)/0x384c5b0 2014-06-09 16:22:12.080837 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.080845 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.19f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.080854 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.080854 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32072e80] 2014-06-09 16:22:12.080854 7f868cf5c700 10 op_submit_start 3996 2014-06-09 16:22:12.080856 7f868cf5c700 5 queue_transactions (trailing journal) 3996 2014-06-09 16:22:12.080857 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32072e80] 2014-06-09 16:22:12.080859 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080860 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080858 7f868cf5c700 5 queue_op 0x300612c0 seq 3996 osr(2.c6 0x2ed0490) 826 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.080862 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115320 (1 active) 2014-06-09 16:22:12.080864 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cb40 (2 active) 2014-06-09 16:22:12.080863 7f8689f56700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080867 7f86a5610700 5 _do_op 0x324b2b40 seq 3951 osr(3.271 0x3ed0bc0)/0x3ed0bc0 start 2014-06-09 16:22:12.080870 7f86a5610700 10 _do_transaction on 0x4dcdec0 2014-06-09 16:22:12.080859 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.080869 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080874 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.080874 7f8691765700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080878 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080873 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.080883 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080887 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.080882 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x270bea00,0x2c06ab00 2014-06-09 16:22:12.080892 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080897 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080902 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080898 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.080907 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080908 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.080911 7f868d75d700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080907 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.080921 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3951.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.080916 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.080924 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.080932 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080935 7f86a5610700 10 _do_op 0x324b2b40 seq 3951 r = 0, finisher 0x30e15520 0 2014-06-09 16:22:12.080928 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1052 2014-06-09 16:22:12.080937 7f86a5610700 10 _finish_op 0x324b2b40 seq 3951 osr(3.271 0x3ed0bc0)/0x3ed0bc0 2014-06-09 16:22:12.080938 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080943 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080938 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.080946 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.080954 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080957 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.080958 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e15520] 2014-06-09 16:22:12.080959 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080961 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e15520] 2014-06-09 16:22:12.080963 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.080964 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.080963 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080966 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cb40 (1 active) 2014-06-09 16:22:12.080967 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.080968 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780240 (2 active) 2014-06-09 16:22:12.080962 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.080972 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.080973 7f868cf5c700 10 op_submit_finish 3996 2014-06-09 16:22:12.080976 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080977 7f868ff62700 10 op_submit_start 3997 2014-06-09 16:22:12.080979 7f868ff62700 5 queue_transactions (trailing journal) 3997 2014-06-09 16:22:12.080973 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.080980 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.080981 7f868ff62700 5 queue_op 0x30a3500 seq 3997 osr(2.13c 0x2ed0610) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.080984 7f868ff62700 10 op_submit_finish 3997 2014-06-09 16:22:12.080981 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.080979 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.236( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.080988 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.080989 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.33a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.080996 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081003 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081011 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.081015 7f868cf5c700 5 queue_transactions existing osr(2.236 0x2ed0ad0)/0x2ed0ad0 2014-06-09 16:22:12.081017 7f868cf5c700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081019 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.081014 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:12.081021 7f86a5e11700 10 _finish_op 0xc88a840 seq 3942 osr(0.25d 0x2bd6a00)/0x2bd6a00 2014-06-09 16:22:12.081024 7f868ff62700 5 queue_transactions existing osr(2.33a 0x2ed0ed0)/0x2ed0ed0 2014-06-09 16:22:12.081023 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.081031 7f868b759700 10 op_submit_start 3998 2014-06-09 16:22:12.081032 7f868b759700 5 queue_transactions (trailing journal) 3998 2014-06-09 16:22:12.081031 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc1696a0] 2014-06-09 16:22:12.081033 7f868b759700 5 queue_op 0xc88a120 seq 3998 osr(3.124 0x3ed07c0) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.081031 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.15e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.081038 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc1696a0] 2014-06-09 16:22:12.081039 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.081040 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.081040 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.081043 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524fc0 (1 active) 2014-06-09 16:22:12.081045 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286120 (2 active) 2014-06-09 16:22:12.081049 7f86a5e11700 5 _do_op 0x30a2c00 seq 3953 osr(3.37a 0x3ed1040)/0x3ed1040 start 2014-06-09 16:22:12.081051 7f86a5e11700 10 _do_transaction on 0x394ed80 2014-06-09 16:22:12.081051 7f868b759700 10 op_submit_finish 3998 2014-06-09 16:22:12.081044 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081056 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.081056 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.081055 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.26d( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.081063 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.081036 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081071 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.081078 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081078 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081089 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.081086 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081093 7f868b759700 5 queue_transactions existing osr(2.26d 0x2ed0de0)/0x2ed0de0 2014-06-09 16:22:12.081095 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081098 7f8691765700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.081102 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.081103 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081097 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:186 2014-06-09 16:22:12.081108 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081106 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.081113 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.081115 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3953.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.081117 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.081118 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.081113 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.614( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.081122 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.081121 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081125 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081131 7f868cf5c700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081125 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081137 7f868ff62700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081138 7f86a5e11700 10 _do_op 0x30a2c00 seq 3953 r = 0, finisher 0x3206f300 0 2014-06-09 16:22:12.081142 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081137 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.081147 7f86a5e11700 10 _finish_op 0x30a2c00 seq 3953 osr(3.37a 0x3ed1040)/0x3ed1040 2014-06-09 16:22:12.081147 7f868d75d700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081145 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.081156 7f8691f66700 10 op_submit_start 3999 2014-06-09 16:22:12.081155 7f868d75d700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081157 7f8691f66700 5 queue_transactions (trailing journal) 3999 2014-06-09 16:22:12.081153 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.081161 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081166 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3206f300] 2014-06-09 16:22:12.081161 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081166 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081158 7f8691f66700 5 queue_op 0x3250e180 seq 3999 osr(1.517 0x384db70) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.081169 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3206f300] 2014-06-09 16:22:12.081171 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.081172 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.081172 7f8691765700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.081174 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286120 (1 active) 2014-06-09 16:22:12.081168 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081176 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081176 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2867e0 (2 active) 2014-06-09 16:22:12.081180 7f86a5e11700 5 _do_op 0xcca70e0 seq 3954 osr(3.76 0x3ed0240)/0x3ed0240 start 2014-06-09 16:22:12.081181 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081183 7f86a5e11700 10 _do_transaction on 0x3859980 2014-06-09 16:22:12.081181 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 2014-06-09 16:22:12.081187 7f8691f66700 10 op_submit_finish 3999 2014-06-09 16:22:12.081188 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.081189 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.081196 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.29a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.081191 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.631( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:12.081204 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.081209 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081223 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.081220 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.081230 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.081228 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.081235 7f8691f66700 5 queue_transactions existing osr(0.631 0x2bd7c70)/0x2bd7c70 2014-06-09 16:22:12.081235 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3954.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.081238 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081239 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.081235 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.081242 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.081247 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.081243 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081252 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081253 7f86a5e11700 10 _do_op 0xcca70e0 seq 3954 r = 0, finisher 0xbd60640 0 2014-06-09 16:22:12.081255 7f86a5e11700 10 _finish_op 0xcca70e0 seq 3954 osr(3.76 0x3ed0240)/0x3ed0240 2014-06-09 16:22:12.081250 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081257 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081262 7f868cf5c700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081259 7f86a5610700 5 _do_op 0xbf7f3e0 seq 3952 osr(3.124 0x3ed07c0)/0x3ed07c0 start 2014-06-09 16:22:12.081267 7f868ff62700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081269 7f86a5610700 10 _do_transaction on 0xc317740 2014-06-09 16:22:12.081262 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.081273 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081271 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.081278 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081280 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.081278 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.4ef( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.081286 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.081291 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081302 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.081314 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.081309 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.081317 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.081326 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3952.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.081330 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.081324 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081330 7f86a6b95700 5 -- op tracker -- , seq: 3634, time: 2014-06-09 16:22:12.080285, event: header_read, op: osd_op(client.4542.0:34498 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2482176~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.081332 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081343 7f86a6b95700 5 -- op tracker -- , seq: 3634, time: 2014-06-09 16:22:12.080286, event: throttled, op: osd_op(client.4542.0:34498 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2482176~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.081347 7f86a5610700 10 _do_op 0xbf7f3e0 seq 3952 r = 0, finisher 0xbdf06e0 0 2014-06-09 16:22:12.081344 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.081352 7f86a6b95700 5 -- op tracker -- , seq: 3634, time: 2014-06-09 16:22:12.081306, event: all_read, op: osd_op(client.4542.0:34498 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2482176~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.081352 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.081358 7f868ef60700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081360 7f868d75d700 10 op_submit_start 4000 2014-06-09 16:22:12.081361 7f868d75d700 5 queue_transactions (trailing journal) 4000 2014-06-09 16:22:12.081363 7f8691765700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081362 7f86a6b95700 5 -- op tracker -- , seq: 3634, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34498 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2482176~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.081363 7f868d75d700 5 queue_op 0x10039920 seq 4000 osr(3.2d6 0x3ed0e60) 828 bytes (queue has 50 ops and 927949 bytes) 2014-06-09 16:22:12.081360 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.3a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.081367 7f868b759700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081369 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.081372 7f8690f64700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd60640] 2014-06-09 16:22:12.081377 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd60640] 2014-06-09 16:22:12.081378 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.081379 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.081377 7f8691f66700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081373 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081382 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2867e0 (1 active) 2014-06-09 16:22:12.081383 7f868e75f700 2 waiting 51 > 50 ops || 928810 > 104857600 2014-06-09 16:22:12.081383 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f65a0 (2 active) 2014-06-09 16:22:12.081387 7f86a5e11700 5 _do_op 0x30a2660 seq 3955 osr(1.73b 0x2ed00c0)/0x2ed00c0 start 2014-06-09 16:22:12.081387 7f8689755700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081389 7f86a5e11700 10 _do_transaction on 0xc2f49c0 2014-06-09 16:22:12.081385 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.081391 7f868df5e700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081394 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.081396 7f8690763700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081392 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.081402 7f868cf5c700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081399 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.081407 7f868ff62700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081413 7f868af58700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081407 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081415 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081428 7f868bf5a700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081426 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.081434 7f868f761700 2 waiting 51 > 50 ops || 928777 > 104857600 2014-06-09 16:22:12.081408 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.081435 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.081443 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.081443 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.498( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.081451 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.081456 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3955.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.081459 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.081454 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6d500 prio 63 cost 516096 latency 0.001169 osd_op(client.4542.0:34498 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2482176~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.081455 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081470 7f86a5e11700 10 _do_op 0x30a2660 seq 3955 r = 0, finisher 0x1e08ff80 0 2014-06-09 16:22:12.081466 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.081473 7f86a5e11700 10 _finish_op 0x30a2660 seq 3955 osr(1.73b 0x2ed00c0)/0x2ed00c0 2014-06-09 16:22:12.081474 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.081487 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081482 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.081492 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1e08ff80] 2014-06-09 16:22:12.081494 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081495 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1e08ff80] 2014-06-09 16:22:12.081497 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.081491 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081498 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.081499 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081500 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f65a0 (1 active) 2014-06-09 16:22:12.081502 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc000 (2 active) 2014-06-09 16:22:12.081498 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081505 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.081506 7f86a5e11700 5 _do_op 0xc705260 seq 3956 osr(1.2a3 0x384d1f0)/0x384d1f0 start 2014-06-09 16:22:12.081508 7f86a5e11700 10 _do_transaction on 0xbfa4480 2014-06-09 16:22:12.081508 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.081509 7f868d75d700 10 op_submit_finish 4000 2014-06-09 16:22:12.081512 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081513 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.081516 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081510 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.081520 7f8689f56700 10 op_submit_start 4001 2014-06-09 16:22:12.081521 7f8689f56700 5 queue_transactions (trailing journal) 4001 2014-06-09 16:22:12.081521 7f868cf5c700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081523 7f8689f56700 5 queue_op 0x308cbe00 seq 4001 osr(0.186 0x2bd67d0) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.081518 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.081515 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.38a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.38a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.081525 7f8689f56700 10 op_submit_finish 4001 2014-06-09 16:22:12.081529 7f8691765700 10 op_submit_start 4002 2014-06-09 16:22:12.081530 7f8691765700 5 queue_transactions (trailing journal) 4002 2014-06-09 16:22:12.081526 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.13b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.081531 7f8691765700 5 queue_op 0x159965a0 seq 4002 osr(3.f0 0x3ed04a0) 826 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.081535 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.081526 7f86a5610700 10 _finish_op 0xbf7f3e0 seq 3952 osr(3.124 0x3ed07c0)/0x3ed07c0 2014-06-09 16:22:12.081548 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.081549 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdf06e0] 2014-06-09 16:22:12.081553 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdf06e0] 2014-06-09 16:22:12.081539 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081554 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.081553 7f868d75d700 5 queue_transactions existing osr(2.38a 0x2ed10f0)/0x2ed10f0 2014-06-09 16:22:12.081555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.081556 7f868d75d700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081557 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780240 (1 active) 2014-06-09 16:22:12.081559 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0c60 (2 active) 2014-06-09 16:22:12.081560 7f868ff62700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081563 7f8691765700 10 op_submit_finish 4002 2014-06-09 16:22:12.081556 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 active m=1] check_local 2014-06-09 16:22:12.081567 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 2014-06-09 16:22:12.081530 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:12.081568 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 243/201 242/242/200) 2014-06-09 16:22:12.081576 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing 2014-06-09 16:22:12.081580 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.081587 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.081591 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.081600 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.081604 7f8691765700 5 queue_transactions existing osr(3.1d5 0x3ed09f0)/0x3ed09f0 2014-06-09 16:22:12.081600 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.081606 7f8691765700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081610 7f868af58700 10 op_submit_start 4003 2014-06-09 16:22:12.081611 7f868af58700 5 queue_transactions (trailing journal) 4003 2014-06-09 16:22:12.081603 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1057 2014-06-09 16:22:12.081612 7f868af58700 5 queue_op 0x324a4f60 seq 4003 osr(0.3a6 0x2bd7330) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.081615 7f868af58700 10 op_submit_finish 4003 2014-06-09 16:22:12.081614 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.081614 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear 2014-06-09 16:22:12.081618 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.113( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.113( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.081624 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.6d3( v 214'632 lc 214'631 (0'0,214'632] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'632 lcod 0'0 mlcod 0'0 active m=1] agent_stop 2014-06-09 16:22:12.081634 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.081632 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:180 2014-06-09 16:22:12.081646 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.081638 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081648 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.081650 7f868af58700 5 queue_transactions existing osr(1.113 0x384c910)/0x384c910 2014-06-09 16:22:12.081652 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081649 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.081657 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.081664 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.081671 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081678 7f86a5610700 5 _do_op 0x324f4840 seq 3957 osr(0.3b7 0x2bd7290)/0x2bd7290 start 2014-06-09 16:22:12.081678 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081681 7f86a5610700 10 _do_transaction on 0xc3fc180 2014-06-09 16:22:12.081689 7f8692f68700 10 op_submit_start 4004 2014-06-09 16:22:12.081689 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.081691 7f8692f68700 5 queue_transactions (trailing journal) 4004 2014-06-09 16:22:12.081689 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f25d0 (19 active) 2014-06-09 16:22:12.081693 7f8692f68700 5 queue_op 0xc88aea0 seq 4004 osr(0.25d 0x2bd6a00) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.081695 7f8692f68700 10 op_submit_finish 4004 2014-06-09 16:22:12.081698 7f8689f56700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.081699 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.375( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.081705 7f8689f56700 10 osd.26 243 dequeue_op 0x22008d00 prio 63 cost 151552 latency 1.408050 osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.081725 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.081732 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.081730 7f8689f56700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.081725, event: reached_pg, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.081737 7f8692f68700 5 queue_transactions existing osr(1.375 0x384d230)/0x384d230 2014-06-09 16:22:12.081739 7f8692f68700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081738 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3956.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.081742 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.081744 7f868a757700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.081738 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.081749 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081749 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30d603a0,0x33a7b20,0x3206ec00,0x318c3ea0,0xbd70e20,0x32075360] 2014-06-09 16:22:12.081752 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30d603a0,0x33a7b20,0x3206ec00,0x318c3ea0,0xbd70e20,0x32075360] 2014-06-09 16:22:12.081754 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081755 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.081756 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.081759 7f868c75b700 10 op_submit_start 4005 2014-06-09 16:22:12.081758 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081760 7f868c75b700 5 queue_transactions (trailing journal) 4005 2014-06-09 16:22:12.081752 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x22008d00 2014-06-09 16:22:12.081761 7f868c75b700 5 queue_op 0xbf7e1e0 seq 4005 osr(2.1e3 0x2ed0810) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.081762 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.081764 7f868c75b700 10 op_submit_finish 4005 2014-06-09 16:22:12.081766 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.081770 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081774 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081767 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.081778 7f868cf5c700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081764 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.081781 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081780 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.081788 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.081795 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.081803 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.081809 7f8689f56700 15 getattr 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 '_' 2014-06-09 16:22:12.081810 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.081816 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081820 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.081822 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:167 2014-06-09 16:22:12.081831 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.081838 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.77a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.081846 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.081850 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.718( v 214'639 lc 213'626 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.081863 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:12.081877 7f8689f56700 10 getattr got.size() is 0 2014-06-09 16:22:12.081872 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.081880 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.081879 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 but can_create 2014-06-09 16:22:12.081892 7f8689f56700 15 getattr 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 'snapset' 2014-06-09 16:22:12.081893 7f868c75b700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.081909 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:998 2014-06-09 16:22:12.081920 7f8689f56700 10 getattr got.size() is 0 2014-06-09 16:22:12.081922 7f8689f56700 15 getattr 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/snapdir//3 'snapset' 2014-06-09 16:22:12.081923 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.081942 7f868c75b700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41050 (19 active) 2014-06-09 16:22:12.081948 7f8689f56700 10 getattr got.size() is 0 2014-06-09 16:22:12.081957 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x3105d080 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.081971 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41050 (20 active) 2014-06-09 16:22:12.081968 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.081975 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.461( v 213'587 (0'0,213'587] local-les=243 n=110 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:12.081977 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.081986 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x3105d080 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 rwstate(none n=0 w=0) oi: 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x26a468c0 snapset: 0=[]:[] 2014-06-09 16:22:12.082004 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.082008 7f868c75b700 5 queue_transactions existing osr(3.461 0x3ed1120)/0x3ed1120 2014-06-09 16:22:12.082011 7f868c75b700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082001 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 @head oi=2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.082020 7f8689f56700 15 getattr 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/snapdir//3 '_' 2014-06-09 16:22:12.082018 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:12.082028 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.082035 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.6ab( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.082044 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.082049 7f8689f56700 10 getattr got.size() is 0 2014-06-09 16:22:12.082048 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.082051 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 2c355206/rbd_data.117d6b8b4567.00000000000016b8/snapdir//3 and !can_create 2014-06-09 16:22:12.082059 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.082066 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.082069 7f8689f56700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.082063, event: started, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.082074 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.082078 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x2d735e00 2014-06-09 16:22:12.082081 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.082088 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.082095 7f8689f56700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.082090, event: started, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.082100 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.082101 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] ov 0'0 av 243'597 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.082108 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.082115 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.648( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.082113 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 2014-06-09 16:22:12.082123 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.082123 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.082126 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.082138 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.082137 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~151552 2014-06-09 16:22:12.082145 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x306bd00,0x1c769300,0x306bc00,0x1c769200,0xe033500,0x1c769100,0xe033400,0x1c769000,0xe033300,0x1c768f00 2014-06-09 16:22:12.082153 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229600 2014-06-09 16:22:12.082163 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 snapset=0x26a468f8 snapc=0=[] 2014-06-09 16:22:12.082173 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.082180 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.082182 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.082188 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.082191 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 0x2d735e00 op modify 2014-06-09 16:22:12.082196 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.082202 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:06.717212 2014-06-09 16:22:12.082207 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:963 2014-06-09 16:22:12.082217 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.082221 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.082225 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.38c( v 213'596 (0'0,213'596] local-les=243 n=114 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.082233 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.082237 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.082249 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f3eb0 (19 active) 2014-06-09 16:22:12.082254 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.082248 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 25 on osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.082258 7f8692767700 10 osd.26 243 dequeue_op 0x270bea00 prio 63 cost 1388544 latency 1.252778 osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.082263 7f8689f56700 7 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 25 o 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.082276 7f8692767700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.082272, event: reached_pg, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.082288 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.082301 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x270bea00 2014-06-09 16:22:12.082320 7f86a5e11700 10 _do_op 0xc705260 seq 3956 r = 0, finisher 0x31412b80 0 2014-06-09 16:22:12.082311 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.082326 7f86a5e11700 10 _finish_op 0xc705260 seq 3956 osr(1.2a3 0x384d1f0)/0x384d1f0 2014-06-09 16:22:12.082330 7f8692767700 15 getattr 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 '_' 2014-06-09 16:22:12.082335 7f868d75d700 10 op_submit_start 4006 2014-06-09 16:22:12.082337 7f868d75d700 5 queue_transactions (trailing journal) 4006 2014-06-09 16:22:12.082339 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412b80] 2014-06-09 16:22:12.082339 7f868d75d700 5 queue_op 0x10038a20 seq 4006 osr(2.38a 0x2ed10f0) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.082343 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412b80] 2014-06-09 16:22:12.082343 7f868ff62700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082344 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.082346 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.082349 7f8691765700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082353 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082358 7f8692f68700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082362 7f868a757700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.082366 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082370 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082374 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082378 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.082382 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.082386 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082390 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082394 7f868cf5c700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082397 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082401 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc000 (1 active) 2014-06-09 16:22:12.082402 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082404 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd9e0 (2 active) 2014-06-09 16:22:12.082409 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.082408 7f86a5e11700 5 _do_op 0x11fcf200 seq 3958 osr(2.5b6 0x2ed1b90)/0x2ed1b90 start 2014-06-09 16:22:12.082409 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082410 7f868d75d700 10 op_submit_finish 4006 2014-06-09 16:22:12.082412 7f86a5e11700 10 _do_transaction on 0xc05b8c0 2014-06-09 16:22:12.082417 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.082411 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 but can_create 2014-06-09 16:22:12.082423 7f8692767700 15 getattr 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 'snapset' 2014-06-09 16:22:12.082413 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.65d( v 214'536 (0'0,214'536] local-les=243 n=106 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:12.082440 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5be80] 2014-06-09 16:22:12.082432 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.4 activated and committed 2014-06-09 16:22:12.082442 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5be80] 2014-06-09 16:22:12.082445 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.082446 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.082445 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.082458 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.082454 7f8689f56700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.082279, event: waiting for subops from 3, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.082456 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.082470 7f868d75d700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.082474 7f8689f56700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.082472 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3958.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.082477 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.082489 7f86a5e11700 10 _do_op 0x11fcf200 seq 3958 r = 0, finisher 0x32477c80 0 2014-06-09 16:22:12.082484 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.65d( v 214'536 lc 214'535 (0'0,214'536] local-les=243 n=106 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'536 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:875 2014-06-09 16:22:12.082500 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.082492 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 214'596 (0'0,214'596] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'596], crt=0'0) [243'597 (0'0) modify 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 by client.4573.0:34037 2014-06-09 10:20:06.717212] 2014-06-09 16:22:12.082504 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.1ef( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.1ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.082515 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=214'596 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'597 (0'0) modify 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 by client.4573.0:34037 2014-06-09 10:20:06.717212 2014-06-09 16:22:12.082529 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.082531 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.082535 7f868d75d700 5 queue_transactions existing osr(1.1ef 0x384cdb0)/0x384cdb0 2014-06-09 16:22:12.082537 7f868d75d700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082543 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3957.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.082536 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=214'596 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.082546 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.082564 7f8689f56700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'597, trimmed: 2014-06-09 16:22:12.082567 7f86a5610700 10 _do_op 0x324f4840 seq 3957 r = 0, finisher 0x325357c0 0 2014-06-09 16:22:12.082576 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc1695c0] 2014-06-09 16:22:12.082578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc1695c0] 2014-06-09 16:22:12.082579 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.082580 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.082584 7f8689f56700 5 queue_transactions existing osr(3.206 0x3ed0cf0)/0x3ed0cf0 2014-06-09 16:22:12.082587 7f8689f56700 2 waiting 51 > 50 ops || 1081897 > 104857600 2014-06-09 16:22:12.082588 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32535f00] 2014-06-09 16:22:12.082589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32535f00] 2014-06-09 16:22:12.082590 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.082591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.082594 7f86a5610700 10 _finish_op 0x324f4840 seq 3957 osr(0.3b7 0x2bd7290)/0x2bd7290 2014-06-09 16:22:12.082604 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x325357c0] 2014-06-09 16:22:12.082609 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x325357c0] 2014-06-09 16:22:12.082610 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.082611 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.082614 7f8691765700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082613 7f868c75b700 10 op_submit_start 4007 2014-06-09 16:22:12.082618 7f868c75b700 5 queue_transactions (trailing journal) 4007 2014-06-09 16:22:12.082618 7f868af58700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082619 7f868c75b700 5 queue_op 0x30607f80 seq 4007 osr(3.461 0x3ed1120) 828 bytes (queue has 50 ops and 927947 bytes) 2014-06-09 16:22:12.082622 7f8692f68700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082626 7f868a757700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.082630 7f868ef60700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082634 7f8690f64700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082638 7f8691f66700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082642 7f868e75f700 2 waiting 51 > 50 ops || 928808 > 104857600 2014-06-09 16:22:12.082645 7f8689755700 2 waiting 51 > 50 ops || 928773 > 104857600 2014-06-09 16:22:12.082649 7f868df5e700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082653 7f8690763700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082657 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.082657 7f868cf5c700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082659 7f8692767700 15 getattr 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/snapdir//3 'snapset' 2014-06-09 16:22:12.082661 7f868b759700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082666 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0c60 (1 active) 2014-06-09 16:22:12.082666 7f868f761700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082668 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcd80 (2 active) 2014-06-09 16:22:12.082672 7f86a5610700 5 _do_op 0xd470840 seq 3959 osr(3.7a 0x3ed0220)/0x3ed0220 start 2014-06-09 16:22:12.082674 7f868bf5a700 2 waiting 51 > 50 ops || 928775 > 104857600 2014-06-09 16:22:12.082675 7f86a5610700 10 _do_transaction on 0xc29e9c0 2014-06-09 16:22:12.082680 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.082685 7f86a5e11700 10 _finish_op 0x11fcf200 seq 3958 osr(2.5b6 0x2ed1b90)/0x2ed1b90 2014-06-09 16:22:12.082690 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.082696 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30e06840 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.082711 7f86a6b95700 5 -- op tracker -- , seq: 3635, time: 2014-06-09 16:22:12.081473, event: header_read, op: osd_op(client.4542.0:34499 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2998272~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.082709 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.082722 7f86a6b95700 5 -- op tracker -- , seq: 3635, time: 2014-06-09 16:22:12.081474, event: throttled, op: osd_op(client.4542.0:34499 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2998272~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.082718 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.082727 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.082731 7f86a6b95700 5 -- op tracker -- , seq: 3635, time: 2014-06-09 16:22:12.082687, event: all_read, op: osd_op(client.4542.0:34499 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2998272~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.082728 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30e06840 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 rwstate(none n=0 w=0) oi: c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd4460 snapset: 0=[]:[] 2014-06-09 16:22:12.082740 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3959.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.082740 7f86a6b95700 5 -- op tracker -- , seq: 3635, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34499 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2998272~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.082744 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.082747 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.082742 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 @head oi=c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.082756 7f86a5610700 10 _do_op 0xd470840 seq 3959 r = 0, finisher 0x31435e20 0 2014-06-09 16:22:12.082757 7f8692767700 15 getattr 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/snapdir//3 '_' 2014-06-09 16:22:12.082770 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32477c80] 2014-06-09 16:22:12.082765 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6d400 prio 63 cost 516096 latency 0.001292 osd_op(client.4542.0:34499 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 2998272~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.082775 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32477c80] 2014-06-09 16:22:12.082777 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.082778 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.082781 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd9e0 (1 active) 2014-06-09 16:22:12.082783 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c120 (2 active) 2014-06-09 16:22:12.082785 7f86a5e11700 5 _do_op 0xbf7e120 seq 3960 osr(3.577 0x3ed18c0)/0x3ed18c0 start 2014-06-09 16:22:12.082788 7f86a5e11700 10 _do_transaction on 0xc2ac840 2014-06-09 16:22:12.082790 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.082793 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.082791 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid c0524973/rbd_data.117f6b8b4567.0000000000001573/snapdir//3 and !can_create 2014-06-09 16:22:12.082809 7f8692767700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.082803, event: started, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.082817 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30a8be00 2014-06-09 16:22:12.082829 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.082834 7f8692767700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.082829, event: started, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.082842 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3960.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.082845 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.082841 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] ov 0'0 av 243'625 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.082857 7f86a5e11700 10 _do_op 0xbf7e120 seq 3960 r = 0, finisher 0xbe66f60 0 2014-06-09 16:22:12.082853 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 2014-06-09 16:22:12.082866 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c20a0] 2014-06-09 16:22:12.082868 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c20a0] 2014-06-09 16:22:12.082869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.082870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.082864 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.082873 7f86a5610700 10 _finish_op 0xd470840 seq 3959 osr(3.7a 0x3ed0220)/0x3ed0220 2014-06-09 16:22:12.082881 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31435e20] 2014-06-09 16:22:12.082885 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31435e20] 2014-06-09 16:22:12.082878 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~1388544 2014-06-09 16:22:12.082887 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.082888 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.082890 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcd80 (1 active) 2014-06-09 16:22:12.082892 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcfc0 (2 active) 2014-06-09 16:22:12.082894 7f86a5610700 5 _do_op 0x230b8c60 seq 3961 osr(1.18a 0x384cb50)/0x384cb50 start 2014-06-09 16:22:12.082897 7f86a5610700 10 _do_transaction on 0xc2d8a80 2014-06-09 16:22:12.082891 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229e00 2014-06-09 16:22:12.082902 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.082901 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 snapset=0x30dd4498 snapc=0=[] 2014-06-09 16:22:12.082911 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.082920 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.082937 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.082930 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 0x30a8be00 op modify 2014-06-09 16:22:12.082949 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3961.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.082953 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.082941 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:21:10.623181 2014-06-09 16:22:12.082966 7f86a5610700 10 _do_op 0x230b8c60 seq 3961 r = 0, finisher 0x3281ab40 0 2014-06-09 16:22:12.082965 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=119 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.082977 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x275ce9a0] 2014-06-09 16:22:12.082979 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x275ce9a0] 2014-06-09 16:22:12.082980 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.082981 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.082989 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd775c0] 2014-06-09 16:22:12.082990 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd775c0] 2014-06-09 16:22:12.082982 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.082991 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.082992 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.082995 7f86a5e11700 10 _finish_op 0xbf7e120 seq 3960 osr(3.577 0x3ed18c0)/0x3ed18c0 2014-06-09 16:22:12.083003 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbe66f60] 2014-06-09 16:22:12.082992 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 26 on osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.083006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbe66f60] 2014-06-09 16:22:12.083007 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.083008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.083011 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c120 (1 active) 2014-06-09 16:22:12.083013 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89a480 (2 active) 2014-06-09 16:22:12.083007 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 26 o c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.083015 7f86a5e11700 5 _do_op 0xc705b60 seq 3962 osr(3.587 0x3ed1840)/0x3ed1840 start 2014-06-09 16:22:12.083017 7f86a5e11700 10 _do_transaction on 0xc3269c0 2014-06-09 16:22:12.083022 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.083031 7f868c75b700 10 op_submit_finish 4007 2014-06-09 16:22:12.083034 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.126( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.126( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.083057 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.083066 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.083070 7f868c75b700 5 queue_transactions existing osr(2.126 0x2ed0670)/0x2ed0670 2014-06-09 16:22:12.083070 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3962.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.083073 7f868c75b700 10 op_submit_start 4008 2014-06-09 16:22:12.083074 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.083074 7f868c75b700 5 queue_transactions (trailing journal) 4008 2014-06-09 16:22:12.083084 7f8692f68700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087172 7f868c75b700 5 queue_op 0x324a5b00 seq 4008 osr(2.126 0x2ed0670) 828 bytes (queue has 50 ops and 1081071 bytes) 2014-06-09 16:22:12.087178 7f868a757700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.087184 7f868ef60700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087189 7f8690f64700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.083078 7f86a5610700 10 _finish_op 0x230b8c60 seq 3961 osr(1.18a 0x384cb50)/0x384cb50 2014-06-09 16:22:12.083026 7f8692767700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.083021, event: waiting for subops from 9, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.087194 7f8691f66700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087199 7f868e75f700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.087204 7f8689755700 2 waiting 51 > 50 ops || 1081897 > 104857600 2014-06-09 16:22:12.087209 7f868df5e700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087209 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.087215 7f8690763700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087212 7f86a6b95700 5 -- op tracker -- , seq: 3636, time: 2014-06-09 16:22:12.082784, event: header_read, op: osd_op(client.4542.0:34500 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3514368~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087220 7f868cf5c700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087226 7f868b759700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087224 7f86a6b95700 5 -- op tracker -- , seq: 3636, time: 2014-06-09 16:22:12.082785, event: throttled, op: osd_op(client.4542.0:34500 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3514368~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087232 7f868f761700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087238 7f868bf5a700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087230 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 213'624 (0'0,213'624] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'624], crt=0'0) [243'625 (0'0) modify c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 by client.4575.0:32697 2014-06-09 10:21:10.623181] 2014-06-09 16:22:12.087248 7f86a6b95700 5 -- op tracker -- , seq: 3636, time: 2014-06-09 16:22:12.083684, event: all_read, op: osd_op(client.4542.0:34500 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3514368~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087259 7f86a6b95700 5 -- op tracker -- , seq: 3636, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34500 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3514368~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087254 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=213'624 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'625 (0'0) modify c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 by client.4575.0:32697 2014-06-09 10:21:10.623181 2014-06-09 16:22:12.087265 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.087274 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=213'624 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.087286 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6ce00 prio 63 cost 516096 latency 0.004502 osd_op(client.4542.0:34500 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 3514368~516096] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.087300 7f8692767700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'625, trimmed: 2014-06-09 16:22:12.087317 7f8692767700 5 queue_transactions existing osr(3.173 0x3ed06f0)/0x3ed06f0 2014-06-09 16:22:12.087320 7f8692767700 2 waiting 51 > 50 ops || 2472013 > 104857600 2014-06-09 16:22:12.087330 7f86827e2700 5 -- op tracker -- , seq: 3637, time: 2014-06-09 16:22:12.086169, event: header_read, op: osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.087338 7f86827e2700 5 -- op tracker -- , seq: 3637, time: 2014-06-09 16:22:12.086169, event: throttled, op: osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.087345 7f86827e2700 5 -- op tracker -- , seq: 3637, time: 2014-06-09 16:22:12.086176, event: all_read, op: osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.087352 7f86827e2700 5 -- op tracker -- , seq: 3637, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.087356 7f86827e2700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:12.087362 7f86827e2700 20 osd.26 243 should_share_map osd.3 172.16.96.201:6823/50569 243 2014-06-09 16:22:12.087367 7f86827e2700 15 osd.26 243 enqueue_op 0x30c6d900 prio 196 cost 0 latency 0.001198 osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:12.087392 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281ab40] 2014-06-09 16:22:12.087394 7f868d75d700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087397 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281ab40] 2014-06-09 16:22:12.087401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.087402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.087404 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcfc0 (1 active) 2014-06-09 16:22:12.087406 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3faa20 (2 active) 2014-06-09 16:22:12.087410 7f868c75b700 10 op_submit_finish 4008 2014-06-09 16:22:12.087414 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.219( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.219( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.087446 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.087450 7f8689f56700 10 op_submit_start 4009 2014-06-09 16:22:12.087452 7f8689f56700 5 queue_transactions (trailing journal) 4009 2014-06-09 16:22:12.087453 7f8689f56700 5 queue_op 0x3248bf80 seq 4009 osr(3.206 0x3ed0cf0) 153950 bytes (queue has 50 ops and 1081071 bytes) 2014-06-09 16:22:12.087456 7f8689f56700 10 op_submit_finish 4009 2014-06-09 16:22:12.087460 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=214'596 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc143200 243'597 rep_tid=25 committed?=0 applied?=0 lock=0 op=osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.087479 7f8689f56700 10 osd.26 243 dequeue_op 0x22008d00 finish 2014-06-09 16:22:12.087482 7f8689f56700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.087515 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41950 (20 active) 2014-06-09 16:22:12.087518 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.48f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.48f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.087523 7f86a6b95700 5 -- op tracker -- , seq: 3638, time: 2014-06-09 16:22:12.087308, event: header_read, op: osd_op(client.4542.0:34501 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 4030464~163840] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087536 7f86a6b95700 5 -- op tracker -- , seq: 3638, time: 2014-06-09 16:22:12.087310, event: throttled, op: osd_op(client.4542.0:34501 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 4030464~163840] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087545 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.087549 7f868af58700 10 op_submit_start 4010 2014-06-09 16:22:12.087550 7f868af58700 5 queue_transactions (trailing journal) 4010 2014-06-09 16:22:12.087551 7f868af58700 5 queue_op 0x324a5d40 seq 4010 osr(1.113 0x384c910) 828 bytes (queue has 50 ops and 1081071 bytes) 2014-06-09 16:22:12.087554 7f868af58700 10 op_submit_finish 4010 2014-06-09 16:22:12.087546 7f86a6b95700 5 -- op tracker -- , seq: 3638, time: 2014-06-09 16:22:12.087498, event: all_read, op: osd_op(client.4542.0:34501 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 4030464~163840] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087558 7f868ff62700 10 op_submit_start 4011 2014-06-09 16:22:12.087560 7f868ff62700 5 queue_transactions (trailing journal) 4011 2014-06-09 16:22:12.087562 7f868ff62700 5 queue_op 0x30a3f80 seq 4011 osr(2.33a 0x2ed0ed0) 828 bytes (queue has 50 ops and 1081071 bytes) 2014-06-09 16:22:12.087566 7f868ff62700 10 op_submit_finish 4011 2014-06-09 16:22:12.087558 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.652( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.652( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.087570 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.473( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:12.087582 7f86a6b95700 5 -- op tracker -- , seq: 3638, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34501 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 4030464~163840] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087587 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.087591 7f86a5610700 5 _do_op 0xc34bda0 seq 3963 osr(2.52d 0x2ed19f0)/0x2ed19f0 start 2014-06-09 16:22:12.087589 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.087594 7f86a5610700 10 _do_transaction on 0xbf36b40 2014-06-09 16:22:12.087599 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.087604 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.087606 7f86a6b95700 15 osd.26 243 enqueue_op 0xdfa6200 prio 63 cost 163840 latency 0.000297 osd_op(client.4542.0:34501 rbd_data.115e6b8b4567.00000000000017ff [set-alloc-hint object_size 4194304 write_size 4194304,write 4030464~163840] 3.7121e06e RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.087636 7f868a757700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.087643 7f868ef60700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087647 7f8690f64700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087652 7f8691f66700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087656 7f868e75f700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.087660 7f8689755700 2 waiting 51 > 50 ops || 1081897 > 104857600 2014-06-09 16:22:12.087836 7f86a6b95700 5 -- op tracker -- , seq: 3639, time: 2014-06-09 16:22:12.087635, event: header_read, op: osd_op(client.4542.0:34527 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~159744] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087847 7f86a6b95700 5 -- op tracker -- , seq: 3639, time: 2014-06-09 16:22:12.087636, event: throttled, op: osd_op(client.4542.0:34527 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~159744] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087857 7f86a6b95700 5 -- op tracker -- , seq: 3639, time: 2014-06-09 16:22:12.087812, event: all_read, op: osd_op(client.4542.0:34527 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~159744] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087867 7f86a6b95700 5 -- op tracker -- , seq: 3639, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34527 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~159744] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.087880 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.087895 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6df00 prio 63 cost 159744 latency 0.000260 osd_op(client.4542.0:34527 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~159744] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.087961 7f868c75b700 5 queue_transactions existing osr(0.219 0x2bd6ad0)/0x2bd6ad0 2014-06-09 16:22:12.087965 7f868c75b700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087972 7f8689f56700 5 queue_transactions existing osr(2.48f 0x2ed1450)/0x2ed1450 2014-06-09 16:22:12.087974 7f8689f56700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087981 7f868af58700 5 queue_transactions existing osr(0.652 0x2bd7b90)/0x2bd7b90 2014-06-09 16:22:12.087983 7f868af58700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.087986 7f868ff62700 5 queue_transactions existing osr(0.473 0x2bd74d0)/0x2bd74d0 2014-06-09 16:22:12.087990 7f868ff62700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088122 7f868df5e700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088128 7f8690763700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088133 7f868cf5c700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088138 7f868b759700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088144 7f868f761700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088152 7f868bf5a700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088158 7f8692767700 2 waiting 51 > 50 ops || 2472013 > 104857600 2014-06-09 16:22:12.088164 7f8692f68700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088167 7f8691765700 2 waiting 51 > 50 ops || 1081899 > 104857600 2014-06-09 16:22:12.088173 7f86a5e11700 10 _do_op 0xc705b60 seq 3962 r = 0, finisher 0x31412a00 0 2014-06-09 16:22:12.088184 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd72bc0] 2014-06-09 16:22:12.088187 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd72bc0] 2014-06-09 16:22:12.088188 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.088189 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.088193 7f86a5e11700 10 _finish_op 0xc705b60 seq 3962 osr(3.587 0x3ed1840)/0x3ed1840 2014-06-09 16:22:12.088209 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.088216 7f868a757700 10 op_submit_start 4012 2014-06-09 16:22:12.088217 7f868a757700 5 queue_transactions (trailing journal) 4012 2014-06-09 16:22:12.088219 7f868a757700 5 queue_op 0xc64bce0 seq 4012 osr(1.327 0x384d370) 861 bytes (queue has 50 ops and 1081104 bytes) 2014-06-09 16:22:12.088224 7f868ef60700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088229 7f8690f64700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088227 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3963.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.088232 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.088233 7f8691f66700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088236 7f868e75f700 2 waiting 51 > 50 ops || 1081965 > 104857600 2014-06-09 16:22:12.088242 7f8689755700 2 waiting 51 > 50 ops || 1081930 > 104857600 2014-06-09 16:22:12.088243 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89a480 (1 active) 2014-06-09 16:22:12.088245 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0480 (2 active) 2014-06-09 16:22:12.088247 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412a00] 2014-06-09 16:22:12.088250 7f868a757700 10 op_submit_finish 4012 2014-06-09 16:22:12.088251 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412a00] 2014-06-09 16:22:12.088253 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.088254 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.088256 7f868c75b700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088261 7f8689f56700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088265 7f868af58700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088271 7f868ff62700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088274 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.088290 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.019839 7 0.003660 2014-06-09 16:22:12.088300 7f86a5610700 10 _do_op 0xc34bda0 seq 3963 r = 0, finisher 0xbd67020 0 2014-06-09 16:22:12.088309 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3045a420] 2014-06-09 16:22:12.088303 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.088311 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3045a420] 2014-06-09 16:22:12.088312 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.088313 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.088315 7f86a5610700 10 _finish_op 0xc34bda0 seq 3963 osr(2.52d 0x2ed19f0)/0x2ed19f0 2014-06-09 16:22:12.088326 7f8690763700 10 op_submit_start 4013 2014-06-09 16:22:12.088327 7f8690763700 5 queue_transactions (trailing journal) 4013 2014-06-09 16:22:12.088329 7f8690763700 5 queue_op 0x324b3920 seq 4013 osr(2.182 0x2ed0990) 828 bytes (queue has 50 ops and 1081104 bytes) 2014-06-09 16:22:12.088334 7f868cf5c700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088315 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.088338 7f868b759700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088346 7f868f761700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088348 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd67020] 2014-06-09 16:22:12.088343 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000040 0 0.000000 2014-06-09 16:22:12.088352 7f868bf5a700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088356 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd67020] 2014-06-09 16:22:12.088358 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.088359 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.088354 7f868a757700 5 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.088362 7f8692767700 2 waiting 51 > 50 ops || 2472046 > 104857600 2014-06-09 16:22:12.088366 7f8692f68700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088363 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.088370 7f8691765700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088374 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3faa20 (1 active) 2014-06-09 16:22:12.088374 7f868d75d700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088376 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd320 (2 active) 2014-06-09 16:22:12.088371 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.088380 7f8690763700 10 op_submit_finish 4013 2014-06-09 16:22:12.088381 7f8690f64700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088385 7f8691f66700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088390 7f868e75f700 2 waiting 51 > 50 ops || 1081965 > 104857600 2014-06-09 16:22:12.088385 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.494( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.494( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.088393 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.088404 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.088418 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.088413 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.088423 7f8690763700 5 queue_transactions existing osr(1.494 0x384d990)/0x384d990 2014-06-09 16:22:12.088426 7f8690763700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088427 7f868a757700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.088430 7f8689755700 2 waiting 51 > 50 ops || 1081930 > 104857600 2014-06-09 16:22:12.088435 7f868c75b700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088439 7f8689f56700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088442 7f868af58700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088447 7f86a5610700 5 _do_op 0xd471c20 seq 3965 osr(2.5be 0x2ed1b30)/0x2ed1b30 start 2014-06-09 16:22:12.088447 7f868ff62700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088450 7f86a5610700 10 _do_transaction on 0xc2c5380 2014-06-09 16:22:12.088453 7f868ef60700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088461 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.088444 7f868a757700 15 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.088489 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.088493 7f868a757700 5 queue_transactions existing osr(0.75 0x2bd6370)/0x2bd6370 2014-06-09 16:22:12.088496 7f868a757700 2 waiting 51 > 50 ops || 1081962 > 104857600 2014-06-09 16:22:12.088499 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.088501 7f868df5e700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088513 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3965.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.088517 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.088530 7f86a5610700 10 _do_op 0xd471c20 seq 3965 r = 0, finisher 0xbd744c0 0 2014-06-09 16:22:12.088540 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509140] 2014-06-09 16:22:12.088542 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509140] 2014-06-09 16:22:12.088543 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.088544 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.088547 7f86a5610700 10 _finish_op 0xd471c20 seq 3965 osr(2.5be 0x2ed1b30)/0x2ed1b30 2014-06-09 16:22:12.088557 7f868b759700 10 op_submit_start 4014 2014-06-09 16:22:12.088559 7f868b759700 5 queue_transactions (trailing journal) 4014 2014-06-09 16:22:12.088559 7f868f761700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088560 7f868b759700 5 queue_op 0xc88b3e0 seq 4014 osr(2.26d 0x2ed0de0) 828 bytes (queue has 50 ops and 1081104 bytes) 2014-06-09 16:22:12.088566 7f868bf5a700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088572 7f8692767700 2 waiting 51 > 50 ops || 2472046 > 104857600 2014-06-09 16:22:12.088577 7f8692f68700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088581 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd744c0] 2014-06-09 16:22:12.088586 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd744c0] 2014-06-09 16:22:12.088587 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.088588 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.088591 7f8691765700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088596 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd320 (1 active) 2014-06-09 16:22:12.088596 7f868d75d700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088598 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406d80 (2 active) 2014-06-09 16:22:12.088602 7f868b759700 10 op_submit_finish 4014 2014-06-09 16:22:12.088602 7f8690f64700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088605 7f8691f66700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088609 7f868e75f700 2 waiting 51 > 50 ops || 1081965 > 104857600 2014-06-09 16:22:12.088606 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.213( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.213( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:12.088633 7f86a5e11700 5 _do_op 0xd04f800 seq 3964 osr(3.2b3 0x3ed0f00)/0x3ed0f00 start 2014-06-09 16:22:12.088637 7f86a5e11700 10 _do_transaction on 0xc161140 2014-06-09 16:22:12.088641 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.088642 7f86a5e11700 15 _omap_setkeys meta/1fe0c935/pglog_3.2b3/0//-1 2014-06-09 16:22:12.088645 7f868b759700 5 queue_transactions existing osr(2.213 0x2ed0b70)/0x2ed0b70 2014-06-09 16:22:12.088647 7f868b759700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088651 7f8690763700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088656 7f8689755700 2 waiting 51 > 50 ops || 1081930 > 104857600 2014-06-09 16:22:12.088660 7f868c75b700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088664 7f8689f56700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088668 7f868af58700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088672 7f86a5610700 5 _do_op 0xc88ac00 seq 3966 osr(0.6c0 0x2bd7f50)/0x2bd7f50 start 2014-06-09 16:22:12.088672 7f868ff62700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088675 7f86a5610700 10 _do_transaction on 0x3489440 2014-06-09 16:22:12.088678 7f868ef60700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088686 7f868a757700 2 waiting 51 > 50 ops || 1081962 > 104857600 2014-06-09 16:22:12.088686 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.088691 7f868df5e700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088696 7f868cf5c700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088706 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe0c935/pglog_3.2b3/0//-1 2014-06-09 16:22:12.088721 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.088741 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.088755 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3966.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.088759 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.088781 7f86a5610700 10 _do_op 0xc88ac00 seq 3966 r = 0, finisher 0xbd73e40 0 2014-06-09 16:22:12.088791 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d1320] 2014-06-09 16:22:12.088795 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d1320] 2014-06-09 16:22:12.088797 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.088798 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.088801 7f86a5610700 10 _finish_op 0xc88ac00 seq 3966 osr(0.6c0 0x2bd7f50)/0x2bd7f50 2014-06-09 16:22:12.088809 7f868f761700 10 op_submit_start 4015 2014-06-09 16:22:12.088810 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.088812 7f868f761700 5 queue_transactions (trailing journal) 4015 2014-06-09 16:22:12.088812 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd73e40] 2014-06-09 16:22:12.088815 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd73e40] 2014-06-09 16:22:12.088814 7f868f761700 5 queue_op 0x11fce360 seq 4015 osr(3.18e 0x3ed0af0) 828 bytes (queue has 50 ops and 1081104 bytes) 2014-06-09 16:22:12.088816 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.088817 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.088819 7f868bf5a700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088820 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406d80 (1 active) 2014-06-09 16:22:12.088825 7f8692767700 2 waiting 51 > 50 ops || 2472046 > 104857600 2014-06-09 16:22:12.088827 7f86a5e11700 15 _omap_rmkeys meta/1fe0c935/pglog_3.2b3/0//-1 2014-06-09 16:22:12.088829 7f8692f68700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088833 7f8691765700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088833 7f86a5e11700 10 _omap_rmkeys meta/1fe0c935/pglog_3.2b3/0//-1 r = 0 2014-06-09 16:22:12.088822 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9226c0 (2 active) 2014-06-09 16:22:12.088838 7f868d75d700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088839 7f86a5e11700 15 _omap_setkeys meta/1fe0c935/pglog_3.2b3/0//-1 2014-06-09 16:22:12.088841 7f86a5610700 5 _do_op 0xcc87c20 seq 3967 osr(0.771 0x384c310)/0x384c310 start 2014-06-09 16:22:12.088842 7f868f761700 10 op_submit_finish 4015 2014-06-09 16:22:12.088843 7f86a5610700 10 _do_transaction on 0xbf90240 2014-06-09 16:22:12.088843 7f8690f64700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088847 7f8691f66700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088848 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.088851 7f868e75f700 2 waiting 51 > 50 ops || 1081965 > 104857600 2014-06-09 16:22:12.088851 7f86a5e11700 15 _touch 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.088855 7f868b759700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088846 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.6e7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.6e7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.088860 7f8690763700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088864 7f8689755700 2 waiting 51 > 50 ops || 1081930 > 104857600 2014-06-09 16:22:12.088869 7f868c75b700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088875 7f8689f56700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088880 7f868af58700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088886 7f868ff62700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088892 7f868ef60700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088897 7f868a757700 2 waiting 51 > 50 ops || 1081962 > 104857600 2014-06-09 16:22:12.088902 7f868df5e700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088906 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.088906 7f868cf5c700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088912 7f868f761700 5 queue_transactions existing osr(0.6e7 0x384c1b0)/0x384c1b0 2014-06-09 16:22:12.088915 7f868f761700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.088943 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.2b3_head oid 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.088961 7f86a5e11700 20 write_state seq is 30163 2014-06-09 16:22:12.088966 7f86a5e11700 20 set_header setting 30162 cid 3.2b3_head oid 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 parent seq 0 2014-06-09 16:22:12.088975 7f86a5e11700 10 _touch 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 = 0 2014-06-09 16:22:12.088979 7f86a5e11700 15 _write 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 0~884736 2014-06-09 16:22:12.088992 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.089058 7f86a6b95700 5 -- op tracker -- , seq: 3640, time: 2014-06-09 16:22:12.087912, event: header_read, op: osd_op(client.4542.0:34528 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 159744~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.089069 7f86a6b95700 5 -- op tracker -- , seq: 3640, time: 2014-06-09 16:22:12.087913, event: throttled, op: osd_op(client.4542.0:34528 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 159744~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.089079 7f86a6b95700 5 -- op tracker -- , seq: 3640, time: 2014-06-09 16:22:12.089032, event: all_read, op: osd_op(client.4542.0:34528 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 159744~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.089092 7f86a6b95700 5 -- op tracker -- , seq: 3640, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34528 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 159744~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.089098 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.089135 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6de00 prio 63 cost 516096 latency 0.001206 osd_op(client.4542.0:34528 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 159744~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.090037 7f86a6b95700 5 -- op tracker -- , seq: 3641, time: 2014-06-09 16:22:12.089159, event: header_read, op: osd_op(client.4542.0:34529 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 675840~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.090049 7f86a6b95700 5 -- op tracker -- , seq: 3641, time: 2014-06-09 16:22:12.089160, event: throttled, op: osd_op(client.4542.0:34529 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 675840~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.090060 7f86a6b95700 5 -- op tracker -- , seq: 3641, time: 2014-06-09 16:22:12.089999, event: all_read, op: osd_op(client.4542.0:34529 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 675840~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.090070 7f86a6b95700 5 -- op tracker -- , seq: 3641, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34529 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 675840~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.090076 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.090094 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6dd00 prio 63 cost 516096 latency 0.000934 osd_op(client.4542.0:34529 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 675840~516096] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.091840 7f86a6b95700 5 -- op tracker -- , seq: 3642, time: 2014-06-09 16:22:12.090119, event: header_read, op: osd_op(client.4542.0:34530 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 1191936~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.091853 7f86a6b95700 5 -- op tracker -- , seq: 3642, time: 2014-06-09 16:22:12.090120, event: throttled, op: osd_op(client.4542.0:34530 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 1191936~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.091863 7f86a6b95700 5 -- op tracker -- , seq: 3642, time: 2014-06-09 16:22:12.091795, event: all_read, op: osd_op(client.4542.0:34530 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 1191936~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.091873 7f86a6b95700 5 -- op tracker -- , seq: 3642, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34530 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 1191936~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.091881 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.091906 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6dc00 prio 63 cost 1032192 latency 0.001786 osd_op(client.4542.0:34530 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 1191936~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.092225 7f86a5e11700 10 _generic_write 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 0~884736 = 0 2014-06-09 16:22:12.092280 7f86a5e11700 15 _setattrs 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.092294 7f86a5e11700 10 _setattrs 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 = 0 2014-06-09 16:22:12.092298 7f86a5e11700 15 _setattrs 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 2014-06-09 16:22:12.092303 7f86a5e11700 10 _setattrs 3.2b3_head/8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 = 0 2014-06-09 16:22:12.092430 7f86a5e11700 10 cid: oid: 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 not skipping op, *spos 3964.0.0 2014-06-09 16:22:12.092435 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.092463 7f86a5e11700 20 set_header setting 30162 cid 3.2b3_head oid 8ee862b3/rbd_data.117b6b8b4567.00000000000017ef/head//3 parent seq 0 2014-06-09 16:22:12.092531 7f86a5e11700 10 cid: oid: 1fe0c935/pglog_3.2b3/0//-1 skipping op, spos 3964.0.0 <= header.spos 150272.0.0 2014-06-09 16:22:12.092537 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3964.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.092540 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.093457 7f86a6b95700 5 -- op tracker -- , seq: 3643, time: 2014-06-09 16:22:12.091966, event: header_read, op: osd_op(client.4542.0:34531 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.093469 7f86a6b95700 5 -- op tracker -- , seq: 3643, time: 2014-06-09 16:22:12.091968, event: throttled, op: osd_op(client.4542.0:34531 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.093484 7f86a6b95700 5 -- op tracker -- , seq: 3643, time: 2014-06-09 16:22:12.093416, event: all_read, op: osd_op(client.4542.0:34531 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.093495 7f86a6b95700 5 -- op tracker -- , seq: 3643, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34531 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.093502 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.093527 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6db00 prio 63 cost 1032192 latency 0.001560 osd_op(client.4542.0:34531 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 2224128~1032192] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.095123 7f86a6b95700 5 -- op tracker -- , seq: 3644, time: 2014-06-09 16:22:12.093554, event: header_read, op: osd_op(client.4542.0:34532 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 3256320~937984] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.095135 7f86a6b95700 5 -- op tracker -- , seq: 3644, time: 2014-06-09 16:22:12.093556, event: throttled, op: osd_op(client.4542.0:34532 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 3256320~937984] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.095145 7f86a6b95700 5 -- op tracker -- , seq: 3644, time: 2014-06-09 16:22:12.095075, event: all_read, op: osd_op(client.4542.0:34532 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 3256320~937984] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.095155 7f86a6b95700 5 -- op tracker -- , seq: 3644, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34532 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 3256320~937984] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.095163 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.095219 7f86a6b95700 15 osd.26 243 enqueue_op 0x30c6da00 prio 63 cost 937984 latency 0.001664 osd_op(client.4542.0:34532 rbd_data.115e6b8b4567.0000000000001803 [set-alloc-hint object_size 4194304 write_size 4194304,write 3256320~937984] 3.b67825f9 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.095809 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.095821 7f86a6b95700 5 -- op tracker -- , seq: 3645, time: 2014-06-09 16:22:12.095242, event: header_read, op: osd_op(client.4542.0:34600 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~405504] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.095832 7f86a6b95700 5 -- op tracker -- , seq: 3645, time: 2014-06-09 16:22:12.095243, event: throttled, op: osd_op(client.4542.0:34600 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~405504] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.095836 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3967.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.095841 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.095842 7f86a6b95700 5 -- op tracker -- , seq: 3645, time: 2014-06-09 16:22:12.095766, event: all_read, op: osd_op(client.4542.0:34600 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~405504] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.095857 7f86a6b95700 5 -- op tracker -- , seq: 3645, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34600 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~405504] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.095864 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.095887 7f86a5610700 10 _do_op 0xcc87c20 seq 3967 r = 0, finisher 0x30e15f80 0 2014-06-09 16:22:12.095882 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976c600 prio 63 cost 405504 latency 0.000639 osd_op(client.4542.0:34600 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 0~405504] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.095899 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a0020c0] 2014-06-09 16:22:12.095901 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a0020c0] 2014-06-09 16:22:12.095903 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.095903 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.095907 7f86a5610700 10 _finish_op 0xcc87c20 seq 3967 osr(0.771 0x384c310)/0x384c310 2014-06-09 16:22:12.095922 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e15f80] 2014-06-09 16:22:12.095927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e15f80] 2014-06-09 16:22:12.095928 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.095929 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.095932 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9226c0 (1 active) 2014-06-09 16:22:12.095935 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9226c0 (2 active) 2014-06-09 16:22:12.095937 7f868bf5a700 10 op_submit_start 4016 2014-06-09 16:22:12.095944 7f86a5610700 5 _do_op 0xcc86780 seq 3968 osr(0.771 0x384c310)/0x384c310 start 2014-06-09 16:22:12.095938 7f8692767700 2 waiting 51 > 50 ops || 2472046 > 104857600 2014-06-09 16:22:12.095947 7f868bf5a700 5 queue_transactions (trailing journal) 4016 2014-06-09 16:22:12.095950 7f868bf5a700 5 queue_op 0xc34a0c0 seq 4016 osr(0.529 0x2bd7930) 828 bytes (queue has 50 ops and 1081104 bytes) 2014-06-09 16:22:12.095952 7f8692f68700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.095954 7f868bf5a700 10 op_submit_finish 4016 2014-06-09 16:22:12.095957 7f8691765700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.095962 7f86a5610700 10 _do_transaction on 0xc2c4000 2014-06-09 16:22:12.095964 7f868d75d700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.095968 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.095972 7f8690f64700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.095977 7f8691f66700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.095981 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41980 (19 active) 2014-06-09 16:22:12.095982 7f868e75f700 2 waiting 51 > 50 ops || 1081965 > 104857600 2014-06-09 16:22:12.095990 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.096000 7f868b759700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.096008 7f8690763700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.096013 7f8689755700 2 waiting 51 > 50 ops || 1081930 > 104857600 2014-06-09 16:22:12.095996 7f868bf5a700 10 osd.26 243 dequeue_op 0x30c6d900 prio 196 cost 0 latency 0.009826 osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) v2 pg pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=214'596 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.096018 7f868c75b700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.096019 7f868bf5a700 5 -- op tracker -- , seq: 3637, time: 2014-06-09 16:22:12.096016, event: reached_pg, op: osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.096033 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=214'596 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x30c6d900 2014-06-09 16:22:12.096068 7f8689f56700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.096068 7f868bf5a700 5 -- op tracker -- , seq: 3637, time: 2014-06-09 16:22:12.096064, event: started, op: osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.096075 7f868af58700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.096081 7f868ff62700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.096074 7f868bf5a700 7 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=214'596 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 25 op ack_type 4 from 3 2014-06-09 16:22:12.096089 7f868ef60700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.096095 7f868a757700 2 waiting 51 > 50 ops || 1081962 > 104857600 2014-06-09 16:22:12.096101 7f868df5e700 2 waiting 51 > 50 ops || 1081932 > 104857600 2014-06-09 16:22:12.096094 7f868bf5a700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.096088, event: sub_op_commit_rec, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.096107 7f868bf5a700 10 osd.26 243 dequeue_op 0x30c6d900 finish 2014-06-09 16:22:12.096108 7f86a5e11700 10 _do_op 0xd04f800 seq 3964 r = 0, finisher 0x13965440 0x39a9780 2014-06-09 16:22:12.096115 7f86a5e11700 10 _finish_op 0xd04f800 seq 3964 osr(3.2b3 0x3ed0f00)/0x3ed0f00 2014-06-09 16:22:12.096115 7f868bf5a700 5 -- op tracker -- , seq: 3637, time: 2014-06-09 16:22:12.096111, event: done, op: osd_sub_op_reply(client.4573.0:34037 3.206 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.096129 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.096132 7f868cf5c700 10 op_submit_start 4017 2014-06-09 16:22:12.096131 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13965440] 2014-06-09 16:22:12.096134 7f868cf5c700 5 queue_transactions (trailing journal) 4017 2014-06-09 16:22:12.096135 7f868cf5c700 5 queue_op 0x30060c60 seq 4017 osr(2.236 0x2ed0ad0) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.096141 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x139642e0] 2014-06-09 16:22:12.096144 7f8692f68700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096148 7f8691765700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096135 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 24 2014-06-09 16:22:12.096158 7f86a4e0f700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.096151, event: op_applied, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.096168 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'637 2014-06-09 16:22:12.096165 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) 2014-06-09 16:22:12.096179 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_applied: repop tid 24 all applied 2014-06-09 16:22:12.096192 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.52d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.52d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.096190 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc218d80 243'637 rep_tid=24 committed?=0 applied?=1 lock=0 op=osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.096206 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096104 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.096212 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096216 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096207 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] sending ack on repgather(0xc218d80 243'637 rep_tid=24 committed?=0 applied?=1 lock=0 op=osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4) 0x257c6780 2014-06-09 16:22:12.096220 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.096225 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096237 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.096241 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096245 7f868bf5a700 5 queue_transactions existing osr(2.52d 0x2ed19f0)/0x2ed19f0 2014-06-09 16:22:12.096249 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096254 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13965440] 2014-06-09 16:22:12.096254 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096256 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.096257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.096246 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 24 2014-06-09 16:22:12.096260 7f868c75b700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096265 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0480 (1 active) 2014-06-09 16:22:12.096268 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383eb40 (2 active) 2014-06-09 16:22:12.096273 7f868cf5c700 10 op_submit_finish 4017 2014-06-09 16:22:12.096268 7f86a460e700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.096259, event: op_commit, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.096279 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'636 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 24 all committed 2014-06-09 16:22:12.096278 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:12.096289 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 0'0 active] eval_repop repgather(0xc218d80 243'637 rep_tid=24 committed?=1 applied?=1 lock=0 op=osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.096313 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.096304 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 0'0 active] log_op_stats osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4 inb 885120 outb 0 rlat 1.794271 lat 1.794336 2014-06-09 16:22:12.096318 7f868cf5c700 5 queue_transactions existing osr(3.76f 0x3ed1e20)/0x3ed1e20 2014-06-09 16:22:12.096320 7f868cf5c700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096325 7f86a5e11700 5 _do_op 0xc88aba0 seq 3969 osr(3.7bc 0xb7f0190)/0xb7f0190 start 2014-06-09 16:22:12.096328 7f86a5e11700 10 _do_transaction on 0xc2becc0 2014-06-09 16:22:12.096325 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 0'0 active] publish_stats_to_osd 243:1061 2014-06-09 16:22:12.096333 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.096338 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 0'0 active] sending commit on repgather(0xc218d80 243'637 rep_tid=24 committed?=1 applied?=1 lock=0 op=osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4) 0x9a09180 2014-06-09 16:22:12.096386 7f86a460e700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.096381, event: commit_sent, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.096393 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.096400 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] removing repgather(0xc218d80 243'637 rep_tid=24 committed?=1 applied?=1 op=osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.096409 7f86a460e700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] q front is repgather(0xc218d80 243'637 rep_tid=24 committed?=1 applied?=1 op=osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.096417 7f86a460e700 20 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] remove_repop repgather(0xc218d80 243'637 rep_tid=24 committed?=1 applied?=1 op=osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.096425 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] requeue_ops 2014-06-09 16:22:12.096443 7f86a460e700 5 -- op tracker -- , seq: 251, time: 2014-06-09 16:22:12.096440, event: done, op: osd_op(client.4571.0:35826 rbd_data.117b6b8b4567.00000000000017ef [set-alloc-hint object_size 4194304 write_size 4194304,write 0~884736] 3.8ee862b3 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.096462 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x139642e0] 2014-06-09 16:22:12.096463 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.096464 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.096467 7f8689755700 2 waiting 51 > 50 ops || 195624 > 104857600 2014-06-09 16:22:12.096471 7f8689f56700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096476 7f868ff62700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096484 7f868ef60700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096491 7f868a757700 2 waiting 51 > 50 ops || 195656 > 104857600 2014-06-09 16:22:12.096495 7f868df5e700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096500 7f8692767700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.096504 7f868af58700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096573 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3968.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.096578 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.096592 7f86a5610700 10 _do_op 0xcc86780 seq 3968 r = 0, finisher 0x3061e600 0 2014-06-09 16:22:12.096600 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7c4e0] 2014-06-09 16:22:12.096602 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7c4e0] 2014-06-09 16:22:12.096603 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.096604 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.096628 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.096645 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3969.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.096649 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.096663 7f86a5e11700 10 _do_op 0xc88aba0 seq 3969 r = 0, finisher 0x324d11a0 0 2014-06-09 16:22:12.096670 7f86a5e11700 10 _finish_op 0xc88aba0 seq 3969 osr(3.7bc 0xb7f0190)/0xb7f0190 2014-06-09 16:22:12.096682 7f8691765700 10 op_submit_start 4018 2014-06-09 16:22:12.096683 7f8691765700 5 queue_transactions (trailing journal) 4018 2014-06-09 16:22:12.096683 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096685 7f8691765700 5 queue_op 0x15997bc0 seq 4018 osr(3.1d5 0x3ed09f0) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.096689 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096693 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096695 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d11a0] 2014-06-09 16:22:12.096696 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.096701 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d11a0] 2014-06-09 16:22:12.096702 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.096704 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.096707 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383eb40 (1 active) 2014-06-09 16:22:12.096709 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4aca20 (2 active) 2014-06-09 16:22:12.096712 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096714 7f8691765700 10 op_submit_finish 4018 2014-06-09 16:22:12.096717 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096722 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5e220] 2014-06-09 16:22:12.096724 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5e220] 2014-06-09 16:22:12.096723 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096726 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.096727 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.096730 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096718 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.48d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.48d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.096729 7f86a5610700 10 _finish_op 0xcc86780 seq 3968 osr(0.771 0x384c310)/0x384c310 2014-06-09 16:22:12.096757 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.096758 7f868d75d700 10 op_submit_start 4019 2014-06-09 16:22:12.096760 7f868d75d700 5 queue_transactions (trailing journal) 4019 2014-06-09 16:22:12.096761 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061e600] 2014-06-09 16:22:12.096762 7f8691765700 5 queue_transactions existing osr(2.48d 0x2ed1470)/0x2ed1470 2014-06-09 16:22:12.096762 7f868d75d700 5 queue_op 0x16fd4e40 seq 4019 osr(1.1ef 0x384cdb0) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.096764 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061e600] 2014-06-09 16:22:12.096764 7f8691765700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096766 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.096767 7f868cf5c700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096767 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.096772 7f8689f56700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096775 7f868c75b700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096779 7f868ef60700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096778 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9226c0 (1 active) 2014-06-09 16:22:12.096781 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fdd40 (2 active) 2014-06-09 16:22:12.096783 7f868a757700 2 waiting 51 > 50 ops || 195656 > 104857600 2014-06-09 16:22:12.096785 7f86a5610700 5 _do_op 0xcca7260 seq 3971 osr(1.14c 0x384c810)/0x384c810 start 2014-06-09 16:22:12.096786 7f868d75d700 10 op_submit_finish 4019 2014-06-09 16:22:12.096787 7f868df5e700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096788 7f86a5610700 10 _do_transaction on 0x3472240 2014-06-09 16:22:12.096791 7f8692767700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.096794 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.096795 7f868af58700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096799 7f8692f68700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096791 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.1e8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.096772 7f86a5e11700 5 _do_op 0xcca68a0 seq 3970 osr(2.40c 0x2ed1250)/0x2ed1250 start 2014-06-09 16:22:12.096805 7f86a5e11700 10 _do_transaction on 0xc2395c0 2014-06-09 16:22:12.096810 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.096825 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.096830 7f868d75d700 5 queue_transactions existing osr(1.1e8 0x384cdf0)/0x384cdf0 2014-06-09 16:22:12.096832 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096831 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.096836 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096840 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096843 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.096848 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096854 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096884 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096891 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096896 7f868ff62700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.096902 7f8689755700 2 waiting 51 > 50 ops || 195624 > 104857600 2014-06-09 16:22:12.096952 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3971.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.096956 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.096966 7f86a6b95700 5 -- op tracker -- , seq: 3646, time: 2014-06-09 16:22:12.095906, event: header_read, op: osd_op(client.4542.0:34601 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 405504~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.096980 7f86a6b95700 5 -- op tracker -- , seq: 3646, time: 2014-06-09 16:22:12.095907, event: throttled, op: osd_op(client.4542.0:34601 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 405504~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.096993 7f86a6b95700 5 -- op tracker -- , seq: 3646, time: 2014-06-09 16:22:12.096928, event: all_read, op: osd_op(client.4542.0:34601 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 405504~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.097007 7f86a6b95700 5 -- op tracker -- , seq: 3646, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34601 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 405504~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.097009 7f86a5610700 10 _do_op 0xcca7260 seq 3971 r = 0, finisher 0x3062a8c0 0 2014-06-09 16:22:12.097014 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.097020 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd79700] 2014-06-09 16:22:12.097022 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd79700] 2014-06-09 16:22:12.097024 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.097025 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.097028 7f86a5610700 10 _finish_op 0xcca7260 seq 3971 osr(1.14c 0x384c810)/0x384c810 2014-06-09 16:22:12.097032 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976c500 prio 63 cost 516096 latency 0.001126 osd_op(client.4542.0:34601 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 405504~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.097041 7f868cf5c700 10 op_submit_start 4020 2014-06-09 16:22:12.097043 7f868cf5c700 5 queue_transactions (trailing journal) 4020 2014-06-09 16:22:12.097042 7f8689f56700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097044 7f868cf5c700 5 queue_op 0x30061320 seq 4020 osr(3.76f 0x3ed1e20) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.097047 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062a8c0] 2014-06-09 16:22:12.097051 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062a8c0] 2014-06-09 16:22:12.097053 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.097054 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.097056 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fdd40 (1 active) 2014-06-09 16:22:12.097058 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89eb40 (2 active) 2014-06-09 16:22:12.097063 7f86a5610700 5 _do_op 0xcca6a80 seq 3972 osr(2.48f 0x2ed1450)/0x2ed1450 start 2014-06-09 16:22:12.097064 7f868cf5c700 10 op_submit_finish 4020 2014-06-09 16:22:12.097064 7f868c75b700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097065 7f86a5610700 10 _do_transaction on 0xbf89380 2014-06-09 16:22:12.097071 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.097070 7f868ef60700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097077 7f868a757700 2 waiting 51 > 50 ops || 195656 > 104857600 2014-06-09 16:22:12.097081 7f868df5e700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097069 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.381( v 213'647 (0'0,213'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:12.097084 7f8692767700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.097088 7f868af58700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097092 7f8692f68700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097108 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.097113 7f868cf5c700 5 queue_transactions existing osr(3.381 0x3ed0fe0)/0x3ed0fe0 2014-06-09 16:22:12.097117 7f868cf5c700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097121 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097125 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097129 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097132 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.097138 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097143 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097134 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.097150 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097157 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097162 7f868ff62700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097162 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3972.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.097166 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.097183 7f86a5610700 10 _do_op 0xcca6a80 seq 3972 r = 0, finisher 0x318c3fc0 0 2014-06-09 16:22:12.097192 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c3880] 2014-06-09 16:22:12.097194 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c3880] 2014-06-09 16:22:12.097195 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.097197 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.097200 7f86a5610700 10 _finish_op 0xcca6a80 seq 3972 osr(2.48f 0x2ed1450)/0x2ed1450 2014-06-09 16:22:12.097210 7f8691765700 10 op_submit_start 4021 2014-06-09 16:22:12.097212 7f8691765700 5 queue_transactions (trailing journal) 4021 2014-06-09 16:22:12.097213 7f8691765700 5 queue_op 0x15996d20 seq 4021 osr(2.48d 0x2ed1470) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.097217 7f868c75b700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097222 7f868ef60700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097226 7f868a757700 2 waiting 51 > 50 ops || 195656 > 104857600 2014-06-09 16:22:12.097235 7f868df5e700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097240 7f8692767700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.097245 7f868af58700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097250 7f8692f68700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097255 7f868cf5c700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097260 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c3fc0] 2014-06-09 16:22:12.097261 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097266 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c3fc0] 2014-06-09 16:22:12.097265 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097267 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.097269 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.097270 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097272 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89eb40 (1 active) 2014-06-09 16:22:12.097274 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e240 (2 active) 2014-06-09 16:22:12.097275 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.097280 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097284 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097289 7f8691765700 10 op_submit_finish 4021 2014-06-09 16:22:12.097289 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097296 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097301 7f868ff62700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097312 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x153584b0 (19 active) 2014-06-09 16:22:12.097320 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.097333 7f8689755700 2 waiting 51 > 50 ops || 195624 > 104857600 2014-06-09 16:22:12.097325 7f8691765700 10 osd.26 243 dequeue_op 0x1804df00 prio 63 cost 1531904 latency 1.781982 osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.097350 7f8691765700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.097345, event: reached_pg, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.097359 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.097373 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x1804df00 2014-06-09 16:22:12.097384 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.097408 7f8691765700 15 getattr 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 '_' 2014-06-09 16:22:12.097415 7f8689f56700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097419 7f86a5610700 5 _do_op 0x30060a20 seq 3973 osr(3.3a8 0x3ed0f80)/0x3ed0f80 start 2014-06-09 16:22:12.097422 7f86a5610700 10 _do_transaction on 0xbf7ca80 2014-06-09 16:22:12.097428 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.097463 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.097542 7f8691765700 10 getattr got.size() is 0 2014-06-09 16:22:12.097544 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3970.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.097549 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.097545 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 but can_create 2014-06-09 16:22:12.097559 7f8691765700 15 getattr 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 'snapset' 2014-06-09 16:22:12.097593 7f8691765700 10 getattr got.size() is 0 2014-06-09 16:22:12.097596 7f8691765700 15 getattr 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/snapdir//3 'snapset' 2014-06-09 16:22:12.097626 7f8691765700 10 getattr got.size() is 0 2014-06-09 16:22:12.097636 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x31064b00 b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.097648 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.097658 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.097668 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x31064b00 b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 rwstate(none n=0 w=0) oi: b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2e877180 snapset: 0=[]:[] 2014-06-09 16:22:12.097683 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 @head oi=b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.097695 7f86a5e11700 10 _do_op 0xcca68a0 seq 3970 r = 0, finisher 0xc5a6000 0 2014-06-09 16:22:12.097701 7f8691765700 15 getattr 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/snapdir//3 '_' 2014-06-09 16:22:12.097721 7f86a6b95700 5 -- op tracker -- , seq: 3647, time: 2014-06-09 16:22:12.097054, event: header_read, op: osd_op(client.4542.0:34602 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 921600~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.097733 7f8691765700 10 getattr got.size() is 0 2014-06-09 16:22:12.097733 7f86a6b95700 5 -- op tracker -- , seq: 3647, time: 2014-06-09 16:22:12.097056, event: throttled, op: osd_op(client.4542.0:34602 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 921600~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.097701 7f86a5e11700 10 _finish_op 0xcca68a0 seq 3970 osr(2.40c 0x2ed1250)/0x2ed1250 2014-06-09 16:22:12.097736 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid b4f7744d/rbd_data.11806b8b4567.000000000000181b/snapdir//3 and !can_create 2014-06-09 16:22:12.097744 7f86a6b95700 5 -- op tracker -- , seq: 3647, time: 2014-06-09 16:22:12.097687, event: all_read, op: osd_op(client.4542.0:34602 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 921600~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.097749 7f868ef60700 10 op_submit_start 4022 2014-06-09 16:22:12.097751 7f868ef60700 5 queue_transactions (trailing journal) 4022 2014-06-09 16:22:12.097752 7f868ef60700 5 queue_op 0x760ec60 seq 4022 osr(1.289 0x384ceb0) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.097756 7f8691765700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.097750, event: started, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.097766 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30a8ac00 2014-06-09 16:22:12.097785 7f8691765700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.097780, event: started, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.097795 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a6000] 2014-06-09 16:22:12.097800 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a6000] 2014-06-09 16:22:12.097801 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.097800 7f86a6b95700 5 -- op tracker -- , seq: 3647, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34602 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 921600~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.097803 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.097792 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] ov 0'0 av 243'636 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.097805 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4aca20 (1 active) 2014-06-09 16:22:12.097807 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1e60 (2 active) 2014-06-09 16:22:12.097807 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.097812 7f868ef60700 10 op_submit_finish 4022 2014-06-09 16:22:12.097806 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 2014-06-09 16:22:12.097817 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.097825 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976c400 prio 63 cost 516096 latency 0.000771 osd_op(client.4542.0:34602 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 921600~516096] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.097816 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 242 MNotifyRec from 8 notify: (query_epoch:242, epoch_sent:243, info:3.1ff( v 214'683 (0'0,214'683] local-les=212 n=131 ec=118 les/c 212/213 242/242/242)) 2014-06-09 16:22:12.097830 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~1531904 2014-06-09 16:22:12.097837 7f868ef60700 10 osd.26 pg_epoch: 243 pg[3.1ff( v 214'683 lc 214'682 (0'0,214'683] local-les=243 n=131 ec=118 les/c 212/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'683 lcod 0'0 mlcod 0'0 inactive m=1] state: Active: got notify from 8, already have info from that osd, ignoring 2014-06-09 16:22:12.097851 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.097845 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0x1078a600 2014-06-09 16:22:12.097856 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 snapset=0x2e8771b8 snapc=0=[] 2014-06-09 16:22:12.097858 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.34d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=8-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.34d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.097867 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.097876 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.097885 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 0x30a8ac00 op modify 2014-06-09 16:22:12.097904 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.097909 7f868ef60700 5 queue_transactions existing osr(2.34d 0x2ed0e70)/0x2ed0e70 2014-06-09 16:22:12.097897 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:05.777968 2014-06-09 16:22:12.097912 7f868ef60700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.097922 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=122 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.097944 7f86a5e11700 5 _do_op 0x324cc300 seq 3974 osr(1.3cb 0x384d4f0)/0x384d4f0 start 2014-06-09 16:22:12.097948 7f86a5e11700 10 _do_transaction on 0xc0dfb00 2014-06-09 16:22:12.097941 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.097951 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 27 on osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.097967 7f8691765700 7 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 27 o b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.097994 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.097995 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd77f80] 2014-06-09 16:22:12.097997 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd77f80] 2014-06-09 16:22:12.097999 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.098000 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.097991 7f8691765700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.097986, event: waiting for subops from 9, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.098003 7f868a757700 2 waiting 51 > 50 ops || 195656 > 104857600 2014-06-09 16:22:12.098008 7f868df5e700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098012 7f8692767700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.098016 7f868af58700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098020 7f8692f68700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098018 7f8691765700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.098086 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.098041 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 214'635 (0'0,214'635] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'635], crt=0'0) [243'636 (0'0) modify b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 by client.4576.0:35874 2014-06-09 10:19:05.777968] 2014-06-09 16:22:12.098142 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3974.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.098146 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.098140 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'636 (0'0) modify b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 by client.4576.0:35874 2014-06-09 10:19:05.777968 2014-06-09 16:22:12.098162 7f86a5e11700 10 _do_op 0x324cc300 seq 3974 r = 0, finisher 0x139656c0 0 2014-06-09 16:22:12.098160 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.098189 7f8691765700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'636, trimmed: 2014-06-09 16:22:12.098209 7f8691765700 5 queue_transactions existing osr(3.44d 0x3ed1180)/0x3ed1180 2014-06-09 16:22:12.098213 7f8691765700 2 waiting 51 > 50 ops || 1729100 > 104857600 2014-06-09 16:22:12.098218 7f868cf5c700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098224 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098228 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098167 7f86a5e11700 10 _finish_op 0x324cc300 seq 3974 osr(1.3cb 0x384d4f0)/0x384d4f0 2014-06-09 16:22:12.098232 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098237 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.098243 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098257 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098275 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098283 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.098289 7f868ff62700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.099056 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6c0a0] 2014-06-09 16:22:12.099061 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6c0a0] 2014-06-09 16:22:12.099062 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.099063 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.099153 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.099173 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3973.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.099178 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.099208 7f86a5610700 10 _do_op 0x30060a20 seq 3973 r = 0, finisher 0x3251d0e0 0 2014-06-09 16:22:12.099218 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e8400] 2014-06-09 16:22:12.099220 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e8400] 2014-06-09 16:22:12.099222 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.099222 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.100881 7f8689755700 2 waiting 51 > 50 ops || 195624 > 104857600 2014-06-09 16:22:12.100889 7f8689f56700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100897 7f868c75b700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100913 7f868a757700 2 waiting 51 > 50 ops || 195656 > 104857600 2014-06-09 16:22:12.100914 7f868ef60700 10 op_submit_start 4023 2014-06-09 16:22:12.100915 7f868ef60700 5 queue_transactions (trailing journal) 4023 2014-06-09 16:22:12.100918 7f868df5e700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100918 7f868ef60700 5 queue_op 0x760ec00 seq 4023 osr(2.34d 0x2ed0e70) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.100924 7f868af58700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100902 7f8682ee9700 5 -- op tracker -- , seq: 3648, time: 2014-06-09 16:22:12.089683, event: header_read, op: MOSDPGPush(3.21e 243 [PushOp(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3, version: 221'617, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3@221'617, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.100929 7f8692f68700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100934 7f8692767700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.100936 7f8682ee9700 5 -- op tracker -- , seq: 3648, time: 2014-06-09 16:22:12.089685, event: throttled, op: MOSDPGPush(3.21e 243 [PushOp(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3, version: 221'617, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3@221'617, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.100941 7f868cf5c700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100944 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1e60 (1 active) 2014-06-09 16:22:12.100947 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x139656c0] 2014-06-09 16:22:12.100948 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359440 (2 active) 2014-06-09 16:22:12.100950 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100952 7f86a5e11700 5 _do_op 0xbf7ff20 seq 3975 osr(2.654 0x2ed1fd0)/0x2ed1fd0 start 2014-06-09 16:22:12.100955 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x139656c0] 2014-06-09 16:22:12.100956 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100956 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.100956 7f86a5e11700 10 _do_transaction on 0xc0e5440 2014-06-09 16:22:12.100958 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.100961 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100964 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.100966 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.100972 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100979 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100985 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100967 7f8682ee9700 5 -- op tracker -- , seq: 3648, time: 2014-06-09 16:22:12.100849, event: all_read, op: MOSDPGPush(3.21e 243 [PushOp(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3, version: 221'617, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3@221'617, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.100992 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.100998 7f8682ee9700 5 -- op tracker -- , seq: 3648, time: 0.000000, event: dispatched, op: MOSDPGPush(3.21e 243 [PushOp(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3, version: 221'617, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3@221'617, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.101000 7f868ff62700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101006 7f8689755700 2 waiting 51 > 50 ops || 195624 > 104857600 2014-06-09 16:22:12.101003 7f8682ee9700 10 osd.26 243 handle_replica_op MOSDPGPush(3.21e 243 [PushOp(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3, version: 221'617, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3@221'617, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 epoch 243 2014-06-09 16:22:12.101011 7f8689f56700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101015 7f868c75b700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101012 7f8682ee9700 20 osd.26 243 should_share_map osd.4 172.16.96.201:6832/51134 243 2014-06-09 16:22:12.101020 7f8691765700 2 waiting 51 > 50 ops || 1729100 > 104857600 2014-06-09 16:22:12.101022 7f86a5610700 10 _finish_op 0x30060a20 seq 3973 osr(3.3a8 0x3ed0f80)/0x3ed0f80 2014-06-09 16:22:12.101022 7f8682ee9700 15 osd.26 243 enqueue_op 0x2976c700 prio 63 cost 4195304 latency 0.011338 MOSDPGPush(3.21e 243 [PushOp(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3, version: 221'617, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3@221'617, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 2014-06-09 16:22:12.101034 7f868af58700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101038 7f8692f68700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101041 7f8692767700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101045 7f868cf5c700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101047 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e240 (1 active) 2014-06-09 16:22:12.101049 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251d0e0] 2014-06-09 16:22:12.101050 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52efc0 (2 active) 2014-06-09 16:22:12.101050 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101054 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251d0e0] 2014-06-09 16:22:12.101053 7f86a5610700 5 _do_op 0x16fd5020 seq 3976 osr(2.350 0x2ed0e50)/0x2ed0e50 start 2014-06-09 16:22:12.101054 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101058 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.101058 7f86a5610700 10 _do_transaction on 0xc131b00 2014-06-09 16:22:12.101059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.101061 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101062 7f868ef60700 10 op_submit_finish 4023 2014-06-09 16:22:12.101065 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.101066 7f868df5e700 10 op_submit_start 4024 2014-06-09 16:22:12.101067 7f868df5e700 5 queue_transactions (trailing journal) 4024 2014-06-09 16:22:12.101069 7f868df5e700 5 queue_op 0xbf7f1a0 seq 4024 osr(3.579 0x3ed18d0) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.101072 7f868df5e700 10 op_submit_finish 4024 2014-06-09 16:22:12.101070 7f868ef60700 10 osd.26 pg_epoch: 243 pg[1.3f0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.3f0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.101075 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.555( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.555( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.101111 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.101111 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.101120 7f868ef60700 5 queue_transactions existing osr(1.3f0 0x384d400)/0x384d400 2014-06-09 16:22:12.101121 7f868df5e700 5 queue_transactions existing osr(1.555 0x384dab0)/0x384dab0 2014-06-09 16:22:12.101123 7f868df5e700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101128 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101065 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.101133 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101137 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101144 7f868ff62700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101149 7f8689755700 2 waiting 51 > 50 ops || 195624 > 104857600 2014-06-09 16:22:12.101155 7f8689f56700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101159 7f868c75b700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101164 7f8691765700 2 waiting 51 > 50 ops || 1729100 > 104857600 2014-06-09 16:22:12.101169 7f868a757700 2 waiting 51 > 50 ops || 195656 > 104857600 2014-06-09 16:22:12.101174 7f868ef60700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101179 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.101180 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101182 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.101183 7f869af78700 20 osd.26 243 _dispatch 0x3104a8c0 MRecoveryReserve REQUEST pgid: 3.21e, query_epoch: 243 v2 2014-06-09 16:22:12.101193 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.101190 7f869af78700 5 -- op tracker -- , seq: 3649, time: 2014-06-09 16:22:12.101140, event: header_read, op: MRecoveryReserve REQUEST pgid: 3.21e, query_epoch: 243 2014-06-09 16:22:12.101199 7f869af78700 5 -- op tracker -- , seq: 3649, time: 2014-06-09 16:22:12.101141, event: throttled, op: MRecoveryReserve REQUEST pgid: 3.21e, query_epoch: 243 2014-06-09 16:22:12.101206 7f869af78700 5 -- op tracker -- , seq: 3649, time: 2014-06-09 16:22:12.101151, event: all_read, op: MRecoveryReserve REQUEST pgid: 3.21e, query_epoch: 243 2014-06-09 16:22:12.101210 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3975.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.101214 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.101212 7f869af78700 5 -- op tracker -- , seq: 3649, time: 2014-06-09 16:22:12.101178, event: dispatched, op: MRecoveryReserve REQUEST pgid: 3.21e, query_epoch: 243 2014-06-09 16:22:12.101222 7f869af78700 5 -- op tracker -- , seq: 3649, time: 2014-06-09 16:22:12.101220, event: waiting_for_osdmap, op: MRecoveryReserve REQUEST pgid: 3.21e, query_epoch: 243 2014-06-09 16:22:12.101228 7f869af78700 15 osd.26 243 require_same_or_newer_map 243 (i am 243) 0x3104a8c0 2014-06-09 16:22:12.101242 7f869af78700 5 -- op tracker -- , seq: 3649, time: 2014-06-09 16:22:12.101240, event: done, op: MRecoveryReserve REQUEST pgid: 3.21e, query_epoch: 243 2014-06-09 16:22:12.101254 7f86a5e11700 10 _do_op 0xbf7ff20 seq 3975 r = 0, finisher 0xcdb2b20 0 2014-06-09 16:22:12.101258 7f869af78700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.101261 7f869af78700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.101266 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc40dee0] 2014-06-09 16:22:12.101269 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc40dee0] 2014-06-09 16:22:12.101270 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.101271 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.101274 7f86a5e11700 10 _finish_op 0xbf7ff20 seq 3975 osr(2.654 0x2ed1fd0)/0x2ed1fd0 2014-06-09 16:22:12.101283 7f8692f68700 10 op_submit_start 4025 2014-06-09 16:22:12.101285 7f8692f68700 5 queue_transactions (trailing journal) 4025 2014-06-09 16:22:12.101286 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.101287 7f8692f68700 5 queue_op 0xc88bf20 seq 4025 osr(1.375 0x384d230) 828 bytes (queue has 50 ops and 194798 bytes) 2014-06-09 16:22:12.101291 7f8692767700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101297 7f868cf5c700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101299 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359440 (1 active) 2014-06-09 16:22:12.101300 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcdb2b20] 2014-06-09 16:22:12.101301 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1147e0 (2 active) 2014-06-09 16:22:12.101303 7f868d75d700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101305 7f8692f68700 10 op_submit_finish 4025 2014-06-09 16:22:12.101307 7f8690f64700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101308 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcdb2b20] 2014-06-09 16:22:12.101309 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.101310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.101310 7f8691f66700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101314 7f868e75f700 2 waiting 51 > 50 ops || 195659 > 104857600 2014-06-09 16:22:12.101316 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3976.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.101320 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.101309 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.101333 7f86a5610700 10 _do_op 0x16fd5020 seq 3976 r = 0, finisher 0xbcc9120 0 2014-06-09 16:22:12.101331 7f86a6b95700 5 -- op tracker -- , seq: 3650, time: 2014-06-09 16:22:12.097846, event: header_read, op: osd_op(client.4542.0:34603 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 1437696~2064384] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.101343 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe66220] 2014-06-09 16:22:12.101344 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.101345 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe66220] 2014-06-09 16:22:12.101346 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.101347 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.101345 7f86a6b95700 5 -- op tracker -- , seq: 3650, time: 2014-06-09 16:22:12.097847, event: throttled, op: osd_op(client.4542.0:34603 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 1437696~2064384] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.101348 7f8692f68700 5 queue_transactions existing osr(1.a 0x384c270)/0x384c270 2014-06-09 16:22:12.101350 7f86a5610700 10 _finish_op 0x16fd5020 seq 3976 osr(2.350 0x2ed0e50)/0x2ed0e50 2014-06-09 16:22:12.101351 7f8692f68700 2 waiting 51 > 50 ops || 195622 > 104857600 2014-06-09 16:22:12.101356 7f8690763700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101357 7f86a5e11700 5 _do_op 0xc34b980 seq 3977 osr(1.f2 0x384c990)/0x384c990 start 2014-06-09 16:22:12.101356 7f86a6b95700 5 -- op tracker -- , seq: 3650, time: 2014-06-09 16:22:12.101291, event: all_read, op: osd_op(client.4542.0:34603 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 1437696~2064384] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.101360 7f86a5e11700 10 _do_transaction on 0xc270900 2014-06-09 16:22:12.101361 7f868b759700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101366 7f868f761700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101366 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.101372 7f868ff62700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101374 7f86a6b95700 5 -- op tracker -- , seq: 3650, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34603 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 1437696~2064384] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.101378 7f8689755700 2 waiting 51 > 50 ops || 195624 > 104857600 2014-06-09 16:22:12.101384 7f8689f56700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101382 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.101389 7f868c75b700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101394 7f8691765700 2 waiting 51 > 50 ops || 1729100 > 104857600 2014-06-09 16:22:12.101400 7f868a757700 2 waiting 51 > 50 ops || 195656 > 104857600 2014-06-09 16:22:12.101404 7f868ef60700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101403 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976c300 prio 63 cost 2064384 latency 0.003557 osd_op(client.4542.0:34603 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 1437696~2064384] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.101410 7f868bf5a700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101410 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.101415 7f868af58700 2 waiting 51 > 50 ops || 195626 > 104857600 2014-06-09 16:22:12.101425 7f8692767700 10 op_submit_start 4026 2014-06-09 16:22:12.101427 7f8692767700 5 queue_transactions (trailing journal) 4026 2014-06-09 16:22:12.101427 7f868cf5c700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101429 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbcc9120] 2014-06-09 16:22:12.101429 7f8692767700 5 queue_op 0x2e20420 seq 4026 osr(3.173 0x3ed06f0) 1390942 bytes (queue has 50 ops and 1584912 bytes) 2014-06-09 16:22:12.101433 7f868d75d700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101434 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbcc9120] 2014-06-09 16:22:12.101436 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.101437 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.101437 7f8690f64700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101440 7f8691f66700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101440 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52efc0 (1 active) 2014-06-09 16:22:12.101440 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3977.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.101442 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f680 (2 active) 2014-06-09 16:22:12.101443 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.101444 7f868e75f700 2 waiting 51 > 50 ops || 1585773 > 104857600 2014-06-09 16:22:12.101446 7f86a5610700 5 _do_op 0xbf7eb40 seq 3978 osr(3.27 0x3ed0340)/0x3ed0340 start 2014-06-09 16:22:12.101447 7f8692767700 10 op_submit_finish 4026 2014-06-09 16:22:12.101449 7f86a5610700 10 _do_transaction on 0xc2afbc0 2014-06-09 16:22:12.101454 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.101453 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=213'624 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc23e300 243'625 rep_tid=26 committed?=0 applied?=0 lock=0 op=osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.101480 7f8692767700 10 osd.26 243 dequeue_op 0x270bea00 finish 2014-06-09 16:22:12.101485 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.101527 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153591c0 (20 active) 2014-06-09 16:22:12.101530 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.6f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.6f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.101556 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.101560 7f8692767700 5 queue_transactions existing osr(1.6f 0x384c570)/0x384c570 2014-06-09 16:22:12.101561 7f8692767700 2 waiting 51 > 50 ops || 1585738 > 104857600 2014-06-09 16:22:12.101565 7f8692f68700 2 waiting 51 > 50 ops || 1585736 > 104857600 2014-06-09 16:22:12.101569 7f8690763700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101574 7f868b759700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101580 7f868f761700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101582 7f86a5e11700 10 _do_op 0xc34b980 seq 3977 r = 0, finisher 0x32535740 0 2014-06-09 16:22:12.101587 7f868ff62700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101588 7f86a5e11700 10 _finish_op 0xc34b980 seq 3977 osr(1.f2 0x384c990)/0x384c990 2014-06-09 16:22:12.101589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32535ea0] 2014-06-09 16:22:12.101603 7f868cf5c700 10 op_submit_start 4027 2014-06-09 16:22:12.101592 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.101605 7f868cf5c700 5 queue_transactions (trailing journal) 4027 2014-06-09 16:22:12.101605 7f8689f56700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101607 7f868cf5c700 5 queue_op 0xcd902a0 seq 4027 osr(3.381 0x3ed0fe0) 828 bytes (queue has 50 ops and 1584882 bytes) 2014-06-09 16:22:12.101605 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.101612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32535740] 2014-06-09 16:22:12.101616 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32535740] 2014-06-09 16:22:12.101617 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.101618 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.101621 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1147e0 (1 active) 2014-06-09 16:22:12.101616 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.101623 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4078c0 (2 active) 2014-06-09 16:22:12.101625 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32535ea0] 2014-06-09 16:22:12.101626 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.101627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.101627 7f86a5e11700 5 _do_op 0xcabcb40 seq 3979 osr(3.218 0x3ed0cd0)/0x3ed0cd0 start 2014-06-09 16:22:12.101628 7f868cf5c700 10 op_submit_finish 4027 2014-06-09 16:22:12.101629 7f86a5e11700 10 _do_transaction on 0xc2ee3c0 2014-06-09 16:22:12.101630 7f8689755700 2 waiting 51 > 50 ops || 1585708 > 104857600 2014-06-09 16:22:12.101635 7f868c75b700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101635 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.101639 7f868a757700 2 waiting 51 > 50 ops || 1585740 > 104857600 2014-06-09 16:22:12.101644 7f868ef60700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101649 7f868bf5a700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101650 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8f040 (19 active) 2014-06-09 16:22:12.101653 7f868af58700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101659 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.101658 7f868df5e700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101663 7f868d75d700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101668 7f8690f64700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101671 7f8691f66700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101676 7f868e75f700 2 waiting 51 > 50 ops || 1585743 > 104857600 2014-06-09 16:22:12.101663 7f868cf5c700 10 osd.26 243 dequeue_op 0x29f5ab00 prio 63 cost 811008 latency 1.279040 osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.101680 7f8692767700 2 waiting 51 > 50 ops || 1585708 > 104857600 2014-06-09 16:22:12.101684 7f8692f68700 2 waiting 51 > 50 ops || 1585706 > 104857600 2014-06-09 16:22:12.101682 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.101688 7f8690763700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101686 7f868cf5c700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.101682, event: reached_pg, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.101692 7f868b759700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101697 7f868f761700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101695 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.101708 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3978.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.101713 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.101709 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x29f5ab00 2014-06-09 16:22:12.101718 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.101728 7f86a5610700 10 _do_op 0xbf7eb40 seq 3978 r = 0, finisher 0x4eb98c0 0 2014-06-09 16:22:12.101738 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd637c0] 2014-06-09 16:22:12.101739 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd637c0] 2014-06-09 16:22:12.101739 7f868cf5c700 15 getattr 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 '_' 2014-06-09 16:22:12.101741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.101742 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.101745 7f8691765700 2 waiting 51 > 50 ops || 3119184 > 104857600 2014-06-09 16:22:12.101745 7f86a5610700 10 _finish_op 0xbf7eb40 seq 3978 osr(3.27 0x3ed0340)/0x3ed0340 2014-06-09 16:22:12.101752 7f868ff62700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101753 7f8689f56700 10 op_submit_start 4028 2014-06-09 16:22:12.101754 7f868c75b700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101756 7f8689f56700 5 queue_transactions (trailing journal) 4028 2014-06-09 16:22:12.101757 7f8689f56700 5 queue_op 0x3248b980 seq 4028 osr(2.48f 0x2ed1450) 828 bytes (queue has 50 ops and 1584884 bytes) 2014-06-09 16:22:12.101759 7f868a757700 2 waiting 51 > 50 ops || 1585742 > 104857600 2014-06-09 16:22:12.101763 7f868bf5a700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101768 7f868ef60700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101772 7f868df5e700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101772 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x4eb98c0] 2014-06-09 16:22:12.101775 7f868d75d700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101778 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x4eb98c0] 2014-06-09 16:22:12.101779 7f8690f64700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101780 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.101781 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.101782 7f8691f66700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101783 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f680 (1 active) 2014-06-09 16:22:12.101785 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3010e0 (2 active) 2014-06-09 16:22:12.101786 7f868e75f700 2 waiting 51 > 50 ops || 1585745 > 104857600 2014-06-09 16:22:12.101790 7f8689f56700 10 op_submit_finish 4028 2014-06-09 16:22:12.101789 7f86a5610700 5 _do_op 0x3250e300 seq 3980 osr(3.27b 0x3ed0bd0)/0x3ed0bd0 start 2014-06-09 16:22:12.101792 7f86a5610700 10 _do_transaction on 0xbf50840 2014-06-09 16:22:12.101798 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.101793 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.51e( v 214'681 lc 213'678 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.101809 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.101817 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.101822 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.101833 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.101845 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1148 2014-06-09 16:22:12.101861 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.101865 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.1aa( v 214'499 lc 214'498 (0'0,214'499] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.101878 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.1 activated and committed 2014-06-09 16:22:12.101885 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 191/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.101889 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.101897 7f8689f56700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.101908 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:793 2014-06-09 16:22:12.101926 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.101930 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.3c3( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.101956 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.101968 7f8689f56700 5 queue_transactions existing osr(1.3c3 0x384d530)/0x384d530 2014-06-09 16:22:12.101971 7f8689f56700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101975 7f8692767700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.101980 7f8692f68700 2 waiting 51 > 50 ops || 1585708 > 104857600 2014-06-09 16:22:12.101984 7f8690763700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101989 7f868b759700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101995 7f868f761700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.101997 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.102008 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3979.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.102011 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.102071 7f868cf5c700 10 getattr got.size() is 0 2014-06-09 16:22:12.102074 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 but can_create 2014-06-09 16:22:12.102087 7f868cf5c700 15 getattr 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 'snapset' 2014-06-09 16:22:12.102091 7f8691765700 2 waiting 51 > 50 ops || 3119186 > 104857600 2014-06-09 16:22:12.102098 7f8689755700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102103 7f868af58700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102108 7f86a5e11700 10 _do_op 0xcabcb40 seq 3979 r = 0, finisher 0xbd7c000 0 2014-06-09 16:22:12.102114 7f86a5e11700 10 _finish_op 0xcabcb40 seq 3979 osr(3.218 0x3ed0cd0)/0x3ed0cd0 2014-06-09 16:22:12.102115 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30c544e0] 2014-06-09 16:22:12.102117 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30c544e0] 2014-06-09 16:22:12.102118 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.102119 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.102121 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4078c0 (1 active) 2014-06-09 16:22:12.102122 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7c000] 2014-06-09 16:22:12.102122 7f868ff62700 10 op_submit_start 4029 2014-06-09 16:22:12.102123 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fde60 (2 active) 2014-06-09 16:22:12.102124 7f868ff62700 5 queue_transactions (trailing journal) 4029 2014-06-09 16:22:12.102126 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7c000] 2014-06-09 16:22:12.102125 7f86a5e11700 5 _do_op 0x32816180 seq 3981 osr(1.27c 0x384cf30)/0x384cf30 start 2014-06-09 16:22:12.102128 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.102126 7f868ff62700 5 queue_op 0x30a3140 seq 4029 osr(0.473 0x2bd74d0) 828 bytes (queue has 50 ops and 1584884 bytes) 2014-06-09 16:22:12.102128 7f86a5e11700 10 _do_transaction on 0xc271800 2014-06-09 16:22:12.102129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.102129 7f868ff62700 10 op_submit_finish 4029 2014-06-09 16:22:12.102134 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.102133 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.c6( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.c6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.102155 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.102163 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.102168 7f868ff62700 5 queue_transactions existing osr(1.c6 0x384c610)/0x384c610 2014-06-09 16:22:12.102170 7f868ff62700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102169 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3980.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.102173 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.102174 7f868c75b700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102179 7f868a757700 2 waiting 51 > 50 ops || 1585742 > 104857600 2014-06-09 16:22:12.102191 7f868cf5c700 10 getattr got.size() is 0 2014-06-09 16:22:12.102192 7f86a5610700 10 _do_op 0x3250e300 seq 3980 r = 0, finisher 0x318c3aa0 0 2014-06-09 16:22:12.102194 7f868cf5c700 15 getattr 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/snapdir//3 'snapset' 2014-06-09 16:22:12.102201 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30629f40] 2014-06-09 16:22:12.102203 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30629f40] 2014-06-09 16:22:12.102205 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.102206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.102209 7f86a5610700 10 _finish_op 0x3250e300 seq 3980 osr(3.27b 0x3ed0bd0)/0x3ed0bd0 2014-06-09 16:22:12.102235 7f868cf5c700 10 getattr got.size() is 0 2014-06-09 16:22:12.102245 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x148b2000 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.102258 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.102262 7f8682de8700 5 -- op tracker -- , seq: 3651, time: 2014-06-09 16:22:12.102234, event: header_read, op: osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.102271 7f8682de8700 5 -- op tracker -- , seq: 3651, time: 2014-06-09 16:22:12.102235, event: throttled, op: osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.102268 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.102280 7f8682de8700 5 -- op tracker -- , seq: 3651, time: 2014-06-09 16:22:12.102243, event: all_read, op: osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.102287 7f8682de8700 5 -- op tracker -- , seq: 3651, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.102277 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x148b2000 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 rwstate(none n=0 w=0) oi: 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x2e876a80 snapset: 0=[]:[] 2014-06-09 16:22:12.102291 7f8682de8700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:12.102295 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:12.102292 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 @head oi=92539e61/rbd_data.11816b8b4567.000000000000194d/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.102301 7f8682de8700 15 osd.26 243 enqueue_op 0x30c6cd00 prio 196 cost 0 latency 0.000067 osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:12.102309 7f868cf5c700 15 getattr 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/snapdir//3 '_' 2014-06-09 16:22:12.102312 7f86a6b95700 5 -- op tracker -- , seq: 3652, time: 2014-06-09 16:22:12.101426, event: header_read, op: osd_op(client.4542.0:34604 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 3502080~692224] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.102321 7f868ef60700 10 op_submit_start 4030 2014-06-09 16:22:12.102323 7f868ef60700 5 queue_transactions (trailing journal) 4030 2014-06-09 16:22:12.102325 7f868ef60700 5 queue_op 0x760f0e0 seq 4030 osr(1.3f0 0x384d400) 828 bytes (queue has 50 ops and 1584884 bytes) 2014-06-09 16:22:12.102323 7f86a6b95700 5 -- op tracker -- , seq: 3652, time: 2014-06-09 16:22:12.101427, event: throttled, op: osd_op(client.4542.0:34604 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 3502080~692224] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.102330 7f868df5e700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102334 7f868d75d700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102338 7f8690f64700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102342 7f8691f66700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102343 7f868cf5c700 10 getattr got.size() is 0 2014-06-09 16:22:12.102346 7f868e75f700 2 waiting 51 > 50 ops || 1585745 > 104857600 2014-06-09 16:22:12.102335 7f86a6b95700 5 -- op tracker -- , seq: 3652, time: 2014-06-09 16:22:12.102275, event: all_read, op: osd_op(client.4542.0:34604 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 3502080~692224] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.102350 7f8689f56700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102354 7f8692767700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102345 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 92539e61/rbd_data.11816b8b4567.000000000000194d/snapdir//3 and !can_create 2014-06-09 16:22:12.102354 7f86a6b95700 5 -- op tracker -- , seq: 3652, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34604 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 3502080~692224] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.102358 7f8692f68700 2 waiting 51 > 50 ops || 1585708 > 104857600 2014-06-09 16:22:12.102362 7f8690763700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102365 7f868b759700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102365 7f868cf5c700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.102359, event: started, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.102370 7f868f761700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102375 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c3aa0] 2014-06-09 16:22:12.102364 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.102377 7f8691765700 2 waiting 51 > 50 ops || 3119186 > 104857600 2014-06-09 16:22:12.102379 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c3aa0] 2014-06-09 16:22:12.102380 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.102373 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x306dac00 2014-06-09 16:22:12.102381 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.102383 7f8689755700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102384 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3010e0 (1 active) 2014-06-09 16:22:12.102388 7f868af58700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102393 7f868c75b700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102393 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0ea0 (2 active) 2014-06-09 16:22:12.102389 7f868cf5c700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.102385, event: started, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.102389 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976c200 prio 63 cost 692224 latency 0.000963 osd_op(client.4542.0:34604 rbd_data.115e6b8b4567.000000000000180f [set-alloc-hint object_size 4194304 write_size 4194304,write 3502080~692224] 3.11fd9505 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.102398 7f868ef60700 10 op_submit_finish 4030 2014-06-09 16:22:12.102396 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] ov 0'0 av 243'550 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.102401 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.fd( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.fd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.102410 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 2014-06-09 16:22:12.102419 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.102435 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.102433 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~811008 2014-06-09 16:22:12.102440 7f868ef60700 5 queue_transactions existing osr(2.fd 0x2ed0770)/0x2ed0770 2014-06-09 16:22:12.102442 7f868ef60700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102446 7f868a757700 2 waiting 51 > 50 ops || 1585742 > 104857600 2014-06-09 16:22:12.102451 7f86a5610700 5 _do_op 0x30a25a0 seq 3982 osr(3.537 0x3ed1620)/0x3ed1620 start 2014-06-09 16:22:12.102451 7f868ff62700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102454 7f86a5610700 10 _do_transaction on 0xc2aa6c0 2014-06-09 16:22:12.102448 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0x1078ae00 2014-06-09 16:22:12.102459 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.102458 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 snapset=0x2e876ab8 snapc=0=[] 2014-06-09 16:22:12.102469 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.102479 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.102496 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.102489 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 0x306dac00 op modify 2014-06-09 16:22:12.102509 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3982.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.102501 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:38.907455 2014-06-09 16:22:12.102513 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.102524 7f86a5610700 10 _do_op 0x30a25a0 seq 3982 r = 0, finisher 0x2e0ec20 0 2014-06-09 16:22:12.102521 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=103 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.102534 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x29283b80] 2014-06-09 16:22:12.102538 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x29283b80] 2014-06-09 16:22:12.102539 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.102540 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.102543 7f86a5610700 10 _finish_op 0x30a25a0 seq 3982 osr(3.537 0x3ed1620)/0x3ed1620 2014-06-09 16:22:12.102539 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.102553 7f868d75d700 10 op_submit_start 4031 2014-06-09 16:22:12.102554 7f868d75d700 5 queue_transactions (trailing journal) 4031 2014-06-09 16:22:12.102555 7f868d75d700 5 queue_op 0x16fd5860 seq 4031 osr(1.1e8 0x384cdf0) 828 bytes (queue has 50 ops and 1584884 bytes) 2014-06-09 16:22:12.102560 7f8690f64700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102551 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 28 on osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.102564 7f8691f66700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102568 7f868e75f700 2 waiting 51 > 50 ops || 1585745 > 104857600 2014-06-09 16:22:12.102572 7f8689f56700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102566 7f868cf5c700 7 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 28 o 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.102575 7f8692767700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102579 7f8692f68700 2 waiting 51 > 50 ops || 1585708 > 104857600 2014-06-09 16:22:12.102583 7f8690763700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102586 7f868b759700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102592 7f868f761700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102590 7f868cf5c700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.102585, event: waiting for subops from 6, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.102597 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0ec20] 2014-06-09 16:22:12.102599 7f8691765700 2 waiting 51 > 50 ops || 3119186 > 104857600 2014-06-09 16:22:12.102601 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0ec20] 2014-06-09 16:22:12.102603 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.102604 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.102605 7f8689755700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102609 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.102610 7f868af58700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102606 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0ea0 (1 active) 2014-06-09 16:22:12.102614 7f868c75b700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102615 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525c20 (2 active) 2014-06-09 16:22:12.102620 7f868ef60700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102625 7f868d75d700 10 op_submit_finish 4031 2014-06-09 16:22:12.102629 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.2cb( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.2cb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.102627 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 213'549 (0'0,213'549] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'549], crt=0'0) [243'550 (0'0) modify 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 by client.4577.0:37501 2014-06-09 10:19:38.907455] 2014-06-09 16:22:12.102661 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.102649 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=213'549 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'550 (0'0) modify 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 by client.4577.0:37501 2014-06-09 10:19:38.907455 2014-06-09 16:22:12.102666 7f868d75d700 5 queue_transactions existing osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 2014-06-09 16:22:12.102668 7f868d75d700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102672 7f868a757700 2 waiting 51 > 50 ops || 1585742 > 104857600 2014-06-09 16:22:12.102677 7f868ff62700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102677 7f86a5610700 5 _do_op 0xbf7fe60 seq 3983 osr(3.98 0x3ed0620)/0x3ed0620 start 2014-06-09 16:22:12.102670 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=213'549 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.102681 7f86a5610700 10 _do_transaction on 0xbe89680 2014-06-09 16:22:12.102683 7f868df5e700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102693 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.102701 7f868cf5c700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'550, trimmed: 2014-06-09 16:22:12.102720 7f868cf5c700 5 queue_transactions existing osr(3.661 0x3ed19f0)/0x3ed19f0 2014-06-09 16:22:12.102723 7f868cf5c700 2 waiting 51 > 50 ops || 2398290 > 104857600 2014-06-09 16:22:12.102728 7f868bf5a700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102730 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.102744 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3983.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.102749 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.102774 7f86a5610700 10 _do_op 0xbf7fe60 seq 3983 r = 0, finisher 0x31456340 0 2014-06-09 16:22:12.102784 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31434b80] 2014-06-09 16:22:12.102786 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31434b80] 2014-06-09 16:22:12.102787 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.102788 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.102791 7f86a5610700 10 _finish_op 0xbf7fe60 seq 3983 osr(3.98 0x3ed0620)/0x3ed0620 2014-06-09 16:22:12.102802 7f8691f66700 10 op_submit_start 4032 2014-06-09 16:22:12.102803 7f8691f66700 5 queue_transactions (trailing journal) 4032 2014-06-09 16:22:12.102802 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.102804 7f8691f66700 5 queue_op 0x3250e000 seq 4032 osr(0.631 0x2bd7c70) 828 bytes (queue has 50 ops and 1584886 bytes) 2014-06-09 16:22:12.102809 7f868e75f700 2 waiting 51 > 50 ops || 1585747 > 104857600 2014-06-09 16:22:12.102813 7f8689f56700 2 waiting 51 > 50 ops || 1585714 > 104857600 2014-06-09 16:22:12.102817 7f8692767700 2 waiting 51 > 50 ops || 1585712 > 104857600 2014-06-09 16:22:12.102817 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3981.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.102821 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.102821 7f8692f68700 2 waiting 51 > 50 ops || 1585710 > 104857600 2014-06-09 16:22:12.102825 7f8690763700 2 waiting 51 > 50 ops || 1585714 > 104857600 2014-06-09 16:22:12.102828 7f868b759700 2 waiting 51 > 50 ops || 1585714 > 104857600 2014-06-09 16:22:12.102833 7f86a5e11700 10 _do_op 0x32816180 seq 3981 r = 0, finisher 0x17001fa0 0 2014-06-09 16:22:12.102839 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd77aa0] 2014-06-09 16:22:12.102842 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd77aa0] 2014-06-09 16:22:12.102843 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.102844 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.102846 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31456340] 2014-06-09 16:22:12.102848 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31456340] 2014-06-09 16:22:12.102849 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.102850 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.102851 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525c20 (1 active) 2014-06-09 16:22:12.102853 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62fc20 (2 active) 2014-06-09 16:22:12.102856 7f86a5610700 5 _do_op 0xc88ae40 seq 3984 osr(1.c9 0x384c9f0)/0x384c9f0 start 2014-06-09 16:22:12.102859 7f86a5610700 10 _do_transaction on 0xc0e5ec0 2014-06-09 16:22:12.102860 7f8691f66700 10 op_submit_finish 4032 2014-06-09 16:22:12.102864 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.102863 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.72c( v 213'558 lc 213'557 (0'0,213'558] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.102879 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.0 activated and committed 2014-06-09 16:22:12.102891 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 191/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.102899 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.102900 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.102912 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3984.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.102915 7f8691f66700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.102916 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.102935 7f86a5610700 10 _do_op 0xc88ae40 seq 3984 r = 0, finisher 0xbd5e2c0 0 2014-06-09 16:22:12.102930 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:942 2014-06-09 16:22:12.102945 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050bc80] 2014-06-09 16:22:12.102947 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050bc80] 2014-06-09 16:22:12.102947 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.102950 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.102952 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.102954 7f86a5610700 10 _finish_op 0xc88ae40 seq 3984 osr(1.c9 0x384c9f0)/0x384c9f0 2014-06-09 16:22:12.102951 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.2d9( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.102966 7f8691765700 10 op_submit_start 4033 2014-06-09 16:22:12.102968 7f8691765700 5 queue_transactions (trailing journal) 4033 2014-06-09 16:22:12.102967 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5e2c0] 2014-06-09 16:22:12.102971 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5e2c0] 2014-06-09 16:22:12.102969 7f8691765700 5 queue_op 0x159960c0 seq 4033 osr(3.44d 0x3ed1180) 1534302 bytes (queue has 50 ops and 3118362 bytes) 2014-06-09 16:22:12.102972 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.102973 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.102974 7f8689755700 2 waiting 51 > 50 ops || 3119188 > 104857600 2014-06-09 16:22:12.102979 7f868af58700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.102982 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.102983 7f868c75b700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.102976 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62fc20 (1 active) 2014-06-09 16:22:12.102986 7f8691f66700 5 queue_transactions existing osr(0.2d9 0x2bd6e90)/0x2bd6e90 2014-06-09 16:22:12.102988 7f868ef60700 2 waiting 51 > 50 ops || 3119188 > 104857600 2014-06-09 16:22:12.102993 7f868d75d700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.102986 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4adb00 (2 active) 2014-06-09 16:22:12.102998 7f868a757700 2 waiting 51 > 50 ops || 3119220 > 104857600 2014-06-09 16:22:12.103000 7f86a5610700 5 _do_op 0x15997a40 seq 3985 osr(0.71b 0x384c050)/0x384c050 start 2014-06-09 16:22:12.103004 7f86a5610700 10 _do_transaction on 0xbe68d80 2014-06-09 16:22:12.103003 7f8691765700 10 op_submit_finish 4033 2014-06-09 16:22:12.103005 7f86a5e11700 10 _finish_op 0x32816180 seq 3981 osr(1.27c 0x384cf30)/0x384cf30 2014-06-09 16:22:12.103009 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.103016 7f868df5e700 10 op_submit_start 4034 2014-06-09 16:22:12.103015 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fde60 (1 active) 2014-06-09 16:22:12.103017 7f868df5e700 5 queue_transactions (trailing journal) 4034 2014-06-09 16:22:12.103017 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2865a0 (2 active) 2014-06-09 16:22:12.103018 7f868df5e700 5 queue_op 0xbf7f920 seq 4034 osr(1.555 0x384dab0) 828 bytes (queue has 50 ops and 3118362 bytes) 2014-06-09 16:22:12.103020 7f86a5e11700 5 _do_op 0x309a35c0 seq 3986 osr(0.33f 0x2bd7060)/0x2bd7060 start 2014-06-09 16:22:12.103021 7f868df5e700 10 op_submit_finish 4034 2014-06-09 16:22:12.103022 7f86a5e11700 10 _do_transaction on 0xc139080 2014-06-09 16:22:12.103007 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xbfe6540 243'636 rep_tid=27 committed?=0 applied?=0 lock=0 op=osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.103026 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.103029 7f8691765700 10 osd.26 243 dequeue_op 0x1804df00 finish 2014-06-09 16:22:12.103033 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.103024 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.61f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.103049 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.103052 7f868df5e700 5 queue_transactions existing osr(0.61f 0x2bd7d30)/0x2bd7d30 2014-06-09 16:22:12.103053 7f868df5e700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103057 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17001fa0] 2014-06-09 16:22:12.103058 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x1fc880c0 (20 active) 2014-06-09 16:22:12.103059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17001fa0] 2014-06-09 16:22:12.103060 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.103061 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.103062 7f868cf5c700 2 waiting 51 > 50 ops || 3931768 > 104857600 2014-06-09 16:22:12.103067 7f868bf5a700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103061 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.236( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.236( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.103073 7f8690f64700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103078 7f8689f56700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103083 7f8692767700 2 waiting 51 > 50 ops || 3119188 > 104857600 2014-06-09 16:22:12.103087 7f8692f68700 2 waiting 51 > 50 ops || 3119186 > 104857600 2014-06-09 16:22:12.103092 7f8690763700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103095 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.103097 7f868b759700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103099 7f8691765700 5 queue_transactions existing osr(2.236 0x2ed0ad0)/0x2ed0ad0 2014-06-09 16:22:12.103102 7f868e75f700 2 waiting 51 > 50 ops || 3119223 > 104857600 2014-06-09 16:22:12.103106 7f8691f66700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103109 7f868ff62700 2 waiting 51 > 50 ops || 3119188 > 104857600 2014-06-09 16:22:12.103115 7f868c75b700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103120 7f868ef60700 2 waiting 51 > 50 ops || 3119188 > 104857600 2014-06-09 16:22:12.103125 7f868d75d700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103130 7f868a757700 2 waiting 51 > 50 ops || 3119220 > 104857600 2014-06-09 16:22:12.103134 7f8689755700 2 waiting 51 > 50 ops || 3119188 > 104857600 2014-06-09 16:22:12.103138 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.103139 7f8691765700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103146 7f868af58700 2 waiting 51 > 50 ops || 3119190 > 104857600 2014-06-09 16:22:12.103158 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3985.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.103162 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.103176 7f86a5610700 10 _do_op 0x15997a40 seq 3985 r = 0, finisher 0x29283d40 0 2014-06-09 16:22:12.103185 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x29283f00] 2014-06-09 16:22:12.103187 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x29283f00] 2014-06-09 16:22:12.103188 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.103189 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.103192 7f86a5610700 10 _finish_op 0x15997a40 seq 3985 osr(0.71b 0x384c050)/0x384c050 2014-06-09 16:22:12.103202 7f868cf5c700 10 op_submit_start 4035 2014-06-09 16:22:12.103204 7f868cf5c700 5 queue_transactions (trailing journal) 4035 2014-06-09 16:22:12.103203 7f868bf5a700 2 waiting 51 > 50 ops || 3931768 > 104857600 2014-06-09 16:22:12.103205 7f868cf5c700 5 queue_op 0xcd90540 seq 4035 osr(3.661 0x3ed19f0) 813406 bytes (queue has 50 ops and 3930940 bytes) 2014-06-09 16:22:12.103209 7f8690f64700 2 waiting 51 > 50 ops || 3931768 > 104857600 2014-06-09 16:22:12.103213 7f8689f56700 2 waiting 51 > 50 ops || 3931768 > 104857600 2014-06-09 16:22:12.103217 7f8692767700 2 waiting 51 > 50 ops || 3931766 > 104857600 2014-06-09 16:22:12.103218 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.103222 7f8692f68700 2 waiting 51 > 50 ops || 3931764 > 104857600 2014-06-09 16:22:12.103224 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x29283d40] 2014-06-09 16:22:12.103225 7f868f761700 2 waiting 51 > 50 ops || 3931768 > 104857600 2014-06-09 16:22:12.103228 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x29283d40] 2014-06-09 16:22:12.103230 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.103231 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.103233 7f868b759700 2 waiting 51 > 50 ops || 3931768 > 104857600 2014-06-09 16:22:12.103237 7f868e75f700 2 waiting 51 > 50 ops || 3931801 > 104857600 2014-06-09 16:22:12.103234 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4adb00 (1 active) 2014-06-09 16:22:12.103242 7f8691f66700 2 waiting 51 > 50 ops || 3931768 > 104857600 2014-06-09 16:22:12.103242 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff4b40 (2 active) 2014-06-09 16:22:12.103246 7f868cf5c700 10 op_submit_finish 4035 2014-06-09 16:22:12.103247 7f868ff62700 2 waiting 51 > 50 ops || 3931766 > 104857600 2014-06-09 16:22:12.103250 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3986.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.103253 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.103253 7f868c75b700 2 waiting 51 > 50 ops || 3931768 > 104857600 2014-06-09 16:22:12.103258 7f868ef60700 2 waiting 51 > 50 ops || 3931766 > 104857600 2014-06-09 16:22:12.103249 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=213'549 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc014e40 243'550 rep_tid=28 committed?=0 applied?=0 lock=0 op=osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.103268 7f868cf5c700 10 osd.26 243 dequeue_op 0x29f5ab00 finish 2014-06-09 16:22:12.103271 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.103271 7f86a5e11700 10 _do_op 0x309a35c0 seq 3986 r = 0, finisher 0x3006d2e0 0 2014-06-09 16:22:12.103277 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.103280 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006d2a0] 2014-06-09 16:22:12.103281 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006d2a0] 2014-06-09 16:22:12.103283 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.103284 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.103286 7f86a5e11700 10 _finish_op 0x309a35c0 seq 3986 osr(0.33f 0x2bd7060)/0x2bd7060 2014-06-09 16:22:12.103281 7f868cf5c700 10 osd.26 243 dequeue_op 0x30c6cd00 prio 196 cost 0 latency 0.001047 osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) v2 pg pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=213'624 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.103295 7f868a757700 10 op_submit_start 4036 2014-06-09 16:22:12.103296 7f868a757700 5 queue_transactions (trailing journal) 4036 2014-06-09 16:22:12.103297 7f868a757700 5 queue_op 0x14dd6780 seq 4036 osr(0.75 0x2bd6370) 858 bytes (queue has 50 ops and 3930970 bytes) 2014-06-09 16:22:12.103302 7f8689755700 2 waiting 51 > 50 ops || 3931796 > 104857600 2014-06-09 16:22:12.103298 7f868cf5c700 5 -- op tracker -- , seq: 3651, time: 2014-06-09 16:22:12.103295, event: reached_pg, op: osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.103308 7f8691765700 2 waiting 51 > 50 ops || 3931798 > 104857600 2014-06-09 16:22:12.103313 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006d2e0] 2014-06-09 16:22:12.103314 7f868af58700 2 waiting 51 > 50 ops || 3931798 > 104857600 2014-06-09 16:22:12.103307 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=213'624 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x30c6cd00 2014-06-09 16:22:12.103317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006d2e0] 2014-06-09 16:22:12.103318 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.103319 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.103319 7f868df5e700 2 waiting 51 > 50 ops || 3931798 > 104857600 2014-06-09 16:22:12.103322 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2865a0 (1 active) 2014-06-09 16:22:12.103323 7f8690763700 2 waiting 51 > 50 ops || 3931798 > 104857600 2014-06-09 16:22:12.103323 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcadb200 (2 active) 2014-06-09 16:22:12.103321 7f868cf5c700 5 -- op tracker -- , seq: 3651, time: 2014-06-09 16:22:12.103318, event: started, op: osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.103327 7f8690f64700 2 waiting 51 > 50 ops || 3931798 > 104857600 2014-06-09 16:22:12.103337 7f868a757700 10 op_submit_finish 4036 2014-06-09 16:22:12.103328 7f868cf5c700 7 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=213'624 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 26 op ack_type 4 from 9 2014-06-09 16:22:12.103345 7f868cf5c700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.103340, event: sub_op_commit_rec, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.103341 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.5c6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.5c6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.103352 7f868cf5c700 10 osd.26 243 dequeue_op 0x30c6cd00 finish 2014-06-09 16:22:12.103358 7f868cf5c700 5 -- op tracker -- , seq: 3651, time: 2014-06-09 16:22:12.103355, event: done, op: osd_sub_op_reply(client.4575.0:32697 3.173 c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.103370 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.103374 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.103379 7f868a757700 5 queue_transactions existing osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 2014-06-09 16:22:12.103381 7f868a757700 2 waiting 51 > 50 ops || 3931798 > 104857600 2014-06-09 16:22:12.103385 7f8689f56700 2 waiting 51 > 50 ops || 3931798 > 104857600 2014-06-09 16:22:12.103394 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40b80 (20 active) 2014-06-09 16:22:12.103403 7f86a5e11700 5 _do_op 0x32503b00 seq 3988 osr(3.18b 0x3ed06a0)/0x3ed06a0 start 2014-06-09 16:22:12.103406 7f86a5e11700 10 _do_transaction on 0xbfca540 2014-06-09 16:22:12.103398 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.103412 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.103411 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.103419 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.103427 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.103435 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.103448 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.103443 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.103457 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:12.103464 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3988.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.103467 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.103466 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.103474 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.442( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.103480 7f86a5e11700 10 _do_op 0x32503b00 seq 3988 r = 0, finisher 0x318c3740 0 2014-06-09 16:22:12.103482 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.103489 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314ac1a0] 2014-06-09 16:22:12.103493 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314ac1a0] 2014-06-09 16:22:12.103494 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.103495 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.103487 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.103499 7f86a5e11700 10 _finish_op 0x32503b00 seq 3988 osr(3.18b 0x3ed06a0)/0x3ed06a0 2014-06-09 16:22:12.103500 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.103508 7f8692f68700 10 op_submit_start 4037 2014-06-09 16:22:12.103509 7f8692f68700 5 queue_transactions (trailing journal) 4037 2014-06-09 16:22:12.103510 7f868f761700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103511 7f8692f68700 5 queue_op 0xc88a2a0 seq 4037 osr(1.a 0x384c270) 824 bytes (queue has 50 ops and 3930741 bytes) 2014-06-09 16:22:12.103509 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1c768300,0x22009000,0x1c768200,0x22008f00,0x1c768100,0x22008e00,0x1c768000 2014-06-09 16:22:12.103519 7f868b759700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103526 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c3740] 2014-06-09 16:22:12.103529 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] _activate_committed 243 peer_activated now 26 last_epoch_started 191 same_interval_since 242 2014-06-09 16:22:12.103537 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.103543 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.18b( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] flushed 2014-06-09 16:22:12.103546 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.103558 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c3740] 2014-06-09 16:22:12.103559 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.103553 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.103560 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.103563 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcadb200 (1 active) 2014-06-09 16:22:12.103565 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407560 (2 active) 2014-06-09 16:22:12.103570 7f8692f68700 10 op_submit_finish 4037 2014-06-09 16:22:12.103566 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1035 2014-06-09 16:22:12.103575 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.103573 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 243/213 242/242/211) 2014-06-09 16:22:12.103583 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.583( v 213'614 (0'0,213'614] local-les=243 n=113 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.103591 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.103602 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.103595 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.103606 7f8692f68700 5 queue_transactions existing osr(3.5f8 0x3ed1b30)/0x3ed1b30 2014-06-09 16:22:12.103609 7f8692f68700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103625 7f868d75d700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103631 7f8691f66700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103612 7f86a5e11700 5 _do_op 0xbf7e2a0 seq 3989 osr(1.b5 0x384c690)/0x384c690 start 2014-06-09 16:22:12.103634 7f86a5e11700 10 _do_transaction on 0x34e95c0 2014-06-09 16:22:12.103635 7f868e75f700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.103640 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.103640 7f868c75b700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103608 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.103647 7f868ef60700 2 waiting 51 > 50 ops || 3931567 > 104857600 2014-06-09 16:22:12.103652 7f868bf5a700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103646 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16bd7700,0x16bd7600,0x16bd7500,0x16bd7400 2014-06-09 16:22:12.103658 7f8691765700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103663 7f868af58700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103667 7f868df5e700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103671 7f8690763700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103665 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.1 has 3 missing 2014-06-09 16:22:12.103675 7f8690f64700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103680 7f868a757700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103675 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.103684 7f8689f56700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103688 7f8689755700 2 waiting 51 > 50 ops || 3931567 > 104857600 2014-06-09 16:22:12.103693 7f868ff62700 2 waiting 51 > 50 ops || 3931567 > 104857600 2014-06-09 16:22:12.103693 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:903 2014-06-09 16:22:12.103704 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.103703 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.103712 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.308( v 214'552 (0'0,214'552] local-les=243 n=108 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.103717 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3989.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.103721 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.103721 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.103726 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.103739 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.103746 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.103750 7f86a5e11700 10 _do_op 0xbf7e2a0 seq 3989 r = 0, finisher 0x13964b00 0 2014-06-09 16:22:12.103760 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251ce20] 2014-06-09 16:22:12.103754 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.103762 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251ce20] 2014-06-09 16:22:12.103763 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.103764 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.103767 7f86a5e11700 10 _finish_op 0xbf7e2a0 seq 3989 osr(1.b5 0x384c690)/0x384c690 2014-06-09 16:22:12.103762 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.103769 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.103778 7f868b759700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103777 7f868f761700 10 op_submit_start 4038 2014-06-09 16:22:12.103780 7f868f761700 5 queue_transactions (trailing journal) 4038 2014-06-09 16:22:12.103782 7f8692f68700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103782 7f868f761700 5 queue_op 0x230b94a0 seq 4038 osr(0.6e7 0x384c1b0) 828 bytes (queue has 50 ops and 3930743 bytes) 2014-06-09 16:22:12.103787 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13964b00] 2014-06-09 16:22:12.103782 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:12.103791 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13964b00] 2014-06-09 16:22:12.103792 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.103793 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.103796 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407560 (1 active) 2014-06-09 16:22:12.103791 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.103798 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3017a0 (2 active) 2014-06-09 16:22:12.103802 7f86a5e11700 5 _do_op 0x15997aa0 seq 3990 osr(0.654 0x2bd7b50)/0x2bd7b50 start 2014-06-09 16:22:12.103798 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.5ed( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.103804 7f86a5e11700 10 _do_transaction on 0xc1f2a80 2014-06-09 16:22:12.103805 7f868f761700 10 op_submit_finish 4038 2014-06-09 16:22:12.103806 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.103809 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.103810 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.103822 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.916389 7 0.000406 2014-06-09 16:22:12.103809 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.6f5( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.6f5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.103836 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.103844 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.103846 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.103856 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3990.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.103859 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.103860 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.103857 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.103865 7f868f761700 5 queue_transactions existing osr(1.6f5 0x2ed0170)/0x2ed0170 2014-06-09 16:22:12.103868 7f868f761700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103870 7f86a5e11700 10 _do_op 0x15997aa0 seq 3990 r = 0, finisher 0x325359e0 0 2014-06-09 16:22:12.103867 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.103880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x4eb8860] 2014-06-09 16:22:12.103881 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x4eb8860] 2014-06-09 16:22:12.103876 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.103883 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.103884 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.103887 7f86a5e11700 10 _finish_op 0x15997aa0 seq 3990 osr(0.654 0x2bd7b50)/0x2bd7b50 2014-06-09 16:22:12.103883 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.103895 7f8691f66700 10 op_submit_start 4039 2014-06-09 16:22:12.103897 7f8691f66700 5 queue_transactions (trailing journal) 4039 2014-06-09 16:22:12.103898 7f8691f66700 5 queue_op 0x3250f500 seq 4039 osr(0.2d9 0x2bd6e90) 828 bytes (queue has 50 ops and 3930743 bytes) 2014-06-09 16:22:12.103902 7f868e75f700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.103895 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.103908 7f868c75b700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103912 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x325359e0] 2014-06-09 16:22:12.103907 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.103915 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x325359e0] 2014-06-09 16:22:12.103917 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.103918 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.103920 7f868ef60700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103917 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.103925 7f868bf5a700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103929 7f868cf5c700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:12.103929 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3017a0 (1 active) 2014-06-09 16:22:12.103930 7f8691765700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103932 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115680 (2 active) 2014-06-09 16:22:12.103936 7f8691f66700 10 op_submit_finish 4039 2014-06-09 16:22:12.103935 7f868af58700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103940 7f868df5e700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103945 7f8690763700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103948 7f8690f64700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103939 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.7ae( v 213'629 (0'0,213'629] local-les=243 n=119 ec=118 les/c 243/243 242/242/223) 2014-06-09 16:22:12.103945 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:175 2014-06-09 16:22:12.103953 7f868a757700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103957 7f8689f56700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103961 7f8689755700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103965 7f868ff62700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103969 7f8692f68700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103970 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.103973 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.103974 7f868f761700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103975 7f8691f66700 5 queue_transactions existing osr(3.7ae 0xb7f01d0)/0xb7f01d0 2014-06-09 16:22:12.103977 7f868cf5c700 5 queue_transactions existing osr(2.45b 0x2ed1530)/0x2ed1530 2014-06-09 16:22:12.103977 7f8691f66700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103979 7f868cf5c700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.103985 7f868d75d700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103990 7f868b759700 2 waiting 51 > 50 ops || 3931571 > 104857600 2014-06-09 16:22:12.103991 7f86a5610700 5 _do_op 0x324a5560 seq 3987 osr(3.616 0x3ed1a90)/0x3ed1a90 start 2014-06-09 16:22:12.103995 7f8692767700 2 waiting 51 > 50 ops || 3931569 > 104857600 2014-06-09 16:22:12.103999 7f86a5610700 10 _do_transaction on 0x394f200 2014-06-09 16:22:12.104004 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.104004 7f86a5e11700 5 _do_op 0x16096a20 seq 3991 osr(1.65e 0x384df30)/0x384df30 start 2014-06-09 16:22:12.104010 7f86a5e11700 10 _do_transaction on 0xc245d40 2014-06-09 16:22:12.104015 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.104040 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.104054 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3987.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.104058 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.104076 7f86a5610700 10 _do_op 0x324a5560 seq 3987 r = 0, finisher 0xbd7c720 0 2014-06-09 16:22:12.104082 7f86a5610700 10 _finish_op 0x324a5560 seq 3987 osr(3.616 0x3ed1a90)/0x3ed1a90 2014-06-09 16:22:12.104083 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189aa500] 2014-06-09 16:22:12.104085 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189aa500] 2014-06-09 16:22:12.104087 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.104088 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.104093 7f868e75f700 10 op_submit_start 4040 2014-06-09 16:22:12.104092 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7c720] 2014-06-09 16:22:12.104094 7f868e75f700 5 queue_transactions (trailing journal) 4040 2014-06-09 16:22:12.104094 7f868c75b700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104096 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7c720] 2014-06-09 16:22:12.104097 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.104095 7f868e75f700 5 queue_op 0x309a2a80 seq 4040 osr(3.310 0x3ed0d80) 861 bytes (queue has 50 ops and 3930776 bytes) 2014-06-09 16:22:12.104098 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.104100 7f868ef60700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104100 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff4b40 (1 active) 2014-06-09 16:22:12.104102 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406240 (2 active) 2014-06-09 16:22:12.104105 7f868bf5a700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104106 7f86a5610700 5 _do_op 0x160977a0 seq 3992 osr(0.6a8 0x2bd7d90)/0x2bd7d90 start 2014-06-09 16:22:12.104108 7f86a5610700 10 _do_transaction on 0xc058fc0 2014-06-09 16:22:12.104109 7f868e75f700 10 op_submit_finish 4040 2014-06-09 16:22:12.104109 7f8691765700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104113 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.104114 7f868af58700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104118 7f868df5e700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104112 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.104122 7f8690763700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104126 7f8690f64700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104131 7f868a757700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104135 7f8689f56700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104138 7f8689755700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104125 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.104143 7f868ff62700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104147 7f8692f68700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104142 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.104151 7f868f761700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104151 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.104158 7f8691f66700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104157 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.104163 7f868cf5c700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104168 7f868d75d700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104165 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.104174 7f868b759700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104180 7f8692767700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104178 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:147 2014-06-09 16:22:12.104186 7f868e75f700 20 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.104192 7f868e75f700 20 osd.26 pg_epoch: 243 pg[0.4e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.104197 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.104201 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.104209 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.104214 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x26aa8900,0x26aa8800,0x26aa8700,0x26aa8600,0x26aa8500,0x26aa8400 2014-06-09 16:22:12.104231 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.104236 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.104241 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.104249 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1043 2014-06-09 16:22:12.104254 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.104259 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.4e2( v 213'643 (0'0,213'643] local-les=243 n=121 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.104264 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.104267 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.104275 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.988002 7 0.000417 2014-06-09 16:22:12.104282 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.104288 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.104294 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000012 0 0.000000 2014-06-09 16:22:12.104300 7f868e75f700 5 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.104305 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.104310 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.104316 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.104323 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.104329 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.104337 7f868e75f700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:12.104348 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:975 2014-06-09 16:22:12.104379 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.104382 7f868e75f700 5 queue_transactions existing osr(3.de 0x3ed0520)/0x3ed0520 2014-06-09 16:22:12.104384 7f868e75f700 2 waiting 51 > 50 ops || 3931634 > 104857600 2014-06-09 16:22:12.104420 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.104435 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3991.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.104439 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.104458 7f86a5e11700 10 _do_op 0x16096a20 seq 3991 r = 0, finisher 0xc6e89e0 0 2014-06-09 16:22:12.104468 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205d600] 2014-06-09 16:22:12.104470 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205d600] 2014-06-09 16:22:12.104472 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.104473 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.104476 7f86a5e11700 10 _finish_op 0x16096a20 seq 3991 osr(1.65e 0x384df30)/0x384df30 2014-06-09 16:22:12.104486 7f868ef60700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104485 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e89e0] 2014-06-09 16:22:12.104485 7f868c75b700 10 op_submit_start 4041 2014-06-09 16:22:12.104488 7f868c75b700 5 queue_transactions (trailing journal) 4041 2014-06-09 16:22:12.104490 7f868bf5a700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104491 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e89e0] 2014-06-09 16:22:12.104490 7f868c75b700 5 queue_op 0xc64b9e0 seq 4041 osr(0.219 0x2bd6ad0) 828 bytes (queue has 50 ops and 3930776 bytes) 2014-06-09 16:22:12.104493 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.104494 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.104484 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115680 (1 active) 2014-06-09 16:22:12.104496 7f8691765700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104496 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fcb40 (2 active) 2014-06-09 16:22:12.104500 7f868af58700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104502 7f868c75b700 10 op_submit_finish 4041 2014-06-09 16:22:12.104502 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.104504 7f868df5e700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104509 7f8690763700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104513 7f8690f64700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104505 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.1e3( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.1e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.104517 7f868a757700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104516 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3992.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.104519 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.104521 7f8689f56700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104525 7f8689755700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104529 7f868ff62700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104530 7f86a5610700 10 _do_op 0x160977a0 seq 3992 r = 0, finisher 0x324d0300 0 2014-06-09 16:22:12.104533 7f8692f68700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104536 7f86a5610700 10 _finish_op 0x160977a0 seq 3992 osr(0.6a8 0x2bd7d90)/0x2bd7d90 2014-06-09 16:22:12.104538 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.104543 7f868c75b700 5 queue_transactions existing osr(2.1e3 0x2ed0810)/0x2ed0810 2014-06-09 16:22:12.104545 7f868c75b700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104546 7f8691f66700 10 op_submit_start 4042 2014-06-09 16:22:12.104548 7f8691f66700 5 queue_transactions (trailing journal) 4042 2014-06-09 16:22:12.104549 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a7c40] 2014-06-09 16:22:12.104549 7f8691f66700 5 queue_op 0x3250f380 seq 4042 osr(3.7ae 0xb7f01d0) 828 bytes (queue has 50 ops and 3930776 bytes) 2014-06-09 16:22:12.104551 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a7c40] 2014-06-09 16:22:12.104552 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.104553 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.104553 7f868cf5c700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104557 7f868d75d700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104556 7f86a5e11700 5 _do_op 0xbf7f740 seq 3993 osr(2.30b 0x2ed0f50)/0x2ed0f50 start 2014-06-09 16:22:12.104559 7f86a5e11700 10 _do_transaction on 0xc2df440 2014-06-09 16:22:12.104562 7f868b759700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104565 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.104566 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d0300] 2014-06-09 16:22:12.104570 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d0300] 2014-06-09 16:22:12.104577 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.104579 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.104582 7f8692767700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104587 7f868e75f700 2 waiting 51 > 50 ops || 3931634 > 104857600 2014-06-09 16:22:12.104587 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406240 (1 active) 2014-06-09 16:22:12.104589 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd680 (2 active) 2014-06-09 16:22:12.104592 7f868bf5a700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104593 7f8691f66700 10 op_submit_finish 4042 2014-06-09 16:22:12.104597 7f8691765700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104602 7f868af58700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104606 7f868df5e700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104610 7f8690763700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104613 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (19 active) 2014-06-09 16:22:12.104614 7f8690f64700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104614 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.104618 7f868a757700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104620 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.104622 7f8689f56700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104626 7f8689755700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104626 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3993.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.104630 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.104630 7f868ff62700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104634 7f8692f68700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104625 7f8691f66700 10 osd.26 243 dequeue_op 0x2976c700 prio 63 cost 4195304 latency 0.014941 MOSDPGPush(3.21e 243 [PushOp(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3, version: 221'617, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3@221'617, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 pg pg[3.21e( v 221'617 lc 221'616 (0'0,221'617] local-les=243 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=221'617 lcod 0'0 active m=1] 2014-06-09 16:22:12.104650 7f86a5e11700 10 _do_op 0xbf7f740 seq 3993 r = 0, finisher 0x31435cc0 0 2014-06-09 16:22:12.104659 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30628080] 2014-06-09 16:22:12.104659 7f868ef60700 2 waiting 51 > 50 ops || 3931602 > 104857600 2014-06-09 16:22:12.104661 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30628080] 2014-06-09 16:22:12.104663 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.104664 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.104655 7f8691f66700 5 -- op tracker -- , seq: 3648, time: 2014-06-09 16:22:12.104647, event: reached_pg, op: MOSDPGPush(3.21e 243 [PushOp(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3, version: 221'617, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(90df621e/rbd_data.11796b8b4567.0000000000001ad8/head//3@221'617, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.104665 7f86a5610700 5 _do_op 0x30a2060 seq 3994 osr(1.2ef 0x384d0f0)/0x384d0f0 start 2014-06-09 16:22:12.104665 7f868f761700 2 waiting 51 > 50 ops || 3931604 > 104857600 2014-06-09 16:22:12.104667 7f86a5e11700 10 _finish_op 0xbf7f740 seq 3993 osr(2.30b 0x2ed0f50)/0x2ed0f50 2014-06-09 16:22:12.104668 7f86a5610700 10 _do_transaction on 0xc0cb080 2014-06-09 16:22:12.104678 7f868cf5c700 10 op_submit_start 4043 2014-06-09 16:22:12.104677 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fcb40 (1 active) 2014-06-09 16:22:12.104679 7f868cf5c700 5 queue_transactions (trailing journal) 4043 2014-06-09 16:22:12.104680 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525320 (2 active) 2014-06-09 16:22:12.104669 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.21e( v 221'617 lc 221'616 (0'0,221'617] local-les=243 n=121 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=221'617 lcod 0'0 active m=1] 1 flushes_in_progress pending waiting for active on 0x2976c700 2014-06-09 16:22:12.104680 7f868cf5c700 5 queue_op 0x324e6a80 seq 4043 osr(2.45b 0x2ed1530) 861 bytes (queue has 50 ops and 3930809 bytes) 2014-06-09 16:22:12.104682 7f86a5e11700 5 _do_op 0x16097200 seq 3995 osr(3.527 0x3ed1660)/0x3ed1660 start 2014-06-09 16:22:12.104683 7f868cf5c700 10 op_submit_finish 4043 2014-06-09 16:22:12.104682 7f8691f66700 10 osd.26 243 dequeue_op 0x2976c700 finish 2014-06-09 16:22:12.104684 7f86a5e11700 10 _do_transaction on 0xbf2e6c0 2014-06-09 16:22:12.104686 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.104690 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.104685 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.104698 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.952062 7 0.000374 2014-06-09 16:22:12.104712 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (20 active) 2014-06-09 16:22:12.104708 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.104718 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.104716 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.104725 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.104727 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.104739 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3995.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.104742 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.104737 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.104750 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.104746 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.104753 7f86a5e11700 10 _do_op 0x16097200 seq 3995 r = 0, finisher 0x318c3980 0 2014-06-09 16:22:12.104755 7f8691f66700 5 queue_transactions existing osr(0.5e 0x2bd63b0)/0x2bd63b0 2014-06-09 16:22:12.104758 7f8691f66700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104753 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.104762 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c3a00] 2014-06-09 16:22:12.104764 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c3a00] 2014-06-09 16:22:12.104765 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.104766 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.104769 7f86a5e11700 10 _finish_op 0x16097200 seq 3995 osr(3.527 0x3ed1660)/0x3ed1660 2014-06-09 16:22:12.104763 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.104774 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525320 (1 active) 2014-06-09 16:22:12.104777 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359320 (2 active) 2014-06-09 16:22:12.104778 7f8691f66700 10 op_submit_start 4044 2014-06-09 16:22:12.104780 7f8691f66700 5 queue_transactions (trailing journal) 4044 2014-06-09 16:22:12.104779 7f86a5e11700 5 _do_op 0x300612c0 seq 3996 osr(2.c6 0x2ed0490)/0x2ed0490 start 2014-06-09 16:22:12.104779 7f868b759700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104781 7f86a5e11700 10 _do_transaction on 0xc1458c0 2014-06-09 16:22:12.104774 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.104781 7f8691f66700 5 queue_op 0x3250f200 seq 4044 osr(0.5e 0x2bd63b0) 826 bytes (queue has 50 ops and 3930807 bytes) 2014-06-09 16:22:12.104784 7f8691f66700 10 op_submit_finish 4044 2014-06-09 16:22:12.104785 7f8692767700 2 waiting 51 > 50 ops || 3931633 > 104857600 2014-06-09 16:22:12.104783 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.104789 7f868e75f700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.104794 7f868bf5a700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104795 7f868cf5c700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.104786 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.104786 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.456( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.456( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.104818 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.104812 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 2014-06-09 16:22:12.104823 7f8691765700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104828 7f868af58700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104831 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.104833 7f868df5e700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104837 7f8690763700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104840 7f8690f64700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104842 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.104843 7f868a757700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104845 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3996.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.104847 7f8689f56700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104848 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31435cc0,0x318c3980] 2014-06-09 16:22:12.104849 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.104851 7f8689755700 2 waiting 51 > 50 ops || 3931633 > 104857600 2014-06-09 16:22:12.104853 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31435cc0,0x318c3980] 2014-06-09 16:22:12.104855 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.104855 7f868ff62700 2 waiting 51 > 50 ops || 3931633 > 104857600 2014-06-09 16:22:12.104856 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.104859 7f8692f68700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104674 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.104860 7f86a5e11700 10 _do_op 0x300612c0 seq 3996 r = 0, finisher 0x3042b020 0 2014-06-09 16:22:12.104864 7f868ef60700 2 waiting 51 > 50 ops || 3931633 > 104857600 2014-06-09 16:22:12.104867 7f8691f66700 5 queue_transactions existing osr(0.456 0x2bd7530)/0x2bd7530 2014-06-09 16:22:12.104869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a7920] 2014-06-09 16:22:12.104871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a7920] 2014-06-09 16:22:12.104872 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.104872 7f868cf5c700 5 queue_transactions existing osr(2.5f1 0x2ed1a10)/0x2ed1a10 2014-06-09 16:22:12.104872 7f8691f66700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104873 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.104874 7f868cf5c700 2 waiting 51 > 50 ops || 3931668 > 104857600 2014-06-09 16:22:12.104876 7f86a5e11700 10 _finish_op 0x300612c0 seq 3996 osr(2.c6 0x2ed0490)/0x2ed0490 2014-06-09 16:22:12.104884 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359320 (1 active) 2014-06-09 16:22:12.104885 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b020] 2014-06-09 16:22:12.104886 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca49b00 (2 active) 2014-06-09 16:22:12.104888 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b020] 2014-06-09 16:22:12.104889 7f868c75b700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104890 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.104889 7f868f761700 10 op_submit_start 4045 2014-06-09 16:22:12.104891 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.104891 7f868f761700 5 queue_transactions (trailing journal) 4045 2014-06-09 16:22:12.104894 7f868d75d700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104893 7f868f761700 5 queue_op 0x230b9740 seq 4045 osr(1.6f5 0x2ed0170) 828 bytes (queue has 50 ops and 3930809 bytes) 2014-06-09 16:22:12.104894 7f86a5e11700 5 _do_op 0x30a3500 seq 3997 osr(2.13c 0x2ed0610)/0x2ed0610 start 2014-06-09 16:22:12.104897 7f868f761700 10 op_submit_finish 4045 2014-06-09 16:22:12.104897 7f86a5e11700 10 _do_transaction on 0xc22a780 2014-06-09 16:22:12.104902 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.104900 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.4a5( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 2.4a5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.104925 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.104936 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.104941 7f868f761700 5 queue_transactions existing osr(2.4a5 0x2ed17f0)/0x2ed17f0 2014-06-09 16:22:12.104939 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3994.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.104943 7f868f761700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104944 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.104948 7f868b759700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104953 7f8692767700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.104959 7f868bf5a700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104969 7f8691765700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104975 7f868af58700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104979 7f868df5e700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104980 7f86a5610700 10 _do_op 0x30a2060 seq 3994 r = 0, finisher 0x3251c6e0 0 2014-06-09 16:22:12.104983 7f8690763700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.104988 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd1e0] 2014-06-09 16:22:12.104989 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd1e0] 2014-06-09 16:22:12.104991 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.104992 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.104994 7f86a5610700 10 _finish_op 0x30a2060 seq 3994 osr(1.2ef 0x384d0f0)/0x384d0f0 2014-06-09 16:22:12.105025 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105036 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3997.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105039 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.105052 7f86a5e11700 10 _do_op 0x30a3500 seq 3997 r = 0, finisher 0x3206f980 0 2014-06-09 16:22:12.105062 7f868c75b700 10 op_submit_start 4046 2014-06-09 16:22:12.105061 7f868a757700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105063 7f868c75b700 5 queue_transactions (trailing journal) 4046 2014-06-09 16:22:12.105065 7f868c75b700 5 queue_op 0x288f9f80 seq 4046 osr(2.1e3 0x2ed0810) 828 bytes (queue has 50 ops and 3930809 bytes) 2014-06-09 16:22:12.105066 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251c6e0] 2014-06-09 16:22:12.105067 7f8689f56700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105072 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3206fa00] 2014-06-09 16:22:12.105072 7f8689755700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.105073 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251c6e0] 2014-06-09 16:22:12.105073 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3206fa00] 2014-06-09 16:22:12.105074 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105075 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.105075 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105076 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.105076 7f868ff62700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.105078 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd680 (1 active) 2014-06-09 16:22:12.105081 7f8692f68700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105080 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780240 (2 active) 2014-06-09 16:22:12.105085 7f8691f66700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105085 7f86a5610700 5 _do_op 0xc88a120 seq 3998 osr(3.124 0x3ed07c0)/0x3ed07c0 start 2014-06-09 16:22:12.105087 7f868c75b700 10 op_submit_finish 4046 2014-06-09 16:22:12.105090 7f868cf5c700 2 waiting 51 > 50 ops || 3931670 > 104857600 2014-06-09 16:22:12.105094 7f86a5610700 10 _do_transaction on 0xc05a240 2014-06-09 16:22:12.105090 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.b5( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.b5( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.105100 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.105122 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.105127 7f868c75b700 5 queue_transactions existing osr(1.b5 0x384c690)/0x384c690 2014-06-09 16:22:12.105129 7f868c75b700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.105134 7f868e75f700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105139 7f868d75d700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105139 7f86a5e11700 10 _finish_op 0x30a3500 seq 3997 osr(2.13c 0x2ed0610)/0x2ed0610 2014-06-09 16:22:12.105137 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105151 7f868a757700 10 op_submit_start 4047 2014-06-09 16:22:12.105151 7f868b759700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105152 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49b00 (1 active) 2014-06-09 16:22:12.105153 7f868a757700 5 queue_transactions (trailing journal) 4047 2014-06-09 16:22:12.105153 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3206f980] 2014-06-09 16:22:12.105157 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3206f980] 2014-06-09 16:22:12.105157 7f868bf5a700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105155 7f868a757700 5 queue_op 0xcabc360 seq 4047 osr(2.5c6 0x2ed1ad0) 828 bytes (queue has 50 ops and 3930809 bytes) 2014-06-09 16:22:12.105158 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105159 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105155 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3387e0 (2 active) 2014-06-09 16:22:12.105161 7f8691765700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105165 7f86a5e11700 5 _do_op 0x3250e180 seq 3999 osr(1.517 0x384db70)/0x384db70 start 2014-06-09 16:22:12.105166 7f868af58700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105168 7f86a5e11700 10 _do_transaction on 0x344c000 2014-06-09 16:22:12.105170 7f868df5e700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105170 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3998.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105174 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.105173 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.105174 7f8690763700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105178 7f8690f64700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105184 7f868a757700 10 op_submit_finish 4047 2014-06-09 16:22:12.105184 7f868ef60700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.105190 7f868f761700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105188 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.105201 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.945917 7 0.000328 2014-06-09 16:22:12.105215 7f86a5610700 10 _do_op 0xc88a120 seq 3998 r = 0, finisher 0xbd5ff00 0 2014-06-09 16:22:12.105213 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.105222 7f86a5610700 10 _finish_op 0xc88a120 seq 3998 osr(3.124 0x3ed07c0)/0x3ed07c0 2014-06-09 16:22:12.105223 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc410e80] 2014-06-09 16:22:12.105225 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc410e80] 2014-06-09 16:22:12.105227 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.105228 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.105222 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.105230 7f868b759700 10 op_submit_start 4048 2014-06-09 16:22:12.105232 7f868b759700 5 queue_transactions (trailing journal) 4048 2014-06-09 16:22:12.105233 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ff00] 2014-06-09 16:22:12.105233 7f868b759700 5 queue_op 0xc88ab40 seq 4048 osr(2.213 0x2ed0b70) 828 bytes (queue has 50 ops and 3930809 bytes) 2014-06-09 16:22:12.105237 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ff00] 2014-06-09 16:22:12.105238 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105239 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105233 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.105242 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780240 (1 active) 2014-06-09 16:22:12.105244 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b9e0 (2 active) 2014-06-09 16:22:12.105248 7f86a5610700 5 _do_op 0x10039920 seq 4000 osr(3.2d6 0x3ed0e60)/0x3ed0e60 start 2014-06-09 16:22:12.105250 7f868b759700 10 op_submit_finish 4048 2014-06-09 16:22:12.105243 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.105251 7f86a5610700 10 _do_transaction on 0x390c240 2014-06-09 16:22:12.105254 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105256 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.105252 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.105259 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.105253 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.7bc( v 213'570 (0'0,213'570] local-les=243 n=110 ec=118 les/c 243/243 242/242/223) 2014-06-09 16:22:12.105268 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 3999.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105272 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.105268 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.105286 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.105278 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.105290 7f868b759700 5 queue_transactions existing osr(3.7bc 0xb7f0190)/0xb7f0190 2014-06-09 16:22:12.105292 7f868b759700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105287 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.105295 7f86a5e11700 10 _do_op 0x3250e180 seq 3999 r = 0, finisher 0x324f9e60 0 2014-06-09 16:22:12.105296 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.105304 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6e4c0] 2014-06-09 16:22:12.105306 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6e4c0] 2014-06-09 16:22:12.105308 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.105309 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.105305 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.105312 7f86a5e11700 10 _finish_op 0x3250e180 seq 3999 osr(1.517 0x384db70)/0x384db70 2014-06-09 16:22:12.105318 7f868a757700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.105319 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3387e0 (1 active) 2014-06-09 16:22:12.105320 7f868b759700 10 op_submit_start 4049 2014-06-09 16:22:12.105322 7f868b759700 5 queue_transactions (trailing journal) 4049 2014-06-09 16:22:12.105320 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f9e60] 2014-06-09 16:22:12.105321 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e240 (2 active) 2014-06-09 16:22:12.105323 7f868b759700 5 queue_op 0xc88af60 seq 4049 osr(3.7bc 0xb7f0190) 828 bytes (queue has 50 ops and 3930809 bytes) 2014-06-09 16:22:12.105324 7f86a5e11700 5 _do_op 0x308cbe00 seq 4001 osr(0.186 0x2bd67d0)/0x2bd67d0 start 2014-06-09 16:22:12.105326 7f868b759700 10 op_submit_finish 4049 2014-06-09 16:22:12.105327 7f86a5e11700 10 _do_transaction on 0xc2d3bc0 2014-06-09 16:22:12.105327 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f9e60] 2014-06-09 16:22:12.105329 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105330 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105332 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.105329 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 243/203 242/242/202) 2014-06-09 16:22:12.105334 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.105340 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105353 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4000.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105357 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.105358 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.105362 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.105362 7f868b759700 5 queue_transactions existing osr(3.4fe 0x3ed1730)/0x3ed1730 2014-06-09 16:22:12.105364 7f868b759700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105366 7f868a757700 5 queue_transactions existing osr(1.109 0x384c950)/0x384c950 2014-06-09 16:22:12.105368 7f868a757700 2 waiting 51 > 50 ops || 3931670 > 104857600 2014-06-09 16:22:12.105373 7f8689f56700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105377 7f8689755700 2 waiting 51 > 50 ops || 3931635 > 104857600 2014-06-09 16:22:12.105377 7f86a5610700 10 _do_op 0x10039920 seq 4000 r = 0, finisher 0xbd689a0 0 2014-06-09 16:22:12.105382 7f8692f68700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105387 7f8691f66700 2 waiting 51 > 50 ops || 3931637 > 104857600 2014-06-09 16:22:12.105387 7f86a5610700 10 _finish_op 0x10039920 seq 4000 osr(3.2d6 0x3ed0e60)/0x3ed0e60 2014-06-09 16:22:12.105389 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32480a40] 2014-06-09 16:22:12.105392 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32480a40] 2014-06-09 16:22:12.105393 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.105394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.105398 7f868b759700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105399 7f868e75f700 10 op_submit_start 4050 2014-06-09 16:22:12.105402 7f868e75f700 5 queue_transactions (trailing journal) 4050 2014-06-09 16:22:12.105403 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd689a0] 2014-06-09 16:22:12.105403 7f868e75f700 5 queue_op 0xc64a660 seq 4050 osr(3.de 0x3ed0520) 858 bytes (queue has 50 ops and 3930839 bytes) 2014-06-09 16:22:12.105405 7f8692767700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105406 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd689a0] 2014-06-09 16:22:12.105408 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105409 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105410 7f868bf5a700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105411 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b9e0 (1 active) 2014-06-09 16:22:12.105413 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52dd40 (2 active) 2014-06-09 16:22:12.105415 7f8691765700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105417 7f86a5610700 5 _do_op 0x159965a0 seq 4002 osr(3.f0 0x3ed04a0)/0x3ed04a0 start 2014-06-09 16:22:12.105420 7f86a5610700 10 _do_transaction on 0xbf82300 2014-06-09 16:22:12.105420 7f868af58700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105424 7f868df5e700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105426 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.105427 7f868e75f700 10 op_submit_finish 4050 2014-06-09 16:22:12.105428 7f8690763700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105432 7f8690f64700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105436 7f868ef60700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105431 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.105441 7f868f761700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105441 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.930784 7 0.000432 2014-06-09 16:22:12.105447 7f868ff62700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105452 7f868cf5c700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.105448 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.105458 7f868c75b700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105462 7f868a757700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.105457 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.105468 7f8689f56700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105467 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.105472 7f8689755700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105473 7f868e75f700 5 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.105476 7f8692f68700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105480 7f8691f66700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105478 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.105485 7f868d75d700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105482 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.105491 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.105498 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.105503 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.105511 7f868e75f700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.105523 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.105553 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.105556 7f868e75f700 5 queue_transactions existing osr(2.59f 0x2ed1830)/0x2ed1830 2014-06-09 16:22:12.105558 7f868e75f700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.105587 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105601 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4001.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105605 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.105627 7f86a5e11700 10 _do_op 0x308cbe00 seq 4001 r = 0, finisher 0xcd32300 0 2014-06-09 16:22:12.105637 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32073ac0] 2014-06-09 16:22:12.105639 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32073ac0] 2014-06-09 16:22:12.105640 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.105641 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.105644 7f86a5e11700 10 _finish_op 0x308cbe00 seq 4001 osr(0.186 0x2bd67d0)/0x2bd67d0 2014-06-09 16:22:12.105651 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e240 (1 active) 2014-06-09 16:22:12.105654 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd9e0 (2 active) 2014-06-09 16:22:12.105653 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd32300] 2014-06-09 16:22:12.105654 7f868b759700 10 op_submit_start 4051 2014-06-09 16:22:12.105657 7f868b759700 5 queue_transactions (trailing journal) 4051 2014-06-09 16:22:12.105656 7f86a5e11700 5 _do_op 0x324a4f60 seq 4003 osr(0.3a6 0x2bd7330)/0x2bd7330 start 2014-06-09 16:22:12.105658 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd32300] 2014-06-09 16:22:12.105659 7f86a5e11700 10 _do_transaction on 0x37e4480 2014-06-09 16:22:12.105660 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105658 7f868b759700 5 queue_op 0xc88a0c0 seq 4051 osr(3.4fe 0x3ed1730) 828 bytes (queue has 50 ops and 3930839 bytes) 2014-06-09 16:22:12.105660 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105661 7f868b759700 10 op_submit_finish 4051 2014-06-09 16:22:12.105664 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.105670 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105664 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.4c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.105684 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4002.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105688 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.105703 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.105708 7f868b759700 5 queue_transactions existing osr(0.4c8 0x2bd7670)/0x2bd7670 2014-06-09 16:22:12.105711 7f868b759700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105711 7f86a5610700 10 _do_op 0x159965a0 seq 4002 r = 0, finisher 0xbd5ab80 0 2014-06-09 16:22:12.105716 7f8692767700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105717 7f86a5610700 10 _finish_op 0x159965a0 seq 4002 osr(3.f0 0x3ed04a0)/0x3ed04a0 2014-06-09 16:22:12.105718 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd0f8e0] 2014-06-09 16:22:12.105721 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd0f8e0] 2014-06-09 16:22:12.105722 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.105723 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.105724 7f868b759700 10 op_submit_start 4052 2014-06-09 16:22:12.105726 7f868b759700 5 queue_transactions (trailing journal) 4052 2014-06-09 16:22:12.105727 7f868bf5a700 2 waiting 51 > 50 ops || 3931669 > 104857600 2014-06-09 16:22:12.105727 7f868b759700 5 queue_op 0xc88a960 seq 4052 osr(0.4c8 0x2bd7670) 828 bytes (queue has 50 ops and 3930841 bytes) 2014-06-09 16:22:12.105728 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ab80] 2014-06-09 16:22:12.105731 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ab80] 2014-06-09 16:22:12.105733 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105736 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52dd40 (1 active) 2014-06-09 16:22:12.105739 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524fc0 (2 active) 2014-06-09 16:22:12.105742 7f86a5610700 5 _do_op 0xc88aea0 seq 4004 osr(0.25d 0x2bd6a00)/0x2bd6a00 start 2014-06-09 16:22:12.105744 7f868b759700 10 op_submit_finish 4052 2014-06-09 16:22:12.105745 7f86a5610700 10 _do_transaction on 0xbf88480 2014-06-09 16:22:12.105750 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.105747 7f868b759700 10 osd.26 pg_epoch: 243 pg[2.26d( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.26d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.105759 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105772 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4003.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105776 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.105777 7f8691765700 2 waiting 51 > 50 ops || 3931669 > 104857600 2014-06-09 16:22:12.105782 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.105783 7f868af58700 2 waiting 51 > 50 ops || 3931669 > 104857600 2014-06-09 16:22:12.105787 7f868df5e700 2 waiting 51 > 50 ops || 3931669 > 104857600 2014-06-09 16:22:12.105787 7f868b759700 5 queue_transactions existing osr(2.26d 0x2ed0de0)/0x2ed0de0 2014-06-09 16:22:12.105789 7f868b759700 2 waiting 51 > 50 ops || 3931669 > 104857600 2014-06-09 16:22:12.105793 7f868ef60700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105793 7f86a5e11700 10 _do_op 0x324a4f60 seq 4003 r = 0, finisher 0xbd70d40 0 2014-06-09 16:22:12.105798 7f8690f64700 2 waiting 51 > 50 ops || 3931669 > 104857600 2014-06-09 16:22:12.105803 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7a1c0] 2014-06-09 16:22:12.105805 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7a1c0] 2014-06-09 16:22:12.105806 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.105807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.105809 7f868f761700 2 waiting 51 > 50 ops || 3931669 > 104857600 2014-06-09 16:22:12.105810 7f86a5e11700 10 _finish_op 0x324a4f60 seq 4003 osr(0.3a6 0x2bd7330)/0x2bd7330 2014-06-09 16:22:12.105824 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd9e0 (1 active) 2014-06-09 16:22:12.105826 7f868c75b700 10 op_submit_start 4053 2014-06-09 16:22:12.105827 7f868c75b700 5 queue_transactions (trailing journal) 4053 2014-06-09 16:22:12.105827 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922480 (2 active) 2014-06-09 16:22:12.105827 7f868a757700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.105829 7f86a5e11700 5 _do_op 0xbf7e1e0 seq 4005 osr(2.1e3 0x2ed0810)/0x2ed0810 start 2014-06-09 16:22:12.105829 7f868c75b700 5 queue_op 0x30060600 seq 4053 osr(1.b5 0x384c690) 826 bytes (queue has 50 ops and 3930839 bytes) 2014-06-09 16:22:12.105831 7f86a5e11700 10 _do_transaction on 0xbfec6c0 2014-06-09 16:22:12.105831 7f868c75b700 10 op_submit_finish 4053 2014-06-09 16:22:12.105833 7f8689f56700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105836 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.105837 7f8689755700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105842 7f8692f68700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105835 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.1f0( v 213'588 (0'0,213'588] local-les=243 n=115 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.105846 7f8691f66700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105851 7f868d75d700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105856 7f868e75f700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.105863 7f8692767700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105866 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.105869 7f8690763700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105870 7f868c75b700 5 queue_transactions existing osr(3.1f0 0x3ed0990)/0x3ed0990 2014-06-09 16:22:12.105872 7f868c75b700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105874 7f868cf5c700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.105875 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd70d40] 2014-06-09 16:22:12.105878 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd70d40] 2014-06-09 16:22:12.105879 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105878 7f8691765700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105879 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105883 7f868af58700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105885 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105886 7f868df5e700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105892 7f868b759700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105897 7f868ef60700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105898 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4004.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105902 7f8690f64700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105903 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.105907 7f868f761700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105913 7f868ff62700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105918 7f868bf5a700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105926 7f86a5610700 10 _do_op 0xc88aea0 seq 4004 r = 0, finisher 0x324d0a00 0 2014-06-09 16:22:12.105931 7f86a5610700 10 _finish_op 0xc88aea0 seq 4004 osr(0.25d 0x2bd6a00)/0x2bd6a00 2014-06-09 16:22:12.105933 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31434600] 2014-06-09 16:22:12.105935 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31434600] 2014-06-09 16:22:12.105937 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.105938 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.105942 7f8689f56700 10 op_submit_start 4054 2014-06-09 16:22:12.105943 7f8689f56700 5 queue_transactions (trailing journal) 4054 2014-06-09 16:22:12.105944 7f8689f56700 5 queue_op 0x32833620 seq 4054 osr(1.3c3 0x384d530) 828 bytes (queue has 50 ops and 3930839 bytes) 2014-06-09 16:22:12.105949 7f8689755700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105954 7f8692f68700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105959 7f8691f66700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105965 7f868d75d700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105970 7f868e75f700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.105966 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.105976 7f8692767700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.105979 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d0a00] 2014-06-09 16:22:12.105982 7f8690763700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105984 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d0a00] 2014-06-09 16:22:12.105986 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.105987 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.105990 7f868c75b700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.105988 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4005.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.105992 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.105995 7f868cf5c700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.105995 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524fc0 (1 active) 2014-06-09 16:22:12.105997 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301b00 (2 active) 2014-06-09 16:22:12.106000 7f8691765700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106002 7f8689f56700 10 op_submit_finish 4054 2014-06-09 16:22:12.106004 7f868af58700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106003 7f86a5e11700 10 _do_op 0xbf7e1e0 seq 4005 r = 0, finisher 0xbd72280 0 2014-06-09 16:22:12.106008 7f868df5e700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc8f40] 2014-06-09 16:22:12.106013 7f868b759700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106015 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc8f40] 2014-06-09 16:22:12.106016 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.106018 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.106006 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.106018 7f868ef60700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.106020 7f86a5e11700 10 _finish_op 0xbf7e1e0 seq 4005 osr(2.1e3 0x2ed0810)/0x2ed0810 2014-06-09 16:22:12.106030 7f868f761700 10 op_submit_start 4055 2014-06-09 16:22:12.106032 7f868f761700 5 queue_transactions (trailing journal) 4055 2014-06-09 16:22:12.106032 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd72280] 2014-06-09 16:22:12.106035 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd72280] 2014-06-09 16:22:12.106033 7f868f761700 5 queue_op 0xc2ea6c0 seq 4055 osr(2.4a5 0x2ed17f0) 828 bytes (queue has 50 ops and 3930839 bytes) 2014-06-09 16:22:12.106036 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.106037 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.106039 7f868ff62700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.106040 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.106040 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922480 (1 active) 2014-06-09 16:22:12.106042 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b7a0 (2 active) 2014-06-09 16:22:12.106044 7f868bf5a700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106045 7f8689f56700 5 queue_transactions existing osr(3.58f 0x3ed1850)/0x3ed1850 2014-06-09 16:22:12.106045 7f86a5e11700 5 _do_op 0x30607f80 seq 4007 osr(3.461 0x3ed1120)/0x3ed1120 start 2014-06-09 16:22:12.106047 7f8689f56700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106048 7f86a5e11700 10 _do_transaction on 0xc149680 2014-06-09 16:22:12.106050 7f868f761700 10 op_submit_finish 4055 2014-06-09 16:22:12.106051 7f8689755700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.106053 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.106056 7f868a757700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.106061 7f8692f68700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106054 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.66e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.106066 7f868d75d700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106066 7f86a5610700 5 _do_op 0x10038a20 seq 4006 osr(2.38a 0x2ed10f0)/0x2ed10f0 start 2014-06-09 16:22:12.106069 7f86a5610700 10 _do_transaction on 0xc058000 2014-06-09 16:22:12.106071 7f868e75f700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.106074 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.106090 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.106095 7f868f761700 5 queue_transactions existing osr(1.66e 0x384ded0)/0x384ded0 2014-06-09 16:22:12.106095 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.106098 7f868f761700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106102 7f8692767700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.106108 7f8690763700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106109 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4007.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.106113 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.106114 7f868c75b700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106120 7f868cf5c700 2 waiting 51 > 50 ops || 3931700 > 104857600 2014-06-09 16:22:12.106125 7f8691765700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106129 7f868af58700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106133 7f868df5e700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106138 7f868b759700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106144 7f86a5e11700 10 _do_op 0x30607f80 seq 4007 r = 0, finisher 0x32075b60 0 2014-06-09 16:22:12.106147 7f868ef60700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.106152 7f8690f64700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106154 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32480a00] 2014-06-09 16:22:12.106156 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32480a00] 2014-06-09 16:22:12.106156 7f8691f66700 2 waiting 51 > 50 ops || 3931667 > 104857600 2014-06-09 16:22:12.106157 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.106158 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.106161 7f86a5e11700 10 _finish_op 0x30607f80 seq 4007 osr(3.461 0x3ed1120)/0x3ed1120 2014-06-09 16:22:12.106169 7f868ff62700 10 op_submit_start 4056 2014-06-09 16:22:12.108350 7f868ff62700 5 queue_transactions (trailing journal) 4056 2014-06-09 16:22:12.108354 7f868ff62700 5 queue_op 0x30a3ec0 seq 4056 osr(1.c6 0x384c610) 826 bytes (queue has 50 ops and 3930837 bytes) 2014-06-09 16:22:12.106172 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32075b60] 2014-06-09 16:22:12.108369 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32075b60] 2014-06-09 16:22:12.108372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.108374 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.106172 7f868bf5a700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.106175 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b7a0 (1 active) 2014-06-09 16:22:12.108385 7f8689f56700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108387 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7805a0 (2 active) 2014-06-09 16:22:12.108395 7f868ff62700 10 op_submit_finish 4056 2014-06-09 16:22:12.108397 7f8689755700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.108402 7f868a757700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.108408 7f8692f68700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.106477 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.108402 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.640( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:12.108416 7f868d75d700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108422 7f868e75f700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.108434 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4006.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.108439 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.108443 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.108452 7f868ff62700 5 queue_transactions existing osr(0.640 0x2bd7c10)/0x2bd7c10 2014-06-09 16:22:12.108455 7f868ff62700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108457 7f86a5610700 10 _do_op 0x10038a20 seq 4006 r = 0, finisher 0x318c39e0 0 2014-06-09 16:22:12.108459 7f868f761700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108466 7f8692767700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.108470 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32480980] 2014-06-09 16:22:12.108472 7f8690763700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108476 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32480980] 2014-06-09 16:22:12.108478 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.108480 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.108483 7f868c75b700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108485 7f86a6b95700 5 -- op tracker -- , seq: 3653, time: 2014-06-09 16:22:12.102408, event: header_read, op: osd_op(client.4542.0:34690 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2781184] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.108489 7f868cf5c700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.108490 7f86a5610700 10 _finish_op 0x10038a20 seq 4006 osr(2.38a 0x2ed10f0)/0x2ed10f0 2014-06-09 16:22:12.108504 7f868bf5a700 10 op_submit_start 4057 2014-06-09 16:22:12.108503 7f868af58700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108505 7f868bf5a700 5 queue_transactions (trailing journal) 4057 2014-06-09 16:22:12.108506 7f868bf5a700 5 queue_op 0xc34b6e0 seq 4057 osr(2.52d 0x2ed19f0) 828 bytes (queue has 50 ops and 3930837 bytes) 2014-06-09 16:22:12.108508 7f868df5e700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108508 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c39e0] 2014-06-09 16:22:12.108514 7f868b759700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108517 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c39e0] 2014-06-09 16:22:12.108519 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.108512 7f86a5e11700 5 _do_op 0x324a5b00 seq 4008 osr(2.126 0x2ed0670)/0x2ed0670 start 2014-06-09 16:22:12.108520 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.108520 7f8690f64700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108523 7f86a5e11700 10 _do_transaction on 0xbfa5740 2014-06-09 16:22:12.108523 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301b00 (1 active) 2014-06-09 16:22:12.108525 7f8691f66700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108526 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114900 (2 active) 2014-06-09 16:22:12.108529 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.108530 7f8689f56700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108535 7f8689755700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.108540 7f868a757700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.108543 7f8692f68700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108548 7f868d75d700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108550 7f86a5610700 5 _do_op 0x3248bf80 seq 4009 osr(3.206 0x3ed0cf0)/0x3ed0cf0 start 2014-06-09 16:22:12.108552 7f868bf5a700 10 op_submit_finish 4057 2014-06-09 16:22:12.108553 7f86a5610700 10 _do_transaction on 0xc262300 2014-06-09 16:22:12.108554 7f868e75f700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.108558 7f86a5610700 15 _omap_setkeys meta/1fe240ba/pglog_3.206/0//-1 2014-06-09 16:22:12.108556 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.529( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.529( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.108590 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.108596 7f868bf5a700 5 queue_transactions existing osr(0.529 0x2bd7930)/0x2bd7930 2014-06-09 16:22:12.108598 7f868bf5a700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108603 7f868ff62700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108608 7f868f761700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108612 7f8692767700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.108617 7f8690763700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108497 7f86a6b95700 5 -- op tracker -- , seq: 3653, time: 2014-06-09 16:22:12.102409, event: throttled, op: osd_op(client.4542.0:34690 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2781184] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.108623 7f868c75b700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108630 7f868cf5c700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.108629 7f86a6b95700 5 -- op tracker -- , seq: 3653, time: 2014-06-09 16:22:12.108169, event: all_read, op: osd_op(client.4542.0:34690 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2781184] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.108634 7f8691765700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.108639 7f868ef60700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.108640 7f86a6b95700 5 -- op tracker -- , seq: 3653, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34690 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2781184] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.108648 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.108673 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976c100 prio 63 cost 2781184 latency 0.006265 osd_op(client.4542.0:34690 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2781184] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.109576 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.109594 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4008.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.109598 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.109622 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe240ba/pglog_3.206/0//-1 2014-06-09 16:22:12.109626 7f86a5e11700 10 _do_op 0x324a5b00 seq 4008 r = 0, finisher 0x3281b080 0 2014-06-09 16:22:12.109633 7f86a5e11700 10 _finish_op 0x324a5b00 seq 4008 osr(2.126 0x2ed0670)/0x2ed0670 2014-06-09 16:22:12.109635 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3281b7c0] 2014-06-09 16:22:12.109637 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3281b7c0] 2014-06-09 16:22:12.109639 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.109640 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.109640 7f868af58700 10 op_submit_start 4058 2014-06-09 16:22:12.109640 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7805a0 (1 active) 2014-06-09 16:22:12.109642 7f868af58700 5 queue_transactions (trailing journal) 4058 2014-06-09 16:22:12.109643 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653440 (2 active) 2014-06-09 16:22:12.109644 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281b080] 2014-06-09 16:22:12.109644 7f868af58700 5 queue_op 0x324a4960 seq 4058 osr(0.652 0x2bd7b90) 828 bytes (queue has 50 ops and 3930837 bytes) 2014-06-09 16:22:12.109645 7f86a5e11700 5 _do_op 0x324a5d40 seq 4010 osr(1.113 0x384c910)/0x384c910 start 2014-06-09 16:22:12.109647 7f868af58700 10 op_submit_finish 4058 2014-06-09 16:22:12.109647 7f86a5e11700 10 _do_transaction on 0xc134a80 2014-06-09 16:22:12.109650 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281b080] 2014-06-09 16:22:12.109652 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.109653 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.109653 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.109651 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.778( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 243/243 242/242/220) 2014-06-09 16:22:12.109665 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.109685 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.109691 7f868af58700 5 queue_transactions existing osr(3.778 0x3ed1df0)/0x3ed1df0 2014-06-09 16:22:12.109692 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.109693 7f868af58700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109697 7f868df5e700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109703 7f868b759700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109709 7f8690f64700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109714 7f8691f66700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109715 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4010.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.109719 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.109719 7f8689f56700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109726 7f8689755700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.109731 7f868a757700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.109736 7f8692f68700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109741 7f868d75d700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109747 7f868e75f700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.109752 7f868bf5a700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109757 7f868ff62700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109762 7f868f761700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109766 7f8692767700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.109772 7f8690763700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109776 7f86a5e11700 10 _do_op 0x324a5d40 seq 4010 r = 0, finisher 0x20051be0 0 2014-06-09 16:22:12.109778 7f868c75b700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109781 7f86a5e11700 10 _finish_op 0x324a5d40 seq 4010 osr(1.113 0x384c910)/0x384c910 2014-06-09 16:22:12.109785 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a0027c0] 2014-06-09 16:22:12.109787 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a0027c0] 2014-06-09 16:22:12.109789 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.109788 7f868af58700 10 op_submit_start 4059 2014-06-09 16:22:12.109790 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653440 (1 active) 2014-06-09 16:22:12.109791 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.109791 7f868af58700 5 queue_transactions (trailing journal) 4059 2014-06-09 16:22:12.109792 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0120 (2 active) 2014-06-09 16:22:12.109793 7f868af58700 5 queue_op 0x324a4360 seq 4059 osr(3.778 0x3ed1df0) 828 bytes (queue has 50 ops and 3930837 bytes) 2014-06-09 16:22:12.109794 7f86a5e11700 5 _do_op 0x30a3f80 seq 4011 osr(2.33a 0x2ed0ed0)/0x2ed0ed0 start 2014-06-09 16:22:12.109795 7f868af58700 10 op_submit_finish 4059 2014-06-09 16:22:12.109794 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x20051be0] 2014-06-09 16:22:12.109796 7f86a5e11700 10 _do_transaction on 0xbe89440 2014-06-09 16:22:12.109798 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x20051be0] 2014-06-09 16:22:12.109800 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.109801 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.109802 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.109798 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 3.1fc( v 213'579 (0'0,213'579] local-les=243 n=113 ec=118 les/c 243/243 242/242/220) 2014-06-09 16:22:12.109811 7f86a6b95700 5 -- op tracker -- , seq: 3654, time: 2014-06-09 16:22:12.108704, event: header_read, op: osd_op(client.4542.0:34691 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 2781184~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.109822 7f86a6b95700 5 -- op tracker -- , seq: 3654, time: 2014-06-09 16:22:12.108706, event: throttled, op: osd_op(client.4542.0:34691 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 2781184~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.109831 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.109831 7f86a6b95700 5 -- op tracker -- , seq: 3654, time: 2014-06-09 16:22:12.109516, event: all_read, op: osd_op(client.4542.0:34691 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 2781184~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.109835 7f868af58700 5 queue_transactions existing osr(3.1fc 0x3ed0930)/0x3ed0930 2014-06-09 16:22:12.109838 7f868af58700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109838 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.109841 7f86a6b95700 5 -- op tracker -- , seq: 3654, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34691 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 2781184~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.109842 7f8691765700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109847 7f868ef60700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.109851 7f868df5e700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109856 7f8690f64700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109858 7f86a5610700 15 _omap_rmkeys meta/1fe240ba/pglog_3.206/0//-1 2014-06-09 16:22:12.109860 7f8691f66700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109865 7f8689f56700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109871 7f8689755700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.109874 7f868a757700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.109878 7f8692f68700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109884 7f868d75d700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109884 7f86a5610700 10 _omap_rmkeys meta/1fe240ba/pglog_3.206/0//-1 r = 0 2014-06-09 16:22:12.109889 7f868e75f700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.109846 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.109891 7f86a5610700 15 _omap_setkeys meta/1fe240ba/pglog_3.206/0//-1 2014-06-09 16:22:12.109894 7f868bf5a700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109900 7f868ff62700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109904 7f868f761700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109906 7f86a5610700 15 _touch 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.109909 7f8692767700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.109914 7f8690763700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109920 7f868c75b700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109927 7f868cf5c700 2 waiting 51 > 50 ops || 3931698 > 104857600 2014-06-09 16:22:12.109931 7f868b759700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.109925 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976c000 prio 63 cost 516096 latency 0.001221 osd_op(client.4542.0:34691 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 2781184~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.109958 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.206_head oid 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.109987 7f86a5610700 20 write_state seq is 30164 2014-06-09 16:22:12.109995 7f86a5610700 20 set_header setting 30163 cid 3.206_head oid 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 parent seq 0 2014-06-09 16:22:12.110010 7f86a5610700 10 _touch 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 = 0 2014-06-09 16:22:12.110017 7f86a5610700 15 _write 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 0~151552 2014-06-09 16:22:12.110027 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:12.110630 7f86a5610700 10 _generic_write 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 0~151552 = 0 2014-06-09 16:22:12.110648 7f86a5610700 15 _setattrs 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.110659 7f86a5610700 10 _setattrs 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 = 0 2014-06-09 16:22:12.110665 7f86a5610700 15 _setattrs 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 2014-06-09 16:22:12.110674 7f86a5610700 10 _setattrs 3.206_head/2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 = 0 2014-06-09 16:22:12.110693 7f86a6b95700 5 -- op tracker -- , seq: 3655, time: 2014-06-09 16:22:12.109953, event: header_read, op: osd_op(client.4542.0:34692 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3297280~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.110704 7f86a5610700 10 cid: oid: 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 not skipping op, *spos 4009.0.0 2014-06-09 16:22:12.110708 7f86a5610700 10 > header.spos 0.0.0 2014-06-09 16:22:12.110705 7f86a6b95700 5 -- op tracker -- , seq: 3655, time: 2014-06-09 16:22:12.109954, event: throttled, op: osd_op(client.4542.0:34692 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3297280~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.110717 7f86a5610700 20 set_header setting 30163 cid 3.206_head oid 2c355206/rbd_data.117d6b8b4567.00000000000016b8/head//3 parent seq 0 2014-06-09 16:22:12.110717 7f86a6b95700 5 -- op tracker -- , seq: 3655, time: 2014-06-09 16:22:12.110651, event: all_read, op: osd_op(client.4542.0:34692 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3297280~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.110730 7f86a6b95700 5 -- op tracker -- , seq: 3655, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34692 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3297280~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.110736 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.110742 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4009.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.110747 7f86a5610700 10 cid: oid: 1fe240ba/pglog_3.206/0//-1 skipping op, spos 4009.0.0 <= header.spos 150125.0.0 2014-06-09 16:22:12.110750 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.110752 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976cd00 prio 63 cost 516096 latency 0.000799 osd_op(client.4542.0:34692 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3297280~516096] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.111185 7f86a6b95700 5 -- op tracker -- , seq: 3656, time: 2014-06-09 16:22:12.110771, event: header_read, op: osd_op(client.4542.0:34693 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3813376~380928] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.111196 7f86a6b95700 5 -- op tracker -- , seq: 3656, time: 2014-06-09 16:22:12.110772, event: throttled, op: osd_op(client.4542.0:34693 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3813376~380928] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.111206 7f86a6b95700 5 -- op tracker -- , seq: 3656, time: 2014-06-09 16:22:12.111152, event: all_read, op: osd_op(client.4542.0:34693 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3813376~380928] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.111216 7f86a6b95700 5 -- op tracker -- , seq: 3656, time: 0.000000, event: dispatched, op: osd_op(client.4542.0:34693 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3813376~380928] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.111224 7f86a6b95700 20 osd.26 243 should_share_map client.4542 172.16.96.220:0/1123748 243 2014-06-09 16:22:12.111240 7f86a6b95700 15 osd.26 243 enqueue_op 0x2976cc00 prio 63 cost 380928 latency 0.000469 osd_op(client.4542.0:34693 rbd_data.115e6b8b4567.0000000000001822 [set-alloc-hint object_size 4194304 write_size 4194304,write 3813376~380928] 3.e8e75f2b RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.111429 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.111431 7f86a5610700 10 _do_op 0x3248bf80 seq 4009 r = 0, finisher 0x324f83e0 0x2fe6dc80 2014-06-09 16:22:12.111446 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050a980] 2014-06-09 16:22:12.111446 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4011.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.111450 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.111449 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=214'596 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 25 2014-06-09 16:22:12.111471 7f86a5e11700 10 _do_op 0x30a3f80 seq 4011 r = 0, finisher 0x324d0ac0 0 2014-06-09 16:22:12.111475 7f86a5e11700 10 _finish_op 0x30a3f80 seq 4011 osr(2.33a 0x2ed0ed0)/0x2ed0ed0 2014-06-09 16:22:12.111473 7f86a460e700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.111466, event: op_commit, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.111485 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0120 (1 active) 2014-06-09 16:22:12.111487 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cfc0 (2 active) 2014-06-09 16:22:12.111489 7f86a5e11700 5 _do_op 0xc64bce0 seq 4012 osr(1.327 0x384d370)/0x384d370 start 2014-06-09 16:22:12.111492 7f86a5e11700 10 _do_transaction on 0xc209680 2014-06-09 16:22:12.111482 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 luod=214'596 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 25 all committed 2014-06-09 16:22:12.111498 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.111494 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] eval_repop repgather(0xc143200 243'597 rep_tid=25 committed?=1 applied?=0 lock=0 op=osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.111522 7f868af58700 10 op_submit_start 4060 2014-06-09 16:22:12.111525 7f868af58700 5 queue_transactions (trailing journal) 4060 2014-06-09 16:22:12.111513 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] log_op_stats osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4 inb 151936 outb 0 rlat 0.000000 lat 1.437857 2014-06-09 16:22:12.111527 7f868af58700 5 queue_op 0xd04f620 seq 4060 osr(3.1fc 0x3ed0930) 828 bytes (queue has 50 ops and 3930837 bytes) 2014-06-09 16:22:12.111530 7f868af58700 10 op_submit_finish 4060 2014-06-09 16:22:12.111534 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.111533 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] publish_stats_to_osd 243:975 2014-06-09 16:22:12.111534 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.463( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.463( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.111549 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4012.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.111553 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.111545 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] sending commit on repgather(0xc143200 243'597 rep_tid=25 committed?=1 applied?=0 lock=0 op=osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4) 0x32070a00 2014-06-09 16:22:12.111568 7f86a5e11700 10 _do_op 0xc64bce0 seq 4012 r = 0, finisher 0xc169ea0 0 2014-06-09 16:22:12.111569 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.111572 7f86a5e11700 10 _finish_op 0xc64bce0 seq 4012 osr(1.327 0x384d370)/0x384d370 2014-06-09 16:22:12.111574 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cfc0 (1 active) 2014-06-09 16:22:12.111576 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e6c0 (2 active) 2014-06-09 16:22:12.111576 7f868af58700 5 queue_transactions existing osr(2.463 0x2ed1510)/0x2ed1510 2014-06-09 16:22:12.111579 7f868af58700 10 op_submit_start 4061 2014-06-09 16:22:12.111578 7f86a5e11700 5 _do_op 0x324b3920 seq 4013 osr(2.182 0x2ed0990)/0x2ed0990 start 2014-06-09 16:22:12.111580 7f868af58700 5 queue_transactions (trailing journal) 4061 2014-06-09 16:22:12.111580 7f86a5e11700 10 _do_transaction on 0x2ed4e40 2014-06-09 16:22:12.111581 7f868af58700 5 queue_op 0xd04eea0 seq 4061 osr(2.463 0x2ed1510) 828 bytes (queue has 50 ops and 3930804 bytes) 2014-06-09 16:22:12.111583 7f868af58700 10 op_submit_finish 4061 2014-06-09 16:22:12.111585 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.111583 7f86a460e700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.111577, event: commit_sent, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.111592 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050a980] 2014-06-09 16:22:12.111585 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.759( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.759( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.111594 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e18a00,0x29283b20] 2014-06-09 16:22:12.111598 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.111623 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.111626 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.111608 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.111629 7f868af58700 5 queue_transactions existing osr(2.759 0x3ed00b0)/0x3ed00b0 2014-06-09 16:22:12.111631 7f868af58700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111636 7f8691765700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111642 7f868ef60700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.111641 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4013.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.111636 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.327( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:12.111645 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.111647 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e18a00,0x29283b20] 2014-06-09 16:22:12.111647 7f868df5e700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111649 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.111650 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.111652 7f8690f64700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111655 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324d0ac0,0xc169ea0] 2014-06-09 16:22:12.111657 7f8691f66700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111657 7f86a5e11700 10 _do_op 0x324b3920 seq 4013 r = 0, finisher 0x31413e00 0 2014-06-09 16:22:12.111663 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324d0ac0,0xc169ea0] 2014-06-09 16:22:12.111665 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.111666 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.111666 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe306e0] 2014-06-09 16:22:12.111669 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe306e0] 2014-06-09 16:22:12.111670 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.111670 7f8689f56700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111672 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.111675 7f86a5e11700 10 _finish_op 0x324b3920 seq 4013 osr(2.182 0x2ed0990)/0x2ed0990 2014-06-09 16:22:12.111677 7f8689755700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.111684 7f868a757700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.111693 7f8692f68700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111699 7f868d75d700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111706 7f868e75f700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.111715 7f868bf5a700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111722 7f868ff62700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111727 7f868f761700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111732 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.111737 7f8690763700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111744 7f868c75b700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111753 7f868cf5c700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.111760 7f868b759700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111772 7f868af58700 10 op_submit_start 4062 2014-06-09 16:22:12.111771 7f8691765700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111774 7f868af58700 5 queue_transactions (trailing journal) 4062 2014-06-09 16:22:12.111773 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e6c0 (1 active) 2014-06-09 16:22:12.111775 7f868af58700 5 queue_op 0xd04f020 seq 4062 osr(2.759 0x3ed00b0) 828 bytes (queue has 50 ops and 3930804 bytes) 2014-06-09 16:22:12.111777 7f868ef60700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.111777 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cc60 (2 active) 2014-06-09 16:22:12.111782 7f86a5e11700 5 _do_op 0xc88b3e0 seq 4014 osr(2.26d 0x2ed0de0)/0x2ed0de0 start 2014-06-09 16:22:12.111786 7f86a5e11700 10 _do_transaction on 0xc10a9c0 2014-06-09 16:22:12.111792 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.111781 7f868df5e700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111800 7f8691f66700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111806 7f8689f56700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111812 7f8689755700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.111817 7f868a757700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.111821 7f8692f68700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111826 7f868d75d700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111833 7f868e75f700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.111840 7f868bf5a700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111846 7f868ff62700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111846 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.111851 7f868f761700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111855 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.111860 7f8690763700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111862 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4014.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.111864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31413e00] 2014-06-09 16:22:12.111866 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.111873 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31413e00] 2014-06-09 16:22:12.111875 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.111876 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.111877 7f86a5e11700 10 _do_op 0xc88b3e0 seq 4014 r = 0, finisher 0x31412a80 0 2014-06-09 16:22:12.111880 7f868c75b700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111883 7f86a5e11700 10 _finish_op 0xc88b3e0 seq 4014 osr(2.26d 0x2ed0de0)/0x2ed0de0 2014-06-09 16:22:12.111884 7f868cf5c700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.111890 7f868b759700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111895 7f8690f64700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111900 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3fc20] 2014-06-09 16:22:12.111902 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3fc20] 2014-06-09 16:22:12.111904 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.111905 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.111906 7f868ef60700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.111913 7f8691f66700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111915 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cc60 (1 active) 2014-06-09 16:22:12.111918 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcd80 (2 active) 2014-06-09 16:22:12.111919 7f8689f56700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111924 7f8689755700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.111923 7f86a5e11700 5 _do_op 0x11fce360 seq 4015 osr(3.18e 0x3ed0af0)/0x3ed0af0 start 2014-06-09 16:22:12.111926 7f86a5e11700 10 _do_transaction on 0x3144cc0 2014-06-09 16:22:12.111928 7f868a757700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.111931 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.111932 7f8692f68700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111937 7f868d75d700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111939 7f868af58700 10 op_submit_finish 4062 2014-06-09 16:22:12.111942 7f8691765700 10 op_submit_start 4063 2014-06-09 16:22:12.111943 7f8691765700 5 queue_transactions (trailing journal) 4063 2014-06-09 16:22:12.111943 7f868e75f700 2 waiting 51 > 50 ops || 3931665 > 104857600 2014-06-09 16:22:12.111945 7f8691765700 5 queue_op 0x159966c0 seq 4063 osr(2.236 0x2ed0ad0) 828 bytes (queue has 50 ops and 3930804 bytes) 2014-06-09 16:22:12.111948 7f8691765700 10 op_submit_finish 4063 2014-06-09 16:22:12.111943 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.111952 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.2d9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.2d9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.111959 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.111969 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x26aa8d00,0x26aa8c00,0x26aa8b00,0x26aa8a00,0x4679c00,0x2e075b00,0x2e075a00,0x2e075900,0x2e075800,0x2e075700 2014-06-09 16:22:12.111981 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.111984 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.111988 7f8691765700 5 queue_transactions existing osr(2.2d9 0x2ed0c30)/0x2ed0c30 2014-06-09 16:22:12.111990 7f8691765700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.111995 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4015.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.112000 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.112012 7f86a5e11700 10 _do_op 0x11fce360 seq 4015 r = 0, finisher 0x20051100 0 2014-06-09 16:22:12.112004 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112018 7f86a5e11700 10 _finish_op 0x11fce360 seq 4015 osr(3.18e 0x3ed0af0)/0x3ed0af0 2014-06-09 16:22:12.112016 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112029 7f868ff62700 10 op_submit_start 4064 2014-06-09 16:22:12.112031 7f868ff62700 5 queue_transactions (trailing journal) 4064 2014-06-09 16:22:12.112026 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112033 7f868ff62700 5 queue_op 0x30a2180 seq 4064 osr(0.640 0x2bd7c10) 828 bytes (queue has 50 ops and 3930804 bytes) 2014-06-09 16:22:12.112038 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcd80 (1 active) 2014-06-09 16:22:12.112041 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287c20 (2 active) 2014-06-09 16:22:12.112046 7f868ff62700 10 op_submit_finish 4064 2014-06-09 16:22:12.112041 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:988 2014-06-09 16:22:12.112050 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.3a9( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.3a9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.112052 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112063 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.464( v 213'621 (0'0,213'621] local-les=243 n=120 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112073 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112075 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412a80,0x20051100] 2014-06-09 16:22:12.112080 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412a80,0x20051100] 2014-06-09 16:22:12.112082 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.112083 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.112085 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.112079 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112090 7f868ff62700 5 queue_transactions existing osr(0.3a9 0x2bd7310)/0x2bd7310 2014-06-09 16:22:12.112093 7f868ff62700 2 waiting 51 > 50 ops || 3931632 > 104857600 2014-06-09 16:22:12.112091 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112097 7f86a5e11700 5 _do_op 0xc34a0c0 seq 4016 osr(0.529 0x2bd7930)/0x2bd7930 start 2014-06-09 16:22:12.112102 7f86a5e11700 10 _do_transaction on 0xc05f500 2014-06-09 16:22:12.112101 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.112107 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.112109 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112118 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112125 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112138 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:12.112147 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.112147 7f868af58700 20 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112155 7f868af58700 20 osd.26 pg_epoch: 243 pg[0.1a7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112164 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112162 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4016.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.112166 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.112168 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112180 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112188 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.112195 7f86a5e11700 10 _do_op 0xc34a0c0 seq 4016 r = 0, finisher 0x3045ace0 0 2014-06-09 16:22:12.112200 7f86a5e11700 10 _finish_op 0xc34a0c0 seq 4016 osr(0.529 0x2bd7930)/0x2bd7930 2014-06-09 16:22:12.112197 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112205 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112212 7f8692767700 10 op_submit_start 4065 2014-06-09 16:22:12.112214 7f8692767700 5 queue_transactions (trailing journal) 4065 2014-06-09 16:22:12.112214 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112215 7f8692767700 5 queue_op 0x16097500 seq 4065 osr(1.6f 0x384c570) 826 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.112212 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112227 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3045ace0] 2014-06-09 16:22:12.112231 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3045ace0] 2014-06-09 16:22:12.112225 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.112233 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.112234 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.112237 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287c20 (1 active) 2014-06-09 16:22:12.112240 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1320 (2 active) 2014-06-09 16:22:12.112234 7f868af58700 20 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112245 7f8692767700 10 op_submit_finish 4065 2014-06-09 16:22:12.112243 7f868af58700 20 osd.26 pg_epoch: 243 pg[2.521( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112252 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112249 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.66a( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.66a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.112256 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112270 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112287 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.112280 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1b0b6e00,0x1b0b6d00,0x1b0b6c00,0x1b0b6b00,0x1c66bf00,0x2c06aa00 2014-06-09 16:22:12.112292 7f8692767700 5 queue_transactions existing osr(0.66a 0x2bd7ef0)/0x2bd7ef0 2014-06-09 16:22:12.112295 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112299 7f86a5e11700 5 _do_op 0x30060c60 seq 4017 osr(2.236 0x2ed0ad0)/0x2ed0ad0 start 2014-06-09 16:22:12.112302 7f86a5e11700 10 _do_transaction on 0xbf44cc0 2014-06-09 16:22:12.112308 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.112308 7f868c75b700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112306 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112317 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112326 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112346 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.112340 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:917 2014-06-09 16:22:12.112351 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112361 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4017.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.112365 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.112360 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.7e4( v 214'548 (0'0,214'548] local-les=243 n=108 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112371 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112376 7f86a5e11700 10 _do_op 0x30060c60 seq 4017 r = 0, finisher 0x3205cdc0 0 2014-06-09 16:22:12.112380 7f86a5e11700 10 _finish_op 0x30060c60 seq 4017 osr(2.236 0x2ed0ad0)/0x2ed0ad0 2014-06-09 16:22:12.112375 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112390 7f8690763700 10 op_submit_start 4066 2014-06-09 16:22:12.112392 7f8690763700 5 queue_transactions (trailing journal) 4066 2014-06-09 16:22:12.112393 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205cdc0] 2014-06-09 16:22:12.112387 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112393 7f8690763700 5 queue_op 0x324b2540 seq 4066 osr(1.494 0x384d990) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.112396 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205cdc0] 2014-06-09 16:22:12.112398 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.112398 7f868b759700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112399 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.112395 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.112402 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1320 (1 active) 2014-06-09 16:22:12.112403 7f8690f64700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112405 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a6c0 (2 active) 2014-06-09 16:22:12.112408 7f868df5e700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112409 7f86a5e11700 5 _do_op 0x15997bc0 seq 4018 osr(3.1d5 0x3ed09f0)/0x3ed09f0 start 2014-06-09 16:22:12.112404 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112413 7f86a5e11700 10 _do_transaction on 0xc2bdb00 2014-06-09 16:22:12.112412 7f8691f66700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112417 7f8690763700 10 op_submit_finish 4066 2014-06-09 16:22:12.112413 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112418 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.112420 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112420 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.35c( v 213'649 (0'0,213'649] local-les=243 n=123 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:12.112432 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.112439 7f868af58700 20 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112446 7f868af58700 20 osd.26 pg_epoch: 243 pg[1.373( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112452 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.112455 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112455 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.112457 7f8690763700 5 queue_transactions existing osr(3.35c 0x3ed10c0)/0x3ed10c0 2014-06-09 16:22:12.112459 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112463 7f8689f56700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112467 7f8689755700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.112458 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112471 7f868a757700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.112469 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4018.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.112473 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.112475 7f8692f68700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112479 7f868d75d700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112472 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112485 7f868e75f700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.112486 7f86a5e11700 10 _do_op 0x15997bc0 seq 4018 r = 0, finisher 0x318c2700 0 2014-06-09 16:22:12.112489 7f86a5e11700 10 _finish_op 0x15997bc0 seq 4018 osr(3.1d5 0x3ed09f0)/0x3ed09f0 2014-06-09 16:22:12.112481 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16058e00,0x16058d00,0x16058c00,0x16058b00,0x16058a00,0x1804d000 2014-06-09 16:22:12.112497 7f868ef60700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.112498 7f868b759700 10 op_submit_start 4067 2014-06-09 16:22:12.112500 7f868b759700 5 queue_transactions (trailing journal) 4067 2014-06-09 16:22:12.112502 7f868b759700 5 queue_op 0xc88b5c0 seq 4067 osr(2.26d 0x2ed0de0) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.112503 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112509 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c2700] 2014-06-09 16:22:12.112506 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.1 has 4 missing 2014-06-09 16:22:12.112515 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c2700] 2014-06-09 16:22:12.112517 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.112518 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.112521 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a6c0 (1 active) 2014-06-09 16:22:12.112524 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781440 (2 active) 2014-06-09 16:22:12.112516 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.112528 7f86a5e11700 5 _do_op 0x16fd4e40 seq 4019 osr(1.1ef 0x384cdb0)/0x384cdb0 start 2014-06-09 16:22:12.112529 7f868b759700 10 op_submit_finish 4067 2014-06-09 16:22:12.112531 7f86a5e11700 10 _do_transaction on 0xc2da480 2014-06-09 16:22:12.112537 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.112536 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:862 2014-06-09 16:22:12.112533 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.786( v 213'522 (0'0,213'522] local-les=243 n=105 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.112544 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112555 7f868c75b700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112552 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.7d4( v 214'543 (0'0,214'543] local-les=243 n=110 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112562 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112563 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.112567 7f868b759700 5 queue_transactions existing osr(3.786 0x3ed1d90)/0x3ed1d90 2014-06-09 16:22:12.112569 7f868b759700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112573 7f8691765700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112566 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112574 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.112577 7f8690f64700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112581 7f868df5e700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112585 7f8691f66700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112579 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112589 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112589 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4019.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.112592 7f8689f56700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112593 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.112597 7f8689755700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.112588 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x2e074700,0x2e074600,0x2e074500,0x2e074400,0x2e074300,0x2e074200,0x2e074100,0x2e074000,0x2e075000,0x2e074f00,0x2e074e00,0x2e074d00,0x2e074c00,0x2e074b00,0x2e074a00,0x2f096c00 2014-06-09 16:22:12.112600 7f868a757700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.112604 7f8692f68700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112604 7f86a5e11700 10 _do_op 0x16fd4e40 seq 4019 r = 0, finisher 0xcd0ee20 0 2014-06-09 16:22:12.112607 7f868d75d700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112608 7f86a5e11700 10 _finish_op 0x16fd4e40 seq 4019 osr(1.1ef 0x384cdb0)/0x384cdb0 2014-06-09 16:22:12.112621 7f868ff62700 10 op_submit_start 4068 2014-06-09 16:22:12.112623 7f868ff62700 5 queue_transactions (trailing journal) 4068 2014-06-09 16:22:12.112625 7f868ff62700 5 queue_op 0x30a20c0 seq 4068 osr(0.3a9 0x2bd7310) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.112629 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112634 7f868c75b700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112640 7f868b759700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112635 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112644 7f8691765700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112646 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd0ee20] 2014-06-09 16:22:12.112649 7f8690f64700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112649 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd0ee20] 2014-06-09 16:22:12.112651 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.112652 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.112653 7f868df5e700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112646 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112655 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781440 (1 active) 2014-06-09 16:22:12.112657 7f8691f66700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112658 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a480 (2 active) 2014-06-09 16:22:12.112661 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112655 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112663 7f868ff62700 10 op_submit_finish 4068 2014-06-09 16:22:12.112664 7f8689f56700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112668 7f8689755700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.112671 7f868a757700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.112675 7f8692f68700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112668 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:843 2014-06-09 16:22:12.112666 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.530( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.112679 7f868d75d700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112678 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112688 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112698 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112702 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.112707 7f868ff62700 5 queue_transactions existing osr(0.530 0x2bd78f0)/0x2bd78f0 2014-06-09 16:22:12.112709 7f868ff62700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112702 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112714 7f868ef60700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.112718 7f86a5e11700 5 _do_op 0x30061320 seq 4020 osr(3.76f 0x3ed1e20)/0x3ed1e20 start 2014-06-09 16:22:12.112721 7f86a5e11700 10 _do_transaction on 0xbdeb5c0 2014-06-09 16:22:12.112716 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112726 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.112725 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.112735 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112744 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112753 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112762 7f868e75f700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.112764 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.112766 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1120 2014-06-09 16:22:12.112778 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4020.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.112782 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.112776 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112785 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.6( v 213'682 (0'0,213'682] local-les=243 n=136 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112795 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112800 7f86a5e11700 10 _do_op 0x30061320 seq 4020 r = 0, finisher 0x30508c20 0 2014-06-09 16:22:12.112804 7f86a5e11700 10 _finish_op 0x30061320 seq 4020 osr(3.76f 0x3ed1e20)/0x3ed1e20 2014-06-09 16:22:12.112800 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112817 7f868c75b700 10 op_submit_start 4069 2014-06-09 16:22:12.112811 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112818 7f868b759700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112818 7f868c75b700 5 queue_transactions (trailing journal) 4069 2014-06-09 16:22:12.112820 7f868c75b700 5 queue_op 0xd3262a0 seq 4069 osr(3.1f0 0x3ed0990) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.112823 7f8691765700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112819 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.112827 7f8690f64700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112831 7f868df5e700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30508c20] 2014-06-09 16:22:12.112828 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112835 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30508c20] 2014-06-09 16:22:12.112836 7f8691f66700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112837 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.112838 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.112840 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112836 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112841 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a480 (1 active) 2014-06-09 16:22:12.112844 7f8689f56700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112844 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c06c0 (2 active) 2014-06-09 16:22:12.112847 7f8689755700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.112849 7f868c75b700 10 op_submit_finish 4069 2014-06-09 16:22:12.112843 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112851 7f868a757700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.112855 7f8692f68700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112858 7f868d75d700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.112856 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:145 2014-06-09 16:22:12.112853 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 RequestRecovery 2014-06-09 16:22:12.112865 7f868af58700 20 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112874 7f868af58700 20 osd.26 pg_epoch: 243 pg[0.198( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112870 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] exit Started/ReplicaActive/RepNotRecovering 0.469842 3 0.003177 2014-06-09 16:22:12.112883 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112886 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] enter Started/ReplicaActive/RepWaitRecoveryReserved 2014-06-09 16:22:12.112887 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112907 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.112901 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112907 7f8686f50700 10 finisher(0x2c5e890) finisher_thread doing [0x1b0ca9f0] 2014-06-09 16:22:12.112910 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x29f5b600,0x29f5b500,0x29f5b400,0x29f5b300 2014-06-09 16:22:12.112919 7f8686f50700 10 finisher(0x2c5e890) finisher_thread done with [0x1b0ca9f0] 2014-06-09 16:22:12.112921 7f8686f50700 10 finisher(0x2c5e890) finisher_thread empty 2014-06-09 16:22:12.112922 7f8686f50700 10 finisher(0x2c5e890) finisher_thread sleeping 2014-06-09 16:22:12.112912 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.112929 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.112931 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112938 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.112941 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112949 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.112951 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112959 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.112964 7f868af58700 15 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:964 2014-06-09 16:22:12.112969 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.112975 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112985 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.66f( v 214'585 (0'0,214'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.112985 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:933 2014-06-09 16:22:12.112995 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.112996 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.112999 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.113006 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.ed( v 214'568 (0'0,214'568] local-les=243 n=105 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.113017 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.113011 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.113019 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.113021 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.113027 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.113034 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.113036 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.113042 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.113044 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.113051 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.113057 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.113059 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.113066 7f868af58700 20 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.113068 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.113074 7f868af58700 20 osd.26 pg_epoch: 243 pg[1.293( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.113083 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.113082 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.113091 7f868c75b700 20 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.113099 7f868c75b700 20 osd.26 pg_epoch: 243 pg[2.5e2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.113108 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.113108 7f868af58700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xc711a90 (19 active) 2014-06-09 16:22:12.113117 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.113113 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.113125 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.113133 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.113122 7f868af58700 10 osd.26 243 dequeue_op 0x32826100 prio 63 cost 4195304 latency 0.321091 MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 pg pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] 2014-06-09 16:22:12.113141 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.113150 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.113153 7f868af58700 5 -- op tracker -- , seq: 3239, time: 2014-06-09 16:22:12.113145, event: reached_pg, op: MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 4194304, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.113158 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.113163 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] handle_message: 0x32826100 2014-06-09 16:22:12.113171 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:150 2014-06-09 16:22:12.113180 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.113182 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] handle_push ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {})ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true) 2014-06-09 16:22:12.113188 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.28a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.113198 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.113205 7f868af58700 10 collection_exists 2014-06-09 16:22:12.113202 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.113214 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.113222 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.113230 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.113238 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.113246 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.113252 7f868af58700 20 lookup_strip_header failed to get strip_header cid COLLECTIONS oid 8e1f8b20/3.27_TEMP/head//-1 2014-06-09 16:22:12.113259 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:12.113269 7f868c75b700 20 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.113267 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] on_local_recover: 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 2014-06-09 16:22:12.113277 7f868c75b700 20 osd.26 pg_epoch: 243 pg[1.4d3( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.113287 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.113284 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 lc 215'667 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active m=1] got missing 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 v 215'668 2014-06-09 16:22:12.113292 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.113297 7f868af58700 10 osd.26 pg_epoch: 243 pg[3.27( v 215'668 (0'0,215'668] local-les=243 n=123 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=215'668 lcod 0'0 active] last_complete now 215'668 log.complete_to at end 2014-06-09 16:22:12.113305 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.113315 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x2c06b000 2014-06-09 16:22:12.113326 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.113331 7f868af58700 5 queue_transactions existing osr(3.27 0x3ed0340)/0x3ed0340 2014-06-09 16:22:12.113334 7f868af58700 2 waiting 51 > 50 ops || 8127022 > 104857600 2014-06-09 16:22:12.113329 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.113339 7f868f761700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113345 7f868cf5c700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.113339 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.113350 7f868bf5a700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113355 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113348 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.113359 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd76fc0,0xcc82620,0x1b0c1560,0x318c2000,0x307f0020,0x19941f80] 2014-06-09 16:22:12.113362 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd76fc0,0xcc82620,0x1b0c1560,0x318c2000,0x307f0020,0x19941f80] 2014-06-09 16:22:12.113365 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.113367 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.113362 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:980 2014-06-09 16:22:12.113373 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.113382 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.113393 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.113397 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.113408 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.113416 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.113424 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.113433 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.113441 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.113453 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:12.113463 7f868c75b700 20 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.113470 7f868c75b700 20 osd.26 pg_epoch: 243 pg[1.299( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.113480 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.113484 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.113496 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.974326 7 0.000972 2014-06-09 16:22:12.113508 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.113519 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.113530 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.113542 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.113551 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.113559 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.113572 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.113583 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.113592 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.113602 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.113612 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.113626 7f868c75b700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.113643 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.113677 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.113681 7f868c75b700 5 queue_transactions existing osr(0.29d 0x2bd6c40)/0x2bd6c40 2014-06-09 16:22:12.113683 7f868c75b700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.113688 7f868ef60700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.113692 7f868ff62700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113696 7f86a5e11700 5 _do_op 0x15996d20 seq 4021 osr(2.48d 0x2ed1470)/0x2ed1470 start 2014-06-09 16:22:12.113699 7f86a5e11700 10 _do_transaction on 0xc2c7e00 2014-06-09 16:22:12.113729 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.113769 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.113777 7f868e75f700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.113784 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4021.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.113788 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.113804 7f86a5e11700 10 _do_op 0x15996d20 seq 4021 r = 0, finisher 0x3062a4e0 0 2014-06-09 16:22:12.113810 7f86a5e11700 10 _finish_op 0x15996d20 seq 4021 osr(2.48d 0x2ed1470)/0x2ed1470 2014-06-09 16:22:12.113812 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6b520] 2014-06-09 16:22:12.113815 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6b520] 2014-06-09 16:22:12.113817 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.113818 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.113820 7f868b759700 10 op_submit_start 4070 2014-06-09 16:22:12.113822 7f868b759700 5 queue_transactions (trailing journal) 4070 2014-06-09 16:22:12.113823 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062a4e0] 2014-06-09 16:22:12.113823 7f868b759700 5 queue_op 0xc88b9e0 seq 4070 osr(3.786 0x3ed1d90) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.113827 7f8691765700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113828 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062a4e0] 2014-06-09 16:22:12.113830 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.113831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.113831 7f8690f64700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113834 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c06c0 (1 active) 2014-06-09 16:22:12.113836 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652480 (2 active) 2014-06-09 16:22:12.113838 7f868df5e700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113841 7f86a5e11700 5 _do_op 0x760ec60 seq 4022 osr(1.289 0x384ceb0)/0x384ceb0 start 2014-06-09 16:22:12.113842 7f868b759700 10 op_submit_finish 4070 2014-06-09 16:22:12.113844 7f86a5e11700 10 _do_transaction on 0xc262e40 2014-06-09 16:22:12.113843 7f8691f66700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113848 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113849 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.113851 7f8689f56700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113855 7f8689755700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.113846 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.6c0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.6c0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.113858 7f868a757700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.113862 7f8692f68700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113866 7f868d75d700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113887 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.113893 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.113898 7f868b759700 5 queue_transactions existing osr(0.6c0 0x2bd7f50)/0x2bd7f50 2014-06-09 16:22:12.113900 7f868b759700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113901 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4022.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.113906 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.113918 7f86a5e11700 10 _do_op 0x760ec60 seq 4022 r = 0, finisher 0x31791dc0 0 2014-06-09 16:22:12.113924 7f86a5e11700 10 _finish_op 0x760ec60 seq 4022 osr(1.289 0x384ceb0)/0x384ceb0 2014-06-09 16:22:12.113933 7f8691765700 10 op_submit_start 4071 2014-06-09 16:22:12.113935 7f8691765700 5 queue_transactions (trailing journal) 4071 2014-06-09 16:22:12.113936 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31791dc0] 2014-06-09 16:22:12.113937 7f8691765700 5 queue_op 0xcd9d0e0 seq 4071 osr(2.2d9 0x2ed0c30) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.113940 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31791dc0] 2014-06-09 16:22:12.113941 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.113943 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.113946 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652480 (1 active) 2014-06-09 16:22:12.113948 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd440 (2 active) 2014-06-09 16:22:12.113953 7f86a5e11700 5 _do_op 0x760ec00 seq 4023 osr(2.34d 0x2ed0e70)/0x2ed0e70 start 2014-06-09 16:22:12.113955 7f8691765700 10 op_submit_finish 4071 2014-06-09 16:22:12.113956 7f86a5e11700 10 _do_transaction on 0xc239380 2014-06-09 16:22:12.113952 7f8683620700 5 -- op tracker -- , seq: 3657, time: 2014-06-09 16:22:12.113919, event: header_read, op: osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.113961 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.113962 7f8683620700 5 -- op tracker -- , seq: 3657, time: 2014-06-09 16:22:12.113920, event: throttled, op: osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.113958 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.76f( v 213'666 (0'0,213'666] local-les=243 n=127 ec=118 les/c 243/243 242/242/222) 2014-06-09 16:22:12.113969 7f8683620700 5 -- op tracker -- , seq: 3657, time: 2014-06-09 16:22:12.113930, event: all_read, op: osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.113976 7f8683620700 5 -- op tracker -- , seq: 3657, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.113980 7f8683620700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:12.113985 7f8683620700 20 osd.26 243 should_share_map osd.6 172.16.96.201:6849/52045 243 2014-06-09 16:22:12.113989 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.113993 7f8691765700 5 queue_transactions existing osr(3.76f 0x3ed1e20)/0x3ed1e20 2014-06-09 16:22:12.113995 7f8691765700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.113992 7f8683620700 15 osd.26 243 enqueue_op 0x2976cf00 prio 196 cost 0 latency 0.000073 osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:12.113998 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.114013 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4023.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.114017 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.114016 7f868f761700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114023 7f868cf5c700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114027 7f868bf5a700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114030 7f86a5e11700 10 _do_op 0x760ec00 seq 4023 r = 0, finisher 0xbd7d4c0 0 2014-06-09 16:22:12.114031 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114037 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x170010c0,0xbd6c000] 2014-06-09 16:22:12.114037 7f868c75b700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114039 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x170010c0,0xbd6c000] 2014-06-09 16:22:12.114041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.114042 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.114043 7f868ef60700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114059 7f868ff62700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114034 7f86a5e11700 10 _finish_op 0x760ec00 seq 4023 osr(2.34d 0x2ed0e70)/0x2ed0e70 2014-06-09 16:22:12.114073 7f8691765700 10 op_submit_start 4072 2014-06-09 16:22:12.114075 7f8691765700 5 queue_transactions (trailing journal) 4072 2014-06-09 16:22:12.114075 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7d4c0] 2014-06-09 16:22:12.114077 7f8691765700 5 queue_op 0xcd9d200 seq 4072 osr(3.76f 0x3ed1e20) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.114079 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7d4c0] 2014-06-09 16:22:12.114080 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.114082 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.114082 7f8690f64700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114085 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd440 (1 active) 2014-06-09 16:22:12.114087 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0360 (2 active) 2014-06-09 16:22:12.114087 7f868df5e700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114091 7f86a5e11700 5 _do_op 0xbf7f1a0 seq 4024 osr(3.579 0x3ed18d0)/0x3ed18d0 start 2014-06-09 16:22:12.114093 7f8691765700 10 op_submit_finish 4072 2014-06-09 16:22:12.114093 7f8691f66700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114094 7f86a5e11700 10 _do_transaction on 0xc294840 2014-06-09 16:22:12.114097 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114100 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.114101 7f8689f56700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114104 7f8689755700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114108 7f868a757700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114097 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.e8( v 213'654 (0'0,213'654] local-les=243 n=125 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.114112 7f8692f68700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114116 7f868d75d700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114131 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.114136 7f8691765700 5 queue_transactions existing osr(3.e8 0x3ed0510)/0x3ed0510 2014-06-09 16:22:12.114139 7f8691765700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114138 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.114143 7f868b759700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114148 7f868af58700 2 waiting 51 > 50 ops || 8127022 > 104857600 2014-06-09 16:22:12.114153 7f868f761700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114153 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4024.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.114157 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.114157 7f868cf5c700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114162 7f868bf5a700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114166 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114172 7f868c75b700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114179 7f868ef60700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114183 7f868ff62700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114188 7f86a5e11700 10 _do_op 0xbf7f1a0 seq 4024 r = 0, finisher 0xbebff80 0 2014-06-09 16:22:12.114193 7f86a5e11700 10 _finish_op 0xbf7f1a0 seq 4024 osr(3.579 0x3ed18d0)/0x3ed18d0 2014-06-09 16:22:12.114195 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062bc40] 2014-06-09 16:22:12.114197 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062bc40] 2014-06-09 16:22:12.114199 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.114200 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.114203 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbebff80] 2014-06-09 16:22:12.114206 7f8690f64700 10 op_submit_start 4073 2014-06-09 16:22:12.114207 7f8690f64700 5 queue_transactions (trailing journal) 4073 2014-06-09 16:22:12.114206 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbebff80] 2014-06-09 16:22:12.114207 7f868df5e700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114208 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.114209 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.114208 7f8690f64700 5 queue_op 0x30a3b60 seq 4073 osr(1.11d 0x384c8b0) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.114212 7f8691f66700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114212 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0360 (1 active) 2014-06-09 16:22:12.114214 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e240 (2 active) 2014-06-09 16:22:12.114217 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114218 7f86a5e11700 5 _do_op 0xc88bf20 seq 4025 osr(1.375 0x384d230)/0x384d230 start 2014-06-09 16:22:12.114220 7f86a5e11700 10 _do_transaction on 0xc28b200 2014-06-09 16:22:12.114222 7f8690f64700 10 op_submit_finish 4073 2014-06-09 16:22:12.114222 7f8689f56700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114225 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.114226 7f8689755700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114230 7f868a757700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114233 7f868e75f700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114226 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.71e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.71e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.114239 7f868d75d700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114222 7f8682de8700 5 -- op tracker -- , seq: 3658, time: 2014-06-09 16:22:12.114193, event: header_read, op: osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.114253 7f8682de8700 5 -- op tracker -- , seq: 3658, time: 2014-06-09 16:22:12.114194, event: throttled, op: osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.114260 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.114261 7f8682de8700 5 -- op tracker -- , seq: 3658, time: 2014-06-09 16:22:12.114202, event: all_read, op: osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.114264 7f8690f64700 5 queue_transactions existing osr(1.71e 0x2ed00b0)/0x2ed00b0 2014-06-09 16:22:12.114264 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.114267 7f8690f64700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114268 7f8682de8700 5 -- op tracker -- , seq: 3658, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.114271 7f8691765700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114275 7f868b759700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114272 7f8682de8700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:12.114279 7f868af58700 2 waiting 51 > 50 ops || 8127022 > 104857600 2014-06-09 16:22:12.114278 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4025.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.114278 7f8682de8700 20 osd.26 243 should_share_map osd.9 172.16.96.201:6865/53518 243 2014-06-09 16:22:12.114283 7f868f761700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114283 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.114287 7f868cf5c700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114284 7f8682de8700 15 osd.26 243 enqueue_op 0x30c6cc00 prio 196 cost 0 latency 0.000091 osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:12.114291 7f868bf5a700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114295 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114300 7f868c75b700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114307 7f868ef60700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114311 7f868ff62700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114318 7f8692f68700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114323 7f86a5e11700 10 _do_op 0xc88bf20 seq 4025 r = 0, finisher 0x33a78e0 0 2014-06-09 16:22:12.114329 7f86a5e11700 10 _finish_op 0xc88bf20 seq 4025 osr(1.375 0x384d230)/0x384d230 2014-06-09 16:22:12.114330 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x306cfdc0] 2014-06-09 16:22:12.114333 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x306cfdc0] 2014-06-09 16:22:12.114335 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.114336 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.114337 7f868df5e700 10 op_submit_start 4074 2014-06-09 16:22:12.114339 7f868df5e700 5 queue_transactions (trailing journal) 4074 2014-06-09 16:22:12.114340 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e240 (1 active) 2014-06-09 16:22:12.114341 7f868df5e700 5 queue_op 0xbf7ecc0 seq 4074 osr(0.61f 0x2bd7d30) 828 bytes (queue has 50 ops and 3930802 bytes) 2014-06-09 16:22:12.114343 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780fc0 (2 active) 2014-06-09 16:22:12.114342 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a78e0] 2014-06-09 16:22:12.114344 7f8691f66700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114347 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a78e0] 2014-06-09 16:22:12.114347 7f86a5e11700 5 _do_op 0x2e20420 seq 4026 osr(3.173 0x3ed06f0)/0x3ed06f0 start 2014-06-09 16:22:12.114349 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.114349 7f8690763700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114350 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.114350 7f86a5e11700 10 _do_transaction on 0xc219e00 2014-06-09 16:22:12.114355 7f868df5e700 10 op_submit_finish 4074 2014-06-09 16:22:12.114354 7f8689f56700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114355 7f86a5e11700 15 _omap_setkeys meta/1fe2a34a/pglog_3.173/0//-1 2014-06-09 16:22:12.114358 7f8689755700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114362 7f868a757700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114358 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.114372 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.114380 7f868e75f700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114381 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.114388 7f868d75d700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114390 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.114399 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe2a34a/pglog_3.173/0//-1 2014-06-09 16:22:12.114399 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.114407 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.114415 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.114421 7f868df5e700 15 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.114431 7f868df5e700 20 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.114439 7f868df5e700 20 osd.26 pg_epoch: 243 pg[1.1b1( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.114448 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.114451 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.114453 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.58a( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.58a( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.114466 7f86a5e11700 15 _omap_rmkeys meta/1fe2a34a/pglog_3.173/0//-1 2014-06-09 16:22:12.114472 7f86a5e11700 10 _omap_rmkeys meta/1fe2a34a/pglog_3.173/0//-1 r = 0 2014-06-09 16:22:12.114478 7f86a5e11700 15 _omap_setkeys meta/1fe2a34a/pglog_3.173/0//-1 2014-06-09 16:22:12.114488 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.114491 7f86a5e11700 15 _touch 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.114493 7f868df5e700 5 queue_transactions existing osr(0.58a 0x2bd7770)/0x2bd7770 2014-06-09 16:22:12.114495 7f868df5e700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114500 7f8690f64700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114505 7f8691765700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114509 7f868b759700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114514 7f868af58700 2 waiting 51 > 50 ops || 8127022 > 104857600 2014-06-09 16:22:12.114518 7f868f761700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114523 7f868cf5c700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114527 7f868bf5a700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114531 7f8692767700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114537 7f868c75b700 2 waiting 51 > 50 ops || 3931663 > 104857600 2014-06-09 16:22:12.114544 7f868ef60700 2 waiting 51 > 50 ops || 3931628 > 104857600 2014-06-09 16:22:12.114548 7f868ff62700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114532 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.173_head oid c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.114554 7f8692f68700 2 waiting 51 > 50 ops || 3931630 > 104857600 2014-06-09 16:22:12.114581 7f86a5e11700 20 write_state seq is 30165 2014-06-09 16:22:12.114589 7f86a5e11700 20 set_header setting 30164 cid 3.173_head oid c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 parent seq 0 2014-06-09 16:22:12.114603 7f86a5e11700 10 _touch 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 = 0 2014-06-09 16:22:12.114610 7f86a5e11700 15 _write 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 0~1388544 2014-06-09 16:22:12.114644 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.115630 7f86a5610700 10 _finish_op 0x3248bf80 seq 4009 osr(3.206 0x3ed0cf0)/0x3ed0cf0 2014-06-09 16:22:12.115642 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114900 (1 active) 2014-06-09 16:22:12.115643 7f8691f66700 10 op_submit_start 4075 2014-06-09 16:22:12.115644 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f83e0] 2014-06-09 16:22:12.116488 7f8691f66700 5 queue_transactions (trailing journal) 4075 2014-06-09 16:22:12.116492 7f8691f66700 5 queue_op 0x2ea53a40 seq 4075 osr(0.456 0x2bd7530) 828 bytes (queue has 50 ops and 3777680 bytes) 2014-06-09 16:22:12.115646 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d560 (2 active) 2014-06-09 16:22:12.115648 7f8690763700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116495 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] op_applied: 25 2014-06-09 16:22:12.116510 7f8691f66700 10 op_submit_finish 4075 2014-06-09 16:22:12.116508 7f8689f56700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116514 7f8689755700 2 waiting 51 > 50 ops || 3778506 > 104857600 2014-06-09 16:22:12.116519 7f868a757700 2 waiting 51 > 50 ops || 3778541 > 104857600 2014-06-09 16:22:12.116519 7f86a4e0f700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.116511, event: op_applied, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.116514 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.517( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.517( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.116525 7f868e75f700 2 waiting 51 > 50 ops || 3778541 > 104857600 2014-06-09 16:22:12.116532 7f868d75d700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116527 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] op_applied on primary on version 243'597 2014-06-09 16:22:12.116539 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] repop_all_applied: repop tid 25 all applied 2014-06-09 16:22:12.116553 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.116559 7f8691f66700 5 queue_transactions existing osr(1.517 0x384db70)/0x384db70 2014-06-09 16:22:12.116562 7f8691f66700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116549 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] eval_repop repgather(0xc143200 243'597 rep_tid=25 committed?=1 applied?=1 lock=0 op=osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.116566 7f868df5e700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116571 7f8690f64700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116576 7f8691765700 2 waiting 51 > 50 ops || 3778506 > 104857600 2014-06-09 16:22:12.116566 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] log_op_stats osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4 inb 151936 outb 0 rlat 0.000000 lat 1.442910 2014-06-09 16:22:12.116581 7f868b759700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116587 7f868af58700 2 waiting 51 > 50 ops || 7973900 > 104857600 2014-06-09 16:22:12.116591 7f868f761700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116584 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] publish_stats_to_osd 243:976 2014-06-09 16:22:12.116596 7f868cf5c700 2 waiting 51 > 50 ops || 3778541 > 104857600 2014-06-09 16:22:12.116600 7f868bf5a700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116595 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.116605 7f8692767700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116610 7f86a5610700 5 _do_op 0xcd902a0 seq 4027 osr(3.381 0x3ed0fe0)/0x3ed0fe0 start 2014-06-09 16:22:12.116611 7f868c75b700 2 waiting 51 > 50 ops || 3778541 > 104857600 2014-06-09 16:22:12.116614 7f86a5610700 10 _do_transaction on 0x2ed43c0 2014-06-09 16:22:12.116618 7f868ef60700 2 waiting 51 > 50 ops || 3778506 > 104857600 2014-06-09 16:22:12.116606 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] removing repgather(0xc143200 243'597 rep_tid=25 committed?=1 applied?=1 op=osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.116620 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.116624 7f868ff62700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116631 7f8692f68700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.116621 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] q front is repgather(0xc143200 243'597 rep_tid=25 committed?=1 applied?=1 op=osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.116635 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] remove_repop repgather(0xc143200 243'597 rep_tid=25 committed?=1 applied?=1 op=osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.116650 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] requeue_ops 2014-06-09 16:22:12.116679 7f86a4e0f700 5 -- op tracker -- , seq: 575, time: 2014-06-09 16:22:12.116673, event: done, op: osd_op(client.4573.0:34037 rbd_data.117d6b8b4567.00000000000016b8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~151552] 3.2c355206 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.116713 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f83e0] 2014-06-09 16:22:12.116715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.116716 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.120894 7f86a5e11700 10 _generic_write 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 0~1388544 = 0 2014-06-09 16:22:12.120974 7f86a5e11700 15 _setattrs 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.120989 7f86a5e11700 10 _setattrs 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 = 0 2014-06-09 16:22:12.120996 7f86a5e11700 15 _setattrs 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 2014-06-09 16:22:12.121004 7f86a5e11700 10 _setattrs 3.173_head/c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 = 0 2014-06-09 16:22:12.121220 7f86a5e11700 10 cid: oid: c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 not skipping op, *spos 4026.0.0 2014-06-09 16:22:12.121225 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.121267 7f86a5e11700 20 set_header setting 30164 cid 3.173_head oid c0524973/rbd_data.117f6b8b4567.0000000000001573/head//3 parent seq 0 2014-06-09 16:22:12.121393 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4026.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.121400 7f86a5e11700 10 cid: oid: 1fe2a34a/pglog_3.173/0//-1 skipping op, spos 4026.0.0 <= header.spos 149956.0.0 2014-06-09 16:22:12.121403 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.127834 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.127863 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4027.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.127871 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.127891 7f86a5610700 10 _do_op 0xcd902a0 seq 4027 r = 0, finisher 0xbd5c4a0 0 2014-06-09 16:22:12.127908 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32075e80] 2014-06-09 16:22:12.127917 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32075e80] 2014-06-09 16:22:12.127923 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.127924 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.127928 7f86a5610700 10 _finish_op 0xcd902a0 seq 4027 osr(3.381 0x3ed0fe0)/0x3ed0fe0 2014-06-09 16:22:12.127945 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d560 (1 active) 2014-06-09 16:22:12.127947 7f8689f56700 10 op_submit_start 4076 2014-06-09 16:22:12.127949 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89eb40 (2 active) 2014-06-09 16:22:12.127950 7f8689f56700 5 queue_transactions (trailing journal) 4076 2014-06-09 16:22:12.127951 7f86a5610700 5 _do_op 0x3248b980 seq 4028 osr(2.48f 0x2ed1450)/0x2ed1450 start 2014-06-09 16:22:12.127951 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5c4a0] 2014-06-09 16:22:12.127951 7f8689f56700 5 queue_op 0x32833020 seq 4076 osr(3.58f 0x3ed1850) 828 bytes (queue has 50 ops and 3777680 bytes) 2014-06-09 16:22:12.127954 7f86a5610700 10 _do_transaction on 0x3858240 2014-06-09 16:22:12.127956 7f8689f56700 10 op_submit_finish 4076 2014-06-09 16:22:12.127958 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5c4a0] 2014-06-09 16:22:12.127960 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.127961 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.127960 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.127961 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.3c3( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.3c3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.127998 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.128008 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.128011 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4028.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.128015 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.128016 7f8689f56700 5 queue_transactions existing osr(1.3c3 0x384d530)/0x384d530 2014-06-09 16:22:12.128019 7f8689f56700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128025 7f8689755700 2 waiting 51 > 50 ops || 3778506 > 104857600 2014-06-09 16:22:12.128031 7f868a757700 2 waiting 51 > 50 ops || 3778541 > 104857600 2014-06-09 16:22:12.128033 7f86a5610700 10 _do_op 0x3248b980 seq 4028 r = 0, finisher 0xbd5c6a0 0 2014-06-09 16:22:12.128043 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6c9e0] 2014-06-09 16:22:12.128045 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6c9e0] 2014-06-09 16:22:12.128049 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.128050 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.128054 7f86a5610700 10 _finish_op 0x3248b980 seq 4028 osr(2.48f 0x2ed1450)/0x2ed1450 2014-06-09 16:22:12.128052 7f868e75f700 2 waiting 51 > 50 ops || 3778541 > 104857600 2014-06-09 16:22:12.128062 7f868d75d700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128070 7f8691f66700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128078 7f868df5e700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128078 7f86a5e11700 10 _do_op 0x2e20420 seq 4026 r = 0, finisher 0x189aa5a0 0x6c98240 2014-06-09 16:22:12.128084 7f8690f64700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128090 7f8691765700 2 waiting 51 > 50 ops || 3778506 > 104857600 2014-06-09 16:22:12.128096 7f868b759700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128103 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062be60] 2014-06-09 16:22:12.128106 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=213'624 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 26 2014-06-09 16:22:12.128133 7f86a460e700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.128122, event: op_commit, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.128143 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=213'624 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 26 all committed 2014-06-09 16:22:12.128155 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] eval_repop repgather(0xc23e300 243'625 rep_tid=26 committed?=1 applied?=0 lock=0 op=osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.128173 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] log_op_stats osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1388928 outb 0 rlat 0.000000 lat 1.298693 2014-06-09 16:22:12.128194 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] publish_stats_to_osd 243:1053 2014-06-09 16:22:12.128206 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] sending commit on repgather(0xc23e300 243'625 rep_tid=26 committed?=1 applied?=0 lock=0 op=osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4) 0x9a08780 2014-06-09 16:22:12.128243 7f86a460e700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.128237, event: commit_sent, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.128251 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062be60] 2014-06-09 16:22:12.128253 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.128255 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.128295 7f868af58700 2 waiting 51 > 50 ops || 7973900 > 104857600 2014-06-09 16:22:12.128302 7f868f761700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128309 7f868cf5c700 2 waiting 51 > 50 ops || 3778541 > 104857600 2014-06-09 16:22:12.128318 7f868bf5a700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128325 7f8692767700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128331 7f868c75b700 2 waiting 51 > 50 ops || 3778541 > 104857600 2014-06-09 16:22:12.128338 7f868ef60700 2 waiting 51 > 50 ops || 3778506 > 104857600 2014-06-09 16:22:12.128354 7f868ff62700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128363 7f8692f68700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128369 7f8690763700 2 waiting 51 > 50 ops || 3778508 > 104857600 2014-06-09 16:22:12.128379 7f8689f56700 10 op_submit_start 4077 2014-06-09 16:22:12.128380 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89eb40 (1 active) 2014-06-09 16:22:12.128381 7f8689f56700 5 queue_transactions (trailing journal) 4077 2014-06-09 16:22:12.128383 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6240 (2 active) 2014-06-09 16:22:12.128383 7f8689f56700 5 queue_op 0x324989c0 seq 4077 osr(1.3c3 0x384d530) 828 bytes (queue has 50 ops and 3777680 bytes) 2014-06-09 16:22:12.128383 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5c6a0] 2014-06-09 16:22:12.128385 7f8689f56700 10 op_submit_finish 4077 2014-06-09 16:22:12.128387 7f86a5610700 5 _do_op 0x30a3140 seq 4029 osr(0.473 0x2bd74d0)/0x2bd74d0 start 2014-06-09 16:22:12.128389 7f86a5610700 10 _do_transaction on 0xbe89c80 2014-06-09 16:22:12.128389 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5c6a0] 2014-06-09 16:22:12.128391 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.128389 7f86a5e11700 10 _finish_op 0x2e20420 seq 4026 osr(3.173 0x3ed06f0)/0x3ed06f0 2014-06-09 16:22:12.128393 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.128395 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.128388 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.128399 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780fc0 (1 active) 2014-06-09 16:22:12.128401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aa5a0] 2014-06-09 16:22:12.128403 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301440 (2 active) 2014-06-09 16:22:12.128407 7f86a5e11700 5 _do_op 0x760f0e0 seq 4030 osr(1.3f0 0x384d400)/0x384d400 start 2014-06-09 16:22:12.128410 7f86a5e11700 10 _do_transaction on 0xc262840 2014-06-09 16:22:12.128404 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.128416 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.128404 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] op_applied: 26 2014-06-09 16:22:12.128415 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16bd6400,0x16bd6300,0x16bd6200,0x16bd6100,0x16bd6000,0x10c54000,0x10c55000 2014-06-09 16:22:12.128428 7f86a4e0f700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.128421, event: op_applied, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.128434 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.128436 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] op_applied on primary on version 243'625 2014-06-09 16:22:12.128448 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4029.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.128452 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.128445 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.128447 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] repop_all_applied: repop tid 26 all applied 2014-06-09 16:22:12.128456 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.128464 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.128457 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] eval_repop repgather(0xc23e300 243'625 rep_tid=26 committed?=1 applied?=1 lock=0 op=osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.128478 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:992 2014-06-09 16:22:12.128473 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] log_op_stats osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4 inb 1388928 outb 0 rlat 0.000000 lat 1.298994 2014-06-09 16:22:12.128489 7f86a5610700 10 _do_op 0x30a3140 seq 4029 r = 0, finisher 0xbd71f80 0 2014-06-09 16:22:12.128489 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.128498 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509940] 2014-06-09 16:22:12.128491 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] publish_stats_to_osd 243:1054 2014-06-09 16:22:12.128500 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509940] 2014-06-09 16:22:12.128502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.128503 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.128498 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.3ae( v 213'576 (0'0,213'576] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.128506 7f86a5610700 10 _finish_op 0x30a3140 seq 4029 osr(0.473 0x2bd74d0)/0x2bd74d0 2014-06-09 16:22:12.128509 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.128509 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6240 (1 active) 2014-06-09 16:22:12.128511 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781b00 (2 active) 2014-06-09 16:22:12.128503 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.128513 7f86a5610700 5 _do_op 0x16fd5860 seq 4031 osr(1.1e8 0x384cdf0)/0x384cdf0 start 2014-06-09 16:22:12.128515 7f86a5610700 10 _do_transaction on 0xbf2e000 2014-06-09 16:22:12.128520 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.128519 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.128514 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.128514 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 213'624 active] removing repgather(0xc23e300 243'625 rep_tid=26 committed?=1 applied?=1 op=osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.128525 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.128533 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.128536 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4030.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.128529 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 213'624 active] q front is repgather(0xc23e300 243'625 rep_tid=26 committed?=1 applied?=1 op=osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.128542 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.128542 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.128550 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.128544 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 213'624 active] remove_repop repgather(0xc23e300 243'625 rep_tid=26 committed?=1 applied?=1 op=osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.128558 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.128559 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.173( v 243'625 (0'0,243'625] local-les=243 n=120 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 213'624 mlcod 213'624 active] requeue_ops 2014-06-09 16:22:12.128566 7f86a5e11700 10 _do_op 0x760f0e0 seq 4030 r = 0, finisher 0x170010a0 0 2014-06-09 16:22:12.128573 7f86a5e11700 10 _finish_op 0x760f0e0 seq 4030 osr(1.3f0 0x384d400)/0x384d400 2014-06-09 16:22:12.128574 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31bff8c0] 2014-06-09 16:22:12.128577 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31bff8c0] 2014-06-09 16:22:12.128570 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.128576 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301440 (1 active) 2014-06-09 16:22:12.128578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.128579 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.128580 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34977a0 (2 active) 2014-06-09 16:22:12.128582 7f86a5e11700 5 _do_op 0x3250e000 seq 4032 osr(0.631 0x2bd7c70)/0x2bd7c70 start 2014-06-09 16:22:12.128579 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.128585 7f86a5e11700 10 _do_transaction on 0x3145c80 2014-06-09 16:22:12.128591 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.128587 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.316( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.128588 7f86a4e0f700 5 -- op tracker -- , seq: 825, time: 2014-06-09 16:22:12.128582, event: done, op: osd_op(client.4575.0:32697 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1388544] 3.c0524973 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.128595 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.128600 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.128608 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.128614 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aa5a0] 2014-06-09 16:22:12.128616 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd71f80,0x170010a0] 2014-06-09 16:22:12.128612 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.128620 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd71f80,0x170010a0] 2014-06-09 16:22:12.128622 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.128623 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.128622 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4031.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.128626 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.128620 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.128628 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.128635 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.128643 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.128650 7f86a5610700 10 _do_op 0x16fd5860 seq 4031 r = 0, finisher 0xc6e8540 0 2014-06-09 16:22:12.128659 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc83120] 2014-06-09 16:22:12.128660 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc83120] 2014-06-09 16:22:12.128655 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:144 2014-06-09 16:22:12.128662 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.128663 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.128666 7f86a5610700 10 _finish_op 0x16fd5860 seq 4031 osr(1.1e8 0x384cdf0)/0x384cdf0 2014-06-09 16:22:12.128663 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.128670 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781b00 (1 active) 2014-06-09 16:22:12.128672 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e360 (2 active) 2014-06-09 16:22:12.128673 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8540] 2014-06-09 16:22:12.128674 7f86a5610700 5 _do_op 0x159960c0 seq 4033 osr(3.44d 0x3ed1180)/0x3ed1180 start 2014-06-09 16:22:12.128671 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.744( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.128677 7f86a5610700 10 _do_transaction on 0xbe88000 2014-06-09 16:22:12.128679 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.128679 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8540] 2014-06-09 16:22:12.128681 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.128682 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.128681 7f86a5610700 15 _omap_setkeys meta/1fe1a41b/pglog_3.44d/0//-1 2014-06-09 16:22:12.128680 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.128684 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.128696 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4032.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.128700 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.128695 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.128703 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.128711 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.128718 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.128725 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.128734 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe1a41b/pglog_3.44d/0//-1 2014-06-09 16:22:12.128737 7f86a5e11700 10 _do_op 0x3250e000 seq 4032 r = 0, finisher 0x31413a80 0 2014-06-09 16:22:12.128737 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.128744 7f86a5e11700 10 _finish_op 0x3250e000 seq 4032 osr(0.631 0x2bd7c70)/0x2bd7c70 2014-06-09 16:22:12.128745 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd60aa0] 2014-06-09 16:22:12.128748 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd60aa0] 2014-06-09 16:22:12.128749 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.128749 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34977a0 (1 active) 2014-06-09 16:22:12.128750 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.128751 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dde60 (2 active) 2014-06-09 16:22:12.128751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31413a80] 2014-06-09 16:22:12.128754 7f86a5e11700 5 _do_op 0xbf7f920 seq 4034 osr(1.555 0x384dab0)/0x384dab0 start 2014-06-09 16:22:12.128757 7f86a5e11700 10 _do_transaction on 0xc252000 2014-06-09 16:22:12.128756 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31413a80] 2014-06-09 16:22:12.128759 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.128760 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.128761 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.128762 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.128795 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.128800 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.128810 7f86a5610700 15 _omap_rmkeys meta/1fe1a41b/pglog_3.44d/0//-1 2014-06-09 16:22:12.128809 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.158( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.128815 7f86a5610700 10 _omap_rmkeys meta/1fe1a41b/pglog_3.44d/0//-1 r = 0 2014-06-09 16:22:12.128818 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.128820 7f86a5610700 15 _omap_setkeys meta/1fe1a41b/pglog_3.44d/0//-1 2014-06-09 16:22:12.128823 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.128832 7f86a5610700 15 _touch 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.128834 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.128841 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.128848 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.128855 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.128862 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.128874 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.44d_head oid b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.128874 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.128883 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.128890 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.6b4( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.128898 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.128903 7f86a5610700 20 write_state seq is 30166 2014-06-09 16:22:12.128903 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.128910 7f86a5610700 20 set_header setting 30165 cid 3.44d_head oid b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 parent seq 0 2014-06-09 16:22:12.128914 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.128923 7f86a5610700 10 _touch 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 = 0 2014-06-09 16:22:12.128921 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.128929 7f86a5610700 15 _write 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 0~1531904 2014-06-09 16:22:12.128929 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.128936 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.128943 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.128954 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:12.128954 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:193 2014-06-09 16:22:12.128962 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.128970 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.1f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.128978 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.128982 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.128993 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.129000 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.129007 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129015 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.129022 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.129033 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:153 2014-06-09 16:22:12.129041 7f8689f56700 20 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.129048 7f8689f56700 20 osd.26 pg_epoch: 243 pg[1.30e( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.129056 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129060 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.129070 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.129077 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.129084 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129092 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.129099 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.129110 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.129118 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.129126 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.138( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.129134 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129137 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.129148 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.129155 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.129162 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129169 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.129177 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.129188 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:148 2014-06-09 16:22:12.129196 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.129203 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.292( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.129211 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129215 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.129227 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.129235 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.129244 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129252 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.129260 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.129272 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1062 2014-06-09 16:22:12.129281 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.129290 7f8689f56700 20 osd.26 pg_epoch: 243 pg[3.491( v 214'640 (0'0,214'640] local-les=243 n=123 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.129299 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129302 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.129314 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.129321 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.129328 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129335 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.129343 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.129354 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.129362 7f8689f56700 20 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.129370 7f8689f56700 20 osd.26 pg_epoch: 243 pg[1.4a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.129377 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129381 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.129391 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.129398 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.129406 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129413 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.129420 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.129431 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:185 2014-06-09 16:22:12.129440 7f8689f56700 20 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.129447 7f8689f56700 20 osd.26 pg_epoch: 243 pg[1.5e1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.129455 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129458 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.129469 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.129476 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.129483 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129491 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.129498 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.129509 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.129517 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.129524 7f8689f56700 20 osd.26 pg_epoch: 243 pg[2.587( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.129533 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129536 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.129546 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.129554 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.129561 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129568 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.129575 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.129586 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.129595 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.129602 7f8689f56700 20 osd.26 pg_epoch: 243 pg[0.594( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.129610 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129632 7f8689f56700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41950 (19 active) 2014-06-09 16:22:12.129661 7f8689f56700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41950 (20 active) 2014-06-09 16:22:12.129664 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.58f( v 213'596 (0'0,213'596] local-les=243 n=120 ec=118 les/c 243/243 242/242/223) 2014-06-09 16:22:12.129696 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129701 7f8689f56700 5 queue_transactions existing osr(3.58f 0x3ed1850)/0x3ed1850 2014-06-09 16:22:12.129703 7f8689f56700 10 op_submit_start 4078 2014-06-09 16:22:12.129704 7f8689f56700 5 queue_transactions (trailing journal) 4078 2014-06-09 16:22:12.129705 7f8689f56700 5 queue_op 0x2fcaef00 seq 4078 osr(3.58f 0x3ed1850) 828 bytes (queue has 46 ops and 2384254 bytes) 2014-06-09 16:22:12.129708 7f8689f56700 10 op_submit_finish 4078 2014-06-09 16:22:12.129710 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.129722 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.930640 7 0.000259 2014-06-09 16:22:12.129734 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.129745 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129755 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.129765 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.129773 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.129780 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.129792 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.129803 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.129812 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.129821 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.129829 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.129843 7f8689f56700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.129860 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.129889 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.129893 7f8689f56700 5 queue_transactions existing osr(2.28a 0x2ed0d20)/0x2ed0d20 2014-06-09 16:22:12.129895 7f8689f56700 10 op_submit_start 4079 2014-06-09 16:22:12.129896 7f8689f56700 5 queue_transactions (trailing journal) 4079 2014-06-09 16:22:12.129897 7f8689f56700 5 queue_op 0x30061f20 seq 4079 osr(2.28a 0x2ed0d20) 861 bytes (queue has 47 ops and 2385115 bytes) 2014-06-09 16:22:12.129900 7f8689f56700 10 op_submit_finish 4079 2014-06-09 16:22:12.129902 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.129915 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.967972 7 0.002373 2014-06-09 16:22:12.129925 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.129935 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.129945 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.129955 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.129964 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.129972 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.129983 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.129994 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.130004 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.130016 7f8689f56700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.130030 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1095 2014-06-09 16:22:12.130061 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.130064 7f8689f56700 5 queue_transactions existing osr(3.5d1 0x3ed1b70)/0x3ed1b70 2014-06-09 16:22:12.130066 7f8689f56700 10 op_submit_start 4080 2014-06-09 16:22:12.130067 7f8689f56700 5 queue_transactions (trailing journal) 4080 2014-06-09 16:22:12.130068 7f8689f56700 5 queue_op 0x2fc8cf60 seq 4080 osr(3.5d1 0x3ed1b70) 861 bytes (queue has 48 ops and 2385976 bytes) 2014-06-09 16:22:12.130071 7f8689f56700 10 op_submit_finish 4080 2014-06-09 16:22:12.130073 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.130084 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.949378 7 0.000426 2014-06-09 16:22:12.130094 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.130103 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.130112 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.130122 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.130130 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.130137 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.130146 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.130157 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.130165 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.130174 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.130183 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.130194 7f8689f56700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.130209 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.130238 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.130241 7f8689f56700 5 queue_transactions existing osr(2.65c 0x2ed1fb0)/0x2ed1fb0 2014-06-09 16:22:12.130244 7f8689f56700 10 op_submit_start 4081 2014-06-09 16:22:12.130245 7f8689f56700 5 queue_transactions (trailing journal) 4081 2014-06-09 16:22:12.130246 7f8689f56700 5 queue_op 0x2fc8c660 seq 4081 osr(2.65c 0x2ed1fb0) 861 bytes (queue has 49 ops and 2386837 bytes) 2014-06-09 16:22:12.130249 7f8689f56700 10 op_submit_finish 4081 2014-06-09 16:22:12.130251 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.130264 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] exit Started/Primary/Active/Activating 0.994442 7 0.000491 2014-06-09 16:22:12.130276 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.130286 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] needs_recovery is recovered 2014-06-09 16:22:12.130296 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.130307 7f8689f56700 5 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.130316 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] finish_recovery 2014-06-09 16:22:12.130324 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active] clear_recovery_state 2014-06-09 16:22:12.130336 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.130347 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.130357 7f8689f56700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] share_pg_info 2014-06-09 16:22:12.130369 7f8689f56700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.130385 7f8689f56700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] publish_stats_to_osd 243:1116 2014-06-09 16:22:12.130416 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.130420 7f8689f56700 5 queue_transactions existing osr(3.f6 0x3ed04b0)/0x3ed04b0 2014-06-09 16:22:12.130422 7f8689f56700 10 op_submit_start 4082 2014-06-09 16:22:12.130423 7f8689f56700 5 queue_transactions (trailing journal) 4082 2014-06-09 16:22:12.130424 7f8689f56700 5 queue_op 0x14dd61e0 seq 4082 osr(3.f6 0x3ed04b0) 858 bytes (queue has 50 ops and 2387695 bytes) 2014-06-09 16:22:12.130427 7f8689f56700 10 op_submit_finish 4082 2014-06-09 16:22:12.130431 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.130444 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.840310 7 0.000620 2014-06-09 16:22:12.130454 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.130463 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.130472 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.130482 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.130490 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.130497 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.130506 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.130516 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.130525 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.130534 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.130545 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.130558 7f8689f56700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.130573 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.130602 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.130607 7f8689f56700 5 queue_transactions existing osr(0.479 0x2bd7490)/0x2bd7490 2014-06-09 16:22:12.130609 7f8689f56700 2 waiting 51 > 50 ops || 2388556 > 104857600 2014-06-09 16:22:12.130614 7f8689755700 2 waiting 51 > 50 ops || 2388521 > 104857600 2014-06-09 16:22:12.130620 7f868e75f700 2 waiting 51 > 50 ops || 2388556 > 104857600 2014-06-09 16:22:12.130628 7f868d75d700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130634 7f8691f66700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130640 7f868df5e700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130645 7f8690f64700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130650 7f8691765700 2 waiting 51 > 50 ops || 2388521 > 104857600 2014-06-09 16:22:12.130654 7f868b759700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130659 7f868af58700 2 waiting 51 > 50 ops || 6583915 > 104857600 2014-06-09 16:22:12.130664 7f868f761700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130671 7f868cf5c700 2 waiting 51 > 50 ops || 2388556 > 104857600 2014-06-09 16:22:12.130677 7f868bf5a700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130682 7f8692767700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130686 7f868c75b700 2 waiting 51 > 50 ops || 2388556 > 104857600 2014-06-09 16:22:12.130692 7f868ef60700 2 waiting 51 > 50 ops || 2388521 > 104857600 2014-06-09 16:22:12.130708 7f868ff62700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130715 7f8692f68700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130721 7f8690763700 2 waiting 51 > 50 ops || 2388523 > 104857600 2014-06-09 16:22:12.130726 7f868a757700 2 waiting 51 > 50 ops || 2388556 > 104857600 2014-06-09 16:22:12.134654 7f867bb63700 5 -- op tracker -- , seq: 3659, time: 2014-06-09 16:22:12.132449, event: header_read, op: osd_op(client.4575.0:32703 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~1036288] 3.c0524973 ack+ondisk+write e242) 2014-06-09 16:22:12.134666 7f867bb63700 5 -- op tracker -- , seq: 3659, time: 2014-06-09 16:22:12.132453, event: throttled, op: osd_op(client.4575.0:32703 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~1036288] 3.c0524973 ack+ondisk+write e242) 2014-06-09 16:22:12.134677 7f867bb63700 5 -- op tracker -- , seq: 3659, time: 2014-06-09 16:22:12.134616, event: all_read, op: osd_op(client.4575.0:32703 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~1036288] 3.c0524973 ack+ondisk+write e242) 2014-06-09 16:22:12.134686 7f867bb63700 5 -- op tracker -- , seq: 3659, time: 0.000000, event: dispatched, op: osd_op(client.4575.0:32703 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~1036288] 3.c0524973 ack+ondisk+write e242) 2014-06-09 16:22:12.134694 7f867bb63700 20 osd.26 243 should_share_map client.4575 172.16.96.220:0/1127679 242 2014-06-09 16:22:12.134700 7f867bb63700 20 osd.26 243 client session last_sent_epoch: 0 versus osdmap epoch 243 2014-06-09 16:22:12.134724 7f867bb63700 15 osd.26 243 enqueue_op 0x2c06a800 prio 63 cost 1036288 latency 0.002274 osd_op(client.4575.0:32703 rbd_data.117f6b8b4567.0000000000001573 [set-alloc-hint object_size 4194304 write_size 4194304,write 2936832~1036288] 3.c0524973 ack+ondisk+write e242) v4 2014-06-09 16:22:12.135852 7f86a5610700 10 _generic_write 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 0~1531904 = 0 2014-06-09 16:22:12.135919 7f86a5610700 15 _setattrs 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.135936 7f86a5610700 10 _setattrs 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 = 0 2014-06-09 16:22:12.135944 7f86a5610700 15 _setattrs 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 2014-06-09 16:22:12.135952 7f86a5610700 10 _setattrs 3.44d_head/b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 = 0 2014-06-09 16:22:12.136127 7f86a5610700 10 cid: oid: b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 not skipping op, *spos 4033.0.0 2014-06-09 16:22:12.136133 7f86a5610700 10 > header.spos 0.0.0 2014-06-09 16:22:12.136174 7f86a5610700 20 set_header setting 30165 cid 3.44d_head oid b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 parent seq 0 2014-06-09 16:22:12.136264 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4033.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.136272 7f86a5610700 10 cid: oid: 1fe1a41b/pglog_3.44d/0//-1 skipping op, spos 4033.0.0 <= header.spos 150346.0.0 2014-06-09 16:22:12.136275 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.141115 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.141147 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4034.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.141154 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.141318 7f86a5610700 10 _do_op 0x159960c0 seq 4033 r = 0, finisher 0x32079a20 0x30fd6f40 2014-06-09 16:22:12.141335 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc8a60] 2014-06-09 16:22:12.141339 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 27 2014-06-09 16:22:12.141373 7f86a460e700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.141363, event: op_commit, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.141385 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc8a60] 2014-06-09 16:22:12.141388 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.141389 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.141394 7f86a5610700 10 _finish_op 0x159960c0 seq 4033 osr(3.44d 0x3ed1180)/0x3ed1180 2014-06-09 16:22:12.141407 7f8689f56700 10 op_submit_start 4083 2014-06-09 16:22:12.141410 7f8689f56700 5 queue_transactions (trailing journal) 4083 2014-06-09 16:22:12.141410 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e360 (1 active) 2014-06-09 16:22:12.141410 7f868e75f700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.141412 7f8689f56700 5 queue_op 0xc2d5e60 seq 4083 osr(0.479 0x2bd7490) 861 bytes (queue has 50 ops and 854254 bytes) 2014-06-09 16:22:12.141414 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300b40 (2 active) 2014-06-09 16:22:12.141416 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32079a20] 2014-06-09 16:22:12.141419 7f86a5610700 5 _do_op 0xcd90540 seq 4035 osr(3.661 0x3ed19f0)/0x3ed19f0 start 2014-06-09 16:22:12.141421 7f868d75d700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141422 7f86a5610700 10 _do_transaction on 0xbf538c0 2014-06-09 16:22:12.141427 7f8689755700 2 waiting 51 > 50 ops || 855080 > 104857600 2014-06-09 16:22:12.141433 7f8689f56700 10 op_submit_finish 4083 2014-06-09 16:22:12.141433 7f868df5e700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141421 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 27 2014-06-09 16:22:12.141438 7f8690f64700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141443 7f8691765700 2 waiting 51 > 50 ops || 855080 > 104857600 2014-06-09 16:22:12.141438 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.141443 7f86a4e0f700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.141436, event: op_applied, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.141449 7f868b759700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141454 7f868af58700 2 waiting 51 > 50 ops || 5050474 > 104857600 2014-06-09 16:22:12.141460 7f86a5610700 15 _omap_setkeys meta/1fe0ca4a/pglog_3.661/0//-1 2014-06-09 16:22:12.141452 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'636 2014-06-09 16:22:12.141454 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.988178 7 0.000362 2014-06-09 16:22:12.141461 7f868f761700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141470 7f868cf5c700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.141471 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32079a20] 2014-06-09 16:22:12.141473 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.141474 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.141468 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.141479 7f868bf5a700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141485 7f8692767700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141480 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.141492 7f868c75b700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.141492 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 2014-06-09 16:22:12.141499 7f868ef60700 2 waiting 51 > 50 ops || 855080 > 104857600 2014-06-09 16:22:12.141501 7f8689f56700 5 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.141505 7f868ff62700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141506 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.141514 7f8692f68700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141511 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.141520 7f8690763700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.141524 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.141534 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.141540 7f8689f56700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.141552 7f8689f56700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.141568 7f8689f56700 15 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 2014-06-09 16:22:12.141592 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.141596 7f8689f56700 5 queue_transactions existing osr(2.3fa 0x2ed12f0)/0x2ed12f0 2014-06-09 16:22:12.141598 7f8689f56700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.141601 7f868a757700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.141607 7f8691f66700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142665 7f86a5610700 10 lookup_strip_header done cid meta oid 1fe0ca4a/pglog_3.661/0//-1 2014-06-09 16:22:12.142668 7f86a5e11700 10 _do_op 0xbf7f920 seq 4034 r = 0, finisher 0x31435f00 0 2014-06-09 16:22:12.142677 7f86a5e11700 10 _finish_op 0xbf7f920 seq 4034 osr(1.555 0x384dab0)/0x384dab0 2014-06-09 16:22:12.142678 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31412c20] 2014-06-09 16:22:12.142681 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31412c20] 2014-06-09 16:22:12.142683 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.142684 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.142689 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dde60 (1 active) 2014-06-09 16:22:12.142692 7f868d75d700 10 op_submit_start 4084 2014-06-09 16:22:12.142692 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2866c0 (2 active) 2014-06-09 16:22:12.142695 7f868d75d700 5 queue_transactions (trailing journal) 4084 2014-06-09 16:22:12.142693 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31435f00] 2014-06-09 16:22:12.142696 7f86a5e11700 5 _do_op 0x14dd6780 seq 4036 osr(0.75 0x2bd6370)/0x2bd6370 start 2014-06-09 16:22:12.142696 7f868d75d700 5 queue_op 0x16fd5500 seq 4084 osr(0.2cb 0x2bd6ed0) 828 bytes (queue has 50 ops and 854254 bytes) 2014-06-09 16:22:12.142699 7f86a5e11700 10 _do_transaction on 0xc303bc0 2014-06-09 16:22:12.142700 7f868d75d700 10 op_submit_finish 4084 2014-06-09 16:22:12.142701 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31435f00] 2014-06-09 16:22:12.142701 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.142703 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.142704 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.142706 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.142712 7f8689755700 2 waiting 51 > 50 ops || 855080 > 104857600 2014-06-09 16:22:12.142719 7f868df5e700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142723 7f868d75d700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f35f0 (19 active) 2014-06-09 16:22:12.142725 7f8690f64700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142731 7f8691765700 2 waiting 51 > 50 ops || 855080 > 104857600 2014-06-09 16:22:12.142733 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.142736 7f868b759700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142740 7f868af58700 2 waiting 51 > 50 ops || 5050474 > 104857600 2014-06-09 16:22:12.142744 7f868f761700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142749 7f868cf5c700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.142739 7f868d75d700 10 osd.26 243 dequeue_op 0x2976cf00 prio 196 cost 0 latency 0.028819 osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) v2 pg pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=213'549 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.142756 7f868bf5a700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142763 7f8692767700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142760 7f868d75d700 5 -- op tracker -- , seq: 3657, time: 2014-06-09 16:22:12.142756, event: reached_pg, op: osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.142769 7f868c75b700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.142774 7f868ef60700 2 waiting 51 > 50 ops || 855080 > 104857600 2014-06-09 16:22:12.142775 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.142769 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=213'549 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2976cf00 2014-06-09 16:22:12.142783 7f868ff62700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142784 7f868d75d700 5 -- op tracker -- , seq: 3657, time: 2014-06-09 16:22:12.142781, event: started, op: osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.142789 7f8692f68700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142794 7f8690763700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142791 7f868d75d700 7 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=213'549 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 28 op ack_type 4 from 6 2014-06-09 16:22:12.142802 7f86a5610700 15 _omap_rmkeys meta/1fe0ca4a/pglog_3.661/0//-1 2014-06-09 16:22:12.142808 7f86a5610700 10 _omap_rmkeys meta/1fe0ca4a/pglog_3.661/0//-1 r = 0 2014-06-09 16:22:12.142810 7f868d75d700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.142803, event: sub_op_commit_rec, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.142814 7f86a5610700 15 _omap_setkeys meta/1fe0ca4a/pglog_3.661/0//-1 2014-06-09 16:22:12.142817 7f868d75d700 10 osd.26 243 dequeue_op 0x2976cf00 finish 2014-06-09 16:22:12.142826 7f86a5610700 15 _touch 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.142824 7f868d75d700 5 -- op tracker -- , seq: 3657, time: 2014-06-09 16:22:12.142820, event: done, op: osd_sub_op_reply(client.4577.0:37501 3.661 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.142839 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.142838 7f8689f56700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.142847 7f868a757700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.142853 7f8691f66700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142859 7f868e75f700 2 waiting 51 > 50 ops || 855115 > 104857600 2014-06-09 16:22:12.142869 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f35f0 (20 active) 2014-06-09 16:22:12.142874 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.3cd( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.3cd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.142884 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.661_head oid 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.142913 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.142916 7f86a5610700 20 write_state seq is 30167 2014-06-09 16:22:12.142919 7f868d75d700 5 queue_transactions existing osr(1.3cd 0x384d4b0)/0x384d4b0 2014-06-09 16:22:12.142921 7f868d75d700 2 waiting 51 > 50 ops || 855082 > 104857600 2014-06-09 16:22:12.142922 7f86a5610700 20 set_header setting 30166 cid 3.661_head oid 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 parent seq 0 2014-06-09 16:22:12.142939 7f86a5610700 10 _touch 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 = 0 2014-06-09 16:22:12.142946 7f86a5610700 15 _write 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 0~811008 2014-06-09 16:22:12.142962 7f86a5610700 10 file_to_extents done 2014-06-09 16:22:12.144887 7f86a5610700 10 _generic_write 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 0~811008 = 0 2014-06-09 16:22:12.144933 7f86a5610700 15 _setattrs 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.144947 7f86a5610700 10 _setattrs 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 = 0 2014-06-09 16:22:12.144954 7f86a5610700 15 _setattrs 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 2014-06-09 16:22:12.144963 7f86a5610700 10 _setattrs 3.661_head/92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 = 0 2014-06-09 16:22:12.145050 7f86a5610700 10 cid: oid: 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 not skipping op, *spos 4035.0.0 2014-06-09 16:22:12.145055 7f86a5610700 10 > header.spos 0.0.0 2014-06-09 16:22:12.145082 7f86a5610700 20 set_header setting 30166 cid 3.661_head oid 92539e61/rbd_data.11816b8b4567.000000000000194d/head//3 parent seq 0 2014-06-09 16:22:12.145136 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4035.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.145142 7f86a5610700 10 cid: oid: 1fe0ca4a/pglog_3.661/0//-1 skipping op, spos 4035.0.0 <= header.spos 149744.0.0 2014-06-09 16:22:12.145146 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.147413 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.147442 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4036.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.147448 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.147458 7f86a5e11700 10 _do_op 0x14dd6780 seq 4036 r = 0, finisher 0xbd73da0 0 2014-06-09 16:22:12.147465 7f86a5e11700 10 _finish_op 0x14dd6780 seq 4036 osr(0.75 0x2bd6370)/0x2bd6370 2014-06-09 16:22:12.147469 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324806c0] 2014-06-09 16:22:12.147475 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2866c0 (1 active) 2014-06-09 16:22:12.147478 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b0e0 (2 active) 2014-06-09 16:22:12.147479 7f8689755700 10 op_submit_start 4085 2014-06-09 16:22:12.147481 7f86a5e11700 5 _do_op 0xc88a2a0 seq 4037 osr(1.a 0x384c270)/0x384c270 start 2014-06-09 16:22:12.147482 7f8689755700 5 queue_transactions (trailing journal) 4085 2014-06-09 16:22:12.147484 7f86a5e11700 10 _do_transaction on 0xc0dad80 2014-06-09 16:22:12.147484 7f8689755700 5 queue_op 0x309a21e0 seq 4085 osr(1.a7 0x384c730) 826 bytes (queue has 50 ops and 854222 bytes) 2014-06-09 16:22:12.147474 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.147488 7f8689755700 10 op_submit_finish 4085 2014-06-09 16:22:12.147490 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.147488 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.147493 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.298( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=68-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 2.298( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.147500 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.75( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 2014-06-09 16:22:12.147511 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324806c0] 2014-06-09 16:22:12.147513 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.147514 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.147520 7f868df5e700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147529 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.147529 7f8690f64700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147537 7f8691765700 2 waiting 51 > 50 ops || 855048 > 104857600 2014-06-09 16:22:12.147538 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.147544 7f868b759700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147545 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4037.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.147547 7f8689755700 5 queue_transactions existing osr(2.298 0x2ed0c70)/0x2ed0c70 2014-06-09 16:22:12.147549 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.147550 7f868af58700 2 waiting 51 > 50 ops || 5050442 > 104857600 2014-06-09 16:22:12.147556 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd73da0] 2014-06-09 16:22:12.147557 7f868f761700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147563 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd73da0] 2014-06-09 16:22:12.147564 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.147566 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.147565 7f868cf5c700 2 waiting 51 > 50 ops || 855083 > 104857600 2014-06-09 16:22:12.147573 7f868bf5a700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147574 7f86a5e11700 10 _do_op 0xc88a2a0 seq 4037 r = 0, finisher 0xc2d6860 0 2014-06-09 16:22:12.147579 7f86a5e11700 10 _finish_op 0xc88a2a0 seq 4037 osr(1.a 0x384c270)/0x384c270 2014-06-09 16:22:12.147581 7f8692767700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147587 7f868c75b700 2 waiting 51 > 50 ops || 855083 > 104857600 2014-06-09 16:22:12.147587 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7d840] 2014-06-09 16:22:12.147590 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7d840] 2014-06-09 16:22:12.147591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.147593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.147592 7f868ef60700 2 waiting 51 > 50 ops || 855048 > 104857600 2014-06-09 16:22:12.147599 7f868ff62700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147605 7f8692f68700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147611 7f8690763700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147643 7f8689f56700 2 waiting 51 > 50 ops || 855083 > 104857600 2014-06-09 16:22:12.147649 7f868a757700 2 waiting 51 > 50 ops || 855083 > 104857600 2014-06-09 16:22:12.147656 7f8691f66700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147665 7f868e75f700 2 waiting 51 > 50 ops || 855083 > 104857600 2014-06-09 16:22:12.147673 7f868d75d700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147681 7f8689755700 2 waiting 51 > 50 ops || 855050 > 104857600 2014-06-09 16:22:12.147692 7f8690f64700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147693 7f868df5e700 10 op_submit_start 4086 2014-06-09 16:22:12.147693 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b0e0 (1 active) 2014-06-09 16:22:12.147695 7f868df5e700 5 queue_transactions (trailing journal) 4086 2014-06-09 16:22:12.147696 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d6860] 2014-06-09 16:22:12.147696 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c6c0 (2 active) 2014-06-09 16:22:12.147698 7f868b759700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147697 7f868df5e700 5 queue_op 0x2e21b60 seq 4086 osr(0.58a 0x2bd7770) 828 bytes (queue has 50 ops and 854226 bytes) 2014-06-09 16:22:12.147699 7f86a5e11700 5 _do_op 0x230b94a0 seq 4038 osr(0.6e7 0x384c1b0)/0x384c1b0 start 2014-06-09 16:22:12.147701 7f868df5e700 10 op_submit_finish 4086 2014-06-09 16:22:12.147701 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d6860] 2014-06-09 16:22:12.147702 7f868af58700 2 waiting 51 > 50 ops || 5050446 > 104857600 2014-06-09 16:22:12.147702 7f86a5e11700 10 _do_transaction on 0x3145b00 2014-06-09 16:22:12.147703 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.147704 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.147706 7f868f761700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147711 7f868cf5c700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147709 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.147705 7f868df5e700 10 osd.26 pg_epoch: 243 pg[1.3d6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.3d6( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.147716 7f868bf5a700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147723 7f8692767700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147728 7f868c75b700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147735 7f868ef60700 2 waiting 51 > 50 ops || 855052 > 104857600 2014-06-09 16:22:12.147740 7f868ff62700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147741 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.147745 7f8692f68700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147747 7f868df5e700 5 queue_transactions existing osr(1.3d6 0x384d470)/0x384d470 2014-06-09 16:22:12.147750 7f8690763700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147757 7f8689f56700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147762 7f868a757700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147768 7f8691f66700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147775 7f868e75f700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147780 7f868d75d700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147787 7f8689755700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147793 7f868df5e700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147798 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.147799 7f8691765700 2 waiting 51 > 50 ops || 855052 > 104857600 2014-06-09 16:22:12.147805 7f86a5610700 10 _do_op 0xcd90540 seq 4035 r = 0, finisher 0x33a68e0 0x30c403c0 2014-06-09 16:22:12.147813 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4038.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.147815 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32534f00] 2014-06-09 16:22:12.147817 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.147828 7f86a5e11700 10 _do_op 0x230b94a0 seq 4038 r = 0, finisher 0x318c2020 0 2014-06-09 16:22:12.147819 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=213'549 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 28 2014-06-09 16:22:12.147832 7f86a5e11700 10 _finish_op 0x230b94a0 seq 4038 osr(0.6e7 0x384c1b0)/0x384c1b0 2014-06-09 16:22:12.147840 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c6c0 (1 active) 2014-06-09 16:22:12.147841 7f8690f64700 10 op_submit_start 4087 2014-06-09 16:22:12.147843 7f8690f64700 5 queue_transactions (trailing journal) 4087 2014-06-09 16:22:12.147843 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b680 (2 active) 2014-06-09 16:22:12.147843 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c2020] 2014-06-09 16:22:12.147845 7f86a5e11700 5 _do_op 0x3250f500 seq 4039 osr(0.2d9 0x2bd6e90)/0x2bd6e90 start 2014-06-09 16:22:12.147844 7f8690f64700 5 queue_op 0x30a35c0 seq 4087 osr(1.71e 0x2ed00b0) 828 bytes (queue has 50 ops and 854226 bytes) 2014-06-09 16:22:12.147847 7f86a5e11700 10 _do_transaction on 0x398be00 2014-06-09 16:22:12.147848 7f8690f64700 10 op_submit_finish 4087 2014-06-09 16:22:12.147843 7f86a460e700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.147834, event: op_commit, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.147851 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c2020] 2014-06-09 16:22:12.147852 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.147852 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.147854 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.147852 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=213'549 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 28 all committed 2014-06-09 16:22:12.147851 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.151( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.151( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.147864 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] eval_repop repgather(0xc014e40 243'550 rep_tid=28 committed?=1 applied?=0 lock=0 op=osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.147886 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.147889 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.147892 7f8690f64700 5 queue_transactions existing osr(0.151 0x2bd65e0)/0x2bd65e0 2014-06-09 16:22:12.147894 7f8690f64700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147883 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] log_op_stats osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4 inb 811392 outb 0 rlat 0.000000 lat 1.325259 2014-06-09 16:22:12.147899 7f868b759700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147904 7f868af58700 2 waiting 51 > 50 ops || 5050446 > 104857600 2014-06-09 16:22:12.147903 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4039.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.147907 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.147907 7f868f761700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147903 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] publish_stats_to_osd 243:926 2014-06-09 16:22:12.147912 7f868cf5c700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147919 7f868bf5a700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147924 7f86a5e11700 10 _do_op 0x3250f500 seq 4039 r = 0, finisher 0x3206f480 0 2014-06-09 16:22:12.147926 7f8692767700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147915 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] sending commit on repgather(0xc014e40 243'550 rep_tid=28 committed?=1 applied?=0 lock=0 op=osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4) 0x32071180 2014-06-09 16:22:12.147931 7f868c75b700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147928 7f86a5e11700 10 _finish_op 0x3250f500 seq 4039 osr(0.2d9 0x2bd6e90)/0x2bd6e90 2014-06-09 16:22:12.147936 7f868ef60700 2 waiting 51 > 50 ops || 855052 > 104857600 2014-06-09 16:22:12.147942 7f868ff62700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147947 7f8692f68700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147952 7f8690763700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147957 7f8689f56700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147962 7f868a757700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147968 7f8691f66700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147974 7f868e75f700 2 waiting 51 > 50 ops || 855087 > 104857600 2014-06-09 16:22:12.147980 7f868d75d700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147952 7f86a460e700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.147947, event: commit_sent, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.147989 7f8689755700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.147993 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32534f00] 2014-06-09 16:22:12.147996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2d738e60,0x30d61be0] 2014-06-09 16:22:12.147998 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2d738e60,0x30d61be0] 2014-06-09 16:22:12.148000 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.148001 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.148004 7f868df5e700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.148009 7f8691765700 2 waiting 51 > 50 ops || 855052 > 104857600 2014-06-09 16:22:12.148020 7f8690f64700 10 op_submit_start 4088 2014-06-09 16:22:12.148020 7f868b759700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.148022 7f8690f64700 5 queue_transactions (trailing journal) 4088 2014-06-09 16:22:12.148025 7f868f761700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.148023 7f8690f64700 5 queue_op 0x30a26c0 seq 4088 osr(0.151 0x2bd65e0) 828 bytes (queue has 50 ops and 854226 bytes) 2014-06-09 16:22:12.148028 7f868af58700 2 waiting 51 > 50 ops || 5050446 > 104857600 2014-06-09 16:22:12.148032 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b680 (1 active) 2014-06-09 16:22:12.148035 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524d80 (2 active) 2014-06-09 16:22:12.148034 7f868bf5a700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.148039 7f86a5e11700 5 _do_op 0x309a2a80 seq 4040 osr(3.310 0x3ed0d80)/0x3ed0d80 start 2014-06-09 16:22:12.148040 7f8692767700 2 waiting 51 > 50 ops || 855054 > 104857600 2014-06-09 16:22:12.148042 7f86a5e11700 10 _do_transaction on 0x2f28840 2014-06-09 16:22:12.148041 7f86a5610700 10 _finish_op 0xcd90540 seq 4035 osr(3.661 0x3ed19f0)/0x3ed19f0 2014-06-09 16:22:12.148045 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3206f480] 2014-06-09 16:22:12.148050 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3206f480] 2014-06-09 16:22:12.148050 7f868ef60700 2 waiting 51 > 50 ops || 42474 > 104857600 2014-06-09 16:22:12.148051 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300b40 (1 active) 2014-06-09 16:22:12.148052 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a68e0] 2014-06-09 16:22:12.148054 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4079e0 (2 active) 2014-06-09 16:22:12.148056 7f8692f68700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148059 7f86a5610700 5 _do_op 0xc64b9e0 seq 4041 osr(0.219 0x2bd6ad0)/0x2bd6ad0 start 2014-06-09 16:22:12.148060 7f8690763700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148061 7f8690f64700 10 op_submit_finish 4088 2014-06-09 16:22:12.148062 7f86a5610700 10 _do_transaction on 0xbf2e900 2014-06-09 16:22:12.148065 7f8689f56700 2 waiting 51 > 50 ops || 42509 > 104857600 2014-06-09 16:22:12.148055 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] op_applied: 28 2014-06-09 16:22:12.148069 7f868b759700 10 op_submit_start 4089 2014-06-09 16:22:12.148068 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.148070 7f868b759700 5 queue_transactions (trailing journal) 4089 2014-06-09 16:22:12.148071 7f868b759700 5 queue_op 0xc88acc0 seq 4089 osr(0.6c0 0x2bd7f50) 828 bytes (queue has 50 ops and 41648 bytes) 2014-06-09 16:22:12.148074 7f868b759700 10 op_submit_finish 4089 2014-06-09 16:22:12.148066 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.1f0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) 2014-06-09 16:22:12.148076 7f86a4e0f700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.148069, event: op_applied, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.148082 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.148083 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] op_applied on primary on version 243'550 2014-06-09 16:22:12.148091 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.148098 7f868b759700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f3fc0 (19 active) 2014-06-09 16:22:12.148094 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] repop_all_applied: repop tid 28 all applied 2014-06-09 16:22:12.148099 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.148107 7f868b759700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.148113 7f8690f64700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.148104 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] eval_repop repgather(0xc014e40 243'550 rep_tid=28 committed?=1 applied?=1 lock=0 op=osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.148124 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.148112 7f868b759700 10 osd.26 243 dequeue_op 0x30c6cc00 prio 196 cost 0 latency 0.033918 osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) v2 pg pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.148120 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] log_op_stats osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4 inb 811392 outb 0 rlat 0.000000 lat 1.325496 2014-06-09 16:22:12.148128 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.1f0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:192 2014-06-09 16:22:12.148131 7f868b759700 5 -- op tracker -- , seq: 3658, time: 2014-06-09 16:22:12.148128, event: reached_pg, op: osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.148140 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4041.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.148144 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.148145 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.148137 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] publish_stats_to_osd 243:927 2014-06-09 16:22:12.148142 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x30c6cc00 2014-06-09 16:22:12.148155 7f86a5610700 10 _do_op 0xc64b9e0 seq 4041 r = 0, finisher 0x3281b4c0 0 2014-06-09 16:22:12.148148 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.4b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) 2014-06-09 16:22:12.148149 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.148156 7f868b759700 5 -- op tracker -- , seq: 3658, time: 2014-06-09 16:22:12.148153, event: started, op: osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.148166 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3281bb40] 2014-06-09 16:22:12.148161 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] state: peer osd.4 activated and committed 2014-06-09 16:22:12.148162 7f868b759700 7 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 27 op ack_type 4 from 9 2014-06-09 16:22:12.148172 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3281bb40] 2014-06-09 16:22:12.148160 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] removing repgather(0xc014e40 243'550 rep_tid=28 committed?=1 applied?=1 op=osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.148174 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.148169 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.148176 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.148179 7f86a5610700 10 _finish_op 0xc64b9e0 seq 4041 osr(0.219 0x2bd6ad0)/0x2bd6ad0 2014-06-09 16:22:12.148177 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.148179 7f868b759700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.148173, event: sub_op_commit_rec, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.148188 7f8690f64700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 243 2014-06-09 16:22:12.148175 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] q front is repgather(0xc014e40 243'550 rep_tid=28 committed?=1 applied?=1 op=osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.148190 7f8691f66700 10 op_submit_start 4090 2014-06-09 16:22:12.148192 7f8691f66700 5 queue_transactions (trailing journal) 4090 2014-06-09 16:22:12.148193 7f8691f66700 5 queue_op 0x2ea53aa0 seq 4090 osr(1.517 0x384db70) 828 bytes (queue has 50 ops and 41648 bytes) 2014-06-09 16:22:12.148186 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_applied: repop tid 27 all applied 2014-06-09 16:22:12.148199 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4079e0 (1 active) 2014-06-09 16:22:12.148201 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f8c0 (2 active) 2014-06-09 16:22:12.148190 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] remove_repop repgather(0xc014e40 243'550 rep_tid=28 committed?=1 applied?=1 op=osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.148207 7f8691f66700 10 op_submit_finish 4090 2014-06-09 16:22:12.148202 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.4b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:151 2014-06-09 16:22:12.148196 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xbfe6540 243'636 rep_tid=27 committed?=0 applied?=1 lock=0 op=osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.148205 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] requeue_ops 2014-06-09 16:22:12.148216 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.148211 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.631( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.631( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.148212 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] sending ack on repgather(0xbfe6540 243'636 rep_tid=27 committed?=0 applied?=1 lock=0 op=osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4) 0x30bbf180 2014-06-09 16:22:12.148220 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.468( v 213'658 (0'0,213'658] local-les=243 n=124 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:12.148232 7f86a4e0f700 5 -- op tracker -- , seq: 801, time: 2014-06-09 16:22:12.148226, event: done, op: osd_op(client.4577.0:37501 rbd_data.11816b8b4567.000000000000194d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~811008] 3.92539e61 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.148248 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.148240 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 luod=214'635 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 27 all committed 2014-06-09 16:22:12.148251 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.148253 7f8691f66700 5 queue_transactions existing osr(0.631 0x2bd7c70)/0x2bd7c70 2014-06-09 16:22:12.148256 7f8691f66700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148256 7f8690f64700 5 queue_transactions existing osr(3.468 0x3ed1590)/0x3ed1590 2014-06-09 16:22:12.148258 7f8690f64700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148260 7f86a5610700 5 _do_op 0x3250f380 seq 4042 osr(3.7ae 0xb7f01d0)/0xb7f01d0 start 2014-06-09 16:22:12.148263 7f86a5610700 10 _do_transaction on 0xbfec780 2014-06-09 16:22:12.148250 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 0'0 active] eval_repop repgather(0xbfe6540 243'636 rep_tid=27 committed?=1 applied?=1 lock=0 op=osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.148266 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a68e0] 2014-06-09 16:22:12.148269 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281b4c0] 2014-06-09 16:22:12.148269 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.148271 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281b4c0] 2014-06-09 16:22:12.148273 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.148274 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.148266 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 0'0 active] log_op_stats osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4 inb 1532288 outb 0 rlat 1.832897 lat 1.832923 2014-06-09 16:22:12.148286 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 0'0 active] publish_stats_to_osd 243:1050 2014-06-09 16:22:12.148312 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.148300 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 0'0 active] sending commit on repgather(0xbfe6540 243'636 rep_tid=27 committed?=1 applied?=1 lock=0 op=osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4) 0x257c6a00 2014-06-09 16:22:12.148327 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4042.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.148331 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.148335 7f868b759700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.148330, event: commit_sent, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.148341 7f86a5610700 10 _do_op 0x3250f380 seq 4042 r = 0, finisher 0x3251c560 0 2014-06-09 16:22:12.148343 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.148351 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f85c0] 2014-06-09 16:22:12.148354 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f85c0] 2014-06-09 16:22:12.148355 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.148357 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.148360 7f86a5610700 10 _finish_op 0x3250f380 seq 4042 osr(3.7ae 0xb7f01d0)/0xb7f01d0 2014-06-09 16:22:12.148370 7f868d75d700 10 op_submit_start 4091 2014-06-09 16:22:12.148372 7f868d75d700 5 queue_transactions (trailing journal) 4091 2014-06-09 16:22:12.148373 7f8689755700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148373 7f868d75d700 5 queue_op 0x16fd5260 seq 4091 osr(1.3cd 0x384d4b0) 828 bytes (queue has 50 ops and 41648 bytes) 2014-06-09 16:22:12.148380 7f868df5e700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148385 7f8691765700 2 waiting 51 > 50 ops || 42474 > 104857600 2014-06-09 16:22:12.148385 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251c560] 2014-06-09 16:22:12.148381 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f8c0 (1 active) 2014-06-09 16:22:12.148389 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251c560] 2014-06-09 16:22:12.148354 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] removing repgather(0xbfe6540 243'636 rep_tid=27 committed?=1 applied?=1 op=osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.148390 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.148390 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c1d40 (2 active) 2014-06-09 16:22:12.148392 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.148395 7f868d75d700 10 op_submit_finish 4091 2014-06-09 16:22:12.148392 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] q front is repgather(0xbfe6540 243'636 rep_tid=27 committed?=1 applied?=1 op=osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.148399 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.67( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.67( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.148410 7f868b759700 20 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] remove_repop repgather(0xbfe6540 243'636 rep_tid=27 committed?=1 applied?=1 op=osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.148425 7f868b759700 15 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] requeue_ops 2014-06-09 16:22:12.148438 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.148443 7f868d75d700 5 queue_transactions existing osr(1.67 0x384c5b0)/0x384c5b0 2014-06-09 16:22:12.148446 7f868d75d700 2 waiting 51 > 50 ops || 42474 > 104857600 2014-06-09 16:22:12.148450 7f86a5610700 5 _do_op 0x324e6a80 seq 4043 osr(2.45b 0x2ed1530)/0x2ed1530 start 2014-06-09 16:22:12.148449 7f868b759700 5 -- op tracker -- , seq: 270, time: 2014-06-09 16:22:12.148443, event: done, op: osd_op(client.4576.0:35874 rbd_data.11806b8b4567.000000000000181b [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1531904] 3.b4f7744d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.148453 7f86a5610700 10 _do_transaction on 0x3859a40 2014-06-09 16:22:12.148459 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.148469 7f868b759700 10 osd.26 243 dequeue_op 0x30c6cc00 finish 2014-06-09 16:22:12.148477 7f868b759700 5 -- op tracker -- , seq: 3658, time: 2014-06-09 16:22:12.148473, event: done, op: osd_sub_op_reply(client.4576.0:35874 3.44d b4f7744d/rbd_data.11806b8b4567.000000000000181b/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.148488 7f868b759700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.148502 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.148516 7f868b759700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f3fc0 (20 active) 2014-06-09 16:22:12.148519 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4043.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.148523 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.148520 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.4fe( v 213'605 (0'0,213'605] local-les=243 n=114 ec=118 les/c 243/243 242/242/202) 2014-06-09 16:22:12.148547 7f86a5610700 10 _do_op 0x324e6a80 seq 4043 r = 0, finisher 0xbd5ffa0 0 2014-06-09 16:22:12.148555 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.148559 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31478020] 2014-06-09 16:22:12.148560 7f868b759700 5 queue_transactions existing osr(3.4fe 0x3ed1730)/0x3ed1730 2014-06-09 16:22:12.148562 7f868b759700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148562 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.148573 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.148582 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.45b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.148592 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31478020] 2014-06-09 16:22:12.148594 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.148590 7f868e75f700 2 waiting 51 > 50 ops || 42509 > 104857600 2014-06-09 16:22:12.148595 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.148598 7f86a5610700 10 _finish_op 0x324e6a80 seq 4043 osr(2.45b 0x2ed1530)/0x2ed1530 2014-06-09 16:22:12.148600 7f868cf5c700 2 waiting 51 > 50 ops || 42509 > 104857600 2014-06-09 16:22:12.148605 7f868a757700 2 waiting 51 > 50 ops || 42509 > 104857600 2014-06-09 16:22:12.148610 7f868f761700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148613 7f868bf5a700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148620 7f868c75b700 2 waiting 51 > 50 ops || 42509 > 104857600 2014-06-09 16:22:12.148625 7f868ff62700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148618 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.148631 7f8692f68700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148638 7f8690763700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148643 7f8689f56700 2 waiting 51 > 50 ops || 42509 > 104857600 2014-06-09 16:22:12.148648 7f868ef60700 2 waiting 51 > 50 ops || 42474 > 104857600 2014-06-09 16:22:12.148656 7f8690f64700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148663 7f8691f66700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148675 7f8689755700 10 op_submit_start 4092 2014-06-09 16:22:12.148674 7f868af58700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.148675 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c1d40 (1 active) 2014-06-09 16:22:12.148677 7f8689755700 5 queue_transactions (trailing journal) 4092 2014-06-09 16:22:12.148679 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cea0 (2 active) 2014-06-09 16:22:12.148679 7f8689755700 5 queue_op 0x324cd5c0 seq 4092 osr(2.298 0x2ed0c70) 828 bytes (queue has 50 ops and 41615 bytes) 2014-06-09 16:22:12.148682 7f8689755700 10 op_submit_finish 4092 2014-06-09 16:22:12.148681 7f86a5610700 5 _do_op 0x3250f200 seq 4044 osr(0.5e 0x2bd63b0)/0x2bd63b0 start 2014-06-09 16:22:12.148685 7f86a5610700 10 _do_transaction on 0xbf82fc0 2014-06-09 16:22:12.148685 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.148690 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.148685 7f8689755700 10 osd.26 pg_epoch: 243 pg[2.69d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.69d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.148703 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4040.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.148708 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.148719 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.148724 7f8689755700 5 queue_transactions existing osr(2.69d 0x2ed1ef0)/0x2ed1ef0 2014-06-09 16:22:12.148726 7f8689755700 2 waiting 51 > 50 ops || 42443 > 104857600 2014-06-09 16:22:12.148730 7f868df5e700 2 waiting 51 > 50 ops || 42443 > 104857600 2014-06-09 16:22:12.148736 7f8692767700 2 waiting 51 > 50 ops || 42443 > 104857600 2014-06-09 16:22:12.148739 7f86a5e11700 10 _do_op 0x309a2a80 seq 4040 r = 0, finisher 0xbd6d860 0 2014-06-09 16:22:12.148741 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ffa0] 2014-06-09 16:22:12.148741 7f868d75d700 2 waiting 51 > 50 ops || 42441 > 104857600 2014-06-09 16:22:12.148745 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ffa0] 2014-06-09 16:22:12.148746 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.148747 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.148748 7f868b759700 2 waiting 51 > 50 ops || 42443 > 104857600 2014-06-09 16:22:12.148754 7f868e75f700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148758 7f868cf5c700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148763 7f868a757700 2 waiting 51 > 50 ops || 42476 > 104857600 2014-06-09 16:22:12.148767 7f868f761700 2 waiting 51 > 50 ops || 42443 > 104857600 2014-06-09 16:22:12.148774 7f868bf5a700 2 waiting 51 > 50 ops || 42443 > 104857600 2014-06-09 16:22:12.148774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32535820] 2014-06-09 16:22:12.148780 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.148778 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.148795 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4044.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.148800 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.148792 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.148810 7f86a5610700 10 _do_op 0x3250f200 seq 4044 r = 0, finisher 0x30508ea0 0 2014-06-09 16:22:12.148803 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.310( v 213'668 (0'0,213'668] local-les=243 n=130 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1077 2014-06-09 16:22:12.148814 7f86a5610700 10 _finish_op 0x3250f200 seq 4044 osr(0.5e 0x2bd63b0)/0x2bd63b0 2014-06-09 16:22:12.148814 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32535820] 2014-06-09 16:22:12.148816 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32534da0] 2014-06-09 16:22:12.148818 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32534da0] 2014-06-09 16:22:12.148819 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.148820 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.148822 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cea0 (1 active) 2014-06-09 16:22:12.148823 7f868c75b700 2 waiting 51 > 50 ops || 4237870 > 104857600 2014-06-09 16:22:12.148824 7f868af58700 10 op_submit_start 4093 2014-06-09 16:22:12.148824 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338000 (2 active) 2014-06-09 16:22:12.148826 7f868af58700 5 queue_transactions (trailing journal) 4093 2014-06-09 16:22:12.148825 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30508ea0] 2014-06-09 16:22:12.148828 7f868ff62700 2 waiting 51 > 50 ops || 4237837 > 104857600 2014-06-09 16:22:12.148827 7f86a5610700 5 _do_op 0x230b9740 seq 4045 osr(1.6f5 0x2ed0170)/0x2ed0170 start 2014-06-09 16:22:12.148829 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30508ea0] 2014-06-09 16:22:12.148828 7f868af58700 5 queue_op 0x300601e0 seq 4093 osr(3.27 0x3ed0340) 4196220 bytes (queue has 50 ops and 4237009 bytes) 2014-06-09 16:22:12.148831 7f86a5610700 10 _do_transaction on 0xc128180 2014-06-09 16:22:12.148831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.148831 7f868af58700 10 op_submit_finish 4093 2014-06-09 16:22:12.148832 7f8690763700 2 waiting 51 > 50 ops || 4237837 > 104857600 2014-06-09 16:22:12.148832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.148834 7f868af58700 10 osd.26 243 dequeue_op 0x32826100 finish 2014-06-09 16:22:12.148835 7f8689f56700 2 waiting 51 > 50 ops || 4237870 > 104857600 2014-06-09 16:22:12.148836 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.148840 7f86a5e11700 10 _finish_op 0x309a2a80 seq 4040 osr(3.310 0x3ed0d80)/0x3ed0d80 2014-06-09 16:22:12.148840 7f868ef60700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.148855 7f8691f66700 10 op_submit_start 4094 2014-06-09 16:22:12.148854 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524d80 (1 active) 2014-06-09 16:22:12.148857 7f8691f66700 5 queue_transactions (trailing journal) 4094 2014-06-09 16:22:12.148857 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6d860] 2014-06-09 16:22:12.148857 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922480 (2 active) 2014-06-09 16:22:12.148858 7f8691f66700 5 queue_op 0x2ea52d20 seq 4094 osr(0.631 0x2bd7c70) 828 bytes (queue has 50 ops and 4236976 bytes) 2014-06-09 16:22:12.148861 7f8691f66700 10 op_submit_finish 4094 2014-06-09 16:22:12.148849 7f868af58700 5 -- op tracker -- , seq: 3239, time: 2014-06-09 16:22:12.148838, event: done, op: MOSDPGPush(3.27 243 [PushOp(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3, version: 215'668, data_included: [0~4194304], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3@215'668, copy_subset: [0~4194304], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.148860 7f86a5e11700 5 _do_op 0x288f9f80 seq 4046 osr(2.1e3 0x2ed0810)/0x2ed0810 start 2014-06-09 16:22:12.148863 7f86a5e11700 10 _do_transaction on 0xbf2e300 2014-06-09 16:22:12.148864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6d860] 2014-06-09 16:22:12.148866 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.148867 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.148868 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.148870 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.148864 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.2d9( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.2d9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.148876 7f868af58700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.148879 7f868af58700 10 osd.26 243 dequeue_op 0x32827d00 prio 63 cost 3777512 latency 0.117182 MOSDPGPush(3.113 243 [PushOp(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3, version: 227'654, data_included: [0~3776512], data_size: 3776512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3@227'654, copy_subset: [0~3776512], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3776512, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) v2 pg pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] 2014-06-09 16:22:12.148899 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.148904 7f8691f66700 5 queue_transactions existing osr(0.2d9 0x2bd6e90)/0x2bd6e90 2014-06-09 16:22:12.148897 7f868af58700 5 -- op tracker -- , seq: 3611, time: 2014-06-09 16:22:12.148893, event: reached_pg, op: MOSDPGPush(3.113 243 [PushOp(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3, version: 227'654, data_included: [0~3776512], data_size: 3776512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2594913/rbd_data.11756b8b4567.00000000000015fb/head//3@227'654, copy_subset: [0~3776512], clone_subset: {}), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3776512, data_complete:true, omap_recovered_to:, omap_complete:true), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false))]) 2014-06-09 16:22:12.148907 7f8691f66700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148911 7f8691765700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.148908 7f868af58700 20 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] 1 flushes_in_progress pending waiting for active on 0x32827d00 2014-06-09 16:22:12.148915 7f868af58700 10 osd.26 243 dequeue_op 0x32827d00 finish 2014-06-09 16:22:12.148917 7f868af58700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.148917 7f8689755700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148923 7f868df5e700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148928 7f8692767700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148933 7f868d75d700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.148935 7f868af58700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8fef0 (20 active) 2014-06-09 16:22:12.148939 7f868b759700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148944 7f868e75f700 2 waiting 51 > 50 ops || 4237837 > 104857600 2014-06-09 16:22:12.148938 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.148948 7f868cf5c700 2 waiting 51 > 50 ops || 4237837 > 104857600 2014-06-09 16:22:12.148951 7f868a757700 2 waiting 51 > 50 ops || 4237837 > 104857600 2014-06-09 16:22:12.148955 7f868f761700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148950 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.967291 7 0.000367 2014-06-09 16:22:12.148960 7f868bf5a700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148965 7f8692f68700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148965 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.148962 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.148970 7f868ff62700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148975 7f8690763700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148978 7f8689f56700 2 waiting 51 > 50 ops || 4237837 > 104857600 2014-06-09 16:22:12.148973 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.148984 7f868ef60700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.148990 7f8690f64700 2 waiting 51 > 50 ops || 4237804 > 104857600 2014-06-09 16:22:12.148984 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.148995 7f868c75b700 2 waiting 51 > 50 ops || 4237837 > 104857600 2014-06-09 16:22:12.148981 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4045.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.148999 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.148996 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.149004 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.149011 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.149023 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.149031 7f86a5610700 10 _do_op 0x230b9740 seq 4045 r = 0, finisher 0xbd6c980 0 2014-06-09 16:22:12.149038 7f86a5610700 10 _finish_op 0x230b9740 seq 4045 osr(1.6f5 0x2ed0170)/0x2ed0170 2014-06-09 16:22:12.149039 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd60a80] 2014-06-09 16:22:12.149041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd60a80] 2014-06-09 16:22:12.149034 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.149042 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.149043 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.149043 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.149049 7f8691765700 10 op_submit_start 4095 2014-06-09 16:22:12.149050 7f8691765700 5 queue_transactions (trailing journal) 4095 2014-06-09 16:22:12.149051 7f8689755700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149052 7f8691765700 5 queue_op 0xcd9cd20 seq 4095 osr(3.e8 0x3ed0510) 826 bytes (queue has 50 ops and 4236974 bytes) 2014-06-09 16:22:12.149055 7f868af58700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.149058 7f868df5e700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149059 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338000 (1 active) 2014-06-09 16:22:12.149062 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652900 (2 active) 2014-06-09 16:22:12.149062 7f8692767700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149067 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6c980] 2014-06-09 16:22:12.149067 7f868d75d700 2 waiting 51 > 50 ops || 4237800 > 104857600 2014-06-09 16:22:12.149073 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6c980] 2014-06-09 16:22:12.149074 7f868b759700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149075 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.149076 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.149079 7f868e75f700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149072 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 2014-06-09 16:22:12.149082 7f868cf5c700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149086 7f868a757700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149090 7f868f761700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149095 7f868bf5a700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149095 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.149100 7f8692f68700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149099 7f8691765700 10 op_submit_finish 4095 2014-06-09 16:22:12.149104 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.149105 7f868ff62700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149109 7f868af58700 5 queue_transactions existing osr(0.718 0x384c090)/0x384c090 2014-06-09 16:22:12.149109 7f8690763700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149111 7f868af58700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149113 7f8689f56700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149107 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.149125 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4046.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.149130 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.149122 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.991482 7 0.000330 2014-06-09 16:22:12.149134 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.149148 7f86a5e11700 10 _do_op 0x288f9f80 seq 4046 r = 0, finisher 0x3281af80 0 2014-06-09 16:22:12.149144 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.149159 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc4117a0] 2014-06-09 16:22:12.149161 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc4117a0] 2014-06-09 16:22:12.149163 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.149155 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.149164 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.149167 7f86a5e11700 10 _finish_op 0x288f9f80 seq 4046 osr(2.1e3 0x2ed0810)/0x2ed0810 2014-06-09 16:22:12.149166 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.149175 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922480 (1 active) 2014-06-09 16:22:12.149177 7f8689755700 10 op_submit_start 4096 2014-06-09 16:22:12.149179 7f8689755700 5 queue_transactions (trailing journal) 4096 2014-06-09 16:22:12.149178 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb0e0 (2 active) 2014-06-09 16:22:12.149175 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.149181 7f86a5e11700 5 _do_op 0xc88ab40 seq 4048 osr(2.213 0x2ed0b70)/0x2ed0b70 start 2014-06-09 16:22:12.149180 7f8689755700 5 queue_op 0x324cca80 seq 4096 osr(2.69d 0x2ed1ef0) 828 bytes (queue has 50 ops and 4236974 bytes) 2014-06-09 16:22:12.149184 7f8689755700 10 op_submit_finish 4096 2014-06-09 16:22:12.149184 7f86a5e11700 10 _do_transaction on 0xc29a780 2014-06-09 16:22:12.149190 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.149188 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.1fd( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:12.149183 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.149205 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.149221 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.149217 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.149226 7f8689755700 5 queue_transactions existing osr(0.1fd 0x2bd6b30)/0x2bd6b30 2014-06-09 16:22:12.149228 7f8689755700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149232 7f8690f64700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149231 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.149226 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.149238 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281af80] 2014-06-09 16:22:12.149238 7f868c75b700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149241 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281af80] 2014-06-09 16:22:12.149243 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.149244 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.149236 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.149244 7f8691f66700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149250 7f868df5e700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149246 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.149254 7f8692767700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149259 7f868d75d700 2 waiting 51 > 50 ops || 4237800 > 104857600 2014-06-09 16:22:12.149260 7f8691765700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.149264 7f868b759700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149269 7f868e75f700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149274 7f868cf5c700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149279 7f868a757700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149284 7f868f761700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149277 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:207 2014-06-09 16:22:12.149260 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4048.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.149289 7f868bf5a700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149291 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.149294 7f8692f68700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149299 7f868ff62700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149304 7f8690763700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149308 7f868af58700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149312 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.149313 7f8689f56700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149316 7f8691765700 5 queue_transactions existing osr(1.86 0x384c490)/0x384c490 2014-06-09 16:22:12.149319 7f8691765700 2 waiting 51 > 50 ops || 4237832 > 104857600 2014-06-09 16:22:12.149323 7f868ef60700 2 waiting 51 > 50 ops || 4237800 > 104857600 2014-06-09 16:22:12.149323 7f86a5e11700 10 _do_op 0xc88ab40 seq 4048 r = 0, finisher 0x30d61ac0 0 2014-06-09 16:22:12.149333 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd62f60] 2014-06-09 16:22:12.149335 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd62f60] 2014-06-09 16:22:12.149337 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.149338 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.149341 7f86a5e11700 10 _finish_op 0xc88ab40 seq 4048 osr(2.213 0x2ed0b70)/0x2ed0b70 2014-06-09 16:22:12.149348 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb0e0 (1 active) 2014-06-09 16:22:12.149349 7f8689755700 10 op_submit_start 4097 2014-06-09 16:22:12.149352 7f8689755700 5 queue_transactions (trailing journal) 4097 2014-06-09 16:22:12.149351 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383eb40 (2 active) 2014-06-09 16:22:12.149354 7f86a5e11700 5 _do_op 0xc88af60 seq 4049 osr(3.7bc 0xb7f0190)/0xb7f0190 start 2014-06-09 16:22:12.149353 7f8689755700 5 queue_op 0x324c1f20 seq 4097 osr(0.1fd 0x2bd6b30) 828 bytes (queue has 50 ops and 4236974 bytes) 2014-06-09 16:22:12.149350 7f86a5610700 5 _do_op 0xcabc360 seq 4047 osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 start 2014-06-09 16:22:12.149357 7f86a5e11700 10 _do_transaction on 0x336ae40 2014-06-09 16:22:12.149357 7f8689755700 10 op_submit_finish 4097 2014-06-09 16:22:12.149358 7f86a5610700 10 _do_transaction on 0xc3269c0 2014-06-09 16:22:12.149362 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.149364 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.149361 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.129( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:12.149393 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.149397 7f8689755700 5 queue_transactions existing osr(0.129 0x2bd66b0)/0x2bd66b0 2014-06-09 16:22:12.149400 7f8689755700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149404 7f8690f64700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149404 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.149409 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30d61ac0] 2014-06-09 16:22:12.149410 7f868c75b700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149419 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30d61ac0] 2014-06-09 16:22:12.149421 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.149422 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.149423 7f8691f66700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149428 7f868df5e700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149433 7f8692767700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149437 7f868d75d700 2 waiting 51 > 50 ops || 4237800 > 104857600 2014-06-09 16:22:12.149442 7f868b759700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149447 7f868e75f700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149452 7f868cf5c700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149457 7f868a757700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149461 7f868f761700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149437 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4049.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.149467 7f868bf5a700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149468 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.149472 7f8692f68700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149476 7f868ff62700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149482 7f8690763700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149487 7f868af58700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149491 7f8689f56700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149497 7f8691765700 2 waiting 51 > 50 ops || 4237832 > 104857600 2014-06-09 16:22:12.149504 7f868ef60700 2 waiting 51 > 50 ops || 4237800 > 104857600 2014-06-09 16:22:12.149504 7f86a5e11700 10 _do_op 0xc88af60 seq 4049 r = 0, finisher 0x2d738100 0 2014-06-09 16:22:12.149547 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.149561 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4047.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.149565 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.149575 7f86a5610700 10 _do_op 0xcabc360 seq 4047 r = 0, finisher 0x3050a0a0 0 2014-06-09 16:22:12.149584 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30d61b20] 2014-06-09 16:22:12.149586 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30d61b20] 2014-06-09 16:22:12.149588 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.149589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.149593 7f86a5e11700 10 _finish_op 0xc88af60 seq 4049 osr(3.7bc 0xb7f0190)/0xb7f0190 2014-06-09 16:22:12.149600 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383eb40 (1 active) 2014-06-09 16:22:12.149602 7f8689755700 10 op_submit_start 4098 2014-06-09 16:22:12.149603 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287680 (2 active) 2014-06-09 16:22:12.149604 7f8689755700 5 queue_transactions (trailing journal) 4098 2014-06-09 16:22:12.149605 7f86a5e11700 5 _do_op 0xc64a660 seq 4050 osr(3.de 0x3ed0520)/0x3ed0520 start 2014-06-09 16:22:12.149605 7f8689755700 5 queue_op 0x324c1f80 seq 4098 osr(0.129 0x2bd66b0) 828 bytes (queue has 50 ops and 4236974 bytes) 2014-06-09 16:22:12.149609 7f8689755700 10 op_submit_finish 4098 2014-06-09 16:22:12.149608 7f86a5e11700 10 _do_transaction on 0x2fb6240 2014-06-09 16:22:12.149615 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.149611 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.1fd( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.1fd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.149642 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.149646 7f8689755700 5 queue_transactions existing osr(0.1fd 0x2bd6b30)/0x2bd6b30 2014-06-09 16:22:12.149649 7f8689755700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149653 7f8690f64700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149658 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d738100] 2014-06-09 16:22:12.149658 7f868c75b700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149661 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d738100] 2014-06-09 16:22:12.149663 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.149664 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.149665 7f8691f66700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149670 7f868df5e700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149675 7f8692767700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149676 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xd9fbb60] 2014-06-09 16:22:12.149678 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xd9fbb60] 2014-06-09 16:22:12.149680 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.149679 7f868d75d700 2 waiting 51 > 50 ops || 4237800 > 104857600 2014-06-09 16:22:12.149681 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.149684 7f868b759700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149684 7f86a5610700 10 _finish_op 0xcabc360 seq 4047 osr(2.5c6 0x2ed1ad0)/0x2ed1ad0 2014-06-09 16:22:12.149695 7f868cf5c700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149696 7f8689755700 10 op_submit_start 4099 2014-06-09 16:22:12.149697 7f8689755700 5 queue_transactions (trailing journal) 4099 2014-06-09 16:22:12.149698 7f8689755700 5 queue_op 0x324c1320 seq 4099 osr(0.1fd 0x2bd6b30) 828 bytes (queue has 50 ops and 4236974 bytes) 2014-06-09 16:22:12.149700 7f868f761700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149705 7f868bf5a700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149710 7f868a757700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149710 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652900 (1 active) 2014-06-09 16:22:12.149715 7f868ff62700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149716 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a0a0] 2014-06-09 16:22:12.149720 7f8690763700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149722 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a0a0] 2014-06-09 16:22:12.149724 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.149725 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.149724 7f868af58700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149729 7f8689f56700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149655 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.149735 7f8691765700 2 waiting 51 > 50 ops || 4237832 > 104857600 2014-06-09 16:22:12.149741 7f868ef60700 2 waiting 51 > 50 ops || 4237800 > 104857600 2014-06-09 16:22:12.149747 7f868e75f700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149750 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4050.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.149753 7f8690f64700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149758 7f868c75b700 2 waiting 51 > 50 ops || 4237835 > 104857600 2014-06-09 16:22:12.149763 7f8691f66700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149768 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5560 (2 active) 2014-06-09 16:22:12.149768 7f868df5e700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149772 7f86a5610700 5 _do_op 0xc88a0c0 seq 4051 osr(3.4fe 0x3ed1730)/0x3ed1730 start 2014-06-09 16:22:12.149774 7f8692767700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149775 7f86a5610700 10 _do_transaction on 0xc001500 2014-06-09 16:22:12.149778 7f868d75d700 2 waiting 51 > 50 ops || 4237800 > 104857600 2014-06-09 16:22:12.149780 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.149782 7f868b759700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149786 7f8692f68700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149789 7f8689755700 10 op_submit_finish 4099 2014-06-09 16:22:12.149792 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.129( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.129( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.149754 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.149821 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.149825 7f8689755700 5 queue_transactions existing osr(0.129 0x2bd66b0)/0x2bd66b0 2014-06-09 16:22:12.149827 7f8689755700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149834 7f86a5e11700 10 _do_op 0xc64a660 seq 4050 r = 0, finisher 0xbcc91e0 0 2014-06-09 16:22:12.149845 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050bfe0] 2014-06-09 16:22:12.149847 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.149859 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.149870 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.de( v 214'581 (0'0,214'581] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:976 2014-06-09 16:22:12.149880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050bfe0] 2014-06-09 16:22:12.149882 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.149883 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.149887 7f86a5e11700 10 _finish_op 0xc64a660 seq 4050 osr(3.de 0x3ed0520)/0x3ed0520 2014-06-09 16:22:12.149899 7f868f761700 10 op_submit_start 4100 2014-06-09 16:22:12.149900 7f868f761700 5 queue_transactions (trailing journal) 4100 2014-06-09 16:22:12.149901 7f868bf5a700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149902 7f868f761700 5 queue_op 0xbe7dce0 seq 4100 osr(1.66e 0x384ded0) 828 bytes (queue has 50 ops and 4236944 bytes) 2014-06-09 16:22:12.149907 7f868a757700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.149912 7f868ff62700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149912 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbcc91e0] 2014-06-09 16:22:12.149916 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbcc91e0] 2014-06-09 16:22:12.149917 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.149917 7f8690763700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149918 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.149921 7f868af58700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.149926 7f8689f56700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.149909 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287680 (1 active) 2014-06-09 16:22:12.149932 7f8691765700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.149933 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcfc0 (2 active) 2014-06-09 16:22:12.149938 7f868f761700 10 op_submit_finish 4100 2014-06-09 16:22:12.149938 7f868ef60700 2 waiting 51 > 50 ops || 4237770 > 104857600 2014-06-09 16:22:12.149944 7f868e75f700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.149948 7f8690f64700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149942 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.616( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.616( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.149953 7f868c75b700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.149957 7f8691f66700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149961 7f868df5e700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149966 7f8692767700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149969 7f868d75d700 2 waiting 51 > 50 ops || 4237770 > 104857600 2014-06-09 16:22:12.149973 7f868b759700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149977 7f8692f68700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149979 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.149981 7f8689755700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.149984 7f868cf5c700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.149984 7f868f761700 5 queue_transactions existing osr(0.616 0x2bd7950)/0x2bd7950 2014-06-09 16:22:12.149987 7f868f761700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150026 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.150041 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4051.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.150046 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.150063 7f86a5610700 10 _do_op 0xc88a0c0 seq 4051 r = 0, finisher 0x30508d40 0 2014-06-09 16:22:12.150072 7f86a5e11700 5 _do_op 0xc88a960 seq 4052 osr(0.4c8 0x2bd7670)/0x2bd7670 start 2014-06-09 16:22:12.150076 7f86a5e11700 10 _do_transaction on 0x32fa300 2014-06-09 16:22:12.150081 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.150121 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.150135 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4052.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.150139 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.150149 7f86a5e11700 10 _do_op 0xc88a960 seq 4052 r = 0, finisher 0xcd0e860 0 2014-06-09 16:22:12.150158 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a72c0] 2014-06-09 16:22:12.150161 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a72c0] 2014-06-09 16:22:12.150162 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.150163 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.150167 7f86a5610700 10 _finish_op 0xc88a0c0 seq 4051 osr(3.4fe 0x3ed1730)/0x3ed1730 2014-06-09 16:22:12.150174 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5560 (1 active) 2014-06-09 16:22:12.150176 7f868bf5a700 10 op_submit_start 4101 2014-06-09 16:22:12.150177 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407560 (2 active) 2014-06-09 16:22:12.150178 7f868bf5a700 5 queue_transactions (trailing journal) 4101 2014-06-09 16:22:12.150179 7f86a5610700 5 _do_op 0x30060600 seq 4053 osr(1.b5 0x384c690)/0x384c690 start 2014-06-09 16:22:12.150179 7f868bf5a700 5 queue_op 0x32524720 seq 4101 osr(0.529 0x2bd7930) 828 bytes (queue has 50 ops and 4236944 bytes) 2014-06-09 16:22:12.150182 7f86a5610700 10 _do_transaction on 0xbf0fd40 2014-06-09 16:22:12.150183 7f868bf5a700 10 op_submit_finish 4101 2014-06-09 16:22:12.150187 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.150186 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.2a2( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.2a2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.150216 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.150221 7f868bf5a700 5 queue_transactions existing osr(0.2a2 0x2bd6c20)/0x2bd6c20 2014-06-09 16:22:12.150223 7f868bf5a700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150225 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.150227 7f868a757700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150233 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30508d40] 2014-06-09 16:22:12.150233 7f868ff62700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150236 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30508d40] 2014-06-09 16:22:12.150238 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.150239 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.150239 7f8690763700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150244 7f868af58700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150248 7f8689f56700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150253 7f8691765700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.150258 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b020] 2014-06-09 16:22:12.150259 7f868ef60700 2 waiting 51 > 50 ops || 4237770 > 104857600 2014-06-09 16:22:12.150261 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b020] 2014-06-09 16:22:12.150263 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.150264 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.150265 7f868e75f700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150267 7f86a5e11700 10 _finish_op 0xc88a960 seq 4052 osr(0.4c8 0x2bd7670)/0x2bd7670 2014-06-09 16:22:12.150275 7f868bf5a700 10 op_submit_start 4102 2014-06-09 16:22:12.150276 7f868bf5a700 5 queue_transactions (trailing journal) 4102 2014-06-09 16:22:12.150278 7f868bf5a700 5 queue_op 0x32524a20 seq 4102 osr(0.2a2 0x2bd6c20) 828 bytes (queue has 50 ops and 4236944 bytes) 2014-06-09 16:22:12.150280 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd0e860] 2014-06-09 16:22:12.150282 7f8691f66700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150283 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd0e860] 2014-06-09 16:22:12.150285 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.150286 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.150286 7f868c75b700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150289 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcfc0 (1 active) 2014-06-09 16:22:12.150290 7f8692767700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150291 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407d40 (2 active) 2014-06-09 16:22:12.150294 7f868d75d700 2 waiting 51 > 50 ops || 4237770 > 104857600 2014-06-09 16:22:12.150296 7f86a5e11700 5 _do_op 0x32833620 seq 4054 osr(1.3c3 0x384d530)/0x384d530 start 2014-06-09 16:22:12.150298 7f868b759700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150299 7f86a5e11700 10 _do_transaction on 0x394fbc0 2014-06-09 16:22:12.150302 7f8692f68700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150304 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.150306 7f868df5e700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150310 7f868cf5c700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150314 7f868bf5a700 10 op_submit_finish 4102 2014-06-09 16:22:12.150315 7f868f761700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150318 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.5c3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.5c3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.150348 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.150353 7f868bf5a700 5 queue_transactions existing osr(0.5c3 0x2bd7a50)/0x2bd7a50 2014-06-09 16:22:12.150355 7f868bf5a700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150359 7f868a757700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150364 7f8690f64700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150369 7f8690763700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150374 7f868af58700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150379 7f8689f56700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150362 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4053.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.150384 7f8691765700 2 waiting 51 > 50 ops || 4237802 > 104857600 2014-06-09 16:22:12.150386 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.150390 7f868ef60700 2 waiting 51 > 50 ops || 4237770 > 104857600 2014-06-09 16:22:12.150396 7f868e75f700 2 waiting 51 > 50 ops || 4237805 > 104857600 2014-06-09 16:22:12.150402 7f868ff62700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150407 7f8689755700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150441 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.150455 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4054.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.150459 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.150470 7f86a5e11700 10 _do_op 0x32833620 seq 4054 r = 0, finisher 0xc2d7080 0 2014-06-09 16:22:12.150399 7f86a5610700 10 _do_op 0x30060600 seq 4053 r = 0, finisher 0x324809c0 0 2014-06-09 16:22:12.150485 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd72720] 2014-06-09 16:22:12.150487 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd72720] 2014-06-09 16:22:12.150488 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.150489 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.150497 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe67f40] 2014-06-09 16:22:12.150499 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe67f40] 2014-06-09 16:22:12.150501 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.150502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.150504 7f86a5610700 10 _finish_op 0x30060600 seq 4053 osr(1.b5 0x384c690)/0x384c690 2014-06-09 16:22:12.150513 7f8691f66700 10 op_submit_start 4103 2014-06-09 16:22:12.150515 7f8691f66700 5 queue_transactions (trailing journal) 4103 2014-06-09 16:22:12.150516 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324809c0] 2014-06-09 16:22:12.150516 7f8691f66700 5 queue_op 0x2ea53b60 seq 4103 osr(0.2d9 0x2bd6e90) 828 bytes (queue has 50 ops and 4236946 bytes) 2014-06-09 16:22:12.150519 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324809c0] 2014-06-09 16:22:12.150521 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.150521 7f868c75b700 2 waiting 51 > 50 ops || 4237807 > 104857600 2014-06-09 16:22:12.150522 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.150525 7f8692767700 2 waiting 51 > 50 ops || 4237774 > 104857600 2014-06-09 16:22:12.150525 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407560 (1 active) 2014-06-09 16:22:12.150528 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc922900 (2 active) 2014-06-09 16:22:12.150529 7f868d75d700 2 waiting 51 > 50 ops || 4237772 > 104857600 2014-06-09 16:22:12.150532 7f86a5610700 5 _do_op 0xc2ea6c0 seq 4055 osr(2.4a5 0x2ed17f0)/0x2ed17f0 start 2014-06-09 16:22:12.150534 7f8691f66700 10 op_submit_finish 4103 2014-06-09 16:22:12.150534 7f86a5610700 10 _do_transaction on 0xc258fc0 2014-06-09 16:22:12.150540 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.150537 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.150550 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.150558 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.150566 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.150578 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.150574 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.150582 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.150593 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4055.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.150597 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.150595 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.150605 7f8691f66700 20 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.150612 7f86a5610700 10 _do_op 0xc2ea6c0 seq 4055 r = 0, finisher 0x3006d2e0 0 2014-06-09 16:22:12.150613 7f8691f66700 20 osd.26 pg_epoch: 243 pg[2.3d7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.150622 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.150621 7f86a5e11700 10 _finish_op 0x32833620 seq 4054 osr(1.3c3 0x384d530)/0x384d530 2014-06-09 16:22:12.150631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d7080] 2014-06-09 16:22:12.150626 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.150636 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d7080] 2014-06-09 16:22:12.150638 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.150639 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.150642 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407d40 (1 active) 2014-06-09 16:22:12.150644 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62fe60 (2 active) 2014-06-09 16:22:12.150640 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.150646 7f86a5e11700 5 _do_op 0x30a3ec0 seq 4056 osr(1.c6 0x384c610)/0x384c610 start 2014-06-09 16:22:12.150649 7f86a5e11700 10 _do_transaction on 0xbe88cc0 2014-06-09 16:22:12.150655 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.150648 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x14052000,0x14053300,0x14053200,0x14053100,0x14053000 2014-06-09 16:22:12.150672 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.150681 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.150696 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.150690 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.150704 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:989 2014-06-09 16:22:12.150710 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4056.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.150714 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.150714 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.150725 7f86a5e11700 10 _do_op 0x30a3ec0 seq 4056 r = 0, finisher 0x30509de0 0 2014-06-09 16:22:12.150722 7f8691f66700 20 osd.26 pg_epoch: 243 pg[3.4d1( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.150732 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.150734 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3f820] 2014-06-09 16:22:12.150737 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3f820] 2014-06-09 16:22:12.150738 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.150739 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.150742 7f86a5610700 10 _finish_op 0xc2ea6c0 seq 4055 osr(2.4a5 0x2ed17f0)/0x2ed17f0 2014-06-09 16:22:12.150736 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.150751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006d2e0] 2014-06-09 16:22:12.150755 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006d2e0] 2014-06-09 16:22:12.150748 7f8691f66700 5 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.969825 7 0.000329 2014-06-09 16:22:12.150757 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.150758 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.150761 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc922900 (1 active) 2014-06-09 16:22:12.150763 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3faa20 (2 active) 2014-06-09 16:22:12.150759 7f8691f66700 5 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.150765 7f86a5610700 5 _do_op 0xc34b6e0 seq 4057 osr(2.52d 0x2ed19f0)/0x2ed19f0 start 2014-06-09 16:22:12.150768 7f86a5610700 10 _do_transaction on 0xc327380 2014-06-09 16:22:12.150773 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.150769 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.150779 7f8691f66700 5 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.150788 7f8691f66700 5 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.150797 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.150804 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.150812 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.150814 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.150826 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4057.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.150831 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.150825 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.150834 7f8691f66700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.150840 7f86a5610700 10 _do_op 0xc34b6e0 seq 4057 r = 0, finisher 0xbd5d1a0 0 2014-06-09 16:22:12.150845 7f86a5610700 10 _finish_op 0xc34b6e0 seq 4057 osr(2.52d 0x2ed19f0)/0x2ed19f0 2014-06-09 16:22:12.150847 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006cb20] 2014-06-09 16:22:12.150847 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.150849 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006cb20] 2014-06-09 16:22:12.150851 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.150852 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.150855 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5d1a0] 2014-06-09 16:22:12.150858 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5d1a0] 2014-06-09 16:22:12.150859 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.150860 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.150870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061fa40] 2014-06-09 16:22:12.150863 7f8691f66700 15 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:163 2014-06-09 16:22:12.150872 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061fa40] 2014-06-09 16:22:12.150873 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.150874 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.150878 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3faa20 (1 active) 2014-06-09 16:22:12.150880 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3019e0 (2 active) 2014-06-09 16:22:12.150884 7f86a5610700 5 _do_op 0x324a4960 seq 4058 osr(0.652 0x2bd7b90)/0x2bd7b90 start 2014-06-09 16:22:12.150886 7f86a5610700 10 _do_transaction on 0xbf44240 2014-06-09 16:22:12.150891 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.150895 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.150899 7f8691f66700 5 queue_transactions existing osr(2.285 0x2ed0d50)/0x2ed0d50 2014-06-09 16:22:12.150902 7f8691f66700 10 op_submit_start 4104 2014-06-09 16:22:12.150903 7f8691f66700 5 queue_transactions (trailing journal) 4104 2014-06-09 16:22:12.150904 7f8691f66700 5 queue_op 0x11fce4e0 seq 4104 osr(2.285 0x2ed0d50) 861 bytes (queue has 48 ops and 4235323 bytes) 2014-06-09 16:22:12.150907 7f8691f66700 10 op_submit_finish 4104 2014-06-09 16:22:12.150909 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.150924 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.060115 7 0.000433 2014-06-09 16:22:12.150936 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.150935 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.150947 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4058.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.150949 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.150945 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.150956 7f86a5610700 10 _do_op 0x324a4960 seq 4058 r = 0, finisher 0xbd79540 0 2014-06-09 16:22:12.150961 7f86a5610700 10 _finish_op 0x324a4960 seq 4058 osr(0.652 0x2bd7b90)/0x2bd7b90 2014-06-09 16:22:12.150954 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.150963 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6dce0] 2014-06-09 16:22:12.150966 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6dce0] 2014-06-09 16:22:12.150967 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.150968 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.150970 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd79540] 2014-06-09 16:22:12.150964 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.150973 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd79540] 2014-06-09 16:22:12.150975 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.150975 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.150977 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3019e0 (1 active) 2014-06-09 16:22:12.150978 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a7e0 (2 active) 2014-06-09 16:22:12.150973 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.150981 7f86a5610700 5 _do_op 0x324a4360 seq 4059 osr(3.778 0x3ed1df0)/0x3ed1df0 start 2014-06-09 16:22:12.150982 7f86a5e11700 10 _finish_op 0x30a3ec0 seq 4056 osr(1.c6 0x384c610)/0x384c610 2014-06-09 16:22:12.150984 7f86a5610700 10 _do_transaction on 0xc239200 2014-06-09 16:22:12.150981 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.150989 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30509de0] 2014-06-09 16:22:12.150988 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62fe60 (1 active) 2014-06-09 16:22:12.150991 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd0e0 (2 active) 2014-06-09 16:22:12.150992 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30509de0] 2014-06-09 16:22:12.150993 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.150993 7f86a5e11700 5 _do_op 0xd04f620 seq 4060 osr(3.1fc 0x3ed0930)/0x3ed0930 start 2014-06-09 16:22:12.150994 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.150996 7f86a5e11700 10 _do_transaction on 0x344ccc0 2014-06-09 16:22:12.150998 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.151000 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.150993 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.151004 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.151015 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.151028 7f8691f66700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.151040 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.151043 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1058 2014-06-09 16:22:12.151053 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4059.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.151058 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.151076 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.151080 7f8691f66700 5 queue_transactions existing osr(3.d3 0x3ed0570)/0x3ed0570 2014-06-09 16:22:12.151083 7f8691f66700 10 op_submit_start 4105 2014-06-09 16:22:12.151080 7f86a5610700 10 _do_op 0x324a4360 seq 4059 r = 0, finisher 0xbd6d4e0 0 2014-06-09 16:22:12.151084 7f8691f66700 5 queue_transactions (trailing journal) 4105 2014-06-09 16:22:12.151085 7f8691f66700 5 queue_op 0xc2ea0c0 seq 4105 osr(3.d3 0x3ed0570) 858 bytes (queue has 47 ops and 4234527 bytes) 2014-06-09 16:22:12.151087 7f8691f66700 10 op_submit_finish 4105 2014-06-09 16:22:12.151091 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc1681e0] 2014-06-09 16:22:12.151093 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc1681e0] 2014-06-09 16:22:12.151095 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.151096 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.151090 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.151099 7f86a5610700 10 _finish_op 0x324a4360 seq 4059 osr(3.778 0x3ed1df0)/0x3ed1df0 2014-06-09 16:22:12.151107 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6d4e0] 2014-06-09 16:22:12.151103 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.060985 7 0.000451 2014-06-09 16:22:12.151113 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6d4e0] 2014-06-09 16:22:12.151115 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.151116 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.151119 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a7e0 (1 active) 2014-06-09 16:22:12.151113 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.151121 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e7e0 (2 active) 2014-06-09 16:22:12.151123 7f86a5610700 5 _do_op 0xd04eea0 seq 4061 osr(2.463 0x2ed1510)/0x2ed1510 start 2014-06-09 16:22:12.151126 7f86a5610700 10 _do_transaction on 0xc23e900 2014-06-09 16:22:12.151123 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.151131 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.151137 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.151133 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.151143 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.151152 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4060.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.151156 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.151151 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.151159 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.151169 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.151180 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.151189 7f86a5e11700 10 _do_op 0xd04f620 seq 4060 r = 0, finisher 0xbcc89e0 0 2014-06-09 16:22:12.151189 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.151200 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd76860] 2014-06-09 16:22:12.151203 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd76860] 2014-06-09 16:22:12.151205 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.151198 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.151207 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.151210 7f86a5e11700 10 _finish_op 0xd04f620 seq 4060 osr(3.1fc 0x3ed0930)/0x3ed0930 2014-06-09 16:22:12.151208 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.151219 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbcc89e0] 2014-06-09 16:22:12.151220 7f8691f66700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.151222 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbcc89e0] 2014-06-09 16:22:12.151224 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.151225 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.151228 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd0e0 (1 active) 2014-06-09 16:22:12.151231 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd7a0 (2 active) 2014-06-09 16:22:12.151233 7f86a5e11700 5 _do_op 0xd04f020 seq 4062 osr(2.759 0x3ed00b0)/0x3ed00b0 start 2014-06-09 16:22:12.151236 7f86a5e11700 10 _do_transaction on 0xc2c5740 2014-06-09 16:22:12.151241 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.151236 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:181 2014-06-09 16:22:12.151267 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.151271 7f8691f66700 5 queue_transactions existing osr(0.633 0x2bd7c50)/0x2bd7c50 2014-06-09 16:22:12.151273 7f8691f66700 10 op_submit_start 4106 2014-06-09 16:22:12.151274 7f8691f66700 5 queue_transactions (trailing journal) 4106 2014-06-09 16:22:12.151275 7f8691f66700 5 queue_op 0xc2ea060 seq 4106 osr(0.633 0x2bd7c50) 861 bytes (queue has 46 ops and 4233732 bytes) 2014-06-09 16:22:12.151278 7f8691f66700 10 op_submit_finish 4106 2014-06-09 16:22:12.151285 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.151280 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.151292 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.006085 7 0.004057 2014-06-09 16:22:12.151300 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4062.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.151305 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.151303 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.151318 7f86a5e11700 10 _do_op 0xd04f020 seq 4062 r = 0, finisher 0xbd6da40 0 2014-06-09 16:22:12.151312 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.151328 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1322d7a0] 2014-06-09 16:22:12.151330 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1322d7a0] 2014-06-09 16:22:12.151331 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.151323 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.151332 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.151335 7f86a5e11700 10 _finish_op 0xd04f020 seq 4062 osr(2.759 0x3ed00b0)/0x3ed00b0 2014-06-09 16:22:12.151334 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.151344 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6da40] 2014-06-09 16:22:12.151346 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6da40] 2014-06-09 16:22:12.151348 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.151349 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.151342 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.151352 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd7a0 (1 active) 2014-06-09 16:22:12.151354 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1320 (2 active) 2014-06-09 16:22:12.151350 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.151356 7f86a5e11700 5 _do_op 0x159966c0 seq 4063 osr(2.236 0x2ed0ad0)/0x2ed0ad0 start 2014-06-09 16:22:12.151359 7f86a5e11700 10 _do_transaction on 0xbf512c0 2014-06-09 16:22:12.151364 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.151360 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.151371 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.151380 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.151392 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.151409 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.151407 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:187 2014-06-09 16:22:12.151424 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4061.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.151428 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.151437 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.151438 7f86a5610700 10 _do_op 0xd04eea0 seq 4061 r = 0, finisher 0xc2d7d40 0 2014-06-09 16:22:12.151441 7f8691f66700 5 queue_transactions existing osr(1.b9 0x384c630)/0x384c630 2014-06-09 16:22:12.151443 7f8691f66700 10 op_submit_start 4107 2014-06-09 16:22:12.151445 7f8691f66700 5 queue_transactions (trailing journal) 4107 2014-06-09 16:22:12.151445 7f86a5610700 10 _finish_op 0xd04eea0 seq 4061 osr(2.463 0x2ed1510)/0x2ed1510 2014-06-09 16:22:12.151446 7f8691f66700 5 queue_op 0xc2ebb00 seq 4107 osr(1.b9 0x384c630) 858 bytes (queue has 46 ops and 4233762 bytes) 2014-06-09 16:22:12.151459 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc411460] 2014-06-09 16:22:12.151463 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc411460] 2014-06-09 16:22:12.151464 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.151465 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.151489 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.151507 7f868f761700 2 waiting 51 > 50 ops || 4237968 > 104857600 2014-06-09 16:22:12.151504 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4063.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.151514 7f868a757700 2 waiting 51 > 50 ops || 4238001 > 104857600 2014-06-09 16:22:12.151517 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.151520 7f8690f64700 2 waiting 51 > 50 ops || 4237968 > 104857600 2014-06-09 16:22:12.151526 7f8690763700 2 waiting 51 > 50 ops || 4237968 > 104857600 2014-06-09 16:22:12.151533 7f868af58700 2 waiting 51 > 50 ops || 4238001 > 104857600 2014-06-09 16:22:12.151539 7f8689f56700 2 waiting 51 > 50 ops || 4238001 > 104857600 2014-06-09 16:22:12.151530 7f86a5e11700 10 _do_op 0x159966c0 seq 4063 r = 0, finisher 0x31bff660 0 2014-06-09 16:22:12.151544 7f8691765700 2 waiting 51 > 50 ops || 4237998 > 104857600 2014-06-09 16:22:12.151551 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d7d40] 2014-06-09 16:22:12.151551 7f868ef60700 2 waiting 51 > 50 ops || 4237966 > 104857600 2014-06-09 16:22:12.151550 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x320747c0] 2014-06-09 16:22:12.151555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d7d40] 2014-06-09 16:22:12.151556 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.151557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.151557 7f868e75f700 2 waiting 51 > 50 ops || 4238001 > 104857600 2014-06-09 16:22:12.151554 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x320747c0] 2014-06-09 16:22:12.151564 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.151566 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.151575 7f868ff62700 2 waiting 51 > 50 ops || 4237968 > 104857600 2014-06-09 16:22:12.151561 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e7e0 (1 active) 2014-06-09 16:22:12.151582 7f8689755700 2 waiting 51 > 50 ops || 4237968 > 104857600 2014-06-09 16:22:12.151582 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5c20 (2 active) 2014-06-09 16:22:12.151587 7f86a5610700 5 _do_op 0x30a2180 seq 4064 osr(0.640 0x2bd7c10)/0x2bd7c10 start 2014-06-09 16:22:12.151588 7f8692767700 2 waiting 51 > 50 ops || 4237968 > 104857600 2014-06-09 16:22:12.151590 7f86a5610700 10 _do_transaction on 0xbe883c0 2014-06-09 16:22:12.151592 7f868d75d700 2 waiting 51 > 50 ops || 4237966 > 104857600 2014-06-09 16:22:12.151596 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.151597 7f868bf5a700 2 waiting 51 > 50 ops || 4237968 > 104857600 2014-06-09 16:22:12.151602 7f8691f66700 10 op_submit_finish 4107 2014-06-09 16:22:12.151606 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.151625 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.040464 7 0.000484 2014-06-09 16:22:12.151641 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.151637 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.151647 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.151657 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4064.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.151661 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.151658 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.151669 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.151682 7f86a5610700 10 _do_op 0x30a2180 seq 4064 r = 0, finisher 0x20050340 0 2014-06-09 16:22:12.151678 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.151689 7f86a5610700 10 _finish_op 0x30a2180 seq 4064 osr(0.640 0x2bd7c10)/0x2bd7c10 2014-06-09 16:22:12.151687 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.151702 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a0025a0] 2014-06-09 16:22:12.151704 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x20050340] 2014-06-09 16:22:12.151705 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a0025a0] 2014-06-09 16:22:12.151707 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.151707 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x20050340] 2014-06-09 16:22:12.151708 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.151708 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.151698 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.151709 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.151711 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5c20 (1 active) 2014-06-09 16:22:12.151713 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0360 (2 active) 2014-06-09 16:22:12.151717 7f86a5610700 5 _do_op 0x16097500 seq 4065 osr(1.6f 0x384c570)/0x384c570 start 2014-06-09 16:22:12.151710 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.151720 7f86a5610700 10 _do_transaction on 0xc2aa0c0 2014-06-09 16:22:12.151725 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.151720 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.151733 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.151749 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1223 2014-06-09 16:22:12.151771 7f86a5e11700 10 _finish_op 0x159966c0 seq 4063 osr(2.236 0x2ed0ad0)/0x2ed0ad0 2014-06-09 16:22:12.151781 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.151784 7f8691f66700 5 queue_transactions existing osr(3.505 0x3ed1700)/0x3ed1700 2014-06-09 16:22:12.151784 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1320 (1 active) 2014-06-09 16:22:12.151787 7f8691f66700 10 op_submit_start 4108 2014-06-09 16:22:12.151786 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31bff660] 2014-06-09 16:22:12.151787 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fd200 (2 active) 2014-06-09 16:22:12.151788 7f8691f66700 5 queue_transactions (trailing journal) 4108 2014-06-09 16:22:12.151790 7f86a5e11700 5 _do_op 0x324b2540 seq 4066 osr(1.494 0x384d990)/0x384d990 start 2014-06-09 16:22:12.151790 7f8691f66700 5 queue_op 0x2c06e900 seq 4108 osr(3.505 0x3ed1700) 861 bytes (queue has 49 ops and 4236345 bytes) 2014-06-09 16:22:12.151792 7f86a5e11700 10 _do_transaction on 0x3085740 2014-06-09 16:22:12.151792 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31bff660] 2014-06-09 16:22:12.151792 7f8691f66700 10 op_submit_finish 4108 2014-06-09 16:22:12.151794 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.151796 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.151797 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.151795 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.151808 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.015448 7 0.000321 2014-06-09 16:22:12.151818 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.151825 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.151827 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.151840 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4065.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.151837 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.151845 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.151847 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.151854 7f86a5610700 10 _do_op 0x16097500 seq 4065 r = 0, finisher 0xbd5ea60 0 2014-06-09 16:22:12.151859 7f86a5610700 10 _finish_op 0x16097500 seq 4065 osr(1.6f 0x384c570)/0x384c570 2014-06-09 16:22:12.151855 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.151863 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.151869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x306280c0] 2014-06-09 16:22:12.151872 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x306280c0] 2014-06-09 16:22:12.151874 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.151875 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.151872 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.151883 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.151892 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.151903 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.151901 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.151911 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.151917 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4066.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.151921 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.151924 7f8691f66700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.151934 7f86a5e11700 10 _do_op 0x324b2540 seq 4066 r = 0, finisher 0xcdb3ee0 0 2014-06-09 16:22:12.151945 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ea60] 2014-06-09 16:22:12.151938 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.151947 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32534f60] 2014-06-09 16:22:12.151949 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ea60] 2014-06-09 16:22:12.151950 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32534f60] 2014-06-09 16:22:12.151951 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.151951 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.151951 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.151952 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.151954 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0360 (1 active) 2014-06-09 16:22:12.151956 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52cc60 (2 active) 2014-06-09 16:22:12.151959 7f86a5610700 5 _do_op 0xc88b5c0 seq 4067 osr(2.26d 0x2ed0de0)/0x2ed0de0 start 2014-06-09 16:22:12.151962 7f86a5610700 10 _do_transaction on 0xc1f35c0 2014-06-09 16:22:12.151968 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.151969 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.151973 7f8691f66700 5 queue_transactions existing osr(0.ca 0x2bd6590)/0x2bd6590 2014-06-09 16:22:12.151976 7f8691f66700 10 op_submit_start 4109 2014-06-09 16:22:12.151977 7f8691f66700 5 queue_transactions (trailing journal) 4109 2014-06-09 16:22:12.151978 7f8691f66700 5 queue_op 0xc2ebec0 seq 4109 osr(0.ca 0x2bd6590) 858 bytes (queue has 49 ops and 4236377 bytes) 2014-06-09 16:22:12.151980 7f8691f66700 10 op_submit_finish 4109 2014-06-09 16:22:12.151983 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.151995 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.989624 7 0.000351 2014-06-09 16:22:12.152007 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.152005 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.152015 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.152021 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4067.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.152025 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.152024 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.152038 7f86a5610700 10 _do_op 0xc88b5c0 seq 4067 r = 0, finisher 0x320740a0 0 2014-06-09 16:22:12.152034 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.152045 7f86a5610700 10 _finish_op 0xc88b5c0 seq 4067 osr(2.26d 0x2ed0de0)/0x2ed0de0 2014-06-09 16:22:12.152046 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c3980] 2014-06-09 16:22:12.152042 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.152049 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c3980] 2014-06-09 16:22:12.152051 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.152052 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.152054 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x320740a0] 2014-06-09 16:22:12.152049 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.152057 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x320740a0] 2014-06-09 16:22:12.152059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.152060 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.152063 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52cc60 (1 active) 2014-06-09 16:22:12.152065 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e480 (2 active) 2014-06-09 16:22:12.152059 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.152069 7f86a5610700 5 _do_op 0x30a20c0 seq 4068 osr(0.3a9 0x2bd7310)/0x2bd7310 start 2014-06-09 16:22:12.152072 7f86a5610700 10 _do_transaction on 0xbe89800 2014-06-09 16:22:12.152070 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.152078 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.152079 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.152089 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.152103 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.152124 7f86a5e11700 10 _finish_op 0x324b2540 seq 4066 osr(1.494 0x384d990)/0x384d990 2014-06-09 16:22:12.152132 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.152136 7f8691f66700 5 queue_transactions existing osr(0.89 0x2bd64a0)/0x2bd64a0 2014-06-09 16:22:12.152137 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcdb3ee0] 2014-06-09 16:22:12.152138 7f8691f66700 10 op_submit_start 4110 2014-06-09 16:22:12.152139 7f8691f66700 5 queue_transactions (trailing journal) 4110 2014-06-09 16:22:12.152141 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcdb3ee0] 2014-06-09 16:22:12.152141 7f8691f66700 5 queue_op 0x2c06e0c0 seq 4110 osr(0.89 0x2bd64a0) 858 bytes (queue has 48 ops and 4235579 bytes) 2014-06-09 16:22:12.152143 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.152144 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.152146 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fd200 (1 active) 2014-06-09 16:22:12.152148 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77bc20 (2 active) 2014-06-09 16:22:12.152153 7f86a5e11700 5 _do_op 0xd3262a0 seq 4069 osr(3.1f0 0x3ed0990)/0x3ed0990 start 2014-06-09 16:22:12.152156 7f86a5e11700 10 _do_transaction on 0x331f080 2014-06-09 16:22:12.152171 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.152174 7f8690763700 2 waiting 51 > 50 ops || 4238096 > 104857600 2014-06-09 16:22:12.152181 7f8691f66700 10 op_submit_finish 4110 2014-06-09 16:22:12.152181 7f868af58700 2 waiting 51 > 50 ops || 4238129 > 104857600 2014-06-09 16:22:12.152188 7f8689f56700 2 waiting 51 > 50 ops || 4238129 > 104857600 2014-06-09 16:22:12.152185 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.152195 7f8691765700 2 waiting 51 > 50 ops || 4238126 > 104857600 2014-06-09 16:22:12.152203 7f868ef60700 2 waiting 51 > 50 ops || 4238094 > 104857600 2014-06-09 16:22:12.152197 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.059308 7 0.000306 2014-06-09 16:22:12.152206 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.152208 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.152221 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4068.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.152217 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.152225 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.152227 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.152235 7f86a5610700 10 _do_op 0x30a20c0 seq 4068 r = 0, finisher 0xbd75500 0 2014-06-09 16:22:12.152241 7f86a5610700 10 _finish_op 0x30a20c0 seq 4068 osr(0.3a9 0x2bd7310)/0x2bd7310 2014-06-09 16:22:12.152237 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.152245 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.152251 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32535aa0] 2014-06-09 16:22:12.152254 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32535aa0] 2014-06-09 16:22:12.152256 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.152257 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.152252 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.152261 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.152271 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.152281 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.152287 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.152292 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.152302 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4069.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.152306 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.152306 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 2014-06-09 16:22:12.152322 7f86a5e11700 10 _do_op 0xd3262a0 seq 4069 r = 0, finisher 0x3205d540 0 2014-06-09 16:22:12.152332 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd75500] 2014-06-09 16:22:12.152335 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.152336 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd75500] 2014-06-09 16:22:12.152338 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.152339 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.152339 7f8691f66700 5 queue_transactions existing osr(0.e2 0x2bd6560)/0x2bd6560 2014-06-09 16:22:12.152341 7f8691f66700 10 op_submit_start 4111 2014-06-09 16:22:12.152342 7f8691f66700 5 queue_transactions (trailing journal) 4111 2014-06-09 16:22:12.152342 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e480 (1 active) 2014-06-09 16:22:12.152344 7f8691f66700 5 queue_op 0x2c06f080 seq 4111 osr(0.e2 0x2bd6560) 858 bytes (queue has 50 ops and 4237298 bytes) 2014-06-09 16:22:12.152345 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77bd40 (2 active) 2014-06-09 16:22:12.152349 7f8690763700 2 waiting 51 > 50 ops || 4238126 > 104857600 2014-06-09 16:22:12.152349 7f86a5610700 5 _do_op 0xc88b9e0 seq 4070 osr(3.786 0x3ed1d90)/0x3ed1d90 start 2014-06-09 16:22:12.152352 7f86a5610700 10 _do_transaction on 0xc3218c0 2014-06-09 16:22:12.152355 7f868ff62700 2 waiting 51 > 50 ops || 4238126 > 104857600 2014-06-09 16:22:12.152360 7f8689755700 2 waiting 51 > 50 ops || 4238126 > 104857600 2014-06-09 16:22:12.152361 7f86a5e11700 10 _finish_op 0xd3262a0 seq 4069 osr(3.1f0 0x3ed0990)/0x3ed0990 2014-06-09 16:22:12.152369 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.152409 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.152424 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4070.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.152428 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.152437 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050bec0] 2014-06-09 16:22:12.152438 7f86a5610700 10 _do_op 0xc88b9e0 seq 4070 r = 0, finisher 0x292823a0 0 2014-06-09 16:22:12.152441 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050bec0] 2014-06-09 16:22:12.152442 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31435cc0] 2014-06-09 16:22:12.152447 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31435cc0] 2014-06-09 16:22:12.152449 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.152450 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.152451 7f868bf5a700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.152455 7f868f761700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.152465 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205d540] 2014-06-09 16:22:12.152470 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205d540] 2014-06-09 16:22:12.152471 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.152472 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.152475 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77bc20 (1 active) 2014-06-09 16:22:12.152478 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad7a0 (2 active) 2014-06-09 16:22:12.152484 7f8691f66700 10 op_submit_finish 4111 2014-06-09 16:22:12.152488 7f868af58700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.152489 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.152502 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.042510 7 0.000355 2014-06-09 16:22:12.152514 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.152525 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.152536 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.152546 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.152556 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.152563 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.152574 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.152585 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.152593 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.152603 7f8691f66700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.152620 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.152653 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.152658 7f8691f66700 5 queue_transactions existing osr(1.207 0x384ccd0)/0x384ccd0 2014-06-09 16:22:12.152660 7f8691f66700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.152665 7f8689f56700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.152669 7f8692f68700 10 op_submit_start 4112 2014-06-09 16:22:12.152671 7f8692f68700 5 queue_transactions (trailing journal) 4112 2014-06-09 16:22:12.152670 7f8691765700 2 waiting 51 > 50 ops || 4238154 > 104857600 2014-06-09 16:22:12.152672 7f8692f68700 5 queue_op 0xc88a1e0 seq 4112 osr(3.5f8 0x3ed1b30) 828 bytes (queue has 50 ops and 4237296 bytes) 2014-06-09 16:22:12.152676 7f8692f68700 10 op_submit_finish 4112 2014-06-09 16:22:12.152678 7f868ef60700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.152683 7f868e75f700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.152688 7f8692767700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.152682 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.508( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.152693 7f868ff62700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.152698 7f8689755700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.152717 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.152722 7f8692f68700 5 queue_transactions existing osr(1.508 0x384db90)/0x384db90 2014-06-09 16:22:12.152725 7f8692f68700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.152728 7f868cf5c700 10 op_submit_start 4113 2014-06-09 16:22:12.152729 7f868cf5c700 5 queue_transactions (trailing journal) 4113 2014-06-09 16:22:12.152730 7f868cf5c700 5 queue_op 0x2ea529c0 seq 4113 osr(2.5f1 0x2ed1a10) 861 bytes (queue has 50 ops and 4237296 bytes) 2014-06-09 16:22:12.152732 7f868cf5c700 10 op_submit_finish 4113 2014-06-09 16:22:12.152736 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.152751 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.053721 7 0.000387 2014-06-09 16:22:12.152763 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.152774 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.152784 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.152795 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.152803 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.152812 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.152822 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.152834 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.152844 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.152856 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.152872 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:12.152906 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.152910 7f868cf5c700 5 queue_transactions existing osr(0.34e 0x2bd7020)/0x2bd7020 2014-06-09 16:22:12.152913 7f868cf5c700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.152936 7f868df5e700 10 op_submit_start 4114 2014-06-09 16:22:12.152941 7f868df5e700 5 queue_transactions (trailing journal) 4114 2014-06-09 16:22:12.152943 7f868df5e700 5 queue_op 0xbf7f4a0 seq 4114 osr(1.3d6 0x384d470) 828 bytes (queue has 50 ops and 4237296 bytes) 2014-06-09 16:22:12.152946 7f868df5e700 10 op_submit_finish 4114 2014-06-09 16:22:12.152952 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.172( v 213'603 (0'0,213'603] local-les=243 n=116 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:12.152984 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.152989 7f868df5e700 5 queue_transactions existing osr(3.172 0x3ed06e0)/0x3ed06e0 2014-06-09 16:22:12.152992 7f868df5e700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.152996 7f868c75b700 10 op_submit_start 4115 2014-06-09 16:22:12.152998 7f868c75b700 5 queue_transactions (trailing journal) 4115 2014-06-09 16:22:12.152999 7f868c75b700 5 queue_op 0x2fc8c060 seq 4115 osr(0.29d 0x2bd6c40) 861 bytes (queue has 50 ops and 4237296 bytes) 2014-06-09 16:22:12.153003 7f868c75b700 10 op_submit_finish 4115 2014-06-09 16:22:12.153008 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.153023 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.153032 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.153040 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.153049 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.153057 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.153069 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.153075 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.153079 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.7d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.153087 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.153091 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.153104 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.153112 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.153121 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.153130 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.153138 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.153150 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.153160 7f868c75b700 20 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.153169 7f868c75b700 20 osd.26 pg_epoch: 243 pg[1.1e4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.153178 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.153182 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.153194 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.153202 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.153210 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.153219 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.153227 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.153240 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:175 2014-06-09 16:22:12.153250 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.153257 7f868c75b700 20 osd.26 pg_epoch: 243 pg[0.1fb( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.153266 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.153270 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.153284 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.153293 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.153303 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.153312 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.153322 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.153335 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:894 2014-06-09 16:22:12.153346 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.153353 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.19b( v 214'526 (0'0,214'526] local-les=243 n=101 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.153364 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.153368 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.153380 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.153387 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.153396 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.153403 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.153411 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.153422 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.153428 7f868c75b700 20 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.153432 7f868c75b700 20 osd.26 pg_epoch: 243 pg[2.53d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.153440 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.153444 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.15c( v 214'638 lc 213'628 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.153458 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.153468 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.153477 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.153491 7f868c75b700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 243 2014-06-09 16:22:12.153506 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1022 2014-06-09 16:22:12.153523 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.153544 7f868c75b700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41050 (19 active) 2014-06-09 16:22:12.153552 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.153556 7f868c75b700 10 osd.26 243 dequeue_op 0x1b0b6800 prio 63 cost 1548288 latency 0.429266 osd_op(client.4538.0:36646 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 1019904~1548288] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:12.153581 7f868c75b700 5 -- op tracker -- , seq: 2780, time: 2014-06-09 16:22:12.153576, event: reached_pg, op: osd_op(client.4538.0:36646 rbd_data.115a6b8b4567.0000000000001a30 [set-alloc-hint object_size 4194304 write_size 4194304,write 1019904~1548288] 3.356aed1e RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.153588 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.153601 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 1 flushes_in_progress pending waiting for active on 0x1b0b6800 2014-06-09 16:22:12.153611 7f868c75b700 10 osd.26 243 dequeue_op 0x1b0b6800 finish 2014-06-09 16:22:12.153615 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.153639 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41050 (20 active) 2014-06-09 16:22:12.153644 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.153657 7f868c75b700 5 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.985343 7 0.004805 2014-06-09 16:22:12.153669 7f868c75b700 5 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.153679 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.153689 7f868c75b700 5 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.153700 7f868c75b700 5 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.153708 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.153716 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.153726 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.153737 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.153746 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.153755 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.153765 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.153778 7f868c75b700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.153791 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:201 2014-06-09 16:22:12.153825 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.153830 7f868c75b700 5 queue_transactions existing osr(2.412 0x2ed15d0)/0x2ed15d0 2014-06-09 16:22:12.153833 7f868c75b700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.153859 7f868a757700 10 op_submit_start 4116 2014-06-09 16:22:12.153863 7f868a757700 5 queue_transactions (trailing journal) 4116 2014-06-09 16:22:12.153865 7f868a757700 5 queue_op 0x14dd6840 seq 4116 osr(1.109 0x384c950) 861 bytes (queue has 50 ops and 4237296 bytes) 2014-06-09 16:22:12.153869 7f868a757700 10 op_submit_finish 4116 2014-06-09 16:22:12.153875 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.153889 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.006555 7 0.000346 2014-06-09 16:22:12.153900 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.153911 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.153921 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.153932 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.153941 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.153948 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.153959 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.153970 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.153980 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.153993 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.154010 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:163 2014-06-09 16:22:12.154016 7f8690f64700 10 op_submit_start 4117 2014-06-09 16:22:12.154020 7f8690f64700 5 queue_transactions (trailing journal) 4117 2014-06-09 16:22:12.154022 7f8690f64700 5 queue_op 0x30a33e0 seq 4117 osr(3.468 0x3ed1590) 828 bytes (queue has 50 ops and 4237296 bytes) 2014-06-09 16:22:12.154026 7f8690f64700 10 op_submit_finish 4117 2014-06-09 16:22:12.154032 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.5c1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.154044 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.154048 7f868a757700 5 queue_transactions existing osr(1.2e0 0x384d130)/0x384d130 2014-06-09 16:22:12.154051 7f868a757700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.154056 7f86a5e11700 5 _do_op 0xcd9d0e0 seq 4071 osr(2.2d9 0x2ed0c30)/0x2ed0c30 start 2014-06-09 16:22:12.154059 7f86a5e11700 10 _do_transaction on 0x3a875c0 2014-06-09 16:22:12.154065 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.154067 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.154073 7f8690f64700 5 queue_transactions existing osr(1.5c1 0x384dd90)/0x384dd90 2014-06-09 16:22:12.154075 7f8690f64700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.154080 7f86a5610700 10 _finish_op 0xc88b9e0 seq 4070 osr(3.786 0x3ed1d90)/0x3ed1d90 2014-06-09 16:22:12.154090 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77bd40 (1 active) 2014-06-09 16:22:12.154091 7f868bf5a700 10 op_submit_start 4118 2014-06-09 16:22:12.154092 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x292823a0] 2014-06-09 16:22:12.154093 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a480 (2 active) 2014-06-09 16:22:12.154094 7f868bf5a700 5 queue_transactions (trailing journal) 4118 2014-06-09 16:22:12.154095 7f868bf5a700 5 queue_op 0x3252b500 seq 4118 osr(0.5c3 0x2bd7a50) 828 bytes (queue has 50 ops and 4237296 bytes) 2014-06-09 16:22:12.154096 7f86a5610700 5 _do_op 0xcd9d200 seq 4072 osr(3.76f 0x3ed1e20)/0x3ed1e20 start 2014-06-09 16:22:12.154097 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x292823a0] 2014-06-09 16:22:12.154098 7f868bf5a700 10 op_submit_finish 4118 2014-06-09 16:22:12.154099 7f86a5610700 10 _do_transaction on 0xc258d80 2014-06-09 16:22:12.154100 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.154101 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.154104 7f868f761700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.154104 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.154108 7f868af58700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.154102 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.2a3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.2a3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.154113 7f8691f66700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.154117 7f8689f56700 2 waiting 51 > 50 ops || 4238157 > 104857600 2014-06-09 16:22:12.154133 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.154138 7f868bf5a700 5 queue_transactions existing osr(1.2a3 0x384d1f0)/0x384d1f0 2014-06-09 16:22:12.154141 7f868bf5a700 2 waiting 51 > 50 ops || 4238124 > 104857600 2014-06-09 16:22:12.154149 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.154165 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4071.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.154170 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.154185 7f86a5e11700 10 _do_op 0xcd9d0e0 seq 4071 r = 0, finisher 0x3281a120 0 2014-06-09 16:22:12.154196 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3206e440] 2014-06-09 16:22:12.154198 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3206e440] 2014-06-09 16:22:12.154200 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.154201 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.154204 7f86a5e11700 10 _finish_op 0xcd9d0e0 seq 4071 osr(2.2d9 0x2ed0c30)/0x2ed0c30 2014-06-09 16:22:12.154215 7f868ef60700 10 op_submit_start 4119 2014-06-09 16:22:12.154216 7f868ef60700 5 queue_transactions (trailing journal) 4119 2014-06-09 16:22:12.154218 7f868ef60700 5 queue_op 0x39d9920 seq 4119 osr(2.fd 0x2ed0770) 826 bytes (queue has 50 ops and 4237294 bytes) 2014-06-09 16:22:12.154229 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281a120] 2014-06-09 16:22:12.154233 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281a120] 2014-06-09 16:22:12.154234 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.154235 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.154238 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad7a0 (1 active) 2014-06-09 16:22:12.154240 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc7807e0 (2 active) 2014-06-09 16:22:12.154245 7f868ef60700 10 op_submit_finish 4119 2014-06-09 16:22:12.154249 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5f8( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.5f8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.154264 7f868e75f700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154271 7f8692767700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154276 7f868ff62700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154281 7f8691765700 2 waiting 51 > 50 ops || 4238152 > 104857600 2014-06-09 16:22:12.154284 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.154287 7f8689755700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154290 7f868ef60700 5 queue_transactions existing osr(2.5f8 0x2ed1db0)/0x2ed1db0 2014-06-09 16:22:12.154293 7f868ef60700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154297 7f8692f68700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154302 7f868df5e700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154307 7f868cf5c700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154311 7f868a757700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154315 7f868c75b700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154320 7f868af58700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154324 7f8691f66700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154328 7f8689f56700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154333 7f86a5e11700 5 _do_op 0x30a3b60 seq 4073 osr(1.11d 0x384c8b0)/0x384c8b0 start 2014-06-09 16:22:12.154335 7f8690f64700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154337 7f86a5e11700 10 _do_transaction on 0x35be900 2014-06-09 16:22:12.154342 7f868f761700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154345 7f868bf5a700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154348 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.154361 7f8690763700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154366 7f868b759700 10 op_submit_start 4120 2014-06-09 16:22:12.154368 7f868b759700 5 queue_transactions (trailing journal) 4120 2014-06-09 16:22:12.154369 7f868b759700 5 queue_op 0x159968a0 seq 4120 osr(3.4fe 0x3ed1730) 828 bytes (queue has 50 ops and 4237294 bytes) 2014-06-09 16:22:12.154373 7f868b759700 10 op_submit_finish 4120 2014-06-09 16:22:12.154378 7f868d75d700 10 op_submit_start 4121 2014-06-09 16:22:12.154381 7f868d75d700 5 queue_transactions (trailing journal) 4121 2014-06-09 16:22:12.154382 7f868d75d700 5 queue_op 0x16fd4de0 seq 4121 osr(1.67 0x384c5b0) 826 bytes (queue has 50 ops and 4237294 bytes) 2014-06-09 16:22:12.154386 7f868d75d700 10 op_submit_finish 4121 2014-06-09 16:22:12.154378 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.4c8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.4c8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.154390 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.1e8( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.1e8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.154413 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.154418 7f868b759700 5 queue_transactions existing osr(0.4c8 0x2bd7670)/0x2bd7670 2014-06-09 16:22:12.154421 7f868b759700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154425 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.154430 7f868d75d700 5 queue_transactions existing osr(1.1e8 0x384cdf0)/0x384cdf0 2014-06-09 16:22:12.154433 7f868d75d700 2 waiting 51 > 50 ops || 4238122 > 104857600 2014-06-09 16:22:12.154457 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.154469 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4073.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.154473 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.154486 7f86a5e11700 10 _do_op 0x30a3b60 seq 4073 r = 0, finisher 0x13965a80 0 2014-06-09 16:22:12.154533 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.154548 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4072.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.154553 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.154564 7f86a5610700 10 _do_op 0xcd9d200 seq 4072 r = 0, finisher 0x32079f00 0 2014-06-09 16:22:12.154573 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6f6e0] 2014-06-09 16:22:12.154576 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6f6e0] 2014-06-09 16:22:12.154577 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.154578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.154581 7f86a5e11700 10 _finish_op 0x30a3b60 seq 4073 osr(1.11d 0x384c8b0)/0x384c8b0 2014-06-09 16:22:12.154592 7f868e75f700 10 op_submit_start 4122 2014-06-09 16:22:12.154593 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13965a80] 2014-06-09 16:22:12.154595 7f868e75f700 5 queue_transactions (trailing journal) 4122 2014-06-09 16:22:12.154596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13965a80] 2014-06-09 16:22:12.154598 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.154597 7f868e75f700 5 queue_op 0x30d96000 seq 4122 osr(2.59f 0x2ed1830) 861 bytes (queue has 50 ops and 4237327 bytes) 2014-06-09 16:22:12.154599 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.154602 7f8692767700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154602 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc7807e0 (1 active) 2014-06-09 16:22:12.154605 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52eb40 (2 active) 2014-06-09 16:22:12.154607 7f868ff62700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154610 7f86a5e11700 5 _do_op 0xbf7ecc0 seq 4074 osr(0.61f 0x2bd7d30)/0x2bd7d30 start 2014-06-09 16:22:12.154613 7f86a5e11700 10 _do_transaction on 0xc128480 2014-06-09 16:22:12.154613 7f868e75f700 10 op_submit_finish 4122 2014-06-09 16:22:12.154619 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.154617 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.154631 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.088711 7 0.000374 2014-06-09 16:22:12.154643 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.154654 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.154660 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.154665 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.154675 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4074.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.154680 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.154676 7f868e75f700 5 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.154685 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.154693 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.154697 7f86a5e11700 10 _do_op 0xbf7ecc0 seq 4074 r = 0, finisher 0x3045bee0 0 2014-06-09 16:22:12.154705 7f86a5e11700 10 _finish_op 0xbf7ecc0 seq 4074 osr(0.61f 0x2bd7d30)/0x2bd7d30 2014-06-09 16:22:12.154706 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3045bd00] 2014-06-09 16:22:12.154709 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3045bd00] 2014-06-09 16:22:12.154711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.154712 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.154703 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.154716 7f8689755700 10 op_submit_start 4123 2014-06-09 16:22:12.154717 7f8689755700 5 queue_transactions (trailing journal) 4123 2014-06-09 16:22:12.154719 7f8689755700 5 queue_op 0x324b3320 seq 4123 osr(0.129 0x2bd66b0) 828 bytes (queue has 50 ops and 4237327 bytes) 2014-06-09 16:22:12.154715 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.154724 7f868ef60700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154729 7f8692f68700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154725 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.154733 7f868df5e700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154738 7f868cf5c700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154738 7f868e75f700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.154742 7f868a757700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154747 7f868c75b700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154751 7f868af58700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154756 7f8691f66700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154760 7f8689f56700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154756 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:183 2014-06-09 16:22:12.154764 7f8690f64700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154768 7f868f761700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154772 7f868bf5a700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154776 7f8690763700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154780 7f868b759700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154784 7f868d75d700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154788 7f868ff62700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154789 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.154793 7f8692767700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154793 7f868e75f700 5 queue_transactions existing osr(0.622 0x2bd7cf0)/0x2bd7cf0 2014-06-09 16:22:12.154796 7f868e75f700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154800 7f8691765700 2 waiting 51 > 50 ops || 4238185 > 104857600 2014-06-09 16:22:12.154802 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074f80] 2014-06-09 16:22:12.154805 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074f80] 2014-06-09 16:22:12.154806 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.154807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.154817 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3045bee0] 2014-06-09 16:22:12.154820 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3045bee0] 2014-06-09 16:22:12.154822 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.154823 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.154826 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52eb40 (1 active) 2014-06-09 16:22:12.154828 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcc60 (2 active) 2014-06-09 16:22:12.154834 7f8689755700 10 op_submit_finish 4123 2014-06-09 16:22:12.154838 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.1d4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:12.154869 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.154874 7f8689755700 5 queue_transactions existing osr(0.1d4 0x2bd6940)/0x2bd6940 2014-06-09 16:22:12.154876 7f8689755700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154880 7f86a5610700 10 _finish_op 0xcd9d200 seq 4072 osr(3.76f 0x3ed1e20)/0x3ed1e20 2014-06-09 16:22:12.154890 7f8692f68700 10 op_submit_start 4124 2014-06-09 16:22:12.154891 7f8692f68700 5 queue_transactions (trailing journal) 4124 2014-06-09 16:22:12.154893 7f8692f68700 5 queue_op 0xd327d40 seq 4124 osr(1.508 0x384db90) 828 bytes (queue has 50 ops and 4237327 bytes) 2014-06-09 16:22:12.154897 7f868df5e700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154901 7f868cf5c700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154904 7f868a757700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154908 7f868c75b700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154912 7f868af58700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154915 7f8691f66700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154919 7f8689f56700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154923 7f8690f64700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154927 7f868f761700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154930 7f868bf5a700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154934 7f8690763700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154938 7f868b759700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154942 7f868d75d700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154946 7f868ff62700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154949 7f8692767700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154959 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32079f00] 2014-06-09 16:22:12.154963 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32079f00] 2014-06-09 16:22:12.154964 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.154965 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.154967 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a480 (1 active) 2014-06-09 16:22:12.154969 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5247e0 (2 active) 2014-06-09 16:22:12.154968 7f868e75f700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.154973 7f8692f68700 10 op_submit_finish 4124 2014-06-09 16:22:12.154975 7f8691765700 2 waiting 51 > 50 ops || 4238185 > 104857600 2014-06-09 16:22:12.154980 7f8689755700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154985 7f868ef60700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.154977 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.375( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.375( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.155011 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.155016 7f8692f68700 5 queue_transactions existing osr(1.375 0x384d230)/0x384d230 2014-06-09 16:22:12.155018 7f8692f68700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.155022 7f86a5610700 5 _do_op 0x32833020 seq 4076 osr(3.58f 0x3ed1850)/0x3ed1850 start 2014-06-09 16:22:12.155025 7f86a5610700 10 _do_transaction on 0x3419380 2014-06-09 16:22:12.155030 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.155071 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.155085 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4076.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.155090 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.155101 7f86a5610700 10 _do_op 0x32833020 seq 4076 r = 0, finisher 0x324f9f80 0 2014-06-09 16:22:12.155109 7f86a5e11700 5 _do_op 0x2ea53a40 seq 4075 osr(0.456 0x2bd7530)/0x2bd7530 start 2014-06-09 16:22:12.155112 7f86a5e11700 10 _do_transaction on 0x390d680 2014-06-09 16:22:12.155117 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.155155 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.155169 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4075.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.155173 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.155181 7f86a5e11700 10 _do_op 0x2ea53a40 seq 4075 r = 0, finisher 0x32534f80 0 2014-06-09 16:22:12.155190 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5bc40] 2014-06-09 16:22:12.155192 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5bc40] 2014-06-09 16:22:12.155193 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.155194 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.155197 7f86a5610700 10 _finish_op 0x32833020 seq 4076 osr(3.58f 0x3ed1850)/0x3ed1850 2014-06-09 16:22:12.155207 7f868cf5c700 10 op_submit_start 4125 2014-06-09 16:22:12.155209 7f868cf5c700 5 queue_transactions (trailing journal) 4125 2014-06-09 16:22:12.155210 7f868cf5c700 5 queue_op 0x300619e0 seq 4125 osr(0.34e 0x2bd7020) 861 bytes (queue has 50 ops and 4237360 bytes) 2014-06-09 16:22:12.155214 7f868a757700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.155219 7f868c75b700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.155223 7f868af58700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.155227 7f8691f66700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.155232 7f8689f56700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.155236 7f8690f64700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155240 7f868f761700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155246 7f868bf5a700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155250 7f8690763700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155253 7f868b759700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155257 7f868d75d700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155261 7f868ff62700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155265 7f8692767700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155271 7f868e75f700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.155275 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f9f80] 2014-06-09 16:22:12.155278 7f8691765700 2 waiting 51 > 50 ops || 4238218 > 104857600 2014-06-09 16:22:12.155281 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f9f80] 2014-06-09 16:22:12.155282 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.155283 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.155283 7f8689755700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155286 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5247e0 (1 active) 2014-06-09 16:22:12.155288 7f868ef60700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155288 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407d40 (2 active) 2014-06-09 16:22:12.155293 7f868cf5c700 10 op_submit_finish 4125 2014-06-09 16:22:12.155293 7f8692f68700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155298 7f868df5e700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.155297 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.155312 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.155321 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.155329 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.155338 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.155346 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.155360 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.155369 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.155376 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.38c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.155386 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.155391 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.155410 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 active m=2] check_local 2014-06-09 16:22:12.155421 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 active m=2] requeue_ops 0x10c54200,0x10c54100,0x13023c00,0x13023b00,0x1c768800,0x13023a00,0x1804d900,0x1c768700,0x13023900,0x1804ca00,0x1c54bf00,0x13023800,0x14052f00,0x1c54be00,0x13023700,0xe2ac600,0x1c54bd00,0x1c54bc00,0x1b0b7100,0x1b0b7000,0x1b0b6f00 2014-06-09 16:22:12.155486 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 active m=2] needs_recovery primary has 2 missing 2014-06-09 16:22:12.155499 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 active m=2] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.155516 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 active m=2] publish_stats_to_osd 243:1122 2014-06-09 16:22:12.155528 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 active m=2] hit_set_clear 2014-06-09 16:22:12.155538 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.3b4( v 214'677 lc 214'675 (0'0,214'677] local-les=243 n=133 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=214'677 lcod 0'0 mlcod 0'0 active m=2] agent_stop 2014-06-09 16:22:12.155550 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.155555 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.155567 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.155574 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.155582 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.155589 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.155596 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.155608 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.155638 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.155645 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.3f9( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.155654 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.155659 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.155671 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.155679 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.155686 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.155693 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.155699 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.155711 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:12.155720 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.155728 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.155736 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.155741 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.155752 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.155760 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x21966700,0x21966600 2014-06-09 16:22:12.155775 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.155784 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.155792 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.155804 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:993 2014-06-09 16:22:12.155813 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.155821 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.60e( v 213'574 (0'0,213'574] local-les=243 n=109 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.155830 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.155834 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.155844 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.155851 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.155859 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.155866 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.155873 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.155885 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 2014-06-09 16:22:12.155893 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.155901 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.5ad( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.155909 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.155912 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.155923 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.155929 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.155935 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.155942 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.155949 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.155960 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.155968 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.155975 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.238( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.155983 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.155987 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.155998 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.156006 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.156014 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.156021 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.156029 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.156042 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:149 2014-06-09 16:22:12.156051 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.156059 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.74a( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.156068 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.156073 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.156084 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.156092 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.156100 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.156109 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.156117 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.156129 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.156138 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.156146 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.6c7( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.156155 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.156159 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.156171 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.156179 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.156187 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.156196 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.156204 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.156216 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:157 2014-06-09 16:22:12.156226 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.156233 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[1.359( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.156242 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.156246 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.156258 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.940155 7 0.006132 2014-06-09 16:22:12.156271 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.156282 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.156292 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.156303 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.156312 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.156320 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.156332 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.156343 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.156353 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.156366 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.156384 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.156419 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.156423 7f868cf5c700 5 queue_transactions existing osr(2.d 0x2ed02b0)/0x2ed02b0 2014-06-09 16:22:12.156426 7f868cf5c700 2 waiting 51 > 50 ops || 4238215 > 104857600 2014-06-09 16:22:12.156431 7f86a5610700 5 _do_op 0x324989c0 seq 4077 osr(1.3c3 0x384d530)/0x384d530 start 2014-06-09 16:22:12.156435 7f86a5610700 10 _do_transaction on 0xbfe72c0 2014-06-09 16:22:12.156441 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.156498 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.156513 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4077.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.156518 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.156532 7f86a5610700 10 _do_op 0x324989c0 seq 4077 r = 0, finisher 0x3042a320 0 2014-06-09 16:22:12.156546 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042a1c0] 2014-06-09 16:22:12.156548 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042a1c0] 2014-06-09 16:22:12.156549 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.156551 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.156560 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251d0a0] 2014-06-09 16:22:12.156562 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251d0a0] 2014-06-09 16:22:12.156563 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.156564 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.156568 7f86a5e11700 10 _finish_op 0x2ea53a40 seq 4075 osr(0.456 0x2bd7530)/0x2bd7530 2014-06-09 16:22:12.156579 7f868c75b700 10 op_submit_start 4126 2014-06-09 16:22:12.156581 7f868c75b700 5 queue_transactions (trailing journal) 4126 2014-06-09 16:22:12.156583 7f868c75b700 5 queue_op 0xc2ebb60 seq 4126 osr(2.412 0x2ed15d0) 861 bytes (queue has 50 ops and 4237393 bytes) 2014-06-09 16:22:12.156588 7f868af58700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156593 7f8691f66700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156598 7f8689f56700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156603 7f8690f64700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156607 7f868f761700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156612 7f868bf5a700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156617 7f8690763700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156621 7f868b759700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156626 7f868d75d700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156630 7f868ff62700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156635 7f8692767700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156642 7f868e75f700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156647 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32534f80] 2014-06-09 16:22:12.156650 7f8691765700 2 waiting 51 > 50 ops || 4238251 > 104857600 2014-06-09 16:22:12.156653 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32534f80] 2014-06-09 16:22:12.156655 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.156656 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.156655 7f8689755700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156659 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcc60 (1 active) 2014-06-09 16:22:12.156661 7f868ef60700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156662 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5247e0 (2 active) 2014-06-09 16:22:12.156666 7f8692f68700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156667 7f868c75b700 10 op_submit_finish 4126 2014-06-09 16:22:12.156670 7f868df5e700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.156675 7f868cf5c700 2 waiting 51 > 50 ops || 4238248 > 104857600 2014-06-09 16:22:12.156680 7f868a757700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156671 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.156685 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.004946 7 0.000401 2014-06-09 16:22:12.156697 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.156698 7f8678d96700 5 -- op tracker -- , seq: 3660, time: 2014-06-09 16:22:12.154414, event: header_read, op: osd_op(client.4577.0:38077 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1470464] 3.73fa29ff ack+ondisk+write e243) 2014-06-09 16:22:12.156707 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.156715 7f8678d96700 5 -- op tracker -- , seq: 3660, time: 2014-06-09 16:22:12.154416, event: throttled, op: osd_op(client.4577.0:38077 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1470464] 3.73fa29ff ack+ondisk+write e243) 2014-06-09 16:22:12.156718 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.156725 7f8678d96700 5 -- op tracker -- , seq: 3660, time: 2014-06-09 16:22:12.156670, event: all_read, op: osd_op(client.4577.0:38077 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1470464] 3.73fa29ff ack+ondisk+write e243) 2014-06-09 16:22:12.156729 7f868c75b700 5 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.156740 7f8678d96700 5 -- op tracker -- , seq: 3660, time: 0.000000, event: dispatched, op: osd_op(client.4577.0:38077 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1470464] 3.73fa29ff ack+ondisk+write e243) 2014-06-09 16:22:12.156738 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.156747 7f8678d96700 20 osd.26 243 should_share_map client.4577 172.16.96.220:0/1127735 243 2014-06-09 16:22:12.156746 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.156757 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.156770 7f8678d96700 15 osd.26 243 enqueue_op 0x2f097000 prio 63 cost 1470464 latency 0.002355 osd_op(client.4577.0:38077 rbd_data.11816b8b4567.00000000000019a6 [set-alloc-hint object_size 4194304 write_size 4194304,write 2723840~1470464] 3.73fa29ff ack+ondisk+write e243) v4 2014-06-09 16:22:12.156768 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.156778 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.156791 7f868c75b700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.156805 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.156839 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.156843 7f868c75b700 5 queue_transactions existing osr(0.230 0x2bd6a10)/0x2bd6a10 2014-06-09 16:22:12.156846 7f868c75b700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156851 7f86a5e11700 5 _do_op 0x2fcaef00 seq 4078 osr(3.58f 0x3ed1850)/0x3ed1850 start 2014-06-09 16:22:12.156854 7f86a5e11700 10 _do_transaction on 0xc1f6cc0 2014-06-09 16:22:12.156859 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.156906 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.156921 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4078.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.156925 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.156950 7f86a5e11700 10 _do_op 0x2fcaef00 seq 4078 r = 0, finisher 0xbd75ce0 0 2014-06-09 16:22:12.156960 7f86a5610700 10 _finish_op 0x324989c0 seq 4077 osr(1.3c3 0x384d530)/0x384d530 2014-06-09 16:22:12.156971 7f8691f66700 10 op_submit_start 4127 2014-06-09 16:22:12.156973 7f8691f66700 5 queue_transactions (trailing journal) 4127 2014-06-09 16:22:12.156974 7f8691f66700 5 queue_op 0x16096420 seq 4127 osr(1.207 0x384ccd0) 861 bytes (queue has 50 ops and 4237426 bytes) 2014-06-09 16:22:12.156980 7f8689f56700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.156984 7f8690f64700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156989 7f868f761700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156994 7f868bf5a700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.156998 7f8690763700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157003 7f868b759700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157007 7f868d75d700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157012 7f868ff62700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157016 7f8692767700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157022 7f868e75f700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157027 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042a320] 2014-06-09 16:22:12.157029 7f8691765700 2 waiting 51 > 50 ops || 4238284 > 104857600 2014-06-09 16:22:12.157034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042a320] 2014-06-09 16:22:12.157034 7f8689755700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157036 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.157037 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.157038 7f868ef60700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157040 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407d40 (1 active) 2014-06-09 16:22:12.157043 7f8692f68700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157043 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6525a0 (2 active) 2014-06-09 16:22:12.157047 7f868df5e700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157048 7f8691f66700 10 op_submit_finish 4127 2014-06-09 16:22:12.157051 7f868cf5c700 2 waiting 51 > 50 ops || 4238281 > 104857600 2014-06-09 16:22:12.157056 7f868a757700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157061 7f868c75b700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157065 7f868af58700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157052 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.5( v 214'624 (0'0,214'624] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) 2014-06-09 16:22:12.157071 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.3 activated and committed 2014-06-09 16:22:12.157083 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 201/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.157094 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.157111 7f8691f66700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.157127 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.5( v 214'624 lc 214'623 (0'0,214'624] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'624 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1029 2014-06-09 16:22:12.157146 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.157151 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.157164 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.981128 7 0.000378 2014-06-09 16:22:12.157176 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.157186 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.157196 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.157207 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.157215 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.157223 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.157234 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.157245 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.157254 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.157264 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.157273 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.157285 7f8691f66700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.157300 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.157332 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.157336 7f8691f66700 5 queue_transactions existing osr(1.54c 0x384dad0)/0x384dad0 2014-06-09 16:22:12.157338 7f8691f66700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157343 7f86a5610700 5 _do_op 0x30061f20 seq 4079 osr(2.28a 0x2ed0d20)/0x2ed0d20 start 2014-06-09 16:22:12.157347 7f86a5610700 10 _do_transaction on 0x3339ec0 2014-06-09 16:22:12.157354 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.157396 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.157415 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4079.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.157419 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.157439 7f86a5610700 10 _do_op 0x30061f20 seq 4079 r = 0, finisher 0xbd5d860 0 2014-06-09 16:22:12.157448 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5c7c0] 2014-06-09 16:22:12.157450 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5c7c0] 2014-06-09 16:22:12.157452 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.157453 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.157456 7f86a5e11700 10 _finish_op 0x2fcaef00 seq 4078 osr(3.58f 0x3ed1850)/0x3ed1850 2014-06-09 16:22:12.157466 7f8690f64700 10 op_submit_start 4128 2014-06-09 16:22:12.157468 7f8690f64700 5 queue_transactions (trailing journal) 4128 2014-06-09 16:22:12.157469 7f8690f64700 5 queue_op 0x30a23c0 seq 4128 osr(1.5c1 0x384dd90) 828 bytes (queue has 50 ops and 4237426 bytes) 2014-06-09 16:22:12.157475 7f868f761700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157479 7f868bf5a700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157484 7f8690763700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157488 7f868b759700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157493 7f868d75d700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157497 7f868ff62700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157502 7f8692767700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157513 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd75ce0] 2014-06-09 16:22:12.157512 7f868e75f700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157517 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd75ce0] 2014-06-09 16:22:12.157518 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.157519 7f8691765700 2 waiting 51 > 50 ops || 4238284 > 104857600 2014-06-09 16:22:12.157520 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.157523 7f8689755700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157523 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5247e0 (1 active) 2014-06-09 16:22:12.157525 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca49200 (2 active) 2014-06-09 16:22:12.157526 7f868ef60700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157530 7f8692f68700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157530 7f8690f64700 10 op_submit_finish 4128 2014-06-09 16:22:12.157533 7f868df5e700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157537 7f868cf5c700 2 waiting 51 > 50 ops || 4238281 > 104857600 2014-06-09 16:22:12.157540 7f868a757700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157544 7f868c75b700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157547 7f868af58700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157554 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358400 (19 active) 2014-06-09 16:22:12.157560 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.157565 7f8690f64700 10 osd.26 243 dequeue_op 0x32826700 prio 63 cost 516096 latency 0.338657 osd_op(client.4539.0:34391 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:12.157588 7f8690f64700 5 -- op tracker -- , seq: 3277, time: 2014-06-09 16:22:12.157583, event: reached_pg, op: osd_op(client.4539.0:34391 rbd_data.115b6b8b4567.00000000000018a9 [set-alloc-hint object_size 4194304 write_size 4194304,write 2506752~516096] 3.3ca15f4f RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.157613 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.157627 7f86a68f6700 10 osd.26 243 new session 0xcbdd800 con=0x30c3c2c0 addr=172.16.96.220:0/1123776 2014-06-09 16:22:12.157628 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 1 flushes_in_progress pending waiting for active on 0x32826700 2014-06-09 16:22:12.157638 7f8690f64700 10 osd.26 243 dequeue_op 0x32826700 finish 2014-06-09 16:22:12.157640 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.157663 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358400 (20 active) 2014-06-09 16:22:12.157666 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.73b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.73b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.157689 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.157693 7f8690f64700 5 queue_transactions existing osr(1.73b 0x2ed00c0)/0x2ed00c0 2014-06-09 16:22:12.157696 7f8690f64700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157700 7f8691f66700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157706 7f8689f56700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.157709 7f86a5e11700 5 _do_op 0x2fc8cf60 seq 4080 osr(3.5d1 0x3ed1b70)/0x3ed1b70 start 2014-06-09 16:22:12.157713 7f86a5e11700 10 _do_transaction on 0xc0e2f00 2014-06-09 16:22:12.157719 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.157763 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.157781 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4080.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.157785 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.157799 7f86a5e11700 10 _do_op 0x2fc8cf60 seq 4080 r = 0, finisher 0x1322c400 0 2014-06-09 16:22:12.157812 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32075720] 2014-06-09 16:22:12.157815 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.157828 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.157838 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1096 2014-06-09 16:22:12.157849 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32075720] 2014-06-09 16:22:12.157851 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.157852 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.157861 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd75ac0] 2014-06-09 16:22:12.157863 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.157872 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.157881 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:12.157890 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd75ac0] 2014-06-09 16:22:12.157892 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.157893 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.157896 7f86a5610700 10 _finish_op 0x30061f20 seq 4079 osr(2.28a 0x2ed0d20)/0x2ed0d20 2014-06-09 16:22:12.157908 7f868bf5a700 10 op_submit_start 4129 2014-06-09 16:22:12.157909 7f868bf5a700 5 queue_transactions (trailing journal) 4129 2014-06-09 16:22:12.157911 7f868bf5a700 5 queue_op 0x230b9d40 seq 4129 osr(1.2a3 0x384d1f0) 828 bytes (queue has 50 ops and 4237393 bytes) 2014-06-09 16:22:12.157916 7f8690763700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157920 7f868b759700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157925 7f868d75d700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157929 7f868ff62700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157934 7f8692767700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157940 7f868e75f700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157944 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5d860] 2014-06-09 16:22:12.157948 7f8691765700 2 waiting 51 > 50 ops || 4238251 > 104857600 2014-06-09 16:22:12.157948 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5d860] 2014-06-09 16:22:12.157950 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.157951 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.157952 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6525a0 (1 active) 2014-06-09 16:22:12.157953 7f8689755700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157955 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1158c0 (2 active) 2014-06-09 16:22:12.157959 7f868ef60700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157959 7f86a5610700 5 _do_op 0x2fc8c660 seq 4081 osr(2.65c 0x2ed1fb0)/0x2ed1fb0 start 2014-06-09 16:22:12.157961 7f868bf5a700 10 op_submit_finish 4129 2014-06-09 16:22:12.157962 7f86a5610700 10 _do_transaction on 0xc31cb40 2014-06-09 16:22:12.157963 7f8692f68700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157967 7f868df5e700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157968 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.157972 7f868cf5c700 2 waiting 51 > 50 ops || 4238248 > 104857600 2014-06-09 16:22:12.157965 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.157976 7f868a757700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157981 7f868c75b700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157985 7f868af58700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157979 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.157989 7f8690f64700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.157987 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.157994 7f8691f66700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157999 7f8689f56700 2 waiting 51 > 50 ops || 4238254 > 104857600 2014-06-09 16:22:12.157995 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.158003 7f868f761700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.158007 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.158003 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.158011 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.158022 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4081.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.158025 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.158024 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:12.158033 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.158039 7f86a5610700 10 _do_op 0x2fc8c660 seq 4081 r = 0, finisher 0xbd7ba00 0 2014-06-09 16:22:12.158040 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.158047 7f86a5e11700 10 _finish_op 0x2fc8cf60 seq 4080 osr(3.5d1 0x3ed1b70)/0x3ed1b70 2014-06-09 16:22:12.158049 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.158057 7f868b759700 10 op_submit_start 4130 2014-06-09 16:22:12.158058 7f868b759700 5 queue_transactions (trailing journal) 4130 2014-06-09 16:22:12.158059 7f868b759700 5 queue_op 0xc88b4a0 seq 4130 osr(0.4c8 0x2bd7670) 828 bytes (queue has 50 ops and 4237360 bytes) 2014-06-09 16:22:12.158064 7f868d75d700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158054 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.158068 7f868ff62700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158072 7f8692767700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158068 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.158077 7f868e75f700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.158081 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322c400] 2014-06-09 16:22:12.158083 7f8691765700 2 waiting 51 > 50 ops || 4238218 > 104857600 2014-06-09 16:22:12.158086 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322c400] 2014-06-09 16:22:12.158077 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x14053a00,0x16059f00,0x16059e00,0x16059d00,0x16059c00,0x1804d800 2014-06-09 16:22:12.158087 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.158088 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.158088 7f8689755700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158091 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49200 (1 active) 2014-06-09 16:22:12.158092 7f868ef60700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158093 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34965a0 (2 active) 2014-06-09 16:22:12.158097 7f8692f68700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158098 7f868b759700 10 op_submit_finish 4130 2014-06-09 16:22:12.158101 7f868df5e700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158106 7f868cf5c700 2 waiting 51 > 50 ops || 4238215 > 104857600 2014-06-09 16:22:12.158103 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.158101 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.9d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.158110 7f868a757700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.158115 7f868c75b700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.158111 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.158119 7f868af58700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.158124 7f8690f64700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158119 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.158128 7f8691f66700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.158132 7f8689f56700 2 waiting 51 > 50 ops || 4238221 > 104857600 2014-06-09 16:22:12.158134 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.158137 7f868f761700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158131 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1030 2014-06-09 16:22:12.158139 7f868b759700 5 queue_transactions existing osr(0.9d 0x2bd6490)/0x2bd6490 2014-06-09 16:22:12.158141 7f8690763700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158144 7f86a5e11700 5 _do_op 0x14dd61e0 seq 4082 osr(3.f6 0x3ed04b0)/0x3ed04b0 start 2014-06-09 16:22:12.158147 7f86a5e11700 10 _do_transaction on 0xc22a240 2014-06-09 16:22:12.158146 7f868b759700 2 waiting 51 > 50 ops || 4238186 > 104857600 2014-06-09 16:22:12.158141 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.158153 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.158149 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.158158 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.158163 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.40c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.158192 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.158191 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.158196 7f868bf5a700 5 queue_transactions existing osr(2.40c 0x2ed1250)/0x2ed1250 2014-06-09 16:22:12.158198 7f868bf5a700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158207 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4082.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.158211 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.158231 7f86a5e11700 10 _do_op 0x14dd61e0 seq 4082 r = 0, finisher 0x31456a20 0 2014-06-09 16:22:12.158241 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6f3e0] 2014-06-09 16:22:12.158243 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.158254 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.158262 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.158272 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6f3e0] 2014-06-09 16:22:12.158274 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.158275 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.158279 7f86a5610700 10 _finish_op 0x2fc8c660 seq 4081 osr(2.65c 0x2ed1fb0)/0x2ed1fb0 2014-06-09 16:22:12.158291 7f868ff62700 10 op_submit_start 4131 2014-06-09 16:22:12.158292 7f868ff62700 5 queue_transactions (trailing journal) 4131 2014-06-09 16:22:12.158294 7f868ff62700 5 queue_op 0x30a3860 seq 4131 osr(0.530 0x2bd78f0) 828 bytes (queue has 50 ops and 4237327 bytes) 2014-06-09 16:22:12.158299 7f8692767700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158305 7f868e75f700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7ba00] 2014-06-09 16:22:12.158311 7f8691765700 2 waiting 51 > 50 ops || 4238185 > 104857600 2014-06-09 16:22:12.158314 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7ba00] 2014-06-09 16:22:12.158315 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.158315 7f8689755700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158316 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.158320 7f868ef60700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158319 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1158c0 (1 active) 2014-06-09 16:22:12.158322 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89fc20 (2 active) 2014-06-09 16:22:12.158324 7f8692f68700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158327 7f868ff62700 10 op_submit_finish 4131 2014-06-09 16:22:12.158329 7f868df5e700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158333 7f868cf5c700 2 waiting 51 > 50 ops || 4238182 > 104857600 2014-06-09 16:22:12.158337 7f868a757700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158331 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.32f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.158341 7f868c75b700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158346 7f868af58700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158350 7f8690f64700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158354 7f8691f66700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158358 7f8689f56700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158363 7f868f761700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158365 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.158367 7f8690763700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158370 7f868ff62700 5 queue_transactions existing osr(1.32f 0x384d350)/0x384d350 2014-06-09 16:22:12.158371 7f868b759700 2 waiting 51 > 50 ops || 4238153 > 104857600 2014-06-09 16:22:12.158375 7f86a5610700 5 _do_op 0xc2d5e60 seq 4083 osr(0.479 0x2bd7490)/0x2bd7490 start 2014-06-09 16:22:12.158376 7f868bf5a700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158379 7f86a5610700 10 _do_transaction on 0xc24cf00 2014-06-09 16:22:12.158381 7f868d75d700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158385 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.158386 7f868ff62700 2 waiting 51 > 50 ops || 4238155 > 104857600 2014-06-09 16:22:12.158425 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.158442 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4083.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.158447 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.158459 7f86a5610700 10 _do_op 0xc2d5e60 seq 4083 r = 0, finisher 0xbfcfb00 0 2014-06-09 16:22:12.158471 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1b0c04e0] 2014-06-09 16:22:12.158474 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.158484 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.158493 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.158502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1b0c04e0] 2014-06-09 16:22:12.158504 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.158505 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.158514 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7b780] 2014-06-09 16:22:12.158517 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] _finish_recovery 2014-06-09 16:22:12.158527 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] purge_strays 2014-06-09 16:22:12.158538 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] publish_stats_to_osd 243:1117 2014-06-09 16:22:12.158550 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7b780] 2014-06-09 16:22:12.158552 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.158553 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.158557 7f86a5e11700 10 _finish_op 0x14dd61e0 seq 4082 osr(3.f6 0x3ed04b0)/0x3ed04b0 2014-06-09 16:22:12.158570 7f868e75f700 10 op_submit_start 4132 2014-06-09 16:22:12.158571 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31456a20] 2014-06-09 16:22:12.158573 7f868e75f700 5 queue_transactions (trailing journal) 4132 2014-06-09 16:22:12.158575 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31456a20] 2014-06-09 16:22:12.158574 7f868e75f700 5 queue_op 0x30d966c0 seq 4132 osr(0.622 0x2bd7cf0) 861 bytes (queue has 50 ops and 4237330 bytes) 2014-06-09 16:22:12.158577 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.158578 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.158579 7f8691765700 2 waiting 51 > 50 ops || 4238188 > 104857600 2014-06-09 16:22:12.158581 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34965a0 (1 active) 2014-06-09 16:22:12.158583 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6b40 (2 active) 2014-06-09 16:22:12.158584 7f8689755700 2 waiting 51 > 50 ops || 4238158 > 104857600 2014-06-09 16:22:12.158587 7f86a5e11700 5 _do_op 0x16fd5500 seq 4084 osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 start 2014-06-09 16:22:12.158589 7f868ef60700 2 waiting 51 > 50 ops || 4238158 > 104857600 2014-06-09 16:22:12.158590 7f86a5e11700 10 _do_transaction on 0xc14af00 2014-06-09 16:22:12.158593 7f8692f68700 2 waiting 51 > 50 ops || 4238158 > 104857600 2014-06-09 16:22:12.158596 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.158598 7f868df5e700 2 waiting 51 > 50 ops || 4238158 > 104857600 2014-06-09 16:22:12.158602 7f868cf5c700 2 waiting 51 > 50 ops || 4238185 > 104857600 2014-06-09 16:22:12.158606 7f868a757700 2 waiting 51 > 50 ops || 4238191 > 104857600 2014-06-09 16:22:12.158610 7f868c75b700 2 waiting 51 > 50 ops || 4238191 > 104857600 2014-06-09 16:22:12.158614 7f868af58700 2 waiting 51 > 50 ops || 4238191 > 104857600 2014-06-09 16:22:12.158620 7f868e75f700 10 op_submit_finish 4132 2014-06-09 16:22:12.158636 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.158623 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.158650 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4084.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.158643 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] check_local 2014-06-09 16:22:12.158655 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.158665 7f86a5e11700 10 _do_op 0x16fd5500 seq 4084 r = 0, finisher 0x2e0eca0 0 2014-06-09 16:22:12.158655 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 0x19fe1b00,0x1b0b6100,0x19fe1a00,0x1b0b6000,0x19fe1900,0x1b0b7a00,0x19fe1800,0x1b0b7900,0x19fe1700,0x19fe1600,0x1c769c00 2014-06-09 16:22:12.158674 7f86a5610700 10 _finish_op 0xc2d5e60 seq 4083 osr(0.479 0x2bd7490)/0x2bd7490 2014-06-09 16:22:12.158686 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfcfb00] 2014-06-09 16:22:12.158690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfcfb00] 2014-06-09 16:22:12.158692 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.158693 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.158696 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89fc20 (1 active) 2014-06-09 16:22:12.158698 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3587e0 (2 active) 2014-06-09 16:22:12.158701 7f86a5610700 5 _do_op 0x309a21e0 seq 4085 osr(1.a7 0x384c730)/0x384c730 start 2014-06-09 16:22:12.158703 7f86a5610700 10 _do_transaction on 0x2ed5800 2014-06-09 16:22:12.158696 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing 2014-06-09 16:22:12.158709 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.158709 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.158727 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1050 2014-06-09 16:22:12.158739 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear 2014-06-09 16:22:12.158748 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.158750 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] agent_stop 2014-06-09 16:22:12.158762 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.158763 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4085.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.158767 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.158768 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.158780 7f86a5610700 10 _do_op 0x309a21e0 seq 4085 r = 0, finisher 0x318c3820 0 2014-06-09 16:22:12.158780 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.158789 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d72a0] 2014-06-09 16:22:12.158792 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d72a0] 2014-06-09 16:22:12.158793 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.158794 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.158789 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.158797 7f86a5e11700 10 _finish_op 0x16fd5500 seq 4084 osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 2014-06-09 16:22:12.158798 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.158806 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0eca0] 2014-06-09 16:22:12.158811 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0eca0] 2014-06-09 16:22:12.158812 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.158813 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.158807 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.158816 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6b40 (1 active) 2014-06-09 16:22:12.158819 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524a20 (2 active) 2014-06-09 16:22:12.158815 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.158821 7f86a5e11700 5 _do_op 0x2e21b60 seq 4086 osr(0.58a 0x2bd7770)/0x2bd7770 start 2014-06-09 16:22:12.158824 7f86a5e11700 10 _do_transaction on 0xbe89200 2014-06-09 16:22:12.158829 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.158829 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:145 2014-06-09 16:22:12.158839 7f868e75f700 20 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.158847 7f868e75f700 20 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.158856 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.158869 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.158875 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40e90 (19 active) 2014-06-09 16:22:12.158883 7f868e75f700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.158882 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4086.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.158887 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.158897 7f86a5e11700 10 _do_op 0x2e21b60 seq 4086 r = 0, finisher 0x30d61620 0 2014-06-09 16:22:12.158887 7f868e75f700 10 osd.26 243 dequeue_op 0x2c06b000 prio 63 cost 8192 latency 0.259421 osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.158909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5a260] 2014-06-09 16:22:12.158911 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5a260] 2014-06-09 16:22:12.158913 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.158914 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.158912 7f868e75f700 5 -- op tracker -- , seq: 3335, time: 2014-06-09 16:22:12.158906, event: reached_pg, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.158921 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006dc00] 2014-06-09 16:22:12.158923 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006dc00] 2014-06-09 16:22:12.158925 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.158926 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.158929 7f86a5610700 10 _finish_op 0x309a21e0 seq 4085 osr(1.a7 0x384c730)/0x384c730 2014-06-09 16:22:12.158919 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.158938 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c3820] 2014-06-09 16:22:12.158933 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2c06b000 2014-06-09 16:22:12.158942 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c3820] 2014-06-09 16:22:12.158944 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.158945 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.158948 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3587e0 (1 active) 2014-06-09 16:22:12.158950 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fca20 (2 active) 2014-06-09 16:22:12.158952 7f86a5610700 5 _do_op 0x30a35c0 seq 4087 osr(1.71e 0x2ed00b0)/0x2ed00b0 start 2014-06-09 16:22:12.158955 7f86a5610700 10 _do_transaction on 0xc2e06c0 2014-06-09 16:22:12.158943 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.158960 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.158965 7f868e75f700 3 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4540.0:33543 was 213'587 2014-06-09 16:22:12.158993 7f868e75f700 10 osd.26 243 dequeue_op 0x2c06b000 finish 2014-06-09 16:22:12.158999 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.159008 7f8691765700 10 op_submit_start 4133 2014-06-09 16:22:12.159010 7f8691765700 5 queue_transactions (trailing journal) 4133 2014-06-09 16:22:12.159012 7f8691765700 5 queue_op 0x306075c0 seq 4133 osr(1.86 0x384c490) 858 bytes (queue has 48 ops and 4235673 bytes) 2014-06-09 16:22:12.159015 7f8691765700 10 op_submit_finish 4133 2014-06-09 16:22:12.159014 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4087.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.159018 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.159018 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.159037 7f86a5610700 10 _do_op 0x30a35c0 seq 4087 r = 0, finisher 0x324f9c80 0 2014-06-09 16:22:12.159031 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.102565 6 0.000333 2014-06-09 16:22:12.159046 7f86a5e11700 10 _finish_op 0x2e21b60 seq 4086 osr(0.58a 0x2bd7770)/0x2bd7770 2014-06-09 16:22:12.159044 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.159056 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30d61620] 2014-06-09 16:22:12.159059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30d61620] 2014-06-09 16:22:12.159061 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.159062 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.159054 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.159065 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524a20 (1 active) 2014-06-09 16:22:12.159068 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fdb00 (2 active) 2014-06-09 16:22:12.159070 7f86a5e11700 5 _do_op 0x30a26c0 seq 4088 osr(0.151 0x2bd65e0)/0x2bd65e0 start 2014-06-09 16:22:12.159073 7f86a5e11700 10 _do_transaction on 0xc2ae900 2014-06-09 16:22:12.159065 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.159079 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.159076 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.159085 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.159093 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.159104 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.159117 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.159115 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.159125 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.159132 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4088.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.159136 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.159138 7f8691765700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.159145 7f86a5e11700 10 _do_op 0x30a26c0 seq 4088 r = 0, finisher 0x31bffbe0 0 2014-06-09 16:22:12.159154 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7d5c0] 2014-06-09 16:22:12.159156 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7d5c0] 2014-06-09 16:22:12.159158 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.159159 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.159154 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.159162 7f86a5610700 10 _finish_op 0x30a35c0 seq 4087 osr(1.71e 0x2ed00b0)/0x2ed00b0 2014-06-09 16:22:12.159171 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f9c80] 2014-06-09 16:22:12.159175 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f9c80] 2014-06-09 16:22:12.159176 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.159177 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.159180 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fca20 (1 active) 2014-06-09 16:22:12.159183 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406d80 (2 active) 2014-06-09 16:22:12.159185 7f86a5610700 5 _do_op 0xc88acc0 seq 4089 osr(0.6c0 0x2bd7f50)/0x2bd7f50 start 2014-06-09 16:22:12.159188 7f86a5610700 10 _do_transaction on 0xbf59380 2014-06-09 16:22:12.159189 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.159192 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.159193 7f8691765700 5 queue_transactions existing osr(2.19 0x2ed0270)/0x2ed0270 2014-06-09 16:22:12.159196 7f8691765700 10 op_submit_start 4134 2014-06-09 16:22:12.159197 7f8691765700 5 queue_transactions (trailing journal) 4134 2014-06-09 16:22:12.159198 7f8691765700 5 queue_op 0x2fcafbc0 seq 4134 osr(2.19 0x2ed0270) 858 bytes (queue has 47 ops and 4234875 bytes) 2014-06-09 16:22:12.159201 7f8691765700 10 op_submit_finish 4134 2014-06-09 16:22:12.159203 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.159216 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.000381 7 0.000323 2014-06-09 16:22:12.159231 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.159227 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.159237 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.159245 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4089.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.159250 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.159247 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.159262 7f86a5610700 10 _do_op 0xc88acc0 seq 4089 r = 0, finisher 0x32480180 0 2014-06-09 16:22:12.159257 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.159265 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.159275 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d0300] 2014-06-09 16:22:12.159277 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d0300] 2014-06-09 16:22:12.159278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.159273 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.159280 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.159289 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31bffc00] 2014-06-09 16:22:12.159290 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31bffc00] 2014-06-09 16:22:12.159292 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.159293 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.159284 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.159296 7f86a5e11700 10 _finish_op 0x30a26c0 seq 4088 osr(0.151 0x2bd65e0)/0x2bd65e0 2014-06-09 16:22:12.159295 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.159302 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fdb00 (1 active) 2014-06-09 16:22:12.159304 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31bffbe0] 2014-06-09 16:22:12.159305 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3387e0 (2 active) 2014-06-09 16:22:12.159307 7f86a5e11700 5 _do_op 0x2ea53aa0 seq 4090 osr(1.517 0x384db70)/0x384db70 start 2014-06-09 16:22:12.159308 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31bffbe0] 2014-06-09 16:22:12.159310 7f86a5e11700 10 _do_transaction on 0xbfec840 2014-06-09 16:22:12.159310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.159311 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.159304 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.159315 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.159314 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.159323 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.159336 7f8691765700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.159354 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.159351 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.159368 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4090.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.159372 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.159383 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.159386 7f86a5e11700 10 _do_op 0x2ea53aa0 seq 4090 r = 0, finisher 0x3251d2e0 0 2014-06-09 16:22:12.159387 7f8691765700 5 queue_transactions existing osr(1.2aa 0x384d1d0)/0x384d1d0 2014-06-09 16:22:12.159390 7f8691765700 10 op_submit_start 4135 2014-06-09 16:22:12.159392 7f8691765700 5 queue_transactions (trailing journal) 4135 2014-06-09 16:22:12.159393 7f8691765700 5 queue_op 0x15996b40 seq 4135 osr(1.2aa 0x384d1d0) 861 bytes (queue has 47 ops and 4234908 bytes) 2014-06-09 16:22:12.159395 7f86a5610700 10 _finish_op 0xc88acc0 seq 4089 osr(0.6c0 0x2bd7f50)/0x2bd7f50 2014-06-09 16:22:12.159401 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406d80 (1 active) 2014-06-09 16:22:12.159404 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1d40 (2 active) 2014-06-09 16:22:12.159409 7f8691765700 10 op_submit_finish 4135 2014-06-09 16:22:12.159413 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.159426 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.054902 7 0.000351 2014-06-09 16:22:12.159439 7f8690f64700 10 op_submit_start 4136 2014-06-09 16:22:12.159441 7f8690f64700 5 queue_transactions (trailing journal) 4136 2014-06-09 16:22:12.159442 7f8690f64700 5 queue_op 0x30a3320 seq 4136 osr(1.73b 0x2ed00c0) 828 bytes (queue has 47 ops and 4234908 bytes) 2014-06-09 16:22:12.159437 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.159445 7f8690f64700 10 op_submit_finish 4136 2014-06-09 16:22:12.159447 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.159448 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/243 242/242/212) 2014-06-09 16:22:12.159458 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.159469 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.159477 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.159485 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.159490 7f8690f64700 5 queue_transactions existing osr(3.537 0x3ed1620)/0x3ed1620 2014-06-09 16:22:12.159485 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.159493 7f8690f64700 10 op_submit_start 4137 2014-06-09 16:22:12.159494 7f8690f64700 5 queue_transactions (trailing journal) 4137 2014-06-09 16:22:12.159495 7f8690f64700 5 queue_op 0x30a3da0 seq 4137 osr(3.537 0x3ed1620) 828 bytes (queue has 48 ops and 4235736 bytes) 2014-06-09 16:22:12.159498 7f8690f64700 10 op_submit_finish 4137 2014-06-09 16:22:12.159495 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.159501 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.5c1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.159506 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.159516 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.159529 7f8691765700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.159534 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.159538 7f8690f64700 5 queue_transactions existing osr(1.5c1 0x384dd90)/0x384dd90 2014-06-09 16:22:12.159541 7f8690f64700 10 op_submit_start 4138 2014-06-09 16:22:12.159542 7f8690f64700 5 queue_transactions (trailing journal) 4138 2014-06-09 16:22:12.159543 7f8690f64700 5 queue_op 0x30a2ae0 seq 4138 osr(1.5c1 0x384dd90) 828 bytes (queue has 49 ops and 4236564 bytes) 2014-06-09 16:22:12.159546 7f8690f64700 10 op_submit_finish 4138 2014-06-09 16:22:12.159545 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.159548 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.348( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.159578 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.159579 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.159582 7f8691765700 5 queue_transactions existing osr(0.21b 0x2bd6ab0)/0x2bd6ab0 2014-06-09 16:22:12.159584 7f8690f64700 5 queue_transactions existing osr(1.348 0x384d310)/0x384d310 2014-06-09 16:22:12.159585 7f8691765700 10 op_submit_start 4139 2014-06-09 16:22:12.159586 7f8691765700 5 queue_transactions (trailing journal) 4139 2014-06-09 16:22:12.159586 7f8690f64700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159587 7f8691765700 5 queue_op 0x1e08b440 seq 4139 osr(0.21b 0x2bd6ab0) 861 bytes (queue has 50 ops and 4237425 bytes) 2014-06-09 16:22:12.159590 7f8691765700 10 op_submit_finish 4139 2014-06-09 16:22:12.159591 7f8689f56700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.159597 7f868f761700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159593 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.159609 7f8690763700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159605 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.066104 7 0.000350 2014-06-09 16:22:12.159622 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.159632 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.159641 7f868b759700 2 waiting 51 > 50 ops || 4238251 > 104857600 2014-06-09 16:22:12.159647 7f868bf5a700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159642 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.159649 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32480180] 2014-06-09 16:22:12.159651 7f868d75d700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159655 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32480180] 2014-06-09 16:22:12.159656 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.159656 7f868ff62700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159657 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.159653 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.159002 7f868e75f700 5 -- op tracker -- , seq: 3335, time: 2014-06-09 16:22:12.158997, event: done, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.159661 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.159669 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.159677 7f868e75f700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.159678 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.159689 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.159698 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40e90 (20 active) 2014-06-09 16:22:12.159699 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.159702 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.24c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.159711 7f8691765700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.159727 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:186 2014-06-09 16:22:12.159737 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.159742 7f868e75f700 5 queue_transactions existing osr(1.24c 0x384cfd0)/0x384cfd0 2014-06-09 16:22:12.159744 7f868e75f700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159748 7f8691f66700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.159753 7f8689755700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159758 7f868ef60700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159759 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.159763 7f8692f68700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159764 7f8691765700 5 queue_transactions existing osr(0.678 0x2bd7e90)/0x2bd7e90 2014-06-09 16:22:12.159766 7f8691765700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.159773 7f868cf5c700 2 waiting 51 > 50 ops || 4238280 > 104857600 2014-06-09 16:22:12.159774 7f86a5610700 5 _do_op 0x16fd5260 seq 4091 osr(1.3cd 0x384d4b0)/0x384d4b0 start 2014-06-09 16:22:12.159778 7f86a5610700 10 _do_transaction on 0xc121e00 2014-06-09 16:22:12.159778 7f868a757700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.159783 7f868c75b700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.159783 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.159787 7f868af58700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.159791 7f868df5e700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159795 7f8692767700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159825 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.159839 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4091.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.159844 7f86a5610700 5 submit_transaction r = 0 2014-06-09 16:22:12.159867 7f86a5610700 10 _do_op 0x16fd5260 seq 4091 r = 0, finisher 0x3061fc20 0 2014-06-09 16:22:12.159876 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251c520] 2014-06-09 16:22:12.159878 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251c520] 2014-06-09 16:22:12.159879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.159880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.159883 7f86a5e11700 10 _finish_op 0x2ea53aa0 seq 4090 osr(1.517 0x384db70)/0x384db70 2014-06-09 16:22:12.159891 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3387e0 (1 active) 2014-06-09 16:22:12.159892 7f8690f64700 10 op_submit_start 4140 2014-06-09 16:22:12.159895 7f8690f64700 5 queue_transactions (trailing journal) 4140 2014-06-09 16:22:12.159894 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc6c0 (2 active) 2014-06-09 16:22:12.159897 7f86a5e11700 5 _do_op 0x324cd5c0 seq 4092 osr(2.298 0x2ed0c70)/0x2ed0c70 start 2014-06-09 16:22:12.159896 7f8690f64700 5 queue_op 0x30a2540 seq 4140 osr(1.348 0x384d310) 828 bytes (queue has 50 ops and 4237425 bytes) 2014-06-09 16:22:12.159899 7f8690f64700 10 op_submit_finish 4140 2014-06-09 16:22:12.159899 7f86a5e11700 10 _do_transaction on 0xbfcd080 2014-06-09 16:22:12.159906 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.159902 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.2ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.159934 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.159938 7f8690f64700 5 queue_transactions existing osr(1.2ef 0x384d0f0)/0x384d0f0 2014-06-09 16:22:12.159941 7f8690f64700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159945 7f8689f56700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.159946 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.159950 7f868f761700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159954 7f8690763700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159959 7f868b759700 2 waiting 51 > 50 ops || 4238251 > 104857600 2014-06-09 16:22:12.159961 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4092.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.159963 7f868bf5a700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159965 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.159967 7f868d75d700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159972 7f868ff62700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159976 7f868e75f700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159980 7f8691f66700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.159979 7f86a5e11700 10 _do_op 0x324cd5c0 seq 4092 r = 0, finisher 0x3042b1e0 0 2014-06-09 16:22:12.159985 7f8689755700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159989 7f868ef60700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159992 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc83e40] 2014-06-09 16:22:12.159994 7f8692f68700 2 waiting 51 > 50 ops || 4238253 > 104857600 2014-06-09 16:22:12.159995 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc83e40] 2014-06-09 16:22:12.159996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.159998 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.159998 7f8691765700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.160003 7f868cf5c700 2 waiting 51 > 50 ops || 4238280 > 104857600 2014-06-09 16:22:12.160006 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b9a0] 2014-06-09 16:22:12.160007 7f868a757700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.160008 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b9a0] 2014-06-09 16:22:12.160011 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.160011 7f868c75b700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.160013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.160016 7f868af58700 2 waiting 51 > 50 ops || 4238286 > 104857600 2014-06-09 16:22:12.160016 7f86a5610700 10 _finish_op 0x16fd5260 seq 4091 osr(1.3cd 0x384d4b0)/0x384d4b0 2014-06-09 16:22:12.160029 7f8690f64700 10 op_submit_start 4141 2014-06-09 16:22:12.160028 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1d40 (1 active) 2014-06-09 16:22:12.160030 7f8690f64700 5 queue_transactions (trailing journal) 4141 2014-06-09 16:22:12.160031 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f680 (2 active) 2014-06-09 16:22:12.160032 7f8690f64700 5 queue_op 0x2fcaea20 seq 4141 osr(1.2ef 0x384d0f0) 828 bytes (queue has 50 ops and 4237425 bytes) 2014-06-09 16:22:12.160035 7f8690f64700 10 op_submit_finish 4141 2014-06-09 16:22:12.160035 7f86a5610700 5 _do_op 0x300601e0 seq 4093 osr(3.27 0x3ed0340)/0x3ed0340 start 2014-06-09 16:22:12.160038 7f86a5610700 10 _do_transaction on 0xbf883c0 2014-06-09 16:22:12.160044 7f86a5610700 15 _remove 3.27_head/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 2014-06-09 16:22:12.160037 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.160048 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251d2e0,0x3061fc20] 2014-06-09 16:22:12.160053 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251d2e0,0x3061fc20] 2014-06-09 16:22:12.160055 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.160056 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.160051 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.160059 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.160068 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.160077 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.160085 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.160098 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:157 2014-06-09 16:22:12.160108 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.160117 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.160126 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160131 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160144 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160154 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160158 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160172 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.160183 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160193 7f86a5610700 10 lookup_strip_header done cid 3.27_head oid 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 2014-06-09 16:22:12.160187 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160201 7f86a5610700 20 remove_header removing 29931 cid 3.27_head oid 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 2014-06-09 16:22:12.160206 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160209 7f86a5610700 20 clear_header clearing seq 29931 2014-06-09 16:22:12.160210 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160223 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 2014-06-09 16:22:12.160235 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160239 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160252 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.160263 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160267 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160279 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160289 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160293 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160306 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.160317 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160321 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160334 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160343 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160348 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160359 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160368 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160372 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160384 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160393 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160397 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160409 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160419 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160423 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160437 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:12.160448 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160452 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160463 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160473 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160477 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160488 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160497 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160513 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358400 (19 active) 2014-06-09 16:22:12.160520 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.160524 7f8690f64700 10 osd.26 243 dequeue_op 0x2c06ad00 prio 63 cost 401408 latency 0.207053 osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] 2014-06-09 16:22:12.160548 7f8690f64700 5 -- op tracker -- , seq: 3363, time: 2014-06-09 16:22:12.160543, event: reached_pg, op: osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.160556 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.160570 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] handle_message: 0x2c06ad00 2014-06-09 16:22:12.160580 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] do_op osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.160598 7f8690f64700 3 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] do_op dup client.4541.0:31903 was 213'734 2014-06-09 16:22:12.160625 7f8690f64700 10 osd.26 243 dequeue_op 0x2c06ad00 finish 2014-06-09 16:22:12.160628 7f86a5610700 10 _remove 3.27_head/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 = 0 2014-06-09 16:22:12.160636 7f86a5610700 15 _create_collection 3.27_TEMP 2014-06-09 16:22:12.160634 7f8690f64700 5 -- op tracker -- , seq: 3363, time: 2014-06-09 16:22:12.160629, event: done, op: osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.160646 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.160666 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358400 (20 active) 2014-06-09 16:22:12.160674 7f86a5610700 20 lookup_strip_header failed to get strip_header cid COLLECTIONS oid 8e1f8b20/3.27_TEMP/head//-1 2014-06-09 16:22:12.160679 7f86a5610700 10 lookup_cached_header COLLECTIONS/8e1f8b20/3.27_TEMP/head//-1 r = -2 2014-06-09 16:22:12.160670 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160684 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.160696 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160701 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160711 7f86a5610700 20 lookup_strip_header failed to get strip_header cid COLLECTIONS oid 8e1f8b20/3.27_TEMP/head//-1 2014-06-09 16:22:12.160712 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160721 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160725 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160737 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160743 7f86a5610700 20 write_state seq is 30168 2014-06-09 16:22:12.160741 7f86a68f6700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.157947, event: header_read, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.160747 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160751 7f86a5610700 20 set_header setting 30167 cid COLLECTIONS oid 8e1f8b20/3.27_TEMP/head//-1 parent seq 0 2014-06-09 16:22:12.160751 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160754 7f86a68f6700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.157949, event: throttled, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.160765 7f86a5610700 10 _create_collection cid 3.27_TEMP r = 0 2014-06-09 16:22:12.160770 7f86a5610700 15 _remove 3.27_TEMP/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 2014-06-09 16:22:12.160771 7f86a68f6700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.160705, event: all_read, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.160765 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x1b0d6b00,0x1b0d6a00,0x16bd6800,0x1b0d6900,0x16bd6700,0x1c54a600,0x1c54a500,0x306b400,0x1c54a400,0x306b600,0x1c54a300,0x10c55f00,0x1b0d6800,0x1c54a200,0x11e62900,0x22009100,0x11e62800,0x11e62700,0x11e62600,0x11e62500 2014-06-09 16:22:12.160785 7f86a68f6700 5 -- op tracker -- , seq: 3661, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.160792 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.160813 7f86a68f6700 15 osd.26 243 enqueue_op 0x32827b00 prio 63 cost 516096 latency 0.002866 osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.160820 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.27_TEMP oid 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 2014-06-09 16:22:12.160823 7f86a5610700 10 lookup_cached_header 3.27_TEMP/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 r = -2 2014-06-09 16:22:12.160826 7f86a5610700 10 _remove 3.27_TEMP/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 failed to get header: r = -2 2014-06-09 16:22:12.160830 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160831 7f86a5610700 15 _touch 3.27_head/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 2014-06-09 16:22:12.160834 7f86a5610700 10 _touch 3.27_head/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 failed to get header: r = -2 2014-06-09 16:22:12.160835 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160837 7f86a5610700 0 error (22) Invalid argument not handled on operation 9 (4093.0.3, or op 3, counting from 0) 2014-06-09 16:22:12.160846 7f86a5610700 0 unexpected error code 2014-06-09 16:22:12.160848 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160858 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160862 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160875 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.160886 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160890 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160902 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160911 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160916 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160930 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.160940 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160944 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160956 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.160965 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.160969 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.160987 7f86a5e11700 10 _finish_op 0x324cd5c0 seq 4092 osr(2.298 0x2ed0c70)/0x2ed0c70 2014-06-09 16:22:12.160982 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x230dae00,0x230dad00,0x230dac00,0x230dab00,0x230daa00,0x270bf700,0x270bf600 2014-06-09 16:22:12.161000 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b1e0] 2014-06-09 16:22:12.161004 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b1e0] 2014-06-09 16:22:12.161006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.161007 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.161010 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc6c0 (1 active) 2014-06-09 16:22:12.161012 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161013 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34977a0 (2 active) 2014-06-09 16:22:12.161016 7f86a5e11700 5 _do_op 0x2ea52d20 seq 4094 osr(0.631 0x2bd7c70)/0x2bd7c70 start 2014-06-09 16:22:12.161019 7f86a5e11700 10 _do_transaction on 0xc2dc480 2014-06-09 16:22:12.161016 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161025 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.161028 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.161038 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161042 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161056 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0xe2ad800,0xe2ad700,0xe2ad600,0x11e63700,0xe2ad500,0x13023100,0x10c55e00,0x13023000,0x14052e00,0xe2ac500,0xe2ac700,0x2e075600,0x2f097800,0x2f097700,0x2f097600,0x2f097500,0x2f097400 2014-06-09 16:22:12.161069 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.161084 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4094.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.161089 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.161101 7f86a5e11700 10 _do_op 0x2ea52d20 seq 4094 r = 0, finisher 0x2d7399e0 0 2014-06-09 16:22:12.161108 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161111 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x281b6280] 2014-06-09 16:22:12.161114 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x281b6280] 2014-06-09 16:22:12.161115 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.161116 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.161120 7f86a5e11700 10 _finish_op 0x2ea52d20 seq 4094 osr(0.631 0x2bd7c70)/0x2bd7c70 2014-06-09 16:22:12.161112 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d7399e0] 2014-06-09 16:22:12.161124 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.161132 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d7399e0] 2014-06-09 16:22:12.161134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.161134 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.161137 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34977a0 (1 active) 2014-06-09 16:22:12.161140 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497440 (2 active) 2014-06-09 16:22:12.161142 7f86a5e11700 5 _do_op 0xcd9cd20 seq 4095 osr(3.e8 0x3ed0510)/0x3ed0510 start 2014-06-09 16:22:12.161145 7f86a5e11700 10 _do_transaction on 0xc0df440 2014-06-09 16:22:12.161138 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161150 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.161156 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161160 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161173 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 2014-06-09 16:22:12.161186 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161192 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.161189 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161207 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161208 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4095.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.161213 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.161211 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161224 7f86a5e11700 10 _do_op 0xcd9cd20 seq 4095 r = 0, finisher 0xbd750e0 0 2014-06-09 16:22:12.161224 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 2014-06-09 16:22:12.161235 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078800] 2014-06-09 16:22:12.161237 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161237 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078800] 2014-06-09 16:22:12.161239 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.161240 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.161244 7f86a5e11700 10 _finish_op 0xcd9cd20 seq 4095 osr(3.e8 0x3ed0510)/0x3ed0510 2014-06-09 16:22:12.161241 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161252 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd750e0] 2014-06-09 16:22:12.161257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd750e0] 2014-06-09 16:22:12.161258 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.161259 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.161255 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.161262 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497440 (1 active) 2014-06-09 16:22:12.161266 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161265 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338b40 (2 active) 2014-06-09 16:22:12.161268 7f86a5e11700 5 _do_op 0x324cca80 seq 4096 osr(2.69d 0x2ed1ef0)/0x2ed1ef0 start 2014-06-09 16:22:12.161271 7f86a5e11700 10 _do_transaction on 0xc05acc0 2014-06-09 16:22:12.161276 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.161270 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161281 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.161290 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161294 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161305 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.161314 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161316 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.161327 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358400 (19 active) 2014-06-09 16:22:12.161331 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4096.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.161333 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.161336 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.161347 7f86a5e11700 10 _do_op 0x324cca80 seq 4096 r = 0, finisher 0xbd79080 0 2014-06-09 16:22:12.161338 7f8690f64700 10 osd.26 243 dequeue_op 0xdfa6300 prio 63 cost 516096 latency 0.108603 osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:12.161358 7f86a5e11700 10 _finish_op 0x324cca80 seq 4096 osr(2.69d 0x2ed1ef0)/0x2ed1ef0 2014-06-09 16:22:12.161359 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006c960] 2014-06-09 16:22:12.161362 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006c960] 2014-06-09 16:22:12.161363 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.161359 7f8690f64700 5 -- op tracker -- , seq: 3617, time: 2014-06-09 16:22:12.161354, event: reached_pg, op: osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.161364 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.161367 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd79080] 2014-06-09 16:22:12.161372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd79080] 2014-06-09 16:22:12.161374 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.161375 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.161366 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.161378 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338b40 (1 active) 2014-06-09 16:22:12.161382 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb6a20 (2 active) 2014-06-09 16:22:12.161378 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xdfa6300 2014-06-09 16:22:12.161384 7f86a5e11700 5 _do_op 0x324c1f20 seq 4097 osr(0.1fd 0x2bd6b30)/0x2bd6b30 start 2014-06-09 16:22:12.161386 7f8690f64700 10 osd.26 243 dequeue_op 0xdfa6300 finish 2014-06-09 16:22:12.161387 7f86a5e11700 10 _do_transaction on 0xc14ba40 2014-06-09 16:22:12.161389 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.161392 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.161409 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358400 (20 active) 2014-06-09 16:22:12.161413 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161425 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.161433 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.161436 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161440 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.161447 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4097.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.161452 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.161452 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.161462 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161470 7f86a5e11700 10 _do_op 0x324c1f20 seq 4097 r = 0, finisher 0xc6e8740 0 2014-06-09 16:22:12.161466 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.161476 7f86a5e11700 10 _finish_op 0x324c1f20 seq 4097 osr(0.1fd 0x2bd6b30)/0x2bd6b30 2014-06-09 16:22:12.161477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e8ca0] 2014-06-09 16:22:12.161482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e8ca0] 2014-06-09 16:22:12.161484 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.161484 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8740] 2014-06-09 16:22:12.161485 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.161479 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.161489 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8740] 2014-06-09 16:22:12.161491 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.161492 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.161488 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.161495 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb6a20 (1 active) 2014-06-09 16:22:12.161498 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780000 (2 active) 2014-06-09 16:22:12.161500 7f86a5e11700 5 _do_op 0x324c1f80 seq 4098 osr(0.129 0x2bd66b0)/0x2bd66b0 start 2014-06-09 16:22:12.161496 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.161503 7f86a5e11700 10 _do_transaction on 0xc2c78c0 2014-06-09 16:22:12.161503 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.161509 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.161510 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.161521 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:12.161529 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.161536 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.161543 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161548 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.161547 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.161557 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.980216 7 0.003017 2014-06-09 16:22:12.161563 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4098.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.161567 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.161567 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.161578 7f86a5e11700 10 _do_op 0x324c1f80 seq 4098 r = 0, finisher 0xbd70460 0 2014-06-09 16:22:12.161576 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.161589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd980] 2014-06-09 16:22:12.161591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd980] 2014-06-09 16:22:12.161593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.161594 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.161586 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.161598 7f86a5e11700 10 _finish_op 0x324c1f80 seq 4098 osr(0.129 0x2bd66b0)/0x2bd66b0 2014-06-09 16:22:12.161599 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.161607 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd70460] 2014-06-09 16:22:12.161610 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd70460] 2014-06-09 16:22:12.161612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.161613 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.161608 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.161616 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780000 (1 active) 2014-06-09 16:22:12.161619 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb6a20 (2 active) 2014-06-09 16:22:12.161621 7f86a5e11700 5 _do_op 0x324c1320 seq 4099 osr(0.1fd 0x2bd6b30)/0x2bd6b30 start 2014-06-09 16:22:12.161617 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.161624 7f86a5e11700 10 _do_transaction on 0xc05e480 2014-06-09 16:22:12.161629 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.161628 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.161639 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.161649 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.161663 7f8690f64700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.161668 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.161683 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4099.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.161687 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.161680 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 2014-06-09 16:22:12.161696 7f86a5e11700 10 _do_op 0x324c1320 seq 4099 r = 0, finisher 0x324f8d60 0 2014-06-09 16:22:12.161707 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f8c60] 2014-06-09 16:22:12.161709 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f8c60] 2014-06-09 16:22:12.161711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.161712 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.161715 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161716 7f86a5e11700 10 _finish_op 0x324c1320 seq 4099 osr(0.1fd 0x2bd6b30)/0x2bd6b30 2014-06-09 16:22:12.161720 7f8690f64700 5 queue_transactions existing osr(0.44f 0x2bd71a0)/0x2bd71a0 2014-06-09 16:22:12.161723 7f8690f64700 10 op_submit_start 4142 2014-06-09 16:22:12.161724 7f8690f64700 5 queue_transactions (trailing journal) 4142 2014-06-09 16:22:12.161725 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f8d60] 2014-06-09 16:22:12.161725 7f8690f64700 5 queue_op 0xc2d4d20 seq 4142 osr(0.44f 0x2bd71a0) 861 bytes (queue has 44 ops and 4232492 bytes) 2014-06-09 16:22:12.161728 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f8d60] 2014-06-09 16:22:12.161732 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.161733 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.161736 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb6a20 (1 active) 2014-06-09 16:22:12.161739 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd8c0 (2 active) 2014-06-09 16:22:12.161744 7f86a5e11700 5 _do_op 0xbe7dce0 seq 4100 osr(1.66e 0x384ded0)/0x384ded0 start 2014-06-09 16:22:12.161747 7f86a5e11700 10 _do_transaction on 0xc22be00 2014-06-09 16:22:12.161746 7f8690f64700 10 op_submit_finish 4142 2014-06-09 16:22:12.161752 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.161750 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.161764 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.161773 7f8692767700 10 op_submit_start 4143 2014-06-09 16:22:12.161776 7f8692767700 5 queue_transactions (trailing journal) 4143 2014-06-09 16:22:12.161772 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.161778 7f8692767700 5 queue_op 0x327fd800 seq 4143 osr(0.66a 0x2bd7ef0) 828 bytes (queue has 45 ops and 4233320 bytes) 2014-06-09 16:22:12.161781 7f8692767700 10 op_submit_finish 4143 2014-06-09 16:22:12.161780 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.161788 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.161788 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.161785 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.654( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.161796 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.161801 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4100.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.161805 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.161808 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:12.161817 7f86a5e11700 10 _do_op 0xbe7dce0 seq 4100 r = 0, finisher 0x29283d40 0 2014-06-09 16:22:12.161817 7f8690f64700 20 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.161827 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9ea60] 2014-06-09 16:22:12.161829 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9ea60] 2014-06-09 16:22:12.161830 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.161831 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.161825 7f8690f64700 20 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.161834 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.161833 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.161834 7f86a5e11700 10 _finish_op 0xbe7dce0 seq 4100 osr(1.66e 0x384ded0)/0x384ded0 2014-06-09 16:22:12.161840 7f8692767700 5 queue_transactions existing osr(0.654 0x2bd7b50)/0x2bd7b50 2014-06-09 16:22:12.161841 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x29283d40] 2014-06-09 16:22:12.161843 7f8692767700 10 op_submit_start 4144 2014-06-09 16:22:12.161844 7f8692767700 5 queue_transactions (trailing journal) 4144 2014-06-09 16:22:12.161844 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x29283d40] 2014-06-09 16:22:12.161846 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.161847 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.161846 7f8692767700 5 queue_op 0x327fc420 seq 4144 osr(0.654 0x2bd7b50) 828 bytes (queue has 45 ops and 4233320 bytes) 2014-06-09 16:22:12.161838 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.161849 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd8c0 (1 active) 2014-06-09 16:22:12.161852 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287c20 (2 active) 2014-06-09 16:22:12.161856 7f8692767700 10 op_submit_finish 4144 2014-06-09 16:22:12.161851 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.023913 7 0.000360 2014-06-09 16:22:12.161864 7f8689f56700 10 op_submit_start 4145 2014-06-09 16:22:12.161866 7f8689f56700 5 queue_transactions (trailing journal) 4145 2014-06-09 16:22:12.161861 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.161868 7f8689f56700 5 queue_op 0xc2d5aa0 seq 4145 osr(2.3fa 0x2ed12f0) 861 bytes (queue has 46 ops and 4234181 bytes) 2014-06-09 16:22:12.161860 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 243/243 242/242/202) 2014-06-09 16:22:12.161871 7f8689f56700 10 op_submit_finish 4145 2014-06-09 16:22:12.161870 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.161876 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.161879 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.161892 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.161890 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.161897 7f8692767700 5 queue_transactions existing osr(3.f0 0x3ed04a0)/0x3ed04a0 2014-06-09 16:22:12.161899 7f8692767700 10 op_submit_start 4146 2014-06-09 16:22:12.161900 7f8692767700 5 queue_transactions (trailing journal) 4146 2014-06-09 16:22:12.161890 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.126743 8 0.000412 2014-06-09 16:22:12.161901 7f8692767700 5 queue_op 0x327fc060 seq 4146 osr(3.f0 0x3ed04a0) 826 bytes (queue has 47 ops and 4235007 bytes) 2014-06-09 16:22:12.161897 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.161904 7f8692767700 10 op_submit_finish 4146 2014-06-09 16:22:12.161904 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.161903 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.161907 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.112( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.161914 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.161914 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.161925 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.161925 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.161940 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.161933 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.161936 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.161944 7f8692767700 5 queue_transactions existing osr(2.112 0x2ed06d0)/0x2ed06d0 2014-06-09 16:22:12.161946 7f8692767700 10 op_submit_start 4147 2014-06-09 16:22:12.161947 7f8692767700 5 queue_transactions (trailing journal) 4147 2014-06-09 16:22:12.161948 7f8692767700 5 queue_op 0x327fd7a0 seq 4147 osr(2.112 0x2ed06d0) 828 bytes (queue has 48 ops and 4235835 bytes) 2014-06-09 16:22:12.161942 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.161951 7f8692767700 10 op_submit_finish 4147 2014-06-09 16:22:12.161945 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.161952 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.161954 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.161954 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.765( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.161963 7f8690f64700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.161966 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.161984 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.161978 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.161979 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.161988 7f8692767700 5 queue_transactions existing osr(0.765 0x384c360)/0x384c360 2014-06-09 16:22:12.161990 7f8692767700 10 op_submit_start 4148 2014-06-09 16:22:12.161991 7f8692767700 5 queue_transactions (trailing journal) 4148 2014-06-09 16:22:12.161992 7f8692767700 5 queue_op 0x327fc540 seq 4148 osr(0.765 0x384c360) 828 bytes (queue has 49 ops and 4236663 bytes) 2014-06-09 16:22:12.161988 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.161995 7f8692767700 10 op_submit_finish 4148 2014-06-09 16:22:12.162001 7f8689f56700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.161997 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 243/243 242/242/200) 2014-06-09 16:22:12.162009 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162012 7f8690f64700 5 queue_transactions existing osr(2.3bd 0x2ed1010)/0x2ed1010 2014-06-09 16:22:12.162015 7f8690f64700 10 op_submit_start 4149 2014-06-09 16:22:12.162015 7f8690f64700 5 queue_transactions (trailing journal) 4149 2014-06-09 16:22:12.162016 7f8690f64700 5 queue_op 0xc2d4000 seq 4149 osr(2.3bd 0x2ed1010) 861 bytes (queue has 50 ops and 4237524 bytes) 2014-06-09 16:22:12.162019 7f8690f64700 10 op_submit_finish 4149 2014-06-09 16:22:12.162017 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 2014-06-09 16:22:12.162026 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.162030 7f8692767700 5 queue_transactions existing osr(3.1d5 0x3ed09f0)/0x3ed09f0 2014-06-09 16:22:12.162021 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162032 7f8692767700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162037 7f86a5e11700 5 _do_op 0x32524720 seq 4101 osr(0.529 0x2bd7930)/0x2bd7930 start 2014-06-09 16:22:12.162040 7f86a5e11700 10 _do_transaction on 0x37e4300 2014-06-09 16:22:12.162035 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162045 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162045 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.162041 7f86a68f6700 5 -- op tracker -- , seq: 3662, time: 2014-06-09 16:22:12.160865, event: header_read, op: osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.162052 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.162056 7f8689f56700 5 queue_transactions existing osr(0.3f9 0x2bd7230)/0x2bd7230 2014-06-09 16:22:12.162049 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162059 7f8689f56700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.162060 7f86a68f6700 5 -- op tracker -- , seq: 3662, time: 2014-06-09 16:22:12.160866, event: throttled, op: osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.162063 7f868f761700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162061 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162070 7f868df5e700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162072 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162082 7f8690763700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162082 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.162076 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162087 7f868b759700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.162070 7f86a68f6700 5 -- op tracker -- , seq: 3662, time: 2014-06-09 16:22:12.161990, event: all_read, op: osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.162088 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162095 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4101.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.162096 7f86a68f6700 5 -- op tracker -- , seq: 3662, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.162098 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162098 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.162101 7f868d75d700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162102 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.162107 7f868ff62700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162112 7f868bf5a700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162103 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162115 7f86a5e11700 10 _do_op 0x32524720 seq 4101 r = 0, finisher 0x32074fe0 0 2014-06-09 16:22:12.162117 7f868e75f700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162122 7f8689755700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162116 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162124 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006c560] 2014-06-09 16:22:12.162119 7f86a68f6700 15 osd.26 243 enqueue_op 0xdfa7200 prio 63 cost 516096 latency 0.001254 osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.162126 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006c560] 2014-06-09 16:22:12.162127 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162127 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.162128 7f868ef60700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162128 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.162131 7f86a5e11700 10 _finish_op 0x32524720 seq 4101 osr(0.529 0x2bd7930)/0x2bd7930 2014-06-09 16:22:12.162133 7f8692f68700 2 waiting 51 > 50 ops || 4238352 > 104857600 2014-06-09 16:22:12.162139 7f8691765700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.162130 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162144 7f868cf5c700 2 waiting 51 > 50 ops || 4238379 > 104857600 2014-06-09 16:22:12.162143 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162150 7f868a757700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.162151 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162155 7f868c75b700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.162154 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162160 7f868af58700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.162162 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.162169 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162176 7f8689f56700 10 op_submit_start 4150 2014-06-09 16:22:12.162177 7f8689f56700 5 queue_transactions (trailing journal) 4150 2014-06-09 16:22:12.162177 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32074fe0] 2014-06-09 16:22:12.162179 7f8689f56700 5 queue_op 0xc2d43c0 seq 4150 osr(0.3f9 0x2bd7230) 861 bytes (queue has 50 ops and 4237557 bytes) 2014-06-09 16:22:12.162173 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162183 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32074fe0] 2014-06-09 16:22:12.162185 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.162184 7f868f761700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.162186 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.162188 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287c20 (1 active) 2014-06-09 16:22:12.162189 7f868df5e700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.162184 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.162191 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc120 (2 active) 2014-06-09 16:22:12.162193 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162194 7f8690763700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.162195 7f86a5e11700 5 _do_op 0x32524a20 seq 4102 osr(0.2a2 0x2bd6c20)/0x2bd6c20 start 2014-06-09 16:22:12.162197 7f86a5e11700 10 _do_transaction on 0xbdebd40 2014-06-09 16:22:12.162199 7f8689f56700 10 op_submit_finish 4150 2014-06-09 16:22:12.162199 7f868b759700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.162202 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.162197 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162203 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.162210 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162221 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162217 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.052888 7 0.004506 2014-06-09 16:22:12.162225 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.162228 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.162240 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.162237 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.162238 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.162245 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162253 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4102.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.162256 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.162250 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.162254 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.162266 7f86a5e11700 10 _do_op 0x32524a20 seq 4102 r = 0, finisher 0x320747a0 0 2014-06-09 16:22:12.162262 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.162261 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.162275 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074620] 2014-06-09 16:22:12.162269 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.162277 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074620] 2014-06-09 16:22:12.162270 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.162278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.162279 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.162282 7f86a5e11700 10 _finish_op 0x32524a20 seq 4102 osr(0.2a2 0x2bd6c20)/0x2bd6c20 2014-06-09 16:22:12.162279 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.162283 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.162292 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x320747a0] 2014-06-09 16:22:12.162295 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x320747a0] 2014-06-09 16:22:12.162296 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.162297 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.162294 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.162290 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.162300 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc120 (1 active) 2014-06-09 16:22:12.162302 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b680 (2 active) 2014-06-09 16:22:12.162304 7f86a5e11700 5 _do_op 0x2ea53b60 seq 4103 osr(0.2d9 0x2bd6e90)/0x2bd6e90 start 2014-06-09 16:22:12.162306 7f86a5e11700 10 _do_transaction on 0xc3f0600 2014-06-09 16:22:12.162301 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.162310 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162302 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.162311 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.162312 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.162314 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162326 7f8689f56700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.162326 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.162335 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162346 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.162338 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.162343 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 2014-06-09 16:22:12.162351 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162358 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4103.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.162362 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.162362 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162371 7f86a5e11700 10 _do_op 0x2ea53b60 seq 4103 r = 0, finisher 0xbd68d40 0 2014-06-09 16:22:12.162366 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.162378 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.162380 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc40dee0] 2014-06-09 16:22:12.162382 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc40dee0] 2014-06-09 16:22:12.162382 7f8689f56700 5 queue_transactions existing osr(1.6b7 0x2ed01d0)/0x2ed01d0 2014-06-09 16:22:12.162384 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.162385 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.162385 7f8689f56700 10 op_submit_start 4151 2014-06-09 16:22:12.162377 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.119200 7 0.000397 2014-06-09 16:22:12.162386 7f8689f56700 5 queue_transactions (trailing journal) 4151 2014-06-09 16:22:12.162388 7f86a5e11700 10 _finish_op 0x2ea53b60 seq 4103 osr(0.2d9 0x2bd6e90)/0x2bd6e90 2014-06-09 16:22:12.162388 7f8689f56700 5 queue_op 0xc2d46c0 seq 4151 osr(1.6b7 0x2ed01d0) 861 bytes (queue has 50 ops and 4237590 bytes) 2014-06-09 16:22:12.162388 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.162395 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd68d40] 2014-06-09 16:22:12.162398 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd68d40] 2014-06-09 16:22:12.162397 7f868f761700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.162399 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.162400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.162403 7f868ff62700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.162403 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b680 (1 active) 2014-06-09 16:22:12.162398 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.162405 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e6c0 (2 active) 2014-06-09 16:22:12.162408 7f868bf5a700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.162409 7f86a5e11700 5 _do_op 0x11fce4e0 seq 4104 osr(2.285 0x2ed0d50)/0x2ed0d50 start 2014-06-09 16:22:12.162411 7f86a5e11700 10 _do_transaction on 0xc28ed80 2014-06-09 16:22:12.162413 7f8689755700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.162407 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.162417 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.162418 7f868ef60700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.162417 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.162423 7f8692f68700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.162428 7f8689f56700 10 op_submit_finish 4151 2014-06-09 16:22:12.162425 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.162432 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.162442 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.162432 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.162453 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.162453 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.162455 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.998208 7 0.000485 2014-06-09 16:22:12.162462 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.162468 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4104.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.162472 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.162467 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.162479 7f8690f64700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.162477 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.162490 7f86a5e11700 10 _do_op 0x11fce4e0 seq 4104 r = 0, finisher 0x318c3880 0 2014-06-09 16:22:12.162488 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.162500 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5e220] 2014-06-09 16:22:12.162494 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.162498 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.162502 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.162507 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.162512 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.162516 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.162524 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.162521 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:164 2014-06-09 16:22:12.162528 7f8690f64700 5 queue_transactions existing osr(0.29 0x2bd62b0)/0x2bd62b0 2014-06-09 16:22:12.162530 7f8690f64700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.162530 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5e220] 2014-06-09 16:22:12.162532 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.162533 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.162526 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.162536 7f86a5e11700 10 _finish_op 0x11fce4e0 seq 4104 osr(2.285 0x2ed0d50)/0x2ed0d50 2014-06-09 16:22:12.162538 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.162546 7f868cf5c700 10 op_submit_start 4152 2014-06-09 16:22:12.162548 7f868cf5c700 5 queue_transactions (trailing journal) 4152 2014-06-09 16:22:12.162550 7f868cf5c700 5 queue_op 0x324e6780 seq 4152 osr(2.d 0x2ed02b0) 855 bytes (queue has 50 ops and 4237584 bytes) 2014-06-09 16:22:12.162547 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.162555 7f868a757700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.162559 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:12.162560 7f868c75b700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.162565 7f868af58700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.162570 7f8691f66700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.162574 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.162600 7f8692767700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162606 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.162609 7f868df5e700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162611 7f8689f56700 5 queue_transactions existing osr(0.5e4 0x2bd79d0)/0x2bd79d0 2014-06-09 16:22:12.162613 7f8690763700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162619 7f868b759700 2 waiting 51 > 50 ops || 4238410 > 104857600 2014-06-09 16:22:12.162625 7f868ff62700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162625 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c3880] 2014-06-09 16:22:12.162629 7f868bf5a700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162630 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c3880] 2014-06-09 16:22:12.162632 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.162633 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.162634 7f8689755700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162636 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e6c0 (1 active) 2014-06-09 16:22:12.162638 7f868ef60700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162639 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300a20 (2 active) 2014-06-09 16:22:12.162644 7f868cf5c700 10 op_submit_finish 4152 2014-06-09 16:22:12.162657 7f8692f68700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162650 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.162665 7f868e75f700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.162664 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.162677 7f8691765700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.162673 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162681 7f8689f56700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.162685 7f8690f64700 2 waiting 51 > 50 ops || 4238442 > 104857600 2014-06-09 16:22:12.162682 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.162691 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.162699 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.162713 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:12.162722 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.162730 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.162739 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.162744 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.162756 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.162764 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.162773 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.162781 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.162789 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.162801 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.162811 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.162818 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.162827 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.162852 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40b80 (19 active) 2014-06-09 16:22:12.162859 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.162863 7f868cf5c700 10 osd.26 243 dequeue_op 0x32827b00 prio 63 cost 516096 latency 0.004916 osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.162887 7f868cf5c700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.162882, event: reached_pg, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.162904 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.162918 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x32827b00 2014-06-09 16:22:12.162929 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.162947 7f868cf5c700 3 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4543.0:33160 was 213'552 2014-06-09 16:22:12.162974 7f868cf5c700 10 osd.26 243 dequeue_op 0x32827b00 finish 2014-06-09 16:22:12.162983 7f868cf5c700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.162977, event: done, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.162996 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.163019 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40b80 (20 active) 2014-06-09 16:22:12.163023 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.163035 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.023293 7 0.000477 2014-06-09 16:22:12.163047 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.163058 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.163068 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.163079 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.163087 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.163094 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.163103 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.163115 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.163124 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.163134 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.163143 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.163156 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.163172 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.163205 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.163209 7f868cf5c700 5 queue_transactions existing osr(1.3ef 0x384d3f0)/0x384d3f0 2014-06-09 16:22:12.163212 7f868cf5c700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.163217 7f868d75d700 10 op_submit_start 4153 2014-06-09 16:22:12.163218 7f868d75d700 5 queue_transactions (trailing journal) 4153 2014-06-09 16:22:12.163220 7f868d75d700 5 queue_op 0x16fd4c60 seq 4153 osr(1.1e8 0x384cdf0) 828 bytes (queue has 50 ops and 4237584 bytes) 2014-06-09 16:22:12.163223 7f868d75d700 10 op_submit_finish 4153 2014-06-09 16:22:12.163227 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.163261 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.163266 7f868d75d700 5 queue_transactions existing osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 2014-06-09 16:22:12.163269 7f868d75d700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.163274 7f868f761700 2 waiting 51 > 50 ops || 4238412 > 104857600 2014-06-09 16:22:12.163277 7f86a5e11700 5 _do_op 0xc2ea0c0 seq 4105 osr(3.d3 0x3ed0570)/0x3ed0570 start 2014-06-09 16:22:12.163281 7f86a5e11700 10 _do_transaction on 0xc31fbc0 2014-06-09 16:22:12.163288 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.163333 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.163350 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4105.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.163354 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.163371 7f86a5e11700 10 _do_op 0xc2ea0c0 seq 4105 r = 0, finisher 0x31434ea0 0 2014-06-09 16:22:12.163383 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6db40] 2014-06-09 16:22:12.163386 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.163397 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.163408 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1059 2014-06-09 16:22:12.163418 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6db40] 2014-06-09 16:22:12.163420 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.163421 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.163425 7f86a5e11700 10 _finish_op 0xc2ea0c0 seq 4105 osr(3.d3 0x3ed0570)/0x3ed0570 2014-06-09 16:22:12.163436 7f868c75b700 10 op_submit_start 4154 2014-06-09 16:22:12.163438 7f868c75b700 5 queue_transactions (trailing journal) 4154 2014-06-09 16:22:12.163440 7f868c75b700 5 queue_op 0xc34a6c0 seq 4154 osr(0.230 0x2bd6a10) 861 bytes (queue has 50 ops and 4237587 bytes) 2014-06-09 16:22:12.163445 7f868af58700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163450 7f8691f66700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163455 7f8692767700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163459 7f868df5e700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163465 7f8690763700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163471 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31434ea0] 2014-06-09 16:22:12.163472 7f868b759700 2 waiting 51 > 50 ops || 4238413 > 104857600 2014-06-09 16:22:12.163475 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31434ea0] 2014-06-09 16:22:12.163477 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.163478 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.163481 7f868ff62700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163481 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300a20 (1 active) 2014-06-09 16:22:12.163484 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339b00 (2 active) 2014-06-09 16:22:12.163486 7f868bf5a700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163488 7f868c75b700 10 op_submit_finish 4154 2014-06-09 16:22:12.163490 7f8689755700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163495 7f868ef60700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163500 7f8692f68700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163493 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.163506 7f868e75f700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163512 7f8691765700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163517 7f8689f56700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163508 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] exit Started/Primary/Active/Activating 1.052078 7 0.005100 2014-06-09 16:22:12.163522 7f8690f64700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.163522 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.163533 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] needs_recovery is recovered 2014-06-09 16:22:12.163545 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.163555 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.163565 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] finish_recovery 2014-06-09 16:22:12.163574 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] clear_recovery_state 2014-06-09 16:22:12.163585 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.163597 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'739 mlcod 214'739 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.163608 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 214'739 mlcod 214'739 active+clean] share_pg_info 2014-06-09 16:22:12.163636 7f868c75b700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.163653 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 214'739 mlcod 214'739 active+clean] publish_stats_to_osd 243:1239 2014-06-09 16:22:12.163689 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.163693 7f868c75b700 5 queue_transactions existing osr(3.224 0x3ed0c80)/0x3ed0c80 2014-06-09 16:22:12.163696 7f868c75b700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163700 7f868cf5c700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163705 7f868d75d700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163710 7f868f761700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163714 7f868a757700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163718 7f86a5e11700 5 _do_op 0xc2ea060 seq 4106 osr(0.633 0x2bd7c50)/0x2bd7c50 start 2014-06-09 16:22:12.163722 7f86a5e11700 10 _do_transaction on 0xc29b5c0 2014-06-09 16:22:12.163729 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.163771 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.163788 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4106.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.163793 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.163826 7f86a5e11700 10 _do_op 0xc2ea060 seq 4106 r = 0, finisher 0x189abd80 0 2014-06-09 16:22:12.163837 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcf8e0] 2014-06-09 16:22:12.163840 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.163850 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.163859 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:182 2014-06-09 16:22:12.163869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcf8e0] 2014-06-09 16:22:12.163870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.163872 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.163875 7f86a5e11700 10 _finish_op 0xc2ea060 seq 4106 osr(0.633 0x2bd7c50)/0x2bd7c50 2014-06-09 16:22:12.163887 7f8691f66700 10 op_submit_start 4155 2014-06-09 16:22:12.163889 7f8691f66700 5 queue_transactions (trailing journal) 4155 2014-06-09 16:22:12.163891 7f8691f66700 5 queue_op 0x16097080 seq 4155 osr(1.54c 0x384dad0) 861 bytes (queue has 50 ops and 4237587 bytes) 2014-06-09 16:22:12.163896 7f8692767700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163901 7f868df5e700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163906 7f8690763700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163912 7f868b759700 2 waiting 51 > 50 ops || 4238413 > 104857600 2014-06-09 16:22:12.163915 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189abd80] 2014-06-09 16:22:12.163920 7f868ff62700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163923 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189abd80] 2014-06-09 16:22:12.163924 7f868bf5a700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163925 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.163927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.163929 7f8689755700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163930 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339b00 (1 active) 2014-06-09 16:22:12.163933 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d680 (2 active) 2014-06-09 16:22:12.163934 7f868ef60700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163938 7f8691f66700 10 op_submit_finish 4155 2014-06-09 16:22:12.163939 7f8692f68700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163944 7f868e75f700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.163942 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.163951 7f8691765700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163957 7f8689f56700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.163963 7f8690f64700 2 waiting 51 > 50 ops || 4238445 > 104857600 2014-06-09 16:22:12.163955 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.995027 7 0.000282 2014-06-09 16:22:12.163968 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.163978 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.163988 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.163999 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.164008 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.164015 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.164026 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.164037 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.164047 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.164060 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.164076 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:163 2014-06-09 16:22:12.164109 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.164113 7f8691f66700 5 queue_transactions existing osr(1.25a 0x384cfb0)/0x384cfb0 2014-06-09 16:22:12.164116 7f8691f66700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.164120 7f868c75b700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.164125 7f868cf5c700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.164130 7f868d75d700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.164134 7f868f761700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.164138 7f868a757700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.164143 7f868af58700 2 waiting 51 > 50 ops || 4238448 > 104857600 2014-06-09 16:22:12.164147 7f86a5e11700 5 _do_op 0xc2ebb00 seq 4107 osr(1.b9 0x384c630)/0x384c630 start 2014-06-09 16:22:12.164151 7f86a5e11700 10 _do_transaction on 0xc2d8240 2014-06-09 16:22:12.164157 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.164198 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.164215 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4107.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.164220 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.164235 7f86a5e11700 10 _do_op 0xc2ebb00 seq 4107 r = 0, finisher 0x31457460 0 2014-06-09 16:22:12.164246 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3206fb80] 2014-06-09 16:22:12.164248 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.164259 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.164267 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:188 2014-06-09 16:22:12.164276 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3206fb80] 2014-06-09 16:22:12.164278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.164278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.164282 7f86a5e11700 10 _finish_op 0xc2ebb00 seq 4107 osr(1.b9 0x384c630)/0x384c630 2014-06-09 16:22:12.164293 7f868df5e700 10 op_submit_start 4156 2014-06-09 16:22:12.164294 7f868df5e700 5 queue_transactions (trailing journal) 4156 2014-06-09 16:22:12.164296 7f868df5e700 5 queue_op 0xbf7e000 seq 4156 osr(3.172 0x3ed06e0) 828 bytes (queue has 50 ops and 4237557 bytes) 2014-06-09 16:22:12.164301 7f8690763700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164308 7f868b759700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.164313 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31457460] 2014-06-09 16:22:12.164314 7f868ff62700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31457460] 2014-06-09 16:22:12.164318 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.164319 7f868bf5a700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164320 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.164323 7f8689755700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164323 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d680 (1 active) 2014-06-09 16:22:12.164327 7f868ef60700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164335 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f560 (2 active) 2014-06-09 16:22:12.164341 7f868df5e700 10 op_submit_finish 4156 2014-06-09 16:22:12.164340 7f8692f68700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164346 7f868e75f700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164351 7f8691765700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.164355 7f8689f56700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.164345 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:12.164360 7f8690f64700 2 waiting 51 > 50 ops || 4238415 > 104857600 2014-06-09 16:22:12.164382 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.164387 7f868df5e700 5 queue_transactions existing osr(3.400 0x3ed1260)/0x3ed1260 2014-06-09 16:22:12.164393 7f868df5e700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164399 7f8691f66700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.164404 7f868c75b700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.164408 7f868cf5c700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.164413 7f868d75d700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164417 7f868f761700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164422 7f868a757700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.164425 7f868af58700 2 waiting 51 > 50 ops || 4238418 > 104857600 2014-06-09 16:22:12.164429 7f8692767700 2 waiting 51 > 50 ops || 4238385 > 104857600 2014-06-09 16:22:12.164433 7f86a5e11700 5 _do_op 0x2c06e900 seq 4108 osr(3.505 0x3ed1700)/0x3ed1700 start 2014-06-09 16:22:12.164437 7f86a5e11700 10 _do_transaction on 0xc1120c0 2014-06-09 16:22:12.164443 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.164485 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.164502 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4108.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.164507 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.164520 7f86a5e11700 10 _do_op 0x2c06e900 seq 4108 r = 0, finisher 0x3050b520 0 2014-06-09 16:22:12.164531 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9e5e0] 2014-06-09 16:22:12.164534 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.164546 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.164557 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1224 2014-06-09 16:22:12.164567 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9e5e0] 2014-06-09 16:22:12.164569 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.164570 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.164574 7f86a5e11700 10 _finish_op 0x2c06e900 seq 4108 osr(3.505 0x3ed1700)/0x3ed1700 2014-06-09 16:22:12.164588 7f868b759700 10 op_submit_start 4157 2014-06-09 16:22:12.164591 7f868b759700 5 queue_transactions (trailing journal) 4157 2014-06-09 16:22:12.164590 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050b520] 2014-06-09 16:22:12.164594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050b520] 2014-06-09 16:22:12.164593 7f868b759700 5 queue_op 0xc88bce0 seq 4157 osr(0.9d 0x2bd6490) 826 bytes (queue has 50 ops and 4237522 bytes) 2014-06-09 16:22:12.164596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.164597 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.164598 7f868ff62700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164600 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f560 (1 active) 2014-06-09 16:22:12.164603 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7d40 (2 active) 2014-06-09 16:22:12.164604 7f868bf5a700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164607 7f86a5e11700 5 _do_op 0xc2ebec0 seq 4109 osr(0.ca 0x2bd6590)/0x2bd6590 start 2014-06-09 16:22:12.164610 7f86a5e11700 10 _do_transaction on 0xc0d9980 2014-06-09 16:22:12.164609 7f868b759700 10 op_submit_finish 4157 2014-06-09 16:22:12.164616 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.164614 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.c9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.164651 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.164655 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.164657 7f868b759700 5 queue_transactions existing osr(1.c9 0x384c9f0)/0x384c9f0 2014-06-09 16:22:12.164663 7f868b759700 2 waiting 51 > 50 ops || 4238348 > 104857600 2014-06-09 16:22:12.164668 7f8689755700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164673 7f868ef60700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164672 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4109.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.164677 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.164678 7f8692f68700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164684 7f868e75f700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164689 7f8691765700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.164691 7f86a5e11700 10 _do_op 0xc2ebec0 seq 4109 r = 0, finisher 0xbef1640 0 2014-06-09 16:22:12.164694 7f8689f56700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.164698 7f8690f64700 2 waiting 51 > 50 ops || 4238380 > 104857600 2014-06-09 16:22:12.164702 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd74e00] 2014-06-09 16:22:12.164704 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.164714 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.164724 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:12.164733 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd74e00] 2014-06-09 16:22:12.164735 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.164737 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.164740 7f868df5e700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164746 7f8691f66700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.164750 7f868c75b700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.164755 7f868cf5c700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.164759 7f868d75d700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164764 7f868f761700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164768 7f868a757700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.164772 7f868af58700 2 waiting 51 > 50 ops || 4238383 > 104857600 2014-06-09 16:22:12.164776 7f8692767700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164780 7f8690763700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164784 7f86a5e11700 10 _finish_op 0xc2ebec0 seq 4109 osr(0.ca 0x2bd6590)/0x2bd6590 2014-06-09 16:22:12.164794 7f868ff62700 10 op_submit_start 4158 2014-06-09 16:22:12.164796 7f868ff62700 5 queue_transactions (trailing journal) 4158 2014-06-09 16:22:12.164796 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef1640] 2014-06-09 16:22:12.164798 7f868ff62700 5 queue_op 0x30a2c60 seq 4158 osr(1.32f 0x384d350) 828 bytes (queue has 50 ops and 4237492 bytes) 2014-06-09 16:22:12.164800 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef1640] 2014-06-09 16:22:12.164802 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.164803 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.164804 7f868bf5a700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164806 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7d40 (1 active) 2014-06-09 16:22:12.164809 7f868b759700 2 waiting 51 > 50 ops || 4238318 > 104857600 2014-06-09 16:22:12.164809 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52eea0 (2 active) 2014-06-09 16:22:12.164813 7f8689755700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164814 7f86a5e11700 5 _do_op 0x2c06e0c0 seq 4110 osr(0.89 0x2bd64a0)/0x2bd64a0 start 2014-06-09 16:22:12.164817 7f86a5e11700 10 _do_transaction on 0xbff6a80 2014-06-09 16:22:12.164818 7f868ef60700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164823 7f868ff62700 10 op_submit_finish 4158 2014-06-09 16:22:12.164823 7f8692f68700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164824 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.164826 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.31f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.164865 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.164864 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.164870 7f868ff62700 5 queue_transactions existing osr(0.31f 0x2bd6d50)/0x2bd6d50 2014-06-09 16:22:12.164872 7f868ff62700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164877 7f868e75f700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164881 7f8691765700 2 waiting 51 > 50 ops || 4238353 > 104857600 2014-06-09 16:22:12.164880 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4110.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.164885 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.164886 7f8689f56700 2 waiting 51 > 50 ops || 4238353 > 104857600 2014-06-09 16:22:12.164891 7f8690f64700 2 waiting 51 > 50 ops || 4238350 > 104857600 2014-06-09 16:22:12.164898 7f868df5e700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164904 7f8691f66700 2 waiting 51 > 50 ops || 4238353 > 104857600 2014-06-09 16:22:12.164908 7f868c75b700 2 waiting 51 > 50 ops || 4238353 > 104857600 2014-06-09 16:22:12.164913 7f868cf5c700 2 waiting 51 > 50 ops || 4238353 > 104857600 2014-06-09 16:22:12.164917 7f868d75d700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164921 7f868f761700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164926 7f868a757700 2 waiting 51 > 50 ops || 4238353 > 104857600 2014-06-09 16:22:12.164930 7f868af58700 2 waiting 51 > 50 ops || 4238353 > 104857600 2014-06-09 16:22:12.164934 7f8692767700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164938 7f8690763700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.164953 7f86a5e11700 10 _do_op 0x2c06e0c0 seq 4110 r = 0, finisher 0xbd79ba0 0 2014-06-09 16:22:12.164963 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x18742140] 2014-06-09 16:22:12.164965 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.164975 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.164984 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.164993 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x18742140] 2014-06-09 16:22:12.164995 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.164996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.165000 7f86a5e11700 10 _finish_op 0x2c06e0c0 seq 4110 osr(0.89 0x2bd64a0)/0x2bd64a0 2014-06-09 16:22:12.165009 7f868bf5a700 10 op_submit_start 4159 2014-06-09 16:22:12.165012 7f868bf5a700 5 queue_transactions (trailing journal) 4159 2014-06-09 16:22:12.165012 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd79ba0] 2014-06-09 16:22:12.165014 7f868bf5a700 5 queue_op 0x324e76e0 seq 4159 osr(2.40c 0x2ed1250) 828 bytes (queue has 50 ops and 4237462 bytes) 2014-06-09 16:22:12.165019 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd79ba0] 2014-06-09 16:22:12.165019 7f868b759700 2 waiting 51 > 50 ops || 4238288 > 104857600 2014-06-09 16:22:12.165021 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.165022 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.165024 7f8689755700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165025 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52eea0 (1 active) 2014-06-09 16:22:12.165028 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e120 (2 active) 2014-06-09 16:22:12.165028 7f868ef60700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165032 7f86a5e11700 5 _do_op 0x2c06f080 seq 4111 osr(0.e2 0x2bd6560)/0x2bd6560 start 2014-06-09 16:22:12.165034 7f868bf5a700 10 op_submit_finish 4159 2014-06-09 16:22:12.165034 7f8692f68700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165035 7f86a5e11700 10 _do_transaction on 0x4734180 2014-06-09 16:22:12.165042 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.165037 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.165073 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.165074 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.165079 7f868bf5a700 5 queue_transactions existing osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 2014-06-09 16:22:12.165081 7f868bf5a700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165086 7f868ff62700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165086 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4111.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.165088 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.165091 7f868e75f700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165096 7f8691765700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.165097 7f86a5e11700 10 _do_op 0x2c06f080 seq 4111 r = 0, finisher 0x1322c820 0 2014-06-09 16:22:12.165100 7f8689f56700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.165105 7f8690f64700 2 waiting 51 > 50 ops || 4238320 > 104857600 2014-06-09 16:22:12.165106 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205cfc0] 2014-06-09 16:22:12.165110 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.165119 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.165128 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.165138 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205cfc0] 2014-06-09 16:22:12.165140 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.165141 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.165145 7f868df5e700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165150 7f8691f66700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.165155 7f868c75b700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.165160 7f868cf5c700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.165160 7f86a5e11700 10 _finish_op 0x2c06f080 seq 4111 osr(0.e2 0x2bd6560)/0x2bd6560 2014-06-09 16:22:12.165174 7f868f761700 10 op_submit_start 4160 2014-06-09 16:22:12.165176 7f868f761700 5 queue_transactions (trailing journal) 4160 2014-06-09 16:22:12.165177 7f868f761700 5 queue_op 0x2e21c20 seq 4160 osr(0.616 0x2bd7950) 828 bytes (queue has 50 ops and 4237432 bytes) 2014-06-09 16:22:12.165183 7f868a757700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165187 7f868af58700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165192 7f8692767700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165196 7f8690763700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165200 7f868d75d700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165211 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322c820] 2014-06-09 16:22:12.165214 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322c820] 2014-06-09 16:22:12.165216 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.165218 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.165221 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e120 (1 active) 2014-06-09 16:22:12.165223 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f9e0 (2 active) 2014-06-09 16:22:12.165229 7f868f761700 10 op_submit_finish 4160 2014-06-09 16:22:12.165232 7f8689755700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165239 7f868ef60700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165244 7f8692f68700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165250 7f868bf5a700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165250 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x153599f0 (19 active) 2014-06-09 16:22:12.165255 7f868ff62700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165258 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.165259 7f868e75f700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165264 7f8691765700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165268 7f8689f56700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165272 7f8690f64700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165263 7f868f761700 10 osd.26 243 dequeue_op 0x16bd6800 prio 63 cost 45056 latency 1.850850 osd_op(client.4564.0:36004 rbd_data.11746b8b4567.00000000000018a8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.c629f55d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:12.165289 7f868f761700 5 -- op tracker -- , seq: 262, time: 2014-06-09 16:22:12.165283, event: reached_pg, op: osd_op(client.4564.0:36004 rbd_data.11746b8b4567.00000000000018a8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.c629f55d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.165298 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.165312 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16bd6800 2014-06-09 16:22:12.165322 7f868f761700 10 osd.26 243 dequeue_op 0x16bd6800 finish 2014-06-09 16:22:12.165325 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.165348 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153599f0 (20 active) 2014-06-09 16:22:12.165353 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.4fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.165388 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.165394 7f868f761700 5 queue_transactions existing osr(1.4fb 0x384d7f0)/0x384d7f0 2014-06-09 16:22:12.165396 7f868f761700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165401 7f868df5e700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165405 7f8691f66700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165405 7f86a5e11700 5 _do_op 0xc88a1e0 seq 4112 osr(3.5f8 0x3ed1b30)/0x3ed1b30 start 2014-06-09 16:22:12.165411 7f868c75b700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165409 7f86a5e11700 10 _do_transaction on 0x390d140 2014-06-09 16:22:12.165421 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.165444 7f868cf5c700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165451 7f868b759700 2 waiting 51 > 50 ops || 4238258 > 104857600 2014-06-09 16:22:12.165462 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.165476 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4112.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.165481 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.165492 7f86a5e11700 10 _do_op 0xc88a1e0 seq 4112 r = 0, finisher 0xbd76640 0 2014-06-09 16:22:12.165502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf1c60] 2014-06-09 16:22:12.165505 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf1c60] 2014-06-09 16:22:12.165507 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.165508 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.165511 7f86a5e11700 10 _finish_op 0xc88a1e0 seq 4112 osr(3.5f8 0x3ed1b30)/0x3ed1b30 2014-06-09 16:22:12.165522 7f868af58700 10 op_submit_start 4161 2014-06-09 16:22:12.165524 7f868af58700 5 queue_transactions (trailing journal) 4161 2014-06-09 16:22:12.165526 7f868af58700 5 queue_op 0x230b9c80 seq 4161 osr(0.718 0x384c090) 861 bytes (queue has 50 ops and 4237465 bytes) 2014-06-09 16:22:12.165531 7f8692767700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165536 7f8690763700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165540 7f868d75d700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165546 7f8689755700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165551 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165552 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76640] 2014-06-09 16:22:12.165556 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76640] 2014-06-09 16:22:12.165556 7f8692f68700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.165558 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.165561 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165561 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f9e0 (1 active) 2014-06-09 16:22:12.165565 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcadbe60 (2 active) 2014-06-09 16:22:12.165566 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165569 7f868af58700 10 op_submit_finish 4161 2014-06-09 16:22:12.165570 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165575 7f8691765700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.165579 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.165573 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.165584 7f8690f64700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.165587 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.065761 7 0.000331 2014-06-09 16:22:12.165599 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.165610 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.165621 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.165633 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.165641 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.165649 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.165661 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.165672 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.165682 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.165696 7f868af58700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.165712 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 2014-06-09 16:22:12.165744 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.165748 7f868af58700 5 queue_transactions existing osr(2.67f 0x2ed1f50)/0x2ed1f50 2014-06-09 16:22:12.165751 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.165756 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165761 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165766 7f86a5e11700 5 _do_op 0x2ea529c0 seq 4113 osr(2.5f1 0x2ed1a10)/0x2ed1a10 start 2014-06-09 16:22:12.165770 7f86a5e11700 10 _do_transaction on 0xc0e3800 2014-06-09 16:22:12.165777 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.165801 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.165809 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.165827 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.165833 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.165841 7f868b759700 2 waiting 51 > 50 ops || 4238291 > 104857600 2014-06-09 16:22:12.165848 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.165843 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4113.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.165855 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.165871 7f86a5e11700 10 _do_op 0x2ea529c0 seq 4113 r = 0, finisher 0x30629f20 0 2014-06-09 16:22:12.165881 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050b0e0] 2014-06-09 16:22:12.165884 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.165894 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.165904 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 2014-06-09 16:22:12.165913 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050b0e0] 2014-06-09 16:22:12.165915 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.165916 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.165920 7f86a5e11700 10 _finish_op 0x2ea529c0 seq 4113 osr(2.5f1 0x2ed1a10)/0x2ed1a10 2014-06-09 16:22:12.165931 7f8690763700 10 op_submit_start 4162 2014-06-09 16:22:12.165933 7f8690763700 5 queue_transactions (trailing journal) 4162 2014-06-09 16:22:12.165934 7f8690763700 5 queue_op 0x3280ac00 seq 4162 osr(3.35c 0x3ed10c0) 828 bytes (queue has 50 ops and 4237432 bytes) 2014-06-09 16:22:12.165940 7f868d75d700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165946 7f8689755700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165950 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629f20] 2014-06-09 16:22:12.165951 7f868ef60700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165954 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629f20] 2014-06-09 16:22:12.165956 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.165956 7f8692f68700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165957 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.165960 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcadbe60 (1 active) 2014-06-09 16:22:12.165961 7f868bf5a700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165963 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338900 (2 active) 2014-06-09 16:22:12.165966 7f868ff62700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165968 7f8690763700 10 op_submit_finish 4162 2014-06-09 16:22:12.165971 7f868e75f700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.165975 7f8691765700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165979 7f8689f56700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.165984 7f8690f64700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.165985 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8fe40 (19 active) 2014-06-09 16:22:12.165992 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.165997 7f8690763700 10 osd.26 243 dequeue_op 0x2e075000 prio 63 cost 237568 latency 1.231416 osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.166021 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.166016, event: reached_pg, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.166029 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.166042 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2e075000 2014-06-09 16:22:12.166053 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.166078 7f8690763700 15 getattr 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 '_' 2014-06-09 16:22:12.166153 7f8690763700 10 getattr got.size() is 0 2014-06-09 16:22:12.166157 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 but can_create 2014-06-09 16:22:12.166172 7f8690763700 15 getattr 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 'snapset' 2014-06-09 16:22:12.166194 7f86a68f6700 5 -- op tracker -- , seq: 3663, time: 2014-06-09 16:22:12.162137, event: header_read, op: osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.166206 7f8690763700 10 getattr got.size() is 0 2014-06-09 16:22:12.166210 7f8690763700 15 getattr 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/snapdir//3 'snapset' 2014-06-09 16:22:12.166208 7f86a68f6700 5 -- op tracker -- , seq: 3663, time: 2014-06-09 16:22:12.162138, event: throttled, op: osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.166220 7f86a68f6700 5 -- op tracker -- , seq: 3663, time: 2014-06-09 16:22:12.166162, event: all_read, op: osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.166229 7f86a68f6700 5 -- op tracker -- , seq: 3663, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.166237 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.166242 7f8690763700 10 getattr got.size() is 0 2014-06-09 16:22:12.166252 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30acf340 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.166257 7f86a68f6700 15 osd.26 243 enqueue_op 0x2c06b000 prio 63 cost 1818624 latency 0.004120 osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.166265 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.166275 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.166285 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30acf340 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 rwstate(none n=0 w=0) oi: 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc3960 snapset: 0=[]:[] 2014-06-09 16:22:12.166299 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 @head oi=6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.166319 7f8690763700 15 getattr 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/snapdir//3 '_' 2014-06-09 16:22:12.166353 7f8690763700 10 getattr got.size() is 0 2014-06-09 16:22:12.166356 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/snapdir//3 and !can_create 2014-06-09 16:22:12.166374 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.166368, event: started, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.166384 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30647000 2014-06-09 16:22:12.166403 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.166397, event: started, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.166410 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] ov 0'0 av 243'534 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.166424 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 2014-06-09 16:22:12.166435 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.166451 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~237568 2014-06-09 16:22:12.166469 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229000 2014-06-09 16:22:12.166481 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 snapset=0x14dc3998 snapc=0=[] 2014-06-09 16:22:12.166492 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.166503 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.166510 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 0x30647000 op modify 2014-06-09 16:22:12.166522 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:30.043285 2014-06-09 16:22:12.166544 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.166565 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.166577 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 29 on osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.166594 7f8690763700 7 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 29 o 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.166620 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.166614, event: waiting for subops from 6, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.166637 7f8690763700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.166658 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'533], crt=0'0) [243'534 (0'0) modify 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 by client.4565.0:33578 2014-06-09 10:20:30.043285] 2014-06-09 16:22:12.166683 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'534 (0'0) modify 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 by client.4565.0:33578 2014-06-09 10:20:30.043285 2014-06-09 16:22:12.166698 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.166729 7f8690763700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'534, trimmed: 2014-06-09 16:22:12.166750 7f8690763700 5 queue_transactions existing osr(3.7f 0x3ed0200)/0x3ed0200 2014-06-09 16:22:12.166753 7f8690763700 2 waiting 51 > 50 ops || 4477388 > 104857600 2014-06-09 16:22:12.166758 7f868af58700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166762 7f868f761700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.166767 7f868df5e700 2 waiting 51 > 50 ops || 4238260 > 104857600 2014-06-09 16:22:12.166772 7f86a5e11700 5 _do_op 0xbf7f4a0 seq 4114 osr(1.3d6 0x384d470)/0x384d470 start 2014-06-09 16:22:12.166773 7f8691f66700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166776 7f86a5e11700 10 _do_transaction on 0x3472840 2014-06-09 16:22:12.166779 7f868c75b700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166789 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.166793 7f868cf5c700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166822 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.166833 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4114.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.166835 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.166838 7f86a68f6700 5 -- op tracker -- , seq: 3664, time: 2014-06-09 16:22:12.166279, event: header_read, op: osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.166852 7f86a5e11700 10 _do_op 0xbf7f4a0 seq 4114 r = 0, finisher 0xcd0efe0 0 2014-06-09 16:22:12.166850 7f86a68f6700 5 -- op tracker -- , seq: 3664, time: 2014-06-09 16:22:12.166280, event: throttled, op: osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.166857 7f86a5e11700 10 _finish_op 0xbf7f4a0 seq 4114 osr(1.3d6 0x384d470)/0x384d470 2014-06-09 16:22:12.166859 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9ec40] 2014-06-09 16:22:12.166862 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9ec40] 2014-06-09 16:22:12.166864 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.166865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.166860 7f86a68f6700 5 -- op tracker -- , seq: 3664, time: 2014-06-09 16:22:12.166813, event: all_read, op: osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.166868 7f868a757700 10 op_submit_start 4163 2014-06-09 16:22:12.166869 7f8692767700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166870 7f868a757700 5 queue_transactions (trailing journal) 4163 2014-06-09 16:22:12.166871 7f86a68f6700 5 -- op tracker -- , seq: 3664, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.166871 7f868a757700 5 queue_op 0xc64a360 seq 4163 osr(1.2e0 0x384d130) 861 bytes (queue has 50 ops and 4237465 bytes) 2014-06-09 16:22:12.166873 7f8689755700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166878 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166877 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.166882 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd0efe0] 2014-06-09 16:22:12.166892 7f868b759700 2 waiting 51 > 50 ops || 4238291 > 104857600 2014-06-09 16:22:12.166893 7f86a68f6700 15 osd.26 243 enqueue_op 0x2976d000 prio 63 cost 516096 latency 0.000613 osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.166897 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd0efe0] 2014-06-09 16:22:12.166900 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.166901 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.166903 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166905 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338900 (1 active) 2014-06-09 16:22:12.166908 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166908 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc360 (2 active) 2014-06-09 16:22:12.166912 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166915 7f868a757700 10 op_submit_finish 4163 2014-06-09 16:22:12.166916 7f8691765700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.166921 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.166919 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.166926 7f8690f64700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.166930 7f8690763700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.166930 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.166934 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.166935 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.166939 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166943 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166940 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.166948 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.166946 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.166955 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.166953 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.166959 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.166964 7f868d75d700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166970 7f8692f68700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.166968 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:12.166975 7f868a757700 20 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.166980 7f868a757700 20 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.166985 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.166990 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 243/243 242/242/222) 2014-06-09 16:22:12.167013 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.167016 7f868a757700 5 queue_transactions existing osr(3.761 0x3ed1e10)/0x3ed1e10 2014-06-09 16:22:12.167017 7f868a757700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.167021 7f86a5e11700 5 _do_op 0x2fc8c060 seq 4115 osr(0.29d 0x2bd6c40)/0x2bd6c40 start 2014-06-09 16:22:12.167023 7f86a5e11700 10 _do_transaction on 0xc14a6c0 2014-06-09 16:22:12.167028 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.167057 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.167073 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4115.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.167078 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.167088 7f86a5e11700 10 _do_op 0x2fc8c060 seq 4115 r = 0, finisher 0x3281ba00 0 2014-06-09 16:22:12.167093 7f86a5e11700 10 _finish_op 0x2fc8c060 seq 4115 osr(0.29d 0x2bd6c40)/0x2bd6c40 2014-06-09 16:22:12.167094 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfce5a0] 2014-06-09 16:22:12.167104 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281ba00] 2014-06-09 16:22:12.167099 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.167109 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281ba00] 2014-06-09 16:22:12.167111 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.167112 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.167110 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.167115 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc360 (1 active) 2014-06-09 16:22:12.167117 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca49320 (2 active) 2014-06-09 16:22:12.167119 7f86a5e11700 5 _do_op 0x14dd6840 seq 4116 osr(1.109 0x384c950)/0x384c950 start 2014-06-09 16:22:12.167122 7f86a5e11700 10 _do_transaction on 0xc2d8a80 2014-06-09 16:22:12.167119 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.167127 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfce5a0] 2014-06-09 16:22:12.167127 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.167129 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.167130 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.167135 7f8692767700 10 op_submit_start 4164 2014-06-09 16:22:12.167137 7f8692767700 5 queue_transactions (trailing journal) 4164 2014-06-09 16:22:12.167139 7f8692767700 5 queue_op 0xd471020 seq 4164 osr(3.1d5 0x3ed09f0) 828 bytes (queue has 50 ops and 4237432 bytes) 2014-06-09 16:22:12.167142 7f8692767700 10 op_submit_finish 4164 2014-06-09 16:22:12.167146 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.167166 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.167160 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.123272 8 0.007437 2014-06-09 16:22:12.167172 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.167182 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4116.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.167185 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.167182 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.167196 7f86a5e11700 10 _do_op 0x14dd6840 seq 4116 r = 0, finisher 0x3042b2e0 0 2014-06-09 16:22:12.167192 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.167206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189abb00] 2014-06-09 16:22:12.167202 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.167210 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.167209 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.167217 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.167217 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.167225 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.167233 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189abb00] 2014-06-09 16:22:12.167235 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.167236 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.167229 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.167239 7f86a5e11700 10 _finish_op 0x14dd6840 seq 4116 osr(1.109 0x384c950)/0x384c950 2014-06-09 16:22:12.167239 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.167247 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b2e0] 2014-06-09 16:22:12.167251 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b2e0] 2014-06-09 16:22:12.167252 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.167253 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.167249 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.167256 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49320 (1 active) 2014-06-09 16:22:12.167258 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc923b00 (2 active) 2014-06-09 16:22:12.167260 7f86a5e11700 5 _do_op 0x30a33e0 seq 4117 osr(3.468 0x3ed1590)/0x3ed1590 start 2014-06-09 16:22:12.167262 7f86a5e11700 10 _do_transaction on 0x3562300 2014-06-09 16:22:12.167262 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.167267 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.167279 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:12.167311 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.167315 7f8692767700 5 queue_transactions existing osr(2.25c 0x2ed0a30)/0x2ed0a30 2014-06-09 16:22:12.167318 7f8692767700 10 op_submit_start 4165 2014-06-09 16:22:12.167319 7f8692767700 5 queue_transactions (trailing journal) 4165 2014-06-09 16:22:12.167320 7f8692767700 5 queue_op 0x24105ce0 seq 4165 osr(2.25c 0x2ed0a30) 861 bytes (queue has 50 ops and 4237432 bytes) 2014-06-09 16:22:12.167323 7f8692767700 10 op_submit_finish 4165 2014-06-09 16:22:12.167326 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.167342 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.167339 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.998569 7 0.000592 2014-06-09 16:22:12.167350 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.167357 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4117.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.167360 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.167359 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.167372 7f86a5e11700 10 _do_op 0x30a33e0 seq 4117 r = 0, finisher 0x2d739020 0 2014-06-09 16:22:12.167368 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.167381 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc9ae0] 2014-06-09 16:22:12.167384 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc9ae0] 2014-06-09 16:22:12.167378 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.167385 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.167386 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.167389 7f86a5e11700 10 _finish_op 0x30a33e0 seq 4117 osr(3.468 0x3ed1590)/0x3ed1590 2014-06-09 16:22:12.167386 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.167397 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d739020] 2014-06-09 16:22:12.167393 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.167400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d739020] 2014-06-09 16:22:12.167401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.167402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.167405 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc923b00 (1 active) 2014-06-09 16:22:12.167407 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652a20 (2 active) 2014-06-09 16:22:12.167409 7f86a5e11700 5 _do_op 0x3252b500 seq 4118 osr(0.5c3 0x2bd7a50)/0x2bd7a50 start 2014-06-09 16:22:12.167411 7f86a5e11700 10 _do_transaction on 0xbf4e9c0 2014-06-09 16:22:12.167404 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.167416 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.167414 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.167423 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.167434 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 2014-06-09 16:22:12.167450 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.167481 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.167482 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.167485 7f8692767700 5 queue_transactions existing osr(2.3fb 0x2ed12b0)/0x2ed12b0 2014-06-09 16:22:12.167488 7f8692767700 10 op_submit_start 4166 2014-06-09 16:22:12.167489 7f8692767700 5 queue_transactions (trailing journal) 4166 2014-06-09 16:22:12.167490 7f8692767700 5 queue_op 0x308ca000 seq 4166 osr(2.3fb 0x2ed12b0) 861 bytes (queue has 50 ops and 4237465 bytes) 2014-06-09 16:22:12.167492 7f8692767700 10 op_submit_finish 4166 2014-06-09 16:22:12.167496 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4118.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.167499 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.167495 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.167511 7f86a5e11700 10 _do_op 0x3252b500 seq 4118 r = 0, finisher 0x3251c260 0 2014-06-09 16:22:12.167520 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f9d00] 2014-06-09 16:22:12.167525 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f9d00] 2014-06-09 16:22:12.167526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.167508 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] exit Started/Primary/Active/Activating 1.099836 7 0.000588 2014-06-09 16:22:12.167528 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.167530 7f86a5e11700 10 _finish_op 0x3252b500 seq 4118 osr(0.5c3 0x2bd7a50)/0x2bd7a50 2014-06-09 16:22:12.167530 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.167538 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251c260] 2014-06-09 16:22:12.167541 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251c260] 2014-06-09 16:22:12.167543 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.167544 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.167546 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652a20 (1 active) 2014-06-09 16:22:12.167541 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] needs_recovery is recovered 2014-06-09 16:22:12.167549 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358b40 (2 active) 2014-06-09 16:22:12.167550 7f86a5e11700 5 _do_op 0x39d9920 seq 4119 osr(2.fd 0x2ed0770)/0x2ed0770 start 2014-06-09 16:22:12.167553 7f86a5e11700 10 _do_transaction on 0xc316540 2014-06-09 16:22:12.167558 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.167551 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.167562 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.167571 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] finish_recovery 2014-06-09 16:22:12.167579 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] clear_recovery_state 2014-06-09 16:22:12.167595 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.167590 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.167608 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4119.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.167601 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.167612 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.167623 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] share_pg_info 2014-06-09 16:22:12.167637 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 2014-06-09 16:22:12.167651 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] publish_stats_to_osd 243:978 2014-06-09 16:22:12.167658 7f86a5e11700 10 _do_op 0x39d9920 seq 4119 r = 0, finisher 0x31bffa40 0 2014-06-09 16:22:12.167668 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31bffb60] 2014-06-09 16:22:12.167670 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31bffb60] 2014-06-09 16:22:12.167671 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.167672 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.167675 7f86a5e11700 10 _finish_op 0x39d9920 seq 4119 osr(2.fd 0x2ed0770)/0x2ed0770 2014-06-09 16:22:12.167684 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.167683 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31bffa40] 2014-06-09 16:22:12.167688 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31bffa40] 2014-06-09 16:22:12.167688 7f8692767700 5 queue_transactions existing osr(3.647 0x3ed1a70)/0x3ed1a70 2014-06-09 16:22:12.167690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.167690 7f8692767700 10 op_submit_start 4167 2014-06-09 16:22:12.167691 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.167691 7f8692767700 5 queue_transactions (trailing journal) 4167 2014-06-09 16:22:12.167693 7f8692767700 5 queue_op 0xc705380 seq 4167 osr(3.647 0x3ed1a70) 861 bytes (queue has 49 ops and 4236672 bytes) 2014-06-09 16:22:12.167693 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358b40 (1 active) 2014-06-09 16:22:12.167696 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5560 (2 active) 2014-06-09 16:22:12.167701 7f8692767700 10 op_submit_finish 4167 2014-06-09 16:22:12.167702 7f868ef60700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167708 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.167705 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.167717 7f868bf5a700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167722 7f868ff62700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167718 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.167726 7f868e75f700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167732 7f8691765700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.167727 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.167737 7f8689f56700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.167742 7f8690f64700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.167735 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.167747 7f8690763700 2 waiting 51 > 50 ops || 4477456 > 104857600 2014-06-09 16:22:12.167744 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.167752 7f868af58700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.167757 7f868f761700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167752 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.167762 7f868df5e700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167767 7f8691f66700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.167772 7f868c75b700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.167767 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:12.167778 7f868cf5c700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.167777 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.167783 7f868d75d700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167790 7f8692f68700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167786 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.167795 7f868a757700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.167796 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.167801 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.167814 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.167822 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.167831 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.167839 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.167847 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.167859 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.167869 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.167877 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.167886 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.167891 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.167903 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.167911 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.167920 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.167928 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.167936 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.167949 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.167958 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.167966 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.167975 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.167980 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.167992 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.168000 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.168009 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.168016 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.168024 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.168037 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.168046 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.168054 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.168063 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.168067 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.168082 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.168091 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1c54ac00,0x1c54ab00,0x1c54aa00,0x1c54a900,0x1c54a800,0x22009900,0x22009800,0x22009700,0x22009600,0x22009500,0x22009400 2014-06-09 16:22:12.168129 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.168140 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.168149 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.168163 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:949 2014-06-09 16:22:12.168173 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.168183 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.168193 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.168198 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.168210 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.091800 7 0.000359 2014-06-09 16:22:12.168221 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.168231 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.168242 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.168252 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.168261 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.168269 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.168279 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.168290 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.168300 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.168312 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.168328 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:12.168360 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.168364 7f8692767700 5 queue_transactions existing osr(0.b7 0x2bd6440)/0x2bd6440 2014-06-09 16:22:12.168367 7f8692767700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.168392 7f8689755700 10 op_submit_start 4168 2014-06-09 16:22:12.168395 7f8689755700 5 queue_transactions (trailing journal) 4168 2014-06-09 16:22:12.168400 7f8689755700 5 queue_op 0x324b27e0 seq 4168 osr(0.1d4 0x2bd6940) 828 bytes (queue has 50 ops and 4237500 bytes) 2014-06-09 16:22:12.168403 7f8689755700 10 op_submit_finish 4168 2014-06-09 16:22:12.168408 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:12.168445 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.168451 7f8689755700 5 queue_transactions existing osr(1.fa 0x384c970)/0x384c970 2014-06-09 16:22:12.168453 7f8689755700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.168458 7f86a5e11700 5 _do_op 0x159968a0 seq 4120 osr(3.4fe 0x3ed1730)/0x3ed1730 start 2014-06-09 16:22:12.168462 7f86a5e11700 10 _do_transaction on 0xbfe6540 2014-06-09 16:22:12.168467 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.168510 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.168525 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4120.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.168529 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.168541 7f86a5e11700 10 _do_op 0x159968a0 seq 4120 r = 0, finisher 0xc6e8300 0 2014-06-09 16:22:12.168552 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314786e0] 2014-06-09 16:22:12.168554 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314786e0] 2014-06-09 16:22:12.168556 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.168557 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.168560 7f86a5e11700 10 _finish_op 0x159968a0 seq 4120 osr(3.4fe 0x3ed1730)/0x3ed1730 2014-06-09 16:22:12.168575 7f868b759700 10 op_submit_start 4169 2014-06-09 16:22:12.168576 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8300] 2014-06-09 16:22:12.168578 7f868b759700 5 queue_transactions (trailing journal) 4169 2014-06-09 16:22:12.168579 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8300] 2014-06-09 16:22:12.168581 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.168580 7f868b759700 5 queue_op 0x24104ea0 seq 4169 osr(1.c9 0x384c9f0) 826 bytes (queue has 50 ops and 4237498 bytes) 2014-06-09 16:22:12.168582 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.168585 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.168585 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5560 (1 active) 2014-06-09 16:22:12.168588 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115320 (2 active) 2014-06-09 16:22:12.168590 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.168592 7f86a5e11700 5 _do_op 0x16fd4de0 seq 4121 osr(1.67 0x384c5b0)/0x384c5b0 start 2014-06-09 16:22:12.168594 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.168595 7f86a5e11700 10 _do_transaction on 0xc140b40 2014-06-09 16:22:12.168599 7f8691765700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.168601 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.168603 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.168607 7f8690f64700 2 waiting 51 > 50 ops || 4238356 > 104857600 2014-06-09 16:22:12.168612 7f868b759700 10 op_submit_finish 4169 2014-06-09 16:22:12.168616 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.168640 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.168657 7f868b759700 10 log is not dirty 2014-06-09 16:22:12.168655 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4121.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.168660 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.168662 7f868b759700 5 queue_transactions existing osr(3.577 0x3ed18c0)/0x3ed18c0 2014-06-09 16:22:12.168665 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.168670 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.168672 7f86a5e11700 10 _do_op 0x16fd4de0 seq 4121 r = 0, finisher 0x3196c540 0 2014-06-09 16:22:12.168675 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.168679 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.168682 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b9c0] 2014-06-09 16:22:12.168684 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.168685 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b9c0] 2014-06-09 16:22:12.168687 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.168688 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.168688 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.168692 7f86a5e11700 10 _finish_op 0x16fd4de0 seq 4121 osr(1.67 0x384c5b0)/0x384c5b0 2014-06-09 16:22:12.168692 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.168707 7f868bf5a700 10 op_submit_start 4170 2014-06-09 16:22:12.168709 7f868bf5a700 5 queue_transactions (trailing journal) 4170 2014-06-09 16:22:12.168710 7f868bf5a700 5 queue_op 0xc931d40 seq 4170 osr(3.2d3 0x3ed0eb0) 828 bytes (queue has 50 ops and 4237500 bytes) 2014-06-09 16:22:12.168712 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3196c540] 2014-06-09 16:22:12.168715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3196c540] 2014-06-09 16:22:12.168716 7f868d75d700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168717 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.168718 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.168721 7f868a757700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168721 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115320 (1 active) 2014-06-09 16:22:12.168726 7f8692767700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.168731 7f8689755700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.168736 7f868ef60700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168724 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89bb00 (2 active) 2014-06-09 16:22:12.168741 7f868cf5c700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.168744 7f86a5e11700 5 _do_op 0x30d96000 seq 4122 osr(2.59f 0x2ed1830)/0x2ed1830 start 2014-06-09 16:22:12.168747 7f868ff62700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168748 7f86a5e11700 10 _do_transaction on 0xc262900 2014-06-09 16:22:12.168751 7f868e75f700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168754 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.168756 7f8691765700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.168760 7f8689f56700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.168764 7f8690f64700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.168770 7f868bf5a700 10 op_submit_finish 4170 2014-06-09 16:22:12.168773 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.14c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.168794 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.168807 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.168812 7f868bf5a700 5 queue_transactions existing osr(1.14c 0x384c810)/0x384c810 2014-06-09 16:22:12.168811 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4122.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.168814 7f868bf5a700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168816 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.168819 7f868b759700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168823 7f8690763700 2 waiting 51 > 50 ops || 4477456 > 104857600 2014-06-09 16:22:12.168828 7f868af58700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.168829 7f86a5e11700 10 _do_op 0x30d96000 seq 4122 r = 0, finisher 0xbfcefe0 0 2014-06-09 16:22:12.168832 7f868f761700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168839 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7d740] 2014-06-09 16:22:12.168838 7f868df5e700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168846 7f8691f66700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.168842 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.168852 7f868c75b700 2 waiting 51 > 50 ops || 4238361 > 104857600 2014-06-09 16:22:12.168857 7f8692f68700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168853 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.168862 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.168870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7d740] 2014-06-09 16:22:12.168871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.168873 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.168876 7f86a5e11700 10 _finish_op 0x30d96000 seq 4122 osr(2.59f 0x2ed1830)/0x2ed1830 2014-06-09 16:22:12.168885 7f868d75d700 10 op_submit_start 4171 2014-06-09 16:22:12.168887 7f868d75d700 5 queue_transactions (trailing journal) 4171 2014-06-09 16:22:12.168888 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89bb00 (1 active) 2014-06-09 16:22:12.168889 7f868d75d700 5 queue_op 0x288f8ae0 seq 4171 osr(3.6ba 0x3ed1ce0) 828 bytes (queue has 50 ops and 4237467 bytes) 2014-06-09 16:22:12.168889 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfcefe0] 2014-06-09 16:22:12.168891 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780000 (2 active) 2014-06-09 16:22:12.168894 7f868a757700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.168894 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfcefe0] 2014-06-09 16:22:12.168896 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.168895 7f86a5e11700 5 _do_op 0x324b3320 seq 4123 osr(0.129 0x2bd66b0)/0x2bd66b0 start 2014-06-09 16:22:12.168897 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.168900 7f868d75d700 10 op_submit_finish 4171 2014-06-09 16:22:12.168899 7f8692767700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.168904 7f8689755700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.168908 7f868ef60700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.168914 7f86a5e11700 10 _do_transaction on 0xbfe40c0 2014-06-09 16:22:12.168903 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.168920 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.168940 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.168945 7f868d75d700 5 queue_transactions existing osr(3.2d6 0x3ed0e60)/0x3ed0e60 2014-06-09 16:22:12.168947 7f868d75d700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.168952 7f868cf5c700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168957 7f868ff62700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.168961 7f868e75f700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.168961 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.168966 7f8691765700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168970 7f8689f56700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168975 7f8690f64700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.168977 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4123.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.168979 7f868bf5a700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.168981 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.168984 7f868b759700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.168988 7f8690763700 2 waiting 51 > 50 ops || 4477423 > 104857600 2014-06-09 16:22:12.168992 7f868af58700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.168998 7f86a5e11700 10 _do_op 0x324b3320 seq 4123 r = 0, finisher 0x324f9260 0 2014-06-09 16:22:12.169009 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f9160] 2014-06-09 16:22:12.169012 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f9160] 2014-06-09 16:22:12.169013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.169014 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.169018 7f86a5e11700 10 _finish_op 0x324b3320 seq 4123 osr(0.129 0x2bd66b0)/0x2bd66b0 2014-06-09 16:22:12.168996 7f868f761700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169065 7f868df5e700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169071 7f8691f66700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.169077 7f868c75b700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.169082 7f8692f68700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169094 7f868a757700 10 op_submit_start 4172 2014-06-09 16:22:12.169094 7f8692767700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169096 7f868a757700 5 queue_transactions (trailing journal) 4172 2014-06-09 16:22:12.169098 7f868a757700 5 queue_op 0xc704600 seq 4172 osr(3.761 0x3ed1e10) 828 bytes (queue has 50 ops and 4237467 bytes) 2014-06-09 16:22:12.169099 7f868ef60700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169105 7f868d75d700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169110 7f8689755700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.169114 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f9260] 2014-06-09 16:22:12.169115 7f868cf5c700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.169118 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f9260] 2014-06-09 16:22:12.169119 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.169120 7f868e75f700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169120 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.169124 7f8691765700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.169123 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780000 (1 active) 2014-06-09 16:22:12.169126 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f440 (2 active) 2014-06-09 16:22:12.169128 7f8689f56700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.169130 7f86a5e11700 5 _do_op 0xd327d40 seq 4124 osr(1.508 0x384db90)/0x384db90 start 2014-06-09 16:22:12.169134 7f86a5e11700 10 _do_transaction on 0xc3f0fc0 2014-06-09 16:22:12.169133 7f8690f64700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169138 7f868a757700 10 op_submit_finish 4172 2014-06-09 16:22:12.169139 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.169141 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.216( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.169176 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.169179 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.169181 7f868a757700 5 queue_transactions existing osr(2.216 0x2ed0b50)/0x2ed0b50 2014-06-09 16:22:12.169183 7f868a757700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169188 7f868ff62700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169193 7f868bf5a700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169193 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4124.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.169197 7f868b759700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169197 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.169201 7f868af58700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.169206 7f868f761700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169209 7f86a5e11700 10 _do_op 0xd327d40 seq 4124 r = 0, finisher 0x32073a00 0 2014-06-09 16:22:12.169211 7f868df5e700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169216 7f8690763700 2 waiting 51 > 50 ops || 4477423 > 104857600 2014-06-09 16:22:12.169220 7f868c75b700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.169220 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050bec0] 2014-06-09 16:22:12.169222 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050bec0] 2014-06-09 16:22:12.169224 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.169225 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.169224 7f8692f68700 2 waiting 51 > 50 ops || 4238295 > 104857600 2014-06-09 16:22:12.169228 7f8691f66700 2 waiting 51 > 50 ops || 4238328 > 104857600 2014-06-09 16:22:12.169229 7f86a5e11700 10 _finish_op 0xd327d40 seq 4124 osr(1.508 0x384db90)/0x384db90 2014-06-09 16:22:12.169238 7f8692767700 10 op_submit_start 4173 2014-06-09 16:22:12.169238 7f868ef60700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169240 7f8692767700 5 queue_transactions (trailing journal) 4173 2014-06-09 16:22:12.169241 7f8692767700 5 queue_op 0x2fcaf2c0 seq 4173 osr(0.b7 0x2bd6440) 858 bytes (queue has 50 ops and 4237497 bytes) 2014-06-09 16:22:12.169244 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32073a00] 2014-06-09 16:22:12.169244 7f868d75d700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169249 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32073a00] 2014-06-09 16:22:12.169249 7f8689755700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.169251 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.169252 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.169254 7f868cf5c700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.169255 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f440 (1 active) 2014-06-09 16:22:12.169258 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd560 (2 active) 2014-06-09 16:22:12.169258 7f868e75f700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169263 7f8692767700 10 op_submit_finish 4173 2014-06-09 16:22:12.169263 7f8691765700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.169267 7f8689f56700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.169271 7f8690f64700 2 waiting 51 > 50 ops || 4238355 > 104857600 2014-06-09 16:22:12.169276 7f868a757700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169280 7f868ff62700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169267 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.169284 7f868bf5a700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169288 7f868b759700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169293 7f868af58700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.169297 7f868f761700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169287 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] check_local 2014-06-09 16:22:12.169301 7f868df5e700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169306 7f8690763700 2 waiting 51 > 50 ops || 4477453 > 104857600 2014-06-09 16:22:12.169310 7f868c75b700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.169299 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] requeue_ops 0x16059800,0x16059700,0x16059600,0x16059500,0x1804c600,0x1804c500,0x1804c400 2014-06-09 16:22:12.169315 7f8692f68700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169319 7f8691f66700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.169331 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] needs_recovery primary has 2 missing 2014-06-09 16:22:12.169345 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.169365 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] publish_stats_to_osd 243:912 2014-06-09 16:22:12.169378 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] hit_set_clear 2014-06-09 16:22:12.169390 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] agent_stop 2014-06-09 16:22:12.169403 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.169408 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.169420 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.124442 7 0.000397 2014-06-09 16:22:12.169432 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.169442 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.169452 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.169463 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.169471 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.169479 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.169489 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.169501 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.169511 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.169524 7f8692767700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 2014-06-09 16:22:12.169540 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.169571 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.169575 7f8692767700 5 queue_transactions existing osr(0.319 0x2bd6d90)/0x2bd6d90 2014-06-09 16:22:12.169577 7f8692767700 2 waiting 51 > 50 ops || 4238358 > 104857600 2014-06-09 16:22:12.169582 7f86a5e11700 5 _do_op 0x300619e0 seq 4125 osr(0.34e 0x2bd7020)/0x2bd7020 start 2014-06-09 16:22:12.169586 7f86a5e11700 10 _do_transaction on 0xc0498c0 2014-06-09 16:22:12.169592 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.169633 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.169650 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4125.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.169654 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.169668 7f86a5e11700 10 _do_op 0x300619e0 seq 4125 r = 0, finisher 0x33a78c0 0 2014-06-09 16:22:12.169679 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x4eb98c0] 2014-06-09 16:22:12.169682 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.169692 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.169701 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 2014-06-09 16:22:12.169711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x4eb98c0] 2014-06-09 16:22:12.169715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.169716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.169720 7f86a5e11700 10 _finish_op 0x300619e0 seq 4125 osr(0.34e 0x2bd7020)/0x2bd7020 2014-06-09 16:22:12.169735 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a78c0] 2014-06-09 16:22:12.169741 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a78c0] 2014-06-09 16:22:12.169743 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.169744 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.169743 7f868d75d700 10 op_submit_start 4174 2014-06-09 16:22:12.169746 7f868d75d700 5 queue_transactions (trailing journal) 4174 2014-06-09 16:22:12.169747 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd560 (1 active) 2014-06-09 16:22:12.169747 7f868d75d700 5 queue_op 0x288f9ec0 seq 4174 osr(3.2d6 0x3ed0e60) 828 bytes (queue has 50 ops and 4237464 bytes) 2014-06-09 16:22:12.169750 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac900 (2 active) 2014-06-09 16:22:12.169753 7f8689755700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.169755 7f86a5e11700 5 _do_op 0xc2ebb60 seq 4126 osr(2.412 0x2ed15d0)/0x2ed15d0 start 2014-06-09 16:22:12.169757 7f86a5e11700 10 _do_transaction on 0xc04ea80 2014-06-09 16:22:12.169758 7f868cf5c700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169763 7f868e75f700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169764 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.169767 7f8691765700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169771 7f8689f56700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169776 7f8690f64700 2 waiting 51 > 50 ops || 4238322 > 104857600 2014-06-09 16:22:12.169780 7f868a757700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169785 7f868d75d700 10 op_submit_finish 4174 2014-06-09 16:22:12.169788 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.169805 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.169822 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.169821 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4126.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.169826 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.169827 7f868d75d700 5 queue_transactions existing osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 2014-06-09 16:22:12.169830 7f868d75d700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169834 7f868ff62700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169839 7f868bf5a700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169839 7f86a5e11700 10 _do_op 0xc2ebb60 seq 4126 r = 0, finisher 0xbd7bb80 0 2014-06-09 16:22:12.169844 7f868b759700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169848 7f868af58700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169850 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d15a0] 2014-06-09 16:22:12.169852 7f868f761700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169857 7f868df5e700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169853 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.169861 7f8690763700 2 waiting 51 > 50 ops || 4477420 > 104857600 2014-06-09 16:22:12.169866 7f868c75b700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169864 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.169871 7f8692f68700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169875 7f8691f66700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169880 7f8692767700 2 waiting 51 > 50 ops || 4238325 > 104857600 2014-06-09 16:22:12.169873 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:202 2014-06-09 16:22:12.169883 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d15a0] 2014-06-09 16:22:12.169885 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.169884 7f868ef60700 2 waiting 51 > 50 ops || 4238292 > 104857600 2014-06-09 16:22:12.169886 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.169889 7f86a5e11700 10 _finish_op 0xc2ebb60 seq 4126 osr(2.412 0x2ed15d0)/0x2ed15d0 2014-06-09 16:22:12.169899 7f8689755700 10 op_submit_start 4175 2014-06-09 16:22:12.169901 7f8689755700 5 queue_transactions (trailing journal) 4175 2014-06-09 16:22:12.169901 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7bb80] 2014-06-09 16:22:12.169902 7f8689755700 5 queue_op 0x324a4840 seq 4175 osr(1.fa 0x384c970) 826 bytes (queue has 50 ops and 4237429 bytes) 2014-06-09 16:22:12.169905 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7bb80] 2014-06-09 16:22:12.169907 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.169908 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.169907 7f868cf5c700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.169911 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac900 (1 active) 2014-06-09 16:22:12.169912 7f868e75f700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.169914 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34979e0 (2 active) 2014-06-09 16:22:12.169917 7f8691765700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.169918 7f86a5e11700 5 _do_op 0x16096420 seq 4127 osr(1.207 0x384ccd0)/0x384ccd0 start 2014-06-09 16:22:12.169921 7f86a5e11700 10 _do_transaction on 0xc144600 2014-06-09 16:22:12.169921 7f8689755700 10 op_submit_finish 4175 2014-06-09 16:22:12.169927 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.169925 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:12.169959 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.169964 7f8689755700 5 queue_transactions existing osr(3.375 0x3ed1060)/0x3ed1060 2014-06-09 16:22:12.169967 7f8689755700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.169967 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.169971 7f8689f56700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.169976 7f8690f64700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.169980 7f868a757700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.169985 7f868d75d700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.169985 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4127.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.169989 7f868ff62700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.169990 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.169994 7f868bf5a700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.169999 7f868b759700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170003 7f868af58700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170007 7f868f761700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170008 7f86a5e11700 10 _do_op 0x16096420 seq 4127 r = 0, finisher 0xcc82120 0 2014-06-09 16:22:12.170011 7f868df5e700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170015 7f86a5e11700 10 _finish_op 0x16096420 seq 4127 osr(1.207 0x384ccd0)/0x384ccd0 2014-06-09 16:22:12.170016 7f8690763700 2 waiting 51 > 50 ops || 4477385 > 104857600 2014-06-09 16:22:12.170016 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd76e20] 2014-06-09 16:22:12.170020 7f868c75b700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170025 7f8692f68700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170020 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.170030 7f8691f66700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170028 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.170034 7f8692767700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170034 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 2014-06-09 16:22:12.170038 7f868ef60700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd76e20] 2014-06-09 16:22:12.170041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.170041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.170051 7f868cf5c700 10 op_submit_start 4176 2014-06-09 16:22:12.170053 7f868cf5c700 5 queue_transactions (trailing journal) 4176 2014-06-09 16:22:12.170055 7f868cf5c700 5 queue_op 0x309a3800 seq 4176 osr(1.3ef 0x384d3f0) 861 bytes (queue has 50 ops and 4237429 bytes) 2014-06-09 16:22:12.170056 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc82120] 2014-06-09 16:22:12.170060 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc82120] 2014-06-09 16:22:12.170060 7f8691765700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170062 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.170063 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.170065 7f8689755700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170066 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34979e0 (1 active) 2014-06-09 16:22:12.170069 7f868e75f700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170069 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652d80 (2 active) 2014-06-09 16:22:12.170074 7f86a5e11700 5 _do_op 0x30a23c0 seq 4128 osr(1.5c1 0x384dd90)/0x384dd90 start 2014-06-09 16:22:12.170075 7f8690f64700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.170077 7f86a5e11700 10 _do_transaction on 0xbffe840 2014-06-09 16:22:12.170080 7f868cf5c700 10 op_submit_finish 4176 2014-06-09 16:22:12.170082 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.170083 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.170098 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.170107 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.170115 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.170123 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.170124 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.170133 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.170137 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4128.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.170142 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.170147 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.170154 7f86a5e11700 10 _do_op 0x30a23c0 seq 4128 r = 0, finisher 0xbd68ae0 0 2014-06-09 16:22:12.170156 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.170165 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x320744e0] 2014-06-09 16:22:12.170168 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x320744e0] 2014-06-09 16:22:12.170169 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.170171 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.170164 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.170174 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.170174 7f86a5e11700 10 _finish_op 0x30a23c0 seq 4128 osr(1.5c1 0x384dd90)/0x384dd90 2014-06-09 16:22:12.170179 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.170187 7f868d75d700 10 op_submit_start 4177 2014-06-09 16:22:12.170188 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd68ae0] 2014-06-09 16:22:12.170189 7f868d75d700 5 queue_transactions (trailing journal) 4177 2014-06-09 16:22:12.170190 7f868d75d700 5 queue_op 0x39d9da0 seq 4177 osr(3.6cb 0x3ed1cb0) 828 bytes (queue has 50 ops and 4237429 bytes) 2014-06-09 16:22:12.170196 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd68ae0] 2014-06-09 16:22:12.170198 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.170199 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.170190 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.060272 7 0.000333 2014-06-09 16:22:12.170202 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652d80 (1 active) 2014-06-09 16:22:12.170205 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc000 (2 active) 2014-06-09 16:22:12.170202 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.170210 7f868d75d700 10 op_submit_finish 4177 2014-06-09 16:22:12.170212 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.170214 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.170223 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.170234 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.170230 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] exit Started/Primary/Active/Activating 0.946122 7 0.000498 2014-06-09 16:22:12.170242 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.170244 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.170250 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.170256 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] needs_recovery is recovered 2014-06-09 16:22:12.170261 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.170268 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 2014-06-09 16:22:12.170272 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.170282 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.170281 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.170298 7f868cf5c700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 2014-06-09 16:22:12.170291 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] finish_recovery 2014-06-09 16:22:12.170301 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] clear_recovery_state 2014-06-09 16:22:12.170315 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.170313 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.170326 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.170337 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.170349 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.170353 7f868cf5c700 5 queue_transactions existing osr(2.61b 0x2ed1cd0)/0x2ed1cd0 2014-06-09 16:22:12.170356 7f868cf5c700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170348 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.170361 7f868a757700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170359 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] share_pg_info 2014-06-09 16:22:12.170366 7f8689f56700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170370 7f868ff62700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170371 7f868d75d700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.170375 7f868af58700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170379 7f868f761700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170383 7f868df5e700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170388 7f868bf5a700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170392 7f868c75b700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170396 7f8692f68700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170388 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] publish_stats_to_osd 243:875 2014-06-09 16:22:12.170400 7f868b759700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170404 7f8690763700 2 waiting 51 > 50 ops || 4477385 > 104857600 2014-06-09 16:22:12.170408 7f868ef60700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170413 7f8689755700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170416 7f868e75f700 2 waiting 51 > 50 ops || 4238257 > 104857600 2014-06-09 16:22:12.170421 7f8690f64700 2 waiting 51 > 50 ops || 4238287 > 104857600 2014-06-09 16:22:12.170424 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.170425 7f8692767700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170428 7f868d75d700 5 queue_transactions existing osr(3.78a 0x3ed1d80)/0x3ed1d80 2014-06-09 16:22:12.170428 7f8691f66700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170430 7f868d75d700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170434 7f8691765700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170437 7f86a5e11700 5 _do_op 0x230b9d40 seq 4129 osr(1.2a3 0x384d1f0)/0x384d1f0 start 2014-06-09 16:22:12.170441 7f86a5e11700 10 _do_transaction on 0xc128180 2014-06-09 16:22:12.170446 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.170488 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.170503 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4129.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.170508 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.170519 7f86a5e11700 10 _do_op 0x230b9d40 seq 4129 r = 0, finisher 0xccf8120 0 2014-06-09 16:22:12.170529 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd760] 2014-06-09 16:22:12.170532 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd760] 2014-06-09 16:22:12.170533 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.170534 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.170537 7f86a5e11700 10 _finish_op 0x230b9d40 seq 4129 osr(1.2a3 0x384d1f0)/0x384d1f0 2014-06-09 16:22:12.170547 7f868cf5c700 10 op_submit_start 4178 2014-06-09 16:22:12.170549 7f868cf5c700 5 queue_transactions (trailing journal) 4178 2014-06-09 16:22:12.170549 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf8120] 2014-06-09 16:22:12.170551 7f868cf5c700 5 queue_op 0x309a3560 seq 4178 osr(2.61b 0x2ed1cd0) 861 bytes (queue has 50 ops and 4237462 bytes) 2014-06-09 16:22:12.170553 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf8120] 2014-06-09 16:22:12.170554 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.170555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.170556 7f868a757700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170559 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc000 (1 active) 2014-06-09 16:22:12.170560 7f8689f56700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.170562 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcfc0 (2 active) 2014-06-09 16:22:12.170565 7f868ff62700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170566 7f86a5e11700 5 _do_op 0xc88b4a0 seq 4130 osr(0.4c8 0x2bd7670)/0x2bd7670 start 2014-06-09 16:22:12.170569 7f86a5e11700 10 _do_transaction on 0xbf28fc0 2014-06-09 16:22:12.170570 7f868af58700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.170574 7f868f761700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170574 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.170578 7f868df5e700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170583 7f868bf5a700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170587 7f868c75b700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.170592 7f8692f68700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170596 7f868b759700 2 waiting 51 > 50 ops || 4238290 > 104857600 2014-06-09 16:22:12.170601 7f868cf5c700 10 op_submit_finish 4178 2014-06-09 16:22:12.170604 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.170614 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.170618 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.114057 8 0.002010 2014-06-09 16:22:12.170629 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4130.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.170634 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.170629 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.170645 7f86a5e11700 10 _do_op 0xc88b4a0 seq 4130 r = 0, finisher 0x3281b3c0 0 2014-06-09 16:22:12.170640 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.170656 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5fba0] 2014-06-09 16:22:12.170650 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.170658 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5fba0] 2014-06-09 16:22:12.170660 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.170661 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.170665 7f86a5e11700 10 _finish_op 0xc88b4a0 seq 4130 osr(0.4c8 0x2bd7670)/0x2bd7670 2014-06-09 16:22:12.170661 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.170673 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcfc0 (1 active) 2014-06-09 16:22:12.170674 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281b3c0] 2014-06-09 16:22:12.170675 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa900 (2 active) 2014-06-09 16:22:12.170670 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.170678 7f86a5e11700 5 _do_op 0x30a3860 seq 4131 osr(0.530 0x2bd78f0)/0x2bd78f0 start 2014-06-09 16:22:12.170679 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281b3c0] 2014-06-09 16:22:12.170681 7f86a5e11700 10 _do_transaction on 0xbe8b2c0 2014-06-09 16:22:12.170681 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.170682 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.170678 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.170686 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.170689 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.170700 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.170710 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.170725 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.170720 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.170730 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.170738 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4131.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.170742 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.170742 7f868cf5c700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 2014-06-09 16:22:12.170753 7f86a5e11700 10 _do_op 0x30a3860 seq 4131 r = 0, finisher 0x307f0260 0 2014-06-09 16:22:12.170755 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.170763 7f86a5e11700 10 _finish_op 0x30a3860 seq 4131 osr(0.530 0x2bd78f0)/0x2bd78f0 2014-06-09 16:22:12.170764 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074f40] 2014-06-09 16:22:12.170767 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074f40] 2014-06-09 16:22:12.170769 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.170770 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.170772 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f0260] 2014-06-09 16:22:12.170775 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f0260] 2014-06-09 16:22:12.170779 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.170780 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.170783 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa900 (1 active) 2014-06-09 16:22:12.170786 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f7a0 (2 active) 2014-06-09 16:22:12.170787 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.170788 7f86a5e11700 5 _do_op 0x30d966c0 seq 4132 osr(0.622 0x2bd7cf0)/0x2bd7cf0 start 2014-06-09 16:22:12.170791 7f86a5e11700 10 _do_transaction on 0xc2df440 2014-06-09 16:22:12.170792 7f868cf5c700 5 queue_transactions existing osr(1.446 0x384d6b0)/0x384d6b0 2014-06-09 16:22:12.170794 7f868cf5c700 10 op_submit_start 4179 2014-06-09 16:22:12.170795 7f868cf5c700 5 queue_transactions (trailing journal) 4179 2014-06-09 16:22:12.170797 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.170797 7f868cf5c700 5 queue_op 0xc704ba0 seq 4179 osr(1.446 0x384d6b0) 861 bytes (queue has 49 ops and 4236667 bytes) 2014-06-09 16:22:12.170800 7f868cf5c700 10 op_submit_finish 4179 2014-06-09 16:22:12.170802 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.170817 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.170826 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.170836 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.170836 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.170845 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.170853 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4132.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.170858 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.170854 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.170869 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1008 2014-06-09 16:22:12.170878 7f86a5e11700 10 _do_op 0x30d966c0 seq 4132 r = 0, finisher 0x3062b140 0 2014-06-09 16:22:12.170884 7f86a5e11700 10 _finish_op 0x30d966c0 seq 4132 osr(0.622 0x2bd7cf0)/0x2bd7cf0 2014-06-09 16:22:12.170880 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.170886 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x13964600] 2014-06-09 16:22:12.170893 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062b140] 2014-06-09 16:22:12.170896 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062b140] 2014-06-09 16:22:12.170889 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.170897 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.170899 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.170890 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.170899 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.170901 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f7a0 (1 active) 2014-06-09 16:22:12.170904 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd200 (2 active) 2014-06-09 16:22:12.170900 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.170906 7f86a5e11700 5 _do_op 0x306075c0 seq 4133 osr(1.86 0x384c490)/0x384c490 start 2014-06-09 16:22:12.170908 7f86a5e11700 10 _do_transaction on 0xc2af500 2014-06-09 16:22:12.170914 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.170909 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:184 2014-06-09 16:22:12.170917 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x13964600] 2014-06-09 16:22:12.170919 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.170920 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.170953 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.170970 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4133.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.170904 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.170975 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.170978 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.086089 7 0.007362 2014-06-09 16:22:12.170990 7f86a5e11700 10 _do_op 0x306075c0 seq 4133 r = 0, finisher 0x32078c00 0 2014-06-09 16:22:12.170990 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.171001 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32075f20] 2014-06-09 16:22:12.171001 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.171004 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.171011 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.171014 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.171022 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.171022 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:208 2014-06-09 16:22:12.171032 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32075f20] 2014-06-09 16:22:12.171034 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.171035 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.171031 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.171040 7f86a5e11700 10 _finish_op 0x306075c0 seq 4133 osr(1.86 0x384c490)/0x384c490 2014-06-09 16:22:12.171039 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.171049 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32078c00] 2014-06-09 16:22:12.171056 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32078c00] 2014-06-09 16:22:12.171058 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.171059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.171049 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.171062 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd200 (1 active) 2014-06-09 16:22:12.171065 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f9e0 (2 active) 2014-06-09 16:22:12.171067 7f86a5e11700 5 _do_op 0x2fcafbc0 seq 4134 osr(2.19 0x2ed0270)/0x2ed0270 start 2014-06-09 16:22:12.171060 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.171070 7f86a5e11700 10 _do_transaction on 0xc2df680 2014-06-09 16:22:12.171077 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.171070 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.171080 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.171089 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.171105 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.171117 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.171121 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.171133 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4134.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.171138 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.171153 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.171153 7f86a5e11700 10 _do_op 0x2fcafbc0 seq 4134 r = 0, finisher 0x30c540a0 0 2014-06-09 16:22:12.171158 7f868cf5c700 5 queue_transactions existing osr(0.66b 0x2bd7eb0)/0x2bd7eb0 2014-06-09 16:22:12.171161 7f868cf5c700 10 op_submit_start 4180 2014-06-09 16:22:12.171162 7f868cf5c700 5 queue_transactions (trailing journal) 4180 2014-06-09 16:22:12.171163 7f868cf5c700 5 queue_op 0x324e62a0 seq 4180 osr(0.66b 0x2bd7eb0) 861 bytes (queue has 48 ops and 4235809 bytes) 2014-06-09 16:22:12.171163 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x281b76a0] 2014-06-09 16:22:12.171165 7f868cf5c700 10 op_submit_finish 4180 2014-06-09 16:22:12.171167 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.171168 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.171177 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.171180 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.055424 7 0.000370 2014-06-09 16:22:12.171187 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 2014-06-09 16:22:12.171196 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x281b76a0] 2014-06-09 16:22:12.171198 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.171191 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.171199 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.171203 7f86a5e11700 10 _finish_op 0x2fcafbc0 seq 4134 osr(2.19 0x2ed0270)/0x2ed0270 2014-06-09 16:22:12.171201 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.171212 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30c540a0] 2014-06-09 16:22:12.171217 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30c540a0] 2014-06-09 16:22:12.171211 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.171219 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.171220 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.171223 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f9e0 (1 active) 2014-06-09 16:22:12.171226 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c000 (2 active) 2014-06-09 16:22:12.171221 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.171228 7f86a5e11700 5 _do_op 0x15996b40 seq 4135 osr(1.2aa 0x384d1d0)/0x384d1d0 start 2014-06-09 16:22:12.171231 7f86a5e11700 10 _do_transaction on 0xc05f5c0 2014-06-09 16:22:12.171230 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.171237 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.171238 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.171249 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.171260 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.171269 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.171278 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.171281 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 2014-06-09 16:22:12.171294 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4135.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.171299 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.171297 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 2014-06-09 16:22:12.171312 7f86a5e11700 10 _do_op 0x15996b40 seq 4135 r = 0, finisher 0xbce6d60 0 2014-06-09 16:22:12.171321 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002de0] 2014-06-09 16:22:12.171329 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.171324 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.171333 7f868cf5c700 5 queue_transactions existing osr(0.67d 0x2bd7e50)/0x2bd7e50 2014-06-09 16:22:12.171336 7f868cf5c700 10 op_submit_start 4181 2014-06-09 16:22:12.171337 7f868cf5c700 5 queue_transactions (trailing journal) 4181 2014-06-09 16:22:12.171338 7f868cf5c700 5 queue_op 0x160970e0 seq 4181 osr(0.67d 0x2bd7e50) 861 bytes (queue has 48 ops and 4235812 bytes) 2014-06-09 16:22:12.171334 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.171341 7f868cf5c700 10 op_submit_finish 4181 2014-06-09 16:22:12.171343 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.171353 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002de0] 2014-06-09 16:22:12.171344 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.171355 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.171356 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.171360 7f86a5e11700 10 _finish_op 0x15996b40 seq 4135 osr(1.2aa 0x384d1d0)/0x384d1d0 2014-06-09 16:22:12.171358 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.171369 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbce6d60] 2014-06-09 16:22:12.171366 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.172807 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.172820 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.172886 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.172901 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.172911 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.172918 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.172928 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.172933 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.172945 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.019763 7 0.000288 2014-06-09 16:22:12.172956 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.172966 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.172976 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.172986 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.172995 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.173003 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.173014 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.173025 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.173034 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.173046 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 2014-06-09 16:22:12.173062 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:12.173094 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.173098 7f868cf5c700 5 queue_transactions existing osr(2.573 0x2ed18b0)/0x2ed18b0 2014-06-09 16:22:12.171373 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbce6d60] 2014-06-09 16:22:12.173106 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.173107 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.171376 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c000 (1 active) 2014-06-09 16:22:12.171402 7f868a757700 10 op_submit_start 4182 2014-06-09 16:22:12.173114 7f868a757700 5 queue_transactions (trailing journal) 4182 2014-06-09 16:22:12.173116 7f868a757700 5 queue_op 0xcabc780 seq 4182 osr(2.216 0x2ed0b50) 828 bytes (queue has 50 ops and 4237435 bytes) 2014-06-09 16:22:12.171413 7f8690763700 2 waiting 51 > 50 ops || 4477391 > 104857600 2014-06-09 16:22:12.173125 7f8690f64700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173130 7f8692767700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173135 7f8691f66700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173140 7f868d75d700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173137 7f86a68f6700 5 -- op tracker -- , seq: 3665, time: 2014-06-09 16:22:12.166912, event: header_read, op: osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.173145 7f8691765700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173149 7f8689f56700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173150 7f86a68f6700 5 -- op tracker -- , seq: 3665, time: 2014-06-09 16:22:12.166912, event: throttled, op: osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.173154 7f868ff62700 2 waiting 51 > 50 ops || 4238263 > 104857600 2014-06-09 16:22:12.173159 7f868af58700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173163 7f868f761700 2 waiting 51 > 50 ops || 4238263 > 104857600 2014-06-09 16:22:12.173160 7f86a68f6700 5 -- op tracker -- , seq: 3665, time: 2014-06-09 16:22:12.171912, event: all_read, op: osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.173168 7f868df5e700 2 waiting 51 > 50 ops || 4238263 > 104857600 2014-06-09 16:22:12.173172 7f868bf5a700 2 waiting 51 > 50 ops || 4238263 > 104857600 2014-06-09 16:22:12.173174 7f86a68f6700 5 -- op tracker -- , seq: 3665, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.173177 7f868c75b700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173182 7f8692f68700 2 waiting 51 > 50 ops || 4238263 > 104857600 2014-06-09 16:22:12.173186 7f868b759700 2 waiting 51 > 50 ops || 4238263 > 104857600 2014-06-09 16:22:12.173182 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.173191 7f868e75f700 2 waiting 51 > 50 ops || 4238263 > 104857600 2014-06-09 16:22:12.173195 7f868cf5c700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173199 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f65a0 (2 active) 2014-06-09 16:22:12.173205 7f868a757700 10 op_submit_finish 4182 2014-06-09 16:22:12.173206 7f86a68f6700 15 osd.26 243 enqueue_op 0x2976cf00 prio 63 cost 2580480 latency 0.006294 osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.173210 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.173223 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.173232 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.173240 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.173248 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.173256 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.173270 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:170 2014-06-09 16:22:12.173279 7f868a757700 20 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.173287 7f868a757700 20 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.173296 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.173301 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.173315 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.068818 7 0.000428 2014-06-09 16:22:12.173327 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.173338 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.173349 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.173361 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.173371 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.173380 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.173391 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.173403 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.173414 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.173425 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.173435 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.173449 7f868a757700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 2014-06-09 16:22:12.173465 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1009 2014-06-09 16:22:12.173500 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.173504 7f868a757700 5 queue_transactions existing osr(3.362 0x3ed10a0)/0x3ed10a0 2014-06-09 16:22:12.173507 7f868a757700 2 waiting 51 > 50 ops || 4238296 > 104857600 2014-06-09 16:22:12.173511 7f868ef60700 10 op_submit_start 4183 2014-06-09 16:22:12.173513 7f868ef60700 5 queue_transactions (trailing journal) 4183 2014-06-09 16:22:12.173515 7f868ef60700 5 queue_op 0x39d98c0 seq 4183 osr(2.5f8 0x2ed1db0) 828 bytes (queue has 50 ops and 4237435 bytes) 2014-06-09 16:22:12.173518 7f868ef60700 10 op_submit_finish 4183 2014-06-09 16:22:12.173524 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.69c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.173566 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.173571 7f868ef60700 5 queue_transactions existing osr(2.69c 0x2ed1f20)/0x2ed1f20 2014-06-09 16:22:12.173573 7f868ef60700 2 waiting 51 > 50 ops || 4238263 > 104857600 2014-06-09 16:22:12.173600 7f86a5e11700 5 _do_op 0x30a3320 seq 4136 osr(1.73b 0x2ed00c0)/0x2ed00c0 start 2014-06-09 16:22:12.173608 7f86a5e11700 10 _do_transaction on 0xc145a40 2014-06-09 16:22:12.173614 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.173653 7f8683620700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.173621, event: header_read, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) 2014-06-09 16:22:12.173662 7f8683620700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.173622, event: throttled, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) 2014-06-09 16:22:12.173669 7f8683620700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.173630, event: all_read, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) 2014-06-09 16:22:12.173678 7f8683620700 5 -- op tracker -- , seq: 3666, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) 2014-06-09 16:22:12.173683 7f8683620700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) v2 epoch 243 2014-06-09 16:22:12.173688 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.173688 7f8683620700 20 osd.26 243 should_share_map osd.6 172.16.96.201:6849/52045 243 2014-06-09 16:22:12.173694 7f8683620700 15 osd.26 243 enqueue_op 0x2976ce00 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) v2 2014-06-09 16:22:12.173704 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4136.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.173708 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.173723 7f869c77b700 10 osd.26 243 do_waiters -- start 2014-06-09 16:22:12.173725 7f869c77b700 10 osd.26 243 do_waiters -- finish 2014-06-09 16:22:12.173727 7f869c77b700 20 osd.26 243 _dispatch 0x32da8fc0 osd_map(243..244 src has 1..244) v3 2014-06-09 16:22:12.173743 7f86a5e11700 10 _do_op 0x30a3320 seq 4136 r = 0, finisher 0xcd32a20 0 2014-06-09 16:22:12.173744 7f869c77b700 3 osd.26 243 handle_osd_map epochs [243,244], i have 243, src has [1,244] 2014-06-09 16:22:12.173748 7f869c77b700 10 osd.26 243 handle_osd_map got inc map for epoch 244 2014-06-09 16:22:12.173755 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd71c00] 2014-06-09 16:22:12.173757 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd71c00] 2014-06-09 16:22:12.173758 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.173759 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.173762 7f86a5e11700 10 _finish_op 0x30a3320 seq 4136 osr(1.73b 0x2ed00c0)/0x2ed00c0 2014-06-09 16:22:12.173773 7f8690f64700 10 op_submit_start 4184 2014-06-09 16:22:12.173774 7f8690f64700 5 queue_transactions (trailing journal) 4184 2014-06-09 16:22:12.173776 7f8690f64700 5 queue_op 0x324e61e0 seq 4184 osr(0.29 0x2bd62b0) 858 bytes (queue has 50 ops and 4237465 bytes) 2014-06-09 16:22:12.173780 7f8692767700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173784 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173788 7f868d75d700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173792 7f8691765700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173795 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173798 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173802 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173806 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173809 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173812 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173817 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173820 7f8692f68700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173824 7f868b759700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173827 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173831 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173834 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.173837 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.173841 7f8690763700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.173849 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd32a20] 2014-06-09 16:22:12.173853 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd32a20] 2014-06-09 16:22:12.173854 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.173855 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.173858 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f65a0 (1 active) 2014-06-09 16:22:12.173860 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0ea0 (2 active) 2014-06-09 16:22:12.173865 7f8690f64700 10 op_submit_finish 4184 2014-06-09 16:22:12.173868 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.173885 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.173896 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.173900 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.173912 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.173920 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.173926 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.173934 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.173941 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.173952 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 2014-06-09 16:22:12.173961 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.173968 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.173976 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.174004 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358400 (19 active) 2014-06-09 16:22:12.174030 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358400 (20 active) 2014-06-09 16:22:12.174033 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.2fd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.174063 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.174062 7f869c77b700 10 osd.26 243 advance to epoch 244 (<= newest 244) 2014-06-09 16:22:12.174068 7f8690f64700 5 queue_transactions existing osr(0.2fd 0x2bd6e50)/0x2bd6e50 2014-06-09 16:22:12.174070 7f8690f64700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.174075 7f86a5e11700 5 _do_op 0x30a3da0 seq 4137 osr(3.537 0x3ed1620)/0x3ed1620 start 2014-06-09 16:22:12.174079 7f86a5e11700 10 _do_transaction on 0xc1392c0 2014-06-09 16:22:12.174084 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.174126 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.174141 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4137.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.174143 7f869c77b700 7 osd.26 244 advance_map epoch 244 2014-06-09 16:22:12.174146 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.174149 7f869c77b700 10 osd.26 244 write_superblock sb(6336a654-0649-4c42-a04d-d54aefd3bd31 osd.26 d59d5a8a-a977-45a8-b9aa-a778604e89ff e244 [1,244] lci=[242,244]) 2014-06-09 16:22:12.174158 7f86a5e11700 10 _do_op 0x30a3da0 seq 4137 r = 0, finisher 0x2d739300 0 2014-06-09 16:22:12.174166 7f869c77b700 5 queue_transactions existing osr(default 0x2c10280)/0x2c10280 2014-06-09 16:22:12.174167 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d6540] 2014-06-09 16:22:12.174169 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d6540] 2014-06-09 16:22:12.174169 7f869c77b700 2 waiting 51 > 50 ops || 4259089 > 104857600 2014-06-09 16:22:12.174170 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.174172 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.174174 7f86a5e11700 10 _finish_op 0x30a3da0 seq 4137 osr(3.537 0x3ed1620)/0x3ed1620 2014-06-09 16:22:12.174176 7f869af78700 1 osd.26 244 ms_handle_reset con 0xff678c0 session 0x25e5a000 2014-06-09 16:22:12.174183 7f8691f66700 10 op_submit_start 4185 2014-06-09 16:22:12.174185 7f8691f66700 5 queue_transactions (trailing journal) 4185 2014-06-09 16:22:12.174186 7f8691f66700 5 queue_op 0x160967e0 seq 4185 osr(1.25a 0x384cfb0) 861 bytes (queue has 50 ops and 4237498 bytes) 2014-06-09 16:22:12.174191 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174195 7f8691765700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174199 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174203 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174207 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174211 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174214 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174218 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174223 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174227 7f8692f68700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174230 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174233 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174237 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174240 7f868a757700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174244 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174248 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.174251 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.174256 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 2014-06-09 16:22:12.174260 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174270 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d739300] 2014-06-09 16:22:12.174276 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d739300] 2014-06-09 16:22:12.174278 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.174279 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.174282 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0ea0 (1 active) 2014-06-09 16:22:12.174285 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652d80 (2 active) 2014-06-09 16:22:12.174289 7f8691f66700 10 op_submit_finish 4185 2014-06-09 16:22:12.174293 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.174308 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.998106 7 0.001551 2014-06-09 16:22:12.174320 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.174330 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.174340 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.174350 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.174357 7f8683620700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.174323, event: header_read, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.174357 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.174367 7f8683620700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.174324, event: throttled, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.174364 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.174374 7f8683620700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.174336, event: all_read, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.174381 7f8683620700 5 -- op tracker -- , seq: 3667, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.174374 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.174384 7f8683620700 10 osd.26 244 handle_replica_op osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) v2 epoch 243 2014-06-09 16:22:12.174384 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.174389 7f8683620700 20 osd.26 244 should_share_map osd.6 172.16.96.201:6849/52045 243 2014-06-09 16:22:12.174394 7f8683620700 10 osd.26 244 osd.6 172.16.96.201:6849/52045 has old map 243 < 244 2014-06-09 16:22:12.174392 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.174398 7f8683620700 15 osd.26 244 enqueue_op 0x2976d300 prio 196 cost 0 latency 0.000075 osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) v2 2014-06-09 16:22:12.174404 7f8691f66700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xc03a840 172.16.96.201:6823/50569 2014-06-09 16:22:12.174427 7f8691f66700 10 osd.26 243 note_peer_epoch osd.3 has 244 2014-06-09 16:22:12.174439 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 2014-06-09 16:22:12.174474 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.174478 7f8691f66700 5 queue_transactions existing osr(0.5d4 0x2bd7a10)/0x2bd7a10 2014-06-09 16:22:12.174481 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174486 7f86a5e11700 5 _do_op 0x30a2ae0 seq 4138 osr(1.5c1 0x384dd90)/0x384dd90 start 2014-06-09 16:22:12.174489 7f86a5e11700 10 _do_transaction on 0xbf50f00 2014-06-09 16:22:12.174495 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.174540 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.174555 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4138.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.174560 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.174573 7f86a5e11700 10 _do_op 0x30a2ae0 seq 4138 r = 0, finisher 0xc410a40 0 2014-06-09 16:22:12.174584 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3ed20] 2014-06-09 16:22:12.174587 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3ed20] 2014-06-09 16:22:12.174588 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.174590 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.174593 7f86a5e11700 10 _finish_op 0x30a2ae0 seq 4138 osr(1.5c1 0x384dd90)/0x384dd90 2014-06-09 16:22:12.174604 7f8691765700 10 op_submit_start 4186 2014-06-09 16:22:12.174606 7f8691765700 5 queue_transactions (trailing journal) 4186 2014-06-09 16:22:12.174607 7f8691765700 5 queue_op 0x1e08a000 seq 4186 osr(0.678 0x2bd7e90) 861 bytes (queue has 50 ops and 4237531 bytes) 2014-06-09 16:22:12.174613 7f8689f56700 2 waiting 51 > 50 ops || 4238392 > 104857600 2014-06-09 16:22:12.174618 7f868ff62700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174622 7f868af58700 2 waiting 51 > 50 ops || 4238392 > 104857600 2014-06-09 16:22:12.174627 7f868f761700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174632 7f868df5e700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174636 7f868bf5a700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174640 7f868c75b700 2 waiting 51 > 50 ops || 4238392 > 104857600 2014-06-09 16:22:12.174644 7f8692f68700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174649 7f868b759700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174653 7f868e75f700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174657 7f868cf5c700 2 waiting 51 > 50 ops || 4238392 > 104857600 2014-06-09 16:22:12.174661 7f868a757700 2 waiting 51 > 50 ops || 4238392 > 104857600 2014-06-09 16:22:12.174666 7f868ef60700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174670 7f8690763700 2 waiting 51 > 50 ops || 4477487 > 104857600 2014-06-09 16:22:12.174674 7f8690f64700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.174678 7f869c77b700 2 waiting 51 > 50 ops || 4259155 > 104857600 2014-06-09 16:22:12.174683 7f8692767700 2 waiting 51 > 50 ops || 4238392 > 104857600 2014-06-09 16:22:12.174687 7f8691f66700 2 waiting 51 > 50 ops || 4238392 > 104857600 2014-06-09 16:22:12.174691 7f868d75d700 2 waiting 51 > 50 ops || 4238392 > 104857600 2014-06-09 16:22:12.174715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc410a40] 2014-06-09 16:22:12.174719 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc410a40] 2014-06-09 16:22:12.174720 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.174721 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.174717 7f86a68f6700 5 -- op tracker -- , seq: 3668, time: 2014-06-09 16:22:12.173229, event: header_read, op: osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.174724 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652d80 (1 active) 2014-06-09 16:22:12.174727 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4077a0 (2 active) 2014-06-09 16:22:12.174732 7f8691765700 10 op_submit_finish 4186 2014-06-09 16:22:12.174729 7f86a68f6700 5 -- op tracker -- , seq: 3668, time: 2014-06-09 16:22:12.173230, event: throttled, op: osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.174739 7f86a68f6700 5 -- op tracker -- , seq: 3668, time: 2014-06-09 16:22:12.174681, event: all_read, op: osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.174752 7f86a68f6700 5 -- op tracker -- , seq: 3668, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.174760 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.174765 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.174779 7f86a68f6700 15 osd.26 244 enqueue_op 0x1c54ba00 prio 63 cost 1032192 latency 0.001550 osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.174736 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.174816 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.083039 7 0.000975 2014-06-09 16:22:12.174828 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.174839 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.174849 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.174859 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.174868 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.174875 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.174886 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.174897 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.174907 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.174912 7f86a68f6700 5 -- op tracker -- , seq: 3669, time: 2014-06-09 16:22:12.174804, event: header_read, op: osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.174920 7f8691765700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbeb9ce0 172.16.96.201:6859/53035 2014-06-09 16:22:12.174922 7f86a68f6700 5 -- op tracker -- , seq: 3669, time: 2014-06-09 16:22:12.174805, event: throttled, op: osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.174930 7f86a68f6700 5 -- op tracker -- , seq: 3669, time: 2014-06-09 16:22:12.174882, event: all_read, op: osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.174938 7f86a68f6700 5 -- op tracker -- , seq: 3669, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.174941 7f8691765700 10 osd.26 243 note_peer_epoch osd.8 has 244 2014-06-09 16:22:12.174943 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.174947 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.174953 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.174958 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d700 prio 63 cost 65536 latency 0.000154 osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.174988 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.174992 7f8691765700 5 queue_transactions existing osr(0.dc 0x2bd6570)/0x2bd6570 2014-06-09 16:22:12.174995 7f8691765700 2 waiting 51 > 50 ops || 4238389 > 104857600 2014-06-09 16:22:12.175034 7f86a5e11700 5 _do_op 0x1e08b440 seq 4139 osr(0.21b 0x2bd6ab0)/0x2bd6ab0 start 2014-06-09 16:22:12.175040 7f86a5e11700 10 _do_transaction on 0x344ca80 2014-06-09 16:22:12.175046 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.175095 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.175113 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4139.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.175118 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.175131 7f86a5e11700 10 _do_op 0x1e08b440 seq 4139 r = 0, finisher 0x3050a060 0 2014-06-09 16:22:12.175143 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251c240] 2014-06-09 16:22:12.175146 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.175157 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.175167 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:12.175177 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251c240] 2014-06-09 16:22:12.175179 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.175180 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.175184 7f86a5e11700 10 _finish_op 0x1e08b440 seq 4139 osr(0.21b 0x2bd6ab0)/0x2bd6ab0 2014-06-09 16:22:12.175196 7f868ff62700 10 op_submit_start 4187 2014-06-09 16:22:12.175197 7f868ff62700 5 queue_transactions (trailing journal) 4187 2014-06-09 16:22:12.175199 7f868ff62700 5 queue_op 0x30a3d40 seq 4187 osr(0.31f 0x2bd6d50) 828 bytes (queue has 50 ops and 4237498 bytes) 2014-06-09 16:22:12.175204 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175209 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175213 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175218 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175223 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175227 7f8692f68700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175231 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175235 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175236 7f86a68f6700 5 -- op tracker -- , seq: 3670, time: 2014-06-09 16:22:12.174972, event: header_read, op: osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.175247 7f86a68f6700 5 -- op tracker -- , seq: 3670, time: 2014-06-09 16:22:12.174973, event: throttled, op: osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.175257 7f86a68f6700 5 -- op tracker -- , seq: 3670, time: 2014-06-09 16:22:12.175203, event: all_read, op: osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.175289 7f86a68f6700 5 -- op tracker -- , seq: 3670, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.175294 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.175298 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.175309 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d600 prio 63 cost 122880 latency 0.000337 osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.175341 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175347 7f868a757700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175351 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175356 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.175360 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175364 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 2014-06-09 16:22:12.175368 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175373 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175377 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175381 7f8691765700 2 waiting 51 > 50 ops || 4238356 > 104857600 2014-06-09 16:22:12.175386 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175397 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a060] 2014-06-09 16:22:12.175401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a060] 2014-06-09 16:22:12.175403 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.175404 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.175407 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4077a0 (1 active) 2014-06-09 16:22:12.175410 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ade60 (2 active) 2014-06-09 16:22:12.175415 7f868ff62700 10 op_submit_finish 4187 2014-06-09 16:22:12.175420 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.206( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.175457 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.175463 7f868ff62700 5 queue_transactions existing osr(1.206 0x384cd10)/0x384cd10 2014-06-09 16:22:12.175466 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175470 7f86a5e11700 5 _do_op 0x30a2540 seq 4140 osr(1.348 0x384d310)/0x384d310 start 2014-06-09 16:22:12.175474 7f86a5e11700 10 _do_transaction on 0xc257200 2014-06-09 16:22:12.175480 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.175524 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.175539 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4140.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.175543 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.175563 7f86a5e11700 10 _do_op 0x30a2540 seq 4140 r = 0, finisher 0xbef14e0 0 2014-06-09 16:22:12.175574 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d6220] 2014-06-09 16:22:12.175577 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d6220] 2014-06-09 16:22:12.175578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.175580 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.175583 7f86a5e11700 10 _finish_op 0x30a2540 seq 4140 osr(1.348 0x384d310)/0x384d310 2014-06-09 16:22:12.175594 7f868f761700 10 op_submit_start 4188 2014-06-09 16:22:12.175596 7f868f761700 5 queue_transactions (trailing journal) 4188 2014-06-09 16:22:12.175598 7f868f761700 5 queue_op 0x2c06eba0 seq 4188 osr(1.4fb 0x384d7f0) 828 bytes (queue has 50 ops and 4237498 bytes) 2014-06-09 16:22:12.175603 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175608 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175612 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175637 7f8692f68700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175641 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175646 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175651 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175655 7f868a757700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175659 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175664 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.175668 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175673 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 2014-06-09 16:22:12.175677 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175682 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175686 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175690 7f8691765700 2 waiting 51 > 50 ops || 4238356 > 104857600 2014-06-09 16:22:12.175695 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175699 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175704 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef14e0] 2014-06-09 16:22:12.175719 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef14e0] 2014-06-09 16:22:12.175721 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.175722 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.175725 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ade60 (1 active) 2014-06-09 16:22:12.175728 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd680 (2 active) 2014-06-09 16:22:12.175733 7f868f761700 10 op_submit_finish 4188 2014-06-09 16:22:12.175737 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 243/243 242/242/222) 2014-06-09 16:22:12.175774 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.175780 7f868f761700 5 queue_transactions existing osr(3.18e 0x3ed0af0)/0x3ed0af0 2014-06-09 16:22:12.175783 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175788 7f86a5e11700 5 _do_op 0x2fcaea20 seq 4141 osr(1.2ef 0x384d0f0)/0x384d0f0 start 2014-06-09 16:22:12.175792 7f86a5e11700 10 _do_transaction on 0xbf2e840 2014-06-09 16:22:12.175797 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.175841 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.175856 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4141.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.175860 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.175871 7f86a5e11700 10 _do_op 0x2fcaea20 seq 4141 r = 0, finisher 0xbd7ce00 0 2014-06-09 16:22:12.175882 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94480] 2014-06-09 16:22:12.175888 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94480] 2014-06-09 16:22:12.175889 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.175890 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.175894 7f86a5e11700 10 _finish_op 0x2fcaea20 seq 4141 osr(1.2ef 0x384d0f0)/0x384d0f0 2014-06-09 16:22:12.175905 7f868bf5a700 10 op_submit_start 4189 2014-06-09 16:22:12.175907 7f868bf5a700 5 queue_transactions (trailing journal) 4189 2014-06-09 16:22:12.175908 7f868bf5a700 5 queue_op 0x324db080 seq 4189 osr(1.14c 0x384c810) 828 bytes (queue has 50 ops and 4237498 bytes) 2014-06-09 16:22:12.175913 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175918 7f8692f68700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175922 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175927 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175931 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175935 7f868a757700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175940 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175944 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.175949 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175953 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 2014-06-09 16:22:12.175958 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175962 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175966 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175971 7f8691765700 2 waiting 51 > 50 ops || 4238356 > 104857600 2014-06-09 16:22:12.175975 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175979 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175984 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.175988 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.175993 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7ce00] 2014-06-09 16:22:12.176006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7ce00] 2014-06-09 16:22:12.176008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.176009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.176012 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd680 (1 active) 2014-06-09 16:22:12.176015 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e120 (2 active) 2014-06-09 16:22:12.176020 7f868bf5a700 10 op_submit_finish 4189 2014-06-09 16:22:12.176024 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.3fe( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.176058 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.176064 7f868bf5a700 5 queue_transactions existing osr(0.3fe 0x2bd7210)/0x2bd7210 2014-06-09 16:22:12.176066 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176071 7f86a5e11700 5 _do_op 0xc2d4d20 seq 4142 osr(0.44f 0x2bd71a0)/0x2bd71a0 start 2014-06-09 16:22:12.176075 7f86a5e11700 10 _do_transaction on 0xbf0fd40 2014-06-09 16:22:12.176081 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.176124 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.176141 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4142.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.176146 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.176158 7f86a5e11700 10 _do_op 0xc2d4d20 seq 4142 r = 0, finisher 0x318c2b60 0 2014-06-09 16:22:12.176168 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314785a0] 2014-06-09 16:22:12.176171 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.176181 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.176190 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 2014-06-09 16:22:12.176199 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314785a0] 2014-06-09 16:22:12.176201 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.176202 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.176206 7f86a5e11700 10 _finish_op 0xc2d4d20 seq 4142 osr(0.44f 0x2bd71a0)/0x2bd71a0 2014-06-09 16:22:12.176216 7f8692f68700 10 op_submit_start 4190 2014-06-09 16:22:12.176217 7f8692f68700 5 queue_transactions (trailing journal) 4190 2014-06-09 16:22:12.176219 7f8692f68700 5 queue_op 0xd3274a0 seq 4190 osr(1.375 0x384d230) 828 bytes (queue has 50 ops and 4237465 bytes) 2014-06-09 16:22:12.176223 7f868b759700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176227 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176231 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176234 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176237 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176240 7f8690763700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.176243 7f8690f64700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176246 7f869c77b700 2 waiting 51 > 50 ops || 4259089 > 104857600 2014-06-09 16:22:12.176250 7f8692767700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176255 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176258 7f868d75d700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176261 7f8691765700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.176264 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176267 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176271 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176275 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176279 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176282 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176285 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176295 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c2b60] 2014-06-09 16:22:12.176304 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c2b60] 2014-06-09 16:22:12.176306 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.176307 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.176310 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e120 (1 active) 2014-06-09 16:22:12.176313 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1b00 (2 active) 2014-06-09 16:22:12.176317 7f8692f68700 10 op_submit_finish 4190 2014-06-09 16:22:12.176321 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.176360 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.176364 7f8692f68700 5 queue_transactions existing osr(1.a 0x384c270)/0x384c270 2014-06-09 16:22:12.176367 7f8692f68700 2 waiting 51 > 50 ops || 4238289 > 104857600 2014-06-09 16:22:12.176371 7f86a5e11700 5 _do_op 0x327fd800 seq 4143 osr(0.66a 0x2bd7ef0)/0x2bd7ef0 start 2014-06-09 16:22:12.176375 7f86a5e11700 10 _do_transaction on 0xbebd8c0 2014-06-09 16:22:12.176380 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.176422 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.176437 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4143.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.176442 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.176453 7f86a5e11700 10 _do_op 0x327fd800 seq 4143 r = 0, finisher 0x32073640 0 2014-06-09 16:22:12.176464 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a6900] 2014-06-09 16:22:12.176466 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a6900] 2014-06-09 16:22:12.176468 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.176469 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.176472 7f86a5e11700 10 _finish_op 0x327fd800 seq 4143 osr(0.66a 0x2bd7ef0)/0x2bd7ef0 2014-06-09 16:22:12.176483 7f868e75f700 10 op_submit_start 4191 2014-06-09 16:22:12.176485 7f868e75f700 5 queue_transactions (trailing journal) 4191 2014-06-09 16:22:12.176487 7f868e75f700 5 queue_op 0xd327ce0 seq 4191 osr(1.24c 0x384cfd0) 828 bytes (queue has 50 ops and 4237465 bytes) 2014-06-09 16:22:12.176492 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176497 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176501 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176505 7f8690763700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.176510 7f8690f64700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176514 7f869c77b700 2 waiting 51 > 50 ops || 4259089 > 104857600 2014-06-09 16:22:12.176518 7f8692767700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176522 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176527 7f868d75d700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176531 7f8691765700 2 waiting 51 > 50 ops || 4238323 > 104857600 2014-06-09 16:22:12.176535 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176539 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176544 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176548 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176552 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176556 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176561 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176565 7f8692f68700 2 waiting 51 > 50 ops || 4238289 > 104857600 2014-06-09 16:22:12.176569 7f868b759700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176579 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32073640] 2014-06-09 16:22:12.176585 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32073640] 2014-06-09 16:22:12.176587 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.176588 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.176591 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1b00 (1 active) 2014-06-09 16:22:12.176594 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3017a0 (2 active) 2014-06-09 16:22:12.176599 7f868e75f700 10 op_submit_finish 4191 2014-06-09 16:22:12.176603 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.3b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.176642 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.176647 7f868e75f700 5 queue_transactions existing osr(0.3b7 0x2bd7290)/0x2bd7290 2014-06-09 16:22:12.176649 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 2014-06-09 16:22:12.176654 7f86a5e11700 5 _do_op 0x327fc420 seq 4144 osr(0.654 0x2bd7b50)/0x2bd7b50 start 2014-06-09 16:22:12.176657 7f86a5e11700 10 _do_transaction on 0xc22b080 2014-06-09 16:22:12.176729 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.176771 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.176786 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4144.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.176791 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.176801 7f86a5e11700 10 _do_op 0x327fc420 seq 4144 r = 0, finisher 0x307f1a40 0 2014-06-09 16:22:12.176812 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e19600] 2014-06-09 16:22:12.176815 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e19600] 2014-06-09 16:22:12.176816 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.176817 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.176820 7f86a5e11700 10 _finish_op 0x327fc420 seq 4144 osr(0.654 0x2bd7b50)/0x2bd7b50 2014-06-09 16:22:12.176831 7f868a757700 10 op_submit_start 4192 2014-06-09 16:22:12.176833 7f868a757700 5 queue_transactions (trailing journal) 4192 2014-06-09 16:22:12.176835 7f868a757700 5 queue_op 0x16097020 seq 4192 osr(3.362 0x3ed10a0) 861 bytes (queue has 50 ops and 4237498 bytes) 2014-06-09 16:22:12.176840 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176844 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.176849 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176853 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 2014-06-09 16:22:12.176857 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.176862 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.176866 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.176870 7f8691765700 2 waiting 51 > 50 ops || 4238356 > 104857600 2014-06-09 16:22:12.176874 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.176879 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176883 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.176888 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176892 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176896 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176900 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.176904 7f8692f68700 2 waiting 51 > 50 ops || 4238322 > 104857600 2014-06-09 16:22:12.176909 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176913 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176917 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 2014-06-09 16:22:12.176927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f1a40] 2014-06-09 16:22:12.176931 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f1a40] 2014-06-09 16:22:12.176932 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.176933 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.176936 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3017a0 (1 active) 2014-06-09 16:22:12.176939 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406c60 (2 active) 2014-06-09 16:22:12.176944 7f868a757700 10 op_submit_finish 4192 2014-06-09 16:22:12.176948 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.176985 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.176990 7f868a757700 5 queue_transactions existing osr(3.470 0x3ed1540)/0x3ed1540 2014-06-09 16:22:12.176992 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 2014-06-09 16:22:12.176997 7f86a5e11700 5 _do_op 0xc2d5aa0 seq 4145 osr(2.3fa 0x2ed12f0)/0x2ed12f0 start 2014-06-09 16:22:12.177001 7f86a5e11700 10 _do_transaction on 0x33380c0 2014-06-09 16:22:12.177007 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.177017 7f86a68f6700 5 -- op tracker -- , seq: 3671, time: 2014-06-09 16:22:12.175327, event: header_read, op: osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.177029 7f86a68f6700 5 -- op tracker -- , seq: 3671, time: 2014-06-09 16:22:12.175328, event: throttled, op: osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.177039 7f86a68f6700 5 -- op tracker -- , seq: 3671, time: 2014-06-09 16:22:12.176978, event: all_read, op: osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.177048 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.177049 7f86a68f6700 5 -- op tracker -- , seq: 3671, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.177057 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.177061 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.177065 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4145.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.177070 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.177077 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d500 prio 63 cost 1032192 latency 0.001750 osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.177103 7f86a5e11700 10 _do_op 0xc2d5aa0 seq 4145 r = 0, finisher 0xbd71e40 0 2014-06-09 16:22:12.177114 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc411020] 2014-06-09 16:22:12.177117 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.177128 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.177137 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 2014-06-09 16:22:12.177148 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc411020] 2014-06-09 16:22:12.177150 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.177151 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.177155 7f86a5e11700 10 _finish_op 0xc2d5aa0 seq 4145 osr(2.3fa 0x2ed12f0)/0x2ed12f0 2014-06-09 16:22:12.177166 7f8690763700 10 op_submit_start 4193 2014-06-09 16:22:12.177168 7f8690763700 5 queue_transactions (trailing journal) 4193 2014-06-09 16:22:12.177170 7f8690763700 5 queue_op 0x3280b1a0 seq 4193 osr(3.7f 0x3ed0200) 239956 bytes (queue has 50 ops and 4476593 bytes) 2014-06-09 16:22:12.177175 7f8690f64700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177180 7f869c77b700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.177197 7f8692767700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.177202 7f8691f66700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.177206 7f868d75d700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.177211 7f8691765700 2 waiting 51 > 50 ops || 4477451 > 104857600 2014-06-09 16:22:12.177215 7f8689f56700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.177220 7f868ff62700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177224 7f868af58700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.177229 7f868f761700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177233 7f868df5e700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177238 7f868bf5a700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177242 7f868c75b700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.177247 7f8692f68700 2 waiting 51 > 50 ops || 4477417 > 104857600 2014-06-09 16:22:12.177251 7f868b759700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177255 7f868e75f700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177260 7f868cf5c700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.177264 7f868a757700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177269 7f868ef60700 2 waiting 51 > 50 ops || 4477421 > 104857600 2014-06-09 16:22:12.177279 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd71e40] 2014-06-09 16:22:12.177284 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd71e40] 2014-06-09 16:22:12.177285 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.177287 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.177290 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406c60 (1 active) 2014-06-09 16:22:12.177292 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52dd40 (2 active) 2014-06-09 16:22:12.177297 7f8690763700 10 op_submit_finish 4193 2014-06-09 16:22:12.177303 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc1fd740 243'534 rep_tid=29 committed?=0 applied?=0 lock=0 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.177331 7f8690763700 10 osd.26 244 dequeue_op 0x2e075000 finish 2014-06-09 16:22:12.177336 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.177343 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.177348 7f8690763700 10 osd.26 244 dequeue_op 0x2976ce00 prio 196 cost 0 latency 0.003726 osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) v2 pg pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.177366 7f8690763700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.177363, event: reached_pg, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) 2014-06-09 16:22:12.177375 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2976ce00 2014-06-09 16:22:12.177391 7f8690763700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.177387, event: started, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) 2014-06-09 16:22:12.177397 7f8690763700 7 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 29 op ack_type 1 from 6 2014-06-09 16:22:12.177415 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.177409, event: sub_op_applied_rec, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.177423 7f8690763700 10 osd.26 244 dequeue_op 0x2976ce00 finish 2014-06-09 16:22:12.177430 7f8690763700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.177427, event: done, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) 2014-06-09 16:22:12.177443 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.177478 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41810 (20 active) 2014-06-09 16:22:12.177483 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177500 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.177513 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177518 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177530 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.177539 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177543 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177555 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.177564 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177569 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177581 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.177591 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177595 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177607 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.177616 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177620 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177632 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.177642 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177646 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177660 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.177671 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177675 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177692 7f86a68f6700 5 -- op tracker -- , seq: 3672, time: 2014-06-09 16:22:12.177099, event: header_read, op: osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.177689 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.177700 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177703 7f86a68f6700 5 -- op tracker -- , seq: 3672, time: 2014-06-09 16:22:12.177100, event: throttled, op: osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.177704 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177716 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.177726 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177730 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177737 7f86a68f6700 5 -- op tracker -- , seq: 3672, time: 2014-06-09 16:22:12.177658, event: all_read, op: osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.177748 7f86a68f6700 5 -- op tracker -- , seq: 3672, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.177743 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.177755 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177753 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.177758 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.177758 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177770 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d400 prio 63 cost 516096 latency 0.000670 osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.177773 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] requeue_ops 0x32827d00 2014-06-09 16:22:12.177789 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177794 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177810 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x1804c300,0x1c66be00,0x1804c200,0x1c66bd00,0x1804c100,0x1c66bc00,0x1804c000,0x1c66bb00,0x21966400,0x21966300,0x19fe1d00,0x19fe1c00 2014-06-09 16:22:12.177852 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177857 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177871 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0xe032700,0xe2ac100,0xe033f00,0xe033e00,0x11e62300,0x4678300,0x4678200,0x4678100,0x4678000,0x2e074900,0x2e074800 2014-06-09 16:22:12.177910 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177915 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177929 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.177940 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177944 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.177957 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.177968 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.177972 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.177986 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.002198 7 0.000447 2014-06-09 16:22:12.177999 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.178011 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.178022 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 2014-06-09 16:22:12.178033 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.178042 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.178049 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.178061 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.178072 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.178082 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.178097 7f8690763700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.178116 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:185 2014-06-09 16:22:12.178150 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.178154 7f8690763700 5 queue_transactions existing osr(0.164 0x2bd6810)/0x2bd6810 2014-06-09 16:22:12.178157 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 2014-06-09 16:22:12.178187 7f86a5e11700 5 _do_op 0x327fc060 seq 4146 osr(3.f0 0x3ed04a0)/0x3ed04a0 start 2014-06-09 16:22:12.178197 7f86a5e11700 10 _do_transaction on 0xbd8f740 2014-06-09 16:22:12.178203 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.178257 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.178273 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4146.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.178278 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.178292 7f86a5e11700 10 _do_op 0x327fc060 seq 4146 r = 0, finisher 0x307f0bc0 0 2014-06-09 16:22:12.178304 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062bac0] 2014-06-09 16:22:12.178307 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062bac0] 2014-06-09 16:22:12.178308 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.178309 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.178313 7f86a5e11700 10 _finish_op 0x327fc060 seq 4146 osr(3.f0 0x3ed04a0)/0x3ed04a0 2014-06-09 16:22:12.178324 7f869c77b700 10 op_submit_start 4194 2014-06-09 16:22:12.178326 7f869c77b700 5 queue_transactions (trailing journal) 4194 2014-06-09 16:22:12.178328 7f869c77b700 5 queue_op 0xc34a840 seq 4194 osr(default 0x2c10280) 21624 bytes (queue has 50 ops and 4497391 bytes) 2014-06-09 16:22:12.178333 7f8692767700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178338 7f8691f66700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178343 7f868d75d700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178348 7f8691765700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.178353 7f8689f56700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178357 7f868ff62700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178362 7f868af58700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178367 7f868f761700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178371 7f868df5e700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178376 7f868bf5a700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178380 7f868c75b700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178385 7f8692f68700 2 waiting 51 > 50 ops || 4498215 > 104857600 2014-06-09 16:22:12.178389 7f868b759700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178394 7f868e75f700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178398 7f868cf5c700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178403 7f868a757700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178407 7f868ef60700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178411 7f8690763700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178416 7f8690f64700 2 waiting 51 > 50 ops || 4498219 > 104857600 2014-06-09 16:22:12.178425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f0bc0] 2014-06-09 16:22:12.178429 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f0bc0] 2014-06-09 16:22:12.178431 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.178432 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.178435 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52dd40 (1 active) 2014-06-09 16:22:12.178438 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f200 (2 active) 2014-06-09 16:22:12.178443 7f869c77b700 10 op_submit_finish 4194 2014-06-09 16:22:12.178451 7f869c77b700 7 osd.26 244 consume_map version 244 2014-06-09 16:22:12.178486 7f86a5e11700 5 _do_op 0x327fd7a0 seq 4147 osr(2.112 0x2ed06d0)/0x2ed06d0 start 2014-06-09 16:22:12.178490 7f86a5e11700 10 _do_transaction on 0xc2ef800 2014-06-09 16:22:12.178496 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.178539 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.178554 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4147.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.178558 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.178571 7f86a5e11700 10 _do_op 0x327fd7a0 seq 4147 r = 0, finisher 0xcc82760 0 2014-06-09 16:22:12.178648 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f0c80] 2014-06-09 16:22:12.178651 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f0c80] 2014-06-09 16:22:12.178653 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.178654 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.178658 7f86a5e11700 10 _finish_op 0x327fd7a0 seq 4147 osr(2.112 0x2ed06d0)/0x2ed06d0 2014-06-09 16:22:12.178669 7f8691f66700 10 op_submit_start 4195 2014-06-09 16:22:12.178671 7f8691f66700 5 queue_transactions (trailing journal) 4195 2014-06-09 16:22:12.178673 7f8691f66700 5 queue_op 0x30060cc0 seq 4195 osr(0.5d4 0x2bd7a10) 861 bytes (queue has 50 ops and 4497424 bytes) 2014-06-09 16:22:12.178678 7f868d75d700 2 waiting 51 > 50 ops || 4498285 > 104857600 2014-06-09 16:22:12.178682 7f8691765700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.178687 7f8689f56700 2 waiting 51 > 50 ops || 4498285 > 104857600 2014-06-09 16:22:12.178692 7f868ff62700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178696 7f868af58700 2 waiting 51 > 50 ops || 4498285 > 104857600 2014-06-09 16:22:12.178700 7f868f761700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178704 7f868df5e700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178709 7f868bf5a700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178713 7f868c75b700 2 waiting 51 > 50 ops || 4498285 > 104857600 2014-06-09 16:22:12.178717 7f8692f68700 2 waiting 51 > 50 ops || 4498248 > 104857600 2014-06-09 16:22:12.178722 7f868b759700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178726 7f868e75f700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178730 7f868cf5c700 2 waiting 51 > 50 ops || 4498285 > 104857600 2014-06-09 16:22:12.178735 7f868a757700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178739 7f868ef60700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178743 7f8690763700 2 waiting 51 > 50 ops || 4498285 > 104857600 2014-06-09 16:22:12.178748 7f8690f64700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.178753 7f8692767700 2 waiting 51 > 50 ops || 4498285 > 104857600 2014-06-09 16:22:12.178763 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc82760] 2014-06-09 16:22:12.178770 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc82760] 2014-06-09 16:22:12.178772 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.178773 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.178776 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f200 (1 active) 2014-06-09 16:22:12.178779 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b7a0 (2 active) 2014-06-09 16:22:12.178784 7f8691f66700 10 op_submit_finish 4195 2014-06-09 16:22:12.178808 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (19 active) 2014-06-09 16:22:12.178816 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.178820 7f8691f66700 10 osd.26 244 dequeue_op 0x2976d300 prio 196 cost 0 latency 0.004497 osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) v2 pg pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.178836 7f8691f66700 20 osd.26 244 share_map osd.6 172.16.96.201:6849/52045 243 2014-06-09 16:22:12.178841 7f8691f66700 20 osd.26 244 should_share_map osd.6 172.16.96.201:6849/52045 243 2014-06-09 16:22:12.178846 7f8691f66700 10 osd.26 244 osd.6 172.16.96.201:6849/52045 has old map 243 < 244 2014-06-09 16:22:12.178850 7f8691f66700 10 osd.26 244 osd.6 172.16.96.201:6849/52045 has old map 243 < 244 2014-06-09 16:22:12.178854 7f8691f66700 10 osd.26 243 note_peer_epoch osd.6 has 244 2014-06-09 16:22:12.178856 7f8691f66700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbf5adc0 172.16.96.201:6849/52045 2014-06-09 16:22:12.178882 7f8691f66700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.178877, event: reached_pg, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.178889 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2976d300 2014-06-09 16:22:12.178904 7f8691f66700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.178901, event: started, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.178910 7f8691f66700 7 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 29 op ack_type 4 from 6 2014-06-09 16:22:12.178930 7f8691f66700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.178923, event: sub_op_commit_rec, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.178936 7f8691f66700 10 osd.26 244 dequeue_op 0x2976d300 finish 2014-06-09 16:22:12.178943 7f8691f66700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.178940, event: done, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) 2014-06-09 16:22:12.178961 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.178985 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (20 active) 2014-06-09 16:22:12.178989 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.179004 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] exit Started/Primary/Active/Activating 1.015908 7 0.000432 2014-06-09 16:22:12.179017 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.179028 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] needs_recovery is recovered 2014-06-09 16:22:12.179040 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.179052 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.179061 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] finish_recovery 2014-06-09 16:22:12.179070 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] clear_recovery_state 2014-06-09 16:22:12.179082 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.179094 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'636 mlcod 214'636 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.179105 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 214'636 mlcod 214'636 active+clean] share_pg_info 2014-06-09 16:22:12.179120 7f8691f66700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbeb8f20 172.16.96.201:6865/53518 2014-06-09 16:22:12.179140 7f8691f66700 10 osd.26 243 note_peer_epoch osd.9 has 244 2014-06-09 16:22:12.179154 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 214'636 mlcod 214'636 active+clean] publish_stats_to_osd 243:1062 2014-06-09 16:22:12.179183 7f8689755700 10 op_submit_start 4196 2014-06-09 16:22:12.179187 7f8689755700 5 queue_transactions (trailing journal) 4196 2014-06-09 16:22:12.179190 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.179189 7f8689755700 5 queue_op 0x324a5da0 seq 4196 osr(3.375 0x3ed1060) 828 bytes (queue has 50 ops and 4497424 bytes) 2014-06-09 16:22:12.179193 7f8689755700 10 op_submit_finish 4196 2014-06-09 16:22:12.179194 7f8691f66700 5 queue_transactions existing osr(3.2b3 0x3ed0f00)/0x3ed0f00 2014-06-09 16:22:12.179197 7f8691f66700 2 waiting 51 > 50 ops || 4498285 > 104857600 2014-06-09 16:22:12.178947 7f86a5e11700 5 _do_op 0x327fc540 seq 4148 osr(0.765 0x384c360)/0x384c360 start 2014-06-09 16:22:12.179206 7f86a5e11700 10 _do_transaction on 0x399c780 2014-06-09 16:22:12.179211 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.179198 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.1d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.179243 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.179249 7f8689755700 5 queue_transactions existing osr(0.1d4 0x2bd6940)/0x2bd6940 2014-06-09 16:22:12.179252 7f8689755700 2 waiting 51 > 50 ops || 4498252 > 104857600 2014-06-09 16:22:12.179256 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.179271 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4148.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.179275 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.179287 7f86a5e11700 10 _do_op 0x327fc540 seq 4148 r = 0, finisher 0x31456a80 0 2014-06-09 16:22:12.179298 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f0940] 2014-06-09 16:22:12.179301 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f0940] 2014-06-09 16:22:12.179302 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.179303 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.179307 7f86a5e11700 10 _finish_op 0x327fc540 seq 4148 osr(0.765 0x384c360)/0x384c360 2014-06-09 16:22:12.179318 7f8691765700 10 op_submit_start 4197 2014-06-09 16:22:12.179320 7f8691765700 5 queue_transactions (trailing journal) 4197 2014-06-09 16:22:12.179322 7f8691765700 5 queue_op 0xc64a480 seq 4197 osr(0.dc 0x2bd6570) 858 bytes (queue has 50 ops and 4497454 bytes) 2014-06-09 16:22:12.179327 7f8689f56700 2 waiting 51 > 50 ops || 4498315 > 104857600 2014-06-09 16:22:12.179332 7f868ff62700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179336 7f868af58700 2 waiting 51 > 50 ops || 4498315 > 104857600 2014-06-09 16:22:12.179340 7f868f761700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179345 7f868df5e700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179349 7f868bf5a700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179354 7f868c75b700 2 waiting 51 > 50 ops || 4498315 > 104857600 2014-06-09 16:22:12.179358 7f8692f68700 2 waiting 51 > 50 ops || 4498278 > 104857600 2014-06-09 16:22:12.179363 7f868b759700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179367 7f868e75f700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179372 7f868cf5c700 2 waiting 51 > 50 ops || 4498315 > 104857600 2014-06-09 16:22:12.179376 7f868a757700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179381 7f868ef60700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179385 7f8690763700 2 waiting 51 > 50 ops || 4498315 > 104857600 2014-06-09 16:22:12.179389 7f8690f64700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179393 7f8692767700 2 waiting 51 > 50 ops || 4498315 > 104857600 2014-06-09 16:22:12.179398 7f8691f66700 2 waiting 51 > 50 ops || 4498315 > 104857600 2014-06-09 16:22:12.179409 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31456a80] 2014-06-09 16:22:12.179413 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31456a80] 2014-06-09 16:22:12.179414 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.179415 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.179418 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b7a0 (1 active) 2014-06-09 16:22:12.179421 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa240 (2 active) 2014-06-09 16:22:12.179426 7f8691765700 10 op_submit_finish 4197 2014-06-09 16:22:12.179430 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.179444 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.111301 7 0.000311 2014-06-09 16:22:12.179456 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.179466 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.179477 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.179487 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.179496 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.179503 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.179514 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.179525 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.179535 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.179551 7f8691765700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 2014-06-09 16:22:12.179567 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:171 2014-06-09 16:22:12.179598 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.179602 7f8691765700 5 queue_transactions existing osr(1.b3 0x384c6d0)/0x384c6d0 2014-06-09 16:22:12.179605 7f8691765700 2 waiting 51 > 50 ops || 4498312 > 104857600 2014-06-09 16:22:12.179610 7f86a5e11700 5 _do_op 0xc2d4000 seq 4149 osr(2.3bd 0x2ed1010)/0x2ed1010 start 2014-06-09 16:22:12.179633 7f86a5e11700 10 _do_transaction on 0xbf27740 2014-06-09 16:22:12.179633 7f8689755700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179641 7f868d75d700 2 waiting 51 > 50 ops || 4498315 > 104857600 2014-06-09 16:22:12.179646 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.179702 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.179721 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4149.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.179725 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.179762 7f86a5e11700 10 _do_op 0xc2d4000 seq 4149 r = 0, finisher 0xc1685a0 0 2014-06-09 16:22:12.179774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7c640] 2014-06-09 16:22:12.179777 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.179788 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.179797 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:12.179806 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7c640] 2014-06-09 16:22:12.179808 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.179809 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.179813 7f86a5e11700 10 _finish_op 0xc2d4000 seq 4149 osr(2.3bd 0x2ed1010)/0x2ed1010 2014-06-09 16:22:12.179825 7f868ff62700 10 op_submit_start 4198 2014-06-09 16:22:12.179827 7f868ff62700 5 queue_transactions (trailing journal) 4198 2014-06-09 16:22:12.179828 7f868ff62700 5 queue_op 0x30a38c0 seq 4198 osr(1.206 0x384cd10) 828 bytes (queue has 50 ops and 4497421 bytes) 2014-06-09 16:22:12.179833 7f868af58700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179838 7f868f761700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.179842 7f868df5e700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.179847 7f868bf5a700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.179851 7f868c75b700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179856 7f8692f68700 2 waiting 51 > 50 ops || 4498245 > 104857600 2014-06-09 16:22:12.179860 7f868b759700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.179864 7f868e75f700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.179868 7f868cf5c700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179873 7f868a757700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.179877 7f868ef60700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.179881 7f8690763700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179885 7f8690f64700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.179889 7f8692767700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179894 7f8691f66700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.179898 7f8691765700 2 waiting 51 > 50 ops || 4498279 > 104857600 2014-06-09 16:22:12.179909 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc1685a0] 2014-06-09 16:22:12.179914 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc1685a0] 2014-06-09 16:22:12.179915 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.179917 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.179920 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa240 (1 active) 2014-06-09 16:22:12.179922 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524480 (2 active) 2014-06-09 16:22:12.179928 7f868ff62700 10 op_submit_finish 4198 2014-06-09 16:22:12.179947 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e130 (19 active) 2014-06-09 16:22:12.179954 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.179958 7f868ff62700 10 osd.26 244 dequeue_op 0x14053a00 prio 63 cost 16384 latency 1.915519 osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.179984 7f868ff62700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.179978, event: reached_pg, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.179992 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.180006 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x14053a00 2014-06-09 16:22:12.180016 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.180040 7f868ff62700 15 getattr 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 '_' 2014-06-09 16:22:12.180116 7f868ff62700 10 getattr got.size() is 0 2014-06-09 16:22:12.180120 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 but can_create 2014-06-09 16:22:12.180134 7f868ff62700 15 getattr 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 'snapset' 2014-06-09 16:22:12.180165 7f868ff62700 10 getattr got.size() is 0 2014-06-09 16:22:12.180168 7f868ff62700 15 getattr 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/snapdir//3 'snapset' 2014-06-09 16:22:12.180198 7f868ff62700 10 getattr got.size() is 0 2014-06-09 16:22:12.180207 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30bd3600 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.180219 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.180229 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.180239 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30bd3600 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 rwstate(none n=0 w=0) oi: 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc2380 snapset: 0=[]:[] 2014-06-09 16:22:12.180255 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 @head oi=3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.180274 7f868ff62700 15 getattr 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/snapdir//3 '_' 2014-06-09 16:22:12.180308 7f868ff62700 10 getattr got.size() is 0 2014-06-09 16:22:12.180310 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/snapdir//3 and !can_create 2014-06-09 16:22:12.180330 7f868ff62700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.180324, event: started, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.180340 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x240f2000 2014-06-09 16:22:12.180368 7f868ff62700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.180353, event: started, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.180376 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] ov 0'0 av 243'664 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.180391 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 2014-06-09 16:22:12.180402 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.182122 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~16384 2014-06-09 16:22:12.182154 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229200 2014-06-09 16:22:12.182166 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 snapset=0x14dc23b8 snapc=0=[] 2014-06-09 16:22:12.182178 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.182189 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.182199 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 0x240f2000 op modify 2014-06-09 16:22:12.182212 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:10.931984 2014-06-09 16:22:12.182234 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.182253 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.182264 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 30 on osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.182280 7f868ff62700 7 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 30 o 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.182305 7f868ff62700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.182300, event: waiting for subops from 4, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.182326 7f868ff62700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbfb5e40 172.16.96.201:6832/51134 2014-06-09 16:22:12.182349 7f868ff62700 10 osd.26 243 note_peer_epoch osd.4 has 244 2014-06-09 16:22:12.182365 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'663], crt=0'0) [243'664 (0'0) modify 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 by client.4566.0:33893 2014-06-09 10:19:10.931984] 2014-06-09 16:22:12.182387 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'664 (0'0) modify 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 by client.4566.0:33893 2014-06-09 10:19:10.931984 2014-06-09 16:22:12.182472 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.182506 7f868ff62700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'664, trimmed: 2014-06-09 16:22:12.182527 7f868ff62700 5 queue_transactions existing osr(3.2ac 0x3ed0b50)/0x3ed0b50 2014-06-09 16:22:12.180415 7f86a5e11700 5 _do_op 0xc2d43c0 seq 4150 osr(0.3f9 0x2bd7230)/0x2bd7230 start 2014-06-09 16:22:12.180436 7f8689755700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182542 7f868d75d700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.182547 7f8689f56700 2 waiting 51 > 50 ops || 4498282 > 104857600 2014-06-09 16:22:12.182552 7f868ff62700 2 waiting 51 > 50 ops || 4516203 > 104857600 2014-06-09 16:22:12.182557 7f86a5e11700 10 _do_transaction on 0xc161200 2014-06-09 16:22:12.182565 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.182619 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.182638 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4150.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.182643 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.182657 7f86a5e11700 10 _do_op 0xc2d43c0 seq 4150 r = 0, finisher 0xcc94d40 0 2014-06-09 16:22:12.182669 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1b0c1e60] 2014-06-09 16:22:12.182672 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.182683 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.182692 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 2014-06-09 16:22:12.182702 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1b0c1e60] 2014-06-09 16:22:12.182704 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.182705 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.182709 7f86a5e11700 10 _finish_op 0xc2d43c0 seq 4150 osr(0.3f9 0x2bd7230)/0x2bd7230 2014-06-09 16:22:12.182722 7f868f761700 10 op_submit_start 4199 2014-06-09 16:22:12.182724 7f868f761700 5 queue_transactions (trailing journal) 4199 2014-06-09 16:22:12.182725 7f868f761700 5 queue_op 0xbdc82a0 seq 4199 osr(3.18e 0x3ed0af0) 828 bytes (queue has 50 ops and 4497388 bytes) 2014-06-09 16:22:12.182731 7f868df5e700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.182736 7f868bf5a700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.182741 7f868c75b700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182746 7f8692f68700 2 waiting 51 > 50 ops || 4498212 > 104857600 2014-06-09 16:22:12.182750 7f868b759700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.182755 7f868e75f700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.182759 7f868cf5c700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182764 7f868a757700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.182768 7f868ef60700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.182773 7f8690763700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182778 7f8690f64700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.182783 7f8692767700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182787 7f8691f66700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182792 7f8691765700 2 waiting 51 > 50 ops || 4498246 > 104857600 2014-06-09 16:22:12.182796 7f8689755700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.182801 7f868d75d700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182805 7f8689f56700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182809 7f868ff62700 2 waiting 51 > 50 ops || 4516170 > 104857600 2014-06-09 16:22:12.182813 7f868af58700 2 waiting 51 > 50 ops || 4498249 > 104857600 2014-06-09 16:22:12.182825 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc94d40] 2014-06-09 16:22:12.182829 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc94d40] 2014-06-09 16:22:12.182831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.182832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.182835 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524480 (1 active) 2014-06-09 16:22:12.182838 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115c20 (2 active) 2014-06-09 16:22:12.182843 7f868f761700 10 op_submit_finish 4199 2014-06-09 16:22:12.182848 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.ab( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.182889 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.182894 7f868f761700 5 queue_transactions existing osr(1.ab 0x384c6f0)/0x384c6f0 2014-06-09 16:22:12.182897 7f868f761700 2 waiting 51 > 50 ops || 4498214 > 104857600 2014-06-09 16:22:12.182902 7f86a5e11700 5 _do_op 0xc2d46c0 seq 4151 osr(1.6b7 0x2ed01d0)/0x2ed01d0 start 2014-06-09 16:22:12.182906 7f86a5e11700 10 _do_transaction on 0x331f980 2014-06-09 16:22:12.182912 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.182955 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.182972 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4151.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.182976 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.182995 7f86a5e11700 10 _do_op 0xc2d46c0 seq 4151 r = 0, finisher 0x13964780 0 2014-06-09 16:22:12.183006 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6c9e0] 2014-06-09 16:22:12.183009 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.183019 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.183029 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 2014-06-09 16:22:12.183038 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6c9e0] 2014-06-09 16:22:12.183040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.183041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.183044 7f86a5e11700 10 _finish_op 0xc2d46c0 seq 4151 osr(1.6b7 0x2ed01d0)/0x2ed01d0 2014-06-09 16:22:12.183056 7f868bf5a700 10 op_submit_start 4200 2014-06-09 16:22:12.183057 7f868bf5a700 5 queue_transactions (trailing journal) 4200 2014-06-09 16:22:12.183059 7f868bf5a700 5 queue_op 0x324db380 seq 4200 osr(0.3fe 0x2bd7210) 828 bytes (queue has 50 ops and 4497355 bytes) 2014-06-09 16:22:12.183064 7f868c75b700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.183069 7f8692f68700 2 waiting 51 > 50 ops || 4498179 > 104857600 2014-06-09 16:22:12.183073 7f868b759700 2 waiting 51 > 50 ops || 4498183 > 104857600 2014-06-09 16:22:12.183077 7f868e75f700 2 waiting 51 > 50 ops || 4498183 > 104857600 2014-06-09 16:22:12.183082 7f868cf5c700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.183086 7f868a757700 2 waiting 51 > 50 ops || 4498183 > 104857600 2014-06-09 16:22:12.183091 7f868ef60700 2 waiting 51 > 50 ops || 4498183 > 104857600 2014-06-09 16:22:12.183095 7f8690763700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.183099 7f8690f64700 2 waiting 51 > 50 ops || 4498183 > 104857600 2014-06-09 16:22:12.183104 7f8692767700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.183108 7f8691f66700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.183112 7f8691765700 2 waiting 51 > 50 ops || 4498213 > 104857600 2014-06-09 16:22:12.183117 7f8689755700 2 waiting 51 > 50 ops || 4498183 > 104857600 2014-06-09 16:22:12.183121 7f868d75d700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.183125 7f8689f56700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.183129 7f868ff62700 2 waiting 51 > 50 ops || 4516137 > 104857600 2014-06-09 16:22:12.183133 7f868af58700 2 waiting 51 > 50 ops || 4498216 > 104857600 2014-06-09 16:22:12.183138 7f868f761700 2 waiting 51 > 50 ops || 4498181 > 104857600 2014-06-09 16:22:12.183142 7f868df5e700 2 waiting 51 > 50 ops || 4498183 > 104857600 2014-06-09 16:22:12.183152 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13964780] 2014-06-09 16:22:12.183158 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13964780] 2014-06-09 16:22:12.183160 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.183161 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.183164 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115c20 (1 active) 2014-06-09 16:22:12.183167 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89ad80 (2 active) 2014-06-09 16:22:12.183172 7f868bf5a700 10 op_submit_finish 4200 2014-06-09 16:22:12.183176 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.183191 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.183200 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.183209 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.183218 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.183227 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.183242 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 2014-06-09 16:22:12.183251 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.183259 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.183269 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.183275 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 243/243 242/242/202) 2014-06-09 16:22:12.183308 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.183313 7f868bf5a700 5 queue_transactions existing osr(3.527 0x3ed1660)/0x3ed1660 2014-06-09 16:22:12.183315 7f868bf5a700 2 waiting 51 > 50 ops || 4498183 > 104857600 2014-06-09 16:22:12.183320 7f86a5e11700 5 _do_op 0x324e6780 seq 4152 osr(2.d 0x2ed02b0)/0x2ed02b0 start 2014-06-09 16:22:12.183324 7f86a5e11700 10 _do_transaction on 0xc058840 2014-06-09 16:22:12.183330 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.183372 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.183389 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4152.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.183393 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.183408 7f86a5e11700 10 _do_op 0x324e6780 seq 4152 r = 0, finisher 0xbd73580 0 2014-06-09 16:22:12.183418 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccc4a0] 2014-06-09 16:22:12.183421 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.183431 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.183440 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:154 2014-06-09 16:22:12.183449 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccc4a0] 2014-06-09 16:22:12.183451 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.183452 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.183456 7f86a5e11700 10 _finish_op 0x324e6780 seq 4152 osr(2.d 0x2ed02b0)/0x2ed02b0 2014-06-09 16:22:12.183468 7f8692f68700 10 op_submit_start 4201 2014-06-09 16:22:12.183469 7f8692f68700 5 queue_transactions (trailing journal) 4201 2014-06-09 16:22:12.183471 7f8692f68700 5 queue_op 0xd326ae0 seq 4201 osr(1.a 0x384c270) 824 bytes (queue has 50 ops and 4497324 bytes) 2014-06-09 16:22:12.183476 7f868b759700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183481 7f868e75f700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183485 7f868cf5c700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183489 7f868a757700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183494 7f868ef60700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183498 7f8690763700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183503 7f8690f64700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183507 7f8692767700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183511 7f8691f66700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183515 7f8691765700 2 waiting 51 > 50 ops || 4498182 > 104857600 2014-06-09 16:22:12.183519 7f8689755700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183524 7f868d75d700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183528 7f8689f56700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183532 7f868ff62700 2 waiting 51 > 50 ops || 4516106 > 104857600 2014-06-09 16:22:12.183536 7f868af58700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183541 7f868f761700 2 waiting 51 > 50 ops || 4498150 > 104857600 2014-06-09 16:22:12.183545 7f868df5e700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183549 7f868bf5a700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183553 7f868c75b700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183563 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd73580] 2014-06-09 16:22:12.183570 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd73580] 2014-06-09 16:22:12.183571 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.183573 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.183576 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89ad80 (1 active) 2014-06-09 16:22:12.183578 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781b00 (2 active) 2014-06-09 16:22:12.183583 7f8692f68700 10 op_submit_finish 4201 2014-06-09 16:22:12.183587 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.24d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.183626 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.183632 7f8692f68700 5 queue_transactions existing osr(2.24d 0x2ed0a50)/0x2ed0a50 2014-06-09 16:22:12.183634 7f8692f68700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183639 7f86a5e11700 5 _do_op 0x16fd4c60 seq 4153 osr(1.1e8 0x384cdf0)/0x384cdf0 start 2014-06-09 16:22:12.183642 7f86a5e11700 10 _do_transaction on 0xc144cc0 2014-06-09 16:22:12.183648 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.183690 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.183705 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4153.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.183709 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.183734 7f86a5e11700 10 _do_op 0x16fd4c60 seq 4153 r = 0, finisher 0x3196c160 0 2014-06-09 16:22:12.183745 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3196c2e0] 2014-06-09 16:22:12.183748 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3196c2e0] 2014-06-09 16:22:12.183750 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.183751 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.183754 7f86a5e11700 10 _finish_op 0x16fd4c60 seq 4153 osr(1.1e8 0x384cdf0)/0x384cdf0 2014-06-09 16:22:12.183765 7f868e75f700 10 op_submit_start 4202 2014-06-09 16:22:12.183767 7f868e75f700 5 queue_transactions (trailing journal) 4202 2014-06-09 16:22:12.183768 7f868e75f700 5 queue_op 0xc34aba0 seq 4202 osr(0.3b7 0x2bd7290) 828 bytes (queue has 50 ops and 4497324 bytes) 2014-06-09 16:22:12.183773 7f868cf5c700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183778 7f868a757700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183782 7f868ef60700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183787 7f8690763700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183791 7f8690f64700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183795 7f8692767700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183800 7f8691f66700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183804 7f8691765700 2 waiting 51 > 50 ops || 4498182 > 104857600 2014-06-09 16:22:12.183809 7f8689755700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183813 7f868d75d700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183817 7f8689f56700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183821 7f868ff62700 2 waiting 51 > 50 ops || 4516106 > 104857600 2014-06-09 16:22:12.183826 7f868af58700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183830 7f868f761700 2 waiting 51 > 50 ops || 4498150 > 104857600 2014-06-09 16:22:12.183835 7f868df5e700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183839 7f868bf5a700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183843 7f868c75b700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.183848 7f8692f68700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183852 7f868b759700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.183862 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3196c160] 2014-06-09 16:22:12.183865 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3196c160] 2014-06-09 16:22:12.183867 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.183868 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.183871 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781b00 (1 active) 2014-06-09 16:22:12.183874 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1560 (2 active) 2014-06-09 16:22:12.183879 7f868e75f700 10 op_submit_finish 4202 2014-06-09 16:22:12.183893 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.183909 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.183920 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.183926 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.183938 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.183948 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.183952 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.183971 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.183976 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.183989 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x306b800,0x306bb00,0x306ba00,0xe033d00,0xe033c00,0xe033b00,0x10c54d00,0x10c54c00,0x10c54b00,0x16bd7300,0x16bd7200,0x16bd7100,0x16bd7000,0x16bd6f00 2014-06-09 16:22:12.184038 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184043 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184055 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.184065 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184069 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184081 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.184090 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184094 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184108 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.184119 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184123 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184135 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.184144 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184149 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184162 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.184172 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184177 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184189 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.184198 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184202 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184215 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.184226 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184230 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184243 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.184555 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184561 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184574 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.184583 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184587 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184600 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.184612 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184616 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184637 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.184646 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184650 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184662 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.184671 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184676 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184690 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.184700 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184705 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.184717 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.184726 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184746 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40e90 (19 active) 2014-06-09 16:22:12.184771 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40e90 (20 active) 2014-06-09 16:22:12.184775 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.473( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.184810 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.184814 7f868e75f700 5 queue_transactions existing osr(0.473 0x2bd74d0)/0x2bd74d0 2014-06-09 16:22:12.184817 7f868e75f700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.184822 7f86a5e11700 5 _do_op 0xc34a6c0 seq 4154 osr(0.230 0x2bd6a10)/0x2bd6a10 start 2014-06-09 16:22:12.184826 7f86a5e11700 10 _do_transaction on 0xc216e40 2014-06-09 16:22:12.184832 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.184878 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.184896 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4154.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.184900 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.184914 7f86a5e11700 10 _do_op 0xc34a6c0 seq 4154 r = 0, finisher 0x189aabc0 0 2014-06-09 16:22:12.184926 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd766e0] 2014-06-09 16:22:12.184929 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.184940 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.184949 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 2014-06-09 16:22:12.184959 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd766e0] 2014-06-09 16:22:12.184960 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.184962 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.184966 7f86a5e11700 10 _finish_op 0xc34a6c0 seq 4154 osr(0.230 0x2bd6a10)/0x2bd6a10 2014-06-09 16:22:12.184977 7f868a757700 10 op_submit_start 4203 2014-06-09 16:22:12.184979 7f868a757700 5 queue_transactions (trailing journal) 4203 2014-06-09 16:22:12.184980 7f868a757700 5 queue_op 0xd326f00 seq 4203 osr(3.470 0x3ed1540) 828 bytes (queue has 50 ops and 4497291 bytes) 2014-06-09 16:22:12.184985 7f868ef60700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.184991 7f8690763700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.184996 7f8690f64700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185000 7f8692767700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185004 7f8691f66700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185009 7f8691765700 2 waiting 51 > 50 ops || 4498149 > 104857600 2014-06-09 16:22:12.185014 7f8689755700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185018 7f868d75d700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185023 7f8689f56700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185027 7f868ff62700 2 waiting 51 > 50 ops || 4516073 > 104857600 2014-06-09 16:22:12.185031 7f868af58700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185035 7f868f761700 2 waiting 51 > 50 ops || 4498117 > 104857600 2014-06-09 16:22:12.185040 7f868df5e700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185044 7f868bf5a700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185048 7f868c75b700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185053 7f8692f68700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185057 7f868b759700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185062 7f868e75f700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185066 7f868cf5c700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185077 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aabc0] 2014-06-09 16:22:12.185080 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aabc0] 2014-06-09 16:22:12.185082 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.185083 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.185086 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1560 (1 active) 2014-06-09 16:22:12.185089 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e900 (2 active) 2014-06-09 16:22:12.185094 7f868a757700 10 op_submit_finish 4203 2014-06-09 16:22:12.185098 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.4ba( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.185133 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.185137 7f868a757700 5 queue_transactions existing osr(0.4ba 0x2bd7710)/0x2bd7710 2014-06-09 16:22:12.185140 7f868a757700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185145 7f86a5e11700 5 _do_op 0x16097080 seq 4155 osr(1.54c 0x384dad0)/0x384dad0 start 2014-06-09 16:22:12.185149 7f86a5e11700 10 _do_transaction on 0x39db440 2014-06-09 16:22:12.185155 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.185198 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.185215 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4155.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.185219 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.185235 7f86a5e11700 10 _do_op 0x16097080 seq 4155 r = 0, finisher 0x2d7398c0 0 2014-06-09 16:22:12.185246 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc410440] 2014-06-09 16:22:12.185249 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.185260 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.185269 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.185278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc410440] 2014-06-09 16:22:12.185280 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.185281 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.185285 7f86a5e11700 10 _finish_op 0x16097080 seq 4155 osr(1.54c 0x384dad0)/0x384dad0 2014-06-09 16:22:12.185296 7f8690763700 10 op_submit_start 4204 2014-06-09 16:22:12.185298 7f8690763700 5 queue_transactions (trailing journal) 4204 2014-06-09 16:22:12.185299 7f8690763700 5 queue_op 0xc2d4b40 seq 4204 osr(0.164 0x2bd6810) 861 bytes (queue has 50 ops and 4497291 bytes) 2014-06-09 16:22:12.185304 7f8690f64700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185309 7f8692767700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185313 7f8691f66700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185318 7f8691765700 2 waiting 51 > 50 ops || 4498149 > 104857600 2014-06-09 16:22:12.185322 7f8689755700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185327 7f868d75d700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185331 7f8689f56700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185335 7f868ff62700 2 waiting 51 > 50 ops || 4516073 > 104857600 2014-06-09 16:22:12.185340 7f868af58700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185344 7f868f761700 2 waiting 51 > 50 ops || 4498117 > 104857600 2014-06-09 16:22:12.185349 7f868df5e700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185353 7f868bf5a700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185358 7f868c75b700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185362 7f8692f68700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185366 7f868b759700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185370 7f868e75f700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185374 7f868cf5c700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185378 7f868a757700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185382 7f868ef60700 2 waiting 51 > 50 ops || 4498119 > 104857600 2014-06-09 16:22:12.185393 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d7398c0] 2014-06-09 16:22:12.185397 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d7398c0] 2014-06-09 16:22:12.185399 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.185400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.185403 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e900 (1 active) 2014-06-09 16:22:12.185406 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780d80 (2 active) 2014-06-09 16:22:12.185411 7f8690763700 10 op_submit_finish 4204 2014-06-09 16:22:12.185415 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.185436 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.185441 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.185454 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0xe032100,0xe032500,0xe032400,0xe032300,0xe2acd00,0xe2acc00,0xe2acb00,0xe2aca00 2014-06-09 16:22:12.185489 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.185494 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.185507 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.948935 7 0.000495 2014-06-09 16:22:12.185519 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.185530 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.185541 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.185552 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.185561 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.185569 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.185579 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.185591 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.185600 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.185609 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.185619 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.185633 7f8690763700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbf6e6e0 172.16.96.201:6801/57551 2014-06-09 16:22:12.185655 7f8690763700 10 osd.26 243 note_peer_epoch osd.0 has 244 2014-06-09 16:22:12.185667 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 2014-06-09 16:22:12.185702 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.185706 7f8690763700 5 queue_transactions existing osr(2.754 0x3ed00f0)/0x3ed00f0 2014-06-09 16:22:12.185709 7f8690763700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185749 7f86a5e11700 5 _do_op 0xbf7e000 seq 4156 osr(3.172 0x3ed06e0)/0x3ed06e0 start 2014-06-09 16:22:12.185756 7f86a5e11700 10 _do_transaction on 0xbebc480 2014-06-09 16:22:12.185762 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.185816 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.185831 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4156.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.185836 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.185850 7f86a5e11700 10 _do_op 0xbf7e000 seq 4156 r = 0, finisher 0x30d616a0 0 2014-06-09 16:22:12.185861 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32079220] 2014-06-09 16:22:12.185863 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32079220] 2014-06-09 16:22:12.185865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.185866 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.185870 7f86a5e11700 10 _finish_op 0xbf7e000 seq 4156 osr(3.172 0x3ed06e0)/0x3ed06e0 2014-06-09 16:22:12.185881 7f8692767700 10 op_submit_start 4205 2014-06-09 16:22:12.185883 7f8692767700 5 queue_transactions (trailing journal) 4205 2014-06-09 16:22:12.185884 7f8692767700 5 queue_op 0x30607980 seq 4205 osr(0.319 0x2bd6d90) 861 bytes (queue has 50 ops and 4497324 bytes) 2014-06-09 16:22:12.185889 7f8691f66700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.185894 7f8691765700 2 waiting 51 > 50 ops || 4498182 > 104857600 2014-06-09 16:22:12.185898 7f8689755700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185903 7f868d75d700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.185907 7f8689f56700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.185911 7f868ff62700 2 waiting 51 > 50 ops || 4516106 > 104857600 2014-06-09 16:22:12.185916 7f868af58700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.185920 7f868f761700 2 waiting 51 > 50 ops || 4498150 > 104857600 2014-06-09 16:22:12.185924 7f868df5e700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185929 7f868bf5a700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185933 7f868c75b700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.185937 7f8692f68700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185941 7f868b759700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185945 7f868e75f700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185949 7f868cf5c700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.185954 7f868a757700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185958 7f868ef60700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185963 7f8690763700 2 waiting 51 > 50 ops || 4498185 > 104857600 2014-06-09 16:22:12.185967 7f8690f64700 2 waiting 51 > 50 ops || 4498152 > 104857600 2014-06-09 16:22:12.185976 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30d616a0] 2014-06-09 16:22:12.185981 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30d616a0] 2014-06-09 16:22:12.185983 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.185984 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.185987 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780d80 (1 active) 2014-06-09 16:22:12.185990 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114120 (2 active) 2014-06-09 16:22:12.185995 7f8692767700 10 op_submit_finish 4205 2014-06-09 16:22:12.186001 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.186021 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] check_local 2014-06-09 16:22:12.186033 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] requeue_ops 0xe2ad300,0xe2ad200,0xe2ad100,0x10c54500,0x10c54400,0x10c54300 2014-06-09 16:22:12.186061 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] needs_recovery primary has 2 missing 2014-06-09 16:22:12.186075 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.186093 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] publish_stats_to_osd 243:1031 2014-06-09 16:22:12.186106 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] hit_set_clear 2014-06-09 16:22:12.186118 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] agent_stop 2014-06-09 16:22:12.186131 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.186135 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.186148 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.120495 7 0.009110 2014-06-09 16:22:12.186159 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.186169 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.186179 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.186190 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.186199 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.186206 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.186217 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.186229 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.186238 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.186304 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.186313 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.186326 7f8692767700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 2014-06-09 16:22:12.186341 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 2014-06-09 16:22:12.186375 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.186379 7f8692767700 5 queue_transactions existing osr(0.6e 0x2bd6390)/0x2bd6390 2014-06-09 16:22:12.186382 7f8692767700 2 waiting 51 > 50 ops || 4498182 > 104857600 2014-06-09 16:22:12.186425 7f86a5e11700 5 _do_op 0xc88bce0 seq 4157 osr(0.9d 0x2bd6490)/0x2bd6490 start 2014-06-09 16:22:12.186433 7f86a5e11700 10 _do_transaction on 0x2eaafc0 2014-06-09 16:22:12.186439 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.186491 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.186507 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4157.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.186511 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.186523 7f86a5e11700 10 _do_op 0xc88bce0 seq 4157 r = 0, finisher 0xbd5f120 0 2014-06-09 16:22:12.186547 7f86a68f6700 5 -- op tracker -- , seq: 3673, time: 2014-06-09 16:22:12.177787, event: header_read, op: osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.186562 7f86a68f6700 5 -- op tracker -- , seq: 3673, time: 2014-06-09 16:22:12.177788, event: throttled, op: osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.186574 7f86a68f6700 5 -- op tracker -- , seq: 3673, time: 2014-06-09 16:22:12.181593, event: all_read, op: osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.186584 7f86a68f6700 5 -- op tracker -- , seq: 3673, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.186591 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.186597 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.186615 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976da00 prio 63 cost 1548288 latency 0.008828 osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.187240 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31434000] 2014-06-09 16:22:12.187245 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31434000] 2014-06-09 16:22:12.187249 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.187251 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.187255 7f86a5e11700 10 _finish_op 0xc88bce0 seq 4157 osr(0.9d 0x2bd6490)/0x2bd6490 2014-06-09 16:22:12.187267 7f8691765700 10 op_submit_start 4206 2014-06-09 16:22:12.187269 7f8691765700 5 queue_transactions (trailing journal) 4206 2014-06-09 16:22:12.187271 7f8691765700 5 queue_op 0x1e08a2a0 seq 4206 osr(1.b3 0x384c6d0) 858 bytes (queue has 50 ops and 4497356 bytes) 2014-06-09 16:22:12.187276 7f8689755700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187282 7f868d75d700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.187286 7f8689f56700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.187291 7f868ff62700 2 waiting 51 > 50 ops || 4516138 > 104857600 2014-06-09 16:22:12.187295 7f868af58700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.187300 7f868f761700 2 waiting 51 > 50 ops || 4498182 > 104857600 2014-06-09 16:22:12.187305 7f868df5e700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187309 7f868bf5a700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187314 7f868c75b700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.187319 7f8692f68700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187323 7f868b759700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187328 7f868e75f700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187333 7f868cf5c700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.187337 7f868a757700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187342 7f868ef60700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187346 7f8690763700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.187351 7f8690f64700 2 waiting 51 > 50 ops || 4498184 > 104857600 2014-06-09 16:22:12.187355 7f8692767700 2 waiting 51 > 50 ops || 4498214 > 104857600 2014-06-09 16:22:12.187360 7f8691f66700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.187371 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5f120] 2014-06-09 16:22:12.187375 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5f120] 2014-06-09 16:22:12.187377 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.187378 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.187381 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114120 (1 active) 2014-06-09 16:22:12.187384 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115440 (2 active) 2014-06-09 16:22:12.187389 7f8691765700 10 op_submit_finish 4206 2014-06-09 16:22:12.187394 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.187409 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.040696 7 0.000316 2014-06-09 16:22:12.187421 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.187432 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.187443 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.187453 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.187462 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.187470 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.187481 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.187493 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.187503 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.187517 7f8691765700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbf7a840 172.16.96.201:6837/51639 2014-06-09 16:22:12.187540 7f8691765700 10 osd.26 243 note_peer_epoch osd.5 has 244 2014-06-09 16:22:12.187552 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 2014-06-09 16:22:12.187588 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.187593 7f8691765700 5 queue_transactions existing osr(1.b8 0x384c650)/0x384c650 2014-06-09 16:22:12.187595 7f8691765700 2 waiting 51 > 50 ops || 4498214 > 104857600 2014-06-09 16:22:12.187600 7f86a5e11700 5 _do_op 0x30a2c60 seq 4158 osr(1.32f 0x384d350)/0x384d350 start 2014-06-09 16:22:12.187604 7f86a5e11700 10 _do_transaction on 0xbf360c0 2014-06-09 16:22:12.187610 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.187667 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.187684 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4158.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.187688 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.188062 7f86a5e11700 10 _do_op 0x30a2c60 seq 4158 r = 0, finisher 0x307f00e0 0 2014-06-09 16:22:12.188075 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f03e0] 2014-06-09 16:22:12.188077 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f03e0] 2014-06-09 16:22:12.188079 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.188080 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.188084 7f86a5e11700 10 _finish_op 0x30a2c60 seq 4158 osr(1.32f 0x384d350)/0x384d350 2014-06-09 16:22:12.188095 7f868d75d700 10 op_submit_start 4207 2014-06-09 16:22:12.188097 7f868d75d700 5 queue_transactions (trailing journal) 4207 2014-06-09 16:22:12.188098 7f868d75d700 5 queue_op 0xbe7daa0 seq 4207 osr(3.78a 0x3ed1d80) 861 bytes (queue has 50 ops and 4497389 bytes) 2014-06-09 16:22:12.188103 7f8689f56700 2 waiting 51 > 50 ops || 4498250 > 104857600 2014-06-09 16:22:12.188108 7f868ff62700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188113 7f868af58700 2 waiting 51 > 50 ops || 4498250 > 104857600 2014-06-09 16:22:12.188117 7f868f761700 2 waiting 51 > 50 ops || 4498215 > 104857600 2014-06-09 16:22:12.188121 7f868df5e700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188126 7f868bf5a700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188130 7f868c75b700 2 waiting 51 > 50 ops || 4498250 > 104857600 2014-06-09 16:22:12.188134 7f8692f68700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188139 7f868b759700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188143 7f868e75f700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188148 7f868cf5c700 2 waiting 51 > 50 ops || 4498250 > 104857600 2014-06-09 16:22:12.188152 7f868a757700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188215 7f868ef60700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188221 7f8690763700 2 waiting 51 > 50 ops || 4498250 > 104857600 2014-06-09 16:22:12.188225 7f8690f64700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188230 7f8692767700 2 waiting 51 > 50 ops || 4498247 > 104857600 2014-06-09 16:22:12.188234 7f8691f66700 2 waiting 51 > 50 ops || 4498250 > 104857600 2014-06-09 16:22:12.188238 7f8691765700 2 waiting 51 > 50 ops || 4498247 > 104857600 2014-06-09 16:22:12.188243 7f8689755700 2 waiting 51 > 50 ops || 4498217 > 104857600 2014-06-09 16:22:12.188253 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f00e0] 2014-06-09 16:22:12.188257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f00e0] 2014-06-09 16:22:12.188259 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.188260 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.188263 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115440 (1 active) 2014-06-09 16:22:12.188266 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4aca20 (2 active) 2014-06-09 16:22:12.188271 7f868d75d700 10 op_submit_finish 4207 2014-06-09 16:22:12.188275 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.188292 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.188301 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.188310 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.188319 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.188321 7f86a68f6700 5 -- op tracker -- , seq: 3674, time: 2014-06-09 16:22:12.186641, event: header_read, op: osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.188328 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.188334 7f86a68f6700 5 -- op tracker -- , seq: 3674, time: 2014-06-09 16:22:12.186643, event: throttled, op: osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.188342 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:184 2014-06-09 16:22:12.188348 7f86a68f6700 5 -- op tracker -- , seq: 3674, time: 2014-06-09 16:22:12.188280, event: all_read, op: osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.188352 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.188358 7f86a68f6700 5 -- op tracker -- , seq: 3674, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.188361 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.188365 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.188370 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.188371 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.188376 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.188388 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d900 prio 63 cost 974848 latency 0.001747 osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.188390 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.188400 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x230da800,0x10c54e00,0xe032d00,0x230da700,0x11e62b00,0x10c54900,0x230da600,0x11e62a00,0x10c54800,0x230da500,0x14052200,0x10c54700,0x230da400,0x14052100,0x230da300,0x10c54600,0x13022f00 2014-06-09 16:22:12.188456 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.188466 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.188475 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.188489 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:962 2014-06-09 16:22:12.188499 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.188508 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.188518 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.188523 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.188536 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.133427 8 0.000873 2014-06-09 16:22:12.188548 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.188558 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.188569 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.188580 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.188588 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.188596 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.188607 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.188618 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.188627 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.188641 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.188658 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.188692 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.188696 7f868d75d700 5 queue_transactions existing osr(1.223 0x384cc70)/0x384cc70 2014-06-09 16:22:12.188699 7f868d75d700 2 waiting 51 > 50 ops || 4498250 > 104857600 2014-06-09 16:22:12.188729 7f86a5e11700 5 _do_op 0x324e76e0 seq 4159 osr(2.40c 0x2ed1250)/0x2ed1250 start 2014-06-09 16:22:12.188727 7f86a68f6700 5 -- op tracker -- , seq: 3675, time: 2014-06-09 16:22:12.188411, event: header_read, op: osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.188736 7f86a5e11700 10 _do_transaction on 0x2f28840 2014-06-09 16:22:12.188742 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.188741 7f86a68f6700 5 -- op tracker -- , seq: 3675, time: 2014-06-09 16:22:12.188413, event: throttled, op: osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.188750 7f86a68f6700 5 -- op tracker -- , seq: 3675, time: 2014-06-09 16:22:12.188677, event: all_read, op: osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.188759 7f86a68f6700 5 -- op tracker -- , seq: 3675, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.188764 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.188768 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.188781 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d800 prio 63 cost 249856 latency 0.000370 osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.188791 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.188807 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4159.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.188811 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.188824 7f86a5e11700 10 _do_op 0x324e76e0 seq 4159 r = 0, finisher 0xbd75440 0 2014-06-09 16:22:12.188835 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002060] 2014-06-09 16:22:12.188838 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002060] 2014-06-09 16:22:12.188840 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.188841 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.188844 7f86a5e11700 10 _finish_op 0x324e76e0 seq 4159 osr(2.40c 0x2ed1250)/0x2ed1250 2014-06-09 16:22:12.188856 7f868ff62700 10 op_submit_start 4208 2014-06-09 16:22:12.188857 7f868ff62700 5 queue_transactions (trailing journal) 4208 2014-06-09 16:22:12.188859 7f868ff62700 5 queue_op 0x30a24e0 seq 4208 osr(3.2ac 0x3ed0b50) 18782 bytes (queue has 50 ops and 4515343 bytes) 2014-06-09 16:22:12.188864 7f868af58700 2 waiting 51 > 50 ops || 4516204 > 104857600 2014-06-09 16:22:12.188869 7f868f761700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.188874 7f868df5e700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188878 7f868bf5a700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188883 7f868c75b700 2 waiting 51 > 50 ops || 4516204 > 104857600 2014-06-09 16:22:12.188887 7f8692f68700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188892 7f868b759700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188896 7f868e75f700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188901 7f868cf5c700 2 waiting 51 > 50 ops || 4516204 > 104857600 2014-06-09 16:22:12.188905 7f868a757700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188910 7f868ef60700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188914 7f8690763700 2 waiting 51 > 50 ops || 4516204 > 104857600 2014-06-09 16:22:12.188918 7f8690f64700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188923 7f8692767700 2 waiting 51 > 50 ops || 4516201 > 104857600 2014-06-09 16:22:12.188927 7f8691f66700 2 waiting 51 > 50 ops || 4516204 > 104857600 2014-06-09 16:22:12.188931 7f8691765700 2 waiting 51 > 50 ops || 4516201 > 104857600 2014-06-09 16:22:12.188936 7f8689755700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.188940 7f868d75d700 2 waiting 51 > 50 ops || 4516204 > 104857600 2014-06-09 16:22:12.188945 7f8689f56700 2 waiting 51 > 50 ops || 4516204 > 104857600 2014-06-09 16:22:12.188954 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd75440] 2014-06-09 16:22:12.188959 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd75440] 2014-06-09 16:22:12.188960 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.188961 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.188964 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4aca20 (1 active) 2014-06-09 16:22:12.188967 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300c60 (2 active) 2014-06-09 16:22:12.188973 7f868ff62700 10 op_submit_finish 4208 2014-06-09 16:22:12.188978 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc29a3c0 243'664 rep_tid=30 committed?=0 applied?=0 lock=0 op=osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.189005 7f868ff62700 10 osd.26 244 dequeue_op 0x14053a00 finish 2014-06-09 16:22:12.189010 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.189019 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.189023 7f868ff62700 10 osd.26 244 dequeue_op 0x306b800 prio 63 cost 475136 latency 2.269949 osd_op(client.4567.0:34713 rbd_data.11776b8b4567.0000000000001761 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.d6347f30 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:12.189049 7f868ff62700 5 -- op tracker -- , seq: 3, time: 2014-06-09 16:22:12.189044, event: reached_pg, op: osd_op(client.4567.0:34713 rbd_data.11776b8b4567.0000000000001761 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.d6347f30 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.189060 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.189076 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x306b800 2014-06-09 16:22:12.189088 7f868ff62700 10 osd.26 244 dequeue_op 0x306b800 finish 2014-06-09 16:22:12.189092 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.189131 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41be0 (20 active) 2014-06-09 16:22:12.189135 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.189149 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.189157 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.189166 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.189174 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.189182 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.189196 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.189205 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.189214 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.189223 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.189228 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.233( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) 2014-06-09 16:22:12.189266 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.189271 7f868ff62700 5 queue_transactions existing osr(1.233 0x384cc30)/0x384cc30 2014-06-09 16:22:12.189274 7f868ff62700 2 waiting 51 > 50 ops || 4516171 > 104857600 2014-06-09 16:22:12.189279 7f86a5e11700 5 _do_op 0x2e21c20 seq 4160 osr(0.616 0x2bd7950)/0x2bd7950 start 2014-06-09 16:22:12.189283 7f86a5e11700 10 _do_transaction on 0xbcb0b40 2014-06-09 16:22:12.189288 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.189332 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.189347 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4160.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.189351 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.189367 7f86a5e11700 10 _do_op 0x2e21c20 seq 4160 r = 0, finisher 0x17001fa0 0 2014-06-09 16:22:12.189378 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002400] 2014-06-09 16:22:12.189381 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002400] 2014-06-09 16:22:12.189382 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.189383 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.189387 7f86a5e11700 10 _finish_op 0x2e21c20 seq 4160 osr(0.616 0x2bd7950)/0x2bd7950 2014-06-09 16:22:12.189397 7f868f761700 10 op_submit_start 4209 2014-06-09 16:22:12.189399 7f868f761700 5 queue_transactions (trailing journal) 4209 2014-06-09 16:22:12.189400 7f868f761700 5 queue_op 0x324e7f80 seq 4209 osr(1.ab 0x384c6f0) 826 bytes (queue has 50 ops and 4515341 bytes) 2014-06-09 16:22:12.189405 7f868df5e700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189410 7f868bf5a700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189414 7f868c75b700 2 waiting 51 > 50 ops || 4516202 > 104857600 2014-06-09 16:22:12.189419 7f8692f68700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189423 7f868b759700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189428 7f868e75f700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189432 7f868cf5c700 2 waiting 51 > 50 ops || 4516202 > 104857600 2014-06-09 16:22:12.189436 7f868a757700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189440 7f868ef60700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189445 7f8690763700 2 waiting 51 > 50 ops || 4516202 > 104857600 2014-06-09 16:22:12.189449 7f8690f64700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189453 7f8692767700 2 waiting 51 > 50 ops || 4516199 > 104857600 2014-06-09 16:22:12.189457 7f8691f66700 2 waiting 51 > 50 ops || 4516202 > 104857600 2014-06-09 16:22:12.189461 7f8691765700 2 waiting 51 > 50 ops || 4516199 > 104857600 2014-06-09 16:22:12.189466 7f8689755700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189470 7f868d75d700 2 waiting 51 > 50 ops || 4516202 > 104857600 2014-06-09 16:22:12.189474 7f8689f56700 2 waiting 51 > 50 ops || 4516202 > 104857600 2014-06-09 16:22:12.189478 7f868ff62700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189482 7f868af58700 2 waiting 51 > 50 ops || 4516202 > 104857600 2014-06-09 16:22:12.189491 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17001fa0] 2014-06-09 16:22:12.189495 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17001fa0] 2014-06-09 16:22:12.189497 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.189498 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.189501 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300c60 (1 active) 2014-06-09 16:22:12.189504 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e000 (2 active) 2014-06-09 16:22:12.189509 7f868f761700 10 op_submit_finish 4209 2014-06-09 16:22:12.189513 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.189551 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.189556 7f868f761700 5 queue_transactions existing osr(0.b9 0x2bd6400)/0x2bd6400 2014-06-09 16:22:12.189559 7f868f761700 2 waiting 51 > 50 ops || 4516167 > 104857600 2014-06-09 16:22:12.189563 7f86a5e11700 5 _do_op 0x230b9c80 seq 4161 osr(0.718 0x384c090)/0x384c090 start 2014-06-09 16:22:12.189567 7f86a5e11700 10 _do_transaction on 0x2fb6fc0 2014-06-09 16:22:12.189573 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.189627 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.189644 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4161.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.189649 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.189662 7f86a5e11700 10 _do_op 0x230b9c80 seq 4161 r = 0, finisher 0x324f8ee0 0 2014-06-09 16:22:12.189673 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050bc40] 2014-06-09 16:22:12.189676 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.189688 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.189698 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 2014-06-09 16:22:12.189707 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050bc40] 2014-06-09 16:22:12.189709 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.189711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.189714 7f86a5e11700 10 _finish_op 0x230b9c80 seq 4161 osr(0.718 0x384c090)/0x384c090 2014-06-09 16:22:12.189725 7f868bf5a700 10 op_submit_start 4210 2014-06-09 16:22:12.189727 7f868bf5a700 5 queue_transactions (trailing journal) 4210 2014-06-09 16:22:12.189729 7f868bf5a700 5 queue_op 0x230b9560 seq 4210 osr(3.527 0x3ed1660) 828 bytes (queue has 50 ops and 4515308 bytes) 2014-06-09 16:22:12.189734 7f868c75b700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189739 7f8692f68700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189743 7f868b759700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189748 7f868e75f700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189752 7f868cf5c700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189756 7f868a757700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189761 7f868ef60700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189765 7f8690763700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189769 7f8690f64700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189774 7f8692767700 2 waiting 51 > 50 ops || 4516166 > 104857600 2014-06-09 16:22:12.189778 7f8691f66700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189783 7f8691765700 2 waiting 51 > 50 ops || 4516166 > 104857600 2014-06-09 16:22:12.189787 7f8689755700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189791 7f868d75d700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189795 7f8689f56700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189800 7f868ff62700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189804 7f868af58700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.189808 7f868f761700 2 waiting 51 > 50 ops || 4516134 > 104857600 2014-06-09 16:22:12.189813 7f868df5e700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189823 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f8ee0] 2014-06-09 16:22:12.189830 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f8ee0] 2014-06-09 16:22:12.189832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.189833 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.189837 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e000 (1 active) 2014-06-09 16:22:12.189840 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb440 (2 active) 2014-06-09 16:22:12.189845 7f868bf5a700 10 op_submit_finish 4210 2014-06-09 16:22:12.189849 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.189863 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.189872 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.189881 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.189890 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.189898 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.189912 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:172 2014-06-09 16:22:12.189921 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.189929 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.189939 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.189943 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 243/243 242/242/223) 2014-06-09 16:22:12.189977 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.189982 7f868bf5a700 5 queue_transactions existing osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 2014-06-09 16:22:12.189984 7f868bf5a700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.189989 7f86a5e11700 5 _do_op 0x3280ac00 seq 4162 osr(3.35c 0x3ed10c0)/0x3ed10c0 start 2014-06-09 16:22:12.189993 7f86a5e11700 10 _do_transaction on 0x35629c0 2014-06-09 16:22:12.189998 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.190040 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.190055 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4162.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.190059 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.190142 7f86a5e11700 10 _do_op 0x3280ac00 seq 4162 r = 0, finisher 0x281b6600 0 2014-06-09 16:22:12.190153 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251c7a0] 2014-06-09 16:22:12.190156 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251c7a0] 2014-06-09 16:22:12.190157 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.190159 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.190162 7f86a5e11700 10 _finish_op 0x3280ac00 seq 4162 osr(3.35c 0x3ed10c0)/0x3ed10c0 2014-06-09 16:22:12.190173 7f8692f68700 10 op_submit_start 4211 2014-06-09 16:22:12.190175 7f8692f68700 5 queue_transactions (trailing journal) 4211 2014-06-09 16:22:12.190176 7f8692f68700 5 queue_op 0xd327aa0 seq 4211 osr(2.24d 0x2ed0a50) 828 bytes (queue has 50 ops and 4515308 bytes) 2014-06-09 16:22:12.190181 7f868b759700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190184 7f868e75f700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190188 7f868cf5c700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.190191 7f868a757700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190194 7f868ef60700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190197 7f8690763700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.190200 7f8690f64700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190203 7f8692767700 2 waiting 51 > 50 ops || 4516166 > 104857600 2014-06-09 16:22:12.190207 7f8691f66700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.190210 7f8691765700 2 waiting 51 > 50 ops || 4516166 > 104857600 2014-06-09 16:22:12.190213 7f8689755700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190216 7f868d75d700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.190219 7f8689f56700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.190222 7f868ff62700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190226 7f868af58700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.190229 7f868f761700 2 waiting 51 > 50 ops || 4516134 > 104857600 2014-06-09 16:22:12.190232 7f868df5e700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190235 7f868bf5a700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190238 7f868c75b700 2 waiting 51 > 50 ops || 4516169 > 104857600 2014-06-09 16:22:12.190246 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x281b6600] 2014-06-09 16:22:12.190252 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x281b6600] 2014-06-09 16:22:12.190253 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.190254 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.190257 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb440 (1 active) 2014-06-09 16:22:12.190259 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad200 (2 active) 2014-06-09 16:22:12.190263 7f8692f68700 10 op_submit_finish 4211 2014-06-09 16:22:12.190266 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.508( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.190297 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.190302 7f8692f68700 5 queue_transactions existing osr(1.508 0x384db90)/0x384db90 2014-06-09 16:22:12.190304 7f8692f68700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190307 7f86a5e11700 5 _do_op 0xc64a360 seq 4163 osr(1.2e0 0x384d130)/0x384d130 start 2014-06-09 16:22:12.190310 7f86a5e11700 10 _do_transaction on 0xbfcc900 2014-06-09 16:22:12.190316 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.190352 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.190367 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4163.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.190371 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.190385 7f86a5e11700 10 _do_op 0xc64a360 seq 4163 r = 0, finisher 0x189aacc0 0 2014-06-09 16:22:12.190396 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3196c260] 2014-06-09 16:22:12.190399 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.190409 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.190417 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:164 2014-06-09 16:22:12.190430 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3196c260] 2014-06-09 16:22:12.190432 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.190433 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.190437 7f86a5e11700 10 _finish_op 0xc64a360 seq 4163 osr(1.2e0 0x384d130)/0x384d130 2014-06-09 16:22:12.190448 7f868e75f700 10 op_submit_start 4212 2014-06-09 16:22:12.190450 7f868e75f700 5 queue_transactions (trailing journal) 4212 2014-06-09 16:22:12.190451 7f868e75f700 5 queue_op 0x324f49c0 seq 4212 osr(0.473 0x2bd74d0) 828 bytes (queue has 50 ops and 4515275 bytes) 2014-06-09 16:22:12.190456 7f868cf5c700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190461 7f868a757700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190465 7f868ef60700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190470 7f8690763700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190474 7f8690f64700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190479 7f8692767700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.190483 7f8691f66700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190488 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.190492 7f8689755700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190496 7f868d75d700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190501 7f8689f56700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190505 7f868ff62700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190510 7f868af58700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190514 7f868f761700 2 waiting 51 > 50 ops || 4516101 > 104857600 2014-06-09 16:22:12.190518 7f868df5e700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190522 7f868bf5a700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190526 7f868c75b700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190530 7f8692f68700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190534 7f868b759700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190544 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aacc0] 2014-06-09 16:22:12.190549 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aacc0] 2014-06-09 16:22:12.190553 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.190555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.190558 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad200 (1 active) 2014-06-09 16:22:12.190561 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a6c0 (2 active) 2014-06-09 16:22:12.190566 7f868e75f700 10 op_submit_finish 4212 2014-06-09 16:22:12.190570 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.5f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.190604 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.190610 7f868e75f700 5 queue_transactions existing osr(2.5f2 0x2ed1df0)/0x2ed1df0 2014-06-09 16:22:12.190613 7f868e75f700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190617 7f86a5e11700 5 _do_op 0xd471020 seq 4164 osr(3.1d5 0x3ed09f0)/0x3ed09f0 start 2014-06-09 16:22:12.190621 7f86a5e11700 10 _do_transaction on 0xc14b980 2014-06-09 16:22:12.190626 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.190668 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.190682 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4164.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.190686 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.190701 7f86a5e11700 10 _do_op 0xd471020 seq 4164 r = 0, finisher 0x307f0a40 0 2014-06-09 16:22:12.190712 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f0ac0] 2014-06-09 16:22:12.190715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f0ac0] 2014-06-09 16:22:12.190716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.190717 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.190721 7f86a5e11700 10 _finish_op 0xd471020 seq 4164 osr(3.1d5 0x3ed09f0)/0x3ed09f0 2014-06-09 16:22:12.190732 7f868a757700 10 op_submit_start 4213 2014-06-09 16:22:12.190733 7f868a757700 5 queue_transactions (trailing journal) 4213 2014-06-09 16:22:12.190735 7f868a757700 5 queue_op 0xcabde00 seq 4213 osr(0.4ba 0x2bd7710) 828 bytes (queue has 50 ops and 4515275 bytes) 2014-06-09 16:22:12.190739 7f868ef60700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190744 7f8690763700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190748 7f8690f64700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190752 7f8692767700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.190757 7f8691f66700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190761 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.190765 7f8689755700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190770 7f868d75d700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190774 7f8689f56700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190778 7f868ff62700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190782 7f868af58700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190787 7f868f761700 2 waiting 51 > 50 ops || 4516101 > 104857600 2014-06-09 16:22:12.190791 7f868df5e700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190795 7f868bf5a700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190799 7f868c75b700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190804 7f8692f68700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190808 7f868b759700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190812 7f868e75f700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190816 7f868cf5c700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.190826 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f0a40] 2014-06-09 16:22:12.190830 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f0a40] 2014-06-09 16:22:12.190831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.190832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.190835 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a6c0 (1 active) 2014-06-09 16:22:12.190838 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286480 (2 active) 2014-06-09 16:22:12.190843 7f868a757700 10 op_submit_finish 4213 2014-06-09 16:22:12.190866 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e340 (19 active) 2014-06-09 16:22:12.190873 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.190877 7f868a757700 10 osd.26 244 dequeue_op 0xe2ad300 prio 63 cost 385024 latency 2.160916 osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] 2014-06-09 16:22:12.190907 7f868a757700 5 -- op tracker -- , seq: 55, time: 2014-06-09 16:22:12.190901, event: reached_pg, op: osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.190922 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.190939 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] handle_message: 0xe2ad300 2014-06-09 16:22:12.190952 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] do_op osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.190975 7f868a757700 3 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] do_op dup client.4568.0:34735 was 213'599 2014-06-09 16:22:12.191008 7f868a757700 10 osd.26 244 dequeue_op 0xe2ad300 finish 2014-06-09 16:22:12.191885 7f868a757700 5 -- op tracker -- , seq: 55, time: 2014-06-09 16:22:12.191011, event: done, op: osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.191910 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.191940 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e340 (20 active) 2014-06-09 16:22:12.191944 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.2c0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.191981 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.191986 7f868a757700 5 queue_transactions existing osr(1.2c0 0x384d170)/0x384d170 2014-06-09 16:22:12.192047 7f868a757700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.190911 7f86a5e11700 5 _do_op 0x24105ce0 seq 4165 osr(2.25c 0x2ed0a30)/0x2ed0a30 start 2014-06-09 16:22:12.192054 7f86a5e11700 10 _do_transaction on 0x394e240 2014-06-09 16:22:12.192062 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.192109 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.192128 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4165.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.192132 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.192149 7f86a5e11700 10 _do_op 0x24105ce0 seq 4165 r = 0, finisher 0xc6e8ca0 0 2014-06-09 16:22:12.192161 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078800] 2014-06-09 16:22:12.192163 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.192174 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.192184 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 2014-06-09 16:22:12.192193 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078800] 2014-06-09 16:22:12.192195 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.192196 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.192200 7f86a5e11700 10 _finish_op 0x24105ce0 seq 4165 osr(2.25c 0x2ed0a30)/0x2ed0a30 2014-06-09 16:22:12.192212 7f8690763700 10 op_submit_start 4214 2014-06-09 16:22:12.192214 7f8690763700 5 queue_transactions (trailing journal) 4214 2014-06-09 16:22:12.192215 7f8690763700 5 queue_op 0xc2d4d80 seq 4214 osr(2.754 0x3ed00f0) 861 bytes (queue has 50 ops and 4515275 bytes) 2014-06-09 16:22:12.192221 7f8690f64700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192225 7f8692767700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192230 7f8691f66700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.192235 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192239 7f8689755700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192243 7f868d75d700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.192248 7f8689f56700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.192252 7f868ff62700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192257 7f868af58700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.192261 7f868f761700 2 waiting 51 > 50 ops || 4516101 > 104857600 2014-06-09 16:22:12.192266 7f868df5e700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192270 7f868bf5a700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192274 7f868c75b700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.192279 7f8692f68700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192283 7f868b759700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192287 7f868e75f700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192292 7f868cf5c700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.192296 7f868a757700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192300 7f868ef60700 2 waiting 51 > 50 ops || 4516103 > 104857600 2014-06-09 16:22:12.192310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8ca0] 2014-06-09 16:22:12.192319 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8ca0] 2014-06-09 16:22:12.192321 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.192322 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.192325 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286480 (1 active) 2014-06-09 16:22:12.192328 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5245a0 (2 active) 2014-06-09 16:22:12.192333 7f8690763700 10 op_submit_finish 4214 2014-06-09 16:22:12.192337 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.192355 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.192368 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.192388 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41810 (19 active) 2014-06-09 16:22:12.192396 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.192400 7f8690763700 10 osd.26 244 dequeue_op 0xe032700 prio 63 cost 1015808 latency 2.219964 osd_op(client.4569.0:38360 rbd_data.11796b8b4567.0000000000001a6e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1015808] 3.9f12c15c RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:12.192424 7f8690763700 5 -- op tracker -- , seq: 28, time: 2014-06-09 16:22:12.192419, event: reached_pg, op: osd_op(client.4569.0:38360 rbd_data.11796b8b4567.0000000000001a6e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1015808] 3.9f12c15c RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.192434 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.192447 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xe032700 2014-06-09 16:22:12.192457 7f8690763700 10 osd.26 244 dequeue_op 0xe032700 finish 2014-06-09 16:22:12.192461 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.192483 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41810 (20 active) 2014-06-09 16:22:12.192487 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.192501 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.076281 7 0.000431 2014-06-09 16:22:12.192512 7f86a68f6700 5 -- op tracker -- , seq: 3676, time: 2014-06-09 16:22:12.188802, event: header_read, op: osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.192514 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.192524 7f86a68f6700 5 -- op tracker -- , seq: 3676, time: 2014-06-09 16:22:12.188803, event: throttled, op: osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.192525 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.192534 7f86a68f6700 5 -- op tracker -- , seq: 3676, time: 2014-06-09 16:22:12.192396, event: all_read, op: osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.192537 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.192547 7f86a68f6700 5 -- op tracker -- , seq: 3676, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.192548 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.192553 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.192558 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.192558 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.192566 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.192578 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976ce00 prio 63 cost 1548288 latency 0.003776 osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.192577 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.192588 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.192597 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.192607 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.192617 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.192633 7f8690763700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xc13cdc0 172.16.96.201:6811/57862 2014-06-09 16:22:12.192659 7f8690763700 10 osd.26 243 note_peer_epoch osd.1 has 244 2014-06-09 16:22:12.192671 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.192707 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.192711 7f8690763700 5 queue_transactions existing osr(0.686 0x2bd7e20)/0x2bd7e20 2014-06-09 16:22:12.192713 7f8690763700 2 waiting 51 > 50 ops || 4516136 > 104857600 2014-06-09 16:22:12.192719 7f86a5e11700 5 _do_op 0x308ca000 seq 4166 osr(2.3fb 0x2ed12b0)/0x2ed12b0 start 2014-06-09 16:22:12.192723 7f86a5e11700 10 _do_transaction on 0xbe880c0 2014-06-09 16:22:12.192739 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.192786 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.192803 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4166.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.192808 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.192830 7f86a5e11700 10 _do_op 0x308ca000 seq 4166 r = 0, finisher 0x1322d9c0 0 2014-06-09 16:22:12.192842 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5a260] 2014-06-09 16:22:12.192845 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.192856 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.192865 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:12.192877 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5a260] 2014-06-09 16:22:12.192879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.192880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.192883 7f86a5e11700 10 _finish_op 0x308ca000 seq 4166 osr(2.3fb 0x2ed12b0)/0x2ed12b0 2014-06-09 16:22:12.192895 7f8692767700 10 op_submit_start 4215 2014-06-09 16:22:12.192896 7f8692767700 5 queue_transactions (trailing journal) 4215 2014-06-09 16:22:12.192898 7f8692767700 5 queue_op 0x2fcafc20 seq 4215 osr(0.6e 0x2bd6390) 858 bytes (queue has 50 ops and 4515272 bytes) 2014-06-09 16:22:12.192903 7f8691f66700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192908 7f8691765700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.192912 7f8689755700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192916 7f868d75d700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192921 7f8689f56700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192925 7f868ff62700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192929 7f868af58700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192934 7f868f761700 2 waiting 51 > 50 ops || 4516098 > 104857600 2014-06-09 16:22:12.192938 7f868df5e700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192942 7f868bf5a700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192947 7f868c75b700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192952 7f8692f68700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192956 7f868b759700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192960 7f868e75f700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192964 7f868cf5c700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192968 7f868a757700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192973 7f868ef60700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192977 7f8690763700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.192981 7f8690f64700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.192991 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322d9c0] 2014-06-09 16:22:12.192996 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322d9c0] 2014-06-09 16:22:12.192997 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.192998 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.193001 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5245a0 (1 active) 2014-06-09 16:22:12.193004 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286000 (2 active) 2014-06-09 16:22:12.193009 7f8692767700 10 op_submit_finish 4215 2014-06-09 16:22:12.193032 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x153591c0 (19 active) 2014-06-09 16:22:12.193039 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.193043 7f8692767700 10 osd.26 244 dequeue_op 0x230da800 prio 63 cost 344064 latency 1.502615 osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.193067 7f8692767700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.193062, event: reached_pg, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.193075 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.193090 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x230da800 2014-06-09 16:22:12.193100 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.193124 7f8692767700 15 getattr 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 '_' 2014-06-09 16:22:12.193483 7f86a68f6700 5 -- op tracker -- , seq: 3677, time: 2014-06-09 16:22:12.192601, event: header_read, op: osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.193495 7f86a68f6700 5 -- op tracker -- , seq: 3677, time: 2014-06-09 16:22:12.192602, event: throttled, op: osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.193504 7f86a68f6700 5 -- op tracker -- , seq: 3677, time: 2014-06-09 16:22:12.193158, event: all_read, op: osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.193514 7f86a68f6700 5 -- op tracker -- , seq: 3677, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.193519 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.193524 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.193537 7f86a68f6700 15 osd.26 244 enqueue_op 0x32826500 prio 63 cost 516096 latency 0.000936 osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.193774 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.193777 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 but can_create 2014-06-09 16:22:12.193793 7f8692767700 15 getattr 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 'snapset' 2014-06-09 16:22:12.193829 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.193831 7f8692767700 15 getattr 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/snapdir//3 'snapset' 2014-06-09 16:22:12.193862 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.193872 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x3042c2c0 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.193885 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.193896 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.193973 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x3042c2c0 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 rwstate(none n=0 w=0) oi: 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd5c00 snapset: 0=[]:[] 2014-06-09 16:22:12.193990 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 @head oi=888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.194013 7f8692767700 15 getattr 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/snapdir//3 '_' 2014-06-09 16:22:12.194048 7f8692767700 10 getattr got.size() is 0 2014-06-09 16:22:12.194051 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/snapdir//3 and !can_create 2014-06-09 16:22:12.194072 7f8692767700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.194065, event: started, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.194079 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x308a9600 2014-06-09 16:22:12.194097 7f8692767700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.194092, event: started, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.194104 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] ov 0'0 av 243'576 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.194119 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 2014-06-09 16:22:12.194130 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.194146 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~344064 2014-06-09 16:22:12.194165 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228a00 2014-06-09 16:22:12.194177 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 snapset=0x30dd5c38 snapc=0=[] 2014-06-09 16:22:12.194188 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.194199 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.194210 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 0x308a9600 op modify 2014-06-09 16:22:12.194224 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:32.257474 2014-06-09 16:22:12.194245 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.194266 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.194277 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 31 on osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.194293 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 31 o 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.194318 7f8692767700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.194312, event: waiting for subops from 8, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.194339 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.194360 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'575], crt=0'0) [243'576 (0'0) modify 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 by client.4570.0:34004 2014-06-09 10:19:32.257474] 2014-06-09 16:22:12.194384 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'576 (0'0) modify 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 by client.4570.0:34004 2014-06-09 10:19:32.257474 2014-06-09 16:22:12.194404 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.194436 7f8692767700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'576, trimmed: 2014-06-09 16:22:12.194458 7f8692767700 5 queue_transactions existing osr(3.56a 0x3ed1900)/0x3ed1900 2014-06-09 16:22:12.194461 7f8692767700 2 waiting 51 > 50 ops || 4861734 > 104857600 2014-06-09 16:22:12.194708 7f86a5e11700 5 _do_op 0xc705380 seq 4167 osr(3.647 0x3ed1a70)/0x3ed1a70 start 2014-06-09 16:22:12.194714 7f86a5e11700 10 _do_transaction on 0xc0155c0 2014-06-09 16:22:12.194724 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.194778 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.194798 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4167.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.194802 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.194846 7f86a5e11700 10 _do_op 0xc705380 seq 4167 r = 0, finisher 0xbd6f340 0 2014-06-09 16:22:12.194860 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7c100] 2014-06-09 16:22:12.194863 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] _finish_recovery 2014-06-09 16:22:12.194876 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] purge_strays 2014-06-09 16:22:12.194888 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] publish_stats_to_osd 243:979 2014-06-09 16:22:12.194898 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7c100] 2014-06-09 16:22:12.194900 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.194901 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.194905 7f86a5e11700 10 _finish_op 0xc705380 seq 4167 osr(3.647 0x3ed1a70)/0x3ed1a70 2014-06-09 16:22:12.194919 7f8691765700 10 op_submit_start 4216 2014-06-09 16:22:12.194921 7f8691765700 5 queue_transactions (trailing journal) 4216 2014-06-09 16:22:12.194922 7f8691765700 5 queue_op 0xd4715c0 seq 4216 osr(1.b8 0x384c650) 858 bytes (queue has 50 ops and 4515269 bytes) 2014-06-09 16:22:12.194928 7f8689755700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.194934 7f868d75d700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.194939 7f8689f56700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.194945 7f868ff62700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.194950 7f868af58700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.194955 7f868f761700 2 waiting 51 > 50 ops || 4516095 > 104857600 2014-06-09 16:22:12.194960 7f868df5e700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.194965 7f868bf5a700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.194969 7f868c75b700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.194974 7f8692f68700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.194979 7f868b759700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.194984 7f868e75f700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.194988 7f868cf5c700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.194993 7f868a757700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.194998 7f868ef60700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.195003 7f8690763700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195008 7f8690f64700 2 waiting 51 > 50 ops || 4516097 > 104857600 2014-06-09 16:22:12.195012 7f8692767700 2 waiting 51 > 50 ops || 4861731 > 104857600 2014-06-09 16:22:12.195016 7f8691f66700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195027 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6f340] 2014-06-09 16:22:12.195032 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6f340] 2014-06-09 16:22:12.195034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.195035 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.195038 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286000 (1 active) 2014-06-09 16:22:12.195042 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb7560 (2 active) 2014-06-09 16:22:12.195047 7f8691765700 10 op_submit_finish 4216 2014-06-09 16:22:12.195052 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.195067 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.126555 7 0.000304 2014-06-09 16:22:12.195080 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.195091 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.195103 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.195114 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.195123 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.195130 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.195142 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.195153 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.195163 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.195178 7f8691765700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 2014-06-09 16:22:12.195190 7f86a68f6700 5 -- op tracker -- , seq: 3678, time: 2014-06-09 16:22:12.193557, event: header_read, op: osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.195195 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.195201 7f86a68f6700 5 -- op tracker -- , seq: 3678, time: 2014-06-09 16:22:12.193558, event: throttled, op: osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.195222 7f86a68f6700 5 -- op tracker -- , seq: 3678, time: 2014-06-09 16:22:12.195024, event: all_read, op: osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.195228 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.195232 7f86a68f6700 5 -- op tracker -- , seq: 3678, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.195233 7f8691765700 5 queue_transactions existing osr(1.37d 0x384d210)/0x384d210 2014-06-09 16:22:12.195236 7f8691765700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195237 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.195241 7f86a5e11700 5 _do_op 0x324b27e0 seq 4168 osr(0.1d4 0x2bd6940)/0x2bd6940 start 2014-06-09 16:22:12.195242 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.195245 7f86a5e11700 10 _do_transaction on 0x3145380 2014-06-09 16:22:12.195250 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.195258 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976dd00 prio 63 cost 1032192 latency 0.001701 osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.195294 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.195310 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4168.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.195314 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.195324 7f86a5e11700 10 _do_op 0x324b27e0 seq 4168 r = 0, finisher 0x2e0e1e0 0 2014-06-09 16:22:12.195336 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509180] 2014-06-09 16:22:12.195339 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509180] 2014-06-09 16:22:12.195340 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.195341 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.195345 7f86a5e11700 10 _finish_op 0x324b27e0 seq 4168 osr(0.1d4 0x2bd6940)/0x2bd6940 2014-06-09 16:22:12.195356 7f868d75d700 10 op_submit_start 4217 2014-06-09 16:22:12.195357 7f868d75d700 5 queue_transactions (trailing journal) 4217 2014-06-09 16:22:12.195359 7f868d75d700 5 queue_op 0xbe7de00 seq 4217 osr(1.223 0x384cc70) 861 bytes (queue has 50 ops and 4515302 bytes) 2014-06-09 16:22:12.195364 7f8689f56700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.195369 7f868ff62700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195373 7f868af58700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.195378 7f868f761700 2 waiting 51 > 50 ops || 4516128 > 104857600 2014-06-09 16:22:12.195383 7f868df5e700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195387 7f868bf5a700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195391 7f868c75b700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.195395 7f8692f68700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195400 7f868b759700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195404 7f868e75f700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195408 7f868cf5c700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.195412 7f868a757700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195416 7f868ef60700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195420 7f8690763700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.195425 7f8690f64700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195429 7f8692767700 2 waiting 51 > 50 ops || 4861764 > 104857600 2014-06-09 16:22:12.195433 7f8691f66700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.195437 7f8691765700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.195442 7f8689755700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195451 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0e1e0] 2014-06-09 16:22:12.195456 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0e1e0] 2014-06-09 16:22:12.195457 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.195458 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.195462 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb7560 (1 active) 2014-06-09 16:22:12.195465 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62fc20 (2 active) 2014-06-09 16:22:12.195492 7f868d75d700 10 op_submit_finish 4217 2014-06-09 16:22:12.195497 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.195511 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.150645 8 0.000898 2014-06-09 16:22:12.195523 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.195534 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.195544 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.195555 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.195563 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.195571 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.195582 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.195594 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.195603 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.195625 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.195635 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.195649 7f868d75d700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 244 2014-06-09 16:22:12.195666 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:175 2014-06-09 16:22:12.195701 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.195705 7f868d75d700 5 queue_transactions existing osr(1.764 0x2ed03f0)/0x2ed03f0 2014-06-09 16:22:12.195708 7f868d75d700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.195734 7f86a5e11700 5 _do_op 0x24104ea0 seq 4169 osr(1.c9 0x384c9f0)/0x384c9f0 start 2014-06-09 16:22:12.195742 7f86a5e11700 10 _do_transaction on 0xbf89e00 2014-06-09 16:22:12.195748 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.195793 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.195809 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4169.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.195813 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.195891 7f86a5e11700 10 _do_op 0x24104ea0 seq 4169 r = 0, finisher 0x33a7ac0 0 2014-06-09 16:22:12.195903 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc410b40] 2014-06-09 16:22:12.195906 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc410b40] 2014-06-09 16:22:12.195907 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.195909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.195912 7f86a5e11700 10 _finish_op 0x24104ea0 seq 4169 osr(1.c9 0x384c9f0)/0x384c9f0 2014-06-09 16:22:12.195923 7f868ff62700 10 op_submit_start 4218 2014-06-09 16:22:12.195925 7f868ff62700 5 queue_transactions (trailing journal) 4218 2014-06-09 16:22:12.195926 7f868ff62700 5 queue_op 0x16fd4780 seq 4218 osr(1.233 0x384cc30) 828 bytes (queue has 50 ops and 4515304 bytes) 2014-06-09 16:22:12.195932 7f868af58700 2 waiting 51 > 50 ops || 4516165 > 104857600 2014-06-09 16:22:12.195936 7f868f761700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.195941 7f868df5e700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.195945 7f868bf5a700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.195950 7f868c75b700 2 waiting 51 > 50 ops || 4516165 > 104857600 2014-06-09 16:22:12.195954 7f8692f68700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.195958 7f868b759700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.195963 7f868e75f700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.195967 7f868cf5c700 2 waiting 51 > 50 ops || 4516165 > 104857600 2014-06-09 16:22:12.195971 7f868a757700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.195976 7f868ef60700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.195981 7f8690763700 2 waiting 51 > 50 ops || 4516165 > 104857600 2014-06-09 16:22:12.195985 7f8690f64700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.195990 7f8692767700 2 waiting 51 > 50 ops || 4861766 > 104857600 2014-06-09 16:22:12.195994 7f8691f66700 2 waiting 51 > 50 ops || 4516165 > 104857600 2014-06-09 16:22:12.195998 7f8691765700 2 waiting 51 > 50 ops || 4516165 > 104857600 2014-06-09 16:22:12.196003 7f8689755700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.196008 7f868d75d700 2 waiting 51 > 50 ops || 4516165 > 104857600 2014-06-09 16:22:12.196012 7f8689f56700 2 waiting 51 > 50 ops || 4516165 > 104857600 2014-06-09 16:22:12.196022 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a7ac0] 2014-06-09 16:22:12.196027 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a7ac0] 2014-06-09 16:22:12.196029 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.196030 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.196033 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62fc20 (1 active) 2014-06-09 16:22:12.196036 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6900 (2 active) 2014-06-09 16:22:12.196041 7f868ff62700 10 op_submit_finish 4218 2014-06-09 16:22:12.196045 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.196063 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.196073 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x2e075f00,0x2e075e00,0x2e075d00,0x2e075c00 2014-06-09 16:22:12.196095 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.196106 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.196115 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.196131 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1081 2014-06-09 16:22:12.196142 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.196152 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.196163 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.196169 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.196184 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.196193 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x270be300,0x270bf500,0x1b0b7b00,0xe2ac000,0x270be200,0xe032a00,0x270be100,0xe032900,0x270be000,0xe2ad000,0x29f5bf00,0x29f5be00,0x4679f00,0x22009b00,0x22009e00,0x257c4400,0x1804c700 2014-06-09 16:22:12.196249 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.196259 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.196268 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.196282 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:882 2014-06-09 16:22:12.196292 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.196301 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.196312 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.196316 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.196328 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.196337 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.196345 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.196353 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.196362 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.196374 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 2014-06-09 16:22:12.196384 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.196392 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.196400 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.196405 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.4bb( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) 2014-06-09 16:22:12.196442 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.196447 7f868ff62700 5 queue_transactions existing osr(1.4bb 0x384d970)/0x384d970 2014-06-09 16:22:12.196449 7f868ff62700 2 waiting 51 > 50 ops || 4516132 > 104857600 2014-06-09 16:22:12.196454 7f86a5e11700 5 _do_op 0xc931d40 seq 4170 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 start 2014-06-09 16:22:12.196458 7f86a5e11700 10 _do_transaction on 0xc2bdb00 2014-06-09 16:22:12.196463 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.196507 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.196522 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4170.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.196527 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.196540 7f86a5e11700 10 _do_op 0xc931d40 seq 4170 r = 0, finisher 0x33a63a0 0 2014-06-09 16:22:12.196551 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5d840] 2014-06-09 16:22:12.196553 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5d840] 2014-06-09 16:22:12.196555 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.196556 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.196560 7f86a5e11700 10 _finish_op 0xc931d40 seq 4170 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 2014-06-09 16:22:12.196571 7f868f761700 10 op_submit_start 4219 2014-06-09 16:22:12.196573 7f868f761700 5 queue_transactions (trailing journal) 4219 2014-06-09 16:22:12.196574 7f868f761700 5 queue_op 0x324e7d40 seq 4219 osr(0.b9 0x2bd6400) 826 bytes (queue has 50 ops and 4515302 bytes) 2014-06-09 16:22:12.196580 7f868df5e700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196585 7f868bf5a700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196589 7f868c75b700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196594 7f8692f68700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196598 7f868b759700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196603 7f868e75f700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196607 7f868cf5c700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196612 7f868a757700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196616 7f868ef60700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196621 7f8690763700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196625 7f8690f64700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196629 7f8692767700 2 waiting 51 > 50 ops || 4861764 > 104857600 2014-06-09 16:22:12.196628 7f86a68f6700 5 -- op tracker -- , seq: 3679, time: 2014-06-09 16:22:12.195280, event: header_read, op: osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.196634 7f8691f66700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196639 7f8691765700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196643 7f8689755700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196647 7f868d75d700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196651 7f8689f56700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196655 7f868ff62700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196660 7f868af58700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196670 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a63a0] 2014-06-09 16:22:12.196677 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a63a0] 2014-06-09 16:22:12.196679 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.196680 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.196683 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6900 (1 active) 2014-06-09 16:22:12.196686 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301560 (2 active) 2014-06-09 16:22:12.196691 7f868f761700 10 op_submit_finish 4219 2014-06-09 16:22:12.196697 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 243/243 242/242/205) 2014-06-09 16:22:12.196734 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.196739 7f868f761700 5 queue_transactions existing osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 2014-06-09 16:22:12.196741 7f868f761700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196772 7f86a5e11700 5 _do_op 0x288f8ae0 seq 4171 osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 start 2014-06-09 16:22:12.196776 7f86a5e11700 10 _do_transaction on 0xc015e00 2014-06-09 16:22:12.196782 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.196824 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.196839 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4171.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.196843 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.196871 7f86a5e11700 10 _do_op 0x288f8ae0 seq 4171 r = 0, finisher 0x3205dc60 0 2014-06-09 16:22:12.196883 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205dea0] 2014-06-09 16:22:12.196885 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205dea0] 2014-06-09 16:22:12.196887 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.196888 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.196891 7f86a5e11700 10 _finish_op 0x288f8ae0 seq 4171 osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 2014-06-09 16:22:12.196903 7f868bf5a700 10 op_submit_start 4220 2014-06-09 16:22:12.196904 7f868bf5a700 5 queue_transactions (trailing journal) 4220 2014-06-09 16:22:12.196906 7f868bf5a700 5 queue_op 0x230b9e60 seq 4220 osr(3.2d3 0x3ed0eb0) 828 bytes (queue has 50 ops and 4515302 bytes) 2014-06-09 16:22:12.196910 7f868c75b700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196915 7f8692f68700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196920 7f868b759700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196924 7f868e75f700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196928 7f868cf5c700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196932 7f868a757700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196936 7f868ef60700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196941 7f8690763700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196945 7f8690f64700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196948 7f8692767700 2 waiting 51 > 50 ops || 4861764 > 104857600 2014-06-09 16:22:12.196951 7f8691f66700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196955 7f8691765700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196959 7f8689755700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196964 7f868d75d700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196968 7f8689f56700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196972 7f868ff62700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196977 7f868af58700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.196981 7f868f761700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196986 7f868df5e700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.196995 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205dc60] 2014-06-09 16:22:12.197002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205dc60] 2014-06-09 16:22:12.197003 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.197005 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.197008 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301560 (1 active) 2014-06-09 16:22:12.197011 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b9e0 (2 active) 2014-06-09 16:22:12.197016 7f868bf5a700 10 op_submit_finish 4220 2014-06-09 16:22:12.197020 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.3fe( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.197054 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.197059 7f868bf5a700 5 queue_transactions existing osr(0.3fe 0x2bd7210)/0x2bd7210 2014-06-09 16:22:12.197061 7f868bf5a700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197066 7f86a5e11700 5 _do_op 0xc704600 seq 4172 osr(3.761 0x3ed1e10)/0x3ed1e10 start 2014-06-09 16:22:12.197070 7f86a5e11700 10 _do_transaction on 0xbf91bc0 2014-06-09 16:22:12.197075 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.197116 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.197130 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4172.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.197135 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.197145 7f86a5e11700 10 _do_op 0xc704600 seq 4172 r = 0, finisher 0xbccda40 0 2014-06-09 16:22:12.197155 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3eaa0] 2014-06-09 16:22:12.197158 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3eaa0] 2014-06-09 16:22:12.197159 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.197160 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.197163 7f86a5e11700 10 _finish_op 0xc704600 seq 4172 osr(3.761 0x3ed1e10)/0x3ed1e10 2014-06-09 16:22:12.197174 7f8692f68700 10 op_submit_start 4221 2014-06-09 16:22:12.197176 7f8692f68700 5 queue_transactions (trailing journal) 4221 2014-06-09 16:22:12.197178 7f8692f68700 5 queue_op 0xd327c20 seq 4221 osr(1.508 0x384db90) 828 bytes (queue has 50 ops and 4515302 bytes) 2014-06-09 16:22:12.197183 7f868b759700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197187 7f868e75f700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197192 7f868cf5c700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.197196 7f868a757700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197201 7f868ef60700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197205 7f8690763700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.197209 7f8690f64700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197214 7f8692767700 2 waiting 51 > 50 ops || 4861764 > 104857600 2014-06-09 16:22:12.197218 7f8691f66700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.197222 7f8691765700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.197227 7f8689755700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197231 7f868d75d700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.197236 7f8689f56700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.197240 7f868ff62700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197244 7f868af58700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.197249 7f868f761700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197253 7f868df5e700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197257 7f868bf5a700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197261 7f868c75b700 2 waiting 51 > 50 ops || 4516163 > 104857600 2014-06-09 16:22:12.197270 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbccda40] 2014-06-09 16:22:12.197274 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbccda40] 2014-06-09 16:22:12.197275 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.197276 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.197278 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b9e0 (1 active) 2014-06-09 16:22:12.197281 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4070e0 (2 active) 2014-06-09 16:22:12.197285 7f8692f68700 10 op_submit_finish 4221 2014-06-09 16:22:12.197288 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.302( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.197319 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.197323 7f8692f68700 5 queue_transactions existing osr(1.302 0x384d010)/0x384d010 2014-06-09 16:22:12.197325 7f8692f68700 2 waiting 51 > 50 ops || 4516130 > 104857600 2014-06-09 16:22:12.197329 7f86a5e11700 5 _do_op 0x2fcaf2c0 seq 4173 osr(0.b7 0x2bd6440)/0x2bd6440 start 2014-06-09 16:22:12.197333 7f86a5e11700 10 _do_transaction on 0x97bab40 2014-06-09 16:22:12.197338 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.197374 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.197388 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4173.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.197392 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.197403 7f86a5e11700 10 _do_op 0x2fcaf2c0 seq 4173 r = 0, finisher 0xbd76ae0 0 2014-06-09 16:22:12.197412 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6f6e0] 2014-06-09 16:22:12.197414 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.197423 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.197431 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 2014-06-09 16:22:12.197441 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6f6e0] 2014-06-09 16:22:12.197442 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.197443 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.197447 7f86a5e11700 10 _finish_op 0x2fcaf2c0 seq 4173 osr(0.b7 0x2bd6440)/0x2bd6440 2014-06-09 16:22:12.197458 7f868e75f700 10 op_submit_start 4222 2014-06-09 16:22:12.197459 7f868e75f700 5 queue_transactions (trailing journal) 4222 2014-06-09 16:22:12.197461 7f868e75f700 5 queue_op 0x324f4a80 seq 4222 osr(2.5f2 0x2ed1df0) 828 bytes (queue has 50 ops and 4515272 bytes) 2014-06-09 16:22:12.197466 7f868cf5c700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197471 7f868a757700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197475 7f868ef60700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197479 7f8690763700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197484 7f8690f64700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197488 7f8692767700 2 waiting 51 > 50 ops || 4861734 > 104857600 2014-06-09 16:22:12.197492 7f8691f66700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197496 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197501 7f8689755700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197505 7f868d75d700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197509 7f8689f56700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197513 7f868ff62700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197518 7f868af58700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197522 7f868f761700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197526 7f868df5e700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197530 7f868bf5a700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197534 7f868c75b700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197538 7f8692f68700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197543 7f868b759700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197552 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76ae0] 2014-06-09 16:22:12.197556 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76ae0] 2014-06-09 16:22:12.197558 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.197559 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.197562 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4070e0 (1 active) 2014-06-09 16:22:12.197565 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b9e0 (2 active) 2014-06-09 16:22:12.197570 7f868e75f700 10 op_submit_finish 4222 2014-06-09 16:22:12.197574 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.2f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.197608 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.197613 7f868e75f700 5 queue_transactions existing osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 2014-06-09 16:22:12.197616 7f868e75f700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197621 7f86a5e11700 5 _do_op 0x288f9ec0 seq 4174 osr(3.2d6 0x3ed0e60)/0x3ed0e60 start 2014-06-09 16:22:12.197625 7f86a5e11700 10 _do_transaction on 0x390da40 2014-06-09 16:22:12.197630 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.197671 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.197685 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4174.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.197689 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.197700 7f86a5e11700 10 _do_op 0x288f9ec0 seq 4174 r = 0, finisher 0xbd6f060 0 2014-06-09 16:22:12.197711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3206f280] 2014-06-09 16:22:12.197714 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3206f280] 2014-06-09 16:22:12.197715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.197716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.197720 7f86a5e11700 10 _finish_op 0x288f9ec0 seq 4174 osr(3.2d6 0x3ed0e60)/0x3ed0e60 2014-06-09 16:22:12.197730 7f868a757700 10 op_submit_start 4223 2014-06-09 16:22:12.197732 7f868a757700 5 queue_transactions (trailing journal) 4223 2014-06-09 16:22:12.197790 7f868a757700 5 queue_op 0xcabc000 seq 4223 osr(1.2c0 0x384d170) 828 bytes (queue has 50 ops and 4515272 bytes) 2014-06-09 16:22:12.197796 7f868ef60700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197801 7f8690763700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197806 7f8690f64700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197810 7f8692767700 2 waiting 51 > 50 ops || 4861734 > 104857600 2014-06-09 16:22:12.197815 7f8691f66700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197819 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197823 7f8689755700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197828 7f868d75d700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197832 7f8689f56700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197837 7f868ff62700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197841 7f868af58700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197845 7f868f761700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197850 7f868df5e700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197854 7f868bf5a700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197858 7f868c75b700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197863 7f8692f68700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197867 7f868b759700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197871 7f868e75f700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197875 7f868cf5c700 2 waiting 51 > 50 ops || 4516133 > 104857600 2014-06-09 16:22:12.197885 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6f060] 2014-06-09 16:22:12.197892 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6f060] 2014-06-09 16:22:12.197894 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.197895 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.197898 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b9e0 (1 active) 2014-06-09 16:22:12.197900 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358fc0 (2 active) 2014-06-09 16:22:12.197905 7f868a757700 10 op_submit_finish 4223 2014-06-09 16:22:12.197909 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 243/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:12.197950 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.197955 7f868a757700 5 queue_transactions existing osr(3.218 0x3ed0cd0)/0x3ed0cd0 2014-06-09 16:22:12.197958 7f868a757700 2 waiting 51 > 50 ops || 4516100 > 104857600 2014-06-09 16:22:12.197962 7f86a5e11700 5 _do_op 0x324a4840 seq 4175 osr(1.fa 0x384c970)/0x384c970 start 2014-06-09 16:22:12.197966 7f86a5e11700 10 _do_transaction on 0xbf0e300 2014-06-09 16:22:12.197972 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.198013 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.198028 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4175.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.198032 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.198048 7f86a5e11700 10 _do_op 0x324a4840 seq 4175 r = 0, finisher 0x314129e0 0 2014-06-09 16:22:12.198059 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a7bc0] 2014-06-09 16:22:12.198062 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a7bc0] 2014-06-09 16:22:12.198064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.198065 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.198068 7f86a5e11700 10 _finish_op 0x324a4840 seq 4175 osr(1.fa 0x384c970)/0x384c970 2014-06-09 16:22:12.198078 7f8690763700 10 op_submit_start 4224 2014-06-09 16:22:12.198080 7f8690763700 5 queue_transactions (trailing journal) 4224 2014-06-09 16:22:12.198082 7f8690763700 5 queue_op 0xc34bc80 seq 4224 osr(0.686 0x2bd7e20) 861 bytes (queue has 50 ops and 4515307 bytes) 2014-06-09 16:22:12.198087 7f8690f64700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198091 7f8692767700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198096 7f8691f66700 2 waiting 51 > 50 ops || 4516168 > 104857600 2014-06-09 16:22:12.198100 7f8691765700 2 waiting 51 > 50 ops || 4516168 > 104857600 2014-06-09 16:22:12.198105 7f8689755700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198109 7f868d75d700 2 waiting 51 > 50 ops || 4516168 > 104857600 2014-06-09 16:22:12.198113 7f8689f56700 2 waiting 51 > 50 ops || 4516168 > 104857600 2014-06-09 16:22:12.198118 7f868ff62700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198122 7f868af58700 2 waiting 51 > 50 ops || 4516168 > 104857600 2014-06-09 16:22:12.198126 7f868f761700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198130 7f868df5e700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198134 7f868bf5a700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198139 7f868c75b700 2 waiting 51 > 50 ops || 4516168 > 104857600 2014-06-09 16:22:12.198143 7f8692f68700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198147 7f868b759700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198151 7f868e75f700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198155 7f868cf5c700 2 waiting 51 > 50 ops || 4516168 > 104857600 2014-06-09 16:22:12.198159 7f868a757700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198163 7f868ef60700 2 waiting 51 > 50 ops || 4516135 > 104857600 2014-06-09 16:22:12.198173 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x314129e0] 2014-06-09 16:22:12.198178 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x314129e0] 2014-06-09 16:22:12.198180 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.198181 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.198184 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358fc0 (1 active) 2014-06-09 16:22:12.198187 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286a20 (2 active) 2014-06-09 16:22:12.198192 7f8690763700 10 op_submit_finish 4224 2014-06-09 16:22:12.198195 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.198211 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.062160 7 0.000955 2014-06-09 16:22:12.198224 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.198236 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.198247 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.198259 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.198269 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.198278 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.198289 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.198301 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.198313 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.198327 7f8690763700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 2014-06-09 16:22:12.198344 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1113 2014-06-09 16:22:12.198379 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.198383 7f8690763700 5 queue_transactions existing osr(3.65f 0x3ed1a10)/0x3ed1a10 2014-06-09 16:22:12.198385 7f8690763700 2 waiting 51 > 50 ops || 4516168 > 104857600 2014-06-09 16:22:12.198390 7f86a5e11700 5 _do_op 0x309a3800 seq 4176 osr(1.3ef 0x384d3f0)/0x384d3f0 start 2014-06-09 16:22:12.198394 7f86a5e11700 10 _do_transaction on 0xc290900 2014-06-09 16:22:12.198399 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.198441 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.198457 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4176.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.198461 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.198476 7f86a5e11700 10 _do_op 0x309a3800 seq 4176 r = 0, finisher 0xccf8800 0 2014-06-09 16:22:12.198487 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd744c0] 2014-06-09 16:22:12.198490 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.198500 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.198510 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.198519 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd744c0] 2014-06-09 16:22:12.198521 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.198522 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.198526 7f86a5e11700 10 _finish_op 0x309a3800 seq 4176 osr(1.3ef 0x384d3f0)/0x384d3f0 2014-06-09 16:22:12.198537 7f8692767700 10 op_submit_start 4225 2014-06-09 16:22:12.198538 7f8692767700 5 queue_transactions (trailing journal) 4225 2014-06-09 16:22:12.198540 7f8692767700 5 queue_op 0x32539980 seq 4225 osr(3.56a 0x3ed1900) 346462 bytes (queue has 50 ops and 4860908 bytes) 2014-06-09 16:22:12.198544 7f8691f66700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198549 7f8691765700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198554 7f8689755700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198558 7f868d75d700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198562 7f8689f56700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198566 7f868ff62700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198571 7f868af58700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198575 7f868f761700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198579 7f868df5e700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198584 7f868bf5a700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198588 7f868c75b700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198592 7f8692f68700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198596 7f868b759700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198600 7f868e75f700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198605 7f868cf5c700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198609 7f868a757700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198613 7f868ef60700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198617 7f8690763700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198622 7f8690f64700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.198631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf8800] 2014-06-09 16:22:12.198635 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf8800] 2014-06-09 16:22:12.198637 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.198638 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.198641 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286a20 (1 active) 2014-06-09 16:22:12.198644 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524000 (2 active) 2014-06-09 16:22:12.198649 7f8692767700 10 op_submit_finish 4225 2014-06-09 16:22:12.198653 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc20d140 243'576 rep_tid=31 committed?=0 applied?=0 lock=0 op=osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.198676 7f8692767700 10 osd.26 244 dequeue_op 0x230da800 finish 2014-06-09 16:22:12.198681 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.198726 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e560 (20 active) 2014-06-09 16:22:12.198730 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.198743 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.999816 7 0.000446 2014-06-09 16:22:12.198754 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.198765 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.198775 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.198785 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.198794 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.198801 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.198812 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.198822 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.198832 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.198845 7f8692767700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbfa6580 172.16.96.201:6851/52636 2014-06-09 16:22:12.198868 7f8692767700 10 osd.26 243 note_peer_epoch osd.7 has 244 2014-06-09 16:22:12.198880 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.198913 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.198917 7f8692767700 5 queue_transactions existing osr(1.4bc 0x384d930)/0x384d930 2014-06-09 16:22:12.198919 7f8692767700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.198924 7f86a5e11700 5 _do_op 0x39d9da0 seq 4177 osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 start 2014-06-09 16:22:12.198927 7f86a5e11700 10 _do_transaction on 0xbfb2600 2014-06-09 16:22:12.198933 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.198974 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.198988 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4177.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.198993 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.199003 7f86a5e11700 10 _do_op 0x39d9da0 seq 4177 r = 0, finisher 0xbd5bca0 0 2014-06-09 16:22:12.199013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e19d40] 2014-06-09 16:22:12.199016 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e19d40] 2014-06-09 16:22:12.199017 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.199019 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.199022 7f86a5e11700 10 _finish_op 0x39d9da0 seq 4177 osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 2014-06-09 16:22:12.199033 7f8691765700 10 op_submit_start 4226 2014-06-09 16:22:12.199034 7f8691765700 5 queue_transactions (trailing journal) 4226 2014-06-09 16:22:12.199036 7f8691765700 5 queue_op 0x16fd4240 seq 4226 osr(1.37d 0x384d210) 861 bytes (queue has 50 ops and 4860941 bytes) 2014-06-09 16:22:12.199041 7f8689755700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199045 7f868d75d700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199050 7f8689f56700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199054 7f868ff62700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199058 7f868af58700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199063 7f868f761700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199067 7f868df5e700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199071 7f868bf5a700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199075 7f868c75b700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199079 7f8692f68700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199084 7f868b759700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199088 7f868e75f700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199093 7f868cf5c700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199097 7f868a757700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199102 7f868ef60700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199106 7f8690763700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199110 7f8690f64700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199114 7f8692767700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199118 7f8691f66700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199127 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5bca0] 2014-06-09 16:22:12.199131 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5bca0] 2014-06-09 16:22:12.199133 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.199134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.199137 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524000 (1 active) 2014-06-09 16:22:12.199140 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd320 (2 active) 2014-06-09 16:22:12.199145 7f8691765700 10 op_submit_finish 4226 2014-06-09 16:22:12.199149 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.199162 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.037109 7 0.000295 2014-06-09 16:22:12.199174 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.199184 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.199194 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.199205 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.199213 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.199221 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.199231 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.199243 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.199252 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.199262 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.199271 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.199284 7f8691765700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 2014-06-09 16:22:12.199300 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:206 2014-06-09 16:22:12.199332 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.199336 7f8691765700 5 queue_transactions existing osr(2.3c7 0x2ed13d0)/0x2ed13d0 2014-06-09 16:22:12.199339 7f8691765700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199366 7f86a5e11700 5 _do_op 0x309a3560 seq 4178 osr(2.61b 0x2ed1cd0)/0x2ed1cd0 start 2014-06-09 16:22:12.199374 7f86a5e11700 10 _do_transaction on 0xc04f380 2014-06-09 16:22:12.199381 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.199448 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.199466 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4178.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.199470 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.199484 7f86a5e11700 10 _do_op 0x309a3560 seq 4178 r = 0, finisher 0xbd779a0 0 2014-06-09 16:22:12.199495 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf0b40] 2014-06-09 16:22:12.199498 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.199508 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.199517 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:12.199526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf0b40] 2014-06-09 16:22:12.199528 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.199529 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.199532 7f86a5e11700 10 _finish_op 0x309a3560 seq 4178 osr(2.61b 0x2ed1cd0)/0x2ed1cd0 2014-06-09 16:22:12.199544 7f868d75d700 10 op_submit_start 4227 2014-06-09 16:22:12.199545 7f868d75d700 5 queue_transactions (trailing journal) 4227 2014-06-09 16:22:12.199547 7f868d75d700 5 queue_op 0xc2d4060 seq 4227 osr(1.764 0x2ed03f0) 861 bytes (queue has 50 ops and 4860941 bytes) 2014-06-09 16:22:12.199551 7f8689f56700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199556 7f868ff62700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199560 7f868af58700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199565 7f868f761700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199569 7f868df5e700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199573 7f868bf5a700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199577 7f868c75b700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199582 7f8692f68700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199586 7f868b759700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199590 7f868e75f700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199594 7f868cf5c700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199598 7f868a757700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199602 7f868ef60700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199606 7f8690763700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199610 7f8690f64700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199619 7f8692767700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199624 7f8691f66700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199628 7f8691765700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.199633 7f8689755700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.199642 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd779a0] 2014-06-09 16:22:12.199646 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd779a0] 2014-06-09 16:22:12.199648 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.199649 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.199707 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd320 (1 active) 2014-06-09 16:22:12.199711 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1320 (2 active) 2014-06-09 16:22:12.199716 7f868d75d700 10 op_submit_finish 4227 2014-06-09 16:22:12.199720 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.199733 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.199742 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.199750 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.199759 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.199767 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.199781 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 2014-06-09 16:22:12.199790 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.199798 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.199808 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.199813 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.199824 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.024301 7 0.000342 2014-06-09 16:22:12.199835 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.199846 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.199856 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.199866 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.199874 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.199882 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.199893 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.199904 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.199914 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.199923 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.199932 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.199944 7f868d75d700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 2014-06-09 16:22:12.199960 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.199992 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.199996 7f868d75d700 5 queue_transactions existing osr(2.5f4 0x2ed1dd0)/0x2ed1dd0 2014-06-09 16:22:12.199999 7f868d75d700 2 waiting 51 > 50 ops || 4861802 > 104857600 2014-06-09 16:22:12.200028 7f86a5e11700 5 _do_op 0xc704ba0 seq 4179 osr(1.446 0x384d6b0)/0x384d6b0 start 2014-06-09 16:22:12.200035 7f86a5e11700 10 _do_transaction on 0xc2d9140 2014-06-09 16:22:12.200041 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.200083 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.200101 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4179.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.200105 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.200119 7f86a5e11700 10 _do_op 0xc704ba0 seq 4179 r = 0, finisher 0xbd6b5c0 0 2014-06-09 16:22:12.200129 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31791ec0] 2014-06-09 16:22:12.200132 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.200142 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.200151 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:12.200160 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31791ec0] 2014-06-09 16:22:12.200162 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.200163 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.200167 7f86a5e11700 10 _finish_op 0xc704ba0 seq 4179 osr(1.446 0x384d6b0)/0x384d6b0 2014-06-09 16:22:12.200179 7f868ff62700 10 op_submit_start 4228 2014-06-09 16:22:12.200180 7f868ff62700 5 queue_transactions (trailing journal) 4228 2014-06-09 16:22:12.200182 7f868ff62700 5 queue_op 0x3250f9e0 seq 4228 osr(1.4bb 0x384d970) 828 bytes (queue has 50 ops and 4860908 bytes) 2014-06-09 16:22:12.200187 7f868af58700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200191 7f868f761700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200196 7f868df5e700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200200 7f868bf5a700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200204 7f868c75b700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200208 7f8692f68700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200213 7f868b759700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200217 7f868e75f700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200221 7f868cf5c700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200226 7f868a757700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200230 7f868ef60700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200234 7f8690763700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200238 7f8690f64700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200242 7f8692767700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200247 7f8691f66700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200251 7f8691765700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200255 7f8689755700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200259 7f868d75d700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200263 7f8689f56700 2 waiting 51 > 50 ops || 4861769 > 104857600 2014-06-09 16:22:12.200273 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6b5c0] 2014-06-09 16:22:12.200278 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6b5c0] 2014-06-09 16:22:12.200280 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.200281 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.200284 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1320 (1 active) 2014-06-09 16:22:12.200287 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d19e0 (2 active) 2014-06-09 16:22:12.200292 7f868ff62700 10 op_submit_finish 4228 2014-06-09 16:22:12.200296 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.200311 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.200321 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x14052400,0x1c54b000,0x14052300,0x1c54af00,0x21966c00,0x21966b00,0x21966a00,0x21966900,0x21966800,0x1804db00,0x1804da00,0x4679200,0x4679100,0x4679000,0x4678f00,0x4678e00,0x4678d00 2014-06-09 16:22:12.200374 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.200384 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.200392 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.200406 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1100 2014-06-09 16:22:12.200417 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.200426 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.200436 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.200441 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.6f0( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:12.200474 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.200479 7f868ff62700 5 queue_transactions existing osr(0.6f0 0x384c170)/0x384c170 2014-06-09 16:22:12.200481 7f868ff62700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200486 7f86a5e11700 5 _do_op 0x324e62a0 seq 4180 osr(0.66b 0x2bd7eb0)/0x2bd7eb0 start 2014-06-09 16:22:12.200490 7f86a5e11700 10 _do_transaction on 0x398ba40 2014-06-09 16:22:12.200495 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.200537 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.200554 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4180.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.200559 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.200581 7f86a5e11700 10 _do_op 0x324e62a0 seq 4180 r = 0, finisher 0xc6e91c0 0 2014-06-09 16:22:12.200593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5f780] 2014-06-09 16:22:12.200596 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.200606 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.200615 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.200624 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5f780] 2014-06-09 16:22:12.200626 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.200627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.200631 7f86a5e11700 10 _finish_op 0x324e62a0 seq 4180 osr(0.66b 0x2bd7eb0)/0x2bd7eb0 2014-06-09 16:22:12.200642 7f868f761700 10 op_submit_start 4229 2014-06-09 16:22:12.200644 7f868f761700 5 queue_transactions (trailing journal) 4229 2014-06-09 16:22:12.200645 7f868f761700 5 queue_op 0x324e6c60 seq 4229 osr(3.3a5 0x3ed0fb0) 828 bytes (queue has 50 ops and 4860875 bytes) 2014-06-09 16:22:12.200650 7f868df5e700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200654 7f868bf5a700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200658 7f868c75b700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200663 7f8692f68700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200667 7f868b759700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200671 7f868e75f700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200675 7f868cf5c700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200680 7f868a757700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200684 7f868ef60700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200688 7f8690763700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200693 7f8690f64700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200697 7f8692767700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200701 7f8691f66700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200705 7f8691765700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200709 7f8689755700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200714 7f868d75d700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200718 7f8689f56700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200722 7f868ff62700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200726 7f868af58700 2 waiting 51 > 50 ops || 4861736 > 104857600 2014-06-09 16:22:12.200736 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e91c0] 2014-06-09 16:22:12.200741 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e91c0] 2014-06-09 16:22:12.200743 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.200744 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.200747 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d19e0 (1 active) 2014-06-09 16:22:12.200749 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497200 (2 active) 2014-06-09 16:22:12.200754 7f868f761700 10 op_submit_finish 4229 2014-06-09 16:22:12.200760 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.48e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.200796 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.200801 7f868f761700 5 queue_transactions existing osr(1.48e 0x384d9b0)/0x384d9b0 2014-06-09 16:22:12.200804 7f868f761700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200824 7f86a5e11700 5 _do_op 0x160970e0 seq 4181 osr(0.67d 0x2bd7e50)/0x2bd7e50 start 2014-06-09 16:22:12.200827 7f86a5e11700 10 _do_transaction on 0x2f28d80 2014-06-09 16:22:12.200833 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.200874 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.200891 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4181.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.200896 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.200908 7f86a5e11700 10 _do_op 0x160970e0 seq 4181 r = 0, finisher 0xbdf1dc0 0 2014-06-09 16:22:12.200918 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050b0e0] 2014-06-09 16:22:12.200921 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.200931 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.200940 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 2014-06-09 16:22:12.200949 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050b0e0] 2014-06-09 16:22:12.200951 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.200952 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.200956 7f86a5e11700 10 _finish_op 0x160970e0 seq 4181 osr(0.67d 0x2bd7e50)/0x2bd7e50 2014-06-09 16:22:12.200967 7f868bf5a700 10 op_submit_start 4230 2014-06-09 16:22:12.200969 7f868bf5a700 5 queue_transactions (trailing journal) 4230 2014-06-09 16:22:12.200970 7f868bf5a700 5 queue_op 0x324da120 seq 4230 osr(0.3fe 0x2bd7210) 828 bytes (queue has 50 ops and 4860842 bytes) 2014-06-09 16:22:12.200975 7f868c75b700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200979 7f8692f68700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.200983 7f868b759700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.200988 7f868e75f700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.200992 7f868cf5c700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.200996 7f868a757700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201000 7f868ef60700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201005 7f8690763700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201009 7f8690f64700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201013 7f8692767700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201018 7f8691f66700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201022 7f8691765700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201026 7f8689755700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201030 7f868d75d700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201035 7f8689f56700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201039 7f868ff62700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201043 7f868af58700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201047 7f868f761700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201051 7f868df5e700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201061 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdf1dc0] 2014-06-09 16:22:12.201065 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdf1dc0] 2014-06-09 16:22:12.201067 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.201068 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.201071 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497200 (1 active) 2014-06-09 16:22:12.201074 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3faea0 (2 active) 2014-06-09 16:22:12.201079 7f868bf5a700 10 op_submit_finish 4230 2014-06-09 16:22:12.201082 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.201095 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.201104 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.201112 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.201120 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.201128 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.201141 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.201150 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.201159 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.201168 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.201172 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.582( v 214'602 lc 214'601 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.201186 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.201196 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.201205 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.201219 7f868bf5a700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbf7b080 172.16.96.201:6828/58271 2014-06-09 16:22:12.201240 7f868bf5a700 10 osd.26 243 note_peer_epoch osd.2 has 244 2014-06-09 16:22:12.201252 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1029 2014-06-09 16:22:12.201267 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.201271 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.428( v 214'563 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) 2014-06-09 16:22:12.201288 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.6 activated and committed 2014-06-09 16:22:12.201300 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.201311 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.201325 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 2014-06-09 16:22:12.201340 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:910 2014-06-09 16:22:12.201368 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.196639 7f86a68f6700 5 -- op tracker -- , seq: 3679, time: 2014-06-09 16:22:12.195281, event: throttled, op: osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.201373 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.201382 7f86a68f6700 5 -- op tracker -- , seq: 3679, time: 2014-06-09 16:22:12.196466, event: all_read, op: osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.201384 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.034071 7 0.003518 2014-06-09 16:22:12.201392 7f86a68f6700 5 -- op tracker -- , seq: 3679, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.201394 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.201399 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.201403 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.201404 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.201412 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.201421 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.201429 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.201429 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976dc00 prio 63 cost 847872 latency 0.006149 osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.201436 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.201445 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.201455 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.201464 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.201475 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 2014-06-09 16:22:12.201491 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 2014-06-09 16:22:12.201520 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.201524 7f868bf5a700 5 queue_transactions existing osr(2.40f 0x2ed1210)/0x2ed1210 2014-06-09 16:22:12.201526 7f868bf5a700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201531 7f86a5e11700 5 _do_op 0xcabc780 seq 4182 osr(2.216 0x2ed0b50)/0x2ed0b50 start 2014-06-09 16:22:12.201534 7f86a5e11700 10 _do_transaction on 0xc2efe00 2014-06-09 16:22:12.201539 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.201647 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.201662 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4182.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.201666 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.201678 7f86a5e11700 10 _do_op 0xcabc780 seq 4182 r = 0, finisher 0x30e3ea00 0 2014-06-09 16:22:12.201687 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3e9c0] 2014-06-09 16:22:12.201689 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3e9c0] 2014-06-09 16:22:12.201690 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.201691 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.201694 7f86a5e11700 10 _finish_op 0xcabc780 seq 4182 osr(2.216 0x2ed0b50)/0x2ed0b50 2014-06-09 16:22:12.201702 7f8692f68700 10 op_submit_start 4231 2014-06-09 16:22:12.201703 7f8692f68700 5 queue_transactions (trailing journal) 4231 2014-06-09 16:22:12.201705 7f8692f68700 5 queue_op 0xd327260 seq 4231 osr(1.302 0x384d010) 828 bytes (queue has 50 ops and 4860842 bytes) 2014-06-09 16:22:12.201708 7f868b759700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201712 7f868e75f700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201715 7f868cf5c700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201718 7f868a757700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201721 7f868ef60700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201725 7f8690763700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201728 7f8690f64700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201731 7f8692767700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201734 7f8691f66700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201737 7f8691765700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201740 7f8689755700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201743 7f868d75d700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201746 7f8689f56700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201749 7f868ff62700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201752 7f868af58700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201755 7f868f761700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201758 7f868df5e700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201761 7f868bf5a700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201764 7f868c75b700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201772 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e3ea00] 2014-06-09 16:22:12.201776 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e3ea00] 2014-06-09 16:22:12.201777 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.201778 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.201780 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3faea0 (1 active) 2014-06-09 16:22:12.201782 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac000 (2 active) 2014-06-09 16:22:12.201786 7f8692f68700 10 op_submit_finish 4231 2014-06-09 16:22:12.201789 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 243/191 242/242/222) 2014-06-09 16:22:12.201821 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.201825 7f8692f68700 5 queue_transactions existing osr(3.111 0x3ed0830)/0x3ed0830 2014-06-09 16:22:12.201827 7f8692f68700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201831 7f86a5e11700 5 _do_op 0x39d98c0 seq 4183 osr(2.5f8 0x2ed1db0)/0x2ed1db0 start 2014-06-09 16:22:12.201834 7f86a5e11700 10 _do_transaction on 0xbfb2780 2014-06-09 16:22:12.201838 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.201874 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.201886 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4183.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.201890 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.201899 7f86a5e11700 10 _do_op 0x39d98c0 seq 4183 r = 0, finisher 0x3042b500 0 2014-06-09 16:22:12.201907 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6c6e0] 2014-06-09 16:22:12.201909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6c6e0] 2014-06-09 16:22:12.201910 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.201911 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.201914 7f86a5e11700 10 _finish_op 0x39d98c0 seq 4183 osr(2.5f8 0x2ed1db0)/0x2ed1db0 2014-06-09 16:22:12.201922 7f868e75f700 10 op_submit_start 4232 2014-06-09 16:22:12.201923 7f868e75f700 5 queue_transactions (trailing journal) 4232 2014-06-09 16:22:12.201924 7f868e75f700 5 queue_op 0x324f4b40 seq 4232 osr(2.2f2 0x2ed0fb0) 828 bytes (queue has 50 ops and 4860842 bytes) 2014-06-09 16:22:12.201928 7f868cf5c700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201931 7f868a757700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201935 7f868ef60700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201938 7f8690763700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201941 7f8690f64700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201944 7f8692767700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201947 7f8691f66700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201950 7f8691765700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201953 7f8689755700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201956 7f868d75d700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201959 7f8689f56700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201962 7f868ff62700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201965 7f868af58700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201968 7f868f761700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201971 7f868df5e700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201974 7f868bf5a700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201977 7f868c75b700 2 waiting 51 > 50 ops || 4861703 > 104857600 2014-06-09 16:22:12.201980 7f8692f68700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201983 7f868b759700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.201990 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b500] 2014-06-09 16:22:12.201994 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b500] 2014-06-09 16:22:12.201995 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.201996 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.201998 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac000 (1 active) 2014-06-09 16:22:12.202000 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383eea0 (2 active) 2014-06-09 16:22:12.202004 7f868e75f700 10 op_submit_finish 4232 2014-06-09 16:22:12.202007 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.32f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.202035 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.202039 7f868e75f700 5 queue_transactions existing osr(1.32f 0x384d350)/0x384d350 2014-06-09 16:22:12.202041 7f868e75f700 2 waiting 51 > 50 ops || 4861670 > 104857600 2014-06-09 16:22:12.202045 7f86a5e11700 5 _do_op 0x324e61e0 seq 4184 osr(0.29 0x2bd62b0)/0x2bd62b0 start 2014-06-09 16:22:12.202048 7f86a5e11700 10 _do_transaction on 0x2ed4b40 2014-06-09 16:22:12.202053 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.202089 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.202105 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4184.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.202109 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.202111 7f86a68f6700 5 -- op tracker -- , seq: 3680, time: 2014-06-09 16:22:12.201451, event: header_read, op: osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.202122 7f86a68f6700 5 -- op tracker -- , seq: 3680, time: 2014-06-09 16:22:12.201452, event: throttled, op: osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.202132 7f86a5e11700 10 _do_op 0x324e61e0 seq 4184 r = 0, finisher 0xcc94340 0 2014-06-09 16:22:12.202136 7f86a68f6700 5 -- op tracker -- , seq: 3680, time: 2014-06-09 16:22:12.202074, event: all_read, op: osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.202142 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd622e0] 2014-06-09 16:22:12.202146 7f86a68f6700 5 -- op tracker -- , seq: 3680, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.202145 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.202151 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.202155 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.202154 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.202163 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 2014-06-09 16:22:12.202171 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd622e0] 2014-06-09 16:22:12.202173 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.202174 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.202168 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976db00 prio 63 cost 626688 latency 0.000717 osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.202177 7f86a5e11700 10 _finish_op 0x324e61e0 seq 4184 osr(0.29 0x2bd62b0)/0x2bd62b0 2014-06-09 16:22:12.202187 7f868a757700 10 op_submit_start 4233 2014-06-09 16:22:12.202189 7f868a757700 5 queue_transactions (trailing journal) 4233 2014-06-09 16:22:12.202190 7f868a757700 5 queue_op 0xcabcfc0 seq 4233 osr(3.218 0x3ed0cd0) 828 bytes (queue has 50 ops and 4860812 bytes) 2014-06-09 16:22:12.202195 7f868ef60700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202199 7f8690763700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202204 7f8690f64700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202207 7f8692767700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202211 7f8691f66700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202214 7f8691765700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202218 7f8689755700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202222 7f868d75d700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202225 7f8689f56700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202228 7f868ff62700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202232 7f868af58700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202235 7f868f761700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202238 7f868df5e700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202242 7f868bf5a700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202246 7f868c75b700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202249 7f8692f68700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202253 7f868b759700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202257 7f868e75f700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202260 7f868cf5c700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202269 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc94340] 2014-06-09 16:22:12.202282 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc94340] 2014-06-09 16:22:12.202283 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.202284 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.202287 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383eea0 (1 active) 2014-06-09 16:22:12.202290 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5250e0 (2 active) 2014-06-09 16:22:12.202295 7f868a757700 10 op_submit_finish 4233 2014-06-09 16:22:12.202299 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.4ba( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.202331 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.202337 7f868a757700 5 queue_transactions existing osr(0.4ba 0x2bd7710)/0x2bd7710 2014-06-09 16:22:12.202340 7f868a757700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202344 7f86a5e11700 5 _do_op 0x160967e0 seq 4185 osr(1.25a 0x384cfb0)/0x384cfb0 start 2014-06-09 16:22:12.202348 7f86a5e11700 10 _do_transaction on 0xc1449c0 2014-06-09 16:22:12.202353 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.202395 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.202410 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4185.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.202414 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.202428 7f86a5e11700 10 _do_op 0x160967e0 seq 4185 r = 0, finisher 0x30629dc0 0 2014-06-09 16:22:12.202437 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3e600] 2014-06-09 16:22:12.202440 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.202448 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.202456 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:164 2014-06-09 16:22:12.202468 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3e600] 2014-06-09 16:22:12.202470 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.202471 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.202473 7f86a5e11700 10 _finish_op 0x160967e0 seq 4185 osr(1.25a 0x384cfb0)/0x384cfb0 2014-06-09 16:22:12.202483 7f8690763700 10 op_submit_start 4234 2014-06-09 16:22:12.202484 7f8690763700 5 queue_transactions (trailing journal) 4234 2014-06-09 16:22:12.202485 7f8690763700 5 queue_op 0xc34a660 seq 4234 osr(3.65f 0x3ed1a10) 861 bytes (queue has 50 ops and 4860812 bytes) 2014-06-09 16:22:12.202489 7f8690f64700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202492 7f8692767700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202496 7f8691f66700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202499 7f8691765700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202502 7f8689755700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202505 7f868d75d700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202508 7f8689f56700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202511 7f868ff62700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202514 7f868af58700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202517 7f868f761700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202520 7f868df5e700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202523 7f868bf5a700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202526 7f868c75b700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202529 7f8692f68700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202532 7f868b759700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202535 7f868e75f700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202538 7f868cf5c700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202541 7f868a757700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202544 7f868ef60700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202551 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629dc0] 2014-06-09 16:22:12.202555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629dc0] 2014-06-09 16:22:12.202557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.202557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.202560 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5250e0 (1 active) 2014-06-09 16:22:12.202562 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62eea0 (2 active) 2014-06-09 16:22:12.202566 7f8690763700 10 op_submit_finish 4234 2014-06-09 16:22:12.202569 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.202582 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.202589 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.202597 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.202604 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.202610 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.202622 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 2014-06-09 16:22:12.202630 7f8690763700 20 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.202636 7f8690763700 20 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.202644 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.202648 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.202660 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] exit Started/Primary/Active/Activating 1.125374 7 0.001337 2014-06-09 16:22:12.202671 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.202681 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] needs_recovery is recovered 2014-06-09 16:22:12.202690 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.202700 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.202708 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] finish_recovery 2014-06-09 16:22:12.202715 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] clear_recovery_state 2014-06-09 16:22:12.202725 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.202736 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.202745 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] share_pg_info 2014-06-09 16:22:12.202760 7f8690763700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 2014-06-09 16:22:12.202775 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] publish_stats_to_osd 243:928 2014-06-09 16:22:12.202807 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.202805 7f86a68f6700 5 -- op tracker -- , seq: 3681, time: 2014-06-09 16:22:12.202189, event: header_read, op: osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.202811 7f8690763700 5 queue_transactions existing osr(3.661 0x3ed19f0)/0x3ed19f0 2014-06-09 16:22:12.202813 7f8690763700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202816 7f86a68f6700 5 -- op tracker -- , seq: 3681, time: 2014-06-09 16:22:12.202190, event: throttled, op: osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.202819 7f86a5e11700 5 _do_op 0x1e08a000 seq 4186 osr(0.678 0x2bd7e90)/0x2bd7e90 start 2014-06-09 16:22:12.202822 7f86a5e11700 10 _do_transaction on 0xc2d9c80 2014-06-09 16:22:12.202828 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.202826 7f86a68f6700 5 -- op tracker -- , seq: 3681, time: 2014-06-09 16:22:12.202744, event: all_read, op: osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.202835 7f86a68f6700 5 -- op tracker -- , seq: 3681, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.202840 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.202845 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.202857 7f86a68f6700 15 osd.26 244 enqueue_op 0x1c54bb00 prio 63 cost 516096 latency 0.000668 osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.202867 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.202884 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4186.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.202888 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.202899 7f86a5e11700 10 _do_op 0x1e08a000 seq 4186 r = 0, finisher 0xccf9c00 0 2014-06-09 16:22:12.202908 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6ace0] 2014-06-09 16:22:12.202911 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.202920 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.202928 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:187 2014-06-09 16:22:12.202937 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6ace0] 2014-06-09 16:22:12.202938 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.202940 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.202943 7f86a5e11700 10 _finish_op 0x1e08a000 seq 4186 osr(0.678 0x2bd7e90)/0x2bd7e90 2014-06-09 16:22:12.202953 7f8692767700 10 op_submit_start 4235 2014-06-09 16:22:12.202955 7f8692767700 5 queue_transactions (trailing journal) 4235 2014-06-09 16:22:12.202956 7f8692767700 5 queue_op 0xc2eaba0 seq 4235 osr(1.4bc 0x384d930) 861 bytes (queue has 50 ops and 4860812 bytes) 2014-06-09 16:22:12.202961 7f8691f66700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202965 7f8691765700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202969 7f8689755700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202973 7f868d75d700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202976 7f8689f56700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202980 7f868ff62700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202984 7f868af58700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202987 7f868f761700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202990 7f868df5e700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.202994 7f868bf5a700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.202997 7f868c75b700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203001 7f8692f68700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.203005 7f868b759700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.203008 7f868e75f700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.203012 7f868cf5c700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203016 7f868a757700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.203019 7f868ef60700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.203023 7f8690763700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203026 7f8690f64700 2 waiting 51 > 50 ops || 4861640 > 104857600 2014-06-09 16:22:12.203034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf9c00] 2014-06-09 16:22:12.203038 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf9c00] 2014-06-09 16:22:12.203040 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.203041 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.203043 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62eea0 (1 active) 2014-06-09 16:22:12.203046 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358000 (2 active) 2014-06-09 16:22:12.203051 7f8692767700 10 op_submit_finish 4235 2014-06-09 16:22:12.203054 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.203067 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.046040 7 0.000306 2014-06-09 16:22:12.203078 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.203087 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.203097 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.203106 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.203114 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.203121 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.203130 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.203141 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.203150 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.203159 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.203167 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.203181 7f8692767700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 244 2014-06-09 16:22:12.203197 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.203229 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.203233 7f8692767700 5 queue_transactions existing osr(2.3de 0x2ed1330)/0x2ed1330 2014-06-09 16:22:12.203235 7f8692767700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203261 7f86a5e11700 5 _do_op 0x30a3d40 seq 4187 osr(0.31f 0x2bd6d50)/0x2bd6d50 start 2014-06-09 16:22:12.203269 7f86a5e11700 10 _do_transaction on 0xc270b40 2014-06-09 16:22:12.203274 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.203315 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.203329 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4187.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.203333 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.203345 7f86a5e11700 10 _do_op 0x30a3d40 seq 4187 r = 0, finisher 0x307f0340 0 2014-06-09 16:22:12.203355 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f02c0] 2014-06-09 16:22:12.203357 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f02c0] 2014-06-09 16:22:12.203359 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.203360 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.203363 7f86a5e11700 10 _finish_op 0x30a3d40 seq 4187 osr(0.31f 0x2bd6d50)/0x2bd6d50 2014-06-09 16:22:12.203373 7f8691765700 10 op_submit_start 4236 2014-06-09 16:22:12.203374 7f8691765700 5 queue_transactions (trailing journal) 4236 2014-06-09 16:22:12.203376 7f8691765700 5 queue_op 0xc64a960 seq 4236 osr(2.3c7 0x2ed13d0) 861 bytes (queue has 50 ops and 4860845 bytes) 2014-06-09 16:22:12.203380 7f8689755700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203384 7f868d75d700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203388 7f8689f56700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203392 7f868ff62700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203395 7f868af58700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203399 7f868f761700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203403 7f868df5e700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203406 7f868bf5a700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203410 7f868c75b700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203414 7f8692f68700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203418 7f868b759700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203421 7f868e75f700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203425 7f868cf5c700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203428 7f868a757700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203432 7f868ef60700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203436 7f8690763700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203440 7f8690f64700 2 waiting 51 > 50 ops || 4861673 > 104857600 2014-06-09 16:22:12.203444 7f8692767700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203447 7f8691f66700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203456 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f0340] 2014-06-09 16:22:12.203459 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f0340] 2014-06-09 16:22:12.203461 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.203462 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.203464 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358000 (1 active) 2014-06-09 16:22:12.203467 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5680 (2 active) 2014-06-09 16:22:12.203472 7f8691765700 10 op_submit_finish 4236 2014-06-09 16:22:12.203475 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.203555 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.203564 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16bd6e00,0x21967900,0x16bd6d00,0x21967800,0x1c54ae00,0x1c54ad00,0x21967700,0x21967600 2014-06-09 16:22:12.203595 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.203605 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.203630 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.203645 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:965 2014-06-09 16:22:12.203655 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.203664 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.203674 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.203679 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.203690 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.051479 7 0.001368 2014-06-09 16:22:12.203701 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.203711 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.203721 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.203730 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.203738 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.203745 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.203755 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.203765 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.203774 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.203786 7f8691765700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 2014-06-09 16:22:12.203802 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:162 2014-06-09 16:22:12.203833 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.203838 7f86a5e11700 5 _do_op 0x2c06eba0 seq 4188 osr(1.4fb 0x384d7f0)/0x384d7f0 start 2014-06-09 16:22:12.203846 7f8691765700 5 queue_transactions existing osr(1.65d 0x384df50)/0x384df50 2014-06-09 16:22:12.203849 7f8691765700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.203842 7f86a5e11700 10 _do_transaction on 0xc0d9e00 2014-06-09 16:22:12.203858 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.203900 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.203915 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4188.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.203919 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.203955 7f86a5e11700 10 _do_op 0x2c06eba0 seq 4188 r = 0, finisher 0xbd67540 0 2014-06-09 16:22:12.203966 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205da60] 2014-06-09 16:22:12.203968 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205da60] 2014-06-09 16:22:12.203970 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.203971 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.203974 7f86a5e11700 10 _finish_op 0x2c06eba0 seq 4188 osr(1.4fb 0x384d7f0)/0x384d7f0 2014-06-09 16:22:12.203985 7f868d75d700 10 op_submit_start 4237 2014-06-09 16:22:12.203986 7f868d75d700 5 queue_transactions (trailing journal) 4237 2014-06-09 16:22:12.203988 7f868d75d700 5 queue_op 0x308cb7a0 seq 4237 osr(2.5f4 0x2ed1dd0) 861 bytes (queue has 50 ops and 4860878 bytes) 2014-06-09 16:22:12.203993 7f8689f56700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.203997 7f868ff62700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204001 7f868af58700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204005 7f868f761700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204009 7f868df5e700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204013 7f868bf5a700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204016 7f868c75b700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204020 7f8692f68700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204024 7f868b759700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204027 7f868e75f700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204031 7f868cf5c700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204035 7f868a757700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204039 7f868ef60700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204044 7f8690763700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204048 7f8690f64700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204053 7f8692767700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204057 7f8691f66700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204061 7f8691765700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204065 7f8689755700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204074 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd67540] 2014-06-09 16:22:12.204084 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd67540] 2014-06-09 16:22:12.204085 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.204086 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.204089 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5680 (1 active) 2014-06-09 16:22:12.204092 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fdd40 (2 active) 2014-06-09 16:22:12.204097 7f868d75d700 10 op_submit_finish 4237 2014-06-09 16:22:12.204101 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.204116 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.204125 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.204133 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.204141 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.204148 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.204161 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 2014-06-09 16:22:12.204169 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.204177 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.204186 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.204191 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.204203 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.103693 7 0.000357 2014-06-09 16:22:12.204213 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.204222 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.204231 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.204241 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.204248 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.204255 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.204264 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.204275 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.204284 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.204297 7f868d75d700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 244 2014-06-09 16:22:12.204313 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.204343 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.204347 7f868d75d700 5 queue_transactions existing osr(0.6b3 0x2bd7ff0)/0x2bd7ff0 2014-06-09 16:22:12.204349 7f868d75d700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204354 7f86a5e11700 5 _do_op 0x324db080 seq 4189 osr(1.14c 0x384c810)/0x384c810 start 2014-06-09 16:22:12.204357 7f86a5e11700 10 _do_transaction on 0xbf44cc0 2014-06-09 16:22:12.204363 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.204405 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.204420 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4189.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.204424 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.204434 7f86a5e11700 10 _do_op 0x324db080 seq 4189 r = 0, finisher 0x4eb8b60 0 2014-06-09 16:22:12.204444 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314574e0] 2014-06-09 16:22:12.204446 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314574e0] 2014-06-09 16:22:12.204451 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.204452 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.204455 7f86a5e11700 10 _finish_op 0x324db080 seq 4189 osr(1.14c 0x384c810)/0x384c810 2014-06-09 16:22:12.204465 7f868ff62700 10 op_submit_start 4238 2014-06-09 16:22:12.204466 7f868ff62700 5 queue_transactions (trailing journal) 4238 2014-06-09 16:22:12.204468 7f868ff62700 5 queue_op 0x325034a0 seq 4238 osr(0.6f0 0x384c170) 828 bytes (queue has 50 ops and 4860878 bytes) 2014-06-09 16:22:12.204473 7f868af58700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204477 7f868f761700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204481 7f868df5e700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204484 7f868bf5a700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204488 7f868c75b700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204492 7f8692f68700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204496 7f868b759700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204500 7f868e75f700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204504 7f868cf5c700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204508 7f868a757700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204512 7f868ef60700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204515 7f8690763700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204519 7f8690f64700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204523 7f8692767700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204527 7f8691f66700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204531 7f8691765700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204535 7f8689755700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.204539 7f868d75d700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204542 7f8689f56700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.204555 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fdd40 (1 active) 2014-06-09 16:22:12.204551 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x4eb8b60] 2014-06-09 16:22:12.206060 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x4eb8b60] 2014-06-09 16:22:12.206064 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.206065 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.206068 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e240 (2 active) 2014-06-09 16:22:12.206073 7f868ff62700 10 op_submit_finish 4238 2014-06-09 16:22:12.206078 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.448( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:12.206117 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.206122 7f868ff62700 5 queue_transactions existing osr(0.448 0x2bd7190)/0x2bd7190 2014-06-09 16:22:12.206124 7f868ff62700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206128 7f86a5e11700 5 _do_op 0xd3274a0 seq 4190 osr(1.375 0x384d230)/0x384d230 start 2014-06-09 16:22:12.206132 7f86a5e11700 10 _do_transaction on 0xc2ae6c0 2014-06-09 16:22:12.206136 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.206181 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.206194 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4190.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.206198 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.206208 7f86a5e11700 10 _do_op 0xd3274a0 seq 4190 r = 0, finisher 0x306280c0 0 2014-06-09 16:22:12.206217 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x320747c0] 2014-06-09 16:22:12.206219 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x320747c0] 2014-06-09 16:22:12.206221 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.206222 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.206224 7f86a5e11700 10 _finish_op 0xd3274a0 seq 4190 osr(1.375 0x384d230)/0x384d230 2014-06-09 16:22:12.206234 7f868f761700 10 op_submit_start 4239 2014-06-09 16:22:12.206235 7f868f761700 5 queue_transactions (trailing journal) 4239 2014-06-09 16:22:12.206236 7f868f761700 5 queue_op 0x324e7c80 seq 4239 osr(1.48e 0x384d9b0) 828 bytes (queue has 50 ops and 4860878 bytes) 2014-06-09 16:22:12.206241 7f868df5e700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206246 7f868bf5a700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206249 7f868c75b700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206253 7f8692f68700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206257 7f868b759700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206261 7f868e75f700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206264 7f868cf5c700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206267 7f868a757700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206270 7f868ef60700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206274 7f8690763700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206278 7f8690f64700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206281 7f8692767700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206284 7f8691f66700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206287 7f8691765700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206291 7f8689755700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206294 7f868d75d700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206297 7f8689f56700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206301 7f868ff62700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206304 7f868af58700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206311 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306280c0] 2014-06-09 16:22:12.206315 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306280c0] 2014-06-09 16:22:12.206316 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.206317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.206319 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e240 (1 active) 2014-06-09 16:22:12.206321 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300000 (2 active) 2014-06-09 16:22:12.206325 7f868f761700 10 op_submit_finish 4239 2014-06-09 16:22:12.206328 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.66e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.206357 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.206362 7f868f761700 5 queue_transactions existing osr(1.66e 0x384ded0)/0x384ded0 2014-06-09 16:22:12.206364 7f868f761700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.206367 7f86a5e11700 5 _do_op 0xd327ce0 seq 4191 osr(1.24c 0x384cfd0)/0x384cfd0 start 2014-06-09 16:22:12.206370 7f86a5e11700 10 _do_transaction on 0x3472240 2014-06-09 16:22:12.206375 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.206410 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.206422 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4191.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.206425 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.206435 7f86a5e11700 10 _do_op 0xd327ce0 seq 4191 r = 0, finisher 0xc169f40 0 2014-06-09 16:22:12.206443 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d16c0] 2014-06-09 16:22:12.206446 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d16c0] 2014-06-09 16:22:12.206447 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.206448 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.206450 7f86a5e11700 10 _finish_op 0xd327ce0 seq 4191 osr(1.24c 0x384cfd0)/0x384cfd0 2014-06-09 16:22:12.206458 7f868bf5a700 10 op_submit_start 4240 2014-06-09 16:22:12.206459 7f868bf5a700 5 queue_transactions (trailing journal) 4240 2014-06-09 16:22:12.206461 7f868bf5a700 5 queue_op 0x324e6120 seq 4240 osr(2.40f 0x2ed1210) 861 bytes (queue has 50 ops and 4860911 bytes) 2014-06-09 16:22:12.206465 7f868c75b700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206468 7f8692f68700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206471 7f868b759700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206474 7f868e75f700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206477 7f868cf5c700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206480 7f868a757700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206483 7f868ef60700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206487 7f8690763700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206490 7f8690f64700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206493 7f8692767700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206496 7f8691f66700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206499 7f8691765700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206501 7f8689755700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206505 7f868d75d700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206508 7f8689f56700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206511 7f868ff62700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206514 7f868af58700 2 waiting 51 > 50 ops || 4861772 > 104857600 2014-06-09 16:22:12.206517 7f868f761700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206520 7f868df5e700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206527 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc169f40] 2014-06-09 16:22:12.206531 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc169f40] 2014-06-09 16:22:12.206532 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.206533 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.206535 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300000 (1 active) 2014-06-09 16:22:12.206537 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4067e0 (2 active) 2014-06-09 16:22:12.206541 7f868bf5a700 10 op_submit_finish 4240 2014-06-09 16:22:12.206566 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41980 (19 active) 2014-06-09 16:22:12.206574 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.206578 7f868bf5a700 10 osd.26 244 dequeue_op 0x306b400 prio 63 cost 180224 latency 2.141140 osd_op(client.4571.0:35796 rbd_data.117b6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~180224] 3.d889555d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:12.206602 7f868bf5a700 5 -- op tracker -- , seq: 69, time: 2014-06-09 16:22:12.206597, event: reached_pg, op: osd_op(client.4571.0:35796 rbd_data.117b6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~180224] 3.d889555d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.206611 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.206625 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x306b400 2014-06-09 16:22:12.206634 7f868bf5a700 10 osd.26 244 dequeue_op 0x306b400 finish 2014-06-09 16:22:12.206637 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.206650 7f86a68f6700 5 -- op tracker -- , seq: 3682, time: 2014-06-09 16:22:12.202878, event: header_read, op: osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.206661 7f86a68f6700 5 -- op tracker -- , seq: 3682, time: 2014-06-09 16:22:12.202879, event: throttled, op: osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.206673 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) 2014-06-09 16:22:12.206670 7f86a68f6700 5 -- op tracker -- , seq: 3682, time: 2014-06-09 16:22:12.206613, event: all_read, op: osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.206681 7f86a68f6700 5 -- op tracker -- , seq: 3682, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.206677 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.206688 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.206692 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.206693 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x1804d700,0x1804d600,0x1804d500,0x1804d400,0x1804d300,0x4678c00,0x1804d200,0x4678b00,0x1804d100,0x4678a00,0x1c768e00,0x4678900,0x2c06bf00,0x2c06be00 2014-06-09 16:22:12.206710 7f86a68f6700 15 osd.26 244 enqueue_op 0x270bff00 prio 63 cost 2580480 latency 0.003832 osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.206745 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.206751 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.206768 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.206772 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.206785 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x16059b00,0x1804dd00,0x1804dc00,0x1c66a200,0x1c66a100,0x1c66a000,0x1b0d6200,0x1b0d6100,0x1b0d6000,0x22008600,0x1c66a800,0x22008500,0x1c66a700,0x22008400,0x1c66a600,0x22008300,0x22008200,0x22008100,0x22008000,0x270bf000,0x270bef00,0x270bee00,0x270bed00,0x270bec00,0x270beb00 2014-06-09 16:22:12.206856 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.206861 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.206893 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.206897 7f868bf5a700 5 queue_transactions existing osr(3.251 0x3ed0c20)/0x3ed0c20 2014-06-09 16:22:12.206899 7f868bf5a700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.206725 7f86a5e11700 5 _do_op 0x16097020 seq 4192 osr(3.362 0x3ed10a0)/0x3ed10a0 start 2014-06-09 16:22:12.206906 7f86a5e11700 10 _do_transaction on 0x97bb8c0 2014-06-09 16:22:12.206912 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.206953 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.206970 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4192.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.206974 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.206988 7f86a5e11700 10 _do_op 0x16097020 seq 4192 r = 0, finisher 0x30e3e980 0 2014-06-09 16:22:12.206998 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31791120] 2014-06-09 16:22:12.207001 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.207014 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.207024 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1010 2014-06-09 16:22:12.207034 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31791120] 2014-06-09 16:22:12.207036 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.207037 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.207041 7f86a5e11700 10 _finish_op 0x16097020 seq 4192 osr(3.362 0x3ed10a0)/0x3ed10a0 2014-06-09 16:22:12.207051 7f8692f68700 10 op_submit_start 4241 2014-06-09 16:22:12.207053 7f8692f68700 5 queue_transactions (trailing journal) 4241 2014-06-09 16:22:12.207054 7f8692f68700 5 queue_op 0x2fc8dc80 seq 4241 osr(3.111 0x3ed0830) 828 bytes (queue has 50 ops and 4860878 bytes) 2014-06-09 16:22:12.207059 7f868b759700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207063 7f868e75f700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207066 7f868cf5c700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207069 7f868a757700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207072 7f868ef60700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207075 7f8690763700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207078 7f8690f64700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207081 7f8692767700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207084 7f8691f66700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207087 7f8691765700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207090 7f8689755700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207093 7f868d75d700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207097 7f8689f56700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207100 7f868ff62700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207103 7f868af58700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207107 7f868f761700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207110 7f868df5e700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207113 7f868bf5a700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207116 7f868c75b700 2 waiting 51 > 50 ops || 4861739 > 104857600 2014-06-09 16:22:12.207123 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e3e980] 2014-06-09 16:22:12.207127 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e3e980] 2014-06-09 16:22:12.207128 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.207129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.207131 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4067e0 (1 active) 2014-06-09 16:22:12.207133 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359b00 (2 active) 2014-06-09 16:22:12.207137 7f8692f68700 10 op_submit_finish 4241 2014-06-09 16:22:12.207140 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 243/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 243/243 242/242/211) 2014-06-09 16:22:12.207171 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.207176 7f8692f68700 5 queue_transactions existing osr(3.5f8 0x3ed1b30)/0x3ed1b30 2014-06-09 16:22:12.207178 7f8692f68700 2 waiting 51 > 50 ops || 4861706 > 104857600 2014-06-09 16:22:12.207182 7f86a5e11700 5 _do_op 0x3280b1a0 seq 4193 osr(3.7f 0x3ed0200)/0x3ed0200 start 2014-06-09 16:22:12.207185 7f86a5e11700 10 _do_transaction on 0xc29b380 2014-06-09 16:22:12.207189 7f86a5e11700 15 _omap_setkeys meta/516fc4fc/pglog_3.7f/0//-1 2014-06-09 16:22:12.207232 7f86a5e11700 10 lookup_strip_header done cid meta oid 516fc4fc/pglog_3.7f/0//-1 2014-06-09 16:22:12.207247 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.207280 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.207295 7f86a5e11700 15 _omap_rmkeys meta/516fc4fc/pglog_3.7f/0//-1 2014-06-09 16:22:12.207299 7f86a5e11700 10 _omap_rmkeys meta/516fc4fc/pglog_3.7f/0//-1 r = 0 2014-06-09 16:22:12.207304 7f86a5e11700 15 _omap_setkeys meta/516fc4fc/pglog_3.7f/0//-1 2014-06-09 16:22:12.207375 7f86a5e11700 15 _touch 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.207402 7f86a68f6700 5 -- op tracker -- , seq: 3683, time: 2014-06-09 16:22:12.206738, event: header_read, op: osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.207414 7f86a68f6700 5 -- op tracker -- , seq: 3683, time: 2014-06-09 16:22:12.206740, event: throttled, op: osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.207419 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.7f_head oid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.207423 7f86a68f6700 5 -- op tracker -- , seq: 3683, time: 2014-06-09 16:22:12.207369, event: all_read, op: osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.207432 7f86a68f6700 5 -- op tracker -- , seq: 3683, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) 2014-06-09 16:22:12.207437 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 2014-06-09 16:22:12.207442 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 2014-06-09 16:22:12.207448 7f86a5e11700 20 write_state seq is 30169 2014-06-09 16:22:12.207456 7f86a5e11700 20 set_header setting 30168 cid 3.7f_head oid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 parent seq 0 2014-06-09 16:22:12.207454 7f86a68f6700 15 osd.26 244 enqueue_op 0x2f097b00 prio 63 cost 471040 latency 0.000716 osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 2014-06-09 16:22:12.207469 7f86a5e11700 10 _touch 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 = 0 2014-06-09 16:22:12.207476 7f86a5e11700 15 _write 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 0~237568 2014-06-09 16:22:12.207484 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.208074 7f86a5e11700 10 _generic_write 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 0~237568 = 0 2014-06-09 16:22:12.208089 7f86a5e11700 15 _setattrs 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.208095 7f86a5e11700 10 _setattrs 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 = 0 2014-06-09 16:22:12.208099 7f86a5e11700 15 _setattrs 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 2014-06-09 16:22:12.208103 7f86a5e11700 10 _setattrs 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 = 0 2014-06-09 16:22:12.208133 7f86a5e11700 10 cid: oid: 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 not skipping op, *spos 4193.0.0 2014-06-09 16:22:12.208134 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.208141 7f86a5e11700 20 set_header setting 30168 cid 3.7f_head oid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 parent seq 0 2014-06-09 16:22:12.208159 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4193.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.208162 7f86a5e11700 10 cid: oid: 516fc4fc/pglog_3.7f/0//-1 skipping op, spos 4193.0.0 <= header.spos 150197.0.0 2014-06-09 16:22:12.208164 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.208771 7f86a5e11700 10 _do_op 0x3280b1a0 seq 4193 r = 0, finisher 0x32480180 0x447c540 2014-06-09 16:22:12.208778 7f86a5e11700 10 _finish_op 0x3280b1a0 seq 4193 osr(3.7f 0x3ed0200)/0x3ed0200 2014-06-09 16:22:12.208782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c3be0] 2014-06-09 16:22:12.208790 7f868e75f700 10 op_submit_start 4242 2014-06-09 16:22:12.208791 7f868e75f700 5 queue_transactions (trailing journal) 4242 2014-06-09 16:22:12.208792 7f868e75f700 5 queue_op 0x324f5560 seq 4242 osr(1.32f 0x384d350) 828 bytes (queue has 50 ops and 4621750 bytes) 2014-06-09 16:22:12.208797 7f868cf5c700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208786 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 29 2014-06-09 16:22:12.208801 7f868a757700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208806 7f868ef60700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208810 7f8690763700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208813 7f8690f64700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208808 7f86a460e700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.208801, event: op_commit, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.208817 7f8692767700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208821 7f8691f66700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208825 7f8691765700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208818 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 29 all committed 2014-06-09 16:22:12.208829 7f8689755700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208832 7f868d75d700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208836 7f8689f56700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208840 7f868ff62700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208829 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] eval_repop repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=0 lock=0 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.208844 7f868af58700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208848 7f868f761700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208852 7f868df5e700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208856 7f868bf5a700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208846 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] log_op_stats osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 inb 237949 outb 0 rlat 0.000000 lat 1.274265 2014-06-09 16:22:12.208860 7f868c75b700 2 waiting 51 > 50 ops || 4622611 > 104857600 2014-06-09 16:22:12.208864 7f8692f68700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208867 7f868b759700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208865 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] publish_stats_to_osd 243:844 2014-06-09 16:22:12.208878 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32480180] 2014-06-09 16:22:12.208882 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359b00 (1 active) 2014-06-09 16:22:12.208884 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb7680 (2 active) 2014-06-09 16:22:12.208888 7f868e75f700 10 op_submit_finish 4242 2014-06-09 16:22:12.208875 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] sending commit on repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=0 lock=0 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) 0x32070500 2014-06-09 16:22:12.208892 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.31f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.208912 7f86a460e700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.208907, event: commit_sent, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.208924 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c3be0] 2014-06-09 16:22:12.208925 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.208926 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.208927 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.208931 7f868e75f700 5 queue_transactions existing osr(0.31f 0x2bd6d50)/0x2bd6d50 2014-06-09 16:22:12.208933 7f868e75f700 2 waiting 51 > 50 ops || 4622578 > 104857600 2014-06-09 16:22:12.208937 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] op_applied: 29 2014-06-09 16:22:12.208956 7f86a4e0f700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.208950, event: op_applied, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.208964 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] op_applied on primary on version 243'534 2014-06-09 16:22:12.208973 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] repop_all_applied: repop tid 29 all applied 2014-06-09 16:22:12.208982 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] eval_repop repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=1 lock=0 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.208997 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] log_op_stats osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 inb 237949 outb 0 rlat 0.000000 lat 1.274416 2014-06-09 16:22:12.209013 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] publish_stats_to_osd 243:845 2014-06-09 16:22:12.209025 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.209035 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 214'533 active] removing repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=1 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.209050 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 214'533 active] q front is repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=1 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.209065 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 214'533 active] remove_repop repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=1 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) 2014-06-09 16:22:12.209079 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 214'533 active] requeue_ops 2014-06-09 16:22:12.209110 7f86a4e0f700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.209104, event: done, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.209178 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32480180] 2014-06-09 16:22:12.209181 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.209182 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.209115 7f86a5e11700 5 _do_op 0xc34a840 seq 4194 osr(default 0x2c10280)/0x2c10280 start 2014-06-09 16:22:12.209189 7f86a5e11700 10 _do_transaction on 0xc2af140 2014-06-09 16:22:12.209193 7f86a5e11700 15 _write meta/c67cc332/inc_osdmap.244/0//-1 0~171 2014-06-09 16:22:12.209300 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid meta oid c67cc332/inc_osdmap.244/0//-1 2014-06-09 16:22:12.209335 7f86a5e11700 20 write_state seq is 30170 2014-06-09 16:22:12.209342 7f86a5e11700 20 set_header setting 30169 cid meta oid c67cc332/inc_osdmap.244/0//-1 parent seq 0 2014-06-09 16:22:12.209357 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.209367 7f86a5e11700 10 _generic_write meta/c67cc332/inc_osdmap.244/0//-1 0~171 = 0 2014-06-09 16:22:12.209375 7f86a5e11700 15 _write meta/ac9736c5/osdmap.244/0//-1 0~20781 2014-06-09 16:22:12.209419 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid meta oid ac9736c5/osdmap.244/0//-1 2014-06-09 16:22:12.209450 7f86a5e11700 20 write_state seq is 30171 2014-06-09 16:22:12.209457 7f86a5e11700 20 set_header setting 30170 cid meta oid ac9736c5/osdmap.244/0//-1 parent seq 0 2014-06-09 16:22:12.209471 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.209534 7f86a5e11700 10 _generic_write meta/ac9736c5/osdmap.244/0//-1 0~20781 = 0 2014-06-09 16:22:12.209541 7f86a5e11700 15 _write meta/23c2fcde/osd_superblock/0//-1 0~358 2014-06-09 16:22:12.209586 7f86a5e11700 10 lookup_strip_header done cid meta oid 23c2fcde/osd_superblock/0//-1 2014-06-09 16:22:12.209592 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.209644 7f86a5e11700 10 _generic_write meta/23c2fcde/osd_superblock/0//-1 0~358 = 0 2014-06-09 16:22:12.209649 7f86a5e11700 10 cid: oid: c67cc332/inc_osdmap.244/0//-1 not skipping op, *spos 4194.0.0 2014-06-09 16:22:12.209652 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.209655 7f86a5e11700 20 set_header setting 30169 cid meta oid c67cc332/inc_osdmap.244/0//-1 parent seq 0 2014-06-09 16:22:12.209669 7f86a5e11700 10 cid: oid: ac9736c5/osdmap.244/0//-1 not skipping op, *spos 4194.0.0 2014-06-09 16:22:12.209672 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.209676 7f86a5e11700 20 set_header setting 30170 cid meta oid ac9736c5/osdmap.244/0//-1 parent seq 0 2014-06-09 16:22:12.209688 7f86a5e11700 10 cid: oid: 23c2fcde/osd_superblock/0//-1 skipping op, spos 4194.0.0 <= header.spos 150250.0.0 2014-06-09 16:22:12.209692 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.209793 7f86a5e11700 10 _do_op 0xc34a840 seq 4194 r = 0, finisher 0x33630390 0xbd7ccc0 2014-06-09 16:22:12.209797 7f86a5e11700 10 _finish_op 0xc34a840 seq 4194 osr(default 0x2c10280)/0x2c10280 2014-06-09 16:22:12.209810 7f868a757700 10 op_submit_start 4243 2014-06-09 16:22:12.209812 7f868a757700 5 queue_transactions (trailing journal) 4243 2014-06-09 16:22:12.209813 7f868a757700 5 queue_op 0x32539f80 seq 4243 osr(0.4ba 0x2bd7710) 828 bytes (queue has 50 ops and 4600954 bytes) 2014-06-09 16:22:12.209818 7f868ef60700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209824 7f8690763700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209828 7f8690f64700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209833 7f8692767700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209837 7f8691f66700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209842 7f8691765700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209846 7f8689755700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209850 7f868d75d700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209855 7f8689f56700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209859 7f868ff62700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209863 7f868af58700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209868 7f868f761700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209872 7f868df5e700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209877 7f868bf5a700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209881 7f868c75b700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209885 7f8692f68700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209890 7f868b759700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209894 7f868e75f700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209898 7f868cf5c700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.209908 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33630390] 2014-06-09 16:22:12.209928 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33630390] 2014-06-09 16:22:12.209930 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.209931 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.209934 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb7680 (1 active) 2014-06-09 16:22:12.209937 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca490e0 (2 active) 2014-06-09 16:22:12.209942 7f868a757700 10 op_submit_finish 4243 2014-06-09 16:22:12.209946 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 243/243 242/242/200) 2014-06-09 16:22:12.209982 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.209987 7f868a757700 5 queue_transactions existing osr(3.52a 0x3ed1670)/0x3ed1670 2014-06-09 16:22:12.209990 7f868a757700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.209994 7f86a5e11700 5 _do_op 0x30060cc0 seq 4195 osr(0.5d4 0x2bd7a10)/0x2bd7a10 start 2014-06-09 16:22:12.209998 7f86a5e11700 10 _do_transaction on 0x2d82480 2014-06-09 16:22:12.210004 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.210049 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.210066 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4195.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.210070 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.210087 7f86a5e11700 10 _do_op 0x30060cc0 seq 4195 r = 0, finisher 0xc410880 0 2014-06-09 16:22:12.210094 7f86a5e11700 10 _finish_op 0x30060cc0 seq 4195 osr(0.5d4 0x2bd7a10)/0x2bd7a10 2014-06-09 16:22:12.210096 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3e4e0] 2014-06-09 16:22:12.210104 7f8690763700 10 op_submit_start 4244 2014-06-09 16:22:12.210106 7f8690763700 5 queue_transactions (trailing journal) 4244 2014-06-09 16:22:12.210107 7f8690763700 5 queue_op 0xbe7de60 seq 4244 osr(3.661 0x3ed19f0) 861 bytes (queue has 50 ops and 4600954 bytes) 2014-06-09 16:22:12.210099 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.210112 7f8690f64700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210116 7f8692767700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210110 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.210119 7f8691f66700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210123 7f8691765700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210119 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 2014-06-09 16:22:12.210127 7f8689755700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210128 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3e4e0] 2014-06-09 16:22:12.210130 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.210130 7f868d75d700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210131 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.210134 7f8689f56700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210138 7f868ff62700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210142 7f868af58700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210146 7f868f761700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210151 7f868df5e700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210155 7f868bf5a700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210159 7f868c75b700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210163 7f8692f68700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210167 7f868b759700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210171 7f868e75f700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210175 7f868cf5c700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210180 7f868a757700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210184 7f868ef60700 2 waiting 51 > 50 ops || 4601782 > 104857600 2014-06-09 16:22:12.210194 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc410880] 2014-06-09 16:22:12.210198 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc410880] 2014-06-09 16:22:12.210200 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.210201 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.210204 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca490e0 (1 active) 2014-06-09 16:22:12.210207 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e120 (2 active) 2014-06-09 16:22:12.210212 7f8690763700 10 op_submit_finish 4244 2014-06-09 16:22:12.210216 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.210231 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.000369 7 0.000370 2014-06-09 16:22:12.210244 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.210255 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.210267 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.210278 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.210286 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.210294 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.210306 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.210317 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.210327 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.210343 7f8690763700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.210361 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 2014-06-09 16:22:12.210395 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.210399 7f8690763700 5 queue_transactions existing osr(2.5bf 0x2ed1b10)/0x2ed1b10 2014-06-09 16:22:12.210402 7f8690763700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210407 7f86a5e11700 5 _do_op 0x324a5da0 seq 4196 osr(3.375 0x3ed1060)/0x3ed1060 start 2014-06-09 16:22:12.210410 7f86a5e11700 10 _do_transaction on 0xc291980 2014-06-09 16:22:12.210416 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.210458 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.210473 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4196.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.210477 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.210489 7f86a5e11700 10 _do_op 0x324a5da0 seq 4196 r = 0, finisher 0x30509280 0 2014-06-09 16:22:12.210495 7f86a5e11700 10 _finish_op 0x324a5da0 seq 4196 osr(3.375 0x3ed1060)/0x3ed1060 2014-06-09 16:22:12.210497 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc82f80] 2014-06-09 16:22:12.210500 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc82f80] 2014-06-09 16:22:12.210502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.210503 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.210506 7f8692767700 10 op_submit_start 4245 2014-06-09 16:22:12.210507 7f8692767700 5 queue_transactions (trailing journal) 4245 2014-06-09 16:22:12.210508 7f8692767700 5 queue_op 0xc2eaf00 seq 4245 osr(2.3de 0x2ed1330) 861 bytes (queue has 50 ops and 4600987 bytes) 2014-06-09 16:22:12.210513 7f8691f66700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210518 7f8691765700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210523 7f8689755700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210527 7f868d75d700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210532 7f8689f56700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210536 7f868ff62700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210541 7f868af58700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210546 7f868f761700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210550 7f868df5e700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210554 7f868bf5a700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210558 7f868c75b700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210562 7f8692f68700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210566 7f868b759700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210571 7f868e75f700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210575 7f868cf5c700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210579 7f868a757700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210583 7f868ef60700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210587 7f8690763700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210591 7f8690f64700 2 waiting 51 > 50 ops || 4601815 > 104857600 2014-06-09 16:22:12.210601 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30509280] 2014-06-09 16:22:12.210605 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30509280] 2014-06-09 16:22:12.210607 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.210608 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.210611 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e120 (1 active) 2014-06-09 16:22:12.210614 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496ea0 (2 active) 2014-06-09 16:22:12.210619 7f8692767700 10 op_submit_finish 4245 2014-06-09 16:22:12.210623 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.210637 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.069326 7 0.000328 2014-06-09 16:22:12.210649 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.210659 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.210669 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.210680 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.210688 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.210696 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.210707 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.210717 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.210727 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.210739 7f8692767700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 244 2014-06-09 16:22:12.210756 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 2014-06-09 16:22:12.210789 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.210794 7f8692767700 5 queue_transactions existing osr(2.6f4 0x2ed1e20)/0x2ed1e20 2014-06-09 16:22:12.210796 7f8692767700 2 waiting 51 > 50 ops || 4601848 > 104857600 2014-06-09 16:22:12.210801 7f86a5e11700 5 _do_op 0xc64a480 seq 4197 osr(0.dc 0x2bd6570)/0x2bd6570 start 2014-06-09 16:22:12.210804 7f86a5e11700 10 _do_transaction on 0xc1fde00 2014-06-09 16:22:12.210811 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.210854 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.210870 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4197.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.210875 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.210886 7f86a5e11700 10 _do_op 0xc64a480 seq 4197 r = 0, finisher 0x306cf980 0 2014-06-09 16:22:12.210893 7f86a5e11700 10 _finish_op 0xc64a480 seq 4197 osr(0.dc 0x2bd6570)/0x2bd6570 2014-06-09 16:22:12.210895 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30629a60] 2014-06-09 16:22:12.210903 7f8691765700 10 op_submit_start 4246 2014-06-09 16:22:12.210904 7f8691765700 5 queue_transactions (trailing journal) 4246 2014-06-09 16:22:12.210906 7f8691765700 5 queue_op 0xc64b3e0 seq 4246 osr(1.65d 0x384df50) 861 bytes (queue has 50 ops and 4600990 bytes) 2014-06-09 16:22:12.210899 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.210910 7f8689755700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210914 7f868d75d700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.210908 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.210917 7f8689f56700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.210921 7f868ff62700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210918 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:154 2014-06-09 16:22:12.210925 7f868af58700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.210927 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30629a60] 2014-06-09 16:22:12.210929 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.210928 7f868f761700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210929 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.210932 7f868df5e700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210936 7f868bf5a700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210940 7f868c75b700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.210944 7f8692f68700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210949 7f868b759700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210953 7f868e75f700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210957 7f868cf5c700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.210962 7f868a757700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210966 7f868ef60700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210970 7f8690763700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.210974 7f8690f64700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.210978 7f8692767700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.210983 7f8691f66700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.210992 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306cf980] 2014-06-09 16:22:12.210997 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306cf980] 2014-06-09 16:22:12.210999 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.211000 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.211003 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496ea0 (1 active) 2014-06-09 16:22:12.211006 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301e60 (2 active) 2014-06-09 16:22:12.211011 7f8691765700 10 op_submit_finish 4246 2014-06-09 16:22:12.211015 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.211030 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.211039 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.211047 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.211056 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.211064 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.211078 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:193 2014-06-09 16:22:12.211087 7f8691765700 20 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.211096 7f8691765700 20 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.211105 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.211110 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.211122 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.211130 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.211139 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.211202 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.211210 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.211224 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.211234 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.211242 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.211251 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.211255 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.211269 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.211278 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16bd6b00,0x16bd6a00,0x16bd6900,0x13022e00,0x13022d00,0x13022c00,0x13022b00,0x16bd7900,0x16bd7800,0x2c06bd00,0x2e075500,0x2e075400,0x2e075300,0x2e075200,0x2e075100 2014-06-09 16:22:12.211326 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.211337 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.211346 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.211359 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1163 2014-06-09 16:22:12.211370 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.211379 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.211389 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.211394 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.55b( v 214'716 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) 2014-06-09 16:22:12.211410 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.4 activated and committed 2014-06-09 16:22:12.211423 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.211433 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.211448 7f8691765700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 244 2014-06-09 16:22:12.211465 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1128 2014-06-09 16:22:12.211483 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.211507 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x1fc880c0 (19 active) 2014-06-09 16:22:12.211514 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.211520 7f8691765700 10 osd.26 244 dequeue_op 0xe032100 prio 63 cost 57344 latency 2.251505 osd_op(client.4572.0:29986 rbd_data.117c6b8b4567.000000000000140e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.d8b29cb9 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] 2014-06-09 16:22:12.211545 7f8691765700 5 -- op tracker -- , seq: 12, time: 2014-06-09 16:22:12.211540, event: reached_pg, op: osd_op(client.4572.0:29986 rbd_data.117c6b8b4567.000000000000140e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.d8b29cb9 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.211554 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.211569 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0xe032100 2014-06-09 16:22:12.211581 7f8691765700 10 osd.26 244 dequeue_op 0xe032100 finish 2014-06-09 16:22:12.211585 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.211622 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x1fc880c0 (20 active) 2014-06-09 16:22:12.211626 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.211645 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.211650 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.211663 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x13022600,0x13022500,0x16bd6600,0x16bd6500 2014-06-09 16:22:12.211686 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.211690 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.54a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.211726 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.211731 7f8691765700 5 queue_transactions existing osr(1.54a 0x384daf0)/0x384daf0 2014-06-09 16:22:12.211733 7f8691765700 2 waiting 51 > 50 ops || 4601818 > 104857600 2014-06-09 16:22:12.211738 7f86a5e11700 5 _do_op 0x30a38c0 seq 4198 osr(1.206 0x384cd10)/0x384cd10 start 2014-06-09 16:22:12.211743 7f86a5e11700 10 _do_transaction on 0xbe892c0 2014-06-09 16:22:12.211748 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.211792 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.211807 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4198.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.211811 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.211823 7f86a5e11700 10 _do_op 0x30a38c0 seq 4198 r = 0, finisher 0xbebfe60 0 2014-06-09 16:22:12.211830 7f86a5e11700 10 _finish_op 0x30a38c0 seq 4198 osr(1.206 0x384cd10)/0x384cd10 2014-06-09 16:22:12.211832 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbebf9c0] 2014-06-09 16:22:12.211835 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbebf9c0] 2014-06-09 16:22:12.211837 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.211838 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.211841 7f868d75d700 10 op_submit_start 4247 2014-06-09 16:22:12.211842 7f868d75d700 5 queue_transactions (trailing journal) 4247 2014-06-09 16:22:12.211843 7f868d75d700 5 queue_op 0xc2d5020 seq 4247 osr(0.6b3 0x2bd7ff0) 861 bytes (queue has 50 ops and 4601023 bytes) 2014-06-09 16:22:12.211848 7f8689f56700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.211853 7f868ff62700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211857 7f868af58700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.211861 7f868f761700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211865 7f868df5e700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211870 7f868bf5a700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211874 7f868c75b700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.211878 7f8692f68700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211882 7f868b759700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211887 7f868e75f700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211891 7f868cf5c700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.211895 7f868a757700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211900 7f868ef60700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211904 7f8690763700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.211908 7f8690f64700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211913 7f8692767700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.211917 7f8691f66700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.211921 7f8691765700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211925 7f8689755700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.211935 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbebfe60] 2014-06-09 16:22:12.211939 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbebfe60] 2014-06-09 16:22:12.211941 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.211942 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.211945 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301e60 (1 active) 2014-06-09 16:22:12.211948 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcd80 (2 active) 2014-06-09 16:22:12.211953 7f868d75d700 10 op_submit_finish 4247 2014-06-09 16:22:12.211957 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.211973 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.126881 8 0.008879 2014-06-09 16:22:12.211985 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.211997 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.212008 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.212019 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.212029 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.212038 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.212049 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.212061 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.212073 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.212089 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.212105 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1141 2014-06-09 16:22:12.212139 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.212143 7f868d75d700 5 queue_transactions existing osr(3.22d 0x3ed0c40)/0x3ed0c40 2014-06-09 16:22:12.212145 7f868d75d700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212150 7f86a5e11700 5 _do_op 0xbdc82a0 seq 4199 osr(3.18e 0x3ed0af0)/0x3ed0af0 start 2014-06-09 16:22:12.212154 7f86a5e11700 10 _do_transaction on 0x39db740 2014-06-09 16:22:12.212159 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.212201 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.212217 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4199.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.212221 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.212249 7f86a5e11700 10 _do_op 0xbdc82a0 seq 4199 r = 0, finisher 0x3205d4a0 0 2014-06-09 16:22:12.212256 7f86a5e11700 10 _finish_op 0xbdc82a0 seq 4199 osr(3.18e 0x3ed0af0)/0x3ed0af0 2014-06-09 16:22:12.212258 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205d560] 2014-06-09 16:22:12.212261 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205d560] 2014-06-09 16:22:12.212262 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.212263 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.212266 7f868ff62700 10 op_submit_start 4248 2014-06-09 16:22:12.212268 7f868ff62700 5 queue_transactions (trailing journal) 4248 2014-06-09 16:22:12.212269 7f868ff62700 5 queue_op 0xd326120 seq 4248 osr(0.448 0x2bd7190) 828 bytes (queue has 50 ops and 4601023 bytes) 2014-06-09 16:22:12.212274 7f868af58700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212279 7f868f761700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212283 7f868df5e700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212288 7f868bf5a700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212292 7f868c75b700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212297 7f8692f68700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212301 7f868b759700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212305 7f868e75f700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212310 7f868cf5c700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212314 7f868a757700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212318 7f868ef60700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212322 7f8690763700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212327 7f8690f64700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212331 7f8692767700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212335 7f8691f66700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212339 7f8691765700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212344 7f8689755700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212348 7f868d75d700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212352 7f8689f56700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212362 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205d4a0] 2014-06-09 16:22:12.212368 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205d4a0] 2014-06-09 16:22:12.212370 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.212371 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.212374 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcd80 (1 active) 2014-06-09 16:22:12.212377 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524120 (2 active) 2014-06-09 16:22:12.212382 7f868ff62700 10 op_submit_finish 4248 2014-06-09 16:22:12.212386 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.212421 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.212426 7f868ff62700 5 queue_transactions existing osr(3.5ce 0x3ed17c0)/0x3ed17c0 2014-06-09 16:22:12.212429 7f868ff62700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212433 7f86a5e11700 5 _do_op 0x324db380 seq 4200 osr(0.3fe 0x2bd7210)/0x2bd7210 start 2014-06-09 16:22:12.212437 7f86a5e11700 10 _do_transaction on 0xc05f500 2014-06-09 16:22:12.212442 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.212484 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.212499 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4200.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.212504 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.212515 7f86a5e11700 10 _do_op 0x324db380 seq 4200 r = 0, finisher 0x3050a0a0 0 2014-06-09 16:22:12.212521 7f86a5e11700 10 _finish_op 0x324db380 seq 4200 osr(0.3fe 0x2bd7210)/0x2bd7210 2014-06-09 16:22:12.212523 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdba1e0] 2014-06-09 16:22:12.212526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdba1e0] 2014-06-09 16:22:12.212527 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.212528 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.212532 7f868f761700 10 op_submit_start 4249 2014-06-09 16:22:12.212533 7f868f761700 5 queue_transactions (trailing journal) 4249 2014-06-09 16:22:12.212535 7f868f761700 5 queue_op 0x324e7bc0 seq 4249 osr(1.66e 0x384ded0) 828 bytes (queue has 50 ops and 4601023 bytes) 2014-06-09 16:22:12.212540 7f868df5e700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212545 7f868bf5a700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212549 7f868c75b700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212554 7f8692f68700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212558 7f868b759700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212562 7f868e75f700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212566 7f868cf5c700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212571 7f868a757700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212575 7f868ef60700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212579 7f8690763700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212583 7f8690f64700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212588 7f8692767700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212591 7f8691f66700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212596 7f8691765700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212600 7f8689755700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212604 7f868d75d700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212608 7f8689f56700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212612 7f868ff62700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.212616 7f868af58700 2 waiting 51 > 50 ops || 4601884 > 104857600 2014-06-09 16:22:12.212626 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a0a0] 2014-06-09 16:22:12.212629 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a0a0] 2014-06-09 16:22:12.212631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.212632 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.212635 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524120 (1 active) 2014-06-09 16:22:12.212638 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b0e0 (2 active) 2014-06-09 16:22:12.212643 7f868f761700 10 op_submit_finish 4249 2014-06-09 16:22:12.212647 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.212667 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.212673 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.212686 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 2014-06-09 16:22:12.212699 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.212703 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.212720 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.212724 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.212737 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x1804cf00,0x257c5d00,0x1804ce00,0x1804cd00,0x1804cc00,0x1804cb00,0x1c768a00,0x1c768900,0x257c5c00,0x257c5b00,0x257c5a00,0x257c5900,0x257c5800 2014-06-09 16:22:12.212781 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.212786 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.212800 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x22009f00,0x22009d00,0x1804c800,0x1804c900,0x14052d00,0xdfa7d00,0x32826800,0x32826700 2014-06-09 16:22:12.212832 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.212836 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.212849 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.212857 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.212866 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.212874 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.212882 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.212895 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.212904 7f868f761700 20 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.212912 7f868f761700 20 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.212921 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.212925 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.212938 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.212947 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.212952 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.212965 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.212976 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.212980 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.212992 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.213001 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.213005 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.213018 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:12.213028 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.213032 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 243/210 242/242/209) 2014-06-09 16:22:12.213122 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.213128 7f868f761700 5 queue_transactions existing osr(3.2df 0x3ed0e40)/0x3ed0e40 2014-06-09 16:22:12.213130 7f868f761700 2 waiting 51 > 50 ops || 4601851 > 104857600 2014-06-09 16:22:12.213135 7f86a5e11700 5 _do_op 0xd326ae0 seq 4201 osr(1.a 0x384c270)/0x384c270 start 2014-06-09 16:22:12.213139 7f86a5e11700 10 _do_transaction on 0x36b2900 2014-06-09 16:22:12.213145 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.213187 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.213202 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4201.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.213206 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.213219 7f86a5e11700 10 _do_op 0xd326ae0 seq 4201 r = 0, finisher 0xc1681e0 0 2014-06-09 16:22:12.213226 7f86a5e11700 10 _finish_op 0xd326ae0 seq 4201 osr(1.a 0x384c270)/0x384c270 2014-06-09 16:22:12.213228 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061fa40] 2014-06-09 16:22:12.213231 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061fa40] 2014-06-09 16:22:12.213232 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.213233 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.213237 7f868bf5a700 10 op_submit_start 4250 2014-06-09 16:22:12.213238 7f868bf5a700 5 queue_transactions (trailing journal) 4250 2014-06-09 16:22:12.213240 7f868bf5a700 5 queue_op 0x324dbb00 seq 4250 osr(3.251 0x3ed0c20) 828 bytes (queue has 50 ops and 4601027 bytes) 2014-06-09 16:22:12.213245 7f868c75b700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213250 7f8692f68700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213254 7f868b759700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213259 7f868e75f700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213263 7f868cf5c700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213267 7f868a757700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213271 7f868ef60700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213276 7f8690763700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213280 7f8690f64700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213284 7f8692767700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213289 7f8691f66700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213293 7f8691765700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213297 7f8689755700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213302 7f868d75d700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213306 7f8689f56700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213310 7f868ff62700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213314 7f868af58700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213318 7f868f761700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213322 7f868df5e700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213332 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc1681e0] 2014-06-09 16:22:12.213336 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc1681e0] 2014-06-09 16:22:12.213338 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.213339 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.213342 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b0e0 (1 active) 2014-06-09 16:22:12.213344 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0c60 (2 active) 2014-06-09 16:22:12.213350 7f868bf5a700 10 op_submit_finish 4250 2014-06-09 16:22:12.213354 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.4bd( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/211) 2014-06-09 16:22:12.213387 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.213392 7f868bf5a700 5 queue_transactions existing osr(0.4bd 0x2bd76f0)/0x2bd76f0 2014-06-09 16:22:12.213395 7f868bf5a700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213400 7f86a5e11700 5 _do_op 0xc34aba0 seq 4202 osr(0.3b7 0x2bd7290)/0x2bd7290 start 2014-06-09 16:22:12.213403 7f86a5e11700 10 _do_transaction on 0xc2c4000 2014-06-09 16:22:12.213409 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.213450 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.213464 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4202.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.213469 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.213479 7f86a5e11700 10 _do_op 0xc34aba0 seq 4202 r = 0, finisher 0x1322dd80 0 2014-06-09 16:22:12.213485 7f86a5e11700 10 _finish_op 0xc34aba0 seq 4202 osr(0.3b7 0x2bd7290)/0x2bd7290 2014-06-09 16:22:12.213487 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x182cae80] 2014-06-09 16:22:12.213490 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x182cae80] 2014-06-09 16:22:12.213491 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.213492 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.213496 7f8692f68700 10 op_submit_start 4251 2014-06-09 16:22:12.213498 7f8692f68700 5 queue_transactions (trailing journal) 4251 2014-06-09 16:22:12.213499 7f8692f68700 5 queue_op 0x2fc8c120 seq 4251 osr(3.5f8 0x3ed1b30) 828 bytes (queue has 50 ops and 4601027 bytes) 2014-06-09 16:22:12.213504 7f868b759700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213509 7f868e75f700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213513 7f868cf5c700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213517 7f868a757700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213522 7f868ef60700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213526 7f8690763700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213530 7f8690f64700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213535 7f8692767700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213539 7f8691f66700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213543 7f8691765700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213547 7f8689755700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213551 7f868d75d700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213556 7f8689f56700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213560 7f868ff62700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213564 7f868af58700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213568 7f868f761700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213572 7f868df5e700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213576 7f868bf5a700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.213580 7f868c75b700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.213590 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322dd80] 2014-06-09 16:22:12.213594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322dd80] 2014-06-09 16:22:12.213596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.213597 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.213600 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0c60 (1 active) 2014-06-09 16:22:12.213603 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6360 (2 active) 2014-06-09 16:22:12.213608 7f8692f68700 10 op_submit_finish 4251 2014-06-09 16:22:12.213624 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (19 active) 2014-06-09 16:22:12.213631 7f8692f68700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.213635 7f8692f68700 10 osd.26 244 dequeue_op 0x270be300 prio 63 cost 450560 latency 1.399389 osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.213660 7f8692f68700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.213655, event: reached_pg, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.213668 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.213681 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x270be300 2014-06-09 16:22:12.213692 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.213717 7f8692f68700 15 getattr 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 '_' 2014-06-09 16:22:12.213791 7f8692f68700 10 getattr got.size() is 0 2014-06-09 16:22:12.213794 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 but can_create 2014-06-09 16:22:12.213809 7f8692f68700 15 getattr 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 'snapset' 2014-06-09 16:22:12.213841 7f8692f68700 10 getattr got.size() is 0 2014-06-09 16:22:12.213844 7f8692f68700 15 getattr 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/snapdir//3 'snapset' 2014-06-09 16:22:12.213874 7f8692f68700 10 getattr got.size() is 0 2014-06-09 16:22:12.213883 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30e06580 e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.213896 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.213906 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.213917 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30e06580 e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 rwstate(none n=0 w=0) oi: e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc2700 snapset: 0=[]:[] 2014-06-09 16:22:12.213933 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 @head oi=e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.213952 7f8692f68700 15 getattr 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/snapdir//3 '_' 2014-06-09 16:22:12.213985 7f8692f68700 10 getattr got.size() is 0 2014-06-09 16:22:12.213987 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/snapdir//3 and !can_create 2014-06-09 16:22:12.214008 7f8692f68700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.214002, event: started, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.214015 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30a9aa00 2014-06-09 16:22:12.214033 7f8692f68700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.214028, event: started, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.214040 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] ov 0'0 av 243'519 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.214054 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 2014-06-09 16:22:12.214065 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.214080 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~450560 2014-06-09 16:22:12.214097 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229600 2014-06-09 16:22:12.214108 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 snapset=0x14dc2738 snapc=0=[] 2014-06-09 16:22:12.214120 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.214130 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.214142 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 0x30a9aa00 op modify 2014-06-09 16:22:12.214155 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:43.783744 2014-06-09 16:22:12.214175 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.214195 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.214206 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 32 on osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.214222 7f8692f68700 7 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 32 o e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.214246 7f8692f68700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.214240, event: waiting for subops from 8, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.214278 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.214298 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'518], crt=0'0) [243'519 (0'0) modify e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 by client.4573.0:34234 2014-06-09 10:20:43.783744] 2014-06-09 16:22:12.214319 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'519 (0'0) modify e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 by client.4573.0:34234 2014-06-09 10:20:43.783744 2014-06-09 16:22:12.214339 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.214370 7f8692f68700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'519, trimmed: 2014-06-09 16:22:12.214389 7f8692f68700 5 queue_transactions existing osr(3.750 0x3ed1eb0)/0x3ed1eb0 2014-06-09 16:22:12.214392 7f8692f68700 2 waiting 51 > 50 ops || 5053985 > 104857600 2014-06-09 16:22:12.214397 7f86a5e11700 5 _do_op 0xd326f00 seq 4203 osr(3.470 0x3ed1540)/0x3ed1540 start 2014-06-09 16:22:12.214401 7f86a5e11700 10 _do_transaction on 0xbf36cc0 2014-06-09 16:22:12.214407 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.214450 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.214466 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4203.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.214470 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.214490 7f86a5e11700 10 _do_op 0xd326f00 seq 4203 r = 0, finisher 0x139656e0 0 2014-06-09 16:22:12.214497 7f86a5e11700 10 _finish_op 0xd326f00 seq 4203 osr(3.470 0x3ed1540)/0x3ed1540 2014-06-09 16:22:12.214499 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3e840] 2014-06-09 16:22:12.214502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3e840] 2014-06-09 16:22:12.214504 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.214505 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.214508 7f868e75f700 10 op_submit_start 4252 2014-06-09 16:22:12.214510 7f868e75f700 5 queue_transactions (trailing journal) 4252 2014-06-09 16:22:12.214511 7f868e75f700 5 queue_op 0x324f53e0 seq 4252 osr(0.31f 0x2bd6d50) 828 bytes (queue has 50 ops and 4601027 bytes) 2014-06-09 16:22:12.214517 7f868cf5c700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.214521 7f868a757700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214526 7f868ef60700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214531 7f8690763700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.214535 7f8690f64700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214540 7f8692767700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.214544 7f8691f66700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.214548 7f8691765700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214553 7f8689755700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214558 7f868d75d700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.214562 7f8689f56700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.214566 7f868ff62700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214570 7f868af58700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.214575 7f868f761700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214579 7f868df5e700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214583 7f868bf5a700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214587 7f868c75b700 2 waiting 51 > 50 ops || 4601888 > 104857600 2014-06-09 16:22:12.214592 7f8692f68700 2 waiting 51 > 50 ops || 5053985 > 104857600 2014-06-09 16:22:12.214596 7f868b759700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214606 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x139656e0] 2014-06-09 16:22:12.214611 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x139656e0] 2014-06-09 16:22:12.214612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.214613 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.214616 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6360 (1 active) 2014-06-09 16:22:12.214619 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0d80 (2 active) 2014-06-09 16:22:12.214624 7f868e75f700 10 op_submit_finish 4252 2014-06-09 16:22:12.214628 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.13c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) 2014-06-09 16:22:12.214664 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.214669 7f868e75f700 5 queue_transactions existing osr(2.13c 0x2ed0610)/0x2ed0610 2014-06-09 16:22:12.214671 7f868e75f700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214676 7f86a5e11700 5 _do_op 0xc2d4b40 seq 4204 osr(0.164 0x2bd6810)/0x2bd6810 start 2014-06-09 16:22:12.214679 7f86a5e11700 10 _do_transaction on 0xbe8ab40 2014-06-09 16:22:12.214686 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.214728 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.214746 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4204.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.214750 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.214762 7f86a5e11700 10 _do_op 0xc2d4b40 seq 4204 r = 0, finisher 0xccf9ac0 0 2014-06-09 16:22:12.214769 7f86a5e11700 10 _finish_op 0xc2d4b40 seq 4204 osr(0.164 0x2bd6810)/0x2bd6810 2014-06-09 16:22:12.214770 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6e0c0] 2014-06-09 16:22:12.214779 7f868a757700 10 op_submit_start 4253 2014-06-09 16:22:12.214780 7f868a757700 5 queue_transactions (trailing journal) 4253 2014-06-09 16:22:12.214782 7f868a757700 5 queue_op 0x32539f20 seq 4253 osr(3.52a 0x3ed1670) 828 bytes (queue has 50 ops and 4600994 bytes) 2014-06-09 16:22:12.214774 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.214786 7f868ef60700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214790 7f8690763700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214786 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.214794 7f8690f64700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214798 7f8692767700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214801 7f8691f66700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214796 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:186 2014-06-09 16:22:12.214805 7f8691765700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214805 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6e0c0] 2014-06-09 16:22:12.214807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.214808 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.214809 7f8689755700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214813 7f868d75d700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214817 7f8689f56700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214822 7f868ff62700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214826 7f868af58700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214830 7f868f761700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214834 7f868df5e700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214839 7f868bf5a700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214843 7f868c75b700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214847 7f8692f68700 2 waiting 51 > 50 ops || 5053952 > 104857600 2014-06-09 16:22:12.214851 7f868b759700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214856 7f868e75f700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214860 7f868cf5c700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.214870 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf9ac0] 2014-06-09 16:22:12.214875 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf9ac0] 2014-06-09 16:22:12.214877 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.214878 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.214881 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0d80 (1 active) 2014-06-09 16:22:12.214884 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338fc0 (2 active) 2014-06-09 16:22:12.214889 7f868a757700 10 op_submit_finish 4253 2014-06-09 16:22:12.214893 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/207) 2014-06-09 16:22:12.214929 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.214934 7f868a757700 5 queue_transactions existing osr(3.470 0x3ed1540)/0x3ed1540 2014-06-09 16:22:12.214936 7f868a757700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.214941 7f86a5e11700 5 _do_op 0x30607980 seq 4205 osr(0.319 0x2bd6d90)/0x2bd6d90 start 2014-06-09 16:22:12.214945 7f86a5e11700 10 _do_transaction on 0x2c19ec0 2014-06-09 16:22:12.214951 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.215052 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.215071 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4205.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.215075 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.215089 7f86a5e11700 10 _do_op 0x30607980 seq 4205 r = 0, finisher 0xc2d7b20 0 2014-06-09 16:22:12.215096 7f86a5e11700 10 _finish_op 0x30607980 seq 4205 osr(0.319 0x2bd6d90)/0x2bd6d90 2014-06-09 16:22:12.215097 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd72d80] 2014-06-09 16:22:12.215106 7f8690763700 10 op_submit_start 4254 2014-06-09 16:22:12.215107 7f8690763700 5 queue_transactions (trailing journal) 4254 2014-06-09 16:22:12.215100 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.215109 7f8690763700 5 queue_op 0x30060900 seq 4254 osr(2.5bf 0x2ed1b10) 861 bytes (queue has 50 ops and 4600994 bytes) 2014-06-09 16:22:12.215113 7f8690f64700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215110 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.215117 7f8692767700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.215121 7f8691f66700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.215124 7f8691765700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215119 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 2014-06-09 16:22:12.215128 7f8689755700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215128 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd72d80] 2014-06-09 16:22:12.215130 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.215131 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.215131 7f868d75d700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.215135 7f8689f56700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.215139 7f868ff62700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215143 7f868af58700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.215147 7f868f761700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215152 7f868df5e700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215156 7f868bf5a700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215160 7f868c75b700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.215164 7f8692f68700 2 waiting 51 > 50 ops || 5053952 > 104857600 2014-06-09 16:22:12.215168 7f868b759700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215173 7f868e75f700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215177 7f868cf5c700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.215181 7f868a757700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215185 7f868ef60700 2 waiting 51 > 50 ops || 4601822 > 104857600 2014-06-09 16:22:12.215195 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d7b20] 2014-06-09 16:22:12.215200 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d7b20] 2014-06-09 16:22:12.215201 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.215202 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.215206 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338fc0 (1 active) 2014-06-09 16:22:12.216763 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa6c0 (2 active) 2014-06-09 16:22:12.216773 7f8690763700 10 op_submit_finish 4254 2014-06-09 16:22:12.216777 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.216794 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.216804 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x26aa9200,0x26aa9100,0x26aa9000,0x26aa8f00,0x26aa8e00 2014-06-09 16:22:12.216830 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.216841 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.216851 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.216867 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:967 2014-06-09 16:22:12.216878 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.216887 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.216959 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.216966 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.216979 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.035029 7 0.000341 2014-06-09 16:22:12.216992 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.217002 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.217013 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.217024 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.217033 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.217041 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.217051 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.217062 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.217072 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.217088 7f8690763700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.217105 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 2014-06-09 16:22:12.217140 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.217144 7f8690763700 5 queue_transactions existing osr(2.185 0x2ed09a0)/0x2ed09a0 2014-06-09 16:22:12.217147 7f8690763700 2 waiting 51 > 50 ops || 4601855 > 104857600 2014-06-09 16:22:12.217173 7f86a5e11700 5 _do_op 0x1e08a2a0 seq 4206 osr(1.b3 0x384c6d0)/0x384c6d0 start 2014-06-09 16:22:12.217178 7f86a5e11700 10 _do_transaction on 0xc245b00 2014-06-09 16:22:12.217185 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.217238 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.217256 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4206.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.217260 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.217276 7f86a5e11700 10 _do_op 0x1e08a2a0 seq 4206 r = 0, finisher 0xc2d7a40 0 2014-06-09 16:22:12.217283 7f86a5e11700 10 _finish_op 0x1e08a2a0 seq 4206 osr(1.b3 0x384c6d0)/0x384c6d0 2014-06-09 16:22:12.217286 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd620] 2014-06-09 16:22:12.217295 7f8692767700 10 op_submit_start 4255 2014-06-09 16:22:12.217296 7f8692767700 5 queue_transactions (trailing journal) 4255 2014-06-09 16:22:12.217289 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.217297 7f8692767700 5 queue_op 0xc2eb980 seq 4255 osr(2.6f4 0x2ed1e20) 861 bytes (queue has 50 ops and 4600997 bytes) 2014-06-09 16:22:12.217302 7f8691f66700 2 waiting 51 > 50 ops || 4601858 > 104857600 2014-06-09 16:22:12.217299 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.217306 7f8691765700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217311 7f8689755700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217315 7f868d75d700 2 waiting 51 > 50 ops || 4601858 > 104857600 2014-06-09 16:22:12.217309 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:172 2014-06-09 16:22:12.217318 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd620] 2014-06-09 16:22:12.217320 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.217320 7f8689f56700 2 waiting 51 > 50 ops || 4601858 > 104857600 2014-06-09 16:22:12.217321 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.217324 7f868ff62700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217328 7f868af58700 2 waiting 51 > 50 ops || 4601858 > 104857600 2014-06-09 16:22:12.217333 7f868f761700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217337 7f868df5e700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217342 7f868bf5a700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217347 7f868c75b700 2 waiting 51 > 50 ops || 4601858 > 104857600 2014-06-09 16:22:12.217351 7f8692f68700 2 waiting 51 > 50 ops || 5053955 > 104857600 2014-06-09 16:22:12.217356 7f868b759700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217361 7f868e75f700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217366 7f868cf5c700 2 waiting 51 > 50 ops || 4601858 > 104857600 2014-06-09 16:22:12.217370 7f868a757700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217375 7f868ef60700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217379 7f8690763700 2 waiting 51 > 50 ops || 4601858 > 104857600 2014-06-09 16:22:12.217383 7f8690f64700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217394 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d7a40] 2014-06-09 16:22:12.217398 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d7a40] 2014-06-09 16:22:12.217400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.217401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.217404 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa6c0 (1 active) 2014-06-09 16:22:12.217407 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b8c0 (2 active) 2014-06-09 16:22:12.217412 7f8692767700 10 op_submit_finish 4255 2014-06-09 16:22:12.217416 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.217430 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.217439 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.217448 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.217457 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.217464 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.217478 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:156 2014-06-09 16:22:12.217487 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.217495 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.217504 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.217509 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.217523 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.217532 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1b0b6700,0x1b0b6600,0x1b0b6500,0x1b0b6400,0x1b0b6300,0x1b0b6200 2014-06-09 16:22:12.217558 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:12.217568 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.217585 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1060 2014-06-09 16:22:12.217596 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.217605 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.217615 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.217619 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.217631 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.217639 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.217647 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.217656 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.217664 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.217676 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:152 2014-06-09 16:22:12.217686 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.217694 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.217703 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.217708 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.179( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.217741 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.217745 7f8692767700 5 queue_transactions existing osr(0.179 0x2bd6800)/0x2bd6800 2014-06-09 16:22:12.217748 7f8692767700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217753 7f86a5e11700 5 _do_op 0xbe7daa0 seq 4207 osr(3.78a 0x3ed1d80)/0x3ed1d80 start 2014-06-09 16:22:12.217756 7f86a5e11700 10 _do_transaction on 0xc160f00 2014-06-09 16:22:12.217763 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.217806 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.217823 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4207.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.217828 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.217862 7f86a5e11700 10 _do_op 0xbe7daa0 seq 4207 r = 0, finisher 0x31412300 0 2014-06-09 16:22:12.217869 7f86a5e11700 10 _finish_op 0xbe7daa0 seq 4207 osr(3.78a 0x3ed1d80)/0x3ed1d80 2014-06-09 16:22:12.217871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189ab360] 2014-06-09 16:22:12.217879 7f8691765700 10 op_submit_start 4256 2014-06-09 16:22:12.217881 7f8691765700 5 queue_transactions (trailing journal) 4256 2014-06-09 16:22:12.217882 7f8691765700 5 queue_op 0xc88b680 seq 4256 osr(1.54a 0x384daf0) 828 bytes (queue has 50 ops and 4600964 bytes) 2014-06-09 16:22:12.217874 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] _finish_recovery 2014-06-09 16:22:12.217886 7f8689755700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217890 7f868d75d700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217886 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] purge_strays 2014-06-09 16:22:12.217894 7f8689f56700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217898 7f868ff62700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217901 7f868af58700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217896 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] publish_stats_to_osd 243:876 2014-06-09 16:22:12.217905 7f868f761700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217906 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189ab360] 2014-06-09 16:22:12.217908 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.217908 7f868df5e700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.217912 7f868bf5a700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217916 7f868c75b700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217920 7f8692f68700 2 waiting 51 > 50 ops || 5053922 > 104857600 2014-06-09 16:22:12.217924 7f868b759700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217929 7f868e75f700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217933 7f868cf5c700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217937 7f868a757700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217942 7f868ef60700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217946 7f8690763700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217950 7f8690f64700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217954 7f8692767700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.217958 7f8691f66700 2 waiting 51 > 50 ops || 4601825 > 104857600 2014-06-09 16:22:12.217969 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412300] 2014-06-09 16:22:12.217974 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412300] 2014-06-09 16:22:12.217976 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.217977 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.217980 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b8c0 (1 active) 2014-06-09 16:22:12.217984 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d07e0 (2 active) 2014-06-09 16:22:12.217989 7f8691765700 10 op_submit_finish 4256 2014-06-09 16:22:12.217993 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.54a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.218030 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.218035 7f8691765700 5 queue_transactions existing osr(1.54a 0x384daf0)/0x384daf0 2014-06-09 16:22:12.218037 7f8691765700 2 waiting 51 > 50 ops || 4601792 > 104857600 2014-06-09 16:22:12.218042 7f86a5e11700 5 _do_op 0x30a24e0 seq 4208 osr(3.2ac 0x3ed0b50)/0x3ed0b50 start 2014-06-09 16:22:12.218045 7f86a5e11700 10 _do_transaction on 0xc1252c0 2014-06-09 16:22:12.218050 7f86a5e11700 15 _omap_setkeys meta/1fe0d7a6/pglog_3.2ac/0//-1 2014-06-09 16:22:12.218099 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe0d7a6/pglog_3.2ac/0//-1 2014-06-09 16:22:12.218115 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.218155 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.218170 7f86a5e11700 15 _omap_rmkeys meta/1fe0d7a6/pglog_3.2ac/0//-1 2014-06-09 16:22:12.218175 7f86a5e11700 10 _omap_rmkeys meta/1fe0d7a6/pglog_3.2ac/0//-1 r = 0 2014-06-09 16:22:12.218182 7f86a5e11700 15 _omap_setkeys meta/1fe0d7a6/pglog_3.2ac/0//-1 2014-06-09 16:22:12.218194 7f86a5e11700 15 _touch 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.218240 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.2ac_head oid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.218272 7f86a5e11700 20 write_state seq is 30172 2014-06-09 16:22:12.218279 7f86a5e11700 20 set_header setting 30171 cid 3.2ac_head oid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 parent seq 0 2014-06-09 16:22:12.218293 7f86a5e11700 10 _touch 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 = 0 2014-06-09 16:22:12.218300 7f86a5e11700 15 _write 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 0~16384 2014-06-09 16:22:12.218305 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.218366 7f86a5e11700 10 _generic_write 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 0~16384 = 0 2014-06-09 16:22:12.218376 7f86a5e11700 15 _setattrs 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.218386 7f86a5e11700 10 _setattrs 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 = 0 2014-06-09 16:22:12.218391 7f86a5e11700 15 _setattrs 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 2014-06-09 16:22:12.218400 7f86a5e11700 10 _setattrs 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 = 0 2014-06-09 16:22:12.218406 7f86a5e11700 10 cid: oid: 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 not skipping op, *spos 4208.0.0 2014-06-09 16:22:12.218409 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.218414 7f86a5e11700 20 set_header setting 30171 cid 3.2ac_head oid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 parent seq 0 2014-06-09 16:22:12.218427 7f86a5e11700 10 cid: oid: 1fe0d7a6/pglog_3.2ac/0//-1 skipping op, spos 4208.0.0 <= header.spos 149708.0.0 2014-06-09 16:22:12.218431 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4208.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.218434 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.218533 7f86a5e11700 10 _do_op 0x30a24e0 seq 4208 r = 0, finisher 0x15a3ed20 0x86d9ec0 2014-06-09 16:22:12.218541 7f86a5e11700 10 _finish_op 0x30a24e0 seq 4208 osr(3.2ac 0x3ed0b50)/0x3ed0b50 2014-06-09 16:22:12.218543 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94480] 2014-06-09 16:22:12.218552 7f868d75d700 10 op_submit_start 4257 2014-06-09 16:22:12.218553 7f868d75d700 5 queue_transactions (trailing journal) 4257 2014-06-09 16:22:12.218554 7f868d75d700 5 queue_op 0x14dd7140 seq 4257 osr(3.22d 0x3ed0c40) 861 bytes (queue has 50 ops and 4583043 bytes) 2014-06-09 16:22:12.218546 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 30 2014-06-09 16:22:12.218559 7f8689f56700 2 waiting 51 > 50 ops || 4583904 > 104857600 2014-06-09 16:22:12.218563 7f868ff62700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218566 7f868af58700 2 waiting 51 > 50 ops || 4583904 > 104857600 2014-06-09 16:22:12.218570 7f868f761700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218568 7f86a460e700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.218560, event: op_commit, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.218573 7f868df5e700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218576 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94480] 2014-06-09 16:22:12.218577 7f868bf5a700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.218579 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.218581 7f868c75b700 2 waiting 51 > 50 ops || 4583904 > 104857600 2014-06-09 16:22:12.218585 7f8692f68700 2 waiting 51 > 50 ops || 5036001 > 104857600 2014-06-09 16:22:12.218589 7f868b759700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218594 7f868e75f700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218598 7f868cf5c700 2 waiting 51 > 50 ops || 4583904 > 104857600 2014-06-09 16:22:12.218602 7f868a757700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218607 7f868ef60700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218611 7f8690763700 2 waiting 51 > 50 ops || 4583904 > 104857600 2014-06-09 16:22:12.218615 7f8690f64700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218619 7f8692767700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218623 7f8691f66700 2 waiting 51 > 50 ops || 4583904 > 104857600 2014-06-09 16:22:12.218628 7f8691765700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218632 7f8689755700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.218644 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x15a3ed20] 2014-06-09 16:22:12.218647 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 30 2014-06-09 16:22:12.218667 7f86a4e0f700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.218661, event: op_applied, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.218675 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'664 2014-06-09 16:22:12.218702 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x15a3ed20] 2014-06-09 16:22:12.218704 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.218705 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.218709 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d07e0 (1 active) 2014-06-09 16:22:12.218712 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e360 (2 active) 2014-06-09 16:22:12.218717 7f868d75d700 10 op_submit_finish 4257 2014-06-09 16:22:12.218722 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.218735 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.182743 7 0.000343 2014-06-09 16:22:12.218746 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.218757 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.218767 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.218777 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.218786 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.218793 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.218804 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.218873 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.218883 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.218897 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 2014-06-09 16:22:12.218915 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 2014-06-09 16:22:12.218949 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.218953 7f868d75d700 5 queue_transactions existing osr(1.301 0x384d030)/0x384d030 2014-06-09 16:22:12.218956 7f868d75d700 2 waiting 51 > 50 ops || 4583904 > 104857600 2014-06-09 16:22:12.218961 7f86a5e11700 5 _do_op 0x324e7f80 seq 4209 osr(1.ab 0x384c6f0)/0x384c6f0 start 2014-06-09 16:22:12.218964 7f86a5e11700 10 _do_transaction on 0x2e63500 2014-06-09 16:22:12.218969 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.219013 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.219028 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4209.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.219032 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.219049 7f86a5e11700 10 _do_op 0x324e7f80 seq 4209 r = 0, finisher 0x3205d740 0 2014-06-09 16:22:12.219057 7f86a5e11700 10 _finish_op 0x324e7f80 seq 4209 osr(1.ab 0x384c6f0)/0x384c6f0 2014-06-09 16:22:12.219058 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205d060] 2014-06-09 16:22:12.219061 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205d060] 2014-06-09 16:22:12.219063 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.219064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.219067 7f868ff62700 10 op_submit_start 4258 2014-06-09 16:22:12.219069 7f868ff62700 5 queue_transactions (trailing journal) 4258 2014-06-09 16:22:12.219070 7f868ff62700 5 queue_op 0xcd90060 seq 4258 osr(3.5ce 0x3ed17c0) 828 bytes (queue has 50 ops and 4583045 bytes) 2014-06-09 16:22:12.219075 7f868af58700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219080 7f868f761700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219085 7f868df5e700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219089 7f868bf5a700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219094 7f868c75b700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219098 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.219103 7f868b759700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219107 7f868e75f700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219111 7f868cf5c700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219115 7f868a757700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219120 7f868ef60700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219124 7f8690763700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219128 7f8690f64700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219133 7f8692767700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219137 7f8691f66700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219141 7f8691765700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219145 7f8689755700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219149 7f868d75d700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219153 7f8689f56700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219163 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205d740] 2014-06-09 16:22:12.219167 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205d740] 2014-06-09 16:22:12.219168 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.219169 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.219172 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e360 (1 active) 2014-06-09 16:22:12.219174 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525320 (2 active) 2014-06-09 16:22:12.219178 7f868ff62700 10 op_submit_finish 4258 2014-06-09 16:22:12.219183 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.504( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.219223 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.219228 7f868ff62700 5 queue_transactions existing osr(1.504 0x384dbd0)/0x384dbd0 2014-06-09 16:22:12.219230 7f868ff62700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219235 7f86a5e11700 5 _do_op 0x230b9560 seq 4210 osr(3.527 0x3ed1660)/0x3ed1660 start 2014-06-09 16:22:12.219238 7f86a5e11700 10 _do_transaction on 0x3144cc0 2014-06-09 16:22:12.219244 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.219286 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.219301 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4210.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.219305 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.219316 7f86a5e11700 10 _do_op 0x230b9560 seq 4210 r = 0, finisher 0x3062af60 0 2014-06-09 16:22:12.219323 7f86a5e11700 10 _finish_op 0x230b9560 seq 4210 osr(3.527 0x3ed1660)/0x3ed1660 2014-06-09 16:22:12.219324 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078780] 2014-06-09 16:22:12.219332 7f868f761700 10 op_submit_start 4259 2014-06-09 16:22:12.219331 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078780] 2014-06-09 16:22:12.219333 7f868f761700 5 queue_transactions (trailing journal) 4259 2014-06-09 16:22:12.219334 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.219335 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.219334 7f868f761700 5 queue_op 0x16096540 seq 4259 osr(3.2df 0x3ed0e40) 828 bytes (queue has 50 ops and 4583045 bytes) 2014-06-09 16:22:12.219339 7f868df5e700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219343 7f868bf5a700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219348 7f868c75b700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219352 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.219356 7f868b759700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219361 7f868e75f700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219365 7f868cf5c700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219369 7f868a757700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219373 7f868ef60700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219378 7f8690763700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219382 7f8690f64700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219386 7f8692767700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219390 7f8691f66700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219395 7f8691765700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219399 7f8689755700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219403 7f868d75d700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219407 7f8689f56700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219412 7f868ff62700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219416 7f868af58700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062af60] 2014-06-09 16:22:12.219432 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062af60] 2014-06-09 16:22:12.219436 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.219438 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.219441 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525320 (1 active) 2014-06-09 16:22:12.219444 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3386c0 (2 active) 2014-06-09 16:22:12.219449 7f868f761700 10 op_submit_finish 4259 2014-06-09 16:22:12.219453 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.219471 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x270bf400,0x1c66b100,0x270bf300,0x1c66b000,0x270bf200,0x1c66af00,0x270bf100,0x1c66ae00,0x1c66ad00,0x1c66ac00,0x1c66ab00,0x1b0d7800,0x1b0b6800 2014-06-09 16:22:12.219518 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.219524 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.219536 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.219545 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.219569 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x153599f0 (19 active) 2014-06-09 16:22:12.219610 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153599f0 (20 active) 2014-06-09 16:22:12.219635 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 243/243 242/242/222) 2014-06-09 16:22:12.219673 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.219678 7f868f761700 5 queue_transactions existing osr(3.111 0x3ed0830)/0x3ed0830 2014-06-09 16:22:12.219681 7f868f761700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219686 7f86a5e11700 5 _do_op 0xd327aa0 seq 4211 osr(2.24d 0x2ed0a50)/0x2ed0a50 start 2014-06-09 16:22:12.219690 7f86a5e11700 10 _do_transaction on 0xbf0fd40 2014-06-09 16:22:12.219695 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.219738 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.219752 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4211.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.219757 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.219769 7f86a5e11700 10 _do_op 0xd327aa0 seq 4211 r = 0, finisher 0xbd72720 0 2014-06-09 16:22:12.219776 7f86a5e11700 10 _finish_op 0xd327aa0 seq 4211 osr(2.24d 0x2ed0a50)/0x2ed0a50 2014-06-09 16:22:12.219777 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a72c0] 2014-06-09 16:22:12.219780 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a72c0] 2014-06-09 16:22:12.219782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.219783 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.219786 7f868bf5a700 10 op_submit_start 4260 2014-06-09 16:22:12.219787 7f868bf5a700 5 queue_transactions (trailing journal) 4260 2014-06-09 16:22:12.219789 7f868bf5a700 5 queue_op 0x324e7620 seq 4260 osr(0.4bd 0x2bd76f0) 828 bytes (queue has 50 ops and 4583045 bytes) 2014-06-09 16:22:12.219794 7f868c75b700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219798 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.219803 7f868b759700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219807 7f868e75f700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219812 7f868cf5c700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219816 7f868a757700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219820 7f868ef60700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219825 7f8690763700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219829 7f8690f64700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219833 7f8692767700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219837 7f8691f66700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219841 7f8691765700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219846 7f8689755700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219850 7f868d75d700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219854 7f8689f56700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219858 7f868ff62700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219862 7f868af58700 2 waiting 51 > 50 ops || 4583906 > 104857600 2014-06-09 16:22:12.219867 7f868f761700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219871 7f868df5e700 2 waiting 51 > 50 ops || 4583873 > 104857600 2014-06-09 16:22:12.219881 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd72720] 2014-06-09 16:22:12.219886 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd72720] 2014-06-09 16:22:12.219887 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.219888 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.219892 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3386c0 (1 active) 2014-06-09 16:22:12.219894 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6240 (2 active) 2014-06-09 16:22:12.219899 7f868bf5a700 10 op_submit_finish 4260 2014-06-09 16:22:12.219903 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:12.219939 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.219944 7f868bf5a700 5 queue_transactions existing osr(3.9f 0x3ed0600)/0x3ed0600 2014-06-09 16:22:12.219946 7f868bf5a700 2 waiting 51 > 50 ops || 4583871 > 104857600 2014-06-09 16:22:12.219951 7f86a5e11700 5 _do_op 0x324f49c0 seq 4212 osr(0.473 0x2bd74d0)/0x2bd74d0 start 2014-06-09 16:22:12.219954 7f86a5e11700 10 _do_transaction on 0xc2becc0 2014-06-09 16:22:12.219960 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.220001 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.220015 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4212.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.220020 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.220032 7f86a5e11700 10 _do_op 0x324f49c0 seq 4212 r = 0, finisher 0x3042b6a0 0 2014-06-09 16:22:12.220038 7f86a5e11700 10 _finish_op 0x324f49c0 seq 4212 osr(0.473 0x2bd74d0)/0x2bd74d0 2014-06-09 16:22:12.220040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc9940] 2014-06-09 16:22:12.220042 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc9940] 2014-06-09 16:22:12.220044 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.220045 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.220048 7f8692f68700 10 op_submit_start 4261 2014-06-09 16:22:12.220050 7f8692f68700 5 queue_transactions (trailing journal) 4261 2014-06-09 16:22:12.220051 7f8692f68700 5 queue_op 0x2fc8d800 seq 4261 osr(3.750 0x3ed1eb0) 452958 bytes (queue has 50 ops and 5035175 bytes) 2014-06-09 16:22:12.220056 7f868b759700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220061 7f868e75f700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220065 7f868cf5c700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220069 7f868a757700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220074 7f868ef60700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220078 7f8690763700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220082 7f8690f64700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220087 7f8692767700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220091 7f8691f66700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220095 7f8691765700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220100 7f8689755700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220104 7f868d75d700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220108 7f8689f56700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220113 7f868ff62700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220117 7f868af58700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220121 7f868f761700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220126 7f868df5e700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220130 7f868bf5a700 2 waiting 51 > 50 ops || 5036001 > 104857600 2014-06-09 16:22:12.220135 7f868c75b700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220145 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b6a0] 2014-06-09 16:22:12.220151 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b6a0] 2014-06-09 16:22:12.220153 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.220155 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.220158 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6240 (1 active) 2014-06-09 16:22:12.220161 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac45a0 (2 active) 2014-06-09 16:22:12.220166 7f8692f68700 10 op_submit_finish 4261 2014-06-09 16:22:12.220171 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xbffe840 243'519 rep_tid=32 committed?=0 applied?=0 lock=0 op=osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.220198 7f8692f68700 10 osd.26 244 dequeue_op 0x270be300 finish 2014-06-09 16:22:12.220202 7f8692f68700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.220209 7f8692f68700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.220214 7f8692f68700 10 osd.26 244 dequeue_op 0x22009f00 prio 63 cost 1097728 latency 1.540870 osd_op(client.4574.0:32457 rbd_data.117e6b8b4567.0000000000001536 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1097728] 3.4db0ef4f RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:12.220237 7f8692f68700 5 -- op tracker -- , seq: 601, time: 2014-06-09 16:22:12.220231, event: reached_pg, op: osd_op(client.4574.0:32457 rbd_data.117e6b8b4567.0000000000001536 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1097728] 3.4db0ef4f RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.220267 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.220279 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x22009f00 2014-06-09 16:22:12.220287 7f8692f68700 10 osd.26 244 dequeue_op 0x22009f00 finish 2014-06-09 16:22:12.220291 7f8692f68700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.220319 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153589a0 (20 active) 2014-06-09 16:22:12.220323 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.1da( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.220360 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.220365 7f8692f68700 5 queue_transactions existing osr(0.1da 0x2bd6920)/0x2bd6920 2014-06-09 16:22:12.220367 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220372 7f86a5e11700 5 _do_op 0xcabde00 seq 4213 osr(0.4ba 0x2bd7710)/0x2bd7710 start 2014-06-09 16:22:12.220375 7f86a5e11700 10 _do_transaction on 0xc2af200 2014-06-09 16:22:12.220381 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.220424 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.220439 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4213.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.220443 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.220470 7f86a5e11700 10 _do_op 0xcabde00 seq 4213 r = 0, finisher 0x3042a1e0 0 2014-06-09 16:22:12.220477 7f86a5e11700 10 _finish_op 0xcabde00 seq 4213 osr(0.4ba 0x2bd7710)/0x2bd7710 2014-06-09 16:22:12.220479 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x327f38c0] 2014-06-09 16:22:12.220482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x327f38c0] 2014-06-09 16:22:12.220483 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.220484 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.220488 7f868e75f700 10 op_submit_start 4262 2014-06-09 16:22:12.220490 7f868e75f700 5 queue_transactions (trailing journal) 4262 2014-06-09 16:22:12.220491 7f868e75f700 5 queue_op 0x324f5260 seq 4262 osr(2.13c 0x2ed0610) 828 bytes (queue has 50 ops and 5035175 bytes) 2014-06-09 16:22:12.220497 7f868cf5c700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220502 7f868a757700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220507 7f868ef60700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220511 7f8690763700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220516 7f8690f64700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220520 7f8692767700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220525 7f8691f66700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220529 7f8691765700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220534 7f8689755700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220538 7f868d75d700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220543 7f8689f56700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220547 7f868ff62700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220552 7f868af58700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220556 7f868f761700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220560 7f868df5e700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220564 7f868bf5a700 2 waiting 51 > 50 ops || 5036001 > 104857600 2014-06-09 16:22:12.220569 7f868c75b700 2 waiting 51 > 50 ops || 5036036 > 104857600 2014-06-09 16:22:12.220573 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220577 7f868b759700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220586 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042a1e0] 2014-06-09 16:22:12.220591 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042a1e0] 2014-06-09 16:22:12.220593 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.220594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.220597 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac45a0 (1 active) 2014-06-09 16:22:12.220600 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89ec60 (2 active) 2014-06-09 16:22:12.220605 7f868e75f700 10 op_submit_finish 4262 2014-06-09 16:22:12.220609 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.640( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.220644 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.220649 7f868e75f700 5 queue_transactions existing osr(0.640 0x2bd7c10)/0x2bd7c10 2014-06-09 16:22:12.220651 7f868e75f700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220656 7f86a5e11700 5 _do_op 0xc2d4d80 seq 4214 osr(2.754 0x3ed00f0)/0x3ed00f0 start 2014-06-09 16:22:12.220659 7f86a5e11700 10 _do_transaction on 0x2fb6900 2014-06-09 16:22:12.220665 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.220707 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.220790 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4214.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.220795 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.220812 7f86a5e11700 10 _do_op 0xc2d4d80 seq 4214 r = 0, finisher 0xcc941e0 0 2014-06-09 16:22:12.220818 7f86a5e11700 10 _finish_op 0xc2d4d80 seq 4214 osr(2.754 0x3ed00f0)/0x3ed00f0 2014-06-09 16:22:12.220820 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2f40] 2014-06-09 16:22:12.220829 7f868a757700 10 op_submit_start 4263 2014-06-09 16:22:12.220830 7f868a757700 5 queue_transactions (trailing journal) 4263 2014-06-09 16:22:12.220831 7f868a757700 5 queue_op 0x32539ec0 seq 4263 osr(3.470 0x3ed1540) 828 bytes (queue has 50 ops and 5035142 bytes) 2014-06-09 16:22:12.220824 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.220836 7f868ef60700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220840 7f8690763700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220835 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.220844 7f8690f64700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220847 7f8692767700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220851 7f8691f66700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220845 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.220854 7f8691765700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220855 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2f40] 2014-06-09 16:22:12.220856 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.220857 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.220858 7f8689755700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220862 7f868d75d700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220866 7f8689f56700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220871 7f868ff62700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220875 7f868af58700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220879 7f868f761700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220884 7f868df5e700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220888 7f868bf5a700 2 waiting 51 > 50 ops || 5035968 > 104857600 2014-06-09 16:22:12.220892 7f868c75b700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220897 7f8692f68700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220901 7f868b759700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220906 7f868e75f700 2 waiting 51 > 50 ops || 5035970 > 104857600 2014-06-09 16:22:12.220910 7f868cf5c700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.220920 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc941e0] 2014-06-09 16:22:12.220924 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc941e0] 2014-06-09 16:22:12.220926 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.220927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.220930 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89ec60 (1 active) 2014-06-09 16:22:12.220933 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0480 (2 active) 2014-06-09 16:22:12.220938 7f868a757700 10 op_submit_finish 4263 2014-06-09 16:22:12.220943 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.220959 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.136007 7 0.007363 2014-06-09 16:22:12.220973 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.220986 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.220998 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000025 0 0.000000 2014-06-09 16:22:12.221009 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.221019 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.221028 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.221040 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.221053 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.221064 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.221081 7f868a757700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 2014-06-09 16:22:12.221099 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1012 2014-06-09 16:22:12.221132 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.221136 7f868a757700 5 queue_transactions existing osr(3.672 0x3ed1350)/0x3ed1350 2014-06-09 16:22:12.221138 7f868a757700 2 waiting 51 > 50 ops || 5036003 > 104857600 2014-06-09 16:22:12.221143 7f86a5e11700 5 _do_op 0x2fcafc20 seq 4215 osr(0.6e 0x2bd6390)/0x2bd6390 start 2014-06-09 16:22:12.221147 7f86a5e11700 10 _do_transaction on 0xc3f0300 2014-06-09 16:22:12.221154 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.221197 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.221213 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4215.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.221218 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.221232 7f86a5e11700 10 _do_op 0x2fcafc20 seq 4215 r = 0, finisher 0x17000080 0 2014-06-09 16:22:12.221239 7f86a5e11700 10 _finish_op 0x2fcafc20 seq 4215 osr(0.6e 0x2bd6390)/0x2bd6390 2014-06-09 16:22:12.221240 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5c860] 2014-06-09 16:22:12.221249 7f8690763700 10 op_submit_start 4264 2014-06-09 16:22:12.221250 7f8690763700 5 queue_transactions (trailing journal) 4264 2014-06-09 16:22:12.221243 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.221252 7f8690763700 5 queue_op 0x30d969c0 seq 4264 osr(2.185 0x2ed09a0) 861 bytes (queue has 50 ops and 5035145 bytes) 2014-06-09 16:22:12.221256 7f8690f64700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221253 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.221260 7f8692767700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221264 7f8691f66700 2 waiting 51 > 50 ops || 5036006 > 104857600 2014-06-09 16:22:12.221268 7f8691765700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221262 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.221271 7f8689755700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221272 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5c860] 2014-06-09 16:22:12.221273 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.221275 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.221275 7f868d75d700 2 waiting 51 > 50 ops || 5036006 > 104857600 2014-06-09 16:22:12.221279 7f8689f56700 2 waiting 51 > 50 ops || 5036006 > 104857600 2014-06-09 16:22:12.221283 7f868ff62700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221287 7f868af58700 2 waiting 51 > 50 ops || 5036006 > 104857600 2014-06-09 16:22:12.221291 7f868f761700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221295 7f868df5e700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221300 7f868bf5a700 2 waiting 51 > 50 ops || 5035971 > 104857600 2014-06-09 16:22:12.221304 7f868c75b700 2 waiting 51 > 50 ops || 5036006 > 104857600 2014-06-09 16:22:12.221308 7f8692f68700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221312 7f868b759700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221316 7f868e75f700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221320 7f868cf5c700 2 waiting 51 > 50 ops || 5036006 > 104857600 2014-06-09 16:22:12.221324 7f868a757700 2 waiting 51 > 50 ops || 5036006 > 104857600 2014-06-09 16:22:12.221329 7f868ef60700 2 waiting 51 > 50 ops || 5035973 > 104857600 2014-06-09 16:22:12.221339 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17000080] 2014-06-09 16:22:12.221348 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17000080] 2014-06-09 16:22:12.221350 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.221351 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.221354 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0480 (1 active) 2014-06-09 16:22:12.221357 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114240 (2 active) 2014-06-09 16:22:12.221362 7f8690763700 10 op_submit_finish 4264 2014-06-09 16:22:12.221366 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.221380 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.069641 7 0.000337 2014-06-09 16:22:12.221392 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.221402 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.221413 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.221423 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.221431 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.221439 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.221450 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.221461 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.221471 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.221484 7f8690763700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 2014-06-09 16:22:12.221499 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.221532 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.221537 7f8690763700 5 queue_transactions existing osr(2.102 0x2ed0710)/0x2ed0710 2014-06-09 16:22:12.221539 7f8690763700 2 waiting 51 > 50 ops || 5036006 > 104857600 2014-06-09 16:22:12.221544 7f86a5e11700 5 _do_op 0xd4715c0 seq 4216 osr(1.b8 0x384c650)/0x384c650 start 2014-06-09 16:22:12.221548 7f86a5e11700 10 _do_transaction on 0x344d500 2014-06-09 16:22:12.221554 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.221597 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.221614 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4216.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.221619 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.221634 7f86a5e11700 10 _do_op 0xd4715c0 seq 4216 r = 0, finisher 0x30508dc0 0 2014-06-09 16:22:12.221641 7f86a5e11700 10 _finish_op 0xd4715c0 seq 4216 osr(1.b8 0x384c650)/0x384c650 2014-06-09 16:22:12.221643 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324a8f60] 2014-06-09 16:22:12.221651 7f8692767700 10 op_submit_start 4265 2014-06-09 16:22:12.221653 7f8692767700 5 queue_transactions (trailing journal) 4265 2014-06-09 16:22:12.221646 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.221654 7f8692767700 5 queue_op 0xbf7f860 seq 4265 osr(0.179 0x2bd6800) 828 bytes (queue has 50 ops and 5035115 bytes) 2014-06-09 16:22:12.221658 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.221656 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.221662 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221666 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221670 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.221666 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 2014-06-09 16:22:12.221673 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.221674 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324a8f60] 2014-06-09 16:22:12.221676 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.221677 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.221677 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221681 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.221685 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221689 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221693 7f868bf5a700 2 waiting 51 > 50 ops || 5035941 > 104857600 2014-06-09 16:22:12.221698 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.221702 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221706 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221710 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221715 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.221719 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.221724 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221728 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.221733 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221743 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30508dc0] 2014-06-09 16:22:12.221748 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30508dc0] 2014-06-09 16:22:12.221750 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.221751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.221754 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114240 (1 active) 2014-06-09 16:22:12.221757 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497c20 (2 active) 2014-06-09 16:22:12.221762 7f8692767700 10 op_submit_finish 4265 2014-06-09 16:22:12.221766 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.221783 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.221793 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1c66b300,0x257c5f00,0x257c5e00,0x270bfe00,0x270bfd00,0x270bfc00,0x270bfb00,0x270bfa00,0x270bf900 2014-06-09 16:22:12.221829 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.221839 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.221848 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.221863 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1023 2014-06-09 16:22:12.221875 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.221884 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.221895 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.221899 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.57b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.221934 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.221938 7f8692767700 5 queue_transactions existing osr(0.57b 0x2bd77b0)/0x2bd77b0 2014-06-09 16:22:12.221941 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.221946 7f86a5e11700 5 _do_op 0xbe7de00 seq 4217 osr(1.223 0x384cc70)/0x384cc70 start 2014-06-09 16:22:12.221950 7f86a5e11700 10 _do_transaction on 0xc1f0f00 2014-06-09 16:22:12.221956 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.221999 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.222016 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4217.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.222021 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.222044 7f86a5e11700 10 _do_op 0xbe7de00 seq 4217 r = 0, finisher 0xc40d740 0 2014-06-09 16:22:12.222052 7f86a5e11700 10 _finish_op 0xbe7de00 seq 4217 osr(1.223 0x384cc70)/0x384cc70 2014-06-09 16:22:12.222053 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcefe0] 2014-06-09 16:22:12.222062 7f8691765700 10 op_submit_start 4266 2014-06-09 16:22:12.222063 7f8691765700 5 queue_transactions (trailing journal) 4266 2014-06-09 16:22:12.222057 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.222064 7f8691765700 5 queue_op 0xc88be60 seq 4266 osr(1.54a 0x384daf0) 828 bytes (queue has 50 ops and 5035082 bytes) 2014-06-09 16:22:12.222068 7f8689755700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222066 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.222072 7f868d75d700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222076 7f8689f56700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222079 7f868ff62700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222075 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:154 2014-06-09 16:22:12.222083 7f868af58700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222085 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcefe0] 2014-06-09 16:22:12.222087 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.222087 7f868f761700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222088 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.222090 7f868df5e700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222094 7f868bf5a700 2 waiting 51 > 50 ops || 5035908 > 104857600 2014-06-09 16:22:12.222098 7f868c75b700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222103 7f8692f68700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222107 7f868b759700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222112 7f868e75f700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222116 7f868cf5c700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222120 7f868a757700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222125 7f868ef60700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222129 7f8690763700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222133 7f8690f64700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222138 7f8692767700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222142 7f8691f66700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222152 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc40d740] 2014-06-09 16:22:12.222159 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc40d740] 2014-06-09 16:22:12.222160 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.222161 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.222165 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497c20 (1 active) 2014-06-09 16:22:12.222168 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1440 (2 active) 2014-06-09 16:22:12.222172 7f8691765700 10 op_submit_finish 4266 2014-06-09 16:22:12.222176 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:12.222212 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.222217 7f8691765700 5 queue_transactions existing osr(3.48d 0x3ed14e0)/0x3ed14e0 2014-06-09 16:22:12.222220 7f8691765700 2 waiting 51 > 50 ops || 5035910 > 104857600 2014-06-09 16:22:12.222224 7f86a5e11700 5 _do_op 0x16fd4780 seq 4218 osr(1.233 0x384cc30)/0x384cc30 start 2014-06-09 16:22:12.222228 7f86a5e11700 10 _do_transaction on 0xc28be00 2014-06-09 16:22:12.222233 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.222275 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.222290 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4218.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.222294 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.222305 7f86a5e11700 10 _do_op 0x16fd4780 seq 4218 r = 0, finisher 0x3251ca20 0 2014-06-09 16:22:12.222311 7f86a5e11700 10 _finish_op 0x16fd4780 seq 4218 osr(1.233 0x384cc30)/0x384cc30 2014-06-09 16:22:12.222313 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f9d60] 2014-06-09 16:22:12.222316 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f9d60] 2014-06-09 16:22:12.222317 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.222318 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.222322 7f868d75d700 10 op_submit_start 4267 2014-06-09 16:22:12.222323 7f868d75d700 5 queue_transactions (trailing journal) 4267 2014-06-09 16:22:12.222325 7f868d75d700 5 queue_op 0x14dd6060 seq 4267 osr(1.301 0x384d030) 861 bytes (queue has 50 ops and 5035115 bytes) 2014-06-09 16:22:12.222330 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.222335 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222339 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.222344 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222348 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222353 7f868bf5a700 2 waiting 51 > 50 ops || 5035941 > 104857600 2014-06-09 16:22:12.222356 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.222361 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222365 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222369 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222374 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.222378 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.222382 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222387 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.222391 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222395 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222400 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.222404 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222408 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222418 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251ca20] 2014-06-09 16:22:12.222422 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251ca20] 2014-06-09 16:22:12.222424 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.222425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.222428 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1440 (1 active) 2014-06-09 16:22:12.222431 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114360 (2 active) 2014-06-09 16:22:12.222435 7f868d75d700 10 op_submit_finish 4267 2014-06-09 16:22:12.222439 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.222453 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.222462 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.222470 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.222479 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.222487 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.222501 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 2014-06-09 16:22:12.222510 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.222519 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.222528 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.222533 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.222545 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.118381 7 0.000369 2014-06-09 16:22:12.222557 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.222567 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.222577 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.222587 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.222596 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.222603 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.222614 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.222625 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.222634 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.222710 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.222727 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 2014-06-09 16:22:12.222760 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.222764 7f868d75d700 5 queue_transactions existing osr(0.629 0x2bd7c90)/0x2bd7c90 2014-06-09 16:22:12.222766 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.222794 7f86a5e11700 5 _do_op 0x324e7d40 seq 4219 osr(0.b9 0x2bd6400)/0x2bd6400 start 2014-06-09 16:22:12.222799 7f86a5e11700 10 _do_transaction on 0x480d080 2014-06-09 16:22:12.222805 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.222847 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.222862 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4219.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.222866 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.222878 7f86a5e11700 10 _do_op 0x324e7d40 seq 4219 r = 0, finisher 0xbcc9b60 0 2014-06-09 16:22:12.222889 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5e2c0] 2014-06-09 16:22:12.222891 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5e2c0] 2014-06-09 16:22:12.222893 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.222894 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.222897 7f86a5e11700 10 _finish_op 0x324e7d40 seq 4219 osr(0.b9 0x2bd6400)/0x2bd6400 2014-06-09 16:22:12.222908 7f868ff62700 10 op_submit_start 4268 2014-06-09 16:22:12.222910 7f868ff62700 5 queue_transactions (trailing journal) 4268 2014-06-09 16:22:12.222911 7f868ff62700 5 queue_op 0xd235c20 seq 4268 osr(1.504 0x384dbd0) 828 bytes (queue has 50 ops and 5035117 bytes) 2014-06-09 16:22:12.222917 7f868af58700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.222921 7f868f761700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222925 7f868df5e700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222930 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.222934 7f868c75b700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.222938 7f8692f68700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222942 7f868b759700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222946 7f868e75f700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222950 7f868cf5c700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.222955 7f868a757700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.222960 7f868ef60700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222964 7f8690763700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.222968 7f8690f64700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222973 7f8692767700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222977 7f8691f66700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.222981 7f8691765700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222986 7f8689755700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.222990 7f868d75d700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.222994 7f8689f56700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223004 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbcc9b60] 2014-06-09 16:22:12.223010 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbcc9b60] 2014-06-09 16:22:12.223012 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.223013 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.223016 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114360 (1 active) 2014-06-09 16:22:12.223019 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6900 (2 active) 2014-06-09 16:22:12.223024 7f868ff62700 10 op_submit_finish 4268 2014-06-09 16:22:12.223028 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 243/201 242/242/200) 2014-06-09 16:22:12.223063 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.223069 7f868ff62700 5 queue_transactions existing osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 2014-06-09 16:22:12.223071 7f868ff62700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223076 7f86a5e11700 5 _do_op 0x230b9e60 seq 4220 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 start 2014-06-09 16:22:12.223080 7f86a5e11700 10 _do_transaction on 0xbfa5740 2014-06-09 16:22:12.223085 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.223126 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.223140 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4220.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.223144 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.223158 7f86a5e11700 10 _do_op 0x230b9e60 seq 4220 r = 0, finisher 0x3281b100 0 2014-06-09 16:22:12.223168 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd78440] 2014-06-09 16:22:12.223171 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd78440] 2014-06-09 16:22:12.223172 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.223173 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.223177 7f86a5e11700 10 _finish_op 0x230b9e60 seq 4220 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 2014-06-09 16:22:12.223188 7f868f761700 10 op_submit_start 4269 2014-06-09 16:22:12.223189 7f868f761700 5 queue_transactions (trailing journal) 4269 2014-06-09 16:22:12.223191 7f868f761700 5 queue_op 0x324f40c0 seq 4269 osr(3.111 0x3ed0830) 828 bytes (queue has 50 ops and 5035117 bytes) 2014-06-09 16:22:12.223196 7f868df5e700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223201 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223205 7f868c75b700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223210 7f8692f68700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223214 7f868b759700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223218 7f868e75f700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223222 7f868cf5c700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223227 7f868a757700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223231 7f868ef60700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223235 7f8690763700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223240 7f8690f64700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223243 7f8692767700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223248 7f8691f66700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223252 7f8691765700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223256 7f8689755700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223261 7f868d75d700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223265 7f8689f56700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223269 7f868ff62700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223274 7f868af58700 2 waiting 51 > 50 ops || 5035978 > 104857600 2014-06-09 16:22:12.223283 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281b100] 2014-06-09 16:22:12.223291 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281b100] 2014-06-09 16:22:12.223293 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.223294 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.223297 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6900 (1 active) 2014-06-09 16:22:12.223300 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f440 (2 active) 2014-06-09 16:22:12.223305 7f868f761700 10 op_submit_finish 4269 2014-06-09 16:22:12.223308 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.302( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.223343 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.223348 7f868f761700 5 queue_transactions existing osr(1.302 0x384d010)/0x384d010 2014-06-09 16:22:12.223351 7f868f761700 2 waiting 51 > 50 ops || 5035945 > 104857600 2014-06-09 16:22:12.223355 7f86a5e11700 5 _do_op 0xd327c20 seq 4221 osr(1.508 0x384db90)/0x384db90 start 2014-06-09 16:22:12.223359 7f86a5e11700 10 _do_transaction on 0xbf2e840 2014-06-09 16:22:12.223364 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.223405 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.223419 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4221.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.223424 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.223434 7f86a5e11700 10 _do_op 0xd327c20 seq 4221 r = 0, finisher 0x30d61b20 0 2014-06-09 16:22:12.223445 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc4117a0] 2014-06-09 16:22:12.223447 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc4117a0] 2014-06-09 16:22:12.223449 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.223450 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.223453 7f86a5e11700 10 _finish_op 0xd327c20 seq 4221 osr(1.508 0x384db90)/0x384db90 2014-06-09 16:22:12.223465 7f868bf5a700 10 op_submit_start 4270 2014-06-09 16:22:12.223466 7f868bf5a700 5 queue_transactions (trailing journal) 4270 2014-06-09 16:22:12.223468 7f868bf5a700 5 queue_op 0x324e73e0 seq 4270 osr(3.9f 0x3ed0600) 826 bytes (queue has 50 ops and 5035115 bytes) 2014-06-09 16:22:12.223473 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223478 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223482 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223486 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223491 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223495 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223499 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223504 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223508 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223512 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223517 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223521 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223525 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223530 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223534 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223538 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223543 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223547 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223552 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223561 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30d61b20] 2014-06-09 16:22:12.223566 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30d61b20] 2014-06-09 16:22:12.223568 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.223569 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.223572 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f440 (1 active) 2014-06-09 16:22:12.223575 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780120 (2 active) 2014-06-09 16:22:12.223580 7f868bf5a700 10 op_submit_finish 4270 2014-06-09 16:22:12.223584 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.465( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.223625 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.223630 7f868bf5a700 5 queue_transactions existing osr(1.465 0x384d650)/0x384d650 2014-06-09 16:22:12.223633 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223637 7f86a5e11700 5 _do_op 0x324f4a80 seq 4222 osr(2.5f2 0x2ed1df0)/0x2ed1df0 start 2014-06-09 16:22:12.223641 7f86a5e11700 10 _do_transaction on 0xc22a180 2014-06-09 16:22:12.223647 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.223688 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.223703 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4222.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.223707 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.223730 7f86a5e11700 10 _do_op 0x324f4a80 seq 4222 r = 0, finisher 0x307f18a0 0 2014-06-09 16:22:12.223741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31456ce0] 2014-06-09 16:22:12.223743 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31456ce0] 2014-06-09 16:22:12.223747 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.223749 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.223752 7f86a5e11700 10 _finish_op 0x324f4a80 seq 4222 osr(2.5f2 0x2ed1df0)/0x2ed1df0 2014-06-09 16:22:12.223763 7f8692f68700 10 op_submit_start 4271 2014-06-09 16:22:12.223765 7f8692f68700 5 queue_transactions (trailing journal) 4271 2014-06-09 16:22:12.223766 7f8692f68700 5 queue_op 0x1e08b9e0 seq 4271 osr(0.1da 0x2bd6920) 828 bytes (queue has 50 ops and 5035115 bytes) 2014-06-09 16:22:12.223771 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223776 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223780 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223785 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223789 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223793 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223797 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223802 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223806 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223810 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223815 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223819 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223823 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223827 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223831 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223836 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223840 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223844 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223848 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.223858 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f18a0] 2014-06-09 16:22:12.223862 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f18a0] 2014-06-09 16:22:12.223863 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.223865 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.223867 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780120 (1 active) 2014-06-09 16:22:12.223870 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524b40 (2 active) 2014-06-09 16:22:12.223875 7f8692f68700 10 op_submit_finish 4271 2014-06-09 16:22:12.223879 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.4af( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.223914 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.223919 7f8692f68700 5 queue_transactions existing osr(2.4af 0x2ed1790)/0x2ed1790 2014-06-09 16:22:12.223922 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.223926 7f86a5e11700 5 _do_op 0xcabc000 seq 4223 osr(1.2c0 0x384d170)/0x384d170 start 2014-06-09 16:22:12.223930 7f86a5e11700 10 _do_transaction on 0xbf7ccc0 2014-06-09 16:22:12.223936 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.223977 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.223991 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4223.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.223996 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.224007 7f86a5e11700 10 _do_op 0xcabc000 seq 4223 r = 0, finisher 0x3006da40 0 2014-06-09 16:22:12.224018 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1b0c0e60] 2014-06-09 16:22:12.224020 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1b0c0e60] 2014-06-09 16:22:12.224021 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.224023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.224026 7f86a5e11700 10 _finish_op 0xcabc000 seq 4223 osr(1.2c0 0x384d170)/0x384d170 2014-06-09 16:22:12.224036 7f868e75f700 10 op_submit_start 4272 2014-06-09 16:22:12.224038 7f868e75f700 5 queue_transactions (trailing journal) 4272 2014-06-09 16:22:12.224039 7f868e75f700 5 queue_op 0x324f50e0 seq 4272 osr(0.640 0x2bd7c10) 828 bytes (queue has 50 ops and 5035115 bytes) 2014-06-09 16:22:12.224044 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.224049 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.224053 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224058 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.224062 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224066 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224071 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.224075 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224080 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224084 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.224088 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.224092 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224096 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.224100 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224104 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224108 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224113 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.224117 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224121 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224130 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006da40] 2014-06-09 16:22:12.224134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006da40] 2014-06-09 16:22:12.224136 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.224137 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.224140 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524b40 (1 active) 2014-06-09 16:22:12.224143 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34966c0 (2 active) 2014-06-09 16:22:12.224148 7f868e75f700 10 op_submit_finish 4272 2014-06-09 16:22:12.224152 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.530( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.224185 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.225349 7f868e75f700 5 queue_transactions existing osr(0.530 0x2bd78f0)/0x2bd78f0 2014-06-09 16:22:12.225353 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.224189 7f86a5e11700 5 _do_op 0xc34bc80 seq 4224 osr(0.686 0x2bd7e20)/0x2bd7e20 start 2014-06-09 16:22:12.225359 7f86a5e11700 10 _do_transaction on 0xc321680 2014-06-09 16:22:12.225367 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.225414 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.225431 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4224.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.225436 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.225449 7f86a5e11700 10 _do_op 0xc34bc80 seq 4224 r = 0, finisher 0x19941cc0 0 2014-06-09 16:22:12.225460 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x12050240] 2014-06-09 16:22:12.225463 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.225474 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.225484 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:12.225494 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x12050240] 2014-06-09 16:22:12.225496 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.225497 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.225501 7f86a5e11700 10 _finish_op 0xc34bc80 seq 4224 osr(0.686 0x2bd7e20)/0x2bd7e20 2014-06-09 16:22:12.225513 7f868a757700 10 op_submit_start 4273 2014-06-09 16:22:12.225515 7f868a757700 5 queue_transactions (trailing journal) 4273 2014-06-09 16:22:12.225516 7f868a757700 5 queue_op 0xc2d48a0 seq 4273 osr(3.672 0x3ed1350) 861 bytes (queue has 50 ops and 5035115 bytes) 2014-06-09 16:22:12.225522 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225527 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.225531 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225535 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225540 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.225544 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225548 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225553 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.225557 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.225562 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225566 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.225570 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225575 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225579 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225583 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.225588 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225593 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225597 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 2014-06-09 16:22:12.225601 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.225611 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x19941cc0] 2014-06-09 16:22:12.225615 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x19941cc0] 2014-06-09 16:22:12.225617 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.225618 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.225621 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34966c0 (1 active) 2014-06-09 16:22:12.225624 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc240 (2 active) 2014-06-09 16:22:12.225630 7f868a757700 10 op_submit_finish 4273 2014-06-09 16:22:12.225634 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.225649 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.084456 7 0.000348 2014-06-09 16:22:12.225661 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.225672 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.225684 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.225694 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.225703 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.225711 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.225721 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.225734 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.225743 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.225758 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 2014-06-09 16:22:12.225777 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 2014-06-09 16:22:12.225812 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.225816 7f868a757700 5 queue_transactions existing osr(2.523 0x2ed1610)/0x2ed1610 2014-06-09 16:22:12.225819 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 2014-06-09 16:22:12.225854 7f86a5e11700 5 _do_op 0x32539980 seq 4225 osr(3.56a 0x3ed1900)/0x3ed1900 start 2014-06-09 16:22:12.225863 7f86a5e11700 10 _do_transaction on 0x390c540 2014-06-09 16:22:12.225868 7f86a5e11700 15 _omap_setkeys meta/1fe1045b/pglog_3.56a/0//-1 2014-06-09 16:22:12.225916 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe1045b/pglog_3.56a/0//-1 2014-06-09 16:22:12.225932 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.225973 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.225988 7f86a5e11700 15 _omap_rmkeys meta/1fe1045b/pglog_3.56a/0//-1 2014-06-09 16:22:12.225995 7f86a5e11700 10 _omap_rmkeys meta/1fe1045b/pglog_3.56a/0//-1 r = 0 2014-06-09 16:22:12.226001 7f86a5e11700 15 _omap_setkeys meta/1fe1045b/pglog_3.56a/0//-1 2014-06-09 16:22:12.226014 7f86a5e11700 15 _touch 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.226061 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.56a_head oid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.226093 7f86a5e11700 20 write_state seq is 30173 2014-06-09 16:22:12.226101 7f86a5e11700 20 set_header setting 30172 cid 3.56a_head oid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 parent seq 0 2014-06-09 16:22:12.226114 7f86a5e11700 10 _touch 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 = 0 2014-06-09 16:22:12.226121 7f86a5e11700 15 _write 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 0~344064 2014-06-09 16:22:12.226133 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.227598 7f86a5e11700 10 _generic_write 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 0~344064 = 0 2014-06-09 16:22:12.227649 7f86a5e11700 15 _setattrs 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.227662 7f86a5e11700 10 _setattrs 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 = 0 2014-06-09 16:22:12.227669 7f86a5e11700 15 _setattrs 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 2014-06-09 16:22:12.227678 7f86a5e11700 10 _setattrs 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 = 0 2014-06-09 16:22:12.227739 7f86a5e11700 10 cid: oid: 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 not skipping op, *spos 4225.0.0 2014-06-09 16:22:12.227743 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.227757 7f86a5e11700 20 set_header setting 30172 cid 3.56a_head oid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 parent seq 0 2014-06-09 16:22:12.227800 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4225.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.227806 7f86a5e11700 10 cid: oid: 1fe1045b/pglog_3.56a/0//-1 skipping op, spos 4225.0.0 <= header.spos 150148.0.0 2014-06-09 16:22:12.227809 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.229132 7f86a5e11700 10 _do_op 0x32539980 seq 4225 r = 0, finisher 0x30e3eb00 0x6c8dd40 2014-06-09 16:22:12.229147 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078b00] 2014-06-09 16:22:12.229150 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 31 2014-06-09 16:22:12.229175 7f86a460e700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.229167, event: op_commit, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.229184 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078b00] 2014-06-09 16:22:12.229186 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.229187 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.229192 7f86a5e11700 10 _finish_op 0x32539980 seq 4225 osr(3.56a 0x3ed1900)/0x3ed1900 2014-06-09 16:22:12.229207 7f8690763700 10 op_submit_start 4274 2014-06-09 16:22:12.229209 7f8690763700 5 queue_transactions (trailing journal) 4274 2014-06-09 16:22:12.229211 7f8690763700 5 queue_op 0x16fd4900 seq 4274 osr(2.102 0x2ed0710) 861 bytes (queue has 50 ops and 4689514 bytes) 2014-06-09 16:22:12.229217 7f8690f64700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229223 7f8692767700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229228 7f8691f66700 2 waiting 51 > 50 ops || 4690375 > 104857600 2014-06-09 16:22:12.229234 7f8691765700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229240 7f8689755700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229245 7f868d75d700 2 waiting 51 > 50 ops || 4690375 > 104857600 2014-06-09 16:22:12.229251 7f8689f56700 2 waiting 51 > 50 ops || 4690375 > 104857600 2014-06-09 16:22:12.229257 7f868ff62700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229262 7f868af58700 2 waiting 51 > 50 ops || 4690375 > 104857600 2014-06-09 16:22:12.229268 7f868f761700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229273 7f868df5e700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229279 7f868bf5a700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229284 7f868c75b700 2 waiting 51 > 50 ops || 4690375 > 104857600 2014-06-09 16:22:12.229289 7f8692f68700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229295 7f868b759700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229300 7f868e75f700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229306 7f868cf5c700 2 waiting 51 > 50 ops || 4690375 > 104857600 2014-06-09 16:22:12.229312 7f868a757700 2 waiting 51 > 50 ops || 4690375 > 104857600 2014-06-09 16:22:12.229316 7f868ef60700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229329 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e3eb00] 2014-06-09 16:22:12.229332 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 31 2014-06-09 16:22:12.229351 7f86a4e0f700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.229345, event: op_applied, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.229359 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'576 2014-06-09 16:22:12.229385 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e3eb00] 2014-06-09 16:22:12.229387 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.229389 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.229393 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc240 (1 active) 2014-06-09 16:22:12.229396 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa480 (2 active) 2014-06-09 16:22:12.229402 7f8690763700 10 op_submit_finish 4274 2014-06-09 16:22:12.229408 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.298( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.229451 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.229460 7f8690763700 5 queue_transactions existing osr(0.298 0x2bd6c60)/0x2bd6c60 2014-06-09 16:22:12.229463 7f8690763700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229468 7f86a5e11700 5 _do_op 0x16fd4240 seq 4226 osr(1.37d 0x384d210)/0x384d210 start 2014-06-09 16:22:12.229472 7f86a5e11700 10 _do_transaction on 0xbfe4780 2014-06-09 16:22:12.229478 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.229547 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.229565 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4226.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.229570 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.229583 7f86a5e11700 10 _do_op 0x16fd4240 seq 4226 r = 0, finisher 0x307f1ae0 0 2014-06-09 16:22:12.229594 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2d7398c0] 2014-06-09 16:22:12.229597 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.229608 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.229618 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 2014-06-09 16:22:12.229627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2d7398c0] 2014-06-09 16:22:12.229629 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.229630 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.229634 7f86a5e11700 10 _finish_op 0x16fd4240 seq 4226 osr(1.37d 0x384d210)/0x384d210 2014-06-09 16:22:12.229645 7f8692767700 10 op_submit_start 4275 2014-06-09 16:22:12.229647 7f8692767700 5 queue_transactions (trailing journal) 4275 2014-06-09 16:22:12.229648 7f8692767700 5 queue_op 0x2e21aa0 seq 4275 osr(0.57b 0x2bd77b0) 828 bytes (queue has 50 ops and 4689481 bytes) 2014-06-09 16:22:12.229653 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229658 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229662 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229667 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229671 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229675 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229679 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229683 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229688 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229692 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229696 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229700 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229705 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229709 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229714 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229718 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.229722 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229726 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229730 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229740 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f1ae0] 2014-06-09 16:22:12.229744 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f1ae0] 2014-06-09 16:22:12.229745 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.229747 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.229749 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa480 (1 active) 2014-06-09 16:22:12.229752 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b680 (2 active) 2014-06-09 16:22:12.229757 7f8692767700 10 op_submit_finish 4275 2014-06-09 16:22:12.229761 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 243/201 242/242/200) 2014-06-09 16:22:12.229798 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.229804 7f8692767700 5 queue_transactions existing osr(3.4c4 0x3ed1420)/0x3ed1420 2014-06-09 16:22:12.229807 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229812 7f86a5e11700 5 _do_op 0xc2d4060 seq 4227 osr(1.764 0x2ed03f0)/0x2ed03f0 start 2014-06-09 16:22:12.229815 7f86a5e11700 10 _do_transaction on 0x2ed5bc0 2014-06-09 16:22:12.229821 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.229864 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.229881 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4227.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.229886 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.229900 7f86a5e11700 10 _do_op 0xc2d4060 seq 4227 r = 0, finisher 0xc5a6a60 0 2014-06-09 16:22:12.229910 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061f620] 2014-06-09 16:22:12.229913 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.229923 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.229932 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.229942 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061f620] 2014-06-09 16:22:12.229944 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.229945 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.229948 7f86a5e11700 10 _finish_op 0xc2d4060 seq 4227 osr(1.764 0x2ed03f0)/0x2ed03f0 2014-06-09 16:22:12.229960 7f8691765700 10 op_submit_start 4276 2014-06-09 16:22:12.229962 7f8691765700 5 queue_transactions (trailing journal) 4276 2014-06-09 16:22:12.229963 7f8691765700 5 queue_op 0xc88bec0 seq 4276 osr(3.48d 0x3ed14e0) 828 bytes (queue has 50 ops and 4689448 bytes) 2014-06-09 16:22:12.229968 7f8689755700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.229973 7f868d75d700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229977 7f8689f56700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229982 7f868ff62700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.229986 7f868af58700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.229990 7f868f761700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.229994 7f868df5e700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.229998 7f868bf5a700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230003 7f868c75b700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230007 7f8692f68700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230011 7f868b759700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230016 7f868e75f700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230020 7f868cf5c700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230024 7f868a757700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230028 7f868ef60700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230033 7f8690763700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230037 7f8690f64700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230041 7f8692767700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230046 7f8691f66700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230055 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a6a60] 2014-06-09 16:22:12.230062 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a6a60] 2014-06-09 16:22:12.230064 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.230065 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.230069 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b680 (1 active) 2014-06-09 16:22:12.230072 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4480 (2 active) 2014-06-09 16:22:12.230077 7f8691765700 10 op_submit_finish 4276 2014-06-09 16:22:12.230081 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.482( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.230118 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.230124 7f8691765700 5 queue_transactions existing osr(2.482 0x2ed14f0)/0x2ed14f0 2014-06-09 16:22:12.230126 7f8691765700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.230130 7f86a5e11700 5 _do_op 0x3250f9e0 seq 4228 osr(1.4bb 0x384d970)/0x384d970 start 2014-06-09 16:22:12.230134 7f86a5e11700 10 _do_transaction on 0xc1452c0 2014-06-09 16:22:12.230139 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.230181 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.230196 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4228.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.230200 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.230214 7f86a5e11700 10 _do_op 0x3250f9e0 seq 4228 r = 0, finisher 0xc2d6540 0 2014-06-09 16:22:12.230224 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002de0] 2014-06-09 16:22:12.230226 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002de0] 2014-06-09 16:22:12.230228 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.230229 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.230233 7f86a5e11700 10 _finish_op 0x3250f9e0 seq 4228 osr(1.4bb 0x384d970)/0x384d970 2014-06-09 16:22:12.230244 7f868d75d700 10 op_submit_start 4277 2014-06-09 16:22:12.230246 7f868d75d700 5 queue_transactions (trailing journal) 4277 2014-06-09 16:22:12.230248 7f868d75d700 5 queue_op 0xd471ce0 seq 4277 osr(0.629 0x2bd7c90) 861 bytes (queue has 50 ops and 4689481 bytes) 2014-06-09 16:22:12.230253 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.230258 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230262 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.230266 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230270 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230275 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230279 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.230283 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230288 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230292 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230296 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.230300 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.230366 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230371 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230375 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230379 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230384 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.230388 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230392 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.230402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d6540] 2014-06-09 16:22:12.230406 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d6540] 2014-06-09 16:22:12.230407 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.230408 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.230412 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4480 (1 active) 2014-06-09 16:22:12.230414 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2877a0 (2 active) 2014-06-09 16:22:12.230420 7f868d75d700 10 op_submit_finish 4277 2014-06-09 16:22:12.230423 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.230439 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.230448 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.230457 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.230467 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.230475 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.230490 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:164 2014-06-09 16:22:12.230500 7f868d75d700 20 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.230509 7f868d75d700 20 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.230518 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.230547 7f868d75d700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f35f0 (19 active) 2014-06-09 16:22:12.230557 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.230561 7f868d75d700 10 osd.26 244 dequeue_op 0x270bf400 prio 63 cost 307200 latency 1.467174 osd_op(client.4575.0:32478 rbd_data.117f6b8b4567.0000000000001554 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~307200] 3.d7c85d1e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] 2014-06-09 16:22:12.230585 7f868d75d700 5 -- op tracker -- , seq: 693, time: 2014-06-09 16:22:12.230580, event: reached_pg, op: osd_op(client.4575.0:32478 rbd_data.117f6b8b4567.0000000000001554 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~307200] 3.d7c85d1e RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.230601 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.230615 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bf400 2014-06-09 16:22:12.230624 7f868d75d700 10 osd.26 244 dequeue_op 0x270bf400 finish 2014-06-09 16:22:12.230628 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.230671 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f35f0 (20 active) 2014-06-09 16:22:12.230675 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.230690 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] exit Started/Primary/Active/Activating 1.125597 7 0.000432 2014-06-09 16:22:12.230704 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.230715 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] needs_recovery is recovered 2014-06-09 16:22:12.230728 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] exit Started/Primary/Active/Recovered 0.000025 0 0.000000 2014-06-09 16:22:12.230741 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.230750 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] finish_recovery 2014-06-09 16:22:12.230759 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] clear_recovery_state 2014-06-09 16:22:12.230771 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) 2014-06-09 16:22:12.230783 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) 2014-06-09 16:22:12.230794 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] share_pg_info 2014-06-09 16:22:12.230811 7f868d75d700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 244 2014-06-09 16:22:12.230830 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] publish_stats_to_osd 243:977 2014-06-09 16:22:12.230865 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.230869 7f868d75d700 5 queue_transactions existing osr(3.206 0x3ed0cf0)/0x3ed0cf0 2014-06-09 16:22:12.230872 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.230590 7f86a5e11700 5 _do_op 0x324e6c60 seq 4229 osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 start 2014-06-09 16:22:12.230880 7f86a5e11700 10 _do_transaction on 0xc2bec00 2014-06-09 16:22:12.230885 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.230929 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.230944 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4229.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.230948 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.230960 7f86a5e11700 10 _do_op 0x324e6c60 seq 4229 r = 0, finisher 0x31456340 0 2014-06-09 16:22:12.230970 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31434280] 2014-06-09 16:22:12.230972 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31434280] 2014-06-09 16:22:12.230974 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.230975 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.230979 7f86a5e11700 10 _finish_op 0x324e6c60 seq 4229 osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 2014-06-09 16:22:12.230990 7f868ff62700 10 op_submit_start 4278 2014-06-09 16:22:12.230992 7f868ff62700 5 queue_transactions (trailing journal) 4278 2014-06-09 16:22:12.230993 7f868ff62700 5 queue_op 0xc88a9c0 seq 4278 osr(3.1a0 0x3ed0ab0) 828 bytes (queue has 50 ops and 4689481 bytes) 2014-06-09 16:22:12.230999 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231003 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231008 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231012 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231016 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231021 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231025 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231029 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231034 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231038 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231042 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231047 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231051 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231055 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231059 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231063 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231067 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231071 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231076 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231085 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31456340] 2014-06-09 16:22:12.231090 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31456340] 2014-06-09 16:22:12.231092 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.231093 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.231096 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2877a0 (1 active) 2014-06-09 16:22:12.231099 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524120 (2 active) 2014-06-09 16:22:12.231104 7f868ff62700 10 op_submit_finish 4278 2014-06-09 16:22:12.231108 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.5cb( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:12.231143 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.231148 7f868ff62700 5 queue_transactions existing osr(1.5cb 0x384dd70)/0x384dd70 2014-06-09 16:22:12.231150 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231155 7f86a5e11700 5 _do_op 0x324da120 seq 4230 osr(0.3fe 0x2bd7210)/0x2bd7210 start 2014-06-09 16:22:12.231158 7f86a5e11700 10 _do_transaction on 0xbf82300 2014-06-09 16:22:12.231163 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.231205 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.231220 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4230.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.231224 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.231244 7f86a5e11700 10 _do_op 0x324da120 seq 4230 r = 0, finisher 0x19940080 0 2014-06-09 16:22:12.231255 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcf240] 2014-06-09 16:22:12.231257 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcf240] 2014-06-09 16:22:12.231259 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.231260 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.231263 7f86a5e11700 10 _finish_op 0x324da120 seq 4230 osr(0.3fe 0x2bd7210)/0x2bd7210 2014-06-09 16:22:12.231274 7f868f761700 10 op_submit_start 4279 2014-06-09 16:22:12.231276 7f868f761700 5 queue_transactions (trailing journal) 4279 2014-06-09 16:22:12.231277 7f868f761700 5 queue_op 0x24104300 seq 4279 osr(1.302 0x384d010) 828 bytes (queue has 50 ops and 4689481 bytes) 2014-06-09 16:22:12.231283 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231287 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231292 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231297 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231301 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231306 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231310 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231314 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231319 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231323 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231327 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231332 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231336 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231340 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231344 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231349 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231353 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231357 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231362 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x19940080] 2014-06-09 16:22:12.231375 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x19940080] 2014-06-09 16:22:12.231377 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.231378 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.231381 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524120 (1 active) 2014-06-09 16:22:12.231384 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287200 (2 active) 2014-06-09 16:22:12.231389 7f868f761700 10 op_submit_finish 4279 2014-06-09 16:22:12.231393 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.231406 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.231415 7f868f761700 15 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.231424 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.231432 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.231441 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.231454 7f868f761700 15 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:152 2014-06-09 16:22:12.231464 7f868f761700 20 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.231472 7f868f761700 20 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.231481 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.231485 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:12.231518 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.231523 7f868f761700 5 queue_transactions existing osr(3.2df 0x3ed0e40)/0x3ed0e40 2014-06-09 16:22:12.231525 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231530 7f86a5e11700 5 _do_op 0xd327260 seq 4231 osr(1.302 0x384d010)/0x384d010 start 2014-06-09 16:22:12.231533 7f86a5e11700 10 _do_transaction on 0xc257200 2014-06-09 16:22:12.231538 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.231579 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.231594 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4231.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.231598 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.231610 7f86a5e11700 10 _do_op 0xd327260 seq 4231 r = 0, finisher 0x31bffec0 0 2014-06-09 16:22:12.231624 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbce7120] 2014-06-09 16:22:12.231626 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbce7120] 2014-06-09 16:22:12.231627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.231628 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.231631 7f86a5e11700 10 _finish_op 0xd327260 seq 4231 osr(1.302 0x384d010)/0x384d010 2014-06-09 16:22:12.231640 7f868bf5a700 10 op_submit_start 4280 2014-06-09 16:22:12.231641 7f868bf5a700 5 queue_transactions (trailing journal) 4280 2014-06-09 16:22:12.231643 7f868bf5a700 5 queue_op 0x30032c60 seq 4280 osr(1.465 0x384d650) 828 bytes (queue has 50 ops and 4689481 bytes) 2014-06-09 16:22:12.231647 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231651 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231654 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231658 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231662 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231666 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231669 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231673 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231677 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231681 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231685 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231689 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231694 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231698 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231703 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231707 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231711 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231716 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231720 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231730 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31bffec0] 2014-06-09 16:22:12.231734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31bffec0] 2014-06-09 16:22:12.231736 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.231737 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.231740 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287200 (1 active) 2014-06-09 16:22:12.231743 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd440 (2 active) 2014-06-09 16:22:12.231747 7f868bf5a700 10 op_submit_finish 4280 2014-06-09 16:22:12.231751 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.566( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.231786 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.231791 7f868bf5a700 5 queue_transactions existing osr(2.566 0x2ed18f0)/0x2ed18f0 2014-06-09 16:22:12.231794 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231798 7f86a5e11700 5 _do_op 0x324f4b40 seq 4232 osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 start 2014-06-09 16:22:12.231802 7f86a5e11700 10 _do_transaction on 0xc22a6c0 2014-06-09 16:22:12.231807 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.231849 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.231864 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4232.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.231868 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.231883 7f86a5e11700 10 _do_op 0x324f4b40 seq 4232 r = 0, finisher 0xcd337e0 0 2014-06-09 16:22:12.231894 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314adfa0] 2014-06-09 16:22:12.231896 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314adfa0] 2014-06-09 16:22:12.231898 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.231899 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.231902 7f86a5e11700 10 _finish_op 0x324f4b40 seq 4232 osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 2014-06-09 16:22:12.231913 7f8692f68700 10 op_submit_start 4281 2014-06-09 16:22:12.231914 7f8692f68700 5 queue_transactions (trailing journal) 4281 2014-06-09 16:22:12.231916 7f8692f68700 5 queue_op 0x1e08b7a0 seq 4281 osr(2.4af 0x2ed1790) 828 bytes (queue has 50 ops and 4689481 bytes) 2014-06-09 16:22:12.231921 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231925 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231930 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231934 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231938 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231943 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231947 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231952 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231956 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231960 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231965 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231969 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231973 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231977 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231981 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.231986 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231990 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231994 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.231998 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd337e0] 2014-06-09 16:22:12.232016 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd337e0] 2014-06-09 16:22:12.232018 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.232019 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.232022 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd440 (1 active) 2014-06-09 16:22:12.232025 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4078c0 (2 active) 2014-06-09 16:22:12.232030 7f8692f68700 10 op_submit_finish 4281 2014-06-09 16:22:12.232034 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.1da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.232072 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.232077 7f8692f68700 5 queue_transactions existing osr(0.1da 0x2bd6920)/0x2bd6920 2014-06-09 16:22:12.232080 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232084 7f86a5e11700 5 _do_op 0xcabcfc0 seq 4233 osr(3.218 0x3ed0cd0)/0x3ed0cd0 start 2014-06-09 16:22:12.232088 7f86a5e11700 10 _do_transaction on 0xbf88180 2014-06-09 16:22:12.232093 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.232134 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.232148 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4233.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.232153 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.232165 7f86a5e11700 10 _do_op 0xcabcfc0 seq 4233 r = 0, finisher 0x3042b0a0 0 2014-06-09 16:22:12.232175 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a61c0] 2014-06-09 16:22:12.232178 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a61c0] 2014-06-09 16:22:12.232179 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.232180 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.232184 7f86a5e11700 10 _finish_op 0xcabcfc0 seq 4233 osr(3.218 0x3ed0cd0)/0x3ed0cd0 2014-06-09 16:22:12.232195 7f868e75f700 10 op_submit_start 4282 2014-06-09 16:22:12.232196 7f868e75f700 5 queue_transactions (trailing journal) 4282 2014-06-09 16:22:12.232198 7f868e75f700 5 queue_op 0x324f4f60 seq 4282 osr(0.530 0x2bd78f0) 828 bytes (queue has 50 ops and 4689481 bytes) 2014-06-09 16:22:12.232203 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232208 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232213 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232276 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232282 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232286 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232291 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232295 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232299 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232304 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232308 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232312 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232316 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232320 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232325 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232329 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232333 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232338 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232342 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232351 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b0a0] 2014-06-09 16:22:12.232356 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b0a0] 2014-06-09 16:22:12.232357 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.232358 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.232361 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4078c0 (1 active) 2014-06-09 16:22:12.232364 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7e60 (2 active) 2014-06-09 16:22:12.232369 7f868e75f700 10 op_submit_finish 4282 2014-06-09 16:22:12.232373 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.26a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.232410 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.232415 7f868e75f700 5 queue_transactions existing osr(2.26a 0x2ed0dd0)/0x2ed0dd0 2014-06-09 16:22:12.232418 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232422 7f86a5e11700 5 _do_op 0xc34a660 seq 4234 osr(3.65f 0x3ed1a10)/0x3ed1a10 start 2014-06-09 16:22:12.232426 7f86a5e11700 10 _do_transaction on 0xc129200 2014-06-09 16:22:12.232432 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.232474 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.232491 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4234.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.232495 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.232526 7f86a5e11700 10 _do_op 0xc34a660 seq 4234 r = 0, finisher 0x3006dbc0 0 2014-06-09 16:22:12.232538 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9f160] 2014-06-09 16:22:12.232540 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.232553 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.232564 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1114 2014-06-09 16:22:12.232576 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9f160] 2014-06-09 16:22:12.232578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.232579 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.232583 7f86a5e11700 10 _finish_op 0xc34a660 seq 4234 osr(3.65f 0x3ed1a10)/0x3ed1a10 2014-06-09 16:22:12.232594 7f868a757700 10 op_submit_start 4283 2014-06-09 16:22:12.232596 7f868a757700 5 queue_transactions (trailing journal) 4283 2014-06-09 16:22:12.232598 7f868a757700 5 queue_op 0xc2d4840 seq 4283 osr(2.523 0x2ed1610) 861 bytes (queue has 50 ops and 4689481 bytes) 2014-06-09 16:22:12.232603 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232607 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232612 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232616 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232620 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232625 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232629 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232633 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232638 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232642 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232646 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232651 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232655 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232660 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232664 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232668 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232672 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232676 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.232680 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006dbc0] 2014-06-09 16:22:12.232695 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006dbc0] 2014-06-09 16:22:12.232697 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.232698 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.232701 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7e60 (1 active) 2014-06-09 16:22:12.232704 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b9e0 (2 active) 2014-06-09 16:22:12.232709 7f868a757700 10 op_submit_finish 4283 2014-06-09 16:22:12.232713 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.232728 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.062975 7 0.000322 2014-06-09 16:22:12.232741 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.232752 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.232764 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.232775 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.232785 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.232794 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.232805 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.232818 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.232829 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.232844 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 2014-06-09 16:22:12.232859 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1105 2014-06-09 16:22:12.232894 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.232898 7f868a757700 5 queue_transactions existing osr(3.380 0x3ed1020)/0x3ed1020 2014-06-09 16:22:12.232901 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 2014-06-09 16:22:12.232929 7f86a5e11700 5 _do_op 0xc2eaba0 seq 4235 osr(1.4bc 0x384d930)/0x384d930 start 2014-06-09 16:22:12.232938 7f86a5e11700 10 _do_transaction on 0x2ed4780 2014-06-09 16:22:12.232944 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.232987 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.233003 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4235.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.233008 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.233020 7f86a5e11700 10 _do_op 0xc2eaba0 seq 4235 r = 0, finisher 0x2e0e7e0 0 2014-06-09 16:22:12.233031 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x305087c0] 2014-06-09 16:22:12.233034 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.233044 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.233053 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:12.233063 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x305087c0] 2014-06-09 16:22:12.233064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.233066 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.233069 7f86a5e11700 10 _finish_op 0xc2eaba0 seq 4235 osr(1.4bc 0x384d930)/0x384d930 2014-06-09 16:22:12.233080 7f8690763700 10 op_submit_start 4284 2014-06-09 16:22:12.233082 7f8690763700 5 queue_transactions (trailing journal) 4284 2014-06-09 16:22:12.233083 7f8690763700 5 queue_op 0xc2ea540 seq 4284 osr(0.298 0x2bd6c60) 828 bytes (queue has 50 ops and 4689448 bytes) 2014-06-09 16:22:12.233088 7f8690f64700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233092 7f8692767700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233097 7f8691f66700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.233101 7f8691765700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233105 7f8689755700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233109 7f868d75d700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.233114 7f8689f56700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.233118 7f868ff62700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233122 7f868af58700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.233126 7f868f761700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233130 7f868df5e700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233135 7f868bf5a700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233139 7f868c75b700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.233143 7f8692f68700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233147 7f868b759700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233151 7f868e75f700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233155 7f868cf5c700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.233160 7f868a757700 2 waiting 51 > 50 ops || 4690309 > 104857600 2014-06-09 16:22:12.233164 7f868ef60700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233174 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0e7e0] 2014-06-09 16:22:12.233178 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0e7e0] 2014-06-09 16:22:12.233180 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.233181 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.233184 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b9e0 (1 active) 2014-06-09 16:22:12.233187 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358d80 (2 active) 2014-06-09 16:22:12.233192 7f8690763700 10 op_submit_finish 4284 2014-06-09 16:22:12.233196 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.233214 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] check_local 2014-06-09 16:22:12.233226 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 2014-06-09 16:22:12.233236 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing 2014-06-09 16:22:12.233248 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.233265 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:982 2014-06-09 16:22:12.233277 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear 2014-06-09 16:22:12.233288 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] agent_stop 2014-06-09 16:22:12.233299 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.233304 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.648( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.233337 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.233342 7f8690763700 5 queue_transactions existing osr(0.648 0x2bd7bd0)/0x2bd7bd0 2014-06-09 16:22:12.233344 7f8690763700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233349 7f86a5e11700 5 _do_op 0xc64a960 seq 4236 osr(2.3c7 0x2ed13d0)/0x2ed13d0 start 2014-06-09 16:22:12.233352 7f86a5e11700 10 _do_transaction on 0xc22a840 2014-06-09 16:22:12.233358 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.233400 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.233418 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4236.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.233422 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.233434 7f86a5e11700 10 _do_op 0xc64a960 seq 4236 r = 0, finisher 0xbd71e20 0 2014-06-09 16:22:12.233445 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e8460] 2014-06-09 16:22:12.233448 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.233458 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.233467 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:207 2014-06-09 16:22:12.233476 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e8460] 2014-06-09 16:22:12.233478 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.233479 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.233483 7f86a5e11700 10 _finish_op 0xc64a960 seq 4236 osr(2.3c7 0x2ed13d0)/0x2ed13d0 2014-06-09 16:22:12.233494 7f8692767700 10 op_submit_start 4285 2014-06-09 16:22:12.233495 7f8692767700 5 queue_transactions (trailing journal) 4285 2014-06-09 16:22:12.233497 7f8692767700 5 queue_op 0x325398c0 seq 4285 osr(3.4c4 0x3ed1420) 828 bytes (queue has 50 ops and 4689415 bytes) 2014-06-09 16:22:12.233502 7f8691f66700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233507 7f8691765700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233511 7f8689755700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233515 7f868d75d700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233520 7f8689f56700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233524 7f868ff62700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233528 7f868af58700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233532 7f868f761700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233537 7f868df5e700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233541 7f868bf5a700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233545 7f868c75b700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233549 7f8692f68700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233554 7f868b759700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233558 7f868e75f700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233562 7f868cf5c700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233566 7f868a757700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.233571 7f868ef60700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233575 7f8690763700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233579 7f8690f64700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233589 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd71e20] 2014-06-09 16:22:12.233594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd71e20] 2014-06-09 16:22:12.233596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.233597 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.233600 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358d80 (1 active) 2014-06-09 16:22:12.233603 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac5a0 (2 active) 2014-06-09 16:22:12.233608 7f8692767700 10 op_submit_finish 4285 2014-06-09 16:22:12.233611 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.113( v 227'654 (0'0,227'654] local-les=243 n=129 ec=118 les/c 243/213 242/242/212) 2014-06-09 16:22:12.233650 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.233655 7f8692767700 5 queue_transactions existing osr(3.113 0x3ed0800)/0x3ed0800 2014-06-09 16:22:12.233657 7f8692767700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233662 7f86a5e11700 5 _do_op 0x308cb7a0 seq 4237 osr(2.5f4 0x2ed1dd0)/0x2ed1dd0 start 2014-06-09 16:22:12.233665 7f86a5e11700 10 _do_transaction on 0xbeba6c0 2014-06-09 16:22:12.233671 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.233707 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.233722 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4237.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.233725 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.233738 7f86a5e11700 10 _do_op 0x308cb7a0 seq 4237 r = 0, finisher 0x19941780 0 2014-06-09 16:22:12.233747 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd33a80] 2014-06-09 16:22:12.233750 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.233758 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.233766 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 2014-06-09 16:22:12.233774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd33a80] 2014-06-09 16:22:12.233775 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.233776 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.233779 7f86a5e11700 10 _finish_op 0x308cb7a0 seq 4237 osr(2.5f4 0x2ed1dd0)/0x2ed1dd0 2014-06-09 16:22:12.233788 7f8691765700 10 op_submit_start 4286 2014-06-09 16:22:12.233790 7f8691765700 5 queue_transactions (trailing journal) 4286 2014-06-09 16:22:12.233791 7f8691765700 5 queue_op 0xc88aa20 seq 4286 osr(2.482 0x2ed14f0) 828 bytes (queue has 50 ops and 4689382 bytes) 2014-06-09 16:22:12.233794 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233798 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233801 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233805 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233808 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233811 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233814 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233817 7f868bf5a700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233821 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233824 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233827 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233830 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233833 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233836 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233839 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233842 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233845 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233848 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233851 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.233859 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x19941780] 2014-06-09 16:22:12.233862 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x19941780] 2014-06-09 16:22:12.233864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.233864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.233867 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac5a0 (1 active) 2014-06-09 16:22:12.233869 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338120 (2 active) 2014-06-09 16:22:12.233874 7f8691765700 10 op_submit_finish 4286 2014-06-09 16:22:12.233877 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 243/243 242/242/205) 2014-06-09 16:22:12.233914 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.233919 7f8691765700 5 queue_transactions existing osr(3.48d 0x3ed14e0)/0x3ed14e0 2014-06-09 16:22:12.233921 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.233926 7f86a5e11700 5 _do_op 0x325034a0 seq 4238 osr(0.6f0 0x384c170)/0x384c170 start 2014-06-09 16:22:12.233930 7f86a5e11700 10 _do_transaction on 0xbf39080 2014-06-09 16:22:12.233934 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.233976 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.233990 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4238.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.233994 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.234012 7f86a5e11700 10 _do_op 0x325034a0 seq 4238 r = 0, finisher 0x2d7399a0 0 2014-06-09 16:22:12.234023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6bb00] 2014-06-09 16:22:12.234025 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6bb00] 2014-06-09 16:22:12.234027 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.234028 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.234031 7f86a5e11700 10 _finish_op 0x325034a0 seq 4238 osr(0.6f0 0x384c170)/0x384c170 2014-06-09 16:22:12.234042 7f868d75d700 10 op_submit_start 4287 2014-06-09 16:22:12.234043 7f868d75d700 5 queue_transactions (trailing journal) 4287 2014-06-09 16:22:12.234044 7f868d75d700 5 queue_op 0xc2eb680 seq 4287 osr(3.206 0x3ed0cf0) 861 bytes (queue has 50 ops and 4689415 bytes) 2014-06-09 16:22:12.234049 7f8689f56700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234054 7f868ff62700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234058 7f868af58700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234062 7f868f761700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234067 7f868df5e700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234071 7f868bf5a700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234075 7f868c75b700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234079 7f8692f68700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234084 7f868b759700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234088 7f868e75f700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234092 7f868cf5c700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234096 7f868a757700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234101 7f868ef60700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234105 7f8690763700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234109 7f8690f64700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234113 7f8692767700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234118 7f8691f66700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234122 7f8691765700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234126 7f8689755700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234194 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d7399a0] 2014-06-09 16:22:12.234199 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d7399a0] 2014-06-09 16:22:12.234201 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.234202 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.234205 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338120 (1 active) 2014-06-09 16:22:12.234208 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c05a0 (2 active) 2014-06-09 16:22:12.234213 7f868d75d700 10 op_submit_finish 4287 2014-06-09 16:22:12.234216 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.234230 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.024258 7 0.000347 2014-06-09 16:22:12.234241 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.234252 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.234262 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.234272 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.234281 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.234288 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.234299 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.234310 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.234319 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.234329 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.234339 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.234351 7f868d75d700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 244 2014-06-09 16:22:12.234368 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.234399 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.234404 7f868d75d700 5 queue_transactions existing osr(2.2da 0x2ed0c10)/0x2ed0c10 2014-06-09 16:22:12.234406 7f868d75d700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234434 7f86a5e11700 5 _do_op 0x324e7c80 seq 4239 osr(1.48e 0x384d9b0)/0x384d9b0 start 2014-06-09 16:22:12.234441 7f86a5e11700 10 _do_transaction on 0xbe1b200 2014-06-09 16:22:12.234447 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.234490 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.234505 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4239.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.234510 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.234521 7f86a5e11700 10 _do_op 0x324e7c80 seq 4239 r = 0, finisher 0x2e0ec20 0 2014-06-09 16:22:12.234532 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf1ba0] 2014-06-09 16:22:12.234534 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf1ba0] 2014-06-09 16:22:12.234535 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.234536 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.234540 7f86a5e11700 10 _finish_op 0x324e7c80 seq 4239 osr(1.48e 0x384d9b0)/0x384d9b0 2014-06-09 16:22:12.234551 7f868ff62700 10 op_submit_start 4288 2014-06-09 16:22:12.234552 7f868ff62700 5 queue_transactions (trailing journal) 4288 2014-06-09 16:22:12.234554 7f868ff62700 5 queue_op 0x159975c0 seq 4288 osr(1.5cb 0x384dd70) 828 bytes (queue has 50 ops and 4689415 bytes) 2014-06-09 16:22:12.234558 7f868af58700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234563 7f868f761700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234568 7f868df5e700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234572 7f868bf5a700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234577 7f868c75b700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234581 7f8692f68700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234586 7f868b759700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234590 7f868e75f700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234594 7f868cf5c700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234599 7f868a757700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234603 7f868ef60700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234607 7f8690763700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234612 7f8690f64700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234616 7f8692767700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234620 7f8691f66700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234624 7f8691765700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234628 7f8689755700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234632 7f868d75d700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234637 7f8689f56700 2 waiting 51 > 50 ops || 4690276 > 104857600 2014-06-09 16:22:12.234647 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0ec20] 2014-06-09 16:22:12.234650 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0ec20] 2014-06-09 16:22:12.234652 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.234653 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.234656 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c05a0 (1 active) 2014-06-09 16:22:12.234659 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4acb40 (2 active) 2014-06-09 16:22:12.234665 7f868ff62700 10 op_submit_finish 4288 2014-06-09 16:22:12.234668 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.234683 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.234693 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x257c4900,0x10c54f00,0x257c4800,0x11e63200,0x257c4700,0x11e63100,0x257c4600,0x11e63000,0x11e63600,0x11e63500,0x11e63400 2014-06-09 16:22:12.234734 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.2 has 1 missing 2014-06-09 16:22:12.234745 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.234762 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1199 2014-06-09 16:22:12.234772 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.234782 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.234792 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.234797 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.234809 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.234818 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.234826 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.234835 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.234843 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.234856 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.234865 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.234873 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.234883 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.234887 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.35c( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) 2014-06-09 16:22:12.234920 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.234925 7f868ff62700 5 queue_transactions existing osr(1.35c 0x384d2b0)/0x384d2b0 2014-06-09 16:22:12.234927 7f868ff62700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.234932 7f86a5e11700 5 _do_op 0x324e6120 seq 4240 osr(2.40f 0x2ed1210)/0x2ed1210 start 2014-06-09 16:22:12.234936 7f86a5e11700 10 _do_transaction on 0xc2d3bc0 2014-06-09 16:22:12.234942 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.234984 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.235001 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4240.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.235005 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.235020 7f86a5e11700 10 _do_op 0x324e6120 seq 4240 r = 0, finisher 0xccf8ea0 0 2014-06-09 16:22:12.235031 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5f620] 2014-06-09 16:22:12.235033 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.235043 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.235053 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 2014-06-09 16:22:12.235062 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5f620] 2014-06-09 16:22:12.235064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.235065 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.235069 7f86a5e11700 10 _finish_op 0x324e6120 seq 4240 osr(2.40f 0x2ed1210)/0x2ed1210 2014-06-09 16:22:12.235080 7f868f761700 10 op_submit_start 4289 2014-06-09 16:22:12.235082 7f868f761700 5 queue_transactions (trailing journal) 4289 2014-06-09 16:22:12.235083 7f868f761700 5 queue_op 0x324e7860 seq 4289 osr(3.2df 0x3ed0e40) 828 bytes (queue has 50 ops and 4689382 bytes) 2014-06-09 16:22:12.235088 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235092 7f868bf5a700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235097 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235101 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235105 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235110 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235114 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235119 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235123 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235127 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235131 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235136 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235140 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235144 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235148 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235153 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235157 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235161 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235165 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235175 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf8ea0] 2014-06-09 16:22:12.235179 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf8ea0] 2014-06-09 16:22:12.235181 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.235182 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.235185 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4acb40 (1 active) 2014-06-09 16:22:12.235188 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653680 (2 active) 2014-06-09 16:22:12.235193 7f868f761700 10 op_submit_finish 4289 2014-06-09 16:22:12.235196 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.235214 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] check_local 2014-06-09 16:22:12.235226 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 2014-06-09 16:22:12.235236 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing 2014-06-09 16:22:12.235248 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.235264 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1104 2014-06-09 16:22:12.235276 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear 2014-06-09 16:22:12.235287 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] agent_stop 2014-06-09 16:22:12.235299 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.235304 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.235317 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.235326 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x22009f00 2014-06-09 16:22:12.235340 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.0 has 2 missing 2014-06-09 16:22:12.235350 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.235366 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:880 2014-06-09 16:22:12.235377 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.235386 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.235396 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.235401 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.235417 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] check_local 2014-06-09 16:22:12.235427 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 2014-06-09 16:22:12.235438 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing 2014-06-09 16:22:12.235449 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.235465 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:997 2014-06-09 16:22:12.235477 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear 2014-06-09 16:22:12.235488 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] agent_stop 2014-06-09 16:22:12.235499 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.235503 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 243/203 242/242/202) 2014-06-09 16:22:12.235536 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.235540 7f868f761700 5 queue_transactions existing osr(3.2a9 0x3ed0b40)/0x3ed0b40 2014-06-09 16:22:12.235543 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235547 7f86a5e11700 5 _do_op 0x2fc8dc80 seq 4241 osr(3.111 0x3ed0830)/0x3ed0830 start 2014-06-09 16:22:12.235551 7f86a5e11700 10 _do_transaction on 0x344ca80 2014-06-09 16:22:12.235556 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.235598 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.235618 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4241.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.235623 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.235635 7f86a5e11700 10 _do_op 0x2fc8dc80 seq 4241 r = 0, finisher 0x3251c7a0 0 2014-06-09 16:22:12.235644 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2d40] 2014-06-09 16:22:12.235646 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2d40] 2014-06-09 16:22:12.235647 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.235648 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.235650 7f86a5e11700 10 _finish_op 0x2fc8dc80 seq 4241 osr(3.111 0x3ed0830)/0x3ed0830 2014-06-09 16:22:12.235659 7f868bf5a700 10 op_submit_start 4290 2014-06-09 16:22:12.235660 7f868bf5a700 5 queue_transactions (trailing journal) 4290 2014-06-09 16:22:12.235662 7f868bf5a700 5 queue_op 0xbf7f320 seq 4290 osr(2.566 0x2ed18f0) 828 bytes (queue has 50 ops and 4689382 bytes) 2014-06-09 16:22:12.235666 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235670 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235673 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235676 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235679 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235683 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235686 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235690 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235693 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235697 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235700 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235703 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235707 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235710 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235713 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235716 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235719 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235723 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235726 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251c7a0] 2014-06-09 16:22:12.235740 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251c7a0] 2014-06-09 16:22:12.235742 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.235742 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.235745 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653680 (1 active) 2014-06-09 16:22:12.235747 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115440 (2 active) 2014-06-09 16:22:12.235751 7f868bf5a700 10 op_submit_finish 4290 2014-06-09 16:22:12.235754 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 243/243 242/242/205) 2014-06-09 16:22:12.235785 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.235789 7f868bf5a700 5 queue_transactions existing osr(3.9f 0x3ed0600)/0x3ed0600 2014-06-09 16:22:12.235792 7f868bf5a700 2 waiting 51 > 50 ops || 4690208 > 104857600 2014-06-09 16:22:12.235796 7f86a5e11700 5 _do_op 0x324f5560 seq 4242 osr(1.32f 0x384d350)/0x384d350 start 2014-06-09 16:22:12.235799 7f86a5e11700 10 _do_transaction on 0xc22a0c0 2014-06-09 16:22:12.235804 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.235842 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.235854 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4242.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.235858 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.235868 7f86a5e11700 10 _do_op 0x324f5560 seq 4242 r = 0, finisher 0xbd78820 0 2014-06-09 16:22:12.235876 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x182cac80] 2014-06-09 16:22:12.235878 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x182cac80] 2014-06-09 16:22:12.235879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.235880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.235883 7f86a5e11700 10 _finish_op 0x324f5560 seq 4242 osr(1.32f 0x384d350)/0x384d350 2014-06-09 16:22:12.235892 7f8692f68700 10 op_submit_start 4291 2014-06-09 16:22:12.235893 7f8692f68700 5 queue_transactions (trailing journal) 4291 2014-06-09 16:22:12.235894 7f8692f68700 5 queue_op 0xcc866c0 seq 4291 osr(0.1da 0x2bd6920) 828 bytes (queue has 50 ops and 4689382 bytes) 2014-06-09 16:22:12.235898 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235901 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235904 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235907 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235911 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235914 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235918 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235921 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235924 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235927 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235930 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235934 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235937 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235940 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235944 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235947 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235950 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.235953 7f868bf5a700 2 waiting 51 > 50 ops || 4690208 > 104857600 2014-06-09 16:22:12.235956 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.235963 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd78820] 2014-06-09 16:22:12.235967 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd78820] 2014-06-09 16:22:12.235970 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.235971 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.235973 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115440 (1 active) 2014-06-09 16:22:12.235976 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac45a0 (2 active) 2014-06-09 16:22:12.235979 7f8692f68700 10 op_submit_finish 4291 2014-06-09 16:22:12.235983 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.236016 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.236021 7f8692f68700 5 queue_transactions existing osr(3.1e1 0x3ed09b0)/0x3ed09b0 2014-06-09 16:22:12.236023 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236028 7f86a5e11700 5 _do_op 0x32539f80 seq 4243 osr(0.4ba 0x2bd7710)/0x2bd7710 start 2014-06-09 16:22:12.236031 7f86a5e11700 10 _do_transaction on 0xc22ae40 2014-06-09 16:22:12.236037 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.236145 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.236161 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4243.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.236165 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.236192 7f86a5e11700 10 _do_op 0x32539f80 seq 4243 r = 0, finisher 0x187423c0 0 2014-06-09 16:22:12.236204 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc1699c0] 2014-06-09 16:22:12.236206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc1699c0] 2014-06-09 16:22:12.236207 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.236209 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.236212 7f86a5e11700 10 _finish_op 0x32539f80 seq 4243 osr(0.4ba 0x2bd7710)/0x2bd7710 2014-06-09 16:22:12.236223 7f868e75f700 10 op_submit_start 4292 2014-06-09 16:22:12.236224 7f868e75f700 5 queue_transactions (trailing journal) 4292 2014-06-09 16:22:12.236226 7f868e75f700 5 queue_op 0x324f4de0 seq 4292 osr(2.26a 0x2ed0dd0) 828 bytes (queue has 50 ops and 4689382 bytes) 2014-06-09 16:22:12.236231 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236235 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236239 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236243 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236246 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236249 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236253 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236256 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236259 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236262 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236265 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236268 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236271 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236274 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236277 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236280 7f868bf5a700 2 waiting 51 > 50 ops || 4690208 > 104857600 2014-06-09 16:22:12.236284 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236287 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236290 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236297 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x187423c0] 2014-06-09 16:22:12.236303 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x187423c0] 2014-06-09 16:22:12.236305 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.236305 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.236308 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac45a0 (1 active) 2014-06-09 16:22:12.236310 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300b40 (2 active) 2014-06-09 16:22:12.236314 7f868e75f700 10 op_submit_finish 4292 2014-06-09 16:22:12.236317 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.33a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) 2014-06-09 16:22:12.236347 7f868e75f700 10 log is not dirty 2014-06-09 16:22:12.236351 7f868e75f700 5 queue_transactions existing osr(2.33a 0x2ed0ed0)/0x2ed0ed0 2014-06-09 16:22:12.236354 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236357 7f86a5e11700 5 _do_op 0xbe7de60 seq 4244 osr(3.661 0x3ed19f0)/0x3ed19f0 start 2014-06-09 16:22:12.236361 7f86a5e11700 10 _do_transaction on 0xc238cc0 2014-06-09 16:22:12.236366 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.236403 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.236417 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4244.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.236421 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.236434 7f86a5e11700 10 _do_op 0xbe7de60 seq 4244 r = 0, finisher 0x3006c0a0 0 2014-06-09 16:22:12.236442 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b100] 2014-06-09 16:22:12.236444 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] _finish_recovery 2014-06-09 16:22:12.236455 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] purge_strays 2014-06-09 16:22:12.236465 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] publish_stats_to_osd 243:929 2014-06-09 16:22:12.236474 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b100] 2014-06-09 16:22:12.236475 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.236476 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.236479 7f86a5e11700 10 _finish_op 0xbe7de60 seq 4244 osr(3.661 0x3ed19f0)/0x3ed19f0 2014-06-09 16:22:12.236487 7f868a757700 10 op_submit_start 4293 2014-06-09 16:22:12.236488 7f868a757700 5 queue_transactions (trailing journal) 4293 2014-06-09 16:22:12.236489 7f868a757700 5 queue_op 0xc64aea0 seq 4293 osr(3.380 0x3ed1020) 861 bytes (queue has 50 ops and 4689382 bytes) 2014-06-09 16:22:12.236493 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236497 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236500 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236503 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236506 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236510 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236513 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236516 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236519 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236522 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236525 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236528 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236531 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236534 7f868bf5a700 2 waiting 51 > 50 ops || 4690208 > 104857600 2014-06-09 16:22:12.236538 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236541 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236544 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236547 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236550 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 2014-06-09 16:22:12.236557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006c0a0] 2014-06-09 16:22:12.236563 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006c0a0] 2014-06-09 16:22:12.236564 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.236565 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.236567 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300b40 (1 active) 2014-06-09 16:22:12.236570 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406480 (2 active) 2014-06-09 16:22:12.236574 7f868a757700 10 op_submit_finish 4293 2014-06-09 16:22:12.236578 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.236593 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.236601 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x11e63c00,0x11e63b00,0x11e63a00 2014-06-09 16:22:12.236618 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.236626 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.236634 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.236648 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:969 2014-06-09 16:22:12.236658 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.236666 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.236675 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.236679 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.10d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.236707 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.236711 7f868a757700 5 queue_transactions existing osr(1.10d 0x384c930)/0x384c930 2014-06-09 16:22:12.236713 7f868a757700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236717 7f86a5e11700 5 _do_op 0xc2eaf00 seq 4245 osr(2.3de 0x2ed1330)/0x2ed1330 start 2014-06-09 16:22:12.236720 7f86a5e11700 10 _do_transaction on 0xc0155c0 2014-06-09 16:22:12.236725 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.236762 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.236777 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4245.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.236780 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.236792 7f86a5e11700 10 _do_op 0xc2eaf00 seq 4245 r = 0, finisher 0x3006c660 0 2014-06-09 16:22:12.236801 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x281b6600] 2014-06-09 16:22:12.236803 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.236811 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.236819 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:12.236827 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x281b6600] 2014-06-09 16:22:12.236828 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.236829 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.236832 7f86a5e11700 10 _finish_op 0xc2eaf00 seq 4245 osr(2.3de 0x2ed1330)/0x2ed1330 2014-06-09 16:22:12.236841 7f8690763700 10 op_submit_start 4294 2014-06-09 16:22:12.236842 7f8690763700 5 queue_transactions (trailing journal) 4294 2014-06-09 16:22:12.236843 7f8690763700 5 queue_op 0x11fce300 seq 4294 osr(0.648 0x2bd7bd0) 828 bytes (queue has 50 ops and 4689349 bytes) 2014-06-09 16:22:12.236847 7f8690f64700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236851 7f8692767700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236854 7f8691f66700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236857 7f8691765700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236860 7f8689755700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236863 7f868d75d700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236866 7f8689f56700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236870 7f868ff62700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236873 7f868af58700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236876 7f868f761700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236879 7f868df5e700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236882 7f868bf5a700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.236885 7f868c75b700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236888 7f8692f68700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236891 7f868b759700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236894 7f868e75f700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236897 7f868cf5c700 2 waiting 51 > 50 ops || 4690210 > 104857600 2014-06-09 16:22:12.236900 7f868a757700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236903 7f868ef60700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.236910 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006c660] 2014-06-09 16:22:12.236917 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006c660] 2014-06-09 16:22:12.236919 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.236920 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.236922 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406480 (1 active) 2014-06-09 16:22:12.236924 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7c20 (2 active) 2014-06-09 16:22:12.236928 7f8690763700 10 op_submit_finish 4294 2014-06-09 16:22:12.236931 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.236943 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.236950 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.236957 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.236963 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.236970 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.236981 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 2014-06-09 16:22:12.236988 7f8690763700 20 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.236994 7f8690763700 20 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.237002 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.237006 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.237016 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.237022 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.237029 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.237036 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.237041 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.237052 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 2014-06-09 16:22:12.237061 7f8690763700 20 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.237068 7f8690763700 20 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.237075 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.237079 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.237108 7f8690763700 10 log is not dirty 2014-06-09 16:22:12.237112 7f8690763700 5 queue_transactions existing osr(3.408 0x3ed1270)/0x3ed1270 2014-06-09 16:22:12.237114 7f8690763700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.237118 7f86a5e11700 5 _do_op 0xc64b3e0 seq 4246 osr(1.65d 0x384df50)/0x384df50 start 2014-06-09 16:22:12.237121 7f86a5e11700 10 _do_transaction on 0xc1fd080 2014-06-09 16:22:12.237126 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.237163 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.237177 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4246.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.237181 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.237194 7f86a5e11700 10 _do_op 0xc64b3e0 seq 4246 r = 0, finisher 0xbd757c0 0 2014-06-09 16:22:12.237202 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205dc00] 2014-06-09 16:22:12.237205 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.237213 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.237221 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:163 2014-06-09 16:22:12.237231 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205dc00] 2014-06-09 16:22:12.237232 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.237233 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.237236 7f86a5e11700 10 _finish_op 0xc64b3e0 seq 4246 osr(1.65d 0x384df50)/0x384df50 2014-06-09 16:22:12.237244 7f8692767700 10 op_submit_start 4295 2014-06-09 16:22:12.237246 7f8692767700 5 queue_transactions (trailing journal) 4295 2014-06-09 16:22:12.237247 7f8692767700 5 queue_op 0x32539860 seq 4295 osr(3.113 0x3ed0800) 828 bytes (queue has 50 ops and 4689316 bytes) 2014-06-09 16:22:12.237251 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.237254 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237257 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237260 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.237264 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.237267 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237270 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.237273 7f868f761700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237276 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237280 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.237282 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.237286 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237289 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237292 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237295 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.237299 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237302 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237305 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237308 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd757c0] 2014-06-09 16:22:12.237321 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd757c0] 2014-06-09 16:22:12.237322 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.237323 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.237325 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7c20 (1 active) 2014-06-09 16:22:12.237328 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525680 (2 active) 2014-06-09 16:22:12.237332 7f8692767700 10 op_submit_finish 4295 2014-06-09 16:22:12.237335 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.237347 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.237354 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.237361 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.237368 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.237374 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.237385 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:135 2014-06-09 16:22:12.237393 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.237399 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.237407 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.237411 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.237421 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.237427 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.237434 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.237442 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.237449 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.237460 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:194 2014-06-09 16:22:12.237468 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.237474 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.237481 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.237485 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.237496 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.237503 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x10c55400,0x10c55300,0x10c55200,0x10c55100,0x13023600 2014-06-09 16:22:12.237523 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.237531 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.237538 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.237549 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1054 2014-06-09 16:22:12.237557 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.237565 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.237573 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.237577 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.237587 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.237594 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.237600 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.237607 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.237613 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.237623 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 2014-06-09 16:22:12.237631 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.237637 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.237644 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.237649 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.237676 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.237680 7f8692767700 5 queue_transactions existing osr(3.105 0x3ed0820)/0x3ed0820 2014-06-09 16:22:12.237682 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237686 7f86a5e11700 5 _do_op 0xc2d5020 seq 4247 osr(0.6b3 0x2bd7ff0)/0x2bd7ff0 start 2014-06-09 16:22:12.237689 7f86a5e11700 10 _do_transaction on 0x2eaa840 2014-06-09 16:22:12.237695 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.237734 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.237749 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4247.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.237754 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.237776 7f86a5e11700 10 _do_op 0xc2d5020 seq 4247 r = 0, finisher 0xc169480 0 2014-06-09 16:22:12.237788 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078a20] 2014-06-09 16:22:12.237790 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.237801 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.237810 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 2014-06-09 16:22:12.237819 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078a20] 2014-06-09 16:22:12.237821 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.237822 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.237826 7f86a5e11700 10 _finish_op 0xc2d5020 seq 4247 osr(0.6b3 0x2bd7ff0)/0x2bd7ff0 2014-06-09 16:22:12.237838 7f8691765700 10 op_submit_start 4296 2014-06-09 16:22:12.237839 7f8691765700 5 queue_transactions (trailing journal) 4296 2014-06-09 16:22:12.237841 7f8691765700 5 queue_op 0xc88bf80 seq 4296 osr(3.48d 0x3ed14e0) 828 bytes (queue has 50 ops and 4689283 bytes) 2014-06-09 16:22:12.237846 7f8689755700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237851 7f868d75d700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237855 7f8689f56700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237860 7f868ff62700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237864 7f868af58700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237869 7f868f761700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237873 7f868df5e700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237877 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.237882 7f868c75b700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237886 7f8692f68700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237890 7f868b759700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237894 7f868e75f700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237899 7f868cf5c700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237903 7f868a757700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237907 7f868ef60700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237912 7f8690763700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237916 7f8690f64700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237921 7f8692767700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.237925 7f8691f66700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.237934 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc169480] 2014-06-09 16:22:12.237939 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc169480] 2014-06-09 16:22:12.237941 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.237942 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.237945 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525680 (1 active) 2014-06-09 16:22:12.237948 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d10e0 (2 active) 2014-06-09 16:22:12.237953 7f8691765700 10 op_submit_finish 4296 2014-06-09 16:22:12.237957 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.482( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.238060 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.238065 7f8691765700 5 queue_transactions existing osr(2.482 0x2ed14f0)/0x2ed14f0 2014-06-09 16:22:12.238069 7f8691765700 2 waiting 51 > 50 ops || 4690111 > 104857600 2014-06-09 16:22:12.238074 7f86a5e11700 5 _do_op 0xd326120 seq 4248 osr(0.448 0x2bd7190)/0x2bd7190 start 2014-06-09 16:22:12.238078 7f86a5e11700 10 _do_transaction on 0xbf0ea80 2014-06-09 16:22:12.238084 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.238127 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.238142 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4248.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.238146 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.238159 7f86a5e11700 10 _do_op 0xd326120 seq 4248 r = 0, finisher 0x30628960 0 2014-06-09 16:22:12.238170 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe677e0] 2014-06-09 16:22:12.238172 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe677e0] 2014-06-09 16:22:12.238174 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.238175 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.238178 7f86a5e11700 10 _finish_op 0xd326120 seq 4248 osr(0.448 0x2bd7190)/0x2bd7190 2014-06-09 16:22:12.238189 7f868d75d700 10 op_submit_start 4297 2014-06-09 16:22:12.238191 7f868d75d700 5 queue_transactions (trailing journal) 4297 2014-06-09 16:22:12.238192 7f868d75d700 5 queue_op 0xc2d4c60 seq 4297 osr(2.2da 0x2ed0c10) 861 bytes (queue has 50 ops and 4689316 bytes) 2014-06-09 16:22:12.238197 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238202 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238206 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238211 7f868f761700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238215 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238220 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.238224 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238228 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238232 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238237 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238242 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238247 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238251 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238256 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238260 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238264 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238269 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238273 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238277 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238287 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30628960] 2014-06-09 16:22:12.238292 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30628960] 2014-06-09 16:22:12.238293 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.238294 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.238298 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d10e0 (1 active) 2014-06-09 16:22:12.238301 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd8c0 (2 active) 2014-06-09 16:22:12.238306 7f868d75d700 10 op_submit_finish 4297 2014-06-09 16:22:12.238310 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.238325 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.007867 7 0.000379 2014-06-09 16:22:12.238339 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.238351 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.238363 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 2014-06-09 16:22:12.238374 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.238384 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.238391 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.238405 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.238417 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.238427 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.238443 7f868d75d700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 2014-06-09 16:22:12.238463 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 2014-06-09 16:22:12.238497 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.238501 7f868d75d700 5 queue_transactions existing osr(2.446 0x2ed1550)/0x2ed1550 2014-06-09 16:22:12.238504 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238544 7f86a5e11700 5 _do_op 0x324e7bc0 seq 4249 osr(1.66e 0x384ded0)/0x384ded0 start 2014-06-09 16:22:12.238554 7f86a5e11700 10 _do_transaction on 0xc23ee40 2014-06-09 16:22:12.238560 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.238605 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.238620 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4249.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.238624 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.238636 7f86a5e11700 10 _do_op 0x324e7bc0 seq 4249 r = 0, finisher 0xbd5fe80 0 2014-06-09 16:22:12.238647 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509bc0] 2014-06-09 16:22:12.238649 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509bc0] 2014-06-09 16:22:12.238651 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.238652 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.238655 7f86a5e11700 10 _finish_op 0x324e7bc0 seq 4249 osr(1.66e 0x384ded0)/0x384ded0 2014-06-09 16:22:12.238667 7f868ff62700 10 op_submit_start 4298 2014-06-09 16:22:12.238668 7f868ff62700 5 queue_transactions (trailing journal) 4298 2014-06-09 16:22:12.238670 7f868ff62700 5 queue_op 0xbf7f020 seq 4298 osr(1.35c 0x384d2b0) 828 bytes (queue has 50 ops and 4689316 bytes) 2014-06-09 16:22:12.238675 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238680 7f868f761700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238684 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238689 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.238693 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238697 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238702 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238706 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238711 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238715 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238719 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238724 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238729 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238733 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238737 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238741 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238746 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238750 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238756 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238762 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5fe80] 2014-06-09 16:22:12.238766 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5fe80] 2014-06-09 16:22:12.238768 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.238769 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.238773 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd8c0 (1 active) 2014-06-09 16:22:12.238775 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3385a0 (2 active) 2014-06-09 16:22:12.238779 7f86a5e11700 5 _do_op 0x324dbb00 seq 4250 osr(3.251 0x3ed0c20)/0x3ed0c20 start 2014-06-09 16:22:12.238781 7f868ff62700 10 op_submit_finish 4298 2014-06-09 16:22:12.238782 7f86a5e11700 10 _do_transaction on 0xc29cd80 2014-06-09 16:22:12.238788 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.238785 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.233( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.238814 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.238818 7f868ff62700 5 queue_transactions existing osr(1.233 0x384cc30)/0x384cc30 2014-06-09 16:22:12.238820 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238830 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.238844 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4250.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.238849 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.238860 7f86a5e11700 10 _do_op 0x324dbb00 seq 4250 r = 0, finisher 0x1b0c13c0 0 2014-06-09 16:22:12.238866 7f86a5e11700 10 _finish_op 0x324dbb00 seq 4250 osr(3.251 0x3ed0c20)/0x3ed0c20 2014-06-09 16:22:12.238867 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5f880] 2014-06-09 16:22:12.238869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5f880] 2014-06-09 16:22:12.238870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.238871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.238879 7f868f761700 10 op_submit_start 4299 2014-06-09 16:22:12.238881 7f868f761700 5 queue_transactions (trailing journal) 4299 2014-06-09 16:22:12.238882 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1b0c13c0] 2014-06-09 16:22:12.238882 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238883 7f868f761700 5 queue_op 0xc2d56e0 seq 4299 osr(3.2a9 0x3ed0b40) 828 bytes (queue has 50 ops and 4689316 bytes) 2014-06-09 16:22:12.238887 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1b0c13c0] 2014-06-09 16:22:12.238889 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.238889 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.238890 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.238893 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238893 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3385a0 (1 active) 2014-06-09 16:22:12.238896 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f9e0 (2 active) 2014-06-09 16:22:12.238899 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238900 7f86a5e11700 5 _do_op 0x2fc8c120 seq 4251 osr(3.5f8 0x3ed1b30)/0x3ed1b30 start 2014-06-09 16:22:12.238902 7f868f761700 10 op_submit_finish 4299 2014-06-09 16:22:12.238903 7f86a5e11700 10 _do_transaction on 0xbf50f00 2014-06-09 16:22:12.238904 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238908 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238912 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238915 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238906 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.238917 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.238919 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238923 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238927 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238931 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238934 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238938 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238940 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.238942 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238944 7f868f761700 5 queue_transactions existing osr(0.f 0x2bd60e0)/0x2bd60e0 2014-06-09 16:22:12.238946 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238950 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238955 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.238958 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.238960 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.238965 7f868f761700 2 waiting 51 > 50 ops || 4690140 > 104857600 2014-06-09 16:22:12.238973 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4251.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.238977 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.238989 7f86a5e11700 10 _do_op 0x2fc8c120 seq 4251 r = 0, finisher 0x31434000 0 2014-06-09 16:22:12.238996 7f86a5e11700 10 _finish_op 0x2fc8c120 seq 4251 osr(3.5f8 0x3ed1b30)/0x3ed1b30 2014-06-09 16:22:12.238996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc410440] 2014-06-09 16:22:12.238998 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc410440] 2014-06-09 16:22:12.238999 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.239000 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.239003 7f868df5e700 10 op_submit_start 4300 2014-06-09 16:22:12.239004 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239005 7f868df5e700 5 queue_transactions (trailing journal) 4300 2014-06-09 16:22:12.239008 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.239006 7f868df5e700 5 queue_op 0xcca7560 seq 4300 osr(3.400 0x3ed1260) 828 bytes (queue has 50 ops and 4689316 bytes) 2014-06-09 16:22:12.239008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31434000] 2014-06-09 16:22:12.239011 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239012 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31434000] 2014-06-09 16:22:12.239014 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.239015 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.239015 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239018 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f9e0 (1 active) 2014-06-09 16:22:12.239020 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239021 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358000 (2 active) 2014-06-09 16:22:12.239025 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.239025 7f86a5e11700 5 _do_op 0x324f53e0 seq 4252 osr(0.31f 0x2bd6d50)/0x2bd6d50 start 2014-06-09 16:22:12.239027 7f868df5e700 10 op_submit_finish 4300 2014-06-09 16:22:12.239028 7f86a5e11700 10 _do_transaction on 0xc2f4e40 2014-06-09 16:22:12.239034 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.239032 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.101( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.239056 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.239060 7f868df5e700 5 queue_transactions existing osr(2.101 0x2ed0730)/0x2ed0730 2014-06-09 16:22:12.239061 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239064 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239068 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239074 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239073 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.239078 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239083 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239086 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.239088 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4252.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.239091 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239092 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.239095 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239100 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.239104 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.239109 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 2014-06-09 16:22:12.239113 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 2014-06-09 16:22:12.239113 7f86a5e11700 10 _do_op 0x324f53e0 seq 4252 r = 0, finisher 0x314ad1c0 0 2014-06-09 16:22:12.239117 7f868f761700 2 waiting 51 > 50 ops || 4690140 > 104857600 2014-06-09 16:22:12.239120 7f86a5e11700 10 _finish_op 0x324f53e0 seq 4252 osr(0.31f 0x2bd6d50)/0x2bd6d50 2014-06-09 16:22:12.239121 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94740] 2014-06-09 16:22:12.239123 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94740] 2014-06-09 16:22:12.239124 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.239125 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.239128 7f868bf5a700 10 op_submit_start 4301 2014-06-09 16:22:12.239128 7f868c75b700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239129 7f868bf5a700 5 queue_transactions (trailing journal) 4301 2014-06-09 16:22:12.239131 7f868bf5a700 5 queue_op 0x3280a000 seq 4301 osr(3.9f 0x3ed0600) 826 bytes (queue has 50 ops and 4689314 bytes) 2014-06-09 16:22:12.239132 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x314ad1c0] 2014-06-09 16:22:12.239135 7f8692f68700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239138 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x314ad1c0] 2014-06-09 16:22:12.239140 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.239139 7f868b759700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239141 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.239144 7f868e75f700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239144 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358000 (1 active) 2014-06-09 16:22:12.239147 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287d40 (2 active) 2014-06-09 16:22:12.239148 7f868cf5c700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239151 7f86a5e11700 5 _do_op 0x32539f20 seq 4253 osr(3.52a 0x3ed1670)/0x3ed1670 start 2014-06-09 16:22:12.239153 7f868bf5a700 10 op_submit_finish 4301 2014-06-09 16:22:12.239154 7f86a5e11700 10 _do_transaction on 0xc2af740 2014-06-09 16:22:12.239160 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.239156 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.5a4( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.239178 7f868bf5a700 10 log is not dirty 2014-06-09 16:22:12.239181 7f868bf5a700 5 queue_transactions existing osr(2.5a4 0x2ed1810)/0x2ed1810 2014-06-09 16:22:12.239183 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239186 7f868df5e700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239189 7f868a757700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239195 7f868ef60700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239199 7f8690763700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239200 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.239203 7f8690f64700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239207 7f8692767700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239212 7f8691f66700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239214 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4253.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.239217 7f8691765700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239218 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.239221 7f8689755700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239225 7f868d75d700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239230 7f8689f56700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239229 7f86a5e11700 10 _do_op 0x32539f20 seq 4253 r = 0, finisher 0xbd7da60 0 2014-06-09 16:22:12.239233 7f868ff62700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.239236 7f86a5e11700 10 _finish_op 0x32539f20 seq 4253 osr(3.52a 0x3ed1670)/0x3ed1670 2014-06-09 16:22:12.239237 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3045a3c0] 2014-06-09 16:22:12.239238 7f868af58700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239240 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3045a3c0] 2014-06-09 16:22:12.239241 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.239242 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.239245 7f868f761700 2 waiting 51 > 50 ops || 4690138 > 104857600 2014-06-09 16:22:12.239612 7f868c75b700 10 op_submit_start 4302 2014-06-09 16:22:12.239612 7f8692f68700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239621 7f868c75b700 5 queue_transactions (trailing journal) 4302 2014-06-09 16:22:12.239625 7f868e75f700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239623 7f868c75b700 5 queue_op 0x2e21380 seq 4302 osr(3.224 0x3ed0c80) 861 bytes (queue has 50 ops and 4689347 bytes) 2014-06-09 16:22:12.239629 7f868cf5c700 2 waiting 51 > 50 ops || 4690208 > 104857600 2014-06-09 16:22:12.239633 7f868bf5a700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239633 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7da60] 2014-06-09 16:22:12.239639 7f868df5e700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239641 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7da60] 2014-06-09 16:22:12.239643 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.239644 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.239645 7f868a757700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239647 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287d40 (1 active) 2014-06-09 16:22:12.239649 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc923e60 (2 active) 2014-06-09 16:22:12.239650 7f868ef60700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239653 7f86a5e11700 5 _do_op 0x30060900 seq 4254 osr(2.5bf 0x2ed1b10)/0x2ed1b10 start 2014-06-09 16:22:12.239655 7f868c75b700 10 op_submit_finish 4302 2014-06-09 16:22:12.239655 7f86a5e11700 10 _do_transaction on 0xc2c6900 2014-06-09 16:22:12.239655 7f8690763700 2 waiting 51 > 50 ops || 4690175 > 104857600 2014-06-09 16:22:12.239661 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.239660 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239674 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239682 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239687 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239694 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239699 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239699 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.239702 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239709 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239714 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239714 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4254.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.239718 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.239716 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239723 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239728 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239732 7f86a5e11700 10 _do_op 0x30060900 seq 4254 r = 0, finisher 0x17001c80 0 2014-06-09 16:22:12.239730 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239737 7f86a5e11700 10 _finish_op 0x30060900 seq 4254 osr(2.5bf 0x2ed1b10)/0x2ed1b10 2014-06-09 16:22:12.239739 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfce060] 2014-06-09 16:22:12.239737 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239745 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239746 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17001c80] 2014-06-09 16:22:12.239750 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17001c80] 2014-06-09 16:22:12.239742 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.239752 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.239752 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.239755 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc923e60 (1 active) 2014-06-09 16:22:12.239749 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239752 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.239757 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d17a0 (2 active) 2014-06-09 16:22:12.239759 7f86a5e11700 5 _do_op 0xc2eb980 seq 4255 osr(2.6f4 0x2ed1e20)/0x2ed1e20 start 2014-06-09 16:22:12.239762 7f86a5e11700 10 _do_transaction on 0x39db440 2014-06-09 16:22:12.239760 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239760 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.239767 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.239768 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239769 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfce060] 2014-06-09 16:22:12.239773 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.239774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.239772 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239784 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.239791 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239794 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239802 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.239801 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239806 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239808 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239816 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4255.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.239815 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239820 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239820 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.239823 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239829 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239832 7f86a5e11700 10 _do_op 0xc2eb980 seq 4255 r = 0, finisher 0x3042b340 0 2014-06-09 16:22:12.239834 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239838 7f86a5e11700 10 _finish_op 0xc2eb980 seq 4255 osr(2.6f4 0x2ed1e20)/0x2ed1e20 2014-06-09 16:22:12.239839 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5b2e0] 2014-06-09 16:22:12.239845 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b340] 2014-06-09 16:22:12.239837 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239841 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.239849 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.239854 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b340] 2014-06-09 16:22:12.239855 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.239856 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.239849 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.239859 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239858 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d17a0 (1 active) 2014-06-09 16:22:12.239861 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fdc20 (2 active) 2014-06-09 16:22:12.239857 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 2014-06-09 16:22:12.239863 7f86a5e11700 5 _do_op 0xc88b680 seq 4256 osr(1.54a 0x384daf0)/0x384daf0 start 2014-06-09 16:22:12.239865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5b2e0] 2014-06-09 16:22:12.239865 7f86a5e11700 10 _do_transaction on 0xc2bda40 2014-06-09 16:22:12.239866 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.239867 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.239869 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.239863 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239874 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239879 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239882 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239889 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239895 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239897 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239903 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239908 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239910 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239917 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops 2014-06-09 16:22:12.239922 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239924 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239933 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.239939 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239941 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.239950 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 2014-06-09 16:22:12.239956 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.239965 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.239977 7f868c75b700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41050 (19 active) 2014-06-09 16:22:12.239978 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4256.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.239982 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.239985 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) 2014-06-09 16:22:12.239989 7f868c75b700 10 osd.26 244 dequeue_op 0x257c4900 prio 63 cost 217088 latency 1.456234 osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] 2014-06-09 16:22:12.240007 7f868c75b700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:12.240003, event: reached_pg, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.240011 7f86a5e11700 10 _do_op 0xc88b680 seq 4256 r = 0, finisher 0x12051880 0 2014-06-09 16:22:12.240017 7f86a5e11700 10 _finish_op 0xc88b680 seq 4256 osr(1.54a 0x384daf0)/0x384daf0 2014-06-09 16:22:12.240019 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d1760] 2014-06-09 16:22:12.240014 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes 2014-06-09 16:22:12.240024 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x12051880] 2014-06-09 16:22:12.240027 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d1760] 2014-06-09 16:22:12.240028 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.240028 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x12051880] 2014-06-09 16:22:12.240029 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.240030 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.240031 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.240033 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fdc20 (1 active) 2014-06-09 16:22:12.240026 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x257c4900 2014-06-09 16:22:12.240035 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1680 (2 active) 2014-06-09 16:22:12.240037 7f86a5e11700 5 _do_op 0x14dd7140 seq 4257 osr(3.22d 0x3ed0c40)/0x3ed0c40 start 2014-06-09 16:22:12.240040 7f86a5e11700 10 _do_transaction on 0xbf7cb40 2014-06-09 16:22:12.240037 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write 2014-06-09 16:22:12.240046 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.240055 7f868c75b700 15 getattr 3.67d_head/119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 '_' 2014-06-09 16:22:12.240061 7f8690f64700 10 op_submit_start 4303 2014-06-09 16:22:12.240063 7f8690f64700 5 queue_transactions (trailing journal) 4303 2014-06-09 16:22:12.240064 7f8690f64700 5 queue_op 0x324f5080 seq 4303 osr(0.2fd 0x2bd6e50) 828 bytes (queue has 49 ops and 4688453 bytes) 2014-06-09 16:22:12.240068 7f8690f64700 10 op_submit_finish 4303 2014-06-09 16:22:12.240071 7f8692767700 10 op_submit_start 4304 2014-06-09 16:22:12.240072 7f8692767700 5 queue_transactions (trailing journal) 4304 2014-06-09 16:22:12.240073 7f8692767700 5 queue_op 0xcabd0e0 seq 4304 osr(3.105 0x3ed0820) 828 bytes (queue has 49 ops and 4688453 bytes) 2014-06-09 16:22:12.240076 7f8692767700 10 op_submit_finish 4304 2014-06-09 16:22:12.240072 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 243/243 242/242/209) 2014-06-09 16:22:12.240086 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.240079 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.512( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.240102 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.241912 7f8690f64700 5 queue_transactions existing osr(3.271 0x3ed0bc0)/0x3ed0bc0 2014-06-09 16:22:12.240106 7f8691f66700 10 op_submit_start 4305 2014-06-09 16:22:12.241919 7f8691f66700 5 queue_transactions (trailing journal) 4305 2014-06-09 16:22:12.241918 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4257.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.241921 7f8691f66700 5 queue_op 0x324e6420 seq 4305 osr(3.2b3 0x3ed0f00) 861 bytes (queue has 50 ops and 4689314 bytes) 2014-06-09 16:22:12.241924 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.241924 7f8691f66700 10 op_submit_finish 4305 2014-06-09 16:22:12.241928 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.241933 7f8692767700 5 queue_transactions existing osr(0.512 0x2bd7590)/0x2bd7590 2014-06-09 16:22:12.240107 7f8691765700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.241928 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.241940 7f86a5e11700 10 _do_op 0x14dd7140 seq 4257 r = 0, finisher 0xc168500 0 2014-06-09 16:22:12.241947 7f86a5e11700 10 _finish_op 0x14dd7140 seq 4257 osr(3.22d 0x3ed0c40)/0x3ed0c40 2014-06-09 16:22:12.241944 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.198162 8 0.000473 2014-06-09 16:22:12.241956 7f8691765700 10 op_submit_start 4306 2014-06-09 16:22:12.241957 7f8691765700 5 queue_transactions (trailing journal) 4306 2014-06-09 16:22:12.241959 7f8691765700 5 queue_op 0x325267e0 seq 4306 osr(2.482 0x2ed14f0) 828 bytes (queue has 50 ops and 4689281 bytes) 2014-06-09 16:22:12.241961 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc168500] 2014-06-09 16:22:12.241964 7f868d75d700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.241959 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.241967 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc168500] 2014-06-09 16:22:12.241969 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.241969 7f868ff62700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.241970 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.241973 7f868af58700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.241974 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1680 (1 active) 2014-06-09 16:22:12.241977 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc360 (2 active) 2014-06-09 16:22:12.241979 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b440] 2014-06-09 16:22:12.241971 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.241981 7f86a5e11700 5 _do_op 0xcd90060 seq 4258 osr(3.5ce 0x3ed17c0)/0x3ed17c0 start 2014-06-09 16:22:12.241984 7f86a5e11700 10 _do_transaction on 0x37e4c00 2014-06-09 16:22:12.241981 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.241990 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.241984 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000026 0 0.000000 2014-06-09 16:22:12.241992 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.241994 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.242002 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1142 2014-06-09 16:22:12.242003 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.242011 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b440] 2014-06-09 16:22:12.242013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.242014 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.242018 7f8691765700 10 op_submit_finish 4306 2014-06-09 16:22:12.242011 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.242021 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 243/203 242/242/202) 2014-06-09 16:22:12.242026 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.242037 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.242043 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.242053 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.242047 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.242057 7f8691765700 5 queue_transactions existing osr(3.465 0x3ed1580)/0x3ed1580 2014-06-09 16:22:12.242060 7f8691765700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242058 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4258.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.242062 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.242063 7f8691f66700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 2014-06-09 16:22:12.242068 7f8689f56700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242073 7f868b759700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242075 7f86a5e11700 10 _do_op 0xcd90060 seq 4258 r = 0, finisher 0xbd68f20 0 2014-06-09 16:22:12.242078 7f868f761700 2 waiting 51 > 50 ops || 4690105 > 104857600 2014-06-09 16:22:12.242081 7f86a5e11700 10 _finish_op 0xcd90060 seq 4258 osr(3.5ce 0x3ed17c0)/0x3ed17c0 2014-06-09 16:22:12.242083 7f868cf5c700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242088 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050a5a0] 2014-06-09 16:22:12.242090 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050a5a0] 2014-06-09 16:22:12.242082 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1091 2014-06-09 16:22:12.242092 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.242093 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.242098 7f868e75f700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242103 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242107 7f868df5e700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242111 7f868ef60700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242115 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.242116 7f8690763700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242119 7f8691f66700 5 queue_transactions existing osr(3.48a 0x3ed1520)/0x3ed1520 2014-06-09 16:22:12.242122 7f8691f66700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242126 7f868a757700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242131 7f8692f68700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242135 7f8690f64700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.240169 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:12.242140 7f8689755700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242145 7f8692767700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242141 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 but can_create 2014-06-09 16:22:12.242154 7f868c75b700 15 getattr 3.67d_head/119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 'snapset' 2014-06-09 16:22:12.242160 7f868ff62700 10 op_submit_start 4307 2014-06-09 16:22:12.242163 7f868ff62700 5 queue_transactions (trailing journal) 4307 2014-06-09 16:22:12.242165 7f868ff62700 5 queue_op 0x159961e0 seq 4307 osr(1.233 0x384cc30) 828 bytes (queue has 50 ops and 4689281 bytes) 2014-06-09 16:22:12.242171 7f868af58700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242176 7f8691765700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242181 7f8689f56700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242185 7f868b759700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242190 7f868f761700 2 waiting 51 > 50 ops || 4690105 > 104857600 2014-06-09 16:22:12.242194 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:12.242196 7f868d75d700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242197 7f868c75b700 15 getattr 3.67d_head/119d067d/rbd_data.11806b8b4567.000000000000189d/snapdir//3 'snapset' 2014-06-09 16:22:12.242201 7f868cf5c700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242202 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd68f20] 2014-06-09 16:22:12.242205 7f868e75f700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242206 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd68f20] 2014-06-09 16:22:12.242208 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.242209 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.242209 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242213 7f868ef60700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242212 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc360 (1 active) 2014-06-09 16:22:12.242215 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad680 (2 active) 2014-06-09 16:22:12.242217 7f8690763700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242220 7f868ff62700 10 op_submit_finish 4307 2014-06-09 16:22:12.242221 7f868df5e700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242231 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:12.242224 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) 2014-06-09 16:22:12.242245 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30b62b00 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 2014-06-09 16:22:12.242266 7f868ff62700 10 log is not dirty 2014-06-09 16:22:12.242259 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 2014-06-09 16:22:12.242271 7f868ff62700 5 queue_transactions existing osr(1.3bc 0x384d550)/0x384d550 2014-06-09 16:22:12.242274 7f868ff62700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242270 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 2014-06-09 16:22:12.242279 7f86a5e11700 5 _do_op 0x16096540 seq 4259 osr(3.2df 0x3ed0e40)/0x3ed0e40 start 2014-06-09 16:22:12.242283 7f86a5e11700 10 _do_transaction on 0xc2dd140 2014-06-09 16:22:12.242288 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.242280 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30b62b00 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 rwstate(none n=0 w=0) oi: 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd41c0 snapset: 0=[]:[] 2014-06-09 16:22:12.242297 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 @head oi=119d067d/rbd_data.11806b8b4567.000000000000189d/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) 2014-06-09 16:22:12.242318 7f868c75b700 15 getattr 3.67d_head/119d067d/rbd_data.11806b8b4567.000000000000189d/snapdir//3 '_' 2014-06-09 16:22:12.242324 7f868a757700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242329 7f8692f68700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242333 7f8690f64700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242335 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.242337 7f8691f66700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242342 7f8692767700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242350 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4259.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.242355 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.242363 7f8689755700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242380 7f868c75b700 10 getattr got.size() is 0 2014-06-09 16:22:12.242383 7f86a5e11700 10 _do_op 0x16096540 seq 4259 r = 0, finisher 0xcc950c0 0 2014-06-09 16:22:12.242390 7f86a5e11700 10 _finish_op 0x16096540 seq 4259 osr(3.2df 0x3ed0e40)/0x3ed0e40 2014-06-09 16:22:12.242382 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 119d067d/rbd_data.11806b8b4567.000000000000189d/snapdir//3 and !can_create 2014-06-09 16:22:12.242401 7f8691765700 10 op_submit_start 4308 2014-06-09 16:22:12.242402 7f8691765700 5 queue_transactions (trailing journal) 4308 2014-06-09 16:22:12.242404 7f8691765700 5 queue_op 0x32526780 seq 4308 osr(3.465 0x3ed1580) 828 bytes (queue has 50 ops and 4689281 bytes) 2014-06-09 16:22:12.242408 7f8689f56700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242412 7f868b759700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242416 7f868f761700 2 waiting 51 > 50 ops || 4690105 > 104857600 2014-06-09 16:22:12.242401 7f868c75b700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:12.242395, event: started, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.242420 7f868d75d700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242425 7f868cf5c700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.242420 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30643600 2014-06-09 16:22:12.242430 7f868e75f700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242430 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc950c0] 2014-06-09 16:22:12.242435 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242435 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc950c0] 2014-06-09 16:22:12.242436 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.242438 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.242438 7f868ef60700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242442 7f8690763700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242441 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad680 (1 active) 2014-06-09 16:22:12.242438 7f868c75b700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:12.242433, event: started, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.242445 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77bb00 (2 active) 2014-06-09 16:22:12.242446 7f868df5e700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242450 7f8691765700 10 op_submit_finish 4308 2014-06-09 16:22:12.242451 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32534060] 2014-06-09 16:22:12.242453 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32534060] 2014-06-09 16:22:12.242455 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.242456 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.242446 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] ov 0'0 av 243'685 snapc 0=[] snapset 0=[]:[] 2014-06-09 16:22:12.242453 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.242460 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 2014-06-09 16:22:12.242468 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.242471 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 2014-06-09 16:22:12.242477 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.242487 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.242486 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~217088 2014-06-09 16:22:12.242496 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.242505 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.242504 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0x1078a600 2014-06-09 16:22:12.242516 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 snapset=0x30dd41f8 snapc=0=[] 2014-06-09 16:22:12.242521 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:193 2014-06-09 16:22:12.242527 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag 2014-06-09 16:22:12.242531 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.242537 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 done, snapset=0=[]:[]+head 2014-06-09 16:22:12.242540 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.242549 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.242548 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 0x30643600 op modify 2014-06-09 16:22:12.242555 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.4b9( v 214'558 (0'0,214'558] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) 2014-06-09 16:22:12.242561 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:21.594716 2014-06-09 16:22:12.242573 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.5 activated and committed 2014-06-09 16:22:12.242585 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed 2014-06-09 16:22:12.242587 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 2014-06-09 16:22:12.242597 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info 2014-06-09 16:22:12.242606 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 2014-06-09 16:22:12.242614 7f8691765700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 2014-06-09 16:22:12.242617 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 33 on osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 2014-06-09 16:22:12.242632 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 33 o 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 2014-06-09 16:22:12.242631 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:893 2014-06-09 16:22:12.242649 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.242655 7f868c75b700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:12.242650, event: waiting for subops from 2, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.242654 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.438( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.242675 7f868c75b700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 244 2014-06-09 16:22:12.242690 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.242695 7f8691765700 5 queue_transactions existing osr(1.438 0x384d710)/0x384d710 2014-06-09 16:22:12.242698 7f8691765700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242695 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'684], crt=0'0) [243'685 (0'0) modify 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 by client.4576.0:36701 2014-06-09 10:20:21.594716] 2014-06-09 16:22:12.242726 7f868ff62700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.242717 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 243'685 (0'0,243'685] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 luod=214'684 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'685 (0'0) modify 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 by client.4576.0:36701 2014-06-09 10:20:21.594716 2014-06-09 16:22:12.242735 7f86a5e11700 5 _do_op 0x324e7620 seq 4260 osr(0.4bd 0x2bd76f0)/0x2bd76f0 start 2014-06-09 16:22:12.242739 7f86a5e11700 10 _do_transaction on 0xbfb2d80 2014-06-09 16:22:12.242745 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.242738 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 243'685 (0'0,243'685] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 luod=214'684 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys 2014-06-09 16:22:12.242772 7f868c75b700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'685, trimmed: 2014-06-09 16:22:12.242789 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.242792 7f868c75b700 5 queue_transactions existing osr(3.67d 0x3ed1d70)/0x3ed1d70 2014-06-09 16:22:12.242795 7f868c75b700 2 waiting 51 > 50 ops || 4908767 > 104857600 2014-06-09 16:22:12.242804 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4260.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.242808 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.242831 7f86a5e11700 10 _do_op 0x324e7620 seq 4260 r = 0, finisher 0xbd671a0 0 2014-06-09 16:22:12.242838 7f86a5e11700 10 _finish_op 0x324e7620 seq 4260 osr(0.4bd 0x2bd76f0)/0x2bd76f0 2014-06-09 16:22:12.242853 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd671a0] 2014-06-09 16:22:12.242858 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd671a0] 2014-06-09 16:22:12.242860 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.242861 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.242864 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77bb00 (1 active) 2014-06-09 16:22:12.242867 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f0e0 (2 active) 2014-06-09 16:22:12.242870 7f86a5e11700 5 _do_op 0x2fc8d800 seq 4261 osr(3.750 0x3ed1eb0)/0x3ed1eb0 start 2014-06-09 16:22:12.242873 7f86a5e11700 10 _do_transaction on 0xbe8b2c0 2014-06-09 16:22:12.242878 7f86a5e11700 15 _omap_setkeys meta/1fe0815a/pglog_3.750/0//-1 2014-06-09 16:22:12.242926 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe0815a/pglog_3.750/0//-1 2014-06-09 16:22:12.242942 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.242980 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.242982 7f868a757700 10 op_submit_start 4309 2014-06-09 16:22:12.242985 7f868a757700 5 queue_transactions (trailing journal) 4309 2014-06-09 16:22:12.242986 7f868a757700 5 queue_op 0xc2d54a0 seq 4309 osr(1.10d 0x384c930) 828 bytes (queue has 50 ops and 4689281 bytes) 2014-06-09 16:22:12.242993 7f868a757700 10 op_submit_finish 4309 2014-06-09 16:22:12.242996 7f86a5e11700 15 _omap_rmkeys meta/1fe0815a/pglog_3.750/0//-1 2014-06-09 16:22:12.243002 7f86a5e11700 10 _omap_rmkeys meta/1fe0815a/pglog_3.750/0//-1 r = 0 2014-06-09 16:22:12.242997 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.243008 7f86a5e11700 15 _omap_setkeys meta/1fe0815a/pglog_3.750/0//-1 2014-06-09 16:22:12.243011 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.090474 7 0.000334 2014-06-09 16:22:12.243020 7f86a5e11700 15 _touch 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.243035 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6b6c0] 2014-06-09 16:22:12.243038 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6b6c0] 2014-06-09 16:22:12.243039 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.243040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.243034 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.243046 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.243055 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.243062 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.750_head oid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.243066 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.243074 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.243081 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.243094 7f86a5e11700 20 write_state seq is 30174 2014-06-09 16:22:12.243092 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.243102 7f86a5e11700 20 set_header setting 30173 cid 3.750_head oid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 parent seq 0 2014-06-09 16:22:12.243101 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.243110 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.243116 7f86a5e11700 10 _touch 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 = 0 2014-06-09 16:22:12.243123 7f86a5e11700 15 _write 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 0~450560 2014-06-09 16:22:12.243124 7f868a757700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 2014-06-09 16:22:12.243134 7f86a5e11700 10 file_to_extents done 2014-06-09 16:22:12.243139 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.243172 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.243177 7f868a757700 5 queue_transactions existing osr(1.56c 0x384da70)/0x384da70 2014-06-09 16:22:12.243179 7f868a757700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.243201 7f8692f68700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243206 7f8690f64700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243210 7f8692767700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243214 7f8689755700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243220 7f868af58700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.243227 7f868b759700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243231 7f868f761700 2 waiting 51 > 50 ops || 4690105 > 104857600 2014-06-09 16:22:12.243236 7f868d75d700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.243258 7f8691f66700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.243263 7f868e75f700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243267 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243273 7f868ef60700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243276 7f868cf5c700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.243280 7f868df5e700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243285 7f8690763700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243289 7f868ff62700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.243325 7f868c75b700 2 waiting 51 > 50 ops || 4908767 > 104857600 2014-06-09 16:22:12.243332 7f8689f56700 2 waiting 51 > 50 ops || 4690142 > 104857600 2014-06-09 16:22:12.243338 7f8691765700 2 waiting 51 > 50 ops || 4690109 > 104857600 2014-06-09 16:22:12.245111 7f86a5e11700 10 _generic_write 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 0~450560 = 0 2014-06-09 16:22:12.245145 7f86a5e11700 15 _setattrs 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.245162 7f86a5e11700 10 _setattrs 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 = 0 2014-06-09 16:22:12.245168 7f86a5e11700 15 _setattrs 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 2014-06-09 16:22:12.245176 7f86a5e11700 10 _setattrs 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 = 0 2014-06-09 16:22:12.245251 7f86a5e11700 10 cid: oid: e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 not skipping op, *spos 4261.0.0 2014-06-09 16:22:12.245255 7f86a5e11700 10 > header.spos 0.0.0 2014-06-09 16:22:12.245272 7f86a5e11700 20 set_header setting 30173 cid 3.750_head oid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 parent seq 0 2014-06-09 16:22:12.245341 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4261.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.245347 7f86a5e11700 10 cid: oid: 1fe0815a/pglog_3.750/0//-1 skipping op, spos 4261.0.0 <= header.spos 150418.0.0 2014-06-09 16:22:12.245351 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.247673 7f86a5e11700 10 _do_op 0x2fc8d800 seq 4261 r = 0, finisher 0x30629000 0x30439780 2014-06-09 16:22:12.247685 7f86a5e11700 10 _finish_op 0x2fc8d800 seq 4261 osr(3.750 0x3ed1eb0)/0x3ed1eb0 2014-06-09 16:22:12.247688 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xccf94a0] 2014-06-09 16:22:12.247705 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629000] 2014-06-09 16:22:12.247693 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 32 2014-06-09 16:22:12.247711 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f0e0 (1 active) 2014-06-09 16:22:12.247715 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca49b00 (2 active) 2014-06-09 16:22:12.247718 7f86a5e11700 5 _do_op 0x324f5260 seq 4262 osr(2.13c 0x2ed0610)/0x2ed0610 start 2014-06-09 16:22:12.247721 7f86a5e11700 10 _do_transaction on 0x344c180 2014-06-09 16:22:12.247728 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.247722 7f86a460e700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.247713, event: op_commit, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.247741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xccf94a0] 2014-06-09 16:22:12.247743 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.247744 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.247742 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 32 2014-06-09 16:22:12.247765 7f86a4e0f700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.247758, event: op_applied, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) 2014-06-09 16:22:12.247773 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'519 2014-06-09 16:22:12.247796 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629000] 2014-06-09 16:22:12.247799 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.247800 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.247917 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.247950 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4262.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.247955 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.247974 7f86a5e11700 10 _do_op 0x324f5260 seq 4262 r = 0, finisher 0xbd79480 0 2014-06-09 16:22:12.247980 7f86a5e11700 10 _finish_op 0x324f5260 seq 4262 osr(2.13c 0x2ed0610)/0x2ed0610 2014-06-09 16:22:12.247982 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd440] 2014-06-09 16:22:12.247985 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd440] 2014-06-09 16:22:12.247986 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.247987 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.247991 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd79480] 2014-06-09 16:22:12.248000 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd79480] 2014-06-09 16:22:12.248002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.248003 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.248006 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49b00 (1 active) 2014-06-09 16:22:12.248009 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6360 (2 active) 2014-06-09 16:22:12.248011 7f86a5e11700 5 _do_op 0x32539ec0 seq 4263 osr(3.470 0x3ed1540)/0x3ed1540 start 2014-06-09 16:22:12.248014 7f86a5e11700 10 _do_transaction on 0xc263980 2014-06-09 16:22:12.248020 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.248063 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.248077 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4263.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.248081 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.248094 7f86a5e11700 10 _do_op 0x32539ec0 seq 4263 r = 0, finisher 0xc5a73c0 0 2014-06-09 16:22:12.248102 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe31c80] 2014-06-09 16:22:12.248104 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe31c80] 2014-06-09 16:22:12.248106 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.248107 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.248101 7f86a5e11700 10 _finish_op 0x32539ec0 seq 4263 osr(3.470 0x3ed1540)/0x3ed1540 2014-06-09 16:22:12.248129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a73c0] 2014-06-09 16:22:12.248133 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a73c0] 2014-06-09 16:22:12.248135 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.248136 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.248139 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6360 (1 active) 2014-06-09 16:22:12.248142 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcadb320 (2 active) 2014-06-09 16:22:12.248144 7f86a5e11700 5 _do_op 0x30d969c0 seq 4264 osr(2.185 0x2ed09a0)/0x2ed09a0 start 2014-06-09 16:22:12.248147 7f86a5e11700 10 _do_transaction on 0xc245200 2014-06-09 16:22:12.248153 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.248209 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.248242 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4264.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.248261 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.248279 7f86a5e11700 10 _do_op 0x30d969c0 seq 4264 r = 0, finisher 0x3006c160 0 2014-06-09 16:22:12.248285 7f86a5e11700 10 _finish_op 0x30d969c0 seq 4264 osr(2.185 0x2ed09a0)/0x2ed09a0 2014-06-09 16:22:12.248287 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062a700] 2014-06-09 16:22:12.248295 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006c160] 2014-06-09 16:22:12.248299 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006c160] 2014-06-09 16:22:12.248291 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.248301 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.248302 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.248302 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.248313 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.248322 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062a700] 2014-06-09 16:22:12.248324 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.248325 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.248323 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcadb320 (1 active) 2014-06-09 16:22:12.248339 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcada6c0 (2 active) 2014-06-09 16:22:12.248342 7f86a5e11700 5 _do_op 0xbf7f860 seq 4265 osr(0.179 0x2bd6800)/0x2bd6800 start 2014-06-09 16:22:12.248345 7f86a5e11700 10 _do_transaction on 0x33380c0 2014-06-09 16:22:12.248351 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.248393 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.248407 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4265.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.248412 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.248435 7f86a5e11700 10 _do_op 0xbf7f860 seq 4265 r = 0, finisher 0x3006c720 0 2014-06-09 16:22:12.248442 7f86a5e11700 10 _finish_op 0xbf7f860 seq 4265 osr(0.179 0x2bd6800)/0x2bd6800 2014-06-09 16:22:12.248444 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdba860] 2014-06-09 16:22:12.248447 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdba860] 2014-06-09 16:22:12.248448 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.248449 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.248451 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006c720] 2014-06-09 16:22:12.248455 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006c720] 2014-06-09 16:22:12.248457 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.248458 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.248461 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcada6c0 (1 active) 2014-06-09 16:22:12.248464 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fdc20 (2 active) 2014-06-09 16:22:12.248466 7f86a5e11700 5 _do_op 0xc88be60 seq 4266 osr(1.54a 0x384daf0)/0x384daf0 start 2014-06-09 16:22:12.248469 7f86a5e11700 10 _do_transaction on 0x394efc0 2014-06-09 16:22:12.248474 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.248516 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.248530 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4266.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.248534 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.248535 7f868a757700 10 op_submit_start 4310 2014-06-09 16:22:12.248539 7f868a757700 5 queue_transactions (trailing journal) 4310 2014-06-09 16:22:12.248541 7f868a757700 5 queue_op 0xbe7cfc0 seq 4310 osr(1.56c 0x384da70) 861 bytes (queue has 47 ops and 4234667 bytes) 2014-06-09 16:22:12.248544 7f868a757700 10 op_submit_finish 4310 2014-06-09 16:22:12.248547 7f86a5e11700 10 _do_op 0xc88be60 seq 4266 r = 0, finisher 0xc6e98c0 0 2014-06-09 16:22:12.248548 7f8692767700 10 op_submit_start 4311 2014-06-09 16:22:12.248550 7f8692767700 5 queue_transactions (trailing journal) 4311 2014-06-09 16:22:12.248551 7f8692767700 5 queue_op 0x32538f60 seq 4311 osr(0.512 0x2bd7590) 828 bytes (queue has 49 ops and 4236323 bytes) 2014-06-09 16:22:12.248553 7f86a5e11700 10 _finish_op 0xc88be60 seq 4266 osr(1.54a 0x384daf0)/0x384daf0 2014-06-09 16:22:12.248557 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcf400] 2014-06-09 16:22:12.248559 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcf400] 2014-06-09 16:22:12.248560 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.248562 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.248563 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e98c0] 2014-06-09 16:22:12.248551 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 243/208 242/242/207) 2014-06-09 16:22:12.248567 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e98c0] 2014-06-09 16:22:12.248569 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.248570 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.248573 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fdc20 (1 active) 2014-06-09 16:22:12.248576 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114fc0 (2 active) 2014-06-09 16:22:12.248580 7f86a5e11700 5 _do_op 0x14dd6060 seq 4267 osr(1.301 0x384d030)/0x384d030 start 2014-06-09 16:22:12.248582 7f8692767700 10 op_submit_finish 4311 2014-06-09 16:22:12.248583 7f86a5e11700 10 _do_transaction on 0xc1fc9c0 2014-06-09 16:22:12.248589 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.248593 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.248588 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.286( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) 2014-06-09 16:22:12.248601 7f868a757700 5 queue_transactions existing osr(3.544 0x3ed19a0)/0x3ed19a0 2014-06-09 16:22:12.248604 7f868a757700 10 op_submit_start 4312 2014-06-09 16:22:12.248605 7f868a757700 5 queue_transactions (trailing journal) 4312 2014-06-09 16:22:12.248606 7f868a757700 5 queue_op 0x32539e00 seq 4312 osr(3.544 0x3ed19a0) 828 bytes (queue has 49 ops and 4236323 bytes) 2014-06-09 16:22:12.248609 7f868a757700 10 op_submit_finish 4312 2014-06-09 16:22:12.248619 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.248611 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 243/206 242/242/205) 2014-06-09 16:22:12.248624 7f8692767700 5 queue_transactions existing osr(1.286 0x384ced0)/0x384ced0 2014-06-09 16:22:12.248626 7f8692767700 10 op_submit_start 4313 2014-06-09 16:22:12.248627 7f8692767700 5 queue_transactions (trailing journal) 4313 2014-06-09 16:22:12.248628 7f8692767700 5 queue_op 0x32538f00 seq 4313 osr(1.286 0x384ced0) 828 bytes (queue has 50 ops and 4237151 bytes) 2014-06-09 16:22:12.248631 7f8692767700 10 op_submit_finish 4313 2014-06-09 16:22:12.248630 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.248640 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.248633 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.330( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.248644 7f868a757700 5 queue_transactions existing osr(3.46f 0x3ed1560)/0x3ed1560 2014-06-09 16:22:12.248646 7f868a757700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248647 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4267.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.248651 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248652 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.248659 7f868af58700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.248661 7f8692767700 10 log is not dirty 2014-06-09 16:22:12.248666 7f8692767700 5 queue_transactions existing osr(0.330 0x2bd70a0)/0x2bd70a0 2014-06-09 16:22:12.248667 7f868b759700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.160847 7f86a5610700 0 transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3"}, { "op_num": 1, "op_name": "mkcoll", "collection": "3.27_TEMP"}, { "op_num": 2, "op_name": "remove", "collection": "3.27_TEMP", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3"}, { "op_num": 3, "op_name": "touch", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3"}, { "op_num": 4, "op_name": "omap_setheader", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3", "header_length": "0"}, { "op_num": 5, "op_name": "write", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3", "length": 4194304, "offset": 0, "bufferlist length": 4194304}, { "op_num": 6, "op_name": "omap_setkeys", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3", "attr_lens": {}}, { "op_num": 7, "op_name": "setattrs", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3", "attr_lens": { "_": 257, "snapset": 31}}, { "op_num": 8, "op_name": "omap_setkeys", "collection": "meta", "oid": "16ef7597\/infos\/head\/\/-1", "attr_lens": { "3.27_epoch": 4, "3.27_info": 684}}]} 2014-06-09 16:22:12.248669 7f8692767700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248673 7f8690f64700 10 op_submit_start 4314 2014-06-09 16:22:12.248674 7f8690f64700 5 queue_transactions (trailing journal) 4314 2014-06-09 16:22:12.248673 7f868f761700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.248675 7f8690f64700 5 queue_op 0x30d96e40 seq 4314 osr(3.271 0x3ed0bc0) 828 bytes (queue has 50 ops and 4237151 bytes) 2014-06-09 16:22:12.248678 7f8690f64700 10 op_submit_finish 4314 2014-06-09 16:22:12.248679 7f868d75d700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.248683 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.a6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) 2014-06-09 16:22:12.248697 7f86a5e11700 10 _do_op 0x14dd6060 seq 4267 r = 0, finisher 0x3061f0c0 0 2014-06-09 16:22:12.248704 7f86a5e11700 10 _finish_op 0x14dd6060 seq 4267 osr(1.301 0x384d030)/0x384d030 2014-06-09 16:22:12.248715 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.248714 7f8691f66700 10 op_submit_start 4315 2014-06-09 16:22:12.248715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061f0c0] 2014-06-09 16:22:12.248717 7f8691f66700 5 queue_transactions (trailing journal) 4315 2014-06-09 16:22:12.248720 7f8690f64700 5 queue_transactions existing osr(1.a6 0x384c750)/0x384c750 2014-06-09 16:22:12.248719 7f8691f66700 5 queue_op 0x30d967e0 seq 4315 osr(3.48a 0x3ed1520) 861 bytes (queue has 50 ops and 4237151 bytes) 2014-06-09 16:22:12.248722 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061f0c0] 2014-06-09 16:22:12.248723 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.248724 7f8690f64700 2 waiting 51 > 50 ops || 4237977 > 104857600 2014-06-09 16:22:12.248725 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.248728 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114fc0 (1 active) 2014-06-09 16:22:12.248729 7f868bf5a700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248731 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f680 (2 active) 2014-06-09 16:22:12.248734 7f868a757700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248735 7f86a5e11700 5 _do_op 0xd235c20 seq 4268 osr(1.504 0x384dbd0)/0x384dbd0 start 2014-06-09 16:22:12.248738 7f86a5e11700 10 _do_transaction on 0xc0d83c0 2014-06-09 16:22:12.248740 7f868cf5c700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.248743 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.248732 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc82a0] 2014-06-09 16:22:12.248745 7f8691f66700 10 op_submit_finish 4315 2014-06-09 16:22:12.248747 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.248751 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.248759 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.248765 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.194473 8 0.009393 2014-06-09 16:22:12.248768 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 2014-06-09 16:22:12.248776 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc82a0] 2014-06-09 16:22:12.248777 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.248778 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.248782 7f868ef60700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248778 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.248784 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.248789 7f8690763700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248794 7f8692f68700 10 op_submit_start 4316 2014-06-09 16:22:12.248796 7f8692f68700 5 queue_transactions (trailing journal) 4316 2014-06-09 16:22:12.248789 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.248796 7f868ff62700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248797 7f8692f68700 5 queue_op 0xcc87f80 seq 4316 osr(3.1e1 0x3ed09b0) 828 bytes (queue has 50 ops and 4237151 bytes) 2014-06-09 16:22:12.248800 7f8692f68700 10 op_submit_finish 4316 2014-06-09 16:22:12.248800 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 2014-06-09 16:22:12.248809 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4268.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.248803 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt 2014-06-09 16:22:12.248813 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.248810 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.248818 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.248824 7f86a5e11700 10 _do_op 0xd235c20 seq 4268 r = 0, finisher 0x30629c60 0 2014-06-09 16:22:12.248818 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops 2014-06-09 16:22:12.248830 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.248824 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.248830 7f86a5e11700 10 _finish_op 0xd235c20 seq 4268 osr(1.504 0x384dbd0)/0x384dbd0 2014-06-09 16:22:12.248841 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629c60] 2014-06-09 16:22:12.248836 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.248834 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.778( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.248845 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629c60] 2014-06-09 16:22:12.248847 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.248848 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.248851 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f680 (1 active) 2014-06-09 16:22:12.248846 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.248854 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653680 (2 active) 2014-06-09 16:22:12.248856 7f86a5e11700 5 _do_op 0x324f40c0 seq 4269 osr(3.111 0x3ed0830)/0x3ed0830 start 2014-06-09 16:22:12.248859 7f86a5e11700 10 _do_transaction on 0xc259380 2014-06-09 16:22:12.248856 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.248865 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.248865 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.248870 7f8692f68700 5 queue_transactions existing osr(2.778 0x3ed0050)/0x3ed0050 2014-06-09 16:22:12.248872 7f8692f68700 10 op_submit_start 4317 2014-06-09 16:22:12.248873 7f8692f68700 5 queue_transactions (trailing journal) 4317 2014-06-09 16:22:12.248872 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.248874 7f8692f68700 5 queue_op 0xcc877a0 seq 4317 osr(2.778 0x3ed0050) 828 bytes (queue has 50 ops and 4237151 bytes) 2014-06-09 16:22:12.248876 7f8692f68700 10 op_submit_finish 4317 2014-06-09 16:22:12.248879 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 243/203 242/242/202) 2014-06-09 16:22:12.248892 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.248908 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.248912 7f8692f68700 5 queue_transactions existing osr(3.516 0x3ed16d0)/0x3ed16d0 2014-06-09 16:22:12.248914 7f8692f68700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248919 7f868df5e700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248920 7f8691f66700 10 log is not dirty 2014-06-09 16:22:12.248924 7f8691f66700 5 queue_transactions existing osr(0.221 0x2bd6a50)/0x2bd6a50 2014-06-09 16:22:12.248925 7f8690f64700 2 waiting 51 > 50 ops || 4237977 > 104857600 2014-06-09 16:22:12.248928 7f8691f66700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.248932 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94120] 2014-06-09 16:22:12.248934 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94120] 2014-06-09 16:22:12.248934 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248936 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.248937 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.248940 7f868af58700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.248947 7f868b759700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248952 7f8692767700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248958 7f868f761700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.248964 7f868d75d700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.248969 7f868e75f700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.248976 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.248991 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4269.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.248996 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.249016 7f86a5e11700 10 _do_op 0x324f40c0 seq 4269 r = 0, finisher 0x3062a120 0 2014-06-09 16:22:12.249023 7f86a5e11700 10 _finish_op 0x324f40c0 seq 4269 osr(3.111 0x3ed0830)/0x3ed0830 2014-06-09 16:22:12.249024 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xccf87e0] 2014-06-09 16:22:12.249026 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xccf87e0] 2014-06-09 16:22:12.249027 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.249028 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.249033 7f8692f68700 10 op_submit_start 4318 2014-06-09 16:22:12.249034 7f8692f68700 5 queue_transactions (trailing journal) 4318 2014-06-09 16:22:12.249034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062a120] 2014-06-09 16:22:12.249035 7f8692f68700 5 queue_op 0xcc86000 seq 4318 osr(3.516 0x3ed16d0) 828 bytes (queue has 50 ops and 4237151 bytes) 2014-06-09 16:22:12.249038 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062a120] 2014-06-09 16:22:12.249039 7f868a757700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249040 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.249041 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.249044 7f868cf5c700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.249044 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653680 (1 active) 2014-06-09 16:22:12.249048 7f868ef60700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249047 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62ec60 (2 active) 2014-06-09 16:22:12.249051 7f8690763700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249056 7f8692f68700 10 op_submit_finish 4318 2014-06-09 16:22:12.249056 7f868ff62700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249061 7f8689f56700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.249052 7f86a5e11700 5 _do_op 0x324e73e0 seq 4270 osr(3.9f 0x3ed0600)/0x3ed0600 start 2014-06-09 16:22:12.249067 7f86a5e11700 10 _do_transaction on 0xc140e40 2014-06-09 16:22:12.249067 7f868bf5a700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249059 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 243/212 242/242/211) 2014-06-09 16:22:12.249073 7f868c75b700 2 waiting 51 > 50 ops || 4456637 > 104857600 2014-06-09 16:22:12.249074 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.249088 7f8692f68700 10 log is not dirty 2014-06-09 16:22:12.249093 7f8692f68700 5 queue_transactions existing osr(3.292 0x3ed0ba0)/0x3ed0ba0 2014-06-09 16:22:12.249096 7f8692f68700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249099 7f8691765700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249105 7f868df5e700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249109 7f8690f64700 2 waiting 51 > 50 ops || 4237977 > 104857600 2014-06-09 16:22:12.249112 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249118 7f868af58700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.249125 7f868b759700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249129 7f8692767700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249129 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.249135 7f868f761700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.249140 7f868d75d700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.249144 7f868e75f700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249156 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4270.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.249160 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.249171 7f86a5e11700 10 _do_op 0x324e73e0 seq 4270 r = 0, finisher 0xbd6c0e0 0 2014-06-09 16:22:12.249177 7f86a5e11700 10 _finish_op 0x324e73e0 seq 4270 osr(3.9f 0x3ed0600)/0x3ed0600 2014-06-09 16:22:12.249178 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd709e0] 2014-06-09 16:22:12.249180 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd709e0] 2014-06-09 16:22:12.249181 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.249182 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.249186 7f868a757700 10 op_submit_start 4319 2014-06-09 16:22:12.249188 7f868a757700 5 queue_transactions (trailing journal) 4319 2014-06-09 16:22:12.249188 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6c0e0] 2014-06-09 16:22:12.249189 7f868a757700 5 queue_op 0x32539da0 seq 4319 osr(3.46f 0x3ed1560) 828 bytes (queue has 50 ops and 4237153 bytes) 2014-06-09 16:22:12.249193 7f868cf5c700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249193 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6c0e0] 2014-06-09 16:22:12.249195 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.249196 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.249197 7f868ef60700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249199 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62ec60 (1 active) 2014-06-09 16:22:12.249201 7f8690763700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249205 7f868ff62700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249210 7f8689f56700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249202 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a360 (2 active) 2014-06-09 16:22:12.249215 7f868bf5a700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249217 7f86a5e11700 5 _do_op 0x1e08b9e0 seq 4271 osr(0.1da 0x2bd6920)/0x2bd6920 start 2014-06-09 16:22:12.249221 7f868a757700 10 op_submit_finish 4319 2014-06-09 16:22:12.249221 7f86a5e11700 10 _do_transaction on 0xbfe40c0 2014-06-09 16:22:12.249221 7f8691f66700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249225 7f868c75b700 2 waiting 51 > 50 ops || 4456639 > 104857600 2014-06-09 16:22:12.249226 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.249223 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.4b9( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.249253 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.249257 7f868a757700 5 queue_transactions existing osr(2.4b9 0x2ed1730)/0x2ed1730 2014-06-09 16:22:12.249259 7f868a757700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249263 7f8691765700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249267 7f868df5e700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249266 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.249271 7f8690f64700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249274 7f8689755700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249278 7f8692f68700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249283 7f868b759700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249289 7f8692767700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249294 7f868f761700 2 waiting 51 > 50 ops || 4237977 > 104857600 2014-06-09 16:22:12.249299 7f868d75d700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249304 7f868e75f700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249309 7f868af58700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249305 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4271.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.249316 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.249327 7f86a5e11700 10 _do_op 0x1e08b9e0 seq 4271 r = 0, finisher 0x30509480 0 2014-06-09 16:22:12.249333 7f86a5e11700 10 _finish_op 0x1e08b9e0 seq 4271 osr(0.1da 0x2bd6920)/0x2bd6920 2014-06-09 16:22:12.249334 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2660] 2014-06-09 16:22:12.249336 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2660] 2014-06-09 16:22:12.249337 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.249338 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.249342 7f868cf5c700 10 op_submit_start 4320 2014-06-09 16:22:12.249344 7f868cf5c700 5 queue_transactions (trailing journal) 4320 2014-06-09 16:22:12.249344 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30509480] 2014-06-09 16:22:12.249345 7f868cf5c700 5 queue_op 0x324e6360 seq 4320 osr(2.573 0x2ed18b0) 861 bytes (queue has 50 ops and 4237186 bytes) 2014-06-09 16:22:12.249348 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30509480] 2014-06-09 16:22:12.249349 7f868ef60700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249350 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.249351 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.249353 7f8690763700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249354 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a360 (1 active) 2014-06-09 16:22:12.249358 7f868ff62700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249363 7f8689f56700 2 waiting 51 > 50 ops || 4238047 > 104857600 2014-06-09 16:22:12.249367 7f868bf5a700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249368 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5c20 (2 active) 2014-06-09 16:22:12.249373 7f868cf5c700 10 op_submit_finish 4320 2014-06-09 16:22:12.249372 7f86a5e11700 5 _do_op 0x324f50e0 seq 4272 osr(0.640 0x2bd7c10)/0x2bd7c10 start 2014-06-09 16:22:12.249376 7f86a5e11700 10 _do_transaction on 0x32fa780 2014-06-09 16:22:12.249381 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.249377 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.249389 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.073976 7 0.000316 2014-06-09 16:22:12.249399 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.249408 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.249421 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.249417 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.249427 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.249435 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4272.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.249440 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.249435 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.249442 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.249450 7f86a5e11700 10 _do_op 0x324f50e0 seq 4272 r = 0, finisher 0xbef1b00 0 2014-06-09 16:22:12.249456 7f86a5e11700 10 _finish_op 0x324f50e0 seq 4272 osr(0.640 0x2bd7c10)/0x2bd7c10 2014-06-09 16:22:12.249452 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) 2014-06-09 16:22:12.249467 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef1b00] 2014-06-09 16:22:12.249463 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) 2014-06-09 16:22:12.249471 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef1b00] 2014-06-09 16:22:12.249473 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.249474 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.249472 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.249477 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5c20 (1 active) 2014-06-09 16:22:12.249480 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd7a0 (2 active) 2014-06-09 16:22:12.249482 7f86a5e11700 5 _do_op 0xc2d48a0 seq 4273 osr(3.672 0x3ed1350)/0x3ed1350 start 2014-06-09 16:22:12.249485 7f86a5e11700 10 _do_transaction on 0x2c80f00 2014-06-09 16:22:12.249484 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 2014-06-09 16:22:12.249490 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.249499 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 2014-06-09 16:22:12.249529 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.249533 7f868cf5c700 5 queue_transactions existing osr(2.6dd 0x2ed1e50)/0x2ed1e50 2014-06-09 16:22:12.249536 7f868cf5c700 10 op_submit_start 4321 2014-06-09 16:22:12.249537 7f868cf5c700 5 queue_transactions (trailing journal) 4321 2014-06-09 16:22:12.249538 7f868cf5c700 5 queue_op 0x2e20c00 seq 4321 osr(2.6dd 0x2ed1e50) 861 bytes (queue has 50 ops and 4237219 bytes) 2014-06-09 16:22:12.249541 7f868cf5c700 10 op_submit_finish 4321 2014-06-09 16:22:12.249543 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.55b( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) 2014-06-09 16:22:12.249559 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.249573 7f868cf5c700 10 log is not dirty 2014-06-09 16:22:12.249577 7f868cf5c700 5 queue_transactions existing osr(2.55b 0x2ed1930)/0x2ed1930 2014-06-09 16:22:12.249576 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4273.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.249580 7f868cf5c700 2 waiting 51 > 50 ops || 4238047 > 104857600 2014-06-09 16:22:12.249581 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.249584 7f868ef60700 2 waiting 51 > 50 ops || 4238047 > 104857600 2014-06-09 16:22:12.249589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314566e0] 2014-06-09 16:22:12.249591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314566e0] 2014-06-09 16:22:12.249590 7f8691f66700 2 waiting 51 > 50 ops || 4238080 > 104857600 2014-06-09 16:22:12.249592 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.249593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.249595 7f868a757700 2 waiting 51 > 50 ops || 4238047 > 104857600 2014-06-09 16:22:12.249600 7f8691765700 2 waiting 51 > 50 ops || 4238047 > 104857600 2014-06-09 16:22:12.249600 7f86a5e11700 10 _do_op 0xc2d48a0 seq 4273 r = 0, finisher 0x324767c0 0 2014-06-09 16:22:12.249603 7f868df5e700 2 waiting 51 > 50 ops || 4238047 > 104857600 2014-06-09 16:22:12.249606 7f86a5e11700 10 _finish_op 0xc2d48a0 seq 4273 osr(3.672 0x3ed1350)/0x3ed1350 2014-06-09 16:22:12.249607 7f8690f64700 2 waiting 51 > 50 ops || 4238045 > 104857600 2014-06-09 16:22:12.249616 7f8689755700 10 op_submit_start 4322 2014-06-09 16:22:12.249617 7f8689755700 5 queue_transactions (trailing journal) 4322 2014-06-09 16:22:12.249618 7f868b759700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249619 7f8689755700 5 queue_op 0x33a2720 seq 4322 osr(0.1d4 0x2bd6940) 828 bytes (queue has 50 ops and 4237186 bytes) 2014-06-09 16:22:12.249624 7f8692767700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249627 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324767c0] 2014-06-09 16:22:12.249629 7f8692f68700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324767c0] 2014-06-09 16:22:12.249633 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.249633 7f868d75d700 2 waiting 51 > 50 ops || 4238047 > 104857600 2014-06-09 16:22:12.249634 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.249637 7f868e75f700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249641 7f868cf5c700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249638 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd7a0 (1 active) 2014-06-09 16:22:12.249644 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc6c0 (2 active) 2014-06-09 16:22:12.249645 7f8690763700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249650 7f8689755700 10 op_submit_finish 4322 2014-06-09 16:22:12.249650 7f868ff62700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249655 7f8689f56700 2 waiting 51 > 50 ops || 4238047 > 104857600 2014-06-09 16:22:12.249659 7f86a5e11700 5 _do_op 0x16fd4900 seq 4274 osr(2.102 0x2ed0710)/0x2ed0710 start 2014-06-09 16:22:12.249663 7f86a5e11700 10 _do_transaction on 0x3472780 2014-06-09 16:22:12.249653 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) 2014-06-09 16:22:12.249669 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.249687 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.249691 7f8689755700 5 queue_transactions existing osr(1.fa 0x384c970)/0x384c970 2014-06-09 16:22:12.249693 7f8689755700 2 waiting 51 > 50 ops || 4238012 > 104857600 2014-06-09 16:22:12.249697 7f868bf5a700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249702 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32476580] 2014-06-09 16:22:12.249710 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.249704 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.249715 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.249726 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4274.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.249730 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.249724 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1013 2014-06-09 16:22:12.249733 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32476580] 2014-06-09 16:22:12.249734 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.249735 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.249747 7f86a5e11700 10 _do_op 0x16fd4900 seq 4274 r = 0, finisher 0x1b0c19c0 0 2014-06-09 16:22:12.249753 7f86a5e11700 10 _finish_op 0x16fd4900 seq 4274 osr(2.102 0x2ed0710)/0x2ed0710 2014-06-09 16:22:12.249754 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3281af60] 2014-06-09 16:22:12.249757 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.249767 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1b0c19c0] 2014-06-09 16:22:12.249765 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.249771 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1b0c19c0] 2014-06-09 16:22:12.249775 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.249776 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.249773 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 2014-06-09 16:22:12.249779 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc6c0 (1 active) 2014-06-09 16:22:12.249781 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3281af60] 2014-06-09 16:22:12.249782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.249782 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0120 (2 active) 2014-06-09 16:22:12.249783 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.249784 7f86a5e11700 5 _do_op 0x2e21aa0 seq 4275 osr(0.57b 0x2bd77b0)/0x2bd77b0 start 2014-06-09 16:22:12.249787 7f86a5e11700 10 _do_transaction on 0xc0d58c0 2014-06-09 16:22:12.249788 7f868ef60700 10 op_submit_start 4323 2014-06-09 16:22:12.249790 7f868ef60700 5 queue_transactions (trailing journal) 4323 2014-06-09 16:22:12.249791 7f868ef60700 5 queue_op 0xcdaa9c0 seq 4323 osr(2.69c 0x2ed1f20) 828 bytes (queue has 50 ops and 4237153 bytes) 2014-06-09 16:22:12.249792 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.249794 7f868ef60700 10 op_submit_finish 4323 2014-06-09 16:22:12.249797 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.220( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) 2014-06-09 16:22:12.249827 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.249831 7f868ef60700 5 queue_transactions existing osr(2.220 0x2ed0b10)/0x2ed0b10 2014-06-09 16:22:12.249831 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.249833 7f868ef60700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249837 7f8691f66700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249841 7f868a757700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249845 7f8691765700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249845 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4275.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.249848 7f868df5e700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249849 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.249852 7f8690f64700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249857 7f868f761700 2 waiting 51 > 50 ops || 4237977 > 104857600 2014-06-09 16:22:12.249863 7f868af58700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249868 7f8692767700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249873 7f8692f68700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249876 7f868d75d700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249876 7f86a5e11700 10 _do_op 0x2e21aa0 seq 4275 r = 0, finisher 0x2d739160 0 2014-06-09 16:22:12.249881 7f868e75f700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249882 7f86a5e11700 10 _finish_op 0x2e21aa0 seq 4275 osr(0.57b 0x2bd77b0)/0x2bd77b0 2014-06-09 16:22:12.249884 7f868cf5c700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249888 7f8690763700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249892 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc168520] 2014-06-09 16:22:12.249894 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc168520] 2014-06-09 16:22:12.249896 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.249895 7f868ff62700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249897 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.249900 7f8689f56700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249905 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249909 7f868bf5a700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249913 7f868b759700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249925 7f868ef60700 10 op_submit_start 4324 2014-06-09 16:22:12.249927 7f868ef60700 5 queue_transactions (trailing journal) 4324 2014-06-09 16:22:12.249928 7f868ef60700 5 queue_op 0xcdaad20 seq 4324 osr(2.220 0x2ed0b10) 828 bytes (queue has 50 ops and 4237153 bytes) 2014-06-09 16:22:12.249930 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d739160] 2014-06-09 16:22:12.249932 7f868a757700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249934 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d739160] 2014-06-09 16:22:12.249936 7f8691765700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249936 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.249937 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.249939 7f868df5e700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249940 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0120 (1 active) 2014-06-09 16:22:12.249943 7f8690f64700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249943 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c0900 (2 active) 2014-06-09 16:22:12.249947 7f8691f66700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249950 7f868f761700 2 waiting 51 > 50 ops || 4237977 > 104857600 2014-06-09 16:22:12.249952 7f868ef60700 10 op_submit_finish 4324 2014-06-09 16:22:12.249956 7f868af58700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.249961 7f86a5e11700 5 _do_op 0xc88bec0 seq 4276 osr(3.48d 0x3ed14e0)/0x3ed14e0 start 2014-06-09 16:22:12.249964 7f86a5e11700 10 _do_transaction on 0xc262b40 2014-06-09 16:22:12.249955 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) 2014-06-09 16:22:12.249969 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.249985 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.249989 7f868ef60700 5 queue_transactions existing osr(2.f2 0x2ed07e0)/0x2ed07e0 2014-06-09 16:22:12.249991 7f868ef60700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.249994 7f8692767700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.249999 7f868d75d700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.250003 7f868e75f700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250007 7f8692f68700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250010 7f8690763700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250010 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.250014 7f868cf5c700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250019 7f8689f56700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.250025 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.250030 7f868bf5a700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250033 7f868b759700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250037 7f868ff62700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250042 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4276.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.250046 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.250058 7f86a5e11700 10 _do_op 0xc88bec0 seq 4276 r = 0, finisher 0xbd7d420 0 2014-06-09 16:22:12.250064 7f86a5e11700 10 _finish_op 0xc88bec0 seq 4276 osr(3.48d 0x3ed14e0)/0x3ed14e0 2014-06-09 16:22:12.250066 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc168d40] 2014-06-09 16:22:12.250068 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc168d40] 2014-06-09 16:22:12.250070 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.250071 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.250075 7f868a757700 10 op_submit_start 4325 2014-06-09 16:22:12.250076 7f868a757700 5 queue_transactions (trailing journal) 4325 2014-06-09 16:22:12.250075 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7d420] 2014-06-09 16:22:12.250077 7f868a757700 5 queue_op 0x32539d40 seq 4325 osr(2.4b9 0x2ed1730) 828 bytes (queue has 50 ops and 4237153 bytes) 2014-06-09 16:22:12.250081 7f8691765700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250081 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7d420] 2014-06-09 16:22:12.250084 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.250085 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.250085 7f868df5e700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250078 7f86a5610700 -1 os/KeyValueStore.cc: In function 'unsigned int KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)' thread 7f86a5610700 time 2014-06-09 16:22:12.248673 os/KeyValueStore.cc: 1524: FAILED assert(0 == "unexpected error") ceph version 0.80-820-g5d606cd (5d606cd0d00698699c91a378a1bd9f71cc8a77c9) 1: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x750) [0x9e6f20] 2: (KeyValueStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9e8d1e] 3: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x9a) [0x9e8e2a] 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb62a3a] 5: (ThreadPool::WorkThread::entry()+0x10) [0xb63c90] 6: (()+0x7e9a) [0x7f86aed2de9a] 7: (clone()+0x6d) [0x7f86ad2d8ccd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2014-06-09 16:22:12.250088 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c0900 (1 active) 2014-06-09 16:22:12.250089 7f8690f64700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.250090 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406900 (2 active) 2014-06-09 16:22:12.250093 7f8691f66700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.250098 7f868a757700 10 op_submit_finish 4325 2014-06-09 16:22:12.250098 7f868f761700 2 waiting 51 > 50 ops || 4237977 > 104857600 2014-06-09 16:22:12.250103 7f868af58700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.250094 7f86a5e11700 5 _do_op 0xd471ce0 seq 4277 osr(0.629 0x2bd7c90)/0x2bd7c90 start 2014-06-09 16:22:12.250109 7f86a5e11700 10 _do_transaction on 0xc28cf00 2014-06-09 16:22:12.250101 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.33a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) 2014-06-09 16:22:12.250116 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.250130 7f868a757700 10 log is not dirty 2014-06-09 16:22:12.250134 7f868a757700 5 queue_transactions existing osr(0.33a 0x2bd70b0)/0x2bd70b0 2014-06-09 16:22:12.250136 7f868a757700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250140 7f868ef60700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.250145 7f8692767700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250148 7f868d75d700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.250152 7f868e75f700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250156 7f8692f68700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250156 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.250160 7f8690763700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250163 7f868cf5c700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250169 7f8689f56700 2 waiting 51 > 50 ops || 4238014 > 104857600 2014-06-09 16:22:12.250174 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.250179 7f868bf5a700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250178 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4277.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.250182 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.250182 7f868b759700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250187 7f868ff62700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250198 7f86a5e11700 10 _do_op 0xd471ce0 seq 4277 r = 0, finisher 0xbd6fda0 0 2014-06-09 16:22:12.250203 7f86a5e11700 10 _finish_op 0xd471ce0 seq 4277 osr(0.629 0x2bd7c90)/0x2bd7c90 2014-06-09 16:22:12.250205 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2e0e9a0] 2014-06-09 16:22:12.250213 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6fda0] 2014-06-09 16:22:12.250208 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.250218 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6fda0] 2014-06-09 16:22:12.250219 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.250220 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.250223 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406900 (1 active) 2014-06-09 16:22:12.250218 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.250225 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9238c0 (2 active) 2014-06-09 16:22:12.250227 7f86a5e11700 5 _do_op 0xc88a9c0 seq 4278 osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 start 2014-06-09 16:22:12.250230 7f86a5e11700 10 _do_transaction on 0xbf7dc80 2014-06-09 16:22:12.250227 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 2014-06-09 16:22:12.250234 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.250236 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2e0e9a0] 2014-06-09 16:22:12.250238 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.250239 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.250243 7f8691765700 10 op_submit_start 4326 2014-06-09 16:22:12.250245 7f8691765700 5 queue_transactions (trailing journal) 4326 2014-06-09 16:22:12.250246 7f8691765700 5 queue_op 0x325266c0 seq 4326 osr(1.438 0x384d710) 828 bytes (queue has 50 ops and 4237120 bytes) 2014-06-09 16:22:12.250249 7f8691765700 10 op_submit_finish 4326 2014-06-09 16:22:12.250252 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.250288 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.250292 7f8691765700 5 queue_transactions existing osr(3.1b5 0x3ed0a80)/0x3ed0a80 2014-06-09 16:22:12.250295 7f8691765700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250299 7f868df5e700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250303 7f8690f64700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250307 7f8691f66700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250312 7f868f761700 2 waiting 51 > 50 ops || 4237944 > 104857600 2014-06-09 16:22:12.250318 7f868af58700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250323 7f868a757700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250327 7f868ef60700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250331 7f8692767700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250335 7f868d75d700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250339 7f868e75f700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250340 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.250342 7f8692f68700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250346 7f8690763700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250350 7f868cf5c700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250355 7f8689f56700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250360 7f8689755700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250361 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4278.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.250364 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.250365 7f868bf5a700 2 waiting 51 > 50 ops || 4237948 > 104857600 --- begin dump of recent events --- -10930> 2014-06-09 16:22:12.157785 7f86a5e11700 5 submit_transaction r = 0 -10929> 2014-06-09 16:22:12.157799 7f86a5e11700 10 _do_op 0x2fc8cf60 seq 4080 r = 0, finisher 0x1322c400 0 -10928> 2014-06-09 16:22:12.157812 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32075720] -10927> 2014-06-09 16:22:12.157815 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery -10926> 2014-06-09 16:22:12.157828 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays -10925> 2014-06-09 16:22:12.157838 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.5d1( v 213'644 (0'0,213'644] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1096 -10924> 2014-06-09 16:22:12.157849 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32075720] -10923> 2014-06-09 16:22:12.157851 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10922> 2014-06-09 16:22:12.157852 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10921> 2014-06-09 16:22:12.157861 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd75ac0] -10920> 2014-06-09 16:22:12.157863 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -10919> 2014-06-09 16:22:12.157872 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -10918> 2014-06-09 16:22:12.157881 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.28a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 -10917> 2014-06-09 16:22:12.157890 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd75ac0] -10916> 2014-06-09 16:22:12.157892 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10915> 2014-06-09 16:22:12.157893 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10914> 2014-06-09 16:22:12.157896 7f86a5610700 10 _finish_op 0x30061f20 seq 4079 osr(2.28a 0x2ed0d20)/0x2ed0d20 -10913> 2014-06-09 16:22:12.157908 7f868bf5a700 10 op_submit_start 4129 -10912> 2014-06-09 16:22:12.157909 7f868bf5a700 5 queue_transactions (trailing journal) 4129 -10911> 2014-06-09 16:22:12.157911 7f868bf5a700 5 queue_op 0x230b9d40 seq 4129 osr(1.2a3 0x384d1f0) 828 bytes (queue has 50 ops and 4237393 bytes) -10910> 2014-06-09 16:22:12.157916 7f8690763700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10909> 2014-06-09 16:22:12.157920 7f868b759700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10908> 2014-06-09 16:22:12.157925 7f868d75d700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10907> 2014-06-09 16:22:12.157929 7f868ff62700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10906> 2014-06-09 16:22:12.157934 7f8692767700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10905> 2014-06-09 16:22:12.157940 7f868e75f700 2 waiting 51 > 50 ops || 4238254 > 104857600 -10904> 2014-06-09 16:22:12.157944 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5d860] -10903> 2014-06-09 16:22:12.157948 7f8691765700 2 waiting 51 > 50 ops || 4238251 > 104857600 -10902> 2014-06-09 16:22:12.157948 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5d860] -10901> 2014-06-09 16:22:12.157950 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10900> 2014-06-09 16:22:12.157951 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10899> 2014-06-09 16:22:12.157952 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6525a0 (1 active) -10898> 2014-06-09 16:22:12.157953 7f8689755700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10897> 2014-06-09 16:22:12.157955 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc1158c0 (2 active) -10896> 2014-06-09 16:22:12.157959 7f868ef60700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10895> 2014-06-09 16:22:12.157959 7f86a5610700 5 _do_op 0x2fc8c660 seq 4081 osr(2.65c 0x2ed1fb0)/0x2ed1fb0 start -10894> 2014-06-09 16:22:12.157961 7f868bf5a700 10 op_submit_finish 4129 -10893> 2014-06-09 16:22:12.157962 7f86a5610700 10 _do_transaction on 0xc31cb40 -10892> 2014-06-09 16:22:12.157963 7f8692f68700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10891> 2014-06-09 16:22:12.157967 7f868df5e700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10890> 2014-06-09 16:22:12.157968 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10889> 2014-06-09 16:22:12.157972 7f868cf5c700 2 waiting 51 > 50 ops || 4238248 > 104857600 -10888> 2014-06-09 16:22:12.157965 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -10887> 2014-06-09 16:22:12.157976 7f868a757700 2 waiting 51 > 50 ops || 4238254 > 104857600 -10886> 2014-06-09 16:22:12.157981 7f868c75b700 2 waiting 51 > 50 ops || 4238254 > 104857600 -10885> 2014-06-09 16:22:12.157985 7f868af58700 2 waiting 51 > 50 ops || 4238254 > 104857600 -10884> 2014-06-09 16:22:12.157979 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local -10883> 2014-06-09 16:22:12.157989 7f8690f64700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10882> 2014-06-09 16:22:12.157987 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -10881> 2014-06-09 16:22:12.157994 7f8691f66700 2 waiting 51 > 50 ops || 4238254 > 104857600 -10880> 2014-06-09 16:22:12.157999 7f8689f56700 2 waiting 51 > 50 ops || 4238254 > 104857600 -10879> 2014-06-09 16:22:12.157995 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10878> 2014-06-09 16:22:12.158003 7f868f761700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10877> 2014-06-09 16:22:12.158007 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10876> 2014-06-09 16:22:12.158003 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -10875> 2014-06-09 16:22:12.158011 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -10874> 2014-06-09 16:22:12.158022 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4081.0.0 <= header.spos 150443.0.0 -10873> 2014-06-09 16:22:12.158025 7f86a5610700 5 submit_transaction r = 0 -10872> 2014-06-09 16:22:12.158024 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 -10871> 2014-06-09 16:22:12.158033 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -10870> 2014-06-09 16:22:12.158039 7f86a5610700 10 _do_op 0x2fc8c660 seq 4081 r = 0, finisher 0xbd7ba00 0 -10869> 2014-06-09 16:22:12.158040 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.746( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop -10868> 2014-06-09 16:22:12.158047 7f86a5e11700 10 _finish_op 0x2fc8cf60 seq 4080 osr(3.5d1 0x3ed1b70)/0x3ed1b70 -10867> 2014-06-09 16:22:12.158049 7f868bf5a700 10 log is not dirty -10866> 2014-06-09 16:22:12.158057 7f868b759700 10 op_submit_start 4130 -10865> 2014-06-09 16:22:12.158058 7f868b759700 5 queue_transactions (trailing journal) 4130 -10864> 2014-06-09 16:22:12.158059 7f868b759700 5 queue_op 0xc88b4a0 seq 4130 osr(0.4c8 0x2bd7670) 828 bytes (queue has 50 ops and 4237360 bytes) -10863> 2014-06-09 16:22:12.158064 7f868d75d700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10862> 2014-06-09 16:22:12.158054 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -10861> 2014-06-09 16:22:12.158068 7f868ff62700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10860> 2014-06-09 16:22:12.158072 7f8692767700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10859> 2014-06-09 16:22:12.158068 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -10858> 2014-06-09 16:22:12.158077 7f868e75f700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10857> 2014-06-09 16:22:12.158081 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322c400] -10856> 2014-06-09 16:22:12.158083 7f8691765700 2 waiting 51 > 50 ops || 4238218 > 104857600 -10855> 2014-06-09 16:22:12.158086 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322c400] -10854> 2014-06-09 16:22:12.158077 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x14053a00,0x16059f00,0x16059e00,0x16059d00,0x16059c00,0x1804d800 -10853> 2014-06-09 16:22:12.158087 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10852> 2014-06-09 16:22:12.158088 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10851> 2014-06-09 16:22:12.158088 7f8689755700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10850> 2014-06-09 16:22:12.158091 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49200 (1 active) -10849> 2014-06-09 16:22:12.158092 7f868ef60700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10848> 2014-06-09 16:22:12.158093 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34965a0 (2 active) -10847> 2014-06-09 16:22:12.158097 7f8692f68700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10846> 2014-06-09 16:22:12.158098 7f868b759700 10 op_submit_finish 4130 -10845> 2014-06-09 16:22:12.158101 7f868df5e700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10844> 2014-06-09 16:22:12.158106 7f868cf5c700 2 waiting 51 > 50 ops || 4238215 > 104857600 -10843> 2014-06-09 16:22:12.158103 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -10842> 2014-06-09 16:22:12.158101 7f868b759700 10 osd.26 pg_epoch: 243 pg[0.9d( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.9d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -10841> 2014-06-09 16:22:12.158110 7f868a757700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10840> 2014-06-09 16:22:12.158115 7f868c75b700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10839> 2014-06-09 16:22:12.158111 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -10838> 2014-06-09 16:22:12.158119 7f868af58700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10837> 2014-06-09 16:22:12.158124 7f8690f64700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10836> 2014-06-09 16:22:12.158119 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -10835> 2014-06-09 16:22:12.158128 7f8691f66700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10834> 2014-06-09 16:22:12.158132 7f8689f56700 2 waiting 51 > 50 ops || 4238221 > 104857600 -10833> 2014-06-09 16:22:12.158134 7f868b759700 10 log is not dirty -10832> 2014-06-09 16:22:12.158137 7f868f761700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10831> 2014-06-09 16:22:12.158131 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1030 -10830> 2014-06-09 16:22:12.158139 7f868b759700 5 queue_transactions existing osr(0.9d 0x2bd6490)/0x2bd6490 -10829> 2014-06-09 16:22:12.158141 7f8690763700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10828> 2014-06-09 16:22:12.158144 7f86a5e11700 5 _do_op 0x14dd61e0 seq 4082 osr(3.f6 0x3ed04b0)/0x3ed04b0 start -10827> 2014-06-09 16:22:12.158147 7f86a5e11700 10 _do_transaction on 0xc22a240 -10826> 2014-06-09 16:22:12.158146 7f868b759700 2 waiting 51 > 50 ops || 4238186 > 104857600 -10825> 2014-06-09 16:22:12.158141 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -10824> 2014-06-09 16:22:12.158153 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10823> 2014-06-09 16:22:12.158149 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -10822> 2014-06-09 16:22:12.158158 7f868bf5a700 10 log is not dirty -10821> 2014-06-09 16:22:12.158163 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.40c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -10820> 2014-06-09 16:22:12.158192 7f868bf5a700 10 log is not dirty -10819> 2014-06-09 16:22:12.158191 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10818> 2014-06-09 16:22:12.158196 7f868bf5a700 5 queue_transactions existing osr(2.40c 0x2ed1250)/0x2ed1250 -10817> 2014-06-09 16:22:12.158198 7f868bf5a700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10816> 2014-06-09 16:22:12.158207 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4082.0.0 <= header.spos 150443.0.0 -10815> 2014-06-09 16:22:12.158211 7f86a5e11700 5 submit_transaction r = 0 -10814> 2014-06-09 16:22:12.158231 7f86a5e11700 10 _do_op 0x14dd61e0 seq 4082 r = 0, finisher 0x31456a20 0 -10813> 2014-06-09 16:22:12.158241 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6f3e0] -10812> 2014-06-09 16:22:12.158243 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -10811> 2014-06-09 16:22:12.158254 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -10810> 2014-06-09 16:22:12.158262 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.65c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -10809> 2014-06-09 16:22:12.158272 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6f3e0] -10808> 2014-06-09 16:22:12.158274 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10807> 2014-06-09 16:22:12.158275 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10806> 2014-06-09 16:22:12.158279 7f86a5610700 10 _finish_op 0x2fc8c660 seq 4081 osr(2.65c 0x2ed1fb0)/0x2ed1fb0 -10805> 2014-06-09 16:22:12.158291 7f868ff62700 10 op_submit_start 4131 -10804> 2014-06-09 16:22:12.158292 7f868ff62700 5 queue_transactions (trailing journal) 4131 -10803> 2014-06-09 16:22:12.158294 7f868ff62700 5 queue_op 0x30a3860 seq 4131 osr(0.530 0x2bd78f0) 828 bytes (queue has 50 ops and 4237327 bytes) -10802> 2014-06-09 16:22:12.158299 7f8692767700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10801> 2014-06-09 16:22:12.158305 7f868e75f700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10800> 2014-06-09 16:22:12.158310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7ba00] -10799> 2014-06-09 16:22:12.158311 7f8691765700 2 waiting 51 > 50 ops || 4238185 > 104857600 -10798> 2014-06-09 16:22:12.158314 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7ba00] -10797> 2014-06-09 16:22:12.158315 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10796> 2014-06-09 16:22:12.158315 7f8689755700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10795> 2014-06-09 16:22:12.158316 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10794> 2014-06-09 16:22:12.158320 7f868ef60700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10793> 2014-06-09 16:22:12.158319 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc1158c0 (1 active) -10792> 2014-06-09 16:22:12.158322 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89fc20 (2 active) -10791> 2014-06-09 16:22:12.158324 7f8692f68700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10790> 2014-06-09 16:22:12.158327 7f868ff62700 10 op_submit_finish 4131 -10789> 2014-06-09 16:22:12.158329 7f868df5e700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10788> 2014-06-09 16:22:12.158333 7f868cf5c700 2 waiting 51 > 50 ops || 4238182 > 104857600 -10787> 2014-06-09 16:22:12.158337 7f868a757700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10786> 2014-06-09 16:22:12.158331 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.32f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -10785> 2014-06-09 16:22:12.158341 7f868c75b700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10784> 2014-06-09 16:22:12.158346 7f868af58700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10783> 2014-06-09 16:22:12.158350 7f8690f64700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10782> 2014-06-09 16:22:12.158354 7f8691f66700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10781> 2014-06-09 16:22:12.158358 7f8689f56700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10780> 2014-06-09 16:22:12.158363 7f868f761700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10779> 2014-06-09 16:22:12.158365 7f868ff62700 10 log is not dirty -10778> 2014-06-09 16:22:12.158367 7f8690763700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10777> 2014-06-09 16:22:12.158370 7f868ff62700 5 queue_transactions existing osr(1.32f 0x384d350)/0x384d350 -10776> 2014-06-09 16:22:12.158371 7f868b759700 2 waiting 51 > 50 ops || 4238153 > 104857600 -10775> 2014-06-09 16:22:12.158375 7f86a5610700 5 _do_op 0xc2d5e60 seq 4083 osr(0.479 0x2bd7490)/0x2bd7490 start -10774> 2014-06-09 16:22:12.158376 7f868bf5a700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10773> 2014-06-09 16:22:12.158379 7f86a5610700 10 _do_transaction on 0xc24cf00 -10772> 2014-06-09 16:22:12.158381 7f868d75d700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10771> 2014-06-09 16:22:12.158385 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10770> 2014-06-09 16:22:12.158386 7f868ff62700 2 waiting 51 > 50 ops || 4238155 > 104857600 -10769> 2014-06-09 16:22:12.158425 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10768> 2014-06-09 16:22:12.158442 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4083.0.0 <= header.spos 150443.0.0 -10767> 2014-06-09 16:22:12.158447 7f86a5610700 5 submit_transaction r = 0 -10766> 2014-06-09 16:22:12.158459 7f86a5610700 10 _do_op 0xc2d5e60 seq 4083 r = 0, finisher 0xbfcfb00 0 -10765> 2014-06-09 16:22:12.158471 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1b0c04e0] -10764> 2014-06-09 16:22:12.158474 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -10763> 2014-06-09 16:22:12.158484 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -10762> 2014-06-09 16:22:12.158493 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.479( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -10761> 2014-06-09 16:22:12.158502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1b0c04e0] -10760> 2014-06-09 16:22:12.158504 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10759> 2014-06-09 16:22:12.158505 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10758> 2014-06-09 16:22:12.158514 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7b780] -10757> 2014-06-09 16:22:12.158517 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] _finish_recovery -10756> 2014-06-09 16:22:12.158527 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] purge_strays -10755> 2014-06-09 16:22:12.158538 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.f6( v 243'684 (0'0,243'684] local-les=243 n=128 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 213'683 mlcod 213'683 active+clean] publish_stats_to_osd 243:1117 -10754> 2014-06-09 16:22:12.158550 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7b780] -10753> 2014-06-09 16:22:12.158552 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10752> 2014-06-09 16:22:12.158553 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10751> 2014-06-09 16:22:12.158557 7f86a5e11700 10 _finish_op 0x14dd61e0 seq 4082 osr(3.f6 0x3ed04b0)/0x3ed04b0 -10750> 2014-06-09 16:22:12.158570 7f868e75f700 10 op_submit_start 4132 -10749> 2014-06-09 16:22:12.158571 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31456a20] -10748> 2014-06-09 16:22:12.158573 7f868e75f700 5 queue_transactions (trailing journal) 4132 -10747> 2014-06-09 16:22:12.158575 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31456a20] -10746> 2014-06-09 16:22:12.158574 7f868e75f700 5 queue_op 0x30d966c0 seq 4132 osr(0.622 0x2bd7cf0) 861 bytes (queue has 50 ops and 4237330 bytes) -10745> 2014-06-09 16:22:12.158577 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10744> 2014-06-09 16:22:12.158578 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10743> 2014-06-09 16:22:12.158579 7f8691765700 2 waiting 51 > 50 ops || 4238188 > 104857600 -10742> 2014-06-09 16:22:12.158581 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34965a0 (1 active) -10741> 2014-06-09 16:22:12.158583 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6b40 (2 active) -10740> 2014-06-09 16:22:12.158584 7f8689755700 2 waiting 51 > 50 ops || 4238158 > 104857600 -10739> 2014-06-09 16:22:12.158587 7f86a5e11700 5 _do_op 0x16fd5500 seq 4084 osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 start -10738> 2014-06-09 16:22:12.158589 7f868ef60700 2 waiting 51 > 50 ops || 4238158 > 104857600 -10737> 2014-06-09 16:22:12.158590 7f86a5e11700 10 _do_transaction on 0xc14af00 -10736> 2014-06-09 16:22:12.158593 7f8692f68700 2 waiting 51 > 50 ops || 4238158 > 104857600 -10735> 2014-06-09 16:22:12.158596 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10734> 2014-06-09 16:22:12.158598 7f868df5e700 2 waiting 51 > 50 ops || 4238158 > 104857600 -10733> 2014-06-09 16:22:12.158602 7f868cf5c700 2 waiting 51 > 50 ops || 4238185 > 104857600 -10732> 2014-06-09 16:22:12.158606 7f868a757700 2 waiting 51 > 50 ops || 4238191 > 104857600 -10731> 2014-06-09 16:22:12.158610 7f868c75b700 2 waiting 51 > 50 ops || 4238191 > 104857600 -10730> 2014-06-09 16:22:12.158614 7f868af58700 2 waiting 51 > 50 ops || 4238191 > 104857600 -10729> 2014-06-09 16:22:12.158620 7f868e75f700 10 op_submit_finish 4132 -10728> 2014-06-09 16:22:12.158636 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10727> 2014-06-09 16:22:12.158623 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -10726> 2014-06-09 16:22:12.158650 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4084.0.0 <= header.spos 150443.0.0 -10725> 2014-06-09 16:22:12.158643 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] check_local -10724> 2014-06-09 16:22:12.158655 7f86a5e11700 5 submit_transaction r = 0 -10723> 2014-06-09 16:22:12.158665 7f86a5e11700 10 _do_op 0x16fd5500 seq 4084 r = 0, finisher 0x2e0eca0 0 -10722> 2014-06-09 16:22:12.158655 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 0x19fe1b00,0x1b0b6100,0x19fe1a00,0x1b0b6000,0x19fe1900,0x1b0b7a00,0x19fe1800,0x1b0b7900,0x19fe1700,0x19fe1600,0x1c769c00 -10721> 2014-06-09 16:22:12.158674 7f86a5610700 10 _finish_op 0xc2d5e60 seq 4083 osr(0.479 0x2bd7490)/0x2bd7490 -10720> 2014-06-09 16:22:12.158686 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfcfb00] -10719> 2014-06-09 16:22:12.158690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfcfb00] -10718> 2014-06-09 16:22:12.158692 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10717> 2014-06-09 16:22:12.158693 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10716> 2014-06-09 16:22:12.158696 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89fc20 (1 active) -10715> 2014-06-09 16:22:12.158698 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3587e0 (2 active) -10714> 2014-06-09 16:22:12.158701 7f86a5610700 5 _do_op 0x309a21e0 seq 4085 osr(1.a7 0x384c730)/0x384c730 start -10713> 2014-06-09 16:22:12.158703 7f86a5610700 10 _do_transaction on 0x2ed5800 -10712> 2014-06-09 16:22:12.158696 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing -10711> 2014-06-09 16:22:12.158709 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10710> 2014-06-09 16:22:12.158709 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery -10709> 2014-06-09 16:22:12.158727 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1050 -10708> 2014-06-09 16:22:12.158739 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear -10707> 2014-06-09 16:22:12.158748 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10706> 2014-06-09 16:22:12.158750 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.21b( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=123 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] agent_stop -10705> 2014-06-09 16:22:12.158762 7f868e75f700 10 log is not dirty -10704> 2014-06-09 16:22:12.158763 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4085.0.0 <= header.spos 150443.0.0 -10703> 2014-06-09 16:22:12.158767 7f86a5610700 5 submit_transaction r = 0 -10702> 2014-06-09 16:22:12.158768 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -10701> 2014-06-09 16:22:12.158780 7f86a5610700 10 _do_op 0x309a21e0 seq 4085 r = 0, finisher 0x318c3820 0 -10700> 2014-06-09 16:22:12.158780 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local -10699> 2014-06-09 16:22:12.158789 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d72a0] -10698> 2014-06-09 16:22:12.158792 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d72a0] -10697> 2014-06-09 16:22:12.158793 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10696> 2014-06-09 16:22:12.158794 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10695> 2014-06-09 16:22:12.158789 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -10694> 2014-06-09 16:22:12.158797 7f86a5e11700 10 _finish_op 0x16fd5500 seq 4084 osr(0.2cb 0x2bd6ed0)/0x2bd6ed0 -10693> 2014-06-09 16:22:12.158798 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10692> 2014-06-09 16:22:12.158806 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0eca0] -10691> 2014-06-09 16:22:12.158811 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0eca0] -10690> 2014-06-09 16:22:12.158812 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10689> 2014-06-09 16:22:12.158813 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10688> 2014-06-09 16:22:12.158807 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -10687> 2014-06-09 16:22:12.158816 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6b40 (1 active) -10686> 2014-06-09 16:22:12.158819 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524a20 (2 active) -10685> 2014-06-09 16:22:12.158815 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -10684> 2014-06-09 16:22:12.158821 7f86a5e11700 5 _do_op 0x2e21b60 seq 4086 osr(0.58a 0x2bd7770)/0x2bd7770 start -10683> 2014-06-09 16:22:12.158824 7f86a5e11700 10 _do_transaction on 0xbe89200 -10682> 2014-06-09 16:22:12.158829 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10681> 2014-06-09 16:22:12.158829 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:145 -10680> 2014-06-09 16:22:12.158839 7f868e75f700 20 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -10679> 2014-06-09 16:22:12.158847 7f868e75f700 20 osd.26 pg_epoch: 243 pg[1.303( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop -10678> 2014-06-09 16:22:12.158856 7f868e75f700 10 log is not dirty -10677> 2014-06-09 16:22:12.158869 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10676> 2014-06-09 16:22:12.158875 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40e90 (19 active) -10675> 2014-06-09 16:22:12.158883 7f868e75f700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -10674> 2014-06-09 16:22:12.158882 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4086.0.0 <= header.spos 150443.0.0 -10673> 2014-06-09 16:22:12.158887 7f86a5e11700 5 submit_transaction r = 0 -10672> 2014-06-09 16:22:12.158897 7f86a5e11700 10 _do_op 0x2e21b60 seq 4086 r = 0, finisher 0x30d61620 0 -10671> 2014-06-09 16:22:12.158887 7f868e75f700 10 osd.26 243 dequeue_op 0x2c06b000 prio 63 cost 8192 latency 0.259421 osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] -10670> 2014-06-09 16:22:12.158909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5a260] -10669> 2014-06-09 16:22:12.158911 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5a260] -10668> 2014-06-09 16:22:12.158913 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10667> 2014-06-09 16:22:12.158914 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10666> 2014-06-09 16:22:12.158912 7f868e75f700 5 -- op tracker -- , seq: 3335, time: 2014-06-09 16:22:12.158906, event: reached_pg, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) -10665> 2014-06-09 16:22:12.158921 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006dc00] -10664> 2014-06-09 16:22:12.158923 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006dc00] -10663> 2014-06-09 16:22:12.158925 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10662> 2014-06-09 16:22:12.158926 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10661> 2014-06-09 16:22:12.158929 7f86a5610700 10 _finish_op 0x309a21e0 seq 4085 osr(1.a7 0x384c730)/0x384c730 -10660> 2014-06-09 16:22:12.158919 7f868e75f700 20 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -10659> 2014-06-09 16:22:12.158938 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c3820] -10658> 2014-06-09 16:22:12.158933 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2c06b000 -10657> 2014-06-09 16:22:12.158942 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c3820] -10656> 2014-06-09 16:22:12.158944 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10655> 2014-06-09 16:22:12.158945 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10654> 2014-06-09 16:22:12.158948 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3587e0 (1 active) -10653> 2014-06-09 16:22:12.158950 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fca20 (2 active) -10652> 2014-06-09 16:22:12.158952 7f86a5610700 5 _do_op 0x30a35c0 seq 4087 osr(1.71e 0x2ed00b0)/0x2ed00b0 start -10651> 2014-06-09 16:22:12.158955 7f86a5610700 10 _do_transaction on 0xc2e06c0 -10650> 2014-06-09 16:22:12.158943 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) v4 may_write -> write-ordered flags ack+ondisk+retry+write -10649> 2014-06-09 16:22:12.158960 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10648> 2014-06-09 16:22:12.158965 7f868e75f700 3 osd.26 pg_epoch: 243 pg[3.4c7( v 213'593 (0'0,213'593] local-les=243 n=118 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4540.0:33543 was 213'587 -10647> 2014-06-09 16:22:12.158979 7f868e75f700 1 -- 172.16.96.202:6830/30904 --> 172.16.96.220:0/1123692 -- osd_op_reply(33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] v213'587 uv587 ondisk = 0) v6 -- ?+0 0xdfb1180 con 0x30ba09a0 -10646> 2014-06-09 16:22:12.158993 7f868e75f700 10 osd.26 243 dequeue_op 0x2c06b000 finish -10645> 2014-06-09 16:22:12.158999 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10644> 2014-06-09 16:22:12.159008 7f8691765700 10 op_submit_start 4133 -10643> 2014-06-09 16:22:12.159010 7f8691765700 5 queue_transactions (trailing journal) 4133 -10642> 2014-06-09 16:22:12.159012 7f8691765700 5 queue_op 0x306075c0 seq 4133 osr(1.86 0x384c490) 858 bytes (queue has 48 ops and 4235673 bytes) -10641> 2014-06-09 16:22:12.159015 7f8691765700 10 op_submit_finish 4133 -10640> 2014-06-09 16:22:12.159014 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4087.0.0 <= header.spos 150443.0.0 -10639> 2014-06-09 16:22:12.159018 7f86a5610700 5 submit_transaction r = 0 -10638> 2014-06-09 16:22:12.159018 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -10637> 2014-06-09 16:22:12.159037 7f86a5610700 10 _do_op 0x30a35c0 seq 4087 r = 0, finisher 0x324f9c80 0 -10636> 2014-06-09 16:22:12.159031 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.102565 6 0.000333 -10635> 2014-06-09 16:22:12.159046 7f86a5e11700 10 _finish_op 0x2e21b60 seq 4086 osr(0.58a 0x2bd7770)/0x2bd7770 -10634> 2014-06-09 16:22:12.159044 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -10633> 2014-06-09 16:22:12.159056 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30d61620] -10632> 2014-06-09 16:22:12.159059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30d61620] -10631> 2014-06-09 16:22:12.159061 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10630> 2014-06-09 16:22:12.159062 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10629> 2014-06-09 16:22:12.159054 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10628> 2014-06-09 16:22:12.159065 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524a20 (1 active) -10627> 2014-06-09 16:22:12.159068 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fdb00 (2 active) -10626> 2014-06-09 16:22:12.159070 7f86a5e11700 5 _do_op 0x30a26c0 seq 4088 osr(0.151 0x2bd65e0)/0x2bd65e0 start -10625> 2014-06-09 16:22:12.159073 7f86a5e11700 10 _do_transaction on 0xc2ae900 -10624> 2014-06-09 16:22:12.159065 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -10623> 2014-06-09 16:22:12.159079 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10622> 2014-06-09 16:22:12.159076 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -10621> 2014-06-09 16:22:12.159085 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -10620> 2014-06-09 16:22:12.159093 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -10619> 2014-06-09 16:22:12.159104 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -10618> 2014-06-09 16:22:12.159117 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10617> 2014-06-09 16:22:12.159115 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -10616> 2014-06-09 16:22:12.159125 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -10615> 2014-06-09 16:22:12.159132 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4088.0.0 <= header.spos 150443.0.0 -10614> 2014-06-09 16:22:12.159136 7f86a5e11700 5 submit_transaction r = 0 -10613> 2014-06-09 16:22:12.159138 7f8691765700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 -10612> 2014-06-09 16:22:12.159145 7f86a5e11700 10 _do_op 0x30a26c0 seq 4088 r = 0, finisher 0x31bffbe0 0 -10611> 2014-06-09 16:22:12.159141 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:2.19) v4 -- ?+0 0x310db880 -10610> 2014-06-09 16:22:12.159154 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7d5c0] -10609> 2014-06-09 16:22:12.159156 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7d5c0] -10608> 2014-06-09 16:22:12.159158 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10607> 2014-06-09 16:22:12.159159 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10606> 2014-06-09 16:22:12.159154 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 -10605> 2014-06-09 16:22:12.159162 7f86a5610700 10 _finish_op 0x30a35c0 seq 4087 osr(1.71e 0x2ed00b0)/0x2ed00b0 -10604> 2014-06-09 16:22:12.159171 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f9c80] -10603> 2014-06-09 16:22:12.159175 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f9c80] -10602> 2014-06-09 16:22:12.159176 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10601> 2014-06-09 16:22:12.159177 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10600> 2014-06-09 16:22:12.159180 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fca20 (1 active) -10599> 2014-06-09 16:22:12.159183 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406d80 (2 active) -10598> 2014-06-09 16:22:12.159185 7f86a5610700 5 _do_op 0xc88acc0 seq 4089 osr(0.6c0 0x2bd7f50)/0x2bd7f50 start -10597> 2014-06-09 16:22:12.159188 7f86a5610700 10 _do_transaction on 0xbf59380 -10596> 2014-06-09 16:22:12.159189 7f8691765700 10 log is not dirty -10595> 2014-06-09 16:22:12.159192 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10594> 2014-06-09 16:22:12.159193 7f8691765700 5 queue_transactions existing osr(2.19 0x2ed0270)/0x2ed0270 -10593> 2014-06-09 16:22:12.159196 7f8691765700 10 op_submit_start 4134 -10592> 2014-06-09 16:22:12.159197 7f8691765700 5 queue_transactions (trailing journal) 4134 -10591> 2014-06-09 16:22:12.159198 7f8691765700 5 queue_op 0x2fcafbc0 seq 4134 osr(2.19 0x2ed0270) 858 bytes (queue has 47 ops and 4234875 bytes) -10590> 2014-06-09 16:22:12.159201 7f8691765700 10 op_submit_finish 4134 -10589> 2014-06-09 16:22:12.159203 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -10588> 2014-06-09 16:22:12.159216 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.000381 7 0.000323 -10587> 2014-06-09 16:22:12.159231 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10586> 2014-06-09 16:22:12.159227 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -10585> 2014-06-09 16:22:12.159237 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10584> 2014-06-09 16:22:12.159245 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4089.0.0 <= header.spos 150443.0.0 -10583> 2014-06-09 16:22:12.159250 7f86a5610700 5 submit_transaction r = 0 -10582> 2014-06-09 16:22:12.159247 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 -10581> 2014-06-09 16:22:12.159262 7f86a5610700 10 _do_op 0xc88acc0 seq 4089 r = 0, finisher 0x32480180 0 -10580> 2014-06-09 16:22:12.159257 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -10579> 2014-06-09 16:22:12.159265 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -10578> 2014-06-09 16:22:12.159275 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d0300] -10577> 2014-06-09 16:22:12.159277 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d0300] -10576> 2014-06-09 16:22:12.159278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10575> 2014-06-09 16:22:12.159273 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -10574> 2014-06-09 16:22:12.159280 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10573> 2014-06-09 16:22:12.159289 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31bffc00] -10572> 2014-06-09 16:22:12.159290 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31bffc00] -10571> 2014-06-09 16:22:12.159292 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10570> 2014-06-09 16:22:12.159293 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10569> 2014-06-09 16:22:12.159284 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -10568> 2014-06-09 16:22:12.159296 7f86a5e11700 10 _finish_op 0x30a26c0 seq 4088 osr(0.151 0x2bd65e0)/0x2bd65e0 -10567> 2014-06-09 16:22:12.159295 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -10566> 2014-06-09 16:22:12.159302 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fdb00 (1 active) -10565> 2014-06-09 16:22:12.159304 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31bffbe0] -10564> 2014-06-09 16:22:12.159305 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3387e0 (2 active) -10563> 2014-06-09 16:22:12.159307 7f86a5e11700 5 _do_op 0x2ea53aa0 seq 4090 osr(1.517 0x384db70)/0x384db70 start -10562> 2014-06-09 16:22:12.159308 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31bffbe0] -10561> 2014-06-09 16:22:12.159310 7f86a5e11700 10 _do_transaction on 0xbfec840 -10560> 2014-06-09 16:22:12.159310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10559> 2014-06-09 16:22:12.159311 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10558> 2014-06-09 16:22:12.159304 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -10557> 2014-06-09 16:22:12.159315 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10556> 2014-06-09 16:22:12.159314 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -10555> 2014-06-09 16:22:12.159323 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -10554> 2014-06-09 16:22:12.159336 7f8691765700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 -10553> 2014-06-09 16:22:12.159338 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:1.2aa) v4 -- ?+0 0x310db500 -10552> 2014-06-09 16:22:12.159354 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10551> 2014-06-09 16:22:12.159351 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 -10550> 2014-06-09 16:22:12.159368 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4090.0.0 <= header.spos 150443.0.0 -10549> 2014-06-09 16:22:12.159372 7f86a5e11700 5 submit_transaction r = 0 -10548> 2014-06-09 16:22:12.159383 7f8691765700 10 log is not dirty -10547> 2014-06-09 16:22:12.159386 7f86a5e11700 10 _do_op 0x2ea53aa0 seq 4090 r = 0, finisher 0x3251d2e0 0 -10546> 2014-06-09 16:22:12.159387 7f8691765700 5 queue_transactions existing osr(1.2aa 0x384d1d0)/0x384d1d0 -10545> 2014-06-09 16:22:12.159390 7f8691765700 10 op_submit_start 4135 -10544> 2014-06-09 16:22:12.159392 7f8691765700 5 queue_transactions (trailing journal) 4135 -10543> 2014-06-09 16:22:12.159393 7f8691765700 5 queue_op 0x15996b40 seq 4135 osr(1.2aa 0x384d1d0) 861 bytes (queue has 47 ops and 4234908 bytes) -10542> 2014-06-09 16:22:12.159395 7f86a5610700 10 _finish_op 0xc88acc0 seq 4089 osr(0.6c0 0x2bd7f50)/0x2bd7f50 -10541> 2014-06-09 16:22:12.159401 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406d80 (1 active) -10540> 2014-06-09 16:22:12.159404 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1d40 (2 active) -10539> 2014-06-09 16:22:12.159409 7f8691765700 10 op_submit_finish 4135 -10538> 2014-06-09 16:22:12.159413 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -10537> 2014-06-09 16:22:12.159426 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.054902 7 0.000351 -10536> 2014-06-09 16:22:12.159439 7f8690f64700 10 op_submit_start 4136 -10535> 2014-06-09 16:22:12.159441 7f8690f64700 5 queue_transactions (trailing journal) 4136 -10534> 2014-06-09 16:22:12.159442 7f8690f64700 5 queue_op 0x30a3320 seq 4136 osr(1.73b 0x2ed00c0) 828 bytes (queue has 47 ops and 4234908 bytes) -10533> 2014-06-09 16:22:12.159437 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -10532> 2014-06-09 16:22:12.159445 7f8690f64700 10 op_submit_finish 4136 -10531> 2014-06-09 16:22:12.159447 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10530> 2014-06-09 16:22:12.159448 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.537( v 213'606 (0'0,213'606] local-les=243 n=120 ec=118 les/c 243/243 242/242/212) -10529> 2014-06-09 16:22:12.159458 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -10528> 2014-06-09 16:22:12.159469 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -10527> 2014-06-09 16:22:12.159477 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -10526> 2014-06-09 16:22:12.159485 7f8690f64700 10 log is not dirty -10525> 2014-06-09 16:22:12.159490 7f8690f64700 5 queue_transactions existing osr(3.537 0x3ed1620)/0x3ed1620 -10524> 2014-06-09 16:22:12.159485 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -10523> 2014-06-09 16:22:12.159493 7f8690f64700 10 op_submit_start 4137 -10522> 2014-06-09 16:22:12.159494 7f8690f64700 5 queue_transactions (trailing journal) 4137 -10521> 2014-06-09 16:22:12.159495 7f8690f64700 5 queue_op 0x30a3da0 seq 4137 osr(3.537 0x3ed1620) 828 bytes (queue has 48 ops and 4235736 bytes) -10520> 2014-06-09 16:22:12.159498 7f8690f64700 10 op_submit_finish 4137 -10519> 2014-06-09 16:22:12.159495 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) -10518> 2014-06-09 16:22:12.159501 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5c1( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.5c1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) -10517> 2014-06-09 16:22:12.159506 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) -10516> 2014-06-09 16:22:12.159516 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -10515> 2014-06-09 16:22:12.159529 7f8691765700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 -10514> 2014-06-09 16:22:12.159534 7f8690f64700 10 log is not dirty -10513> 2014-06-09 16:22:12.159532 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.3 172.16.96.201:6823/50569 -- pg_info(1 pgs e243:0.21b) v4 -- ?+0 0x30d9c8c0 -10512> 2014-06-09 16:22:12.159538 7f8690f64700 5 queue_transactions existing osr(1.5c1 0x384dd90)/0x384dd90 -10511> 2014-06-09 16:22:12.159541 7f8690f64700 10 op_submit_start 4138 -10510> 2014-06-09 16:22:12.159542 7f8690f64700 5 queue_transactions (trailing journal) 4138 -10509> 2014-06-09 16:22:12.159543 7f8690f64700 5 queue_op 0x30a2ae0 seq 4138 osr(1.5c1 0x384dd90) 828 bytes (queue has 49 ops and 4236564 bytes) -10508> 2014-06-09 16:22:12.159546 7f8690f64700 10 op_submit_finish 4138 -10507> 2014-06-09 16:22:12.159545 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 -10506> 2014-06-09 16:22:12.159548 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.348( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.348( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) -10505> 2014-06-09 16:22:12.159578 7f8691765700 10 log is not dirty -10504> 2014-06-09 16:22:12.159579 7f8690f64700 10 log is not dirty -10503> 2014-06-09 16:22:12.159582 7f8691765700 5 queue_transactions existing osr(0.21b 0x2bd6ab0)/0x2bd6ab0 -10502> 2014-06-09 16:22:12.159584 7f8690f64700 5 queue_transactions existing osr(1.348 0x384d310)/0x384d310 -10501> 2014-06-09 16:22:12.159585 7f8691765700 10 op_submit_start 4139 -10500> 2014-06-09 16:22:12.159586 7f8691765700 5 queue_transactions (trailing journal) 4139 -10499> 2014-06-09 16:22:12.159586 7f8690f64700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10498> 2014-06-09 16:22:12.159587 7f8691765700 5 queue_op 0x1e08b440 seq 4139 osr(0.21b 0x2bd6ab0) 861 bytes (queue has 50 ops and 4237425 bytes) -10497> 2014-06-09 16:22:12.159590 7f8691765700 10 op_submit_finish 4139 -10496> 2014-06-09 16:22:12.159591 7f8689f56700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10495> 2014-06-09 16:22:12.159597 7f868f761700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10494> 2014-06-09 16:22:12.159593 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -10493> 2014-06-09 16:22:12.159609 7f8690763700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10492> 2014-06-09 16:22:12.159605 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.066104 7 0.000350 -10491> 2014-06-09 16:22:12.159622 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -10490> 2014-06-09 16:22:12.159632 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10489> 2014-06-09 16:22:12.159641 7f868b759700 2 waiting 51 > 50 ops || 4238251 > 104857600 -10488> 2014-06-09 16:22:12.159647 7f868bf5a700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10487> 2014-06-09 16:22:12.159642 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -10486> 2014-06-09 16:22:12.159649 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32480180] -10485> 2014-06-09 16:22:12.159651 7f868d75d700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10484> 2014-06-09 16:22:12.159655 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32480180] -10483> 2014-06-09 16:22:12.159656 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10482> 2014-06-09 16:22:12.159656 7f868ff62700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10481> 2014-06-09 16:22:12.159657 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10480> 2014-06-09 16:22:12.159653 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -10479> 2014-06-09 16:22:12.159002 7f868e75f700 5 -- op tracker -- , seq: 3335, time: 2014-06-09 16:22:12.158997, event: done, op: osd_op(client.4540.0:33543 rbd_data.115c6b8b4567.00000000000017ac [set-alloc-hint object_size 4194304 write_size 4194304,write 4186112~8192] 3.b22e6cc7 RETRY=2 ack+ondisk+retry+write e243) -10478> 2014-06-09 16:22:12.159661 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -10477> 2014-06-09 16:22:12.159669 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -10476> 2014-06-09 16:22:12.159677 7f868e75f700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -10475> 2014-06-09 16:22:12.159678 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -10474> 2014-06-09 16:22:12.159689 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -10473> 2014-06-09 16:22:12.159698 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40e90 (20 active) -10472> 2014-06-09 16:22:12.159699 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -10471> 2014-06-09 16:22:12.159702 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.24c( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/25 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 1.24c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) -10470> 2014-06-09 16:22:12.159711 7f8691765700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 -10469> 2014-06-09 16:22:12.159714 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:0.678) v4 -- ?+0 0x30d9c700 -10468> 2014-06-09 16:22:12.159727 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:186 -10467> 2014-06-09 16:22:12.159737 7f868e75f700 10 log is not dirty -10466> 2014-06-09 16:22:12.159742 7f868e75f700 5 queue_transactions existing osr(1.24c 0x384cfd0)/0x384cfd0 -10465> 2014-06-09 16:22:12.159744 7f868e75f700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10464> 2014-06-09 16:22:12.159748 7f8691f66700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10463> 2014-06-09 16:22:12.159753 7f8689755700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10462> 2014-06-09 16:22:12.159758 7f868ef60700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10461> 2014-06-09 16:22:12.159759 7f8691765700 10 log is not dirty -10460> 2014-06-09 16:22:12.159763 7f8692f68700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10459> 2014-06-09 16:22:12.159764 7f8691765700 5 queue_transactions existing osr(0.678 0x2bd7e90)/0x2bd7e90 -10458> 2014-06-09 16:22:12.159766 7f8691765700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10457> 2014-06-09 16:22:12.159773 7f868cf5c700 2 waiting 51 > 50 ops || 4238280 > 104857600 -10456> 2014-06-09 16:22:12.159774 7f86a5610700 5 _do_op 0x16fd5260 seq 4091 osr(1.3cd 0x384d4b0)/0x384d4b0 start -10455> 2014-06-09 16:22:12.159778 7f86a5610700 10 _do_transaction on 0xc121e00 -10454> 2014-06-09 16:22:12.159778 7f868a757700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10453> 2014-06-09 16:22:12.159783 7f868c75b700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10452> 2014-06-09 16:22:12.159783 7f86a5610700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10451> 2014-06-09 16:22:12.159787 7f868af58700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10450> 2014-06-09 16:22:12.159791 7f868df5e700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10449> 2014-06-09 16:22:12.159795 7f8692767700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10448> 2014-06-09 16:22:12.159825 7f86a5610700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10447> 2014-06-09 16:22:12.159839 7f86a5610700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4091.0.0 <= header.spos 150443.0.0 -10446> 2014-06-09 16:22:12.159844 7f86a5610700 5 submit_transaction r = 0 -10445> 2014-06-09 16:22:12.159867 7f86a5610700 10 _do_op 0x16fd5260 seq 4091 r = 0, finisher 0x3061fc20 0 -10444> 2014-06-09 16:22:12.159876 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251c520] -10443> 2014-06-09 16:22:12.159878 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251c520] -10442> 2014-06-09 16:22:12.159879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10441> 2014-06-09 16:22:12.159880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10440> 2014-06-09 16:22:12.159883 7f86a5e11700 10 _finish_op 0x2ea53aa0 seq 4090 osr(1.517 0x384db70)/0x384db70 -10439> 2014-06-09 16:22:12.159891 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3387e0 (1 active) -10438> 2014-06-09 16:22:12.159892 7f8690f64700 10 op_submit_start 4140 -10437> 2014-06-09 16:22:12.159895 7f8690f64700 5 queue_transactions (trailing journal) 4140 -10436> 2014-06-09 16:22:12.159894 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc6c0 (2 active) -10435> 2014-06-09 16:22:12.159897 7f86a5e11700 5 _do_op 0x324cd5c0 seq 4092 osr(2.298 0x2ed0c70)/0x2ed0c70 start -10434> 2014-06-09 16:22:12.159896 7f8690f64700 5 queue_op 0x30a2540 seq 4140 osr(1.348 0x384d310) 828 bytes (queue has 50 ops and 4237425 bytes) -10433> 2014-06-09 16:22:12.159899 7f8690f64700 10 op_submit_finish 4140 -10432> 2014-06-09 16:22:12.159899 7f86a5e11700 10 _do_transaction on 0xbfcd080 -10431> 2014-06-09 16:22:12.159906 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10430> 2014-06-09 16:22:12.159902 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.2ef( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.2ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) -10429> 2014-06-09 16:22:12.159934 7f8690f64700 10 log is not dirty -10428> 2014-06-09 16:22:12.159938 7f8690f64700 5 queue_transactions existing osr(1.2ef 0x384d0f0)/0x384d0f0 -10427> 2014-06-09 16:22:12.159941 7f8690f64700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10426> 2014-06-09 16:22:12.159945 7f8689f56700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10425> 2014-06-09 16:22:12.159946 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10424> 2014-06-09 16:22:12.159950 7f868f761700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10423> 2014-06-09 16:22:12.159954 7f8690763700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10422> 2014-06-09 16:22:12.159959 7f868b759700 2 waiting 51 > 50 ops || 4238251 > 104857600 -10421> 2014-06-09 16:22:12.159961 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4092.0.0 <= header.spos 150443.0.0 -10420> 2014-06-09 16:22:12.159963 7f868bf5a700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10419> 2014-06-09 16:22:12.159965 7f86a5e11700 5 submit_transaction r = 0 -10418> 2014-06-09 16:22:12.159967 7f868d75d700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10417> 2014-06-09 16:22:12.159972 7f868ff62700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10416> 2014-06-09 16:22:12.159976 7f868e75f700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10415> 2014-06-09 16:22:12.159980 7f8691f66700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10414> 2014-06-09 16:22:12.159979 7f86a5e11700 10 _do_op 0x324cd5c0 seq 4092 r = 0, finisher 0x3042b1e0 0 -10413> 2014-06-09 16:22:12.159985 7f8689755700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10412> 2014-06-09 16:22:12.159989 7f868ef60700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10411> 2014-06-09 16:22:12.159992 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc83e40] -10410> 2014-06-09 16:22:12.159994 7f8692f68700 2 waiting 51 > 50 ops || 4238253 > 104857600 -10409> 2014-06-09 16:22:12.159995 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc83e40] -10408> 2014-06-09 16:22:12.159996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10407> 2014-06-09 16:22:12.159998 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10406> 2014-06-09 16:22:12.159998 7f8691765700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10405> 2014-06-09 16:22:12.160003 7f868cf5c700 2 waiting 51 > 50 ops || 4238280 > 104857600 -10404> 2014-06-09 16:22:12.160006 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b9a0] -10403> 2014-06-09 16:22:12.160007 7f868a757700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10402> 2014-06-09 16:22:12.160008 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b9a0] -10401> 2014-06-09 16:22:12.160011 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10400> 2014-06-09 16:22:12.160011 7f868c75b700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10399> 2014-06-09 16:22:12.160013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10398> 2014-06-09 16:22:12.160016 7f868af58700 2 waiting 51 > 50 ops || 4238286 > 104857600 -10397> 2014-06-09 16:22:12.160016 7f86a5610700 10 _finish_op 0x16fd5260 seq 4091 osr(1.3cd 0x384d4b0)/0x384d4b0 -10396> 2014-06-09 16:22:12.160029 7f8690f64700 10 op_submit_start 4141 -10395> 2014-06-09 16:22:12.160028 7f86a5610700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1d40 (1 active) -10394> 2014-06-09 16:22:12.160030 7f8690f64700 5 queue_transactions (trailing journal) 4141 -10393> 2014-06-09 16:22:12.160031 7f86a5610700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f680 (2 active) -10392> 2014-06-09 16:22:12.160032 7f8690f64700 5 queue_op 0x2fcaea20 seq 4141 osr(1.2ef 0x384d0f0) 828 bytes (queue has 50 ops and 4237425 bytes) -10391> 2014-06-09 16:22:12.160035 7f8690f64700 10 op_submit_finish 4141 -10390> 2014-06-09 16:22:12.160035 7f86a5610700 5 _do_op 0x300601e0 seq 4093 osr(3.27 0x3ed0340)/0x3ed0340 start -10389> 2014-06-09 16:22:12.160038 7f86a5610700 10 _do_transaction on 0xbf883c0 -10388> 2014-06-09 16:22:12.160044 7f86a5610700 15 _remove 3.27_head/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 -10387> 2014-06-09 16:22:12.160037 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -10386> 2014-06-09 16:22:12.160048 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251d2e0,0x3061fc20] -10385> 2014-06-09 16:22:12.160053 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251d2e0,0x3061fc20] -10384> 2014-06-09 16:22:12.160055 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10383> 2014-06-09 16:22:12.160056 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10382> 2014-06-09 16:22:12.160051 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -10381> 2014-06-09 16:22:12.160059 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -10380> 2014-06-09 16:22:12.160068 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10379> 2014-06-09 16:22:12.160077 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -10378> 2014-06-09 16:22:12.160085 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -10377> 2014-06-09 16:22:12.160098 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:157 -10376> 2014-06-09 16:22:12.160108 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -10375> 2014-06-09 16:22:12.160117 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.19a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -10374> 2014-06-09 16:22:12.160126 7f8690f64700 10 log is not dirty -10373> 2014-06-09 16:22:12.160131 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10372> 2014-06-09 16:22:12.160144 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops -10371> 2014-06-09 16:22:12.160154 7f8690f64700 10 log is not dirty -10370> 2014-06-09 16:22:12.160158 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10369> 2014-06-09 16:22:12.160172 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -10368> 2014-06-09 16:22:12.160183 7f8690f64700 10 log is not dirty -10367> 2014-06-09 16:22:12.160193 7f86a5610700 10 lookup_strip_header done cid 3.27_head oid 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 -10366> 2014-06-09 16:22:12.160187 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10365> 2014-06-09 16:22:12.160201 7f86a5610700 20 remove_header removing 29931 cid 3.27_head oid 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 -10364> 2014-06-09 16:22:12.160206 7f8690f64700 10 log is not dirty -10363> 2014-06-09 16:22:12.160209 7f86a5610700 20 clear_header clearing seq 29931 -10362> 2014-06-09 16:22:12.160210 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10361> 2014-06-09 16:22:12.160223 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops -10360> 2014-06-09 16:22:12.160235 7f8690f64700 10 log is not dirty -10359> 2014-06-09 16:22:12.160239 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10358> 2014-06-09 16:22:12.160252 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -10357> 2014-06-09 16:22:12.160263 7f8690f64700 10 log is not dirty -10356> 2014-06-09 16:22:12.160267 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10355> 2014-06-09 16:22:12.160279 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10354> 2014-06-09 16:22:12.160289 7f8690f64700 10 log is not dirty -10353> 2014-06-09 16:22:12.160293 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10352> 2014-06-09 16:22:12.160306 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -10351> 2014-06-09 16:22:12.160317 7f8690f64700 10 log is not dirty -10350> 2014-06-09 16:22:12.160321 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10349> 2014-06-09 16:22:12.160334 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.4d5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -10348> 2014-06-09 16:22:12.160343 7f8690f64700 10 log is not dirty -10347> 2014-06-09 16:22:12.160348 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10346> 2014-06-09 16:22:12.160359 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10345> 2014-06-09 16:22:12.160368 7f8690f64700 10 log is not dirty -10344> 2014-06-09 16:22:12.160372 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10343> 2014-06-09 16:22:12.160384 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.1cd( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10342> 2014-06-09 16:22:12.160393 7f8690f64700 10 log is not dirty -10341> 2014-06-09 16:22:12.160397 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10340> 2014-06-09 16:22:12.160409 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -10339> 2014-06-09 16:22:12.160419 7f8690f64700 10 log is not dirty -10338> 2014-06-09 16:22:12.160423 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10337> 2014-06-09 16:22:12.160437 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.709( v 214'577 (0'0,214'577] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops -10336> 2014-06-09 16:22:12.160448 7f8690f64700 10 log is not dirty -10335> 2014-06-09 16:22:12.160452 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10334> 2014-06-09 16:22:12.160463 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.384( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10333> 2014-06-09 16:22:12.160473 7f8690f64700 10 log is not dirty -10332> 2014-06-09 16:22:12.160477 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10331> 2014-06-09 16:22:12.160488 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10330> 2014-06-09 16:22:12.160497 7f8690f64700 10 log is not dirty -10329> 2014-06-09 16:22:12.160513 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358400 (19 active) -10328> 2014-06-09 16:22:12.160520 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -10327> 2014-06-09 16:22:12.160524 7f8690f64700 10 osd.26 243 dequeue_op 0x2c06ad00 prio 63 cost 401408 latency 0.207053 osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] -10326> 2014-06-09 16:22:12.160548 7f8690f64700 5 -- op tracker -- , seq: 3363, time: 2014-06-09 16:22:12.160543, event: reached_pg, op: osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) -10325> 2014-06-09 16:22:12.160556 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -10324> 2014-06-09 16:22:12.160570 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] handle_message: 0x2c06ad00 -10323> 2014-06-09 16:22:12.160580 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] do_op osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) v4 may_write -> write-ordered flags ack+ondisk+retry+write -10322> 2014-06-09 16:22:12.160598 7f8690f64700 3 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] do_op dup client.4541.0:31903 was 213'734 -10321> 2014-06-09 16:22:12.160611 7f8690f64700 1 -- 172.16.96.202:6830/30904 --> 172.16.96.220:0/1123720 -- osd_op_reply(31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] v213'734 uv734 ondisk = 0) v6 -- ?+0 0x32070780 con 0x30ba0c60 -10320> 2014-06-09 16:22:12.160625 7f8690f64700 10 osd.26 243 dequeue_op 0x2c06ad00 finish -10319> 2014-06-09 16:22:12.160628 7f86a5610700 10 _remove 3.27_head/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 = 0 -10318> 2014-06-09 16:22:12.160636 7f86a5610700 15 _create_collection 3.27_TEMP -10317> 2014-06-09 16:22:12.160634 7f8690f64700 5 -- op tracker -- , seq: 3363, time: 2014-06-09 16:22:12.160629, event: done, op: osd_op(client.4541.0:31903 rbd_data.115d6b8b4567.0000000000001606 [set-alloc-hint object_size 4194304 write_size 4194304,write 3792896~401408] 3.da44b224 RETRY=2 ack+ondisk+retry+write e243) -10316> 2014-06-09 16:22:12.160646 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -10315> 2014-06-09 16:22:12.160666 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358400 (20 active) -10314> 2014-06-09 16:22:12.160674 7f86a5610700 20 lookup_strip_header failed to get strip_header cid COLLECTIONS oid 8e1f8b20/3.27_TEMP/head//-1 -10313> 2014-06-09 16:22:12.160679 7f86a5610700 10 lookup_cached_header COLLECTIONS/8e1f8b20/3.27_TEMP/head//-1 r = -2 -10312> 2014-06-09 16:22:12.160670 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10311> 2014-06-09 16:22:12.160684 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -10310> 2014-06-09 16:22:12.160696 7f8690f64700 10 log is not dirty -10309> 2014-06-09 16:22:12.160701 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10308> 2014-06-09 16:22:12.160711 7f86a5610700 20 lookup_strip_header failed to get strip_header cid COLLECTIONS oid 8e1f8b20/3.27_TEMP/head//-1 -10307> 2014-06-09 16:22:12.160712 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -10306> 2014-06-09 16:22:12.160721 7f8690f64700 10 log is not dirty -10305> 2014-06-09 16:22:12.160710 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 1 ==== osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+516096 (1297654895 0 0) 0x2a08cb40 con 0x30c3c2c0 -10304> 2014-06-09 16:22:12.160725 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10303> 2014-06-09 16:22:12.160737 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.719( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10302> 2014-06-09 16:22:12.160743 7f86a5610700 20 write_state seq is 30168 -10301> 2014-06-09 16:22:12.160741 7f86a68f6700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.157947, event: header_read, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -10300> 2014-06-09 16:22:12.160747 7f8690f64700 10 log is not dirty -10299> 2014-06-09 16:22:12.160751 7f86a5610700 20 set_header setting 30167 cid COLLECTIONS oid 8e1f8b20/3.27_TEMP/head//-1 parent seq 0 -10298> 2014-06-09 16:22:12.160751 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10297> 2014-06-09 16:22:12.160754 7f86a68f6700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.157949, event: throttled, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -10296> 2014-06-09 16:22:12.160765 7f86a5610700 10 _create_collection cid 3.27_TEMP r = 0 -10295> 2014-06-09 16:22:12.160770 7f86a5610700 15 _remove 3.27_TEMP/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 -10294> 2014-06-09 16:22:12.160771 7f86a68f6700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.160705, event: all_read, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -10293> 2014-06-09 16:22:12.160765 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x1b0d6b00,0x1b0d6a00,0x16bd6800,0x1b0d6900,0x16bd6700,0x1c54a600,0x1c54a500,0x306b400,0x1c54a400,0x306b600,0x1c54a300,0x10c55f00,0x1b0d6800,0x1c54a200,0x11e62900,0x22009100,0x11e62800,0x11e62700,0x11e62600,0x11e62500 -10292> 2014-06-09 16:22:12.160785 7f86a68f6700 5 -- op tracker -- , seq: 3661, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -10291> 2014-06-09 16:22:12.160792 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 -10290> 2014-06-09 16:22:12.160813 7f86a68f6700 15 osd.26 243 enqueue_op 0x32827b00 prio 63 cost 516096 latency 0.002866 osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 -10289> 2014-06-09 16:22:12.160820 7f86a5610700 20 lookup_strip_header failed to get strip_header cid 3.27_TEMP oid 97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 -10288> 2014-06-09 16:22:12.160823 7f86a5610700 10 lookup_cached_header 3.27_TEMP/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 r = -2 -10287> 2014-06-09 16:22:12.160826 7f86a5610700 10 _remove 3.27_TEMP/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 failed to get header: r = -2 -10286> 2014-06-09 16:22:12.160830 7f8690f64700 10 log is not dirty -10285> 2014-06-09 16:22:12.160831 7f86a5610700 15 _touch 3.27_head/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 -10284> 2014-06-09 16:22:12.160834 7f86a5610700 10 _touch 3.27_head/97a47827/rbd_data.11756b8b4567.00000000000015f7/head//3 failed to get header: r = -2 -10283> 2014-06-09 16:22:12.160835 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10282> 2014-06-09 16:22:12.160837 7f86a5610700 0 error (22) Invalid argument not handled on operation 9 (4093.0.3, or op 3, counting from 0) -10281> 2014-06-09 16:22:12.160846 7f86a5610700 0 unexpected error code -10280> 2014-06-09 16:22:12.160848 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10279> 2014-06-09 16:22:12.160858 7f8690f64700 10 log is not dirty -10278> 2014-06-09 16:22:12.160862 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10277> 2014-06-09 16:22:12.160875 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.437( v 213'621 (0'0,213'621] local-les=243 n=122 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -10276> 2014-06-09 16:22:12.160886 7f8690f64700 10 log is not dirty -10275> 2014-06-09 16:22:12.160890 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10274> 2014-06-09 16:22:12.160902 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -10273> 2014-06-09 16:22:12.160911 7f8690f64700 10 log is not dirty -10272> 2014-06-09 16:22:12.160916 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10271> 2014-06-09 16:22:12.160930 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.ad( v 213'578 (0'0,213'578] local-les=243 n=106 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -10270> 2014-06-09 16:22:12.160940 7f8690f64700 10 log is not dirty -10269> 2014-06-09 16:22:12.160944 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10268> 2014-06-09 16:22:12.160956 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.40a( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -10267> 2014-06-09 16:22:12.160965 7f8690f64700 10 log is not dirty -10266> 2014-06-09 16:22:12.160969 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10265> 2014-06-09 16:22:12.160987 7f86a5e11700 10 _finish_op 0x324cd5c0 seq 4092 osr(2.298 0x2ed0c70)/0x2ed0c70 -10264> 2014-06-09 16:22:12.160982 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.718( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x230dae00,0x230dad00,0x230dac00,0x230dab00,0x230daa00,0x270bf700,0x270bf600 -10263> 2014-06-09 16:22:12.161000 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b1e0] -10262> 2014-06-09 16:22:12.161004 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b1e0] -10261> 2014-06-09 16:22:12.161006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10260> 2014-06-09 16:22:12.161007 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10259> 2014-06-09 16:22:12.161010 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc6c0 (1 active) -10258> 2014-06-09 16:22:12.161012 7f8690f64700 10 log is not dirty -10257> 2014-06-09 16:22:12.161013 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34977a0 (2 active) -10256> 2014-06-09 16:22:12.161016 7f86a5e11700 5 _do_op 0x2ea52d20 seq 4094 osr(0.631 0x2bd7c70)/0x2bd7c70 start -10255> 2014-06-09 16:22:12.161019 7f86a5e11700 10 _do_transaction on 0xc2dc480 -10254> 2014-06-09 16:22:12.161016 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10253> 2014-06-09 16:22:12.161025 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10252> 2014-06-09 16:22:12.161028 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops -10251> 2014-06-09 16:22:12.161038 7f8690f64700 10 log is not dirty -10250> 2014-06-09 16:22:12.161042 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10249> 2014-06-09 16:22:12.161056 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.75f( v 213'605 (0'0,213'605] local-les=243 n=115 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0xe2ad800,0xe2ad700,0xe2ad600,0x11e63700,0xe2ad500,0x13023100,0x10c55e00,0x13023000,0x14052e00,0xe2ac500,0xe2ac700,0x2e075600,0x2f097800,0x2f097700,0x2f097600,0x2f097500,0x2f097400 -10248> 2014-06-09 16:22:12.161069 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10247> 2014-06-09 16:22:12.161084 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4094.0.0 <= header.spos 150443.0.0 -10246> 2014-06-09 16:22:12.161089 7f86a5e11700 5 submit_transaction r = 0 -10245> 2014-06-09 16:22:12.161101 7f86a5e11700 10 _do_op 0x2ea52d20 seq 4094 r = 0, finisher 0x2d7399e0 0 -10244> 2014-06-09 16:22:12.161108 7f8690f64700 10 log is not dirty -10243> 2014-06-09 16:22:12.161111 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x281b6280] -10242> 2014-06-09 16:22:12.161114 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x281b6280] -10241> 2014-06-09 16:22:12.161115 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10240> 2014-06-09 16:22:12.161116 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10239> 2014-06-09 16:22:12.161120 7f86a5e11700 10 _finish_op 0x2ea52d20 seq 4094 osr(0.631 0x2bd7c70)/0x2bd7c70 -10238> 2014-06-09 16:22:12.161112 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10237> 2014-06-09 16:22:12.161129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d7399e0] -10236> 2014-06-09 16:22:12.161124 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops -10235> 2014-06-09 16:22:12.161132 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d7399e0] -10234> 2014-06-09 16:22:12.161134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10233> 2014-06-09 16:22:12.161134 7f8690f64700 10 log is not dirty -10232> 2014-06-09 16:22:12.161134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10231> 2014-06-09 16:22:12.161137 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34977a0 (1 active) -10230> 2014-06-09 16:22:12.161140 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497440 (2 active) -10229> 2014-06-09 16:22:12.161142 7f86a5e11700 5 _do_op 0xcd9cd20 seq 4095 osr(3.e8 0x3ed0510)/0x3ed0510 start -10228> 2014-06-09 16:22:12.161145 7f86a5e11700 10 _do_transaction on 0xc0df440 -10227> 2014-06-09 16:22:12.161138 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10226> 2014-06-09 16:22:12.161150 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10225> 2014-06-09 16:22:12.161156 7f8690f64700 10 log is not dirty -10224> 2014-06-09 16:22:12.161160 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10223> 2014-06-09 16:22:12.161173 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops -10222> 2014-06-09 16:22:12.161186 7f8690f64700 10 log is not dirty -10221> 2014-06-09 16:22:12.161192 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10220> 2014-06-09 16:22:12.161189 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10219> 2014-06-09 16:22:12.161207 7f8690f64700 10 log is not dirty -10218> 2014-06-09 16:22:12.161208 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4095.0.0 <= header.spos 150443.0.0 -10217> 2014-06-09 16:22:12.161213 7f86a5e11700 5 submit_transaction r = 0 -10216> 2014-06-09 16:22:12.161211 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10215> 2014-06-09 16:22:12.161224 7f86a5e11700 10 _do_op 0xcd9cd20 seq 4095 r = 0, finisher 0xbd750e0 0 -10214> 2014-06-09 16:22:12.161224 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.5f9( v 214'611 lc 213'610 (0'0,214'611] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'611 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops -10213> 2014-06-09 16:22:12.161235 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078800] -10212> 2014-06-09 16:22:12.161237 7f8690f64700 10 log is not dirty -10211> 2014-06-09 16:22:12.161237 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078800] -10210> 2014-06-09 16:22:12.161239 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10209> 2014-06-09 16:22:12.161240 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10208> 2014-06-09 16:22:12.161244 7f86a5e11700 10 _finish_op 0xcd9cd20 seq 4095 osr(3.e8 0x3ed0510)/0x3ed0510 -10207> 2014-06-09 16:22:12.161241 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10206> 2014-06-09 16:22:12.161252 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd750e0] -10205> 2014-06-09 16:22:12.161257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd750e0] -10204> 2014-06-09 16:22:12.161258 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10203> 2014-06-09 16:22:12.161259 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10202> 2014-06-09 16:22:12.161255 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.342( v 213'649 (0'0,213'649] local-les=243 n=125 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -10201> 2014-06-09 16:22:12.161262 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497440 (1 active) -10200> 2014-06-09 16:22:12.161266 7f8690f64700 10 log is not dirty -10199> 2014-06-09 16:22:12.161265 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338b40 (2 active) -10198> 2014-06-09 16:22:12.161268 7f86a5e11700 5 _do_op 0x324cca80 seq 4096 osr(2.69d 0x2ed1ef0)/0x2ed1ef0 start -10197> 2014-06-09 16:22:12.161271 7f86a5e11700 10 _do_transaction on 0xc05acc0 -10196> 2014-06-09 16:22:12.161276 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10195> 2014-06-09 16:22:12.161270 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10194> 2014-06-09 16:22:12.161281 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10193> 2014-06-09 16:22:12.161290 7f8690f64700 10 log is not dirty -10192> 2014-06-09 16:22:12.161294 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10191> 2014-06-09 16:22:12.161305 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10190> 2014-06-09 16:22:12.161314 7f8690f64700 10 log is not dirty -10189> 2014-06-09 16:22:12.161316 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10188> 2014-06-09 16:22:12.161327 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358400 (19 active) -10187> 2014-06-09 16:22:12.161331 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4096.0.0 <= header.spos 150443.0.0 -10186> 2014-06-09 16:22:12.161333 7f8690f64700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -10185> 2014-06-09 16:22:12.161336 7f86a5e11700 5 submit_transaction r = 0 -10184> 2014-06-09 16:22:12.161347 7f86a5e11700 10 _do_op 0x324cca80 seq 4096 r = 0, finisher 0xbd79080 0 -10183> 2014-06-09 16:22:12.161338 7f8690f64700 10 osd.26 243 dequeue_op 0xdfa6300 prio 63 cost 516096 latency 0.108603 osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] -10182> 2014-06-09 16:22:12.161358 7f86a5e11700 10 _finish_op 0x324cca80 seq 4096 osr(2.69d 0x2ed1ef0)/0x2ed1ef0 -10181> 2014-06-09 16:22:12.161359 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006c960] -10180> 2014-06-09 16:22:12.161362 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006c960] -10179> 2014-06-09 16:22:12.161363 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10178> 2014-06-09 16:22:12.161359 7f8690f64700 5 -- op tracker -- , seq: 3617, time: 2014-06-09 16:22:12.161354, event: reached_pg, op: osd_op(client.4542.0:34421 rbd_data.115e6b8b4567.00000000000017f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 2818048~516096] 3.5db8c41b RETRY=2 ack+ondisk+retry+write e243) -10177> 2014-06-09 16:22:12.161364 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10176> 2014-06-09 16:22:12.161367 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd79080] -10175> 2014-06-09 16:22:12.161372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd79080] -10174> 2014-06-09 16:22:12.161374 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10173> 2014-06-09 16:22:12.161375 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10172> 2014-06-09 16:22:12.161366 7f8690f64700 20 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -10171> 2014-06-09 16:22:12.161378 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338b40 (1 active) -10170> 2014-06-09 16:22:12.161382 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb6a20 (2 active) -10169> 2014-06-09 16:22:12.161378 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.41b( v 214'609 (0'0,214'609] local-les=243 n=115 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xdfa6300 -10168> 2014-06-09 16:22:12.161384 7f86a5e11700 5 _do_op 0x324c1f20 seq 4097 osr(0.1fd 0x2bd6b30)/0x2bd6b30 start -10167> 2014-06-09 16:22:12.161386 7f8690f64700 10 osd.26 243 dequeue_op 0xdfa6300 finish -10166> 2014-06-09 16:22:12.161387 7f86a5e11700 10 _do_transaction on 0xc14ba40 -10165> 2014-06-09 16:22:12.161389 7f8690f64700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -10164> 2014-06-09 16:22:12.161392 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10163> 2014-06-09 16:22:12.161409 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358400 (20 active) -10162> 2014-06-09 16:22:12.161413 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10161> 2014-06-09 16:22:12.161425 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] requeue_ops -10160> 2014-06-09 16:22:12.161433 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10159> 2014-06-09 16:22:12.161436 7f8690f64700 10 log is not dirty -10158> 2014-06-09 16:22:12.161440 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -10157> 2014-06-09 16:22:12.161447 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4097.0.0 <= header.spos 150443.0.0 -10156> 2014-06-09 16:22:12.161452 7f86a5e11700 5 submit_transaction r = 0 -10155> 2014-06-09 16:22:12.161452 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -10154> 2014-06-09 16:22:12.161462 7f8690f64700 10 log is not dirty -10153> 2014-06-09 16:22:12.161470 7f86a5e11700 10 _do_op 0x324c1f20 seq 4097 r = 0, finisher 0xc6e8740 0 -10152> 2014-06-09 16:22:12.161466 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -10151> 2014-06-09 16:22:12.161476 7f86a5e11700 10 _finish_op 0x324c1f20 seq 4097 osr(0.1fd 0x2bd6b30)/0x2bd6b30 -10150> 2014-06-09 16:22:12.161477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e8ca0] -10149> 2014-06-09 16:22:12.161482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e8ca0] -10148> 2014-06-09 16:22:12.161484 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10147> 2014-06-09 16:22:12.161484 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8740] -10146> 2014-06-09 16:22:12.161485 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10145> 2014-06-09 16:22:12.161479 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local -10144> 2014-06-09 16:22:12.161489 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8740] -10143> 2014-06-09 16:22:12.161491 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10142> 2014-06-09 16:22:12.161492 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10141> 2014-06-09 16:22:12.161488 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -10140> 2014-06-09 16:22:12.161495 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb6a20 (1 active) -10139> 2014-06-09 16:22:12.161498 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780000 (2 active) -10138> 2014-06-09 16:22:12.161500 7f86a5e11700 5 _do_op 0x324c1f80 seq 4098 osr(0.129 0x2bd66b0)/0x2bd66b0 start -10137> 2014-06-09 16:22:12.161496 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10136> 2014-06-09 16:22:12.161503 7f86a5e11700 10 _do_transaction on 0xc2c78c0 -10135> 2014-06-09 16:22:12.161503 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -10134> 2014-06-09 16:22:12.161509 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10133> 2014-06-09 16:22:12.161510 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -10132> 2014-06-09 16:22:12.161521 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 -10131> 2014-06-09 16:22:12.161529 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -10130> 2014-06-09 16:22:12.161536 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.1eb( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop -10129> 2014-06-09 16:22:12.161543 7f8690f64700 10 log is not dirty -10128> 2014-06-09 16:22:12.161548 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10127> 2014-06-09 16:22:12.161547 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -10126> 2014-06-09 16:22:12.161557 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.980216 7 0.003017 -10125> 2014-06-09 16:22:12.161563 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4098.0.0 <= header.spos 150443.0.0 -10124> 2014-06-09 16:22:12.161567 7f86a5e11700 5 submit_transaction r = 0 -10123> 2014-06-09 16:22:12.161567 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -10122> 2014-06-09 16:22:12.161578 7f86a5e11700 10 _do_op 0x324c1f80 seq 4098 r = 0, finisher 0xbd70460 0 -10121> 2014-06-09 16:22:12.161576 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10120> 2014-06-09 16:22:12.161589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd980] -10119> 2014-06-09 16:22:12.161591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd980] -10118> 2014-06-09 16:22:12.161593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10117> 2014-06-09 16:22:12.161594 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10116> 2014-06-09 16:22:12.161586 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 -10115> 2014-06-09 16:22:12.161598 7f86a5e11700 10 _finish_op 0x324c1f80 seq 4098 osr(0.129 0x2bd66b0)/0x2bd66b0 -10114> 2014-06-09 16:22:12.161599 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -10113> 2014-06-09 16:22:12.161607 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd70460] -10112> 2014-06-09 16:22:12.161610 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd70460] -10111> 2014-06-09 16:22:12.161612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10110> 2014-06-09 16:22:12.161613 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10109> 2014-06-09 16:22:12.161608 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -10108> 2014-06-09 16:22:12.161616 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780000 (1 active) -10107> 2014-06-09 16:22:12.161619 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb6a20 (2 active) -10106> 2014-06-09 16:22:12.161621 7f86a5e11700 5 _do_op 0x324c1320 seq 4099 osr(0.1fd 0x2bd6b30)/0x2bd6b30 start -10105> 2014-06-09 16:22:12.161617 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -10104> 2014-06-09 16:22:12.161624 7f86a5e11700 10 _do_transaction on 0xc05e480 -10103> 2014-06-09 16:22:12.161629 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10102> 2014-06-09 16:22:12.161628 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) -10101> 2014-06-09 16:22:12.161639 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) -10100> 2014-06-09 16:22:12.161649 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -10099> 2014-06-09 16:22:12.161663 7f8690f64700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 -10098> 2014-06-09 16:22:12.161668 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10097> 2014-06-09 16:22:12.161666 7f8690f64700 1 -- 172.16.96.202:6850/30904 --> osd.3 172.16.96.201:6823/50569 -- pg_info(1 pgs e243:0.44f) v4 -- ?+0 0xcc741c0 -10096> 2014-06-09 16:22:12.161683 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4099.0.0 <= header.spos 150443.0.0 -10095> 2014-06-09 16:22:12.161687 7f86a5e11700 5 submit_transaction r = 0 -10094> 2014-06-09 16:22:12.161680 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 -10093> 2014-06-09 16:22:12.161696 7f86a5e11700 10 _do_op 0x324c1320 seq 4099 r = 0, finisher 0x324f8d60 0 -10092> 2014-06-09 16:22:12.161707 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f8c60] -10091> 2014-06-09 16:22:12.161709 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f8c60] -10090> 2014-06-09 16:22:12.161711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10089> 2014-06-09 16:22:12.161712 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10088> 2014-06-09 16:22:12.161715 7f8690f64700 10 log is not dirty -10087> 2014-06-09 16:22:12.161716 7f86a5e11700 10 _finish_op 0x324c1320 seq 4099 osr(0.1fd 0x2bd6b30)/0x2bd6b30 -10086> 2014-06-09 16:22:12.161720 7f8690f64700 5 queue_transactions existing osr(0.44f 0x2bd71a0)/0x2bd71a0 -10085> 2014-06-09 16:22:12.161723 7f8690f64700 10 op_submit_start 4142 -10084> 2014-06-09 16:22:12.161724 7f8690f64700 5 queue_transactions (trailing journal) 4142 -10083> 2014-06-09 16:22:12.161725 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f8d60] -10082> 2014-06-09 16:22:12.161725 7f8690f64700 5 queue_op 0xc2d4d20 seq 4142 osr(0.44f 0x2bd71a0) 861 bytes (queue has 44 ops and 4232492 bytes) -10081> 2014-06-09 16:22:12.161728 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f8d60] -10080> 2014-06-09 16:22:12.161732 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10079> 2014-06-09 16:22:12.161733 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10078> 2014-06-09 16:22:12.161736 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb6a20 (1 active) -10077> 2014-06-09 16:22:12.161739 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd8c0 (2 active) -10076> 2014-06-09 16:22:12.161744 7f86a5e11700 5 _do_op 0xbe7dce0 seq 4100 osr(1.66e 0x384ded0)/0x384ded0 start -10075> 2014-06-09 16:22:12.161747 7f86a5e11700 10 _do_transaction on 0xc22be00 -10074> 2014-06-09 16:22:12.161746 7f8690f64700 10 op_submit_finish 4142 -10073> 2014-06-09 16:22:12.161752 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -10072> 2014-06-09 16:22:12.161750 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -10071> 2014-06-09 16:22:12.161764 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local -10070> 2014-06-09 16:22:12.161773 7f8692767700 10 op_submit_start 4143 -10069> 2014-06-09 16:22:12.161776 7f8692767700 5 queue_transactions (trailing journal) 4143 -10068> 2014-06-09 16:22:12.161772 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -10067> 2014-06-09 16:22:12.161778 7f8692767700 5 queue_op 0x327fd800 seq 4143 osr(0.66a 0x2bd7ef0) 828 bytes (queue has 45 ops and 4233320 bytes) -10066> 2014-06-09 16:22:12.161781 7f8692767700 10 op_submit_finish 4143 -10065> 2014-06-09 16:22:12.161780 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10064> 2014-06-09 16:22:12.161788 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -10063> 2014-06-09 16:22:12.161788 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -10062> 2014-06-09 16:22:12.161785 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.654( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.654( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) -10061> 2014-06-09 16:22:12.161796 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -10060> 2014-06-09 16:22:12.161801 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4100.0.0 <= header.spos 150443.0.0 -10059> 2014-06-09 16:22:12.161805 7f86a5e11700 5 submit_transaction r = 0 -10058> 2014-06-09 16:22:12.161808 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 -10057> 2014-06-09 16:22:12.161817 7f86a5e11700 10 _do_op 0xbe7dce0 seq 4100 r = 0, finisher 0x29283d40 0 -10056> 2014-06-09 16:22:12.161817 7f8690f64700 20 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -10055> 2014-06-09 16:22:12.161827 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9ea60] -10054> 2014-06-09 16:22:12.161829 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9ea60] -10053> 2014-06-09 16:22:12.161830 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -10052> 2014-06-09 16:22:12.161831 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -10051> 2014-06-09 16:22:12.161825 7f8690f64700 20 osd.26 pg_epoch: 243 pg[2.266( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop -10050> 2014-06-09 16:22:12.161834 7f8690f64700 10 log is not dirty -10049> 2014-06-09 16:22:12.161833 7f8692767700 10 log is not dirty -10048> 2014-06-09 16:22:12.161834 7f86a5e11700 10 _finish_op 0xbe7dce0 seq 4100 osr(1.66e 0x384ded0)/0x384ded0 -10047> 2014-06-09 16:22:12.161840 7f8692767700 5 queue_transactions existing osr(0.654 0x2bd7b50)/0x2bd7b50 -10046> 2014-06-09 16:22:12.161841 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x29283d40] -10045> 2014-06-09 16:22:12.161843 7f8692767700 10 op_submit_start 4144 -10044> 2014-06-09 16:22:12.161844 7f8692767700 5 queue_transactions (trailing journal) 4144 -10043> 2014-06-09 16:22:12.161844 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x29283d40] -10042> 2014-06-09 16:22:12.161846 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -10041> 2014-06-09 16:22:12.161847 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -10040> 2014-06-09 16:22:12.161846 7f8692767700 5 queue_op 0x327fc420 seq 4144 osr(0.654 0x2bd7b50) 828 bytes (queue has 45 ops and 4233320 bytes) -10039> 2014-06-09 16:22:12.161838 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -10038> 2014-06-09 16:22:12.161849 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd8c0 (1 active) -10037> 2014-06-09 16:22:12.161852 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287c20 (2 active) -10036> 2014-06-09 16:22:12.161856 7f8692767700 10 op_submit_finish 4144 -10035> 2014-06-09 16:22:12.161851 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.023913 7 0.000360 -10034> 2014-06-09 16:22:12.161864 7f8689f56700 10 op_submit_start 4145 -10033> 2014-06-09 16:22:12.161866 7f8689f56700 5 queue_transactions (trailing journal) 4145 -10032> 2014-06-09 16:22:12.161861 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -10031> 2014-06-09 16:22:12.161868 7f8689f56700 5 queue_op 0xc2d5aa0 seq 4145 osr(2.3fa 0x2ed12f0) 861 bytes (queue has 46 ops and 4234181 bytes) -10030> 2014-06-09 16:22:12.161860 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.f0( v 213'549 (0'0,213'549] local-les=243 n=107 ec=118 les/c 243/243 242/242/202) -10029> 2014-06-09 16:22:12.161871 7f8689f56700 10 op_submit_finish 4145 -10028> 2014-06-09 16:22:12.161870 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10027> 2014-06-09 16:22:12.161876 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -10026> 2014-06-09 16:22:12.161879 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -10025> 2014-06-09 16:22:12.161892 7f8692767700 10 log is not dirty -10024> 2014-06-09 16:22:12.161890 7f8690f64700 5 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -10023> 2014-06-09 16:22:12.161897 7f8692767700 5 queue_transactions existing osr(3.f0 0x3ed04a0)/0x3ed04a0 -10022> 2014-06-09 16:22:12.161899 7f8692767700 10 op_submit_start 4146 -10021> 2014-06-09 16:22:12.161900 7f8692767700 5 queue_transactions (trailing journal) 4146 -10020> 2014-06-09 16:22:12.161890 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.126743 8 0.000412 -10019> 2014-06-09 16:22:12.161901 7f8692767700 5 queue_op 0x327fc060 seq 4146 osr(3.f0 0x3ed04a0) 826 bytes (queue has 47 ops and 4235007 bytes) -10018> 2014-06-09 16:22:12.161897 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -10017> 2014-06-09 16:22:12.161904 7f8692767700 10 op_submit_finish 4146 -10016> 2014-06-09 16:22:12.161904 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -10015> 2014-06-09 16:22:12.161903 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -10014> 2014-06-09 16:22:12.161907 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.112( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.112( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) -10013> 2014-06-09 16:22:12.161914 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -10012> 2014-06-09 16:22:12.161914 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) -10011> 2014-06-09 16:22:12.161925 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) -10010> 2014-06-09 16:22:12.161925 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -10009> 2014-06-09 16:22:12.161940 7f8692767700 10 log is not dirty -10008> 2014-06-09 16:22:12.161933 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) -10007> 2014-06-09 16:22:12.161936 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -10006> 2014-06-09 16:22:12.161944 7f8692767700 5 queue_transactions existing osr(2.112 0x2ed06d0)/0x2ed06d0 -10005> 2014-06-09 16:22:12.161946 7f8692767700 10 op_submit_start 4147 -10004> 2014-06-09 16:22:12.161947 7f8692767700 5 queue_transactions (trailing journal) 4147 -10003> 2014-06-09 16:22:12.161948 7f8692767700 5 queue_op 0x327fd7a0 seq 4147 osr(2.112 0x2ed06d0) 828 bytes (queue has 48 ops and 4235835 bytes) -10002> 2014-06-09 16:22:12.161942 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) -10001> 2014-06-09 16:22:12.161951 7f8692767700 10 op_submit_finish 4147 -10000> 2014-06-09 16:22:12.161945 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -9999> 2014-06-09 16:22:12.161952 7f8690f64700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9998> 2014-06-09 16:22:12.161954 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -9997> 2014-06-09 16:22:12.161954 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.765( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 0.765( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) -9996> 2014-06-09 16:22:12.161963 7f8690f64700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 -9995> 2014-06-09 16:22:12.161966 7f8690f64700 1 -- 172.16.96.202:6850/30904 --> osd.1 172.16.96.201:6811/57862 -- pg_info(1 pgs e243:2.3bd) v4 -- ?+0 0x33440000 -9994> 2014-06-09 16:22:12.161966 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) -9993> 2014-06-09 16:22:12.161984 7f8692767700 10 log is not dirty -9992> 2014-06-09 16:22:12.161978 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) -9991> 2014-06-09 16:22:12.161979 7f8690f64700 15 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 -9990> 2014-06-09 16:22:12.161988 7f8692767700 5 queue_transactions existing osr(0.765 0x384c360)/0x384c360 -9989> 2014-06-09 16:22:12.161990 7f8692767700 10 op_submit_start 4148 -9988> 2014-06-09 16:22:12.161991 7f8692767700 5 queue_transactions (trailing journal) 4148 -9987> 2014-06-09 16:22:12.161992 7f8692767700 5 queue_op 0x327fc540 seq 4148 osr(0.765 0x384c360) 828 bytes (queue has 49 ops and 4236663 bytes) -9986> 2014-06-09 16:22:12.161988 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9985> 2014-06-09 16:22:12.161995 7f8692767700 10 op_submit_finish 4148 -9984> 2014-06-09 16:22:12.162001 7f8689f56700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 -9983> 2014-06-09 16:22:12.161997 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.1d5( v 213'531 (0'0,213'531] local-les=243 n=103 ec=118 les/c 243/243 242/242/200) -9982> 2014-06-09 16:22:12.162009 7f8690f64700 10 log is not dirty -9981> 2014-06-09 16:22:12.161994 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 2 ==== osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+516096 (651835643 0 0) 0xc13e6c0 con 0x30c3c2c0 -9980> 2014-06-09 16:22:12.162003 7f8689f56700 1 -- 172.16.96.202:6850/30904 --> osd.3 172.16.96.201:6823/50569 -- pg_info(1 pgs e243:0.3f9) v4 -- ?+0 0x333c2000 -9979> 2014-06-09 16:22:12.162012 7f8690f64700 5 queue_transactions existing osr(2.3bd 0x2ed1010)/0x2ed1010 -9978> 2014-06-09 16:22:12.162015 7f8690f64700 10 op_submit_start 4149 -9977> 2014-06-09 16:22:12.162015 7f8690f64700 5 queue_transactions (trailing journal) 4149 -9976> 2014-06-09 16:22:12.162016 7f8690f64700 5 queue_op 0xc2d4000 seq 4149 osr(2.3bd 0x2ed1010) 861 bytes (queue has 50 ops and 4237524 bytes) -9975> 2014-06-09 16:22:12.162019 7f8690f64700 10 op_submit_finish 4149 -9974> 2014-06-09 16:22:12.162017 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 -9973> 2014-06-09 16:22:12.162026 7f8692767700 10 log is not dirty -9972> 2014-06-09 16:22:12.162030 7f8692767700 5 queue_transactions existing osr(3.1d5 0x3ed09f0)/0x3ed09f0 -9971> 2014-06-09 16:22:12.162021 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9970> 2014-06-09 16:22:12.162032 7f8692767700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9969> 2014-06-09 16:22:12.162037 7f86a5e11700 5 _do_op 0x32524720 seq 4101 osr(0.529 0x2bd7930)/0x2bd7930 start -9968> 2014-06-09 16:22:12.162040 7f86a5e11700 10 _do_transaction on 0x37e4300 -9967> 2014-06-09 16:22:12.162035 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -9966> 2014-06-09 16:22:12.162045 7f8690f64700 10 log is not dirty -9965> 2014-06-09 16:22:12.162045 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9964> 2014-06-09 16:22:12.162041 7f86a68f6700 5 -- op tracker -- , seq: 3662, time: 2014-06-09 16:22:12.160865, event: header_read, op: osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9963> 2014-06-09 16:22:12.162052 7f8689f56700 10 log is not dirty -9962> 2014-06-09 16:22:12.162056 7f8689f56700 5 queue_transactions existing osr(0.3f9 0x2bd7230)/0x2bd7230 -9961> 2014-06-09 16:22:12.162049 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9960> 2014-06-09 16:22:12.162059 7f8689f56700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9959> 2014-06-09 16:22:12.162060 7f86a68f6700 5 -- op tracker -- , seq: 3662, time: 2014-06-09 16:22:12.160866, event: throttled, op: osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9958> 2014-06-09 16:22:12.162063 7f868f761700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9957> 2014-06-09 16:22:12.162061 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -9956> 2014-06-09 16:22:12.162070 7f868df5e700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9955> 2014-06-09 16:22:12.162072 7f8690f64700 10 log is not dirty -9954> 2014-06-09 16:22:12.162082 7f8690763700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9953> 2014-06-09 16:22:12.162082 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9952> 2014-06-09 16:22:12.162076 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9951> 2014-06-09 16:22:12.162087 7f868b759700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9950> 2014-06-09 16:22:12.162070 7f86a68f6700 5 -- op tracker -- , seq: 3662, time: 2014-06-09 16:22:12.161990, event: all_read, op: osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9949> 2014-06-09 16:22:12.162088 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -9948> 2014-06-09 16:22:12.162095 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4101.0.0 <= header.spos 150443.0.0 -9947> 2014-06-09 16:22:12.162096 7f86a68f6700 5 -- op tracker -- , seq: 3662, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9946> 2014-06-09 16:22:12.162098 7f8690f64700 10 log is not dirty -9945> 2014-06-09 16:22:12.162098 7f86a5e11700 5 submit_transaction r = 0 -9944> 2014-06-09 16:22:12.162101 7f868d75d700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9943> 2014-06-09 16:22:12.162102 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 -9942> 2014-06-09 16:22:12.162107 7f868ff62700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9941> 2014-06-09 16:22:12.162112 7f868bf5a700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9940> 2014-06-09 16:22:12.162103 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9939> 2014-06-09 16:22:12.162115 7f86a5e11700 10 _do_op 0x32524720 seq 4101 r = 0, finisher 0x32074fe0 0 -9938> 2014-06-09 16:22:12.162117 7f868e75f700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9937> 2014-06-09 16:22:12.162122 7f8689755700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9936> 2014-06-09 16:22:12.162116 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -9935> 2014-06-09 16:22:12.162124 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3006c560] -9934> 2014-06-09 16:22:12.162119 7f86a68f6700 15 osd.26 243 enqueue_op 0xdfa7200 prio 63 cost 516096 latency 0.001254 osd_op(client.4543.0:33161 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1859584~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 -9933> 2014-06-09 16:22:12.162126 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3006c560] -9932> 2014-06-09 16:22:12.162127 7f8690f64700 10 log is not dirty -9931> 2014-06-09 16:22:12.162127 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9930> 2014-06-09 16:22:12.162128 7f868ef60700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9929> 2014-06-09 16:22:12.162128 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9928> 2014-06-09 16:22:12.162131 7f86a5e11700 10 _finish_op 0x32524720 seq 4101 osr(0.529 0x2bd7930)/0x2bd7930 -9927> 2014-06-09 16:22:12.162133 7f8692f68700 2 waiting 51 > 50 ops || 4238352 > 104857600 -9926> 2014-06-09 16:22:12.162139 7f8691765700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9925> 2014-06-09 16:22:12.162130 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9924> 2014-06-09 16:22:12.162144 7f868cf5c700 2 waiting 51 > 50 ops || 4238379 > 104857600 -9923> 2014-06-09 16:22:12.162143 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -9922> 2014-06-09 16:22:12.162150 7f868a757700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9921> 2014-06-09 16:22:12.162151 7f8690f64700 10 log is not dirty -9920> 2014-06-09 16:22:12.162155 7f868c75b700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9919> 2014-06-09 16:22:12.162154 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9918> 2014-06-09 16:22:12.162160 7f868af58700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9917> 2014-06-09 16:22:12.162162 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] requeue_ops -9916> 2014-06-09 16:22:12.162169 7f8690f64700 10 log is not dirty -9915> 2014-06-09 16:22:12.162176 7f8689f56700 10 op_submit_start 4150 -9914> 2014-06-09 16:22:12.162177 7f8689f56700 5 queue_transactions (trailing journal) 4150 -9913> 2014-06-09 16:22:12.162177 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32074fe0] -9912> 2014-06-09 16:22:12.162179 7f8689f56700 5 queue_op 0xc2d43c0 seq 4150 osr(0.3f9 0x2bd7230) 861 bytes (queue has 50 ops and 4237557 bytes) -9911> 2014-06-09 16:22:12.162173 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9910> 2014-06-09 16:22:12.162183 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32074fe0] -9909> 2014-06-09 16:22:12.162185 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9908> 2014-06-09 16:22:12.162184 7f868f761700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9907> 2014-06-09 16:22:12.162186 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9906> 2014-06-09 16:22:12.162188 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287c20 (1 active) -9905> 2014-06-09 16:22:12.162189 7f868df5e700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9904> 2014-06-09 16:22:12.162184 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -9903> 2014-06-09 16:22:12.162191 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc120 (2 active) -9902> 2014-06-09 16:22:12.162193 7f8690f64700 10 log is not dirty -9901> 2014-06-09 16:22:12.162194 7f8690763700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9900> 2014-06-09 16:22:12.162195 7f86a5e11700 5 _do_op 0x32524a20 seq 4102 osr(0.2a2 0x2bd6c20)/0x2bd6c20 start -9899> 2014-06-09 16:22:12.162197 7f86a5e11700 10 _do_transaction on 0xbdebd40 -9898> 2014-06-09 16:22:12.162199 7f8689f56700 10 op_submit_finish 4150 -9897> 2014-06-09 16:22:12.162199 7f868b759700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9896> 2014-06-09 16:22:12.162202 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9895> 2014-06-09 16:22:12.162197 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9894> 2014-06-09 16:22:12.162203 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9893> 2014-06-09 16:22:12.162210 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -9892> 2014-06-09 16:22:12.162221 7f8690f64700 10 log is not dirty -9891> 2014-06-09 16:22:12.162217 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.052888 7 0.004506 -9890> 2014-06-09 16:22:12.162225 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -9889> 2014-06-09 16:22:12.162228 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -9888> 2014-06-09 16:22:12.162240 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9887> 2014-06-09 16:22:12.162237 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local -9886> 2014-06-09 16:22:12.162238 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9885> 2014-06-09 16:22:12.162245 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -9884> 2014-06-09 16:22:12.162253 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4102.0.0 <= header.spos 150443.0.0 -9883> 2014-06-09 16:22:12.162256 7f86a5e11700 5 submit_transaction r = 0 -9882> 2014-06-09 16:22:12.162250 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -9881> 2014-06-09 16:22:12.162254 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9880> 2014-06-09 16:22:12.162266 7f86a5e11700 10 _do_op 0x32524a20 seq 4102 r = 0, finisher 0x320747a0 0 -9879> 2014-06-09 16:22:12.162262 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -9878> 2014-06-09 16:22:12.162261 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -9877> 2014-06-09 16:22:12.162275 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074620] -9876> 2014-06-09 16:22:12.162269 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -9875> 2014-06-09 16:22:12.162277 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074620] -9874> 2014-06-09 16:22:12.162270 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -9873> 2014-06-09 16:22:12.162278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9872> 2014-06-09 16:22:12.162279 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9871> 2014-06-09 16:22:12.162282 7f86a5e11700 10 _finish_op 0x32524a20 seq 4102 osr(0.2a2 0x2bd6c20)/0x2bd6c20 -9870> 2014-06-09 16:22:12.162279 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -9869> 2014-06-09 16:22:12.162283 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 -9868> 2014-06-09 16:22:12.162292 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x320747a0] -9867> 2014-06-09 16:22:12.162295 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x320747a0] -9866> 2014-06-09 16:22:12.162296 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9865> 2014-06-09 16:22:12.162297 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9864> 2014-06-09 16:22:12.162294 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -9863> 2014-06-09 16:22:12.162290 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) -9862> 2014-06-09 16:22:12.162300 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc120 (1 active) -9861> 2014-06-09 16:22:12.162302 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b680 (2 active) -9860> 2014-06-09 16:22:12.162304 7f86a5e11700 5 _do_op 0x2ea53b60 seq 4103 osr(0.2d9 0x2bd6e90)/0x2bd6e90 start -9859> 2014-06-09 16:22:12.162306 7f86a5e11700 10 _do_transaction on 0xc3f0600 -9858> 2014-06-09 16:22:12.162301 7f8690f64700 20 osd.26 pg_epoch: 243 pg[0.1c8( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop -9857> 2014-06-09 16:22:12.162310 7f8690f64700 10 log is not dirty -9856> 2014-06-09 16:22:12.162302 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) -9855> 2014-06-09 16:22:12.162311 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9854> 2014-06-09 16:22:12.162312 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9853> 2014-06-09 16:22:12.162314 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9852> 2014-06-09 16:22:12.162326 7f8689f56700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 -9851> 2014-06-09 16:22:12.162326 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -9850> 2014-06-09 16:22:12.162335 7f8690f64700 10 log is not dirty -9849> 2014-06-09 16:22:12.162329 7f8689f56700 1 -- 172.16.96.202:6850/30904 --> osd.3 172.16.96.201:6823/50569 -- pg_info(1 pgs e243:1.6b7) v4 -- ?+0 0x31776fc0 -9848> 2014-06-09 16:22:12.162346 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9847> 2014-06-09 16:22:12.162338 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -9846> 2014-06-09 16:22:12.162343 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 -9845> 2014-06-09 16:22:12.162351 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -9844> 2014-06-09 16:22:12.162358 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4103.0.0 <= header.spos 150443.0.0 -9843> 2014-06-09 16:22:12.162362 7f86a5e11700 5 submit_transaction r = 0 -9842> 2014-06-09 16:22:12.162362 7f8690f64700 10 log is not dirty -9841> 2014-06-09 16:22:12.162371 7f86a5e11700 10 _do_op 0x2ea53b60 seq 4103 r = 0, finisher 0xbd68d40 0 -9840> 2014-06-09 16:22:12.162366 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9839> 2014-06-09 16:22:12.162378 7f8689f56700 10 log is not dirty -9838> 2014-06-09 16:22:12.162380 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc40dee0] -9837> 2014-06-09 16:22:12.162382 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc40dee0] -9836> 2014-06-09 16:22:12.162382 7f8689f56700 5 queue_transactions existing osr(1.6b7 0x2ed01d0)/0x2ed01d0 -9835> 2014-06-09 16:22:12.162384 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9834> 2014-06-09 16:22:12.162385 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9833> 2014-06-09 16:22:12.162385 7f8689f56700 10 op_submit_start 4151 -9832> 2014-06-09 16:22:12.162377 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.119200 7 0.000397 -9831> 2014-06-09 16:22:12.162386 7f8689f56700 5 queue_transactions (trailing journal) 4151 -9830> 2014-06-09 16:22:12.162388 7f86a5e11700 10 _finish_op 0x2ea53b60 seq 4103 osr(0.2d9 0x2bd6e90)/0x2bd6e90 -9829> 2014-06-09 16:22:12.162388 7f8689f56700 5 queue_op 0xc2d46c0 seq 4151 osr(1.6b7 0x2ed01d0) 861 bytes (queue has 50 ops and 4237590 bytes) -9828> 2014-06-09 16:22:12.162388 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -9827> 2014-06-09 16:22:12.162395 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd68d40] -9826> 2014-06-09 16:22:12.162398 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd68d40] -9825> 2014-06-09 16:22:12.162397 7f868f761700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9824> 2014-06-09 16:22:12.162399 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9823> 2014-06-09 16:22:12.162400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9822> 2014-06-09 16:22:12.162403 7f868ff62700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9821> 2014-06-09 16:22:12.162403 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b680 (1 active) -9820> 2014-06-09 16:22:12.162398 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9819> 2014-06-09 16:22:12.162405 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e6c0 (2 active) -9818> 2014-06-09 16:22:12.162408 7f868bf5a700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9817> 2014-06-09 16:22:12.162409 7f86a5e11700 5 _do_op 0x11fce4e0 seq 4104 osr(2.285 0x2ed0d50)/0x2ed0d50 start -9816> 2014-06-09 16:22:12.162411 7f86a5e11700 10 _do_transaction on 0xc28ed80 -9815> 2014-06-09 16:22:12.162413 7f8689755700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9814> 2014-06-09 16:22:12.162407 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 -9813> 2014-06-09 16:22:12.162417 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9812> 2014-06-09 16:22:12.162418 7f868ef60700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9811> 2014-06-09 16:22:12.162417 7f8690f64700 5 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -9810> 2014-06-09 16:22:12.162423 7f8692f68700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9809> 2014-06-09 16:22:12.162428 7f8689f56700 10 op_submit_finish 4151 -9808> 2014-06-09 16:22:12.162425 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -9807> 2014-06-09 16:22:12.162432 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -9806> 2014-06-09 16:22:12.162442 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -9805> 2014-06-09 16:22:12.162432 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9804> 2014-06-09 16:22:12.162453 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9803> 2014-06-09 16:22:12.162453 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -9802> 2014-06-09 16:22:12.162455 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.998208 7 0.000485 -9801> 2014-06-09 16:22:12.162462 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9800> 2014-06-09 16:22:12.162468 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4104.0.0 <= header.spos 150443.0.0 -9799> 2014-06-09 16:22:12.162472 7f86a5e11700 5 submit_transaction r = 0 -9798> 2014-06-09 16:22:12.162467 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -9797> 2014-06-09 16:22:12.162479 7f8690f64700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 -9796> 2014-06-09 16:22:12.162477 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9795> 2014-06-09 16:22:12.162481 7f8690f64700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:0.29) v4 -- ?+0 0x3343c540 -9794> 2014-06-09 16:22:12.162490 7f86a5e11700 10 _do_op 0x11fce4e0 seq 4104 r = 0, finisher 0x318c3880 0 -9793> 2014-06-09 16:22:12.162488 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -9792> 2014-06-09 16:22:12.162500 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5e220] -9791> 2014-06-09 16:22:12.162494 7f8690f64700 15 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 -9790> 2014-06-09 16:22:12.162498 7f8689f56700 5 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -9789> 2014-06-09 16:22:12.162502 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9788> 2014-06-09 16:22:12.162507 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -9787> 2014-06-09 16:22:12.162512 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9786> 2014-06-09 16:22:12.162516 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -9785> 2014-06-09 16:22:12.162524 7f8690f64700 10 log is not dirty -9784> 2014-06-09 16:22:12.162521 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.285( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:164 -9783> 2014-06-09 16:22:12.162528 7f8690f64700 5 queue_transactions existing osr(0.29 0x2bd62b0)/0x2bd62b0 -9782> 2014-06-09 16:22:12.162530 7f8690f64700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9781> 2014-06-09 16:22:12.162530 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5e220] -9780> 2014-06-09 16:22:12.162532 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9779> 2014-06-09 16:22:12.162533 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9778> 2014-06-09 16:22:12.162526 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) -9777> 2014-06-09 16:22:12.162536 7f86a5e11700 10 _finish_op 0x11fce4e0 seq 4104 osr(2.285 0x2ed0d50)/0x2ed0d50 -9776> 2014-06-09 16:22:12.162538 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) -9775> 2014-06-09 16:22:12.162546 7f868cf5c700 10 op_submit_start 4152 -9774> 2014-06-09 16:22:12.162548 7f868cf5c700 5 queue_transactions (trailing journal) 4152 -9773> 2014-06-09 16:22:12.162550 7f868cf5c700 5 queue_op 0x324e6780 seq 4152 osr(2.d 0x2ed02b0) 855 bytes (queue has 50 ops and 4237584 bytes) -9772> 2014-06-09 16:22:12.162547 7f8689f56700 10 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9771> 2014-06-09 16:22:12.162555 7f868a757700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9770> 2014-06-09 16:22:12.162559 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 -9769> 2014-06-09 16:22:12.162560 7f868c75b700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9768> 2014-06-09 16:22:12.162565 7f868af58700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9767> 2014-06-09 16:22:12.162562 7f8689f56700 1 -- 172.16.96.202:6850/30904 --> osd.7 172.16.96.201:6851/52636 -- pg_info(1 pgs e243:0.5e4) v4 -- ?+0 0x31777500 -9766> 2014-06-09 16:22:12.162570 7f8691f66700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9765> 2014-06-09 16:22:12.162574 7f8689f56700 15 osd.26 pg_epoch: 243 pg[0.5e4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 -9764> 2014-06-09 16:22:12.162600 7f8692767700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9763> 2014-06-09 16:22:12.162606 7f8689f56700 10 log is not dirty -9762> 2014-06-09 16:22:12.162609 7f868df5e700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9761> 2014-06-09 16:22:12.162611 7f8689f56700 5 queue_transactions existing osr(0.5e4 0x2bd79d0)/0x2bd79d0 -9760> 2014-06-09 16:22:12.162613 7f8690763700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9759> 2014-06-09 16:22:12.162619 7f868b759700 2 waiting 51 > 50 ops || 4238410 > 104857600 -9758> 2014-06-09 16:22:12.162625 7f868ff62700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9757> 2014-06-09 16:22:12.162625 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c3880] -9756> 2014-06-09 16:22:12.162629 7f868bf5a700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9755> 2014-06-09 16:22:12.162630 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c3880] -9754> 2014-06-09 16:22:12.162632 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9753> 2014-06-09 16:22:12.162633 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9752> 2014-06-09 16:22:12.162634 7f8689755700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9751> 2014-06-09 16:22:12.162636 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e6c0 (1 active) -9750> 2014-06-09 16:22:12.162638 7f868ef60700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9749> 2014-06-09 16:22:12.162639 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300a20 (2 active) -9748> 2014-06-09 16:22:12.162644 7f868cf5c700 10 op_submit_finish 4152 -9747> 2014-06-09 16:22:12.162657 7f8692f68700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9746> 2014-06-09 16:22:12.162650 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -9745> 2014-06-09 16:22:12.162665 7f868e75f700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9744> 2014-06-09 16:22:12.162664 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local -9743> 2014-06-09 16:22:12.162677 7f8691765700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9742> 2014-06-09 16:22:12.162673 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -9741> 2014-06-09 16:22:12.162681 7f8689f56700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9740> 2014-06-09 16:22:12.162685 7f8690f64700 2 waiting 51 > 50 ops || 4238442 > 104857600 -9739> 2014-06-09 16:22:12.162682 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9738> 2014-06-09 16:22:12.162691 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -9737> 2014-06-09 16:22:12.162699 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -9736> 2014-06-09 16:22:12.162713 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 -9735> 2014-06-09 16:22:12.162722 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -9734> 2014-06-09 16:22:12.162730 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.4b5( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop -9733> 2014-06-09 16:22:12.162739 7f868cf5c700 10 log is not dirty -9732> 2014-06-09 16:22:12.162744 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -9731> 2014-06-09 16:22:12.162756 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -9730> 2014-06-09 16:22:12.162764 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -9729> 2014-06-09 16:22:12.162773 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9728> 2014-06-09 16:22:12.162781 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -9727> 2014-06-09 16:22:12.162789 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -9726> 2014-06-09 16:22:12.162801 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 -9725> 2014-06-09 16:22:12.162811 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -9724> 2014-06-09 16:22:12.162818 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[0.69f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -9723> 2014-06-09 16:22:12.162827 7f868cf5c700 10 log is not dirty -9722> 2014-06-09 16:22:12.162852 7f868cf5c700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40b80 (19 active) -9721> 2014-06-09 16:22:12.162859 7f868cf5c700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -9720> 2014-06-09 16:22:12.162863 7f868cf5c700 10 osd.26 243 dequeue_op 0x32827b00 prio 63 cost 516096 latency 0.004916 osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 pg pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] -9719> 2014-06-09 16:22:12.162887 7f868cf5c700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.162882, event: reached_pg, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9718> 2014-06-09 16:22:12.162904 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -9717> 2014-06-09 16:22:12.162918 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x32827b00 -9716> 2014-06-09 16:22:12.162929 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 may_write -> write-ordered flags ack+ondisk+retry+write -9715> 2014-06-09 16:22:12.162947 7f868cf5c700 3 osd.26 pg_epoch: 243 pg[3.77c( v 213'558 (0'0,213'558] local-les=243 n=109 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op dup client.4543.0:33160 was 213'552 -9714> 2014-06-09 16:22:12.162961 7f868cf5c700 1 -- 172.16.96.202:6830/30904 --> 172.16.96.220:0/1123776 -- osd_op_reply(33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] v213'552 uv552 ondisk = 0) v6 -- ?+0 0x9a08280 con 0x30c3c2c0 -9713> 2014-06-09 16:22:12.162974 7f868cf5c700 10 osd.26 243 dequeue_op 0x32827b00 finish -9712> 2014-06-09 16:22:12.162983 7f868cf5c700 5 -- op tracker -- , seq: 3661, time: 2014-06-09 16:22:12.162977, event: done, op: osd_op(client.4543.0:33160 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 1343488~516096] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9711> 2014-06-09 16:22:12.162996 7f868cf5c700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -9710> 2014-06-09 16:22:12.163019 7f868cf5c700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40b80 (20 active) -9709> 2014-06-09 16:22:12.163023 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9708> 2014-06-09 16:22:12.163035 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.023293 7 0.000477 -9707> 2014-06-09 16:22:12.163047 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -9706> 2014-06-09 16:22:12.163058 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9705> 2014-06-09 16:22:12.163068 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -9704> 2014-06-09 16:22:12.163079 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -9703> 2014-06-09 16:22:12.163087 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -9702> 2014-06-09 16:22:12.163094 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -9701> 2014-06-09 16:22:12.163103 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -9700> 2014-06-09 16:22:12.163115 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -9699> 2014-06-09 16:22:12.163124 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -9698> 2014-06-09 16:22:12.163134 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -9697> 2014-06-09 16:22:12.163143 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9696> 2014-06-09 16:22:12.163156 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 -9695> 2014-06-09 16:22:12.163158 7f868cf5c700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:1.3ef) v4 -- ?+0 0xca42000 -9694> 2014-06-09 16:22:12.163172 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 -9693> 2014-06-09 16:22:12.163205 7f868cf5c700 10 log is not dirty -9692> 2014-06-09 16:22:12.163209 7f868cf5c700 5 queue_transactions existing osr(1.3ef 0x384d3f0)/0x384d3f0 -9691> 2014-06-09 16:22:12.163212 7f868cf5c700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9690> 2014-06-09 16:22:12.163217 7f868d75d700 10 op_submit_start 4153 -9689> 2014-06-09 16:22:12.163218 7f868d75d700 5 queue_transactions (trailing journal) 4153 -9688> 2014-06-09 16:22:12.163220 7f868d75d700 5 queue_op 0x16fd4c60 seq 4153 osr(1.1e8 0x384cdf0) 828 bytes (queue has 50 ops and 4237584 bytes) -9687> 2014-06-09 16:22:12.163223 7f868d75d700 10 op_submit_finish 4153 -9686> 2014-06-09 16:22:12.163227 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6ba( v 213'670 (0'0,213'670] local-les=243 n=133 ec=118 les/c 243/243 242/242/207) -9685> 2014-06-09 16:22:12.163261 7f868d75d700 10 log is not dirty -9684> 2014-06-09 16:22:12.163266 7f868d75d700 5 queue_transactions existing osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 -9683> 2014-06-09 16:22:12.163269 7f868d75d700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9682> 2014-06-09 16:22:12.163274 7f868f761700 2 waiting 51 > 50 ops || 4238412 > 104857600 -9681> 2014-06-09 16:22:12.163277 7f86a5e11700 5 _do_op 0xc2ea0c0 seq 4105 osr(3.d3 0x3ed0570)/0x3ed0570 start -9680> 2014-06-09 16:22:12.163281 7f86a5e11700 10 _do_transaction on 0xc31fbc0 -9679> 2014-06-09 16:22:12.163288 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9678> 2014-06-09 16:22:12.163333 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9677> 2014-06-09 16:22:12.163350 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4105.0.0 <= header.spos 150443.0.0 -9676> 2014-06-09 16:22:12.163354 7f86a5e11700 5 submit_transaction r = 0 -9675> 2014-06-09 16:22:12.163371 7f86a5e11700 10 _do_op 0xc2ea0c0 seq 4105 r = 0, finisher 0x31434ea0 0 -9674> 2014-06-09 16:22:12.163383 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6db40] -9673> 2014-06-09 16:22:12.163386 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery -9672> 2014-06-09 16:22:12.163397 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays -9671> 2014-06-09 16:22:12.163408 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.d3( v 214'647 (0'0,214'647] local-les=243 n=126 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1059 -9670> 2014-06-09 16:22:12.163418 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6db40] -9669> 2014-06-09 16:22:12.163420 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9668> 2014-06-09 16:22:12.163421 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9667> 2014-06-09 16:22:12.163425 7f86a5e11700 10 _finish_op 0xc2ea0c0 seq 4105 osr(3.d3 0x3ed0570)/0x3ed0570 -9666> 2014-06-09 16:22:12.163436 7f868c75b700 10 op_submit_start 4154 -9665> 2014-06-09 16:22:12.163438 7f868c75b700 5 queue_transactions (trailing journal) 4154 -9664> 2014-06-09 16:22:12.163440 7f868c75b700 5 queue_op 0xc34a6c0 seq 4154 osr(0.230 0x2bd6a10) 861 bytes (queue has 50 ops and 4237587 bytes) -9663> 2014-06-09 16:22:12.163445 7f868af58700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9662> 2014-06-09 16:22:12.163450 7f8691f66700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9661> 2014-06-09 16:22:12.163455 7f8692767700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9660> 2014-06-09 16:22:12.163459 7f868df5e700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9659> 2014-06-09 16:22:12.163465 7f8690763700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9658> 2014-06-09 16:22:12.163471 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31434ea0] -9657> 2014-06-09 16:22:12.163472 7f868b759700 2 waiting 51 > 50 ops || 4238413 > 104857600 -9656> 2014-06-09 16:22:12.163475 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31434ea0] -9655> 2014-06-09 16:22:12.163477 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9654> 2014-06-09 16:22:12.163478 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9653> 2014-06-09 16:22:12.163481 7f868ff62700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9652> 2014-06-09 16:22:12.163481 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300a20 (1 active) -9651> 2014-06-09 16:22:12.163484 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc339b00 (2 active) -9650> 2014-06-09 16:22:12.163486 7f868bf5a700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9649> 2014-06-09 16:22:12.163488 7f868c75b700 10 op_submit_finish 4154 -9648> 2014-06-09 16:22:12.163490 7f8689755700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9647> 2014-06-09 16:22:12.163495 7f868ef60700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9646> 2014-06-09 16:22:12.163500 7f8692f68700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9645> 2014-06-09 16:22:12.163493 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9644> 2014-06-09 16:22:12.163506 7f868e75f700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9643> 2014-06-09 16:22:12.163512 7f8691765700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9642> 2014-06-09 16:22:12.163517 7f8689f56700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9641> 2014-06-09 16:22:12.163508 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] exit Started/Primary/Active/Activating 1.052078 7 0.005100 -9640> 2014-06-09 16:22:12.163522 7f8690f64700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9639> 2014-06-09 16:22:12.163522 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] enter Started/Primary/Active/Recovered -9638> 2014-06-09 16:22:12.163533 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] needs_recovery is recovered -9637> 2014-06-09 16:22:12.163545 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -9636> 2014-06-09 16:22:12.163555 7f868c75b700 5 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] enter Started/Primary/Active/Clean -9635> 2014-06-09 16:22:12.163565 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] finish_recovery -9634> 2014-06-09 16:22:12.163574 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active] clear_recovery_state -9633> 2014-06-09 16:22:12.163585 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'739 mlcod 214'739 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -9632> 2014-06-09 16:22:12.163597 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'739 mlcod 214'739 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -9631> 2014-06-09 16:22:12.163608 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 214'739 mlcod 214'739 active+clean] share_pg_info -9630> 2014-06-09 16:22:12.163636 7f868c75b700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 -9629> 2014-06-09 16:22:12.163639 7f868c75b700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:3.224) v4 -- ?+0 0x2f046000 -9628> 2014-06-09 16:22:12.163653 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.224( v 243'740 (0'0,243'740] local-les=243 n=136 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 214'739 mlcod 214'739 active+clean] publish_stats_to_osd 243:1239 -9627> 2014-06-09 16:22:12.163689 7f868c75b700 10 log is not dirty -9626> 2014-06-09 16:22:12.163693 7f868c75b700 5 queue_transactions existing osr(3.224 0x3ed0c80)/0x3ed0c80 -9625> 2014-06-09 16:22:12.163696 7f868c75b700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9624> 2014-06-09 16:22:12.163700 7f868cf5c700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9623> 2014-06-09 16:22:12.163705 7f868d75d700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9622> 2014-06-09 16:22:12.163710 7f868f761700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9621> 2014-06-09 16:22:12.163714 7f868a757700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9620> 2014-06-09 16:22:12.163718 7f86a5e11700 5 _do_op 0xc2ea060 seq 4106 osr(0.633 0x2bd7c50)/0x2bd7c50 start -9619> 2014-06-09 16:22:12.163722 7f86a5e11700 10 _do_transaction on 0xc29b5c0 -9618> 2014-06-09 16:22:12.163729 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9617> 2014-06-09 16:22:12.163771 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9616> 2014-06-09 16:22:12.163788 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4106.0.0 <= header.spos 150443.0.0 -9615> 2014-06-09 16:22:12.163793 7f86a5e11700 5 submit_transaction r = 0 -9614> 2014-06-09 16:22:12.163826 7f86a5e11700 10 _do_op 0xc2ea060 seq 4106 r = 0, finisher 0x189abd80 0 -9613> 2014-06-09 16:22:12.163837 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcf8e0] -9612> 2014-06-09 16:22:12.163840 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9611> 2014-06-09 16:22:12.163850 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9610> 2014-06-09 16:22:12.163859 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.633( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:182 -9609> 2014-06-09 16:22:12.163869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcf8e0] -9608> 2014-06-09 16:22:12.163870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9607> 2014-06-09 16:22:12.163872 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9606> 2014-06-09 16:22:12.163875 7f86a5e11700 10 _finish_op 0xc2ea060 seq 4106 osr(0.633 0x2bd7c50)/0x2bd7c50 -9605> 2014-06-09 16:22:12.163887 7f8691f66700 10 op_submit_start 4155 -9604> 2014-06-09 16:22:12.163889 7f8691f66700 5 queue_transactions (trailing journal) 4155 -9603> 2014-06-09 16:22:12.163891 7f8691f66700 5 queue_op 0x16097080 seq 4155 osr(1.54c 0x384dad0) 861 bytes (queue has 50 ops and 4237587 bytes) -9602> 2014-06-09 16:22:12.163896 7f8692767700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9601> 2014-06-09 16:22:12.163901 7f868df5e700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9600> 2014-06-09 16:22:12.163906 7f8690763700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9599> 2014-06-09 16:22:12.163912 7f868b759700 2 waiting 51 > 50 ops || 4238413 > 104857600 -9598> 2014-06-09 16:22:12.163915 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189abd80] -9597> 2014-06-09 16:22:12.163920 7f868ff62700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9596> 2014-06-09 16:22:12.163923 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189abd80] -9595> 2014-06-09 16:22:12.163924 7f868bf5a700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9594> 2014-06-09 16:22:12.163925 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9593> 2014-06-09 16:22:12.163927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9592> 2014-06-09 16:22:12.163929 7f8689755700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9591> 2014-06-09 16:22:12.163930 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc339b00 (1 active) -9590> 2014-06-09 16:22:12.163933 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52d680 (2 active) -9589> 2014-06-09 16:22:12.163934 7f868ef60700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9588> 2014-06-09 16:22:12.163938 7f8691f66700 10 op_submit_finish 4155 -9587> 2014-06-09 16:22:12.163939 7f8692f68700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9586> 2014-06-09 16:22:12.163944 7f868e75f700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9585> 2014-06-09 16:22:12.163942 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9584> 2014-06-09 16:22:12.163951 7f8691765700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9583> 2014-06-09 16:22:12.163957 7f8689f56700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9582> 2014-06-09 16:22:12.163963 7f8690f64700 2 waiting 51 > 50 ops || 4238445 > 104857600 -9581> 2014-06-09 16:22:12.163955 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.995027 7 0.000282 -9580> 2014-06-09 16:22:12.163968 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -9579> 2014-06-09 16:22:12.163978 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9578> 2014-06-09 16:22:12.163988 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -9577> 2014-06-09 16:22:12.163999 7f8691f66700 5 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -9576> 2014-06-09 16:22:12.164008 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -9575> 2014-06-09 16:22:12.164015 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -9574> 2014-06-09 16:22:12.164026 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -9573> 2014-06-09 16:22:12.164037 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -9572> 2014-06-09 16:22:12.164047 7f8691f66700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9571> 2014-06-09 16:22:12.164060 7f8691f66700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 -9570> 2014-06-09 16:22:12.164063 7f8691f66700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:1.25a) v4 -- ?+0 0x317776c0 -9569> 2014-06-09 16:22:12.164076 7f8691f66700 15 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:163 -9568> 2014-06-09 16:22:12.164109 7f8691f66700 10 log is not dirty -9567> 2014-06-09 16:22:12.164113 7f8691f66700 5 queue_transactions existing osr(1.25a 0x384cfb0)/0x384cfb0 -9566> 2014-06-09 16:22:12.164116 7f8691f66700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9565> 2014-06-09 16:22:12.164120 7f868c75b700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9564> 2014-06-09 16:22:12.164125 7f868cf5c700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9563> 2014-06-09 16:22:12.164130 7f868d75d700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9562> 2014-06-09 16:22:12.164134 7f868f761700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9561> 2014-06-09 16:22:12.164138 7f868a757700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9560> 2014-06-09 16:22:12.164143 7f868af58700 2 waiting 51 > 50 ops || 4238448 > 104857600 -9559> 2014-06-09 16:22:12.164147 7f86a5e11700 5 _do_op 0xc2ebb00 seq 4107 osr(1.b9 0x384c630)/0x384c630 start -9558> 2014-06-09 16:22:12.164151 7f86a5e11700 10 _do_transaction on 0xc2d8240 -9557> 2014-06-09 16:22:12.164157 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9556> 2014-06-09 16:22:12.164198 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9555> 2014-06-09 16:22:12.164215 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4107.0.0 <= header.spos 150443.0.0 -9554> 2014-06-09 16:22:12.164220 7f86a5e11700 5 submit_transaction r = 0 -9553> 2014-06-09 16:22:12.164235 7f86a5e11700 10 _do_op 0xc2ebb00 seq 4107 r = 0, finisher 0x31457460 0 -9552> 2014-06-09 16:22:12.164246 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3206fb80] -9551> 2014-06-09 16:22:12.164248 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9550> 2014-06-09 16:22:12.164259 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9549> 2014-06-09 16:22:12.164267 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:188 -9548> 2014-06-09 16:22:12.164276 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3206fb80] -9547> 2014-06-09 16:22:12.164278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9546> 2014-06-09 16:22:12.164278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9545> 2014-06-09 16:22:12.164282 7f86a5e11700 10 _finish_op 0xc2ebb00 seq 4107 osr(1.b9 0x384c630)/0x384c630 -9544> 2014-06-09 16:22:12.164293 7f868df5e700 10 op_submit_start 4156 -9543> 2014-06-09 16:22:12.164294 7f868df5e700 5 queue_transactions (trailing journal) 4156 -9542> 2014-06-09 16:22:12.164296 7f868df5e700 5 queue_op 0xbf7e000 seq 4156 osr(3.172 0x3ed06e0) 828 bytes (queue has 50 ops and 4237557 bytes) -9541> 2014-06-09 16:22:12.164301 7f8690763700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9540> 2014-06-09 16:22:12.164308 7f868b759700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9539> 2014-06-09 16:22:12.164313 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31457460] -9538> 2014-06-09 16:22:12.164314 7f868ff62700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9537> 2014-06-09 16:22:12.164317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31457460] -9536> 2014-06-09 16:22:12.164318 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9535> 2014-06-09 16:22:12.164319 7f868bf5a700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9534> 2014-06-09 16:22:12.164320 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9533> 2014-06-09 16:22:12.164323 7f8689755700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9532> 2014-06-09 16:22:12.164323 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52d680 (1 active) -9531> 2014-06-09 16:22:12.164327 7f868ef60700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9530> 2014-06-09 16:22:12.164335 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f560 (2 active) -9529> 2014-06-09 16:22:12.164341 7f868df5e700 10 op_submit_finish 4156 -9528> 2014-06-09 16:22:12.164340 7f8692f68700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9527> 2014-06-09 16:22:12.164346 7f868e75f700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9526> 2014-06-09 16:22:12.164351 7f8691765700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9525> 2014-06-09 16:22:12.164355 7f8689f56700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9524> 2014-06-09 16:22:12.164345 7f868df5e700 10 osd.26 pg_epoch: 243 pg[3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.400( v 213'625 (0'0,213'625] local-les=243 n=125 ec=118 les/c 243/212 242/242/211) -9523> 2014-06-09 16:22:12.164360 7f8690f64700 2 waiting 51 > 50 ops || 4238415 > 104857600 -9522> 2014-06-09 16:22:12.164382 7f868df5e700 10 log is not dirty -9521> 2014-06-09 16:22:12.164387 7f868df5e700 5 queue_transactions existing osr(3.400 0x3ed1260)/0x3ed1260 -9520> 2014-06-09 16:22:12.164393 7f868df5e700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9519> 2014-06-09 16:22:12.164399 7f8691f66700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9518> 2014-06-09 16:22:12.164404 7f868c75b700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9517> 2014-06-09 16:22:12.164408 7f868cf5c700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9516> 2014-06-09 16:22:12.164413 7f868d75d700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9515> 2014-06-09 16:22:12.164417 7f868f761700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9514> 2014-06-09 16:22:12.164422 7f868a757700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9513> 2014-06-09 16:22:12.164425 7f868af58700 2 waiting 51 > 50 ops || 4238418 > 104857600 -9512> 2014-06-09 16:22:12.164429 7f8692767700 2 waiting 51 > 50 ops || 4238385 > 104857600 -9511> 2014-06-09 16:22:12.164433 7f86a5e11700 5 _do_op 0x2c06e900 seq 4108 osr(3.505 0x3ed1700)/0x3ed1700 start -9510> 2014-06-09 16:22:12.164437 7f86a5e11700 10 _do_transaction on 0xc1120c0 -9509> 2014-06-09 16:22:12.164443 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9508> 2014-06-09 16:22:12.164485 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9507> 2014-06-09 16:22:12.164502 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4108.0.0 <= header.spos 150443.0.0 -9506> 2014-06-09 16:22:12.164507 7f86a5e11700 5 submit_transaction r = 0 -9505> 2014-06-09 16:22:12.164520 7f86a5e11700 10 _do_op 0x2c06e900 seq 4108 r = 0, finisher 0x3050b520 0 -9504> 2014-06-09 16:22:12.164531 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9e5e0] -9503> 2014-06-09 16:22:12.164534 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery -9502> 2014-06-09 16:22:12.164546 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays -9501> 2014-06-09 16:22:12.164557 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.505( v 213'732 (0'0,213'732] local-les=243 n=143 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1224 -9500> 2014-06-09 16:22:12.164567 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9e5e0] -9499> 2014-06-09 16:22:12.164569 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9498> 2014-06-09 16:22:12.164570 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9497> 2014-06-09 16:22:12.164574 7f86a5e11700 10 _finish_op 0x2c06e900 seq 4108 osr(3.505 0x3ed1700)/0x3ed1700 -9496> 2014-06-09 16:22:12.164588 7f868b759700 10 op_submit_start 4157 -9495> 2014-06-09 16:22:12.164591 7f868b759700 5 queue_transactions (trailing journal) 4157 -9494> 2014-06-09 16:22:12.164590 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050b520] -9493> 2014-06-09 16:22:12.164594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050b520] -9492> 2014-06-09 16:22:12.164593 7f868b759700 5 queue_op 0xc88bce0 seq 4157 osr(0.9d 0x2bd6490) 826 bytes (queue has 50 ops and 4237522 bytes) -9491> 2014-06-09 16:22:12.164596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9490> 2014-06-09 16:22:12.164597 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9489> 2014-06-09 16:22:12.164598 7f868ff62700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9488> 2014-06-09 16:22:12.164600 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f560 (1 active) -9487> 2014-06-09 16:22:12.164603 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7d40 (2 active) -9486> 2014-06-09 16:22:12.164604 7f868bf5a700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9485> 2014-06-09 16:22:12.164607 7f86a5e11700 5 _do_op 0xc2ebec0 seq 4109 osr(0.ca 0x2bd6590)/0x2bd6590 start -9484> 2014-06-09 16:22:12.164610 7f86a5e11700 10 _do_transaction on 0xc0d9980 -9483> 2014-06-09 16:22:12.164609 7f868b759700 10 op_submit_finish 4157 -9482> 2014-06-09 16:22:12.164616 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9481> 2014-06-09 16:22:12.164614 7f868b759700 10 osd.26 pg_epoch: 243 pg[1.c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.c9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -9480> 2014-06-09 16:22:12.164651 7f868b759700 10 log is not dirty -9479> 2014-06-09 16:22:12.164655 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9478> 2014-06-09 16:22:12.164657 7f868b759700 5 queue_transactions existing osr(1.c9 0x384c9f0)/0x384c9f0 -9477> 2014-06-09 16:22:12.164663 7f868b759700 2 waiting 51 > 50 ops || 4238348 > 104857600 -9476> 2014-06-09 16:22:12.164668 7f8689755700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9475> 2014-06-09 16:22:12.164673 7f868ef60700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9474> 2014-06-09 16:22:12.164672 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4109.0.0 <= header.spos 150443.0.0 -9473> 2014-06-09 16:22:12.164677 7f86a5e11700 5 submit_transaction r = 0 -9472> 2014-06-09 16:22:12.164678 7f8692f68700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9471> 2014-06-09 16:22:12.164684 7f868e75f700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9470> 2014-06-09 16:22:12.164689 7f8691765700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9469> 2014-06-09 16:22:12.164691 7f86a5e11700 10 _do_op 0xc2ebec0 seq 4109 r = 0, finisher 0xbef1640 0 -9468> 2014-06-09 16:22:12.164694 7f8689f56700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9467> 2014-06-09 16:22:12.164698 7f8690f64700 2 waiting 51 > 50 ops || 4238380 > 104857600 -9466> 2014-06-09 16:22:12.164702 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd74e00] -9465> 2014-06-09 16:22:12.164704 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9464> 2014-06-09 16:22:12.164714 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9463> 2014-06-09 16:22:12.164724 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 -9462> 2014-06-09 16:22:12.164733 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd74e00] -9461> 2014-06-09 16:22:12.164735 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9460> 2014-06-09 16:22:12.164737 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9459> 2014-06-09 16:22:12.164740 7f868df5e700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9458> 2014-06-09 16:22:12.164746 7f8691f66700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9457> 2014-06-09 16:22:12.164750 7f868c75b700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9456> 2014-06-09 16:22:12.164755 7f868cf5c700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9455> 2014-06-09 16:22:12.164759 7f868d75d700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9454> 2014-06-09 16:22:12.164764 7f868f761700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9453> 2014-06-09 16:22:12.164768 7f868a757700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9452> 2014-06-09 16:22:12.164772 7f868af58700 2 waiting 51 > 50 ops || 4238383 > 104857600 -9451> 2014-06-09 16:22:12.164776 7f8692767700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9450> 2014-06-09 16:22:12.164780 7f8690763700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9449> 2014-06-09 16:22:12.164784 7f86a5e11700 10 _finish_op 0xc2ebec0 seq 4109 osr(0.ca 0x2bd6590)/0x2bd6590 -9448> 2014-06-09 16:22:12.164794 7f868ff62700 10 op_submit_start 4158 -9447> 2014-06-09 16:22:12.164796 7f868ff62700 5 queue_transactions (trailing journal) 4158 -9446> 2014-06-09 16:22:12.164796 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef1640] -9445> 2014-06-09 16:22:12.164798 7f868ff62700 5 queue_op 0x30a2c60 seq 4158 osr(1.32f 0x384d350) 828 bytes (queue has 50 ops and 4237492 bytes) -9444> 2014-06-09 16:22:12.164800 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef1640] -9443> 2014-06-09 16:22:12.164802 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9442> 2014-06-09 16:22:12.164803 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9441> 2014-06-09 16:22:12.164804 7f868bf5a700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9440> 2014-06-09 16:22:12.164806 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7d40 (1 active) -9439> 2014-06-09 16:22:12.164809 7f868b759700 2 waiting 51 > 50 ops || 4238318 > 104857600 -9438> 2014-06-09 16:22:12.164809 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52eea0 (2 active) -9437> 2014-06-09 16:22:12.164813 7f8689755700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9436> 2014-06-09 16:22:12.164814 7f86a5e11700 5 _do_op 0x2c06e0c0 seq 4110 osr(0.89 0x2bd64a0)/0x2bd64a0 start -9435> 2014-06-09 16:22:12.164817 7f86a5e11700 10 _do_transaction on 0xbff6a80 -9434> 2014-06-09 16:22:12.164818 7f868ef60700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9433> 2014-06-09 16:22:12.164823 7f868ff62700 10 op_submit_finish 4158 -9432> 2014-06-09 16:22:12.164823 7f8692f68700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9431> 2014-06-09 16:22:12.164824 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9430> 2014-06-09 16:22:12.164826 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.31f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -9429> 2014-06-09 16:22:12.164865 7f868ff62700 10 log is not dirty -9428> 2014-06-09 16:22:12.164864 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9427> 2014-06-09 16:22:12.164870 7f868ff62700 5 queue_transactions existing osr(0.31f 0x2bd6d50)/0x2bd6d50 -9426> 2014-06-09 16:22:12.164872 7f868ff62700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9425> 2014-06-09 16:22:12.164877 7f868e75f700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9424> 2014-06-09 16:22:12.164881 7f8691765700 2 waiting 51 > 50 ops || 4238353 > 104857600 -9423> 2014-06-09 16:22:12.164880 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4110.0.0 <= header.spos 150443.0.0 -9422> 2014-06-09 16:22:12.164885 7f86a5e11700 5 submit_transaction r = 0 -9421> 2014-06-09 16:22:12.164886 7f8689f56700 2 waiting 51 > 50 ops || 4238353 > 104857600 -9420> 2014-06-09 16:22:12.164891 7f8690f64700 2 waiting 51 > 50 ops || 4238350 > 104857600 -9419> 2014-06-09 16:22:12.164898 7f868df5e700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9418> 2014-06-09 16:22:12.164904 7f8691f66700 2 waiting 51 > 50 ops || 4238353 > 104857600 -9417> 2014-06-09 16:22:12.164908 7f868c75b700 2 waiting 51 > 50 ops || 4238353 > 104857600 -9416> 2014-06-09 16:22:12.164913 7f868cf5c700 2 waiting 51 > 50 ops || 4238353 > 104857600 -9415> 2014-06-09 16:22:12.164917 7f868d75d700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9414> 2014-06-09 16:22:12.164921 7f868f761700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9413> 2014-06-09 16:22:12.164926 7f868a757700 2 waiting 51 > 50 ops || 4238353 > 104857600 -9412> 2014-06-09 16:22:12.164930 7f868af58700 2 waiting 51 > 50 ops || 4238353 > 104857600 -9411> 2014-06-09 16:22:12.164934 7f8692767700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9410> 2014-06-09 16:22:12.164938 7f8690763700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9409> 2014-06-09 16:22:12.164953 7f86a5e11700 10 _do_op 0x2c06e0c0 seq 4110 r = 0, finisher 0xbd79ba0 0 -9408> 2014-06-09 16:22:12.164963 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x18742140] -9407> 2014-06-09 16:22:12.164965 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9406> 2014-06-09 16:22:12.164975 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9405> 2014-06-09 16:22:12.164984 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.89( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -9404> 2014-06-09 16:22:12.164993 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x18742140] -9403> 2014-06-09 16:22:12.164995 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9402> 2014-06-09 16:22:12.164996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9401> 2014-06-09 16:22:12.165000 7f86a5e11700 10 _finish_op 0x2c06e0c0 seq 4110 osr(0.89 0x2bd64a0)/0x2bd64a0 -9400> 2014-06-09 16:22:12.165009 7f868bf5a700 10 op_submit_start 4159 -9399> 2014-06-09 16:22:12.165012 7f868bf5a700 5 queue_transactions (trailing journal) 4159 -9398> 2014-06-09 16:22:12.165012 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd79ba0] -9397> 2014-06-09 16:22:12.165014 7f868bf5a700 5 queue_op 0x324e76e0 seq 4159 osr(2.40c 0x2ed1250) 828 bytes (queue has 50 ops and 4237462 bytes) -9396> 2014-06-09 16:22:12.165019 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd79ba0] -9395> 2014-06-09 16:22:12.165019 7f868b759700 2 waiting 51 > 50 ops || 4238288 > 104857600 -9394> 2014-06-09 16:22:12.165021 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9393> 2014-06-09 16:22:12.165022 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9392> 2014-06-09 16:22:12.165024 7f8689755700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9391> 2014-06-09 16:22:12.165025 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52eea0 (1 active) -9390> 2014-06-09 16:22:12.165028 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e120 (2 active) -9389> 2014-06-09 16:22:12.165028 7f868ef60700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9388> 2014-06-09 16:22:12.165032 7f86a5e11700 5 _do_op 0x2c06f080 seq 4111 osr(0.e2 0x2bd6560)/0x2bd6560 start -9387> 2014-06-09 16:22:12.165034 7f868bf5a700 10 op_submit_finish 4159 -9386> 2014-06-09 16:22:12.165034 7f8692f68700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9385> 2014-06-09 16:22:12.165035 7f86a5e11700 10 _do_transaction on 0x4734180 -9384> 2014-06-09 16:22:12.165042 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9383> 2014-06-09 16:22:12.165037 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 243/191 242/242/223) -9382> 2014-06-09 16:22:12.165073 7f868bf5a700 10 log is not dirty -9381> 2014-06-09 16:22:12.165074 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9380> 2014-06-09 16:22:12.165079 7f868bf5a700 5 queue_transactions existing osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 -9379> 2014-06-09 16:22:12.165081 7f868bf5a700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9378> 2014-06-09 16:22:12.165086 7f868ff62700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9377> 2014-06-09 16:22:12.165086 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4111.0.0 <= header.spos 150443.0.0 -9376> 2014-06-09 16:22:12.165088 7f86a5e11700 5 submit_transaction r = 0 -9375> 2014-06-09 16:22:12.165091 7f868e75f700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9374> 2014-06-09 16:22:12.165096 7f8691765700 2 waiting 51 > 50 ops || 4238323 > 104857600 -9373> 2014-06-09 16:22:12.165097 7f86a5e11700 10 _do_op 0x2c06f080 seq 4111 r = 0, finisher 0x1322c820 0 -9372> 2014-06-09 16:22:12.165100 7f8689f56700 2 waiting 51 > 50 ops || 4238323 > 104857600 -9371> 2014-06-09 16:22:12.165105 7f8690f64700 2 waiting 51 > 50 ops || 4238320 > 104857600 -9370> 2014-06-09 16:22:12.165106 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205cfc0] -9369> 2014-06-09 16:22:12.165110 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9368> 2014-06-09 16:22:12.165119 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9367> 2014-06-09 16:22:12.165128 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.e2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 -9366> 2014-06-09 16:22:12.165138 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205cfc0] -9365> 2014-06-09 16:22:12.165140 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9364> 2014-06-09 16:22:12.165141 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9363> 2014-06-09 16:22:12.165145 7f868df5e700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9362> 2014-06-09 16:22:12.165150 7f8691f66700 2 waiting 51 > 50 ops || 4238323 > 104857600 -9361> 2014-06-09 16:22:12.165155 7f868c75b700 2 waiting 51 > 50 ops || 4238323 > 104857600 -9360> 2014-06-09 16:22:12.165160 7f868cf5c700 2 waiting 51 > 50 ops || 4238323 > 104857600 -9359> 2014-06-09 16:22:12.165160 7f86a5e11700 10 _finish_op 0x2c06f080 seq 4111 osr(0.e2 0x2bd6560)/0x2bd6560 -9358> 2014-06-09 16:22:12.165174 7f868f761700 10 op_submit_start 4160 -9357> 2014-06-09 16:22:12.165176 7f868f761700 5 queue_transactions (trailing journal) 4160 -9356> 2014-06-09 16:22:12.165177 7f868f761700 5 queue_op 0x2e21c20 seq 4160 osr(0.616 0x2bd7950) 828 bytes (queue has 50 ops and 4237432 bytes) -9355> 2014-06-09 16:22:12.165183 7f868a757700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9354> 2014-06-09 16:22:12.165187 7f868af58700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9353> 2014-06-09 16:22:12.165192 7f8692767700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9352> 2014-06-09 16:22:12.165196 7f8690763700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9351> 2014-06-09 16:22:12.165200 7f868d75d700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9350> 2014-06-09 16:22:12.165211 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322c820] -9349> 2014-06-09 16:22:12.165214 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322c820] -9348> 2014-06-09 16:22:12.165216 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9347> 2014-06-09 16:22:12.165218 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9346> 2014-06-09 16:22:12.165221 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e120 (1 active) -9345> 2014-06-09 16:22:12.165223 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f9e0 (2 active) -9344> 2014-06-09 16:22:12.165229 7f868f761700 10 op_submit_finish 4160 -9343> 2014-06-09 16:22:12.165232 7f8689755700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9342> 2014-06-09 16:22:12.165239 7f868ef60700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9341> 2014-06-09 16:22:12.165244 7f8692f68700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9340> 2014-06-09 16:22:12.165250 7f868bf5a700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9339> 2014-06-09 16:22:12.165250 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x153599f0 (19 active) -9338> 2014-06-09 16:22:12.165255 7f868ff62700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9337> 2014-06-09 16:22:12.165258 7f868f761700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -9336> 2014-06-09 16:22:12.165259 7f868e75f700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9335> 2014-06-09 16:22:12.165264 7f8691765700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9334> 2014-06-09 16:22:12.165268 7f8689f56700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9333> 2014-06-09 16:22:12.165272 7f8690f64700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9332> 2014-06-09 16:22:12.165263 7f868f761700 10 osd.26 243 dequeue_op 0x16bd6800 prio 63 cost 45056 latency 1.850850 osd_op(client.4564.0:36004 rbd_data.11746b8b4567.00000000000018a8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.c629f55d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] -9331> 2014-06-09 16:22:12.165289 7f868f761700 5 -- op tracker -- , seq: 262, time: 2014-06-09 16:22:12.165283, event: reached_pg, op: osd_op(client.4564.0:36004 rbd_data.11746b8b4567.00000000000018a8 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~45056] 3.c629f55d RETRY=3 ack+ondisk+retry+write e242) -9330> 2014-06-09 16:22:12.165298 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -9329> 2014-06-09 16:22:12.165312 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x16bd6800 -9328> 2014-06-09 16:22:12.165322 7f868f761700 10 osd.26 243 dequeue_op 0x16bd6800 finish -9327> 2014-06-09 16:22:12.165325 7f868f761700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -9326> 2014-06-09 16:22:12.165348 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153599f0 (20 active) -9325> 2014-06-09 16:22:12.165353 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.4fb( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=12-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.4fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -9324> 2014-06-09 16:22:12.165388 7f868f761700 10 log is not dirty -9323> 2014-06-09 16:22:12.165394 7f868f761700 5 queue_transactions existing osr(1.4fb 0x384d7f0)/0x384d7f0 -9322> 2014-06-09 16:22:12.165396 7f868f761700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9321> 2014-06-09 16:22:12.165401 7f868df5e700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9320> 2014-06-09 16:22:12.165405 7f8691f66700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9319> 2014-06-09 16:22:12.165405 7f86a5e11700 5 _do_op 0xc88a1e0 seq 4112 osr(3.5f8 0x3ed1b30)/0x3ed1b30 start -9318> 2014-06-09 16:22:12.165411 7f868c75b700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9317> 2014-06-09 16:22:12.165409 7f86a5e11700 10 _do_transaction on 0x390d140 -9316> 2014-06-09 16:22:12.165421 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9315> 2014-06-09 16:22:12.165444 7f868cf5c700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9314> 2014-06-09 16:22:12.165451 7f868b759700 2 waiting 51 > 50 ops || 4238258 > 104857600 -9313> 2014-06-09 16:22:12.165462 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9312> 2014-06-09 16:22:12.165476 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4112.0.0 <= header.spos 150443.0.0 -9311> 2014-06-09 16:22:12.165481 7f86a5e11700 5 submit_transaction r = 0 -9310> 2014-06-09 16:22:12.165492 7f86a5e11700 10 _do_op 0xc88a1e0 seq 4112 r = 0, finisher 0xbd76640 0 -9309> 2014-06-09 16:22:12.165502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf1c60] -9308> 2014-06-09 16:22:12.165505 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf1c60] -9307> 2014-06-09 16:22:12.165507 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9306> 2014-06-09 16:22:12.165508 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9305> 2014-06-09 16:22:12.165511 7f86a5e11700 10 _finish_op 0xc88a1e0 seq 4112 osr(3.5f8 0x3ed1b30)/0x3ed1b30 -9304> 2014-06-09 16:22:12.165522 7f868af58700 10 op_submit_start 4161 -9303> 2014-06-09 16:22:12.165524 7f868af58700 5 queue_transactions (trailing journal) 4161 -9302> 2014-06-09 16:22:12.165526 7f868af58700 5 queue_op 0x230b9c80 seq 4161 osr(0.718 0x384c090) 861 bytes (queue has 50 ops and 4237465 bytes) -9301> 2014-06-09 16:22:12.165531 7f8692767700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9300> 2014-06-09 16:22:12.165536 7f8690763700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9299> 2014-06-09 16:22:12.165540 7f868d75d700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9298> 2014-06-09 16:22:12.165546 7f8689755700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9297> 2014-06-09 16:22:12.165551 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9296> 2014-06-09 16:22:12.165552 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76640] -9295> 2014-06-09 16:22:12.165556 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76640] -9294> 2014-06-09 16:22:12.165556 7f8692f68700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9293> 2014-06-09 16:22:12.165557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9292> 2014-06-09 16:22:12.165558 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9291> 2014-06-09 16:22:12.165561 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9290> 2014-06-09 16:22:12.165561 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f9e0 (1 active) -9289> 2014-06-09 16:22:12.165565 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcadbe60 (2 active) -9288> 2014-06-09 16:22:12.165566 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9287> 2014-06-09 16:22:12.165569 7f868af58700 10 op_submit_finish 4161 -9286> 2014-06-09 16:22:12.165570 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9285> 2014-06-09 16:22:12.165575 7f8691765700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9284> 2014-06-09 16:22:12.165579 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9283> 2014-06-09 16:22:12.165573 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9282> 2014-06-09 16:22:12.165584 7f8690f64700 2 waiting 51 > 50 ops || 4238323 > 104857600 -9281> 2014-06-09 16:22:12.165587 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.065761 7 0.000331 -9280> 2014-06-09 16:22:12.165599 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -9279> 2014-06-09 16:22:12.165610 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9278> 2014-06-09 16:22:12.165621 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -9277> 2014-06-09 16:22:12.165633 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -9276> 2014-06-09 16:22:12.165641 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -9275> 2014-06-09 16:22:12.165649 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -9274> 2014-06-09 16:22:12.165661 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -9273> 2014-06-09 16:22:12.165672 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -9272> 2014-06-09 16:22:12.165682 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9271> 2014-06-09 16:22:12.165696 7f868af58700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 -9270> 2014-06-09 16:22:12.165699 7f868af58700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:2.67f) v4 -- ?+0 0x1c039880 -9269> 2014-06-09 16:22:12.165712 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.67f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 -9268> 2014-06-09 16:22:12.165744 7f868af58700 10 log is not dirty -9267> 2014-06-09 16:22:12.165748 7f868af58700 5 queue_transactions existing osr(2.67f 0x2ed1f50)/0x2ed1f50 -9266> 2014-06-09 16:22:12.165751 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9265> 2014-06-09 16:22:12.165756 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9264> 2014-06-09 16:22:12.165761 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9263> 2014-06-09 16:22:12.165766 7f86a5e11700 5 _do_op 0x2ea529c0 seq 4113 osr(2.5f1 0x2ed1a10)/0x2ed1a10 start -9262> 2014-06-09 16:22:12.165770 7f86a5e11700 10 _do_transaction on 0xc0e3800 -9261> 2014-06-09 16:22:12.165777 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9260> 2014-06-09 16:22:12.165801 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9259> 2014-06-09 16:22:12.165809 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9258> 2014-06-09 16:22:12.165827 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9257> 2014-06-09 16:22:12.165833 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9256> 2014-06-09 16:22:12.165841 7f868b759700 2 waiting 51 > 50 ops || 4238291 > 104857600 -9255> 2014-06-09 16:22:12.165848 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9254> 2014-06-09 16:22:12.165843 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4113.0.0 <= header.spos 150443.0.0 -9253> 2014-06-09 16:22:12.165855 7f86a5e11700 5 submit_transaction r = 0 -9252> 2014-06-09 16:22:12.165871 7f86a5e11700 10 _do_op 0x2ea529c0 seq 4113 r = 0, finisher 0x30629f20 0 -9251> 2014-06-09 16:22:12.165881 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050b0e0] -9250> 2014-06-09 16:22:12.165884 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9249> 2014-06-09 16:22:12.165894 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9248> 2014-06-09 16:22:12.165904 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.5f1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 -9247> 2014-06-09 16:22:12.165913 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050b0e0] -9246> 2014-06-09 16:22:12.165915 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9245> 2014-06-09 16:22:12.165916 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9244> 2014-06-09 16:22:12.165920 7f86a5e11700 10 _finish_op 0x2ea529c0 seq 4113 osr(2.5f1 0x2ed1a10)/0x2ed1a10 -9243> 2014-06-09 16:22:12.165931 7f8690763700 10 op_submit_start 4162 -9242> 2014-06-09 16:22:12.165933 7f8690763700 5 queue_transactions (trailing journal) 4162 -9241> 2014-06-09 16:22:12.165934 7f8690763700 5 queue_op 0x3280ac00 seq 4162 osr(3.35c 0x3ed10c0) 828 bytes (queue has 50 ops and 4237432 bytes) -9240> 2014-06-09 16:22:12.165940 7f868d75d700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9239> 2014-06-09 16:22:12.165946 7f8689755700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9238> 2014-06-09 16:22:12.165950 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629f20] -9237> 2014-06-09 16:22:12.165951 7f868ef60700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9236> 2014-06-09 16:22:12.165954 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629f20] -9235> 2014-06-09 16:22:12.165956 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9234> 2014-06-09 16:22:12.165956 7f8692f68700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9233> 2014-06-09 16:22:12.165957 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9232> 2014-06-09 16:22:12.165960 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcadbe60 (1 active) -9231> 2014-06-09 16:22:12.165961 7f868bf5a700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9230> 2014-06-09 16:22:12.165963 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338900 (2 active) -9229> 2014-06-09 16:22:12.165966 7f868ff62700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9228> 2014-06-09 16:22:12.165968 7f8690763700 10 op_submit_finish 4162 -9227> 2014-06-09 16:22:12.165971 7f868e75f700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9226> 2014-06-09 16:22:12.165975 7f8691765700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9225> 2014-06-09 16:22:12.165979 7f8689f56700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9224> 2014-06-09 16:22:12.165984 7f8690f64700 2 waiting 51 > 50 ops || 4238290 > 104857600 -9223> 2014-06-09 16:22:12.165985 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8fe40 (19 active) -9222> 2014-06-09 16:22:12.165992 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -9221> 2014-06-09 16:22:12.165997 7f8690763700 10 osd.26 243 dequeue_op 0x2e075000 prio 63 cost 237568 latency 1.231416 osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] -9220> 2014-06-09 16:22:12.166021 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.166016, event: reached_pg, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -9219> 2014-06-09 16:22:12.166029 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -9218> 2014-06-09 16:22:12.166042 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2e075000 -9217> 2014-06-09 16:22:12.166053 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write -9216> 2014-06-09 16:22:12.166078 7f8690763700 15 getattr 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 '_' -9215> 2014-06-09 16:22:12.166153 7f8690763700 10 getattr got.size() is 0 -9214> 2014-06-09 16:22:12.166157 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 but can_create -9213> 2014-06-09 16:22:12.166172 7f8690763700 15 getattr 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 'snapset' -9212> 2014-06-09 16:22:12.166167 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 3 ==== osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+1818624 (1237681501 0 0) 0xc13f680 con 0x30c3c2c0 -9211> 2014-06-09 16:22:12.166194 7f86a68f6700 5 -- op tracker -- , seq: 3663, time: 2014-06-09 16:22:12.162137, event: header_read, op: osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9210> 2014-06-09 16:22:12.166206 7f8690763700 10 getattr got.size() is 0 -9209> 2014-06-09 16:22:12.166210 7f8690763700 15 getattr 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/snapdir//3 'snapset' -9208> 2014-06-09 16:22:12.166208 7f86a68f6700 5 -- op tracker -- , seq: 3663, time: 2014-06-09 16:22:12.162138, event: throttled, op: osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9207> 2014-06-09 16:22:12.166220 7f86a68f6700 5 -- op tracker -- , seq: 3663, time: 2014-06-09 16:22:12.166162, event: all_read, op: osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9206> 2014-06-09 16:22:12.166229 7f86a68f6700 5 -- op tracker -- , seq: 3663, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) -9205> 2014-06-09 16:22:12.166237 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 -9204> 2014-06-09 16:22:12.166242 7f8690763700 10 getattr got.size() is 0 -9203> 2014-06-09 16:22:12.166252 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30acf340 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -9202> 2014-06-09 16:22:12.166257 7f86a68f6700 15 osd.26 243 enqueue_op 0x2c06b000 prio 63 cost 1818624 latency 0.004120 osd_op(client.4543.0:33162 rbd_data.115f6b8b4567.000000000000166e [set-alloc-hint object_size 4194304 write_size 4194304,write 2375680~1818624] 3.fbf5ef7c RETRY=2 ack+ondisk+retry+write e243) v4 -9201> 2014-06-09 16:22:12.166265 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -9200> 2014-06-09 16:22:12.166275 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -9199> 2014-06-09 16:22:12.166285 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30acf340 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 rwstate(none n=0 w=0) oi: 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc3960 snapset: 0=[]:[] -9198> 2014-06-09 16:22:12.166299 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 @head oi=6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) -9197> 2014-06-09 16:22:12.166319 7f8690763700 15 getattr 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/snapdir//3 '_' -9196> 2014-06-09 16:22:12.166353 7f8690763700 10 getattr got.size() is 0 -9195> 2014-06-09 16:22:12.166356 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/snapdir//3 and !can_create -9194> 2014-06-09 16:22:12.166374 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.166368, event: started, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -9193> 2014-06-09 16:22:12.166384 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30647000 -9192> 2014-06-09 16:22:12.166403 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.166397, event: started, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -9191> 2014-06-09 16:22:12.166410 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] ov 0'0 av 243'534 snapc 0=[] snapset 0=[]:[] -9190> 2014-06-09 16:22:12.166424 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] -9189> 2014-06-09 16:22:12.166435 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 -9188> 2014-06-09 16:22:12.166451 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~237568 -9187> 2014-06-09 16:22:12.166469 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229000 -9186> 2014-06-09 16:22:12.166481 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 snapset=0x14dc3998 snapc=0=[] -9185> 2014-06-09 16:22:12.166492 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag -9184> 2014-06-09 16:22:12.166503 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 done, snapset=0=[]:[]+head -9183> 2014-06-09 16:22:12.166510 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 0x30647000 op modify -9182> 2014-06-09 16:22:12.166522 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:30.043285 -9181> 2014-06-09 16:22:12.166544 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=106 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -9180> 2014-06-09 16:22:12.166565 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 -9179> 2014-06-09 16:22:12.166577 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 29 on osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 -9178> 2014-06-09 16:22:12.166594 7f8690763700 7 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 29 o 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -9177> 2014-06-09 16:22:12.166620 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.166614, event: waiting for subops from 6, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -9176> 2014-06-09 16:22:12.166637 7f8690763700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 -9175> 2014-06-09 16:22:12.166640 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- osd_sub_op(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] v 243'534 snapset=0=[]:[] snapc=0=[]) v10 -- ?+238475 0xe064c00 -9174> 2014-06-09 16:22:12.166658 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 214'533 (0'0,214'533] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'533], crt=0'0) [243'534 (0'0) modify 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 by client.4565.0:33578 2014-06-09 10:20:30.043285] -9173> 2014-06-09 16:22:12.166683 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'534 (0'0) modify 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 by client.4565.0:33578 2014-06-09 10:20:30.043285 -9172> 2014-06-09 16:22:12.166698 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys -9171> 2014-06-09 16:22:12.166729 7f8690763700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'534, trimmed: -9170> 2014-06-09 16:22:12.166750 7f8690763700 5 queue_transactions existing osr(3.7f 0x3ed0200)/0x3ed0200 -9169> 2014-06-09 16:22:12.166753 7f8690763700 2 waiting 51 > 50 ops || 4477388 > 104857600 -9168> 2014-06-09 16:22:12.166758 7f868af58700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9167> 2014-06-09 16:22:12.166762 7f868f761700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9166> 2014-06-09 16:22:12.166767 7f868df5e700 2 waiting 51 > 50 ops || 4238260 > 104857600 -9165> 2014-06-09 16:22:12.166772 7f86a5e11700 5 _do_op 0xbf7f4a0 seq 4114 osr(1.3d6 0x384d470)/0x384d470 start -9164> 2014-06-09 16:22:12.166773 7f8691f66700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9163> 2014-06-09 16:22:12.166776 7f86a5e11700 10 _do_transaction on 0x3472840 -9162> 2014-06-09 16:22:12.166779 7f868c75b700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9161> 2014-06-09 16:22:12.166789 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9160> 2014-06-09 16:22:12.166793 7f868cf5c700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9159> 2014-06-09 16:22:12.166822 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9158> 2014-06-09 16:22:12.166817 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 4 ==== osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+516096 (420636209 0 0) 0x32da8480 con 0x30c3c2c0 -9157> 2014-06-09 16:22:12.166833 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4114.0.0 <= header.spos 150443.0.0 -9156> 2014-06-09 16:22:12.166835 7f86a5e11700 5 submit_transaction r = 0 -9155> 2014-06-09 16:22:12.166838 7f86a68f6700 5 -- op tracker -- , seq: 3664, time: 2014-06-09 16:22:12.166279, event: header_read, op: osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -9154> 2014-06-09 16:22:12.166852 7f86a5e11700 10 _do_op 0xbf7f4a0 seq 4114 r = 0, finisher 0xcd0efe0 0 -9153> 2014-06-09 16:22:12.166850 7f86a68f6700 5 -- op tracker -- , seq: 3664, time: 2014-06-09 16:22:12.166280, event: throttled, op: osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -9152> 2014-06-09 16:22:12.166857 7f86a5e11700 10 _finish_op 0xbf7f4a0 seq 4114 osr(1.3d6 0x384d470)/0x384d470 -9151> 2014-06-09 16:22:12.166859 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9ec40] -9150> 2014-06-09 16:22:12.166862 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9ec40] -9149> 2014-06-09 16:22:12.166864 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9148> 2014-06-09 16:22:12.166865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9147> 2014-06-09 16:22:12.166860 7f86a68f6700 5 -- op tracker -- , seq: 3664, time: 2014-06-09 16:22:12.166813, event: all_read, op: osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -9146> 2014-06-09 16:22:12.166868 7f868a757700 10 op_submit_start 4163 -9145> 2014-06-09 16:22:12.166869 7f8692767700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9144> 2014-06-09 16:22:12.166870 7f868a757700 5 queue_transactions (trailing journal) 4163 -9143> 2014-06-09 16:22:12.166871 7f86a68f6700 5 -- op tracker -- , seq: 3664, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -9142> 2014-06-09 16:22:12.166871 7f868a757700 5 queue_op 0xc64a360 seq 4163 osr(1.2e0 0x384d130) 861 bytes (queue has 50 ops and 4237465 bytes) -9141> 2014-06-09 16:22:12.166873 7f8689755700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9140> 2014-06-09 16:22:12.166878 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9139> 2014-06-09 16:22:12.166877 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 -9138> 2014-06-09 16:22:12.166882 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd0efe0] -9137> 2014-06-09 16:22:12.166892 7f868b759700 2 waiting 51 > 50 ops || 4238291 > 104857600 -9136> 2014-06-09 16:22:12.166893 7f86a68f6700 15 osd.26 243 enqueue_op 0x2976d000 prio 63 cost 516096 latency 0.000613 osd_op(client.4543.0:33290 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~516096] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 -9135> 2014-06-09 16:22:12.166897 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd0efe0] -9134> 2014-06-09 16:22:12.166900 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9133> 2014-06-09 16:22:12.166901 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9132> 2014-06-09 16:22:12.166903 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9131> 2014-06-09 16:22:12.166905 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338900 (1 active) -9130> 2014-06-09 16:22:12.166908 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9129> 2014-06-09 16:22:12.166908 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc360 (2 active) -9128> 2014-06-09 16:22:12.166912 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9127> 2014-06-09 16:22:12.166915 7f868a757700 10 op_submit_finish 4163 -9126> 2014-06-09 16:22:12.166916 7f8691765700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9125> 2014-06-09 16:22:12.166921 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9124> 2014-06-09 16:22:12.166919 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -9123> 2014-06-09 16:22:12.166926 7f8690f64700 2 waiting 51 > 50 ops || 4238323 > 104857600 -9122> 2014-06-09 16:22:12.166930 7f8690763700 2 waiting 51 > 50 ops || 4477421 > 104857600 -9121> 2014-06-09 16:22:12.166930 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local -9120> 2014-06-09 16:22:12.166934 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9119> 2014-06-09 16:22:12.166935 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -9118> 2014-06-09 16:22:12.166939 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9117> 2014-06-09 16:22:12.166943 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9116> 2014-06-09 16:22:12.166940 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9115> 2014-06-09 16:22:12.166948 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9114> 2014-06-09 16:22:12.166946 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -9113> 2014-06-09 16:22:12.166955 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9112> 2014-06-09 16:22:12.166953 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -9111> 2014-06-09 16:22:12.166959 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 -9110> 2014-06-09 16:22:12.166964 7f868d75d700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9109> 2014-06-09 16:22:12.166970 7f8692f68700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9108> 2014-06-09 16:22:12.166968 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 -9107> 2014-06-09 16:22:12.166975 7f868a757700 20 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -9106> 2014-06-09 16:22:12.166980 7f868a757700 20 osd.26 pg_epoch: 243 pg[1.46e( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop -9105> 2014-06-09 16:22:12.166985 7f868a757700 10 log is not dirty -9104> 2014-06-09 16:22:12.166990 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.761( v 213'670 (0'0,213'670] local-les=243 n=131 ec=118 les/c 243/243 242/242/222) -9103> 2014-06-09 16:22:12.167013 7f868a757700 10 log is not dirty -9102> 2014-06-09 16:22:12.167016 7f868a757700 5 queue_transactions existing osr(3.761 0x3ed1e10)/0x3ed1e10 -9101> 2014-06-09 16:22:12.167017 7f868a757700 2 waiting 51 > 50 ops || 4238293 > 104857600 -9100> 2014-06-09 16:22:12.167021 7f86a5e11700 5 _do_op 0x2fc8c060 seq 4115 osr(0.29d 0x2bd6c40)/0x2bd6c40 start -9099> 2014-06-09 16:22:12.167023 7f86a5e11700 10 _do_transaction on 0xc14a6c0 -9098> 2014-06-09 16:22:12.167028 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9097> 2014-06-09 16:22:12.167057 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9096> 2014-06-09 16:22:12.167073 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4115.0.0 <= header.spos 150443.0.0 -9095> 2014-06-09 16:22:12.167078 7f86a5e11700 5 submit_transaction r = 0 -9094> 2014-06-09 16:22:12.167088 7f86a5e11700 10 _do_op 0x2fc8c060 seq 4115 r = 0, finisher 0x3281ba00 0 -9093> 2014-06-09 16:22:12.167093 7f86a5e11700 10 _finish_op 0x2fc8c060 seq 4115 osr(0.29d 0x2bd6c40)/0x2bd6c40 -9092> 2014-06-09 16:22:12.167094 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfce5a0] -9091> 2014-06-09 16:22:12.167104 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281ba00] -9090> 2014-06-09 16:22:12.167099 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9089> 2014-06-09 16:22:12.167109 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281ba00] -9088> 2014-06-09 16:22:12.167111 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9087> 2014-06-09 16:22:12.167112 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9086> 2014-06-09 16:22:12.167110 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9085> 2014-06-09 16:22:12.167115 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc360 (1 active) -9084> 2014-06-09 16:22:12.167117 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca49320 (2 active) -9083> 2014-06-09 16:22:12.167119 7f86a5e11700 5 _do_op 0x14dd6840 seq 4116 osr(1.109 0x384c950)/0x384c950 start -9082> 2014-06-09 16:22:12.167122 7f86a5e11700 10 _do_transaction on 0xc2d8a80 -9081> 2014-06-09 16:22:12.167119 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.29d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -9080> 2014-06-09 16:22:12.167127 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfce5a0] -9079> 2014-06-09 16:22:12.167127 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9078> 2014-06-09 16:22:12.167129 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9077> 2014-06-09 16:22:12.167130 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9076> 2014-06-09 16:22:12.167135 7f8692767700 10 op_submit_start 4164 -9075> 2014-06-09 16:22:12.167137 7f8692767700 5 queue_transactions (trailing journal) 4164 -9074> 2014-06-09 16:22:12.167139 7f8692767700 5 queue_op 0xd471020 seq 4164 osr(3.1d5 0x3ed09f0) 828 bytes (queue has 50 ops and 4237432 bytes) -9073> 2014-06-09 16:22:12.167142 7f8692767700 10 op_submit_finish 4164 -9072> 2014-06-09 16:22:12.167146 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9071> 2014-06-09 16:22:12.167166 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9070> 2014-06-09 16:22:12.167160 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.123272 8 0.007437 -9069> 2014-06-09 16:22:12.167172 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -9068> 2014-06-09 16:22:12.167182 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4116.0.0 <= header.spos 150443.0.0 -9067> 2014-06-09 16:22:12.167185 7f86a5e11700 5 submit_transaction r = 0 -9066> 2014-06-09 16:22:12.167182 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9065> 2014-06-09 16:22:12.167196 7f86a5e11700 10 _do_op 0x14dd6840 seq 4116 r = 0, finisher 0x3042b2e0 0 -9064> 2014-06-09 16:22:12.167192 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -9063> 2014-06-09 16:22:12.167206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189abb00] -9062> 2014-06-09 16:22:12.167202 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -9061> 2014-06-09 16:22:12.167210 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -9060> 2014-06-09 16:22:12.167209 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -9059> 2014-06-09 16:22:12.167217 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -9058> 2014-06-09 16:22:12.167217 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -9057> 2014-06-09 16:22:12.167225 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.109( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -9056> 2014-06-09 16:22:12.167233 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189abb00] -9055> 2014-06-09 16:22:12.167235 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9054> 2014-06-09 16:22:12.167236 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9053> 2014-06-09 16:22:12.167229 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -9052> 2014-06-09 16:22:12.167239 7f86a5e11700 10 _finish_op 0x14dd6840 seq 4116 osr(1.109 0x384c950)/0x384c950 -9051> 2014-06-09 16:22:12.167239 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -9050> 2014-06-09 16:22:12.167247 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b2e0] -9049> 2014-06-09 16:22:12.167251 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b2e0] -9048> 2014-06-09 16:22:12.167252 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9047> 2014-06-09 16:22:12.167253 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9046> 2014-06-09 16:22:12.167249 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9045> 2014-06-09 16:22:12.167256 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49320 (1 active) -9044> 2014-06-09 16:22:12.167258 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc923b00 (2 active) -9043> 2014-06-09 16:22:12.167260 7f86a5e11700 5 _do_op 0x30a33e0 seq 4117 osr(3.468 0x3ed1590)/0x3ed1590 start -9042> 2014-06-09 16:22:12.167262 7f86a5e11700 10 _do_transaction on 0x3562300 -9041> 2014-06-09 16:22:12.167262 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 -9040> 2014-06-09 16:22:12.167267 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9039> 2014-06-09 16:22:12.167264 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:2.25c) v4 -- ?+0 0x31776c40 -9038> 2014-06-09 16:22:12.167279 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 -9037> 2014-06-09 16:22:12.167311 7f8692767700 10 log is not dirty -9036> 2014-06-09 16:22:12.167315 7f8692767700 5 queue_transactions existing osr(2.25c 0x2ed0a30)/0x2ed0a30 -9035> 2014-06-09 16:22:12.167318 7f8692767700 10 op_submit_start 4165 -9034> 2014-06-09 16:22:12.167319 7f8692767700 5 queue_transactions (trailing journal) 4165 -9033> 2014-06-09 16:22:12.167320 7f8692767700 5 queue_op 0x24105ce0 seq 4165 osr(2.25c 0x2ed0a30) 861 bytes (queue has 50 ops and 4237432 bytes) -9032> 2014-06-09 16:22:12.167323 7f8692767700 10 op_submit_finish 4165 -9031> 2014-06-09 16:22:12.167326 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -9030> 2014-06-09 16:22:12.167342 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -9029> 2014-06-09 16:22:12.167339 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.998569 7 0.000592 -9028> 2014-06-09 16:22:12.167350 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -9027> 2014-06-09 16:22:12.167357 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4117.0.0 <= header.spos 150443.0.0 -9026> 2014-06-09 16:22:12.167360 7f86a5e11700 5 submit_transaction r = 0 -9025> 2014-06-09 16:22:12.167359 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -9024> 2014-06-09 16:22:12.167372 7f86a5e11700 10 _do_op 0x30a33e0 seq 4117 r = 0, finisher 0x2d739020 0 -9023> 2014-06-09 16:22:12.167368 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 -9022> 2014-06-09 16:22:12.167381 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc9ae0] -9021> 2014-06-09 16:22:12.167384 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc9ae0] -9020> 2014-06-09 16:22:12.167378 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -9019> 2014-06-09 16:22:12.167385 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -9018> 2014-06-09 16:22:12.167386 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -9017> 2014-06-09 16:22:12.167389 7f86a5e11700 10 _finish_op 0x30a33e0 seq 4117 osr(3.468 0x3ed1590)/0x3ed1590 -9016> 2014-06-09 16:22:12.167386 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -9015> 2014-06-09 16:22:12.167397 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d739020] -9014> 2014-06-09 16:22:12.167393 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -9013> 2014-06-09 16:22:12.167400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d739020] -9012> 2014-06-09 16:22:12.167401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -9011> 2014-06-09 16:22:12.167402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -9010> 2014-06-09 16:22:12.167405 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc923b00 (1 active) -9009> 2014-06-09 16:22:12.167407 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652a20 (2 active) -9008> 2014-06-09 16:22:12.167409 7f86a5e11700 5 _do_op 0x3252b500 seq 4118 osr(0.5c3 0x2bd7a50)/0x2bd7a50 start -9007> 2014-06-09 16:22:12.167411 7f86a5e11700 10 _do_transaction on 0xbf4e9c0 -9006> 2014-06-09 16:22:12.167404 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) -9005> 2014-06-09 16:22:12.167416 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -9004> 2014-06-09 16:22:12.167414 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) -9003> 2014-06-09 16:22:12.167423 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -9002> 2014-06-09 16:22:12.167434 7f8692767700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 243 -9001> 2014-06-09 16:22:12.167437 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.7 172.16.96.201:6851/52636 -- pg_info(1 pgs e243:2.3fb) v4 -- ?+0 0x31776a80 -9000> 2014-06-09 16:22:12.167450 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 -8999> 2014-06-09 16:22:12.167481 7f8692767700 10 log is not dirty -8998> 2014-06-09 16:22:12.167482 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8997> 2014-06-09 16:22:12.167485 7f8692767700 5 queue_transactions existing osr(2.3fb 0x2ed12b0)/0x2ed12b0 -8996> 2014-06-09 16:22:12.167488 7f8692767700 10 op_submit_start 4166 -8995> 2014-06-09 16:22:12.167489 7f8692767700 5 queue_transactions (trailing journal) 4166 -8994> 2014-06-09 16:22:12.167490 7f8692767700 5 queue_op 0x308ca000 seq 4166 osr(2.3fb 0x2ed12b0) 861 bytes (queue has 50 ops and 4237465 bytes) -8993> 2014-06-09 16:22:12.167492 7f8692767700 10 op_submit_finish 4166 -8992> 2014-06-09 16:22:12.167496 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4118.0.0 <= header.spos 150443.0.0 -8991> 2014-06-09 16:22:12.167499 7f86a5e11700 5 submit_transaction r = 0 -8990> 2014-06-09 16:22:12.167495 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8989> 2014-06-09 16:22:12.167511 7f86a5e11700 10 _do_op 0x3252b500 seq 4118 r = 0, finisher 0x3251c260 0 -8988> 2014-06-09 16:22:12.167520 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f9d00] -8987> 2014-06-09 16:22:12.167525 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f9d00] -8986> 2014-06-09 16:22:12.167526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8985> 2014-06-09 16:22:12.167508 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] exit Started/Primary/Active/Activating 1.099836 7 0.000588 -8984> 2014-06-09 16:22:12.167528 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8983> 2014-06-09 16:22:12.167530 7f86a5e11700 10 _finish_op 0x3252b500 seq 4118 osr(0.5c3 0x2bd7a50)/0x2bd7a50 -8982> 2014-06-09 16:22:12.167530 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] enter Started/Primary/Active/Recovered -8981> 2014-06-09 16:22:12.167538 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251c260] -8980> 2014-06-09 16:22:12.167541 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251c260] -8979> 2014-06-09 16:22:12.167543 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8978> 2014-06-09 16:22:12.167544 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8977> 2014-06-09 16:22:12.167546 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652a20 (1 active) -8976> 2014-06-09 16:22:12.167541 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] needs_recovery is recovered -8975> 2014-06-09 16:22:12.167549 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358b40 (2 active) -8974> 2014-06-09 16:22:12.167550 7f86a5e11700 5 _do_op 0x39d9920 seq 4119 osr(2.fd 0x2ed0770)/0x2ed0770 start -8973> 2014-06-09 16:22:12.167553 7f86a5e11700 10 _do_transaction on 0xc316540 -8972> 2014-06-09 16:22:12.167558 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8971> 2014-06-09 16:22:12.167551 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -8970> 2014-06-09 16:22:12.167562 7f8692767700 5 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] enter Started/Primary/Active/Clean -8969> 2014-06-09 16:22:12.167571 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] finish_recovery -8968> 2014-06-09 16:22:12.167579 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active] clear_recovery_state -8967> 2014-06-09 16:22:12.167595 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8966> 2014-06-09 16:22:12.167590 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -8965> 2014-06-09 16:22:12.167608 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4119.0.0 <= header.spos 150443.0.0 -8964> 2014-06-09 16:22:12.167601 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -8963> 2014-06-09 16:22:12.167612 7f86a5e11700 5 submit_transaction r = 0 -8962> 2014-06-09 16:22:12.167623 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] share_pg_info -8961> 2014-06-09 16:22:12.167637 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 243 -8960> 2014-06-09 16:22:12.167639 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:3.647) v4 -- ?+0 0x317768c0 -8959> 2014-06-09 16:22:12.167651 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] publish_stats_to_osd 243:978 -8958> 2014-06-09 16:22:12.167658 7f86a5e11700 10 _do_op 0x39d9920 seq 4119 r = 0, finisher 0x31bffa40 0 -8957> 2014-06-09 16:22:12.167668 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31bffb60] -8956> 2014-06-09 16:22:12.167670 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31bffb60] -8955> 2014-06-09 16:22:12.167671 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8954> 2014-06-09 16:22:12.167672 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8953> 2014-06-09 16:22:12.167675 7f86a5e11700 10 _finish_op 0x39d9920 seq 4119 osr(2.fd 0x2ed0770)/0x2ed0770 -8952> 2014-06-09 16:22:12.167684 7f8692767700 10 log is not dirty -8951> 2014-06-09 16:22:12.167683 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31bffa40] -8950> 2014-06-09 16:22:12.167688 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31bffa40] -8949> 2014-06-09 16:22:12.167688 7f8692767700 5 queue_transactions existing osr(3.647 0x3ed1a70)/0x3ed1a70 -8948> 2014-06-09 16:22:12.167690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8947> 2014-06-09 16:22:12.167690 7f8692767700 10 op_submit_start 4167 -8946> 2014-06-09 16:22:12.167691 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8945> 2014-06-09 16:22:12.167691 7f8692767700 5 queue_transactions (trailing journal) 4167 -8944> 2014-06-09 16:22:12.167693 7f8692767700 5 queue_op 0xc705380 seq 4167 osr(3.647 0x3ed1a70) 861 bytes (queue has 49 ops and 4236672 bytes) -8943> 2014-06-09 16:22:12.167693 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358b40 (1 active) -8942> 2014-06-09 16:22:12.167696 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5560 (2 active) -8941> 2014-06-09 16:22:12.167701 7f8692767700 10 op_submit_finish 4167 -8940> 2014-06-09 16:22:12.167702 7f868ef60700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8939> 2014-06-09 16:22:12.167708 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8938> 2014-06-09 16:22:12.167705 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8937> 2014-06-09 16:22:12.167717 7f868bf5a700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8936> 2014-06-09 16:22:12.167722 7f868ff62700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8935> 2014-06-09 16:22:12.167718 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local -8934> 2014-06-09 16:22:12.167726 7f868e75f700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8933> 2014-06-09 16:22:12.167732 7f8691765700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8932> 2014-06-09 16:22:12.167727 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -8931> 2014-06-09 16:22:12.167737 7f8689f56700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8930> 2014-06-09 16:22:12.167742 7f8690f64700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8929> 2014-06-09 16:22:12.167735 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8928> 2014-06-09 16:22:12.167747 7f8690763700 2 waiting 51 > 50 ops || 4477456 > 104857600 -8927> 2014-06-09 16:22:12.167744 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -8926> 2014-06-09 16:22:12.167752 7f868af58700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8925> 2014-06-09 16:22:12.167757 7f868f761700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8924> 2014-06-09 16:22:12.167752 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8923> 2014-06-09 16:22:12.167762 7f868df5e700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8922> 2014-06-09 16:22:12.167767 7f8691f66700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8921> 2014-06-09 16:22:12.167772 7f868c75b700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8920> 2014-06-09 16:22:12.167767 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 -8919> 2014-06-09 16:22:12.167778 7f868cf5c700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8918> 2014-06-09 16:22:12.167777 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -8917> 2014-06-09 16:22:12.167783 7f868d75d700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8916> 2014-06-09 16:22:12.167790 7f8692f68700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8915> 2014-06-09 16:22:12.167786 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.1b1( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop -8914> 2014-06-09 16:22:12.167795 7f868a757700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8913> 2014-06-09 16:22:12.167796 7f8692767700 10 log is not dirty -8912> 2014-06-09 16:22:12.167801 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8911> 2014-06-09 16:22:12.167814 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -8910> 2014-06-09 16:22:12.167822 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -8909> 2014-06-09 16:22:12.167831 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8908> 2014-06-09 16:22:12.167839 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -8907> 2014-06-09 16:22:12.167847 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8906> 2014-06-09 16:22:12.167859 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 -8905> 2014-06-09 16:22:12.167869 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -8904> 2014-06-09 16:22:12.167877 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.2c2( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -8903> 2014-06-09 16:22:12.167886 7f8692767700 10 log is not dirty -8902> 2014-06-09 16:22:12.167891 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8901> 2014-06-09 16:22:12.167903 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local -8900> 2014-06-09 16:22:12.167911 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -8899> 2014-06-09 16:22:12.167920 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8898> 2014-06-09 16:22:12.167928 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -8897> 2014-06-09 16:22:12.167936 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8896> 2014-06-09 16:22:12.167949 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 -8895> 2014-06-09 16:22:12.167958 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -8894> 2014-06-09 16:22:12.167966 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.4ba( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop -8893> 2014-06-09 16:22:12.167975 7f8692767700 10 log is not dirty -8892> 2014-06-09 16:22:12.167980 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8891> 2014-06-09 16:22:12.167992 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -8890> 2014-06-09 16:22:12.168000 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -8889> 2014-06-09 16:22:12.168009 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8888> 2014-06-09 16:22:12.168016 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -8887> 2014-06-09 16:22:12.168024 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8886> 2014-06-09 16:22:12.168037 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 -8885> 2014-06-09 16:22:12.168046 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -8884> 2014-06-09 16:22:12.168054 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.25c( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -8883> 2014-06-09 16:22:12.168063 7f8692767700 10 log is not dirty -8882> 2014-06-09 16:22:12.168067 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8881> 2014-06-09 16:22:12.168082 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -8880> 2014-06-09 16:22:12.168091 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1c54ac00,0x1c54ab00,0x1c54aa00,0x1c54a900,0x1c54a800,0x22009900,0x22009800,0x22009700,0x22009600,0x22009500,0x22009400 -8879> 2014-06-09 16:22:12.168129 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -8878> 2014-06-09 16:22:12.168140 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -8877> 2014-06-09 16:22:12.168149 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8876> 2014-06-09 16:22:12.168163 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:949 -8875> 2014-06-09 16:22:12.168173 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -8874> 2014-06-09 16:22:12.168183 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.6cd( v 214'577 (0'0,214'577] local-les=243 n=111 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -8873> 2014-06-09 16:22:12.168193 7f8692767700 10 log is not dirty -8872> 2014-06-09 16:22:12.168198 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8871> 2014-06-09 16:22:12.168210 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.091800 7 0.000359 -8870> 2014-06-09 16:22:12.168221 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -8869> 2014-06-09 16:22:12.168231 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8868> 2014-06-09 16:22:12.168242 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -8867> 2014-06-09 16:22:12.168252 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -8866> 2014-06-09 16:22:12.168261 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -8865> 2014-06-09 16:22:12.168269 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -8864> 2014-06-09 16:22:12.168279 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -8863> 2014-06-09 16:22:12.168290 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -8862> 2014-06-09 16:22:12.168300 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -8861> 2014-06-09 16:22:12.168312 7f8692767700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 -8860> 2014-06-09 16:22:12.168315 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:0.b7) v4 -- ?+0 0x31776700 -8859> 2014-06-09 16:22:12.168328 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 -8858> 2014-06-09 16:22:12.168360 7f8692767700 10 log is not dirty -8857> 2014-06-09 16:22:12.168364 7f8692767700 5 queue_transactions existing osr(0.b7 0x2bd6440)/0x2bd6440 -8856> 2014-06-09 16:22:12.168367 7f8692767700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8855> 2014-06-09 16:22:12.168392 7f8689755700 10 op_submit_start 4168 -8854> 2014-06-09 16:22:12.168395 7f8689755700 5 queue_transactions (trailing journal) 4168 -8853> 2014-06-09 16:22:12.168400 7f8689755700 5 queue_op 0x324b27e0 seq 4168 osr(0.1d4 0x2bd6940) 828 bytes (queue has 50 ops and 4237500 bytes) -8852> 2014-06-09 16:22:12.168403 7f8689755700 10 op_submit_finish 4168 -8851> 2014-06-09 16:22:12.168408 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) -8850> 2014-06-09 16:22:12.168445 7f8689755700 10 log is not dirty -8849> 2014-06-09 16:22:12.168451 7f8689755700 5 queue_transactions existing osr(1.fa 0x384c970)/0x384c970 -8848> 2014-06-09 16:22:12.168453 7f8689755700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8847> 2014-06-09 16:22:12.168458 7f86a5e11700 5 _do_op 0x159968a0 seq 4120 osr(3.4fe 0x3ed1730)/0x3ed1730 start -8846> 2014-06-09 16:22:12.168462 7f86a5e11700 10 _do_transaction on 0xbfe6540 -8845> 2014-06-09 16:22:12.168467 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8844> 2014-06-09 16:22:12.168510 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8843> 2014-06-09 16:22:12.168525 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4120.0.0 <= header.spos 150443.0.0 -8842> 2014-06-09 16:22:12.168529 7f86a5e11700 5 submit_transaction r = 0 -8841> 2014-06-09 16:22:12.168541 7f86a5e11700 10 _do_op 0x159968a0 seq 4120 r = 0, finisher 0xc6e8300 0 -8840> 2014-06-09 16:22:12.168552 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314786e0] -8839> 2014-06-09 16:22:12.168554 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314786e0] -8838> 2014-06-09 16:22:12.168556 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8837> 2014-06-09 16:22:12.168557 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8836> 2014-06-09 16:22:12.168560 7f86a5e11700 10 _finish_op 0x159968a0 seq 4120 osr(3.4fe 0x3ed1730)/0x3ed1730 -8835> 2014-06-09 16:22:12.168575 7f868b759700 10 op_submit_start 4169 -8834> 2014-06-09 16:22:12.168576 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8300] -8833> 2014-06-09 16:22:12.168578 7f868b759700 5 queue_transactions (trailing journal) 4169 -8832> 2014-06-09 16:22:12.168579 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8300] -8831> 2014-06-09 16:22:12.168581 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8830> 2014-06-09 16:22:12.168580 7f868b759700 5 queue_op 0x24104ea0 seq 4169 osr(1.c9 0x384c9f0) 826 bytes (queue has 50 ops and 4237498 bytes) -8829> 2014-06-09 16:22:12.168582 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8828> 2014-06-09 16:22:12.168585 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8827> 2014-06-09 16:22:12.168585 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5560 (1 active) -8826> 2014-06-09 16:22:12.168588 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115320 (2 active) -8825> 2014-06-09 16:22:12.168590 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8824> 2014-06-09 16:22:12.168592 7f86a5e11700 5 _do_op 0x16fd4de0 seq 4121 osr(1.67 0x384c5b0)/0x384c5b0 start -8823> 2014-06-09 16:22:12.168594 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8822> 2014-06-09 16:22:12.168595 7f86a5e11700 10 _do_transaction on 0xc140b40 -8821> 2014-06-09 16:22:12.168599 7f8691765700 2 waiting 51 > 50 ops || 4238359 > 104857600 -8820> 2014-06-09 16:22:12.168601 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8819> 2014-06-09 16:22:12.168603 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 -8818> 2014-06-09 16:22:12.168607 7f8690f64700 2 waiting 51 > 50 ops || 4238356 > 104857600 -8817> 2014-06-09 16:22:12.168612 7f868b759700 10 op_submit_finish 4169 -8816> 2014-06-09 16:22:12.168616 7f868b759700 10 osd.26 pg_epoch: 243 pg[3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.577( v 213'564 (0'0,213'564] local-les=243 n=113 ec=118 les/c 243/243 242/242/207) -8815> 2014-06-09 16:22:12.168640 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8814> 2014-06-09 16:22:12.168657 7f868b759700 10 log is not dirty -8813> 2014-06-09 16:22:12.168655 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4121.0.0 <= header.spos 150443.0.0 -8812> 2014-06-09 16:22:12.168660 7f86a5e11700 5 submit_transaction r = 0 -8811> 2014-06-09 16:22:12.168662 7f868b759700 5 queue_transactions existing osr(3.577 0x3ed18c0)/0x3ed18c0 -8810> 2014-06-09 16:22:12.168665 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8809> 2014-06-09 16:22:12.168670 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 -8808> 2014-06-09 16:22:12.168672 7f86a5e11700 10 _do_op 0x16fd4de0 seq 4121 r = 0, finisher 0x3196c540 0 -8807> 2014-06-09 16:22:12.168675 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 -8806> 2014-06-09 16:22:12.168679 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8805> 2014-06-09 16:22:12.168682 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b9c0] -8804> 2014-06-09 16:22:12.168684 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8803> 2014-06-09 16:22:12.168685 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b9c0] -8802> 2014-06-09 16:22:12.168687 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8801> 2014-06-09 16:22:12.168688 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8800> 2014-06-09 16:22:12.168688 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 -8799> 2014-06-09 16:22:12.168692 7f86a5e11700 10 _finish_op 0x16fd4de0 seq 4121 osr(1.67 0x384c5b0)/0x384c5b0 -8798> 2014-06-09 16:22:12.168692 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 -8797> 2014-06-09 16:22:12.168707 7f868bf5a700 10 op_submit_start 4170 -8796> 2014-06-09 16:22:12.168709 7f868bf5a700 5 queue_transactions (trailing journal) 4170 -8795> 2014-06-09 16:22:12.168710 7f868bf5a700 5 queue_op 0xc931d40 seq 4170 osr(3.2d3 0x3ed0eb0) 828 bytes (queue has 50 ops and 4237500 bytes) -8794> 2014-06-09 16:22:12.168712 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3196c540] -8793> 2014-06-09 16:22:12.168715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3196c540] -8792> 2014-06-09 16:22:12.168716 7f868d75d700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8791> 2014-06-09 16:22:12.168717 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8790> 2014-06-09 16:22:12.168718 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8789> 2014-06-09 16:22:12.168721 7f868a757700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8788> 2014-06-09 16:22:12.168721 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115320 (1 active) -8787> 2014-06-09 16:22:12.168726 7f8692767700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8786> 2014-06-09 16:22:12.168731 7f8689755700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8785> 2014-06-09 16:22:12.168736 7f868ef60700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8784> 2014-06-09 16:22:12.168724 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89bb00 (2 active) -8783> 2014-06-09 16:22:12.168741 7f868cf5c700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8782> 2014-06-09 16:22:12.168744 7f86a5e11700 5 _do_op 0x30d96000 seq 4122 osr(2.59f 0x2ed1830)/0x2ed1830 start -8781> 2014-06-09 16:22:12.168747 7f868ff62700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8780> 2014-06-09 16:22:12.168748 7f86a5e11700 10 _do_transaction on 0xc262900 -8779> 2014-06-09 16:22:12.168751 7f868e75f700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8778> 2014-06-09 16:22:12.168754 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8777> 2014-06-09 16:22:12.168756 7f8691765700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8776> 2014-06-09 16:22:12.168760 7f8689f56700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8775> 2014-06-09 16:22:12.168764 7f8690f64700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8774> 2014-06-09 16:22:12.168770 7f868bf5a700 10 op_submit_finish 4170 -8773> 2014-06-09 16:22:12.168773 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.14c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.14c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -8772> 2014-06-09 16:22:12.168794 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8771> 2014-06-09 16:22:12.168807 7f868bf5a700 10 log is not dirty -8770> 2014-06-09 16:22:12.168812 7f868bf5a700 5 queue_transactions existing osr(1.14c 0x384c810)/0x384c810 -8769> 2014-06-09 16:22:12.168811 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4122.0.0 <= header.spos 150443.0.0 -8768> 2014-06-09 16:22:12.168814 7f868bf5a700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8767> 2014-06-09 16:22:12.168816 7f86a5e11700 5 submit_transaction r = 0 -8766> 2014-06-09 16:22:12.168819 7f868b759700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8765> 2014-06-09 16:22:12.168823 7f8690763700 2 waiting 51 > 50 ops || 4477456 > 104857600 -8764> 2014-06-09 16:22:12.168828 7f868af58700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8763> 2014-06-09 16:22:12.168829 7f86a5e11700 10 _do_op 0x30d96000 seq 4122 r = 0, finisher 0xbfcefe0 0 -8762> 2014-06-09 16:22:12.168832 7f868f761700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8761> 2014-06-09 16:22:12.168839 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7d740] -8760> 2014-06-09 16:22:12.168838 7f868df5e700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8759> 2014-06-09 16:22:12.168846 7f8691f66700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8758> 2014-06-09 16:22:12.168842 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -8757> 2014-06-09 16:22:12.168852 7f868c75b700 2 waiting 51 > 50 ops || 4238361 > 104857600 -8756> 2014-06-09 16:22:12.168857 7f8692f68700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8755> 2014-06-09 16:22:12.168853 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -8754> 2014-06-09 16:22:12.168862 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.59f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -8753> 2014-06-09 16:22:12.168870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7d740] -8752> 2014-06-09 16:22:12.168871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8751> 2014-06-09 16:22:12.168873 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8750> 2014-06-09 16:22:12.168876 7f86a5e11700 10 _finish_op 0x30d96000 seq 4122 osr(2.59f 0x2ed1830)/0x2ed1830 -8749> 2014-06-09 16:22:12.168885 7f868d75d700 10 op_submit_start 4171 -8748> 2014-06-09 16:22:12.168887 7f868d75d700 5 queue_transactions (trailing journal) 4171 -8747> 2014-06-09 16:22:12.168888 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89bb00 (1 active) -8746> 2014-06-09 16:22:12.168889 7f868d75d700 5 queue_op 0x288f8ae0 seq 4171 osr(3.6ba 0x3ed1ce0) 828 bytes (queue has 50 ops and 4237467 bytes) -8745> 2014-06-09 16:22:12.168889 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbfcefe0] -8744> 2014-06-09 16:22:12.168891 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780000 (2 active) -8743> 2014-06-09 16:22:12.168894 7f868a757700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8742> 2014-06-09 16:22:12.168894 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbfcefe0] -8741> 2014-06-09 16:22:12.168896 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8740> 2014-06-09 16:22:12.168895 7f86a5e11700 5 _do_op 0x324b3320 seq 4123 osr(0.129 0x2bd66b0)/0x2bd66b0 start -8739> 2014-06-09 16:22:12.168897 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8738> 2014-06-09 16:22:12.168900 7f868d75d700 10 op_submit_finish 4171 -8737> 2014-06-09 16:22:12.168899 7f8692767700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8736> 2014-06-09 16:22:12.168904 7f8689755700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8735> 2014-06-09 16:22:12.168908 7f868ef60700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8734> 2014-06-09 16:22:12.168914 7f86a5e11700 10 _do_transaction on 0xbfe40c0 -8733> 2014-06-09 16:22:12.168903 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.2d6( v 213'618 (0'0,213'618] local-les=243 n=118 ec=118 les/c 243/243 242/242/207) -8732> 2014-06-09 16:22:12.168920 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8731> 2014-06-09 16:22:12.168940 7f868d75d700 10 log is not dirty -8730> 2014-06-09 16:22:12.168945 7f868d75d700 5 queue_transactions existing osr(3.2d6 0x3ed0e60)/0x3ed0e60 -8729> 2014-06-09 16:22:12.168947 7f868d75d700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8728> 2014-06-09 16:22:12.168952 7f868cf5c700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8727> 2014-06-09 16:22:12.168957 7f868ff62700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8726> 2014-06-09 16:22:12.168961 7f868e75f700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8725> 2014-06-09 16:22:12.168961 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8724> 2014-06-09 16:22:12.168966 7f8691765700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8723> 2014-06-09 16:22:12.168970 7f8689f56700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8722> 2014-06-09 16:22:12.168975 7f8690f64700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8721> 2014-06-09 16:22:12.168977 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4123.0.0 <= header.spos 150443.0.0 -8720> 2014-06-09 16:22:12.168979 7f868bf5a700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8719> 2014-06-09 16:22:12.168981 7f86a5e11700 5 submit_transaction r = 0 -8718> 2014-06-09 16:22:12.168984 7f868b759700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8717> 2014-06-09 16:22:12.168988 7f8690763700 2 waiting 51 > 50 ops || 4477423 > 104857600 -8716> 2014-06-09 16:22:12.168992 7f868af58700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8715> 2014-06-09 16:22:12.168998 7f86a5e11700 10 _do_op 0x324b3320 seq 4123 r = 0, finisher 0x324f9260 0 -8714> 2014-06-09 16:22:12.169009 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f9160] -8713> 2014-06-09 16:22:12.169012 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f9160] -8712> 2014-06-09 16:22:12.169013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8711> 2014-06-09 16:22:12.169014 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8710> 2014-06-09 16:22:12.169018 7f86a5e11700 10 _finish_op 0x324b3320 seq 4123 osr(0.129 0x2bd66b0)/0x2bd66b0 -8709> 2014-06-09 16:22:12.168996 7f868f761700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8708> 2014-06-09 16:22:12.169065 7f868df5e700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8707> 2014-06-09 16:22:12.169071 7f8691f66700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8706> 2014-06-09 16:22:12.169077 7f868c75b700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8705> 2014-06-09 16:22:12.169082 7f8692f68700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8704> 2014-06-09 16:22:12.169094 7f868a757700 10 op_submit_start 4172 -8703> 2014-06-09 16:22:12.169094 7f8692767700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8702> 2014-06-09 16:22:12.169096 7f868a757700 5 queue_transactions (trailing journal) 4172 -8701> 2014-06-09 16:22:12.169098 7f868a757700 5 queue_op 0xc704600 seq 4172 osr(3.761 0x3ed1e10) 828 bytes (queue has 50 ops and 4237467 bytes) -8700> 2014-06-09 16:22:12.169099 7f868ef60700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8699> 2014-06-09 16:22:12.169105 7f868d75d700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8698> 2014-06-09 16:22:12.169110 7f8689755700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8697> 2014-06-09 16:22:12.169114 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f9260] -8696> 2014-06-09 16:22:12.169115 7f868cf5c700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8695> 2014-06-09 16:22:12.169118 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f9260] -8694> 2014-06-09 16:22:12.169119 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8693> 2014-06-09 16:22:12.169120 7f868e75f700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8692> 2014-06-09 16:22:12.169120 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8691> 2014-06-09 16:22:12.169124 7f8691765700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8690> 2014-06-09 16:22:12.169123 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780000 (1 active) -8689> 2014-06-09 16:22:12.169126 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f440 (2 active) -8688> 2014-06-09 16:22:12.169128 7f8689f56700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8687> 2014-06-09 16:22:12.169130 7f86a5e11700 5 _do_op 0xd327d40 seq 4124 osr(1.508 0x384db90)/0x384db90 start -8686> 2014-06-09 16:22:12.169134 7f86a5e11700 10 _do_transaction on 0xc3f0fc0 -8685> 2014-06-09 16:22:12.169133 7f8690f64700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8684> 2014-06-09 16:22:12.169138 7f868a757700 10 op_submit_finish 4172 -8683> 2014-06-09 16:22:12.169139 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8682> 2014-06-09 16:22:12.169141 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.216( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.216( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) -8681> 2014-06-09 16:22:12.169176 7f868a757700 10 log is not dirty -8680> 2014-06-09 16:22:12.169179 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8679> 2014-06-09 16:22:12.169181 7f868a757700 5 queue_transactions existing osr(2.216 0x2ed0b50)/0x2ed0b50 -8678> 2014-06-09 16:22:12.169183 7f868a757700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8677> 2014-06-09 16:22:12.169188 7f868ff62700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8676> 2014-06-09 16:22:12.169193 7f868bf5a700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8675> 2014-06-09 16:22:12.169193 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4124.0.0 <= header.spos 150443.0.0 -8674> 2014-06-09 16:22:12.169197 7f868b759700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8673> 2014-06-09 16:22:12.169197 7f86a5e11700 5 submit_transaction r = 0 -8672> 2014-06-09 16:22:12.169201 7f868af58700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8671> 2014-06-09 16:22:12.169206 7f868f761700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8670> 2014-06-09 16:22:12.169209 7f86a5e11700 10 _do_op 0xd327d40 seq 4124 r = 0, finisher 0x32073a00 0 -8669> 2014-06-09 16:22:12.169211 7f868df5e700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8668> 2014-06-09 16:22:12.169216 7f8690763700 2 waiting 51 > 50 ops || 4477423 > 104857600 -8667> 2014-06-09 16:22:12.169220 7f868c75b700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8666> 2014-06-09 16:22:12.169220 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050bec0] -8665> 2014-06-09 16:22:12.169222 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050bec0] -8664> 2014-06-09 16:22:12.169224 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8663> 2014-06-09 16:22:12.169225 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8662> 2014-06-09 16:22:12.169224 7f8692f68700 2 waiting 51 > 50 ops || 4238295 > 104857600 -8661> 2014-06-09 16:22:12.169228 7f8691f66700 2 waiting 51 > 50 ops || 4238328 > 104857600 -8660> 2014-06-09 16:22:12.169229 7f86a5e11700 10 _finish_op 0xd327d40 seq 4124 osr(1.508 0x384db90)/0x384db90 -8659> 2014-06-09 16:22:12.169238 7f8692767700 10 op_submit_start 4173 -8658> 2014-06-09 16:22:12.169238 7f868ef60700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8657> 2014-06-09 16:22:12.169240 7f8692767700 5 queue_transactions (trailing journal) 4173 -8656> 2014-06-09 16:22:12.169241 7f8692767700 5 queue_op 0x2fcaf2c0 seq 4173 osr(0.b7 0x2bd6440) 858 bytes (queue has 50 ops and 4237497 bytes) -8655> 2014-06-09 16:22:12.169244 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32073a00] -8654> 2014-06-09 16:22:12.169244 7f868d75d700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8653> 2014-06-09 16:22:12.169249 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32073a00] -8652> 2014-06-09 16:22:12.169249 7f8689755700 2 waiting 51 > 50 ops || 4238323 > 104857600 -8651> 2014-06-09 16:22:12.169251 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8650> 2014-06-09 16:22:12.169252 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8649> 2014-06-09 16:22:12.169254 7f868cf5c700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8648> 2014-06-09 16:22:12.169255 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f440 (1 active) -8647> 2014-06-09 16:22:12.169258 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd560 (2 active) -8646> 2014-06-09 16:22:12.169258 7f868e75f700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8645> 2014-06-09 16:22:12.169263 7f8692767700 10 op_submit_finish 4173 -8644> 2014-06-09 16:22:12.169263 7f8691765700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8643> 2014-06-09 16:22:12.169267 7f8689f56700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8642> 2014-06-09 16:22:12.169271 7f8690f64700 2 waiting 51 > 50 ops || 4238355 > 104857600 -8641> 2014-06-09 16:22:12.169276 7f868a757700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8640> 2014-06-09 16:22:12.169280 7f868ff62700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8639> 2014-06-09 16:22:12.169267 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8638> 2014-06-09 16:22:12.169284 7f868bf5a700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8637> 2014-06-09 16:22:12.169288 7f868b759700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8636> 2014-06-09 16:22:12.169293 7f868af58700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8635> 2014-06-09 16:22:12.169297 7f868f761700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8634> 2014-06-09 16:22:12.169287 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] check_local -8633> 2014-06-09 16:22:12.169301 7f868df5e700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8632> 2014-06-09 16:22:12.169306 7f8690763700 2 waiting 51 > 50 ops || 4477453 > 104857600 -8631> 2014-06-09 16:22:12.169310 7f868c75b700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8630> 2014-06-09 16:22:12.169299 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] requeue_ops 0x16059800,0x16059700,0x16059600,0x16059500,0x1804c600,0x1804c500,0x1804c400 -8629> 2014-06-09 16:22:12.169315 7f8692f68700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8628> 2014-06-09 16:22:12.169319 7f8691f66700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8627> 2014-06-09 16:22:12.169331 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] needs_recovery primary has 2 missing -8626> 2014-06-09 16:22:12.169345 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] activate not all replicas are up-to-date, queueing recovery -8625> 2014-06-09 16:22:12.169365 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] publish_stats_to_osd 243:912 -8624> 2014-06-09 16:22:12.169378 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] hit_set_clear -8623> 2014-06-09 16:22:12.169390 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.94( v 214'571 lc 214'568 (0'0,214'571] local-les=243 n=112 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'571 lcod 0'0 mlcod 0'0 active m=2] agent_stop -8622> 2014-06-09 16:22:12.169403 7f8692767700 10 log is not dirty -8621> 2014-06-09 16:22:12.169408 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8620> 2014-06-09 16:22:12.169420 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.124442 7 0.000397 -8619> 2014-06-09 16:22:12.169432 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -8618> 2014-06-09 16:22:12.169442 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8617> 2014-06-09 16:22:12.169452 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -8616> 2014-06-09 16:22:12.169463 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -8615> 2014-06-09 16:22:12.169471 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -8614> 2014-06-09 16:22:12.169479 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -8613> 2014-06-09 16:22:12.169489 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -8612> 2014-06-09 16:22:12.169501 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -8611> 2014-06-09 16:22:12.169511 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -8610> 2014-06-09 16:22:12.169524 7f8692767700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 243 -8609> 2014-06-09 16:22:12.169527 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:0.319) v4 -- ?+0 0x31776540 -8608> 2014-06-09 16:22:12.169540 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 -8607> 2014-06-09 16:22:12.169571 7f8692767700 10 log is not dirty -8606> 2014-06-09 16:22:12.169575 7f8692767700 5 queue_transactions existing osr(0.319 0x2bd6d90)/0x2bd6d90 -8605> 2014-06-09 16:22:12.169577 7f8692767700 2 waiting 51 > 50 ops || 4238358 > 104857600 -8604> 2014-06-09 16:22:12.169582 7f86a5e11700 5 _do_op 0x300619e0 seq 4125 osr(0.34e 0x2bd7020)/0x2bd7020 start -8603> 2014-06-09 16:22:12.169586 7f86a5e11700 10 _do_transaction on 0xc0498c0 -8602> 2014-06-09 16:22:12.169592 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8601> 2014-06-09 16:22:12.169633 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8600> 2014-06-09 16:22:12.169650 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4125.0.0 <= header.spos 150443.0.0 -8599> 2014-06-09 16:22:12.169654 7f86a5e11700 5 submit_transaction r = 0 -8598> 2014-06-09 16:22:12.169668 7f86a5e11700 10 _do_op 0x300619e0 seq 4125 r = 0, finisher 0x33a78c0 0 -8597> 2014-06-09 16:22:12.169679 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x4eb98c0] -8596> 2014-06-09 16:22:12.169682 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -8595> 2014-06-09 16:22:12.169692 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -8594> 2014-06-09 16:22:12.169701 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.34e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 -8593> 2014-06-09 16:22:12.169711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x4eb98c0] -8592> 2014-06-09 16:22:12.169715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8591> 2014-06-09 16:22:12.169716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8590> 2014-06-09 16:22:12.169720 7f86a5e11700 10 _finish_op 0x300619e0 seq 4125 osr(0.34e 0x2bd7020)/0x2bd7020 -8589> 2014-06-09 16:22:12.169735 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a78c0] -8588> 2014-06-09 16:22:12.169741 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a78c0] -8587> 2014-06-09 16:22:12.169743 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8586> 2014-06-09 16:22:12.169744 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8585> 2014-06-09 16:22:12.169743 7f868d75d700 10 op_submit_start 4174 -8584> 2014-06-09 16:22:12.169746 7f868d75d700 5 queue_transactions (trailing journal) 4174 -8583> 2014-06-09 16:22:12.169747 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd560 (1 active) -8582> 2014-06-09 16:22:12.169747 7f868d75d700 5 queue_op 0x288f9ec0 seq 4174 osr(3.2d6 0x3ed0e60) 828 bytes (queue has 50 ops and 4237464 bytes) -8581> 2014-06-09 16:22:12.169750 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac900 (2 active) -8580> 2014-06-09 16:22:12.169753 7f8689755700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8579> 2014-06-09 16:22:12.169755 7f86a5e11700 5 _do_op 0xc2ebb60 seq 4126 osr(2.412 0x2ed15d0)/0x2ed15d0 start -8578> 2014-06-09 16:22:12.169757 7f86a5e11700 10 _do_transaction on 0xc04ea80 -8577> 2014-06-09 16:22:12.169758 7f868cf5c700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8576> 2014-06-09 16:22:12.169763 7f868e75f700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8575> 2014-06-09 16:22:12.169764 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8574> 2014-06-09 16:22:12.169767 7f8691765700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8573> 2014-06-09 16:22:12.169771 7f8689f56700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8572> 2014-06-09 16:22:12.169776 7f8690f64700 2 waiting 51 > 50 ops || 4238322 > 104857600 -8571> 2014-06-09 16:22:12.169780 7f868a757700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8570> 2014-06-09 16:22:12.169785 7f868d75d700 10 op_submit_finish 4174 -8569> 2014-06-09 16:22:12.169788 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.6cb( v 213'564 (0'0,213'564] local-les=243 n=112 ec=118 les/c 243/243 242/242/207) -8568> 2014-06-09 16:22:12.169805 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8567> 2014-06-09 16:22:12.169822 7f868d75d700 10 log is not dirty -8566> 2014-06-09 16:22:12.169821 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4126.0.0 <= header.spos 150443.0.0 -8565> 2014-06-09 16:22:12.169826 7f86a5e11700 5 submit_transaction r = 0 -8564> 2014-06-09 16:22:12.169827 7f868d75d700 5 queue_transactions existing osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 -8563> 2014-06-09 16:22:12.169830 7f868d75d700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8562> 2014-06-09 16:22:12.169834 7f868ff62700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8561> 2014-06-09 16:22:12.169839 7f868bf5a700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8560> 2014-06-09 16:22:12.169839 7f86a5e11700 10 _do_op 0xc2ebb60 seq 4126 r = 0, finisher 0xbd7bb80 0 -8559> 2014-06-09 16:22:12.169844 7f868b759700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8558> 2014-06-09 16:22:12.169848 7f868af58700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8557> 2014-06-09 16:22:12.169850 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d15a0] -8556> 2014-06-09 16:22:12.169852 7f868f761700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8555> 2014-06-09 16:22:12.169857 7f868df5e700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8554> 2014-06-09 16:22:12.169853 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -8553> 2014-06-09 16:22:12.169861 7f8690763700 2 waiting 51 > 50 ops || 4477420 > 104857600 -8552> 2014-06-09 16:22:12.169866 7f868c75b700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8551> 2014-06-09 16:22:12.169864 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -8550> 2014-06-09 16:22:12.169871 7f8692f68700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8549> 2014-06-09 16:22:12.169875 7f8691f66700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8548> 2014-06-09 16:22:12.169880 7f8692767700 2 waiting 51 > 50 ops || 4238325 > 104857600 -8547> 2014-06-09 16:22:12.169873 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.412( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:202 -8546> 2014-06-09 16:22:12.169883 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d15a0] -8545> 2014-06-09 16:22:12.169885 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8544> 2014-06-09 16:22:12.169884 7f868ef60700 2 waiting 51 > 50 ops || 4238292 > 104857600 -8543> 2014-06-09 16:22:12.169886 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8542> 2014-06-09 16:22:12.169889 7f86a5e11700 10 _finish_op 0xc2ebb60 seq 4126 osr(2.412 0x2ed15d0)/0x2ed15d0 -8541> 2014-06-09 16:22:12.169899 7f8689755700 10 op_submit_start 4175 -8540> 2014-06-09 16:22:12.169901 7f8689755700 5 queue_transactions (trailing journal) 4175 -8539> 2014-06-09 16:22:12.169901 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7bb80] -8538> 2014-06-09 16:22:12.169902 7f8689755700 5 queue_op 0x324a4840 seq 4175 osr(1.fa 0x384c970) 826 bytes (queue has 50 ops and 4237429 bytes) -8537> 2014-06-09 16:22:12.169905 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7bb80] -8536> 2014-06-09 16:22:12.169907 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8535> 2014-06-09 16:22:12.169908 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8534> 2014-06-09 16:22:12.169907 7f868cf5c700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8533> 2014-06-09 16:22:12.169911 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac900 (1 active) -8532> 2014-06-09 16:22:12.169912 7f868e75f700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8531> 2014-06-09 16:22:12.169914 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34979e0 (2 active) -8530> 2014-06-09 16:22:12.169917 7f8691765700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8529> 2014-06-09 16:22:12.169918 7f86a5e11700 5 _do_op 0x16096420 seq 4127 osr(1.207 0x384ccd0)/0x384ccd0 start -8528> 2014-06-09 16:22:12.169921 7f86a5e11700 10 _do_transaction on 0xc144600 -8527> 2014-06-09 16:22:12.169921 7f8689755700 10 op_submit_finish 4175 -8526> 2014-06-09 16:22:12.169927 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8525> 2014-06-09 16:22:12.169925 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 243/213 242/242/212) -8524> 2014-06-09 16:22:12.169959 7f8689755700 10 log is not dirty -8523> 2014-06-09 16:22:12.169964 7f8689755700 5 queue_transactions existing osr(3.375 0x3ed1060)/0x3ed1060 -8522> 2014-06-09 16:22:12.169967 7f8689755700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8521> 2014-06-09 16:22:12.169967 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8520> 2014-06-09 16:22:12.169971 7f8689f56700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8519> 2014-06-09 16:22:12.169976 7f8690f64700 2 waiting 51 > 50 ops || 4238287 > 104857600 -8518> 2014-06-09 16:22:12.169980 7f868a757700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8517> 2014-06-09 16:22:12.169985 7f868d75d700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8516> 2014-06-09 16:22:12.169985 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4127.0.0 <= header.spos 150443.0.0 -8515> 2014-06-09 16:22:12.169989 7f868ff62700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8514> 2014-06-09 16:22:12.169990 7f86a5e11700 5 submit_transaction r = 0 -8513> 2014-06-09 16:22:12.169994 7f868bf5a700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8512> 2014-06-09 16:22:12.169999 7f868b759700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8511> 2014-06-09 16:22:12.170003 7f868af58700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8510> 2014-06-09 16:22:12.170007 7f868f761700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8509> 2014-06-09 16:22:12.170008 7f86a5e11700 10 _do_op 0x16096420 seq 4127 r = 0, finisher 0xcc82120 0 -8508> 2014-06-09 16:22:12.170011 7f868df5e700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8507> 2014-06-09 16:22:12.170015 7f86a5e11700 10 _finish_op 0x16096420 seq 4127 osr(1.207 0x384ccd0)/0x384ccd0 -8506> 2014-06-09 16:22:12.170016 7f8690763700 2 waiting 51 > 50 ops || 4477385 > 104857600 -8505> 2014-06-09 16:22:12.170016 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd76e20] -8504> 2014-06-09 16:22:12.170020 7f868c75b700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8503> 2014-06-09 16:22:12.170025 7f8692f68700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8502> 2014-06-09 16:22:12.170020 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -8501> 2014-06-09 16:22:12.170030 7f8691f66700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8500> 2014-06-09 16:22:12.170028 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -8499> 2014-06-09 16:22:12.170034 7f8692767700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8498> 2014-06-09 16:22:12.170034 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.207( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 -8497> 2014-06-09 16:22:12.170038 7f868ef60700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8496> 2014-06-09 16:22:12.170040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd76e20] -8495> 2014-06-09 16:22:12.170041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8494> 2014-06-09 16:22:12.170041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8493> 2014-06-09 16:22:12.170051 7f868cf5c700 10 op_submit_start 4176 -8492> 2014-06-09 16:22:12.170053 7f868cf5c700 5 queue_transactions (trailing journal) 4176 -8491> 2014-06-09 16:22:12.170055 7f868cf5c700 5 queue_op 0x309a3800 seq 4176 osr(1.3ef 0x384d3f0) 861 bytes (queue has 50 ops and 4237429 bytes) -8490> 2014-06-09 16:22:12.170056 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc82120] -8489> 2014-06-09 16:22:12.170060 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc82120] -8488> 2014-06-09 16:22:12.170060 7f8691765700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8487> 2014-06-09 16:22:12.170062 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8486> 2014-06-09 16:22:12.170063 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8485> 2014-06-09 16:22:12.170065 7f8689755700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8484> 2014-06-09 16:22:12.170066 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34979e0 (1 active) -8483> 2014-06-09 16:22:12.170069 7f868e75f700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8482> 2014-06-09 16:22:12.170069 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652d80 (2 active) -8481> 2014-06-09 16:22:12.170074 7f86a5e11700 5 _do_op 0x30a23c0 seq 4128 osr(1.5c1 0x384dd90)/0x384dd90 start -8480> 2014-06-09 16:22:12.170075 7f8690f64700 2 waiting 51 > 50 ops || 4238287 > 104857600 -8479> 2014-06-09 16:22:12.170077 7f86a5e11700 10 _do_transaction on 0xbffe840 -8478> 2014-06-09 16:22:12.170080 7f868cf5c700 10 op_submit_finish 4176 -8477> 2014-06-09 16:22:12.170082 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8476> 2014-06-09 16:22:12.170083 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8475> 2014-06-09 16:22:12.170098 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -8474> 2014-06-09 16:22:12.170107 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -8473> 2014-06-09 16:22:12.170115 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8472> 2014-06-09 16:22:12.170123 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8471> 2014-06-09 16:22:12.170124 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -8470> 2014-06-09 16:22:12.170133 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8469> 2014-06-09 16:22:12.170137 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4128.0.0 <= header.spos 150443.0.0 -8468> 2014-06-09 16:22:12.170142 7f86a5e11700 5 submit_transaction r = 0 -8467> 2014-06-09 16:22:12.170147 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 -8466> 2014-06-09 16:22:12.170154 7f86a5e11700 10 _do_op 0x30a23c0 seq 4128 r = 0, finisher 0xbd68ae0 0 -8465> 2014-06-09 16:22:12.170156 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -8464> 2014-06-09 16:22:12.170165 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x320744e0] -8463> 2014-06-09 16:22:12.170168 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x320744e0] -8462> 2014-06-09 16:22:12.170169 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8461> 2014-06-09 16:22:12.170171 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8460> 2014-06-09 16:22:12.170164 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.383( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -8459> 2014-06-09 16:22:12.170174 7f868cf5c700 10 log is not dirty -8458> 2014-06-09 16:22:12.170174 7f86a5e11700 10 _finish_op 0x30a23c0 seq 4128 osr(1.5c1 0x384dd90)/0x384dd90 -8457> 2014-06-09 16:22:12.170179 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8456> 2014-06-09 16:22:12.170187 7f868d75d700 10 op_submit_start 4177 -8455> 2014-06-09 16:22:12.170188 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd68ae0] -8454> 2014-06-09 16:22:12.170189 7f868d75d700 5 queue_transactions (trailing journal) 4177 -8453> 2014-06-09 16:22:12.170190 7f868d75d700 5 queue_op 0x39d9da0 seq 4177 osr(3.6cb 0x3ed1cb0) 828 bytes (queue has 50 ops and 4237429 bytes) -8452> 2014-06-09 16:22:12.170196 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd68ae0] -8451> 2014-06-09 16:22:12.170198 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8450> 2014-06-09 16:22:12.170199 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8449> 2014-06-09 16:22:12.170190 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.060272 7 0.000333 -8448> 2014-06-09 16:22:12.170202 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652d80 (1 active) -8447> 2014-06-09 16:22:12.170205 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc000 (2 active) -8446> 2014-06-09 16:22:12.170202 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -8445> 2014-06-09 16:22:12.170210 7f868d75d700 10 op_submit_finish 4177 -8444> 2014-06-09 16:22:12.170212 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8443> 2014-06-09 16:22:12.170214 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8442> 2014-06-09 16:22:12.170223 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -8441> 2014-06-09 16:22:12.170234 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -8440> 2014-06-09 16:22:12.170230 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] exit Started/Primary/Active/Activating 0.946122 7 0.000498 -8439> 2014-06-09 16:22:12.170242 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -8438> 2014-06-09 16:22:12.170244 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] enter Started/Primary/Active/Recovered -8437> 2014-06-09 16:22:12.170250 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -8436> 2014-06-09 16:22:12.170256 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] needs_recovery is recovered -8435> 2014-06-09 16:22:12.170261 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) -8434> 2014-06-09 16:22:12.170268 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 -8433> 2014-06-09 16:22:12.170272 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) -8432> 2014-06-09 16:22:12.170282 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -8431> 2014-06-09 16:22:12.170281 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] enter Started/Primary/Active/Clean -8430> 2014-06-09 16:22:12.170298 7f868cf5c700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 243 -8429> 2014-06-09 16:22:12.170291 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] finish_recovery -8428> 2014-06-09 16:22:12.170301 7f868cf5c700 1 -- 172.16.96.202:6850/30904 --> osd.3 172.16.96.201:6823/50569 -- pg_info(1 pgs e243:2.61b) v4 -- ?+0 0x334408c0 -8427> 2014-06-09 16:22:12.170301 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active] clear_recovery_state -8426> 2014-06-09 16:22:12.170315 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 -8425> 2014-06-09 16:22:12.170313 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) -8424> 2014-06-09 16:22:12.170326 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) -8423> 2014-06-09 16:22:12.170337 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) -8422> 2014-06-09 16:22:12.170349 7f868cf5c700 10 log is not dirty -8421> 2014-06-09 16:22:12.170353 7f868cf5c700 5 queue_transactions existing osr(2.61b 0x2ed1cd0)/0x2ed1cd0 -8420> 2014-06-09 16:22:12.170356 7f868cf5c700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8419> 2014-06-09 16:22:12.170348 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) -8418> 2014-06-09 16:22:12.170361 7f868a757700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8417> 2014-06-09 16:22:12.170359 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] share_pg_info -8416> 2014-06-09 16:22:12.170366 7f8689f56700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8415> 2014-06-09 16:22:12.170370 7f868ff62700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8414> 2014-06-09 16:22:12.170371 7f868d75d700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 -8413> 2014-06-09 16:22:12.170375 7f868af58700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8412> 2014-06-09 16:22:12.170379 7f868f761700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8411> 2014-06-09 16:22:12.170374 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.1 172.16.96.201:6811/57862 -- pg_info(1 pgs e243:3.78a) v4 -- ?+0 0x240e8700 -8410> 2014-06-09 16:22:12.170383 7f868df5e700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8409> 2014-06-09 16:22:12.170388 7f868bf5a700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8408> 2014-06-09 16:22:12.170392 7f868c75b700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8407> 2014-06-09 16:22:12.170396 7f8692f68700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8406> 2014-06-09 16:22:12.170388 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] publish_stats_to_osd 243:875 -8405> 2014-06-09 16:22:12.170400 7f868b759700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8404> 2014-06-09 16:22:12.170404 7f8690763700 2 waiting 51 > 50 ops || 4477385 > 104857600 -8403> 2014-06-09 16:22:12.170408 7f868ef60700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8402> 2014-06-09 16:22:12.170413 7f8689755700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8401> 2014-06-09 16:22:12.170416 7f868e75f700 2 waiting 51 > 50 ops || 4238257 > 104857600 -8400> 2014-06-09 16:22:12.170421 7f8690f64700 2 waiting 51 > 50 ops || 4238287 > 104857600 -8399> 2014-06-09 16:22:12.170424 7f868d75d700 10 log is not dirty -8398> 2014-06-09 16:22:12.170425 7f8692767700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8397> 2014-06-09 16:22:12.170428 7f868d75d700 5 queue_transactions existing osr(3.78a 0x3ed1d80)/0x3ed1d80 -8396> 2014-06-09 16:22:12.170428 7f8691f66700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8395> 2014-06-09 16:22:12.170430 7f868d75d700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8394> 2014-06-09 16:22:12.170434 7f8691765700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8393> 2014-06-09 16:22:12.170437 7f86a5e11700 5 _do_op 0x230b9d40 seq 4129 osr(1.2a3 0x384d1f0)/0x384d1f0 start -8392> 2014-06-09 16:22:12.170441 7f86a5e11700 10 _do_transaction on 0xc128180 -8391> 2014-06-09 16:22:12.170446 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8390> 2014-06-09 16:22:12.170488 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8389> 2014-06-09 16:22:12.170503 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4129.0.0 <= header.spos 150443.0.0 -8388> 2014-06-09 16:22:12.170508 7f86a5e11700 5 submit_transaction r = 0 -8387> 2014-06-09 16:22:12.170519 7f86a5e11700 10 _do_op 0x230b9d40 seq 4129 r = 0, finisher 0xccf8120 0 -8386> 2014-06-09 16:22:12.170529 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd760] -8385> 2014-06-09 16:22:12.170532 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd760] -8384> 2014-06-09 16:22:12.170533 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8383> 2014-06-09 16:22:12.170534 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8382> 2014-06-09 16:22:12.170537 7f86a5e11700 10 _finish_op 0x230b9d40 seq 4129 osr(1.2a3 0x384d1f0)/0x384d1f0 -8381> 2014-06-09 16:22:12.170547 7f868cf5c700 10 op_submit_start 4178 -8380> 2014-06-09 16:22:12.170549 7f868cf5c700 5 queue_transactions (trailing journal) 4178 -8379> 2014-06-09 16:22:12.170549 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf8120] -8378> 2014-06-09 16:22:12.170551 7f868cf5c700 5 queue_op 0x309a3560 seq 4178 osr(2.61b 0x2ed1cd0) 861 bytes (queue has 50 ops and 4237462 bytes) -8377> 2014-06-09 16:22:12.170553 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf8120] -8376> 2014-06-09 16:22:12.170554 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8375> 2014-06-09 16:22:12.170555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8374> 2014-06-09 16:22:12.170556 7f868a757700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8373> 2014-06-09 16:22:12.170559 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc000 (1 active) -8372> 2014-06-09 16:22:12.170560 7f8689f56700 2 waiting 51 > 50 ops || 4238323 > 104857600 -8371> 2014-06-09 16:22:12.170562 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dcfc0 (2 active) -8370> 2014-06-09 16:22:12.170565 7f868ff62700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8369> 2014-06-09 16:22:12.170566 7f86a5e11700 5 _do_op 0xc88b4a0 seq 4130 osr(0.4c8 0x2bd7670)/0x2bd7670 start -8368> 2014-06-09 16:22:12.170569 7f86a5e11700 10 _do_transaction on 0xbf28fc0 -8367> 2014-06-09 16:22:12.170570 7f868af58700 2 waiting 51 > 50 ops || 4238323 > 104857600 -8366> 2014-06-09 16:22:12.170574 7f868f761700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8365> 2014-06-09 16:22:12.170574 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8364> 2014-06-09 16:22:12.170578 7f868df5e700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8363> 2014-06-09 16:22:12.170583 7f868bf5a700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8362> 2014-06-09 16:22:12.170587 7f868c75b700 2 waiting 51 > 50 ops || 4238323 > 104857600 -8361> 2014-06-09 16:22:12.170592 7f8692f68700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8360> 2014-06-09 16:22:12.170596 7f868b759700 2 waiting 51 > 50 ops || 4238290 > 104857600 -8359> 2014-06-09 16:22:12.170601 7f868cf5c700 10 op_submit_finish 4178 -8358> 2014-06-09 16:22:12.170604 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8357> 2014-06-09 16:22:12.170614 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8356> 2014-06-09 16:22:12.170618 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.114057 8 0.002010 -8355> 2014-06-09 16:22:12.170629 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4130.0.0 <= header.spos 150443.0.0 -8354> 2014-06-09 16:22:12.170634 7f86a5e11700 5 submit_transaction r = 0 -8353> 2014-06-09 16:22:12.170629 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -8352> 2014-06-09 16:22:12.170645 7f86a5e11700 10 _do_op 0xc88b4a0 seq 4130 r = 0, finisher 0x3281b3c0 0 -8351> 2014-06-09 16:22:12.170640 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8350> 2014-06-09 16:22:12.170656 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5fba0] -8349> 2014-06-09 16:22:12.170650 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -8348> 2014-06-09 16:22:12.170658 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5fba0] -8347> 2014-06-09 16:22:12.170660 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8346> 2014-06-09 16:22:12.170661 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8345> 2014-06-09 16:22:12.170665 7f86a5e11700 10 _finish_op 0xc88b4a0 seq 4130 osr(0.4c8 0x2bd7670)/0x2bd7670 -8344> 2014-06-09 16:22:12.170661 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -8343> 2014-06-09 16:22:12.170673 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dcfc0 (1 active) -8342> 2014-06-09 16:22:12.170674 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281b3c0] -8341> 2014-06-09 16:22:12.170675 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa900 (2 active) -8340> 2014-06-09 16:22:12.170670 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -8339> 2014-06-09 16:22:12.170678 7f86a5e11700 5 _do_op 0x30a3860 seq 4131 osr(0.530 0x2bd78f0)/0x2bd78f0 start -8338> 2014-06-09 16:22:12.170679 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281b3c0] -8337> 2014-06-09 16:22:12.170681 7f86a5e11700 10 _do_transaction on 0xbe8b2c0 -8336> 2014-06-09 16:22:12.170681 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8335> 2014-06-09 16:22:12.170682 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8334> 2014-06-09 16:22:12.170678 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -8333> 2014-06-09 16:22:12.170686 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8332> 2014-06-09 16:22:12.170689 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) -8331> 2014-06-09 16:22:12.170700 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) -8330> 2014-06-09 16:22:12.170710 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) -8329> 2014-06-09 16:22:12.170725 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8328> 2014-06-09 16:22:12.170720 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) -8327> 2014-06-09 16:22:12.170730 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -8326> 2014-06-09 16:22:12.170738 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4131.0.0 <= header.spos 150443.0.0 -8325> 2014-06-09 16:22:12.170742 7f86a5e11700 5 submit_transaction r = 0 -8324> 2014-06-09 16:22:12.170742 7f868cf5c700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 243 -8323> 2014-06-09 16:22:12.170745 7f868cf5c700 1 -- 172.16.96.202:6850/30904 --> osd.1 172.16.96.201:6811/57862 -- pg_info(1 pgs e243:1.446) v4 -- ?+0 0x2f0396c0 -8322> 2014-06-09 16:22:12.170753 7f86a5e11700 10 _do_op 0x30a3860 seq 4131 r = 0, finisher 0x307f0260 0 -8321> 2014-06-09 16:22:12.170755 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 -8320> 2014-06-09 16:22:12.170763 7f86a5e11700 10 _finish_op 0x30a3860 seq 4131 osr(0.530 0x2bd78f0)/0x2bd78f0 -8319> 2014-06-09 16:22:12.170764 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074f40] -8318> 2014-06-09 16:22:12.170767 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074f40] -8317> 2014-06-09 16:22:12.170769 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8316> 2014-06-09 16:22:12.170770 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8315> 2014-06-09 16:22:12.170772 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f0260] -8314> 2014-06-09 16:22:12.170775 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f0260] -8313> 2014-06-09 16:22:12.170779 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8312> 2014-06-09 16:22:12.170780 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8311> 2014-06-09 16:22:12.170783 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa900 (1 active) -8310> 2014-06-09 16:22:12.170786 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62f7a0 (2 active) -8309> 2014-06-09 16:22:12.170787 7f868cf5c700 10 log is not dirty -8308> 2014-06-09 16:22:12.170788 7f86a5e11700 5 _do_op 0x30d966c0 seq 4132 osr(0.622 0x2bd7cf0)/0x2bd7cf0 start -8307> 2014-06-09 16:22:12.170791 7f86a5e11700 10 _do_transaction on 0xc2df440 -8306> 2014-06-09 16:22:12.170792 7f868cf5c700 5 queue_transactions existing osr(1.446 0x384d6b0)/0x384d6b0 -8305> 2014-06-09 16:22:12.170794 7f868cf5c700 10 op_submit_start 4179 -8304> 2014-06-09 16:22:12.170795 7f868cf5c700 5 queue_transactions (trailing journal) 4179 -8303> 2014-06-09 16:22:12.170797 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8302> 2014-06-09 16:22:12.170797 7f868cf5c700 5 queue_op 0xc704ba0 seq 4179 osr(1.446 0x384d6b0) 861 bytes (queue has 49 ops and 4236667 bytes) -8301> 2014-06-09 16:22:12.170800 7f868cf5c700 10 op_submit_finish 4179 -8300> 2014-06-09 16:22:12.170802 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8299> 2014-06-09 16:22:12.170817 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -8298> 2014-06-09 16:22:12.170826 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops -8297> 2014-06-09 16:22:12.170836 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8296> 2014-06-09 16:22:12.170836 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -8295> 2014-06-09 16:22:12.170845 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -8294> 2014-06-09 16:22:12.170853 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4132.0.0 <= header.spos 150443.0.0 -8293> 2014-06-09 16:22:12.170858 7f86a5e11700 5 submit_transaction r = 0 -8292> 2014-06-09 16:22:12.170854 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8291> 2014-06-09 16:22:12.170869 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1008 -8290> 2014-06-09 16:22:12.170878 7f86a5e11700 10 _do_op 0x30d966c0 seq 4132 r = 0, finisher 0x3062b140 0 -8289> 2014-06-09 16:22:12.170884 7f86a5e11700 10 _finish_op 0x30d966c0 seq 4132 osr(0.622 0x2bd7cf0)/0x2bd7cf0 -8288> 2014-06-09 16:22:12.170880 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -8287> 2014-06-09 16:22:12.170886 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x13964600] -8286> 2014-06-09 16:22:12.170893 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062b140] -8285> 2014-06-09 16:22:12.170896 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062b140] -8284> 2014-06-09 16:22:12.170889 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[3.59f( v 213'624 (0'0,213'624] local-les=243 n=117 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -8283> 2014-06-09 16:22:12.170897 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8282> 2014-06-09 16:22:12.170899 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8281> 2014-06-09 16:22:12.170890 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -8280> 2014-06-09 16:22:12.170899 7f868cf5c700 10 log is not dirty -8279> 2014-06-09 16:22:12.170901 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62f7a0 (1 active) -8278> 2014-06-09 16:22:12.170904 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd200 (2 active) -8277> 2014-06-09 16:22:12.170900 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -8276> 2014-06-09 16:22:12.170906 7f86a5e11700 5 _do_op 0x306075c0 seq 4133 osr(1.86 0x384c490)/0x384c490 start -8275> 2014-06-09 16:22:12.170908 7f86a5e11700 10 _do_transaction on 0xc2af500 -8274> 2014-06-09 16:22:12.170914 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8273> 2014-06-09 16:22:12.170909 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.622( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:184 -8272> 2014-06-09 16:22:12.170917 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x13964600] -8271> 2014-06-09 16:22:12.170919 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8270> 2014-06-09 16:22:12.170920 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8269> 2014-06-09 16:22:12.170953 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8268> 2014-06-09 16:22:12.170970 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4133.0.0 <= header.spos 150443.0.0 -8267> 2014-06-09 16:22:12.170904 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8266> 2014-06-09 16:22:12.170975 7f86a5e11700 5 submit_transaction r = 0 -8265> 2014-06-09 16:22:12.170978 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.086089 7 0.007362 -8264> 2014-06-09 16:22:12.170990 7f86a5e11700 10 _do_op 0x306075c0 seq 4133 r = 0, finisher 0x32078c00 0 -8263> 2014-06-09 16:22:12.170990 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -8262> 2014-06-09 16:22:12.171001 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32075f20] -8261> 2014-06-09 16:22:12.171001 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8260> 2014-06-09 16:22:12.171004 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -8259> 2014-06-09 16:22:12.171011 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -8258> 2014-06-09 16:22:12.171014 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -8257> 2014-06-09 16:22:12.171022 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -8256> 2014-06-09 16:22:12.171022 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.86( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:208 -8255> 2014-06-09 16:22:12.171032 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32075f20] -8254> 2014-06-09 16:22:12.171034 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8253> 2014-06-09 16:22:12.171035 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8252> 2014-06-09 16:22:12.171031 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -8251> 2014-06-09 16:22:12.171040 7f86a5e11700 10 _finish_op 0x306075c0 seq 4133 osr(1.86 0x384c490)/0x384c490 -8250> 2014-06-09 16:22:12.171039 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -8249> 2014-06-09 16:22:12.171049 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32078c00] -8248> 2014-06-09 16:22:12.171056 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32078c00] -8247> 2014-06-09 16:22:12.171058 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8246> 2014-06-09 16:22:12.171059 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8245> 2014-06-09 16:22:12.171049 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -8244> 2014-06-09 16:22:12.171062 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd200 (1 active) -8243> 2014-06-09 16:22:12.171065 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383f9e0 (2 active) -8242> 2014-06-09 16:22:12.171067 7f86a5e11700 5 _do_op 0x2fcafbc0 seq 4134 osr(2.19 0x2ed0270)/0x2ed0270 start -8241> 2014-06-09 16:22:12.171060 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -8240> 2014-06-09 16:22:12.171070 7f86a5e11700 10 _do_transaction on 0xc2df680 -8239> 2014-06-09 16:22:12.171077 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8238> 2014-06-09 16:22:12.171070 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -8237> 2014-06-09 16:22:12.171080 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -8236> 2014-06-09 16:22:12.171089 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -8235> 2014-06-09 16:22:12.171105 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 -8234> 2014-06-09 16:22:12.171107 7f868cf5c700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:0.66b) v4 -- ?+0 0x30caf6c0 -8233> 2014-06-09 16:22:12.171117 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8232> 2014-06-09 16:22:12.171121 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 -8231> 2014-06-09 16:22:12.171133 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4134.0.0 <= header.spos 150443.0.0 -8230> 2014-06-09 16:22:12.171138 7f86a5e11700 5 submit_transaction r = 0 -8229> 2014-06-09 16:22:12.171153 7f868cf5c700 10 log is not dirty -8228> 2014-06-09 16:22:12.171153 7f86a5e11700 10 _do_op 0x2fcafbc0 seq 4134 r = 0, finisher 0x30c540a0 0 -8227> 2014-06-09 16:22:12.171158 7f868cf5c700 5 queue_transactions existing osr(0.66b 0x2bd7eb0)/0x2bd7eb0 -8226> 2014-06-09 16:22:12.171161 7f868cf5c700 10 op_submit_start 4180 -8225> 2014-06-09 16:22:12.171162 7f868cf5c700 5 queue_transactions (trailing journal) 4180 -8224> 2014-06-09 16:22:12.171163 7f868cf5c700 5 queue_op 0x324e62a0 seq 4180 osr(0.66b 0x2bd7eb0) 861 bytes (queue has 48 ops and 4235809 bytes) -8223> 2014-06-09 16:22:12.171163 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x281b76a0] -8222> 2014-06-09 16:22:12.171165 7f868cf5c700 10 op_submit_finish 4180 -8221> 2014-06-09 16:22:12.171167 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -8220> 2014-06-09 16:22:12.171168 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8219> 2014-06-09 16:22:12.171177 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -8218> 2014-06-09 16:22:12.171180 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.055424 7 0.000370 -8217> 2014-06-09 16:22:12.171187 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.19( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 -8216> 2014-06-09 16:22:12.171196 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x281b76a0] -8215> 2014-06-09 16:22:12.171198 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8214> 2014-06-09 16:22:12.171191 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -8213> 2014-06-09 16:22:12.171199 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8212> 2014-06-09 16:22:12.171203 7f86a5e11700 10 _finish_op 0x2fcafbc0 seq 4134 osr(2.19 0x2ed0270)/0x2ed0270 -8211> 2014-06-09 16:22:12.171201 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8210> 2014-06-09 16:22:12.171212 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30c540a0] -8209> 2014-06-09 16:22:12.171217 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30c540a0] -8208> 2014-06-09 16:22:12.171211 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -8207> 2014-06-09 16:22:12.171219 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8206> 2014-06-09 16:22:12.171220 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8205> 2014-06-09 16:22:12.171223 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383f9e0 (1 active) -8204> 2014-06-09 16:22:12.171226 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c000 (2 active) -8203> 2014-06-09 16:22:12.171221 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -8202> 2014-06-09 16:22:12.171228 7f86a5e11700 5 _do_op 0x15996b40 seq 4135 osr(1.2aa 0x384d1d0)/0x384d1d0 start -8201> 2014-06-09 16:22:12.171231 7f86a5e11700 10 _do_transaction on 0xc05f5c0 -8200> 2014-06-09 16:22:12.171230 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -8199> 2014-06-09 16:22:12.171237 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8198> 2014-06-09 16:22:12.171238 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -8197> 2014-06-09 16:22:12.171249 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -8196> 2014-06-09 16:22:12.171260 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -8195> 2014-06-09 16:22:12.171269 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -8194> 2014-06-09 16:22:12.171278 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8193> 2014-06-09 16:22:12.171281 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 243 -8192> 2014-06-09 16:22:12.171284 7f868cf5c700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:0.67d) v4 -- ?+0 0x30caec40 -8191> 2014-06-09 16:22:12.171294 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4135.0.0 <= header.spos 150443.0.0 -8190> 2014-06-09 16:22:12.171299 7f86a5e11700 5 submit_transaction r = 0 -8189> 2014-06-09 16:22:12.171297 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 -8188> 2014-06-09 16:22:12.171312 7f86a5e11700 10 _do_op 0x15996b40 seq 4135 r = 0, finisher 0xbce6d60 0 -8187> 2014-06-09 16:22:12.171321 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002de0] -8186> 2014-06-09 16:22:12.171329 7f868cf5c700 10 log is not dirty -8185> 2014-06-09 16:22:12.171324 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -8184> 2014-06-09 16:22:12.171333 7f868cf5c700 5 queue_transactions existing osr(0.67d 0x2bd7e50)/0x2bd7e50 -8183> 2014-06-09 16:22:12.171336 7f868cf5c700 10 op_submit_start 4181 -8182> 2014-06-09 16:22:12.171337 7f868cf5c700 5 queue_transactions (trailing journal) 4181 -8181> 2014-06-09 16:22:12.171338 7f868cf5c700 5 queue_op 0x160970e0 seq 4181 osr(0.67d 0x2bd7e50) 861 bytes (queue has 48 ops and 4235812 bytes) -8180> 2014-06-09 16:22:12.171334 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -8179> 2014-06-09 16:22:12.171341 7f868cf5c700 10 op_submit_finish 4181 -8178> 2014-06-09 16:22:12.171343 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.2aa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -8177> 2014-06-09 16:22:12.171353 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002de0] -8176> 2014-06-09 16:22:12.171344 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8175> 2014-06-09 16:22:12.171355 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8174> 2014-06-09 16:22:12.171356 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8173> 2014-06-09 16:22:12.171360 7f86a5e11700 10 _finish_op 0x15996b40 seq 4135 osr(1.2aa 0x384d1d0)/0x384d1d0 -8172> 2014-06-09 16:22:12.171358 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -8171> 2014-06-09 16:22:12.171369 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbce6d60] -8170> 2014-06-09 16:22:12.171366 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -8169> 2014-06-09 16:22:12.172807 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8168> 2014-06-09 16:22:12.172820 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -8167> 2014-06-09 16:22:12.172886 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8166> 2014-06-09 16:22:12.172901 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 -8165> 2014-06-09 16:22:12.172911 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -8164> 2014-06-09 16:22:12.172918 7f868cf5c700 20 osd.26 pg_epoch: 243 pg[2.621( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -8163> 2014-06-09 16:22:12.172928 7f868cf5c700 10 log is not dirty -8162> 2014-06-09 16:22:12.172933 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8161> 2014-06-09 16:22:12.172945 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.019763 7 0.000288 -8160> 2014-06-09 16:22:12.172956 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -8159> 2014-06-09 16:22:12.172966 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8158> 2014-06-09 16:22:12.172976 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -8157> 2014-06-09 16:22:12.172986 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -8156> 2014-06-09 16:22:12.172995 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -8155> 2014-06-09 16:22:12.173003 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -8154> 2014-06-09 16:22:12.173014 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -8153> 2014-06-09 16:22:12.173025 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -8152> 2014-06-09 16:22:12.173034 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -8151> 2014-06-09 16:22:12.173046 7f868cf5c700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 243 -8150> 2014-06-09 16:22:12.173049 7f868cf5c700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:2.573) v4 -- ?+0 0x30c00700 -8149> 2014-06-09 16:22:12.173062 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.573( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 -8148> 2014-06-09 16:22:12.173094 7f868cf5c700 10 log is not dirty -8147> 2014-06-09 16:22:12.173098 7f868cf5c700 5 queue_transactions existing osr(2.573 0x2ed18b0)/0x2ed18b0 -8146> 2014-06-09 16:22:12.171373 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbce6d60] -8145> 2014-06-09 16:22:12.173106 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8144> 2014-06-09 16:22:12.173107 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8143> 2014-06-09 16:22:12.171376 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c000 (1 active) -8142> 2014-06-09 16:22:12.171402 7f868a757700 10 op_submit_start 4182 -8141> 2014-06-09 16:22:12.173114 7f868a757700 5 queue_transactions (trailing journal) 4182 -8140> 2014-06-09 16:22:12.173116 7f868a757700 5 queue_op 0xcabc780 seq 4182 osr(2.216 0x2ed0b50) 828 bytes (queue has 50 ops and 4237435 bytes) -8139> 2014-06-09 16:22:12.171413 7f8690763700 2 waiting 51 > 50 ops || 4477391 > 104857600 -8138> 2014-06-09 16:22:12.171917 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 5 ==== osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+2580480 (2437168134 0 0) 0x32da8240 con 0x30c3c2c0 -8137> 2014-06-09 16:22:12.173125 7f8690f64700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8136> 2014-06-09 16:22:12.173130 7f8692767700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8135> 2014-06-09 16:22:12.173135 7f8691f66700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8134> 2014-06-09 16:22:12.173140 7f868d75d700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8133> 2014-06-09 16:22:12.173137 7f86a68f6700 5 -- op tracker -- , seq: 3665, time: 2014-06-09 16:22:12.166912, event: header_read, op: osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -8132> 2014-06-09 16:22:12.173145 7f8691765700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8131> 2014-06-09 16:22:12.173149 7f8689f56700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8130> 2014-06-09 16:22:12.173150 7f86a68f6700 5 -- op tracker -- , seq: 3665, time: 2014-06-09 16:22:12.166912, event: throttled, op: osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -8129> 2014-06-09 16:22:12.173154 7f868ff62700 2 waiting 51 > 50 ops || 4238263 > 104857600 -8128> 2014-06-09 16:22:12.173159 7f868af58700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8127> 2014-06-09 16:22:12.173163 7f868f761700 2 waiting 51 > 50 ops || 4238263 > 104857600 -8126> 2014-06-09 16:22:12.173160 7f86a68f6700 5 -- op tracker -- , seq: 3665, time: 2014-06-09 16:22:12.171912, event: all_read, op: osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -8125> 2014-06-09 16:22:12.173168 7f868df5e700 2 waiting 51 > 50 ops || 4238263 > 104857600 -8124> 2014-06-09 16:22:12.173172 7f868bf5a700 2 waiting 51 > 50 ops || 4238263 > 104857600 -8123> 2014-06-09 16:22:12.173174 7f86a68f6700 5 -- op tracker -- , seq: 3665, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -8122> 2014-06-09 16:22:12.173177 7f868c75b700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8121> 2014-06-09 16:22:12.173182 7f8692f68700 2 waiting 51 > 50 ops || 4238263 > 104857600 -8120> 2014-06-09 16:22:12.173186 7f868b759700 2 waiting 51 > 50 ops || 4238263 > 104857600 -8119> 2014-06-09 16:22:12.173182 7f86a68f6700 20 osd.26 243 should_share_map client.4543 172.16.96.220:0/1123776 243 -8118> 2014-06-09 16:22:12.173191 7f868e75f700 2 waiting 51 > 50 ops || 4238263 > 104857600 -8117> 2014-06-09 16:22:12.173195 7f868cf5c700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8116> 2014-06-09 16:22:12.173199 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f65a0 (2 active) -8115> 2014-06-09 16:22:12.173205 7f868a757700 10 op_submit_finish 4182 -8114> 2014-06-09 16:22:12.173206 7f86a68f6700 15 osd.26 243 enqueue_op 0x2976cf00 prio 63 cost 2580480 latency 0.006294 osd_op(client.4543.0:33291 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 516096~2580480] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 -8113> 2014-06-09 16:22:12.173210 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8112> 2014-06-09 16:22:12.173223 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local -8111> 2014-06-09 16:22:12.173232 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -8110> 2014-06-09 16:22:12.173240 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8109> 2014-06-09 16:22:12.173248 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -8108> 2014-06-09 16:22:12.173256 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8107> 2014-06-09 16:22:12.173270 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:170 -8106> 2014-06-09 16:22:12.173279 7f868a757700 20 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -8105> 2014-06-09 16:22:12.173287 7f868a757700 20 osd.26 pg_epoch: 243 pg[2.244( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop -8104> 2014-06-09 16:22:12.173296 7f868a757700 10 log is not dirty -8103> 2014-06-09 16:22:12.173301 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -8102> 2014-06-09 16:22:12.173315 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.068818 7 0.000428 -8101> 2014-06-09 16:22:12.173327 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -8100> 2014-06-09 16:22:12.173338 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -8099> 2014-06-09 16:22:12.173349 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -8098> 2014-06-09 16:22:12.173361 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -8097> 2014-06-09 16:22:12.173371 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery -8096> 2014-06-09 16:22:12.173380 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state -8095> 2014-06-09 16:22:12.173391 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -8094> 2014-06-09 16:22:12.173403 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -8093> 2014-06-09 16:22:12.173414 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -8092> 2014-06-09 16:22:12.173425 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -8091> 2014-06-09 16:22:12.173435 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info -8090> 2014-06-09 16:22:12.173449 7f868a757700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 243 -8089> 2014-06-09 16:22:12.173452 7f868a757700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:3.362) v4 -- ?+0 0x310db6c0 -8088> 2014-06-09 16:22:12.173465 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1009 -8087> 2014-06-09 16:22:12.173500 7f868a757700 10 log is not dirty -8086> 2014-06-09 16:22:12.173504 7f868a757700 5 queue_transactions existing osr(3.362 0x3ed10a0)/0x3ed10a0 -8085> 2014-06-09 16:22:12.173507 7f868a757700 2 waiting 51 > 50 ops || 4238296 > 104857600 -8084> 2014-06-09 16:22:12.173511 7f868ef60700 10 op_submit_start 4183 -8083> 2014-06-09 16:22:12.173513 7f868ef60700 5 queue_transactions (trailing journal) 4183 -8082> 2014-06-09 16:22:12.173515 7f868ef60700 5 queue_op 0x39d98c0 seq 4183 osr(2.5f8 0x2ed1db0) 828 bytes (queue has 50 ops and 4237435 bytes) -8081> 2014-06-09 16:22:12.173518 7f868ef60700 10 op_submit_finish 4183 -8080> 2014-06-09 16:22:12.173524 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.69c( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.69c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) -8079> 2014-06-09 16:22:12.173566 7f868ef60700 10 log is not dirty -8078> 2014-06-09 16:22:12.173571 7f868ef60700 5 queue_transactions existing osr(2.69c 0x2ed1f20)/0x2ed1f20 -8077> 2014-06-09 16:22:12.173573 7f868ef60700 2 waiting 51 > 50 ops || 4238263 > 104857600 -8076> 2014-06-09 16:22:12.173600 7f86a5e11700 5 _do_op 0x30a3320 seq 4136 osr(1.73b 0x2ed00c0)/0x2ed00c0 start -8075> 2014-06-09 16:22:12.173608 7f86a5e11700 10 _do_transaction on 0xc145a40 -8074> 2014-06-09 16:22:12.173614 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -8073> 2014-06-09 16:22:12.173635 7f8683620700 1 -- 172.16.96.202:6850/30904 <== osd.6 172.16.96.201:6849/52045 268 ==== osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) v2 ==== 176+0+0 (2571984869 0 0) 0x9a09400 con 0xbf5adc0 -8072> 2014-06-09 16:22:12.173653 7f8683620700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.173621, event: header_read, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) -8071> 2014-06-09 16:22:12.173662 7f8683620700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.173622, event: throttled, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) -8070> 2014-06-09 16:22:12.173669 7f8683620700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.173630, event: all_read, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) -8069> 2014-06-09 16:22:12.173678 7f8683620700 5 -- op tracker -- , seq: 3666, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) -8068> 2014-06-09 16:22:12.173683 7f8683620700 10 osd.26 243 handle_replica_op osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) v2 epoch 243 -8067> 2014-06-09 16:22:12.173688 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -8066> 2014-06-09 16:22:12.173688 7f8683620700 20 osd.26 243 should_share_map osd.6 172.16.96.201:6849/52045 243 -8065> 2014-06-09 16:22:12.173694 7f8683620700 15 osd.26 243 enqueue_op 0x2976ce00 prio 196 cost 0 latency 0.000072 osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) v2 -8064> 2014-06-09 16:22:12.173704 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4136.0.0 <= header.spos 150443.0.0 -8063> 2014-06-09 16:22:12.173708 7f86a5e11700 5 submit_transaction r = 0 -8062> 2014-06-09 16:22:12.173713 7f869c77b700 1 -- 172.16.96.202:6830/30904 <== mon.0 172.16.96.201:6789/0 19 ==== osd_map(243..244 src has 1..244) v3 ==== 473+0+0 (2365343850 0 0) 0x32da8fc0 con 0xbcb86e0 -8061> 2014-06-09 16:22:12.173723 7f869c77b700 10 osd.26 243 do_waiters -- start -8060> 2014-06-09 16:22:12.173725 7f869c77b700 10 osd.26 243 do_waiters -- finish -8059> 2014-06-09 16:22:12.173727 7f869c77b700 20 osd.26 243 _dispatch 0x32da8fc0 osd_map(243..244 src has 1..244) v3 -8058> 2014-06-09 16:22:12.173743 7f86a5e11700 10 _do_op 0x30a3320 seq 4136 r = 0, finisher 0xcd32a20 0 -8057> 2014-06-09 16:22:12.173744 7f869c77b700 3 osd.26 243 handle_osd_map epochs [243,244], i have 243, src has [1,244] -8056> 2014-06-09 16:22:12.173748 7f869c77b700 10 osd.26 243 handle_osd_map got inc map for epoch 244 -8055> 2014-06-09 16:22:12.173755 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd71c00] -8054> 2014-06-09 16:22:12.173757 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd71c00] -8053> 2014-06-09 16:22:12.173758 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -8052> 2014-06-09 16:22:12.173759 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -8051> 2014-06-09 16:22:12.173762 7f86a5e11700 10 _finish_op 0x30a3320 seq 4136 osr(1.73b 0x2ed00c0)/0x2ed00c0 -8050> 2014-06-09 16:22:12.173773 7f8690f64700 10 op_submit_start 4184 -8049> 2014-06-09 16:22:12.173774 7f8690f64700 5 queue_transactions (trailing journal) 4184 -8048> 2014-06-09 16:22:12.173776 7f8690f64700 5 queue_op 0x324e61e0 seq 4184 osr(0.29 0x2bd62b0) 858 bytes (queue has 50 ops and 4237465 bytes) -8047> 2014-06-09 16:22:12.173780 7f8692767700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8046> 2014-06-09 16:22:12.173784 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8045> 2014-06-09 16:22:12.173788 7f868d75d700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8044> 2014-06-09 16:22:12.173792 7f8691765700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8043> 2014-06-09 16:22:12.173795 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8042> 2014-06-09 16:22:12.173798 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8041> 2014-06-09 16:22:12.173802 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8040> 2014-06-09 16:22:12.173806 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8039> 2014-06-09 16:22:12.173809 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8038> 2014-06-09 16:22:12.173812 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8037> 2014-06-09 16:22:12.173817 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8036> 2014-06-09 16:22:12.173820 7f8692f68700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8035> 2014-06-09 16:22:12.173824 7f868b759700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8034> 2014-06-09 16:22:12.173827 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8033> 2014-06-09 16:22:12.173831 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8032> 2014-06-09 16:22:12.173834 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 -8031> 2014-06-09 16:22:12.173837 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8030> 2014-06-09 16:22:12.173841 7f8690763700 2 waiting 51 > 50 ops || 4477421 > 104857600 -8029> 2014-06-09 16:22:12.173849 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd32a20] -8028> 2014-06-09 16:22:12.173853 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd32a20] -8027> 2014-06-09 16:22:12.173854 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -8026> 2014-06-09 16:22:12.173855 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -8025> 2014-06-09 16:22:12.173858 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f65a0 (1 active) -8024> 2014-06-09 16:22:12.173860 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0ea0 (2 active) -8023> 2014-06-09 16:22:12.173865 7f8690f64700 10 op_submit_finish 4184 -8022> 2014-06-09 16:22:12.173868 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -8021> 2014-06-09 16:22:12.173885 7f8690f64700 15 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -8020> 2014-06-09 16:22:12.173896 7f8690f64700 10 log is not dirty -8019> 2014-06-09 16:22:12.173900 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -8018> 2014-06-09 16:22:12.173912 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -8017> 2014-06-09 16:22:12.173920 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -8016> 2014-06-09 16:22:12.173926 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -8015> 2014-06-09 16:22:12.173934 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -8014> 2014-06-09 16:22:12.173941 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -8013> 2014-06-09 16:22:12.173952 7f8690f64700 15 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 -8012> 2014-06-09 16:22:12.173961 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -8011> 2014-06-09 16:22:12.173968 7f8690f64700 20 osd.26 pg_epoch: 243 pg[1.16d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -8010> 2014-06-09 16:22:12.173976 7f8690f64700 10 log is not dirty -8009> 2014-06-09 16:22:12.174004 7f8690f64700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x15358400 (19 active) -8008> 2014-06-09 16:22:12.174030 7f8690f64700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x15358400 (20 active) -8007> 2014-06-09 16:22:12.174033 7f8690f64700 10 osd.26 pg_epoch: 243 pg[0.2fd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.2fd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) -8006> 2014-06-09 16:22:12.174063 7f8690f64700 10 log is not dirty -8005> 2014-06-09 16:22:12.174062 7f869c77b700 10 osd.26 243 advance to epoch 244 (<= newest 244) -8004> 2014-06-09 16:22:12.174068 7f8690f64700 5 queue_transactions existing osr(0.2fd 0x2bd6e50)/0x2bd6e50 -8003> 2014-06-09 16:22:12.174070 7f8690f64700 2 waiting 51 > 50 ops || 4238293 > 104857600 -8002> 2014-06-09 16:22:12.174070 7f869c77b700 1 -- 172.16.96.202:6850/30904 mark_down 172.16.96.202:6873/24417 -- 0x257c6500 -8001> 2014-06-09 16:22:12.174075 7f86a5e11700 5 _do_op 0x30a3da0 seq 4137 osr(3.537 0x3ed1620)/0x3ed1620 start -8000> 2014-06-09 16:22:12.174079 7f86a5e11700 10 _do_transaction on 0xc1392c0 -7999> 2014-06-09 16:22:12.174084 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7998> 2014-06-09 16:22:12.174103 7f869c77b700 1 -- 172.16.96.202:0/30904 mark_down 0x15a38840 -- 0x19fe5400 -7997> 2014-06-09 16:22:12.174126 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7996> 2014-06-09 16:22:12.174125 7f869c77b700 1 -- 172.16.96.202:0/30904 mark_down 0x15a38b00 -- 0x19fe5180 -7995> 2014-06-09 16:22:12.174141 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4137.0.0 <= header.spos 150443.0.0 -7994> 2014-06-09 16:22:12.174143 7f869c77b700 7 osd.26 244 advance_map epoch 244 -7993> 2014-06-09 16:22:12.174146 7f86a5e11700 5 submit_transaction r = 0 -7992> 2014-06-09 16:22:12.174149 7f869c77b700 10 osd.26 244 write_superblock sb(6336a654-0649-4c42-a04d-d54aefd3bd31 osd.26 d59d5a8a-a977-45a8-b9aa-a778604e89ff e244 [1,244] lci=[242,244]) -7991> 2014-06-09 16:22:12.174158 7f86a5e11700 10 _do_op 0x30a3da0 seq 4137 r = 0, finisher 0x2d739300 0 -7990> 2014-06-09 16:22:12.174166 7f869c77b700 5 queue_transactions existing osr(default 0x2c10280)/0x2c10280 -7989> 2014-06-09 16:22:12.174167 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d6540] -7988> 2014-06-09 16:22:12.174169 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d6540] -7987> 2014-06-09 16:22:12.174169 7f869c77b700 2 waiting 51 > 50 ops || 4259089 > 104857600 -7986> 2014-06-09 16:22:12.174170 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7985> 2014-06-09 16:22:12.174172 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7984> 2014-06-09 16:22:12.174174 7f86a5e11700 10 _finish_op 0x30a3da0 seq 4137 osr(3.537 0x3ed1620)/0x3ed1620 -7983> 2014-06-09 16:22:12.174176 7f869af78700 1 osd.26 244 ms_handle_reset con 0xff678c0 session 0x25e5a000 -7982> 2014-06-09 16:22:12.174183 7f8691f66700 10 op_submit_start 4185 -7981> 2014-06-09 16:22:12.174185 7f8691f66700 5 queue_transactions (trailing journal) 4185 -7980> 2014-06-09 16:22:12.174186 7f8691f66700 5 queue_op 0x160967e0 seq 4185 osr(1.25a 0x384cfb0) 861 bytes (queue has 50 ops and 4237498 bytes) -7979> 2014-06-09 16:22:12.174191 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7978> 2014-06-09 16:22:12.174195 7f8691765700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7977> 2014-06-09 16:22:12.174183 7f8674ab4700 2 -- 172.16.96.202:0/30904 >> 172.16.96.202:6875/24417 pipe(0x19fe5400 sd=96 :37673 s=4 pgs=22 cs=1 l=1 c=0x15a38840).reader couldn't read tag, (0) Success -7976> 2014-06-09 16:22:12.174199 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7975> 2014-06-09 16:22:12.174203 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7974> 2014-06-09 16:22:12.174207 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7973> 2014-06-09 16:22:12.174200 7f8674ab4700 2 -- 172.16.96.202:0/30904 >> 172.16.96.202:6875/24417 pipe(0x19fe5400 sd=96 :37673 s=4 pgs=22 cs=1 l=1 c=0x15a38840).fault (0) Success -7972> 2014-06-09 16:22:12.174211 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7971> 2014-06-09 16:22:12.174214 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7970> 2014-06-09 16:22:12.174218 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7969> 2014-06-09 16:22:12.174223 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7968> 2014-06-09 16:22:12.174227 7f8692f68700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7967> 2014-06-09 16:22:12.174230 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7966> 2014-06-09 16:22:12.174218 7f86735a0700 2 -- 172.16.96.202:6850/30904 >> 172.16.96.202:6873/24417 pipe(0x257c6500 sd=97 :6850 s=4 pgs=27 cs=1 l=0 c=0xff678c0).reader couldn't read tag, (0) Success -7965> 2014-06-09 16:22:12.174233 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7964> 2014-06-09 16:22:12.174237 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7963> 2014-06-09 16:22:12.174232 7f86735a0700 2 -- 172.16.96.202:6850/30904 >> 172.16.96.202:6873/24417 pipe(0x257c6500 sd=97 :6850 s=4 pgs=27 cs=1 l=0 c=0xff678c0).fault (0) Success -7962> 2014-06-09 16:22:12.174240 7f868a757700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7961> 2014-06-09 16:22:12.174244 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7960> 2014-06-09 16:22:12.174248 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7959> 2014-06-09 16:22:12.174251 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7958> 2014-06-09 16:22:12.174256 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 -7957> 2014-06-09 16:22:12.174245 7f86749b3700 2 -- 172.16.96.202:0/30904 >> 172.16.96.202:6876/24417 pipe(0x19fe5180 sd=94 :55234 s=4 pgs=22 cs=1 l=1 c=0x15a38b00).reader couldn't read tag, (0) Success -7956> 2014-06-09 16:22:12.174260 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7955> 2014-06-09 16:22:12.174259 7f86749b3700 2 -- 172.16.96.202:0/30904 >> 172.16.96.202:6876/24417 pipe(0x19fe5180 sd=94 :55234 s=4 pgs=22 cs=1 l=1 c=0x15a38b00).fault (0) Success -7954> 2014-06-09 16:22:12.174270 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d739300] -7953> 2014-06-09 16:22:12.174276 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d739300] -7952> 2014-06-09 16:22:12.174278 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7951> 2014-06-09 16:22:12.174279 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7950> 2014-06-09 16:22:12.174282 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0ea0 (1 active) -7949> 2014-06-09 16:22:12.174285 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc652d80 (2 active) -7948> 2014-06-09 16:22:12.174289 7f8691f66700 10 op_submit_finish 4185 -7947> 2014-06-09 16:22:12.174293 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -7946> 2014-06-09 16:22:12.174308 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.998106 7 0.001551 -7945> 2014-06-09 16:22:12.174320 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -7944> 2014-06-09 16:22:12.174330 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -7943> 2014-06-09 16:22:12.174340 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -7942> 2014-06-09 16:22:12.174341 7f8683620700 1 -- 172.16.96.202:6850/30904 <== osd.6 172.16.96.201:6849/52045 269 ==== osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) v2 ==== 176+0+0 (795240069 0 0) 0x257c6500 con 0xbf5adc0 -7941> 2014-06-09 16:22:12.174350 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -7940> 2014-06-09 16:22:12.174357 7f8683620700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.174323, event: header_read, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) -7939> 2014-06-09 16:22:12.174357 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -7938> 2014-06-09 16:22:12.174367 7f8683620700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.174324, event: throttled, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) -7937> 2014-06-09 16:22:12.174364 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -7936> 2014-06-09 16:22:12.174374 7f8683620700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.174336, event: all_read, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) -7935> 2014-06-09 16:22:12.174381 7f8683620700 5 -- op tracker -- , seq: 3667, time: 0.000000, event: dispatched, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) -7934> 2014-06-09 16:22:12.174374 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) -7933> 2014-06-09 16:22:12.174384 7f8683620700 10 osd.26 244 handle_replica_op osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) v2 epoch 243 -7932> 2014-06-09 16:22:12.174384 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) -7931> 2014-06-09 16:22:12.174389 7f8683620700 20 osd.26 244 should_share_map osd.6 172.16.96.201:6849/52045 243 -7930> 2014-06-09 16:22:12.174394 7f8683620700 10 osd.26 244 osd.6 172.16.96.201:6849/52045 has old map 243 < 244 -7929> 2014-06-09 16:22:12.174392 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -7928> 2014-06-09 16:22:12.174398 7f8683620700 15 osd.26 244 enqueue_op 0x2976d300 prio 196 cost 0 latency 0.000075 osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) v2 -7927> 2014-06-09 16:22:12.174404 7f8691f66700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xc03a840 172.16.96.201:6823/50569 -7926> 2014-06-09 16:22:12.174415 7f8691f66700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6823/50569 -- osd_map(244..244 src has 1..243) v3 -- ?+0 0xbcc38c0 con 0xc03a840 -7925> 2014-06-09 16:22:12.174427 7f8691f66700 10 osd.26 243 note_peer_epoch osd.3 has 244 -7924> 2014-06-09 16:22:12.174430 7f8691f66700 1 -- 172.16.96.202:6850/30904 --> osd.3 172.16.96.201:6823/50569 -- pg_info(1 pgs e243:0.5d4) v4 -- ?+0 0x30de5dc0 -7923> 2014-06-09 16:22:12.174439 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 -7922> 2014-06-09 16:22:12.174474 7f8691f66700 10 log is not dirty -7921> 2014-06-09 16:22:12.174478 7f8691f66700 5 queue_transactions existing osr(0.5d4 0x2bd7a10)/0x2bd7a10 -7920> 2014-06-09 16:22:12.174481 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7919> 2014-06-09 16:22:12.174486 7f86a5e11700 5 _do_op 0x30a2ae0 seq 4138 osr(1.5c1 0x384dd90)/0x384dd90 start -7918> 2014-06-09 16:22:12.174489 7f86a5e11700 10 _do_transaction on 0xbf50f00 -7917> 2014-06-09 16:22:12.174495 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7916> 2014-06-09 16:22:12.174540 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7915> 2014-06-09 16:22:12.174555 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4138.0.0 <= header.spos 150443.0.0 -7914> 2014-06-09 16:22:12.174560 7f86a5e11700 5 submit_transaction r = 0 -7913> 2014-06-09 16:22:12.174573 7f86a5e11700 10 _do_op 0x30a2ae0 seq 4138 r = 0, finisher 0xc410a40 0 -7912> 2014-06-09 16:22:12.174584 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x15a3ed20] -7911> 2014-06-09 16:22:12.174587 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x15a3ed20] -7910> 2014-06-09 16:22:12.174588 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7909> 2014-06-09 16:22:12.174590 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7908> 2014-06-09 16:22:12.174593 7f86a5e11700 10 _finish_op 0x30a2ae0 seq 4138 osr(1.5c1 0x384dd90)/0x384dd90 -7907> 2014-06-09 16:22:12.174604 7f8691765700 10 op_submit_start 4186 -7906> 2014-06-09 16:22:12.174606 7f8691765700 5 queue_transactions (trailing journal) 4186 -7905> 2014-06-09 16:22:12.174607 7f8691765700 5 queue_op 0x1e08a000 seq 4186 osr(0.678 0x2bd7e90) 861 bytes (queue has 50 ops and 4237531 bytes) -7904> 2014-06-09 16:22:12.174613 7f8689f56700 2 waiting 51 > 50 ops || 4238392 > 104857600 -7903> 2014-06-09 16:22:12.174618 7f868ff62700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7902> 2014-06-09 16:22:12.174622 7f868af58700 2 waiting 51 > 50 ops || 4238392 > 104857600 -7901> 2014-06-09 16:22:12.174627 7f868f761700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7900> 2014-06-09 16:22:12.174632 7f868df5e700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7899> 2014-06-09 16:22:12.174636 7f868bf5a700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7898> 2014-06-09 16:22:12.174640 7f868c75b700 2 waiting 51 > 50 ops || 4238392 > 104857600 -7897> 2014-06-09 16:22:12.174644 7f8692f68700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7896> 2014-06-09 16:22:12.174649 7f868b759700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7895> 2014-06-09 16:22:12.174653 7f868e75f700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7894> 2014-06-09 16:22:12.174657 7f868cf5c700 2 waiting 51 > 50 ops || 4238392 > 104857600 -7893> 2014-06-09 16:22:12.174661 7f868a757700 2 waiting 51 > 50 ops || 4238392 > 104857600 -7892> 2014-06-09 16:22:12.174666 7f868ef60700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7891> 2014-06-09 16:22:12.174670 7f8690763700 2 waiting 51 > 50 ops || 4477487 > 104857600 -7890> 2014-06-09 16:22:12.174674 7f8690f64700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7889> 2014-06-09 16:22:12.174678 7f869c77b700 2 waiting 51 > 50 ops || 4259155 > 104857600 -7888> 2014-06-09 16:22:12.174683 7f8692767700 2 waiting 51 > 50 ops || 4238392 > 104857600 -7887> 2014-06-09 16:22:12.174687 7f8691f66700 2 waiting 51 > 50 ops || 4238392 > 104857600 -7886> 2014-06-09 16:22:12.174691 7f868d75d700 2 waiting 51 > 50 ops || 4238392 > 104857600 -7885> 2014-06-09 16:22:12.174694 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 6 ==== osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+1032192 (3561745480 0 0) 0x32da8000 con 0x30c3c2c0 -7884> 2014-06-09 16:22:12.174715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc410a40] -7883> 2014-06-09 16:22:12.174719 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc410a40] -7882> 2014-06-09 16:22:12.174720 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7881> 2014-06-09 16:22:12.174721 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7880> 2014-06-09 16:22:12.174717 7f86a68f6700 5 -- op tracker -- , seq: 3668, time: 2014-06-09 16:22:12.173229, event: header_read, op: osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -7879> 2014-06-09 16:22:12.174724 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc652d80 (1 active) -7878> 2014-06-09 16:22:12.174727 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4077a0 (2 active) -7877> 2014-06-09 16:22:12.174732 7f8691765700 10 op_submit_finish 4186 -7876> 2014-06-09 16:22:12.174729 7f86a68f6700 5 -- op tracker -- , seq: 3668, time: 2014-06-09 16:22:12.173230, event: throttled, op: osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -7875> 2014-06-09 16:22:12.174739 7f86a68f6700 5 -- op tracker -- , seq: 3668, time: 2014-06-09 16:22:12.174681, event: all_read, op: osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -7874> 2014-06-09 16:22:12.174752 7f86a68f6700 5 -- op tracker -- , seq: 3668, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -7873> 2014-06-09 16:22:12.174760 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -7872> 2014-06-09 16:22:12.174765 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -7871> 2014-06-09 16:22:12.174779 7f86a68f6700 15 osd.26 244 enqueue_op 0x1c54ba00 prio 63 cost 1032192 latency 0.001550 osd_op(client.4543.0:33292 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 3096576~1032192] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 -7870> 2014-06-09 16:22:12.174736 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -7869> 2014-06-09 16:22:12.174816 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.083039 7 0.000975 -7868> 2014-06-09 16:22:12.174828 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -7867> 2014-06-09 16:22:12.174839 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -7866> 2014-06-09 16:22:12.174849 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -7865> 2014-06-09 16:22:12.174859 7f8691765700 5 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -7864> 2014-06-09 16:22:12.174868 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -7863> 2014-06-09 16:22:12.174875 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -7862> 2014-06-09 16:22:12.174886 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -7861> 2014-06-09 16:22:12.174893 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 7 ==== osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+65536 (4142700753 0 0) 0x32da9d40 con 0x30c3c2c0 -7860> 2014-06-09 16:22:12.174897 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -7859> 2014-06-09 16:22:12.174907 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -7858> 2014-06-09 16:22:12.174912 7f86a68f6700 5 -- op tracker -- , seq: 3669, time: 2014-06-09 16:22:12.174804, event: header_read, op: osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -7857> 2014-06-09 16:22:12.174920 7f8691765700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbeb9ce0 172.16.96.201:6859/53035 -7856> 2014-06-09 16:22:12.174922 7f86a68f6700 5 -- op tracker -- , seq: 3669, time: 2014-06-09 16:22:12.174805, event: throttled, op: osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -7855> 2014-06-09 16:22:12.174930 7f86a68f6700 5 -- op tracker -- , seq: 3669, time: 2014-06-09 16:22:12.174882, event: all_read, op: osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -7854> 2014-06-09 16:22:12.174930 7f8691765700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6859/53035 -- osd_map(244..244 src has 1..243) v3 -- ?+0 0x2d730480 con 0xbeb9ce0 -7853> 2014-06-09 16:22:12.174938 7f86a68f6700 5 -- op tracker -- , seq: 3669, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) -7852> 2014-06-09 16:22:12.174941 7f8691765700 10 osd.26 243 note_peer_epoch osd.8 has 244 -7851> 2014-06-09 16:22:12.174943 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -7850> 2014-06-09 16:22:12.174947 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -7849> 2014-06-09 16:22:12.174944 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:0.dc) v4 -- ?+0 0x30d9c540 -7848> 2014-06-09 16:22:12.174953 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 -7847> 2014-06-09 16:22:12.174958 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d700 prio 63 cost 65536 latency 0.000154 osd_op(client.4543.0:33293 rbd_data.115f6b8b4567.0000000000001682 [set-alloc-hint object_size 4194304 write_size 4194304,write 4128768~65536] 3.cae2c800 RETRY=2 ack+ondisk+retry+write e243) v4 -7846> 2014-06-09 16:22:12.174988 7f8691765700 10 log is not dirty -7845> 2014-06-09 16:22:12.174992 7f8691765700 5 queue_transactions existing osr(0.dc 0x2bd6570)/0x2bd6570 -7844> 2014-06-09 16:22:12.174995 7f8691765700 2 waiting 51 > 50 ops || 4238389 > 104857600 -7843> 2014-06-09 16:22:12.175034 7f86a5e11700 5 _do_op 0x1e08b440 seq 4139 osr(0.21b 0x2bd6ab0)/0x2bd6ab0 start -7842> 2014-06-09 16:22:12.175040 7f86a5e11700 10 _do_transaction on 0x344ca80 -7841> 2014-06-09 16:22:12.175046 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7840> 2014-06-09 16:22:12.175095 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7839> 2014-06-09 16:22:12.175113 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4139.0.0 <= header.spos 150443.0.0 -7838> 2014-06-09 16:22:12.175118 7f86a5e11700 5 submit_transaction r = 0 -7837> 2014-06-09 16:22:12.175131 7f86a5e11700 10 _do_op 0x1e08b440 seq 4139 r = 0, finisher 0x3050a060 0 -7836> 2014-06-09 16:22:12.175143 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251c240] -7835> 2014-06-09 16:22:12.175146 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -7834> 2014-06-09 16:22:12.175157 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -7833> 2014-06-09 16:22:12.175167 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.21b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 -7832> 2014-06-09 16:22:12.175177 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251c240] -7831> 2014-06-09 16:22:12.175179 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7830> 2014-06-09 16:22:12.175180 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7829> 2014-06-09 16:22:12.175184 7f86a5e11700 10 _finish_op 0x1e08b440 seq 4139 osr(0.21b 0x2bd6ab0)/0x2bd6ab0 -7828> 2014-06-09 16:22:12.175196 7f868ff62700 10 op_submit_start 4187 -7827> 2014-06-09 16:22:12.175197 7f868ff62700 5 queue_transactions (trailing journal) 4187 -7826> 2014-06-09 16:22:12.175199 7f868ff62700 5 queue_op 0x30a3d40 seq 4187 osr(0.31f 0x2bd6d50) 828 bytes (queue has 50 ops and 4237498 bytes) -7825> 2014-06-09 16:22:12.175204 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7824> 2014-06-09 16:22:12.175209 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7823> 2014-06-09 16:22:12.175213 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7822> 2014-06-09 16:22:12.175218 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7821> 2014-06-09 16:22:12.175223 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7820> 2014-06-09 16:22:12.175214 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 8 ==== osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+122880 (2351009256 0 0) 0xcbe6900 con 0x30c3c2c0 -7819> 2014-06-09 16:22:12.175227 7f8692f68700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7818> 2014-06-09 16:22:12.175231 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7817> 2014-06-09 16:22:12.175235 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7816> 2014-06-09 16:22:12.175236 7f86a68f6700 5 -- op tracker -- , seq: 3670, time: 2014-06-09 16:22:12.174972, event: header_read, op: osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7815> 2014-06-09 16:22:12.175247 7f86a68f6700 5 -- op tracker -- , seq: 3670, time: 2014-06-09 16:22:12.174973, event: throttled, op: osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7814> 2014-06-09 16:22:12.175257 7f86a68f6700 5 -- op tracker -- , seq: 3670, time: 2014-06-09 16:22:12.175203, event: all_read, op: osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7813> 2014-06-09 16:22:12.175289 7f86a68f6700 5 -- op tracker -- , seq: 3670, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7812> 2014-06-09 16:22:12.175294 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -7811> 2014-06-09 16:22:12.175298 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -7810> 2014-06-09 16:22:12.175309 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d600 prio 63 cost 122880 latency 0.000337 osd_op(client.4543.0:33343 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 0~122880] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 -7809> 2014-06-09 16:22:12.175341 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7808> 2014-06-09 16:22:12.175347 7f868a757700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7807> 2014-06-09 16:22:12.175351 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7806> 2014-06-09 16:22:12.175356 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7805> 2014-06-09 16:22:12.175360 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7804> 2014-06-09 16:22:12.175364 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 -7803> 2014-06-09 16:22:12.175368 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7802> 2014-06-09 16:22:12.175373 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7801> 2014-06-09 16:22:12.175377 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7800> 2014-06-09 16:22:12.175381 7f8691765700 2 waiting 51 > 50 ops || 4238356 > 104857600 -7799> 2014-06-09 16:22:12.175386 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7798> 2014-06-09 16:22:12.175397 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a060] -7797> 2014-06-09 16:22:12.175401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a060] -7796> 2014-06-09 16:22:12.175403 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7795> 2014-06-09 16:22:12.175404 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7794> 2014-06-09 16:22:12.175407 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4077a0 (1 active) -7793> 2014-06-09 16:22:12.175410 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ade60 (2 active) -7792> 2014-06-09 16:22:12.175415 7f868ff62700 10 op_submit_finish 4187 -7791> 2014-06-09 16:22:12.175420 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.206( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.206( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) -7790> 2014-06-09 16:22:12.175457 7f868ff62700 10 log is not dirty -7789> 2014-06-09 16:22:12.175463 7f868ff62700 5 queue_transactions existing osr(1.206 0x384cd10)/0x384cd10 -7788> 2014-06-09 16:22:12.175466 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7787> 2014-06-09 16:22:12.175470 7f86a5e11700 5 _do_op 0x30a2540 seq 4140 osr(1.348 0x384d310)/0x384d310 start -7786> 2014-06-09 16:22:12.175474 7f86a5e11700 10 _do_transaction on 0xc257200 -7785> 2014-06-09 16:22:12.175480 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7784> 2014-06-09 16:22:12.175524 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7783> 2014-06-09 16:22:12.175539 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4140.0.0 <= header.spos 150443.0.0 -7782> 2014-06-09 16:22:12.175543 7f86a5e11700 5 submit_transaction r = 0 -7781> 2014-06-09 16:22:12.175563 7f86a5e11700 10 _do_op 0x30a2540 seq 4140 r = 0, finisher 0xbef14e0 0 -7780> 2014-06-09 16:22:12.175574 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc2d6220] -7779> 2014-06-09 16:22:12.175577 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc2d6220] -7778> 2014-06-09 16:22:12.175578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7777> 2014-06-09 16:22:12.175580 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7776> 2014-06-09 16:22:12.175583 7f86a5e11700 10 _finish_op 0x30a2540 seq 4140 osr(1.348 0x384d310)/0x384d310 -7775> 2014-06-09 16:22:12.175594 7f868f761700 10 op_submit_start 4188 -7774> 2014-06-09 16:22:12.175596 7f868f761700 5 queue_transactions (trailing journal) 4188 -7773> 2014-06-09 16:22:12.175598 7f868f761700 5 queue_op 0x2c06eba0 seq 4188 osr(1.4fb 0x384d7f0) 828 bytes (queue has 50 ops and 4237498 bytes) -7772> 2014-06-09 16:22:12.175603 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7771> 2014-06-09 16:22:12.175608 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7770> 2014-06-09 16:22:12.175612 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7769> 2014-06-09 16:22:12.175637 7f8692f68700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7768> 2014-06-09 16:22:12.175641 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7767> 2014-06-09 16:22:12.175646 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7766> 2014-06-09 16:22:12.175651 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7765> 2014-06-09 16:22:12.175655 7f868a757700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7764> 2014-06-09 16:22:12.175659 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7763> 2014-06-09 16:22:12.175664 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7762> 2014-06-09 16:22:12.175668 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7761> 2014-06-09 16:22:12.175673 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 -7760> 2014-06-09 16:22:12.175677 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7759> 2014-06-09 16:22:12.175682 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7758> 2014-06-09 16:22:12.175686 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7757> 2014-06-09 16:22:12.175690 7f8691765700 2 waiting 51 > 50 ops || 4238356 > 104857600 -7756> 2014-06-09 16:22:12.175695 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7755> 2014-06-09 16:22:12.175699 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7754> 2014-06-09 16:22:12.175704 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7753> 2014-06-09 16:22:12.175715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef14e0] -7752> 2014-06-09 16:22:12.175719 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef14e0] -7751> 2014-06-09 16:22:12.175721 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7750> 2014-06-09 16:22:12.175722 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7749> 2014-06-09 16:22:12.175725 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ade60 (1 active) -7748> 2014-06-09 16:22:12.175728 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd680 (2 active) -7747> 2014-06-09 16:22:12.175733 7f868f761700 10 op_submit_finish 4188 -7746> 2014-06-09 16:22:12.175737 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.18e( v 213'606 (0'0,213'606] local-les=243 n=114 ec=118 les/c 243/243 242/242/222) -7745> 2014-06-09 16:22:12.175774 7f868f761700 10 log is not dirty -7744> 2014-06-09 16:22:12.175780 7f868f761700 5 queue_transactions existing osr(3.18e 0x3ed0af0)/0x3ed0af0 -7743> 2014-06-09 16:22:12.175783 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7742> 2014-06-09 16:22:12.175788 7f86a5e11700 5 _do_op 0x2fcaea20 seq 4141 osr(1.2ef 0x384d0f0)/0x384d0f0 start -7741> 2014-06-09 16:22:12.175792 7f86a5e11700 10 _do_transaction on 0xbf2e840 -7740> 2014-06-09 16:22:12.175797 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7739> 2014-06-09 16:22:12.175841 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7738> 2014-06-09 16:22:12.175856 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4141.0.0 <= header.spos 150443.0.0 -7737> 2014-06-09 16:22:12.175860 7f86a5e11700 5 submit_transaction r = 0 -7736> 2014-06-09 16:22:12.175871 7f86a5e11700 10 _do_op 0x2fcaea20 seq 4141 r = 0, finisher 0xbd7ce00 0 -7735> 2014-06-09 16:22:12.175882 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94480] -7734> 2014-06-09 16:22:12.175888 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94480] -7733> 2014-06-09 16:22:12.175889 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7732> 2014-06-09 16:22:12.175890 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7731> 2014-06-09 16:22:12.175894 7f86a5e11700 10 _finish_op 0x2fcaea20 seq 4141 osr(1.2ef 0x384d0f0)/0x384d0f0 -7730> 2014-06-09 16:22:12.175905 7f868bf5a700 10 op_submit_start 4189 -7729> 2014-06-09 16:22:12.175907 7f868bf5a700 5 queue_transactions (trailing journal) 4189 -7728> 2014-06-09 16:22:12.175908 7f868bf5a700 5 queue_op 0x324db080 seq 4189 osr(1.14c 0x384c810) 828 bytes (queue has 50 ops and 4237498 bytes) -7727> 2014-06-09 16:22:12.175913 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7726> 2014-06-09 16:22:12.175918 7f8692f68700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7725> 2014-06-09 16:22:12.175922 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7724> 2014-06-09 16:22:12.175927 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7723> 2014-06-09 16:22:12.175931 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7722> 2014-06-09 16:22:12.175935 7f868a757700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7721> 2014-06-09 16:22:12.175940 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7720> 2014-06-09 16:22:12.175944 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7719> 2014-06-09 16:22:12.175949 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7718> 2014-06-09 16:22:12.175953 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 -7717> 2014-06-09 16:22:12.175958 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7716> 2014-06-09 16:22:12.175962 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7715> 2014-06-09 16:22:12.175966 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7714> 2014-06-09 16:22:12.175971 7f8691765700 2 waiting 51 > 50 ops || 4238356 > 104857600 -7713> 2014-06-09 16:22:12.175975 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7712> 2014-06-09 16:22:12.175979 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7711> 2014-06-09 16:22:12.175984 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7710> 2014-06-09 16:22:12.175988 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7709> 2014-06-09 16:22:12.175993 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7708> 2014-06-09 16:22:12.176002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7ce00] -7707> 2014-06-09 16:22:12.176006 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7ce00] -7706> 2014-06-09 16:22:12.176008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7705> 2014-06-09 16:22:12.176009 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7704> 2014-06-09 16:22:12.176012 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd680 (1 active) -7703> 2014-06-09 16:22:12.176015 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e120 (2 active) -7702> 2014-06-09 16:22:12.176020 7f868bf5a700 10 op_submit_finish 4189 -7701> 2014-06-09 16:22:12.176024 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.3fe( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -7700> 2014-06-09 16:22:12.176058 7f868bf5a700 10 log is not dirty -7699> 2014-06-09 16:22:12.176064 7f868bf5a700 5 queue_transactions existing osr(0.3fe 0x2bd7210)/0x2bd7210 -7698> 2014-06-09 16:22:12.176066 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7697> 2014-06-09 16:22:12.176071 7f86a5e11700 5 _do_op 0xc2d4d20 seq 4142 osr(0.44f 0x2bd71a0)/0x2bd71a0 start -7696> 2014-06-09 16:22:12.176075 7f86a5e11700 10 _do_transaction on 0xbf0fd40 -7695> 2014-06-09 16:22:12.176081 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7694> 2014-06-09 16:22:12.176124 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7693> 2014-06-09 16:22:12.176141 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4142.0.0 <= header.spos 150443.0.0 -7692> 2014-06-09 16:22:12.176146 7f86a5e11700 5 submit_transaction r = 0 -7691> 2014-06-09 16:22:12.176158 7f86a5e11700 10 _do_op 0xc2d4d20 seq 4142 r = 0, finisher 0x318c2b60 0 -7690> 2014-06-09 16:22:12.176168 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314785a0] -7689> 2014-06-09 16:22:12.176171 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -7688> 2014-06-09 16:22:12.176181 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -7687> 2014-06-09 16:22:12.176190 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.44f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 -7686> 2014-06-09 16:22:12.176199 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314785a0] -7685> 2014-06-09 16:22:12.176201 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7684> 2014-06-09 16:22:12.176202 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7683> 2014-06-09 16:22:12.176206 7f86a5e11700 10 _finish_op 0xc2d4d20 seq 4142 osr(0.44f 0x2bd71a0)/0x2bd71a0 -7682> 2014-06-09 16:22:12.176216 7f8692f68700 10 op_submit_start 4190 -7681> 2014-06-09 16:22:12.176217 7f8692f68700 5 queue_transactions (trailing journal) 4190 -7680> 2014-06-09 16:22:12.176219 7f8692f68700 5 queue_op 0xd3274a0 seq 4190 osr(1.375 0x384d230) 828 bytes (queue has 50 ops and 4237465 bytes) -7679> 2014-06-09 16:22:12.176223 7f868b759700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7678> 2014-06-09 16:22:12.176227 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7677> 2014-06-09 16:22:12.176231 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7676> 2014-06-09 16:22:12.176234 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7675> 2014-06-09 16:22:12.176237 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7674> 2014-06-09 16:22:12.176240 7f8690763700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7673> 2014-06-09 16:22:12.176243 7f8690f64700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7672> 2014-06-09 16:22:12.176246 7f869c77b700 2 waiting 51 > 50 ops || 4259089 > 104857600 -7671> 2014-06-09 16:22:12.176250 7f8692767700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7670> 2014-06-09 16:22:12.176255 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7669> 2014-06-09 16:22:12.176258 7f868d75d700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7668> 2014-06-09 16:22:12.176261 7f8691765700 2 waiting 51 > 50 ops || 4238323 > 104857600 -7667> 2014-06-09 16:22:12.176264 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7666> 2014-06-09 16:22:12.176267 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7665> 2014-06-09 16:22:12.176271 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7664> 2014-06-09 16:22:12.176275 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7663> 2014-06-09 16:22:12.176279 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7662> 2014-06-09 16:22:12.176282 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7661> 2014-06-09 16:22:12.176285 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7660> 2014-06-09 16:22:12.176295 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x318c2b60] -7659> 2014-06-09 16:22:12.176304 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x318c2b60] -7658> 2014-06-09 16:22:12.176306 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7657> 2014-06-09 16:22:12.176307 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7656> 2014-06-09 16:22:12.176310 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e120 (1 active) -7655> 2014-06-09 16:22:12.176313 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1b00 (2 active) -7654> 2014-06-09 16:22:12.176317 7f8692f68700 10 op_submit_finish 4190 -7653> 2014-06-09 16:22:12.176321 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) -7652> 2014-06-09 16:22:12.176360 7f8692f68700 10 log is not dirty -7651> 2014-06-09 16:22:12.176364 7f8692f68700 5 queue_transactions existing osr(1.a 0x384c270)/0x384c270 -7650> 2014-06-09 16:22:12.176367 7f8692f68700 2 waiting 51 > 50 ops || 4238289 > 104857600 -7649> 2014-06-09 16:22:12.176371 7f86a5e11700 5 _do_op 0x327fd800 seq 4143 osr(0.66a 0x2bd7ef0)/0x2bd7ef0 start -7648> 2014-06-09 16:22:12.176375 7f86a5e11700 10 _do_transaction on 0xbebd8c0 -7647> 2014-06-09 16:22:12.176380 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7646> 2014-06-09 16:22:12.176422 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7645> 2014-06-09 16:22:12.176437 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4143.0.0 <= header.spos 150443.0.0 -7644> 2014-06-09 16:22:12.176442 7f86a5e11700 5 submit_transaction r = 0 -7643> 2014-06-09 16:22:12.176453 7f86a5e11700 10 _do_op 0x327fd800 seq 4143 r = 0, finisher 0x32073640 0 -7642> 2014-06-09 16:22:12.176464 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a6900] -7641> 2014-06-09 16:22:12.176466 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a6900] -7640> 2014-06-09 16:22:12.176468 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7639> 2014-06-09 16:22:12.176469 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7638> 2014-06-09 16:22:12.176472 7f86a5e11700 10 _finish_op 0x327fd800 seq 4143 osr(0.66a 0x2bd7ef0)/0x2bd7ef0 -7637> 2014-06-09 16:22:12.176483 7f868e75f700 10 op_submit_start 4191 -7636> 2014-06-09 16:22:12.176485 7f868e75f700 5 queue_transactions (trailing journal) 4191 -7635> 2014-06-09 16:22:12.176487 7f868e75f700 5 queue_op 0xd327ce0 seq 4191 osr(1.24c 0x384cfd0) 828 bytes (queue has 50 ops and 4237465 bytes) -7634> 2014-06-09 16:22:12.176492 7f868cf5c700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7633> 2014-06-09 16:22:12.176497 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7632> 2014-06-09 16:22:12.176501 7f868ef60700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7631> 2014-06-09 16:22:12.176505 7f8690763700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7630> 2014-06-09 16:22:12.176510 7f8690f64700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7629> 2014-06-09 16:22:12.176514 7f869c77b700 2 waiting 51 > 50 ops || 4259089 > 104857600 -7628> 2014-06-09 16:22:12.176518 7f8692767700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7627> 2014-06-09 16:22:12.176522 7f8691f66700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7626> 2014-06-09 16:22:12.176527 7f868d75d700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7625> 2014-06-09 16:22:12.176531 7f8691765700 2 waiting 51 > 50 ops || 4238323 > 104857600 -7624> 2014-06-09 16:22:12.176535 7f8689f56700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7623> 2014-06-09 16:22:12.176539 7f868ff62700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7622> 2014-06-09 16:22:12.176544 7f868af58700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7621> 2014-06-09 16:22:12.176548 7f868f761700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7620> 2014-06-09 16:22:12.176552 7f868df5e700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7619> 2014-06-09 16:22:12.176556 7f868bf5a700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7618> 2014-06-09 16:22:12.176561 7f868c75b700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7617> 2014-06-09 16:22:12.176565 7f8692f68700 2 waiting 51 > 50 ops || 4238289 > 104857600 -7616> 2014-06-09 16:22:12.176569 7f868b759700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7615> 2014-06-09 16:22:12.176579 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32073640] -7614> 2014-06-09 16:22:12.176585 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32073640] -7613> 2014-06-09 16:22:12.176587 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7612> 2014-06-09 16:22:12.176588 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7611> 2014-06-09 16:22:12.176591 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1b00 (1 active) -7610> 2014-06-09 16:22:12.176594 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3017a0 (2 active) -7609> 2014-06-09 16:22:12.176599 7f868e75f700 10 op_submit_finish 4191 -7608> 2014-06-09 16:22:12.176603 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.3b7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.3b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) -7607> 2014-06-09 16:22:12.176642 7f868e75f700 10 log is not dirty -7606> 2014-06-09 16:22:12.176647 7f868e75f700 5 queue_transactions existing osr(0.3b7 0x2bd7290)/0x2bd7290 -7605> 2014-06-09 16:22:12.176649 7f868e75f700 2 waiting 51 > 50 ops || 4238293 > 104857600 -7604> 2014-06-09 16:22:12.176654 7f86a5e11700 5 _do_op 0x327fc420 seq 4144 osr(0.654 0x2bd7b50)/0x2bd7b50 start -7603> 2014-06-09 16:22:12.176657 7f86a5e11700 10 _do_transaction on 0xc22b080 -7602> 2014-06-09 16:22:12.176729 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7601> 2014-06-09 16:22:12.176771 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7600> 2014-06-09 16:22:12.176786 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4144.0.0 <= header.spos 150443.0.0 -7599> 2014-06-09 16:22:12.176791 7f86a5e11700 5 submit_transaction r = 0 -7598> 2014-06-09 16:22:12.176801 7f86a5e11700 10 _do_op 0x327fc420 seq 4144 r = 0, finisher 0x307f1a40 0 -7597> 2014-06-09 16:22:12.176812 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e19600] -7596> 2014-06-09 16:22:12.176815 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e19600] -7595> 2014-06-09 16:22:12.176816 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7594> 2014-06-09 16:22:12.176817 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7593> 2014-06-09 16:22:12.176820 7f86a5e11700 10 _finish_op 0x327fc420 seq 4144 osr(0.654 0x2bd7b50)/0x2bd7b50 -7592> 2014-06-09 16:22:12.176831 7f868a757700 10 op_submit_start 4192 -7591> 2014-06-09 16:22:12.176833 7f868a757700 5 queue_transactions (trailing journal) 4192 -7590> 2014-06-09 16:22:12.176835 7f868a757700 5 queue_op 0x16097020 seq 4192 osr(3.362 0x3ed10a0) 861 bytes (queue has 50 ops and 4237498 bytes) -7589> 2014-06-09 16:22:12.176840 7f868ef60700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7588> 2014-06-09 16:22:12.176844 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7587> 2014-06-09 16:22:12.176849 7f8690f64700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7586> 2014-06-09 16:22:12.176853 7f869c77b700 2 waiting 51 > 50 ops || 4259122 > 104857600 -7585> 2014-06-09 16:22:12.176857 7f8692767700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7584> 2014-06-09 16:22:12.176862 7f8691f66700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7583> 2014-06-09 16:22:12.176866 7f868d75d700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7582> 2014-06-09 16:22:12.176870 7f8691765700 2 waiting 51 > 50 ops || 4238356 > 104857600 -7581> 2014-06-09 16:22:12.176874 7f8689f56700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7580> 2014-06-09 16:22:12.176879 7f868ff62700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7579> 2014-06-09 16:22:12.176883 7f868af58700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7578> 2014-06-09 16:22:12.176888 7f868f761700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7577> 2014-06-09 16:22:12.176892 7f868df5e700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7576> 2014-06-09 16:22:12.176896 7f868bf5a700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7575> 2014-06-09 16:22:12.176900 7f868c75b700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7574> 2014-06-09 16:22:12.176904 7f8692f68700 2 waiting 51 > 50 ops || 4238322 > 104857600 -7573> 2014-06-09 16:22:12.176909 7f868b759700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7572> 2014-06-09 16:22:12.176913 7f868e75f700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7571> 2014-06-09 16:22:12.176917 7f868cf5c700 2 waiting 51 > 50 ops || 4238359 > 104857600 -7570> 2014-06-09 16:22:12.176927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f1a40] -7569> 2014-06-09 16:22:12.176931 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f1a40] -7568> 2014-06-09 16:22:12.176932 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7567> 2014-06-09 16:22:12.176933 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7566> 2014-06-09 16:22:12.176936 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3017a0 (1 active) -7565> 2014-06-09 16:22:12.176939 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406c60 (2 active) -7564> 2014-06-09 16:22:12.176944 7f868a757700 10 op_submit_finish 4192 -7563> 2014-06-09 16:22:12.176948 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 243/208 242/242/207) -7562> 2014-06-09 16:22:12.176985 7f868a757700 10 log is not dirty -7561> 2014-06-09 16:22:12.176990 7f868a757700 5 queue_transactions existing osr(3.470 0x3ed1540)/0x3ed1540 -7560> 2014-06-09 16:22:12.176992 7f868a757700 2 waiting 51 > 50 ops || 4238326 > 104857600 -7559> 2014-06-09 16:22:12.176997 7f86a5e11700 5 _do_op 0xc2d5aa0 seq 4145 osr(2.3fa 0x2ed12f0)/0x2ed12f0 start -7558> 2014-06-09 16:22:12.177001 7f86a5e11700 10 _do_transaction on 0x33380c0 -7557> 2014-06-09 16:22:12.177007 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7556> 2014-06-09 16:22:12.176993 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 9 ==== osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+1032192 (3545258279 0 0) 0xcbe66c0 con 0x30c3c2c0 -7555> 2014-06-09 16:22:12.177017 7f86a68f6700 5 -- op tracker -- , seq: 3671, time: 2014-06-09 16:22:12.175327, event: header_read, op: osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7554> 2014-06-09 16:22:12.177029 7f86a68f6700 5 -- op tracker -- , seq: 3671, time: 2014-06-09 16:22:12.175328, event: throttled, op: osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7553> 2014-06-09 16:22:12.177039 7f86a68f6700 5 -- op tracker -- , seq: 3671, time: 2014-06-09 16:22:12.176978, event: all_read, op: osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7552> 2014-06-09 16:22:12.177048 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7551> 2014-06-09 16:22:12.177049 7f86a68f6700 5 -- op tracker -- , seq: 3671, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7550> 2014-06-09 16:22:12.177057 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -7549> 2014-06-09 16:22:12.177061 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -7548> 2014-06-09 16:22:12.177065 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4145.0.0 <= header.spos 150443.0.0 -7547> 2014-06-09 16:22:12.177070 7f86a5e11700 5 submit_transaction r = 0 -7546> 2014-06-09 16:22:12.177077 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d500 prio 63 cost 1032192 latency 0.001750 osd_op(client.4543.0:33344 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 122880~1032192] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 -7545> 2014-06-09 16:22:12.177103 7f86a5e11700 10 _do_op 0xc2d5aa0 seq 4145 r = 0, finisher 0xbd71e40 0 -7544> 2014-06-09 16:22:12.177114 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc411020] -7543> 2014-06-09 16:22:12.177117 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -7542> 2014-06-09 16:22:12.177128 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -7541> 2014-06-09 16:22:12.177137 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 -7540> 2014-06-09 16:22:12.177148 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc411020] -7539> 2014-06-09 16:22:12.177150 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7538> 2014-06-09 16:22:12.177151 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7537> 2014-06-09 16:22:12.177155 7f86a5e11700 10 _finish_op 0xc2d5aa0 seq 4145 osr(2.3fa 0x2ed12f0)/0x2ed12f0 -7536> 2014-06-09 16:22:12.177166 7f8690763700 10 op_submit_start 4193 -7535> 2014-06-09 16:22:12.177168 7f8690763700 5 queue_transactions (trailing journal) 4193 -7534> 2014-06-09 16:22:12.177170 7f8690763700 5 queue_op 0x3280b1a0 seq 4193 osr(3.7f 0x3ed0200) 239956 bytes (queue has 50 ops and 4476593 bytes) -7533> 2014-06-09 16:22:12.177175 7f8690f64700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7532> 2014-06-09 16:22:12.177180 7f869c77b700 2 waiting 51 > 50 ops || 4498217 > 104857600 -7531> 2014-06-09 16:22:12.177197 7f8692767700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7530> 2014-06-09 16:22:12.177202 7f8691f66700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7529> 2014-06-09 16:22:12.177206 7f868d75d700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7528> 2014-06-09 16:22:12.177211 7f8691765700 2 waiting 51 > 50 ops || 4477451 > 104857600 -7527> 2014-06-09 16:22:12.177215 7f8689f56700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7526> 2014-06-09 16:22:12.177220 7f868ff62700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7525> 2014-06-09 16:22:12.177224 7f868af58700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7524> 2014-06-09 16:22:12.177229 7f868f761700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7523> 2014-06-09 16:22:12.177233 7f868df5e700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7522> 2014-06-09 16:22:12.177238 7f868bf5a700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7521> 2014-06-09 16:22:12.177242 7f868c75b700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7520> 2014-06-09 16:22:12.177247 7f8692f68700 2 waiting 51 > 50 ops || 4477417 > 104857600 -7519> 2014-06-09 16:22:12.177251 7f868b759700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7518> 2014-06-09 16:22:12.177255 7f868e75f700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7517> 2014-06-09 16:22:12.177260 7f868cf5c700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7516> 2014-06-09 16:22:12.177264 7f868a757700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7515> 2014-06-09 16:22:12.177269 7f868ef60700 2 waiting 51 > 50 ops || 4477421 > 104857600 -7514> 2014-06-09 16:22:12.177279 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd71e40] -7513> 2014-06-09 16:22:12.177284 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd71e40] -7512> 2014-06-09 16:22:12.177285 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7511> 2014-06-09 16:22:12.177287 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7510> 2014-06-09 16:22:12.177290 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406c60 (1 active) -7509> 2014-06-09 16:22:12.177292 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52dd40 (2 active) -7508> 2014-06-09 16:22:12.177297 7f8690763700 10 op_submit_finish 4193 -7507> 2014-06-09 16:22:12.177303 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc1fd740 243'534 rep_tid=29 committed?=0 applied?=0 lock=0 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad -7506> 2014-06-09 16:22:12.177331 7f8690763700 10 osd.26 244 dequeue_op 0x2e075000 finish -7505> 2014-06-09 16:22:12.177336 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -7504> 2014-06-09 16:22:12.177343 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -7503> 2014-06-09 16:22:12.177348 7f8690763700 10 osd.26 244 dequeue_op 0x2976ce00 prio 196 cost 0 latency 0.003726 osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) v2 pg pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] -7502> 2014-06-09 16:22:12.177366 7f8690763700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.177363, event: reached_pg, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) -7501> 2014-06-09 16:22:12.177375 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2976ce00 -7500> 2014-06-09 16:22:12.177391 7f8690763700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.177387, event: started, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) -7499> 2014-06-09 16:22:12.177397 7f8690763700 7 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 29 op ack_type 1 from 6 -7498> 2014-06-09 16:22:12.177415 7f8690763700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.177409, event: sub_op_applied_rec, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -7497> 2014-06-09 16:22:12.177423 7f8690763700 10 osd.26 244 dequeue_op 0x2976ce00 finish -7496> 2014-06-09 16:22:12.177430 7f8690763700 5 -- op tracker -- , seq: 3666, time: 2014-06-09 16:22:12.177427, event: done, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ack, result = 0) -7495> 2014-06-09 16:22:12.177443 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -7494> 2014-06-09 16:22:12.177478 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41810 (20 active) -7493> 2014-06-09 16:22:12.177483 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7492> 2014-06-09 16:22:12.177500 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.712( v 213'629 (0'0,213'629] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -7491> 2014-06-09 16:22:12.177513 7f8690763700 10 log is not dirty -7490> 2014-06-09 16:22:12.177518 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7489> 2014-06-09 16:22:12.177530 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -7488> 2014-06-09 16:22:12.177539 7f8690763700 10 log is not dirty -7487> 2014-06-09 16:22:12.177543 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7486> 2014-06-09 16:22:12.177555 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -7485> 2014-06-09 16:22:12.177564 7f8690763700 10 log is not dirty -7484> 2014-06-09 16:22:12.177569 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7483> 2014-06-09 16:22:12.177581 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.774( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -7482> 2014-06-09 16:22:12.177591 7f8690763700 10 log is not dirty -7481> 2014-06-09 16:22:12.177595 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7480> 2014-06-09 16:22:12.177607 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops -7479> 2014-06-09 16:22:12.177616 7f8690763700 10 log is not dirty -7478> 2014-06-09 16:22:12.177620 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7477> 2014-06-09 16:22:12.177632 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -7476> 2014-06-09 16:22:12.177642 7f8690763700 10 log is not dirty -7475> 2014-06-09 16:22:12.177646 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7474> 2014-06-09 16:22:12.177660 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -7473> 2014-06-09 16:22:12.177671 7f8690763700 10 log is not dirty -7472> 2014-06-09 16:22:12.177669 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 10 ==== osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+516096 (3452789998 0 0) 0xcbe6480 con 0x30c3c2c0 -7471> 2014-06-09 16:22:12.177675 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7470> 2014-06-09 16:22:12.177692 7f86a68f6700 5 -- op tracker -- , seq: 3672, time: 2014-06-09 16:22:12.177099, event: header_read, op: osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7469> 2014-06-09 16:22:12.177689 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -7468> 2014-06-09 16:22:12.177700 7f8690763700 10 log is not dirty -7467> 2014-06-09 16:22:12.177703 7f86a68f6700 5 -- op tracker -- , seq: 3672, time: 2014-06-09 16:22:12.177100, event: throttled, op: osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7466> 2014-06-09 16:22:12.177704 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7465> 2014-06-09 16:22:12.177716 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -7464> 2014-06-09 16:22:12.177726 7f8690763700 10 log is not dirty -7463> 2014-06-09 16:22:12.177730 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7462> 2014-06-09 16:22:12.177737 7f86a68f6700 5 -- op tracker -- , seq: 3672, time: 2014-06-09 16:22:12.177658, event: all_read, op: osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7461> 2014-06-09 16:22:12.177748 7f86a68f6700 5 -- op tracker -- , seq: 3672, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -7460> 2014-06-09 16:22:12.177743 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.1eb( v 213'563 (0'0,213'563] local-les=243 n=114 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -7459> 2014-06-09 16:22:12.177755 7f8690763700 10 log is not dirty -7458> 2014-06-09 16:22:12.177753 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -7457> 2014-06-09 16:22:12.177758 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -7456> 2014-06-09 16:22:12.177758 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7455> 2014-06-09 16:22:12.177770 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d400 prio 63 cost 516096 latency 0.000670 osd_op(client.4543.0:33345 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1155072~516096] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 -7454> 2014-06-09 16:22:12.177773 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] requeue_ops 0x32827d00 -7453> 2014-06-09 16:22:12.177789 7f8690763700 10 log is not dirty -7452> 2014-06-09 16:22:12.177794 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7451> 2014-06-09 16:22:12.177810 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.1aa( v 214'499 (0'0,214'499] local-les=243 n=98 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x1804c300,0x1c66be00,0x1804c200,0x1c66bd00,0x1804c100,0x1c66bc00,0x1804c000,0x1c66bb00,0x21966400,0x21966300,0x19fe1d00,0x19fe1c00 -7450> 2014-06-09 16:22:12.177852 7f8690763700 10 log is not dirty -7449> 2014-06-09 16:22:12.177857 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7448> 2014-06-09 16:22:12.177871 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0xe032700,0xe2ac100,0xe033f00,0xe033e00,0x11e62300,0x4678300,0x4678200,0x4678100,0x4678000,0x2e074900,0x2e074800 -7447> 2014-06-09 16:22:12.177910 7f8690763700 10 log is not dirty -7446> 2014-06-09 16:22:12.177915 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7445> 2014-06-09 16:22:12.177929 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -7444> 2014-06-09 16:22:12.177940 7f8690763700 10 log is not dirty -7443> 2014-06-09 16:22:12.177944 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -7442> 2014-06-09 16:22:12.177957 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.555( v 213'588 (0'0,213'588] local-les=243 n=120 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -7441> 2014-06-09 16:22:12.177968 7f8690763700 10 log is not dirty -7440> 2014-06-09 16:22:12.177972 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -7439> 2014-06-09 16:22:12.177986 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.002198 7 0.000447 -7438> 2014-06-09 16:22:12.177999 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -7437> 2014-06-09 16:22:12.178011 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -7436> 2014-06-09 16:22:12.178022 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 -7435> 2014-06-09 16:22:12.178033 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -7434> 2014-06-09 16:22:12.178042 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -7433> 2014-06-09 16:22:12.178049 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -7432> 2014-06-09 16:22:12.178061 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -7431> 2014-06-09 16:22:12.178072 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -7430> 2014-06-09 16:22:12.178082 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -7429> 2014-06-09 16:22:12.178097 7f8690763700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -7428> 2014-06-09 16:22:12.178100 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:0.164) v4 -- ?+0 0x3343da40 -7427> 2014-06-09 16:22:12.178116 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:185 -7426> 2014-06-09 16:22:12.178150 7f8690763700 10 log is not dirty -7425> 2014-06-09 16:22:12.178154 7f8690763700 5 queue_transactions existing osr(0.164 0x2bd6810)/0x2bd6810 -7424> 2014-06-09 16:22:12.178157 7f8690763700 2 waiting 51 > 50 ops || 4477454 > 104857600 -7423> 2014-06-09 16:22:12.178187 7f86a5e11700 5 _do_op 0x327fc060 seq 4146 osr(3.f0 0x3ed04a0)/0x3ed04a0 start -7422> 2014-06-09 16:22:12.178197 7f86a5e11700 10 _do_transaction on 0xbd8f740 -7421> 2014-06-09 16:22:12.178203 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7420> 2014-06-09 16:22:12.178257 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7419> 2014-06-09 16:22:12.178273 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4146.0.0 <= header.spos 150443.0.0 -7418> 2014-06-09 16:22:12.178278 7f86a5e11700 5 submit_transaction r = 0 -7417> 2014-06-09 16:22:12.178292 7f86a5e11700 10 _do_op 0x327fc060 seq 4146 r = 0, finisher 0x307f0bc0 0 -7416> 2014-06-09 16:22:12.178304 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062bac0] -7415> 2014-06-09 16:22:12.178307 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062bac0] -7414> 2014-06-09 16:22:12.178308 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7413> 2014-06-09 16:22:12.178309 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7412> 2014-06-09 16:22:12.178313 7f86a5e11700 10 _finish_op 0x327fc060 seq 4146 osr(3.f0 0x3ed04a0)/0x3ed04a0 -7411> 2014-06-09 16:22:12.178324 7f869c77b700 10 op_submit_start 4194 -7410> 2014-06-09 16:22:12.178326 7f869c77b700 5 queue_transactions (trailing journal) 4194 -7409> 2014-06-09 16:22:12.178328 7f869c77b700 5 queue_op 0xc34a840 seq 4194 osr(default 0x2c10280) 21624 bytes (queue has 50 ops and 4497391 bytes) -7408> 2014-06-09 16:22:12.178333 7f8692767700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7407> 2014-06-09 16:22:12.178338 7f8691f66700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7406> 2014-06-09 16:22:12.178343 7f868d75d700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7405> 2014-06-09 16:22:12.178348 7f8691765700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7404> 2014-06-09 16:22:12.178353 7f8689f56700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7403> 2014-06-09 16:22:12.178357 7f868ff62700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7402> 2014-06-09 16:22:12.178362 7f868af58700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7401> 2014-06-09 16:22:12.178367 7f868f761700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7400> 2014-06-09 16:22:12.178371 7f868df5e700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7399> 2014-06-09 16:22:12.178376 7f868bf5a700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7398> 2014-06-09 16:22:12.178380 7f868c75b700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7397> 2014-06-09 16:22:12.178385 7f8692f68700 2 waiting 51 > 50 ops || 4498215 > 104857600 -7396> 2014-06-09 16:22:12.178389 7f868b759700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7395> 2014-06-09 16:22:12.178394 7f868e75f700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7394> 2014-06-09 16:22:12.178398 7f868cf5c700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7393> 2014-06-09 16:22:12.178403 7f868a757700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7392> 2014-06-09 16:22:12.178407 7f868ef60700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7391> 2014-06-09 16:22:12.178411 7f8690763700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7390> 2014-06-09 16:22:12.178416 7f8690f64700 2 waiting 51 > 50 ops || 4498219 > 104857600 -7389> 2014-06-09 16:22:12.178425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f0bc0] -7388> 2014-06-09 16:22:12.178429 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f0bc0] -7387> 2014-06-09 16:22:12.178431 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7386> 2014-06-09 16:22:12.178432 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7385> 2014-06-09 16:22:12.178435 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52dd40 (1 active) -7384> 2014-06-09 16:22:12.178438 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f200 (2 active) -7383> 2014-06-09 16:22:12.178443 7f869c77b700 10 op_submit_finish 4194 -7382> 2014-06-09 16:22:12.178451 7f869c77b700 7 osd.26 244 consume_map version 244 -7381> 2014-06-09 16:22:12.178486 7f86a5e11700 5 _do_op 0x327fd7a0 seq 4147 osr(2.112 0x2ed06d0)/0x2ed06d0 start -7380> 2014-06-09 16:22:12.178490 7f86a5e11700 10 _do_transaction on 0xc2ef800 -7379> 2014-06-09 16:22:12.178496 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7378> 2014-06-09 16:22:12.178539 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7377> 2014-06-09 16:22:12.178554 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4147.0.0 <= header.spos 150443.0.0 -7376> 2014-06-09 16:22:12.178558 7f86a5e11700 5 submit_transaction r = 0 -7375> 2014-06-09 16:22:12.178571 7f86a5e11700 10 _do_op 0x327fd7a0 seq 4147 r = 0, finisher 0xcc82760 0 -7374> 2014-06-09 16:22:12.178648 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f0c80] -7373> 2014-06-09 16:22:12.178651 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f0c80] -7372> 2014-06-09 16:22:12.178653 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7371> 2014-06-09 16:22:12.178654 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7370> 2014-06-09 16:22:12.178658 7f86a5e11700 10 _finish_op 0x327fd7a0 seq 4147 osr(2.112 0x2ed06d0)/0x2ed06d0 -7369> 2014-06-09 16:22:12.178669 7f8691f66700 10 op_submit_start 4195 -7368> 2014-06-09 16:22:12.178671 7f8691f66700 5 queue_transactions (trailing journal) 4195 -7367> 2014-06-09 16:22:12.178673 7f8691f66700 5 queue_op 0x30060cc0 seq 4195 osr(0.5d4 0x2bd7a10) 861 bytes (queue has 50 ops and 4497424 bytes) -7366> 2014-06-09 16:22:12.178678 7f868d75d700 2 waiting 51 > 50 ops || 4498285 > 104857600 -7365> 2014-06-09 16:22:12.178682 7f8691765700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7364> 2014-06-09 16:22:12.178687 7f8689f56700 2 waiting 51 > 50 ops || 4498285 > 104857600 -7363> 2014-06-09 16:22:12.178692 7f868ff62700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7362> 2014-06-09 16:22:12.178696 7f868af58700 2 waiting 51 > 50 ops || 4498285 > 104857600 -7361> 2014-06-09 16:22:12.178700 7f868f761700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7360> 2014-06-09 16:22:12.178704 7f868df5e700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7359> 2014-06-09 16:22:12.178709 7f868bf5a700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7358> 2014-06-09 16:22:12.178713 7f868c75b700 2 waiting 51 > 50 ops || 4498285 > 104857600 -7357> 2014-06-09 16:22:12.178717 7f8692f68700 2 waiting 51 > 50 ops || 4498248 > 104857600 -7356> 2014-06-09 16:22:12.178722 7f868b759700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7355> 2014-06-09 16:22:12.178726 7f868e75f700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7354> 2014-06-09 16:22:12.178730 7f868cf5c700 2 waiting 51 > 50 ops || 4498285 > 104857600 -7353> 2014-06-09 16:22:12.178735 7f868a757700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7352> 2014-06-09 16:22:12.178739 7f868ef60700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7351> 2014-06-09 16:22:12.178743 7f8690763700 2 waiting 51 > 50 ops || 4498285 > 104857600 -7350> 2014-06-09 16:22:12.178748 7f8690f64700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7349> 2014-06-09 16:22:12.178753 7f8692767700 2 waiting 51 > 50 ops || 4498285 > 104857600 -7348> 2014-06-09 16:22:12.178763 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc82760] -7347> 2014-06-09 16:22:12.178770 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc82760] -7346> 2014-06-09 16:22:12.178772 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7345> 2014-06-09 16:22:12.178773 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7344> 2014-06-09 16:22:12.178776 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f200 (1 active) -7343> 2014-06-09 16:22:12.178779 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b7a0 (2 active) -7342> 2014-06-09 16:22:12.178784 7f8691f66700 10 op_submit_finish 4195 -7341> 2014-06-09 16:22:12.178808 7f8691f66700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xb7f0220 (19 active) -7340> 2014-06-09 16:22:12.178816 7f8691f66700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -7339> 2014-06-09 16:22:12.178820 7f8691f66700 10 osd.26 244 dequeue_op 0x2976d300 prio 196 cost 0 latency 0.004497 osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) v2 pg pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] -7338> 2014-06-09 16:22:12.178836 7f8691f66700 20 osd.26 244 share_map osd.6 172.16.96.201:6849/52045 243 -7337> 2014-06-09 16:22:12.178841 7f8691f66700 20 osd.26 244 should_share_map osd.6 172.16.96.201:6849/52045 243 -7336> 2014-06-09 16:22:12.178846 7f8691f66700 10 osd.26 244 osd.6 172.16.96.201:6849/52045 has old map 243 < 244 -7335> 2014-06-09 16:22:12.178850 7f8691f66700 10 osd.26 244 osd.6 172.16.96.201:6849/52045 has old map 243 < 244 -7334> 2014-06-09 16:22:12.178854 7f8691f66700 10 osd.26 243 note_peer_epoch osd.6 has 244 -7333> 2014-06-09 16:22:12.178856 7f8691f66700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbf5adc0 172.16.96.201:6849/52045 -7332> 2014-06-09 16:22:12.178865 7f8691f66700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6849/52045 -- osd_map(244..244 src has 1..244) v3 -- ?+0 0xbcc3680 con 0xbf5adc0 -7331> 2014-06-09 16:22:12.178882 7f8691f66700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.178877, event: reached_pg, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) -7330> 2014-06-09 16:22:12.178889 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x2976d300 -7329> 2014-06-09 16:22:12.178904 7f8691f66700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.178901, event: started, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) -7328> 2014-06-09 16:22:12.178910 7f8691f66700 7 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] sub_op_modify_reply: tid 29 op ack_type 4 from 6 -7327> 2014-06-09 16:22:12.178930 7f8691f66700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.178923, event: sub_op_commit_rec, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -7326> 2014-06-09 16:22:12.178936 7f8691f66700 10 osd.26 244 dequeue_op 0x2976d300 finish -7325> 2014-06-09 16:22:12.178943 7f8691f66700 5 -- op tracker -- , seq: 3667, time: 2014-06-09 16:22:12.178940, event: done, op: osd_sub_op_reply(client.4565.0:33578 3.7f 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 [] ondisk, result = 0) -7324> 2014-06-09 16:22:12.178961 7f8691f66700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -7323> 2014-06-09 16:22:12.178985 7f8691f66700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xb7f0220 (20 active) -7322> 2014-06-09 16:22:12.178989 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -7321> 2014-06-09 16:22:12.179004 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] exit Started/Primary/Active/Activating 1.015908 7 0.000432 -7320> 2014-06-09 16:22:12.179017 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] enter Started/Primary/Active/Recovered -7319> 2014-06-09 16:22:12.179028 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] needs_recovery is recovered -7318> 2014-06-09 16:22:12.179040 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -7317> 2014-06-09 16:22:12.179052 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] enter Started/Primary/Active/Clean -7316> 2014-06-09 16:22:12.179061 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] finish_recovery -7315> 2014-06-09 16:22:12.179070 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active] clear_recovery_state -7314> 2014-06-09 16:22:12.179082 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'636 mlcod 214'636 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -7313> 2014-06-09 16:22:12.179094 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'636 mlcod 214'636 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -7312> 2014-06-09 16:22:12.179105 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 214'636 mlcod 214'636 active+clean] share_pg_info -7311> 2014-06-09 16:22:12.179120 7f8691f66700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbeb8f20 172.16.96.201:6865/53518 -7310> 2014-06-09 16:22:12.179128 7f8691f66700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6865/53518 -- osd_map(244..244 src has 1..244) v3 -- ?+0 0x32da9b00 con 0xbeb8f20 -7309> 2014-06-09 16:22:12.179140 7f8691f66700 10 osd.26 243 note_peer_epoch osd.9 has 244 -7308> 2014-06-09 16:22:12.179143 7f8691f66700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:3.2b3) v4 -- ?+0 0x30d9ec40 -7307> 2014-06-09 16:22:12.179154 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.2b3( v 243'637 (0'0,243'637] local-les=243 n=124 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 214'636 mlcod 214'636 active+clean] publish_stats_to_osd 243:1062 -7306> 2014-06-09 16:22:12.179183 7f8689755700 10 op_submit_start 4196 -7305> 2014-06-09 16:22:12.179187 7f8689755700 5 queue_transactions (trailing journal) 4196 -7304> 2014-06-09 16:22:12.179190 7f8691f66700 10 log is not dirty -7303> 2014-06-09 16:22:12.179189 7f8689755700 5 queue_op 0x324a5da0 seq 4196 osr(3.375 0x3ed1060) 828 bytes (queue has 50 ops and 4497424 bytes) -7302> 2014-06-09 16:22:12.179193 7f8689755700 10 op_submit_finish 4196 -7301> 2014-06-09 16:22:12.179194 7f8691f66700 5 queue_transactions existing osr(3.2b3 0x3ed0f00)/0x3ed0f00 -7300> 2014-06-09 16:22:12.179197 7f8691f66700 2 waiting 51 > 50 ops || 4498285 > 104857600 -7299> 2014-06-09 16:22:12.178947 7f86a5e11700 5 _do_op 0x327fc540 seq 4148 osr(0.765 0x384c360)/0x384c360 start -7298> 2014-06-09 16:22:12.179206 7f86a5e11700 10 _do_transaction on 0x399c780 -7297> 2014-06-09 16:22:12.179211 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7296> 2014-06-09 16:22:12.179198 7f8689755700 10 osd.26 pg_epoch: 243 pg[0.1d4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.1d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) -7295> 2014-06-09 16:22:12.179243 7f8689755700 10 log is not dirty -7294> 2014-06-09 16:22:12.179249 7f8689755700 5 queue_transactions existing osr(0.1d4 0x2bd6940)/0x2bd6940 -7293> 2014-06-09 16:22:12.179252 7f8689755700 2 waiting 51 > 50 ops || 4498252 > 104857600 -7292> 2014-06-09 16:22:12.179256 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7291> 2014-06-09 16:22:12.179271 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4148.0.0 <= header.spos 150443.0.0 -7290> 2014-06-09 16:22:12.179275 7f86a5e11700 5 submit_transaction r = 0 -7289> 2014-06-09 16:22:12.179287 7f86a5e11700 10 _do_op 0x327fc540 seq 4148 r = 0, finisher 0x31456a80 0 -7288> 2014-06-09 16:22:12.179298 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f0940] -7287> 2014-06-09 16:22:12.179301 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f0940] -7286> 2014-06-09 16:22:12.179302 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7285> 2014-06-09 16:22:12.179303 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7284> 2014-06-09 16:22:12.179307 7f86a5e11700 10 _finish_op 0x327fc540 seq 4148 osr(0.765 0x384c360)/0x384c360 -7283> 2014-06-09 16:22:12.179318 7f8691765700 10 op_submit_start 4197 -7282> 2014-06-09 16:22:12.179320 7f8691765700 5 queue_transactions (trailing journal) 4197 -7281> 2014-06-09 16:22:12.179322 7f8691765700 5 queue_op 0xc64a480 seq 4197 osr(0.dc 0x2bd6570) 858 bytes (queue has 50 ops and 4497454 bytes) -7280> 2014-06-09 16:22:12.179327 7f8689f56700 2 waiting 51 > 50 ops || 4498315 > 104857600 -7279> 2014-06-09 16:22:12.179332 7f868ff62700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7278> 2014-06-09 16:22:12.179336 7f868af58700 2 waiting 51 > 50 ops || 4498315 > 104857600 -7277> 2014-06-09 16:22:12.179340 7f868f761700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7276> 2014-06-09 16:22:12.179345 7f868df5e700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7275> 2014-06-09 16:22:12.179349 7f868bf5a700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7274> 2014-06-09 16:22:12.179354 7f868c75b700 2 waiting 51 > 50 ops || 4498315 > 104857600 -7273> 2014-06-09 16:22:12.179358 7f8692f68700 2 waiting 51 > 50 ops || 4498278 > 104857600 -7272> 2014-06-09 16:22:12.179363 7f868b759700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7271> 2014-06-09 16:22:12.179367 7f868e75f700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7270> 2014-06-09 16:22:12.179372 7f868cf5c700 2 waiting 51 > 50 ops || 4498315 > 104857600 -7269> 2014-06-09 16:22:12.179376 7f868a757700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7268> 2014-06-09 16:22:12.179381 7f868ef60700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7267> 2014-06-09 16:22:12.179385 7f8690763700 2 waiting 51 > 50 ops || 4498315 > 104857600 -7266> 2014-06-09 16:22:12.179389 7f8690f64700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7265> 2014-06-09 16:22:12.179393 7f8692767700 2 waiting 51 > 50 ops || 4498315 > 104857600 -7264> 2014-06-09 16:22:12.179398 7f8691f66700 2 waiting 51 > 50 ops || 4498315 > 104857600 -7263> 2014-06-09 16:22:12.179409 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31456a80] -7262> 2014-06-09 16:22:12.179413 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31456a80] -7261> 2014-06-09 16:22:12.179414 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7260> 2014-06-09 16:22:12.179415 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7259> 2014-06-09 16:22:12.179418 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b7a0 (1 active) -7258> 2014-06-09 16:22:12.179421 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa240 (2 active) -7257> 2014-06-09 16:22:12.179426 7f8691765700 10 op_submit_finish 4197 -7256> 2014-06-09 16:22:12.179430 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -7255> 2014-06-09 16:22:12.179444 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.111301 7 0.000311 -7254> 2014-06-09 16:22:12.179456 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -7253> 2014-06-09 16:22:12.179466 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -7252> 2014-06-09 16:22:12.179477 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -7251> 2014-06-09 16:22:12.179487 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -7250> 2014-06-09 16:22:12.179496 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -7249> 2014-06-09 16:22:12.179503 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -7248> 2014-06-09 16:22:12.179514 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -7247> 2014-06-09 16:22:12.179525 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -7246> 2014-06-09 16:22:12.179535 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -7245> 2014-06-09 16:22:12.179551 7f8691765700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 -7244> 2014-06-09 16:22:12.179554 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:1.b3) v4 -- ?+0 0x30d9c380 -7243> 2014-06-09 16:22:12.179567 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:171 -7242> 2014-06-09 16:22:12.179598 7f8691765700 10 log is not dirty -7241> 2014-06-09 16:22:12.179602 7f8691765700 5 queue_transactions existing osr(1.b3 0x384c6d0)/0x384c6d0 -7240> 2014-06-09 16:22:12.179605 7f8691765700 2 waiting 51 > 50 ops || 4498312 > 104857600 -7239> 2014-06-09 16:22:12.179610 7f86a5e11700 5 _do_op 0xc2d4000 seq 4149 osr(2.3bd 0x2ed1010)/0x2ed1010 start -7238> 2014-06-09 16:22:12.179633 7f86a5e11700 10 _do_transaction on 0xbf27740 -7237> 2014-06-09 16:22:12.179633 7f8689755700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7236> 2014-06-09 16:22:12.179641 7f868d75d700 2 waiting 51 > 50 ops || 4498315 > 104857600 -7235> 2014-06-09 16:22:12.179646 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7234> 2014-06-09 16:22:12.179702 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7233> 2014-06-09 16:22:12.179721 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4149.0.0 <= header.spos 150443.0.0 -7232> 2014-06-09 16:22:12.179725 7f86a5e11700 5 submit_transaction r = 0 -7231> 2014-06-09 16:22:12.179762 7f86a5e11700 10 _do_op 0xc2d4000 seq 4149 r = 0, finisher 0xc1685a0 0 -7230> 2014-06-09 16:22:12.179774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7c640] -7229> 2014-06-09 16:22:12.179777 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -7228> 2014-06-09 16:22:12.179788 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -7227> 2014-06-09 16:22:12.179797 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3bd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 -7226> 2014-06-09 16:22:12.179806 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7c640] -7225> 2014-06-09 16:22:12.179808 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7224> 2014-06-09 16:22:12.179809 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7223> 2014-06-09 16:22:12.179813 7f86a5e11700 10 _finish_op 0xc2d4000 seq 4149 osr(2.3bd 0x2ed1010)/0x2ed1010 -7222> 2014-06-09 16:22:12.179825 7f868ff62700 10 op_submit_start 4198 -7221> 2014-06-09 16:22:12.179827 7f868ff62700 5 queue_transactions (trailing journal) 4198 -7220> 2014-06-09 16:22:12.179828 7f868ff62700 5 queue_op 0x30a38c0 seq 4198 osr(1.206 0x384cd10) 828 bytes (queue has 50 ops and 4497421 bytes) -7219> 2014-06-09 16:22:12.179833 7f868af58700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7218> 2014-06-09 16:22:12.179838 7f868f761700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7217> 2014-06-09 16:22:12.179842 7f868df5e700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7216> 2014-06-09 16:22:12.179847 7f868bf5a700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7215> 2014-06-09 16:22:12.179851 7f868c75b700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7214> 2014-06-09 16:22:12.179856 7f8692f68700 2 waiting 51 > 50 ops || 4498245 > 104857600 -7213> 2014-06-09 16:22:12.179860 7f868b759700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7212> 2014-06-09 16:22:12.179864 7f868e75f700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7211> 2014-06-09 16:22:12.179868 7f868cf5c700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7210> 2014-06-09 16:22:12.179873 7f868a757700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7209> 2014-06-09 16:22:12.179877 7f868ef60700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7208> 2014-06-09 16:22:12.179881 7f8690763700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7207> 2014-06-09 16:22:12.179885 7f8690f64700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7206> 2014-06-09 16:22:12.179889 7f8692767700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7205> 2014-06-09 16:22:12.179894 7f8691f66700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7204> 2014-06-09 16:22:12.179898 7f8691765700 2 waiting 51 > 50 ops || 4498279 > 104857600 -7203> 2014-06-09 16:22:12.179909 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc1685a0] -7202> 2014-06-09 16:22:12.179914 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc1685a0] -7201> 2014-06-09 16:22:12.179915 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7200> 2014-06-09 16:22:12.179917 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7199> 2014-06-09 16:22:12.179920 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa240 (1 active) -7198> 2014-06-09 16:22:12.179922 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524480 (2 active) -7197> 2014-06-09 16:22:12.179928 7f868ff62700 10 op_submit_finish 4198 -7196> 2014-06-09 16:22:12.179947 7f868ff62700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e130 (19 active) -7195> 2014-06-09 16:22:12.179954 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -7194> 2014-06-09 16:22:12.179958 7f868ff62700 10 osd.26 244 dequeue_op 0x14053a00 prio 63 cost 16384 latency 1.915519 osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] -7193> 2014-06-09 16:22:12.179984 7f868ff62700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.179978, event: reached_pg, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) -7192> 2014-06-09 16:22:12.179992 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -7191> 2014-06-09 16:22:12.180006 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x14053a00 -7190> 2014-06-09 16:22:12.180016 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write -7189> 2014-06-09 16:22:12.180040 7f868ff62700 15 getattr 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 '_' -7188> 2014-06-09 16:22:12.180116 7f868ff62700 10 getattr got.size() is 0 -7187> 2014-06-09 16:22:12.180120 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 but can_create -7186> 2014-06-09 16:22:12.180134 7f868ff62700 15 getattr 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 'snapset' -7185> 2014-06-09 16:22:12.180165 7f868ff62700 10 getattr got.size() is 0 -7184> 2014-06-09 16:22:12.180168 7f868ff62700 15 getattr 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/snapdir//3 'snapset' -7183> 2014-06-09 16:22:12.180198 7f868ff62700 10 getattr got.size() is 0 -7182> 2014-06-09 16:22:12.180207 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30bd3600 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -7181> 2014-06-09 16:22:12.180219 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -7180> 2014-06-09 16:22:12.180229 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -7179> 2014-06-09 16:22:12.180239 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30bd3600 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 rwstate(none n=0 w=0) oi: 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc2380 snapset: 0=[]:[] -7178> 2014-06-09 16:22:12.180255 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 @head oi=3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) -7177> 2014-06-09 16:22:12.180274 7f868ff62700 15 getattr 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/snapdir//3 '_' -7176> 2014-06-09 16:22:12.180308 7f868ff62700 10 getattr got.size() is 0 -7175> 2014-06-09 16:22:12.180310 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/snapdir//3 and !can_create -7174> 2014-06-09 16:22:12.180330 7f868ff62700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.180324, event: started, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) -7173> 2014-06-09 16:22:12.180340 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x240f2000 -7172> 2014-06-09 16:22:12.180368 7f868ff62700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.180353, event: started, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) -7171> 2014-06-09 16:22:12.180376 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] ov 0'0 av 243'664 snapc 0=[] snapset 0=[]:[] -7170> 2014-06-09 16:22:12.180391 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] -7169> 2014-06-09 16:22:12.180402 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 -7168> 2014-06-09 16:22:12.182122 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~16384 -7167> 2014-06-09 16:22:12.182154 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229200 -7166> 2014-06-09 16:22:12.182166 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 snapset=0x14dc23b8 snapc=0=[] -7165> 2014-06-09 16:22:12.182178 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag -7164> 2014-06-09 16:22:12.182189 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 done, snapset=0=[]:[]+head -7163> 2014-06-09 16:22:12.182199 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 0x240f2000 op modify -7162> 2014-06-09 16:22:12.182212 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:10.931984 -7161> 2014-06-09 16:22:12.182234 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=133 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -7160> 2014-06-09 16:22:12.182253 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 -7159> 2014-06-09 16:22:12.182264 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 30 on osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) v4 -7158> 2014-06-09 16:22:12.182280 7f868ff62700 7 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 30 o 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -7157> 2014-06-09 16:22:12.182305 7f868ff62700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.182300, event: waiting for subops from 4, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) -7156> 2014-06-09 16:22:12.182326 7f868ff62700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbfb5e40 172.16.96.201:6832/51134 -7155> 2014-06-09 16:22:12.182337 7f868ff62700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6832/51134 -- osd_map(244..244 src has 1..244) v3 -- ?+0 0xbcc3d40 con 0xbfb5e40 -7154> 2014-06-09 16:22:12.182349 7f868ff62700 10 osd.26 243 note_peer_epoch osd.4 has 244 -7153> 2014-06-09 16:22:12.182352 7f868ff62700 1 -- 172.16.96.202:6850/30904 --> osd.4 172.16.96.201:6832/51134 -- osd_sub_op(client.4566.0:33893 3.2ac 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 [] v 243'664 snapset=0=[]:[] snapc=0=[]) v10 -- ?+17296 0xe066200 -7152> 2014-06-09 16:22:12.182365 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 213'663 (0'0,213'663] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,213'663], crt=0'0) [243'664 (0'0) modify 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 by client.4566.0:33893 2014-06-09 10:19:10.931984] -7151> 2014-06-09 16:22:12.182387 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'664 (0'0) modify 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 by client.4566.0:33893 2014-06-09 10:19:10.931984 -7150> 2014-06-09 16:22:12.182472 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys -7149> 2014-06-09 16:22:12.182506 7f868ff62700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'664, trimmed: -7148> 2014-06-09 16:22:12.182527 7f868ff62700 5 queue_transactions existing osr(3.2ac 0x3ed0b50)/0x3ed0b50 -7147> 2014-06-09 16:22:12.180415 7f86a5e11700 5 _do_op 0xc2d43c0 seq 4150 osr(0.3f9 0x2bd7230)/0x2bd7230 start -7146> 2014-06-09 16:22:12.180436 7f8689755700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7145> 2014-06-09 16:22:12.182542 7f868d75d700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7144> 2014-06-09 16:22:12.182547 7f8689f56700 2 waiting 51 > 50 ops || 4498282 > 104857600 -7143> 2014-06-09 16:22:12.182552 7f868ff62700 2 waiting 51 > 50 ops || 4516203 > 104857600 -7142> 2014-06-09 16:22:12.182557 7f86a5e11700 10 _do_transaction on 0xc161200 -7141> 2014-06-09 16:22:12.182565 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7140> 2014-06-09 16:22:12.182619 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7139> 2014-06-09 16:22:12.182638 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4150.0.0 <= header.spos 150443.0.0 -7138> 2014-06-09 16:22:12.182643 7f86a5e11700 5 submit_transaction r = 0 -7137> 2014-06-09 16:22:12.182657 7f86a5e11700 10 _do_op 0xc2d43c0 seq 4150 r = 0, finisher 0xcc94d40 0 -7136> 2014-06-09 16:22:12.182669 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1b0c1e60] -7135> 2014-06-09 16:22:12.182672 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -7134> 2014-06-09 16:22:12.182683 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -7133> 2014-06-09 16:22:12.182692 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.3f9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 -7132> 2014-06-09 16:22:12.182702 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1b0c1e60] -7131> 2014-06-09 16:22:12.182704 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7130> 2014-06-09 16:22:12.182705 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7129> 2014-06-09 16:22:12.182709 7f86a5e11700 10 _finish_op 0xc2d43c0 seq 4150 osr(0.3f9 0x2bd7230)/0x2bd7230 -7128> 2014-06-09 16:22:12.182722 7f868f761700 10 op_submit_start 4199 -7127> 2014-06-09 16:22:12.182724 7f868f761700 5 queue_transactions (trailing journal) 4199 -7126> 2014-06-09 16:22:12.182725 7f868f761700 5 queue_op 0xbdc82a0 seq 4199 osr(3.18e 0x3ed0af0) 828 bytes (queue has 50 ops and 4497388 bytes) -7125> 2014-06-09 16:22:12.182731 7f868df5e700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7124> 2014-06-09 16:22:12.182736 7f868bf5a700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7123> 2014-06-09 16:22:12.182741 7f868c75b700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7122> 2014-06-09 16:22:12.182746 7f8692f68700 2 waiting 51 > 50 ops || 4498212 > 104857600 -7121> 2014-06-09 16:22:12.182750 7f868b759700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7120> 2014-06-09 16:22:12.182755 7f868e75f700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7119> 2014-06-09 16:22:12.182759 7f868cf5c700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7118> 2014-06-09 16:22:12.182764 7f868a757700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7117> 2014-06-09 16:22:12.182768 7f868ef60700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7116> 2014-06-09 16:22:12.182773 7f8690763700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7115> 2014-06-09 16:22:12.182778 7f8690f64700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7114> 2014-06-09 16:22:12.182783 7f8692767700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7113> 2014-06-09 16:22:12.182787 7f8691f66700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7112> 2014-06-09 16:22:12.182792 7f8691765700 2 waiting 51 > 50 ops || 4498246 > 104857600 -7111> 2014-06-09 16:22:12.182796 7f8689755700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7110> 2014-06-09 16:22:12.182801 7f868d75d700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7109> 2014-06-09 16:22:12.182805 7f8689f56700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7108> 2014-06-09 16:22:12.182809 7f868ff62700 2 waiting 51 > 50 ops || 4516170 > 104857600 -7107> 2014-06-09 16:22:12.182813 7f868af58700 2 waiting 51 > 50 ops || 4498249 > 104857600 -7106> 2014-06-09 16:22:12.182825 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc94d40] -7105> 2014-06-09 16:22:12.182829 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc94d40] -7104> 2014-06-09 16:22:12.182831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7103> 2014-06-09 16:22:12.182832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7102> 2014-06-09 16:22:12.182835 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524480 (1 active) -7101> 2014-06-09 16:22:12.182838 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115c20 (2 active) -7100> 2014-06-09 16:22:12.182843 7f868f761700 10 op_submit_finish 4199 -7099> 2014-06-09 16:22:12.182848 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.ab( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.ab( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) -7098> 2014-06-09 16:22:12.182889 7f868f761700 10 log is not dirty -7097> 2014-06-09 16:22:12.182894 7f868f761700 5 queue_transactions existing osr(1.ab 0x384c6f0)/0x384c6f0 -7096> 2014-06-09 16:22:12.182897 7f868f761700 2 waiting 51 > 50 ops || 4498214 > 104857600 -7095> 2014-06-09 16:22:12.182902 7f86a5e11700 5 _do_op 0xc2d46c0 seq 4151 osr(1.6b7 0x2ed01d0)/0x2ed01d0 start -7094> 2014-06-09 16:22:12.182906 7f86a5e11700 10 _do_transaction on 0x331f980 -7093> 2014-06-09 16:22:12.182912 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7092> 2014-06-09 16:22:12.182955 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7091> 2014-06-09 16:22:12.182972 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4151.0.0 <= header.spos 150443.0.0 -7090> 2014-06-09 16:22:12.182976 7f86a5e11700 5 submit_transaction r = 0 -7089> 2014-06-09 16:22:12.182995 7f86a5e11700 10 _do_op 0xc2d46c0 seq 4151 r = 0, finisher 0x13964780 0 -7088> 2014-06-09 16:22:12.183006 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6c9e0] -7087> 2014-06-09 16:22:12.183009 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -7086> 2014-06-09 16:22:12.183019 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -7085> 2014-06-09 16:22:12.183029 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.6b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 -7084> 2014-06-09 16:22:12.183038 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6c9e0] -7083> 2014-06-09 16:22:12.183040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7082> 2014-06-09 16:22:12.183041 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7081> 2014-06-09 16:22:12.183044 7f86a5e11700 10 _finish_op 0xc2d46c0 seq 4151 osr(1.6b7 0x2ed01d0)/0x2ed01d0 -7080> 2014-06-09 16:22:12.183056 7f868bf5a700 10 op_submit_start 4200 -7079> 2014-06-09 16:22:12.183057 7f868bf5a700 5 queue_transactions (trailing journal) 4200 -7078> 2014-06-09 16:22:12.183059 7f868bf5a700 5 queue_op 0x324db380 seq 4200 osr(0.3fe 0x2bd7210) 828 bytes (queue has 50 ops and 4497355 bytes) -7077> 2014-06-09 16:22:12.183064 7f868c75b700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7076> 2014-06-09 16:22:12.183069 7f8692f68700 2 waiting 51 > 50 ops || 4498179 > 104857600 -7075> 2014-06-09 16:22:12.183073 7f868b759700 2 waiting 51 > 50 ops || 4498183 > 104857600 -7074> 2014-06-09 16:22:12.183077 7f868e75f700 2 waiting 51 > 50 ops || 4498183 > 104857600 -7073> 2014-06-09 16:22:12.183082 7f868cf5c700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7072> 2014-06-09 16:22:12.183086 7f868a757700 2 waiting 51 > 50 ops || 4498183 > 104857600 -7071> 2014-06-09 16:22:12.183091 7f868ef60700 2 waiting 51 > 50 ops || 4498183 > 104857600 -7070> 2014-06-09 16:22:12.183095 7f8690763700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7069> 2014-06-09 16:22:12.183099 7f8690f64700 2 waiting 51 > 50 ops || 4498183 > 104857600 -7068> 2014-06-09 16:22:12.183104 7f8692767700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7067> 2014-06-09 16:22:12.183108 7f8691f66700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7066> 2014-06-09 16:22:12.183112 7f8691765700 2 waiting 51 > 50 ops || 4498213 > 104857600 -7065> 2014-06-09 16:22:12.183117 7f8689755700 2 waiting 51 > 50 ops || 4498183 > 104857600 -7064> 2014-06-09 16:22:12.183121 7f868d75d700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7063> 2014-06-09 16:22:12.183125 7f8689f56700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7062> 2014-06-09 16:22:12.183129 7f868ff62700 2 waiting 51 > 50 ops || 4516137 > 104857600 -7061> 2014-06-09 16:22:12.183133 7f868af58700 2 waiting 51 > 50 ops || 4498216 > 104857600 -7060> 2014-06-09 16:22:12.183138 7f868f761700 2 waiting 51 > 50 ops || 4498181 > 104857600 -7059> 2014-06-09 16:22:12.183142 7f868df5e700 2 waiting 51 > 50 ops || 4498183 > 104857600 -7058> 2014-06-09 16:22:12.183152 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13964780] -7057> 2014-06-09 16:22:12.183158 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13964780] -7056> 2014-06-09 16:22:12.183160 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -7055> 2014-06-09 16:22:12.183161 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -7054> 2014-06-09 16:22:12.183164 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115c20 (1 active) -7053> 2014-06-09 16:22:12.183167 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89ad80 (2 active) -7052> 2014-06-09 16:22:12.183172 7f868bf5a700 10 op_submit_finish 4200 -7051> 2014-06-09 16:22:12.183176 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -7050> 2014-06-09 16:22:12.183191 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local -7049> 2014-06-09 16:22:12.183200 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -7048> 2014-06-09 16:22:12.183209 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -7047> 2014-06-09 16:22:12.183218 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -7046> 2014-06-09 16:22:12.183227 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -7045> 2014-06-09 16:22:12.183242 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:151 -7044> 2014-06-09 16:22:12.183251 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -7043> 2014-06-09 16:22:12.183259 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[0.451( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop -7042> 2014-06-09 16:22:12.183269 7f868bf5a700 10 log is not dirty -7041> 2014-06-09 16:22:12.183275 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.527( v 213'671 (0'0,213'671] local-les=243 n=125 ec=118 les/c 243/243 242/242/202) -7040> 2014-06-09 16:22:12.183308 7f868bf5a700 10 log is not dirty -7039> 2014-06-09 16:22:12.183313 7f868bf5a700 5 queue_transactions existing osr(3.527 0x3ed1660)/0x3ed1660 -7038> 2014-06-09 16:22:12.183315 7f868bf5a700 2 waiting 51 > 50 ops || 4498183 > 104857600 -7037> 2014-06-09 16:22:12.183320 7f86a5e11700 5 _do_op 0x324e6780 seq 4152 osr(2.d 0x2ed02b0)/0x2ed02b0 start -7036> 2014-06-09 16:22:12.183324 7f86a5e11700 10 _do_transaction on 0xc058840 -7035> 2014-06-09 16:22:12.183330 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -7034> 2014-06-09 16:22:12.183372 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7033> 2014-06-09 16:22:12.183389 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4152.0.0 <= header.spos 150443.0.0 -7032> 2014-06-09 16:22:12.183393 7f86a5e11700 5 submit_transaction r = 0 -7031> 2014-06-09 16:22:12.183408 7f86a5e11700 10 _do_op 0x324e6780 seq 4152 r = 0, finisher 0xbd73580 0 -7030> 2014-06-09 16:22:12.183418 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccc4a0] -7029> 2014-06-09 16:22:12.183421 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -7028> 2014-06-09 16:22:12.183431 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -7027> 2014-06-09 16:22:12.183440 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:154 -7026> 2014-06-09 16:22:12.183449 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccc4a0] -7025> 2014-06-09 16:22:12.183451 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -7024> 2014-06-09 16:22:12.183452 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -7023> 2014-06-09 16:22:12.183456 7f86a5e11700 10 _finish_op 0x324e6780 seq 4152 osr(2.d 0x2ed02b0)/0x2ed02b0 -7022> 2014-06-09 16:22:12.183468 7f8692f68700 10 op_submit_start 4201 -7021> 2014-06-09 16:22:12.183469 7f8692f68700 5 queue_transactions (trailing journal) 4201 -7020> 2014-06-09 16:22:12.183471 7f8692f68700 5 queue_op 0xd326ae0 seq 4201 osr(1.a 0x384c270) 824 bytes (queue has 50 ops and 4497324 bytes) -7019> 2014-06-09 16:22:12.183476 7f868b759700 2 waiting 51 > 50 ops || 4498152 > 104857600 -7018> 2014-06-09 16:22:12.183481 7f868e75f700 2 waiting 51 > 50 ops || 4498152 > 104857600 -7017> 2014-06-09 16:22:12.183485 7f868cf5c700 2 waiting 51 > 50 ops || 4498185 > 104857600 -7016> 2014-06-09 16:22:12.183489 7f868a757700 2 waiting 51 > 50 ops || 4498152 > 104857600 -7015> 2014-06-09 16:22:12.183494 7f868ef60700 2 waiting 51 > 50 ops || 4498152 > 104857600 -7014> 2014-06-09 16:22:12.183498 7f8690763700 2 waiting 51 > 50 ops || 4498185 > 104857600 -7013> 2014-06-09 16:22:12.183503 7f8690f64700 2 waiting 51 > 50 ops || 4498152 > 104857600 -7012> 2014-06-09 16:22:12.183507 7f8692767700 2 waiting 51 > 50 ops || 4498185 > 104857600 -7011> 2014-06-09 16:22:12.183511 7f8691f66700 2 waiting 51 > 50 ops || 4498185 > 104857600 -7010> 2014-06-09 16:22:12.183515 7f8691765700 2 waiting 51 > 50 ops || 4498182 > 104857600 -7009> 2014-06-09 16:22:12.183519 7f8689755700 2 waiting 51 > 50 ops || 4498152 > 104857600 -7008> 2014-06-09 16:22:12.183524 7f868d75d700 2 waiting 51 > 50 ops || 4498185 > 104857600 -7007> 2014-06-09 16:22:12.183528 7f8689f56700 2 waiting 51 > 50 ops || 4498185 > 104857600 -7006> 2014-06-09 16:22:12.183532 7f868ff62700 2 waiting 51 > 50 ops || 4516106 > 104857600 -7005> 2014-06-09 16:22:12.183536 7f868af58700 2 waiting 51 > 50 ops || 4498185 > 104857600 -7004> 2014-06-09 16:22:12.183541 7f868f761700 2 waiting 51 > 50 ops || 4498150 > 104857600 -7003> 2014-06-09 16:22:12.183545 7f868df5e700 2 waiting 51 > 50 ops || 4498152 > 104857600 -7002> 2014-06-09 16:22:12.183549 7f868bf5a700 2 waiting 51 > 50 ops || 4498152 > 104857600 -7001> 2014-06-09 16:22:12.183553 7f868c75b700 2 waiting 51 > 50 ops || 4498185 > 104857600 -7000> 2014-06-09 16:22:12.183563 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd73580] -6999> 2014-06-09 16:22:12.183570 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd73580] -6998> 2014-06-09 16:22:12.183571 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6997> 2014-06-09 16:22:12.183573 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6996> 2014-06-09 16:22:12.183576 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89ad80 (1 active) -6995> 2014-06-09 16:22:12.183578 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc781b00 (2 active) -6994> 2014-06-09 16:22:12.183583 7f8692f68700 10 op_submit_finish 4201 -6993> 2014-06-09 16:22:12.183587 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.24d( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.24d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) -6992> 2014-06-09 16:22:12.183626 7f8692f68700 10 log is not dirty -6991> 2014-06-09 16:22:12.183632 7f8692f68700 5 queue_transactions existing osr(2.24d 0x2ed0a50)/0x2ed0a50 -6990> 2014-06-09 16:22:12.183634 7f8692f68700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6989> 2014-06-09 16:22:12.183639 7f86a5e11700 5 _do_op 0x16fd4c60 seq 4153 osr(1.1e8 0x384cdf0)/0x384cdf0 start -6988> 2014-06-09 16:22:12.183642 7f86a5e11700 10 _do_transaction on 0xc144cc0 -6987> 2014-06-09 16:22:12.183648 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6986> 2014-06-09 16:22:12.183690 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6985> 2014-06-09 16:22:12.183705 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4153.0.0 <= header.spos 150443.0.0 -6984> 2014-06-09 16:22:12.183709 7f86a5e11700 5 submit_transaction r = 0 -6983> 2014-06-09 16:22:12.183734 7f86a5e11700 10 _do_op 0x16fd4c60 seq 4153 r = 0, finisher 0x3196c160 0 -6982> 2014-06-09 16:22:12.183745 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3196c2e0] -6981> 2014-06-09 16:22:12.183748 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3196c2e0] -6980> 2014-06-09 16:22:12.183750 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6979> 2014-06-09 16:22:12.183751 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6978> 2014-06-09 16:22:12.183754 7f86a5e11700 10 _finish_op 0x16fd4c60 seq 4153 osr(1.1e8 0x384cdf0)/0x384cdf0 -6977> 2014-06-09 16:22:12.183765 7f868e75f700 10 op_submit_start 4202 -6976> 2014-06-09 16:22:12.183767 7f868e75f700 5 queue_transactions (trailing journal) 4202 -6975> 2014-06-09 16:22:12.183768 7f868e75f700 5 queue_op 0xc34aba0 seq 4202 osr(0.3b7 0x2bd7290) 828 bytes (queue has 50 ops and 4497324 bytes) -6974> 2014-06-09 16:22:12.183773 7f868cf5c700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6973> 2014-06-09 16:22:12.183778 7f868a757700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6972> 2014-06-09 16:22:12.183782 7f868ef60700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6971> 2014-06-09 16:22:12.183787 7f8690763700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6970> 2014-06-09 16:22:12.183791 7f8690f64700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6969> 2014-06-09 16:22:12.183795 7f8692767700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6968> 2014-06-09 16:22:12.183800 7f8691f66700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6967> 2014-06-09 16:22:12.183804 7f8691765700 2 waiting 51 > 50 ops || 4498182 > 104857600 -6966> 2014-06-09 16:22:12.183809 7f8689755700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6965> 2014-06-09 16:22:12.183813 7f868d75d700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6964> 2014-06-09 16:22:12.183817 7f8689f56700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6963> 2014-06-09 16:22:12.183821 7f868ff62700 2 waiting 51 > 50 ops || 4516106 > 104857600 -6962> 2014-06-09 16:22:12.183826 7f868af58700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6961> 2014-06-09 16:22:12.183830 7f868f761700 2 waiting 51 > 50 ops || 4498150 > 104857600 -6960> 2014-06-09 16:22:12.183835 7f868df5e700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6959> 2014-06-09 16:22:12.183839 7f868bf5a700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6958> 2014-06-09 16:22:12.183843 7f868c75b700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6957> 2014-06-09 16:22:12.183848 7f8692f68700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6956> 2014-06-09 16:22:12.183852 7f868b759700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6955> 2014-06-09 16:22:12.183862 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3196c160] -6954> 2014-06-09 16:22:12.183865 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3196c160] -6953> 2014-06-09 16:22:12.183867 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6952> 2014-06-09 16:22:12.183868 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6951> 2014-06-09 16:22:12.183871 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc781b00 (1 active) -6950> 2014-06-09 16:22:12.183874 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1560 (2 active) -6949> 2014-06-09 16:22:12.183879 7f868e75f700 10 op_submit_finish 4202 -6948> 2014-06-09 16:22:12.183893 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6947> 2014-06-09 16:22:12.183909 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.70e( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -6946> 2014-06-09 16:22:12.183920 7f868e75f700 10 log is not dirty -6945> 2014-06-09 16:22:12.183926 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6944> 2014-06-09 16:22:12.183938 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -6943> 2014-06-09 16:22:12.183948 7f868e75f700 10 log is not dirty -6942> 2014-06-09 16:22:12.183952 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6941> 2014-06-09 16:22:12.183971 7f868e75f700 10 log is not dirty -6940> 2014-06-09 16:22:12.183976 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6939> 2014-06-09 16:22:12.183989 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x306b800,0x306bb00,0x306ba00,0xe033d00,0xe033c00,0xe033b00,0x10c54d00,0x10c54c00,0x10c54b00,0x16bd7300,0x16bd7200,0x16bd7100,0x16bd7000,0x16bd6f00 -6938> 2014-06-09 16:22:12.184038 7f868e75f700 10 log is not dirty -6937> 2014-06-09 16:22:12.184043 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6936> 2014-06-09 16:22:12.184055 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -6935> 2014-06-09 16:22:12.184065 7f868e75f700 10 log is not dirty -6934> 2014-06-09 16:22:12.184069 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6933> 2014-06-09 16:22:12.184081 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -6932> 2014-06-09 16:22:12.184090 7f868e75f700 10 log is not dirty -6931> 2014-06-09 16:22:12.184094 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6930> 2014-06-09 16:22:12.184108 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -6929> 2014-06-09 16:22:12.184119 7f868e75f700 10 log is not dirty -6928> 2014-06-09 16:22:12.184123 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6927> 2014-06-09 16:22:12.184135 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] requeue_ops -6926> 2014-06-09 16:22:12.184144 7f868e75f700 10 log is not dirty -6925> 2014-06-09 16:22:12.184149 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6924> 2014-06-09 16:22:12.184162 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.4c5( v 213'605 (0'0,213'605] local-les=243 n=123 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -6923> 2014-06-09 16:22:12.184172 7f868e75f700 10 log is not dirty -6922> 2014-06-09 16:22:12.184177 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6921> 2014-06-09 16:22:12.184189 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -6920> 2014-06-09 16:22:12.184198 7f868e75f700 10 log is not dirty -6919> 2014-06-09 16:22:12.184202 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6918> 2014-06-09 16:22:12.184215 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.4c8( v 213'642 (0'0,213'642] local-les=243 n=117 ec=118 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -6917> 2014-06-09 16:22:12.184226 7f868e75f700 10 log is not dirty -6916> 2014-06-09 16:22:12.184230 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6915> 2014-06-09 16:22:12.184243 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -6914> 2014-06-09 16:22:12.184555 7f868e75f700 10 log is not dirty -6913> 2014-06-09 16:22:12.184561 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6912> 2014-06-09 16:22:12.184574 7f868e75f700 15 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -6911> 2014-06-09 16:22:12.184583 7f868e75f700 10 log is not dirty -6910> 2014-06-09 16:22:12.184587 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6909> 2014-06-09 16:22:12.184600 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.148( v 213'639 (0'0,213'639] local-les=243 n=129 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -6908> 2014-06-09 16:22:12.184612 7f868e75f700 10 log is not dirty -6907> 2014-06-09 16:22:12.184616 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6906> 2014-06-09 16:22:12.184637 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.1c6( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -6905> 2014-06-09 16:22:12.184646 7f868e75f700 10 log is not dirty -6904> 2014-06-09 16:22:12.184650 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6903> 2014-06-09 16:22:12.184662 7f868e75f700 15 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] requeue_ops -6902> 2014-06-09 16:22:12.184671 7f868e75f700 10 log is not dirty -6901> 2014-06-09 16:22:12.184676 7f868e75f700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6900> 2014-06-09 16:22:12.184690 7f868e75f700 15 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -6899> 2014-06-09 16:22:12.184700 7f868e75f700 10 log is not dirty -6898> 2014-06-09 16:22:12.184705 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6897> 2014-06-09 16:22:12.184717 7f868e75f700 15 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -6896> 2014-06-09 16:22:12.184726 7f868e75f700 10 log is not dirty -6895> 2014-06-09 16:22:12.184746 7f868e75f700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40e90 (19 active) -6894> 2014-06-09 16:22:12.184771 7f868e75f700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb40e90 (20 active) -6893> 2014-06-09 16:22:12.184775 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.473( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.473( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) -6892> 2014-06-09 16:22:12.184810 7f868e75f700 10 log is not dirty -6891> 2014-06-09 16:22:12.184814 7f868e75f700 5 queue_transactions existing osr(0.473 0x2bd74d0)/0x2bd74d0 -6890> 2014-06-09 16:22:12.184817 7f868e75f700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6889> 2014-06-09 16:22:12.184822 7f86a5e11700 5 _do_op 0xc34a6c0 seq 4154 osr(0.230 0x2bd6a10)/0x2bd6a10 start -6888> 2014-06-09 16:22:12.184826 7f86a5e11700 10 _do_transaction on 0xc216e40 -6887> 2014-06-09 16:22:12.184832 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6886> 2014-06-09 16:22:12.184878 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6885> 2014-06-09 16:22:12.184896 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4154.0.0 <= header.spos 150443.0.0 -6884> 2014-06-09 16:22:12.184900 7f86a5e11700 5 submit_transaction r = 0 -6883> 2014-06-09 16:22:12.184914 7f86a5e11700 10 _do_op 0xc34a6c0 seq 4154 r = 0, finisher 0x189aabc0 0 -6882> 2014-06-09 16:22:12.184926 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd766e0] -6881> 2014-06-09 16:22:12.184929 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -6880> 2014-06-09 16:22:12.184940 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -6879> 2014-06-09 16:22:12.184949 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.230( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 -6878> 2014-06-09 16:22:12.184959 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd766e0] -6877> 2014-06-09 16:22:12.184960 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6876> 2014-06-09 16:22:12.184962 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6875> 2014-06-09 16:22:12.184966 7f86a5e11700 10 _finish_op 0xc34a6c0 seq 4154 osr(0.230 0x2bd6a10)/0x2bd6a10 -6874> 2014-06-09 16:22:12.184977 7f868a757700 10 op_submit_start 4203 -6873> 2014-06-09 16:22:12.184979 7f868a757700 5 queue_transactions (trailing journal) 4203 -6872> 2014-06-09 16:22:12.184980 7f868a757700 5 queue_op 0xd326f00 seq 4203 osr(3.470 0x3ed1540) 828 bytes (queue has 50 ops and 4497291 bytes) -6871> 2014-06-09 16:22:12.184985 7f868ef60700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6870> 2014-06-09 16:22:12.184991 7f8690763700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6869> 2014-06-09 16:22:12.184996 7f8690f64700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6868> 2014-06-09 16:22:12.185000 7f8692767700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6867> 2014-06-09 16:22:12.185004 7f8691f66700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6866> 2014-06-09 16:22:12.185009 7f8691765700 2 waiting 51 > 50 ops || 4498149 > 104857600 -6865> 2014-06-09 16:22:12.185014 7f8689755700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6864> 2014-06-09 16:22:12.185018 7f868d75d700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6863> 2014-06-09 16:22:12.185023 7f8689f56700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6862> 2014-06-09 16:22:12.185027 7f868ff62700 2 waiting 51 > 50 ops || 4516073 > 104857600 -6861> 2014-06-09 16:22:12.185031 7f868af58700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6860> 2014-06-09 16:22:12.185035 7f868f761700 2 waiting 51 > 50 ops || 4498117 > 104857600 -6859> 2014-06-09 16:22:12.185040 7f868df5e700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6858> 2014-06-09 16:22:12.185044 7f868bf5a700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6857> 2014-06-09 16:22:12.185048 7f868c75b700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6856> 2014-06-09 16:22:12.185053 7f8692f68700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6855> 2014-06-09 16:22:12.185057 7f868b759700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6854> 2014-06-09 16:22:12.185062 7f868e75f700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6853> 2014-06-09 16:22:12.185066 7f868cf5c700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6852> 2014-06-09 16:22:12.185077 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aabc0] -6851> 2014-06-09 16:22:12.185080 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aabc0] -6850> 2014-06-09 16:22:12.185082 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6849> 2014-06-09 16:22:12.185083 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6848> 2014-06-09 16:22:12.185086 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1560 (1 active) -6847> 2014-06-09 16:22:12.185089 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52e900 (2 active) -6846> 2014-06-09 16:22:12.185094 7f868a757700 10 op_submit_finish 4203 -6845> 2014-06-09 16:22:12.185098 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.4ba( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) -6844> 2014-06-09 16:22:12.185133 7f868a757700 10 log is not dirty -6843> 2014-06-09 16:22:12.185137 7f868a757700 5 queue_transactions existing osr(0.4ba 0x2bd7710)/0x2bd7710 -6842> 2014-06-09 16:22:12.185140 7f868a757700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6841> 2014-06-09 16:22:12.185145 7f86a5e11700 5 _do_op 0x16097080 seq 4155 osr(1.54c 0x384dad0)/0x384dad0 start -6840> 2014-06-09 16:22:12.185149 7f86a5e11700 10 _do_transaction on 0x39db440 -6839> 2014-06-09 16:22:12.185155 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6838> 2014-06-09 16:22:12.185198 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6837> 2014-06-09 16:22:12.185215 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4155.0.0 <= header.spos 150443.0.0 -6836> 2014-06-09 16:22:12.185219 7f86a5e11700 5 submit_transaction r = 0 -6835> 2014-06-09 16:22:12.185235 7f86a5e11700 10 _do_op 0x16097080 seq 4155 r = 0, finisher 0x2d7398c0 0 -6834> 2014-06-09 16:22:12.185246 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc410440] -6833> 2014-06-09 16:22:12.185249 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -6832> 2014-06-09 16:22:12.185260 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -6831> 2014-06-09 16:22:12.185269 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.54c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -6830> 2014-06-09 16:22:12.185278 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc410440] -6829> 2014-06-09 16:22:12.185280 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6828> 2014-06-09 16:22:12.185281 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6827> 2014-06-09 16:22:12.185285 7f86a5e11700 10 _finish_op 0x16097080 seq 4155 osr(1.54c 0x384dad0)/0x384dad0 -6826> 2014-06-09 16:22:12.185296 7f8690763700 10 op_submit_start 4204 -6825> 2014-06-09 16:22:12.185298 7f8690763700 5 queue_transactions (trailing journal) 4204 -6824> 2014-06-09 16:22:12.185299 7f8690763700 5 queue_op 0xc2d4b40 seq 4204 osr(0.164 0x2bd6810) 861 bytes (queue has 50 ops and 4497291 bytes) -6823> 2014-06-09 16:22:12.185304 7f8690f64700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6822> 2014-06-09 16:22:12.185309 7f8692767700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6821> 2014-06-09 16:22:12.185313 7f8691f66700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6820> 2014-06-09 16:22:12.185318 7f8691765700 2 waiting 51 > 50 ops || 4498149 > 104857600 -6819> 2014-06-09 16:22:12.185322 7f8689755700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6818> 2014-06-09 16:22:12.185327 7f868d75d700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6817> 2014-06-09 16:22:12.185331 7f8689f56700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6816> 2014-06-09 16:22:12.185335 7f868ff62700 2 waiting 51 > 50 ops || 4516073 > 104857600 -6815> 2014-06-09 16:22:12.185340 7f868af58700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6814> 2014-06-09 16:22:12.185344 7f868f761700 2 waiting 51 > 50 ops || 4498117 > 104857600 -6813> 2014-06-09 16:22:12.185349 7f868df5e700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6812> 2014-06-09 16:22:12.185353 7f868bf5a700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6811> 2014-06-09 16:22:12.185358 7f868c75b700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6810> 2014-06-09 16:22:12.185362 7f8692f68700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6809> 2014-06-09 16:22:12.185366 7f868b759700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6808> 2014-06-09 16:22:12.185370 7f868e75f700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6807> 2014-06-09 16:22:12.185374 7f868cf5c700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6806> 2014-06-09 16:22:12.185378 7f868a757700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6805> 2014-06-09 16:22:12.185382 7f868ef60700 2 waiting 51 > 50 ops || 4498119 > 104857600 -6804> 2014-06-09 16:22:12.185393 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d7398c0] -6803> 2014-06-09 16:22:12.185397 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d7398c0] -6802> 2014-06-09 16:22:12.185399 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6801> 2014-06-09 16:22:12.185400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6800> 2014-06-09 16:22:12.185403 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52e900 (1 active) -6799> 2014-06-09 16:22:12.185406 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780d80 (2 active) -6798> 2014-06-09 16:22:12.185411 7f8690763700 10 op_submit_finish 4204 -6797> 2014-06-09 16:22:12.185415 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6796> 2014-06-09 16:22:12.185436 7f8690763700 10 log is not dirty -6795> 2014-06-09 16:22:12.185441 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6794> 2014-06-09 16:22:12.185454 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0xe032100,0xe032500,0xe032400,0xe032300,0xe2acd00,0xe2acc00,0xe2acb00,0xe2aca00 -6793> 2014-06-09 16:22:12.185489 7f8690763700 10 log is not dirty -6792> 2014-06-09 16:22:12.185494 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -6791> 2014-06-09 16:22:12.185507 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.948935 7 0.000495 -6790> 2014-06-09 16:22:12.185519 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -6789> 2014-06-09 16:22:12.185530 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -6788> 2014-06-09 16:22:12.185541 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -6787> 2014-06-09 16:22:12.185552 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -6786> 2014-06-09 16:22:12.185561 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -6785> 2014-06-09 16:22:12.185569 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -6784> 2014-06-09 16:22:12.185579 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -6783> 2014-06-09 16:22:12.185591 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -6782> 2014-06-09 16:22:12.185600 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -6781> 2014-06-09 16:22:12.185609 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -6780> 2014-06-09 16:22:12.185619 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -6779> 2014-06-09 16:22:12.185633 7f8690763700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbf6e6e0 172.16.96.201:6801/57551 -6778> 2014-06-09 16:22:12.185644 7f8690763700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6801/57551 -- osd_map(244..244 src has 1..244) v3 -- ?+0 0x4682240 con 0xbf6e6e0 -6777> 2014-06-09 16:22:12.185655 7f8690763700 10 osd.26 243 note_peer_epoch osd.0 has 244 -6776> 2014-06-09 16:22:12.185658 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:2.754) v4 -- ?+0 0x3343dc00 -6775> 2014-06-09 16:22:12.185667 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 -6774> 2014-06-09 16:22:12.185702 7f8690763700 10 log is not dirty -6773> 2014-06-09 16:22:12.185706 7f8690763700 5 queue_transactions existing osr(2.754 0x3ed00f0)/0x3ed00f0 -6772> 2014-06-09 16:22:12.185709 7f8690763700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6771> 2014-06-09 16:22:12.185749 7f86a5e11700 5 _do_op 0xbf7e000 seq 4156 osr(3.172 0x3ed06e0)/0x3ed06e0 start -6770> 2014-06-09 16:22:12.185756 7f86a5e11700 10 _do_transaction on 0xbebc480 -6769> 2014-06-09 16:22:12.185762 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6768> 2014-06-09 16:22:12.185816 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6767> 2014-06-09 16:22:12.185831 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4156.0.0 <= header.spos 150443.0.0 -6766> 2014-06-09 16:22:12.185836 7f86a5e11700 5 submit_transaction r = 0 -6765> 2014-06-09 16:22:12.185850 7f86a5e11700 10 _do_op 0xbf7e000 seq 4156 r = 0, finisher 0x30d616a0 0 -6764> 2014-06-09 16:22:12.185861 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32079220] -6763> 2014-06-09 16:22:12.185863 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32079220] -6762> 2014-06-09 16:22:12.185865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6761> 2014-06-09 16:22:12.185866 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6760> 2014-06-09 16:22:12.185870 7f86a5e11700 10 _finish_op 0xbf7e000 seq 4156 osr(3.172 0x3ed06e0)/0x3ed06e0 -6759> 2014-06-09 16:22:12.185881 7f8692767700 10 op_submit_start 4205 -6758> 2014-06-09 16:22:12.185883 7f8692767700 5 queue_transactions (trailing journal) 4205 -6757> 2014-06-09 16:22:12.185884 7f8692767700 5 queue_op 0x30607980 seq 4205 osr(0.319 0x2bd6d90) 861 bytes (queue has 50 ops and 4497324 bytes) -6756> 2014-06-09 16:22:12.185889 7f8691f66700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6755> 2014-06-09 16:22:12.185894 7f8691765700 2 waiting 51 > 50 ops || 4498182 > 104857600 -6754> 2014-06-09 16:22:12.185898 7f8689755700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6753> 2014-06-09 16:22:12.185903 7f868d75d700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6752> 2014-06-09 16:22:12.185907 7f8689f56700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6751> 2014-06-09 16:22:12.185911 7f868ff62700 2 waiting 51 > 50 ops || 4516106 > 104857600 -6750> 2014-06-09 16:22:12.185916 7f868af58700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6749> 2014-06-09 16:22:12.185920 7f868f761700 2 waiting 51 > 50 ops || 4498150 > 104857600 -6748> 2014-06-09 16:22:12.185924 7f868df5e700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6747> 2014-06-09 16:22:12.185929 7f868bf5a700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6746> 2014-06-09 16:22:12.185933 7f868c75b700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6745> 2014-06-09 16:22:12.185937 7f8692f68700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6744> 2014-06-09 16:22:12.185941 7f868b759700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6743> 2014-06-09 16:22:12.185945 7f868e75f700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6742> 2014-06-09 16:22:12.185949 7f868cf5c700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6741> 2014-06-09 16:22:12.185954 7f868a757700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6740> 2014-06-09 16:22:12.185958 7f868ef60700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6739> 2014-06-09 16:22:12.185963 7f8690763700 2 waiting 51 > 50 ops || 4498185 > 104857600 -6738> 2014-06-09 16:22:12.185967 7f8690f64700 2 waiting 51 > 50 ops || 4498152 > 104857600 -6737> 2014-06-09 16:22:12.185976 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30d616a0] -6736> 2014-06-09 16:22:12.185981 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30d616a0] -6735> 2014-06-09 16:22:12.185983 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6734> 2014-06-09 16:22:12.185984 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6733> 2014-06-09 16:22:12.185987 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780d80 (1 active) -6732> 2014-06-09 16:22:12.185990 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114120 (2 active) -6731> 2014-06-09 16:22:12.185995 7f8692767700 10 op_submit_finish 4205 -6730> 2014-06-09 16:22:12.186001 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 inactive m=2] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -6729> 2014-06-09 16:22:12.186021 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] check_local -6728> 2014-06-09 16:22:12.186033 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] requeue_ops 0xe2ad300,0xe2ad200,0xe2ad100,0x10c54500,0x10c54400,0x10c54300 -6727> 2014-06-09 16:22:12.186061 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] needs_recovery primary has 2 missing -6726> 2014-06-09 16:22:12.186075 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] activate not all replicas are up-to-date, queueing recovery -6725> 2014-06-09 16:22:12.186093 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] publish_stats_to_osd 243:1031 -6724> 2014-06-09 16:22:12.186106 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] hit_set_clear -6723> 2014-06-09 16:22:12.186118 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] agent_stop -6722> 2014-06-09 16:22:12.186131 7f8692767700 10 log is not dirty -6721> 2014-06-09 16:22:12.186135 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -6720> 2014-06-09 16:22:12.186148 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.120495 7 0.009110 -6719> 2014-06-09 16:22:12.186159 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -6718> 2014-06-09 16:22:12.186169 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -6717> 2014-06-09 16:22:12.186179 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -6716> 2014-06-09 16:22:12.186190 7f8692767700 5 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -6715> 2014-06-09 16:22:12.186199 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -6714> 2014-06-09 16:22:12.186206 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -6713> 2014-06-09 16:22:12.186217 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -6712> 2014-06-09 16:22:12.186229 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -6711> 2014-06-09 16:22:12.186238 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -6710> 2014-06-09 16:22:12.186304 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -6709> 2014-06-09 16:22:12.186313 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -6708> 2014-06-09 16:22:12.186326 7f8692767700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 -6707> 2014-06-09 16:22:12.186329 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:0.6e) v4 -- ?+0 0x31776380 -6706> 2014-06-09 16:22:12.186341 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 -6705> 2014-06-09 16:22:12.186375 7f8692767700 10 log is not dirty -6704> 2014-06-09 16:22:12.186379 7f8692767700 5 queue_transactions existing osr(0.6e 0x2bd6390)/0x2bd6390 -6703> 2014-06-09 16:22:12.186382 7f8692767700 2 waiting 51 > 50 ops || 4498182 > 104857600 -6702> 2014-06-09 16:22:12.186425 7f86a5e11700 5 _do_op 0xc88bce0 seq 4157 osr(0.9d 0x2bd6490)/0x2bd6490 start -6701> 2014-06-09 16:22:12.186433 7f86a5e11700 10 _do_transaction on 0x2eaafc0 -6700> 2014-06-09 16:22:12.186439 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6699> 2014-06-09 16:22:12.186491 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6698> 2014-06-09 16:22:12.186507 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4157.0.0 <= header.spos 150443.0.0 -6697> 2014-06-09 16:22:12.186511 7f86a5e11700 5 submit_transaction r = 0 -6696> 2014-06-09 16:22:12.186523 7f86a5e11700 10 _do_op 0xc88bce0 seq 4157 r = 0, finisher 0xbd5f120 0 -6695> 2014-06-09 16:22:12.181597 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 11 ==== osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+1548288 (917049376 0 0) 0xcbe6240 con 0x30c3c2c0 -6694> 2014-06-09 16:22:12.186547 7f86a68f6700 5 -- op tracker -- , seq: 3673, time: 2014-06-09 16:22:12.177787, event: header_read, op: osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -6693> 2014-06-09 16:22:12.186562 7f86a68f6700 5 -- op tracker -- , seq: 3673, time: 2014-06-09 16:22:12.177788, event: throttled, op: osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -6692> 2014-06-09 16:22:12.186574 7f86a68f6700 5 -- op tracker -- , seq: 3673, time: 2014-06-09 16:22:12.181593, event: all_read, op: osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -6691> 2014-06-09 16:22:12.186584 7f86a68f6700 5 -- op tracker -- , seq: 3673, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -6690> 2014-06-09 16:22:12.186591 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -6689> 2014-06-09 16:22:12.186597 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -6688> 2014-06-09 16:22:12.186615 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976da00 prio 63 cost 1548288 latency 0.008828 osd_op(client.4543.0:33346 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 1671168~1548288] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 -6687> 2014-06-09 16:22:12.187240 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31434000] -6686> 2014-06-09 16:22:12.187245 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31434000] -6685> 2014-06-09 16:22:12.187249 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6684> 2014-06-09 16:22:12.187251 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6683> 2014-06-09 16:22:12.187255 7f86a5e11700 10 _finish_op 0xc88bce0 seq 4157 osr(0.9d 0x2bd6490)/0x2bd6490 -6682> 2014-06-09 16:22:12.187267 7f8691765700 10 op_submit_start 4206 -6681> 2014-06-09 16:22:12.187269 7f8691765700 5 queue_transactions (trailing journal) 4206 -6680> 2014-06-09 16:22:12.187271 7f8691765700 5 queue_op 0x1e08a2a0 seq 4206 osr(1.b3 0x384c6d0) 858 bytes (queue has 50 ops and 4497356 bytes) -6679> 2014-06-09 16:22:12.187276 7f8689755700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6678> 2014-06-09 16:22:12.187282 7f868d75d700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6677> 2014-06-09 16:22:12.187286 7f8689f56700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6676> 2014-06-09 16:22:12.187291 7f868ff62700 2 waiting 51 > 50 ops || 4516138 > 104857600 -6675> 2014-06-09 16:22:12.187295 7f868af58700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6674> 2014-06-09 16:22:12.187300 7f868f761700 2 waiting 51 > 50 ops || 4498182 > 104857600 -6673> 2014-06-09 16:22:12.187305 7f868df5e700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6672> 2014-06-09 16:22:12.187309 7f868bf5a700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6671> 2014-06-09 16:22:12.187314 7f868c75b700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6670> 2014-06-09 16:22:12.187319 7f8692f68700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6669> 2014-06-09 16:22:12.187323 7f868b759700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6668> 2014-06-09 16:22:12.187328 7f868e75f700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6667> 2014-06-09 16:22:12.187333 7f868cf5c700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6666> 2014-06-09 16:22:12.187337 7f868a757700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6665> 2014-06-09 16:22:12.187342 7f868ef60700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6664> 2014-06-09 16:22:12.187346 7f8690763700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6663> 2014-06-09 16:22:12.187351 7f8690f64700 2 waiting 51 > 50 ops || 4498184 > 104857600 -6662> 2014-06-09 16:22:12.187355 7f8692767700 2 waiting 51 > 50 ops || 4498214 > 104857600 -6661> 2014-06-09 16:22:12.187360 7f8691f66700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6660> 2014-06-09 16:22:12.187371 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5f120] -6659> 2014-06-09 16:22:12.187375 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5f120] -6658> 2014-06-09 16:22:12.187377 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6657> 2014-06-09 16:22:12.187378 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6656> 2014-06-09 16:22:12.187381 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114120 (1 active) -6655> 2014-06-09 16:22:12.187384 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115440 (2 active) -6654> 2014-06-09 16:22:12.187389 7f8691765700 10 op_submit_finish 4206 -6653> 2014-06-09 16:22:12.187394 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -6652> 2014-06-09 16:22:12.187409 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.040696 7 0.000316 -6651> 2014-06-09 16:22:12.187421 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -6650> 2014-06-09 16:22:12.187432 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -6649> 2014-06-09 16:22:12.187443 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -6648> 2014-06-09 16:22:12.187453 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -6647> 2014-06-09 16:22:12.187462 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -6646> 2014-06-09 16:22:12.187470 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -6645> 2014-06-09 16:22:12.187481 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -6644> 2014-06-09 16:22:12.187493 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -6643> 2014-06-09 16:22:12.187503 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -6642> 2014-06-09 16:22:12.187517 7f8691765700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbf7a840 172.16.96.201:6837/51639 -6641> 2014-06-09 16:22:12.187528 7f8691765700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6837/51639 -- osd_map(244..244 src has 1..244) v3 -- ?+0 0x2d730240 con 0xbf7a840 -6640> 2014-06-09 16:22:12.187540 7f8691765700 10 osd.26 243 note_peer_epoch osd.5 has 244 -6639> 2014-06-09 16:22:12.187543 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:1.b8) v4 -- ?+0 0x30d9c1c0 -6638> 2014-06-09 16:22:12.187552 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 -6637> 2014-06-09 16:22:12.187588 7f8691765700 10 log is not dirty -6636> 2014-06-09 16:22:12.187593 7f8691765700 5 queue_transactions existing osr(1.b8 0x384c650)/0x384c650 -6635> 2014-06-09 16:22:12.187595 7f8691765700 2 waiting 51 > 50 ops || 4498214 > 104857600 -6634> 2014-06-09 16:22:12.187600 7f86a5e11700 5 _do_op 0x30a2c60 seq 4158 osr(1.32f 0x384d350)/0x384d350 start -6633> 2014-06-09 16:22:12.187604 7f86a5e11700 10 _do_transaction on 0xbf360c0 -6632> 2014-06-09 16:22:12.187610 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6631> 2014-06-09 16:22:12.187667 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6630> 2014-06-09 16:22:12.187684 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4158.0.0 <= header.spos 150443.0.0 -6629> 2014-06-09 16:22:12.187688 7f86a5e11700 5 submit_transaction r = 0 -6628> 2014-06-09 16:22:12.188062 7f86a5e11700 10 _do_op 0x30a2c60 seq 4158 r = 0, finisher 0x307f00e0 0 -6627> 2014-06-09 16:22:12.188075 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f03e0] -6626> 2014-06-09 16:22:12.188077 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f03e0] -6625> 2014-06-09 16:22:12.188079 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6624> 2014-06-09 16:22:12.188080 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6623> 2014-06-09 16:22:12.188084 7f86a5e11700 10 _finish_op 0x30a2c60 seq 4158 osr(1.32f 0x384d350)/0x384d350 -6622> 2014-06-09 16:22:12.188095 7f868d75d700 10 op_submit_start 4207 -6621> 2014-06-09 16:22:12.188097 7f868d75d700 5 queue_transactions (trailing journal) 4207 -6620> 2014-06-09 16:22:12.188098 7f868d75d700 5 queue_op 0xbe7daa0 seq 4207 osr(3.78a 0x3ed1d80) 861 bytes (queue has 50 ops and 4497389 bytes) -6619> 2014-06-09 16:22:12.188103 7f8689f56700 2 waiting 51 > 50 ops || 4498250 > 104857600 -6618> 2014-06-09 16:22:12.188108 7f868ff62700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6617> 2014-06-09 16:22:12.188113 7f868af58700 2 waiting 51 > 50 ops || 4498250 > 104857600 -6616> 2014-06-09 16:22:12.188117 7f868f761700 2 waiting 51 > 50 ops || 4498215 > 104857600 -6615> 2014-06-09 16:22:12.188121 7f868df5e700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6614> 2014-06-09 16:22:12.188126 7f868bf5a700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6613> 2014-06-09 16:22:12.188130 7f868c75b700 2 waiting 51 > 50 ops || 4498250 > 104857600 -6612> 2014-06-09 16:22:12.188134 7f8692f68700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6611> 2014-06-09 16:22:12.188139 7f868b759700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6610> 2014-06-09 16:22:12.188143 7f868e75f700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6609> 2014-06-09 16:22:12.188148 7f868cf5c700 2 waiting 51 > 50 ops || 4498250 > 104857600 -6608> 2014-06-09 16:22:12.188152 7f868a757700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6607> 2014-06-09 16:22:12.188215 7f868ef60700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6606> 2014-06-09 16:22:12.188221 7f8690763700 2 waiting 51 > 50 ops || 4498250 > 104857600 -6605> 2014-06-09 16:22:12.188225 7f8690f64700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6604> 2014-06-09 16:22:12.188230 7f8692767700 2 waiting 51 > 50 ops || 4498247 > 104857600 -6603> 2014-06-09 16:22:12.188234 7f8691f66700 2 waiting 51 > 50 ops || 4498250 > 104857600 -6602> 2014-06-09 16:22:12.188238 7f8691765700 2 waiting 51 > 50 ops || 4498247 > 104857600 -6601> 2014-06-09 16:22:12.188243 7f8689755700 2 waiting 51 > 50 ops || 4498217 > 104857600 -6600> 2014-06-09 16:22:12.188253 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f00e0] -6599> 2014-06-09 16:22:12.188257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f00e0] -6598> 2014-06-09 16:22:12.188259 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6597> 2014-06-09 16:22:12.188260 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6596> 2014-06-09 16:22:12.188263 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115440 (1 active) -6595> 2014-06-09 16:22:12.188266 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4aca20 (2 active) -6594> 2014-06-09 16:22:12.188271 7f868d75d700 10 op_submit_finish 4207 -6593> 2014-06-09 16:22:12.188275 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -6592> 2014-06-09 16:22:12.188292 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local -6591> 2014-06-09 16:22:12.188301 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -6590> 2014-06-09 16:22:12.188296 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 12 ==== osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+974848 (2029520572 0 0) 0xcbe6000 con 0x30c3c2c0 -6589> 2014-06-09 16:22:12.188310 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -6588> 2014-06-09 16:22:12.188319 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -6587> 2014-06-09 16:22:12.188321 7f86a68f6700 5 -- op tracker -- , seq: 3674, time: 2014-06-09 16:22:12.186641, event: header_read, op: osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -6586> 2014-06-09 16:22:12.188328 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -6585> 2014-06-09 16:22:12.188334 7f86a68f6700 5 -- op tracker -- , seq: 3674, time: 2014-06-09 16:22:12.186643, event: throttled, op: osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -6584> 2014-06-09 16:22:12.188342 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:184 -6583> 2014-06-09 16:22:12.188348 7f86a68f6700 5 -- op tracker -- , seq: 3674, time: 2014-06-09 16:22:12.188280, event: all_read, op: osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -6582> 2014-06-09 16:22:12.188352 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -6581> 2014-06-09 16:22:12.188358 7f86a68f6700 5 -- op tracker -- , seq: 3674, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) -6580> 2014-06-09 16:22:12.188361 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.290( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop -6579> 2014-06-09 16:22:12.188365 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -6578> 2014-06-09 16:22:12.188370 7f868d75d700 10 log is not dirty -6577> 2014-06-09 16:22:12.188371 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -6576> 2014-06-09 16:22:12.188376 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -6575> 2014-06-09 16:22:12.188388 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d900 prio 63 cost 974848 latency 0.001747 osd_op(client.4543.0:33347 rbd_data.115f6b8b4567.000000000000168c [set-alloc-hint object_size 4194304 write_size 4194304,write 3219456~974848] 3.81c5bfe4 RETRY=2 ack+ondisk+retry+write e243) v4 -6574> 2014-06-09 16:22:12.188390 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -6573> 2014-06-09 16:22:12.188400 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x230da800,0x10c54e00,0xe032d00,0x230da700,0x11e62b00,0x10c54900,0x230da600,0x11e62a00,0x10c54800,0x230da500,0x14052200,0x10c54700,0x230da400,0x14052100,0x230da300,0x10c54600,0x13022f00 -6572> 2014-06-09 16:22:12.188456 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -6571> 2014-06-09 16:22:12.188466 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -6570> 2014-06-09 16:22:12.188475 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -6569> 2014-06-09 16:22:12.188489 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:962 -6568> 2014-06-09 16:22:12.188499 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -6567> 2014-06-09 16:22:12.188508 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -6566> 2014-06-09 16:22:12.188518 7f868d75d700 10 log is not dirty -6565> 2014-06-09 16:22:12.188523 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -6564> 2014-06-09 16:22:12.188536 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.133427 8 0.000873 -6563> 2014-06-09 16:22:12.188548 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -6562> 2014-06-09 16:22:12.188558 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -6561> 2014-06-09 16:22:12.188569 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -6560> 2014-06-09 16:22:12.188580 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -6559> 2014-06-09 16:22:12.188588 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -6558> 2014-06-09 16:22:12.188596 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -6557> 2014-06-09 16:22:12.188607 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -6556> 2014-06-09 16:22:12.188618 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -6555> 2014-06-09 16:22:12.188627 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -6554> 2014-06-09 16:22:12.188641 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -6553> 2014-06-09 16:22:12.188643 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:1.223) v4 -- ?+0 0x190b68c0 -6552> 2014-06-09 16:22:12.188658 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 -6551> 2014-06-09 16:22:12.188692 7f868d75d700 10 log is not dirty -6550> 2014-06-09 16:22:12.188696 7f868d75d700 5 queue_transactions existing osr(1.223 0x384cc70)/0x384cc70 -6549> 2014-06-09 16:22:12.188699 7f868d75d700 2 waiting 51 > 50 ops || 4498250 > 104857600 -6548> 2014-06-09 16:22:12.188688 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 13 ==== osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+249856 (464399106 0 0) 0x32d81d40 con 0x30c3c2c0 -6547> 2014-06-09 16:22:12.188729 7f86a5e11700 5 _do_op 0x324e76e0 seq 4159 osr(2.40c 0x2ed1250)/0x2ed1250 start -6546> 2014-06-09 16:22:12.188727 7f86a68f6700 5 -- op tracker -- , seq: 3675, time: 2014-06-09 16:22:12.188411, event: header_read, op: osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6545> 2014-06-09 16:22:12.188736 7f86a5e11700 10 _do_transaction on 0x2f28840 -6544> 2014-06-09 16:22:12.188742 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6543> 2014-06-09 16:22:12.188741 7f86a68f6700 5 -- op tracker -- , seq: 3675, time: 2014-06-09 16:22:12.188413, event: throttled, op: osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6542> 2014-06-09 16:22:12.188750 7f86a68f6700 5 -- op tracker -- , seq: 3675, time: 2014-06-09 16:22:12.188677, event: all_read, op: osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6541> 2014-06-09 16:22:12.188759 7f86a68f6700 5 -- op tracker -- , seq: 3675, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6540> 2014-06-09 16:22:12.188764 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -6539> 2014-06-09 16:22:12.188768 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -6538> 2014-06-09 16:22:12.188781 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976d800 prio 63 cost 249856 latency 0.000370 osd_op(client.4543.0:33379 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~249856] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 -6537> 2014-06-09 16:22:12.188791 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6536> 2014-06-09 16:22:12.188807 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4159.0.0 <= header.spos 150443.0.0 -6535> 2014-06-09 16:22:12.188811 7f86a5e11700 5 submit_transaction r = 0 -6534> 2014-06-09 16:22:12.188824 7f86a5e11700 10 _do_op 0x324e76e0 seq 4159 r = 0, finisher 0xbd75440 0 -6533> 2014-06-09 16:22:12.188835 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002060] -6532> 2014-06-09 16:22:12.188838 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002060] -6531> 2014-06-09 16:22:12.188840 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6530> 2014-06-09 16:22:12.188841 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6529> 2014-06-09 16:22:12.188844 7f86a5e11700 10 _finish_op 0x324e76e0 seq 4159 osr(2.40c 0x2ed1250)/0x2ed1250 -6528> 2014-06-09 16:22:12.188856 7f868ff62700 10 op_submit_start 4208 -6527> 2014-06-09 16:22:12.188857 7f868ff62700 5 queue_transactions (trailing journal) 4208 -6526> 2014-06-09 16:22:12.188859 7f868ff62700 5 queue_op 0x30a24e0 seq 4208 osr(3.2ac 0x3ed0b50) 18782 bytes (queue has 50 ops and 4515343 bytes) -6525> 2014-06-09 16:22:12.188864 7f868af58700 2 waiting 51 > 50 ops || 4516204 > 104857600 -6524> 2014-06-09 16:22:12.188869 7f868f761700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6523> 2014-06-09 16:22:12.188874 7f868df5e700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6522> 2014-06-09 16:22:12.188878 7f868bf5a700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6521> 2014-06-09 16:22:12.188883 7f868c75b700 2 waiting 51 > 50 ops || 4516204 > 104857600 -6520> 2014-06-09 16:22:12.188887 7f8692f68700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6519> 2014-06-09 16:22:12.188892 7f868b759700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6518> 2014-06-09 16:22:12.188896 7f868e75f700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6517> 2014-06-09 16:22:12.188901 7f868cf5c700 2 waiting 51 > 50 ops || 4516204 > 104857600 -6516> 2014-06-09 16:22:12.188905 7f868a757700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6515> 2014-06-09 16:22:12.188910 7f868ef60700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6514> 2014-06-09 16:22:12.188914 7f8690763700 2 waiting 51 > 50 ops || 4516204 > 104857600 -6513> 2014-06-09 16:22:12.188918 7f8690f64700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6512> 2014-06-09 16:22:12.188923 7f8692767700 2 waiting 51 > 50 ops || 4516201 > 104857600 -6511> 2014-06-09 16:22:12.188927 7f8691f66700 2 waiting 51 > 50 ops || 4516204 > 104857600 -6510> 2014-06-09 16:22:12.188931 7f8691765700 2 waiting 51 > 50 ops || 4516201 > 104857600 -6509> 2014-06-09 16:22:12.188936 7f8689755700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6508> 2014-06-09 16:22:12.188940 7f868d75d700 2 waiting 51 > 50 ops || 4516204 > 104857600 -6507> 2014-06-09 16:22:12.188945 7f8689f56700 2 waiting 51 > 50 ops || 4516204 > 104857600 -6506> 2014-06-09 16:22:12.188954 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd75440] -6505> 2014-06-09 16:22:12.188959 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd75440] -6504> 2014-06-09 16:22:12.188960 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6503> 2014-06-09 16:22:12.188961 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6502> 2014-06-09 16:22:12.188964 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4aca20 (1 active) -6501> 2014-06-09 16:22:12.188967 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300c60 (2 active) -6500> 2014-06-09 16:22:12.188973 7f868ff62700 10 op_submit_finish 4208 -6499> 2014-06-09 16:22:12.188978 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc29a3c0 243'664 rep_tid=30 committed?=0 applied?=0 lock=0 op=osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad -6498> 2014-06-09 16:22:12.189005 7f868ff62700 10 osd.26 244 dequeue_op 0x14053a00 finish -6497> 2014-06-09 16:22:12.189010 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -6496> 2014-06-09 16:22:12.189019 7f868ff62700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -6495> 2014-06-09 16:22:12.189023 7f868ff62700 10 osd.26 244 dequeue_op 0x306b800 prio 63 cost 475136 latency 2.269949 osd_op(client.4567.0:34713 rbd_data.11776b8b4567.0000000000001761 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.d6347f30 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] -6494> 2014-06-09 16:22:12.189049 7f868ff62700 5 -- op tracker -- , seq: 3, time: 2014-06-09 16:22:12.189044, event: reached_pg, op: osd_op(client.4567.0:34713 rbd_data.11776b8b4567.0000000000001761 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~475136] 3.d6347f30 RETRY=3 ack+ondisk+retry+write e242) -6493> 2014-06-09 16:22:12.189060 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -6492> 2014-06-09 16:22:12.189076 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0x306b800 -6491> 2014-06-09 16:22:12.189088 7f868ff62700 10 osd.26 244 dequeue_op 0x306b800 finish -6490> 2014-06-09 16:22:12.189092 7f868ff62700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -6489> 2014-06-09 16:22:12.189131 7f868ff62700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41be0 (20 active) -6488> 2014-06-09 16:22:12.189135 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -6487> 2014-06-09 16:22:12.189149 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -6486> 2014-06-09 16:22:12.189157 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -6485> 2014-06-09 16:22:12.189166 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -6484> 2014-06-09 16:22:12.189174 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -6483> 2014-06-09 16:22:12.189182 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -6482> 2014-06-09 16:22:12.189196 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 -6481> 2014-06-09 16:22:12.189205 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -6480> 2014-06-09 16:22:12.189214 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.623( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -6479> 2014-06-09 16:22:12.189223 7f868ff62700 10 log is not dirty -6478> 2014-06-09 16:22:12.189228 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.233( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) -6477> 2014-06-09 16:22:12.189266 7f868ff62700 10 log is not dirty -6476> 2014-06-09 16:22:12.189271 7f868ff62700 5 queue_transactions existing osr(1.233 0x384cc30)/0x384cc30 -6475> 2014-06-09 16:22:12.189274 7f868ff62700 2 waiting 51 > 50 ops || 4516171 > 104857600 -6474> 2014-06-09 16:22:12.189279 7f86a5e11700 5 _do_op 0x2e21c20 seq 4160 osr(0.616 0x2bd7950)/0x2bd7950 start -6473> 2014-06-09 16:22:12.189283 7f86a5e11700 10 _do_transaction on 0xbcb0b40 -6472> 2014-06-09 16:22:12.189288 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6471> 2014-06-09 16:22:12.189332 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6470> 2014-06-09 16:22:12.189347 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4160.0.0 <= header.spos 150443.0.0 -6469> 2014-06-09 16:22:12.189351 7f86a5e11700 5 submit_transaction r = 0 -6468> 2014-06-09 16:22:12.189367 7f86a5e11700 10 _do_op 0x2e21c20 seq 4160 r = 0, finisher 0x17001fa0 0 -6467> 2014-06-09 16:22:12.189378 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002400] -6466> 2014-06-09 16:22:12.189381 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002400] -6465> 2014-06-09 16:22:12.189382 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6464> 2014-06-09 16:22:12.189383 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6463> 2014-06-09 16:22:12.189387 7f86a5e11700 10 _finish_op 0x2e21c20 seq 4160 osr(0.616 0x2bd7950)/0x2bd7950 -6462> 2014-06-09 16:22:12.189397 7f868f761700 10 op_submit_start 4209 -6461> 2014-06-09 16:22:12.189399 7f868f761700 5 queue_transactions (trailing journal) 4209 -6460> 2014-06-09 16:22:12.189400 7f868f761700 5 queue_op 0x324e7f80 seq 4209 osr(1.ab 0x384c6f0) 826 bytes (queue has 50 ops and 4515341 bytes) -6459> 2014-06-09 16:22:12.189405 7f868df5e700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6458> 2014-06-09 16:22:12.189410 7f868bf5a700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6457> 2014-06-09 16:22:12.189414 7f868c75b700 2 waiting 51 > 50 ops || 4516202 > 104857600 -6456> 2014-06-09 16:22:12.189419 7f8692f68700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6455> 2014-06-09 16:22:12.189423 7f868b759700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6454> 2014-06-09 16:22:12.189428 7f868e75f700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6453> 2014-06-09 16:22:12.189432 7f868cf5c700 2 waiting 51 > 50 ops || 4516202 > 104857600 -6452> 2014-06-09 16:22:12.189436 7f868a757700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6451> 2014-06-09 16:22:12.189440 7f868ef60700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6450> 2014-06-09 16:22:12.189445 7f8690763700 2 waiting 51 > 50 ops || 4516202 > 104857600 -6449> 2014-06-09 16:22:12.189449 7f8690f64700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6448> 2014-06-09 16:22:12.189453 7f8692767700 2 waiting 51 > 50 ops || 4516199 > 104857600 -6447> 2014-06-09 16:22:12.189457 7f8691f66700 2 waiting 51 > 50 ops || 4516202 > 104857600 -6446> 2014-06-09 16:22:12.189461 7f8691765700 2 waiting 51 > 50 ops || 4516199 > 104857600 -6445> 2014-06-09 16:22:12.189466 7f8689755700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6444> 2014-06-09 16:22:12.189470 7f868d75d700 2 waiting 51 > 50 ops || 4516202 > 104857600 -6443> 2014-06-09 16:22:12.189474 7f8689f56700 2 waiting 51 > 50 ops || 4516202 > 104857600 -6442> 2014-06-09 16:22:12.189478 7f868ff62700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6441> 2014-06-09 16:22:12.189482 7f868af58700 2 waiting 51 > 50 ops || 4516202 > 104857600 -6440> 2014-06-09 16:22:12.189491 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17001fa0] -6439> 2014-06-09 16:22:12.189495 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17001fa0] -6438> 2014-06-09 16:22:12.189497 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6437> 2014-06-09 16:22:12.189498 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6436> 2014-06-09 16:22:12.189501 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300c60 (1 active) -6435> 2014-06-09 16:22:12.189504 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89e000 (2 active) -6434> 2014-06-09 16:22:12.189509 7f868f761700 10 op_submit_finish 4209 -6433> 2014-06-09 16:22:12.189513 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.b9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.b9( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -6432> 2014-06-09 16:22:12.189551 7f868f761700 10 log is not dirty -6431> 2014-06-09 16:22:12.189556 7f868f761700 5 queue_transactions existing osr(0.b9 0x2bd6400)/0x2bd6400 -6430> 2014-06-09 16:22:12.189559 7f868f761700 2 waiting 51 > 50 ops || 4516167 > 104857600 -6429> 2014-06-09 16:22:12.189563 7f86a5e11700 5 _do_op 0x230b9c80 seq 4161 osr(0.718 0x384c090)/0x384c090 start -6428> 2014-06-09 16:22:12.189567 7f86a5e11700 10 _do_transaction on 0x2fb6fc0 -6427> 2014-06-09 16:22:12.189573 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6426> 2014-06-09 16:22:12.189627 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6425> 2014-06-09 16:22:12.189644 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4161.0.0 <= header.spos 150443.0.0 -6424> 2014-06-09 16:22:12.189649 7f86a5e11700 5 submit_transaction r = 0 -6423> 2014-06-09 16:22:12.189662 7f86a5e11700 10 _do_op 0x230b9c80 seq 4161 r = 0, finisher 0x324f8ee0 0 -6422> 2014-06-09 16:22:12.189673 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050bc40] -6421> 2014-06-09 16:22:12.189676 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -6420> 2014-06-09 16:22:12.189688 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -6419> 2014-06-09 16:22:12.189698 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.718( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 -6418> 2014-06-09 16:22:12.189707 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050bc40] -6417> 2014-06-09 16:22:12.189709 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6416> 2014-06-09 16:22:12.189711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6415> 2014-06-09 16:22:12.189714 7f86a5e11700 10 _finish_op 0x230b9c80 seq 4161 osr(0.718 0x384c090)/0x384c090 -6414> 2014-06-09 16:22:12.189725 7f868bf5a700 10 op_submit_start 4210 -6413> 2014-06-09 16:22:12.189727 7f868bf5a700 5 queue_transactions (trailing journal) 4210 -6412> 2014-06-09 16:22:12.189729 7f868bf5a700 5 queue_op 0x230b9560 seq 4210 osr(3.527 0x3ed1660) 828 bytes (queue has 50 ops and 4515308 bytes) -6411> 2014-06-09 16:22:12.189734 7f868c75b700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6410> 2014-06-09 16:22:12.189739 7f8692f68700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6409> 2014-06-09 16:22:12.189743 7f868b759700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6408> 2014-06-09 16:22:12.189748 7f868e75f700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6407> 2014-06-09 16:22:12.189752 7f868cf5c700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6406> 2014-06-09 16:22:12.189756 7f868a757700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6405> 2014-06-09 16:22:12.189761 7f868ef60700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6404> 2014-06-09 16:22:12.189765 7f8690763700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6403> 2014-06-09 16:22:12.189769 7f8690f64700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6402> 2014-06-09 16:22:12.189774 7f8692767700 2 waiting 51 > 50 ops || 4516166 > 104857600 -6401> 2014-06-09 16:22:12.189778 7f8691f66700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6400> 2014-06-09 16:22:12.189783 7f8691765700 2 waiting 51 > 50 ops || 4516166 > 104857600 -6399> 2014-06-09 16:22:12.189787 7f8689755700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6398> 2014-06-09 16:22:12.189791 7f868d75d700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6397> 2014-06-09 16:22:12.189795 7f8689f56700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6396> 2014-06-09 16:22:12.189800 7f868ff62700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6395> 2014-06-09 16:22:12.189804 7f868af58700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6394> 2014-06-09 16:22:12.189808 7f868f761700 2 waiting 51 > 50 ops || 4516134 > 104857600 -6393> 2014-06-09 16:22:12.189813 7f868df5e700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6392> 2014-06-09 16:22:12.189823 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324f8ee0] -6391> 2014-06-09 16:22:12.189830 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324f8ee0] -6390> 2014-06-09 16:22:12.189832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6389> 2014-06-09 16:22:12.189833 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6388> 2014-06-09 16:22:12.189837 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89e000 (1 active) -6387> 2014-06-09 16:22:12.189840 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb440 (2 active) -6386> 2014-06-09 16:22:12.189845 7f868bf5a700 10 op_submit_finish 4210 -6385> 2014-06-09 16:22:12.189849 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -6384> 2014-06-09 16:22:12.189863 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local -6383> 2014-06-09 16:22:12.189872 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -6382> 2014-06-09 16:22:12.189881 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -6381> 2014-06-09 16:22:12.189890 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -6380> 2014-06-09 16:22:12.189898 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -6379> 2014-06-09 16:22:12.189912 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:172 -6378> 2014-06-09 16:22:12.189921 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -6377> 2014-06-09 16:22:12.189929 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[1.685( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop -6376> 2014-06-09 16:22:12.189939 7f868bf5a700 10 log is not dirty -6375> 2014-06-09 16:22:12.189943 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.2d3( v 213'548 (0'0,213'548] local-les=243 n=107 ec=118 les/c 243/243 242/242/223) -6374> 2014-06-09 16:22:12.189977 7f868bf5a700 10 log is not dirty -6373> 2014-06-09 16:22:12.189982 7f868bf5a700 5 queue_transactions existing osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 -6372> 2014-06-09 16:22:12.189984 7f868bf5a700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6371> 2014-06-09 16:22:12.189989 7f86a5e11700 5 _do_op 0x3280ac00 seq 4162 osr(3.35c 0x3ed10c0)/0x3ed10c0 start -6370> 2014-06-09 16:22:12.189993 7f86a5e11700 10 _do_transaction on 0x35629c0 -6369> 2014-06-09 16:22:12.189998 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6368> 2014-06-09 16:22:12.190040 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6367> 2014-06-09 16:22:12.190055 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4162.0.0 <= header.spos 150443.0.0 -6366> 2014-06-09 16:22:12.190059 7f86a5e11700 5 submit_transaction r = 0 -6365> 2014-06-09 16:22:12.190142 7f86a5e11700 10 _do_op 0x3280ac00 seq 4162 r = 0, finisher 0x281b6600 0 -6364> 2014-06-09 16:22:12.190153 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3251c7a0] -6363> 2014-06-09 16:22:12.190156 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3251c7a0] -6362> 2014-06-09 16:22:12.190157 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6361> 2014-06-09 16:22:12.190159 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6360> 2014-06-09 16:22:12.190162 7f86a5e11700 10 _finish_op 0x3280ac00 seq 4162 osr(3.35c 0x3ed10c0)/0x3ed10c0 -6359> 2014-06-09 16:22:12.190173 7f8692f68700 10 op_submit_start 4211 -6358> 2014-06-09 16:22:12.190175 7f8692f68700 5 queue_transactions (trailing journal) 4211 -6357> 2014-06-09 16:22:12.190176 7f8692f68700 5 queue_op 0xd327aa0 seq 4211 osr(2.24d 0x2ed0a50) 828 bytes (queue has 50 ops and 4515308 bytes) -6356> 2014-06-09 16:22:12.190181 7f868b759700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6355> 2014-06-09 16:22:12.190184 7f868e75f700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6354> 2014-06-09 16:22:12.190188 7f868cf5c700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6353> 2014-06-09 16:22:12.190191 7f868a757700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6352> 2014-06-09 16:22:12.190194 7f868ef60700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6351> 2014-06-09 16:22:12.190197 7f8690763700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6350> 2014-06-09 16:22:12.190200 7f8690f64700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6349> 2014-06-09 16:22:12.190203 7f8692767700 2 waiting 51 > 50 ops || 4516166 > 104857600 -6348> 2014-06-09 16:22:12.190207 7f8691f66700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6347> 2014-06-09 16:22:12.190210 7f8691765700 2 waiting 51 > 50 ops || 4516166 > 104857600 -6346> 2014-06-09 16:22:12.190213 7f8689755700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6345> 2014-06-09 16:22:12.190216 7f868d75d700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6344> 2014-06-09 16:22:12.190219 7f8689f56700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6343> 2014-06-09 16:22:12.190222 7f868ff62700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6342> 2014-06-09 16:22:12.190226 7f868af58700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6341> 2014-06-09 16:22:12.190229 7f868f761700 2 waiting 51 > 50 ops || 4516134 > 104857600 -6340> 2014-06-09 16:22:12.190232 7f868df5e700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6339> 2014-06-09 16:22:12.190235 7f868bf5a700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6338> 2014-06-09 16:22:12.190238 7f868c75b700 2 waiting 51 > 50 ops || 4516169 > 104857600 -6337> 2014-06-09 16:22:12.190246 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x281b6600] -6336> 2014-06-09 16:22:12.190252 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x281b6600] -6335> 2014-06-09 16:22:12.190253 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6334> 2014-06-09 16:22:12.190254 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6333> 2014-06-09 16:22:12.190257 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb440 (1 active) -6332> 2014-06-09 16:22:12.190259 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad200 (2 active) -6331> 2014-06-09 16:22:12.190263 7f8692f68700 10 op_submit_finish 4211 -6330> 2014-06-09 16:22:12.190266 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.508( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.508( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) -6329> 2014-06-09 16:22:12.190297 7f8692f68700 10 log is not dirty -6328> 2014-06-09 16:22:12.190302 7f8692f68700 5 queue_transactions existing osr(1.508 0x384db90)/0x384db90 -6327> 2014-06-09 16:22:12.190304 7f8692f68700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6326> 2014-06-09 16:22:12.190307 7f86a5e11700 5 _do_op 0xc64a360 seq 4163 osr(1.2e0 0x384d130)/0x384d130 start -6325> 2014-06-09 16:22:12.190310 7f86a5e11700 10 _do_transaction on 0xbfcc900 -6324> 2014-06-09 16:22:12.190316 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6323> 2014-06-09 16:22:12.190352 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6322> 2014-06-09 16:22:12.190367 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4163.0.0 <= header.spos 150443.0.0 -6321> 2014-06-09 16:22:12.190371 7f86a5e11700 5 submit_transaction r = 0 -6320> 2014-06-09 16:22:12.190385 7f86a5e11700 10 _do_op 0xc64a360 seq 4163 r = 0, finisher 0x189aacc0 0 -6319> 2014-06-09 16:22:12.190396 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3196c260] -6318> 2014-06-09 16:22:12.190399 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -6317> 2014-06-09 16:22:12.190409 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -6316> 2014-06-09 16:22:12.190417 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.2e0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:164 -6315> 2014-06-09 16:22:12.190430 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3196c260] -6314> 2014-06-09 16:22:12.190432 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6313> 2014-06-09 16:22:12.190433 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6312> 2014-06-09 16:22:12.190437 7f86a5e11700 10 _finish_op 0xc64a360 seq 4163 osr(1.2e0 0x384d130)/0x384d130 -6311> 2014-06-09 16:22:12.190448 7f868e75f700 10 op_submit_start 4212 -6310> 2014-06-09 16:22:12.190450 7f868e75f700 5 queue_transactions (trailing journal) 4212 -6309> 2014-06-09 16:22:12.190451 7f868e75f700 5 queue_op 0x324f49c0 seq 4212 osr(0.473 0x2bd74d0) 828 bytes (queue has 50 ops and 4515275 bytes) -6308> 2014-06-09 16:22:12.190456 7f868cf5c700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6307> 2014-06-09 16:22:12.190461 7f868a757700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6306> 2014-06-09 16:22:12.190465 7f868ef60700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6305> 2014-06-09 16:22:12.190470 7f8690763700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6304> 2014-06-09 16:22:12.190474 7f8690f64700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6303> 2014-06-09 16:22:12.190479 7f8692767700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6302> 2014-06-09 16:22:12.190483 7f8691f66700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6301> 2014-06-09 16:22:12.190488 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6300> 2014-06-09 16:22:12.190492 7f8689755700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6299> 2014-06-09 16:22:12.190496 7f868d75d700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6298> 2014-06-09 16:22:12.190501 7f8689f56700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6297> 2014-06-09 16:22:12.190505 7f868ff62700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6296> 2014-06-09 16:22:12.190510 7f868af58700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6295> 2014-06-09 16:22:12.190514 7f868f761700 2 waiting 51 > 50 ops || 4516101 > 104857600 -6294> 2014-06-09 16:22:12.190518 7f868df5e700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6293> 2014-06-09 16:22:12.190522 7f868bf5a700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6292> 2014-06-09 16:22:12.190526 7f868c75b700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6291> 2014-06-09 16:22:12.190530 7f8692f68700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6290> 2014-06-09 16:22:12.190534 7f868b759700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6289> 2014-06-09 16:22:12.190544 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x189aacc0] -6288> 2014-06-09 16:22:12.190549 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x189aacc0] -6287> 2014-06-09 16:22:12.190553 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6286> 2014-06-09 16:22:12.190555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6285> 2014-06-09 16:22:12.190558 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad200 (1 active) -6284> 2014-06-09 16:22:12.190561 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a6c0 (2 active) -6283> 2014-06-09 16:22:12.190566 7f868e75f700 10 op_submit_finish 4212 -6282> 2014-06-09 16:22:12.190570 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.5f2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.5f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) -6281> 2014-06-09 16:22:12.190604 7f868e75f700 10 log is not dirty -6280> 2014-06-09 16:22:12.190610 7f868e75f700 5 queue_transactions existing osr(2.5f2 0x2ed1df0)/0x2ed1df0 -6279> 2014-06-09 16:22:12.190613 7f868e75f700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6278> 2014-06-09 16:22:12.190617 7f86a5e11700 5 _do_op 0xd471020 seq 4164 osr(3.1d5 0x3ed09f0)/0x3ed09f0 start -6277> 2014-06-09 16:22:12.190621 7f86a5e11700 10 _do_transaction on 0xc14b980 -6276> 2014-06-09 16:22:12.190626 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6275> 2014-06-09 16:22:12.190668 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6274> 2014-06-09 16:22:12.190682 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4164.0.0 <= header.spos 150443.0.0 -6273> 2014-06-09 16:22:12.190686 7f86a5e11700 5 submit_transaction r = 0 -6272> 2014-06-09 16:22:12.190701 7f86a5e11700 10 _do_op 0xd471020 seq 4164 r = 0, finisher 0x307f0a40 0 -6271> 2014-06-09 16:22:12.190712 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f0ac0] -6270> 2014-06-09 16:22:12.190715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f0ac0] -6269> 2014-06-09 16:22:12.190716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6268> 2014-06-09 16:22:12.190717 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6267> 2014-06-09 16:22:12.190721 7f86a5e11700 10 _finish_op 0xd471020 seq 4164 osr(3.1d5 0x3ed09f0)/0x3ed09f0 -6266> 2014-06-09 16:22:12.190732 7f868a757700 10 op_submit_start 4213 -6265> 2014-06-09 16:22:12.190733 7f868a757700 5 queue_transactions (trailing journal) 4213 -6264> 2014-06-09 16:22:12.190735 7f868a757700 5 queue_op 0xcabde00 seq 4213 osr(0.4ba 0x2bd7710) 828 bytes (queue has 50 ops and 4515275 bytes) -6263> 2014-06-09 16:22:12.190739 7f868ef60700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6262> 2014-06-09 16:22:12.190744 7f8690763700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6261> 2014-06-09 16:22:12.190748 7f8690f64700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6260> 2014-06-09 16:22:12.190752 7f8692767700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6259> 2014-06-09 16:22:12.190757 7f8691f66700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6258> 2014-06-09 16:22:12.190761 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6257> 2014-06-09 16:22:12.190765 7f8689755700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6256> 2014-06-09 16:22:12.190770 7f868d75d700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6255> 2014-06-09 16:22:12.190774 7f8689f56700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6254> 2014-06-09 16:22:12.190778 7f868ff62700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6253> 2014-06-09 16:22:12.190782 7f868af58700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6252> 2014-06-09 16:22:12.190787 7f868f761700 2 waiting 51 > 50 ops || 4516101 > 104857600 -6251> 2014-06-09 16:22:12.190791 7f868df5e700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6250> 2014-06-09 16:22:12.190795 7f868bf5a700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6249> 2014-06-09 16:22:12.190799 7f868c75b700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6248> 2014-06-09 16:22:12.190804 7f8692f68700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6247> 2014-06-09 16:22:12.190808 7f868b759700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6246> 2014-06-09 16:22:12.190812 7f868e75f700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6245> 2014-06-09 16:22:12.190816 7f868cf5c700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6244> 2014-06-09 16:22:12.190826 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f0a40] -6243> 2014-06-09 16:22:12.190830 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f0a40] -6242> 2014-06-09 16:22:12.190831 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6241> 2014-06-09 16:22:12.190832 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6240> 2014-06-09 16:22:12.190835 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a6c0 (1 active) -6239> 2014-06-09 16:22:12.190838 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286480 (2 active) -6238> 2014-06-09 16:22:12.190843 7f868a757700 10 op_submit_finish 4213 -6237> 2014-06-09 16:22:12.190866 7f868a757700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcd8e340 (19 active) -6236> 2014-06-09 16:22:12.190873 7f868a757700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -6235> 2014-06-09 16:22:12.190877 7f868a757700 10 osd.26 244 dequeue_op 0xe2ad300 prio 63 cost 385024 latency 2.160916 osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] -6234> 2014-06-09 16:22:12.190907 7f868a757700 5 -- op tracker -- , seq: 55, time: 2014-06-09 16:22:12.190901, event: reached_pg, op: osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) -6233> 2014-06-09 16:22:12.190922 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -6232> 2014-06-09 16:22:12.190939 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] handle_message: 0xe2ad300 -6231> 2014-06-09 16:22:12.190952 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] do_op osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write -6230> 2014-06-09 16:22:12.190975 7f868a757700 3 osd.26 pg_epoch: 243 pg[3.c1( v 214'609 lc 214'607 (0'0,214'609] local-les=243 n=123 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'609 lcod 0'0 mlcod 0'0 active m=2] do_op dup client.4568.0:34735 was 213'599 -6229> 2014-06-09 16:22:12.190991 7f868a757700 1 -- 172.16.96.202:6830/30904 --> 172.16.96.220:0/1127482 -- osd_op_reply(34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] v213'599 uv599 ondisk = 0) v6 -- ?+0 0x30bbe500 con 0xcd61b80 -6228> 2014-06-09 16:22:12.191008 7f868a757700 10 osd.26 244 dequeue_op 0xe2ad300 finish -6227> 2014-06-09 16:22:12.191885 7f868a757700 5 -- op tracker -- , seq: 55, time: 2014-06-09 16:22:12.191011, event: done, op: osd_op(client.4568.0:34735 rbd_data.11786b8b4567.00000000000016c7 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~385024] 3.a8c948c1 RETRY=3 ack+ondisk+retry+write e242) -6226> 2014-06-09 16:22:12.191910 7f868a757700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -6225> 2014-06-09 16:22:12.191940 7f868a757700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e340 (20 active) -6224> 2014-06-09 16:22:12.191944 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.2c0( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.2c0( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) -6223> 2014-06-09 16:22:12.191981 7f868a757700 10 log is not dirty -6222> 2014-06-09 16:22:12.191986 7f868a757700 5 queue_transactions existing osr(1.2c0 0x384d170)/0x384d170 -6221> 2014-06-09 16:22:12.192047 7f868a757700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6220> 2014-06-09 16:22:12.190911 7f86a5e11700 5 _do_op 0x24105ce0 seq 4165 osr(2.25c 0x2ed0a30)/0x2ed0a30 start -6219> 2014-06-09 16:22:12.192054 7f86a5e11700 10 _do_transaction on 0x394e240 -6218> 2014-06-09 16:22:12.192062 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6217> 2014-06-09 16:22:12.192109 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6216> 2014-06-09 16:22:12.192128 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4165.0.0 <= header.spos 150443.0.0 -6215> 2014-06-09 16:22:12.192132 7f86a5e11700 5 submit_transaction r = 0 -6214> 2014-06-09 16:22:12.192149 7f86a5e11700 10 _do_op 0x24105ce0 seq 4165 r = 0, finisher 0xc6e8ca0 0 -6213> 2014-06-09 16:22:12.192161 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078800] -6212> 2014-06-09 16:22:12.192163 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -6211> 2014-06-09 16:22:12.192174 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -6210> 2014-06-09 16:22:12.192184 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.25c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 -6209> 2014-06-09 16:22:12.192193 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078800] -6208> 2014-06-09 16:22:12.192195 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6207> 2014-06-09 16:22:12.192196 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6206> 2014-06-09 16:22:12.192200 7f86a5e11700 10 _finish_op 0x24105ce0 seq 4165 osr(2.25c 0x2ed0a30)/0x2ed0a30 -6205> 2014-06-09 16:22:12.192212 7f8690763700 10 op_submit_start 4214 -6204> 2014-06-09 16:22:12.192214 7f8690763700 5 queue_transactions (trailing journal) 4214 -6203> 2014-06-09 16:22:12.192215 7f8690763700 5 queue_op 0xc2d4d80 seq 4214 osr(2.754 0x3ed00f0) 861 bytes (queue has 50 ops and 4515275 bytes) -6202> 2014-06-09 16:22:12.192221 7f8690f64700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6201> 2014-06-09 16:22:12.192225 7f8692767700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6200> 2014-06-09 16:22:12.192230 7f8691f66700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6199> 2014-06-09 16:22:12.192235 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6198> 2014-06-09 16:22:12.192239 7f8689755700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6197> 2014-06-09 16:22:12.192243 7f868d75d700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6196> 2014-06-09 16:22:12.192248 7f8689f56700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6195> 2014-06-09 16:22:12.192252 7f868ff62700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6194> 2014-06-09 16:22:12.192257 7f868af58700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6193> 2014-06-09 16:22:12.192261 7f868f761700 2 waiting 51 > 50 ops || 4516101 > 104857600 -6192> 2014-06-09 16:22:12.192266 7f868df5e700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6191> 2014-06-09 16:22:12.192270 7f868bf5a700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6190> 2014-06-09 16:22:12.192274 7f868c75b700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6189> 2014-06-09 16:22:12.192279 7f8692f68700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6188> 2014-06-09 16:22:12.192283 7f868b759700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6187> 2014-06-09 16:22:12.192287 7f868e75f700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6186> 2014-06-09 16:22:12.192292 7f868cf5c700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6185> 2014-06-09 16:22:12.192296 7f868a757700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6184> 2014-06-09 16:22:12.192300 7f868ef60700 2 waiting 51 > 50 ops || 4516103 > 104857600 -6183> 2014-06-09 16:22:12.192310 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e8ca0] -6182> 2014-06-09 16:22:12.192319 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e8ca0] -6181> 2014-06-09 16:22:12.192321 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6180> 2014-06-09 16:22:12.192322 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6179> 2014-06-09 16:22:12.192325 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286480 (1 active) -6178> 2014-06-09 16:22:12.192328 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5245a0 (2 active) -6177> 2014-06-09 16:22:12.192333 7f8690763700 10 op_submit_finish 4214 -6176> 2014-06-09 16:22:12.192337 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -6175> 2014-06-09 16:22:12.192355 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.70e( v 213'600 (0'0,213'600] local-les=243 n=118 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -6174> 2014-06-09 16:22:12.192368 7f8690763700 10 log is not dirty -6173> 2014-06-09 16:22:12.192388 7f8690763700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41810 (19 active) -6172> 2014-06-09 16:22:12.192396 7f8690763700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -6171> 2014-06-09 16:22:12.192400 7f8690763700 10 osd.26 244 dequeue_op 0xe032700 prio 63 cost 1015808 latency 2.219964 osd_op(client.4569.0:38360 rbd_data.11796b8b4567.0000000000001a6e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1015808] 3.9f12c15c RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] -6170> 2014-06-09 16:22:12.192424 7f8690763700 5 -- op tracker -- , seq: 28, time: 2014-06-09 16:22:12.192419, event: reached_pg, op: osd_op(client.4569.0:38360 rbd_data.11796b8b4567.0000000000001a6e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1015808] 3.9f12c15c RETRY=3 ack+ondisk+retry+write e242) -6169> 2014-06-09 16:22:12.192434 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -6168> 2014-06-09 16:22:12.192447 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.15c( v 214'638 (0'0,214'638] local-les=243 n=126 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0xe032700 -6167> 2014-06-09 16:22:12.192457 7f8690763700 10 osd.26 244 dequeue_op 0xe032700 finish -6166> 2014-06-09 16:22:12.192461 7f8690763700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -6165> 2014-06-09 16:22:12.192483 7f8690763700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41810 (20 active) -6164> 2014-06-09 16:22:12.192487 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -6163> 2014-06-09 16:22:12.192487 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 14 ==== osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+1548288 (340935203 0 0) 0x32d81b00 con 0x30c3c2c0 -6162> 2014-06-09 16:22:12.192501 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.076281 7 0.000431 -6161> 2014-06-09 16:22:12.192512 7f86a68f6700 5 -- op tracker -- , seq: 3676, time: 2014-06-09 16:22:12.188802, event: header_read, op: osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6160> 2014-06-09 16:22:12.192514 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -6159> 2014-06-09 16:22:12.192524 7f86a68f6700 5 -- op tracker -- , seq: 3676, time: 2014-06-09 16:22:12.188803, event: throttled, op: osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6158> 2014-06-09 16:22:12.192525 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -6157> 2014-06-09 16:22:12.192534 7f86a68f6700 5 -- op tracker -- , seq: 3676, time: 2014-06-09 16:22:12.192396, event: all_read, op: osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6156> 2014-06-09 16:22:12.192537 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -6155> 2014-06-09 16:22:12.192547 7f86a68f6700 5 -- op tracker -- , seq: 3676, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6154> 2014-06-09 16:22:12.192548 7f8690763700 5 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -6153> 2014-06-09 16:22:12.192553 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -6152> 2014-06-09 16:22:12.192558 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -6151> 2014-06-09 16:22:12.192558 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -6150> 2014-06-09 16:22:12.192566 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -6149> 2014-06-09 16:22:12.192578 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976ce00 prio 63 cost 1548288 latency 0.003776 osd_op(client.4543.0:33380 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 249856~1548288] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 -6148> 2014-06-09 16:22:12.192577 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) -6147> 2014-06-09 16:22:12.192588 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) -6146> 2014-06-09 16:22:12.192597 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) -6145> 2014-06-09 16:22:12.192607 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) -6144> 2014-06-09 16:22:12.192617 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -6143> 2014-06-09 16:22:12.192633 7f8690763700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xc13cdc0 172.16.96.201:6811/57862 -6142> 2014-06-09 16:22:12.192645 7f8690763700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6811/57862 -- osd_map(244..244 src has 1..244) v3 -- ?+0 0x4682000 con 0xc13cdc0 -6141> 2014-06-09 16:22:12.192659 7f8690763700 10 osd.26 243 note_peer_epoch osd.1 has 244 -6140> 2014-06-09 16:22:12.192662 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.1 172.16.96.201:6811/57862 -- pg_info(1 pgs e243:0.686) v4 -- ?+0 0x3343ddc0 -6139> 2014-06-09 16:22:12.192671 7f8690763700 15 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 -6138> 2014-06-09 16:22:12.192707 7f8690763700 10 log is not dirty -6137> 2014-06-09 16:22:12.192711 7f8690763700 5 queue_transactions existing osr(0.686 0x2bd7e20)/0x2bd7e20 -6136> 2014-06-09 16:22:12.192713 7f8690763700 2 waiting 51 > 50 ops || 4516136 > 104857600 -6135> 2014-06-09 16:22:12.192719 7f86a5e11700 5 _do_op 0x308ca000 seq 4166 osr(2.3fb 0x2ed12b0)/0x2ed12b0 start -6134> 2014-06-09 16:22:12.192723 7f86a5e11700 10 _do_transaction on 0xbe880c0 -6133> 2014-06-09 16:22:12.192739 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6132> 2014-06-09 16:22:12.192786 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6131> 2014-06-09 16:22:12.192803 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4166.0.0 <= header.spos 150443.0.0 -6130> 2014-06-09 16:22:12.192808 7f86a5e11700 5 submit_transaction r = 0 -6129> 2014-06-09 16:22:12.192830 7f86a5e11700 10 _do_op 0x308ca000 seq 4166 r = 0, finisher 0x1322d9c0 0 -6128> 2014-06-09 16:22:12.192842 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5a260] -6127> 2014-06-09 16:22:12.192845 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -6126> 2014-06-09 16:22:12.192856 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -6125> 2014-06-09 16:22:12.192865 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3fb( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 -6124> 2014-06-09 16:22:12.192877 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5a260] -6123> 2014-06-09 16:22:12.192879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6122> 2014-06-09 16:22:12.192880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6121> 2014-06-09 16:22:12.192883 7f86a5e11700 10 _finish_op 0x308ca000 seq 4166 osr(2.3fb 0x2ed12b0)/0x2ed12b0 -6120> 2014-06-09 16:22:12.192895 7f8692767700 10 op_submit_start 4215 -6119> 2014-06-09 16:22:12.192896 7f8692767700 5 queue_transactions (trailing journal) 4215 -6118> 2014-06-09 16:22:12.192898 7f8692767700 5 queue_op 0x2fcafc20 seq 4215 osr(0.6e 0x2bd6390) 858 bytes (queue has 50 ops and 4515272 bytes) -6117> 2014-06-09 16:22:12.192903 7f8691f66700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6116> 2014-06-09 16:22:12.192908 7f8691765700 2 waiting 51 > 50 ops || 4516130 > 104857600 -6115> 2014-06-09 16:22:12.192912 7f8689755700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6114> 2014-06-09 16:22:12.192916 7f868d75d700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6113> 2014-06-09 16:22:12.192921 7f8689f56700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6112> 2014-06-09 16:22:12.192925 7f868ff62700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6111> 2014-06-09 16:22:12.192929 7f868af58700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6110> 2014-06-09 16:22:12.192934 7f868f761700 2 waiting 51 > 50 ops || 4516098 > 104857600 -6109> 2014-06-09 16:22:12.192938 7f868df5e700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6108> 2014-06-09 16:22:12.192942 7f868bf5a700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6107> 2014-06-09 16:22:12.192947 7f868c75b700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6106> 2014-06-09 16:22:12.192952 7f8692f68700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6105> 2014-06-09 16:22:12.192956 7f868b759700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6104> 2014-06-09 16:22:12.192960 7f868e75f700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6103> 2014-06-09 16:22:12.192964 7f868cf5c700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6102> 2014-06-09 16:22:12.192968 7f868a757700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6101> 2014-06-09 16:22:12.192973 7f868ef60700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6100> 2014-06-09 16:22:12.192977 7f8690763700 2 waiting 51 > 50 ops || 4516133 > 104857600 -6099> 2014-06-09 16:22:12.192981 7f8690f64700 2 waiting 51 > 50 ops || 4516100 > 104857600 -6098> 2014-06-09 16:22:12.192991 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322d9c0] -6097> 2014-06-09 16:22:12.192996 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322d9c0] -6096> 2014-06-09 16:22:12.192997 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -6095> 2014-06-09 16:22:12.192998 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -6094> 2014-06-09 16:22:12.193001 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5245a0 (1 active) -6093> 2014-06-09 16:22:12.193004 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286000 (2 active) -6092> 2014-06-09 16:22:12.193009 7f8692767700 10 op_submit_finish 4215 -6091> 2014-06-09 16:22:12.193032 7f8692767700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x153591c0 (19 active) -6090> 2014-06-09 16:22:12.193039 7f8692767700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -6089> 2014-06-09 16:22:12.193043 7f8692767700 10 osd.26 244 dequeue_op 0x230da800 prio 63 cost 344064 latency 1.502615 osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] -6088> 2014-06-09 16:22:12.193067 7f8692767700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.193062, event: reached_pg, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) -6087> 2014-06-09 16:22:12.193075 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -6086> 2014-06-09 16:22:12.193090 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x230da800 -6085> 2014-06-09 16:22:12.193100 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write -6084> 2014-06-09 16:22:12.193124 7f8692767700 15 getattr 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 '_' -6083> 2014-06-09 16:22:12.193462 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 15 ==== osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+516096 (1856534306 0 0) 0x344d6d80 con 0x30c3c2c0 -6082> 2014-06-09 16:22:12.193483 7f86a68f6700 5 -- op tracker -- , seq: 3677, time: 2014-06-09 16:22:12.192601, event: header_read, op: osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6081> 2014-06-09 16:22:12.193495 7f86a68f6700 5 -- op tracker -- , seq: 3677, time: 2014-06-09 16:22:12.192602, event: throttled, op: osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6080> 2014-06-09 16:22:12.193504 7f86a68f6700 5 -- op tracker -- , seq: 3677, time: 2014-06-09 16:22:12.193158, event: all_read, op: osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6079> 2014-06-09 16:22:12.193514 7f86a68f6700 5 -- op tracker -- , seq: 3677, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -6078> 2014-06-09 16:22:12.193519 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -6077> 2014-06-09 16:22:12.193524 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -6076> 2014-06-09 16:22:12.193537 7f86a68f6700 15 osd.26 244 enqueue_op 0x32826500 prio 63 cost 516096 latency 0.000936 osd_op(client.4543.0:33381 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 1798144~516096] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 -6075> 2014-06-09 16:22:12.193774 7f8692767700 10 getattr got.size() is 0 -6074> 2014-06-09 16:22:12.193777 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 but can_create -6073> 2014-06-09 16:22:12.193793 7f8692767700 15 getattr 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 'snapset' -6072> 2014-06-09 16:22:12.193829 7f8692767700 10 getattr got.size() is 0 -6071> 2014-06-09 16:22:12.193831 7f8692767700 15 getattr 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/snapdir//3 'snapset' -6070> 2014-06-09 16:22:12.193862 7f8692767700 10 getattr got.size() is 0 -6069> 2014-06-09 16:22:12.193872 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x3042c2c0 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -6068> 2014-06-09 16:22:12.193885 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -6067> 2014-06-09 16:22:12.193896 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -6066> 2014-06-09 16:22:12.193973 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x3042c2c0 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 rwstate(none n=0 w=0) oi: 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd5c00 snapset: 0=[]:[] -6065> 2014-06-09 16:22:12.193990 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 @head oi=888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) -6064> 2014-06-09 16:22:12.194013 7f8692767700 15 getattr 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/snapdir//3 '_' -6063> 2014-06-09 16:22:12.194048 7f8692767700 10 getattr got.size() is 0 -6062> 2014-06-09 16:22:12.194051 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/snapdir//3 and !can_create -6061> 2014-06-09 16:22:12.194072 7f8692767700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.194065, event: started, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) -6060> 2014-06-09 16:22:12.194079 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x308a9600 -6059> 2014-06-09 16:22:12.194097 7f8692767700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.194092, event: started, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) -6058> 2014-06-09 16:22:12.194104 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] ov 0'0 av 243'576 snapc 0=[] snapset 0=[]:[] -6057> 2014-06-09 16:22:12.194119 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] -6056> 2014-06-09 16:22:12.194130 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 -6055> 2014-06-09 16:22:12.194146 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~344064 -6054> 2014-06-09 16:22:12.194165 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd228a00 -6053> 2014-06-09 16:22:12.194177 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 snapset=0x30dd5c38 snapc=0=[] -6052> 2014-06-09 16:22:12.194188 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag -6051> 2014-06-09 16:22:12.194199 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 done, snapset=0=[]:[]+head -6050> 2014-06-09 16:22:12.194210 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 0x308a9600 op modify -6049> 2014-06-09 16:22:12.194224 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:19:32.257474 -6048> 2014-06-09 16:22:12.194245 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=108 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -6047> 2014-06-09 16:22:12.194266 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 -6046> 2014-06-09 16:22:12.194277 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 31 on osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) v4 -6045> 2014-06-09 16:22:12.194293 7f8692767700 7 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 31 o 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -6044> 2014-06-09 16:22:12.194318 7f8692767700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.194312, event: waiting for subops from 8, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) -6043> 2014-06-09 16:22:12.194339 7f8692767700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -6042> 2014-06-09 16:22:12.194342 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- osd_sub_op(client.4570.0:34004 3.56a 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 [] v 243'576 snapset=0=[]:[] snapc=0=[]) v10 -- ?+344976 0x1921cd00 -6041> 2014-06-09 16:22:12.194360 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 214'575 (0'0,214'575] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'575], crt=0'0) [243'576 (0'0) modify 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 by client.4570.0:34004 2014-06-09 10:19:32.257474] -6040> 2014-06-09 16:22:12.194384 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'576 (0'0) modify 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 by client.4570.0:34004 2014-06-09 10:19:32.257474 -6039> 2014-06-09 16:22:12.194404 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys -6038> 2014-06-09 16:22:12.194436 7f8692767700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'576, trimmed: -6037> 2014-06-09 16:22:12.194458 7f8692767700 5 queue_transactions existing osr(3.56a 0x3ed1900)/0x3ed1900 -6036> 2014-06-09 16:22:12.194461 7f8692767700 2 waiting 51 > 50 ops || 4861734 > 104857600 -6035> 2014-06-09 16:22:12.194708 7f86a5e11700 5 _do_op 0xc705380 seq 4167 osr(3.647 0x3ed1a70)/0x3ed1a70 start -6034> 2014-06-09 16:22:12.194714 7f86a5e11700 10 _do_transaction on 0xc0155c0 -6033> 2014-06-09 16:22:12.194724 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -6032> 2014-06-09 16:22:12.194778 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -6031> 2014-06-09 16:22:12.194798 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4167.0.0 <= header.spos 150443.0.0 -6030> 2014-06-09 16:22:12.194802 7f86a5e11700 5 submit_transaction r = 0 -6029> 2014-06-09 16:22:12.194846 7f86a5e11700 10 _do_op 0xc705380 seq 4167 r = 0, finisher 0xbd6f340 0 -6028> 2014-06-09 16:22:12.194860 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd7c100] -6027> 2014-06-09 16:22:12.194863 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] _finish_recovery -6026> 2014-06-09 16:22:12.194876 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] purge_strays -6025> 2014-06-09 16:22:12.194888 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.647( v 243'592 (0'0,243'592] local-les=243 n=122 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 214'591 mlcod 214'591 active+clean] publish_stats_to_osd 243:979 -6024> 2014-06-09 16:22:12.194898 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd7c100] -6023> 2014-06-09 16:22:12.194900 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -6022> 2014-06-09 16:22:12.194901 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -6021> 2014-06-09 16:22:12.194905 7f86a5e11700 10 _finish_op 0xc705380 seq 4167 osr(3.647 0x3ed1a70)/0x3ed1a70 -6020> 2014-06-09 16:22:12.194919 7f8691765700 10 op_submit_start 4216 -6019> 2014-06-09 16:22:12.194921 7f8691765700 5 queue_transactions (trailing journal) 4216 -6018> 2014-06-09 16:22:12.194922 7f8691765700 5 queue_op 0xd4715c0 seq 4216 osr(1.b8 0x384c650) 858 bytes (queue has 50 ops and 4515269 bytes) -6017> 2014-06-09 16:22:12.194928 7f8689755700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6016> 2014-06-09 16:22:12.194934 7f868d75d700 2 waiting 51 > 50 ops || 4516130 > 104857600 -6015> 2014-06-09 16:22:12.194939 7f8689f56700 2 waiting 51 > 50 ops || 4516130 > 104857600 -6014> 2014-06-09 16:22:12.194945 7f868ff62700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6013> 2014-06-09 16:22:12.194950 7f868af58700 2 waiting 51 > 50 ops || 4516130 > 104857600 -6012> 2014-06-09 16:22:12.194955 7f868f761700 2 waiting 51 > 50 ops || 4516095 > 104857600 -6011> 2014-06-09 16:22:12.194960 7f868df5e700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6010> 2014-06-09 16:22:12.194965 7f868bf5a700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6009> 2014-06-09 16:22:12.194969 7f868c75b700 2 waiting 51 > 50 ops || 4516130 > 104857600 -6008> 2014-06-09 16:22:12.194974 7f8692f68700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6007> 2014-06-09 16:22:12.194979 7f868b759700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6006> 2014-06-09 16:22:12.194984 7f868e75f700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6005> 2014-06-09 16:22:12.194988 7f868cf5c700 2 waiting 51 > 50 ops || 4516130 > 104857600 -6004> 2014-06-09 16:22:12.194993 7f868a757700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6003> 2014-06-09 16:22:12.194998 7f868ef60700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6002> 2014-06-09 16:22:12.195003 7f8690763700 2 waiting 51 > 50 ops || 4516130 > 104857600 -6001> 2014-06-09 16:22:12.195008 7f8690f64700 2 waiting 51 > 50 ops || 4516097 > 104857600 -6000> 2014-06-09 16:22:12.195012 7f8692767700 2 waiting 51 > 50 ops || 4861731 > 104857600 -5999> 2014-06-09 16:22:12.195016 7f8691f66700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5998> 2014-06-09 16:22:12.195027 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6f340] -5997> 2014-06-09 16:22:12.195032 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6f340] -5996> 2014-06-09 16:22:12.195034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5995> 2014-06-09 16:22:12.195035 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5994> 2014-06-09 16:22:12.195038 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286000 (1 active) -5993> 2014-06-09 16:22:12.195042 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb7560 (2 active) -5992> 2014-06-09 16:22:12.195047 7f8691765700 10 op_submit_finish 4216 -5991> 2014-06-09 16:22:12.195052 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -5990> 2014-06-09 16:22:12.195067 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.126555 7 0.000304 -5989> 2014-06-09 16:22:12.195080 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -5988> 2014-06-09 16:22:12.195091 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5987> 2014-06-09 16:22:12.195103 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -5986> 2014-06-09 16:22:12.195114 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -5985> 2014-06-09 16:22:12.195123 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -5984> 2014-06-09 16:22:12.195130 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -5983> 2014-06-09 16:22:12.195142 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -5982> 2014-06-09 16:22:12.195153 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -5981> 2014-06-09 16:22:12.195163 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -5980> 2014-06-09 16:22:12.195163 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 16 ==== osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+1032192 (1321199146 0 0) 0x344d6b40 con 0x30c3c2c0 -5979> 2014-06-09 16:22:12.195178 7f8691765700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 -5978> 2014-06-09 16:22:12.195181 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:1.37d) v4 -- ?+0 0x30d9c000 -5977> 2014-06-09 16:22:12.195190 7f86a68f6700 5 -- op tracker -- , seq: 3678, time: 2014-06-09 16:22:12.193557, event: header_read, op: osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -5976> 2014-06-09 16:22:12.195195 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 -5975> 2014-06-09 16:22:12.195201 7f86a68f6700 5 -- op tracker -- , seq: 3678, time: 2014-06-09 16:22:12.193558, event: throttled, op: osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -5974> 2014-06-09 16:22:12.195222 7f86a68f6700 5 -- op tracker -- , seq: 3678, time: 2014-06-09 16:22:12.195024, event: all_read, op: osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -5973> 2014-06-09 16:22:12.195228 7f8691765700 10 log is not dirty -5972> 2014-06-09 16:22:12.195232 7f86a68f6700 5 -- op tracker -- , seq: 3678, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -5971> 2014-06-09 16:22:12.195233 7f8691765700 5 queue_transactions existing osr(1.37d 0x384d210)/0x384d210 -5970> 2014-06-09 16:22:12.195236 7f8691765700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5969> 2014-06-09 16:22:12.195237 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -5968> 2014-06-09 16:22:12.195241 7f86a5e11700 5 _do_op 0x324b27e0 seq 4168 osr(0.1d4 0x2bd6940)/0x2bd6940 start -5967> 2014-06-09 16:22:12.195242 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -5966> 2014-06-09 16:22:12.195245 7f86a5e11700 10 _do_transaction on 0x3145380 -5965> 2014-06-09 16:22:12.195250 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5964> 2014-06-09 16:22:12.195258 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976dd00 prio 63 cost 1032192 latency 0.001701 osd_op(client.4543.0:33382 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 2314240~1032192] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 -5963> 2014-06-09 16:22:12.195294 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5962> 2014-06-09 16:22:12.195310 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4168.0.0 <= header.spos 150443.0.0 -5961> 2014-06-09 16:22:12.195314 7f86a5e11700 5 submit_transaction r = 0 -5960> 2014-06-09 16:22:12.195324 7f86a5e11700 10 _do_op 0x324b27e0 seq 4168 r = 0, finisher 0x2e0e1e0 0 -5959> 2014-06-09 16:22:12.195336 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509180] -5958> 2014-06-09 16:22:12.195339 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509180] -5957> 2014-06-09 16:22:12.195340 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5956> 2014-06-09 16:22:12.195341 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5955> 2014-06-09 16:22:12.195345 7f86a5e11700 10 _finish_op 0x324b27e0 seq 4168 osr(0.1d4 0x2bd6940)/0x2bd6940 -5954> 2014-06-09 16:22:12.195356 7f868d75d700 10 op_submit_start 4217 -5953> 2014-06-09 16:22:12.195357 7f868d75d700 5 queue_transactions (trailing journal) 4217 -5952> 2014-06-09 16:22:12.195359 7f868d75d700 5 queue_op 0xbe7de00 seq 4217 osr(1.223 0x384cc70) 861 bytes (queue has 50 ops and 4515302 bytes) -5951> 2014-06-09 16:22:12.195364 7f8689f56700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5950> 2014-06-09 16:22:12.195369 7f868ff62700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5949> 2014-06-09 16:22:12.195373 7f868af58700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5948> 2014-06-09 16:22:12.195378 7f868f761700 2 waiting 51 > 50 ops || 4516128 > 104857600 -5947> 2014-06-09 16:22:12.195383 7f868df5e700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5946> 2014-06-09 16:22:12.195387 7f868bf5a700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5945> 2014-06-09 16:22:12.195391 7f868c75b700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5944> 2014-06-09 16:22:12.195395 7f8692f68700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5943> 2014-06-09 16:22:12.195400 7f868b759700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5942> 2014-06-09 16:22:12.195404 7f868e75f700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5941> 2014-06-09 16:22:12.195408 7f868cf5c700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5940> 2014-06-09 16:22:12.195412 7f868a757700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5939> 2014-06-09 16:22:12.195416 7f868ef60700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5938> 2014-06-09 16:22:12.195420 7f8690763700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5937> 2014-06-09 16:22:12.195425 7f8690f64700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5936> 2014-06-09 16:22:12.195429 7f8692767700 2 waiting 51 > 50 ops || 4861764 > 104857600 -5935> 2014-06-09 16:22:12.195433 7f8691f66700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5934> 2014-06-09 16:22:12.195437 7f8691765700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5933> 2014-06-09 16:22:12.195442 7f8689755700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5932> 2014-06-09 16:22:12.195451 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0e1e0] -5931> 2014-06-09 16:22:12.195456 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0e1e0] -5930> 2014-06-09 16:22:12.195457 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5929> 2014-06-09 16:22:12.195458 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5928> 2014-06-09 16:22:12.195462 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb7560 (1 active) -5927> 2014-06-09 16:22:12.195465 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62fc20 (2 active) -5926> 2014-06-09 16:22:12.195492 7f868d75d700 10 op_submit_finish 4217 -5925> 2014-06-09 16:22:12.195497 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -5924> 2014-06-09 16:22:12.195511 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.150645 8 0.000898 -5923> 2014-06-09 16:22:12.195523 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -5922> 2014-06-09 16:22:12.195534 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5921> 2014-06-09 16:22:12.195544 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -5920> 2014-06-09 16:22:12.195555 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -5919> 2014-06-09 16:22:12.195563 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -5918> 2014-06-09 16:22:12.195571 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -5917> 2014-06-09 16:22:12.195582 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) -5916> 2014-06-09 16:22:12.195594 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) -5915> 2014-06-09 16:22:12.195603 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) -5914> 2014-06-09 16:22:12.195625 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) -5913> 2014-06-09 16:22:12.195635 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -5912> 2014-06-09 16:22:12.195649 7f868d75d700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 244 -5911> 2014-06-09 16:22:12.195652 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.1 172.16.96.201:6811/57862 -- pg_info(1 pgs e243:1.764) v4 -- ?+0 0x19fda8c0 -5910> 2014-06-09 16:22:12.195666 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:175 -5909> 2014-06-09 16:22:12.195701 7f868d75d700 10 log is not dirty -5908> 2014-06-09 16:22:12.195705 7f868d75d700 5 queue_transactions existing osr(1.764 0x2ed03f0)/0x2ed03f0 -5907> 2014-06-09 16:22:12.195708 7f868d75d700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5906> 2014-06-09 16:22:12.195734 7f86a5e11700 5 _do_op 0x24104ea0 seq 4169 osr(1.c9 0x384c9f0)/0x384c9f0 start -5905> 2014-06-09 16:22:12.195742 7f86a5e11700 10 _do_transaction on 0xbf89e00 -5904> 2014-06-09 16:22:12.195748 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5903> 2014-06-09 16:22:12.195793 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5902> 2014-06-09 16:22:12.195809 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4169.0.0 <= header.spos 150443.0.0 -5901> 2014-06-09 16:22:12.195813 7f86a5e11700 5 submit_transaction r = 0 -5900> 2014-06-09 16:22:12.195891 7f86a5e11700 10 _do_op 0x24104ea0 seq 4169 r = 0, finisher 0x33a7ac0 0 -5899> 2014-06-09 16:22:12.195903 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc410b40] -5898> 2014-06-09 16:22:12.195906 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc410b40] -5897> 2014-06-09 16:22:12.195907 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5896> 2014-06-09 16:22:12.195909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5895> 2014-06-09 16:22:12.195912 7f86a5e11700 10 _finish_op 0x24104ea0 seq 4169 osr(1.c9 0x384c9f0)/0x384c9f0 -5894> 2014-06-09 16:22:12.195923 7f868ff62700 10 op_submit_start 4218 -5893> 2014-06-09 16:22:12.195925 7f868ff62700 5 queue_transactions (trailing journal) 4218 -5892> 2014-06-09 16:22:12.195926 7f868ff62700 5 queue_op 0x16fd4780 seq 4218 osr(1.233 0x384cc30) 828 bytes (queue has 50 ops and 4515304 bytes) -5891> 2014-06-09 16:22:12.195932 7f868af58700 2 waiting 51 > 50 ops || 4516165 > 104857600 -5890> 2014-06-09 16:22:12.195936 7f868f761700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5889> 2014-06-09 16:22:12.195941 7f868df5e700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5888> 2014-06-09 16:22:12.195945 7f868bf5a700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5887> 2014-06-09 16:22:12.195950 7f868c75b700 2 waiting 51 > 50 ops || 4516165 > 104857600 -5886> 2014-06-09 16:22:12.195954 7f8692f68700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5885> 2014-06-09 16:22:12.195958 7f868b759700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5884> 2014-06-09 16:22:12.195963 7f868e75f700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5883> 2014-06-09 16:22:12.195967 7f868cf5c700 2 waiting 51 > 50 ops || 4516165 > 104857600 -5882> 2014-06-09 16:22:12.195971 7f868a757700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5881> 2014-06-09 16:22:12.195976 7f868ef60700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5880> 2014-06-09 16:22:12.195981 7f8690763700 2 waiting 51 > 50 ops || 4516165 > 104857600 -5879> 2014-06-09 16:22:12.195985 7f8690f64700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5878> 2014-06-09 16:22:12.195990 7f8692767700 2 waiting 51 > 50 ops || 4861766 > 104857600 -5877> 2014-06-09 16:22:12.195994 7f8691f66700 2 waiting 51 > 50 ops || 4516165 > 104857600 -5876> 2014-06-09 16:22:12.195998 7f8691765700 2 waiting 51 > 50 ops || 4516165 > 104857600 -5875> 2014-06-09 16:22:12.196003 7f8689755700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5874> 2014-06-09 16:22:12.196008 7f868d75d700 2 waiting 51 > 50 ops || 4516165 > 104857600 -5873> 2014-06-09 16:22:12.196012 7f8689f56700 2 waiting 51 > 50 ops || 4516165 > 104857600 -5872> 2014-06-09 16:22:12.196022 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a7ac0] -5871> 2014-06-09 16:22:12.196027 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a7ac0] -5870> 2014-06-09 16:22:12.196029 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5869> 2014-06-09 16:22:12.196030 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5868> 2014-06-09 16:22:12.196033 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62fc20 (1 active) -5867> 2014-06-09 16:22:12.196036 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6900 (2 active) -5866> 2014-06-09 16:22:12.196041 7f868ff62700 10 op_submit_finish 4218 -5865> 2014-06-09 16:22:12.196045 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -5864> 2014-06-09 16:22:12.196063 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -5863> 2014-06-09 16:22:12.196073 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x2e075f00,0x2e075e00,0x2e075d00,0x2e075c00 -5862> 2014-06-09 16:22:12.196095 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -5861> 2014-06-09 16:22:12.196106 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -5860> 2014-06-09 16:22:12.196115 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -5859> 2014-06-09 16:22:12.196131 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1081 -5858> 2014-06-09 16:22:12.196142 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -5857> 2014-06-09 16:22:12.196152 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.1bd( v 214'661 (0'0,214'661] local-les=243 n=131 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -5856> 2014-06-09 16:22:12.196163 7f868ff62700 10 log is not dirty -5855> 2014-06-09 16:22:12.196169 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -5854> 2014-06-09 16:22:12.196184 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -5853> 2014-06-09 16:22:12.196193 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x270be300,0x270bf500,0x1b0b7b00,0xe2ac000,0x270be200,0xe032a00,0x270be100,0xe032900,0x270be000,0xe2ad000,0x29f5bf00,0x29f5be00,0x4679f00,0x22009b00,0x22009e00,0x257c4400,0x1804c700 -5852> 2014-06-09 16:22:12.196249 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -5851> 2014-06-09 16:22:12.196259 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -5850> 2014-06-09 16:22:12.196268 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -5849> 2014-06-09 16:22:12.196282 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:882 -5848> 2014-06-09 16:22:12.196292 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -5847> 2014-06-09 16:22:12.196301 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -5846> 2014-06-09 16:22:12.196312 7f868ff62700 10 log is not dirty -5845> 2014-06-09 16:22:12.196316 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -5844> 2014-06-09 16:22:12.196328 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -5843> 2014-06-09 16:22:12.196337 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -5842> 2014-06-09 16:22:12.196345 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5841> 2014-06-09 16:22:12.196353 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -5840> 2014-06-09 16:22:12.196362 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -5839> 2014-06-09 16:22:12.196374 7f868ff62700 15 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 -5838> 2014-06-09 16:22:12.196384 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -5837> 2014-06-09 16:22:12.196392 7f868ff62700 20 osd.26 pg_epoch: 243 pg[1.345( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -5836> 2014-06-09 16:22:12.196400 7f868ff62700 10 log is not dirty -5835> 2014-06-09 16:22:12.196405 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.4bb( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 1.4bb( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) -5834> 2014-06-09 16:22:12.196442 7f868ff62700 10 log is not dirty -5833> 2014-06-09 16:22:12.196447 7f868ff62700 5 queue_transactions existing osr(1.4bb 0x384d970)/0x384d970 -5832> 2014-06-09 16:22:12.196449 7f868ff62700 2 waiting 51 > 50 ops || 4516132 > 104857600 -5831> 2014-06-09 16:22:12.196454 7f86a5e11700 5 _do_op 0xc931d40 seq 4170 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 start -5830> 2014-06-09 16:22:12.196458 7f86a5e11700 10 _do_transaction on 0xc2bdb00 -5829> 2014-06-09 16:22:12.196463 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5828> 2014-06-09 16:22:12.196507 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5827> 2014-06-09 16:22:12.196522 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4170.0.0 <= header.spos 150443.0.0 -5826> 2014-06-09 16:22:12.196527 7f86a5e11700 5 submit_transaction r = 0 -5825> 2014-06-09 16:22:12.196540 7f86a5e11700 10 _do_op 0xc931d40 seq 4170 r = 0, finisher 0x33a63a0 0 -5824> 2014-06-09 16:22:12.196551 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5d840] -5823> 2014-06-09 16:22:12.196553 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5d840] -5822> 2014-06-09 16:22:12.196555 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5821> 2014-06-09 16:22:12.196556 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5820> 2014-06-09 16:22:12.196560 7f86a5e11700 10 _finish_op 0xc931d40 seq 4170 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 -5819> 2014-06-09 16:22:12.196571 7f868f761700 10 op_submit_start 4219 -5818> 2014-06-09 16:22:12.196573 7f868f761700 5 queue_transactions (trailing journal) 4219 -5817> 2014-06-09 16:22:12.196574 7f868f761700 5 queue_op 0x324e7d40 seq 4219 osr(0.b9 0x2bd6400) 826 bytes (queue has 50 ops and 4515302 bytes) -5816> 2014-06-09 16:22:12.196580 7f868df5e700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5815> 2014-06-09 16:22:12.196585 7f868bf5a700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5814> 2014-06-09 16:22:12.196589 7f868c75b700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5813> 2014-06-09 16:22:12.196594 7f8692f68700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5812> 2014-06-09 16:22:12.196598 7f868b759700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5811> 2014-06-09 16:22:12.196603 7f868e75f700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5810> 2014-06-09 16:22:12.196607 7f868cf5c700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5809> 2014-06-09 16:22:12.196612 7f868a757700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5808> 2014-06-09 16:22:12.196616 7f868ef60700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5807> 2014-06-09 16:22:12.196605 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 17 ==== osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+847872 (3296523188 0 0) 0x344d6900 con 0x30c3c2c0 -5806> 2014-06-09 16:22:12.196621 7f8690763700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5805> 2014-06-09 16:22:12.196625 7f8690f64700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5804> 2014-06-09 16:22:12.196629 7f8692767700 2 waiting 51 > 50 ops || 4861764 > 104857600 -5803> 2014-06-09 16:22:12.196628 7f86a68f6700 5 -- op tracker -- , seq: 3679, time: 2014-06-09 16:22:12.195280, event: header_read, op: osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -5802> 2014-06-09 16:22:12.196634 7f8691f66700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5801> 2014-06-09 16:22:12.196639 7f8691765700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5800> 2014-06-09 16:22:12.196643 7f8689755700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5799> 2014-06-09 16:22:12.196647 7f868d75d700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5798> 2014-06-09 16:22:12.196651 7f8689f56700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5797> 2014-06-09 16:22:12.196655 7f868ff62700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5796> 2014-06-09 16:22:12.196660 7f868af58700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5795> 2014-06-09 16:22:12.196670 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a63a0] -5794> 2014-06-09 16:22:12.196677 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a63a0] -5793> 2014-06-09 16:22:12.196679 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5792> 2014-06-09 16:22:12.196680 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5791> 2014-06-09 16:22:12.196683 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6900 (1 active) -5790> 2014-06-09 16:22:12.196686 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301560 (2 active) -5789> 2014-06-09 16:22:12.196691 7f868f761700 10 op_submit_finish 4219 -5788> 2014-06-09 16:22:12.196697 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.3a5( v 213'523 (0'0,213'523] local-les=243 n=102 ec=118 les/c 243/243 242/242/205) -5787> 2014-06-09 16:22:12.196734 7f868f761700 10 log is not dirty -5786> 2014-06-09 16:22:12.196739 7f868f761700 5 queue_transactions existing osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 -5785> 2014-06-09 16:22:12.196741 7f868f761700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5784> 2014-06-09 16:22:12.196772 7f86a5e11700 5 _do_op 0x288f8ae0 seq 4171 osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 start -5783> 2014-06-09 16:22:12.196776 7f86a5e11700 10 _do_transaction on 0xc015e00 -5782> 2014-06-09 16:22:12.196782 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5781> 2014-06-09 16:22:12.196824 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5780> 2014-06-09 16:22:12.196839 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4171.0.0 <= header.spos 150443.0.0 -5779> 2014-06-09 16:22:12.196843 7f86a5e11700 5 submit_transaction r = 0 -5778> 2014-06-09 16:22:12.196871 7f86a5e11700 10 _do_op 0x288f8ae0 seq 4171 r = 0, finisher 0x3205dc60 0 -5777> 2014-06-09 16:22:12.196883 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205dea0] -5776> 2014-06-09 16:22:12.196885 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205dea0] -5775> 2014-06-09 16:22:12.196887 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5774> 2014-06-09 16:22:12.196888 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5773> 2014-06-09 16:22:12.196891 7f86a5e11700 10 _finish_op 0x288f8ae0 seq 4171 osr(3.6ba 0x3ed1ce0)/0x3ed1ce0 -5772> 2014-06-09 16:22:12.196903 7f868bf5a700 10 op_submit_start 4220 -5771> 2014-06-09 16:22:12.196904 7f868bf5a700 5 queue_transactions (trailing journal) 4220 -5770> 2014-06-09 16:22:12.196906 7f868bf5a700 5 queue_op 0x230b9e60 seq 4220 osr(3.2d3 0x3ed0eb0) 828 bytes (queue has 50 ops and 4515302 bytes) -5769> 2014-06-09 16:22:12.196910 7f868c75b700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5768> 2014-06-09 16:22:12.196915 7f8692f68700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5767> 2014-06-09 16:22:12.196920 7f868b759700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5766> 2014-06-09 16:22:12.196924 7f868e75f700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5765> 2014-06-09 16:22:12.196928 7f868cf5c700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5764> 2014-06-09 16:22:12.196932 7f868a757700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5763> 2014-06-09 16:22:12.196936 7f868ef60700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5762> 2014-06-09 16:22:12.196941 7f8690763700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5761> 2014-06-09 16:22:12.196945 7f8690f64700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5760> 2014-06-09 16:22:12.196948 7f8692767700 2 waiting 51 > 50 ops || 4861764 > 104857600 -5759> 2014-06-09 16:22:12.196951 7f8691f66700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5758> 2014-06-09 16:22:12.196955 7f8691765700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5757> 2014-06-09 16:22:12.196959 7f8689755700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5756> 2014-06-09 16:22:12.196964 7f868d75d700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5755> 2014-06-09 16:22:12.196968 7f8689f56700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5754> 2014-06-09 16:22:12.196972 7f868ff62700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5753> 2014-06-09 16:22:12.196977 7f868af58700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5752> 2014-06-09 16:22:12.196981 7f868f761700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5751> 2014-06-09 16:22:12.196986 7f868df5e700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5750> 2014-06-09 16:22:12.196995 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205dc60] -5749> 2014-06-09 16:22:12.197002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205dc60] -5748> 2014-06-09 16:22:12.197003 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5747> 2014-06-09 16:22:12.197005 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5746> 2014-06-09 16:22:12.197008 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301560 (1 active) -5745> 2014-06-09 16:22:12.197011 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b9e0 (2 active) -5744> 2014-06-09 16:22:12.197016 7f868bf5a700 10 op_submit_finish 4220 -5743> 2014-06-09 16:22:12.197020 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.3fe( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.3fe( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) -5742> 2014-06-09 16:22:12.197054 7f868bf5a700 10 log is not dirty -5741> 2014-06-09 16:22:12.197059 7f868bf5a700 5 queue_transactions existing osr(0.3fe 0x2bd7210)/0x2bd7210 -5740> 2014-06-09 16:22:12.197061 7f868bf5a700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5739> 2014-06-09 16:22:12.197066 7f86a5e11700 5 _do_op 0xc704600 seq 4172 osr(3.761 0x3ed1e10)/0x3ed1e10 start -5738> 2014-06-09 16:22:12.197070 7f86a5e11700 10 _do_transaction on 0xbf91bc0 -5737> 2014-06-09 16:22:12.197075 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5736> 2014-06-09 16:22:12.197116 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5735> 2014-06-09 16:22:12.197130 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4172.0.0 <= header.spos 150443.0.0 -5734> 2014-06-09 16:22:12.197135 7f86a5e11700 5 submit_transaction r = 0 -5733> 2014-06-09 16:22:12.197145 7f86a5e11700 10 _do_op 0xc704600 seq 4172 r = 0, finisher 0xbccda40 0 -5732> 2014-06-09 16:22:12.197155 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3eaa0] -5731> 2014-06-09 16:22:12.197158 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3eaa0] -5730> 2014-06-09 16:22:12.197159 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5729> 2014-06-09 16:22:12.197160 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5728> 2014-06-09 16:22:12.197163 7f86a5e11700 10 _finish_op 0xc704600 seq 4172 osr(3.761 0x3ed1e10)/0x3ed1e10 -5727> 2014-06-09 16:22:12.197174 7f8692f68700 10 op_submit_start 4221 -5726> 2014-06-09 16:22:12.197176 7f8692f68700 5 queue_transactions (trailing journal) 4221 -5725> 2014-06-09 16:22:12.197178 7f8692f68700 5 queue_op 0xd327c20 seq 4221 osr(1.508 0x384db90) 828 bytes (queue has 50 ops and 4515302 bytes) -5724> 2014-06-09 16:22:12.197183 7f868b759700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5723> 2014-06-09 16:22:12.197187 7f868e75f700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5722> 2014-06-09 16:22:12.197192 7f868cf5c700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5721> 2014-06-09 16:22:12.197196 7f868a757700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5720> 2014-06-09 16:22:12.197201 7f868ef60700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5719> 2014-06-09 16:22:12.197205 7f8690763700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5718> 2014-06-09 16:22:12.197209 7f8690f64700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5717> 2014-06-09 16:22:12.197214 7f8692767700 2 waiting 51 > 50 ops || 4861764 > 104857600 -5716> 2014-06-09 16:22:12.197218 7f8691f66700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5715> 2014-06-09 16:22:12.197222 7f8691765700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5714> 2014-06-09 16:22:12.197227 7f8689755700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5713> 2014-06-09 16:22:12.197231 7f868d75d700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5712> 2014-06-09 16:22:12.197236 7f8689f56700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5711> 2014-06-09 16:22:12.197240 7f868ff62700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5710> 2014-06-09 16:22:12.197244 7f868af58700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5709> 2014-06-09 16:22:12.197249 7f868f761700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5708> 2014-06-09 16:22:12.197253 7f868df5e700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5707> 2014-06-09 16:22:12.197257 7f868bf5a700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5706> 2014-06-09 16:22:12.197261 7f868c75b700 2 waiting 51 > 50 ops || 4516163 > 104857600 -5705> 2014-06-09 16:22:12.197270 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbccda40] -5704> 2014-06-09 16:22:12.197274 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbccda40] -5703> 2014-06-09 16:22:12.197275 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5702> 2014-06-09 16:22:12.197276 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5701> 2014-06-09 16:22:12.197278 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b9e0 (1 active) -5700> 2014-06-09 16:22:12.197281 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4070e0 (2 active) -5699> 2014-06-09 16:22:12.197285 7f8692f68700 10 op_submit_finish 4221 -5698> 2014-06-09 16:22:12.197288 7f8692f68700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.302( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -5697> 2014-06-09 16:22:12.197319 7f8692f68700 10 log is not dirty -5696> 2014-06-09 16:22:12.197323 7f8692f68700 5 queue_transactions existing osr(1.302 0x384d010)/0x384d010 -5695> 2014-06-09 16:22:12.197325 7f8692f68700 2 waiting 51 > 50 ops || 4516130 > 104857600 -5694> 2014-06-09 16:22:12.197329 7f86a5e11700 5 _do_op 0x2fcaf2c0 seq 4173 osr(0.b7 0x2bd6440)/0x2bd6440 start -5693> 2014-06-09 16:22:12.197333 7f86a5e11700 10 _do_transaction on 0x97bab40 -5692> 2014-06-09 16:22:12.197338 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5691> 2014-06-09 16:22:12.197374 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5690> 2014-06-09 16:22:12.197388 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4173.0.0 <= header.spos 150443.0.0 -5689> 2014-06-09 16:22:12.197392 7f86a5e11700 5 submit_transaction r = 0 -5688> 2014-06-09 16:22:12.197403 7f86a5e11700 10 _do_op 0x2fcaf2c0 seq 4173 r = 0, finisher 0xbd76ae0 0 -5687> 2014-06-09 16:22:12.197412 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6f6e0] -5686> 2014-06-09 16:22:12.197414 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -5685> 2014-06-09 16:22:12.197423 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -5684> 2014-06-09 16:22:12.197431 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.b7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 -5683> 2014-06-09 16:22:12.197441 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6f6e0] -5682> 2014-06-09 16:22:12.197442 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5681> 2014-06-09 16:22:12.197443 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5680> 2014-06-09 16:22:12.197447 7f86a5e11700 10 _finish_op 0x2fcaf2c0 seq 4173 osr(0.b7 0x2bd6440)/0x2bd6440 -5679> 2014-06-09 16:22:12.197458 7f868e75f700 10 op_submit_start 4222 -5678> 2014-06-09 16:22:12.197459 7f868e75f700 5 queue_transactions (trailing journal) 4222 -5677> 2014-06-09 16:22:12.197461 7f868e75f700 5 queue_op 0x324f4a80 seq 4222 osr(2.5f2 0x2ed1df0) 828 bytes (queue has 50 ops and 4515272 bytes) -5676> 2014-06-09 16:22:12.197466 7f868cf5c700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5675> 2014-06-09 16:22:12.197471 7f868a757700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5674> 2014-06-09 16:22:12.197475 7f868ef60700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5673> 2014-06-09 16:22:12.197479 7f8690763700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5672> 2014-06-09 16:22:12.197484 7f8690f64700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5671> 2014-06-09 16:22:12.197488 7f8692767700 2 waiting 51 > 50 ops || 4861734 > 104857600 -5670> 2014-06-09 16:22:12.197492 7f8691f66700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5669> 2014-06-09 16:22:12.197496 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5668> 2014-06-09 16:22:12.197501 7f8689755700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5667> 2014-06-09 16:22:12.197505 7f868d75d700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5666> 2014-06-09 16:22:12.197509 7f8689f56700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5665> 2014-06-09 16:22:12.197513 7f868ff62700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5664> 2014-06-09 16:22:12.197518 7f868af58700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5663> 2014-06-09 16:22:12.197522 7f868f761700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5662> 2014-06-09 16:22:12.197526 7f868df5e700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5661> 2014-06-09 16:22:12.197530 7f868bf5a700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5660> 2014-06-09 16:22:12.197534 7f868c75b700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5659> 2014-06-09 16:22:12.197538 7f8692f68700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5658> 2014-06-09 16:22:12.197543 7f868b759700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5657> 2014-06-09 16:22:12.197552 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd76ae0] -5656> 2014-06-09 16:22:12.197556 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd76ae0] -5655> 2014-06-09 16:22:12.197558 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5654> 2014-06-09 16:22:12.197559 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5653> 2014-06-09 16:22:12.197562 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4070e0 (1 active) -5652> 2014-06-09 16:22:12.197565 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b9e0 (2 active) -5651> 2014-06-09 16:22:12.197570 7f868e75f700 10 op_submit_finish 4222 -5650> 2014-06-09 16:22:12.197574 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.2f2( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.2f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) -5649> 2014-06-09 16:22:12.197608 7f868e75f700 10 log is not dirty -5648> 2014-06-09 16:22:12.197613 7f868e75f700 5 queue_transactions existing osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 -5647> 2014-06-09 16:22:12.197616 7f868e75f700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5646> 2014-06-09 16:22:12.197621 7f86a5e11700 5 _do_op 0x288f9ec0 seq 4174 osr(3.2d6 0x3ed0e60)/0x3ed0e60 start -5645> 2014-06-09 16:22:12.197625 7f86a5e11700 10 _do_transaction on 0x390da40 -5644> 2014-06-09 16:22:12.197630 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5643> 2014-06-09 16:22:12.197671 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5642> 2014-06-09 16:22:12.197685 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4174.0.0 <= header.spos 150443.0.0 -5641> 2014-06-09 16:22:12.197689 7f86a5e11700 5 submit_transaction r = 0 -5640> 2014-06-09 16:22:12.197700 7f86a5e11700 10 _do_op 0x288f9ec0 seq 4174 r = 0, finisher 0xbd6f060 0 -5639> 2014-06-09 16:22:12.197711 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3206f280] -5638> 2014-06-09 16:22:12.197714 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3206f280] -5637> 2014-06-09 16:22:12.197715 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5636> 2014-06-09 16:22:12.197716 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5635> 2014-06-09 16:22:12.197720 7f86a5e11700 10 _finish_op 0x288f9ec0 seq 4174 osr(3.2d6 0x3ed0e60)/0x3ed0e60 -5634> 2014-06-09 16:22:12.197730 7f868a757700 10 op_submit_start 4223 -5633> 2014-06-09 16:22:12.197732 7f868a757700 5 queue_transactions (trailing journal) 4223 -5632> 2014-06-09 16:22:12.197790 7f868a757700 5 queue_op 0xcabc000 seq 4223 osr(1.2c0 0x384d170) 828 bytes (queue has 50 ops and 4515272 bytes) -5631> 2014-06-09 16:22:12.197796 7f868ef60700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5630> 2014-06-09 16:22:12.197801 7f8690763700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5629> 2014-06-09 16:22:12.197806 7f8690f64700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5628> 2014-06-09 16:22:12.197810 7f8692767700 2 waiting 51 > 50 ops || 4861734 > 104857600 -5627> 2014-06-09 16:22:12.197815 7f8691f66700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5626> 2014-06-09 16:22:12.197819 7f8691765700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5625> 2014-06-09 16:22:12.197823 7f8689755700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5624> 2014-06-09 16:22:12.197828 7f868d75d700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5623> 2014-06-09 16:22:12.197832 7f8689f56700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5622> 2014-06-09 16:22:12.197837 7f868ff62700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5621> 2014-06-09 16:22:12.197841 7f868af58700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5620> 2014-06-09 16:22:12.197845 7f868f761700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5619> 2014-06-09 16:22:12.197850 7f868df5e700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5618> 2014-06-09 16:22:12.197854 7f868bf5a700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5617> 2014-06-09 16:22:12.197858 7f868c75b700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5616> 2014-06-09 16:22:12.197863 7f8692f68700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5615> 2014-06-09 16:22:12.197867 7f868b759700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5614> 2014-06-09 16:22:12.197871 7f868e75f700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5613> 2014-06-09 16:22:12.197875 7f868cf5c700 2 waiting 51 > 50 ops || 4516133 > 104857600 -5612> 2014-06-09 16:22:12.197885 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6f060] -5611> 2014-06-09 16:22:12.197892 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6f060] -5610> 2014-06-09 16:22:12.197894 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5609> 2014-06-09 16:22:12.197895 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5608> 2014-06-09 16:22:12.197898 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b9e0 (1 active) -5607> 2014-06-09 16:22:12.197900 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358fc0 (2 active) -5606> 2014-06-09 16:22:12.197905 7f868a757700 10 op_submit_finish 4223 -5605> 2014-06-09 16:22:12.197909 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 243/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.218( v 213'533 (0'0,213'533] local-les=243 n=104 ec=118 les/c 243/243 242/242/211) -5604> 2014-06-09 16:22:12.197950 7f868a757700 10 log is not dirty -5603> 2014-06-09 16:22:12.197955 7f868a757700 5 queue_transactions existing osr(3.218 0x3ed0cd0)/0x3ed0cd0 -5602> 2014-06-09 16:22:12.197958 7f868a757700 2 waiting 51 > 50 ops || 4516100 > 104857600 -5601> 2014-06-09 16:22:12.197962 7f86a5e11700 5 _do_op 0x324a4840 seq 4175 osr(1.fa 0x384c970)/0x384c970 start -5600> 2014-06-09 16:22:12.197966 7f86a5e11700 10 _do_transaction on 0xbf0e300 -5599> 2014-06-09 16:22:12.197972 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5598> 2014-06-09 16:22:12.198013 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5597> 2014-06-09 16:22:12.198028 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4175.0.0 <= header.spos 150443.0.0 -5596> 2014-06-09 16:22:12.198032 7f86a5e11700 5 submit_transaction r = 0 -5595> 2014-06-09 16:22:12.198048 7f86a5e11700 10 _do_op 0x324a4840 seq 4175 r = 0, finisher 0x314129e0 0 -5594> 2014-06-09 16:22:12.198059 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a7bc0] -5593> 2014-06-09 16:22:12.198062 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a7bc0] -5592> 2014-06-09 16:22:12.198064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5591> 2014-06-09 16:22:12.198065 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5590> 2014-06-09 16:22:12.198068 7f86a5e11700 10 _finish_op 0x324a4840 seq 4175 osr(1.fa 0x384c970)/0x384c970 -5589> 2014-06-09 16:22:12.198078 7f8690763700 10 op_submit_start 4224 -5588> 2014-06-09 16:22:12.198080 7f8690763700 5 queue_transactions (trailing journal) 4224 -5587> 2014-06-09 16:22:12.198082 7f8690763700 5 queue_op 0xc34bc80 seq 4224 osr(0.686 0x2bd7e20) 861 bytes (queue has 50 ops and 4515307 bytes) -5586> 2014-06-09 16:22:12.198087 7f8690f64700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5585> 2014-06-09 16:22:12.198091 7f8692767700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5584> 2014-06-09 16:22:12.198096 7f8691f66700 2 waiting 51 > 50 ops || 4516168 > 104857600 -5583> 2014-06-09 16:22:12.198100 7f8691765700 2 waiting 51 > 50 ops || 4516168 > 104857600 -5582> 2014-06-09 16:22:12.198105 7f8689755700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5581> 2014-06-09 16:22:12.198109 7f868d75d700 2 waiting 51 > 50 ops || 4516168 > 104857600 -5580> 2014-06-09 16:22:12.198113 7f8689f56700 2 waiting 51 > 50 ops || 4516168 > 104857600 -5579> 2014-06-09 16:22:12.198118 7f868ff62700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5578> 2014-06-09 16:22:12.198122 7f868af58700 2 waiting 51 > 50 ops || 4516168 > 104857600 -5577> 2014-06-09 16:22:12.198126 7f868f761700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5576> 2014-06-09 16:22:12.198130 7f868df5e700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5575> 2014-06-09 16:22:12.198134 7f868bf5a700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5574> 2014-06-09 16:22:12.198139 7f868c75b700 2 waiting 51 > 50 ops || 4516168 > 104857600 -5573> 2014-06-09 16:22:12.198143 7f8692f68700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5572> 2014-06-09 16:22:12.198147 7f868b759700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5571> 2014-06-09 16:22:12.198151 7f868e75f700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5570> 2014-06-09 16:22:12.198155 7f868cf5c700 2 waiting 51 > 50 ops || 4516168 > 104857600 -5569> 2014-06-09 16:22:12.198159 7f868a757700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5568> 2014-06-09 16:22:12.198163 7f868ef60700 2 waiting 51 > 50 ops || 4516135 > 104857600 -5567> 2014-06-09 16:22:12.198173 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x314129e0] -5566> 2014-06-09 16:22:12.198178 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x314129e0] -5565> 2014-06-09 16:22:12.198180 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5564> 2014-06-09 16:22:12.198181 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5563> 2014-06-09 16:22:12.198184 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358fc0 (1 active) -5562> 2014-06-09 16:22:12.198187 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc286a20 (2 active) -5561> 2014-06-09 16:22:12.198192 7f8690763700 10 op_submit_finish 4224 -5560> 2014-06-09 16:22:12.198195 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -5559> 2014-06-09 16:22:12.198211 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.062160 7 0.000955 -5558> 2014-06-09 16:22:12.198224 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -5557> 2014-06-09 16:22:12.198236 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -5556> 2014-06-09 16:22:12.198247 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -5555> 2014-06-09 16:22:12.198259 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -5554> 2014-06-09 16:22:12.198269 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery -5553> 2014-06-09 16:22:12.198278 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state -5552> 2014-06-09 16:22:12.198289 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -5551> 2014-06-09 16:22:12.198301 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -5550> 2014-06-09 16:22:12.198313 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info -5549> 2014-06-09 16:22:12.198327 7f8690763700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 -5548> 2014-06-09 16:22:12.198330 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:3.65f) v4 -- ?+0 0x3343cc40 -5547> 2014-06-09 16:22:12.198344 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1113 -5546> 2014-06-09 16:22:12.198379 7f8690763700 10 log is not dirty -5545> 2014-06-09 16:22:12.198383 7f8690763700 5 queue_transactions existing osr(3.65f 0x3ed1a10)/0x3ed1a10 -5544> 2014-06-09 16:22:12.198385 7f8690763700 2 waiting 51 > 50 ops || 4516168 > 104857600 -5543> 2014-06-09 16:22:12.198390 7f86a5e11700 5 _do_op 0x309a3800 seq 4176 osr(1.3ef 0x384d3f0)/0x384d3f0 start -5542> 2014-06-09 16:22:12.198394 7f86a5e11700 10 _do_transaction on 0xc290900 -5541> 2014-06-09 16:22:12.198399 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5540> 2014-06-09 16:22:12.198441 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5539> 2014-06-09 16:22:12.198457 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4176.0.0 <= header.spos 150443.0.0 -5538> 2014-06-09 16:22:12.198461 7f86a5e11700 5 submit_transaction r = 0 -5537> 2014-06-09 16:22:12.198476 7f86a5e11700 10 _do_op 0x309a3800 seq 4176 r = 0, finisher 0xccf8800 0 -5536> 2014-06-09 16:22:12.198487 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd744c0] -5535> 2014-06-09 16:22:12.198490 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -5534> 2014-06-09 16:22:12.198500 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -5533> 2014-06-09 16:22:12.198510 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.3ef( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -5532> 2014-06-09 16:22:12.198519 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd744c0] -5531> 2014-06-09 16:22:12.198521 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5530> 2014-06-09 16:22:12.198522 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5529> 2014-06-09 16:22:12.198526 7f86a5e11700 10 _finish_op 0x309a3800 seq 4176 osr(1.3ef 0x384d3f0)/0x384d3f0 -5528> 2014-06-09 16:22:12.198537 7f8692767700 10 op_submit_start 4225 -5527> 2014-06-09 16:22:12.198538 7f8692767700 5 queue_transactions (trailing journal) 4225 -5526> 2014-06-09 16:22:12.198540 7f8692767700 5 queue_op 0x32539980 seq 4225 osr(3.56a 0x3ed1900) 346462 bytes (queue has 50 ops and 4860908 bytes) -5525> 2014-06-09 16:22:12.198544 7f8691f66700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5524> 2014-06-09 16:22:12.198549 7f8691765700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5523> 2014-06-09 16:22:12.198554 7f8689755700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5522> 2014-06-09 16:22:12.198558 7f868d75d700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5521> 2014-06-09 16:22:12.198562 7f8689f56700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5520> 2014-06-09 16:22:12.198566 7f868ff62700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5519> 2014-06-09 16:22:12.198571 7f868af58700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5518> 2014-06-09 16:22:12.198575 7f868f761700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5517> 2014-06-09 16:22:12.198579 7f868df5e700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5516> 2014-06-09 16:22:12.198584 7f868bf5a700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5515> 2014-06-09 16:22:12.198588 7f868c75b700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5514> 2014-06-09 16:22:12.198592 7f8692f68700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5513> 2014-06-09 16:22:12.198596 7f868b759700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5512> 2014-06-09 16:22:12.198600 7f868e75f700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5511> 2014-06-09 16:22:12.198605 7f868cf5c700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5510> 2014-06-09 16:22:12.198609 7f868a757700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5509> 2014-06-09 16:22:12.198613 7f868ef60700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5508> 2014-06-09 16:22:12.198617 7f8690763700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5507> 2014-06-09 16:22:12.198622 7f8690f64700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5506> 2014-06-09 16:22:12.198631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf8800] -5505> 2014-06-09 16:22:12.198635 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf8800] -5504> 2014-06-09 16:22:12.198637 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5503> 2014-06-09 16:22:12.198638 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5502> 2014-06-09 16:22:12.198641 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc286a20 (1 active) -5501> 2014-06-09 16:22:12.198644 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524000 (2 active) -5500> 2014-06-09 16:22:12.198649 7f8692767700 10 op_submit_finish 4225 -5499> 2014-06-09 16:22:12.198653 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc20d140 243'576 rep_tid=31 committed?=0 applied?=0 lock=0 op=osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad -5498> 2014-06-09 16:22:12.198676 7f8692767700 10 osd.26 244 dequeue_op 0x230da800 finish -5497> 2014-06-09 16:22:12.198681 7f8692767700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -5496> 2014-06-09 16:22:12.198726 7f8692767700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8e560 (20 active) -5495> 2014-06-09 16:22:12.198730 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -5494> 2014-06-09 16:22:12.198743 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 0.999816 7 0.000446 -5493> 2014-06-09 16:22:12.198754 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -5492> 2014-06-09 16:22:12.198765 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5491> 2014-06-09 16:22:12.198775 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -5490> 2014-06-09 16:22:12.198785 7f8692767700 5 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -5489> 2014-06-09 16:22:12.198794 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -5488> 2014-06-09 16:22:12.198801 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -5487> 2014-06-09 16:22:12.198812 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) -5486> 2014-06-09 16:22:12.198822 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) -5485> 2014-06-09 16:22:12.198832 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -5484> 2014-06-09 16:22:12.198845 7f8692767700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbfa6580 172.16.96.201:6851/52636 -5483> 2014-06-09 16:22:12.198856 7f8692767700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6851/52636 -- osd_map(244..244 src has 1..244) v3 -- ?+0 0x24107200 con 0xbfa6580 -5482> 2014-06-09 16:22:12.198868 7f8692767700 10 osd.26 243 note_peer_epoch osd.7 has 244 -5481> 2014-06-09 16:22:12.198871 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.7 172.16.96.201:6851/52636 -- pg_info(1 pgs e243:1.4bc) v4 -- ?+0 0x31776000 -5480> 2014-06-09 16:22:12.198880 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 -5479> 2014-06-09 16:22:12.198913 7f8692767700 10 log is not dirty -5478> 2014-06-09 16:22:12.198917 7f8692767700 5 queue_transactions existing osr(1.4bc 0x384d930)/0x384d930 -5477> 2014-06-09 16:22:12.198919 7f8692767700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5476> 2014-06-09 16:22:12.198924 7f86a5e11700 5 _do_op 0x39d9da0 seq 4177 osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 start -5475> 2014-06-09 16:22:12.198927 7f86a5e11700 10 _do_transaction on 0xbfb2600 -5474> 2014-06-09 16:22:12.198933 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5473> 2014-06-09 16:22:12.198974 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5472> 2014-06-09 16:22:12.198988 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4177.0.0 <= header.spos 150443.0.0 -5471> 2014-06-09 16:22:12.198993 7f86a5e11700 5 submit_transaction r = 0 -5470> 2014-06-09 16:22:12.199003 7f86a5e11700 10 _do_op 0x39d9da0 seq 4177 r = 0, finisher 0xbd5bca0 0 -5469> 2014-06-09 16:22:12.199013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e19d40] -5468> 2014-06-09 16:22:12.199016 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e19d40] -5467> 2014-06-09 16:22:12.199017 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5466> 2014-06-09 16:22:12.199019 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5465> 2014-06-09 16:22:12.199022 7f86a5e11700 10 _finish_op 0x39d9da0 seq 4177 osr(3.6cb 0x3ed1cb0)/0x3ed1cb0 -5464> 2014-06-09 16:22:12.199033 7f8691765700 10 op_submit_start 4226 -5463> 2014-06-09 16:22:12.199034 7f8691765700 5 queue_transactions (trailing journal) 4226 -5462> 2014-06-09 16:22:12.199036 7f8691765700 5 queue_op 0x16fd4240 seq 4226 osr(1.37d 0x384d210) 861 bytes (queue has 50 ops and 4860941 bytes) -5461> 2014-06-09 16:22:12.199041 7f8689755700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5460> 2014-06-09 16:22:12.199045 7f868d75d700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5459> 2014-06-09 16:22:12.199050 7f8689f56700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5458> 2014-06-09 16:22:12.199054 7f868ff62700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5457> 2014-06-09 16:22:12.199058 7f868af58700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5456> 2014-06-09 16:22:12.199063 7f868f761700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5455> 2014-06-09 16:22:12.199067 7f868df5e700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5454> 2014-06-09 16:22:12.199071 7f868bf5a700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5453> 2014-06-09 16:22:12.199075 7f868c75b700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5452> 2014-06-09 16:22:12.199079 7f8692f68700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5451> 2014-06-09 16:22:12.199084 7f868b759700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5450> 2014-06-09 16:22:12.199088 7f868e75f700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5449> 2014-06-09 16:22:12.199093 7f868cf5c700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5448> 2014-06-09 16:22:12.199097 7f868a757700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5447> 2014-06-09 16:22:12.199102 7f868ef60700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5446> 2014-06-09 16:22:12.199106 7f8690763700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5445> 2014-06-09 16:22:12.199110 7f8690f64700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5444> 2014-06-09 16:22:12.199114 7f8692767700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5443> 2014-06-09 16:22:12.199118 7f8691f66700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5442> 2014-06-09 16:22:12.199127 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5bca0] -5441> 2014-06-09 16:22:12.199131 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5bca0] -5440> 2014-06-09 16:22:12.199133 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5439> 2014-06-09 16:22:12.199134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5438> 2014-06-09 16:22:12.199137 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524000 (1 active) -5437> 2014-06-09 16:22:12.199140 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd320 (2 active) -5436> 2014-06-09 16:22:12.199145 7f8691765700 10 op_submit_finish 4226 -5435> 2014-06-09 16:22:12.199149 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -5434> 2014-06-09 16:22:12.199162 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.037109 7 0.000295 -5433> 2014-06-09 16:22:12.199174 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -5432> 2014-06-09 16:22:12.199184 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5431> 2014-06-09 16:22:12.199194 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -5430> 2014-06-09 16:22:12.199205 7f8691765700 5 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -5429> 2014-06-09 16:22:12.199213 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -5428> 2014-06-09 16:22:12.199221 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -5427> 2014-06-09 16:22:12.199231 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -5426> 2014-06-09 16:22:12.199243 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -5425> 2014-06-09 16:22:12.199252 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -5424> 2014-06-09 16:22:12.199262 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -5423> 2014-06-09 16:22:12.199271 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -5422> 2014-06-09 16:22:12.199284 7f8691765700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 -5421> 2014-06-09 16:22:12.199287 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:2.3c7) v4 -- ?+0 0x257cdc00 -5420> 2014-06-09 16:22:12.199300 7f8691765700 15 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:206 -5419> 2014-06-09 16:22:12.199332 7f8691765700 10 log is not dirty -5418> 2014-06-09 16:22:12.199336 7f8691765700 5 queue_transactions existing osr(2.3c7 0x2ed13d0)/0x2ed13d0 -5417> 2014-06-09 16:22:12.199339 7f8691765700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5416> 2014-06-09 16:22:12.199366 7f86a5e11700 5 _do_op 0x309a3560 seq 4178 osr(2.61b 0x2ed1cd0)/0x2ed1cd0 start -5415> 2014-06-09 16:22:12.199374 7f86a5e11700 10 _do_transaction on 0xc04f380 -5414> 2014-06-09 16:22:12.199381 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5413> 2014-06-09 16:22:12.199448 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5412> 2014-06-09 16:22:12.199466 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4178.0.0 <= header.spos 150443.0.0 -5411> 2014-06-09 16:22:12.199470 7f86a5e11700 5 submit_transaction r = 0 -5410> 2014-06-09 16:22:12.199484 7f86a5e11700 10 _do_op 0x309a3560 seq 4178 r = 0, finisher 0xbd779a0 0 -5409> 2014-06-09 16:22:12.199495 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf0b40] -5408> 2014-06-09 16:22:12.199498 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -5407> 2014-06-09 16:22:12.199508 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -5406> 2014-06-09 16:22:12.199517 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.61b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 -5405> 2014-06-09 16:22:12.199526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf0b40] -5404> 2014-06-09 16:22:12.199528 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5403> 2014-06-09 16:22:12.199529 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5402> 2014-06-09 16:22:12.199532 7f86a5e11700 10 _finish_op 0x309a3560 seq 4178 osr(2.61b 0x2ed1cd0)/0x2ed1cd0 -5401> 2014-06-09 16:22:12.199544 7f868d75d700 10 op_submit_start 4227 -5400> 2014-06-09 16:22:12.199545 7f868d75d700 5 queue_transactions (trailing journal) 4227 -5399> 2014-06-09 16:22:12.199547 7f868d75d700 5 queue_op 0xc2d4060 seq 4227 osr(1.764 0x2ed03f0) 861 bytes (queue has 50 ops and 4860941 bytes) -5398> 2014-06-09 16:22:12.199551 7f8689f56700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5397> 2014-06-09 16:22:12.199556 7f868ff62700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5396> 2014-06-09 16:22:12.199560 7f868af58700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5395> 2014-06-09 16:22:12.199565 7f868f761700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5394> 2014-06-09 16:22:12.199569 7f868df5e700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5393> 2014-06-09 16:22:12.199573 7f868bf5a700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5392> 2014-06-09 16:22:12.199577 7f868c75b700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5391> 2014-06-09 16:22:12.199582 7f8692f68700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5390> 2014-06-09 16:22:12.199586 7f868b759700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5389> 2014-06-09 16:22:12.199590 7f868e75f700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5388> 2014-06-09 16:22:12.199594 7f868cf5c700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5387> 2014-06-09 16:22:12.199598 7f868a757700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5386> 2014-06-09 16:22:12.199602 7f868ef60700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5385> 2014-06-09 16:22:12.199606 7f8690763700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5384> 2014-06-09 16:22:12.199610 7f8690f64700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5383> 2014-06-09 16:22:12.199619 7f8692767700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5382> 2014-06-09 16:22:12.199624 7f8691f66700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5381> 2014-06-09 16:22:12.199628 7f8691765700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5380> 2014-06-09 16:22:12.199633 7f8689755700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5379> 2014-06-09 16:22:12.199642 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd779a0] -5378> 2014-06-09 16:22:12.199646 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd779a0] -5377> 2014-06-09 16:22:12.199648 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5376> 2014-06-09 16:22:12.199649 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5375> 2014-06-09 16:22:12.199707 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd320 (1 active) -5374> 2014-06-09 16:22:12.199711 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d1320 (2 active) -5373> 2014-06-09 16:22:12.199716 7f868d75d700 10 op_submit_finish 4227 -5372> 2014-06-09 16:22:12.199720 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -5371> 2014-06-09 16:22:12.199733 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local -5370> 2014-06-09 16:22:12.199742 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -5369> 2014-06-09 16:22:12.199750 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5368> 2014-06-09 16:22:12.199759 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -5367> 2014-06-09 16:22:12.199767 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -5366> 2014-06-09 16:22:12.199781 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:136 -5365> 2014-06-09 16:22:12.199790 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -5364> 2014-06-09 16:22:12.199798 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.125( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop -5363> 2014-06-09 16:22:12.199808 7f868d75d700 10 log is not dirty -5362> 2014-06-09 16:22:12.199813 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -5361> 2014-06-09 16:22:12.199824 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.024301 7 0.000342 -5360> 2014-06-09 16:22:12.199835 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -5359> 2014-06-09 16:22:12.199846 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5358> 2014-06-09 16:22:12.199856 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -5357> 2014-06-09 16:22:12.199866 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -5356> 2014-06-09 16:22:12.199874 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -5355> 2014-06-09 16:22:12.199882 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -5354> 2014-06-09 16:22:12.199893 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -5353> 2014-06-09 16:22:12.199904 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -5352> 2014-06-09 16:22:12.199914 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -5351> 2014-06-09 16:22:12.199923 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -5350> 2014-06-09 16:22:12.199932 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -5349> 2014-06-09 16:22:12.199944 7f868d75d700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 -5348> 2014-06-09 16:22:12.199947 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:2.5f4) v4 -- ?+0 0x1fc041c0 -5347> 2014-06-09 16:22:12.199960 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 -5346> 2014-06-09 16:22:12.199992 7f868d75d700 10 log is not dirty -5345> 2014-06-09 16:22:12.199996 7f868d75d700 5 queue_transactions existing osr(2.5f4 0x2ed1dd0)/0x2ed1dd0 -5344> 2014-06-09 16:22:12.199999 7f868d75d700 2 waiting 51 > 50 ops || 4861802 > 104857600 -5343> 2014-06-09 16:22:12.200028 7f86a5e11700 5 _do_op 0xc704ba0 seq 4179 osr(1.446 0x384d6b0)/0x384d6b0 start -5342> 2014-06-09 16:22:12.200035 7f86a5e11700 10 _do_transaction on 0xc2d9140 -5341> 2014-06-09 16:22:12.200041 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5340> 2014-06-09 16:22:12.200083 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5339> 2014-06-09 16:22:12.200101 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4179.0.0 <= header.spos 150443.0.0 -5338> 2014-06-09 16:22:12.200105 7f86a5e11700 5 submit_transaction r = 0 -5337> 2014-06-09 16:22:12.200119 7f86a5e11700 10 _do_op 0xc704ba0 seq 4179 r = 0, finisher 0xbd6b5c0 0 -5336> 2014-06-09 16:22:12.200129 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31791ec0] -5335> 2014-06-09 16:22:12.200132 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -5334> 2014-06-09 16:22:12.200142 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -5333> 2014-06-09 16:22:12.200151 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 -5332> 2014-06-09 16:22:12.200160 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31791ec0] -5331> 2014-06-09 16:22:12.200162 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5330> 2014-06-09 16:22:12.200163 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5329> 2014-06-09 16:22:12.200167 7f86a5e11700 10 _finish_op 0xc704ba0 seq 4179 osr(1.446 0x384d6b0)/0x384d6b0 -5328> 2014-06-09 16:22:12.200179 7f868ff62700 10 op_submit_start 4228 -5327> 2014-06-09 16:22:12.200180 7f868ff62700 5 queue_transactions (trailing journal) 4228 -5326> 2014-06-09 16:22:12.200182 7f868ff62700 5 queue_op 0x3250f9e0 seq 4228 osr(1.4bb 0x384d970) 828 bytes (queue has 50 ops and 4860908 bytes) -5325> 2014-06-09 16:22:12.200187 7f868af58700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5324> 2014-06-09 16:22:12.200191 7f868f761700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5323> 2014-06-09 16:22:12.200196 7f868df5e700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5322> 2014-06-09 16:22:12.200200 7f868bf5a700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5321> 2014-06-09 16:22:12.200204 7f868c75b700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5320> 2014-06-09 16:22:12.200208 7f8692f68700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5319> 2014-06-09 16:22:12.200213 7f868b759700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5318> 2014-06-09 16:22:12.200217 7f868e75f700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5317> 2014-06-09 16:22:12.200221 7f868cf5c700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5316> 2014-06-09 16:22:12.200226 7f868a757700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5315> 2014-06-09 16:22:12.200230 7f868ef60700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5314> 2014-06-09 16:22:12.200234 7f8690763700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5313> 2014-06-09 16:22:12.200238 7f8690f64700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5312> 2014-06-09 16:22:12.200242 7f8692767700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5311> 2014-06-09 16:22:12.200247 7f8691f66700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5310> 2014-06-09 16:22:12.200251 7f8691765700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5309> 2014-06-09 16:22:12.200255 7f8689755700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5308> 2014-06-09 16:22:12.200259 7f868d75d700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5307> 2014-06-09 16:22:12.200263 7f8689f56700 2 waiting 51 > 50 ops || 4861769 > 104857600 -5306> 2014-06-09 16:22:12.200273 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6b5c0] -5305> 2014-06-09 16:22:12.200278 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6b5c0] -5304> 2014-06-09 16:22:12.200280 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5303> 2014-06-09 16:22:12.200281 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5302> 2014-06-09 16:22:12.200284 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d1320 (1 active) -5301> 2014-06-09 16:22:12.200287 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d19e0 (2 active) -5300> 2014-06-09 16:22:12.200292 7f868ff62700 10 op_submit_finish 4228 -5299> 2014-06-09 16:22:12.200296 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -5298> 2014-06-09 16:22:12.200311 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -5297> 2014-06-09 16:22:12.200321 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x14052400,0x1c54b000,0x14052300,0x1c54af00,0x21966c00,0x21966b00,0x21966a00,0x21966900,0x21966800,0x1804db00,0x1804da00,0x4679200,0x4679100,0x4679000,0x4678f00,0x4678e00,0x4678d00 -5296> 2014-06-09 16:22:12.200374 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -5295> 2014-06-09 16:22:12.200384 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -5294> 2014-06-09 16:22:12.200392 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -5293> 2014-06-09 16:22:12.200406 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1100 -5292> 2014-06-09 16:22:12.200417 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -5291> 2014-06-09 16:22:12.200426 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.327( v 214'680 (0'0,214'680] local-les=243 n=132 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -5290> 2014-06-09 16:22:12.200436 7f868ff62700 10 log is not dirty -5289> 2014-06-09 16:22:12.200441 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.6f0( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 0.6f0( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) -5288> 2014-06-09 16:22:12.200474 7f868ff62700 10 log is not dirty -5287> 2014-06-09 16:22:12.200479 7f868ff62700 5 queue_transactions existing osr(0.6f0 0x384c170)/0x384c170 -5286> 2014-06-09 16:22:12.200481 7f868ff62700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5285> 2014-06-09 16:22:12.200486 7f86a5e11700 5 _do_op 0x324e62a0 seq 4180 osr(0.66b 0x2bd7eb0)/0x2bd7eb0 start -5284> 2014-06-09 16:22:12.200490 7f86a5e11700 10 _do_transaction on 0x398ba40 -5283> 2014-06-09 16:22:12.200495 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5282> 2014-06-09 16:22:12.200537 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5281> 2014-06-09 16:22:12.200554 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4180.0.0 <= header.spos 150443.0.0 -5280> 2014-06-09 16:22:12.200559 7f86a5e11700 5 submit_transaction r = 0 -5279> 2014-06-09 16:22:12.200581 7f86a5e11700 10 _do_op 0x324e62a0 seq 4180 r = 0, finisher 0xc6e91c0 0 -5278> 2014-06-09 16:22:12.200593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5f780] -5277> 2014-06-09 16:22:12.200596 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -5276> 2014-06-09 16:22:12.200606 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -5275> 2014-06-09 16:22:12.200615 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.66b( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -5274> 2014-06-09 16:22:12.200624 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5f780] -5273> 2014-06-09 16:22:12.200626 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5272> 2014-06-09 16:22:12.200627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5271> 2014-06-09 16:22:12.200631 7f86a5e11700 10 _finish_op 0x324e62a0 seq 4180 osr(0.66b 0x2bd7eb0)/0x2bd7eb0 -5270> 2014-06-09 16:22:12.200642 7f868f761700 10 op_submit_start 4229 -5269> 2014-06-09 16:22:12.200644 7f868f761700 5 queue_transactions (trailing journal) 4229 -5268> 2014-06-09 16:22:12.200645 7f868f761700 5 queue_op 0x324e6c60 seq 4229 osr(3.3a5 0x3ed0fb0) 828 bytes (queue has 50 ops and 4860875 bytes) -5267> 2014-06-09 16:22:12.200650 7f868df5e700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5266> 2014-06-09 16:22:12.200654 7f868bf5a700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5265> 2014-06-09 16:22:12.200658 7f868c75b700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5264> 2014-06-09 16:22:12.200663 7f8692f68700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5263> 2014-06-09 16:22:12.200667 7f868b759700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5262> 2014-06-09 16:22:12.200671 7f868e75f700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5261> 2014-06-09 16:22:12.200675 7f868cf5c700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5260> 2014-06-09 16:22:12.200680 7f868a757700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5259> 2014-06-09 16:22:12.200684 7f868ef60700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5258> 2014-06-09 16:22:12.200688 7f8690763700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5257> 2014-06-09 16:22:12.200693 7f8690f64700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5256> 2014-06-09 16:22:12.200697 7f8692767700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5255> 2014-06-09 16:22:12.200701 7f8691f66700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5254> 2014-06-09 16:22:12.200705 7f8691765700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5253> 2014-06-09 16:22:12.200709 7f8689755700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5252> 2014-06-09 16:22:12.200714 7f868d75d700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5251> 2014-06-09 16:22:12.200718 7f8689f56700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5250> 2014-06-09 16:22:12.200722 7f868ff62700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5249> 2014-06-09 16:22:12.200726 7f868af58700 2 waiting 51 > 50 ops || 4861736 > 104857600 -5248> 2014-06-09 16:22:12.200736 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e91c0] -5247> 2014-06-09 16:22:12.200741 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e91c0] -5246> 2014-06-09 16:22:12.200743 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5245> 2014-06-09 16:22:12.200744 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5244> 2014-06-09 16:22:12.200747 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d19e0 (1 active) -5243> 2014-06-09 16:22:12.200749 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497200 (2 active) -5242> 2014-06-09 16:22:12.200754 7f868f761700 10 op_submit_finish 4229 -5241> 2014-06-09 16:22:12.200760 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.48e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.48e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) -5240> 2014-06-09 16:22:12.200796 7f868f761700 10 log is not dirty -5239> 2014-06-09 16:22:12.200801 7f868f761700 5 queue_transactions existing osr(1.48e 0x384d9b0)/0x384d9b0 -5238> 2014-06-09 16:22:12.200804 7f868f761700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5237> 2014-06-09 16:22:12.200824 7f86a5e11700 5 _do_op 0x160970e0 seq 4181 osr(0.67d 0x2bd7e50)/0x2bd7e50 start -5236> 2014-06-09 16:22:12.200827 7f86a5e11700 10 _do_transaction on 0x2f28d80 -5235> 2014-06-09 16:22:12.200833 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5234> 2014-06-09 16:22:12.200874 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5233> 2014-06-09 16:22:12.200891 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4181.0.0 <= header.spos 150443.0.0 -5232> 2014-06-09 16:22:12.200896 7f86a5e11700 5 submit_transaction r = 0 -5231> 2014-06-09 16:22:12.200908 7f86a5e11700 10 _do_op 0x160970e0 seq 4181 r = 0, finisher 0xbdf1dc0 0 -5230> 2014-06-09 16:22:12.200918 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050b0e0] -5229> 2014-06-09 16:22:12.200921 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -5228> 2014-06-09 16:22:12.200931 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -5227> 2014-06-09 16:22:12.200940 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.67d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 -5226> 2014-06-09 16:22:12.200949 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050b0e0] -5225> 2014-06-09 16:22:12.200951 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5224> 2014-06-09 16:22:12.200952 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5223> 2014-06-09 16:22:12.200956 7f86a5e11700 10 _finish_op 0x160970e0 seq 4181 osr(0.67d 0x2bd7e50)/0x2bd7e50 -5222> 2014-06-09 16:22:12.200967 7f868bf5a700 10 op_submit_start 4230 -5221> 2014-06-09 16:22:12.200969 7f868bf5a700 5 queue_transactions (trailing journal) 4230 -5220> 2014-06-09 16:22:12.200970 7f868bf5a700 5 queue_op 0x324da120 seq 4230 osr(0.3fe 0x2bd7210) 828 bytes (queue has 50 ops and 4860842 bytes) -5219> 2014-06-09 16:22:12.200975 7f868c75b700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5218> 2014-06-09 16:22:12.200979 7f8692f68700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5217> 2014-06-09 16:22:12.200983 7f868b759700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5216> 2014-06-09 16:22:12.200988 7f868e75f700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5215> 2014-06-09 16:22:12.200992 7f868cf5c700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5214> 2014-06-09 16:22:12.200996 7f868a757700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5213> 2014-06-09 16:22:12.201000 7f868ef60700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5212> 2014-06-09 16:22:12.201005 7f8690763700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5211> 2014-06-09 16:22:12.201009 7f8690f64700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5210> 2014-06-09 16:22:12.201013 7f8692767700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5209> 2014-06-09 16:22:12.201018 7f8691f66700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5208> 2014-06-09 16:22:12.201022 7f8691765700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5207> 2014-06-09 16:22:12.201026 7f8689755700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5206> 2014-06-09 16:22:12.201030 7f868d75d700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5205> 2014-06-09 16:22:12.201035 7f8689f56700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5204> 2014-06-09 16:22:12.201039 7f868ff62700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5203> 2014-06-09 16:22:12.201043 7f868af58700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5202> 2014-06-09 16:22:12.201047 7f868f761700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5201> 2014-06-09 16:22:12.201051 7f868df5e700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5200> 2014-06-09 16:22:12.201061 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbdf1dc0] -5199> 2014-06-09 16:22:12.201065 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbdf1dc0] -5198> 2014-06-09 16:22:12.201067 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5197> 2014-06-09 16:22:12.201068 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5196> 2014-06-09 16:22:12.201071 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497200 (1 active) -5195> 2014-06-09 16:22:12.201074 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3faea0 (2 active) -5194> 2014-06-09 16:22:12.201079 7f868bf5a700 10 op_submit_finish 4230 -5193> 2014-06-09 16:22:12.201082 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -5192> 2014-06-09 16:22:12.201095 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local -5191> 2014-06-09 16:22:12.201104 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -5190> 2014-06-09 16:22:12.201112 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5189> 2014-06-09 16:22:12.201120 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -5188> 2014-06-09 16:22:12.201128 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -5187> 2014-06-09 16:22:12.201141 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 -5186> 2014-06-09 16:22:12.201150 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -5185> 2014-06-09 16:22:12.201159 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[2.777( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop -5184> 2014-06-09 16:22:12.201168 7f868bf5a700 10 log is not dirty -5183> 2014-06-09 16:22:12.201172 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.582( v 214'602 lc 214'601 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) -5182> 2014-06-09 16:22:12.201186 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed -5181> 2014-06-09 16:22:12.201196 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed -5180> 2014-06-09 16:22:12.201205 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info -5179> 2014-06-09 16:22:12.201219 7f868bf5a700 10 osd.26 244 send_incremental_map 243 -> 244 to 0xbf7b080 172.16.96.201:6828/58271 -5178> 2014-06-09 16:22:12.201229 7f868bf5a700 1 -- 172.16.96.202:6850/30904 --> 172.16.96.201:6828/58271 -- osd_map(244..244 src has 1..244) v3 -- ?+0 0x2004f200 con 0xbf7b080 -5177> 2014-06-09 16:22:12.201240 7f868bf5a700 10 osd.26 243 note_peer_epoch osd.2 has 244 -5176> 2014-06-09 16:22:12.201243 7f868bf5a700 1 -- 172.16.96.202:6850/30904 --> osd.2 172.16.96.201:6828/58271 -- pg_info(1 pgs e243:3.582) v4 -- ?+0 0x3411ce00 -5175> 2014-06-09 16:22:12.201252 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1029 -5174> 2014-06-09 16:22:12.201267 7f868bf5a700 10 log is not dirty -5173> 2014-06-09 16:22:12.201271 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.428( v 214'563 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) -5172> 2014-06-09 16:22:12.201288 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.6 activated and committed -5171> 2014-06-09 16:22:12.201300 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 208/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed -5170> 2014-06-09 16:22:12.201311 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info -5169> 2014-06-09 16:22:12.201325 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 -5168> 2014-06-09 16:22:12.201328 7f868bf5a700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:3.428) v4 -- ?+0 0x3411cc40 -5167> 2014-06-09 16:22:12.201340 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:910 -5166> 2014-06-09 16:22:12.201368 7f868bf5a700 10 log is not dirty -5165> 2014-06-09 16:22:12.196639 7f86a68f6700 5 -- op tracker -- , seq: 3679, time: 2014-06-09 16:22:12.195281, event: throttled, op: osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -5164> 2014-06-09 16:22:12.201373 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -5163> 2014-06-09 16:22:12.201382 7f86a68f6700 5 -- op tracker -- , seq: 3679, time: 2014-06-09 16:22:12.196466, event: all_read, op: osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -5162> 2014-06-09 16:22:12.201384 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.034071 7 0.003518 -5161> 2014-06-09 16:22:12.201392 7f86a68f6700 5 -- op tracker -- , seq: 3679, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) -5160> 2014-06-09 16:22:12.201394 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -5159> 2014-06-09 16:22:12.201399 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -5158> 2014-06-09 16:22:12.201403 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -5157> 2014-06-09 16:22:12.201404 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -5156> 2014-06-09 16:22:12.201412 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -5155> 2014-06-09 16:22:12.201421 7f868bf5a700 5 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -5154> 2014-06-09 16:22:12.201429 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -5153> 2014-06-09 16:22:12.201429 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976dc00 prio 63 cost 847872 latency 0.006149 osd_op(client.4543.0:33383 rbd_data.115f6b8b4567.0000000000001692 [set-alloc-hint object_size 4194304 write_size 4194304,write 3346432~847872] 3.22035f2c RETRY=2 ack+ondisk+retry+write e243) v4 -5152> 2014-06-09 16:22:12.201436 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -5151> 2014-06-09 16:22:12.201445 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -5150> 2014-06-09 16:22:12.201455 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -5149> 2014-06-09 16:22:12.201464 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -5148> 2014-06-09 16:22:12.201475 7f868bf5a700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 -5147> 2014-06-09 16:22:12.201478 7f868bf5a700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:2.40f) v4 -- ?+0 0x3411ca80 -5146> 2014-06-09 16:22:12.201491 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 -5145> 2014-06-09 16:22:12.201520 7f868bf5a700 10 log is not dirty -5144> 2014-06-09 16:22:12.201524 7f868bf5a700 5 queue_transactions existing osr(2.40f 0x2ed1210)/0x2ed1210 -5143> 2014-06-09 16:22:12.201526 7f868bf5a700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5142> 2014-06-09 16:22:12.201531 7f86a5e11700 5 _do_op 0xcabc780 seq 4182 osr(2.216 0x2ed0b50)/0x2ed0b50 start -5141> 2014-06-09 16:22:12.201534 7f86a5e11700 10 _do_transaction on 0xc2efe00 -5140> 2014-06-09 16:22:12.201539 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5139> 2014-06-09 16:22:12.201647 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5138> 2014-06-09 16:22:12.201662 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4182.0.0 <= header.spos 150443.0.0 -5137> 2014-06-09 16:22:12.201666 7f86a5e11700 5 submit_transaction r = 0 -5136> 2014-06-09 16:22:12.201678 7f86a5e11700 10 _do_op 0xcabc780 seq 4182 r = 0, finisher 0x30e3ea00 0 -5135> 2014-06-09 16:22:12.201687 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3e9c0] -5134> 2014-06-09 16:22:12.201689 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3e9c0] -5133> 2014-06-09 16:22:12.201690 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5132> 2014-06-09 16:22:12.201691 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5131> 2014-06-09 16:22:12.201694 7f86a5e11700 10 _finish_op 0xcabc780 seq 4182 osr(2.216 0x2ed0b50)/0x2ed0b50 -5130> 2014-06-09 16:22:12.201702 7f8692f68700 10 op_submit_start 4231 -5129> 2014-06-09 16:22:12.201703 7f8692f68700 5 queue_transactions (trailing journal) 4231 -5128> 2014-06-09 16:22:12.201705 7f8692f68700 5 queue_op 0xd327260 seq 4231 osr(1.302 0x384d010) 828 bytes (queue has 50 ops and 4860842 bytes) -5127> 2014-06-09 16:22:12.201708 7f868b759700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5126> 2014-06-09 16:22:12.201712 7f868e75f700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5125> 2014-06-09 16:22:12.201715 7f868cf5c700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5124> 2014-06-09 16:22:12.201718 7f868a757700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5123> 2014-06-09 16:22:12.201721 7f868ef60700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5122> 2014-06-09 16:22:12.201725 7f8690763700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5121> 2014-06-09 16:22:12.201728 7f8690f64700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5120> 2014-06-09 16:22:12.201731 7f8692767700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5119> 2014-06-09 16:22:12.201734 7f8691f66700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5118> 2014-06-09 16:22:12.201737 7f8691765700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5117> 2014-06-09 16:22:12.201740 7f8689755700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5116> 2014-06-09 16:22:12.201743 7f868d75d700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5115> 2014-06-09 16:22:12.201746 7f8689f56700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5114> 2014-06-09 16:22:12.201749 7f868ff62700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5113> 2014-06-09 16:22:12.201752 7f868af58700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5112> 2014-06-09 16:22:12.201755 7f868f761700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5111> 2014-06-09 16:22:12.201758 7f868df5e700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5110> 2014-06-09 16:22:12.201761 7f868bf5a700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5109> 2014-06-09 16:22:12.201764 7f868c75b700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5108> 2014-06-09 16:22:12.201772 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e3ea00] -5107> 2014-06-09 16:22:12.201776 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e3ea00] -5106> 2014-06-09 16:22:12.201777 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5105> 2014-06-09 16:22:12.201778 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5104> 2014-06-09 16:22:12.201780 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3faea0 (1 active) -5103> 2014-06-09 16:22:12.201782 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac000 (2 active) -5102> 2014-06-09 16:22:12.201786 7f8692f68700 10 op_submit_finish 4231 -5101> 2014-06-09 16:22:12.201789 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 243/191 242/242/222) -5100> 2014-06-09 16:22:12.201821 7f8692f68700 10 log is not dirty -5099> 2014-06-09 16:22:12.201825 7f8692f68700 5 queue_transactions existing osr(3.111 0x3ed0830)/0x3ed0830 -5098> 2014-06-09 16:22:12.201827 7f8692f68700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5097> 2014-06-09 16:22:12.201831 7f86a5e11700 5 _do_op 0x39d98c0 seq 4183 osr(2.5f8 0x2ed1db0)/0x2ed1db0 start -5096> 2014-06-09 16:22:12.201834 7f86a5e11700 10 _do_transaction on 0xbfb2780 -5095> 2014-06-09 16:22:12.201838 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5094> 2014-06-09 16:22:12.201874 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5093> 2014-06-09 16:22:12.201886 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4183.0.0 <= header.spos 150443.0.0 -5092> 2014-06-09 16:22:12.201890 7f86a5e11700 5 submit_transaction r = 0 -5091> 2014-06-09 16:22:12.201899 7f86a5e11700 10 _do_op 0x39d98c0 seq 4183 r = 0, finisher 0x3042b500 0 -5090> 2014-06-09 16:22:12.201907 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6c6e0] -5089> 2014-06-09 16:22:12.201909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6c6e0] -5088> 2014-06-09 16:22:12.201910 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5087> 2014-06-09 16:22:12.201911 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5086> 2014-06-09 16:22:12.201914 7f86a5e11700 10 _finish_op 0x39d98c0 seq 4183 osr(2.5f8 0x2ed1db0)/0x2ed1db0 -5085> 2014-06-09 16:22:12.201922 7f868e75f700 10 op_submit_start 4232 -5084> 2014-06-09 16:22:12.201923 7f868e75f700 5 queue_transactions (trailing journal) 4232 -5083> 2014-06-09 16:22:12.201924 7f868e75f700 5 queue_op 0x324f4b40 seq 4232 osr(2.2f2 0x2ed0fb0) 828 bytes (queue has 50 ops and 4860842 bytes) -5082> 2014-06-09 16:22:12.201928 7f868cf5c700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5081> 2014-06-09 16:22:12.201931 7f868a757700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5080> 2014-06-09 16:22:12.201935 7f868ef60700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5079> 2014-06-09 16:22:12.201938 7f8690763700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5078> 2014-06-09 16:22:12.201941 7f8690f64700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5077> 2014-06-09 16:22:12.201944 7f8692767700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5076> 2014-06-09 16:22:12.201947 7f8691f66700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5075> 2014-06-09 16:22:12.201950 7f8691765700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5074> 2014-06-09 16:22:12.201953 7f8689755700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5073> 2014-06-09 16:22:12.201956 7f868d75d700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5072> 2014-06-09 16:22:12.201959 7f8689f56700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5071> 2014-06-09 16:22:12.201962 7f868ff62700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5070> 2014-06-09 16:22:12.201965 7f868af58700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5069> 2014-06-09 16:22:12.201968 7f868f761700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5068> 2014-06-09 16:22:12.201971 7f868df5e700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5067> 2014-06-09 16:22:12.201974 7f868bf5a700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5066> 2014-06-09 16:22:12.201977 7f868c75b700 2 waiting 51 > 50 ops || 4861703 > 104857600 -5065> 2014-06-09 16:22:12.201980 7f8692f68700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5064> 2014-06-09 16:22:12.201983 7f868b759700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5063> 2014-06-09 16:22:12.201990 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b500] -5062> 2014-06-09 16:22:12.201994 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b500] -5061> 2014-06-09 16:22:12.201995 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5060> 2014-06-09 16:22:12.201996 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5059> 2014-06-09 16:22:12.201998 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac000 (1 active) -5058> 2014-06-09 16:22:12.202000 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x383eea0 (2 active) -5057> 2014-06-09 16:22:12.202004 7f868e75f700 10 op_submit_finish 4232 -5056> 2014-06-09 16:22:12.202007 7f868e75f700 10 osd.26 pg_epoch: 243 pg[1.32f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.32f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) -5055> 2014-06-09 16:22:12.202035 7f868e75f700 10 log is not dirty -5054> 2014-06-09 16:22:12.202039 7f868e75f700 5 queue_transactions existing osr(1.32f 0x384d350)/0x384d350 -5053> 2014-06-09 16:22:12.202041 7f868e75f700 2 waiting 51 > 50 ops || 4861670 > 104857600 -5052> 2014-06-09 16:22:12.202045 7f86a5e11700 5 _do_op 0x324e61e0 seq 4184 osr(0.29 0x2bd62b0)/0x2bd62b0 start -5051> 2014-06-09 16:22:12.202048 7f86a5e11700 10 _do_transaction on 0x2ed4b40 -5050> 2014-06-09 16:22:12.202053 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -5049> 2014-06-09 16:22:12.202089 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -5048> 2014-06-09 16:22:12.202089 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 18 ==== osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+626688 (1791361836 0 0) 0x344d66c0 con 0x30c3c2c0 -5047> 2014-06-09 16:22:12.202105 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4184.0.0 <= header.spos 150443.0.0 -5046> 2014-06-09 16:22:12.202109 7f86a5e11700 5 submit_transaction r = 0 -5045> 2014-06-09 16:22:12.202111 7f86a68f6700 5 -- op tracker -- , seq: 3680, time: 2014-06-09 16:22:12.201451, event: header_read, op: osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -5044> 2014-06-09 16:22:12.202122 7f86a68f6700 5 -- op tracker -- , seq: 3680, time: 2014-06-09 16:22:12.201452, event: throttled, op: osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -5043> 2014-06-09 16:22:12.202132 7f86a5e11700 10 _do_op 0x324e61e0 seq 4184 r = 0, finisher 0xcc94340 0 -5042> 2014-06-09 16:22:12.202136 7f86a68f6700 5 -- op tracker -- , seq: 3680, time: 2014-06-09 16:22:12.202074, event: all_read, op: osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -5041> 2014-06-09 16:22:12.202142 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd622e0] -5040> 2014-06-09 16:22:12.202146 7f86a68f6700 5 -- op tracker -- , seq: 3680, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -5039> 2014-06-09 16:22:12.202145 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -5038> 2014-06-09 16:22:12.202151 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -5037> 2014-06-09 16:22:12.202155 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -5036> 2014-06-09 16:22:12.202154 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -5035> 2014-06-09 16:22:12.202163 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.29( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 -5034> 2014-06-09 16:22:12.202171 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd622e0] -5033> 2014-06-09 16:22:12.202173 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -5032> 2014-06-09 16:22:12.202174 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -5031> 2014-06-09 16:22:12.202168 7f86a68f6700 15 osd.26 244 enqueue_op 0x2976db00 prio 63 cost 626688 latency 0.000717 osd_op(client.4543.0:33503 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~626688] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 -5030> 2014-06-09 16:22:12.202177 7f86a5e11700 10 _finish_op 0x324e61e0 seq 4184 osr(0.29 0x2bd62b0)/0x2bd62b0 -5029> 2014-06-09 16:22:12.202187 7f868a757700 10 op_submit_start 4233 -5028> 2014-06-09 16:22:12.202189 7f868a757700 5 queue_transactions (trailing journal) 4233 -5027> 2014-06-09 16:22:12.202190 7f868a757700 5 queue_op 0xcabcfc0 seq 4233 osr(3.218 0x3ed0cd0) 828 bytes (queue has 50 ops and 4860812 bytes) -5026> 2014-06-09 16:22:12.202195 7f868ef60700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5025> 2014-06-09 16:22:12.202199 7f8690763700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5024> 2014-06-09 16:22:12.202204 7f8690f64700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5023> 2014-06-09 16:22:12.202207 7f8692767700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5022> 2014-06-09 16:22:12.202211 7f8691f66700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5021> 2014-06-09 16:22:12.202214 7f8691765700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5020> 2014-06-09 16:22:12.202218 7f8689755700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5019> 2014-06-09 16:22:12.202222 7f868d75d700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5018> 2014-06-09 16:22:12.202225 7f8689f56700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5017> 2014-06-09 16:22:12.202228 7f868ff62700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5016> 2014-06-09 16:22:12.202232 7f868af58700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5015> 2014-06-09 16:22:12.202235 7f868f761700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5014> 2014-06-09 16:22:12.202238 7f868df5e700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5013> 2014-06-09 16:22:12.202242 7f868bf5a700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5012> 2014-06-09 16:22:12.202246 7f868c75b700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5011> 2014-06-09 16:22:12.202249 7f8692f68700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5010> 2014-06-09 16:22:12.202253 7f868b759700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5009> 2014-06-09 16:22:12.202257 7f868e75f700 2 waiting 51 > 50 ops || 4861640 > 104857600 -5008> 2014-06-09 16:22:12.202260 7f868cf5c700 2 waiting 51 > 50 ops || 4861673 > 104857600 -5007> 2014-06-09 16:22:12.202269 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc94340] -5006> 2014-06-09 16:22:12.202282 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc94340] -5005> 2014-06-09 16:22:12.202283 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -5004> 2014-06-09 16:22:12.202284 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -5003> 2014-06-09 16:22:12.202287 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x383eea0 (1 active) -5002> 2014-06-09 16:22:12.202290 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc5250e0 (2 active) -5001> 2014-06-09 16:22:12.202295 7f868a757700 10 op_submit_finish 4233 -5000> 2014-06-09 16:22:12.202299 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.4ba( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.4ba( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) -4999> 2014-06-09 16:22:12.202331 7f868a757700 10 log is not dirty -4998> 2014-06-09 16:22:12.202337 7f868a757700 5 queue_transactions existing osr(0.4ba 0x2bd7710)/0x2bd7710 -4997> 2014-06-09 16:22:12.202340 7f868a757700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4996> 2014-06-09 16:22:12.202344 7f86a5e11700 5 _do_op 0x160967e0 seq 4185 osr(1.25a 0x384cfb0)/0x384cfb0 start -4995> 2014-06-09 16:22:12.202348 7f86a5e11700 10 _do_transaction on 0xc1449c0 -4994> 2014-06-09 16:22:12.202353 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4993> 2014-06-09 16:22:12.202395 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4992> 2014-06-09 16:22:12.202410 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4185.0.0 <= header.spos 150443.0.0 -4991> 2014-06-09 16:22:12.202414 7f86a5e11700 5 submit_transaction r = 0 -4990> 2014-06-09 16:22:12.202428 7f86a5e11700 10 _do_op 0x160967e0 seq 4185 r = 0, finisher 0x30629dc0 0 -4989> 2014-06-09 16:22:12.202437 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3e600] -4988> 2014-06-09 16:22:12.202440 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -4987> 2014-06-09 16:22:12.202448 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -4986> 2014-06-09 16:22:12.202456 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.25a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:164 -4985> 2014-06-09 16:22:12.202468 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3e600] -4984> 2014-06-09 16:22:12.202470 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4983> 2014-06-09 16:22:12.202471 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4982> 2014-06-09 16:22:12.202473 7f86a5e11700 10 _finish_op 0x160967e0 seq 4185 osr(1.25a 0x384cfb0)/0x384cfb0 -4981> 2014-06-09 16:22:12.202483 7f8690763700 10 op_submit_start 4234 -4980> 2014-06-09 16:22:12.202484 7f8690763700 5 queue_transactions (trailing journal) 4234 -4979> 2014-06-09 16:22:12.202485 7f8690763700 5 queue_op 0xc34a660 seq 4234 osr(3.65f 0x3ed1a10) 861 bytes (queue has 50 ops and 4860812 bytes) -4978> 2014-06-09 16:22:12.202489 7f8690f64700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4977> 2014-06-09 16:22:12.202492 7f8692767700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4976> 2014-06-09 16:22:12.202496 7f8691f66700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4975> 2014-06-09 16:22:12.202499 7f8691765700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4974> 2014-06-09 16:22:12.202502 7f8689755700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4973> 2014-06-09 16:22:12.202505 7f868d75d700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4972> 2014-06-09 16:22:12.202508 7f8689f56700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4971> 2014-06-09 16:22:12.202511 7f868ff62700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4970> 2014-06-09 16:22:12.202514 7f868af58700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4969> 2014-06-09 16:22:12.202517 7f868f761700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4968> 2014-06-09 16:22:12.202520 7f868df5e700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4967> 2014-06-09 16:22:12.202523 7f868bf5a700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4966> 2014-06-09 16:22:12.202526 7f868c75b700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4965> 2014-06-09 16:22:12.202529 7f8692f68700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4964> 2014-06-09 16:22:12.202532 7f868b759700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4963> 2014-06-09 16:22:12.202535 7f868e75f700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4962> 2014-06-09 16:22:12.202538 7f868cf5c700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4961> 2014-06-09 16:22:12.202541 7f868a757700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4960> 2014-06-09 16:22:12.202544 7f868ef60700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4959> 2014-06-09 16:22:12.202551 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629dc0] -4958> 2014-06-09 16:22:12.202555 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629dc0] -4957> 2014-06-09 16:22:12.202557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4956> 2014-06-09 16:22:12.202557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4955> 2014-06-09 16:22:12.202560 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc5250e0 (1 active) -4954> 2014-06-09 16:22:12.202562 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62eea0 (2 active) -4953> 2014-06-09 16:22:12.202566 7f8690763700 10 op_submit_finish 4234 -4952> 2014-06-09 16:22:12.202569 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -4951> 2014-06-09 16:22:12.202582 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -4950> 2014-06-09 16:22:12.202589 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -4949> 2014-06-09 16:22:12.202597 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4948> 2014-06-09 16:22:12.202604 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -4947> 2014-06-09 16:22:12.202610 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -4946> 2014-06-09 16:22:12.202622 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:180 -4945> 2014-06-09 16:22:12.202630 7f8690763700 20 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -4944> 2014-06-09 16:22:12.202636 7f8690763700 20 osd.26 pg_epoch: 243 pg[2.740( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -4943> 2014-06-09 16:22:12.202644 7f8690763700 10 log is not dirty -4942> 2014-06-09 16:22:12.202648 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -4941> 2014-06-09 16:22:12.202660 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] exit Started/Primary/Active/Activating 1.125374 7 0.001337 -4940> 2014-06-09 16:22:12.202671 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] enter Started/Primary/Active/Recovered -4939> 2014-06-09 16:22:12.202681 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] needs_recovery is recovered -4938> 2014-06-09 16:22:12.202690 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 -4937> 2014-06-09 16:22:12.202700 7f8690763700 5 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] enter Started/Primary/Active/Clean -4936> 2014-06-09 16:22:12.202708 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] finish_recovery -4935> 2014-06-09 16:22:12.202715 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active] clear_recovery_state -4934> 2014-06-09 16:22:12.202725 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -4933> 2014-06-09 16:22:12.202736 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -4932> 2014-06-09 16:22:12.202745 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] share_pg_info -4931> 2014-06-09 16:22:12.202760 7f8690763700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 -4930> 2014-06-09 16:22:12.202762 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:3.661) v4 -- ?+0 0x3343ce00 -4929> 2014-06-09 16:22:12.202757 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 19 ==== osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+516096 (1007502399 0 0) 0x344d6480 con 0x30c3c2c0 -4928> 2014-06-09 16:22:12.202775 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] publish_stats_to_osd 243:928 -4927> 2014-06-09 16:22:12.202807 7f8690763700 10 log is not dirty -4926> 2014-06-09 16:22:12.202805 7f86a68f6700 5 -- op tracker -- , seq: 3681, time: 2014-06-09 16:22:12.202189, event: header_read, op: osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4925> 2014-06-09 16:22:12.202811 7f8690763700 5 queue_transactions existing osr(3.661 0x3ed19f0)/0x3ed19f0 -4924> 2014-06-09 16:22:12.202813 7f8690763700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4923> 2014-06-09 16:22:12.202816 7f86a68f6700 5 -- op tracker -- , seq: 3681, time: 2014-06-09 16:22:12.202190, event: throttled, op: osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4922> 2014-06-09 16:22:12.202819 7f86a5e11700 5 _do_op 0x1e08a000 seq 4186 osr(0.678 0x2bd7e90)/0x2bd7e90 start -4921> 2014-06-09 16:22:12.202822 7f86a5e11700 10 _do_transaction on 0xc2d9c80 -4920> 2014-06-09 16:22:12.202828 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4919> 2014-06-09 16:22:12.202826 7f86a68f6700 5 -- op tracker -- , seq: 3681, time: 2014-06-09 16:22:12.202744, event: all_read, op: osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4918> 2014-06-09 16:22:12.202835 7f86a68f6700 5 -- op tracker -- , seq: 3681, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4917> 2014-06-09 16:22:12.202840 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -4916> 2014-06-09 16:22:12.202845 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -4915> 2014-06-09 16:22:12.202857 7f86a68f6700 15 osd.26 244 enqueue_op 0x1c54bb00 prio 63 cost 516096 latency 0.000668 osd_op(client.4543.0:33504 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 626688~516096] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 -4914> 2014-06-09 16:22:12.202867 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4913> 2014-06-09 16:22:12.202884 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4186.0.0 <= header.spos 150443.0.0 -4912> 2014-06-09 16:22:12.202888 7f86a5e11700 5 submit_transaction r = 0 -4911> 2014-06-09 16:22:12.202899 7f86a5e11700 10 _do_op 0x1e08a000 seq 4186 r = 0, finisher 0xccf9c00 0 -4910> 2014-06-09 16:22:12.202908 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6ace0] -4909> 2014-06-09 16:22:12.202911 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -4908> 2014-06-09 16:22:12.202920 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -4907> 2014-06-09 16:22:12.202928 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.678( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:187 -4906> 2014-06-09 16:22:12.202937 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6ace0] -4905> 2014-06-09 16:22:12.202938 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4904> 2014-06-09 16:22:12.202940 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4903> 2014-06-09 16:22:12.202943 7f86a5e11700 10 _finish_op 0x1e08a000 seq 4186 osr(0.678 0x2bd7e90)/0x2bd7e90 -4902> 2014-06-09 16:22:12.202953 7f8692767700 10 op_submit_start 4235 -4901> 2014-06-09 16:22:12.202955 7f8692767700 5 queue_transactions (trailing journal) 4235 -4900> 2014-06-09 16:22:12.202956 7f8692767700 5 queue_op 0xc2eaba0 seq 4235 osr(1.4bc 0x384d930) 861 bytes (queue has 50 ops and 4860812 bytes) -4899> 2014-06-09 16:22:12.202961 7f8691f66700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4898> 2014-06-09 16:22:12.202965 7f8691765700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4897> 2014-06-09 16:22:12.202969 7f8689755700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4896> 2014-06-09 16:22:12.202973 7f868d75d700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4895> 2014-06-09 16:22:12.202976 7f8689f56700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4894> 2014-06-09 16:22:12.202980 7f868ff62700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4893> 2014-06-09 16:22:12.202984 7f868af58700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4892> 2014-06-09 16:22:12.202987 7f868f761700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4891> 2014-06-09 16:22:12.202990 7f868df5e700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4890> 2014-06-09 16:22:12.202994 7f868bf5a700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4889> 2014-06-09 16:22:12.202997 7f868c75b700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4888> 2014-06-09 16:22:12.203001 7f8692f68700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4887> 2014-06-09 16:22:12.203005 7f868b759700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4886> 2014-06-09 16:22:12.203008 7f868e75f700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4885> 2014-06-09 16:22:12.203012 7f868cf5c700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4884> 2014-06-09 16:22:12.203016 7f868a757700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4883> 2014-06-09 16:22:12.203019 7f868ef60700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4882> 2014-06-09 16:22:12.203023 7f8690763700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4881> 2014-06-09 16:22:12.203026 7f8690f64700 2 waiting 51 > 50 ops || 4861640 > 104857600 -4880> 2014-06-09 16:22:12.203034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf9c00] -4879> 2014-06-09 16:22:12.203038 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf9c00] -4878> 2014-06-09 16:22:12.203040 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4877> 2014-06-09 16:22:12.203041 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4876> 2014-06-09 16:22:12.203043 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62eea0 (1 active) -4875> 2014-06-09 16:22:12.203046 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358000 (2 active) -4874> 2014-06-09 16:22:12.203051 7f8692767700 10 op_submit_finish 4235 -4873> 2014-06-09 16:22:12.203054 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -4872> 2014-06-09 16:22:12.203067 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.046040 7 0.000306 -4871> 2014-06-09 16:22:12.203078 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -4870> 2014-06-09 16:22:12.203087 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4869> 2014-06-09 16:22:12.203097 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -4868> 2014-06-09 16:22:12.203106 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -4867> 2014-06-09 16:22:12.203114 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -4866> 2014-06-09 16:22:12.203121 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -4865> 2014-06-09 16:22:12.203130 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) -4864> 2014-06-09 16:22:12.203141 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) -4863> 2014-06-09 16:22:12.203150 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) -4862> 2014-06-09 16:22:12.203159 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) -4861> 2014-06-09 16:22:12.203167 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -4860> 2014-06-09 16:22:12.203181 7f8692767700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 244 -4859> 2014-06-09 16:22:12.203184 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.1 172.16.96.201:6811/57862 -- pg_info(1 pgs e243:2.3de) v4 -- ?+0 0x31745340 -4858> 2014-06-09 16:22:12.203197 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 -4857> 2014-06-09 16:22:12.203229 7f8692767700 10 log is not dirty -4856> 2014-06-09 16:22:12.203233 7f8692767700 5 queue_transactions existing osr(2.3de 0x2ed1330)/0x2ed1330 -4855> 2014-06-09 16:22:12.203235 7f8692767700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4854> 2014-06-09 16:22:12.203261 7f86a5e11700 5 _do_op 0x30a3d40 seq 4187 osr(0.31f 0x2bd6d50)/0x2bd6d50 start -4853> 2014-06-09 16:22:12.203269 7f86a5e11700 10 _do_transaction on 0xc270b40 -4852> 2014-06-09 16:22:12.203274 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4851> 2014-06-09 16:22:12.203315 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4850> 2014-06-09 16:22:12.203329 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4187.0.0 <= header.spos 150443.0.0 -4849> 2014-06-09 16:22:12.203333 7f86a5e11700 5 submit_transaction r = 0 -4848> 2014-06-09 16:22:12.203345 7f86a5e11700 10 _do_op 0x30a3d40 seq 4187 r = 0, finisher 0x307f0340 0 -4847> 2014-06-09 16:22:12.203355 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x307f02c0] -4846> 2014-06-09 16:22:12.203357 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x307f02c0] -4845> 2014-06-09 16:22:12.203359 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4844> 2014-06-09 16:22:12.203360 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4843> 2014-06-09 16:22:12.203363 7f86a5e11700 10 _finish_op 0x30a3d40 seq 4187 osr(0.31f 0x2bd6d50)/0x2bd6d50 -4842> 2014-06-09 16:22:12.203373 7f8691765700 10 op_submit_start 4236 -4841> 2014-06-09 16:22:12.203374 7f8691765700 5 queue_transactions (trailing journal) 4236 -4840> 2014-06-09 16:22:12.203376 7f8691765700 5 queue_op 0xc64a960 seq 4236 osr(2.3c7 0x2ed13d0) 861 bytes (queue has 50 ops and 4860845 bytes) -4839> 2014-06-09 16:22:12.203380 7f8689755700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4838> 2014-06-09 16:22:12.203384 7f868d75d700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4837> 2014-06-09 16:22:12.203388 7f8689f56700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4836> 2014-06-09 16:22:12.203392 7f868ff62700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4835> 2014-06-09 16:22:12.203395 7f868af58700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4834> 2014-06-09 16:22:12.203399 7f868f761700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4833> 2014-06-09 16:22:12.203403 7f868df5e700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4832> 2014-06-09 16:22:12.203406 7f868bf5a700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4831> 2014-06-09 16:22:12.203410 7f868c75b700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4830> 2014-06-09 16:22:12.203414 7f8692f68700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4829> 2014-06-09 16:22:12.203418 7f868b759700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4828> 2014-06-09 16:22:12.203421 7f868e75f700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4827> 2014-06-09 16:22:12.203425 7f868cf5c700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4826> 2014-06-09 16:22:12.203428 7f868a757700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4825> 2014-06-09 16:22:12.203432 7f868ef60700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4824> 2014-06-09 16:22:12.203436 7f8690763700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4823> 2014-06-09 16:22:12.203440 7f8690f64700 2 waiting 51 > 50 ops || 4861673 > 104857600 -4822> 2014-06-09 16:22:12.203444 7f8692767700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4821> 2014-06-09 16:22:12.203447 7f8691f66700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4820> 2014-06-09 16:22:12.203456 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f0340] -4819> 2014-06-09 16:22:12.203459 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f0340] -4818> 2014-06-09 16:22:12.203461 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4817> 2014-06-09 16:22:12.203462 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4816> 2014-06-09 16:22:12.203464 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358000 (1 active) -4815> 2014-06-09 16:22:12.203467 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5680 (2 active) -4814> 2014-06-09 16:22:12.203472 7f8691765700 10 op_submit_finish 4236 -4813> 2014-06-09 16:22:12.203475 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -4812> 2014-06-09 16:22:12.203555 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -4811> 2014-06-09 16:22:12.203564 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16bd6e00,0x21967900,0x16bd6d00,0x21967800,0x1c54ae00,0x1c54ad00,0x21967700,0x21967600 -4810> 2014-06-09 16:22:12.203595 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -4809> 2014-06-09 16:22:12.203605 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -4808> 2014-06-09 16:22:12.203630 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -4807> 2014-06-09 16:22:12.203645 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:965 -4806> 2014-06-09 16:22:12.203655 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -4805> 2014-06-09 16:22:12.203664 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.1f4( v 213'599 (0'0,213'599] local-les=243 n=118 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -4804> 2014-06-09 16:22:12.203674 7f8691765700 10 log is not dirty -4803> 2014-06-09 16:22:12.203679 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -4802> 2014-06-09 16:22:12.203690 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.051479 7 0.001368 -4801> 2014-06-09 16:22:12.203701 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -4800> 2014-06-09 16:22:12.203711 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4799> 2014-06-09 16:22:12.203721 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 -4798> 2014-06-09 16:22:12.203730 7f8691765700 5 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -4797> 2014-06-09 16:22:12.203738 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -4796> 2014-06-09 16:22:12.203745 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -4795> 2014-06-09 16:22:12.203755 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -4794> 2014-06-09 16:22:12.203765 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -4793> 2014-06-09 16:22:12.203774 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -4792> 2014-06-09 16:22:12.203786 7f8691765700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 -4791> 2014-06-09 16:22:12.203788 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:1.65d) v4 -- ?+0 0x30a188c0 -4790> 2014-06-09 16:22:12.203802 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:162 -4789> 2014-06-09 16:22:12.203833 7f8691765700 10 log is not dirty -4788> 2014-06-09 16:22:12.203838 7f86a5e11700 5 _do_op 0x2c06eba0 seq 4188 osr(1.4fb 0x384d7f0)/0x384d7f0 start -4787> 2014-06-09 16:22:12.203846 7f8691765700 5 queue_transactions existing osr(1.65d 0x384df50)/0x384df50 -4786> 2014-06-09 16:22:12.203849 7f8691765700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4785> 2014-06-09 16:22:12.203842 7f86a5e11700 10 _do_transaction on 0xc0d9e00 -4784> 2014-06-09 16:22:12.203858 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4783> 2014-06-09 16:22:12.203900 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4782> 2014-06-09 16:22:12.203915 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4188.0.0 <= header.spos 150443.0.0 -4781> 2014-06-09 16:22:12.203919 7f86a5e11700 5 submit_transaction r = 0 -4780> 2014-06-09 16:22:12.203955 7f86a5e11700 10 _do_op 0x2c06eba0 seq 4188 r = 0, finisher 0xbd67540 0 -4779> 2014-06-09 16:22:12.203966 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205da60] -4778> 2014-06-09 16:22:12.203968 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205da60] -4777> 2014-06-09 16:22:12.203970 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4776> 2014-06-09 16:22:12.203971 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4775> 2014-06-09 16:22:12.203974 7f86a5e11700 10 _finish_op 0x2c06eba0 seq 4188 osr(1.4fb 0x384d7f0)/0x384d7f0 -4774> 2014-06-09 16:22:12.203985 7f868d75d700 10 op_submit_start 4237 -4773> 2014-06-09 16:22:12.203986 7f868d75d700 5 queue_transactions (trailing journal) 4237 -4772> 2014-06-09 16:22:12.203988 7f868d75d700 5 queue_op 0x308cb7a0 seq 4237 osr(2.5f4 0x2ed1dd0) 861 bytes (queue has 50 ops and 4860878 bytes) -4771> 2014-06-09 16:22:12.203993 7f8689f56700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4770> 2014-06-09 16:22:12.203997 7f868ff62700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4769> 2014-06-09 16:22:12.204001 7f868af58700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4768> 2014-06-09 16:22:12.204005 7f868f761700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4767> 2014-06-09 16:22:12.204009 7f868df5e700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4766> 2014-06-09 16:22:12.204013 7f868bf5a700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4765> 2014-06-09 16:22:12.204016 7f868c75b700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4764> 2014-06-09 16:22:12.204020 7f8692f68700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4763> 2014-06-09 16:22:12.204024 7f868b759700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4762> 2014-06-09 16:22:12.204027 7f868e75f700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4761> 2014-06-09 16:22:12.204031 7f868cf5c700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4760> 2014-06-09 16:22:12.204035 7f868a757700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4759> 2014-06-09 16:22:12.204039 7f868ef60700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4758> 2014-06-09 16:22:12.204044 7f8690763700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4757> 2014-06-09 16:22:12.204048 7f8690f64700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4756> 2014-06-09 16:22:12.204053 7f8692767700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4755> 2014-06-09 16:22:12.204057 7f8691f66700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4754> 2014-06-09 16:22:12.204061 7f8691765700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4753> 2014-06-09 16:22:12.204065 7f8689755700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4752> 2014-06-09 16:22:12.204074 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd67540] -4751> 2014-06-09 16:22:12.204084 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd67540] -4750> 2014-06-09 16:22:12.204085 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4749> 2014-06-09 16:22:12.204086 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4748> 2014-06-09 16:22:12.204089 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5680 (1 active) -4747> 2014-06-09 16:22:12.204092 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fdd40 (2 active) -4746> 2014-06-09 16:22:12.204097 7f868d75d700 10 op_submit_finish 4237 -4745> 2014-06-09 16:22:12.204101 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -4744> 2014-06-09 16:22:12.204116 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] check_local -4743> 2014-06-09 16:22:12.204125 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -4742> 2014-06-09 16:22:12.204133 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4741> 2014-06-09 16:22:12.204141 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -4740> 2014-06-09 16:22:12.204148 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -4739> 2014-06-09 16:22:12.204161 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:159 -4738> 2014-06-09 16:22:12.204169 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -4737> 2014-06-09 16:22:12.204177 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.479( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] agent_stop -4736> 2014-06-09 16:22:12.204186 7f868d75d700 10 log is not dirty -4735> 2014-06-09 16:22:12.204191 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -4734> 2014-06-09 16:22:12.204203 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.103693 7 0.000357 -4733> 2014-06-09 16:22:12.204213 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -4732> 2014-06-09 16:22:12.204222 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4731> 2014-06-09 16:22:12.204231 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -4730> 2014-06-09 16:22:12.204241 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -4729> 2014-06-09 16:22:12.204248 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -4728> 2014-06-09 16:22:12.204255 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -4727> 2014-06-09 16:22:12.204264 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) -4726> 2014-06-09 16:22:12.204275 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) -4725> 2014-06-09 16:22:12.204284 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -4724> 2014-06-09 16:22:12.204297 7f868d75d700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 244 -4723> 2014-06-09 16:22:12.204299 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.7 172.16.96.201:6851/52636 -- pg_info(1 pgs e243:0.6b3) v4 -- ?+0 0x30cf1a40 -4722> 2014-06-09 16:22:12.204313 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 -4721> 2014-06-09 16:22:12.204343 7f868d75d700 10 log is not dirty -4720> 2014-06-09 16:22:12.204347 7f868d75d700 5 queue_transactions existing osr(0.6b3 0x2bd7ff0)/0x2bd7ff0 -4719> 2014-06-09 16:22:12.204349 7f868d75d700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4718> 2014-06-09 16:22:12.204354 7f86a5e11700 5 _do_op 0x324db080 seq 4189 osr(1.14c 0x384c810)/0x384c810 start -4717> 2014-06-09 16:22:12.204357 7f86a5e11700 10 _do_transaction on 0xbf44cc0 -4716> 2014-06-09 16:22:12.204363 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4715> 2014-06-09 16:22:12.204405 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4714> 2014-06-09 16:22:12.204420 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4189.0.0 <= header.spos 150443.0.0 -4713> 2014-06-09 16:22:12.204424 7f86a5e11700 5 submit_transaction r = 0 -4712> 2014-06-09 16:22:12.204434 7f86a5e11700 10 _do_op 0x324db080 seq 4189 r = 0, finisher 0x4eb8b60 0 -4711> 2014-06-09 16:22:12.204444 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314574e0] -4710> 2014-06-09 16:22:12.204446 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314574e0] -4709> 2014-06-09 16:22:12.204451 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4708> 2014-06-09 16:22:12.204452 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4707> 2014-06-09 16:22:12.204455 7f86a5e11700 10 _finish_op 0x324db080 seq 4189 osr(1.14c 0x384c810)/0x384c810 -4706> 2014-06-09 16:22:12.204465 7f868ff62700 10 op_submit_start 4238 -4705> 2014-06-09 16:22:12.204466 7f868ff62700 5 queue_transactions (trailing journal) 4238 -4704> 2014-06-09 16:22:12.204468 7f868ff62700 5 queue_op 0x325034a0 seq 4238 osr(0.6f0 0x384c170) 828 bytes (queue has 50 ops and 4860878 bytes) -4703> 2014-06-09 16:22:12.204473 7f868af58700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4702> 2014-06-09 16:22:12.204477 7f868f761700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4701> 2014-06-09 16:22:12.204481 7f868df5e700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4700> 2014-06-09 16:22:12.204484 7f868bf5a700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4699> 2014-06-09 16:22:12.204488 7f868c75b700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4698> 2014-06-09 16:22:12.204492 7f8692f68700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4697> 2014-06-09 16:22:12.204496 7f868b759700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4696> 2014-06-09 16:22:12.204500 7f868e75f700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4695> 2014-06-09 16:22:12.204504 7f868cf5c700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4694> 2014-06-09 16:22:12.204508 7f868a757700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4693> 2014-06-09 16:22:12.204512 7f868ef60700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4692> 2014-06-09 16:22:12.204515 7f8690763700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4691> 2014-06-09 16:22:12.204519 7f8690f64700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4690> 2014-06-09 16:22:12.204523 7f8692767700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4689> 2014-06-09 16:22:12.204527 7f8691f66700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4688> 2014-06-09 16:22:12.204531 7f8691765700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4687> 2014-06-09 16:22:12.204535 7f8689755700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4686> 2014-06-09 16:22:12.204539 7f868d75d700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4685> 2014-06-09 16:22:12.204542 7f8689f56700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4684> 2014-06-09 16:22:12.204555 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fdd40 (1 active) -4683> 2014-06-09 16:22:12.204551 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x4eb8b60] -4682> 2014-06-09 16:22:12.206060 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x4eb8b60] -4681> 2014-06-09 16:22:12.206064 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4680> 2014-06-09 16:22:12.206065 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4679> 2014-06-09 16:22:12.206068 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e240 (2 active) -4678> 2014-06-09 16:22:12.206073 7f868ff62700 10 op_submit_finish 4238 -4677> 2014-06-09 16:22:12.206078 7f868ff62700 10 osd.26 pg_epoch: 243 pg[0.448( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 0.448( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) -4676> 2014-06-09 16:22:12.206117 7f868ff62700 10 log is not dirty -4675> 2014-06-09 16:22:12.206122 7f868ff62700 5 queue_transactions existing osr(0.448 0x2bd7190)/0x2bd7190 -4674> 2014-06-09 16:22:12.206124 7f868ff62700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4673> 2014-06-09 16:22:12.206128 7f86a5e11700 5 _do_op 0xd3274a0 seq 4190 osr(1.375 0x384d230)/0x384d230 start -4672> 2014-06-09 16:22:12.206132 7f86a5e11700 10 _do_transaction on 0xc2ae6c0 -4671> 2014-06-09 16:22:12.206136 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4670> 2014-06-09 16:22:12.206181 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4669> 2014-06-09 16:22:12.206194 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4190.0.0 <= header.spos 150443.0.0 -4668> 2014-06-09 16:22:12.206198 7f86a5e11700 5 submit_transaction r = 0 -4667> 2014-06-09 16:22:12.206208 7f86a5e11700 10 _do_op 0xd3274a0 seq 4190 r = 0, finisher 0x306280c0 0 -4666> 2014-06-09 16:22:12.206217 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x320747c0] -4665> 2014-06-09 16:22:12.206219 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x320747c0] -4664> 2014-06-09 16:22:12.206221 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4663> 2014-06-09 16:22:12.206222 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4662> 2014-06-09 16:22:12.206224 7f86a5e11700 10 _finish_op 0xd3274a0 seq 4190 osr(1.375 0x384d230)/0x384d230 -4661> 2014-06-09 16:22:12.206234 7f868f761700 10 op_submit_start 4239 -4660> 2014-06-09 16:22:12.206235 7f868f761700 5 queue_transactions (trailing journal) 4239 -4659> 2014-06-09 16:22:12.206236 7f868f761700 5 queue_op 0x324e7c80 seq 4239 osr(1.48e 0x384d9b0) 828 bytes (queue has 50 ops and 4860878 bytes) -4658> 2014-06-09 16:22:12.206241 7f868df5e700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4657> 2014-06-09 16:22:12.206246 7f868bf5a700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4656> 2014-06-09 16:22:12.206249 7f868c75b700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4655> 2014-06-09 16:22:12.206253 7f8692f68700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4654> 2014-06-09 16:22:12.206257 7f868b759700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4653> 2014-06-09 16:22:12.206261 7f868e75f700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4652> 2014-06-09 16:22:12.206264 7f868cf5c700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4651> 2014-06-09 16:22:12.206267 7f868a757700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4650> 2014-06-09 16:22:12.206270 7f868ef60700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4649> 2014-06-09 16:22:12.206274 7f8690763700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4648> 2014-06-09 16:22:12.206278 7f8690f64700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4647> 2014-06-09 16:22:12.206281 7f8692767700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4646> 2014-06-09 16:22:12.206284 7f8691f66700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4645> 2014-06-09 16:22:12.206287 7f8691765700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4644> 2014-06-09 16:22:12.206291 7f8689755700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4643> 2014-06-09 16:22:12.206294 7f868d75d700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4642> 2014-06-09 16:22:12.206297 7f8689f56700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4641> 2014-06-09 16:22:12.206301 7f868ff62700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4640> 2014-06-09 16:22:12.206304 7f868af58700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4639> 2014-06-09 16:22:12.206311 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306280c0] -4638> 2014-06-09 16:22:12.206315 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306280c0] -4637> 2014-06-09 16:22:12.206316 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4636> 2014-06-09 16:22:12.206317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4635> 2014-06-09 16:22:12.206319 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e240 (1 active) -4634> 2014-06-09 16:22:12.206321 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300000 (2 active) -4633> 2014-06-09 16:22:12.206325 7f868f761700 10 op_submit_finish 4239 -4632> 2014-06-09 16:22:12.206328 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.66e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=65-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.66e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) -4631> 2014-06-09 16:22:12.206357 7f868f761700 10 log is not dirty -4630> 2014-06-09 16:22:12.206362 7f868f761700 5 queue_transactions existing osr(1.66e 0x384ded0)/0x384ded0 -4629> 2014-06-09 16:22:12.206364 7f868f761700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4628> 2014-06-09 16:22:12.206367 7f86a5e11700 5 _do_op 0xd327ce0 seq 4191 osr(1.24c 0x384cfd0)/0x384cfd0 start -4627> 2014-06-09 16:22:12.206370 7f86a5e11700 10 _do_transaction on 0x3472240 -4626> 2014-06-09 16:22:12.206375 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4625> 2014-06-09 16:22:12.206410 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4624> 2014-06-09 16:22:12.206422 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4191.0.0 <= header.spos 150443.0.0 -4623> 2014-06-09 16:22:12.206425 7f86a5e11700 5 submit_transaction r = 0 -4622> 2014-06-09 16:22:12.206435 7f86a5e11700 10 _do_op 0xd327ce0 seq 4191 r = 0, finisher 0xc169f40 0 -4621> 2014-06-09 16:22:12.206443 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d16c0] -4620> 2014-06-09 16:22:12.206446 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d16c0] -4619> 2014-06-09 16:22:12.206447 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4618> 2014-06-09 16:22:12.206448 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4617> 2014-06-09 16:22:12.206450 7f86a5e11700 10 _finish_op 0xd327ce0 seq 4191 osr(1.24c 0x384cfd0)/0x384cfd0 -4616> 2014-06-09 16:22:12.206458 7f868bf5a700 10 op_submit_start 4240 -4615> 2014-06-09 16:22:12.206459 7f868bf5a700 5 queue_transactions (trailing journal) 4240 -4614> 2014-06-09 16:22:12.206461 7f868bf5a700 5 queue_op 0x324e6120 seq 4240 osr(2.40f 0x2ed1210) 861 bytes (queue has 50 ops and 4860911 bytes) -4613> 2014-06-09 16:22:12.206465 7f868c75b700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4612> 2014-06-09 16:22:12.206468 7f8692f68700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4611> 2014-06-09 16:22:12.206471 7f868b759700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4610> 2014-06-09 16:22:12.206474 7f868e75f700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4609> 2014-06-09 16:22:12.206477 7f868cf5c700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4608> 2014-06-09 16:22:12.206480 7f868a757700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4607> 2014-06-09 16:22:12.206483 7f868ef60700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4606> 2014-06-09 16:22:12.206487 7f8690763700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4605> 2014-06-09 16:22:12.206490 7f8690f64700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4604> 2014-06-09 16:22:12.206493 7f8692767700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4603> 2014-06-09 16:22:12.206496 7f8691f66700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4602> 2014-06-09 16:22:12.206499 7f8691765700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4601> 2014-06-09 16:22:12.206501 7f8689755700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4600> 2014-06-09 16:22:12.206505 7f868d75d700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4599> 2014-06-09 16:22:12.206508 7f8689f56700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4598> 2014-06-09 16:22:12.206511 7f868ff62700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4597> 2014-06-09 16:22:12.206514 7f868af58700 2 waiting 51 > 50 ops || 4861772 > 104857600 -4596> 2014-06-09 16:22:12.206517 7f868f761700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4595> 2014-06-09 16:22:12.206520 7f868df5e700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4594> 2014-06-09 16:22:12.206527 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc169f40] -4593> 2014-06-09 16:22:12.206531 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc169f40] -4592> 2014-06-09 16:22:12.206532 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4591> 2014-06-09 16:22:12.206533 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4590> 2014-06-09 16:22:12.206535 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300000 (1 active) -4589> 2014-06-09 16:22:12.206537 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4067e0 (2 active) -4588> 2014-06-09 16:22:12.206541 7f868bf5a700 10 op_submit_finish 4240 -4587> 2014-06-09 16:22:12.206566 7f868bf5a700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41980 (19 active) -4586> 2014-06-09 16:22:12.206574 7f868bf5a700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -4585> 2014-06-09 16:22:12.206578 7f868bf5a700 10 osd.26 244 dequeue_op 0x306b400 prio 63 cost 180224 latency 2.141140 osd_op(client.4571.0:35796 rbd_data.117b6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~180224] 3.d889555d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] -4584> 2014-06-09 16:22:12.206602 7f868bf5a700 5 -- op tracker -- , seq: 69, time: 2014-06-09 16:22:12.206597, event: reached_pg, op: osd_op(client.4571.0:35796 rbd_data.117b6b8b4567.00000000000017ea [set-alloc-hint object_size 4194304 write_size 4194304,write 0~180224] 3.d889555d RETRY=3 ack+ondisk+retry+write e242) -4583> 2014-06-09 16:22:12.206611 7f868bf5a700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -4582> 2014-06-09 16:22:12.206625 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x306b400 -4581> 2014-06-09 16:22:12.206634 7f868bf5a700 10 osd.26 244 dequeue_op 0x306b400 finish -4580> 2014-06-09 16:22:12.206637 7f868bf5a700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -4579> 2014-06-09 16:22:12.206630 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 20 ==== osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+2580480 (3854738583 0 0) 0x344d6240 con 0x30c3c2c0 -4578> 2014-06-09 16:22:12.206650 7f86a68f6700 5 -- op tracker -- , seq: 3682, time: 2014-06-09 16:22:12.202878, event: header_read, op: osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4577> 2014-06-09 16:22:12.206661 7f86a68f6700 5 -- op tracker -- , seq: 3682, time: 2014-06-09 16:22:12.202879, event: throttled, op: osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4576> 2014-06-09 16:22:12.206673 7f868bf5a700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcb41980 (20 active) -4575> 2014-06-09 16:22:12.206670 7f86a68f6700 5 -- op tracker -- , seq: 3682, time: 2014-06-09 16:22:12.206613, event: all_read, op: osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4574> 2014-06-09 16:22:12.206681 7f86a68f6700 5 -- op tracker -- , seq: 3682, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4573> 2014-06-09 16:22:12.206677 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -4572> 2014-06-09 16:22:12.206688 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -4571> 2014-06-09 16:22:12.206692 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -4570> 2014-06-09 16:22:12.206693 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.582( v 214'602 (0'0,214'602] local-les=243 n=119 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x1804d700,0x1804d600,0x1804d500,0x1804d400,0x1804d300,0x4678c00,0x1804d200,0x4678b00,0x1804d100,0x4678a00,0x1c768e00,0x4678900,0x2c06bf00,0x2c06be00 -4569> 2014-06-09 16:22:12.206710 7f86a68f6700 15 osd.26 244 enqueue_op 0x270bff00 prio 63 cost 2580480 latency 0.003832 osd_op(client.4543.0:33505 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 1142784~2580480] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 -4568> 2014-06-09 16:22:12.206745 7f868bf5a700 10 log is not dirty -4567> 2014-06-09 16:22:12.206751 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -4566> 2014-06-09 16:22:12.206768 7f868bf5a700 10 log is not dirty -4565> 2014-06-09 16:22:12.206772 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -4564> 2014-06-09 16:22:12.206785 7f868bf5a700 15 osd.26 pg_epoch: 243 pg[3.428( v 214'563 lc 214'562 (0'0,214'563] local-les=243 n=111 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=214'563 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x16059b00,0x1804dd00,0x1804dc00,0x1c66a200,0x1c66a100,0x1c66a000,0x1b0d6200,0x1b0d6100,0x1b0d6000,0x22008600,0x1c66a800,0x22008500,0x1c66a700,0x22008400,0x1c66a600,0x22008300,0x22008200,0x22008100,0x22008000,0x270bf000,0x270bef00,0x270bee00,0x270bed00,0x270bec00,0x270beb00 -4563> 2014-06-09 16:22:12.206856 7f868bf5a700 10 log is not dirty -4562> 2014-06-09 16:22:12.206861 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.251( v 213'625 (0'0,213'625] local-les=243 n=121 ec=118 les/c 243/191 242/242/223) -4561> 2014-06-09 16:22:12.206893 7f868bf5a700 10 log is not dirty -4560> 2014-06-09 16:22:12.206897 7f868bf5a700 5 queue_transactions existing osr(3.251 0x3ed0c20)/0x3ed0c20 -4559> 2014-06-09 16:22:12.206899 7f868bf5a700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4558> 2014-06-09 16:22:12.206725 7f86a5e11700 5 _do_op 0x16097020 seq 4192 osr(3.362 0x3ed10a0)/0x3ed10a0 start -4557> 2014-06-09 16:22:12.206906 7f86a5e11700 10 _do_transaction on 0x97bb8c0 -4556> 2014-06-09 16:22:12.206912 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4555> 2014-06-09 16:22:12.206953 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4554> 2014-06-09 16:22:12.206970 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4192.0.0 <= header.spos 150443.0.0 -4553> 2014-06-09 16:22:12.206974 7f86a5e11700 5 submit_transaction r = 0 -4552> 2014-06-09 16:22:12.206988 7f86a5e11700 10 _do_op 0x16097020 seq 4192 r = 0, finisher 0x30e3e980 0 -4551> 2014-06-09 16:22:12.206998 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31791120] -4550> 2014-06-09 16:22:12.207001 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery -4549> 2014-06-09 16:22:12.207014 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays -4548> 2014-06-09 16:22:12.207024 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.362( v 213'616 (0'0,213'616] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1010 -4547> 2014-06-09 16:22:12.207034 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31791120] -4546> 2014-06-09 16:22:12.207036 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4545> 2014-06-09 16:22:12.207037 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4544> 2014-06-09 16:22:12.207041 7f86a5e11700 10 _finish_op 0x16097020 seq 4192 osr(3.362 0x3ed10a0)/0x3ed10a0 -4543> 2014-06-09 16:22:12.207051 7f8692f68700 10 op_submit_start 4241 -4542> 2014-06-09 16:22:12.207053 7f8692f68700 5 queue_transactions (trailing journal) 4241 -4541> 2014-06-09 16:22:12.207054 7f8692f68700 5 queue_op 0x2fc8dc80 seq 4241 osr(3.111 0x3ed0830) 828 bytes (queue has 50 ops and 4860878 bytes) -4540> 2014-06-09 16:22:12.207059 7f868b759700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4539> 2014-06-09 16:22:12.207063 7f868e75f700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4538> 2014-06-09 16:22:12.207066 7f868cf5c700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4537> 2014-06-09 16:22:12.207069 7f868a757700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4536> 2014-06-09 16:22:12.207072 7f868ef60700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4535> 2014-06-09 16:22:12.207075 7f8690763700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4534> 2014-06-09 16:22:12.207078 7f8690f64700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4533> 2014-06-09 16:22:12.207081 7f8692767700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4532> 2014-06-09 16:22:12.207084 7f8691f66700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4531> 2014-06-09 16:22:12.207087 7f8691765700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4530> 2014-06-09 16:22:12.207090 7f8689755700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4529> 2014-06-09 16:22:12.207093 7f868d75d700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4528> 2014-06-09 16:22:12.207097 7f8689f56700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4527> 2014-06-09 16:22:12.207100 7f868ff62700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4526> 2014-06-09 16:22:12.207103 7f868af58700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4525> 2014-06-09 16:22:12.207107 7f868f761700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4524> 2014-06-09 16:22:12.207110 7f868df5e700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4523> 2014-06-09 16:22:12.207113 7f868bf5a700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4522> 2014-06-09 16:22:12.207116 7f868c75b700 2 waiting 51 > 50 ops || 4861739 > 104857600 -4521> 2014-06-09 16:22:12.207123 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e3e980] -4520> 2014-06-09 16:22:12.207127 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e3e980] -4519> 2014-06-09 16:22:12.207128 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4518> 2014-06-09 16:22:12.207129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4517> 2014-06-09 16:22:12.207131 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4067e0 (1 active) -4516> 2014-06-09 16:22:12.207133 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc359b00 (2 active) -4515> 2014-06-09 16:22:12.207137 7f8692f68700 10 op_submit_finish 4241 -4514> 2014-06-09 16:22:12.207140 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 243/213 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.5f8( v 213'577 (0'0,213'577] local-les=243 n=119 ec=118 les/c 243/243 242/242/211) -4513> 2014-06-09 16:22:12.207171 7f8692f68700 10 log is not dirty -4512> 2014-06-09 16:22:12.207176 7f8692f68700 5 queue_transactions existing osr(3.5f8 0x3ed1b30)/0x3ed1b30 -4511> 2014-06-09 16:22:12.207178 7f8692f68700 2 waiting 51 > 50 ops || 4861706 > 104857600 -4510> 2014-06-09 16:22:12.207182 7f86a5e11700 5 _do_op 0x3280b1a0 seq 4193 osr(3.7f 0x3ed0200)/0x3ed0200 start -4509> 2014-06-09 16:22:12.207185 7f86a5e11700 10 _do_transaction on 0xc29b380 -4508> 2014-06-09 16:22:12.207189 7f86a5e11700 15 _omap_setkeys meta/516fc4fc/pglog_3.7f/0//-1 -4507> 2014-06-09 16:22:12.207232 7f86a5e11700 10 lookup_strip_header done cid meta oid 516fc4fc/pglog_3.7f/0//-1 -4506> 2014-06-09 16:22:12.207247 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4505> 2014-06-09 16:22:12.207280 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4504> 2014-06-09 16:22:12.207295 7f86a5e11700 15 _omap_rmkeys meta/516fc4fc/pglog_3.7f/0//-1 -4503> 2014-06-09 16:22:12.207299 7f86a5e11700 10 _omap_rmkeys meta/516fc4fc/pglog_3.7f/0//-1 r = 0 -4502> 2014-06-09 16:22:12.207304 7f86a5e11700 15 _omap_setkeys meta/516fc4fc/pglog_3.7f/0//-1 -4501> 2014-06-09 16:22:12.207375 7f86a5e11700 15 _touch 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -4500> 2014-06-09 16:22:12.207381 7f86a68f6700 1 -- 172.16.96.202:6830/30904 <== client.4543 172.16.96.220:0/1123776 21 ==== osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 ==== 227+0+471040 (151452 0 0) 0x344d6000 con 0x30c3c2c0 -4499> 2014-06-09 16:22:12.207402 7f86a68f6700 5 -- op tracker -- , seq: 3683, time: 2014-06-09 16:22:12.206738, event: header_read, op: osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4498> 2014-06-09 16:22:12.207414 7f86a68f6700 5 -- op tracker -- , seq: 3683, time: 2014-06-09 16:22:12.206740, event: throttled, op: osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4497> 2014-06-09 16:22:12.207419 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.7f_head oid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -4496> 2014-06-09 16:22:12.207423 7f86a68f6700 5 -- op tracker -- , seq: 3683, time: 2014-06-09 16:22:12.207369, event: all_read, op: osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4495> 2014-06-09 16:22:12.207432 7f86a68f6700 5 -- op tracker -- , seq: 3683, time: 0.000000, event: dispatched, op: osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) -4494> 2014-06-09 16:22:12.207437 7f86a68f6700 20 osd.26 244 should_share_map client.4543 172.16.96.220:0/1123776 243 -4493> 2014-06-09 16:22:12.207442 7f86a68f6700 20 osd.26 244 client session last_sent_epoch: 0 versus osdmap epoch 244 -4492> 2014-06-09 16:22:12.207448 7f86a5e11700 20 write_state seq is 30169 -4491> 2014-06-09 16:22:12.207456 7f86a5e11700 20 set_header setting 30168 cid 3.7f_head oid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 parent seq 0 -4490> 2014-06-09 16:22:12.207454 7f86a68f6700 15 osd.26 244 enqueue_op 0x2f097b00 prio 63 cost 471040 latency 0.000716 osd_op(client.4543.0:33506 rbd_data.115f6b8b4567.00000000000016a4 [set-alloc-hint object_size 4194304 write_size 4194304,write 3723264~471040] 3.547499ff RETRY=2 ack+ondisk+retry+write e243) v4 -4489> 2014-06-09 16:22:12.207469 7f86a5e11700 10 _touch 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 = 0 -4488> 2014-06-09 16:22:12.207476 7f86a5e11700 15 _write 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 0~237568 -4487> 2014-06-09 16:22:12.207484 7f86a5e11700 10 file_to_extents done -4486> 2014-06-09 16:22:12.208074 7f86a5e11700 10 _generic_write 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 0~237568 = 0 -4485> 2014-06-09 16:22:12.208089 7f86a5e11700 15 _setattrs 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -4484> 2014-06-09 16:22:12.208095 7f86a5e11700 10 _setattrs 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 = 0 -4483> 2014-06-09 16:22:12.208099 7f86a5e11700 15 _setattrs 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 -4482> 2014-06-09 16:22:12.208103 7f86a5e11700 10 _setattrs 3.7f_head/6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 = 0 -4481> 2014-06-09 16:22:12.208133 7f86a5e11700 10 cid: oid: 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 not skipping op, *spos 4193.0.0 -4480> 2014-06-09 16:22:12.208134 7f86a5e11700 10 > header.spos 0.0.0 -4479> 2014-06-09 16:22:12.208141 7f86a5e11700 20 set_header setting 30168 cid 3.7f_head oid 6f79287f/rbd_data.11756b8b4567.00000000000015f2/head//3 parent seq 0 -4478> 2014-06-09 16:22:12.208159 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4193.0.0 <= header.spos 150443.0.0 -4477> 2014-06-09 16:22:12.208162 7f86a5e11700 10 cid: oid: 516fc4fc/pglog_3.7f/0//-1 skipping op, spos 4193.0.0 <= header.spos 150197.0.0 -4476> 2014-06-09 16:22:12.208164 7f86a5e11700 5 submit_transaction r = 0 -4475> 2014-06-09 16:22:12.208771 7f86a5e11700 10 _do_op 0x3280b1a0 seq 4193 r = 0, finisher 0x32480180 0x447c540 -4474> 2014-06-09 16:22:12.208778 7f86a5e11700 10 _finish_op 0x3280b1a0 seq 4193 osr(3.7f 0x3ed0200)/0x3ed0200 -4473> 2014-06-09 16:22:12.208782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c3be0] -4472> 2014-06-09 16:22:12.208790 7f868e75f700 10 op_submit_start 4242 -4471> 2014-06-09 16:22:12.208791 7f868e75f700 5 queue_transactions (trailing journal) 4242 -4470> 2014-06-09 16:22:12.208792 7f868e75f700 5 queue_op 0x324f5560 seq 4242 osr(1.32f 0x384d350) 828 bytes (queue has 50 ops and 4621750 bytes) -4469> 2014-06-09 16:22:12.208797 7f868cf5c700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4468> 2014-06-09 16:22:12.208786 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 29 -4467> 2014-06-09 16:22:12.208801 7f868a757700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4466> 2014-06-09 16:22:12.208806 7f868ef60700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4465> 2014-06-09 16:22:12.208810 7f8690763700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4464> 2014-06-09 16:22:12.208813 7f8690f64700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4463> 2014-06-09 16:22:12.208808 7f86a460e700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.208801, event: op_commit, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -4462> 2014-06-09 16:22:12.208817 7f8692767700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4461> 2014-06-09 16:22:12.208821 7f8691f66700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4460> 2014-06-09 16:22:12.208825 7f8691765700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4459> 2014-06-09 16:22:12.208818 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 luod=214'533 crt=0'0 lcod 0'0 mlcod 0'0 active] repop_all_committed: repop tid 29 all committed -4458> 2014-06-09 16:22:12.208829 7f8689755700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4457> 2014-06-09 16:22:12.208832 7f868d75d700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4456> 2014-06-09 16:22:12.208836 7f8689f56700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4455> 2014-06-09 16:22:12.208840 7f868ff62700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4454> 2014-06-09 16:22:12.208829 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] eval_repop repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=0 lock=0 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad -4453> 2014-06-09 16:22:12.208844 7f868af58700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4452> 2014-06-09 16:22:12.208848 7f868f761700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4451> 2014-06-09 16:22:12.208852 7f868df5e700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4450> 2014-06-09 16:22:12.208856 7f868bf5a700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4449> 2014-06-09 16:22:12.208846 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] log_op_stats osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 inb 237949 outb 0 rlat 0.000000 lat 1.274265 -4448> 2014-06-09 16:22:12.208860 7f868c75b700 2 waiting 51 > 50 ops || 4622611 > 104857600 -4447> 2014-06-09 16:22:12.208864 7f8692f68700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4446> 2014-06-09 16:22:12.208867 7f868b759700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4445> 2014-06-09 16:22:12.208865 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] publish_stats_to_osd 243:844 -4444> 2014-06-09 16:22:12.208878 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32480180] -4443> 2014-06-09 16:22:12.208882 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc359b00 (1 active) -4442> 2014-06-09 16:22:12.208884 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcbb7680 (2 active) -4441> 2014-06-09 16:22:12.208888 7f868e75f700 10 op_submit_finish 4242 -4440> 2014-06-09 16:22:12.208875 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] sending commit on repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=0 lock=0 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) 0x32070500 -4439> 2014-06-09 16:22:12.208890 7f86a460e700 1 -- 172.16.96.202:6830/30904 --> 172.16.96.220:0/1126116 -- osd_op_reply(33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] v243'534 uv534 ondisk = 0) v6 -- ?+0 0x32070500 con 0xcd609a0 -4438> 2014-06-09 16:22:12.208892 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.31f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.31f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -4437> 2014-06-09 16:22:12.208912 7f86a460e700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.208907, event: commit_sent, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -4436> 2014-06-09 16:22:12.208924 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c3be0] -4435> 2014-06-09 16:22:12.208925 7f868e75f700 10 log is not dirty -4434> 2014-06-09 16:22:12.208926 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4433> 2014-06-09 16:22:12.208927 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4432> 2014-06-09 16:22:12.208931 7f868e75f700 5 queue_transactions existing osr(0.31f 0x2bd6d50)/0x2bd6d50 -4431> 2014-06-09 16:22:12.208933 7f868e75f700 2 waiting 51 > 50 ops || 4622578 > 104857600 -4430> 2014-06-09 16:22:12.208937 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] op_applied: 29 -4429> 2014-06-09 16:22:12.208956 7f86a4e0f700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.208950, event: op_applied, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -4428> 2014-06-09 16:22:12.208964 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] op_applied on primary on version 243'534 -4427> 2014-06-09 16:22:12.208973 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] repop_all_applied: repop tid 29 all applied -4426> 2014-06-09 16:22:12.208982 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] eval_repop repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=1 lock=0 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad -4425> 2014-06-09 16:22:12.208997 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] log_op_stats osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4 inb 237949 outb 0 rlat 0.000000 lat 1.274416 -4424> 2014-06-09 16:22:12.209013 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] publish_stats_to_osd 243:845 -4423> 2014-06-09 16:22:12.209025 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 0'0 active] requeue_ops -4422> 2014-06-09 16:22:12.209035 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 214'533 active] removing repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=1 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) -4421> 2014-06-09 16:22:12.209050 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 214'533 active] q front is repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=1 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) -4420> 2014-06-09 16:22:12.209065 7f86a4e0f700 20 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 214'533 active] remove_repop repgather(0xc1fd740 243'534 rep_tid=29 committed?=1 applied?=1 op=osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) v4) -4419> 2014-06-09 16:22:12.209079 7f86a4e0f700 15 osd.26 pg_epoch: 243 pg[3.7f( v 243'534 (0'0,243'534] local-les=243 n=107 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 214'533 mlcod 214'533 active] requeue_ops -4418> 2014-06-09 16:22:12.209110 7f86a4e0f700 5 -- op tracker -- , seq: 1031, time: 2014-06-09 16:22:12.209104, event: done, op: osd_op(client.4565.0:33578 rbd_data.11756b8b4567.00000000000015f2 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~237568] 3.6f79287f RETRY=3 ack+ondisk+retry+write e242) -4417> 2014-06-09 16:22:12.209178 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32480180] -4416> 2014-06-09 16:22:12.209181 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4415> 2014-06-09 16:22:12.209182 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4414> 2014-06-09 16:22:12.209115 7f86a5e11700 5 _do_op 0xc34a840 seq 4194 osr(default 0x2c10280)/0x2c10280 start -4413> 2014-06-09 16:22:12.209189 7f86a5e11700 10 _do_transaction on 0xc2af140 -4412> 2014-06-09 16:22:12.209193 7f86a5e11700 15 _write meta/c67cc332/inc_osdmap.244/0//-1 0~171 -4411> 2014-06-09 16:22:12.209300 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid meta oid c67cc332/inc_osdmap.244/0//-1 -4410> 2014-06-09 16:22:12.209335 7f86a5e11700 20 write_state seq is 30170 -4409> 2014-06-09 16:22:12.209342 7f86a5e11700 20 set_header setting 30169 cid meta oid c67cc332/inc_osdmap.244/0//-1 parent seq 0 -4408> 2014-06-09 16:22:12.209357 7f86a5e11700 10 file_to_extents done -4407> 2014-06-09 16:22:12.209367 7f86a5e11700 10 _generic_write meta/c67cc332/inc_osdmap.244/0//-1 0~171 = 0 -4406> 2014-06-09 16:22:12.209375 7f86a5e11700 15 _write meta/ac9736c5/osdmap.244/0//-1 0~20781 -4405> 2014-06-09 16:22:12.209419 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid meta oid ac9736c5/osdmap.244/0//-1 -4404> 2014-06-09 16:22:12.209450 7f86a5e11700 20 write_state seq is 30171 -4403> 2014-06-09 16:22:12.209457 7f86a5e11700 20 set_header setting 30170 cid meta oid ac9736c5/osdmap.244/0//-1 parent seq 0 -4402> 2014-06-09 16:22:12.209471 7f86a5e11700 10 file_to_extents done -4401> 2014-06-09 16:22:12.209534 7f86a5e11700 10 _generic_write meta/ac9736c5/osdmap.244/0//-1 0~20781 = 0 -4400> 2014-06-09 16:22:12.209541 7f86a5e11700 15 _write meta/23c2fcde/osd_superblock/0//-1 0~358 -4399> 2014-06-09 16:22:12.209586 7f86a5e11700 10 lookup_strip_header done cid meta oid 23c2fcde/osd_superblock/0//-1 -4398> 2014-06-09 16:22:12.209592 7f86a5e11700 10 file_to_extents done -4397> 2014-06-09 16:22:12.209644 7f86a5e11700 10 _generic_write meta/23c2fcde/osd_superblock/0//-1 0~358 = 0 -4396> 2014-06-09 16:22:12.209649 7f86a5e11700 10 cid: oid: c67cc332/inc_osdmap.244/0//-1 not skipping op, *spos 4194.0.0 -4395> 2014-06-09 16:22:12.209652 7f86a5e11700 10 > header.spos 0.0.0 -4394> 2014-06-09 16:22:12.209655 7f86a5e11700 20 set_header setting 30169 cid meta oid c67cc332/inc_osdmap.244/0//-1 parent seq 0 -4393> 2014-06-09 16:22:12.209669 7f86a5e11700 10 cid: oid: ac9736c5/osdmap.244/0//-1 not skipping op, *spos 4194.0.0 -4392> 2014-06-09 16:22:12.209672 7f86a5e11700 10 > header.spos 0.0.0 -4391> 2014-06-09 16:22:12.209676 7f86a5e11700 20 set_header setting 30170 cid meta oid ac9736c5/osdmap.244/0//-1 parent seq 0 -4390> 2014-06-09 16:22:12.209688 7f86a5e11700 10 cid: oid: 23c2fcde/osd_superblock/0//-1 skipping op, spos 4194.0.0 <= header.spos 150250.0.0 -4389> 2014-06-09 16:22:12.209692 7f86a5e11700 5 submit_transaction r = 0 -4388> 2014-06-09 16:22:12.209793 7f86a5e11700 10 _do_op 0xc34a840 seq 4194 r = 0, finisher 0x33630390 0xbd7ccc0 -4387> 2014-06-09 16:22:12.209797 7f86a5e11700 10 _finish_op 0xc34a840 seq 4194 osr(default 0x2c10280)/0x2c10280 -4386> 2014-06-09 16:22:12.209810 7f868a757700 10 op_submit_start 4243 -4385> 2014-06-09 16:22:12.209812 7f868a757700 5 queue_transactions (trailing journal) 4243 -4384> 2014-06-09 16:22:12.209813 7f868a757700 5 queue_op 0x32539f80 seq 4243 osr(0.4ba 0x2bd7710) 828 bytes (queue has 50 ops and 4600954 bytes) -4383> 2014-06-09 16:22:12.209818 7f868ef60700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4382> 2014-06-09 16:22:12.209824 7f8690763700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4381> 2014-06-09 16:22:12.209828 7f8690f64700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4380> 2014-06-09 16:22:12.209833 7f8692767700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4379> 2014-06-09 16:22:12.209837 7f8691f66700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4378> 2014-06-09 16:22:12.209842 7f8691765700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4377> 2014-06-09 16:22:12.209846 7f8689755700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4376> 2014-06-09 16:22:12.209850 7f868d75d700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4375> 2014-06-09 16:22:12.209855 7f8689f56700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4374> 2014-06-09 16:22:12.209859 7f868ff62700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4373> 2014-06-09 16:22:12.209863 7f868af58700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4372> 2014-06-09 16:22:12.209868 7f868f761700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4371> 2014-06-09 16:22:12.209872 7f868df5e700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4370> 2014-06-09 16:22:12.209877 7f868bf5a700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4369> 2014-06-09 16:22:12.209881 7f868c75b700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4368> 2014-06-09 16:22:12.209885 7f8692f68700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4367> 2014-06-09 16:22:12.209890 7f868b759700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4366> 2014-06-09 16:22:12.209894 7f868e75f700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4365> 2014-06-09 16:22:12.209898 7f868cf5c700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4364> 2014-06-09 16:22:12.209908 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33630390] -4363> 2014-06-09 16:22:12.209928 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33630390] -4362> 2014-06-09 16:22:12.209930 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4361> 2014-06-09 16:22:12.209931 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4360> 2014-06-09 16:22:12.209934 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcbb7680 (1 active) -4359> 2014-06-09 16:22:12.209937 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca490e0 (2 active) -4358> 2014-06-09 16:22:12.209942 7f868a757700 10 op_submit_finish 4243 -4357> 2014-06-09 16:22:12.209946 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.52a( v 213'585 (0'0,213'585] local-les=243 n=116 ec=118 les/c 243/243 242/242/200) -4356> 2014-06-09 16:22:12.209982 7f868a757700 10 log is not dirty -4355> 2014-06-09 16:22:12.209987 7f868a757700 5 queue_transactions existing osr(3.52a 0x3ed1670)/0x3ed1670 -4354> 2014-06-09 16:22:12.209990 7f868a757700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4353> 2014-06-09 16:22:12.209994 7f86a5e11700 5 _do_op 0x30060cc0 seq 4195 osr(0.5d4 0x2bd7a10)/0x2bd7a10 start -4352> 2014-06-09 16:22:12.209998 7f86a5e11700 10 _do_transaction on 0x2d82480 -4351> 2014-06-09 16:22:12.210004 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4350> 2014-06-09 16:22:12.210049 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4349> 2014-06-09 16:22:12.210066 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4195.0.0 <= header.spos 150443.0.0 -4348> 2014-06-09 16:22:12.210070 7f86a5e11700 5 submit_transaction r = 0 -4347> 2014-06-09 16:22:12.210087 7f86a5e11700 10 _do_op 0x30060cc0 seq 4195 r = 0, finisher 0xc410880 0 -4346> 2014-06-09 16:22:12.210094 7f86a5e11700 10 _finish_op 0x30060cc0 seq 4195 osr(0.5d4 0x2bd7a10)/0x2bd7a10 -4345> 2014-06-09 16:22:12.210096 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3e4e0] -4344> 2014-06-09 16:22:12.210104 7f8690763700 10 op_submit_start 4244 -4343> 2014-06-09 16:22:12.210106 7f8690763700 5 queue_transactions (trailing journal) 4244 -4342> 2014-06-09 16:22:12.210107 7f8690763700 5 queue_op 0xbe7de60 seq 4244 osr(3.661 0x3ed19f0) 861 bytes (queue has 50 ops and 4600954 bytes) -4341> 2014-06-09 16:22:12.210099 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -4340> 2014-06-09 16:22:12.210112 7f8690f64700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4339> 2014-06-09 16:22:12.210116 7f8692767700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4338> 2014-06-09 16:22:12.210110 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -4337> 2014-06-09 16:22:12.210119 7f8691f66700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4336> 2014-06-09 16:22:12.210123 7f8691765700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4335> 2014-06-09 16:22:12.210119 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.5d4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 -4334> 2014-06-09 16:22:12.210127 7f8689755700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4333> 2014-06-09 16:22:12.210128 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3e4e0] -4332> 2014-06-09 16:22:12.210130 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4331> 2014-06-09 16:22:12.210130 7f868d75d700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4330> 2014-06-09 16:22:12.210131 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4329> 2014-06-09 16:22:12.210134 7f8689f56700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4328> 2014-06-09 16:22:12.210138 7f868ff62700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4327> 2014-06-09 16:22:12.210142 7f868af58700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4326> 2014-06-09 16:22:12.210146 7f868f761700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4325> 2014-06-09 16:22:12.210151 7f868df5e700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4324> 2014-06-09 16:22:12.210155 7f868bf5a700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4323> 2014-06-09 16:22:12.210159 7f868c75b700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4322> 2014-06-09 16:22:12.210163 7f8692f68700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4321> 2014-06-09 16:22:12.210167 7f868b759700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4320> 2014-06-09 16:22:12.210171 7f868e75f700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4319> 2014-06-09 16:22:12.210175 7f868cf5c700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4318> 2014-06-09 16:22:12.210180 7f868a757700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4317> 2014-06-09 16:22:12.210184 7f868ef60700 2 waiting 51 > 50 ops || 4601782 > 104857600 -4316> 2014-06-09 16:22:12.210194 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc410880] -4315> 2014-06-09 16:22:12.210198 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc410880] -4314> 2014-06-09 16:22:12.210200 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4313> 2014-06-09 16:22:12.210201 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4312> 2014-06-09 16:22:12.210204 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca490e0 (1 active) -4311> 2014-06-09 16:22:12.210207 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e120 (2 active) -4310> 2014-06-09 16:22:12.210212 7f8690763700 10 op_submit_finish 4244 -4309> 2014-06-09 16:22:12.210216 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -4308> 2014-06-09 16:22:12.210231 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.000369 7 0.000370 -4307> 2014-06-09 16:22:12.210244 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -4306> 2014-06-09 16:22:12.210255 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4305> 2014-06-09 16:22:12.210267 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -4304> 2014-06-09 16:22:12.210278 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -4303> 2014-06-09 16:22:12.210286 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -4302> 2014-06-09 16:22:12.210294 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -4301> 2014-06-09 16:22:12.210306 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -4300> 2014-06-09 16:22:12.210317 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -4299> 2014-06-09 16:22:12.210327 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -4298> 2014-06-09 16:22:12.210343 7f8690763700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -4297> 2014-06-09 16:22:12.210346 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:2.5bf) v4 -- ?+0 0x3343cfc0 -4296> 2014-06-09 16:22:12.210361 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 -4295> 2014-06-09 16:22:12.210395 7f8690763700 10 log is not dirty -4294> 2014-06-09 16:22:12.210399 7f8690763700 5 queue_transactions existing osr(2.5bf 0x2ed1b10)/0x2ed1b10 -4293> 2014-06-09 16:22:12.210402 7f8690763700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4292> 2014-06-09 16:22:12.210407 7f86a5e11700 5 _do_op 0x324a5da0 seq 4196 osr(3.375 0x3ed1060)/0x3ed1060 start -4291> 2014-06-09 16:22:12.210410 7f86a5e11700 10 _do_transaction on 0xc291980 -4290> 2014-06-09 16:22:12.210416 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4289> 2014-06-09 16:22:12.210458 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4288> 2014-06-09 16:22:12.210473 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4196.0.0 <= header.spos 150443.0.0 -4287> 2014-06-09 16:22:12.210477 7f86a5e11700 5 submit_transaction r = 0 -4286> 2014-06-09 16:22:12.210489 7f86a5e11700 10 _do_op 0x324a5da0 seq 4196 r = 0, finisher 0x30509280 0 -4285> 2014-06-09 16:22:12.210495 7f86a5e11700 10 _finish_op 0x324a5da0 seq 4196 osr(3.375 0x3ed1060)/0x3ed1060 -4284> 2014-06-09 16:22:12.210497 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc82f80] -4283> 2014-06-09 16:22:12.210500 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc82f80] -4282> 2014-06-09 16:22:12.210502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4281> 2014-06-09 16:22:12.210503 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4280> 2014-06-09 16:22:12.210506 7f8692767700 10 op_submit_start 4245 -4279> 2014-06-09 16:22:12.210507 7f8692767700 5 queue_transactions (trailing journal) 4245 -4278> 2014-06-09 16:22:12.210508 7f8692767700 5 queue_op 0xc2eaf00 seq 4245 osr(2.3de 0x2ed1330) 861 bytes (queue has 50 ops and 4600987 bytes) -4277> 2014-06-09 16:22:12.210513 7f8691f66700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4276> 2014-06-09 16:22:12.210518 7f8691765700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4275> 2014-06-09 16:22:12.210523 7f8689755700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4274> 2014-06-09 16:22:12.210527 7f868d75d700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4273> 2014-06-09 16:22:12.210532 7f8689f56700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4272> 2014-06-09 16:22:12.210536 7f868ff62700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4271> 2014-06-09 16:22:12.210541 7f868af58700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4270> 2014-06-09 16:22:12.210546 7f868f761700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4269> 2014-06-09 16:22:12.210550 7f868df5e700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4268> 2014-06-09 16:22:12.210554 7f868bf5a700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4267> 2014-06-09 16:22:12.210558 7f868c75b700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4266> 2014-06-09 16:22:12.210562 7f8692f68700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4265> 2014-06-09 16:22:12.210566 7f868b759700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4264> 2014-06-09 16:22:12.210571 7f868e75f700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4263> 2014-06-09 16:22:12.210575 7f868cf5c700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4262> 2014-06-09 16:22:12.210579 7f868a757700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4261> 2014-06-09 16:22:12.210583 7f868ef60700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4260> 2014-06-09 16:22:12.210587 7f8690763700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4259> 2014-06-09 16:22:12.210591 7f8690f64700 2 waiting 51 > 50 ops || 4601815 > 104857600 -4258> 2014-06-09 16:22:12.210601 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30509280] -4257> 2014-06-09 16:22:12.210605 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30509280] -4256> 2014-06-09 16:22:12.210607 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4255> 2014-06-09 16:22:12.210608 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4254> 2014-06-09 16:22:12.210611 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e120 (1 active) -4253> 2014-06-09 16:22:12.210614 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496ea0 (2 active) -4252> 2014-06-09 16:22:12.210619 7f8692767700 10 op_submit_finish 4245 -4251> 2014-06-09 16:22:12.210623 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -4250> 2014-06-09 16:22:12.210637 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.069326 7 0.000328 -4249> 2014-06-09 16:22:12.210649 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -4248> 2014-06-09 16:22:12.210659 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4247> 2014-06-09 16:22:12.210669 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -4246> 2014-06-09 16:22:12.210680 7f8692767700 5 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -4245> 2014-06-09 16:22:12.210688 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -4244> 2014-06-09 16:22:12.210696 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -4243> 2014-06-09 16:22:12.210707 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) -4242> 2014-06-09 16:22:12.210717 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) -4241> 2014-06-09 16:22:12.210727 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -4240> 2014-06-09 16:22:12.210739 7f8692767700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 244 -4239> 2014-06-09 16:22:12.210742 7f8692767700 1 -- 172.16.96.202:6850/30904 --> osd.3 172.16.96.201:6823/50569 -- pg_info(1 pgs e243:2.6f4) v4 -- ?+0 0x31744e00 -4238> 2014-06-09 16:22:12.210756 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:112 -4237> 2014-06-09 16:22:12.210789 7f8692767700 10 log is not dirty -4236> 2014-06-09 16:22:12.210794 7f8692767700 5 queue_transactions existing osr(2.6f4 0x2ed1e20)/0x2ed1e20 -4235> 2014-06-09 16:22:12.210796 7f8692767700 2 waiting 51 > 50 ops || 4601848 > 104857600 -4234> 2014-06-09 16:22:12.210801 7f86a5e11700 5 _do_op 0xc64a480 seq 4197 osr(0.dc 0x2bd6570)/0x2bd6570 start -4233> 2014-06-09 16:22:12.210804 7f86a5e11700 10 _do_transaction on 0xc1fde00 -4232> 2014-06-09 16:22:12.210811 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4231> 2014-06-09 16:22:12.210854 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4230> 2014-06-09 16:22:12.210870 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4197.0.0 <= header.spos 150443.0.0 -4229> 2014-06-09 16:22:12.210875 7f86a5e11700 5 submit_transaction r = 0 -4228> 2014-06-09 16:22:12.210886 7f86a5e11700 10 _do_op 0xc64a480 seq 4197 r = 0, finisher 0x306cf980 0 -4227> 2014-06-09 16:22:12.210893 7f86a5e11700 10 _finish_op 0xc64a480 seq 4197 osr(0.dc 0x2bd6570)/0x2bd6570 -4226> 2014-06-09 16:22:12.210895 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30629a60] -4225> 2014-06-09 16:22:12.210903 7f8691765700 10 op_submit_start 4246 -4224> 2014-06-09 16:22:12.210904 7f8691765700 5 queue_transactions (trailing journal) 4246 -4223> 2014-06-09 16:22:12.210906 7f8691765700 5 queue_op 0xc64b3e0 seq 4246 osr(1.65d 0x384df50) 861 bytes (queue has 50 ops and 4600990 bytes) -4222> 2014-06-09 16:22:12.210899 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -4221> 2014-06-09 16:22:12.210910 7f8689755700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4220> 2014-06-09 16:22:12.210914 7f868d75d700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4219> 2014-06-09 16:22:12.210908 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -4218> 2014-06-09 16:22:12.210917 7f8689f56700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4217> 2014-06-09 16:22:12.210921 7f868ff62700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4216> 2014-06-09 16:22:12.210918 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.dc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:154 -4215> 2014-06-09 16:22:12.210925 7f868af58700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4214> 2014-06-09 16:22:12.210927 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30629a60] -4213> 2014-06-09 16:22:12.210929 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4212> 2014-06-09 16:22:12.210928 7f868f761700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4211> 2014-06-09 16:22:12.210929 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4210> 2014-06-09 16:22:12.210932 7f868df5e700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4209> 2014-06-09 16:22:12.210936 7f868bf5a700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4208> 2014-06-09 16:22:12.210940 7f868c75b700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4207> 2014-06-09 16:22:12.210944 7f8692f68700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4206> 2014-06-09 16:22:12.210949 7f868b759700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4205> 2014-06-09 16:22:12.210953 7f868e75f700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4204> 2014-06-09 16:22:12.210957 7f868cf5c700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4203> 2014-06-09 16:22:12.210962 7f868a757700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4202> 2014-06-09 16:22:12.210966 7f868ef60700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4201> 2014-06-09 16:22:12.210970 7f8690763700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4200> 2014-06-09 16:22:12.210974 7f8690f64700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4199> 2014-06-09 16:22:12.210978 7f8692767700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4198> 2014-06-09 16:22:12.210983 7f8691f66700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4197> 2014-06-09 16:22:12.210992 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x306cf980] -4196> 2014-06-09 16:22:12.210997 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x306cf980] -4195> 2014-06-09 16:22:12.210999 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4194> 2014-06-09 16:22:12.211000 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4193> 2014-06-09 16:22:12.211003 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496ea0 (1 active) -4192> 2014-06-09 16:22:12.211006 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc301e60 (2 active) -4191> 2014-06-09 16:22:12.211011 7f8691765700 10 op_submit_finish 4246 -4190> 2014-06-09 16:22:12.211015 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -4189> 2014-06-09 16:22:12.211030 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -4188> 2014-06-09 16:22:12.211039 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -4187> 2014-06-09 16:22:12.211047 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4186> 2014-06-09 16:22:12.211056 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -4185> 2014-06-09 16:22:12.211064 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -4184> 2014-06-09 16:22:12.211078 7f8691765700 15 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:193 -4183> 2014-06-09 16:22:12.211087 7f8691765700 20 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -4182> 2014-06-09 16:22:12.211096 7f8691765700 20 osd.26 pg_epoch: 243 pg[1.591( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -4181> 2014-06-09 16:22:12.211105 7f8691765700 10 log is not dirty -4180> 2014-06-09 16:22:12.211110 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -4179> 2014-06-09 16:22:12.211122 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -4178> 2014-06-09 16:22:12.211130 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -4177> 2014-06-09 16:22:12.211139 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -4176> 2014-06-09 16:22:12.211202 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -4175> 2014-06-09 16:22:12.211210 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -4174> 2014-06-09 16:22:12.211224 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 -4173> 2014-06-09 16:22:12.211234 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -4172> 2014-06-09 16:22:12.211242 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.f1( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -4171> 2014-06-09 16:22:12.211251 7f8691765700 10 log is not dirty -4170> 2014-06-09 16:22:12.211255 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -4169> 2014-06-09 16:22:12.211269 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -4168> 2014-06-09 16:22:12.211278 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16bd6b00,0x16bd6a00,0x16bd6900,0x13022e00,0x13022d00,0x13022c00,0x13022b00,0x16bd7900,0x16bd7800,0x2c06bd00,0x2e075500,0x2e075400,0x2e075300,0x2e075200,0x2e075100 -4167> 2014-06-09 16:22:12.211326 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -4166> 2014-06-09 16:22:12.211337 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -4165> 2014-06-09 16:22:12.211346 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -4164> 2014-06-09 16:22:12.211359 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1163 -4163> 2014-06-09 16:22:12.211370 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -4162> 2014-06-09 16:22:12.211379 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.1db( v 214'696 (0'0,214'696] local-les=243 n=139 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -4161> 2014-06-09 16:22:12.211389 7f8691765700 10 log is not dirty -4160> 2014-06-09 16:22:12.211394 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.55b( v 214'716 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) -4159> 2014-06-09 16:22:12.211410 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.4 activated and committed -4158> 2014-06-09 16:22:12.211423 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 203/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed -4157> 2014-06-09 16:22:12.211433 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info -4156> 2014-06-09 16:22:12.211448 7f8691765700 20 osd.26 243 share_map_peer 0xbfb5e40 already has epoch 244 -4155> 2014-06-09 16:22:12.211451 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.4 172.16.96.201:6832/51134 -- pg_info(1 pgs e243:3.55b) v4 -- ?+0 0x1ac95880 -4154> 2014-06-09 16:22:12.211465 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:1128 -4153> 2014-06-09 16:22:12.211483 7f8691765700 10 log is not dirty -4152> 2014-06-09 16:22:12.211507 7f8691765700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x1fc880c0 (19 active) -4151> 2014-06-09 16:22:12.211514 7f8691765700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -4150> 2014-06-09 16:22:12.211520 7f8691765700 10 osd.26 244 dequeue_op 0xe032100 prio 63 cost 57344 latency 2.251505 osd_op(client.4572.0:29986 rbd_data.117c6b8b4567.000000000000140e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.d8b29cb9 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] -4149> 2014-06-09 16:22:12.211545 7f8691765700 5 -- op tracker -- , seq: 12, time: 2014-06-09 16:22:12.211540, event: reached_pg, op: osd_op(client.4572.0:29986 rbd_data.117c6b8b4567.000000000000140e [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.d8b29cb9 RETRY=3 ack+ondisk+retry+write e242) -4148> 2014-06-09 16:22:12.211554 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -4147> 2014-06-09 16:22:12.211569 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] can_handle_while_inactive: 0xe032100 -4146> 2014-06-09 16:22:12.211581 7f8691765700 10 osd.26 244 dequeue_op 0xe032100 finish -4145> 2014-06-09 16:22:12.211585 7f8691765700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -4144> 2014-06-09 16:22:12.211622 7f8691765700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x1fc880c0 (20 active) -4143> 2014-06-09 16:22:12.211626 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -4142> 2014-06-09 16:22:12.211645 7f8691765700 10 log is not dirty -4141> 2014-06-09 16:22:12.211650 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -4140> 2014-06-09 16:22:12.211663 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.55b( v 214'716 lc 214'715 (0'0,214'716] local-les=243 n=138 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=214'716 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x13022600,0x13022500,0x16bd6600,0x16bd6500 -4139> 2014-06-09 16:22:12.211686 7f8691765700 10 log is not dirty -4138> 2014-06-09 16:22:12.211690 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.54a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -4137> 2014-06-09 16:22:12.211726 7f8691765700 10 log is not dirty -4136> 2014-06-09 16:22:12.211731 7f8691765700 5 queue_transactions existing osr(1.54a 0x384daf0)/0x384daf0 -4135> 2014-06-09 16:22:12.211733 7f8691765700 2 waiting 51 > 50 ops || 4601818 > 104857600 -4134> 2014-06-09 16:22:12.211738 7f86a5e11700 5 _do_op 0x30a38c0 seq 4198 osr(1.206 0x384cd10)/0x384cd10 start -4133> 2014-06-09 16:22:12.211743 7f86a5e11700 10 _do_transaction on 0xbe892c0 -4132> 2014-06-09 16:22:12.211748 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4131> 2014-06-09 16:22:12.211792 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4130> 2014-06-09 16:22:12.211807 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4198.0.0 <= header.spos 150443.0.0 -4129> 2014-06-09 16:22:12.211811 7f86a5e11700 5 submit_transaction r = 0 -4128> 2014-06-09 16:22:12.211823 7f86a5e11700 10 _do_op 0x30a38c0 seq 4198 r = 0, finisher 0xbebfe60 0 -4127> 2014-06-09 16:22:12.211830 7f86a5e11700 10 _finish_op 0x30a38c0 seq 4198 osr(1.206 0x384cd10)/0x384cd10 -4126> 2014-06-09 16:22:12.211832 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbebf9c0] -4125> 2014-06-09 16:22:12.211835 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbebf9c0] -4124> 2014-06-09 16:22:12.211837 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4123> 2014-06-09 16:22:12.211838 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4122> 2014-06-09 16:22:12.211841 7f868d75d700 10 op_submit_start 4247 -4121> 2014-06-09 16:22:12.211842 7f868d75d700 5 queue_transactions (trailing journal) 4247 -4120> 2014-06-09 16:22:12.211843 7f868d75d700 5 queue_op 0xc2d5020 seq 4247 osr(0.6b3 0x2bd7ff0) 861 bytes (queue has 50 ops and 4601023 bytes) -4119> 2014-06-09 16:22:12.211848 7f8689f56700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4118> 2014-06-09 16:22:12.211853 7f868ff62700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4117> 2014-06-09 16:22:12.211857 7f868af58700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4116> 2014-06-09 16:22:12.211861 7f868f761700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4115> 2014-06-09 16:22:12.211865 7f868df5e700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4114> 2014-06-09 16:22:12.211870 7f868bf5a700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4113> 2014-06-09 16:22:12.211874 7f868c75b700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4112> 2014-06-09 16:22:12.211878 7f8692f68700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4111> 2014-06-09 16:22:12.211882 7f868b759700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4110> 2014-06-09 16:22:12.211887 7f868e75f700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4109> 2014-06-09 16:22:12.211891 7f868cf5c700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4108> 2014-06-09 16:22:12.211895 7f868a757700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4107> 2014-06-09 16:22:12.211900 7f868ef60700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4106> 2014-06-09 16:22:12.211904 7f8690763700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4105> 2014-06-09 16:22:12.211908 7f8690f64700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4104> 2014-06-09 16:22:12.211913 7f8692767700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4103> 2014-06-09 16:22:12.211917 7f8691f66700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4102> 2014-06-09 16:22:12.211921 7f8691765700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4101> 2014-06-09 16:22:12.211925 7f8689755700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4100> 2014-06-09 16:22:12.211935 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbebfe60] -4099> 2014-06-09 16:22:12.211939 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbebfe60] -4098> 2014-06-09 16:22:12.211941 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4097> 2014-06-09 16:22:12.211942 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4096> 2014-06-09 16:22:12.211945 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc301e60 (1 active) -4095> 2014-06-09 16:22:12.211948 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fcd80 (2 active) -4094> 2014-06-09 16:22:12.211953 7f868d75d700 10 op_submit_finish 4247 -4093> 2014-06-09 16:22:12.211957 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -4092> 2014-06-09 16:22:12.211973 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.126881 8 0.008879 -4091> 2014-06-09 16:22:12.211985 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -4090> 2014-06-09 16:22:12.211997 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -4089> 2014-06-09 16:22:12.212008 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -4088> 2014-06-09 16:22:12.212019 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -4087> 2014-06-09 16:22:12.212029 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery -4086> 2014-06-09 16:22:12.212038 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state -4085> 2014-06-09 16:22:12.212049 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -4084> 2014-06-09 16:22:12.212061 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -4083> 2014-06-09 16:22:12.212073 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info -4082> 2014-06-09 16:22:12.212089 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -4081> 2014-06-09 16:22:12.212091 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:3.22d) v4 -- ?+0 0x30fccc40 -4080> 2014-06-09 16:22:12.212105 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1141 -4079> 2014-06-09 16:22:12.212139 7f868d75d700 10 log is not dirty -4078> 2014-06-09 16:22:12.212143 7f868d75d700 5 queue_transactions existing osr(3.22d 0x3ed0c40)/0x3ed0c40 -4077> 2014-06-09 16:22:12.212145 7f868d75d700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4076> 2014-06-09 16:22:12.212150 7f86a5e11700 5 _do_op 0xbdc82a0 seq 4199 osr(3.18e 0x3ed0af0)/0x3ed0af0 start -4075> 2014-06-09 16:22:12.212154 7f86a5e11700 10 _do_transaction on 0x39db740 -4074> 2014-06-09 16:22:12.212159 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4073> 2014-06-09 16:22:12.212201 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4072> 2014-06-09 16:22:12.212217 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4199.0.0 <= header.spos 150443.0.0 -4071> 2014-06-09 16:22:12.212221 7f86a5e11700 5 submit_transaction r = 0 -4070> 2014-06-09 16:22:12.212249 7f86a5e11700 10 _do_op 0xbdc82a0 seq 4199 r = 0, finisher 0x3205d4a0 0 -4069> 2014-06-09 16:22:12.212256 7f86a5e11700 10 _finish_op 0xbdc82a0 seq 4199 osr(3.18e 0x3ed0af0)/0x3ed0af0 -4068> 2014-06-09 16:22:12.212258 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205d560] -4067> 2014-06-09 16:22:12.212261 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205d560] -4066> 2014-06-09 16:22:12.212262 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4065> 2014-06-09 16:22:12.212263 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4064> 2014-06-09 16:22:12.212266 7f868ff62700 10 op_submit_start 4248 -4063> 2014-06-09 16:22:12.212268 7f868ff62700 5 queue_transactions (trailing journal) 4248 -4062> 2014-06-09 16:22:12.212269 7f868ff62700 5 queue_op 0xd326120 seq 4248 osr(0.448 0x2bd7190) 828 bytes (queue has 50 ops and 4601023 bytes) -4061> 2014-06-09 16:22:12.212274 7f868af58700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4060> 2014-06-09 16:22:12.212279 7f868f761700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4059> 2014-06-09 16:22:12.212283 7f868df5e700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4058> 2014-06-09 16:22:12.212288 7f868bf5a700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4057> 2014-06-09 16:22:12.212292 7f868c75b700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4056> 2014-06-09 16:22:12.212297 7f8692f68700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4055> 2014-06-09 16:22:12.212301 7f868b759700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4054> 2014-06-09 16:22:12.212305 7f868e75f700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4053> 2014-06-09 16:22:12.212310 7f868cf5c700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4052> 2014-06-09 16:22:12.212314 7f868a757700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4051> 2014-06-09 16:22:12.212318 7f868ef60700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4050> 2014-06-09 16:22:12.212322 7f8690763700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4049> 2014-06-09 16:22:12.212327 7f8690f64700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4048> 2014-06-09 16:22:12.212331 7f8692767700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4047> 2014-06-09 16:22:12.212335 7f8691f66700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4046> 2014-06-09 16:22:12.212339 7f8691765700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4045> 2014-06-09 16:22:12.212344 7f8689755700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4044> 2014-06-09 16:22:12.212348 7f868d75d700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4043> 2014-06-09 16:22:12.212352 7f8689f56700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4042> 2014-06-09 16:22:12.212362 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205d4a0] -4041> 2014-06-09 16:22:12.212368 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205d4a0] -4040> 2014-06-09 16:22:12.212370 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -4039> 2014-06-09 16:22:12.212371 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -4038> 2014-06-09 16:22:12.212374 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fcd80 (1 active) -4037> 2014-06-09 16:22:12.212377 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524120 (2 active) -4036> 2014-06-09 16:22:12.212382 7f868ff62700 10 op_submit_finish 4248 -4035> 2014-06-09 16:22:12.212386 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.5ce( v 213'643 (0'0,213'643] local-les=243 n=128 ec=118 les/c 243/191 242/242/223) -4034> 2014-06-09 16:22:12.212421 7f868ff62700 10 log is not dirty -4033> 2014-06-09 16:22:12.212426 7f868ff62700 5 queue_transactions existing osr(3.5ce 0x3ed17c0)/0x3ed17c0 -4032> 2014-06-09 16:22:12.212429 7f868ff62700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4031> 2014-06-09 16:22:12.212433 7f86a5e11700 5 _do_op 0x324db380 seq 4200 osr(0.3fe 0x2bd7210)/0x2bd7210 start -4030> 2014-06-09 16:22:12.212437 7f86a5e11700 10 _do_transaction on 0xc05f500 -4029> 2014-06-09 16:22:12.212442 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -4028> 2014-06-09 16:22:12.212484 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -4027> 2014-06-09 16:22:12.212499 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4200.0.0 <= header.spos 150443.0.0 -4026> 2014-06-09 16:22:12.212504 7f86a5e11700 5 submit_transaction r = 0 -4025> 2014-06-09 16:22:12.212515 7f86a5e11700 10 _do_op 0x324db380 seq 4200 r = 0, finisher 0x3050a0a0 0 -4024> 2014-06-09 16:22:12.212521 7f86a5e11700 10 _finish_op 0x324db380 seq 4200 osr(0.3fe 0x2bd7210)/0x2bd7210 -4023> 2014-06-09 16:22:12.212523 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdba1e0] -4022> 2014-06-09 16:22:12.212526 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdba1e0] -4021> 2014-06-09 16:22:12.212527 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -4020> 2014-06-09 16:22:12.212528 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -4019> 2014-06-09 16:22:12.212532 7f868f761700 10 op_submit_start 4249 -4018> 2014-06-09 16:22:12.212533 7f868f761700 5 queue_transactions (trailing journal) 4249 -4017> 2014-06-09 16:22:12.212535 7f868f761700 5 queue_op 0x324e7bc0 seq 4249 osr(1.66e 0x384ded0) 828 bytes (queue has 50 ops and 4601023 bytes) -4016> 2014-06-09 16:22:12.212540 7f868df5e700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4015> 2014-06-09 16:22:12.212545 7f868bf5a700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4014> 2014-06-09 16:22:12.212549 7f868c75b700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4013> 2014-06-09 16:22:12.212554 7f8692f68700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4012> 2014-06-09 16:22:12.212558 7f868b759700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4011> 2014-06-09 16:22:12.212562 7f868e75f700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4010> 2014-06-09 16:22:12.212566 7f868cf5c700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4009> 2014-06-09 16:22:12.212571 7f868a757700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4008> 2014-06-09 16:22:12.212575 7f868ef60700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4007> 2014-06-09 16:22:12.212579 7f8690763700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4006> 2014-06-09 16:22:12.212583 7f8690f64700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4005> 2014-06-09 16:22:12.212588 7f8692767700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4004> 2014-06-09 16:22:12.212591 7f8691f66700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4003> 2014-06-09 16:22:12.212596 7f8691765700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4002> 2014-06-09 16:22:12.212600 7f8689755700 2 waiting 51 > 50 ops || 4601851 > 104857600 -4001> 2014-06-09 16:22:12.212604 7f868d75d700 2 waiting 51 > 50 ops || 4601884 > 104857600 -4000> 2014-06-09 16:22:12.212608 7f8689f56700 2 waiting 51 > 50 ops || 4601884 > 104857600 -3999> 2014-06-09 16:22:12.212612 7f868ff62700 2 waiting 51 > 50 ops || 4601851 > 104857600 -3998> 2014-06-09 16:22:12.212616 7f868af58700 2 waiting 51 > 50 ops || 4601884 > 104857600 -3997> 2014-06-09 16:22:12.212626 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a0a0] -3996> 2014-06-09 16:22:12.212629 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a0a0] -3995> 2014-06-09 16:22:12.212631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3994> 2014-06-09 16:22:12.212632 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3993> 2014-06-09 16:22:12.212635 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524120 (1 active) -3992> 2014-06-09 16:22:12.212638 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b0e0 (2 active) -3991> 2014-06-09 16:22:12.212643 7f868f761700 10 op_submit_finish 4249 -3990> 2014-06-09 16:22:12.212647 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3989> 2014-06-09 16:22:12.212667 7f868f761700 10 log is not dirty -3988> 2014-06-09 16:22:12.212673 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3987> 2014-06-09 16:22:12.212686 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops -3986> 2014-06-09 16:22:12.212699 7f868f761700 10 log is not dirty -3985> 2014-06-09 16:22:12.212703 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3984> 2014-06-09 16:22:12.212720 7f868f761700 10 log is not dirty -3983> 2014-06-09 16:22:12.212724 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3982> 2014-06-09 16:22:12.212737 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] requeue_ops 0x1804cf00,0x257c5d00,0x1804ce00,0x1804cd00,0x1804cc00,0x1804cb00,0x1c768a00,0x1c768900,0x257c5c00,0x257c5b00,0x257c5a00,0x257c5900,0x257c5800 -3981> 2014-06-09 16:22:12.212781 7f868f761700 10 log is not dirty -3980> 2014-06-09 16:22:12.212786 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3979> 2014-06-09 16:22:12.212800 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x22009f00,0x22009d00,0x1804c800,0x1804c900,0x14052d00,0xdfa7d00,0x32826800,0x32826700 -3978> 2014-06-09 16:22:12.212832 7f868f761700 10 log is not dirty -3977> 2014-06-09 16:22:12.212836 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -3976> 2014-06-09 16:22:12.212849 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -3975> 2014-06-09 16:22:12.212857 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -3974> 2014-06-09 16:22:12.212866 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -3973> 2014-06-09 16:22:12.212874 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -3972> 2014-06-09 16:22:12.212882 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -3971> 2014-06-09 16:22:12.212895 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 -3970> 2014-06-09 16:22:12.212904 7f868f761700 20 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -3969> 2014-06-09 16:22:12.212912 7f868f761700 20 osd.26 pg_epoch: 243 pg[1.779( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -3968> 2014-06-09 16:22:12.212921 7f868f761700 10 log is not dirty -3967> 2014-06-09 16:22:12.212925 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3966> 2014-06-09 16:22:12.212938 7f868f761700 15 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -3965> 2014-06-09 16:22:12.212947 7f868f761700 10 log is not dirty -3964> 2014-06-09 16:22:12.212952 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3963> 2014-06-09 16:22:12.212965 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -3962> 2014-06-09 16:22:12.212976 7f868f761700 10 log is not dirty -3961> 2014-06-09 16:22:12.212980 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3960> 2014-06-09 16:22:12.212992 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.5f8( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -3959> 2014-06-09 16:22:12.213001 7f868f761700 10 log is not dirty -3958> 2014-06-09 16:22:12.213005 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3957> 2014-06-09 16:22:12.213018 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.25a( v 214'581 (0'0,214'581] local-les=243 n=113 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops -3956> 2014-06-09 16:22:12.213028 7f868f761700 10 log is not dirty -3955> 2014-06-09 16:22:12.213032 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 243/210 242/242/209) -3954> 2014-06-09 16:22:12.213122 7f868f761700 10 log is not dirty -3953> 2014-06-09 16:22:12.213128 7f868f761700 5 queue_transactions existing osr(3.2df 0x3ed0e40)/0x3ed0e40 -3952> 2014-06-09 16:22:12.213130 7f868f761700 2 waiting 51 > 50 ops || 4601851 > 104857600 -3951> 2014-06-09 16:22:12.213135 7f86a5e11700 5 _do_op 0xd326ae0 seq 4201 osr(1.a 0x384c270)/0x384c270 start -3950> 2014-06-09 16:22:12.213139 7f86a5e11700 10 _do_transaction on 0x36b2900 -3949> 2014-06-09 16:22:12.213145 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3948> 2014-06-09 16:22:12.213187 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3947> 2014-06-09 16:22:12.213202 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4201.0.0 <= header.spos 150443.0.0 -3946> 2014-06-09 16:22:12.213206 7f86a5e11700 5 submit_transaction r = 0 -3945> 2014-06-09 16:22:12.213219 7f86a5e11700 10 _do_op 0xd326ae0 seq 4201 r = 0, finisher 0xc1681e0 0 -3944> 2014-06-09 16:22:12.213226 7f86a5e11700 10 _finish_op 0xd326ae0 seq 4201 osr(1.a 0x384c270)/0x384c270 -3943> 2014-06-09 16:22:12.213228 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061fa40] -3942> 2014-06-09 16:22:12.213231 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061fa40] -3941> 2014-06-09 16:22:12.213232 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3940> 2014-06-09 16:22:12.213233 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3939> 2014-06-09 16:22:12.213237 7f868bf5a700 10 op_submit_start 4250 -3938> 2014-06-09 16:22:12.213238 7f868bf5a700 5 queue_transactions (trailing journal) 4250 -3937> 2014-06-09 16:22:12.213240 7f868bf5a700 5 queue_op 0x324dbb00 seq 4250 osr(3.251 0x3ed0c20) 828 bytes (queue has 50 ops and 4601027 bytes) -3936> 2014-06-09 16:22:12.213245 7f868c75b700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3935> 2014-06-09 16:22:12.213250 7f8692f68700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3934> 2014-06-09 16:22:12.213254 7f868b759700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3933> 2014-06-09 16:22:12.213259 7f868e75f700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3932> 2014-06-09 16:22:12.213263 7f868cf5c700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3931> 2014-06-09 16:22:12.213267 7f868a757700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3930> 2014-06-09 16:22:12.213271 7f868ef60700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3929> 2014-06-09 16:22:12.213276 7f8690763700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3928> 2014-06-09 16:22:12.213280 7f8690f64700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3927> 2014-06-09 16:22:12.213284 7f8692767700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3926> 2014-06-09 16:22:12.213289 7f8691f66700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3925> 2014-06-09 16:22:12.213293 7f8691765700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3924> 2014-06-09 16:22:12.213297 7f8689755700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3923> 2014-06-09 16:22:12.213302 7f868d75d700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3922> 2014-06-09 16:22:12.213306 7f8689f56700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3921> 2014-06-09 16:22:12.213310 7f868ff62700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3920> 2014-06-09 16:22:12.213314 7f868af58700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3919> 2014-06-09 16:22:12.213318 7f868f761700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3918> 2014-06-09 16:22:12.213322 7f868df5e700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3917> 2014-06-09 16:22:12.213332 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc1681e0] -3916> 2014-06-09 16:22:12.213336 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc1681e0] -3915> 2014-06-09 16:22:12.213338 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3914> 2014-06-09 16:22:12.213339 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3913> 2014-06-09 16:22:12.213342 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b0e0 (1 active) -3912> 2014-06-09 16:22:12.213344 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0c60 (2 active) -3911> 2014-06-09 16:22:12.213350 7f868bf5a700 10 op_submit_finish 4250 -3910> 2014-06-09 16:22:12.213354 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[0.4bd( empty local-les=243 n=0 ec=1 les/c 212/213 242/242/211) [8,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.4bd( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/211) -3909> 2014-06-09 16:22:12.213387 7f868bf5a700 10 log is not dirty -3908> 2014-06-09 16:22:12.213392 7f868bf5a700 5 queue_transactions existing osr(0.4bd 0x2bd76f0)/0x2bd76f0 -3907> 2014-06-09 16:22:12.213395 7f868bf5a700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3906> 2014-06-09 16:22:12.213400 7f86a5e11700 5 _do_op 0xc34aba0 seq 4202 osr(0.3b7 0x2bd7290)/0x2bd7290 start -3905> 2014-06-09 16:22:12.213403 7f86a5e11700 10 _do_transaction on 0xc2c4000 -3904> 2014-06-09 16:22:12.213409 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3903> 2014-06-09 16:22:12.213450 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3902> 2014-06-09 16:22:12.213464 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4202.0.0 <= header.spos 150443.0.0 -3901> 2014-06-09 16:22:12.213469 7f86a5e11700 5 submit_transaction r = 0 -3900> 2014-06-09 16:22:12.213479 7f86a5e11700 10 _do_op 0xc34aba0 seq 4202 r = 0, finisher 0x1322dd80 0 -3899> 2014-06-09 16:22:12.213485 7f86a5e11700 10 _finish_op 0xc34aba0 seq 4202 osr(0.3b7 0x2bd7290)/0x2bd7290 -3898> 2014-06-09 16:22:12.213487 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x182cae80] -3897> 2014-06-09 16:22:12.213490 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x182cae80] -3896> 2014-06-09 16:22:12.213491 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3895> 2014-06-09 16:22:12.213492 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3894> 2014-06-09 16:22:12.213496 7f8692f68700 10 op_submit_start 4251 -3893> 2014-06-09 16:22:12.213498 7f8692f68700 5 queue_transactions (trailing journal) 4251 -3892> 2014-06-09 16:22:12.213499 7f8692f68700 5 queue_op 0x2fc8c120 seq 4251 osr(3.5f8 0x3ed1b30) 828 bytes (queue has 50 ops and 4601027 bytes) -3891> 2014-06-09 16:22:12.213504 7f868b759700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3890> 2014-06-09 16:22:12.213509 7f868e75f700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3889> 2014-06-09 16:22:12.213513 7f868cf5c700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3888> 2014-06-09 16:22:12.213517 7f868a757700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3887> 2014-06-09 16:22:12.213522 7f868ef60700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3886> 2014-06-09 16:22:12.213526 7f8690763700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3885> 2014-06-09 16:22:12.213530 7f8690f64700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3884> 2014-06-09 16:22:12.213535 7f8692767700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3883> 2014-06-09 16:22:12.213539 7f8691f66700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3882> 2014-06-09 16:22:12.213543 7f8691765700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3881> 2014-06-09 16:22:12.213547 7f8689755700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3880> 2014-06-09 16:22:12.213551 7f868d75d700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3879> 2014-06-09 16:22:12.213556 7f8689f56700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3878> 2014-06-09 16:22:12.213560 7f868ff62700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3877> 2014-06-09 16:22:12.213564 7f868af58700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3876> 2014-06-09 16:22:12.213568 7f868f761700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3875> 2014-06-09 16:22:12.213572 7f868df5e700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3874> 2014-06-09 16:22:12.213576 7f868bf5a700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3873> 2014-06-09 16:22:12.213580 7f868c75b700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3872> 2014-06-09 16:22:12.213590 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1322dd80] -3871> 2014-06-09 16:22:12.213594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1322dd80] -3870> 2014-06-09 16:22:12.213596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3869> 2014-06-09 16:22:12.213597 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3868> 2014-06-09 16:22:12.213600 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0c60 (1 active) -3867> 2014-06-09 16:22:12.213603 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6360 (2 active) -3866> 2014-06-09 16:22:12.213608 7f8692f68700 10 op_submit_finish 4251 -3865> 2014-06-09 16:22:12.213624 7f8692f68700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb40520 (19 active) -3864> 2014-06-09 16:22:12.213631 7f8692f68700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -3863> 2014-06-09 16:22:12.213635 7f8692f68700 10 osd.26 244 dequeue_op 0x270be300 prio 63 cost 450560 latency 1.399389 osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] -3862> 2014-06-09 16:22:12.213660 7f8692f68700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.213655, event: reached_pg, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) -3861> 2014-06-09 16:22:12.213668 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -3860> 2014-06-09 16:22:12.213681 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x270be300 -3859> 2014-06-09 16:22:12.213692 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write -3858> 2014-06-09 16:22:12.213717 7f8692f68700 15 getattr 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 '_' -3857> 2014-06-09 16:22:12.213791 7f8692f68700 10 getattr got.size() is 0 -3856> 2014-06-09 16:22:12.213794 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 but can_create -3855> 2014-06-09 16:22:12.213809 7f8692f68700 15 getattr 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 'snapset' -3854> 2014-06-09 16:22:12.213841 7f8692f68700 10 getattr got.size() is 0 -3853> 2014-06-09 16:22:12.213844 7f8692f68700 15 getattr 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/snapdir//3 'snapset' -3852> 2014-06-09 16:22:12.213874 7f8692f68700 10 getattr got.size() is 0 -3851> 2014-06-09 16:22:12.213883 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30e06580 e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -3850> 2014-06-09 16:22:12.213896 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -3849> 2014-06-09 16:22:12.213906 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -3848> 2014-06-09 16:22:12.213917 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30e06580 e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 rwstate(none n=0 w=0) oi: e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x14dc2700 snapset: 0=[]:[] -3847> 2014-06-09 16:22:12.213933 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 @head oi=e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) -3846> 2014-06-09 16:22:12.213952 7f8692f68700 15 getattr 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/snapdir//3 '_' -3845> 2014-06-09 16:22:12.213985 7f8692f68700 10 getattr got.size() is 0 -3844> 2014-06-09 16:22:12.213987 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/snapdir//3 and !can_create -3843> 2014-06-09 16:22:12.214008 7f8692f68700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.214002, event: started, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) -3842> 2014-06-09 16:22:12.214015 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30a9aa00 -3841> 2014-06-09 16:22:12.214033 7f8692f68700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.214028, event: started, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) -3840> 2014-06-09 16:22:12.214040 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] ov 0'0 av 243'519 snapc 0=[] snapset 0=[]:[] -3839> 2014-06-09 16:22:12.214054 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] -3838> 2014-06-09 16:22:12.214065 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 -3837> 2014-06-09 16:22:12.214080 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~450560 -3836> 2014-06-09 16:22:12.214097 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0xd229600 -3835> 2014-06-09 16:22:12.214108 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 snapset=0x14dc2738 snapc=0=[] -3834> 2014-06-09 16:22:12.214120 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag -3833> 2014-06-09 16:22:12.214130 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 done, snapset=0=[]:[]+head -3832> 2014-06-09 16:22:12.214142 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 0x30a9aa00 op modify -3831> 2014-06-09 16:22:12.214155 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:43.783744 -3830> 2014-06-09 16:22:12.214175 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=98 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -3829> 2014-06-09 16:22:12.214195 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 -3828> 2014-06-09 16:22:12.214206 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 32 on osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4 -3827> 2014-06-09 16:22:12.214222 7f8692f68700 7 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 32 o e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -3826> 2014-06-09 16:22:12.214246 7f8692f68700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.214240, event: waiting for subops from 8, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) -3825> 2014-06-09 16:22:12.214278 7f8692f68700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -3824> 2014-06-09 16:22:12.214281 7f8692f68700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- osd_sub_op(client.4573.0:34234 3.750 e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 [] v 243'519 snapset=0=[]:[] snapc=0=[]) v10 -- ?+451472 0x16bbe000 -3823> 2014-06-09 16:22:12.214298 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 214'518 (0'0,214'518] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'518], crt=0'0) [243'519 (0'0) modify e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 by client.4573.0:34234 2014-06-09 10:20:43.783744] -3822> 2014-06-09 16:22:12.214319 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'519 (0'0) modify e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 by client.4573.0:34234 2014-06-09 10:20:43.783744 -3821> 2014-06-09 16:22:12.214339 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys -3820> 2014-06-09 16:22:12.214370 7f8692f68700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'519, trimmed: -3819> 2014-06-09 16:22:12.214389 7f8692f68700 5 queue_transactions existing osr(3.750 0x3ed1eb0)/0x3ed1eb0 -3818> 2014-06-09 16:22:12.214392 7f8692f68700 2 waiting 51 > 50 ops || 5053985 > 104857600 -3817> 2014-06-09 16:22:12.214397 7f86a5e11700 5 _do_op 0xd326f00 seq 4203 osr(3.470 0x3ed1540)/0x3ed1540 start -3816> 2014-06-09 16:22:12.214401 7f86a5e11700 10 _do_transaction on 0xbf36cc0 -3815> 2014-06-09 16:22:12.214407 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3814> 2014-06-09 16:22:12.214450 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3813> 2014-06-09 16:22:12.214466 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4203.0.0 <= header.spos 150443.0.0 -3812> 2014-06-09 16:22:12.214470 7f86a5e11700 5 submit_transaction r = 0 -3811> 2014-06-09 16:22:12.214490 7f86a5e11700 10 _do_op 0xd326f00 seq 4203 r = 0, finisher 0x139656e0 0 -3810> 2014-06-09 16:22:12.214497 7f86a5e11700 10 _finish_op 0xd326f00 seq 4203 osr(3.470 0x3ed1540)/0x3ed1540 -3809> 2014-06-09 16:22:12.214499 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30e3e840] -3808> 2014-06-09 16:22:12.214502 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30e3e840] -3807> 2014-06-09 16:22:12.214504 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3806> 2014-06-09 16:22:12.214505 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3805> 2014-06-09 16:22:12.214508 7f868e75f700 10 op_submit_start 4252 -3804> 2014-06-09 16:22:12.214510 7f868e75f700 5 queue_transactions (trailing journal) 4252 -3803> 2014-06-09 16:22:12.214511 7f868e75f700 5 queue_op 0x324f53e0 seq 4252 osr(0.31f 0x2bd6d50) 828 bytes (queue has 50 ops and 4601027 bytes) -3802> 2014-06-09 16:22:12.214517 7f868cf5c700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3801> 2014-06-09 16:22:12.214521 7f868a757700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3800> 2014-06-09 16:22:12.214526 7f868ef60700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3799> 2014-06-09 16:22:12.214531 7f8690763700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3798> 2014-06-09 16:22:12.214535 7f8690f64700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3797> 2014-06-09 16:22:12.214540 7f8692767700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3796> 2014-06-09 16:22:12.214544 7f8691f66700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3795> 2014-06-09 16:22:12.214548 7f8691765700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3794> 2014-06-09 16:22:12.214553 7f8689755700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3793> 2014-06-09 16:22:12.214558 7f868d75d700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3792> 2014-06-09 16:22:12.214562 7f8689f56700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3791> 2014-06-09 16:22:12.214566 7f868ff62700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3790> 2014-06-09 16:22:12.214570 7f868af58700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3789> 2014-06-09 16:22:12.214575 7f868f761700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3788> 2014-06-09 16:22:12.214579 7f868df5e700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3787> 2014-06-09 16:22:12.214583 7f868bf5a700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3786> 2014-06-09 16:22:12.214587 7f868c75b700 2 waiting 51 > 50 ops || 4601888 > 104857600 -3785> 2014-06-09 16:22:12.214592 7f8692f68700 2 waiting 51 > 50 ops || 5053985 > 104857600 -3784> 2014-06-09 16:22:12.214596 7f868b759700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3783> 2014-06-09 16:22:12.214606 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x139656e0] -3782> 2014-06-09 16:22:12.214611 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x139656e0] -3781> 2014-06-09 16:22:12.214612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3780> 2014-06-09 16:22:12.214613 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3779> 2014-06-09 16:22:12.214616 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6360 (1 active) -3778> 2014-06-09 16:22:12.214619 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0d80 (2 active) -3777> 2014-06-09 16:22:12.214624 7f868e75f700 10 op_submit_finish 4252 -3776> 2014-06-09 16:22:12.214628 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.13c( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/26 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 2.13c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/200) -3775> 2014-06-09 16:22:12.214664 7f868e75f700 10 log is not dirty -3774> 2014-06-09 16:22:12.214669 7f868e75f700 5 queue_transactions existing osr(2.13c 0x2ed0610)/0x2ed0610 -3773> 2014-06-09 16:22:12.214671 7f868e75f700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3772> 2014-06-09 16:22:12.214676 7f86a5e11700 5 _do_op 0xc2d4b40 seq 4204 osr(0.164 0x2bd6810)/0x2bd6810 start -3771> 2014-06-09 16:22:12.214679 7f86a5e11700 10 _do_transaction on 0xbe8ab40 -3770> 2014-06-09 16:22:12.214686 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3769> 2014-06-09 16:22:12.214728 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3768> 2014-06-09 16:22:12.214746 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4204.0.0 <= header.spos 150443.0.0 -3767> 2014-06-09 16:22:12.214750 7f86a5e11700 5 submit_transaction r = 0 -3766> 2014-06-09 16:22:12.214762 7f86a5e11700 10 _do_op 0xc2d4b40 seq 4204 r = 0, finisher 0xccf9ac0 0 -3765> 2014-06-09 16:22:12.214769 7f86a5e11700 10 _finish_op 0xc2d4b40 seq 4204 osr(0.164 0x2bd6810)/0x2bd6810 -3764> 2014-06-09 16:22:12.214770 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6e0c0] -3763> 2014-06-09 16:22:12.214779 7f868a757700 10 op_submit_start 4253 -3762> 2014-06-09 16:22:12.214780 7f868a757700 5 queue_transactions (trailing journal) 4253 -3761> 2014-06-09 16:22:12.214782 7f868a757700 5 queue_op 0x32539f20 seq 4253 osr(3.52a 0x3ed1670) 828 bytes (queue has 50 ops and 4600994 bytes) -3760> 2014-06-09 16:22:12.214774 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -3759> 2014-06-09 16:22:12.214786 7f868ef60700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3758> 2014-06-09 16:22:12.214790 7f8690763700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3757> 2014-06-09 16:22:12.214786 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -3756> 2014-06-09 16:22:12.214794 7f8690f64700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3755> 2014-06-09 16:22:12.214798 7f8692767700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3754> 2014-06-09 16:22:12.214801 7f8691f66700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3753> 2014-06-09 16:22:12.214796 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.164( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:186 -3752> 2014-06-09 16:22:12.214805 7f8691765700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3751> 2014-06-09 16:22:12.214805 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6e0c0] -3750> 2014-06-09 16:22:12.214807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3749> 2014-06-09 16:22:12.214808 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3748> 2014-06-09 16:22:12.214809 7f8689755700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3747> 2014-06-09 16:22:12.214813 7f868d75d700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3746> 2014-06-09 16:22:12.214817 7f8689f56700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3745> 2014-06-09 16:22:12.214822 7f868ff62700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3744> 2014-06-09 16:22:12.214826 7f868af58700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3743> 2014-06-09 16:22:12.214830 7f868f761700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3742> 2014-06-09 16:22:12.214834 7f868df5e700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3741> 2014-06-09 16:22:12.214839 7f868bf5a700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3740> 2014-06-09 16:22:12.214843 7f868c75b700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3739> 2014-06-09 16:22:12.214847 7f8692f68700 2 waiting 51 > 50 ops || 5053952 > 104857600 -3738> 2014-06-09 16:22:12.214851 7f868b759700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3737> 2014-06-09 16:22:12.214856 7f868e75f700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3736> 2014-06-09 16:22:12.214860 7f868cf5c700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3735> 2014-06-09 16:22:12.214870 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf9ac0] -3734> 2014-06-09 16:22:12.214875 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf9ac0] -3733> 2014-06-09 16:22:12.214877 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3732> 2014-06-09 16:22:12.214878 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3731> 2014-06-09 16:22:12.214881 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0d80 (1 active) -3730> 2014-06-09 16:22:12.214884 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338fc0 (2 active) -3729> 2014-06-09 16:22:12.214889 7f868a757700 10 op_submit_finish 4253 -3728> 2014-06-09 16:22:12.214893 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.470( v 213'652 (0'0,213'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/207) -3727> 2014-06-09 16:22:12.214929 7f868a757700 10 log is not dirty -3726> 2014-06-09 16:22:12.214934 7f868a757700 5 queue_transactions existing osr(3.470 0x3ed1540)/0x3ed1540 -3725> 2014-06-09 16:22:12.214936 7f868a757700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3724> 2014-06-09 16:22:12.214941 7f86a5e11700 5 _do_op 0x30607980 seq 4205 osr(0.319 0x2bd6d90)/0x2bd6d90 start -3723> 2014-06-09 16:22:12.214945 7f86a5e11700 10 _do_transaction on 0x2c19ec0 -3722> 2014-06-09 16:22:12.214951 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3721> 2014-06-09 16:22:12.215052 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3720> 2014-06-09 16:22:12.215071 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4205.0.0 <= header.spos 150443.0.0 -3719> 2014-06-09 16:22:12.215075 7f86a5e11700 5 submit_transaction r = 0 -3718> 2014-06-09 16:22:12.215089 7f86a5e11700 10 _do_op 0x30607980 seq 4205 r = 0, finisher 0xc2d7b20 0 -3717> 2014-06-09 16:22:12.215096 7f86a5e11700 10 _finish_op 0x30607980 seq 4205 osr(0.319 0x2bd6d90)/0x2bd6d90 -3716> 2014-06-09 16:22:12.215097 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd72d80] -3715> 2014-06-09 16:22:12.215106 7f8690763700 10 op_submit_start 4254 -3714> 2014-06-09 16:22:12.215107 7f8690763700 5 queue_transactions (trailing journal) 4254 -3713> 2014-06-09 16:22:12.215100 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -3712> 2014-06-09 16:22:12.215109 7f8690763700 5 queue_op 0x30060900 seq 4254 osr(2.5bf 0x2ed1b10) 861 bytes (queue has 50 ops and 4600994 bytes) -3711> 2014-06-09 16:22:12.215113 7f8690f64700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3710> 2014-06-09 16:22:12.215110 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -3709> 2014-06-09 16:22:12.215117 7f8692767700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3708> 2014-06-09 16:22:12.215121 7f8691f66700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3707> 2014-06-09 16:22:12.215124 7f8691765700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3706> 2014-06-09 16:22:12.215119 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.319( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 -3705> 2014-06-09 16:22:12.215128 7f8689755700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3704> 2014-06-09 16:22:12.215128 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd72d80] -3703> 2014-06-09 16:22:12.215130 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3702> 2014-06-09 16:22:12.215131 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3701> 2014-06-09 16:22:12.215131 7f868d75d700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3700> 2014-06-09 16:22:12.215135 7f8689f56700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3699> 2014-06-09 16:22:12.215139 7f868ff62700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3698> 2014-06-09 16:22:12.215143 7f868af58700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3697> 2014-06-09 16:22:12.215147 7f868f761700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3696> 2014-06-09 16:22:12.215152 7f868df5e700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3695> 2014-06-09 16:22:12.215156 7f868bf5a700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3694> 2014-06-09 16:22:12.215160 7f868c75b700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3693> 2014-06-09 16:22:12.215164 7f8692f68700 2 waiting 51 > 50 ops || 5053952 > 104857600 -3692> 2014-06-09 16:22:12.215168 7f868b759700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3691> 2014-06-09 16:22:12.215173 7f868e75f700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3690> 2014-06-09 16:22:12.215177 7f868cf5c700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3689> 2014-06-09 16:22:12.215181 7f868a757700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3688> 2014-06-09 16:22:12.215185 7f868ef60700 2 waiting 51 > 50 ops || 4601822 > 104857600 -3687> 2014-06-09 16:22:12.215195 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d7b20] -3686> 2014-06-09 16:22:12.215200 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d7b20] -3685> 2014-06-09 16:22:12.215201 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3684> 2014-06-09 16:22:12.215202 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3683> 2014-06-09 16:22:12.215206 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338fc0 (1 active) -3682> 2014-06-09 16:22:12.216763 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa6c0 (2 active) -3681> 2014-06-09 16:22:12.216773 7f8690763700 10 op_submit_finish 4254 -3680> 2014-06-09 16:22:12.216777 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -3679> 2014-06-09 16:22:12.216794 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -3678> 2014-06-09 16:22:12.216804 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x26aa9200,0x26aa9100,0x26aa9000,0x26aa8f00,0x26aa8e00 -3677> 2014-06-09 16:22:12.216830 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -3676> 2014-06-09 16:22:12.216841 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -3675> 2014-06-09 16:22:12.216851 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -3674> 2014-06-09 16:22:12.216867 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:967 -3673> 2014-06-09 16:22:12.216878 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -3672> 2014-06-09 16:22:12.216887 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.6b4( v 213'576 (0'0,213'576] local-les=243 n=115 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -3671> 2014-06-09 16:22:12.216959 7f8690763700 10 log is not dirty -3670> 2014-06-09 16:22:12.216966 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -3669> 2014-06-09 16:22:12.216979 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.035029 7 0.000341 -3668> 2014-06-09 16:22:12.216992 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -3667> 2014-06-09 16:22:12.217002 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -3666> 2014-06-09 16:22:12.217013 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -3665> 2014-06-09 16:22:12.217024 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -3664> 2014-06-09 16:22:12.217033 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -3663> 2014-06-09 16:22:12.217041 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -3662> 2014-06-09 16:22:12.217051 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -3661> 2014-06-09 16:22:12.217062 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -3660> 2014-06-09 16:22:12.217072 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -3659> 2014-06-09 16:22:12.217088 7f8690763700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -3658> 2014-06-09 16:22:12.217090 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:2.185) v4 -- ?+0 0x3343d180 -3657> 2014-06-09 16:22:12.217105 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 -3656> 2014-06-09 16:22:12.217140 7f8690763700 10 log is not dirty -3655> 2014-06-09 16:22:12.217144 7f8690763700 5 queue_transactions existing osr(2.185 0x2ed09a0)/0x2ed09a0 -3654> 2014-06-09 16:22:12.217147 7f8690763700 2 waiting 51 > 50 ops || 4601855 > 104857600 -3653> 2014-06-09 16:22:12.217173 7f86a5e11700 5 _do_op 0x1e08a2a0 seq 4206 osr(1.b3 0x384c6d0)/0x384c6d0 start -3652> 2014-06-09 16:22:12.217178 7f86a5e11700 10 _do_transaction on 0xc245b00 -3651> 2014-06-09 16:22:12.217185 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3650> 2014-06-09 16:22:12.217238 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3649> 2014-06-09 16:22:12.217256 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4206.0.0 <= header.spos 150443.0.0 -3648> 2014-06-09 16:22:12.217260 7f86a5e11700 5 submit_transaction r = 0 -3647> 2014-06-09 16:22:12.217276 7f86a5e11700 10 _do_op 0x1e08a2a0 seq 4206 r = 0, finisher 0xc2d7a40 0 -3646> 2014-06-09 16:22:12.217283 7f86a5e11700 10 _finish_op 0x1e08a2a0 seq 4206 osr(1.b3 0x384c6d0)/0x384c6d0 -3645> 2014-06-09 16:22:12.217286 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd620] -3644> 2014-06-09 16:22:12.217295 7f8692767700 10 op_submit_start 4255 -3643> 2014-06-09 16:22:12.217296 7f8692767700 5 queue_transactions (trailing journal) 4255 -3642> 2014-06-09 16:22:12.217289 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -3641> 2014-06-09 16:22:12.217297 7f8692767700 5 queue_op 0xc2eb980 seq 4255 osr(2.6f4 0x2ed1e20) 861 bytes (queue has 50 ops and 4600997 bytes) -3640> 2014-06-09 16:22:12.217302 7f8691f66700 2 waiting 51 > 50 ops || 4601858 > 104857600 -3639> 2014-06-09 16:22:12.217299 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -3638> 2014-06-09 16:22:12.217306 7f8691765700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3637> 2014-06-09 16:22:12.217311 7f8689755700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3636> 2014-06-09 16:22:12.217315 7f868d75d700 2 waiting 51 > 50 ops || 4601858 > 104857600 -3635> 2014-06-09 16:22:12.217309 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:172 -3634> 2014-06-09 16:22:12.217318 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd620] -3633> 2014-06-09 16:22:12.217320 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3632> 2014-06-09 16:22:12.217320 7f8689f56700 2 waiting 51 > 50 ops || 4601858 > 104857600 -3631> 2014-06-09 16:22:12.217321 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3630> 2014-06-09 16:22:12.217324 7f868ff62700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3629> 2014-06-09 16:22:12.217328 7f868af58700 2 waiting 51 > 50 ops || 4601858 > 104857600 -3628> 2014-06-09 16:22:12.217333 7f868f761700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3627> 2014-06-09 16:22:12.217337 7f868df5e700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3626> 2014-06-09 16:22:12.217342 7f868bf5a700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3625> 2014-06-09 16:22:12.217347 7f868c75b700 2 waiting 51 > 50 ops || 4601858 > 104857600 -3624> 2014-06-09 16:22:12.217351 7f8692f68700 2 waiting 51 > 50 ops || 5053955 > 104857600 -3623> 2014-06-09 16:22:12.217356 7f868b759700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3622> 2014-06-09 16:22:12.217361 7f868e75f700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3621> 2014-06-09 16:22:12.217366 7f868cf5c700 2 waiting 51 > 50 ops || 4601858 > 104857600 -3620> 2014-06-09 16:22:12.217370 7f868a757700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3619> 2014-06-09 16:22:12.217375 7f868ef60700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3618> 2014-06-09 16:22:12.217379 7f8690763700 2 waiting 51 > 50 ops || 4601858 > 104857600 -3617> 2014-06-09 16:22:12.217383 7f8690f64700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3616> 2014-06-09 16:22:12.217394 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d7a40] -3615> 2014-06-09 16:22:12.217398 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d7a40] -3614> 2014-06-09 16:22:12.217400 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3613> 2014-06-09 16:22:12.217401 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3612> 2014-06-09 16:22:12.217404 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa6c0 (1 active) -3611> 2014-06-09 16:22:12.217407 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b8c0 (2 active) -3610> 2014-06-09 16:22:12.217412 7f8692767700 10 op_submit_finish 4255 -3609> 2014-06-09 16:22:12.217416 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -3608> 2014-06-09 16:22:12.217430 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -3607> 2014-06-09 16:22:12.217439 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -3606> 2014-06-09 16:22:12.217448 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -3605> 2014-06-09 16:22:12.217457 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -3604> 2014-06-09 16:22:12.217464 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -3603> 2014-06-09 16:22:12.217478 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:156 -3602> 2014-06-09 16:22:12.217487 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -3601> 2014-06-09 16:22:12.217495 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.1c9( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -3600> 2014-06-09 16:22:12.217504 7f8692767700 10 log is not dirty -3599> 2014-06-09 16:22:12.217509 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -3598> 2014-06-09 16:22:12.217523 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -3597> 2014-06-09 16:22:12.217532 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1b0b6700,0x1b0b6600,0x1b0b6500,0x1b0b6400,0x1b0b6300,0x1b0b6200 -3596> 2014-06-09 16:22:12.217558 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.2 has 1 missing -3595> 2014-06-09 16:22:12.217568 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery -3594> 2014-06-09 16:22:12.217585 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1060 -3593> 2014-06-09 16:22:12.217596 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -3592> 2014-06-09 16:22:12.217605 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.562( v 214'607 (0'0,214'607] local-les=243 n=116 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -3591> 2014-06-09 16:22:12.217615 7f8692767700 10 log is not dirty -3590> 2014-06-09 16:22:12.217619 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -3589> 2014-06-09 16:22:12.217631 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -3588> 2014-06-09 16:22:12.217639 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -3587> 2014-06-09 16:22:12.217647 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -3586> 2014-06-09 16:22:12.217656 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -3585> 2014-06-09 16:22:12.217664 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -3584> 2014-06-09 16:22:12.217676 7f8692767700 15 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:152 -3583> 2014-06-09 16:22:12.217686 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -3582> 2014-06-09 16:22:12.217694 7f8692767700 20 osd.26 pg_epoch: 243 pg[1.43b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -3581> 2014-06-09 16:22:12.217703 7f8692767700 10 log is not dirty -3580> 2014-06-09 16:22:12.217708 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.179( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.179( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -3579> 2014-06-09 16:22:12.217741 7f8692767700 10 log is not dirty -3578> 2014-06-09 16:22:12.217745 7f8692767700 5 queue_transactions existing osr(0.179 0x2bd6800)/0x2bd6800 -3577> 2014-06-09 16:22:12.217748 7f8692767700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3576> 2014-06-09 16:22:12.217753 7f86a5e11700 5 _do_op 0xbe7daa0 seq 4207 osr(3.78a 0x3ed1d80)/0x3ed1d80 start -3575> 2014-06-09 16:22:12.217756 7f86a5e11700 10 _do_transaction on 0xc160f00 -3574> 2014-06-09 16:22:12.217763 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3573> 2014-06-09 16:22:12.217806 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3572> 2014-06-09 16:22:12.217823 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4207.0.0 <= header.spos 150443.0.0 -3571> 2014-06-09 16:22:12.217828 7f86a5e11700 5 submit_transaction r = 0 -3570> 2014-06-09 16:22:12.217862 7f86a5e11700 10 _do_op 0xbe7daa0 seq 4207 r = 0, finisher 0x31412300 0 -3569> 2014-06-09 16:22:12.217869 7f86a5e11700 10 _finish_op 0xbe7daa0 seq 4207 osr(3.78a 0x3ed1d80)/0x3ed1d80 -3568> 2014-06-09 16:22:12.217871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x189ab360] -3567> 2014-06-09 16:22:12.217879 7f8691765700 10 op_submit_start 4256 -3566> 2014-06-09 16:22:12.217881 7f8691765700 5 queue_transactions (trailing journal) 4256 -3565> 2014-06-09 16:22:12.217882 7f8691765700 5 queue_op 0xc88b680 seq 4256 osr(1.54a 0x384daf0) 828 bytes (queue has 50 ops and 4600964 bytes) -3564> 2014-06-09 16:22:12.217874 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] _finish_recovery -3563> 2014-06-09 16:22:12.217886 7f8689755700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3562> 2014-06-09 16:22:12.217890 7f868d75d700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3561> 2014-06-09 16:22:12.217886 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] purge_strays -3560> 2014-06-09 16:22:12.217894 7f8689f56700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3559> 2014-06-09 16:22:12.217898 7f868ff62700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3558> 2014-06-09 16:22:12.217901 7f868af58700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3557> 2014-06-09 16:22:12.217896 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.78a( v 243'546 (0'0,243'546] local-les=243 n=111 ec=118 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 lcod 213'545 mlcod 213'545 active+clean] publish_stats_to_osd 243:876 -3556> 2014-06-09 16:22:12.217905 7f868f761700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3555> 2014-06-09 16:22:12.217906 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x189ab360] -3554> 2014-06-09 16:22:12.217908 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3553> 2014-06-09 16:22:12.217908 7f868df5e700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3552> 2014-06-09 16:22:12.217909 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3551> 2014-06-09 16:22:12.217912 7f868bf5a700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3550> 2014-06-09 16:22:12.217916 7f868c75b700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3549> 2014-06-09 16:22:12.217920 7f8692f68700 2 waiting 51 > 50 ops || 5053922 > 104857600 -3548> 2014-06-09 16:22:12.217924 7f868b759700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3547> 2014-06-09 16:22:12.217929 7f868e75f700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3546> 2014-06-09 16:22:12.217933 7f868cf5c700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3545> 2014-06-09 16:22:12.217937 7f868a757700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3544> 2014-06-09 16:22:12.217942 7f868ef60700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3543> 2014-06-09 16:22:12.217946 7f8690763700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3542> 2014-06-09 16:22:12.217950 7f8690f64700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3541> 2014-06-09 16:22:12.217954 7f8692767700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3540> 2014-06-09 16:22:12.217958 7f8691f66700 2 waiting 51 > 50 ops || 4601825 > 104857600 -3539> 2014-06-09 16:22:12.217969 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31412300] -3538> 2014-06-09 16:22:12.217974 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31412300] -3537> 2014-06-09 16:22:12.217976 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3536> 2014-06-09 16:22:12.217977 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3535> 2014-06-09 16:22:12.217980 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b8c0 (1 active) -3534> 2014-06-09 16:22:12.217984 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d07e0 (2 active) -3533> 2014-06-09 16:22:12.217989 7f8691765700 10 op_submit_finish 4256 -3532> 2014-06-09 16:22:12.217993 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.54a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.54a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -3531> 2014-06-09 16:22:12.218030 7f8691765700 10 log is not dirty -3530> 2014-06-09 16:22:12.218035 7f8691765700 5 queue_transactions existing osr(1.54a 0x384daf0)/0x384daf0 -3529> 2014-06-09 16:22:12.218037 7f8691765700 2 waiting 51 > 50 ops || 4601792 > 104857600 -3528> 2014-06-09 16:22:12.218042 7f86a5e11700 5 _do_op 0x30a24e0 seq 4208 osr(3.2ac 0x3ed0b50)/0x3ed0b50 start -3527> 2014-06-09 16:22:12.218045 7f86a5e11700 10 _do_transaction on 0xc1252c0 -3526> 2014-06-09 16:22:12.218050 7f86a5e11700 15 _omap_setkeys meta/1fe0d7a6/pglog_3.2ac/0//-1 -3525> 2014-06-09 16:22:12.218099 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe0d7a6/pglog_3.2ac/0//-1 -3524> 2014-06-09 16:22:12.218115 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3523> 2014-06-09 16:22:12.218155 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3522> 2014-06-09 16:22:12.218170 7f86a5e11700 15 _omap_rmkeys meta/1fe0d7a6/pglog_3.2ac/0//-1 -3521> 2014-06-09 16:22:12.218175 7f86a5e11700 10 _omap_rmkeys meta/1fe0d7a6/pglog_3.2ac/0//-1 r = 0 -3520> 2014-06-09 16:22:12.218182 7f86a5e11700 15 _omap_setkeys meta/1fe0d7a6/pglog_3.2ac/0//-1 -3519> 2014-06-09 16:22:12.218194 7f86a5e11700 15 _touch 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -3518> 2014-06-09 16:22:12.218240 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.2ac_head oid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -3517> 2014-06-09 16:22:12.218272 7f86a5e11700 20 write_state seq is 30172 -3516> 2014-06-09 16:22:12.218279 7f86a5e11700 20 set_header setting 30171 cid 3.2ac_head oid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 parent seq 0 -3515> 2014-06-09 16:22:12.218293 7f86a5e11700 10 _touch 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 = 0 -3514> 2014-06-09 16:22:12.218300 7f86a5e11700 15 _write 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 0~16384 -3513> 2014-06-09 16:22:12.218305 7f86a5e11700 10 file_to_extents done -3512> 2014-06-09 16:22:12.218366 7f86a5e11700 10 _generic_write 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 0~16384 = 0 -3511> 2014-06-09 16:22:12.218376 7f86a5e11700 15 _setattrs 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -3510> 2014-06-09 16:22:12.218386 7f86a5e11700 10 _setattrs 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 = 0 -3509> 2014-06-09 16:22:12.218391 7f86a5e11700 15 _setattrs 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 -3508> 2014-06-09 16:22:12.218400 7f86a5e11700 10 _setattrs 3.2ac_head/3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 = 0 -3507> 2014-06-09 16:22:12.218406 7f86a5e11700 10 cid: oid: 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 not skipping op, *spos 4208.0.0 -3506> 2014-06-09 16:22:12.218409 7f86a5e11700 10 > header.spos 0.0.0 -3505> 2014-06-09 16:22:12.218414 7f86a5e11700 20 set_header setting 30171 cid 3.2ac_head oid 3fad62ac/rbd_data.11766b8b4567.00000000000016bf/head//3 parent seq 0 -3504> 2014-06-09 16:22:12.218427 7f86a5e11700 10 cid: oid: 1fe0d7a6/pglog_3.2ac/0//-1 skipping op, spos 4208.0.0 <= header.spos 149708.0.0 -3503> 2014-06-09 16:22:12.218431 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4208.0.0 <= header.spos 150443.0.0 -3502> 2014-06-09 16:22:12.218434 7f86a5e11700 5 submit_transaction r = 0 -3501> 2014-06-09 16:22:12.218533 7f86a5e11700 10 _do_op 0x30a24e0 seq 4208 r = 0, finisher 0x15a3ed20 0x86d9ec0 -3500> 2014-06-09 16:22:12.218541 7f86a5e11700 10 _finish_op 0x30a24e0 seq 4208 osr(3.2ac 0x3ed0b50)/0x3ed0b50 -3499> 2014-06-09 16:22:12.218543 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94480] -3498> 2014-06-09 16:22:12.218552 7f868d75d700 10 op_submit_start 4257 -3497> 2014-06-09 16:22:12.218553 7f868d75d700 5 queue_transactions (trailing journal) 4257 -3496> 2014-06-09 16:22:12.218554 7f868d75d700 5 queue_op 0x14dd7140 seq 4257 osr(3.22d 0x3ed0c40) 861 bytes (queue has 50 ops and 4583043 bytes) -3495> 2014-06-09 16:22:12.218546 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 30 -3494> 2014-06-09 16:22:12.218559 7f8689f56700 2 waiting 51 > 50 ops || 4583904 > 104857600 -3493> 2014-06-09 16:22:12.218563 7f868ff62700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3492> 2014-06-09 16:22:12.218566 7f868af58700 2 waiting 51 > 50 ops || 4583904 > 104857600 -3491> 2014-06-09 16:22:12.218570 7f868f761700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3490> 2014-06-09 16:22:12.218568 7f86a460e700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.218560, event: op_commit, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) -3489> 2014-06-09 16:22:12.218573 7f868df5e700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3488> 2014-06-09 16:22:12.218576 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94480] -3487> 2014-06-09 16:22:12.218577 7f868bf5a700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3486> 2014-06-09 16:22:12.218578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3485> 2014-06-09 16:22:12.218579 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3484> 2014-06-09 16:22:12.218581 7f868c75b700 2 waiting 51 > 50 ops || 4583904 > 104857600 -3483> 2014-06-09 16:22:12.218585 7f8692f68700 2 waiting 51 > 50 ops || 5036001 > 104857600 -3482> 2014-06-09 16:22:12.218589 7f868b759700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3481> 2014-06-09 16:22:12.218594 7f868e75f700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3480> 2014-06-09 16:22:12.218598 7f868cf5c700 2 waiting 51 > 50 ops || 4583904 > 104857600 -3479> 2014-06-09 16:22:12.218602 7f868a757700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3478> 2014-06-09 16:22:12.218607 7f868ef60700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3477> 2014-06-09 16:22:12.218611 7f8690763700 2 waiting 51 > 50 ops || 4583904 > 104857600 -3476> 2014-06-09 16:22:12.218615 7f8690f64700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3475> 2014-06-09 16:22:12.218619 7f8692767700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3474> 2014-06-09 16:22:12.218623 7f8691f66700 2 waiting 51 > 50 ops || 4583904 > 104857600 -3473> 2014-06-09 16:22:12.218628 7f8691765700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3472> 2014-06-09 16:22:12.218632 7f8689755700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3471> 2014-06-09 16:22:12.218644 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x15a3ed20] -3470> 2014-06-09 16:22:12.218647 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 30 -3469> 2014-06-09 16:22:12.218667 7f86a4e0f700 5 -- op tracker -- , seq: 196, time: 2014-06-09 16:22:12.218661, event: op_applied, op: osd_op(client.4566.0:33893 rbd_data.11766b8b4567.00000000000016bf [set-alloc-hint object_size 4194304 write_size 4194304,write 0~16384] 3.3fad62ac RETRY=3 ack+ondisk+retry+write e242) -3468> 2014-06-09 16:22:12.218675 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.2ac( v 243'664 (0'0,243'664] local-les=243 n=134 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 luod=213'663 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'664 -3467> 2014-06-09 16:22:12.218702 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x15a3ed20] -3466> 2014-06-09 16:22:12.218704 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3465> 2014-06-09 16:22:12.218705 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3464> 2014-06-09 16:22:12.218709 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d07e0 (1 active) -3463> 2014-06-09 16:22:12.218712 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62e360 (2 active) -3462> 2014-06-09 16:22:12.218717 7f868d75d700 10 op_submit_finish 4257 -3461> 2014-06-09 16:22:12.218722 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -3460> 2014-06-09 16:22:12.218735 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.182743 7 0.000343 -3459> 2014-06-09 16:22:12.218746 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -3458> 2014-06-09 16:22:12.218757 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -3457> 2014-06-09 16:22:12.218767 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -3456> 2014-06-09 16:22:12.218777 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -3455> 2014-06-09 16:22:12.218786 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -3454> 2014-06-09 16:22:12.218793 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -3453> 2014-06-09 16:22:12.218804 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -3452> 2014-06-09 16:22:12.218873 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -3451> 2014-06-09 16:22:12.218883 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -3450> 2014-06-09 16:22:12.218897 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 -3449> 2014-06-09 16:22:12.218900 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:1.301) v4 -- ?+0 0x31745a40 -3448> 2014-06-09 16:22:12.218915 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:160 -3447> 2014-06-09 16:22:12.218949 7f868d75d700 10 log is not dirty -3446> 2014-06-09 16:22:12.218953 7f868d75d700 5 queue_transactions existing osr(1.301 0x384d030)/0x384d030 -3445> 2014-06-09 16:22:12.218956 7f868d75d700 2 waiting 51 > 50 ops || 4583904 > 104857600 -3444> 2014-06-09 16:22:12.218961 7f86a5e11700 5 _do_op 0x324e7f80 seq 4209 osr(1.ab 0x384c6f0)/0x384c6f0 start -3443> 2014-06-09 16:22:12.218964 7f86a5e11700 10 _do_transaction on 0x2e63500 -3442> 2014-06-09 16:22:12.218969 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3441> 2014-06-09 16:22:12.219013 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3440> 2014-06-09 16:22:12.219028 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4209.0.0 <= header.spos 150443.0.0 -3439> 2014-06-09 16:22:12.219032 7f86a5e11700 5 submit_transaction r = 0 -3438> 2014-06-09 16:22:12.219049 7f86a5e11700 10 _do_op 0x324e7f80 seq 4209 r = 0, finisher 0x3205d740 0 -3437> 2014-06-09 16:22:12.219057 7f86a5e11700 10 _finish_op 0x324e7f80 seq 4209 osr(1.ab 0x384c6f0)/0x384c6f0 -3436> 2014-06-09 16:22:12.219058 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205d060] -3435> 2014-06-09 16:22:12.219061 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205d060] -3434> 2014-06-09 16:22:12.219063 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3433> 2014-06-09 16:22:12.219064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3432> 2014-06-09 16:22:12.219067 7f868ff62700 10 op_submit_start 4258 -3431> 2014-06-09 16:22:12.219069 7f868ff62700 5 queue_transactions (trailing journal) 4258 -3430> 2014-06-09 16:22:12.219070 7f868ff62700 5 queue_op 0xcd90060 seq 4258 osr(3.5ce 0x3ed17c0) 828 bytes (queue has 50 ops and 4583045 bytes) -3429> 2014-06-09 16:22:12.219075 7f868af58700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3428> 2014-06-09 16:22:12.219080 7f868f761700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3427> 2014-06-09 16:22:12.219085 7f868df5e700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3426> 2014-06-09 16:22:12.219089 7f868bf5a700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3425> 2014-06-09 16:22:12.219094 7f868c75b700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3424> 2014-06-09 16:22:12.219098 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3423> 2014-06-09 16:22:12.219103 7f868b759700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3422> 2014-06-09 16:22:12.219107 7f868e75f700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3421> 2014-06-09 16:22:12.219111 7f868cf5c700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3420> 2014-06-09 16:22:12.219115 7f868a757700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3419> 2014-06-09 16:22:12.219120 7f868ef60700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3418> 2014-06-09 16:22:12.219124 7f8690763700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3417> 2014-06-09 16:22:12.219128 7f8690f64700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3416> 2014-06-09 16:22:12.219133 7f8692767700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3415> 2014-06-09 16:22:12.219137 7f8691f66700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3414> 2014-06-09 16:22:12.219141 7f8691765700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3413> 2014-06-09 16:22:12.219145 7f8689755700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3412> 2014-06-09 16:22:12.219149 7f868d75d700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3411> 2014-06-09 16:22:12.219153 7f8689f56700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3410> 2014-06-09 16:22:12.219163 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205d740] -3409> 2014-06-09 16:22:12.219167 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205d740] -3408> 2014-06-09 16:22:12.219168 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3407> 2014-06-09 16:22:12.219169 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3406> 2014-06-09 16:22:12.219172 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62e360 (1 active) -3405> 2014-06-09 16:22:12.219174 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525320 (2 active) -3404> 2014-06-09 16:22:12.219178 7f868ff62700 10 op_submit_finish 4258 -3403> 2014-06-09 16:22:12.219183 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.504( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.504( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -3402> 2014-06-09 16:22:12.219223 7f868ff62700 10 log is not dirty -3401> 2014-06-09 16:22:12.219228 7f868ff62700 5 queue_transactions existing osr(1.504 0x384dbd0)/0x384dbd0 -3400> 2014-06-09 16:22:12.219230 7f868ff62700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3399> 2014-06-09 16:22:12.219235 7f86a5e11700 5 _do_op 0x230b9560 seq 4210 osr(3.527 0x3ed1660)/0x3ed1660 start -3398> 2014-06-09 16:22:12.219238 7f86a5e11700 10 _do_transaction on 0x3144cc0 -3397> 2014-06-09 16:22:12.219244 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3396> 2014-06-09 16:22:12.219286 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3395> 2014-06-09 16:22:12.219301 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4210.0.0 <= header.spos 150443.0.0 -3394> 2014-06-09 16:22:12.219305 7f86a5e11700 5 submit_transaction r = 0 -3393> 2014-06-09 16:22:12.219316 7f86a5e11700 10 _do_op 0x230b9560 seq 4210 r = 0, finisher 0x3062af60 0 -3392> 2014-06-09 16:22:12.219323 7f86a5e11700 10 _finish_op 0x230b9560 seq 4210 osr(3.527 0x3ed1660)/0x3ed1660 -3391> 2014-06-09 16:22:12.219324 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078780] -3390> 2014-06-09 16:22:12.219332 7f868f761700 10 op_submit_start 4259 -3389> 2014-06-09 16:22:12.219331 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078780] -3388> 2014-06-09 16:22:12.219333 7f868f761700 5 queue_transactions (trailing journal) 4259 -3387> 2014-06-09 16:22:12.219334 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3386> 2014-06-09 16:22:12.219335 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3385> 2014-06-09 16:22:12.219334 7f868f761700 5 queue_op 0x16096540 seq 4259 osr(3.2df 0x3ed0e40) 828 bytes (queue has 50 ops and 4583045 bytes) -3384> 2014-06-09 16:22:12.219339 7f868df5e700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3383> 2014-06-09 16:22:12.219343 7f868bf5a700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3382> 2014-06-09 16:22:12.219348 7f868c75b700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3381> 2014-06-09 16:22:12.219352 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3380> 2014-06-09 16:22:12.219356 7f868b759700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3379> 2014-06-09 16:22:12.219361 7f868e75f700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3378> 2014-06-09 16:22:12.219365 7f868cf5c700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3377> 2014-06-09 16:22:12.219369 7f868a757700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3376> 2014-06-09 16:22:12.219373 7f868ef60700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3375> 2014-06-09 16:22:12.219378 7f8690763700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3374> 2014-06-09 16:22:12.219382 7f8690f64700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3373> 2014-06-09 16:22:12.219386 7f8692767700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3372> 2014-06-09 16:22:12.219390 7f8691f66700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3371> 2014-06-09 16:22:12.219395 7f8691765700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3370> 2014-06-09 16:22:12.219399 7f8689755700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3369> 2014-06-09 16:22:12.219403 7f868d75d700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3368> 2014-06-09 16:22:12.219407 7f8689f56700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3367> 2014-06-09 16:22:12.219412 7f868ff62700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3366> 2014-06-09 16:22:12.219416 7f868af58700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3365> 2014-06-09 16:22:12.219425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062af60] -3364> 2014-06-09 16:22:12.219432 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062af60] -3363> 2014-06-09 16:22:12.219436 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3362> 2014-06-09 16:22:12.219438 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3361> 2014-06-09 16:22:12.219441 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525320 (1 active) -3360> 2014-06-09 16:22:12.219444 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3386c0 (2 active) -3359> 2014-06-09 16:22:12.219449 7f868f761700 10 op_submit_finish 4259 -3358> 2014-06-09 16:22:12.219453 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3357> 2014-06-09 16:22:12.219471 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops 0x270bf400,0x1c66b100,0x270bf300,0x1c66b000,0x270bf200,0x1c66af00,0x270bf100,0x1c66ae00,0x1c66ad00,0x1c66ac00,0x1c66ab00,0x1b0d7800,0x1b0b6800 -3356> 2014-06-09 16:22:12.219518 7f868f761700 10 log is not dirty -3355> 2014-06-09 16:22:12.219524 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -3354> 2014-06-09 16:22:12.219536 7f868f761700 15 osd.26 pg_epoch: 243 pg[1.188( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -3353> 2014-06-09 16:22:12.219545 7f868f761700 10 log is not dirty -3352> 2014-06-09 16:22:12.219569 7f868f761700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0x153599f0 (19 active) -3351> 2014-06-09 16:22:12.219610 7f868f761700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153599f0 (20 active) -3350> 2014-06-09 16:22:12.219635 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 3.111( v 213'733 (0'0,213'733] local-les=243 n=143 ec=118 les/c 243/243 242/242/222) -3349> 2014-06-09 16:22:12.219673 7f868f761700 10 log is not dirty -3348> 2014-06-09 16:22:12.219678 7f868f761700 5 queue_transactions existing osr(3.111 0x3ed0830)/0x3ed0830 -3347> 2014-06-09 16:22:12.219681 7f868f761700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3346> 2014-06-09 16:22:12.219686 7f86a5e11700 5 _do_op 0xd327aa0 seq 4211 osr(2.24d 0x2ed0a50)/0x2ed0a50 start -3345> 2014-06-09 16:22:12.219690 7f86a5e11700 10 _do_transaction on 0xbf0fd40 -3344> 2014-06-09 16:22:12.219695 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3343> 2014-06-09 16:22:12.219738 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3342> 2014-06-09 16:22:12.219752 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4211.0.0 <= header.spos 150443.0.0 -3341> 2014-06-09 16:22:12.219757 7f86a5e11700 5 submit_transaction r = 0 -3340> 2014-06-09 16:22:12.219769 7f86a5e11700 10 _do_op 0xd327aa0 seq 4211 r = 0, finisher 0xbd72720 0 -3339> 2014-06-09 16:22:12.219776 7f86a5e11700 10 _finish_op 0xd327aa0 seq 4211 osr(2.24d 0x2ed0a50)/0x2ed0a50 -3338> 2014-06-09 16:22:12.219777 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc5a72c0] -3337> 2014-06-09 16:22:12.219780 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc5a72c0] -3336> 2014-06-09 16:22:12.219782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3335> 2014-06-09 16:22:12.219783 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3334> 2014-06-09 16:22:12.219786 7f868bf5a700 10 op_submit_start 4260 -3333> 2014-06-09 16:22:12.219787 7f868bf5a700 5 queue_transactions (trailing journal) 4260 -3332> 2014-06-09 16:22:12.219789 7f868bf5a700 5 queue_op 0x324e7620 seq 4260 osr(0.4bd 0x2bd76f0) 828 bytes (queue has 50 ops and 4583045 bytes) -3331> 2014-06-09 16:22:12.219794 7f868c75b700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3330> 2014-06-09 16:22:12.219798 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3329> 2014-06-09 16:22:12.219803 7f868b759700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3328> 2014-06-09 16:22:12.219807 7f868e75f700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3327> 2014-06-09 16:22:12.219812 7f868cf5c700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3326> 2014-06-09 16:22:12.219816 7f868a757700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3325> 2014-06-09 16:22:12.219820 7f868ef60700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3324> 2014-06-09 16:22:12.219825 7f8690763700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3323> 2014-06-09 16:22:12.219829 7f8690f64700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3322> 2014-06-09 16:22:12.219833 7f8692767700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3321> 2014-06-09 16:22:12.219837 7f8691f66700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3320> 2014-06-09 16:22:12.219841 7f8691765700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3319> 2014-06-09 16:22:12.219846 7f8689755700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3318> 2014-06-09 16:22:12.219850 7f868d75d700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3317> 2014-06-09 16:22:12.219854 7f8689f56700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3316> 2014-06-09 16:22:12.219858 7f868ff62700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3315> 2014-06-09 16:22:12.219862 7f868af58700 2 waiting 51 > 50 ops || 4583906 > 104857600 -3314> 2014-06-09 16:22:12.219867 7f868f761700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3313> 2014-06-09 16:22:12.219871 7f868df5e700 2 waiting 51 > 50 ops || 4583873 > 104857600 -3312> 2014-06-09 16:22:12.219881 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd72720] -3311> 2014-06-09 16:22:12.219886 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd72720] -3310> 2014-06-09 16:22:12.219887 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3309> 2014-06-09 16:22:12.219888 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3308> 2014-06-09 16:22:12.219892 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3386c0 (1 active) -3307> 2014-06-09 16:22:12.219894 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6240 (2 active) -3306> 2014-06-09 16:22:12.219899 7f868bf5a700 10 op_submit_finish 4260 -3305> 2014-06-09 16:22:12.219903 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 243/206 242/242/205) -3304> 2014-06-09 16:22:12.219939 7f868bf5a700 10 log is not dirty -3303> 2014-06-09 16:22:12.219944 7f868bf5a700 5 queue_transactions existing osr(3.9f 0x3ed0600)/0x3ed0600 -3302> 2014-06-09 16:22:12.219946 7f868bf5a700 2 waiting 51 > 50 ops || 4583871 > 104857600 -3301> 2014-06-09 16:22:12.219951 7f86a5e11700 5 _do_op 0x324f49c0 seq 4212 osr(0.473 0x2bd74d0)/0x2bd74d0 start -3300> 2014-06-09 16:22:12.219954 7f86a5e11700 10 _do_transaction on 0xc2becc0 -3299> 2014-06-09 16:22:12.219960 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3298> 2014-06-09 16:22:12.220001 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3297> 2014-06-09 16:22:12.220015 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4212.0.0 <= header.spos 150443.0.0 -3296> 2014-06-09 16:22:12.220020 7f86a5e11700 5 submit_transaction r = 0 -3295> 2014-06-09 16:22:12.220032 7f86a5e11700 10 _do_op 0x324f49c0 seq 4212 r = 0, finisher 0x3042b6a0 0 -3294> 2014-06-09 16:22:12.220038 7f86a5e11700 10 _finish_op 0x324f49c0 seq 4212 osr(0.473 0x2bd74d0)/0x2bd74d0 -3293> 2014-06-09 16:22:12.220040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc9940] -3292> 2014-06-09 16:22:12.220042 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc9940] -3291> 2014-06-09 16:22:12.220044 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3290> 2014-06-09 16:22:12.220045 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3289> 2014-06-09 16:22:12.220048 7f8692f68700 10 op_submit_start 4261 -3288> 2014-06-09 16:22:12.220050 7f8692f68700 5 queue_transactions (trailing journal) 4261 -3287> 2014-06-09 16:22:12.220051 7f8692f68700 5 queue_op 0x2fc8d800 seq 4261 osr(3.750 0x3ed1eb0) 452958 bytes (queue has 50 ops and 5035175 bytes) -3286> 2014-06-09 16:22:12.220056 7f868b759700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3285> 2014-06-09 16:22:12.220061 7f868e75f700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3284> 2014-06-09 16:22:12.220065 7f868cf5c700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3283> 2014-06-09 16:22:12.220069 7f868a757700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3282> 2014-06-09 16:22:12.220074 7f868ef60700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3281> 2014-06-09 16:22:12.220078 7f8690763700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3280> 2014-06-09 16:22:12.220082 7f8690f64700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3279> 2014-06-09 16:22:12.220087 7f8692767700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3278> 2014-06-09 16:22:12.220091 7f8691f66700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3277> 2014-06-09 16:22:12.220095 7f8691765700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3276> 2014-06-09 16:22:12.220100 7f8689755700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3275> 2014-06-09 16:22:12.220104 7f868d75d700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3274> 2014-06-09 16:22:12.220108 7f8689f56700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3273> 2014-06-09 16:22:12.220113 7f868ff62700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3272> 2014-06-09 16:22:12.220117 7f868af58700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3271> 2014-06-09 16:22:12.220121 7f868f761700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3270> 2014-06-09 16:22:12.220126 7f868df5e700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3269> 2014-06-09 16:22:12.220130 7f868bf5a700 2 waiting 51 > 50 ops || 5036001 > 104857600 -3268> 2014-06-09 16:22:12.220135 7f868c75b700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3267> 2014-06-09 16:22:12.220145 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b6a0] -3266> 2014-06-09 16:22:12.220151 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b6a0] -3265> 2014-06-09 16:22:12.220153 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3264> 2014-06-09 16:22:12.220155 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3263> 2014-06-09 16:22:12.220158 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6240 (1 active) -3262> 2014-06-09 16:22:12.220161 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac45a0 (2 active) -3261> 2014-06-09 16:22:12.220166 7f8692f68700 10 op_submit_finish 4261 -3260> 2014-06-09 16:22:12.220171 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xbffe840 243'519 rep_tid=32 committed?=0 applied?=0 lock=0 op=osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad -3259> 2014-06-09 16:22:12.220198 7f8692f68700 10 osd.26 244 dequeue_op 0x270be300 finish -3258> 2014-06-09 16:22:12.220202 7f8692f68700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -3257> 2014-06-09 16:22:12.220209 7f8692f68700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -3256> 2014-06-09 16:22:12.220214 7f8692f68700 10 osd.26 244 dequeue_op 0x22009f00 prio 63 cost 1097728 latency 1.540870 osd_op(client.4574.0:32457 rbd_data.117e6b8b4567.0000000000001536 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1097728] 3.4db0ef4f RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] -3255> 2014-06-09 16:22:12.220237 7f8692f68700 5 -- op tracker -- , seq: 601, time: 2014-06-09 16:22:12.220231, event: reached_pg, op: osd_op(client.4574.0:32457 rbd_data.117e6b8b4567.0000000000001536 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1097728] 3.4db0ef4f RETRY=3 ack+ondisk+retry+write e242) -3254> 2014-06-09 16:22:12.220267 7f8692f68700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -3253> 2014-06-09 16:22:12.220279 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x22009f00 -3252> 2014-06-09 16:22:12.220287 7f8692f68700 10 osd.26 244 dequeue_op 0x22009f00 finish -3251> 2014-06-09 16:22:12.220291 7f8692f68700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -3250> 2014-06-09 16:22:12.220319 7f8692f68700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0x153589a0 (20 active) -3249> 2014-06-09 16:22:12.220323 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.1da( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -3248> 2014-06-09 16:22:12.220360 7f8692f68700 10 log is not dirty -3247> 2014-06-09 16:22:12.220365 7f8692f68700 5 queue_transactions existing osr(0.1da 0x2bd6920)/0x2bd6920 -3246> 2014-06-09 16:22:12.220367 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3245> 2014-06-09 16:22:12.220372 7f86a5e11700 5 _do_op 0xcabde00 seq 4213 osr(0.4ba 0x2bd7710)/0x2bd7710 start -3244> 2014-06-09 16:22:12.220375 7f86a5e11700 10 _do_transaction on 0xc2af200 -3243> 2014-06-09 16:22:12.220381 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3242> 2014-06-09 16:22:12.220424 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3241> 2014-06-09 16:22:12.220439 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4213.0.0 <= header.spos 150443.0.0 -3240> 2014-06-09 16:22:12.220443 7f86a5e11700 5 submit_transaction r = 0 -3239> 2014-06-09 16:22:12.220470 7f86a5e11700 10 _do_op 0xcabde00 seq 4213 r = 0, finisher 0x3042a1e0 0 -3238> 2014-06-09 16:22:12.220477 7f86a5e11700 10 _finish_op 0xcabde00 seq 4213 osr(0.4ba 0x2bd7710)/0x2bd7710 -3237> 2014-06-09 16:22:12.220479 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x327f38c0] -3236> 2014-06-09 16:22:12.220482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x327f38c0] -3235> 2014-06-09 16:22:12.220483 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3234> 2014-06-09 16:22:12.220484 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3233> 2014-06-09 16:22:12.220488 7f868e75f700 10 op_submit_start 4262 -3232> 2014-06-09 16:22:12.220490 7f868e75f700 5 queue_transactions (trailing journal) 4262 -3231> 2014-06-09 16:22:12.220491 7f868e75f700 5 queue_op 0x324f5260 seq 4262 osr(2.13c 0x2ed0610) 828 bytes (queue has 50 ops and 5035175 bytes) -3230> 2014-06-09 16:22:12.220497 7f868cf5c700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3229> 2014-06-09 16:22:12.220502 7f868a757700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3228> 2014-06-09 16:22:12.220507 7f868ef60700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3227> 2014-06-09 16:22:12.220511 7f8690763700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3226> 2014-06-09 16:22:12.220516 7f8690f64700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3225> 2014-06-09 16:22:12.220520 7f8692767700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3224> 2014-06-09 16:22:12.220525 7f8691f66700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3223> 2014-06-09 16:22:12.220529 7f8691765700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3222> 2014-06-09 16:22:12.220534 7f8689755700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3221> 2014-06-09 16:22:12.220538 7f868d75d700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3220> 2014-06-09 16:22:12.220543 7f8689f56700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3219> 2014-06-09 16:22:12.220547 7f868ff62700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3218> 2014-06-09 16:22:12.220552 7f868af58700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3217> 2014-06-09 16:22:12.220556 7f868f761700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3216> 2014-06-09 16:22:12.220560 7f868df5e700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3215> 2014-06-09 16:22:12.220564 7f868bf5a700 2 waiting 51 > 50 ops || 5036001 > 104857600 -3214> 2014-06-09 16:22:12.220569 7f868c75b700 2 waiting 51 > 50 ops || 5036036 > 104857600 -3213> 2014-06-09 16:22:12.220573 7f8692f68700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3212> 2014-06-09 16:22:12.220577 7f868b759700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3211> 2014-06-09 16:22:12.220586 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042a1e0] -3210> 2014-06-09 16:22:12.220591 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042a1e0] -3209> 2014-06-09 16:22:12.220593 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3208> 2014-06-09 16:22:12.220594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3207> 2014-06-09 16:22:12.220597 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac45a0 (1 active) -3206> 2014-06-09 16:22:12.220600 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89ec60 (2 active) -3205> 2014-06-09 16:22:12.220605 7f868e75f700 10 op_submit_finish 4262 -3204> 2014-06-09 16:22:12.220609 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.640( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 0.640( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) -3203> 2014-06-09 16:22:12.220644 7f868e75f700 10 log is not dirty -3202> 2014-06-09 16:22:12.220649 7f868e75f700 5 queue_transactions existing osr(0.640 0x2bd7c10)/0x2bd7c10 -3201> 2014-06-09 16:22:12.220651 7f868e75f700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3200> 2014-06-09 16:22:12.220656 7f86a5e11700 5 _do_op 0xc2d4d80 seq 4214 osr(2.754 0x3ed00f0)/0x3ed00f0 start -3199> 2014-06-09 16:22:12.220659 7f86a5e11700 10 _do_transaction on 0x2fb6900 -3198> 2014-06-09 16:22:12.220665 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3197> 2014-06-09 16:22:12.220707 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3196> 2014-06-09 16:22:12.220790 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4214.0.0 <= header.spos 150443.0.0 -3195> 2014-06-09 16:22:12.220795 7f86a5e11700 5 submit_transaction r = 0 -3194> 2014-06-09 16:22:12.220812 7f86a5e11700 10 _do_op 0xc2d4d80 seq 4214 r = 0, finisher 0xcc941e0 0 -3193> 2014-06-09 16:22:12.220818 7f86a5e11700 10 _finish_op 0xc2d4d80 seq 4214 osr(2.754 0x3ed00f0)/0x3ed00f0 -3192> 2014-06-09 16:22:12.220820 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2f40] -3191> 2014-06-09 16:22:12.220829 7f868a757700 10 op_submit_start 4263 -3190> 2014-06-09 16:22:12.220830 7f868a757700 5 queue_transactions (trailing journal) 4263 -3189> 2014-06-09 16:22:12.220831 7f868a757700 5 queue_op 0x32539ec0 seq 4263 osr(3.470 0x3ed1540) 828 bytes (queue has 50 ops and 5035142 bytes) -3188> 2014-06-09 16:22:12.220824 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -3187> 2014-06-09 16:22:12.220836 7f868ef60700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3186> 2014-06-09 16:22:12.220840 7f8690763700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3185> 2014-06-09 16:22:12.220835 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -3184> 2014-06-09 16:22:12.220844 7f8690f64700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3183> 2014-06-09 16:22:12.220847 7f8692767700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3182> 2014-06-09 16:22:12.220851 7f8691f66700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3181> 2014-06-09 16:22:12.220845 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.754( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 -3180> 2014-06-09 16:22:12.220854 7f8691765700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3179> 2014-06-09 16:22:12.220855 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2f40] -3178> 2014-06-09 16:22:12.220856 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3177> 2014-06-09 16:22:12.220857 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3176> 2014-06-09 16:22:12.220858 7f8689755700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3175> 2014-06-09 16:22:12.220862 7f868d75d700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3174> 2014-06-09 16:22:12.220866 7f8689f56700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3173> 2014-06-09 16:22:12.220871 7f868ff62700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3172> 2014-06-09 16:22:12.220875 7f868af58700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3171> 2014-06-09 16:22:12.220879 7f868f761700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3170> 2014-06-09 16:22:12.220884 7f868df5e700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3169> 2014-06-09 16:22:12.220888 7f868bf5a700 2 waiting 51 > 50 ops || 5035968 > 104857600 -3168> 2014-06-09 16:22:12.220892 7f868c75b700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3167> 2014-06-09 16:22:12.220897 7f8692f68700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3166> 2014-06-09 16:22:12.220901 7f868b759700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3165> 2014-06-09 16:22:12.220906 7f868e75f700 2 waiting 51 > 50 ops || 5035970 > 104857600 -3164> 2014-06-09 16:22:12.220910 7f868cf5c700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3163> 2014-06-09 16:22:12.220920 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc941e0] -3162> 2014-06-09 16:22:12.220924 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc941e0] -3161> 2014-06-09 16:22:12.220926 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3160> 2014-06-09 16:22:12.220927 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3159> 2014-06-09 16:22:12.220930 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89ec60 (1 active) -3158> 2014-06-09 16:22:12.220933 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d0480 (2 active) -3157> 2014-06-09 16:22:12.220938 7f868a757700 10 op_submit_finish 4263 -3156> 2014-06-09 16:22:12.220943 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -3155> 2014-06-09 16:22:12.220959 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.136007 7 0.007363 -3154> 2014-06-09 16:22:12.220973 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -3153> 2014-06-09 16:22:12.220986 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -3152> 2014-06-09 16:22:12.220998 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000025 0 0.000000 -3151> 2014-06-09 16:22:12.221009 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -3150> 2014-06-09 16:22:12.221019 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery -3149> 2014-06-09 16:22:12.221028 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state -3148> 2014-06-09 16:22:12.221040 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -3147> 2014-06-09 16:22:12.221053 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -3146> 2014-06-09 16:22:12.221064 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info -3145> 2014-06-09 16:22:12.221081 7f868a757700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 -3144> 2014-06-09 16:22:12.221084 7f868a757700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:3.672) v4 -- ?+0 0xeffae00 -3143> 2014-06-09 16:22:12.221099 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1012 -3142> 2014-06-09 16:22:12.221132 7f868a757700 10 log is not dirty -3141> 2014-06-09 16:22:12.221136 7f868a757700 5 queue_transactions existing osr(3.672 0x3ed1350)/0x3ed1350 -3140> 2014-06-09 16:22:12.221138 7f868a757700 2 waiting 51 > 50 ops || 5036003 > 104857600 -3139> 2014-06-09 16:22:12.221143 7f86a5e11700 5 _do_op 0x2fcafc20 seq 4215 osr(0.6e 0x2bd6390)/0x2bd6390 start -3138> 2014-06-09 16:22:12.221147 7f86a5e11700 10 _do_transaction on 0xc3f0300 -3137> 2014-06-09 16:22:12.221154 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3136> 2014-06-09 16:22:12.221197 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3135> 2014-06-09 16:22:12.221213 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4215.0.0 <= header.spos 150443.0.0 -3134> 2014-06-09 16:22:12.221218 7f86a5e11700 5 submit_transaction r = 0 -3133> 2014-06-09 16:22:12.221232 7f86a5e11700 10 _do_op 0x2fcafc20 seq 4215 r = 0, finisher 0x17000080 0 -3132> 2014-06-09 16:22:12.221239 7f86a5e11700 10 _finish_op 0x2fcafc20 seq 4215 osr(0.6e 0x2bd6390)/0x2bd6390 -3131> 2014-06-09 16:22:12.221240 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5c860] -3130> 2014-06-09 16:22:12.221249 7f8690763700 10 op_submit_start 4264 -3129> 2014-06-09 16:22:12.221250 7f8690763700 5 queue_transactions (trailing journal) 4264 -3128> 2014-06-09 16:22:12.221243 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -3127> 2014-06-09 16:22:12.221252 7f8690763700 5 queue_op 0x30d969c0 seq 4264 osr(2.185 0x2ed09a0) 861 bytes (queue has 50 ops and 5035145 bytes) -3126> 2014-06-09 16:22:12.221256 7f8690f64700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3125> 2014-06-09 16:22:12.221253 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -3124> 2014-06-09 16:22:12.221260 7f8692767700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3123> 2014-06-09 16:22:12.221264 7f8691f66700 2 waiting 51 > 50 ops || 5036006 > 104857600 -3122> 2014-06-09 16:22:12.221268 7f8691765700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3121> 2014-06-09 16:22:12.221262 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.6e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 -3120> 2014-06-09 16:22:12.221271 7f8689755700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3119> 2014-06-09 16:22:12.221272 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5c860] -3118> 2014-06-09 16:22:12.221273 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3117> 2014-06-09 16:22:12.221275 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3116> 2014-06-09 16:22:12.221275 7f868d75d700 2 waiting 51 > 50 ops || 5036006 > 104857600 -3115> 2014-06-09 16:22:12.221279 7f8689f56700 2 waiting 51 > 50 ops || 5036006 > 104857600 -3114> 2014-06-09 16:22:12.221283 7f868ff62700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3113> 2014-06-09 16:22:12.221287 7f868af58700 2 waiting 51 > 50 ops || 5036006 > 104857600 -3112> 2014-06-09 16:22:12.221291 7f868f761700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3111> 2014-06-09 16:22:12.221295 7f868df5e700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3110> 2014-06-09 16:22:12.221300 7f868bf5a700 2 waiting 51 > 50 ops || 5035971 > 104857600 -3109> 2014-06-09 16:22:12.221304 7f868c75b700 2 waiting 51 > 50 ops || 5036006 > 104857600 -3108> 2014-06-09 16:22:12.221308 7f8692f68700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3107> 2014-06-09 16:22:12.221312 7f868b759700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3106> 2014-06-09 16:22:12.221316 7f868e75f700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3105> 2014-06-09 16:22:12.221320 7f868cf5c700 2 waiting 51 > 50 ops || 5036006 > 104857600 -3104> 2014-06-09 16:22:12.221324 7f868a757700 2 waiting 51 > 50 ops || 5036006 > 104857600 -3103> 2014-06-09 16:22:12.221329 7f868ef60700 2 waiting 51 > 50 ops || 5035973 > 104857600 -3102> 2014-06-09 16:22:12.221339 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17000080] -3101> 2014-06-09 16:22:12.221348 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17000080] -3100> 2014-06-09 16:22:12.221350 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3099> 2014-06-09 16:22:12.221351 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3098> 2014-06-09 16:22:12.221354 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d0480 (1 active) -3097> 2014-06-09 16:22:12.221357 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114240 (2 active) -3096> 2014-06-09 16:22:12.221362 7f8690763700 10 op_submit_finish 4264 -3095> 2014-06-09 16:22:12.221366 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -3094> 2014-06-09 16:22:12.221380 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.069641 7 0.000337 -3093> 2014-06-09 16:22:12.221392 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -3092> 2014-06-09 16:22:12.221402 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -3091> 2014-06-09 16:22:12.221413 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -3090> 2014-06-09 16:22:12.221423 7f8690763700 5 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -3089> 2014-06-09 16:22:12.221431 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -3088> 2014-06-09 16:22:12.221439 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -3087> 2014-06-09 16:22:12.221450 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -3086> 2014-06-09 16:22:12.221461 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -3085> 2014-06-09 16:22:12.221471 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -3084> 2014-06-09 16:22:12.221484 7f8690763700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 -3083> 2014-06-09 16:22:12.221487 7f8690763700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:2.102) v4 -- ?+0 0x3343d340 -3082> 2014-06-09 16:22:12.221499 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 -3081> 2014-06-09 16:22:12.221532 7f8690763700 10 log is not dirty -3080> 2014-06-09 16:22:12.221537 7f8690763700 5 queue_transactions existing osr(2.102 0x2ed0710)/0x2ed0710 -3079> 2014-06-09 16:22:12.221539 7f8690763700 2 waiting 51 > 50 ops || 5036006 > 104857600 -3078> 2014-06-09 16:22:12.221544 7f86a5e11700 5 _do_op 0xd4715c0 seq 4216 osr(1.b8 0x384c650)/0x384c650 start -3077> 2014-06-09 16:22:12.221548 7f86a5e11700 10 _do_transaction on 0x344d500 -3076> 2014-06-09 16:22:12.221554 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3075> 2014-06-09 16:22:12.221597 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3074> 2014-06-09 16:22:12.221614 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4216.0.0 <= header.spos 150443.0.0 -3073> 2014-06-09 16:22:12.221619 7f86a5e11700 5 submit_transaction r = 0 -3072> 2014-06-09 16:22:12.221634 7f86a5e11700 10 _do_op 0xd4715c0 seq 4216 r = 0, finisher 0x30508dc0 0 -3071> 2014-06-09 16:22:12.221641 7f86a5e11700 10 _finish_op 0xd4715c0 seq 4216 osr(1.b8 0x384c650)/0x384c650 -3070> 2014-06-09 16:22:12.221643 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324a8f60] -3069> 2014-06-09 16:22:12.221651 7f8692767700 10 op_submit_start 4265 -3068> 2014-06-09 16:22:12.221653 7f8692767700 5 queue_transactions (trailing journal) 4265 -3067> 2014-06-09 16:22:12.221646 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -3066> 2014-06-09 16:22:12.221654 7f8692767700 5 queue_op 0xbf7f860 seq 4265 osr(0.179 0x2bd6800) 828 bytes (queue has 50 ops and 5035115 bytes) -3065> 2014-06-09 16:22:12.221658 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 -3064> 2014-06-09 16:22:12.221656 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -3063> 2014-06-09 16:22:12.221662 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3062> 2014-06-09 16:22:12.221666 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3061> 2014-06-09 16:22:12.221670 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 -3060> 2014-06-09 16:22:12.221666 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.b8( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 -3059> 2014-06-09 16:22:12.221673 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 -3058> 2014-06-09 16:22:12.221674 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324a8f60] -3057> 2014-06-09 16:22:12.221676 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -3056> 2014-06-09 16:22:12.221677 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -3055> 2014-06-09 16:22:12.221677 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3054> 2014-06-09 16:22:12.221681 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 -3053> 2014-06-09 16:22:12.221685 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3052> 2014-06-09 16:22:12.221689 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3051> 2014-06-09 16:22:12.221693 7f868bf5a700 2 waiting 51 > 50 ops || 5035941 > 104857600 -3050> 2014-06-09 16:22:12.221698 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 -3049> 2014-06-09 16:22:12.221702 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3048> 2014-06-09 16:22:12.221706 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3047> 2014-06-09 16:22:12.221710 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3046> 2014-06-09 16:22:12.221715 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 -3045> 2014-06-09 16:22:12.221719 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 -3044> 2014-06-09 16:22:12.221724 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3043> 2014-06-09 16:22:12.221728 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 -3042> 2014-06-09 16:22:12.221733 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3041> 2014-06-09 16:22:12.221743 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30508dc0] -3040> 2014-06-09 16:22:12.221748 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30508dc0] -3039> 2014-06-09 16:22:12.221750 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -3038> 2014-06-09 16:22:12.221751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -3037> 2014-06-09 16:22:12.221754 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114240 (1 active) -3036> 2014-06-09 16:22:12.221757 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3497c20 (2 active) -3035> 2014-06-09 16:22:12.221762 7f8692767700 10 op_submit_finish 4265 -3034> 2014-06-09 16:22:12.221766 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -3033> 2014-06-09 16:22:12.221783 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -3032> 2014-06-09 16:22:12.221793 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x1c66b300,0x257c5f00,0x257c5e00,0x270bfe00,0x270bfd00,0x270bfc00,0x270bfb00,0x270bfa00,0x270bf900 -3031> 2014-06-09 16:22:12.221829 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -3030> 2014-06-09 16:22:12.221839 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -3029> 2014-06-09 16:22:12.221848 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -3028> 2014-06-09 16:22:12.221863 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1023 -3027> 2014-06-09 16:22:12.221875 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -3026> 2014-06-09 16:22:12.221884 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.573( v 213'607 (0'0,213'607] local-les=243 n=120 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -3025> 2014-06-09 16:22:12.221895 7f8692767700 10 log is not dirty -3024> 2014-06-09 16:22:12.221899 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.57b( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.57b( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -3023> 2014-06-09 16:22:12.221934 7f8692767700 10 log is not dirty -3022> 2014-06-09 16:22:12.221938 7f8692767700 5 queue_transactions existing osr(0.57b 0x2bd77b0)/0x2bd77b0 -3021> 2014-06-09 16:22:12.221941 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3020> 2014-06-09 16:22:12.221946 7f86a5e11700 5 _do_op 0xbe7de00 seq 4217 osr(1.223 0x384cc70)/0x384cc70 start -3019> 2014-06-09 16:22:12.221950 7f86a5e11700 10 _do_transaction on 0xc1f0f00 -3018> 2014-06-09 16:22:12.221956 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -3017> 2014-06-09 16:22:12.221999 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -3016> 2014-06-09 16:22:12.222016 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4217.0.0 <= header.spos 150443.0.0 -3015> 2014-06-09 16:22:12.222021 7f86a5e11700 5 submit_transaction r = 0 -3014> 2014-06-09 16:22:12.222044 7f86a5e11700 10 _do_op 0xbe7de00 seq 4217 r = 0, finisher 0xc40d740 0 -3013> 2014-06-09 16:22:12.222052 7f86a5e11700 10 _finish_op 0xbe7de00 seq 4217 osr(1.223 0x384cc70)/0x384cc70 -3012> 2014-06-09 16:22:12.222053 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcefe0] -3011> 2014-06-09 16:22:12.222062 7f8691765700 10 op_submit_start 4266 -3010> 2014-06-09 16:22:12.222063 7f8691765700 5 queue_transactions (trailing journal) 4266 -3009> 2014-06-09 16:22:12.222057 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -3008> 2014-06-09 16:22:12.222064 7f8691765700 5 queue_op 0xc88be60 seq 4266 osr(1.54a 0x384daf0) 828 bytes (queue has 50 ops and 5035082 bytes) -3007> 2014-06-09 16:22:12.222068 7f8689755700 2 waiting 51 > 50 ops || 5035910 > 104857600 -3006> 2014-06-09 16:22:12.222066 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -3005> 2014-06-09 16:22:12.222072 7f868d75d700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3004> 2014-06-09 16:22:12.222076 7f8689f56700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3003> 2014-06-09 16:22:12.222079 7f868ff62700 2 waiting 51 > 50 ops || 5035910 > 104857600 -3002> 2014-06-09 16:22:12.222075 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.223( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:154 -3001> 2014-06-09 16:22:12.222083 7f868af58700 2 waiting 51 > 50 ops || 5035943 > 104857600 -3000> 2014-06-09 16:22:12.222085 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcefe0] -2999> 2014-06-09 16:22:12.222087 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2998> 2014-06-09 16:22:12.222087 7f868f761700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2997> 2014-06-09 16:22:12.222088 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2996> 2014-06-09 16:22:12.222090 7f868df5e700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2995> 2014-06-09 16:22:12.222094 7f868bf5a700 2 waiting 51 > 50 ops || 5035908 > 104857600 -2994> 2014-06-09 16:22:12.222098 7f868c75b700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2993> 2014-06-09 16:22:12.222103 7f8692f68700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2992> 2014-06-09 16:22:12.222107 7f868b759700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2991> 2014-06-09 16:22:12.222112 7f868e75f700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2990> 2014-06-09 16:22:12.222116 7f868cf5c700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2989> 2014-06-09 16:22:12.222120 7f868a757700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2988> 2014-06-09 16:22:12.222125 7f868ef60700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2987> 2014-06-09 16:22:12.222129 7f8690763700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2986> 2014-06-09 16:22:12.222133 7f8690f64700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2985> 2014-06-09 16:22:12.222138 7f8692767700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2984> 2014-06-09 16:22:12.222142 7f8691f66700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2983> 2014-06-09 16:22:12.222152 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc40d740] -2982> 2014-06-09 16:22:12.222159 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc40d740] -2981> 2014-06-09 16:22:12.222160 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2980> 2014-06-09 16:22:12.222161 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2979> 2014-06-09 16:22:12.222165 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3497c20 (1 active) -2978> 2014-06-09 16:22:12.222168 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1440 (2 active) -2977> 2014-06-09 16:22:12.222172 7f8691765700 10 op_submit_finish 4266 -2976> 2014-06-09 16:22:12.222176 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 243/206 242/242/205) -2975> 2014-06-09 16:22:12.222212 7f8691765700 10 log is not dirty -2974> 2014-06-09 16:22:12.222217 7f8691765700 5 queue_transactions existing osr(3.48d 0x3ed14e0)/0x3ed14e0 -2973> 2014-06-09 16:22:12.222220 7f8691765700 2 waiting 51 > 50 ops || 5035910 > 104857600 -2972> 2014-06-09 16:22:12.222224 7f86a5e11700 5 _do_op 0x16fd4780 seq 4218 osr(1.233 0x384cc30)/0x384cc30 start -2971> 2014-06-09 16:22:12.222228 7f86a5e11700 10 _do_transaction on 0xc28be00 -2970> 2014-06-09 16:22:12.222233 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2969> 2014-06-09 16:22:12.222275 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2968> 2014-06-09 16:22:12.222290 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4218.0.0 <= header.spos 150443.0.0 -2967> 2014-06-09 16:22:12.222294 7f86a5e11700 5 submit_transaction r = 0 -2966> 2014-06-09 16:22:12.222305 7f86a5e11700 10 _do_op 0x16fd4780 seq 4218 r = 0, finisher 0x3251ca20 0 -2965> 2014-06-09 16:22:12.222311 7f86a5e11700 10 _finish_op 0x16fd4780 seq 4218 osr(1.233 0x384cc30)/0x384cc30 -2964> 2014-06-09 16:22:12.222313 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324f9d60] -2963> 2014-06-09 16:22:12.222316 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324f9d60] -2962> 2014-06-09 16:22:12.222317 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2961> 2014-06-09 16:22:12.222318 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2960> 2014-06-09 16:22:12.222322 7f868d75d700 10 op_submit_start 4267 -2959> 2014-06-09 16:22:12.222323 7f868d75d700 5 queue_transactions (trailing journal) 4267 -2958> 2014-06-09 16:22:12.222325 7f868d75d700 5 queue_op 0x14dd6060 seq 4267 osr(1.301 0x384d030) 861 bytes (queue has 50 ops and 5035115 bytes) -2957> 2014-06-09 16:22:12.222330 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2956> 2014-06-09 16:22:12.222335 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2955> 2014-06-09 16:22:12.222339 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2954> 2014-06-09 16:22:12.222344 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2953> 2014-06-09 16:22:12.222348 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2952> 2014-06-09 16:22:12.222353 7f868bf5a700 2 waiting 51 > 50 ops || 5035941 > 104857600 -2951> 2014-06-09 16:22:12.222356 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2950> 2014-06-09 16:22:12.222361 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2949> 2014-06-09 16:22:12.222365 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2948> 2014-06-09 16:22:12.222369 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2947> 2014-06-09 16:22:12.222374 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2946> 2014-06-09 16:22:12.222378 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2945> 2014-06-09 16:22:12.222382 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2944> 2014-06-09 16:22:12.222387 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2943> 2014-06-09 16:22:12.222391 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2942> 2014-06-09 16:22:12.222395 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2941> 2014-06-09 16:22:12.222400 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2940> 2014-06-09 16:22:12.222404 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2939> 2014-06-09 16:22:12.222408 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2938> 2014-06-09 16:22:12.222418 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251ca20] -2937> 2014-06-09 16:22:12.222422 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251ca20] -2936> 2014-06-09 16:22:12.222424 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2935> 2014-06-09 16:22:12.222425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2934> 2014-06-09 16:22:12.222428 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1440 (1 active) -2933> 2014-06-09 16:22:12.222431 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114360 (2 active) -2932> 2014-06-09 16:22:12.222435 7f868d75d700 10 op_submit_finish 4267 -2931> 2014-06-09 16:22:12.222439 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -2930> 2014-06-09 16:22:12.222453 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local -2929> 2014-06-09 16:22:12.222462 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -2928> 2014-06-09 16:22:12.222470 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -2927> 2014-06-09 16:22:12.222479 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -2926> 2014-06-09 16:22:12.222487 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -2925> 2014-06-09 16:22:12.222501 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:129 -2924> 2014-06-09 16:22:12.222510 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -2923> 2014-06-09 16:22:12.222519 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.b6( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop -2922> 2014-06-09 16:22:12.222528 7f868d75d700 10 log is not dirty -2921> 2014-06-09 16:22:12.222533 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -2920> 2014-06-09 16:22:12.222545 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.118381 7 0.000369 -2919> 2014-06-09 16:22:12.222557 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -2918> 2014-06-09 16:22:12.222567 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -2917> 2014-06-09 16:22:12.222577 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -2916> 2014-06-09 16:22:12.222587 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -2915> 2014-06-09 16:22:12.222596 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -2914> 2014-06-09 16:22:12.222603 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -2913> 2014-06-09 16:22:12.222614 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -2912> 2014-06-09 16:22:12.222625 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -2911> 2014-06-09 16:22:12.222634 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -2910> 2014-06-09 16:22:12.222710 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -2909> 2014-06-09 16:22:12.222713 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:0.629) v4 -- ?+0 0x1abd5c00 -2908> 2014-06-09 16:22:12.222727 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 -2907> 2014-06-09 16:22:12.222760 7f868d75d700 10 log is not dirty -2906> 2014-06-09 16:22:12.222764 7f868d75d700 5 queue_transactions existing osr(0.629 0x2bd7c90)/0x2bd7c90 -2905> 2014-06-09 16:22:12.222766 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2904> 2014-06-09 16:22:12.222794 7f86a5e11700 5 _do_op 0x324e7d40 seq 4219 osr(0.b9 0x2bd6400)/0x2bd6400 start -2903> 2014-06-09 16:22:12.222799 7f86a5e11700 10 _do_transaction on 0x480d080 -2902> 2014-06-09 16:22:12.222805 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2901> 2014-06-09 16:22:12.222847 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2900> 2014-06-09 16:22:12.222862 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4219.0.0 <= header.spos 150443.0.0 -2899> 2014-06-09 16:22:12.222866 7f86a5e11700 5 submit_transaction r = 0 -2898> 2014-06-09 16:22:12.222878 7f86a5e11700 10 _do_op 0x324e7d40 seq 4219 r = 0, finisher 0xbcc9b60 0 -2897> 2014-06-09 16:22:12.222889 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5e2c0] -2896> 2014-06-09 16:22:12.222891 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5e2c0] -2895> 2014-06-09 16:22:12.222893 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2894> 2014-06-09 16:22:12.222894 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2893> 2014-06-09 16:22:12.222897 7f86a5e11700 10 _finish_op 0x324e7d40 seq 4219 osr(0.b9 0x2bd6400)/0x2bd6400 -2892> 2014-06-09 16:22:12.222908 7f868ff62700 10 op_submit_start 4268 -2891> 2014-06-09 16:22:12.222910 7f868ff62700 5 queue_transactions (trailing journal) 4268 -2890> 2014-06-09 16:22:12.222911 7f868ff62700 5 queue_op 0xd235c20 seq 4268 osr(1.504 0x384dbd0) 828 bytes (queue has 50 ops and 5035117 bytes) -2889> 2014-06-09 16:22:12.222917 7f868af58700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2888> 2014-06-09 16:22:12.222921 7f868f761700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2887> 2014-06-09 16:22:12.222925 7f868df5e700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2886> 2014-06-09 16:22:12.222930 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2885> 2014-06-09 16:22:12.222934 7f868c75b700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2884> 2014-06-09 16:22:12.222938 7f8692f68700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2883> 2014-06-09 16:22:12.222942 7f868b759700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2882> 2014-06-09 16:22:12.222946 7f868e75f700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2881> 2014-06-09 16:22:12.222950 7f868cf5c700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2880> 2014-06-09 16:22:12.222955 7f868a757700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2879> 2014-06-09 16:22:12.222960 7f868ef60700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2878> 2014-06-09 16:22:12.222964 7f8690763700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2877> 2014-06-09 16:22:12.222968 7f8690f64700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2876> 2014-06-09 16:22:12.222973 7f8692767700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2875> 2014-06-09 16:22:12.222977 7f8691f66700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2874> 2014-06-09 16:22:12.222981 7f8691765700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2873> 2014-06-09 16:22:12.222986 7f8689755700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2872> 2014-06-09 16:22:12.222990 7f868d75d700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2871> 2014-06-09 16:22:12.222994 7f8689f56700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2870> 2014-06-09 16:22:12.223004 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbcc9b60] -2869> 2014-06-09 16:22:12.223010 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbcc9b60] -2868> 2014-06-09 16:22:12.223012 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2867> 2014-06-09 16:22:12.223013 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2866> 2014-06-09 16:22:12.223016 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114360 (1 active) -2865> 2014-06-09 16:22:12.223019 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6900 (2 active) -2864> 2014-06-09 16:22:12.223024 7f868ff62700 10 op_submit_finish 4268 -2863> 2014-06-09 16:22:12.223028 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.1a0( v 213'567 (0'0,213'567] local-les=243 n=114 ec=118 les/c 243/201 242/242/200) -2862> 2014-06-09 16:22:12.223063 7f868ff62700 10 log is not dirty -2861> 2014-06-09 16:22:12.223069 7f868ff62700 5 queue_transactions existing osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 -2860> 2014-06-09 16:22:12.223071 7f868ff62700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2859> 2014-06-09 16:22:12.223076 7f86a5e11700 5 _do_op 0x230b9e60 seq 4220 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 start -2858> 2014-06-09 16:22:12.223080 7f86a5e11700 10 _do_transaction on 0xbfa5740 -2857> 2014-06-09 16:22:12.223085 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2856> 2014-06-09 16:22:12.223126 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2855> 2014-06-09 16:22:12.223140 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4220.0.0 <= header.spos 150443.0.0 -2854> 2014-06-09 16:22:12.223144 7f86a5e11700 5 submit_transaction r = 0 -2853> 2014-06-09 16:22:12.223158 7f86a5e11700 10 _do_op 0x230b9e60 seq 4220 r = 0, finisher 0x3281b100 0 -2852> 2014-06-09 16:22:12.223168 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd78440] -2851> 2014-06-09 16:22:12.223171 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd78440] -2850> 2014-06-09 16:22:12.223172 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2849> 2014-06-09 16:22:12.223173 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2848> 2014-06-09 16:22:12.223177 7f86a5e11700 10 _finish_op 0x230b9e60 seq 4220 osr(3.2d3 0x3ed0eb0)/0x3ed0eb0 -2847> 2014-06-09 16:22:12.223188 7f868f761700 10 op_submit_start 4269 -2846> 2014-06-09 16:22:12.223189 7f868f761700 5 queue_transactions (trailing journal) 4269 -2845> 2014-06-09 16:22:12.223191 7f868f761700 5 queue_op 0x324f40c0 seq 4269 osr(3.111 0x3ed0830) 828 bytes (queue has 50 ops and 5035117 bytes) -2844> 2014-06-09 16:22:12.223196 7f868df5e700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2843> 2014-06-09 16:22:12.223201 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2842> 2014-06-09 16:22:12.223205 7f868c75b700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2841> 2014-06-09 16:22:12.223210 7f8692f68700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2840> 2014-06-09 16:22:12.223214 7f868b759700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2839> 2014-06-09 16:22:12.223218 7f868e75f700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2838> 2014-06-09 16:22:12.223222 7f868cf5c700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2837> 2014-06-09 16:22:12.223227 7f868a757700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2836> 2014-06-09 16:22:12.223231 7f868ef60700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2835> 2014-06-09 16:22:12.223235 7f8690763700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2834> 2014-06-09 16:22:12.223240 7f8690f64700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2833> 2014-06-09 16:22:12.223243 7f8692767700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2832> 2014-06-09 16:22:12.223248 7f8691f66700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2831> 2014-06-09 16:22:12.223252 7f8691765700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2830> 2014-06-09 16:22:12.223256 7f8689755700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2829> 2014-06-09 16:22:12.223261 7f868d75d700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2828> 2014-06-09 16:22:12.223265 7f8689f56700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2827> 2014-06-09 16:22:12.223269 7f868ff62700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2826> 2014-06-09 16:22:12.223274 7f868af58700 2 waiting 51 > 50 ops || 5035978 > 104857600 -2825> 2014-06-09 16:22:12.223283 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3281b100] -2824> 2014-06-09 16:22:12.223291 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3281b100] -2823> 2014-06-09 16:22:12.223293 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2822> 2014-06-09 16:22:12.223294 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2821> 2014-06-09 16:22:12.223297 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6900 (1 active) -2820> 2014-06-09 16:22:12.223300 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f440 (2 active) -2819> 2014-06-09 16:22:12.223305 7f868f761700 10 op_submit_finish 4269 -2818> 2014-06-09 16:22:12.223308 7f868f761700 10 osd.26 pg_epoch: 243 pg[1.302( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.302( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -2817> 2014-06-09 16:22:12.223343 7f868f761700 10 log is not dirty -2816> 2014-06-09 16:22:12.223348 7f868f761700 5 queue_transactions existing osr(1.302 0x384d010)/0x384d010 -2815> 2014-06-09 16:22:12.223351 7f868f761700 2 waiting 51 > 50 ops || 5035945 > 104857600 -2814> 2014-06-09 16:22:12.223355 7f86a5e11700 5 _do_op 0xd327c20 seq 4221 osr(1.508 0x384db90)/0x384db90 start -2813> 2014-06-09 16:22:12.223359 7f86a5e11700 10 _do_transaction on 0xbf2e840 -2812> 2014-06-09 16:22:12.223364 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2811> 2014-06-09 16:22:12.223405 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2810> 2014-06-09 16:22:12.223419 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4221.0.0 <= header.spos 150443.0.0 -2809> 2014-06-09 16:22:12.223424 7f86a5e11700 5 submit_transaction r = 0 -2808> 2014-06-09 16:22:12.223434 7f86a5e11700 10 _do_op 0xd327c20 seq 4221 r = 0, finisher 0x30d61b20 0 -2807> 2014-06-09 16:22:12.223445 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc4117a0] -2806> 2014-06-09 16:22:12.223447 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc4117a0] -2805> 2014-06-09 16:22:12.223449 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2804> 2014-06-09 16:22:12.223450 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2803> 2014-06-09 16:22:12.223453 7f86a5e11700 10 _finish_op 0xd327c20 seq 4221 osr(1.508 0x384db90)/0x384db90 -2802> 2014-06-09 16:22:12.223465 7f868bf5a700 10 op_submit_start 4270 -2801> 2014-06-09 16:22:12.223466 7f868bf5a700 5 queue_transactions (trailing journal) 4270 -2800> 2014-06-09 16:22:12.223468 7f868bf5a700 5 queue_op 0x324e73e0 seq 4270 osr(3.9f 0x3ed0600) 826 bytes (queue has 50 ops and 5035115 bytes) -2799> 2014-06-09 16:22:12.223473 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2798> 2014-06-09 16:22:12.223478 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2797> 2014-06-09 16:22:12.223482 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2796> 2014-06-09 16:22:12.223486 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2795> 2014-06-09 16:22:12.223491 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2794> 2014-06-09 16:22:12.223495 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2793> 2014-06-09 16:22:12.223499 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2792> 2014-06-09 16:22:12.223504 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2791> 2014-06-09 16:22:12.223508 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2790> 2014-06-09 16:22:12.223512 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2789> 2014-06-09 16:22:12.223517 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2788> 2014-06-09 16:22:12.223521 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2787> 2014-06-09 16:22:12.223525 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2786> 2014-06-09 16:22:12.223530 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2785> 2014-06-09 16:22:12.223534 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2784> 2014-06-09 16:22:12.223538 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2783> 2014-06-09 16:22:12.223543 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2782> 2014-06-09 16:22:12.223547 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2781> 2014-06-09 16:22:12.223552 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2780> 2014-06-09 16:22:12.223561 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30d61b20] -2779> 2014-06-09 16:22:12.223566 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30d61b20] -2778> 2014-06-09 16:22:12.223568 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2777> 2014-06-09 16:22:12.223569 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2776> 2014-06-09 16:22:12.223572 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f440 (1 active) -2775> 2014-06-09 16:22:12.223575 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc780120 (2 active) -2774> 2014-06-09 16:22:12.223580 7f868bf5a700 10 op_submit_finish 4270 -2773> 2014-06-09 16:22:12.223584 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[1.465( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 1.465( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -2772> 2014-06-09 16:22:12.223625 7f868bf5a700 10 log is not dirty -2771> 2014-06-09 16:22:12.223630 7f868bf5a700 5 queue_transactions existing osr(1.465 0x384d650)/0x384d650 -2770> 2014-06-09 16:22:12.223633 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2769> 2014-06-09 16:22:12.223637 7f86a5e11700 5 _do_op 0x324f4a80 seq 4222 osr(2.5f2 0x2ed1df0)/0x2ed1df0 start -2768> 2014-06-09 16:22:12.223641 7f86a5e11700 10 _do_transaction on 0xc22a180 -2767> 2014-06-09 16:22:12.223647 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2766> 2014-06-09 16:22:12.223688 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2765> 2014-06-09 16:22:12.223703 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4222.0.0 <= header.spos 150443.0.0 -2764> 2014-06-09 16:22:12.223707 7f86a5e11700 5 submit_transaction r = 0 -2763> 2014-06-09 16:22:12.223730 7f86a5e11700 10 _do_op 0x324f4a80 seq 4222 r = 0, finisher 0x307f18a0 0 -2762> 2014-06-09 16:22:12.223741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31456ce0] -2761> 2014-06-09 16:22:12.223743 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31456ce0] -2760> 2014-06-09 16:22:12.223747 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2759> 2014-06-09 16:22:12.223749 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2758> 2014-06-09 16:22:12.223752 7f86a5e11700 10 _finish_op 0x324f4a80 seq 4222 osr(2.5f2 0x2ed1df0)/0x2ed1df0 -2757> 2014-06-09 16:22:12.223763 7f8692f68700 10 op_submit_start 4271 -2756> 2014-06-09 16:22:12.223765 7f8692f68700 5 queue_transactions (trailing journal) 4271 -2755> 2014-06-09 16:22:12.223766 7f8692f68700 5 queue_op 0x1e08b9e0 seq 4271 osr(0.1da 0x2bd6920) 828 bytes (queue has 50 ops and 5035115 bytes) -2754> 2014-06-09 16:22:12.223771 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2753> 2014-06-09 16:22:12.223776 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2752> 2014-06-09 16:22:12.223780 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2751> 2014-06-09 16:22:12.223785 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2750> 2014-06-09 16:22:12.223789 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2749> 2014-06-09 16:22:12.223793 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2748> 2014-06-09 16:22:12.223797 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2747> 2014-06-09 16:22:12.223802 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2746> 2014-06-09 16:22:12.223806 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2745> 2014-06-09 16:22:12.223810 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2744> 2014-06-09 16:22:12.223815 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2743> 2014-06-09 16:22:12.223819 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2742> 2014-06-09 16:22:12.223823 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2741> 2014-06-09 16:22:12.223827 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2740> 2014-06-09 16:22:12.223831 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2739> 2014-06-09 16:22:12.223836 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2738> 2014-06-09 16:22:12.223840 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2737> 2014-06-09 16:22:12.223844 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2736> 2014-06-09 16:22:12.223848 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2735> 2014-06-09 16:22:12.223858 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f18a0] -2734> 2014-06-09 16:22:12.223862 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f18a0] -2733> 2014-06-09 16:22:12.223863 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2732> 2014-06-09 16:22:12.223865 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2731> 2014-06-09 16:22:12.223867 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc780120 (1 active) -2730> 2014-06-09 16:22:12.223870 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524b40 (2 active) -2729> 2014-06-09 16:22:12.223875 7f8692f68700 10 op_submit_finish 4271 -2728> 2014-06-09 16:22:12.223879 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.4af( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.4af( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -2727> 2014-06-09 16:22:12.223914 7f8692f68700 10 log is not dirty -2726> 2014-06-09 16:22:12.223919 7f8692f68700 5 queue_transactions existing osr(2.4af 0x2ed1790)/0x2ed1790 -2725> 2014-06-09 16:22:12.223922 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2724> 2014-06-09 16:22:12.223926 7f86a5e11700 5 _do_op 0xcabc000 seq 4223 osr(1.2c0 0x384d170)/0x384d170 start -2723> 2014-06-09 16:22:12.223930 7f86a5e11700 10 _do_transaction on 0xbf7ccc0 -2722> 2014-06-09 16:22:12.223936 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2721> 2014-06-09 16:22:12.223977 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2720> 2014-06-09 16:22:12.223991 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4223.0.0 <= header.spos 150443.0.0 -2719> 2014-06-09 16:22:12.223996 7f86a5e11700 5 submit_transaction r = 0 -2718> 2014-06-09 16:22:12.224007 7f86a5e11700 10 _do_op 0xcabc000 seq 4223 r = 0, finisher 0x3006da40 0 -2717> 2014-06-09 16:22:12.224018 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1b0c0e60] -2716> 2014-06-09 16:22:12.224020 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1b0c0e60] -2715> 2014-06-09 16:22:12.224021 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2714> 2014-06-09 16:22:12.224023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2713> 2014-06-09 16:22:12.224026 7f86a5e11700 10 _finish_op 0xcabc000 seq 4223 osr(1.2c0 0x384d170)/0x384d170 -2712> 2014-06-09 16:22:12.224036 7f868e75f700 10 op_submit_start 4272 -2711> 2014-06-09 16:22:12.224038 7f868e75f700 5 queue_transactions (trailing journal) 4272 -2710> 2014-06-09 16:22:12.224039 7f868e75f700 5 queue_op 0x324f50e0 seq 4272 osr(0.640 0x2bd7c10) 828 bytes (queue has 50 ops and 5035115 bytes) -2709> 2014-06-09 16:22:12.224044 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2708> 2014-06-09 16:22:12.224049 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2707> 2014-06-09 16:22:12.224053 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2706> 2014-06-09 16:22:12.224058 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2705> 2014-06-09 16:22:12.224062 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2704> 2014-06-09 16:22:12.224066 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2703> 2014-06-09 16:22:12.224071 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2702> 2014-06-09 16:22:12.224075 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2701> 2014-06-09 16:22:12.224080 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2700> 2014-06-09 16:22:12.224084 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2699> 2014-06-09 16:22:12.224088 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2698> 2014-06-09 16:22:12.224092 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2697> 2014-06-09 16:22:12.224096 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2696> 2014-06-09 16:22:12.224100 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2695> 2014-06-09 16:22:12.224104 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2694> 2014-06-09 16:22:12.224108 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2693> 2014-06-09 16:22:12.224113 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2692> 2014-06-09 16:22:12.224117 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2691> 2014-06-09 16:22:12.224121 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2690> 2014-06-09 16:22:12.224130 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006da40] -2689> 2014-06-09 16:22:12.224134 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006da40] -2688> 2014-06-09 16:22:12.224136 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2687> 2014-06-09 16:22:12.224137 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2686> 2014-06-09 16:22:12.224140 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524b40 (1 active) -2685> 2014-06-09 16:22:12.224143 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x34966c0 (2 active) -2684> 2014-06-09 16:22:12.224148 7f868e75f700 10 op_submit_finish 4272 -2683> 2014-06-09 16:22:12.224152 7f868e75f700 10 osd.26 pg_epoch: 243 pg[0.530( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.530( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) -2682> 2014-06-09 16:22:12.224185 7f868e75f700 10 log is not dirty -2681> 2014-06-09 16:22:12.225349 7f868e75f700 5 queue_transactions existing osr(0.530 0x2bd78f0)/0x2bd78f0 -2680> 2014-06-09 16:22:12.225353 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2679> 2014-06-09 16:22:12.224189 7f86a5e11700 5 _do_op 0xc34bc80 seq 4224 osr(0.686 0x2bd7e20)/0x2bd7e20 start -2678> 2014-06-09 16:22:12.225359 7f86a5e11700 10 _do_transaction on 0xc321680 -2677> 2014-06-09 16:22:12.225367 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2676> 2014-06-09 16:22:12.225414 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2675> 2014-06-09 16:22:12.225431 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4224.0.0 <= header.spos 150443.0.0 -2674> 2014-06-09 16:22:12.225436 7f86a5e11700 5 submit_transaction r = 0 -2673> 2014-06-09 16:22:12.225449 7f86a5e11700 10 _do_op 0xc34bc80 seq 4224 r = 0, finisher 0x19941cc0 0 -2672> 2014-06-09 16:22:12.225460 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x12050240] -2671> 2014-06-09 16:22:12.225463 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -2670> 2014-06-09 16:22:12.225474 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -2669> 2014-06-09 16:22:12.225484 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.686( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 -2668> 2014-06-09 16:22:12.225494 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x12050240] -2667> 2014-06-09 16:22:12.225496 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2666> 2014-06-09 16:22:12.225497 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2665> 2014-06-09 16:22:12.225501 7f86a5e11700 10 _finish_op 0xc34bc80 seq 4224 osr(0.686 0x2bd7e20)/0x2bd7e20 -2664> 2014-06-09 16:22:12.225513 7f868a757700 10 op_submit_start 4273 -2663> 2014-06-09 16:22:12.225515 7f868a757700 5 queue_transactions (trailing journal) 4273 -2662> 2014-06-09 16:22:12.225516 7f868a757700 5 queue_op 0xc2d48a0 seq 4273 osr(3.672 0x3ed1350) 861 bytes (queue has 50 ops and 5035115 bytes) -2661> 2014-06-09 16:22:12.225522 7f868ef60700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2660> 2014-06-09 16:22:12.225527 7f8690763700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2659> 2014-06-09 16:22:12.225531 7f8690f64700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2658> 2014-06-09 16:22:12.225535 7f8692767700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2657> 2014-06-09 16:22:12.225540 7f8691f66700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2656> 2014-06-09 16:22:12.225544 7f8691765700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2655> 2014-06-09 16:22:12.225548 7f8689755700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2654> 2014-06-09 16:22:12.225553 7f868d75d700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2653> 2014-06-09 16:22:12.225557 7f8689f56700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2652> 2014-06-09 16:22:12.225562 7f868ff62700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2651> 2014-06-09 16:22:12.225566 7f868af58700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2650> 2014-06-09 16:22:12.225570 7f868f761700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2649> 2014-06-09 16:22:12.225575 7f868df5e700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2648> 2014-06-09 16:22:12.225579 7f868bf5a700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2647> 2014-06-09 16:22:12.225583 7f868c75b700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2646> 2014-06-09 16:22:12.225588 7f8692f68700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2645> 2014-06-09 16:22:12.225593 7f868b759700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2644> 2014-06-09 16:22:12.225597 7f868e75f700 2 waiting 51 > 50 ops || 5035943 > 104857600 -2643> 2014-06-09 16:22:12.225601 7f868cf5c700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2642> 2014-06-09 16:22:12.225611 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x19941cc0] -2641> 2014-06-09 16:22:12.225615 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x19941cc0] -2640> 2014-06-09 16:22:12.225617 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2639> 2014-06-09 16:22:12.225618 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2638> 2014-06-09 16:22:12.225621 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x34966c0 (1 active) -2637> 2014-06-09 16:22:12.225624 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc240 (2 active) -2636> 2014-06-09 16:22:12.225630 7f868a757700 10 op_submit_finish 4273 -2635> 2014-06-09 16:22:12.225634 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -2634> 2014-06-09 16:22:12.225649 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.084456 7 0.000348 -2633> 2014-06-09 16:22:12.225661 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -2632> 2014-06-09 16:22:12.225672 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -2631> 2014-06-09 16:22:12.225684 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -2630> 2014-06-09 16:22:12.225694 7f868a757700 5 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -2629> 2014-06-09 16:22:12.225703 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -2628> 2014-06-09 16:22:12.225711 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -2627> 2014-06-09 16:22:12.225721 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -2626> 2014-06-09 16:22:12.225734 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -2625> 2014-06-09 16:22:12.225743 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -2624> 2014-06-09 16:22:12.225758 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 -2623> 2014-06-09 16:22:12.225762 7f868a757700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:2.523) v4 -- ?+0 0x30c69500 -2622> 2014-06-09 16:22:12.225777 7f868a757700 15 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 -2621> 2014-06-09 16:22:12.225812 7f868a757700 10 log is not dirty -2620> 2014-06-09 16:22:12.225816 7f868a757700 5 queue_transactions existing osr(2.523 0x2ed1610)/0x2ed1610 -2619> 2014-06-09 16:22:12.225819 7f868a757700 2 waiting 51 > 50 ops || 5035976 > 104857600 -2618> 2014-06-09 16:22:12.225854 7f86a5e11700 5 _do_op 0x32539980 seq 4225 osr(3.56a 0x3ed1900)/0x3ed1900 start -2617> 2014-06-09 16:22:12.225863 7f86a5e11700 10 _do_transaction on 0x390c540 -2616> 2014-06-09 16:22:12.225868 7f86a5e11700 15 _omap_setkeys meta/1fe1045b/pglog_3.56a/0//-1 -2615> 2014-06-09 16:22:12.225916 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe1045b/pglog_3.56a/0//-1 -2614> 2014-06-09 16:22:12.225932 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2613> 2014-06-09 16:22:12.225973 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2612> 2014-06-09 16:22:12.225988 7f86a5e11700 15 _omap_rmkeys meta/1fe1045b/pglog_3.56a/0//-1 -2611> 2014-06-09 16:22:12.225995 7f86a5e11700 10 _omap_rmkeys meta/1fe1045b/pglog_3.56a/0//-1 r = 0 -2610> 2014-06-09 16:22:12.226001 7f86a5e11700 15 _omap_setkeys meta/1fe1045b/pglog_3.56a/0//-1 -2609> 2014-06-09 16:22:12.226014 7f86a5e11700 15 _touch 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -2608> 2014-06-09 16:22:12.226061 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.56a_head oid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -2607> 2014-06-09 16:22:12.226093 7f86a5e11700 20 write_state seq is 30173 -2606> 2014-06-09 16:22:12.226101 7f86a5e11700 20 set_header setting 30172 cid 3.56a_head oid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 parent seq 0 -2605> 2014-06-09 16:22:12.226114 7f86a5e11700 10 _touch 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 = 0 -2604> 2014-06-09 16:22:12.226121 7f86a5e11700 15 _write 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 0~344064 -2603> 2014-06-09 16:22:12.226133 7f86a5e11700 10 file_to_extents done -2602> 2014-06-09 16:22:12.227598 7f86a5e11700 10 _generic_write 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 0~344064 = 0 -2601> 2014-06-09 16:22:12.227649 7f86a5e11700 15 _setattrs 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -2600> 2014-06-09 16:22:12.227662 7f86a5e11700 10 _setattrs 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 = 0 -2599> 2014-06-09 16:22:12.227669 7f86a5e11700 15 _setattrs 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 -2598> 2014-06-09 16:22:12.227678 7f86a5e11700 10 _setattrs 3.56a_head/888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 = 0 -2597> 2014-06-09 16:22:12.227739 7f86a5e11700 10 cid: oid: 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 not skipping op, *spos 4225.0.0 -2596> 2014-06-09 16:22:12.227743 7f86a5e11700 10 > header.spos 0.0.0 -2595> 2014-06-09 16:22:12.227757 7f86a5e11700 20 set_header setting 30172 cid 3.56a_head oid 888f0d6a/rbd_data.117a6b8b4567.00000000000016da/head//3 parent seq 0 -2594> 2014-06-09 16:22:12.227800 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4225.0.0 <= header.spos 150443.0.0 -2593> 2014-06-09 16:22:12.227806 7f86a5e11700 10 cid: oid: 1fe1045b/pglog_3.56a/0//-1 skipping op, spos 4225.0.0 <= header.spos 150148.0.0 -2592> 2014-06-09 16:22:12.227809 7f86a5e11700 5 submit_transaction r = 0 -2591> 2014-06-09 16:22:12.229132 7f86a5e11700 10 _do_op 0x32539980 seq 4225 r = 0, finisher 0x30e3eb00 0x6c8dd40 -2590> 2014-06-09 16:22:12.229147 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078b00] -2589> 2014-06-09 16:22:12.229150 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 31 -2588> 2014-06-09 16:22:12.229175 7f86a460e700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.229167, event: op_commit, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) -2587> 2014-06-09 16:22:12.229184 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078b00] -2586> 2014-06-09 16:22:12.229186 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2585> 2014-06-09 16:22:12.229187 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2584> 2014-06-09 16:22:12.229192 7f86a5e11700 10 _finish_op 0x32539980 seq 4225 osr(3.56a 0x3ed1900)/0x3ed1900 -2583> 2014-06-09 16:22:12.229207 7f8690763700 10 op_submit_start 4274 -2582> 2014-06-09 16:22:12.229209 7f8690763700 5 queue_transactions (trailing journal) 4274 -2581> 2014-06-09 16:22:12.229211 7f8690763700 5 queue_op 0x16fd4900 seq 4274 osr(2.102 0x2ed0710) 861 bytes (queue has 50 ops and 4689514 bytes) -2580> 2014-06-09 16:22:12.229217 7f8690f64700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2579> 2014-06-09 16:22:12.229223 7f8692767700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2578> 2014-06-09 16:22:12.229228 7f8691f66700 2 waiting 51 > 50 ops || 4690375 > 104857600 -2577> 2014-06-09 16:22:12.229234 7f8691765700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2576> 2014-06-09 16:22:12.229240 7f8689755700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2575> 2014-06-09 16:22:12.229245 7f868d75d700 2 waiting 51 > 50 ops || 4690375 > 104857600 -2574> 2014-06-09 16:22:12.229251 7f8689f56700 2 waiting 51 > 50 ops || 4690375 > 104857600 -2573> 2014-06-09 16:22:12.229257 7f868ff62700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2572> 2014-06-09 16:22:12.229262 7f868af58700 2 waiting 51 > 50 ops || 4690375 > 104857600 -2571> 2014-06-09 16:22:12.229268 7f868f761700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2570> 2014-06-09 16:22:12.229273 7f868df5e700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2569> 2014-06-09 16:22:12.229279 7f868bf5a700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2568> 2014-06-09 16:22:12.229284 7f868c75b700 2 waiting 51 > 50 ops || 4690375 > 104857600 -2567> 2014-06-09 16:22:12.229289 7f8692f68700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2566> 2014-06-09 16:22:12.229295 7f868b759700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2565> 2014-06-09 16:22:12.229300 7f868e75f700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2564> 2014-06-09 16:22:12.229306 7f868cf5c700 2 waiting 51 > 50 ops || 4690375 > 104857600 -2563> 2014-06-09 16:22:12.229312 7f868a757700 2 waiting 51 > 50 ops || 4690375 > 104857600 -2562> 2014-06-09 16:22:12.229316 7f868ef60700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2561> 2014-06-09 16:22:12.229329 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30e3eb00] -2560> 2014-06-09 16:22:12.229332 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 31 -2559> 2014-06-09 16:22:12.229351 7f86a4e0f700 5 -- op tracker -- , seq: 607, time: 2014-06-09 16:22:12.229345, event: op_applied, op: osd_op(client.4570.0:34004 rbd_data.117a6b8b4567.00000000000016da [set-alloc-hint object_size 4194304 write_size 4194304,write 0~344064] 3.888f0d6a RETRY=3 ack+ondisk+retry+write e242) -2558> 2014-06-09 16:22:12.229359 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.56a( v 243'576 (0'0,243'576] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'575 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'576 -2557> 2014-06-09 16:22:12.229385 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30e3eb00] -2556> 2014-06-09 16:22:12.229387 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2555> 2014-06-09 16:22:12.229389 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2554> 2014-06-09 16:22:12.229393 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc240 (1 active) -2553> 2014-06-09 16:22:12.229396 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa480 (2 active) -2552> 2014-06-09 16:22:12.229402 7f8690763700 10 op_submit_finish 4274 -2551> 2014-06-09 16:22:12.229408 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.298( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=15-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 0.298( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) -2550> 2014-06-09 16:22:12.229451 7f8690763700 10 log is not dirty -2549> 2014-06-09 16:22:12.229460 7f8690763700 5 queue_transactions existing osr(0.298 0x2bd6c60)/0x2bd6c60 -2548> 2014-06-09 16:22:12.229463 7f8690763700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2547> 2014-06-09 16:22:12.229468 7f86a5e11700 5 _do_op 0x16fd4240 seq 4226 osr(1.37d 0x384d210)/0x384d210 start -2546> 2014-06-09 16:22:12.229472 7f86a5e11700 10 _do_transaction on 0xbfe4780 -2545> 2014-06-09 16:22:12.229478 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2544> 2014-06-09 16:22:12.229547 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2543> 2014-06-09 16:22:12.229565 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4226.0.0 <= header.spos 150443.0.0 -2542> 2014-06-09 16:22:12.229570 7f86a5e11700 5 submit_transaction r = 0 -2541> 2014-06-09 16:22:12.229583 7f86a5e11700 10 _do_op 0x16fd4240 seq 4226 r = 0, finisher 0x307f1ae0 0 -2540> 2014-06-09 16:22:12.229594 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2d7398c0] -2539> 2014-06-09 16:22:12.229597 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -2538> 2014-06-09 16:22:12.229608 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -2537> 2014-06-09 16:22:12.229618 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.37d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 -2536> 2014-06-09 16:22:12.229627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2d7398c0] -2535> 2014-06-09 16:22:12.229629 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2534> 2014-06-09 16:22:12.229630 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2533> 2014-06-09 16:22:12.229634 7f86a5e11700 10 _finish_op 0x16fd4240 seq 4226 osr(1.37d 0x384d210)/0x384d210 -2532> 2014-06-09 16:22:12.229645 7f8692767700 10 op_submit_start 4275 -2531> 2014-06-09 16:22:12.229647 7f8692767700 5 queue_transactions (trailing journal) 4275 -2530> 2014-06-09 16:22:12.229648 7f8692767700 5 queue_op 0x2e21aa0 seq 4275 osr(0.57b 0x2bd77b0) 828 bytes (queue has 50 ops and 4689481 bytes) -2529> 2014-06-09 16:22:12.229653 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2528> 2014-06-09 16:22:12.229658 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2527> 2014-06-09 16:22:12.229662 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2526> 2014-06-09 16:22:12.229667 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2525> 2014-06-09 16:22:12.229671 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2524> 2014-06-09 16:22:12.229675 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2523> 2014-06-09 16:22:12.229679 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2522> 2014-06-09 16:22:12.229683 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2521> 2014-06-09 16:22:12.229688 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2520> 2014-06-09 16:22:12.229692 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2519> 2014-06-09 16:22:12.229696 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2518> 2014-06-09 16:22:12.229700 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2517> 2014-06-09 16:22:12.229705 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2516> 2014-06-09 16:22:12.229709 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2515> 2014-06-09 16:22:12.229714 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2514> 2014-06-09 16:22:12.229718 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2513> 2014-06-09 16:22:12.229722 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2512> 2014-06-09 16:22:12.229726 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2511> 2014-06-09 16:22:12.229730 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2510> 2014-06-09 16:22:12.229740 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x307f1ae0] -2509> 2014-06-09 16:22:12.229744 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x307f1ae0] -2508> 2014-06-09 16:22:12.229745 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2507> 2014-06-09 16:22:12.229747 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2506> 2014-06-09 16:22:12.229749 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa480 (1 active) -2505> 2014-06-09 16:22:12.229752 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca3b680 (2 active) -2504> 2014-06-09 16:22:12.229757 7f8692767700 10 op_submit_finish 4275 -2503> 2014-06-09 16:22:12.229761 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.4c4( v 213'560 (0'0,213'560] local-les=243 n=110 ec=118 les/c 243/201 242/242/200) -2502> 2014-06-09 16:22:12.229798 7f8692767700 10 log is not dirty -2501> 2014-06-09 16:22:12.229804 7f8692767700 5 queue_transactions existing osr(3.4c4 0x3ed1420)/0x3ed1420 -2500> 2014-06-09 16:22:12.229807 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2499> 2014-06-09 16:22:12.229812 7f86a5e11700 5 _do_op 0xc2d4060 seq 4227 osr(1.764 0x2ed03f0)/0x2ed03f0 start -2498> 2014-06-09 16:22:12.229815 7f86a5e11700 10 _do_transaction on 0x2ed5bc0 -2497> 2014-06-09 16:22:12.229821 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2496> 2014-06-09 16:22:12.229864 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2495> 2014-06-09 16:22:12.229881 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4227.0.0 <= header.spos 150443.0.0 -2494> 2014-06-09 16:22:12.229886 7f86a5e11700 5 submit_transaction r = 0 -2493> 2014-06-09 16:22:12.229900 7f86a5e11700 10 _do_op 0xc2d4060 seq 4227 r = 0, finisher 0xc5a6a60 0 -2492> 2014-06-09 16:22:12.229910 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3061f620] -2491> 2014-06-09 16:22:12.229913 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -2490> 2014-06-09 16:22:12.229923 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -2489> 2014-06-09 16:22:12.229932 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.764( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 -2488> 2014-06-09 16:22:12.229942 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3061f620] -2487> 2014-06-09 16:22:12.229944 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2486> 2014-06-09 16:22:12.229945 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2485> 2014-06-09 16:22:12.229948 7f86a5e11700 10 _finish_op 0xc2d4060 seq 4227 osr(1.764 0x2ed03f0)/0x2ed03f0 -2484> 2014-06-09 16:22:12.229960 7f8691765700 10 op_submit_start 4276 -2483> 2014-06-09 16:22:12.229962 7f8691765700 5 queue_transactions (trailing journal) 4276 -2482> 2014-06-09 16:22:12.229963 7f8691765700 5 queue_op 0xc88bec0 seq 4276 osr(3.48d 0x3ed14e0) 828 bytes (queue has 50 ops and 4689448 bytes) -2481> 2014-06-09 16:22:12.229968 7f8689755700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2480> 2014-06-09 16:22:12.229973 7f868d75d700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2479> 2014-06-09 16:22:12.229977 7f8689f56700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2478> 2014-06-09 16:22:12.229982 7f868ff62700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2477> 2014-06-09 16:22:12.229986 7f868af58700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2476> 2014-06-09 16:22:12.229990 7f868f761700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2475> 2014-06-09 16:22:12.229994 7f868df5e700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2474> 2014-06-09 16:22:12.229998 7f868bf5a700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2473> 2014-06-09 16:22:12.230003 7f868c75b700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2472> 2014-06-09 16:22:12.230007 7f8692f68700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2471> 2014-06-09 16:22:12.230011 7f868b759700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2470> 2014-06-09 16:22:12.230016 7f868e75f700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2469> 2014-06-09 16:22:12.230020 7f868cf5c700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2468> 2014-06-09 16:22:12.230024 7f868a757700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2467> 2014-06-09 16:22:12.230028 7f868ef60700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2466> 2014-06-09 16:22:12.230033 7f8690763700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2465> 2014-06-09 16:22:12.230037 7f8690f64700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2464> 2014-06-09 16:22:12.230041 7f8692767700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2463> 2014-06-09 16:22:12.230046 7f8691f66700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2462> 2014-06-09 16:22:12.230055 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a6a60] -2461> 2014-06-09 16:22:12.230062 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a6a60] -2460> 2014-06-09 16:22:12.230064 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2459> 2014-06-09 16:22:12.230065 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2458> 2014-06-09 16:22:12.230069 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca3b680 (1 active) -2457> 2014-06-09 16:22:12.230072 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4480 (2 active) -2456> 2014-06-09 16:22:12.230077 7f8691765700 10 op_submit_finish 4276 -2455> 2014-06-09 16:22:12.230081 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.482( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -2454> 2014-06-09 16:22:12.230118 7f8691765700 10 log is not dirty -2453> 2014-06-09 16:22:12.230124 7f8691765700 5 queue_transactions existing osr(2.482 0x2ed14f0)/0x2ed14f0 -2452> 2014-06-09 16:22:12.230126 7f8691765700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2451> 2014-06-09 16:22:12.230130 7f86a5e11700 5 _do_op 0x3250f9e0 seq 4228 osr(1.4bb 0x384d970)/0x384d970 start -2450> 2014-06-09 16:22:12.230134 7f86a5e11700 10 _do_transaction on 0xc1452c0 -2449> 2014-06-09 16:22:12.230139 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2448> 2014-06-09 16:22:12.230181 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2447> 2014-06-09 16:22:12.230196 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4228.0.0 <= header.spos 150443.0.0 -2446> 2014-06-09 16:22:12.230200 7f86a5e11700 5 submit_transaction r = 0 -2445> 2014-06-09 16:22:12.230214 7f86a5e11700 10 _do_op 0x3250f9e0 seq 4228 r = 0, finisher 0xc2d6540 0 -2444> 2014-06-09 16:22:12.230224 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x1a002de0] -2443> 2014-06-09 16:22:12.230226 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x1a002de0] -2442> 2014-06-09 16:22:12.230228 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2441> 2014-06-09 16:22:12.230229 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2440> 2014-06-09 16:22:12.230233 7f86a5e11700 10 _finish_op 0x3250f9e0 seq 4228 osr(1.4bb 0x384d970)/0x384d970 -2439> 2014-06-09 16:22:12.230244 7f868d75d700 10 op_submit_start 4277 -2438> 2014-06-09 16:22:12.230246 7f868d75d700 5 queue_transactions (trailing journal) 4277 -2437> 2014-06-09 16:22:12.230248 7f868d75d700 5 queue_op 0xd471ce0 seq 4277 osr(0.629 0x2bd7c90) 861 bytes (queue has 50 ops and 4689481 bytes) -2436> 2014-06-09 16:22:12.230253 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2435> 2014-06-09 16:22:12.230258 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2434> 2014-06-09 16:22:12.230262 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2433> 2014-06-09 16:22:12.230266 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2432> 2014-06-09 16:22:12.230270 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2431> 2014-06-09 16:22:12.230275 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2430> 2014-06-09 16:22:12.230279 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2429> 2014-06-09 16:22:12.230283 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2428> 2014-06-09 16:22:12.230288 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2427> 2014-06-09 16:22:12.230292 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2426> 2014-06-09 16:22:12.230296 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2425> 2014-06-09 16:22:12.230300 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2424> 2014-06-09 16:22:12.230366 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2423> 2014-06-09 16:22:12.230371 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2422> 2014-06-09 16:22:12.230375 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2421> 2014-06-09 16:22:12.230379 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2420> 2014-06-09 16:22:12.230384 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2419> 2014-06-09 16:22:12.230388 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2418> 2014-06-09 16:22:12.230392 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2417> 2014-06-09 16:22:12.230402 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc2d6540] -2416> 2014-06-09 16:22:12.230406 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc2d6540] -2415> 2014-06-09 16:22:12.230407 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2414> 2014-06-09 16:22:12.230408 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2413> 2014-06-09 16:22:12.230412 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4480 (1 active) -2412> 2014-06-09 16:22:12.230414 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc2877a0 (2 active) -2411> 2014-06-09 16:22:12.230420 7f868d75d700 10 op_submit_finish 4277 -2410> 2014-06-09 16:22:12.230423 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -2409> 2014-06-09 16:22:12.230439 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] check_local -2408> 2014-06-09 16:22:12.230448 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -2407> 2014-06-09 16:22:12.230457 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -2406> 2014-06-09 16:22:12.230467 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -2405> 2014-06-09 16:22:12.230475 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -2404> 2014-06-09 16:22:12.230490 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:164 -2403> 2014-06-09 16:22:12.230500 7f868d75d700 20 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -2402> 2014-06-09 16:22:12.230509 7f868d75d700 20 osd.26 pg_epoch: 243 pg[0.18b( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] agent_stop -2401> 2014-06-09 16:22:12.230518 7f868d75d700 10 log is not dirty -2400> 2014-06-09 16:22:12.230547 7f868d75d700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xd4f35f0 (19 active) -2399> 2014-06-09 16:22:12.230557 7f868d75d700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -2398> 2014-06-09 16:22:12.230561 7f868d75d700 10 osd.26 244 dequeue_op 0x270bf400 prio 63 cost 307200 latency 1.467174 osd_op(client.4575.0:32478 rbd_data.117f6b8b4567.0000000000001554 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~307200] 3.d7c85d1e RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] -2397> 2014-06-09 16:22:12.230585 7f868d75d700 5 -- op tracker -- , seq: 693, time: 2014-06-09 16:22:12.230580, event: reached_pg, op: osd_op(client.4575.0:32478 rbd_data.117f6b8b4567.0000000000001554 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~307200] 3.d7c85d1e RETRY=3 ack+ondisk+retry+write e242) -2396> 2014-06-09 16:22:12.230601 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -2395> 2014-06-09 16:22:12.230615 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.51e( v 214'681 (0'0,214'681] local-les=243 n=129 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] can_handle_while_inactive: 0x270bf400 -2394> 2014-06-09 16:22:12.230624 7f868d75d700 10 osd.26 244 dequeue_op 0x270bf400 finish -2393> 2014-06-09 16:22:12.230628 7f868d75d700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -2392> 2014-06-09 16:22:12.230671 7f868d75d700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xd4f35f0 (20 active) -2391> 2014-06-09 16:22:12.230675 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -2390> 2014-06-09 16:22:12.230690 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] exit Started/Primary/Active/Activating 1.125597 7 0.000432 -2389> 2014-06-09 16:22:12.230704 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] enter Started/Primary/Active/Recovered -2388> 2014-06-09 16:22:12.230715 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] needs_recovery is recovered -2387> 2014-06-09 16:22:12.230728 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] exit Started/Primary/Active/Recovered 0.000025 0 0.000000 -2386> 2014-06-09 16:22:12.230741 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] enter Started/Primary/Active/Clean -2385> 2014-06-09 16:22:12.230750 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] finish_recovery -2384> 2014-06-09 16:22:12.230759 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active] clear_recovery_state -2383> 2014-06-09 16:22:12.230771 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] trim_past_intervals: trimming interval(200-235 up [26,3](26) acting [26,3](26) maybe_went_rw) -2382> 2014-06-09 16:22:12.230783 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] trim_past_intervals: trimming interval(236-241 up [3](3) acting [3](3)) -2381> 2014-06-09 16:22:12.230794 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] share_pg_info -2380> 2014-06-09 16:22:12.230811 7f868d75d700 20 osd.26 243 share_map_peer 0xc03a840 already has epoch 244 -2379> 2014-06-09 16:22:12.230814 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.3 172.16.96.201:6823/50569 -- pg_info(1 pgs e243:3.206) v4 -- ?+0 0x2fc95180 -2378> 2014-06-09 16:22:12.230830 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] publish_stats_to_osd 243:977 -2377> 2014-06-09 16:22:12.230865 7f868d75d700 10 log is not dirty -2376> 2014-06-09 16:22:12.230869 7f868d75d700 5 queue_transactions existing osr(3.206 0x3ed0cf0)/0x3ed0cf0 -2375> 2014-06-09 16:22:12.230872 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2374> 2014-06-09 16:22:12.230590 7f86a5e11700 5 _do_op 0x324e6c60 seq 4229 osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 start -2373> 2014-06-09 16:22:12.230880 7f86a5e11700 10 _do_transaction on 0xc2bec00 -2372> 2014-06-09 16:22:12.230885 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2371> 2014-06-09 16:22:12.230929 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2370> 2014-06-09 16:22:12.230944 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4229.0.0 <= header.spos 150443.0.0 -2369> 2014-06-09 16:22:12.230948 7f86a5e11700 5 submit_transaction r = 0 -2368> 2014-06-09 16:22:12.230960 7f86a5e11700 10 _do_op 0x324e6c60 seq 4229 r = 0, finisher 0x31456340 0 -2367> 2014-06-09 16:22:12.230970 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31434280] -2366> 2014-06-09 16:22:12.230972 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31434280] -2365> 2014-06-09 16:22:12.230974 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2364> 2014-06-09 16:22:12.230975 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2363> 2014-06-09 16:22:12.230979 7f86a5e11700 10 _finish_op 0x324e6c60 seq 4229 osr(3.3a5 0x3ed0fb0)/0x3ed0fb0 -2362> 2014-06-09 16:22:12.230990 7f868ff62700 10 op_submit_start 4278 -2361> 2014-06-09 16:22:12.230992 7f868ff62700 5 queue_transactions (trailing journal) 4278 -2360> 2014-06-09 16:22:12.230993 7f868ff62700 5 queue_op 0xc88a9c0 seq 4278 osr(3.1a0 0x3ed0ab0) 828 bytes (queue has 50 ops and 4689481 bytes) -2359> 2014-06-09 16:22:12.230999 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2358> 2014-06-09 16:22:12.231003 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2357> 2014-06-09 16:22:12.231008 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2356> 2014-06-09 16:22:12.231012 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2355> 2014-06-09 16:22:12.231016 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2354> 2014-06-09 16:22:12.231021 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2353> 2014-06-09 16:22:12.231025 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2352> 2014-06-09 16:22:12.231029 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2351> 2014-06-09 16:22:12.231034 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2350> 2014-06-09 16:22:12.231038 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2349> 2014-06-09 16:22:12.231042 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2348> 2014-06-09 16:22:12.231047 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2347> 2014-06-09 16:22:12.231051 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2346> 2014-06-09 16:22:12.231055 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2345> 2014-06-09 16:22:12.231059 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2344> 2014-06-09 16:22:12.231063 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2343> 2014-06-09 16:22:12.231067 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2342> 2014-06-09 16:22:12.231071 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2341> 2014-06-09 16:22:12.231076 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2340> 2014-06-09 16:22:12.231085 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31456340] -2339> 2014-06-09 16:22:12.231090 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31456340] -2338> 2014-06-09 16:22:12.231092 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2337> 2014-06-09 16:22:12.231093 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2336> 2014-06-09 16:22:12.231096 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc2877a0 (1 active) -2335> 2014-06-09 16:22:12.231099 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc524120 (2 active) -2334> 2014-06-09 16:22:12.231104 7f868ff62700 10 op_submit_finish 4278 -2333> 2014-06-09 16:22:12.231108 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.5cb( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 1.5cb( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) -2332> 2014-06-09 16:22:12.231143 7f868ff62700 10 log is not dirty -2331> 2014-06-09 16:22:12.231148 7f868ff62700 5 queue_transactions existing osr(1.5cb 0x384dd70)/0x384dd70 -2330> 2014-06-09 16:22:12.231150 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2329> 2014-06-09 16:22:12.231155 7f86a5e11700 5 _do_op 0x324da120 seq 4230 osr(0.3fe 0x2bd7210)/0x2bd7210 start -2328> 2014-06-09 16:22:12.231158 7f86a5e11700 10 _do_transaction on 0xbf82300 -2327> 2014-06-09 16:22:12.231163 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2326> 2014-06-09 16:22:12.231205 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2325> 2014-06-09 16:22:12.231220 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4230.0.0 <= header.spos 150443.0.0 -2324> 2014-06-09 16:22:12.231224 7f86a5e11700 5 submit_transaction r = 0 -2323> 2014-06-09 16:22:12.231244 7f86a5e11700 10 _do_op 0x324da120 seq 4230 r = 0, finisher 0x19940080 0 -2322> 2014-06-09 16:22:12.231255 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcf240] -2321> 2014-06-09 16:22:12.231257 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcf240] -2320> 2014-06-09 16:22:12.231259 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2319> 2014-06-09 16:22:12.231260 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2318> 2014-06-09 16:22:12.231263 7f86a5e11700 10 _finish_op 0x324da120 seq 4230 osr(0.3fe 0x2bd7210)/0x2bd7210 -2317> 2014-06-09 16:22:12.231274 7f868f761700 10 op_submit_start 4279 -2316> 2014-06-09 16:22:12.231276 7f868f761700 5 queue_transactions (trailing journal) 4279 -2315> 2014-06-09 16:22:12.231277 7f868f761700 5 queue_op 0x24104300 seq 4279 osr(1.302 0x384d010) 828 bytes (queue has 50 ops and 4689481 bytes) -2314> 2014-06-09 16:22:12.231283 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2313> 2014-06-09 16:22:12.231287 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2312> 2014-06-09 16:22:12.231292 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2311> 2014-06-09 16:22:12.231297 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2310> 2014-06-09 16:22:12.231301 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2309> 2014-06-09 16:22:12.231306 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2308> 2014-06-09 16:22:12.231310 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2307> 2014-06-09 16:22:12.231314 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2306> 2014-06-09 16:22:12.231319 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2305> 2014-06-09 16:22:12.231323 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2304> 2014-06-09 16:22:12.231327 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2303> 2014-06-09 16:22:12.231332 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2302> 2014-06-09 16:22:12.231336 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2301> 2014-06-09 16:22:12.231340 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2300> 2014-06-09 16:22:12.231344 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2299> 2014-06-09 16:22:12.231349 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2298> 2014-06-09 16:22:12.231353 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2297> 2014-06-09 16:22:12.231357 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2296> 2014-06-09 16:22:12.231362 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2295> 2014-06-09 16:22:12.231372 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x19940080] -2294> 2014-06-09 16:22:12.231375 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x19940080] -2293> 2014-06-09 16:22:12.231377 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2292> 2014-06-09 16:22:12.231378 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2291> 2014-06-09 16:22:12.231381 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc524120 (1 active) -2290> 2014-06-09 16:22:12.231384 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287200 (2 active) -2289> 2014-06-09 16:22:12.231389 7f868f761700 10 op_submit_finish 4279 -2288> 2014-06-09 16:22:12.231393 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -2287> 2014-06-09 16:22:12.231406 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] check_local -2286> 2014-06-09 16:22:12.231415 7f868f761700 15 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -2285> 2014-06-09 16:22:12.231424 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -2284> 2014-06-09 16:22:12.231432 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -2283> 2014-06-09 16:22:12.231441 7f868f761700 10 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -2282> 2014-06-09 16:22:12.231454 7f868f761700 15 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:152 -2281> 2014-06-09 16:22:12.231464 7f868f761700 20 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -2280> 2014-06-09 16:22:12.231472 7f868f761700 20 osd.26 pg_epoch: 243 pg[2.1fc( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] agent_stop -2279> 2014-06-09 16:22:12.231481 7f868f761700 10 log is not dirty -2278> 2014-06-09 16:22:12.231485 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.2df( v 213'543 (0'0,213'543] local-les=243 n=98 ec=118 les/c 243/243 242/242/209) -2277> 2014-06-09 16:22:12.231518 7f868f761700 10 log is not dirty -2276> 2014-06-09 16:22:12.231523 7f868f761700 5 queue_transactions existing osr(3.2df 0x3ed0e40)/0x3ed0e40 -2275> 2014-06-09 16:22:12.231525 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2274> 2014-06-09 16:22:12.231530 7f86a5e11700 5 _do_op 0xd327260 seq 4231 osr(1.302 0x384d010)/0x384d010 start -2273> 2014-06-09 16:22:12.231533 7f86a5e11700 10 _do_transaction on 0xc257200 -2272> 2014-06-09 16:22:12.231538 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2271> 2014-06-09 16:22:12.231579 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2270> 2014-06-09 16:22:12.231594 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4231.0.0 <= header.spos 150443.0.0 -2269> 2014-06-09 16:22:12.231598 7f86a5e11700 5 submit_transaction r = 0 -2268> 2014-06-09 16:22:12.231610 7f86a5e11700 10 _do_op 0xd327260 seq 4231 r = 0, finisher 0x31bffec0 0 -2267> 2014-06-09 16:22:12.231624 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbce7120] -2266> 2014-06-09 16:22:12.231626 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbce7120] -2265> 2014-06-09 16:22:12.231627 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2264> 2014-06-09 16:22:12.231628 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2263> 2014-06-09 16:22:12.231631 7f86a5e11700 10 _finish_op 0xd327260 seq 4231 osr(1.302 0x384d010)/0x384d010 -2262> 2014-06-09 16:22:12.231640 7f868bf5a700 10 op_submit_start 4280 -2261> 2014-06-09 16:22:12.231641 7f868bf5a700 5 queue_transactions (trailing journal) 4280 -2260> 2014-06-09 16:22:12.231643 7f868bf5a700 5 queue_op 0x30032c60 seq 4280 osr(1.465 0x384d650) 828 bytes (queue has 50 ops and 4689481 bytes) -2259> 2014-06-09 16:22:12.231647 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2258> 2014-06-09 16:22:12.231651 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2257> 2014-06-09 16:22:12.231654 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2256> 2014-06-09 16:22:12.231658 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2255> 2014-06-09 16:22:12.231662 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2254> 2014-06-09 16:22:12.231666 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2253> 2014-06-09 16:22:12.231669 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2252> 2014-06-09 16:22:12.231673 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2251> 2014-06-09 16:22:12.231677 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2250> 2014-06-09 16:22:12.231681 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2249> 2014-06-09 16:22:12.231685 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2248> 2014-06-09 16:22:12.231689 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2247> 2014-06-09 16:22:12.231694 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2246> 2014-06-09 16:22:12.231698 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2245> 2014-06-09 16:22:12.231703 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2244> 2014-06-09 16:22:12.231707 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2243> 2014-06-09 16:22:12.231711 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2242> 2014-06-09 16:22:12.231716 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2241> 2014-06-09 16:22:12.231720 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2240> 2014-06-09 16:22:12.231730 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31bffec0] -2239> 2014-06-09 16:22:12.231734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31bffec0] -2238> 2014-06-09 16:22:12.231736 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2237> 2014-06-09 16:22:12.231737 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2236> 2014-06-09 16:22:12.231740 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287200 (1 active) -2235> 2014-06-09 16:22:12.231743 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dd440 (2 active) -2234> 2014-06-09 16:22:12.231747 7f868bf5a700 10 op_submit_finish 4280 -2233> 2014-06-09 16:22:12.231751 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.566( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 2.566( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) -2232> 2014-06-09 16:22:12.231786 7f868bf5a700 10 log is not dirty -2231> 2014-06-09 16:22:12.231791 7f868bf5a700 5 queue_transactions existing osr(2.566 0x2ed18f0)/0x2ed18f0 -2230> 2014-06-09 16:22:12.231794 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2229> 2014-06-09 16:22:12.231798 7f86a5e11700 5 _do_op 0x324f4b40 seq 4232 osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 start -2228> 2014-06-09 16:22:12.231802 7f86a5e11700 10 _do_transaction on 0xc22a6c0 -2227> 2014-06-09 16:22:12.231807 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2226> 2014-06-09 16:22:12.231849 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2225> 2014-06-09 16:22:12.231864 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4232.0.0 <= header.spos 150443.0.0 -2224> 2014-06-09 16:22:12.231868 7f86a5e11700 5 submit_transaction r = 0 -2223> 2014-06-09 16:22:12.231883 7f86a5e11700 10 _do_op 0x324f4b40 seq 4232 r = 0, finisher 0xcd337e0 0 -2222> 2014-06-09 16:22:12.231894 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314adfa0] -2221> 2014-06-09 16:22:12.231896 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314adfa0] -2220> 2014-06-09 16:22:12.231898 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2219> 2014-06-09 16:22:12.231899 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2218> 2014-06-09 16:22:12.231902 7f86a5e11700 10 _finish_op 0x324f4b40 seq 4232 osr(2.2f2 0x2ed0fb0)/0x2ed0fb0 -2217> 2014-06-09 16:22:12.231913 7f8692f68700 10 op_submit_start 4281 -2216> 2014-06-09 16:22:12.231914 7f8692f68700 5 queue_transactions (trailing journal) 4281 -2215> 2014-06-09 16:22:12.231916 7f8692f68700 5 queue_op 0x1e08b7a0 seq 4281 osr(2.4af 0x2ed1790) 828 bytes (queue has 50 ops and 4689481 bytes) -2214> 2014-06-09 16:22:12.231921 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2213> 2014-06-09 16:22:12.231925 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2212> 2014-06-09 16:22:12.231930 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2211> 2014-06-09 16:22:12.231934 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2210> 2014-06-09 16:22:12.231938 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2209> 2014-06-09 16:22:12.231943 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2208> 2014-06-09 16:22:12.231947 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2207> 2014-06-09 16:22:12.231952 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2206> 2014-06-09 16:22:12.231956 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2205> 2014-06-09 16:22:12.231960 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2204> 2014-06-09 16:22:12.231965 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2203> 2014-06-09 16:22:12.231969 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2202> 2014-06-09 16:22:12.231973 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2201> 2014-06-09 16:22:12.231977 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2200> 2014-06-09 16:22:12.231981 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2199> 2014-06-09 16:22:12.231986 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2198> 2014-06-09 16:22:12.231990 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2197> 2014-06-09 16:22:12.231994 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2196> 2014-06-09 16:22:12.231998 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2195> 2014-06-09 16:22:12.232008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd337e0] -2194> 2014-06-09 16:22:12.232016 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd337e0] -2193> 2014-06-09 16:22:12.232018 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2192> 2014-06-09 16:22:12.232019 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2191> 2014-06-09 16:22:12.232022 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dd440 (1 active) -2190> 2014-06-09 16:22:12.232025 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4078c0 (2 active) -2189> 2014-06-09 16:22:12.232030 7f8692f68700 10 op_submit_finish 4281 -2188> 2014-06-09 16:22:12.232034 7f8692f68700 10 osd.26 pg_epoch: 243 pg[0.1da( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.1da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -2187> 2014-06-09 16:22:12.232072 7f8692f68700 10 log is not dirty -2186> 2014-06-09 16:22:12.232077 7f8692f68700 5 queue_transactions existing osr(0.1da 0x2bd6920)/0x2bd6920 -2185> 2014-06-09 16:22:12.232080 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2184> 2014-06-09 16:22:12.232084 7f86a5e11700 5 _do_op 0xcabcfc0 seq 4233 osr(3.218 0x3ed0cd0)/0x3ed0cd0 start -2183> 2014-06-09 16:22:12.232088 7f86a5e11700 10 _do_transaction on 0xbf88180 -2182> 2014-06-09 16:22:12.232093 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2181> 2014-06-09 16:22:12.232134 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2180> 2014-06-09 16:22:12.232148 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4233.0.0 <= header.spos 150443.0.0 -2179> 2014-06-09 16:22:12.232153 7f86a5e11700 5 submit_transaction r = 0 -2178> 2014-06-09 16:22:12.232165 7f86a5e11700 10 _do_op 0xcabcfc0 seq 4233 r = 0, finisher 0x3042b0a0 0 -2177> 2014-06-09 16:22:12.232175 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x33a61c0] -2176> 2014-06-09 16:22:12.232178 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x33a61c0] -2175> 2014-06-09 16:22:12.232179 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2174> 2014-06-09 16:22:12.232180 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2173> 2014-06-09 16:22:12.232184 7f86a5e11700 10 _finish_op 0xcabcfc0 seq 4233 osr(3.218 0x3ed0cd0)/0x3ed0cd0 -2172> 2014-06-09 16:22:12.232195 7f868e75f700 10 op_submit_start 4282 -2171> 2014-06-09 16:22:12.232196 7f868e75f700 5 queue_transactions (trailing journal) 4282 -2170> 2014-06-09 16:22:12.232198 7f868e75f700 5 queue_op 0x324f4f60 seq 4282 osr(0.530 0x2bd78f0) 828 bytes (queue has 50 ops and 4689481 bytes) -2169> 2014-06-09 16:22:12.232203 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2168> 2014-06-09 16:22:12.232208 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2167> 2014-06-09 16:22:12.232213 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2166> 2014-06-09 16:22:12.232276 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2165> 2014-06-09 16:22:12.232282 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2164> 2014-06-09 16:22:12.232286 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2163> 2014-06-09 16:22:12.232291 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2162> 2014-06-09 16:22:12.232295 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2161> 2014-06-09 16:22:12.232299 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2160> 2014-06-09 16:22:12.232304 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2159> 2014-06-09 16:22:12.232308 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2158> 2014-06-09 16:22:12.232312 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2157> 2014-06-09 16:22:12.232316 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2156> 2014-06-09 16:22:12.232320 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2155> 2014-06-09 16:22:12.232325 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2154> 2014-06-09 16:22:12.232329 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2153> 2014-06-09 16:22:12.232333 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2152> 2014-06-09 16:22:12.232338 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2151> 2014-06-09 16:22:12.232342 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2150> 2014-06-09 16:22:12.232351 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b0a0] -2149> 2014-06-09 16:22:12.232356 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b0a0] -2148> 2014-06-09 16:22:12.232357 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2147> 2014-06-09 16:22:12.232358 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2146> 2014-06-09 16:22:12.232361 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4078c0 (1 active) -2145> 2014-06-09 16:22:12.232364 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7e60 (2 active) -2144> 2014-06-09 16:22:12.232369 7f868e75f700 10 op_submit_finish 4282 -2143> 2014-06-09 16:22:12.232373 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.26a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.26a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) -2142> 2014-06-09 16:22:12.232410 7f868e75f700 10 log is not dirty -2141> 2014-06-09 16:22:12.232415 7f868e75f700 5 queue_transactions existing osr(2.26a 0x2ed0dd0)/0x2ed0dd0 -2140> 2014-06-09 16:22:12.232418 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2139> 2014-06-09 16:22:12.232422 7f86a5e11700 5 _do_op 0xc34a660 seq 4234 osr(3.65f 0x3ed1a10)/0x3ed1a10 start -2138> 2014-06-09 16:22:12.232426 7f86a5e11700 10 _do_transaction on 0xc129200 -2137> 2014-06-09 16:22:12.232432 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2136> 2014-06-09 16:22:12.232474 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2135> 2014-06-09 16:22:12.232491 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4234.0.0 <= header.spos 150443.0.0 -2134> 2014-06-09 16:22:12.232495 7f86a5e11700 5 submit_transaction r = 0 -2133> 2014-06-09 16:22:12.232526 7f86a5e11700 10 _do_op 0xc34a660 seq 4234 r = 0, finisher 0x3006dbc0 0 -2132> 2014-06-09 16:22:12.232538 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd9f160] -2131> 2014-06-09 16:22:12.232540 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery -2130> 2014-06-09 16:22:12.232553 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays -2129> 2014-06-09 16:22:12.232564 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.65f( v 214'652 (0'0,214'652] local-les=243 n=125 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1114 -2128> 2014-06-09 16:22:12.232576 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd9f160] -2127> 2014-06-09 16:22:12.232578 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2126> 2014-06-09 16:22:12.232579 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2125> 2014-06-09 16:22:12.232583 7f86a5e11700 10 _finish_op 0xc34a660 seq 4234 osr(3.65f 0x3ed1a10)/0x3ed1a10 -2124> 2014-06-09 16:22:12.232594 7f868a757700 10 op_submit_start 4283 -2123> 2014-06-09 16:22:12.232596 7f868a757700 5 queue_transactions (trailing journal) 4283 -2122> 2014-06-09 16:22:12.232598 7f868a757700 5 queue_op 0xc2d4840 seq 4283 osr(2.523 0x2ed1610) 861 bytes (queue has 50 ops and 4689481 bytes) -2121> 2014-06-09 16:22:12.232603 7f868ef60700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2120> 2014-06-09 16:22:12.232607 7f8690763700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2119> 2014-06-09 16:22:12.232612 7f8690f64700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2118> 2014-06-09 16:22:12.232616 7f8692767700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2117> 2014-06-09 16:22:12.232620 7f8691f66700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2116> 2014-06-09 16:22:12.232625 7f8691765700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2115> 2014-06-09 16:22:12.232629 7f8689755700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2114> 2014-06-09 16:22:12.232633 7f868d75d700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2113> 2014-06-09 16:22:12.232638 7f8689f56700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2112> 2014-06-09 16:22:12.232642 7f868ff62700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2111> 2014-06-09 16:22:12.232646 7f868af58700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2110> 2014-06-09 16:22:12.232651 7f868f761700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2109> 2014-06-09 16:22:12.232655 7f868df5e700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2108> 2014-06-09 16:22:12.232660 7f868bf5a700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2107> 2014-06-09 16:22:12.232664 7f868c75b700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2106> 2014-06-09 16:22:12.232668 7f8692f68700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2105> 2014-06-09 16:22:12.232672 7f868b759700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2104> 2014-06-09 16:22:12.232676 7f868e75f700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2103> 2014-06-09 16:22:12.232680 7f868cf5c700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2102> 2014-06-09 16:22:12.232690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006dbc0] -2101> 2014-06-09 16:22:12.232695 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006dbc0] -2100> 2014-06-09 16:22:12.232697 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2099> 2014-06-09 16:22:12.232698 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2098> 2014-06-09 16:22:12.232701 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7e60 (1 active) -2097> 2014-06-09 16:22:12.232704 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b9e0 (2 active) -2096> 2014-06-09 16:22:12.232709 7f868a757700 10 op_submit_finish 4283 -2095> 2014-06-09 16:22:12.232713 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -2094> 2014-06-09 16:22:12.232728 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.062975 7 0.000322 -2093> 2014-06-09 16:22:12.232741 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -2092> 2014-06-09 16:22:12.232752 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -2091> 2014-06-09 16:22:12.232764 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -2090> 2014-06-09 16:22:12.232775 7f868a757700 5 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -2089> 2014-06-09 16:22:12.232785 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery -2088> 2014-06-09 16:22:12.232794 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state -2087> 2014-06-09 16:22:12.232805 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -2086> 2014-06-09 16:22:12.232818 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -2085> 2014-06-09 16:22:12.232829 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info -2084> 2014-06-09 16:22:12.232844 7f868a757700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 -2083> 2014-06-09 16:22:12.232846 7f868a757700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:3.380) v4 -- ?+0 0x30c128c0 -2082> 2014-06-09 16:22:12.232859 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1105 -2081> 2014-06-09 16:22:12.232894 7f868a757700 10 log is not dirty -2080> 2014-06-09 16:22:12.232898 7f868a757700 5 queue_transactions existing osr(3.380 0x3ed1020)/0x3ed1020 -2079> 2014-06-09 16:22:12.232901 7f868a757700 2 waiting 51 > 50 ops || 4690342 > 104857600 -2078> 2014-06-09 16:22:12.232929 7f86a5e11700 5 _do_op 0xc2eaba0 seq 4235 osr(1.4bc 0x384d930)/0x384d930 start -2077> 2014-06-09 16:22:12.232938 7f86a5e11700 10 _do_transaction on 0x2ed4780 -2076> 2014-06-09 16:22:12.232944 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2075> 2014-06-09 16:22:12.232987 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2074> 2014-06-09 16:22:12.233003 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4235.0.0 <= header.spos 150443.0.0 -2073> 2014-06-09 16:22:12.233008 7f86a5e11700 5 submit_transaction r = 0 -2072> 2014-06-09 16:22:12.233020 7f86a5e11700 10 _do_op 0xc2eaba0 seq 4235 r = 0, finisher 0x2e0e7e0 0 -2071> 2014-06-09 16:22:12.233031 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x305087c0] -2070> 2014-06-09 16:22:12.233034 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -2069> 2014-06-09 16:22:12.233044 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -2068> 2014-06-09 16:22:12.233053 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.4bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 -2067> 2014-06-09 16:22:12.233063 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x305087c0] -2066> 2014-06-09 16:22:12.233064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2065> 2014-06-09 16:22:12.233066 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2064> 2014-06-09 16:22:12.233069 7f86a5e11700 10 _finish_op 0xc2eaba0 seq 4235 osr(1.4bc 0x384d930)/0x384d930 -2063> 2014-06-09 16:22:12.233080 7f8690763700 10 op_submit_start 4284 -2062> 2014-06-09 16:22:12.233082 7f8690763700 5 queue_transactions (trailing journal) 4284 -2061> 2014-06-09 16:22:12.233083 7f8690763700 5 queue_op 0xc2ea540 seq 4284 osr(0.298 0x2bd6c60) 828 bytes (queue has 50 ops and 4689448 bytes) -2060> 2014-06-09 16:22:12.233088 7f8690f64700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2059> 2014-06-09 16:22:12.233092 7f8692767700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2058> 2014-06-09 16:22:12.233097 7f8691f66700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2057> 2014-06-09 16:22:12.233101 7f8691765700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2056> 2014-06-09 16:22:12.233105 7f8689755700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2055> 2014-06-09 16:22:12.233109 7f868d75d700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2054> 2014-06-09 16:22:12.233114 7f8689f56700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2053> 2014-06-09 16:22:12.233118 7f868ff62700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2052> 2014-06-09 16:22:12.233122 7f868af58700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2051> 2014-06-09 16:22:12.233126 7f868f761700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2050> 2014-06-09 16:22:12.233130 7f868df5e700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2049> 2014-06-09 16:22:12.233135 7f868bf5a700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2048> 2014-06-09 16:22:12.233139 7f868c75b700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2047> 2014-06-09 16:22:12.233143 7f8692f68700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2046> 2014-06-09 16:22:12.233147 7f868b759700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2045> 2014-06-09 16:22:12.233151 7f868e75f700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2044> 2014-06-09 16:22:12.233155 7f868cf5c700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2043> 2014-06-09 16:22:12.233160 7f868a757700 2 waiting 51 > 50 ops || 4690309 > 104857600 -2042> 2014-06-09 16:22:12.233164 7f868ef60700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2041> 2014-06-09 16:22:12.233174 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0e7e0] -2040> 2014-06-09 16:22:12.233178 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0e7e0] -2039> 2014-06-09 16:22:12.233180 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -2038> 2014-06-09 16:22:12.233181 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -2037> 2014-06-09 16:22:12.233184 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b9e0 (1 active) -2036> 2014-06-09 16:22:12.233187 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358d80 (2 active) -2035> 2014-06-09 16:22:12.233192 7f8690763700 10 op_submit_finish 4284 -2034> 2014-06-09 16:22:12.233196 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -2033> 2014-06-09 16:22:12.233214 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] check_local -2032> 2014-06-09 16:22:12.233226 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] requeue_ops -2031> 2014-06-09 16:22:12.233236 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing -2030> 2014-06-09 16:22:12.233248 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery -2029> 2014-06-09 16:22:12.233265 7f8690763700 15 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:982 -2028> 2014-06-09 16:22:12.233277 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear -2027> 2014-06-09 16:22:12.233288 7f8690763700 20 osd.26 pg_epoch: 243 pg[3.4c0( v 214'586 lc 214'585 (0'0,214'586] local-les=243 n=110 ec=118 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=214'586 lcod 0'0 mlcod 0'0 active m=1] agent_stop -2026> 2014-06-09 16:22:12.233299 7f8690763700 10 log is not dirty -2025> 2014-06-09 16:22:12.233304 7f8690763700 10 osd.26 pg_epoch: 243 pg[0.648( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 0.648( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) -2024> 2014-06-09 16:22:12.233337 7f8690763700 10 log is not dirty -2023> 2014-06-09 16:22:12.233342 7f8690763700 5 queue_transactions existing osr(0.648 0x2bd7bd0)/0x2bd7bd0 -2022> 2014-06-09 16:22:12.233344 7f8690763700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2021> 2014-06-09 16:22:12.233349 7f86a5e11700 5 _do_op 0xc64a960 seq 4236 osr(2.3c7 0x2ed13d0)/0x2ed13d0 start -2020> 2014-06-09 16:22:12.233352 7f86a5e11700 10 _do_transaction on 0xc22a840 -2019> 2014-06-09 16:22:12.233358 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -2018> 2014-06-09 16:22:12.233400 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -2017> 2014-06-09 16:22:12.233418 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4236.0.0 <= header.spos 150443.0.0 -2016> 2014-06-09 16:22:12.233422 7f86a5e11700 5 submit_transaction r = 0 -2015> 2014-06-09 16:22:12.233434 7f86a5e11700 10 _do_op 0xc64a960 seq 4236 r = 0, finisher 0xbd71e20 0 -2014> 2014-06-09 16:22:12.233445 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e8460] -2013> 2014-06-09 16:22:12.233448 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -2012> 2014-06-09 16:22:12.233458 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -2011> 2014-06-09 16:22:12.233467 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3c7( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:207 -2010> 2014-06-09 16:22:12.233476 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e8460] -2009> 2014-06-09 16:22:12.233478 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -2008> 2014-06-09 16:22:12.233479 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -2007> 2014-06-09 16:22:12.233483 7f86a5e11700 10 _finish_op 0xc64a960 seq 4236 osr(2.3c7 0x2ed13d0)/0x2ed13d0 -2006> 2014-06-09 16:22:12.233494 7f8692767700 10 op_submit_start 4285 -2005> 2014-06-09 16:22:12.233495 7f8692767700 5 queue_transactions (trailing journal) 4285 -2004> 2014-06-09 16:22:12.233497 7f8692767700 5 queue_op 0x325398c0 seq 4285 osr(3.4c4 0x3ed1420) 828 bytes (queue has 50 ops and 4689415 bytes) -2003> 2014-06-09 16:22:12.233502 7f8691f66700 2 waiting 51 > 50 ops || 4690276 > 104857600 -2002> 2014-06-09 16:22:12.233507 7f8691765700 2 waiting 51 > 50 ops || 4690243 > 104857600 -2001> 2014-06-09 16:22:12.233511 7f8689755700 2 waiting 51 > 50 ops || 4690243 > 104857600 -2000> 2014-06-09 16:22:12.233515 7f868d75d700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1999> 2014-06-09 16:22:12.233520 7f8689f56700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1998> 2014-06-09 16:22:12.233524 7f868ff62700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1997> 2014-06-09 16:22:12.233528 7f868af58700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1996> 2014-06-09 16:22:12.233532 7f868f761700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1995> 2014-06-09 16:22:12.233537 7f868df5e700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1994> 2014-06-09 16:22:12.233541 7f868bf5a700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1993> 2014-06-09 16:22:12.233545 7f868c75b700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1992> 2014-06-09 16:22:12.233549 7f8692f68700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1991> 2014-06-09 16:22:12.233554 7f868b759700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1990> 2014-06-09 16:22:12.233558 7f868e75f700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1989> 2014-06-09 16:22:12.233562 7f868cf5c700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1988> 2014-06-09 16:22:12.233566 7f868a757700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1987> 2014-06-09 16:22:12.233571 7f868ef60700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1986> 2014-06-09 16:22:12.233575 7f8690763700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1985> 2014-06-09 16:22:12.233579 7f8690f64700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1984> 2014-06-09 16:22:12.233589 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd71e20] -1983> 2014-06-09 16:22:12.233594 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd71e20] -1982> 2014-06-09 16:22:12.233596 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1981> 2014-06-09 16:22:12.233597 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1980> 2014-06-09 16:22:12.233600 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358d80 (1 active) -1979> 2014-06-09 16:22:12.233603 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ac5a0 (2 active) -1978> 2014-06-09 16:22:12.233608 7f8692767700 10 op_submit_finish 4285 -1977> 2014-06-09 16:22:12.233611 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.113( v 227'654 lc 227'653 (0'0,227'654] local-les=243 n=129 ec=118 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=227'654 lcod 0'0 active m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.113( v 227'654 (0'0,227'654] local-les=243 n=129 ec=118 les/c 243/213 242/242/212) -1976> 2014-06-09 16:22:12.233650 7f8692767700 10 log is not dirty -1975> 2014-06-09 16:22:12.233655 7f8692767700 5 queue_transactions existing osr(3.113 0x3ed0800)/0x3ed0800 -1974> 2014-06-09 16:22:12.233657 7f8692767700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1973> 2014-06-09 16:22:12.233662 7f86a5e11700 5 _do_op 0x308cb7a0 seq 4237 osr(2.5f4 0x2ed1dd0)/0x2ed1dd0 start -1972> 2014-06-09 16:22:12.233665 7f86a5e11700 10 _do_transaction on 0xbeba6c0 -1971> 2014-06-09 16:22:12.233671 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1970> 2014-06-09 16:22:12.233707 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1969> 2014-06-09 16:22:12.233722 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4237.0.0 <= header.spos 150443.0.0 -1968> 2014-06-09 16:22:12.233725 7f86a5e11700 5 submit_transaction r = 0 -1967> 2014-06-09 16:22:12.233738 7f86a5e11700 10 _do_op 0x308cb7a0 seq 4237 r = 0, finisher 0x19941780 0 -1966> 2014-06-09 16:22:12.233747 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcd33a80] -1965> 2014-06-09 16:22:12.233750 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -1964> 2014-06-09 16:22:12.233758 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -1963> 2014-06-09 16:22:12.233766 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.5f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:193 -1962> 2014-06-09 16:22:12.233774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcd33a80] -1961> 2014-06-09 16:22:12.233775 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1960> 2014-06-09 16:22:12.233776 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1959> 2014-06-09 16:22:12.233779 7f86a5e11700 10 _finish_op 0x308cb7a0 seq 4237 osr(2.5f4 0x2ed1dd0)/0x2ed1dd0 -1958> 2014-06-09 16:22:12.233788 7f8691765700 10 op_submit_start 4286 -1957> 2014-06-09 16:22:12.233790 7f8691765700 5 queue_transactions (trailing journal) 4286 -1956> 2014-06-09 16:22:12.233791 7f8691765700 5 queue_op 0xc88aa20 seq 4286 osr(2.482 0x2ed14f0) 828 bytes (queue has 50 ops and 4689382 bytes) -1955> 2014-06-09 16:22:12.233794 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1954> 2014-06-09 16:22:12.233798 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1953> 2014-06-09 16:22:12.233801 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1952> 2014-06-09 16:22:12.233805 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1951> 2014-06-09 16:22:12.233808 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1950> 2014-06-09 16:22:12.233811 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1949> 2014-06-09 16:22:12.233814 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1948> 2014-06-09 16:22:12.233817 7f868bf5a700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1947> 2014-06-09 16:22:12.233821 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1946> 2014-06-09 16:22:12.233824 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1945> 2014-06-09 16:22:12.233827 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1944> 2014-06-09 16:22:12.233830 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1943> 2014-06-09 16:22:12.233833 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1942> 2014-06-09 16:22:12.233836 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1941> 2014-06-09 16:22:12.233839 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1940> 2014-06-09 16:22:12.233842 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1939> 2014-06-09 16:22:12.233845 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1938> 2014-06-09 16:22:12.233848 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1937> 2014-06-09 16:22:12.233851 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1936> 2014-06-09 16:22:12.233859 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x19941780] -1935> 2014-06-09 16:22:12.233862 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x19941780] -1934> 2014-06-09 16:22:12.233864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1933> 2014-06-09 16:22:12.233864 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1932> 2014-06-09 16:22:12.233867 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ac5a0 (1 active) -1931> 2014-06-09 16:22:12.233869 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338120 (2 active) -1930> 2014-06-09 16:22:12.233874 7f8691765700 10 op_submit_finish 4286 -1929> 2014-06-09 16:22:12.233877 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.48d( v 213'551 (0'0,213'551] local-les=243 n=109 ec=118 les/c 243/243 242/242/205) -1928> 2014-06-09 16:22:12.233914 7f8691765700 10 log is not dirty -1927> 2014-06-09 16:22:12.233919 7f8691765700 5 queue_transactions existing osr(3.48d 0x3ed14e0)/0x3ed14e0 -1926> 2014-06-09 16:22:12.233921 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1925> 2014-06-09 16:22:12.233926 7f86a5e11700 5 _do_op 0x325034a0 seq 4238 osr(0.6f0 0x384c170)/0x384c170 start -1924> 2014-06-09 16:22:12.233930 7f86a5e11700 10 _do_transaction on 0xbf39080 -1923> 2014-06-09 16:22:12.233934 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1922> 2014-06-09 16:22:12.233976 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1921> 2014-06-09 16:22:12.233990 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4238.0.0 <= header.spos 150443.0.0 -1920> 2014-06-09 16:22:12.233994 7f86a5e11700 5 submit_transaction r = 0 -1919> 2014-06-09 16:22:12.234012 7f86a5e11700 10 _do_op 0x325034a0 seq 4238 r = 0, finisher 0x2d7399a0 0 -1918> 2014-06-09 16:22:12.234023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6bb00] -1917> 2014-06-09 16:22:12.234025 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6bb00] -1916> 2014-06-09 16:22:12.234027 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1915> 2014-06-09 16:22:12.234028 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1914> 2014-06-09 16:22:12.234031 7f86a5e11700 10 _finish_op 0x325034a0 seq 4238 osr(0.6f0 0x384c170)/0x384c170 -1913> 2014-06-09 16:22:12.234042 7f868d75d700 10 op_submit_start 4287 -1912> 2014-06-09 16:22:12.234043 7f868d75d700 5 queue_transactions (trailing journal) 4287 -1911> 2014-06-09 16:22:12.234044 7f868d75d700 5 queue_op 0xc2eb680 seq 4287 osr(3.206 0x3ed0cf0) 861 bytes (queue has 50 ops and 4689415 bytes) -1910> 2014-06-09 16:22:12.234049 7f8689f56700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1909> 2014-06-09 16:22:12.234054 7f868ff62700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1908> 2014-06-09 16:22:12.234058 7f868af58700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1907> 2014-06-09 16:22:12.234062 7f868f761700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1906> 2014-06-09 16:22:12.234067 7f868df5e700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1905> 2014-06-09 16:22:12.234071 7f868bf5a700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1904> 2014-06-09 16:22:12.234075 7f868c75b700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1903> 2014-06-09 16:22:12.234079 7f8692f68700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1902> 2014-06-09 16:22:12.234084 7f868b759700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1901> 2014-06-09 16:22:12.234088 7f868e75f700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1900> 2014-06-09 16:22:12.234092 7f868cf5c700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1899> 2014-06-09 16:22:12.234096 7f868a757700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1898> 2014-06-09 16:22:12.234101 7f868ef60700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1897> 2014-06-09 16:22:12.234105 7f8690763700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1896> 2014-06-09 16:22:12.234109 7f8690f64700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1895> 2014-06-09 16:22:12.234113 7f8692767700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1894> 2014-06-09 16:22:12.234118 7f8691f66700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1893> 2014-06-09 16:22:12.234122 7f8691765700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1892> 2014-06-09 16:22:12.234126 7f8689755700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1891> 2014-06-09 16:22:12.234194 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d7399a0] -1890> 2014-06-09 16:22:12.234199 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d7399a0] -1889> 2014-06-09 16:22:12.234201 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1888> 2014-06-09 16:22:12.234202 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1887> 2014-06-09 16:22:12.234205 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc338120 (1 active) -1886> 2014-06-09 16:22:12.234208 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c05a0 (2 active) -1885> 2014-06-09 16:22:12.234213 7f868d75d700 10 op_submit_finish 4287 -1884> 2014-06-09 16:22:12.234216 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -1883> 2014-06-09 16:22:12.234230 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.024258 7 0.000347 -1882> 2014-06-09 16:22:12.234241 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -1881> 2014-06-09 16:22:12.234252 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -1880> 2014-06-09 16:22:12.234262 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -1879> 2014-06-09 16:22:12.234272 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -1878> 2014-06-09 16:22:12.234281 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -1877> 2014-06-09 16:22:12.234288 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -1876> 2014-06-09 16:22:12.234299 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) -1875> 2014-06-09 16:22:12.234310 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) -1874> 2014-06-09 16:22:12.234319 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) -1873> 2014-06-09 16:22:12.234329 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) -1872> 2014-06-09 16:22:12.234339 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -1871> 2014-06-09 16:22:12.234351 7f868d75d700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 244 -1870> 2014-06-09 16:22:12.234354 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.1 172.16.96.201:6811/57862 -- pg_info(1 pgs e243:2.2da) v4 -- ?+0 0x10d02700 -1869> 2014-06-09 16:22:12.234368 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 -1868> 2014-06-09 16:22:12.234399 7f868d75d700 10 log is not dirty -1867> 2014-06-09 16:22:12.234404 7f868d75d700 5 queue_transactions existing osr(2.2da 0x2ed0c10)/0x2ed0c10 -1866> 2014-06-09 16:22:12.234406 7f868d75d700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1865> 2014-06-09 16:22:12.234434 7f86a5e11700 5 _do_op 0x324e7c80 seq 4239 osr(1.48e 0x384d9b0)/0x384d9b0 start -1864> 2014-06-09 16:22:12.234441 7f86a5e11700 10 _do_transaction on 0xbe1b200 -1863> 2014-06-09 16:22:12.234447 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1862> 2014-06-09 16:22:12.234490 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1861> 2014-06-09 16:22:12.234505 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4239.0.0 <= header.spos 150443.0.0 -1860> 2014-06-09 16:22:12.234510 7f86a5e11700 5 submit_transaction r = 0 -1859> 2014-06-09 16:22:12.234521 7f86a5e11700 10 _do_op 0x324e7c80 seq 4239 r = 0, finisher 0x2e0ec20 0 -1858> 2014-06-09 16:22:12.234532 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdf1ba0] -1857> 2014-06-09 16:22:12.234534 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdf1ba0] -1856> 2014-06-09 16:22:12.234535 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1855> 2014-06-09 16:22:12.234536 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1854> 2014-06-09 16:22:12.234540 7f86a5e11700 10 _finish_op 0x324e7c80 seq 4239 osr(1.48e 0x384d9b0)/0x384d9b0 -1853> 2014-06-09 16:22:12.234551 7f868ff62700 10 op_submit_start 4288 -1852> 2014-06-09 16:22:12.234552 7f868ff62700 5 queue_transactions (trailing journal) 4288 -1851> 2014-06-09 16:22:12.234554 7f868ff62700 5 queue_op 0x159975c0 seq 4288 osr(1.5cb 0x384dd70) 828 bytes (queue has 50 ops and 4689415 bytes) -1850> 2014-06-09 16:22:12.234558 7f868af58700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1849> 2014-06-09 16:22:12.234563 7f868f761700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1848> 2014-06-09 16:22:12.234568 7f868df5e700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1847> 2014-06-09 16:22:12.234572 7f868bf5a700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1846> 2014-06-09 16:22:12.234577 7f868c75b700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1845> 2014-06-09 16:22:12.234581 7f8692f68700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1844> 2014-06-09 16:22:12.234586 7f868b759700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1843> 2014-06-09 16:22:12.234590 7f868e75f700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1842> 2014-06-09 16:22:12.234594 7f868cf5c700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1841> 2014-06-09 16:22:12.234599 7f868a757700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1840> 2014-06-09 16:22:12.234603 7f868ef60700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1839> 2014-06-09 16:22:12.234607 7f8690763700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1838> 2014-06-09 16:22:12.234612 7f8690f64700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1837> 2014-06-09 16:22:12.234616 7f8692767700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1836> 2014-06-09 16:22:12.234620 7f8691f66700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1835> 2014-06-09 16:22:12.234624 7f8691765700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1834> 2014-06-09 16:22:12.234628 7f8689755700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1833> 2014-06-09 16:22:12.234632 7f868d75d700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1832> 2014-06-09 16:22:12.234637 7f8689f56700 2 waiting 51 > 50 ops || 4690276 > 104857600 -1831> 2014-06-09 16:22:12.234647 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0ec20] -1830> 2014-06-09 16:22:12.234650 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0ec20] -1829> 2014-06-09 16:22:12.234652 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1828> 2014-06-09 16:22:12.234653 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1827> 2014-06-09 16:22:12.234656 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c05a0 (1 active) -1826> 2014-06-09 16:22:12.234659 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4acb40 (2 active) -1825> 2014-06-09 16:22:12.234665 7f868ff62700 10 op_submit_finish 4288 -1824> 2014-06-09 16:22:12.234668 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1823> 2014-06-09 16:22:12.234683 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -1822> 2014-06-09 16:22:12.234693 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x257c4900,0x10c54f00,0x257c4800,0x11e63200,0x257c4700,0x11e63100,0x257c4600,0x11e63000,0x11e63600,0x11e63500,0x11e63400 -1821> 2014-06-09 16:22:12.234734 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.2 has 1 missing -1820> 2014-06-09 16:22:12.234745 7f868ff62700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery -1819> 2014-06-09 16:22:12.234762 7f868ff62700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1199 -1818> 2014-06-09 16:22:12.234772 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -1817> 2014-06-09 16:22:12.234782 7f868ff62700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -1816> 2014-06-09 16:22:12.234792 7f868ff62700 10 log is not dirty -1815> 2014-06-09 16:22:12.234797 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1814> 2014-06-09 16:22:12.234809 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -1813> 2014-06-09 16:22:12.234818 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -1812> 2014-06-09 16:22:12.234826 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -1811> 2014-06-09 16:22:12.234835 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -1810> 2014-06-09 16:22:12.234843 7f868ff62700 10 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -1809> 2014-06-09 16:22:12.234856 7f868ff62700 15 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 -1808> 2014-06-09 16:22:12.234865 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -1807> 2014-06-09 16:22:12.234873 7f868ff62700 20 osd.26 pg_epoch: 243 pg[2.1b3( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -1806> 2014-06-09 16:22:12.234883 7f868ff62700 10 log is not dirty -1805> 2014-06-09 16:22:12.234887 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.35c( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=11-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.35c( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) -1804> 2014-06-09 16:22:12.234920 7f868ff62700 10 log is not dirty -1803> 2014-06-09 16:22:12.234925 7f868ff62700 5 queue_transactions existing osr(1.35c 0x384d2b0)/0x384d2b0 -1802> 2014-06-09 16:22:12.234927 7f868ff62700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1801> 2014-06-09 16:22:12.234932 7f86a5e11700 5 _do_op 0x324e6120 seq 4240 osr(2.40f 0x2ed1210)/0x2ed1210 start -1800> 2014-06-09 16:22:12.234936 7f86a5e11700 10 _do_transaction on 0xc2d3bc0 -1799> 2014-06-09 16:22:12.234942 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1798> 2014-06-09 16:22:12.234984 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1797> 2014-06-09 16:22:12.235001 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4240.0.0 <= header.spos 150443.0.0 -1796> 2014-06-09 16:22:12.235005 7f86a5e11700 5 submit_transaction r = 0 -1795> 2014-06-09 16:22:12.235020 7f86a5e11700 10 _do_op 0x324e6120 seq 4240 r = 0, finisher 0xccf8ea0 0 -1794> 2014-06-09 16:22:12.235031 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5f620] -1793> 2014-06-09 16:22:12.235033 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -1792> 2014-06-09 16:22:12.235043 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -1791> 2014-06-09 16:22:12.235053 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.40f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 -1790> 2014-06-09 16:22:12.235062 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5f620] -1789> 2014-06-09 16:22:12.235064 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1788> 2014-06-09 16:22:12.235065 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1787> 2014-06-09 16:22:12.235069 7f86a5e11700 10 _finish_op 0x324e6120 seq 4240 osr(2.40f 0x2ed1210)/0x2ed1210 -1786> 2014-06-09 16:22:12.235080 7f868f761700 10 op_submit_start 4289 -1785> 2014-06-09 16:22:12.235082 7f868f761700 5 queue_transactions (trailing journal) 4289 -1784> 2014-06-09 16:22:12.235083 7f868f761700 5 queue_op 0x324e7860 seq 4289 osr(3.2df 0x3ed0e40) 828 bytes (queue has 50 ops and 4689382 bytes) -1783> 2014-06-09 16:22:12.235088 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1782> 2014-06-09 16:22:12.235092 7f868bf5a700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1781> 2014-06-09 16:22:12.235097 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1780> 2014-06-09 16:22:12.235101 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1779> 2014-06-09 16:22:12.235105 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1778> 2014-06-09 16:22:12.235110 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1777> 2014-06-09 16:22:12.235114 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1776> 2014-06-09 16:22:12.235119 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1775> 2014-06-09 16:22:12.235123 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1774> 2014-06-09 16:22:12.235127 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1773> 2014-06-09 16:22:12.235131 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1772> 2014-06-09 16:22:12.235136 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1771> 2014-06-09 16:22:12.235140 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1770> 2014-06-09 16:22:12.235144 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1769> 2014-06-09 16:22:12.235148 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1768> 2014-06-09 16:22:12.235153 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1767> 2014-06-09 16:22:12.235157 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1766> 2014-06-09 16:22:12.235161 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1765> 2014-06-09 16:22:12.235165 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1764> 2014-06-09 16:22:12.235175 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xccf8ea0] -1763> 2014-06-09 16:22:12.235179 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xccf8ea0] -1762> 2014-06-09 16:22:12.235181 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1761> 2014-06-09 16:22:12.235182 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1760> 2014-06-09 16:22:12.235185 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4acb40 (1 active) -1759> 2014-06-09 16:22:12.235188 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653680 (2 active) -1758> 2014-06-09 16:22:12.235193 7f868f761700 10 op_submit_finish 4289 -1757> 2014-06-09 16:22:12.235196 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1756> 2014-06-09 16:22:12.235214 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] check_local -1755> 2014-06-09 16:22:12.235226 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] requeue_ops -1754> 2014-06-09 16:22:12.235236 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing -1753> 2014-06-09 16:22:12.235248 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery -1752> 2014-06-09 16:22:12.235264 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1104 -1751> 2014-06-09 16:22:12.235276 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear -1750> 2014-06-09 16:22:12.235287 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.799( v 214'665 lc 214'664 (0'0,214'665] local-les=243 n=127 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'665 lcod 0'0 mlcod 0'0 active m=1] agent_stop -1749> 2014-06-09 16:22:12.235299 7f868f761700 10 log is not dirty -1748> 2014-06-09 16:22:12.235304 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1747> 2014-06-09 16:22:12.235317 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -1746> 2014-06-09 16:22:12.235326 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x22009f00 -1745> 2014-06-09 16:22:12.235340 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.0 has 2 missing -1744> 2014-06-09 16:22:12.235350 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery -1743> 2014-06-09 16:22:12.235366 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:880 -1742> 2014-06-09 16:22:12.235377 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -1741> 2014-06-09 16:22:12.235386 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.74f( v 213'540 (0'0,213'540] local-les=243 n=107 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -1740> 2014-06-09 16:22:12.235396 7f868f761700 10 log is not dirty -1739> 2014-06-09 16:22:12.235401 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1738> 2014-06-09 16:22:12.235417 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] check_local -1737> 2014-06-09 16:22:12.235427 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] requeue_ops -1736> 2014-06-09 16:22:12.235438 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing -1735> 2014-06-09 16:22:12.235449 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery -1734> 2014-06-09 16:22:12.235465 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:997 -1733> 2014-06-09 16:22:12.235477 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear -1732> 2014-06-09 16:22:12.235488 7f868f761700 20 osd.26 pg_epoch: 243 pg[3.2a5( v 214'635 lc 214'634 (0'0,214'635] local-les=243 n=128 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=214'635 lcod 0'0 mlcod 0'0 active m=1] agent_stop -1731> 2014-06-09 16:22:12.235499 7f868f761700 10 log is not dirty -1730> 2014-06-09 16:22:12.235503 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.2a9( v 213'660 (0'0,213'660] local-les=243 n=130 ec=118 les/c 243/203 242/242/202) -1729> 2014-06-09 16:22:12.235536 7f868f761700 10 log is not dirty -1728> 2014-06-09 16:22:12.235540 7f868f761700 5 queue_transactions existing osr(3.2a9 0x3ed0b40)/0x3ed0b40 -1727> 2014-06-09 16:22:12.235543 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1726> 2014-06-09 16:22:12.235547 7f86a5e11700 5 _do_op 0x2fc8dc80 seq 4241 osr(3.111 0x3ed0830)/0x3ed0830 start -1725> 2014-06-09 16:22:12.235551 7f86a5e11700 10 _do_transaction on 0x344ca80 -1724> 2014-06-09 16:22:12.235556 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1723> 2014-06-09 16:22:12.235598 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1722> 2014-06-09 16:22:12.235618 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4241.0.0 <= header.spos 150443.0.0 -1721> 2014-06-09 16:22:12.235623 7f86a5e11700 5 submit_transaction r = 0 -1720> 2014-06-09 16:22:12.235635 7f86a5e11700 10 _do_op 0x2fc8dc80 seq 4241 r = 0, finisher 0x3251c7a0 0 -1719> 2014-06-09 16:22:12.235644 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2d40] -1718> 2014-06-09 16:22:12.235646 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2d40] -1717> 2014-06-09 16:22:12.235647 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1716> 2014-06-09 16:22:12.235648 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1715> 2014-06-09 16:22:12.235650 7f86a5e11700 10 _finish_op 0x2fc8dc80 seq 4241 osr(3.111 0x3ed0830)/0x3ed0830 -1714> 2014-06-09 16:22:12.235659 7f868bf5a700 10 op_submit_start 4290 -1713> 2014-06-09 16:22:12.235660 7f868bf5a700 5 queue_transactions (trailing journal) 4290 -1712> 2014-06-09 16:22:12.235662 7f868bf5a700 5 queue_op 0xbf7f320 seq 4290 osr(2.566 0x2ed18f0) 828 bytes (queue has 50 ops and 4689382 bytes) -1711> 2014-06-09 16:22:12.235666 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1710> 2014-06-09 16:22:12.235670 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1709> 2014-06-09 16:22:12.235673 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1708> 2014-06-09 16:22:12.235676 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1707> 2014-06-09 16:22:12.235679 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1706> 2014-06-09 16:22:12.235683 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1705> 2014-06-09 16:22:12.235686 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1704> 2014-06-09 16:22:12.235690 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1703> 2014-06-09 16:22:12.235693 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1702> 2014-06-09 16:22:12.235697 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1701> 2014-06-09 16:22:12.235700 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1700> 2014-06-09 16:22:12.235703 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1699> 2014-06-09 16:22:12.235707 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1698> 2014-06-09 16:22:12.235710 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1697> 2014-06-09 16:22:12.235713 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1696> 2014-06-09 16:22:12.235716 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1695> 2014-06-09 16:22:12.235719 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1694> 2014-06-09 16:22:12.235723 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1693> 2014-06-09 16:22:12.235726 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1692> 2014-06-09 16:22:12.235734 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3251c7a0] -1691> 2014-06-09 16:22:12.235740 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3251c7a0] -1690> 2014-06-09 16:22:12.235742 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1689> 2014-06-09 16:22:12.235742 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1688> 2014-06-09 16:22:12.235745 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653680 (1 active) -1687> 2014-06-09 16:22:12.235747 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc115440 (2 active) -1686> 2014-06-09 16:22:12.235751 7f868bf5a700 10 op_submit_finish 4290 -1685> 2014-06-09 16:22:12.235754 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.9f( v 213'792 (0'0,213'792] local-les=243 n=148 ec=118 les/c 243/243 242/242/205) -1684> 2014-06-09 16:22:12.235785 7f868bf5a700 10 log is not dirty -1683> 2014-06-09 16:22:12.235789 7f868bf5a700 5 queue_transactions existing osr(3.9f 0x3ed0600)/0x3ed0600 -1682> 2014-06-09 16:22:12.235792 7f868bf5a700 2 waiting 51 > 50 ops || 4690208 > 104857600 -1681> 2014-06-09 16:22:12.235796 7f86a5e11700 5 _do_op 0x324f5560 seq 4242 osr(1.32f 0x384d350)/0x384d350 start -1680> 2014-06-09 16:22:12.235799 7f86a5e11700 10 _do_transaction on 0xc22a0c0 -1679> 2014-06-09 16:22:12.235804 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1678> 2014-06-09 16:22:12.235842 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1677> 2014-06-09 16:22:12.235854 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4242.0.0 <= header.spos 150443.0.0 -1676> 2014-06-09 16:22:12.235858 7f86a5e11700 5 submit_transaction r = 0 -1675> 2014-06-09 16:22:12.235868 7f86a5e11700 10 _do_op 0x324f5560 seq 4242 r = 0, finisher 0xbd78820 0 -1674> 2014-06-09 16:22:12.235876 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x182cac80] -1673> 2014-06-09 16:22:12.235878 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x182cac80] -1672> 2014-06-09 16:22:12.235879 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1671> 2014-06-09 16:22:12.235880 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1670> 2014-06-09 16:22:12.235883 7f86a5e11700 10 _finish_op 0x324f5560 seq 4242 osr(1.32f 0x384d350)/0x384d350 -1669> 2014-06-09 16:22:12.235892 7f8692f68700 10 op_submit_start 4291 -1668> 2014-06-09 16:22:12.235893 7f8692f68700 5 queue_transactions (trailing journal) 4291 -1667> 2014-06-09 16:22:12.235894 7f8692f68700 5 queue_op 0xcc866c0 seq 4291 osr(0.1da 0x2bd6920) 828 bytes (queue has 50 ops and 4689382 bytes) -1666> 2014-06-09 16:22:12.235898 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1665> 2014-06-09 16:22:12.235901 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1664> 2014-06-09 16:22:12.235904 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1663> 2014-06-09 16:22:12.235907 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1662> 2014-06-09 16:22:12.235911 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1661> 2014-06-09 16:22:12.235914 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1660> 2014-06-09 16:22:12.235918 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1659> 2014-06-09 16:22:12.235921 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1658> 2014-06-09 16:22:12.235924 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1657> 2014-06-09 16:22:12.235927 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1656> 2014-06-09 16:22:12.235930 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1655> 2014-06-09 16:22:12.235934 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1654> 2014-06-09 16:22:12.235937 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1653> 2014-06-09 16:22:12.235940 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1652> 2014-06-09 16:22:12.235944 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1651> 2014-06-09 16:22:12.235947 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1650> 2014-06-09 16:22:12.235950 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1649> 2014-06-09 16:22:12.235953 7f868bf5a700 2 waiting 51 > 50 ops || 4690208 > 104857600 -1648> 2014-06-09 16:22:12.235956 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1647> 2014-06-09 16:22:12.235963 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd78820] -1646> 2014-06-09 16:22:12.235967 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd78820] -1645> 2014-06-09 16:22:12.235970 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1644> 2014-06-09 16:22:12.235971 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1643> 2014-06-09 16:22:12.235973 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc115440 (1 active) -1642> 2014-06-09 16:22:12.235976 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac45a0 (2 active) -1641> 2014-06-09 16:22:12.235979 7f8692f68700 10 op_submit_finish 4291 -1640> 2014-06-09 16:22:12.235983 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1e1( v 213'564 (0'0,213'564] local-les=243 n=115 ec=118 les/c 243/191 242/242/223) -1639> 2014-06-09 16:22:12.236016 7f8692f68700 10 log is not dirty -1638> 2014-06-09 16:22:12.236021 7f8692f68700 5 queue_transactions existing osr(3.1e1 0x3ed09b0)/0x3ed09b0 -1637> 2014-06-09 16:22:12.236023 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1636> 2014-06-09 16:22:12.236028 7f86a5e11700 5 _do_op 0x32539f80 seq 4243 osr(0.4ba 0x2bd7710)/0x2bd7710 start -1635> 2014-06-09 16:22:12.236031 7f86a5e11700 10 _do_transaction on 0xc22ae40 -1634> 2014-06-09 16:22:12.236037 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1633> 2014-06-09 16:22:12.236145 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1632> 2014-06-09 16:22:12.236161 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4243.0.0 <= header.spos 150443.0.0 -1631> 2014-06-09 16:22:12.236165 7f86a5e11700 5 submit_transaction r = 0 -1630> 2014-06-09 16:22:12.236192 7f86a5e11700 10 _do_op 0x32539f80 seq 4243 r = 0, finisher 0x187423c0 0 -1629> 2014-06-09 16:22:12.236204 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc1699c0] -1628> 2014-06-09 16:22:12.236206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc1699c0] -1627> 2014-06-09 16:22:12.236207 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1626> 2014-06-09 16:22:12.236209 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1625> 2014-06-09 16:22:12.236212 7f86a5e11700 10 _finish_op 0x32539f80 seq 4243 osr(0.4ba 0x2bd7710)/0x2bd7710 -1624> 2014-06-09 16:22:12.236223 7f868e75f700 10 op_submit_start 4292 -1623> 2014-06-09 16:22:12.236224 7f868e75f700 5 queue_transactions (trailing journal) 4292 -1622> 2014-06-09 16:22:12.236226 7f868e75f700 5 queue_op 0x324f4de0 seq 4292 osr(2.26a 0x2ed0dd0) 828 bytes (queue has 50 ops and 4689382 bytes) -1621> 2014-06-09 16:22:12.236231 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1620> 2014-06-09 16:22:12.236235 7f868a757700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1619> 2014-06-09 16:22:12.236239 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1618> 2014-06-09 16:22:12.236243 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1617> 2014-06-09 16:22:12.236246 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1616> 2014-06-09 16:22:12.236249 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1615> 2014-06-09 16:22:12.236253 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1614> 2014-06-09 16:22:12.236256 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1613> 2014-06-09 16:22:12.236259 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1612> 2014-06-09 16:22:12.236262 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1611> 2014-06-09 16:22:12.236265 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1610> 2014-06-09 16:22:12.236268 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1609> 2014-06-09 16:22:12.236271 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1608> 2014-06-09 16:22:12.236274 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1607> 2014-06-09 16:22:12.236277 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1606> 2014-06-09 16:22:12.236280 7f868bf5a700 2 waiting 51 > 50 ops || 4690208 > 104857600 -1605> 2014-06-09 16:22:12.236284 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1604> 2014-06-09 16:22:12.236287 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1603> 2014-06-09 16:22:12.236290 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1602> 2014-06-09 16:22:12.236297 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x187423c0] -1601> 2014-06-09 16:22:12.236303 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x187423c0] -1600> 2014-06-09 16:22:12.236305 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1599> 2014-06-09 16:22:12.236305 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1598> 2014-06-09 16:22:12.236308 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac45a0 (1 active) -1597> 2014-06-09 16:22:12.236310 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc300b40 (2 active) -1596> 2014-06-09 16:22:12.236314 7f868e75f700 10 op_submit_finish 4292 -1595> 2014-06-09 16:22:12.236317 7f868e75f700 10 osd.26 pg_epoch: 243 pg[2.33a( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 2.33a( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/222) -1594> 2014-06-09 16:22:12.236347 7f868e75f700 10 log is not dirty -1593> 2014-06-09 16:22:12.236351 7f868e75f700 5 queue_transactions existing osr(2.33a 0x2ed0ed0)/0x2ed0ed0 -1592> 2014-06-09 16:22:12.236354 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1591> 2014-06-09 16:22:12.236357 7f86a5e11700 5 _do_op 0xbe7de60 seq 4244 osr(3.661 0x3ed19f0)/0x3ed19f0 start -1590> 2014-06-09 16:22:12.236361 7f86a5e11700 10 _do_transaction on 0xc238cc0 -1589> 2014-06-09 16:22:12.236366 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1588> 2014-06-09 16:22:12.236403 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1587> 2014-06-09 16:22:12.236417 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4244.0.0 <= header.spos 150443.0.0 -1586> 2014-06-09 16:22:12.236421 7f86a5e11700 5 submit_transaction r = 0 -1585> 2014-06-09 16:22:12.236434 7f86a5e11700 10 _do_op 0xbe7de60 seq 4244 r = 0, finisher 0x3006c0a0 0 -1584> 2014-06-09 16:22:12.236442 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b100] -1583> 2014-06-09 16:22:12.236444 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] _finish_recovery -1582> 2014-06-09 16:22:12.236455 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] purge_strays -1581> 2014-06-09 16:22:12.236465 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.661( v 243'550 (0'0,243'550] local-les=243 n=104 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 213'549 mlcod 213'549 active+clean] publish_stats_to_osd 243:929 -1580> 2014-06-09 16:22:12.236474 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b100] -1579> 2014-06-09 16:22:12.236475 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1578> 2014-06-09 16:22:12.236476 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1577> 2014-06-09 16:22:12.236479 7f86a5e11700 10 _finish_op 0xbe7de60 seq 4244 osr(3.661 0x3ed19f0)/0x3ed19f0 -1576> 2014-06-09 16:22:12.236487 7f868a757700 10 op_submit_start 4293 -1575> 2014-06-09 16:22:12.236488 7f868a757700 5 queue_transactions (trailing journal) 4293 -1574> 2014-06-09 16:22:12.236489 7f868a757700 5 queue_op 0xc64aea0 seq 4293 osr(3.380 0x3ed1020) 861 bytes (queue has 50 ops and 4689382 bytes) -1573> 2014-06-09 16:22:12.236493 7f868ef60700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1572> 2014-06-09 16:22:12.236497 7f8690763700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1571> 2014-06-09 16:22:12.236500 7f8690f64700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1570> 2014-06-09 16:22:12.236503 7f8692767700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1569> 2014-06-09 16:22:12.236506 7f8691f66700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1568> 2014-06-09 16:22:12.236510 7f8691765700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1567> 2014-06-09 16:22:12.236513 7f8689755700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1566> 2014-06-09 16:22:12.236516 7f868d75d700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1565> 2014-06-09 16:22:12.236519 7f8689f56700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1564> 2014-06-09 16:22:12.236522 7f868ff62700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1563> 2014-06-09 16:22:12.236525 7f868af58700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1562> 2014-06-09 16:22:12.236528 7f868f761700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1561> 2014-06-09 16:22:12.236531 7f868df5e700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1560> 2014-06-09 16:22:12.236534 7f868bf5a700 2 waiting 51 > 50 ops || 4690208 > 104857600 -1559> 2014-06-09 16:22:12.236538 7f868c75b700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1558> 2014-06-09 16:22:12.236541 7f8692f68700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1557> 2014-06-09 16:22:12.236544 7f868b759700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1556> 2014-06-09 16:22:12.236547 7f868e75f700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1555> 2014-06-09 16:22:12.236550 7f868cf5c700 2 waiting 51 > 50 ops || 4690243 > 104857600 -1554> 2014-06-09 16:22:12.236557 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006c0a0] -1553> 2014-06-09 16:22:12.236563 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006c0a0] -1552> 2014-06-09 16:22:12.236564 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1551> 2014-06-09 16:22:12.236565 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1550> 2014-06-09 16:22:12.236567 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc300b40 (1 active) -1549> 2014-06-09 16:22:12.236570 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406480 (2 active) -1548> 2014-06-09 16:22:12.236574 7f868a757700 10 op_submit_finish 4293 -1547> 2014-06-09 16:22:12.236578 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1546> 2014-06-09 16:22:12.236593 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -1545> 2014-06-09 16:22:12.236601 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x11e63c00,0x11e63b00,0x11e63a00 -1544> 2014-06-09 16:22:12.236618 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -1543> 2014-06-09 16:22:12.236626 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -1542> 2014-06-09 16:22:12.236634 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -1541> 2014-06-09 16:22:12.236648 7f868a757700 15 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:969 -1540> 2014-06-09 16:22:12.236658 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -1539> 2014-06-09 16:22:12.236666 7f868a757700 20 osd.26 pg_epoch: 243 pg[3.4d7( v 214'562 (0'0,214'562] local-les=243 n=109 ec=118 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -1538> 2014-06-09 16:22:12.236675 7f868a757700 10 log is not dirty -1537> 2014-06-09 16:22:12.236679 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.10d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 1.10d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) -1536> 2014-06-09 16:22:12.236707 7f868a757700 10 log is not dirty -1535> 2014-06-09 16:22:12.236711 7f868a757700 5 queue_transactions existing osr(1.10d 0x384c930)/0x384c930 -1534> 2014-06-09 16:22:12.236713 7f868a757700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1533> 2014-06-09 16:22:12.236717 7f86a5e11700 5 _do_op 0xc2eaf00 seq 4245 osr(2.3de 0x2ed1330)/0x2ed1330 start -1532> 2014-06-09 16:22:12.236720 7f86a5e11700 10 _do_transaction on 0xc0155c0 -1531> 2014-06-09 16:22:12.236725 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1530> 2014-06-09 16:22:12.236762 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1529> 2014-06-09 16:22:12.236777 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4245.0.0 <= header.spos 150443.0.0 -1528> 2014-06-09 16:22:12.236780 7f86a5e11700 5 submit_transaction r = 0 -1527> 2014-06-09 16:22:12.236792 7f86a5e11700 10 _do_op 0xc2eaf00 seq 4245 r = 0, finisher 0x3006c660 0 -1526> 2014-06-09 16:22:12.236801 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x281b6600] -1525> 2014-06-09 16:22:12.236803 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -1524> 2014-06-09 16:22:12.236811 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -1523> 2014-06-09 16:22:12.236819 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.3de( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 -1522> 2014-06-09 16:22:12.236827 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x281b6600] -1521> 2014-06-09 16:22:12.236828 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1520> 2014-06-09 16:22:12.236829 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1519> 2014-06-09 16:22:12.236832 7f86a5e11700 10 _finish_op 0xc2eaf00 seq 4245 osr(2.3de 0x2ed1330)/0x2ed1330 -1518> 2014-06-09 16:22:12.236841 7f8690763700 10 op_submit_start 4294 -1517> 2014-06-09 16:22:12.236842 7f8690763700 5 queue_transactions (trailing journal) 4294 -1516> 2014-06-09 16:22:12.236843 7f8690763700 5 queue_op 0x11fce300 seq 4294 osr(0.648 0x2bd7bd0) 828 bytes (queue has 50 ops and 4689349 bytes) -1515> 2014-06-09 16:22:12.236847 7f8690f64700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1514> 2014-06-09 16:22:12.236851 7f8692767700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1513> 2014-06-09 16:22:12.236854 7f8691f66700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1512> 2014-06-09 16:22:12.236857 7f8691765700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1511> 2014-06-09 16:22:12.236860 7f8689755700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1510> 2014-06-09 16:22:12.236863 7f868d75d700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1509> 2014-06-09 16:22:12.236866 7f8689f56700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1508> 2014-06-09 16:22:12.236870 7f868ff62700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1507> 2014-06-09 16:22:12.236873 7f868af58700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1506> 2014-06-09 16:22:12.236876 7f868f761700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1505> 2014-06-09 16:22:12.236879 7f868df5e700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1504> 2014-06-09 16:22:12.236882 7f868bf5a700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1503> 2014-06-09 16:22:12.236885 7f868c75b700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1502> 2014-06-09 16:22:12.236888 7f8692f68700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1501> 2014-06-09 16:22:12.236891 7f868b759700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1500> 2014-06-09 16:22:12.236894 7f868e75f700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1499> 2014-06-09 16:22:12.236897 7f868cf5c700 2 waiting 51 > 50 ops || 4690210 > 104857600 -1498> 2014-06-09 16:22:12.236900 7f868a757700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1497> 2014-06-09 16:22:12.236903 7f868ef60700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1496> 2014-06-09 16:22:12.236910 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006c660] -1495> 2014-06-09 16:22:12.236917 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006c660] -1494> 2014-06-09 16:22:12.236919 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1493> 2014-06-09 16:22:12.236920 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1492> 2014-06-09 16:22:12.236922 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406480 (1 active) -1491> 2014-06-09 16:22:12.236924 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7c20 (2 active) -1490> 2014-06-09 16:22:12.236928 7f8690763700 10 op_submit_finish 4294 -1489> 2014-06-09 16:22:12.236931 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1488> 2014-06-09 16:22:12.236943 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -1487> 2014-06-09 16:22:12.236950 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -1486> 2014-06-09 16:22:12.236957 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -1485> 2014-06-09 16:22:12.236963 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -1484> 2014-06-09 16:22:12.236970 7f8690763700 10 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -1483> 2014-06-09 16:22:12.236981 7f8690763700 15 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:181 -1482> 2014-06-09 16:22:12.236988 7f8690763700 20 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -1481> 2014-06-09 16:22:12.236994 7f8690763700 20 osd.26 pg_epoch: 243 pg[1.449( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -1480> 2014-06-09 16:22:12.237002 7f8690763700 10 log is not dirty -1479> 2014-06-09 16:22:12.237006 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1478> 2014-06-09 16:22:12.237016 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -1477> 2014-06-09 16:22:12.237022 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -1476> 2014-06-09 16:22:12.237029 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -1475> 2014-06-09 16:22:12.237036 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -1474> 2014-06-09 16:22:12.237041 7f8690763700 10 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -1473> 2014-06-09 16:22:12.237052 7f8690763700 15 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:121 -1472> 2014-06-09 16:22:12.237061 7f8690763700 20 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -1471> 2014-06-09 16:22:12.237068 7f8690763700 20 osd.26 pg_epoch: 243 pg[2.186( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -1470> 2014-06-09 16:22:12.237075 7f8690763700 10 log is not dirty -1469> 2014-06-09 16:22:12.237079 7f8690763700 10 osd.26 pg_epoch: 243 pg[3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.408( v 213'596 (0'0,213'596] local-les=243 n=115 ec=118 les/c 243/191 242/242/223) -1468> 2014-06-09 16:22:12.237108 7f8690763700 10 log is not dirty -1467> 2014-06-09 16:22:12.237112 7f8690763700 5 queue_transactions existing osr(3.408 0x3ed1270)/0x3ed1270 -1466> 2014-06-09 16:22:12.237114 7f8690763700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1465> 2014-06-09 16:22:12.237118 7f86a5e11700 5 _do_op 0xc64b3e0 seq 4246 osr(1.65d 0x384df50)/0x384df50 start -1464> 2014-06-09 16:22:12.237121 7f86a5e11700 10 _do_transaction on 0xc1fd080 -1463> 2014-06-09 16:22:12.237126 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1462> 2014-06-09 16:22:12.237163 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1461> 2014-06-09 16:22:12.237177 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4246.0.0 <= header.spos 150443.0.0 -1460> 2014-06-09 16:22:12.237181 7f86a5e11700 5 submit_transaction r = 0 -1459> 2014-06-09 16:22:12.237194 7f86a5e11700 10 _do_op 0xc64b3e0 seq 4246 r = 0, finisher 0xbd757c0 0 -1458> 2014-06-09 16:22:12.237202 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3205dc00] -1457> 2014-06-09 16:22:12.237205 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -1456> 2014-06-09 16:22:12.237213 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -1455> 2014-06-09 16:22:12.237221 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.65d( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:163 -1454> 2014-06-09 16:22:12.237231 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3205dc00] -1453> 2014-06-09 16:22:12.237232 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1452> 2014-06-09 16:22:12.237233 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1451> 2014-06-09 16:22:12.237236 7f86a5e11700 10 _finish_op 0xc64b3e0 seq 4246 osr(1.65d 0x384df50)/0x384df50 -1450> 2014-06-09 16:22:12.237244 7f8692767700 10 op_submit_start 4295 -1449> 2014-06-09 16:22:12.237246 7f8692767700 5 queue_transactions (trailing journal) 4295 -1448> 2014-06-09 16:22:12.237247 7f8692767700 5 queue_op 0x32539860 seq 4295 osr(3.113 0x3ed0800) 828 bytes (queue has 50 ops and 4689316 bytes) -1447> 2014-06-09 16:22:12.237251 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1446> 2014-06-09 16:22:12.237254 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1445> 2014-06-09 16:22:12.237257 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1444> 2014-06-09 16:22:12.237260 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1443> 2014-06-09 16:22:12.237264 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1442> 2014-06-09 16:22:12.237267 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1441> 2014-06-09 16:22:12.237270 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1440> 2014-06-09 16:22:12.237273 7f868f761700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1439> 2014-06-09 16:22:12.237276 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1438> 2014-06-09 16:22:12.237280 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1437> 2014-06-09 16:22:12.237282 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1436> 2014-06-09 16:22:12.237286 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1435> 2014-06-09 16:22:12.237289 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1434> 2014-06-09 16:22:12.237292 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1433> 2014-06-09 16:22:12.237295 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1432> 2014-06-09 16:22:12.237299 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1431> 2014-06-09 16:22:12.237302 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1430> 2014-06-09 16:22:12.237305 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1429> 2014-06-09 16:22:12.237308 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1428> 2014-06-09 16:22:12.237317 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd757c0] -1427> 2014-06-09 16:22:12.237321 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd757c0] -1426> 2014-06-09 16:22:12.237322 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1425> 2014-06-09 16:22:12.237323 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1424> 2014-06-09 16:22:12.237325 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7c20 (1 active) -1423> 2014-06-09 16:22:12.237328 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc525680 (2 active) -1422> 2014-06-09 16:22:12.237332 7f8692767700 10 op_submit_finish 4295 -1421> 2014-06-09 16:22:12.237335 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1420> 2014-06-09 16:22:12.237347 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] check_local -1419> 2014-06-09 16:22:12.237354 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -1418> 2014-06-09 16:22:12.237361 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -1417> 2014-06-09 16:22:12.237368 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -1416> 2014-06-09 16:22:12.237374 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -1415> 2014-06-09 16:22:12.237385 7f8692767700 15 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:135 -1414> 2014-06-09 16:22:12.237393 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -1413> 2014-06-09 16:22:12.237399 7f8692767700 20 osd.26 pg_epoch: 243 pg[0.1dc( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] agent_stop -1412> 2014-06-09 16:22:12.237407 7f8692767700 10 log is not dirty -1411> 2014-06-09 16:22:12.237411 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1410> 2014-06-09 16:22:12.237421 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -1409> 2014-06-09 16:22:12.237427 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -1408> 2014-06-09 16:22:12.237434 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -1407> 2014-06-09 16:22:12.237442 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -1406> 2014-06-09 16:22:12.237449 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -1405> 2014-06-09 16:22:12.237460 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:194 -1404> 2014-06-09 16:22:12.237468 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -1403> 2014-06-09 16:22:12.237474 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.12d( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -1402> 2014-06-09 16:22:12.237481 7f8692767700 10 log is not dirty -1401> 2014-06-09 16:22:12.237485 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1400> 2014-06-09 16:22:12.237496 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -1399> 2014-06-09 16:22:12.237503 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x10c55400,0x10c55300,0x10c55200,0x10c55100,0x13023600 -1398> 2014-06-09 16:22:12.237523 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -1397> 2014-06-09 16:22:12.237531 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -1396> 2014-06-09 16:22:12.237538 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -1395> 2014-06-09 16:22:12.237549 7f8692767700 15 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1054 -1394> 2014-06-09 16:22:12.237557 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -1393> 2014-06-09 16:22:12.237565 7f8692767700 20 osd.26 pg_epoch: 243 pg[3.67f( v 214'639 (0'0,214'639] local-les=243 n=124 ec=118 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -1392> 2014-06-09 16:22:12.237573 7f8692767700 10 log is not dirty -1391> 2014-06-09 16:22:12.237577 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -1390> 2014-06-09 16:22:12.237587 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] check_local -1389> 2014-06-09 16:22:12.237594 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -1388> 2014-06-09 16:22:12.237600 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -1387> 2014-06-09 16:22:12.237607 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -1386> 2014-06-09 16:22:12.237613 7f8692767700 10 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -1385> 2014-06-09 16:22:12.237623 7f8692767700 15 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:111 -1384> 2014-06-09 16:22:12.237631 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -1383> 2014-06-09 16:22:12.237637 7f8692767700 20 osd.26 pg_epoch: 243 pg[2.75d( empty local-les=243 n=0 ec=1 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=0'0 mlcod 0'0 active] agent_stop -1382> 2014-06-09 16:22:12.237644 7f8692767700 10 log is not dirty -1381> 2014-06-09 16:22:12.237649 7f8692767700 10 osd.26 pg_epoch: 243 pg[3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.105( v 213'565 (0'0,213'565] local-les=243 n=105 ec=118 les/c 243/191 242/242/223) -1380> 2014-06-09 16:22:12.237676 7f8692767700 10 log is not dirty -1379> 2014-06-09 16:22:12.237680 7f8692767700 5 queue_transactions existing osr(3.105 0x3ed0820)/0x3ed0820 -1378> 2014-06-09 16:22:12.237682 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1377> 2014-06-09 16:22:12.237686 7f86a5e11700 5 _do_op 0xc2d5020 seq 4247 osr(0.6b3 0x2bd7ff0)/0x2bd7ff0 start -1376> 2014-06-09 16:22:12.237689 7f86a5e11700 10 _do_transaction on 0x2eaa840 -1375> 2014-06-09 16:22:12.237695 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1374> 2014-06-09 16:22:12.237734 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1373> 2014-06-09 16:22:12.237749 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4247.0.0 <= header.spos 150443.0.0 -1372> 2014-06-09 16:22:12.237754 7f86a5e11700 5 submit_transaction r = 0 -1371> 2014-06-09 16:22:12.237776 7f86a5e11700 10 _do_op 0xc2d5020 seq 4247 r = 0, finisher 0xc169480 0 -1370> 2014-06-09 16:22:12.237788 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32078a20] -1369> 2014-06-09 16:22:12.237790 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -1368> 2014-06-09 16:22:12.237801 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -1367> 2014-06-09 16:22:12.237810 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.6b3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:146 -1366> 2014-06-09 16:22:12.237819 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32078a20] -1365> 2014-06-09 16:22:12.237821 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1364> 2014-06-09 16:22:12.237822 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1363> 2014-06-09 16:22:12.237826 7f86a5e11700 10 _finish_op 0xc2d5020 seq 4247 osr(0.6b3 0x2bd7ff0)/0x2bd7ff0 -1362> 2014-06-09 16:22:12.237838 7f8691765700 10 op_submit_start 4296 -1361> 2014-06-09 16:22:12.237839 7f8691765700 5 queue_transactions (trailing journal) 4296 -1360> 2014-06-09 16:22:12.237841 7f8691765700 5 queue_op 0xc88bf80 seq 4296 osr(3.48d 0x3ed14e0) 828 bytes (queue has 50 ops and 4689283 bytes) -1359> 2014-06-09 16:22:12.237846 7f8689755700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1358> 2014-06-09 16:22:12.237851 7f868d75d700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1357> 2014-06-09 16:22:12.237855 7f8689f56700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1356> 2014-06-09 16:22:12.237860 7f868ff62700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1355> 2014-06-09 16:22:12.237864 7f868af58700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1354> 2014-06-09 16:22:12.237869 7f868f761700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1353> 2014-06-09 16:22:12.237873 7f868df5e700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1352> 2014-06-09 16:22:12.237877 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 -1351> 2014-06-09 16:22:12.237882 7f868c75b700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1350> 2014-06-09 16:22:12.237886 7f8692f68700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1349> 2014-06-09 16:22:12.237890 7f868b759700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1348> 2014-06-09 16:22:12.237894 7f868e75f700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1347> 2014-06-09 16:22:12.237899 7f868cf5c700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1346> 2014-06-09 16:22:12.237903 7f868a757700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1345> 2014-06-09 16:22:12.237907 7f868ef60700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1344> 2014-06-09 16:22:12.237912 7f8690763700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1343> 2014-06-09 16:22:12.237916 7f8690f64700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1342> 2014-06-09 16:22:12.237921 7f8692767700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1341> 2014-06-09 16:22:12.237925 7f8691f66700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1340> 2014-06-09 16:22:12.237934 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc169480] -1339> 2014-06-09 16:22:12.237939 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc169480] -1338> 2014-06-09 16:22:12.237941 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1337> 2014-06-09 16:22:12.237942 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1336> 2014-06-09 16:22:12.237945 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc525680 (1 active) -1335> 2014-06-09 16:22:12.237948 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d10e0 (2 active) -1334> 2014-06-09 16:22:12.237953 7f8691765700 10 op_submit_finish 4296 -1333> 2014-06-09 16:22:12.237957 7f8691765700 10 osd.26 pg_epoch: 243 pg[2.482( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.482( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -1332> 2014-06-09 16:22:12.238060 7f8691765700 10 log is not dirty -1331> 2014-06-09 16:22:12.238065 7f8691765700 5 queue_transactions existing osr(2.482 0x2ed14f0)/0x2ed14f0 -1330> 2014-06-09 16:22:12.238069 7f8691765700 2 waiting 51 > 50 ops || 4690111 > 104857600 -1329> 2014-06-09 16:22:12.238074 7f86a5e11700 5 _do_op 0xd326120 seq 4248 osr(0.448 0x2bd7190)/0x2bd7190 start -1328> 2014-06-09 16:22:12.238078 7f86a5e11700 10 _do_transaction on 0xbf0ea80 -1327> 2014-06-09 16:22:12.238084 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1326> 2014-06-09 16:22:12.238127 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1325> 2014-06-09 16:22:12.238142 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4248.0.0 <= header.spos 150443.0.0 -1324> 2014-06-09 16:22:12.238146 7f86a5e11700 5 submit_transaction r = 0 -1323> 2014-06-09 16:22:12.238159 7f86a5e11700 10 _do_op 0xd326120 seq 4248 r = 0, finisher 0x30628960 0 -1322> 2014-06-09 16:22:12.238170 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe677e0] -1321> 2014-06-09 16:22:12.238172 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe677e0] -1320> 2014-06-09 16:22:12.238174 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1319> 2014-06-09 16:22:12.238175 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1318> 2014-06-09 16:22:12.238178 7f86a5e11700 10 _finish_op 0xd326120 seq 4248 osr(0.448 0x2bd7190)/0x2bd7190 -1317> 2014-06-09 16:22:12.238189 7f868d75d700 10 op_submit_start 4297 -1316> 2014-06-09 16:22:12.238191 7f868d75d700 5 queue_transactions (trailing journal) 4297 -1315> 2014-06-09 16:22:12.238192 7f868d75d700 5 queue_op 0xc2d4c60 seq 4297 osr(2.2da 0x2ed0c10) 861 bytes (queue has 50 ops and 4689316 bytes) -1314> 2014-06-09 16:22:12.238197 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1313> 2014-06-09 16:22:12.238202 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1312> 2014-06-09 16:22:12.238206 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1311> 2014-06-09 16:22:12.238211 7f868f761700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1310> 2014-06-09 16:22:12.238215 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1309> 2014-06-09 16:22:12.238220 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1308> 2014-06-09 16:22:12.238224 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1307> 2014-06-09 16:22:12.238228 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1306> 2014-06-09 16:22:12.238232 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1305> 2014-06-09 16:22:12.238237 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1304> 2014-06-09 16:22:12.238242 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1303> 2014-06-09 16:22:12.238247 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1302> 2014-06-09 16:22:12.238251 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1301> 2014-06-09 16:22:12.238256 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1300> 2014-06-09 16:22:12.238260 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1299> 2014-06-09 16:22:12.238264 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1298> 2014-06-09 16:22:12.238269 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1297> 2014-06-09 16:22:12.238273 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1296> 2014-06-09 16:22:12.238277 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1295> 2014-06-09 16:22:12.238287 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30628960] -1294> 2014-06-09 16:22:12.238292 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30628960] -1293> 2014-06-09 16:22:12.238293 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1292> 2014-06-09 16:22:12.238294 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1291> 2014-06-09 16:22:12.238298 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d10e0 (1 active) -1290> 2014-06-09 16:22:12.238301 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd8c0 (2 active) -1289> 2014-06-09 16:22:12.238306 7f868d75d700 10 op_submit_finish 4297 -1288> 2014-06-09 16:22:12.238310 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -1287> 2014-06-09 16:22:12.238325 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.007867 7 0.000379 -1286> 2014-06-09 16:22:12.238339 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -1285> 2014-06-09 16:22:12.238351 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -1284> 2014-06-09 16:22:12.238363 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000024 0 0.000000 -1283> 2014-06-09 16:22:12.238374 7f868d75d700 5 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -1282> 2014-06-09 16:22:12.238384 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -1281> 2014-06-09 16:22:12.238391 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -1280> 2014-06-09 16:22:12.238405 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -1279> 2014-06-09 16:22:12.238417 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -1278> 2014-06-09 16:22:12.238427 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -1277> 2014-06-09 16:22:12.238443 7f868d75d700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 -1276> 2014-06-09 16:22:12.238446 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:2.446) v4 -- ?+0 0x2a08a540 -1275> 2014-06-09 16:22:12.238463 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.446( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:130 -1274> 2014-06-09 16:22:12.238497 7f868d75d700 10 log is not dirty -1273> 2014-06-09 16:22:12.238501 7f868d75d700 5 queue_transactions existing osr(2.446 0x2ed1550)/0x2ed1550 -1272> 2014-06-09 16:22:12.238504 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1271> 2014-06-09 16:22:12.238544 7f86a5e11700 5 _do_op 0x324e7bc0 seq 4249 osr(1.66e 0x384ded0)/0x384ded0 start -1270> 2014-06-09 16:22:12.238554 7f86a5e11700 10 _do_transaction on 0xc23ee40 -1269> 2014-06-09 16:22:12.238560 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1268> 2014-06-09 16:22:12.238605 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1267> 2014-06-09 16:22:12.238620 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4249.0.0 <= header.spos 150443.0.0 -1266> 2014-06-09 16:22:12.238624 7f86a5e11700 5 submit_transaction r = 0 -1265> 2014-06-09 16:22:12.238636 7f86a5e11700 10 _do_op 0x324e7bc0 seq 4249 r = 0, finisher 0xbd5fe80 0 -1264> 2014-06-09 16:22:12.238647 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x30509bc0] -1263> 2014-06-09 16:22:12.238649 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x30509bc0] -1262> 2014-06-09 16:22:12.238651 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1261> 2014-06-09 16:22:12.238652 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1260> 2014-06-09 16:22:12.238655 7f86a5e11700 10 _finish_op 0x324e7bc0 seq 4249 osr(1.66e 0x384ded0)/0x384ded0 -1259> 2014-06-09 16:22:12.238667 7f868ff62700 10 op_submit_start 4298 -1258> 2014-06-09 16:22:12.238668 7f868ff62700 5 queue_transactions (trailing journal) 4298 -1257> 2014-06-09 16:22:12.238670 7f868ff62700 5 queue_op 0xbf7f020 seq 4298 osr(1.35c 0x384d2b0) 828 bytes (queue has 50 ops and 4689316 bytes) -1256> 2014-06-09 16:22:12.238675 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1255> 2014-06-09 16:22:12.238680 7f868f761700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1254> 2014-06-09 16:22:12.238684 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1253> 2014-06-09 16:22:12.238689 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1252> 2014-06-09 16:22:12.238693 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1251> 2014-06-09 16:22:12.238697 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1250> 2014-06-09 16:22:12.238702 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1249> 2014-06-09 16:22:12.238706 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1248> 2014-06-09 16:22:12.238711 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1247> 2014-06-09 16:22:12.238715 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1246> 2014-06-09 16:22:12.238719 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1245> 2014-06-09 16:22:12.238724 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1244> 2014-06-09 16:22:12.238729 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1243> 2014-06-09 16:22:12.238733 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1242> 2014-06-09 16:22:12.238737 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1241> 2014-06-09 16:22:12.238741 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1240> 2014-06-09 16:22:12.238746 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1239> 2014-06-09 16:22:12.238750 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1238> 2014-06-09 16:22:12.238756 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1237> 2014-06-09 16:22:12.238762 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5fe80] -1236> 2014-06-09 16:22:12.238766 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5fe80] -1235> 2014-06-09 16:22:12.238768 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1234> 2014-06-09 16:22:12.238769 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1233> 2014-06-09 16:22:12.238773 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd8c0 (1 active) -1232> 2014-06-09 16:22:12.238775 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3385a0 (2 active) -1231> 2014-06-09 16:22:12.238779 7f86a5e11700 5 _do_op 0x324dbb00 seq 4250 osr(3.251 0x3ed0c20)/0x3ed0c20 start -1230> 2014-06-09 16:22:12.238781 7f868ff62700 10 op_submit_finish 4298 -1229> 2014-06-09 16:22:12.238782 7f86a5e11700 10 _do_transaction on 0xc29cd80 -1228> 2014-06-09 16:22:12.238788 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1227> 2014-06-09 16:22:12.238785 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.233( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/20 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 1.233( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) -1226> 2014-06-09 16:22:12.238814 7f868ff62700 10 log is not dirty -1225> 2014-06-09 16:22:12.238818 7f868ff62700 5 queue_transactions existing osr(1.233 0x384cc30)/0x384cc30 -1224> 2014-06-09 16:22:12.238820 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1223> 2014-06-09 16:22:12.238830 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1222> 2014-06-09 16:22:12.238844 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4250.0.0 <= header.spos 150443.0.0 -1221> 2014-06-09 16:22:12.238849 7f86a5e11700 5 submit_transaction r = 0 -1220> 2014-06-09 16:22:12.238860 7f86a5e11700 10 _do_op 0x324dbb00 seq 4250 r = 0, finisher 0x1b0c13c0 0 -1219> 2014-06-09 16:22:12.238866 7f86a5e11700 10 _finish_op 0x324dbb00 seq 4250 osr(3.251 0x3ed0c20)/0x3ed0c20 -1218> 2014-06-09 16:22:12.238867 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5f880] -1217> 2014-06-09 16:22:12.238869 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5f880] -1216> 2014-06-09 16:22:12.238870 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1215> 2014-06-09 16:22:12.238871 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1214> 2014-06-09 16:22:12.238879 7f868f761700 10 op_submit_start 4299 -1213> 2014-06-09 16:22:12.238881 7f868f761700 5 queue_transactions (trailing journal) 4299 -1212> 2014-06-09 16:22:12.238882 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1b0c13c0] -1211> 2014-06-09 16:22:12.238882 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1210> 2014-06-09 16:22:12.238883 7f868f761700 5 queue_op 0xc2d56e0 seq 4299 osr(3.2a9 0x3ed0b40) 828 bytes (queue has 50 ops and 4689316 bytes) -1209> 2014-06-09 16:22:12.238887 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1b0c13c0] -1208> 2014-06-09 16:22:12.238889 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1207> 2014-06-09 16:22:12.238889 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1206> 2014-06-09 16:22:12.238890 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1205> 2014-06-09 16:22:12.238893 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1204> 2014-06-09 16:22:12.238893 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3385a0 (1 active) -1203> 2014-06-09 16:22:12.238896 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f9e0 (2 active) -1202> 2014-06-09 16:22:12.238899 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1201> 2014-06-09 16:22:12.238900 7f86a5e11700 5 _do_op 0x2fc8c120 seq 4251 osr(3.5f8 0x3ed1b30)/0x3ed1b30 start -1200> 2014-06-09 16:22:12.238902 7f868f761700 10 op_submit_finish 4299 -1199> 2014-06-09 16:22:12.238903 7f86a5e11700 10 _do_transaction on 0xbf50f00 -1198> 2014-06-09 16:22:12.238904 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1197> 2014-06-09 16:22:12.238908 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1196> 2014-06-09 16:22:12.238912 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1195> 2014-06-09 16:22:12.238915 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1194> 2014-06-09 16:22:12.238906 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -1193> 2014-06-09 16:22:12.238917 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1192> 2014-06-09 16:22:12.238919 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1191> 2014-06-09 16:22:12.238923 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1190> 2014-06-09 16:22:12.238927 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1189> 2014-06-09 16:22:12.238931 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1188> 2014-06-09 16:22:12.238934 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1187> 2014-06-09 16:22:12.238938 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1186> 2014-06-09 16:22:12.238940 7f868f761700 10 log is not dirty -1185> 2014-06-09 16:22:12.238942 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1184> 2014-06-09 16:22:12.238944 7f868f761700 5 queue_transactions existing osr(0.f 0x2bd60e0)/0x2bd60e0 -1183> 2014-06-09 16:22:12.238946 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1182> 2014-06-09 16:22:12.238950 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1181> 2014-06-09 16:22:12.238955 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1180> 2014-06-09 16:22:12.238958 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1179> 2014-06-09 16:22:12.238960 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1178> 2014-06-09 16:22:12.238965 7f868f761700 2 waiting 51 > 50 ops || 4690140 > 104857600 -1177> 2014-06-09 16:22:12.238973 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4251.0.0 <= header.spos 150443.0.0 -1176> 2014-06-09 16:22:12.238977 7f86a5e11700 5 submit_transaction r = 0 -1175> 2014-06-09 16:22:12.238989 7f86a5e11700 10 _do_op 0x2fc8c120 seq 4251 r = 0, finisher 0x31434000 0 -1174> 2014-06-09 16:22:12.238996 7f86a5e11700 10 _finish_op 0x2fc8c120 seq 4251 osr(3.5f8 0x3ed1b30)/0x3ed1b30 -1173> 2014-06-09 16:22:12.238996 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc410440] -1172> 2014-06-09 16:22:12.238998 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc410440] -1171> 2014-06-09 16:22:12.238999 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1170> 2014-06-09 16:22:12.239000 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1169> 2014-06-09 16:22:12.239003 7f868df5e700 10 op_submit_start 4300 -1168> 2014-06-09 16:22:12.239004 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1167> 2014-06-09 16:22:12.239005 7f868df5e700 5 queue_transactions (trailing journal) 4300 -1166> 2014-06-09 16:22:12.239008 7f868c75b700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1165> 2014-06-09 16:22:12.239006 7f868df5e700 5 queue_op 0xcca7560 seq 4300 osr(3.400 0x3ed1260) 828 bytes (queue has 50 ops and 4689316 bytes) -1164> 2014-06-09 16:22:12.239008 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31434000] -1163> 2014-06-09 16:22:12.239011 7f8692f68700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1162> 2014-06-09 16:22:12.239012 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31434000] -1161> 2014-06-09 16:22:12.239014 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1160> 2014-06-09 16:22:12.239015 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1159> 2014-06-09 16:22:12.239015 7f868b759700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1158> 2014-06-09 16:22:12.239018 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f9e0 (1 active) -1157> 2014-06-09 16:22:12.239020 7f868e75f700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1156> 2014-06-09 16:22:12.239021 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc358000 (2 active) -1155> 2014-06-09 16:22:12.239025 7f868cf5c700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1154> 2014-06-09 16:22:12.239025 7f86a5e11700 5 _do_op 0x324f53e0 seq 4252 osr(0.31f 0x2bd6d50)/0x2bd6d50 start -1153> 2014-06-09 16:22:12.239027 7f868df5e700 10 op_submit_finish 4300 -1152> 2014-06-09 16:22:12.239028 7f86a5e11700 10 _do_transaction on 0xc2f4e40 -1151> 2014-06-09 16:22:12.239034 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1150> 2014-06-09 16:22:12.239032 7f868df5e700 10 osd.26 pg_epoch: 243 pg[2.101( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.101( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -1149> 2014-06-09 16:22:12.239056 7f868df5e700 10 log is not dirty -1148> 2014-06-09 16:22:12.239060 7f868df5e700 5 queue_transactions existing osr(2.101 0x2ed0730)/0x2ed0730 -1147> 2014-06-09 16:22:12.239061 7f868df5e700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1146> 2014-06-09 16:22:12.239064 7f868a757700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1145> 2014-06-09 16:22:12.239068 7f868ef60700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1144> 2014-06-09 16:22:12.239074 7f8690763700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1143> 2014-06-09 16:22:12.239073 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1142> 2014-06-09 16:22:12.239078 7f8690f64700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1141> 2014-06-09 16:22:12.239083 7f8692767700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1140> 2014-06-09 16:22:12.239086 7f8691f66700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1139> 2014-06-09 16:22:12.239088 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4252.0.0 <= header.spos 150443.0.0 -1138> 2014-06-09 16:22:12.239091 7f8691765700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1137> 2014-06-09 16:22:12.239092 7f86a5e11700 5 submit_transaction r = 0 -1136> 2014-06-09 16:22:12.239095 7f8689755700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1135> 2014-06-09 16:22:12.239100 7f868d75d700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1134> 2014-06-09 16:22:12.239104 7f8689f56700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1133> 2014-06-09 16:22:12.239109 7f868ff62700 2 waiting 51 > 50 ops || 4690144 > 104857600 -1132> 2014-06-09 16:22:12.239113 7f868af58700 2 waiting 51 > 50 ops || 4690177 > 104857600 -1131> 2014-06-09 16:22:12.239113 7f86a5e11700 10 _do_op 0x324f53e0 seq 4252 r = 0, finisher 0x314ad1c0 0 -1130> 2014-06-09 16:22:12.239117 7f868f761700 2 waiting 51 > 50 ops || 4690140 > 104857600 -1129> 2014-06-09 16:22:12.239120 7f86a5e11700 10 _finish_op 0x324f53e0 seq 4252 osr(0.31f 0x2bd6d50)/0x2bd6d50 -1128> 2014-06-09 16:22:12.239121 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94740] -1127> 2014-06-09 16:22:12.239123 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94740] -1126> 2014-06-09 16:22:12.239124 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1125> 2014-06-09 16:22:12.239125 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1124> 2014-06-09 16:22:12.239128 7f868bf5a700 10 op_submit_start 4301 -1123> 2014-06-09 16:22:12.239128 7f868c75b700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1122> 2014-06-09 16:22:12.239129 7f868bf5a700 5 queue_transactions (trailing journal) 4301 -1121> 2014-06-09 16:22:12.239131 7f868bf5a700 5 queue_op 0x3280a000 seq 4301 osr(3.9f 0x3ed0600) 826 bytes (queue has 50 ops and 4689314 bytes) -1120> 2014-06-09 16:22:12.239132 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x314ad1c0] -1119> 2014-06-09 16:22:12.239135 7f8692f68700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1118> 2014-06-09 16:22:12.239138 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x314ad1c0] -1117> 2014-06-09 16:22:12.239140 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1116> 2014-06-09 16:22:12.239139 7f868b759700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1115> 2014-06-09 16:22:12.239141 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1114> 2014-06-09 16:22:12.239144 7f868e75f700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1113> 2014-06-09 16:22:12.239144 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc358000 (1 active) -1112> 2014-06-09 16:22:12.239147 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287d40 (2 active) -1111> 2014-06-09 16:22:12.239148 7f868cf5c700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1110> 2014-06-09 16:22:12.239151 7f86a5e11700 5 _do_op 0x32539f20 seq 4253 osr(3.52a 0x3ed1670)/0x3ed1670 start -1109> 2014-06-09 16:22:12.239153 7f868bf5a700 10 op_submit_finish 4301 -1108> 2014-06-09 16:22:12.239154 7f86a5e11700 10 _do_transaction on 0xc2af740 -1107> 2014-06-09 16:22:12.239160 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1106> 2014-06-09 16:22:12.239156 7f868bf5a700 10 osd.26 pg_epoch: 243 pg[2.5a4( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 2.5a4( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) -1105> 2014-06-09 16:22:12.239178 7f868bf5a700 10 log is not dirty -1104> 2014-06-09 16:22:12.239181 7f868bf5a700 5 queue_transactions existing osr(2.5a4 0x2ed1810)/0x2ed1810 -1103> 2014-06-09 16:22:12.239183 7f868bf5a700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1102> 2014-06-09 16:22:12.239186 7f868df5e700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1101> 2014-06-09 16:22:12.239189 7f868a757700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1100> 2014-06-09 16:22:12.239195 7f868ef60700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1099> 2014-06-09 16:22:12.239199 7f8690763700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1098> 2014-06-09 16:22:12.239200 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1097> 2014-06-09 16:22:12.239203 7f8690f64700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1096> 2014-06-09 16:22:12.239207 7f8692767700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1095> 2014-06-09 16:22:12.239212 7f8691f66700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1094> 2014-06-09 16:22:12.239214 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4253.0.0 <= header.spos 150443.0.0 -1093> 2014-06-09 16:22:12.239217 7f8691765700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1092> 2014-06-09 16:22:12.239218 7f86a5e11700 5 submit_transaction r = 0 -1091> 2014-06-09 16:22:12.239221 7f8689755700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1090> 2014-06-09 16:22:12.239225 7f868d75d700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1089> 2014-06-09 16:22:12.239230 7f8689f56700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1088> 2014-06-09 16:22:12.239229 7f86a5e11700 10 _do_op 0x32539f20 seq 4253 r = 0, finisher 0xbd7da60 0 -1087> 2014-06-09 16:22:12.239233 7f868ff62700 2 waiting 51 > 50 ops || 4690142 > 104857600 -1086> 2014-06-09 16:22:12.239236 7f86a5e11700 10 _finish_op 0x32539f20 seq 4253 osr(3.52a 0x3ed1670)/0x3ed1670 -1085> 2014-06-09 16:22:12.239237 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3045a3c0] -1084> 2014-06-09 16:22:12.239238 7f868af58700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1083> 2014-06-09 16:22:12.239240 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3045a3c0] -1082> 2014-06-09 16:22:12.239241 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1081> 2014-06-09 16:22:12.239242 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1080> 2014-06-09 16:22:12.239245 7f868f761700 2 waiting 51 > 50 ops || 4690138 > 104857600 -1079> 2014-06-09 16:22:12.239612 7f868c75b700 10 op_submit_start 4302 -1078> 2014-06-09 16:22:12.239612 7f8692f68700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1077> 2014-06-09 16:22:12.239621 7f868c75b700 5 queue_transactions (trailing journal) 4302 -1076> 2014-06-09 16:22:12.239625 7f868e75f700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1075> 2014-06-09 16:22:12.239623 7f868c75b700 5 queue_op 0x2e21380 seq 4302 osr(3.224 0x3ed0c80) 861 bytes (queue has 50 ops and 4689347 bytes) -1074> 2014-06-09 16:22:12.239629 7f868cf5c700 2 waiting 51 > 50 ops || 4690208 > 104857600 -1073> 2014-06-09 16:22:12.239633 7f868bf5a700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1072> 2014-06-09 16:22:12.239633 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7da60] -1071> 2014-06-09 16:22:12.239639 7f868df5e700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1070> 2014-06-09 16:22:12.239641 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7da60] -1069> 2014-06-09 16:22:12.239643 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1068> 2014-06-09 16:22:12.239644 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1067> 2014-06-09 16:22:12.239645 7f868a757700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1066> 2014-06-09 16:22:12.239647 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287d40 (1 active) -1065> 2014-06-09 16:22:12.239649 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc923e60 (2 active) -1064> 2014-06-09 16:22:12.239650 7f868ef60700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1063> 2014-06-09 16:22:12.239653 7f86a5e11700 5 _do_op 0x30060900 seq 4254 osr(2.5bf 0x2ed1b10)/0x2ed1b10 start -1062> 2014-06-09 16:22:12.239655 7f868c75b700 10 op_submit_finish 4302 -1061> 2014-06-09 16:22:12.239655 7f86a5e11700 10 _do_transaction on 0xc2c6900 -1060> 2014-06-09 16:22:12.239655 7f8690763700 2 waiting 51 > 50 ops || 4690175 > 104857600 -1059> 2014-06-09 16:22:12.239661 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1058> 2014-06-09 16:22:12.239660 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1057> 2014-06-09 16:22:12.239674 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] requeue_ops -1056> 2014-06-09 16:22:12.239682 7f868c75b700 10 log is not dirty -1055> 2014-06-09 16:22:12.239687 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1054> 2014-06-09 16:22:12.239694 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.3aa( empty local-les=243 n=0 ec=1 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops -1053> 2014-06-09 16:22:12.239699 7f868c75b700 10 log is not dirty -1052> 2014-06-09 16:22:12.239699 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1051> 2014-06-09 16:22:12.239702 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1050> 2014-06-09 16:22:12.239709 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.409( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -1049> 2014-06-09 16:22:12.239714 7f868c75b700 10 log is not dirty -1048> 2014-06-09 16:22:12.239714 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4254.0.0 <= header.spos 150443.0.0 -1047> 2014-06-09 16:22:12.239718 7f86a5e11700 5 submit_transaction r = 0 -1046> 2014-06-09 16:22:12.239716 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1045> 2014-06-09 16:22:12.239723 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.47d( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -1044> 2014-06-09 16:22:12.239728 7f868c75b700 10 log is not dirty -1043> 2014-06-09 16:22:12.239732 7f86a5e11700 10 _do_op 0x30060900 seq 4254 r = 0, finisher 0x17001c80 0 -1042> 2014-06-09 16:22:12.239730 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1041> 2014-06-09 16:22:12.239737 7f86a5e11700 10 _finish_op 0x30060900 seq 4254 osr(2.5bf 0x2ed1b10)/0x2ed1b10 -1040> 2014-06-09 16:22:12.239739 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfce060] -1039> 2014-06-09 16:22:12.239737 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.77c( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -1038> 2014-06-09 16:22:12.239745 7f868c75b700 10 log is not dirty -1037> 2014-06-09 16:22:12.239746 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x17001c80] -1036> 2014-06-09 16:22:12.239750 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x17001c80] -1035> 2014-06-09 16:22:12.239742 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -1034> 2014-06-09 16:22:12.239752 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1033> 2014-06-09 16:22:12.239752 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1032> 2014-06-09 16:22:12.239755 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc923e60 (1 active) -1031> 2014-06-09 16:22:12.239749 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1030> 2014-06-09 16:22:12.239752 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -1029> 2014-06-09 16:22:12.239757 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d17a0 (2 active) -1028> 2014-06-09 16:22:12.239759 7f86a5e11700 5 _do_op 0xc2eb980 seq 4255 osr(2.6f4 0x2ed1e20)/0x2ed1e20 start -1027> 2014-06-09 16:22:12.239762 7f86a5e11700 10 _do_transaction on 0x39db440 -1026> 2014-06-09 16:22:12.239760 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.493( empty local-les=243 n=0 ec=1 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=67-241/16 crt=0'0 active] requeue_ops -1025> 2014-06-09 16:22:12.239760 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.5bf( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 -1024> 2014-06-09 16:22:12.239767 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1023> 2014-06-09 16:22:12.239768 7f868c75b700 10 log is not dirty -1022> 2014-06-09 16:22:12.239769 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfce060] -1021> 2014-06-09 16:22:12.239773 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1020> 2014-06-09 16:22:12.239774 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1019> 2014-06-09 16:22:12.239772 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1018> 2014-06-09 16:22:12.239784 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.52b( v 213'697 (0'0,213'697] local-les=243 n=131 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -1017> 2014-06-09 16:22:12.239791 7f868c75b700 10 log is not dirty -1016> 2014-06-09 16:22:12.239794 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1015> 2014-06-09 16:22:12.239802 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1014> 2014-06-09 16:22:12.239801 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.6b6( empty local-les=243 n=0 ec=1 les/c 201/201 242/242/200) [3,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -1013> 2014-06-09 16:22:12.239806 7f868c75b700 10 log is not dirty -1012> 2014-06-09 16:22:12.239808 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1011> 2014-06-09 16:22:12.239816 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4255.0.0 <= header.spos 150443.0.0 -1010> 2014-06-09 16:22:12.239815 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -1009> 2014-06-09 16:22:12.239820 7f868c75b700 10 log is not dirty -1008> 2014-06-09 16:22:12.239820 7f86a5e11700 5 submit_transaction r = 0 -1007> 2014-06-09 16:22:12.239823 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -1006> 2014-06-09 16:22:12.239829 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.26c( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -1005> 2014-06-09 16:22:12.239832 7f86a5e11700 10 _do_op 0xc2eb980 seq 4255 r = 0, finisher 0x3042b340 0 -1004> 2014-06-09 16:22:12.239834 7f868c75b700 10 log is not dirty -1003> 2014-06-09 16:22:12.239838 7f86a5e11700 10 _finish_op 0xc2eb980 seq 4255 osr(2.6f4 0x2ed1e20)/0x2ed1e20 -1002> 2014-06-09 16:22:12.239839 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5b2e0] -1001> 2014-06-09 16:22:12.239845 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3042b340] -1000> 2014-06-09 16:22:12.239837 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -999> 2014-06-09 16:22:12.239841 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -998> 2014-06-09 16:22:12.239849 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -997> 2014-06-09 16:22:12.239854 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3042b340] -996> 2014-06-09 16:22:12.239855 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -995> 2014-06-09 16:22:12.239856 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -994> 2014-06-09 16:22:12.239849 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.6bb( v 213'619 (0'0,213'619] local-les=243 n=116 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -993> 2014-06-09 16:22:12.239859 7f868c75b700 10 log is not dirty -992> 2014-06-09 16:22:12.239858 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d17a0 (1 active) -991> 2014-06-09 16:22:12.239861 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fdc20 (2 active) -990> 2014-06-09 16:22:12.239857 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.6f4( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:113 -989> 2014-06-09 16:22:12.239863 7f86a5e11700 5 _do_op 0xc88b680 seq 4256 osr(1.54a 0x384daf0)/0x384daf0 start -988> 2014-06-09 16:22:12.239865 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5b2e0] -987> 2014-06-09 16:22:12.239865 7f86a5e11700 10 _do_transaction on 0xc2bda40 -986> 2014-06-09 16:22:12.239866 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -985> 2014-06-09 16:22:12.239867 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -984> 2014-06-09 16:22:12.239869 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -983> 2014-06-09 16:22:12.239863 7f868c75b700 10 osd.26 pg_epoch: 243 pg[2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -982> 2014-06-09 16:22:12.239874 7f868c75b700 15 osd.26 pg_epoch: 243 pg[2.492( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -981> 2014-06-09 16:22:12.239879 7f868c75b700 10 log is not dirty -980> 2014-06-09 16:22:12.239882 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -979> 2014-06-09 16:22:12.239889 7f868c75b700 15 osd.26 pg_epoch: 243 pg[0.2c6( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] requeue_ops -978> 2014-06-09 16:22:12.239895 7f868c75b700 10 log is not dirty -977> 2014-06-09 16:22:12.239897 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -976> 2014-06-09 16:22:12.239903 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.6b6( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] requeue_ops -975> 2014-06-09 16:22:12.239908 7f868c75b700 10 log is not dirty -974> 2014-06-09 16:22:12.239910 7f868c75b700 10 osd.26 pg_epoch: 243 pg[1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -973> 2014-06-09 16:22:12.239917 7f868c75b700 15 osd.26 pg_epoch: 243 pg[1.582( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] requeue_ops -972> 2014-06-09 16:22:12.239922 7f868c75b700 10 log is not dirty -971> 2014-06-09 16:22:12.239924 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -970> 2014-06-09 16:22:12.239933 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.f7( v 213'589 (0'0,213'589] local-les=243 n=116 ec=118 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -969> 2014-06-09 16:22:12.239939 7f868c75b700 10 log is not dirty -968> 2014-06-09 16:22:12.239941 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -967> 2014-06-09 16:22:12.239950 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.72c( v 213'558 (0'0,213'558] local-les=243 n=112 ec=118 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] requeue_ops -966> 2014-06-09 16:22:12.239956 7f868c75b700 10 log is not dirty -965> 2014-06-09 16:22:12.239965 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -964> 2014-06-09 16:22:12.239977 7f868c75b700 15 OSD::op_tp worker wq OSD::PeeringWQ done processing 0xcb41050 (19 active) -963> 2014-06-09 16:22:12.239978 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4256.0.0 <= header.spos 150443.0.0 -962> 2014-06-09 16:22:12.239982 7f86a5e11700 5 submit_transaction r = 0 -961> 2014-06-09 16:22:12.239985 7f868c75b700 12 OSD::op_tp worker wq OSD::OpWQ start processing 0x1 (20 active) -960> 2014-06-09 16:22:12.239989 7f868c75b700 10 osd.26 244 dequeue_op 0x257c4900 prio 63 cost 217088 latency 1.456234 osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 pg pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] -959> 2014-06-09 16:22:12.240007 7f868c75b700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:12.240003, event: reached_pg, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) -958> 2014-06-09 16:22:12.240011 7f86a5e11700 10 _do_op 0xc88b680 seq 4256 r = 0, finisher 0x12051880 0 -957> 2014-06-09 16:22:12.240017 7f86a5e11700 10 _finish_op 0xc88b680 seq 4256 osr(1.54a 0x384daf0)/0x384daf0 -956> 2014-06-09 16:22:12.240019 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324d1760] -955> 2014-06-09 16:22:12.240014 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] op_has_sufficient_caps pool=3 (ceph ) owner=0 need_read_cap=0 need_write_cap=1 need_class_read_cap=0 need_class_write_cap=0 -> yes -954> 2014-06-09 16:22:12.240024 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x12051880] -953> 2014-06-09 16:22:12.240027 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324d1760] -952> 2014-06-09 16:22:12.240028 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -951> 2014-06-09 16:22:12.240028 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x12051880] -950> 2014-06-09 16:22:12.240029 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -949> 2014-06-09 16:22:12.240030 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -948> 2014-06-09 16:22:12.240031 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -947> 2014-06-09 16:22:12.240033 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fdc20 (1 active) -946> 2014-06-09 16:22:12.240026 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_message: 0x257c4900 -945> 2014-06-09 16:22:12.240035 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1680 (2 active) -944> 2014-06-09 16:22:12.240037 7f86a5e11700 5 _do_op 0x14dd7140 seq 4257 osr(3.22d 0x3ed0c40)/0x3ed0c40 start -943> 2014-06-09 16:22:12.240040 7f86a5e11700 10 _do_transaction on 0xbf7cb40 -942> 2014-06-09 16:22:12.240037 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 may_write -> write-ordered flags ack+ondisk+retry+write -941> 2014-06-09 16:22:12.240046 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -940> 2014-06-09 16:22:12.240055 7f868c75b700 15 getattr 3.67d_head/119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 '_' -939> 2014-06-09 16:22:12.240061 7f8690f64700 10 op_submit_start 4303 -938> 2014-06-09 16:22:12.240063 7f8690f64700 5 queue_transactions (trailing journal) 4303 -937> 2014-06-09 16:22:12.240064 7f8690f64700 5 queue_op 0x324f5080 seq 4303 osr(0.2fd 0x2bd6e50) 828 bytes (queue has 49 ops and 4688453 bytes) -936> 2014-06-09 16:22:12.240068 7f8690f64700 10 op_submit_finish 4303 -935> 2014-06-09 16:22:12.240071 7f8692767700 10 op_submit_start 4304 -934> 2014-06-09 16:22:12.240072 7f8692767700 5 queue_transactions (trailing journal) 4304 -933> 2014-06-09 16:22:12.240073 7f8692767700 5 queue_op 0xcabd0e0 seq 4304 osr(3.105 0x3ed0820) 828 bytes (queue has 49 ops and 4688453 bytes) -932> 2014-06-09 16:22:12.240076 7f8692767700 10 op_submit_finish 4304 -931> 2014-06-09 16:22:12.240072 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 3.271( v 213'580 (0'0,213'580] local-les=243 n=112 ec=118 les/c 243/243 242/242/209) -930> 2014-06-09 16:22:12.240086 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -929> 2014-06-09 16:22:12.240079 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.512( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.512( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -928> 2014-06-09 16:22:12.240102 7f8690f64700 10 log is not dirty -927> 2014-06-09 16:22:12.241912 7f8690f64700 5 queue_transactions existing osr(3.271 0x3ed0bc0)/0x3ed0bc0 -926> 2014-06-09 16:22:12.240106 7f8691f66700 10 op_submit_start 4305 -925> 2014-06-09 16:22:12.241919 7f8691f66700 5 queue_transactions (trailing journal) 4305 -924> 2014-06-09 16:22:12.241918 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4257.0.0 <= header.spos 150443.0.0 -923> 2014-06-09 16:22:12.241921 7f8691f66700 5 queue_op 0x324e6420 seq 4305 osr(3.2b3 0x3ed0f00) 861 bytes (queue has 50 ops and 4689314 bytes) -922> 2014-06-09 16:22:12.241924 7f86a5e11700 5 submit_transaction r = 0 -921> 2014-06-09 16:22:12.241924 7f8691f66700 10 op_submit_finish 4305 -920> 2014-06-09 16:22:12.241928 7f8692767700 10 log is not dirty -919> 2014-06-09 16:22:12.241933 7f8692767700 5 queue_transactions existing osr(0.512 0x2bd7590)/0x2bd7590 -918> 2014-06-09 16:22:12.240107 7f8691765700 2 waiting 51 > 50 ops || 4690142 > 104857600 -917> 2014-06-09 16:22:12.241928 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -916> 2014-06-09 16:22:12.241940 7f86a5e11700 10 _do_op 0x14dd7140 seq 4257 r = 0, finisher 0xc168500 0 -915> 2014-06-09 16:22:12.241947 7f86a5e11700 10 _finish_op 0x14dd7140 seq 4257 osr(3.22d 0x3ed0c40)/0x3ed0c40 -914> 2014-06-09 16:22:12.241944 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.198162 8 0.000473 -913> 2014-06-09 16:22:12.241956 7f8691765700 10 op_submit_start 4306 -912> 2014-06-09 16:22:12.241957 7f8691765700 5 queue_transactions (trailing journal) 4306 -911> 2014-06-09 16:22:12.241959 7f8691765700 5 queue_op 0x325267e0 seq 4306 osr(2.482 0x2ed14f0) 828 bytes (queue has 50 ops and 4689281 bytes) -910> 2014-06-09 16:22:12.241961 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc168500] -909> 2014-06-09 16:22:12.241964 7f868d75d700 2 waiting 51 > 50 ops || 4690142 > 104857600 -908> 2014-06-09 16:22:12.241959 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -907> 2014-06-09 16:22:12.241967 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc168500] -906> 2014-06-09 16:22:12.241969 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -905> 2014-06-09 16:22:12.241969 7f868ff62700 2 waiting 51 > 50 ops || 4690109 > 104857600 -904> 2014-06-09 16:22:12.241970 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -903> 2014-06-09 16:22:12.241973 7f868af58700 2 waiting 51 > 50 ops || 4690142 > 104857600 -902> 2014-06-09 16:22:12.241974 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1680 (1 active) -901> 2014-06-09 16:22:12.241977 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc360 (2 active) -900> 2014-06-09 16:22:12.241979 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3042b440] -899> 2014-06-09 16:22:12.241971 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -898> 2014-06-09 16:22:12.241981 7f86a5e11700 5 _do_op 0xcd90060 seq 4258 osr(3.5ce 0x3ed17c0)/0x3ed17c0 start -897> 2014-06-09 16:22:12.241984 7f86a5e11700 10 _do_transaction on 0x37e4c00 -896> 2014-06-09 16:22:12.241981 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery -895> 2014-06-09 16:22:12.241990 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -894> 2014-06-09 16:22:12.241984 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000026 0 0.000000 -893> 2014-06-09 16:22:12.241992 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays -892> 2014-06-09 16:22:12.241994 7f8691f66700 5 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -891> 2014-06-09 16:22:12.242002 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.22d( v 214'683 (0'0,214'683] local-les=243 n=133 ec=118 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1142 -890> 2014-06-09 16:22:12.242003 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_recovery -889> 2014-06-09 16:22:12.242011 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3042b440] -888> 2014-06-09 16:22:12.242013 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -887> 2014-06-09 16:22:12.242014 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -886> 2014-06-09 16:22:12.242018 7f8691765700 10 op_submit_finish 4306 -885> 2014-06-09 16:22:12.242011 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active] clear_recovery_state -884> 2014-06-09 16:22:12.242021 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.465( v 213'631 (0'0,213'631] local-les=243 n=120 ec=118 les/c 243/203 242/242/202) -883> 2014-06-09 16:22:12.242026 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -882> 2014-06-09 16:22:12.242037 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -881> 2014-06-09 16:22:12.242043 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -880> 2014-06-09 16:22:12.242053 7f8691765700 10 log is not dirty -879> 2014-06-09 16:22:12.242047 7f8691f66700 10 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] share_pg_info -878> 2014-06-09 16:22:12.242057 7f8691765700 5 queue_transactions existing osr(3.465 0x3ed1580)/0x3ed1580 -877> 2014-06-09 16:22:12.242060 7f8691765700 2 waiting 51 > 50 ops || 4690109 > 104857600 -876> 2014-06-09 16:22:12.242058 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4258.0.0 <= header.spos 150443.0.0 -875> 2014-06-09 16:22:12.242062 7f86a5e11700 5 submit_transaction r = 0 -874> 2014-06-09 16:22:12.242063 7f8691f66700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 -873> 2014-06-09 16:22:12.242068 7f8689f56700 2 waiting 51 > 50 ops || 4690142 > 104857600 -872> 2014-06-09 16:22:12.242073 7f868b759700 2 waiting 51 > 50 ops || 4690109 > 104857600 -871> 2014-06-09 16:22:12.242066 7f8691f66700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:3.48a) v4 -- ?+0 0x190b6e00 -870> 2014-06-09 16:22:12.242075 7f86a5e11700 10 _do_op 0xcd90060 seq 4258 r = 0, finisher 0xbd68f20 0 -869> 2014-06-09 16:22:12.242078 7f868f761700 2 waiting 51 > 50 ops || 4690105 > 104857600 -868> 2014-06-09 16:22:12.242081 7f86a5e11700 10 _finish_op 0xcd90060 seq 4258 osr(3.5ce 0x3ed17c0)/0x3ed17c0 -867> 2014-06-09 16:22:12.242083 7f868cf5c700 2 waiting 51 > 50 ops || 4690142 > 104857600 -866> 2014-06-09 16:22:12.242088 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3050a5a0] -865> 2014-06-09 16:22:12.242090 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3050a5a0] -864> 2014-06-09 16:22:12.242082 7f8691f66700 15 osd.26 pg_epoch: 243 pg[3.48a( v 214'648 (0'0,214'648] local-les=243 n=120 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1091 -863> 2014-06-09 16:22:12.242092 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -862> 2014-06-09 16:22:12.242093 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -861> 2014-06-09 16:22:12.242098 7f868e75f700 2 waiting 51 > 50 ops || 4690109 > 104857600 -860> 2014-06-09 16:22:12.242103 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 -859> 2014-06-09 16:22:12.242107 7f868df5e700 2 waiting 51 > 50 ops || 4690109 > 104857600 -858> 2014-06-09 16:22:12.242111 7f868ef60700 2 waiting 51 > 50 ops || 4690109 > 104857600 -857> 2014-06-09 16:22:12.242115 7f8691f66700 10 log is not dirty -856> 2014-06-09 16:22:12.242116 7f8690763700 2 waiting 51 > 50 ops || 4690109 > 104857600 -855> 2014-06-09 16:22:12.242119 7f8691f66700 5 queue_transactions existing osr(3.48a 0x3ed1520)/0x3ed1520 -854> 2014-06-09 16:22:12.242122 7f8691f66700 2 waiting 51 > 50 ops || 4690142 > 104857600 -853> 2014-06-09 16:22:12.242126 7f868a757700 2 waiting 51 > 50 ops || 4690109 > 104857600 -852> 2014-06-09 16:22:12.242131 7f8692f68700 2 waiting 51 > 50 ops || 4690109 > 104857600 -851> 2014-06-09 16:22:12.242135 7f8690f64700 2 waiting 51 > 50 ops || 4690109 > 104857600 -850> 2014-06-09 16:22:12.240169 7f868c75b700 10 getattr got.size() is 0 -849> 2014-06-09 16:22:12.242140 7f8689755700 2 waiting 51 > 50 ops || 4690109 > 104857600 -848> 2014-06-09 16:22:12.242145 7f8692767700 2 waiting 51 > 50 ops || 4690109 > 104857600 -847> 2014-06-09 16:22:12.242141 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 but can_create -846> 2014-06-09 16:22:12.242154 7f868c75b700 15 getattr 3.67d_head/119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 'snapset' -845> 2014-06-09 16:22:12.242160 7f868ff62700 10 op_submit_start 4307 -844> 2014-06-09 16:22:12.242163 7f868ff62700 5 queue_transactions (trailing journal) 4307 -843> 2014-06-09 16:22:12.242165 7f868ff62700 5 queue_op 0x159961e0 seq 4307 osr(1.233 0x384cc30) 828 bytes (queue has 50 ops and 4689281 bytes) -842> 2014-06-09 16:22:12.242171 7f868af58700 2 waiting 51 > 50 ops || 4690142 > 104857600 -841> 2014-06-09 16:22:12.242176 7f8691765700 2 waiting 51 > 50 ops || 4690109 > 104857600 -840> 2014-06-09 16:22:12.242181 7f8689f56700 2 waiting 51 > 50 ops || 4690142 > 104857600 -839> 2014-06-09 16:22:12.242185 7f868b759700 2 waiting 51 > 50 ops || 4690109 > 104857600 -838> 2014-06-09 16:22:12.242190 7f868f761700 2 waiting 51 > 50 ops || 4690105 > 104857600 -837> 2014-06-09 16:22:12.242194 7f868c75b700 10 getattr got.size() is 0 -836> 2014-06-09 16:22:12.242196 7f868d75d700 2 waiting 51 > 50 ops || 4690142 > 104857600 -835> 2014-06-09 16:22:12.242197 7f868c75b700 15 getattr 3.67d_head/119d067d/rbd_data.11806b8b4567.000000000000189d/snapdir//3 'snapset' -834> 2014-06-09 16:22:12.242201 7f868cf5c700 2 waiting 51 > 50 ops || 4690142 > 104857600 -833> 2014-06-09 16:22:12.242202 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd68f20] -832> 2014-06-09 16:22:12.242205 7f868e75f700 2 waiting 51 > 50 ops || 4690109 > 104857600 -831> 2014-06-09 16:22:12.242206 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd68f20] -830> 2014-06-09 16:22:12.242208 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -829> 2014-06-09 16:22:12.242209 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -828> 2014-06-09 16:22:12.242209 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 -827> 2014-06-09 16:22:12.242213 7f868ef60700 2 waiting 51 > 50 ops || 4690109 > 104857600 -826> 2014-06-09 16:22:12.242212 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc360 (1 active) -825> 2014-06-09 16:22:12.242215 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad680 (2 active) -824> 2014-06-09 16:22:12.242217 7f8690763700 2 waiting 51 > 50 ops || 4690109 > 104857600 -823> 2014-06-09 16:22:12.242220 7f868ff62700 10 op_submit_finish 4307 -822> 2014-06-09 16:22:12.242221 7f868df5e700 2 waiting 51 > 50 ops || 4690109 > 104857600 -821> 2014-06-09 16:22:12.242231 7f868c75b700 10 getattr got.size() is 0 -820> 2014-06-09 16:22:12.242224 7f868ff62700 10 osd.26 pg_epoch: 243 pg[1.3bc( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) -819> 2014-06-09 16:22:12.242245 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] create_object_context 0x30b62b00 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 -818> 2014-06-09 16:22:12.242266 7f868ff62700 10 log is not dirty -817> 2014-06-09 16:22:12.242259 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] populate_obc_watchers 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 -816> 2014-06-09 16:22:12.242271 7f868ff62700 5 queue_transactions existing osr(1.3bc 0x384d550)/0x384d550 -815> 2014-06-09 16:22:12.242274 7f868ff62700 2 waiting 51 > 50 ops || 4690109 > 104857600 -814> 2014-06-09 16:22:12.242270 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] ReplicatedPG::check_blacklisted_obc_watchers for obc 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 -813> 2014-06-09 16:22:12.242279 7f86a5e11700 5 _do_op 0x16096540 seq 4259 osr(3.2df 0x3ed0e40)/0x3ed0e40 start -812> 2014-06-09 16:22:12.242283 7f86a5e11700 10 _do_transaction on 0xc2dd140 -811> 2014-06-09 16:22:12.242288 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -810> 2014-06-09 16:22:12.242280 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: 0x30b62b00 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 rwstate(none n=0 w=0) oi: 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) ssc: 0x30dd41c0 snapset: 0=[]:[] -809> 2014-06-09 16:22:12.242297 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] find_object_context 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 @head oi=119d067d/rbd_data.11806b8b4567.000000000000189d/head//3(0'0 unknown.0.0:0 wrlock_by=unknown.0.0:0 s 0 uv0) -808> 2014-06-09 16:22:12.242318 7f868c75b700 15 getattr 3.67d_head/119d067d/rbd_data.11806b8b4567.000000000000189d/snapdir//3 '_' -807> 2014-06-09 16:22:12.242324 7f868a757700 2 waiting 51 > 50 ops || 4690109 > 104857600 -806> 2014-06-09 16:22:12.242329 7f8692f68700 2 waiting 51 > 50 ops || 4690109 > 104857600 -805> 2014-06-09 16:22:12.242333 7f8690f64700 2 waiting 51 > 50 ops || 4690109 > 104857600 -804> 2014-06-09 16:22:12.242335 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -803> 2014-06-09 16:22:12.242337 7f8691f66700 2 waiting 51 > 50 ops || 4690142 > 104857600 -802> 2014-06-09 16:22:12.242342 7f8692767700 2 waiting 51 > 50 ops || 4690109 > 104857600 -801> 2014-06-09 16:22:12.242350 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4259.0.0 <= header.spos 150443.0.0 -800> 2014-06-09 16:22:12.242355 7f86a5e11700 5 submit_transaction r = 0 -799> 2014-06-09 16:22:12.242363 7f8689755700 2 waiting 51 > 50 ops || 4690109 > 104857600 -798> 2014-06-09 16:22:12.242380 7f868c75b700 10 getattr got.size() is 0 -797> 2014-06-09 16:22:12.242383 7f86a5e11700 10 _do_op 0x16096540 seq 4259 r = 0, finisher 0xcc950c0 0 -796> 2014-06-09 16:22:12.242390 7f86a5e11700 10 _finish_op 0x16096540 seq 4259 osr(3.2df 0x3ed0e40)/0x3ed0e40 -795> 2014-06-09 16:22:12.242382 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] get_object_context: no obc for soid 119d067d/rbd_data.11806b8b4567.000000000000189d/snapdir//3 and !can_create -794> 2014-06-09 16:22:12.242401 7f8691765700 10 op_submit_start 4308 -793> 2014-06-09 16:22:12.242402 7f8691765700 5 queue_transactions (trailing journal) 4308 -792> 2014-06-09 16:22:12.242404 7f8691765700 5 queue_op 0x32526780 seq 4308 osr(3.465 0x3ed1580) 828 bytes (queue has 50 ops and 4689281 bytes) -791> 2014-06-09 16:22:12.242408 7f8689f56700 2 waiting 51 > 50 ops || 4690142 > 104857600 -790> 2014-06-09 16:22:12.242412 7f868b759700 2 waiting 51 > 50 ops || 4690109 > 104857600 -789> 2014-06-09 16:22:12.242416 7f868f761700 2 waiting 51 > 50 ops || 4690105 > 104857600 -788> 2014-06-09 16:22:12.242401 7f868c75b700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:12.242395, event: started, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) -787> 2014-06-09 16:22:12.242420 7f868d75d700 2 waiting 51 > 50 ops || 4690142 > 104857600 -786> 2014-06-09 16:22:12.242425 7f868cf5c700 2 waiting 51 > 50 ops || 4690142 > 104857600 -785> 2014-06-09 16:22:12.242420 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] execute_ctx 0x30643600 -784> 2014-06-09 16:22:12.242430 7f868e75f700 2 waiting 51 > 50 ops || 4690109 > 104857600 -783> 2014-06-09 16:22:12.242430 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcc950c0] -782> 2014-06-09 16:22:12.242435 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 -781> 2014-06-09 16:22:12.242435 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcc950c0] -780> 2014-06-09 16:22:12.242436 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -779> 2014-06-09 16:22:12.242438 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -778> 2014-06-09 16:22:12.242438 7f868ef60700 2 waiting 51 > 50 ops || 4690109 > 104857600 -777> 2014-06-09 16:22:12.242442 7f8690763700 2 waiting 51 > 50 ops || 4690109 > 104857600 -776> 2014-06-09 16:22:12.242441 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad680 (1 active) -775> 2014-06-09 16:22:12.242438 7f868c75b700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:12.242433, event: started, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) -774> 2014-06-09 16:22:12.242445 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77bb00 (2 active) -773> 2014-06-09 16:22:12.242446 7f868df5e700 2 waiting 51 > 50 ops || 4690109 > 104857600 -772> 2014-06-09 16:22:12.242450 7f8691765700 10 op_submit_finish 4308 -771> 2014-06-09 16:22:12.242451 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32534060] -770> 2014-06-09 16:22:12.242453 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32534060] -769> 2014-06-09 16:22:12.242455 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -768> 2014-06-09 16:22:12.242456 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -767> 2014-06-09 16:22:12.242446 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_op 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] ov 0'0 av 243'685 snapc 0=[] snapset 0=[]:[] -766> 2014-06-09 16:22:12.242453 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -765> 2014-06-09 16:22:12.242460 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] -764> 2014-06-09 16:22:12.242468 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] check_local -763> 2014-06-09 16:22:12.242471 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op set-alloc-hint object_size 4194304 write_size 4194304 -762> 2014-06-09 16:22:12.242477 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] requeue_ops -761> 2014-06-09 16:22:12.242487 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -760> 2014-06-09 16:22:12.242486 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op write 0~217088 -759> 2014-06-09 16:22:12.242496 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -758> 2014-06-09 16:22:12.242505 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -757> 2014-06-09 16:22:12.242504 7f868c75b700 15 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] do_osd_op_effects on session 0x1078a600 -756> 2014-06-09 16:22:12.242516 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 snapset=0x30dd41f8 snapc=0=[] -755> 2014-06-09 16:22:12.242521 7f8691765700 15 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:193 -754> 2014-06-09 16:22:12.242527 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] setting DIRTY flag -753> 2014-06-09 16:22:12.242531 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] hit_set_clear -752> 2014-06-09 16:22:12.242537 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] make_writeable 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 done, snapset=0=[]:[]+head -751> 2014-06-09 16:22:12.242540 7f8691765700 20 osd.26 pg_epoch: 243 pg[0.1f1( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] agent_stop -750> 2014-06-09 16:22:12.242549 7f8691765700 10 log is not dirty -749> 2014-06-09 16:22:12.242548 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] finish_ctx 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 0x30643600 op modify -748> 2014-06-09 16:22:12.242555 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.4b9( v 214'558 (0'0,214'558] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) -747> 2014-06-09 16:22:12.242561 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] set mtime to 2014-06-09 10:20:21.594716 -746> 2014-06-09 16:22:12.242573 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] state: peer osd.5 activated and committed -745> 2014-06-09 16:22:12.242585 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 206/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] all_activated_and_committed -744> 2014-06-09 16:22:12.242587 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=131 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] final snapset 0=[]:[]+head in 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 -743> 2014-06-09 16:22:12.242597 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] share_pg_info -742> 2014-06-09 16:22:12.242606 7f868c75b700 20 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] zeroing write result code 0 -741> 2014-06-09 16:22:12.242614 7f8691765700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 -740> 2014-06-09 16:22:12.242617 7f8691765700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:3.4b9) v4 -- ?+0 0x30ef6c40 -739> 2014-06-09 16:22:12.242617 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] new_repop rep_tid 33 on osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4 -738> 2014-06-09 16:22:12.242632 7f868c75b700 7 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] issue_repop rep_tid 33 o 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 -737> 2014-06-09 16:22:12.242631 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.4b9( v 214'558 lc 214'557 (0'0,214'558] local-les=243 n=108 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'558 lcod 0'0 mlcod 0'0 inactive m=1] publish_stats_to_osd 243:893 -736> 2014-06-09 16:22:12.242649 7f8691765700 10 log is not dirty -735> 2014-06-09 16:22:12.242655 7f868c75b700 5 -- op tracker -- , seq: 716, time: 2014-06-09 16:22:12.242650, event: waiting for subops from 2, op: osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) -734> 2014-06-09 16:22:12.242654 7f8691765700 10 osd.26 pg_epoch: 243 pg[1.438( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 1.438( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -733> 2014-06-09 16:22:12.242675 7f868c75b700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 244 -732> 2014-06-09 16:22:12.242678 7f868c75b700 1 -- 172.16.96.202:6850/30904 --> osd.2 172.16.96.201:6828/58271 -- osd_sub_op(client.4576.0:36701 3.67d 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 [] v 243'685 snapset=0=[]:[] snapc=0=[]) v10 -- ?+218000 0x19219600 -731> 2014-06-09 16:22:12.242690 7f8691765700 10 log is not dirty -730> 2014-06-09 16:22:12.242695 7f8691765700 5 queue_transactions existing osr(1.438 0x384d710)/0x384d710 -729> 2014-06-09 16:22:12.242698 7f8691765700 2 waiting 51 > 50 ops || 4690109 > 104857600 -728> 2014-06-09 16:22:12.242695 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 214'684 (0'0,214'684] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log log((0'0,214'684], crt=0'0) [243'685 (0'0) modify 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 by client.4576.0:36701 2014-06-09 10:20:21.594716] -727> 2014-06-09 16:22:12.242726 7f868ff62700 2 waiting 51 > 50 ops || 4690109 > 104857600 -726> 2014-06-09 16:22:12.242717 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 243'685 (0'0,243'685] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 luod=214'684 crt=0'0 lcod 0'0 mlcod 0'0 active] add_log_entry 243'685 (0'0) modify 119d067d/rbd_data.11806b8b4567.000000000000189d/head//3 by client.4576.0:36701 2014-06-09 10:20:21.594716 -725> 2014-06-09 16:22:12.242735 7f86a5e11700 5 _do_op 0x324e7620 seq 4260 osr(0.4bd 0x2bd76f0)/0x2bd76f0 start -724> 2014-06-09 16:22:12.242739 7f86a5e11700 10 _do_transaction on 0xbfb2d80 -723> 2014-06-09 16:22:12.242745 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -722> 2014-06-09 16:22:12.242738 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 243'685 (0'0,243'685] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 luod=214'684 crt=0'0 lcod 0'0 mlcod 0'0 active] append_log adding 1 keys -721> 2014-06-09 16:22:12.242772 7f868c75b700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 243'685, trimmed: -720> 2014-06-09 16:22:12.242789 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -719> 2014-06-09 16:22:12.242792 7f868c75b700 5 queue_transactions existing osr(3.67d 0x3ed1d70)/0x3ed1d70 -718> 2014-06-09 16:22:12.242795 7f868c75b700 2 waiting 51 > 50 ops || 4908767 > 104857600 -717> 2014-06-09 16:22:12.242804 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4260.0.0 <= header.spos 150443.0.0 -716> 2014-06-09 16:22:12.242808 7f86a5e11700 5 submit_transaction r = 0 -715> 2014-06-09 16:22:12.242831 7f86a5e11700 10 _do_op 0x324e7620 seq 4260 r = 0, finisher 0xbd671a0 0 -714> 2014-06-09 16:22:12.242838 7f86a5e11700 10 _finish_op 0x324e7620 seq 4260 osr(0.4bd 0x2bd76f0)/0x2bd76f0 -713> 2014-06-09 16:22:12.242853 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd671a0] -712> 2014-06-09 16:22:12.242858 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd671a0] -711> 2014-06-09 16:22:12.242860 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -710> 2014-06-09 16:22:12.242861 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -709> 2014-06-09 16:22:12.242864 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77bb00 (1 active) -708> 2014-06-09 16:22:12.242867 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f0e0 (2 active) -707> 2014-06-09 16:22:12.242870 7f86a5e11700 5 _do_op 0x2fc8d800 seq 4261 osr(3.750 0x3ed1eb0)/0x3ed1eb0 start -706> 2014-06-09 16:22:12.242873 7f86a5e11700 10 _do_transaction on 0xbe8b2c0 -705> 2014-06-09 16:22:12.242878 7f86a5e11700 15 _omap_setkeys meta/1fe0815a/pglog_3.750/0//-1 -704> 2014-06-09 16:22:12.242926 7f86a5e11700 10 lookup_strip_header done cid meta oid 1fe0815a/pglog_3.750/0//-1 -703> 2014-06-09 16:22:12.242942 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -702> 2014-06-09 16:22:12.242980 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -701> 2014-06-09 16:22:12.242982 7f868a757700 10 op_submit_start 4309 -700> 2014-06-09 16:22:12.242985 7f868a757700 5 queue_transactions (trailing journal) 4309 -699> 2014-06-09 16:22:12.242986 7f868a757700 5 queue_op 0xc2d54a0 seq 4309 osr(1.10d 0x384c930) 828 bytes (queue has 50 ops and 4689281 bytes) -698> 2014-06-09 16:22:12.242993 7f868a757700 10 op_submit_finish 4309 -697> 2014-06-09 16:22:12.242996 7f86a5e11700 15 _omap_rmkeys meta/1fe0815a/pglog_3.750/0//-1 -696> 2014-06-09 16:22:12.243002 7f86a5e11700 10 _omap_rmkeys meta/1fe0815a/pglog_3.750/0//-1 r = 0 -695> 2014-06-09 16:22:12.242997 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -694> 2014-06-09 16:22:12.243008 7f86a5e11700 15 _omap_setkeys meta/1fe0815a/pglog_3.750/0//-1 -693> 2014-06-09 16:22:12.243011 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.090474 7 0.000334 -692> 2014-06-09 16:22:12.243020 7f86a5e11700 15 _touch 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -691> 2014-06-09 16:22:12.243035 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd6b6c0] -690> 2014-06-09 16:22:12.243038 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd6b6c0] -689> 2014-06-09 16:22:12.243039 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -688> 2014-06-09 16:22:12.243040 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -687> 2014-06-09 16:22:12.243034 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -686> 2014-06-09 16:22:12.243046 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -685> 2014-06-09 16:22:12.243055 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -684> 2014-06-09 16:22:12.243062 7f86a5e11700 20 lookup_strip_header failed to get strip_header cid 3.750_head oid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -683> 2014-06-09 16:22:12.243066 7f868a757700 5 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -682> 2014-06-09 16:22:12.243074 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -681> 2014-06-09 16:22:12.243081 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -680> 2014-06-09 16:22:12.243094 7f86a5e11700 20 write_state seq is 30174 -679> 2014-06-09 16:22:12.243092 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -678> 2014-06-09 16:22:12.243102 7f86a5e11700 20 set_header setting 30173 cid 3.750_head oid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 parent seq 0 -677> 2014-06-09 16:22:12.243101 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -676> 2014-06-09 16:22:12.243110 7f868a757700 10 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -675> 2014-06-09 16:22:12.243116 7f86a5e11700 10 _touch 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 = 0 -674> 2014-06-09 16:22:12.243123 7f86a5e11700 15 _write 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 0~450560 -673> 2014-06-09 16:22:12.243124 7f868a757700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 -672> 2014-06-09 16:22:12.243126 7f868a757700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:1.56c) v4 -- ?+0 0x30c12540 -671> 2014-06-09 16:22:12.243134 7f86a5e11700 10 file_to_extents done -670> 2014-06-09 16:22:12.243139 7f868a757700 15 osd.26 pg_epoch: 243 pg[1.56c( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 -669> 2014-06-09 16:22:12.243172 7f868a757700 10 log is not dirty -668> 2014-06-09 16:22:12.243177 7f868a757700 5 queue_transactions existing osr(1.56c 0x384da70)/0x384da70 -667> 2014-06-09 16:22:12.243179 7f868a757700 2 waiting 51 > 50 ops || 4690142 > 104857600 -666> 2014-06-09 16:22:12.243201 7f8692f68700 2 waiting 51 > 50 ops || 4690109 > 104857600 -665> 2014-06-09 16:22:12.243206 7f8690f64700 2 waiting 51 > 50 ops || 4690109 > 104857600 -664> 2014-06-09 16:22:12.243210 7f8692767700 2 waiting 51 > 50 ops || 4690109 > 104857600 -663> 2014-06-09 16:22:12.243214 7f8689755700 2 waiting 51 > 50 ops || 4690109 > 104857600 -662> 2014-06-09 16:22:12.243220 7f868af58700 2 waiting 51 > 50 ops || 4690142 > 104857600 -661> 2014-06-09 16:22:12.243227 7f868b759700 2 waiting 51 > 50 ops || 4690109 > 104857600 -660> 2014-06-09 16:22:12.243231 7f868f761700 2 waiting 51 > 50 ops || 4690105 > 104857600 -659> 2014-06-09 16:22:12.243236 7f868d75d700 2 waiting 51 > 50 ops || 4690142 > 104857600 -658> 2014-06-09 16:22:12.243258 7f8691f66700 2 waiting 51 > 50 ops || 4690142 > 104857600 -657> 2014-06-09 16:22:12.243263 7f868e75f700 2 waiting 51 > 50 ops || 4690109 > 104857600 -656> 2014-06-09 16:22:12.243267 7f868bf5a700 2 waiting 51 > 50 ops || 4690109 > 104857600 -655> 2014-06-09 16:22:12.243273 7f868ef60700 2 waiting 51 > 50 ops || 4690109 > 104857600 -654> 2014-06-09 16:22:12.243276 7f868cf5c700 2 waiting 51 > 50 ops || 4690142 > 104857600 -653> 2014-06-09 16:22:12.243280 7f868df5e700 2 waiting 51 > 50 ops || 4690109 > 104857600 -652> 2014-06-09 16:22:12.243285 7f8690763700 2 waiting 51 > 50 ops || 4690109 > 104857600 -651> 2014-06-09 16:22:12.243289 7f868ff62700 2 waiting 51 > 50 ops || 4690109 > 104857600 -650> 2014-06-09 16:22:12.243325 7f868c75b700 2 waiting 51 > 50 ops || 4908767 > 104857600 -649> 2014-06-09 16:22:12.243332 7f8689f56700 2 waiting 51 > 50 ops || 4690142 > 104857600 -648> 2014-06-09 16:22:12.243338 7f8691765700 2 waiting 51 > 50 ops || 4690109 > 104857600 -647> 2014-06-09 16:22:12.245111 7f86a5e11700 10 _generic_write 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 0~450560 = 0 -646> 2014-06-09 16:22:12.245145 7f86a5e11700 15 _setattrs 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -645> 2014-06-09 16:22:12.245162 7f86a5e11700 10 _setattrs 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 = 0 -644> 2014-06-09 16:22:12.245168 7f86a5e11700 15 _setattrs 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 -643> 2014-06-09 16:22:12.245176 7f86a5e11700 10 _setattrs 3.750_head/e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 = 0 -642> 2014-06-09 16:22:12.245251 7f86a5e11700 10 cid: oid: e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 not skipping op, *spos 4261.0.0 -641> 2014-06-09 16:22:12.245255 7f86a5e11700 10 > header.spos 0.0.0 -640> 2014-06-09 16:22:12.245272 7f86a5e11700 20 set_header setting 30173 cid 3.750_head oid e3f48f50/rbd_data.117d6b8b4567.00000000000016db/head//3 parent seq 0 -639> 2014-06-09 16:22:12.245341 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4261.0.0 <= header.spos 150443.0.0 -638> 2014-06-09 16:22:12.245347 7f86a5e11700 10 cid: oid: 1fe0815a/pglog_3.750/0//-1 skipping op, spos 4261.0.0 <= header.spos 150418.0.0 -637> 2014-06-09 16:22:12.245351 7f86a5e11700 5 submit_transaction r = 0 -636> 2014-06-09 16:22:12.247673 7f86a5e11700 10 _do_op 0x2fc8d800 seq 4261 r = 0, finisher 0x30629000 0x30439780 -635> 2014-06-09 16:22:12.247685 7f86a5e11700 10 _finish_op 0x2fc8d800 seq 4261 osr(3.750 0x3ed1eb0)/0x3ed1eb0 -634> 2014-06-09 16:22:12.247688 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xccf94a0] -633> 2014-06-09 16:22:12.247705 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629000] -632> 2014-06-09 16:22:12.247693 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] op_commit: 32 -631> 2014-06-09 16:22:12.247711 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f0e0 (1 active) -630> 2014-06-09 16:22:12.247715 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca49b00 (2 active) -629> 2014-06-09 16:22:12.247718 7f86a5e11700 5 _do_op 0x324f5260 seq 4262 osr(2.13c 0x2ed0610)/0x2ed0610 start -628> 2014-06-09 16:22:12.247721 7f86a5e11700 10 _do_transaction on 0x344c180 -627> 2014-06-09 16:22:12.247728 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -626> 2014-06-09 16:22:12.247722 7f86a460e700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.247713, event: op_commit, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) -625> 2014-06-09 16:22:12.247741 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xccf94a0] -624> 2014-06-09 16:22:12.247743 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -623> 2014-06-09 16:22:12.247744 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -622> 2014-06-09 16:22:12.247742 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied: 32 -621> 2014-06-09 16:22:12.247765 7f86a4e0f700 5 -- op tracker -- , seq: 766, time: 2014-06-09 16:22:12.247758, event: op_applied, op: osd_op(client.4573.0:34234 rbd_data.117d6b8b4567.00000000000016db [set-alloc-hint object_size 4194304 write_size 4194304,write 0~450560] 3.e3f48f50 RETRY=3 ack+ondisk+retry+write e242) -620> 2014-06-09 16:22:12.247773 7f86a4e0f700 10 osd.26 pg_epoch: 243 pg[3.750( v 243'519 (0'0,243'519] local-les=243 n=99 ec=118 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 luod=214'518 crt=0'0 lcod 0'0 mlcod 0'0 active] op_applied on primary on version 243'519 -619> 2014-06-09 16:22:12.247796 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629000] -618> 2014-06-09 16:22:12.247799 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -617> 2014-06-09 16:22:12.247800 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -616> 2014-06-09 16:22:12.247917 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -615> 2014-06-09 16:22:12.247950 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4262.0.0 <= header.spos 150443.0.0 -614> 2014-06-09 16:22:12.247955 7f86a5e11700 5 submit_transaction r = 0 -613> 2014-06-09 16:22:12.247974 7f86a5e11700 10 _do_op 0x324f5260 seq 4262 r = 0, finisher 0xbd79480 0 -612> 2014-06-09 16:22:12.247980 7f86a5e11700 10 _finish_op 0x324f5260 seq 4262 osr(2.13c 0x2ed0610)/0x2ed0610 -611> 2014-06-09 16:22:12.247982 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbccd440] -610> 2014-06-09 16:22:12.247985 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbccd440] -609> 2014-06-09 16:22:12.247986 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -608> 2014-06-09 16:22:12.247987 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -607> 2014-06-09 16:22:12.247991 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd79480] -606> 2014-06-09 16:22:12.248000 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd79480] -605> 2014-06-09 16:22:12.248002 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -604> 2014-06-09 16:22:12.248003 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -603> 2014-06-09 16:22:12.248006 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca49b00 (1 active) -602> 2014-06-09 16:22:12.248009 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f6360 (2 active) -601> 2014-06-09 16:22:12.248011 7f86a5e11700 5 _do_op 0x32539ec0 seq 4263 osr(3.470 0x3ed1540)/0x3ed1540 start -600> 2014-06-09 16:22:12.248014 7f86a5e11700 10 _do_transaction on 0xc263980 -599> 2014-06-09 16:22:12.248020 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -598> 2014-06-09 16:22:12.248063 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -597> 2014-06-09 16:22:12.248077 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4263.0.0 <= header.spos 150443.0.0 -596> 2014-06-09 16:22:12.248081 7f86a5e11700 5 submit_transaction r = 0 -595> 2014-06-09 16:22:12.248094 7f86a5e11700 10 _do_op 0x32539ec0 seq 4263 r = 0, finisher 0xc5a73c0 0 -594> 2014-06-09 16:22:12.248102 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbe31c80] -593> 2014-06-09 16:22:12.248104 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbe31c80] -592> 2014-06-09 16:22:12.248106 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -591> 2014-06-09 16:22:12.248107 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -590> 2014-06-09 16:22:12.248101 7f86a5e11700 10 _finish_op 0x32539ec0 seq 4263 osr(3.470 0x3ed1540)/0x3ed1540 -589> 2014-06-09 16:22:12.248129 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a73c0] -588> 2014-06-09 16:22:12.248133 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a73c0] -587> 2014-06-09 16:22:12.248135 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -586> 2014-06-09 16:22:12.248136 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -585> 2014-06-09 16:22:12.248139 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f6360 (1 active) -584> 2014-06-09 16:22:12.248142 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcadb320 (2 active) -583> 2014-06-09 16:22:12.248144 7f86a5e11700 5 _do_op 0x30d969c0 seq 4264 osr(2.185 0x2ed09a0)/0x2ed09a0 start -582> 2014-06-09 16:22:12.248147 7f86a5e11700 10 _do_transaction on 0xc245200 -581> 2014-06-09 16:22:12.248153 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -580> 2014-06-09 16:22:12.248209 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -579> 2014-06-09 16:22:12.248242 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4264.0.0 <= header.spos 150443.0.0 -578> 2014-06-09 16:22:12.248261 7f86a5e11700 5 submit_transaction r = 0 -577> 2014-06-09 16:22:12.248279 7f86a5e11700 10 _do_op 0x30d969c0 seq 4264 r = 0, finisher 0x3006c160 0 -576> 2014-06-09 16:22:12.248285 7f86a5e11700 10 _finish_op 0x30d969c0 seq 4264 osr(2.185 0x2ed09a0)/0x2ed09a0 -575> 2014-06-09 16:22:12.248287 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062a700] -574> 2014-06-09 16:22:12.248295 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006c160] -573> 2014-06-09 16:22:12.248299 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006c160] -572> 2014-06-09 16:22:12.248291 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -571> 2014-06-09 16:22:12.248301 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -570> 2014-06-09 16:22:12.248302 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -569> 2014-06-09 16:22:12.248302 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -568> 2014-06-09 16:22:12.248313 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.185( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 -567> 2014-06-09 16:22:12.248322 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062a700] -566> 2014-06-09 16:22:12.248324 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -565> 2014-06-09 16:22:12.248325 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -564> 2014-06-09 16:22:12.248323 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcadb320 (1 active) -563> 2014-06-09 16:22:12.248339 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcada6c0 (2 active) -562> 2014-06-09 16:22:12.248342 7f86a5e11700 5 _do_op 0xbf7f860 seq 4265 osr(0.179 0x2bd6800)/0x2bd6800 start -561> 2014-06-09 16:22:12.248345 7f86a5e11700 10 _do_transaction on 0x33380c0 -560> 2014-06-09 16:22:12.248351 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -559> 2014-06-09 16:22:12.248393 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -558> 2014-06-09 16:22:12.248407 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4265.0.0 <= header.spos 150443.0.0 -557> 2014-06-09 16:22:12.248412 7f86a5e11700 5 submit_transaction r = 0 -556> 2014-06-09 16:22:12.248435 7f86a5e11700 10 _do_op 0xbf7f860 seq 4265 r = 0, finisher 0x3006c720 0 -555> 2014-06-09 16:22:12.248442 7f86a5e11700 10 _finish_op 0xbf7f860 seq 4265 osr(0.179 0x2bd6800)/0x2bd6800 -554> 2014-06-09 16:22:12.248444 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdba860] -553> 2014-06-09 16:22:12.248447 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdba860] -552> 2014-06-09 16:22:12.248448 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -551> 2014-06-09 16:22:12.248449 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -550> 2014-06-09 16:22:12.248451 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3006c720] -549> 2014-06-09 16:22:12.248455 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3006c720] -548> 2014-06-09 16:22:12.248457 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -547> 2014-06-09 16:22:12.248458 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -546> 2014-06-09 16:22:12.248461 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcada6c0 (1 active) -545> 2014-06-09 16:22:12.248464 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fdc20 (2 active) -544> 2014-06-09 16:22:12.248466 7f86a5e11700 5 _do_op 0xc88be60 seq 4266 osr(1.54a 0x384daf0)/0x384daf0 start -543> 2014-06-09 16:22:12.248469 7f86a5e11700 10 _do_transaction on 0x394efc0 -542> 2014-06-09 16:22:12.248474 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -541> 2014-06-09 16:22:12.248516 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -540> 2014-06-09 16:22:12.248530 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4266.0.0 <= header.spos 150443.0.0 -539> 2014-06-09 16:22:12.248534 7f86a5e11700 5 submit_transaction r = 0 -538> 2014-06-09 16:22:12.248535 7f868a757700 10 op_submit_start 4310 -537> 2014-06-09 16:22:12.248539 7f868a757700 5 queue_transactions (trailing journal) 4310 -536> 2014-06-09 16:22:12.248541 7f868a757700 5 queue_op 0xbe7cfc0 seq 4310 osr(1.56c 0x384da70) 861 bytes (queue has 47 ops and 4234667 bytes) -535> 2014-06-09 16:22:12.248544 7f868a757700 10 op_submit_finish 4310 -534> 2014-06-09 16:22:12.248547 7f86a5e11700 10 _do_op 0xc88be60 seq 4266 r = 0, finisher 0xc6e98c0 0 -533> 2014-06-09 16:22:12.248548 7f8692767700 10 op_submit_start 4311 -532> 2014-06-09 16:22:12.248550 7f8692767700 5 queue_transactions (trailing journal) 4311 -531> 2014-06-09 16:22:12.248551 7f8692767700 5 queue_op 0x32538f60 seq 4311 osr(0.512 0x2bd7590) 828 bytes (queue has 49 ops and 4236323 bytes) -530> 2014-06-09 16:22:12.248553 7f86a5e11700 10 _finish_op 0xc88be60 seq 4266 osr(1.54a 0x384daf0)/0x384daf0 -529> 2014-06-09 16:22:12.248557 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfcf400] -528> 2014-06-09 16:22:12.248559 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfcf400] -527> 2014-06-09 16:22:12.248560 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -526> 2014-06-09 16:22:12.248562 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -525> 2014-06-09 16:22:12.248563 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc6e98c0] -524> 2014-06-09 16:22:12.248551 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 208/208 242/242/207) [6,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 3.544( v 213'569 (0'0,213'569] local-les=243 n=105 ec=118 les/c 243/208 242/242/207) -523> 2014-06-09 16:22:12.248567 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc6e98c0] -522> 2014-06-09 16:22:12.248569 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -521> 2014-06-09 16:22:12.248570 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -520> 2014-06-09 16:22:12.248573 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fdc20 (1 active) -519> 2014-06-09 16:22:12.248576 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114fc0 (2 active) -518> 2014-06-09 16:22:12.248580 7f86a5e11700 5 _do_op 0x14dd6060 seq 4267 osr(1.301 0x384d030)/0x384d030 start -517> 2014-06-09 16:22:12.248582 7f8692767700 10 op_submit_finish 4311 -516> 2014-06-09 16:22:12.248583 7f86a5e11700 10 _do_transaction on 0xc1fc9c0 -515> 2014-06-09 16:22:12.248589 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -514> 2014-06-09 16:22:12.248593 7f868a757700 10 log is not dirty -513> 2014-06-09 16:22:12.248588 7f8692767700 10 osd.26 pg_epoch: 243 pg[1.286( empty local-les=243 n=0 ec=1 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.286( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) -512> 2014-06-09 16:22:12.248601 7f868a757700 5 queue_transactions existing osr(3.544 0x3ed19a0)/0x3ed19a0 -511> 2014-06-09 16:22:12.248604 7f868a757700 10 op_submit_start 4312 -510> 2014-06-09 16:22:12.248605 7f868a757700 5 queue_transactions (trailing journal) 4312 -509> 2014-06-09 16:22:12.248606 7f868a757700 5 queue_op 0x32539e00 seq 4312 osr(3.544 0x3ed19a0) 828 bytes (queue has 49 ops and 4236323 bytes) -508> 2014-06-09 16:22:12.248609 7f868a757700 10 op_submit_finish 4312 -507> 2014-06-09 16:22:12.248619 7f8692767700 10 log is not dirty -506> 2014-06-09 16:22:12.248611 7f868a757700 10 osd.26 pg_epoch: 243 pg[3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 206/206 242/242/205) [5,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 3.46f( v 213'549 (0'0,213'549] local-les=243 n=109 ec=118 les/c 243/206 242/242/205) -505> 2014-06-09 16:22:12.248624 7f8692767700 5 queue_transactions existing osr(1.286 0x384ced0)/0x384ced0 -504> 2014-06-09 16:22:12.248626 7f8692767700 10 op_submit_start 4313 -503> 2014-06-09 16:22:12.248627 7f8692767700 5 queue_transactions (trailing journal) 4313 -502> 2014-06-09 16:22:12.248628 7f8692767700 5 queue_op 0x32538f00 seq 4313 osr(1.286 0x384ced0) 828 bytes (queue has 50 ops and 4237151 bytes) -501> 2014-06-09 16:22:12.248631 7f8692767700 10 op_submit_finish 4313 -500> 2014-06-09 16:22:12.248630 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -499> 2014-06-09 16:22:12.248640 7f868a757700 10 log is not dirty -498> 2014-06-09 16:22:12.248633 7f8692767700 10 osd.26 pg_epoch: 243 pg[0.330( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.330( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -497> 2014-06-09 16:22:12.248644 7f868a757700 5 queue_transactions existing osr(3.46f 0x3ed1560)/0x3ed1560 -496> 2014-06-09 16:22:12.248646 7f868a757700 2 waiting 51 > 50 ops || 4237979 > 104857600 -495> 2014-06-09 16:22:12.248647 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4267.0.0 <= header.spos 150443.0.0 -494> 2014-06-09 16:22:12.248651 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 -493> 2014-06-09 16:22:12.248652 7f86a5e11700 5 submit_transaction r = 0 -492> 2014-06-09 16:22:12.248659 7f868af58700 2 waiting 51 > 50 ops || 4238012 > 104857600 -491> 2014-06-09 16:22:12.248661 7f8692767700 10 log is not dirty -490> 2014-06-09 16:22:12.248666 7f8692767700 5 queue_transactions existing osr(0.330 0x2bd70a0)/0x2bd70a0 -489> 2014-06-09 16:22:12.248667 7f868b759700 2 waiting 51 > 50 ops || 4237979 > 104857600 -488> 2014-06-09 16:22:12.160847 7f86a5610700 0 transaction dump: { "ops": [ { "op_num": 0, "op_name": "remove", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3"}, { "op_num": 1, "op_name": "mkcoll", "collection": "3.27_TEMP"}, { "op_num": 2, "op_name": "remove", "collection": "3.27_TEMP", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3"}, { "op_num": 3, "op_name": "touch", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3"}, { "op_num": 4, "op_name": "omap_setheader", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3", "header_length": "0"}, { "op_num": 5, "op_name": "write", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3", "length": 4194304, "offset": 0, "bufferlist length": 4194304}, { "op_num": 6, "op_name": "omap_setkeys", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3", "attr_lens": {}}, { "op_num": 7, "op_name": "setattrs", "collection": "3.27_head", "oid": "97a47827\/rbd_data.11756b8b4567.00000000000015f7\/head\/\/3", "attr_lens": { "_": 257, "snapset": 31}}, { "op_num": 8, "op_name": "omap_setkeys", "collection": "meta", "oid": "16ef7597\/infos\/head\/\/-1", "attr_lens": { "3.27_epoch": 4, "3.27_info": 684}}]} -487> 2014-06-09 16:22:12.248669 7f8692767700 2 waiting 51 > 50 ops || 4237979 > 104857600 -486> 2014-06-09 16:22:12.248673 7f8690f64700 10 op_submit_start 4314 -485> 2014-06-09 16:22:12.248674 7f8690f64700 5 queue_transactions (trailing journal) 4314 -484> 2014-06-09 16:22:12.248673 7f868f761700 2 waiting 51 > 50 ops || 4237975 > 104857600 -483> 2014-06-09 16:22:12.248675 7f8690f64700 5 queue_op 0x30d96e40 seq 4314 osr(3.271 0x3ed0bc0) 828 bytes (queue has 50 ops and 4237151 bytes) -482> 2014-06-09 16:22:12.248678 7f8690f64700 10 op_submit_finish 4314 -481> 2014-06-09 16:22:12.248679 7f868d75d700 2 waiting 51 > 50 ops || 4238012 > 104857600 -480> 2014-06-09 16:22:12.248683 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.a6( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/211) [8,26] r=1 lpr=242 pi=13-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 1.a6( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/211) -479> 2014-06-09 16:22:12.248697 7f86a5e11700 10 _do_op 0x14dd6060 seq 4267 r = 0, finisher 0x3061f0c0 0 -478> 2014-06-09 16:22:12.248704 7f86a5e11700 10 _finish_op 0x14dd6060 seq 4267 osr(1.301 0x384d030)/0x384d030 -477> 2014-06-09 16:22:12.248715 7f8690f64700 10 log is not dirty -476> 2014-06-09 16:22:12.248714 7f8691f66700 10 op_submit_start 4315 -475> 2014-06-09 16:22:12.248715 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3061f0c0] -474> 2014-06-09 16:22:12.248717 7f8691f66700 5 queue_transactions (trailing journal) 4315 -473> 2014-06-09 16:22:12.248720 7f8690f64700 5 queue_transactions existing osr(1.a6 0x384c750)/0x384c750 -472> 2014-06-09 16:22:12.248719 7f8691f66700 5 queue_op 0x30d967e0 seq 4315 osr(3.48a 0x3ed1520) 861 bytes (queue has 50 ops and 4237151 bytes) -471> 2014-06-09 16:22:12.248722 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3061f0c0] -470> 2014-06-09 16:22:12.248723 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -469> 2014-06-09 16:22:12.248724 7f8690f64700 2 waiting 51 > 50 ops || 4237977 > 104857600 -468> 2014-06-09 16:22:12.248725 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -467> 2014-06-09 16:22:12.248728 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114fc0 (1 active) -466> 2014-06-09 16:22:12.248729 7f868bf5a700 2 waiting 51 > 50 ops || 4237979 > 104857600 -465> 2014-06-09 16:22:12.248731 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52f680 (2 active) -464> 2014-06-09 16:22:12.248734 7f868a757700 2 waiting 51 > 50 ops || 4237979 > 104857600 -463> 2014-06-09 16:22:12.248735 7f86a5e11700 5 _do_op 0xd235c20 seq 4268 osr(1.504 0x384dbd0)/0x384dbd0 start -462> 2014-06-09 16:22:12.248738 7f86a5e11700 10 _do_transaction on 0xc0d83c0 -461> 2014-06-09 16:22:12.248740 7f868cf5c700 2 waiting 51 > 50 ops || 4238012 > 104857600 -460> 2014-06-09 16:22:12.248743 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -459> 2014-06-09 16:22:12.248732 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbcc82a0] -458> 2014-06-09 16:22:12.248745 7f8691f66700 10 op_submit_finish 4315 -457> 2014-06-09 16:22:12.248747 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -456> 2014-06-09 16:22:12.248751 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -455> 2014-06-09 16:22:12.248759 7f86a460e700 10 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -454> 2014-06-09 16:22:12.248765 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.194473 8 0.009393 -453> 2014-06-09 16:22:12.248768 7f86a460e700 15 osd.26 pg_epoch: 243 pg[1.301( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:161 -452> 2014-06-09 16:22:12.248776 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbcc82a0] -451> 2014-06-09 16:22:12.248777 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -450> 2014-06-09 16:22:12.248778 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -449> 2014-06-09 16:22:12.248782 7f868ef60700 2 waiting 51 > 50 ops || 4237979 > 104857600 -448> 2014-06-09 16:22:12.248778 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -447> 2014-06-09 16:22:12.248784 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -446> 2014-06-09 16:22:12.248789 7f8690763700 2 waiting 51 > 50 ops || 4237979 > 104857600 -445> 2014-06-09 16:22:12.248794 7f8692f68700 10 op_submit_start 4316 -444> 2014-06-09 16:22:12.248796 7f8692f68700 5 queue_transactions (trailing journal) 4316 -443> 2014-06-09 16:22:12.248789 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -442> 2014-06-09 16:22:12.248796 7f868ff62700 2 waiting 51 > 50 ops || 4237979 > 104857600 -441> 2014-06-09 16:22:12.248797 7f8692f68700 5 queue_op 0xcc87f80 seq 4316 osr(3.1e1 0x3ed09b0) 828 bytes (queue has 50 ops and 4237151 bytes) -440> 2014-06-09 16:22:12.248800 7f8692f68700 10 op_submit_finish 4316 -439> 2014-06-09 16:22:12.248800 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000023 0 0.000000 -438> 2014-06-09 16:22:12.248809 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4268.0.0 <= header.spos 150443.0.0 -437> 2014-06-09 16:22:12.248803 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 FlushedEvt -436> 2014-06-09 16:22:12.248813 7f86a5e11700 5 submit_transaction r = 0 -435> 2014-06-09 16:22:12.248810 7f8691f66700 5 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -434> 2014-06-09 16:22:12.248818 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -433> 2014-06-09 16:22:12.248824 7f86a5e11700 10 _do_op 0xd235c20 seq 4268 r = 0, finisher 0x30629c60 0 -432> 2014-06-09 16:22:12.248818 7f8692f68700 15 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] requeue_ops -431> 2014-06-09 16:22:12.248830 7f8692f68700 10 log is not dirty -430> 2014-06-09 16:22:12.248824 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -429> 2014-06-09 16:22:12.248830 7f86a5e11700 10 _finish_op 0xd235c20 seq 4268 osr(1.504 0x384dbd0)/0x384dbd0 -428> 2014-06-09 16:22:12.248841 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30629c60] -427> 2014-06-09 16:22:12.248836 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -426> 2014-06-09 16:22:12.248834 7f8692f68700 10 osd.26 pg_epoch: 243 pg[2.778( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 2.778( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -425> 2014-06-09 16:22:12.248845 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30629c60] -424> 2014-06-09 16:22:12.248847 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -423> 2014-06-09 16:22:12.248848 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -422> 2014-06-09 16:22:12.248851 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52f680 (1 active) -421> 2014-06-09 16:22:12.248846 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -420> 2014-06-09 16:22:12.248854 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc653680 (2 active) -419> 2014-06-09 16:22:12.248856 7f86a5e11700 5 _do_op 0x324f40c0 seq 4269 osr(3.111 0x3ed0830)/0x3ed0830 start -418> 2014-06-09 16:22:12.248859 7f86a5e11700 10 _do_transaction on 0xc259380 -417> 2014-06-09 16:22:12.248856 7f8691f66700 10 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -416> 2014-06-09 16:22:12.248865 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -415> 2014-06-09 16:22:12.248865 7f8692f68700 10 log is not dirty -414> 2014-06-09 16:22:12.248870 7f8692f68700 5 queue_transactions existing osr(2.778 0x3ed0050)/0x3ed0050 -413> 2014-06-09 16:22:12.248872 7f8692f68700 10 op_submit_start 4317 -412> 2014-06-09 16:22:12.248873 7f8692f68700 5 queue_transactions (trailing journal) 4317 -411> 2014-06-09 16:22:12.248872 7f8691f66700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -410> 2014-06-09 16:22:12.248874 7f8692f68700 5 queue_op 0xcc877a0 seq 4317 osr(2.778 0x3ed0050) 828 bytes (queue has 50 ops and 4237151 bytes) -409> 2014-06-09 16:22:12.248876 7f8692f68700 10 op_submit_finish 4317 -408> 2014-06-09 16:22:12.248875 7f8691f66700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:0.221) v4 -- ?+0 0x190b7180 -407> 2014-06-09 16:22:12.248879 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.516( v 213'643 (0'0,213'643] local-les=243 n=125 ec=118 les/c 243/203 242/242/202) -406> 2014-06-09 16:22:12.248892 7f8691f66700 15 osd.26 pg_epoch: 243 pg[0.221( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 -405> 2014-06-09 16:22:12.248908 7f8692f68700 10 log is not dirty -404> 2014-06-09 16:22:12.248912 7f8692f68700 5 queue_transactions existing osr(3.516 0x3ed16d0)/0x3ed16d0 -403> 2014-06-09 16:22:12.248914 7f8692f68700 2 waiting 51 > 50 ops || 4237979 > 104857600 -402> 2014-06-09 16:22:12.248919 7f868df5e700 2 waiting 51 > 50 ops || 4237979 > 104857600 -401> 2014-06-09 16:22:12.248920 7f8691f66700 10 log is not dirty -400> 2014-06-09 16:22:12.248924 7f8691f66700 5 queue_transactions existing osr(0.221 0x2bd6a50)/0x2bd6a50 -399> 2014-06-09 16:22:12.248925 7f8690f64700 2 waiting 51 > 50 ops || 4237977 > 104857600 -398> 2014-06-09 16:22:12.248928 7f8691f66700 2 waiting 51 > 50 ops || 4238012 > 104857600 -397> 2014-06-09 16:22:12.248932 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc94120] -396> 2014-06-09 16:22:12.248934 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc94120] -395> 2014-06-09 16:22:12.248934 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 -394> 2014-06-09 16:22:12.248936 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -393> 2014-06-09 16:22:12.248937 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -392> 2014-06-09 16:22:12.248940 7f868af58700 2 waiting 51 > 50 ops || 4238012 > 104857600 -391> 2014-06-09 16:22:12.248947 7f868b759700 2 waiting 51 > 50 ops || 4237979 > 104857600 -390> 2014-06-09 16:22:12.248952 7f8692767700 2 waiting 51 > 50 ops || 4237979 > 104857600 -389> 2014-06-09 16:22:12.248958 7f868f761700 2 waiting 51 > 50 ops || 4237975 > 104857600 -388> 2014-06-09 16:22:12.248964 7f868d75d700 2 waiting 51 > 50 ops || 4238012 > 104857600 -387> 2014-06-09 16:22:12.248969 7f868e75f700 2 waiting 51 > 50 ops || 4237979 > 104857600 -386> 2014-06-09 16:22:12.248976 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -385> 2014-06-09 16:22:12.248991 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4269.0.0 <= header.spos 150443.0.0 -384> 2014-06-09 16:22:12.248996 7f86a5e11700 5 submit_transaction r = 0 -383> 2014-06-09 16:22:12.249016 7f86a5e11700 10 _do_op 0x324f40c0 seq 4269 r = 0, finisher 0x3062a120 0 -382> 2014-06-09 16:22:12.249023 7f86a5e11700 10 _finish_op 0x324f40c0 seq 4269 osr(3.111 0x3ed0830)/0x3ed0830 -381> 2014-06-09 16:22:12.249024 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xccf87e0] -380> 2014-06-09 16:22:12.249026 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xccf87e0] -379> 2014-06-09 16:22:12.249027 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -378> 2014-06-09 16:22:12.249028 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -377> 2014-06-09 16:22:12.249033 7f8692f68700 10 op_submit_start 4318 -376> 2014-06-09 16:22:12.249034 7f8692f68700 5 queue_transactions (trailing journal) 4318 -375> 2014-06-09 16:22:12.249034 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3062a120] -374> 2014-06-09 16:22:12.249035 7f8692f68700 5 queue_op 0xcc86000 seq 4318 osr(3.516 0x3ed16d0) 828 bytes (queue has 50 ops and 4237151 bytes) -373> 2014-06-09 16:22:12.249038 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3062a120] -372> 2014-06-09 16:22:12.249039 7f868a757700 2 waiting 51 > 50 ops || 4237979 > 104857600 -371> 2014-06-09 16:22:12.249040 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -370> 2014-06-09 16:22:12.249041 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -369> 2014-06-09 16:22:12.249044 7f868cf5c700 2 waiting 51 > 50 ops || 4238012 > 104857600 -368> 2014-06-09 16:22:12.249044 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc653680 (1 active) -367> 2014-06-09 16:22:12.249048 7f868ef60700 2 waiting 51 > 50 ops || 4237979 > 104857600 -366> 2014-06-09 16:22:12.249047 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62ec60 (2 active) -365> 2014-06-09 16:22:12.249051 7f8690763700 2 waiting 51 > 50 ops || 4237979 > 104857600 -364> 2014-06-09 16:22:12.249056 7f8692f68700 10 op_submit_finish 4318 -363> 2014-06-09 16:22:12.249056 7f868ff62700 2 waiting 51 > 50 ops || 4237979 > 104857600 -362> 2014-06-09 16:22:12.249061 7f8689f56700 2 waiting 51 > 50 ops || 4238012 > 104857600 -361> 2014-06-09 16:22:12.249052 7f86a5e11700 5 _do_op 0x324e73e0 seq 4270 osr(3.9f 0x3ed0600)/0x3ed0600 start -360> 2014-06-09 16:22:12.249067 7f86a5e11700 10 _do_transaction on 0xc140e40 -359> 2014-06-09 16:22:12.249067 7f868bf5a700 2 waiting 51 > 50 ops || 4237979 > 104857600 -358> 2014-06-09 16:22:12.249059 7f8692f68700 10 osd.26 pg_epoch: 243 pg[3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 212/212 242/242/211) [8,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 8 info: 3.292( v 213'634 (0'0,213'634] local-les=243 n=122 ec=118 les/c 243/212 242/242/211) -357> 2014-06-09 16:22:12.249073 7f868c75b700 2 waiting 51 > 50 ops || 4456637 > 104857600 -356> 2014-06-09 16:22:12.249074 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -355> 2014-06-09 16:22:12.249088 7f8692f68700 10 log is not dirty -354> 2014-06-09 16:22:12.249093 7f8692f68700 5 queue_transactions existing osr(3.292 0x3ed0ba0)/0x3ed0ba0 -353> 2014-06-09 16:22:12.249096 7f8692f68700 2 waiting 51 > 50 ops || 4237979 > 104857600 -352> 2014-06-09 16:22:12.249099 7f8691765700 2 waiting 51 > 50 ops || 4237979 > 104857600 -351> 2014-06-09 16:22:12.249105 7f868df5e700 2 waiting 51 > 50 ops || 4237979 > 104857600 -350> 2014-06-09 16:22:12.249109 7f8690f64700 2 waiting 51 > 50 ops || 4237977 > 104857600 -349> 2014-06-09 16:22:12.249112 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 -348> 2014-06-09 16:22:12.249118 7f868af58700 2 waiting 51 > 50 ops || 4238012 > 104857600 -347> 2014-06-09 16:22:12.249125 7f868b759700 2 waiting 51 > 50 ops || 4237979 > 104857600 -346> 2014-06-09 16:22:12.249129 7f8692767700 2 waiting 51 > 50 ops || 4237979 > 104857600 -345> 2014-06-09 16:22:12.249129 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -344> 2014-06-09 16:22:12.249135 7f868f761700 2 waiting 51 > 50 ops || 4237975 > 104857600 -343> 2014-06-09 16:22:12.249140 7f868d75d700 2 waiting 51 > 50 ops || 4238012 > 104857600 -342> 2014-06-09 16:22:12.249144 7f868e75f700 2 waiting 51 > 50 ops || 4237979 > 104857600 -341> 2014-06-09 16:22:12.249156 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4270.0.0 <= header.spos 150443.0.0 -340> 2014-06-09 16:22:12.249160 7f86a5e11700 5 submit_transaction r = 0 -339> 2014-06-09 16:22:12.249171 7f86a5e11700 10 _do_op 0x324e73e0 seq 4270 r = 0, finisher 0xbd6c0e0 0 -338> 2014-06-09 16:22:12.249177 7f86a5e11700 10 _finish_op 0x324e73e0 seq 4270 osr(3.9f 0x3ed0600)/0x3ed0600 -337> 2014-06-09 16:22:12.249178 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd709e0] -336> 2014-06-09 16:22:12.249180 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd709e0] -335> 2014-06-09 16:22:12.249181 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -334> 2014-06-09 16:22:12.249182 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -333> 2014-06-09 16:22:12.249186 7f868a757700 10 op_submit_start 4319 -332> 2014-06-09 16:22:12.249188 7f868a757700 5 queue_transactions (trailing journal) 4319 -331> 2014-06-09 16:22:12.249188 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6c0e0] -330> 2014-06-09 16:22:12.249189 7f868a757700 5 queue_op 0x32539da0 seq 4319 osr(3.46f 0x3ed1560) 828 bytes (queue has 50 ops and 4237153 bytes) -329> 2014-06-09 16:22:12.249193 7f868cf5c700 2 waiting 51 > 50 ops || 4238014 > 104857600 -328> 2014-06-09 16:22:12.249193 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6c0e0] -327> 2014-06-09 16:22:12.249195 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -326> 2014-06-09 16:22:12.249196 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -325> 2014-06-09 16:22:12.249197 7f868ef60700 2 waiting 51 > 50 ops || 4237981 > 104857600 -324> 2014-06-09 16:22:12.249199 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62ec60 (1 active) -323> 2014-06-09 16:22:12.249201 7f8690763700 2 waiting 51 > 50 ops || 4237981 > 104857600 -322> 2014-06-09 16:22:12.249205 7f868ff62700 2 waiting 51 > 50 ops || 4237981 > 104857600 -321> 2014-06-09 16:22:12.249210 7f8689f56700 2 waiting 51 > 50 ops || 4238014 > 104857600 -320> 2014-06-09 16:22:12.249202 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a360 (2 active) -319> 2014-06-09 16:22:12.249215 7f868bf5a700 2 waiting 51 > 50 ops || 4237981 > 104857600 -318> 2014-06-09 16:22:12.249217 7f86a5e11700 5 _do_op 0x1e08b9e0 seq 4271 osr(0.1da 0x2bd6920)/0x2bd6920 start -317> 2014-06-09 16:22:12.249221 7f868a757700 10 op_submit_finish 4319 -316> 2014-06-09 16:22:12.249221 7f86a5e11700 10 _do_transaction on 0xbfe40c0 -315> 2014-06-09 16:22:12.249221 7f8691f66700 2 waiting 51 > 50 ops || 4238014 > 104857600 -314> 2014-06-09 16:22:12.249225 7f868c75b700 2 waiting 51 > 50 ops || 4456639 > 104857600 -313> 2014-06-09 16:22:12.249226 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -312> 2014-06-09 16:22:12.249223 7f868a757700 10 osd.26 pg_epoch: 243 pg[2.4b9( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 2.4b9( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -311> 2014-06-09 16:22:12.249253 7f868a757700 10 log is not dirty -310> 2014-06-09 16:22:12.249257 7f868a757700 5 queue_transactions existing osr(2.4b9 0x2ed1730)/0x2ed1730 -309> 2014-06-09 16:22:12.249259 7f868a757700 2 waiting 51 > 50 ops || 4237981 > 104857600 -308> 2014-06-09 16:22:12.249263 7f8691765700 2 waiting 51 > 50 ops || 4237981 > 104857600 -307> 2014-06-09 16:22:12.249267 7f868df5e700 2 waiting 51 > 50 ops || 4237981 > 104857600 -306> 2014-06-09 16:22:12.249266 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -305> 2014-06-09 16:22:12.249271 7f8690f64700 2 waiting 51 > 50 ops || 4237979 > 104857600 -304> 2014-06-09 16:22:12.249274 7f8689755700 2 waiting 51 > 50 ops || 4237981 > 104857600 -303> 2014-06-09 16:22:12.249278 7f8692f68700 2 waiting 51 > 50 ops || 4237981 > 104857600 -302> 2014-06-09 16:22:12.249283 7f868b759700 2 waiting 51 > 50 ops || 4237981 > 104857600 -301> 2014-06-09 16:22:12.249289 7f8692767700 2 waiting 51 > 50 ops || 4237981 > 104857600 -300> 2014-06-09 16:22:12.249294 7f868f761700 2 waiting 51 > 50 ops || 4237977 > 104857600 -299> 2014-06-09 16:22:12.249299 7f868d75d700 2 waiting 51 > 50 ops || 4238014 > 104857600 -298> 2014-06-09 16:22:12.249304 7f868e75f700 2 waiting 51 > 50 ops || 4237981 > 104857600 -297> 2014-06-09 16:22:12.249309 7f868af58700 2 waiting 51 > 50 ops || 4238014 > 104857600 -296> 2014-06-09 16:22:12.249305 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4271.0.0 <= header.spos 150443.0.0 -295> 2014-06-09 16:22:12.249316 7f86a5e11700 5 submit_transaction r = 0 -294> 2014-06-09 16:22:12.249327 7f86a5e11700 10 _do_op 0x1e08b9e0 seq 4271 r = 0, finisher 0x30509480 0 -293> 2014-06-09 16:22:12.249333 7f86a5e11700 10 _finish_op 0x1e08b9e0 seq 4271 osr(0.1da 0x2bd6920)/0x2bd6920 -292> 2014-06-09 16:22:12.249334 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2660] -291> 2014-06-09 16:22:12.249336 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2660] -290> 2014-06-09 16:22:12.249337 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -289> 2014-06-09 16:22:12.249338 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -288> 2014-06-09 16:22:12.249342 7f868cf5c700 10 op_submit_start 4320 -287> 2014-06-09 16:22:12.249344 7f868cf5c700 5 queue_transactions (trailing journal) 4320 -286> 2014-06-09 16:22:12.249344 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x30509480] -285> 2014-06-09 16:22:12.249345 7f868cf5c700 5 queue_op 0x324e6360 seq 4320 osr(2.573 0x2ed18b0) 861 bytes (queue has 50 ops and 4237186 bytes) -284> 2014-06-09 16:22:12.249348 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x30509480] -283> 2014-06-09 16:22:12.249349 7f868ef60700 2 waiting 51 > 50 ops || 4238014 > 104857600 -282> 2014-06-09 16:22:12.249350 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -281> 2014-06-09 16:22:12.249351 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -280> 2014-06-09 16:22:12.249353 7f8690763700 2 waiting 51 > 50 ops || 4238014 > 104857600 -279> 2014-06-09 16:22:12.249354 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a360 (1 active) -278> 2014-06-09 16:22:12.249358 7f868ff62700 2 waiting 51 > 50 ops || 4238014 > 104857600 -277> 2014-06-09 16:22:12.249363 7f8689f56700 2 waiting 51 > 50 ops || 4238047 > 104857600 -276> 2014-06-09 16:22:12.249367 7f868bf5a700 2 waiting 51 > 50 ops || 4238014 > 104857600 -275> 2014-06-09 16:22:12.249368 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xbff5c20 (2 active) -274> 2014-06-09 16:22:12.249373 7f868cf5c700 10 op_submit_finish 4320 -273> 2014-06-09 16:22:12.249372 7f86a5e11700 5 _do_op 0x324f50e0 seq 4272 osr(0.640 0x2bd7c10)/0x2bd7c10 start -272> 2014-06-09 16:22:12.249376 7f86a5e11700 10 _do_transaction on 0x32fa780 -271> 2014-06-09 16:22:12.249381 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -270> 2014-06-09 16:22:12.249377 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -269> 2014-06-09 16:22:12.249389 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.073976 7 0.000316 -268> 2014-06-09 16:22:12.249399 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -267> 2014-06-09 16:22:12.249408 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -266> 2014-06-09 16:22:12.249421 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -265> 2014-06-09 16:22:12.249417 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -264> 2014-06-09 16:22:12.249427 7f868cf5c700 5 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -263> 2014-06-09 16:22:12.249435 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4272.0.0 <= header.spos 150443.0.0 -262> 2014-06-09 16:22:12.249440 7f86a5e11700 5 submit_transaction r = 0 -261> 2014-06-09 16:22:12.249435 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -260> 2014-06-09 16:22:12.249442 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -259> 2014-06-09 16:22:12.249450 7f86a5e11700 10 _do_op 0x324f50e0 seq 4272 r = 0, finisher 0xbef1b00 0 -258> 2014-06-09 16:22:12.249456 7f86a5e11700 10 _finish_op 0x324f50e0 seq 4272 osr(0.640 0x2bd7c10)/0x2bd7c10 -257> 2014-06-09 16:22:12.249452 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=207-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(207-235 up [26,6](26) acting [26,6](26) maybe_went_rw) -256> 2014-06-09 16:22:12.249467 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbef1b00] -255> 2014-06-09 16:22:12.249463 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [6](6) acting [6](6)) -254> 2014-06-09 16:22:12.249471 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbef1b00] -253> 2014-06-09 16:22:12.249473 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -252> 2014-06-09 16:22:12.249474 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -251> 2014-06-09 16:22:12.249472 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -250> 2014-06-09 16:22:12.249477 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xbff5c20 (1 active) -249> 2014-06-09 16:22:12.249480 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fd7a0 (2 active) -248> 2014-06-09 16:22:12.249482 7f86a5e11700 5 _do_op 0xc2d48a0 seq 4273 osr(3.672 0x3ed1350)/0x3ed1350 start -247> 2014-06-09 16:22:12.249485 7f86a5e11700 10 _do_transaction on 0x2c80f00 -246> 2014-06-09 16:22:12.249484 7f868cf5c700 20 osd.26 243 share_map_peer 0xbf5adc0 already has epoch 244 -245> 2014-06-09 16:22:12.249490 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -244> 2014-06-09 16:22:12.249486 7f868cf5c700 1 -- 172.16.96.202:6850/30904 --> osd.6 172.16.96.201:6849/52045 -- pg_info(1 pgs e243:2.6dd) v4 -- ?+0 0x30c00000 -243> 2014-06-09 16:22:12.249499 7f868cf5c700 15 osd.26 pg_epoch: 243 pg[2.6dd( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:137 -242> 2014-06-09 16:22:12.249529 7f868cf5c700 10 log is not dirty -241> 2014-06-09 16:22:12.249533 7f868cf5c700 5 queue_transactions existing osr(2.6dd 0x2ed1e50)/0x2ed1e50 -240> 2014-06-09 16:22:12.249536 7f868cf5c700 10 op_submit_start 4321 -239> 2014-06-09 16:22:12.249537 7f868cf5c700 5 queue_transactions (trailing journal) 4321 -238> 2014-06-09 16:22:12.249538 7f868cf5c700 5 queue_op 0x2e20c00 seq 4321 osr(2.6dd 0x2ed1e50) 861 bytes (queue has 50 ops and 4237219 bytes) -237> 2014-06-09 16:22:12.249541 7f868cf5c700 10 op_submit_finish 4321 -236> 2014-06-09 16:22:12.249543 7f868cf5c700 10 osd.26 pg_epoch: 243 pg[2.55b( empty local-les=243 n=0 ec=1 les/c 213/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.55b( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) -235> 2014-06-09 16:22:12.249559 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -234> 2014-06-09 16:22:12.249573 7f868cf5c700 10 log is not dirty -233> 2014-06-09 16:22:12.249577 7f868cf5c700 5 queue_transactions existing osr(2.55b 0x2ed1930)/0x2ed1930 -232> 2014-06-09 16:22:12.249576 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4273.0.0 <= header.spos 150443.0.0 -231> 2014-06-09 16:22:12.249580 7f868cf5c700 2 waiting 51 > 50 ops || 4238047 > 104857600 -230> 2014-06-09 16:22:12.249581 7f86a5e11700 5 submit_transaction r = 0 -229> 2014-06-09 16:22:12.249584 7f868ef60700 2 waiting 51 > 50 ops || 4238047 > 104857600 -228> 2014-06-09 16:22:12.249589 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x314566e0] -227> 2014-06-09 16:22:12.249591 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x314566e0] -226> 2014-06-09 16:22:12.249590 7f8691f66700 2 waiting 51 > 50 ops || 4238080 > 104857600 -225> 2014-06-09 16:22:12.249592 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -224> 2014-06-09 16:22:12.249593 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -223> 2014-06-09 16:22:12.249595 7f868a757700 2 waiting 51 > 50 ops || 4238047 > 104857600 -222> 2014-06-09 16:22:12.249600 7f8691765700 2 waiting 51 > 50 ops || 4238047 > 104857600 -221> 2014-06-09 16:22:12.249600 7f86a5e11700 10 _do_op 0xc2d48a0 seq 4273 r = 0, finisher 0x324767c0 0 -220> 2014-06-09 16:22:12.249603 7f868df5e700 2 waiting 51 > 50 ops || 4238047 > 104857600 -219> 2014-06-09 16:22:12.249606 7f86a5e11700 10 _finish_op 0xc2d48a0 seq 4273 osr(3.672 0x3ed1350)/0x3ed1350 -218> 2014-06-09 16:22:12.249607 7f8690f64700 2 waiting 51 > 50 ops || 4238045 > 104857600 -217> 2014-06-09 16:22:12.249616 7f8689755700 10 op_submit_start 4322 -216> 2014-06-09 16:22:12.249617 7f8689755700 5 queue_transactions (trailing journal) 4322 -215> 2014-06-09 16:22:12.249618 7f868b759700 2 waiting 51 > 50 ops || 4238014 > 104857600 -214> 2014-06-09 16:22:12.249619 7f8689755700 5 queue_op 0x33a2720 seq 4322 osr(0.1d4 0x2bd6940) 828 bytes (queue has 50 ops and 4237186 bytes) -213> 2014-06-09 16:22:12.249624 7f8692767700 2 waiting 51 > 50 ops || 4238014 > 104857600 -212> 2014-06-09 16:22:12.249627 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x324767c0] -211> 2014-06-09 16:22:12.249629 7f8692f68700 2 waiting 51 > 50 ops || 4238014 > 104857600 -210> 2014-06-09 16:22:12.249631 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x324767c0] -209> 2014-06-09 16:22:12.249633 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -208> 2014-06-09 16:22:12.249633 7f868d75d700 2 waiting 51 > 50 ops || 4238047 > 104857600 -207> 2014-06-09 16:22:12.249634 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -206> 2014-06-09 16:22:12.249637 7f868e75f700 2 waiting 51 > 50 ops || 4238014 > 104857600 -205> 2014-06-09 16:22:12.249641 7f868cf5c700 2 waiting 51 > 50 ops || 4238014 > 104857600 -204> 2014-06-09 16:22:12.249638 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fd7a0 (1 active) -203> 2014-06-09 16:22:12.249644 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc6c0 (2 active) -202> 2014-06-09 16:22:12.249645 7f8690763700 2 waiting 51 > 50 ops || 4238014 > 104857600 -201> 2014-06-09 16:22:12.249650 7f8689755700 10 op_submit_finish 4322 -200> 2014-06-09 16:22:12.249650 7f868ff62700 2 waiting 51 > 50 ops || 4238014 > 104857600 -199> 2014-06-09 16:22:12.249655 7f8689f56700 2 waiting 51 > 50 ops || 4238047 > 104857600 -198> 2014-06-09 16:22:12.249659 7f86a5e11700 5 _do_op 0x16fd4900 seq 4274 osr(2.102 0x2ed0710)/0x2ed0710 start -197> 2014-06-09 16:22:12.249663 7f86a5e11700 10 _do_transaction on 0x3472780 -196> 2014-06-09 16:22:12.249653 7f8689755700 10 osd.26 pg_epoch: 243 pg[1.fa( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/220) [0,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 0 info: 1.fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/220) -195> 2014-06-09 16:22:12.249669 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -194> 2014-06-09 16:22:12.249687 7f8689755700 10 log is not dirty -193> 2014-06-09 16:22:12.249691 7f8689755700 5 queue_transactions existing osr(1.fa 0x384c970)/0x384c970 -192> 2014-06-09 16:22:12.249693 7f8689755700 2 waiting 51 > 50 ops || 4238012 > 104857600 -191> 2014-06-09 16:22:12.249697 7f868bf5a700 2 waiting 51 > 50 ops || 4238014 > 104857600 -190> 2014-06-09 16:22:12.249702 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32476580] -189> 2014-06-09 16:22:12.249710 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -188> 2014-06-09 16:22:12.249704 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery -187> 2014-06-09 16:22:12.249715 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays -186> 2014-06-09 16:22:12.249726 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4274.0.0 <= header.spos 150443.0.0 -185> 2014-06-09 16:22:12.249730 7f86a5e11700 5 submit_transaction r = 0 -184> 2014-06-09 16:22:12.249724 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.672( v 214'615 (0'0,214'615] local-les=243 n=121 ec=118 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1013 -183> 2014-06-09 16:22:12.249733 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32476580] -182> 2014-06-09 16:22:12.249734 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -181> 2014-06-09 16:22:12.249735 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -180> 2014-06-09 16:22:12.249747 7f86a5e11700 10 _do_op 0x16fd4900 seq 4274 r = 0, finisher 0x1b0c19c0 0 -179> 2014-06-09 16:22:12.249753 7f86a5e11700 10 _finish_op 0x16fd4900 seq 4274 osr(2.102 0x2ed0710)/0x2ed0710 -178> 2014-06-09 16:22:12.249754 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3281af60] -177> 2014-06-09 16:22:12.249757 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -176> 2014-06-09 16:22:12.249767 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x1b0c19c0] -175> 2014-06-09 16:22:12.249765 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -174> 2014-06-09 16:22:12.249771 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x1b0c19c0] -173> 2014-06-09 16:22:12.249775 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -172> 2014-06-09 16:22:12.249776 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -171> 2014-06-09 16:22:12.249773 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.102( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,6] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:138 -170> 2014-06-09 16:22:12.249779 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc6c0 (1 active) -169> 2014-06-09 16:22:12.249781 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3281af60] -168> 2014-06-09 16:22:12.249782 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -167> 2014-06-09 16:22:12.249782 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6d0120 (2 active) -166> 2014-06-09 16:22:12.249783 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -165> 2014-06-09 16:22:12.249784 7f86a5e11700 5 _do_op 0x2e21aa0 seq 4275 osr(0.57b 0x2bd77b0)/0x2bd77b0 start -164> 2014-06-09 16:22:12.249787 7f86a5e11700 10 _do_transaction on 0xc0d58c0 -163> 2014-06-09 16:22:12.249788 7f868ef60700 10 op_submit_start 4323 -162> 2014-06-09 16:22:12.249790 7f868ef60700 5 queue_transactions (trailing journal) 4323 -161> 2014-06-09 16:22:12.249791 7f868ef60700 5 queue_op 0xcdaa9c0 seq 4323 osr(2.69c 0x2ed1f20) 828 bytes (queue has 50 ops and 4237153 bytes) -160> 2014-06-09 16:22:12.249792 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -159> 2014-06-09 16:22:12.249794 7f868ef60700 10 op_submit_finish 4323 -158> 2014-06-09 16:22:12.249797 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.220( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 2.220( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/212) -157> 2014-06-09 16:22:12.249827 7f868ef60700 10 log is not dirty -156> 2014-06-09 16:22:12.249831 7f868ef60700 5 queue_transactions existing osr(2.220 0x2ed0b10)/0x2ed0b10 -155> 2014-06-09 16:22:12.249831 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -154> 2014-06-09 16:22:12.249833 7f868ef60700 2 waiting 51 > 50 ops || 4237981 > 104857600 -153> 2014-06-09 16:22:12.249837 7f8691f66700 2 waiting 51 > 50 ops || 4238014 > 104857600 -152> 2014-06-09 16:22:12.249841 7f868a757700 2 waiting 51 > 50 ops || 4237981 > 104857600 -151> 2014-06-09 16:22:12.249845 7f8691765700 2 waiting 51 > 50 ops || 4237981 > 104857600 -150> 2014-06-09 16:22:12.249845 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4275.0.0 <= header.spos 150443.0.0 -149> 2014-06-09 16:22:12.249848 7f868df5e700 2 waiting 51 > 50 ops || 4237981 > 104857600 -148> 2014-06-09 16:22:12.249849 7f86a5e11700 5 submit_transaction r = 0 -147> 2014-06-09 16:22:12.249852 7f8690f64700 2 waiting 51 > 50 ops || 4237979 > 104857600 -146> 2014-06-09 16:22:12.249857 7f868f761700 2 waiting 51 > 50 ops || 4237977 > 104857600 -145> 2014-06-09 16:22:12.249863 7f868af58700 2 waiting 51 > 50 ops || 4238014 > 104857600 -144> 2014-06-09 16:22:12.249868 7f8692767700 2 waiting 51 > 50 ops || 4237981 > 104857600 -143> 2014-06-09 16:22:12.249873 7f8692f68700 2 waiting 51 > 50 ops || 4237981 > 104857600 -142> 2014-06-09 16:22:12.249876 7f868d75d700 2 waiting 51 > 50 ops || 4238014 > 104857600 -141> 2014-06-09 16:22:12.249876 7f86a5e11700 10 _do_op 0x2e21aa0 seq 4275 r = 0, finisher 0x2d739160 0 -140> 2014-06-09 16:22:12.249881 7f868e75f700 2 waiting 51 > 50 ops || 4237981 > 104857600 -139> 2014-06-09 16:22:12.249882 7f86a5e11700 10 _finish_op 0x2e21aa0 seq 4275 osr(0.57b 0x2bd77b0)/0x2bd77b0 -138> 2014-06-09 16:22:12.249884 7f868cf5c700 2 waiting 51 > 50 ops || 4237981 > 104857600 -137> 2014-06-09 16:22:12.249888 7f8690763700 2 waiting 51 > 50 ops || 4237981 > 104857600 -136> 2014-06-09 16:22:12.249892 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc168520] -135> 2014-06-09 16:22:12.249894 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc168520] -134> 2014-06-09 16:22:12.249896 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -133> 2014-06-09 16:22:12.249895 7f868ff62700 2 waiting 51 > 50 ops || 4237981 > 104857600 -132> 2014-06-09 16:22:12.249897 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -131> 2014-06-09 16:22:12.249900 7f8689f56700 2 waiting 51 > 50 ops || 4238014 > 104857600 -130> 2014-06-09 16:22:12.249905 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 -129> 2014-06-09 16:22:12.249909 7f868bf5a700 2 waiting 51 > 50 ops || 4237981 > 104857600 -128> 2014-06-09 16:22:12.249913 7f868b759700 2 waiting 51 > 50 ops || 4237981 > 104857600 -127> 2014-06-09 16:22:12.249925 7f868ef60700 10 op_submit_start 4324 -126> 2014-06-09 16:22:12.249927 7f868ef60700 5 queue_transactions (trailing journal) 4324 -125> 2014-06-09 16:22:12.249928 7f868ef60700 5 queue_op 0xcdaad20 seq 4324 osr(2.220 0x2ed0b10) 828 bytes (queue has 50 ops and 4237153 bytes) -124> 2014-06-09 16:22:12.249930 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2d739160] -123> 2014-06-09 16:22:12.249932 7f868a757700 2 waiting 51 > 50 ops || 4237981 > 104857600 -122> 2014-06-09 16:22:12.249934 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2d739160] -121> 2014-06-09 16:22:12.249936 7f8691765700 2 waiting 51 > 50 ops || 4237981 > 104857600 -120> 2014-06-09 16:22:12.249936 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -119> 2014-06-09 16:22:12.249937 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -118> 2014-06-09 16:22:12.249939 7f868df5e700 2 waiting 51 > 50 ops || 4237981 > 104857600 -117> 2014-06-09 16:22:12.249940 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6d0120 (1 active) -116> 2014-06-09 16:22:12.249943 7f8690f64700 2 waiting 51 > 50 ops || 4237979 > 104857600 -115> 2014-06-09 16:22:12.249943 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c0900 (2 active) -114> 2014-06-09 16:22:12.249947 7f8691f66700 2 waiting 51 > 50 ops || 4238014 > 104857600 -113> 2014-06-09 16:22:12.249950 7f868f761700 2 waiting 51 > 50 ops || 4237977 > 104857600 -112> 2014-06-09 16:22:12.249952 7f868ef60700 10 op_submit_finish 4324 -111> 2014-06-09 16:22:12.249956 7f868af58700 2 waiting 51 > 50 ops || 4238014 > 104857600 -110> 2014-06-09 16:22:12.249961 7f86a5e11700 5 _do_op 0xc88bec0 seq 4276 osr(3.48d 0x3ed14e0)/0x3ed14e0 start -109> 2014-06-09 16:22:12.249964 7f86a5e11700 10 _do_transaction on 0xc262b40 -108> 2014-06-09 16:22:12.249955 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.f2( empty local-les=243 n=0 ec=1 les/c 243/208 242/242/207) [6,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 6 info: 2.f2( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/207) -107> 2014-06-09 16:22:12.249969 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -106> 2014-06-09 16:22:12.249985 7f868ef60700 10 log is not dirty -105> 2014-06-09 16:22:12.249989 7f868ef60700 5 queue_transactions existing osr(2.f2 0x2ed07e0)/0x2ed07e0 -104> 2014-06-09 16:22:12.249991 7f868ef60700 2 waiting 51 > 50 ops || 4237979 > 104857600 -103> 2014-06-09 16:22:12.249994 7f8692767700 2 waiting 51 > 50 ops || 4237981 > 104857600 -102> 2014-06-09 16:22:12.249999 7f868d75d700 2 waiting 51 > 50 ops || 4238014 > 104857600 -101> 2014-06-09 16:22:12.250003 7f868e75f700 2 waiting 51 > 50 ops || 4237981 > 104857600 -100> 2014-06-09 16:22:12.250007 7f8692f68700 2 waiting 51 > 50 ops || 4237981 > 104857600 -99> 2014-06-09 16:22:12.250010 7f8690763700 2 waiting 51 > 50 ops || 4237981 > 104857600 -98> 2014-06-09 16:22:12.250010 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -97> 2014-06-09 16:22:12.250014 7f868cf5c700 2 waiting 51 > 50 ops || 4237981 > 104857600 -96> 2014-06-09 16:22:12.250019 7f8689f56700 2 waiting 51 > 50 ops || 4238014 > 104857600 -95> 2014-06-09 16:22:12.250025 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 -94> 2014-06-09 16:22:12.250030 7f868bf5a700 2 waiting 51 > 50 ops || 4237981 > 104857600 -93> 2014-06-09 16:22:12.250033 7f868b759700 2 waiting 51 > 50 ops || 4237981 > 104857600 -92> 2014-06-09 16:22:12.250037 7f868ff62700 2 waiting 51 > 50 ops || 4237981 > 104857600 -91> 2014-06-09 16:22:12.250042 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4276.0.0 <= header.spos 150443.0.0 -90> 2014-06-09 16:22:12.250046 7f86a5e11700 5 submit_transaction r = 0 -89> 2014-06-09 16:22:12.250058 7f86a5e11700 10 _do_op 0xc88bec0 seq 4276 r = 0, finisher 0xbd7d420 0 -88> 2014-06-09 16:22:12.250064 7f86a5e11700 10 _finish_op 0xc88bec0 seq 4276 osr(3.48d 0x3ed14e0)/0x3ed14e0 -87> 2014-06-09 16:22:12.250066 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc168d40] -86> 2014-06-09 16:22:12.250068 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc168d40] -85> 2014-06-09 16:22:12.250070 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -84> 2014-06-09 16:22:12.250071 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -83> 2014-06-09 16:22:12.250075 7f868a757700 10 op_submit_start 4325 -82> 2014-06-09 16:22:12.250076 7f868a757700 5 queue_transactions (trailing journal) 4325 -81> 2014-06-09 16:22:12.250075 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd7d420] -80> 2014-06-09 16:22:12.250077 7f868a757700 5 queue_op 0x32539d40 seq 4325 osr(2.4b9 0x2ed1730) 828 bytes (queue has 50 ops and 4237153 bytes) -79> 2014-06-09 16:22:12.250081 7f8691765700 2 waiting 51 > 50 ops || 4237981 > 104857600 -78> 2014-06-09 16:22:12.250081 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd7d420] -77> 2014-06-09 16:22:12.250084 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -76> 2014-06-09 16:22:12.250085 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -75> 2014-06-09 16:22:12.250085 7f868df5e700 2 waiting 51 > 50 ops || 4237981 > 104857600 -74> 2014-06-09 16:22:12.250078 7f86a5610700 -1 os/KeyValueStore.cc: In function 'unsigned int KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)' thread 7f86a5610700 time 2014-06-09 16:22:12.248673 os/KeyValueStore.cc: 1524: FAILED assert(0 == "unexpected error") ceph version 0.80-820-g5d606cd (5d606cd0d00698699c91a378a1bd9f71cc8a77c9) 1: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x750) [0x9e6f20] 2: (KeyValueStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9e8d1e] 3: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x9a) [0x9e8e2a] 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb62a3a] 5: (ThreadPool::WorkThread::entry()+0x10) [0xb63c90] 6: (()+0x7e9a) [0x7f86aed2de9a] 7: (clone()+0x6d) [0x7f86ad2d8ccd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. -73> 2014-06-09 16:22:12.250088 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c0900 (1 active) -72> 2014-06-09 16:22:12.250089 7f8690f64700 2 waiting 51 > 50 ops || 4237979 > 104857600 -71> 2014-06-09 16:22:12.250090 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc406900 (2 active) -70> 2014-06-09 16:22:12.250093 7f8691f66700 2 waiting 51 > 50 ops || 4238014 > 104857600 -69> 2014-06-09 16:22:12.250098 7f868a757700 10 op_submit_finish 4325 -68> 2014-06-09 16:22:12.250098 7f868f761700 2 waiting 51 > 50 ops || 4237977 > 104857600 -67> 2014-06-09 16:22:12.250103 7f868af58700 2 waiting 51 > 50 ops || 4238014 > 104857600 -66> 2014-06-09 16:22:12.250094 7f86a5e11700 5 _do_op 0xd471ce0 seq 4277 osr(0.629 0x2bd7c90)/0x2bd7c90 start -65> 2014-06-09 16:22:12.250109 7f86a5e11700 10 _do_transaction on 0xc28cf00 -64> 2014-06-09 16:22:12.250101 7f868a757700 10 osd.26 pg_epoch: 243 pg[0.33a( empty local-les=243 n=0 ec=1 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.33a( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) -63> 2014-06-09 16:22:12.250116 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -62> 2014-06-09 16:22:12.250130 7f868a757700 10 log is not dirty -61> 2014-06-09 16:22:12.250134 7f868a757700 5 queue_transactions existing osr(0.33a 0x2bd70b0)/0x2bd70b0 -60> 2014-06-09 16:22:12.250136 7f868a757700 2 waiting 51 > 50 ops || 4237981 > 104857600 -59> 2014-06-09 16:22:12.250140 7f868ef60700 2 waiting 51 > 50 ops || 4237979 > 104857600 -58> 2014-06-09 16:22:12.250145 7f8692767700 2 waiting 51 > 50 ops || 4237981 > 104857600 -57> 2014-06-09 16:22:12.250148 7f868d75d700 2 waiting 51 > 50 ops || 4238014 > 104857600 -56> 2014-06-09 16:22:12.250152 7f868e75f700 2 waiting 51 > 50 ops || 4237981 > 104857600 -55> 2014-06-09 16:22:12.250156 7f8692f68700 2 waiting 51 > 50 ops || 4237981 > 104857600 -54> 2014-06-09 16:22:12.250156 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -53> 2014-06-09 16:22:12.250160 7f8690763700 2 waiting 51 > 50 ops || 4237981 > 104857600 -52> 2014-06-09 16:22:12.250163 7f868cf5c700 2 waiting 51 > 50 ops || 4237981 > 104857600 -51> 2014-06-09 16:22:12.250169 7f8689f56700 2 waiting 51 > 50 ops || 4238014 > 104857600 -50> 2014-06-09 16:22:12.250174 7f8689755700 2 waiting 51 > 50 ops || 4237979 > 104857600 -49> 2014-06-09 16:22:12.250179 7f868bf5a700 2 waiting 51 > 50 ops || 4237981 > 104857600 -48> 2014-06-09 16:22:12.250178 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4277.0.0 <= header.spos 150443.0.0 -47> 2014-06-09 16:22:12.250182 7f86a5e11700 5 submit_transaction r = 0 -46> 2014-06-09 16:22:12.250182 7f868b759700 2 waiting 51 > 50 ops || 4237981 > 104857600 -45> 2014-06-09 16:22:12.250187 7f868ff62700 2 waiting 51 > 50 ops || 4237981 > 104857600 -44> 2014-06-09 16:22:12.250198 7f86a5e11700 10 _do_op 0xd471ce0 seq 4277 r = 0, finisher 0xbd6fda0 0 -43> 2014-06-09 16:22:12.250203 7f86a5e11700 10 _finish_op 0xd471ce0 seq 4277 osr(0.629 0x2bd7c90)/0x2bd7c90 -42> 2014-06-09 16:22:12.250205 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x2e0e9a0] -41> 2014-06-09 16:22:12.250213 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6fda0] -40> 2014-06-09 16:22:12.250208 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -39> 2014-06-09 16:22:12.250218 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6fda0] -38> 2014-06-09 16:22:12.250219 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -37> 2014-06-09 16:22:12.250220 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -36> 2014-06-09 16:22:12.250223 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc406900 (1 active) -35> 2014-06-09 16:22:12.250218 7f86a460e700 10 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -34> 2014-06-09 16:22:12.250225 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9238c0 (2 active) -33> 2014-06-09 16:22:12.250227 7f86a5e11700 5 _do_op 0xc88a9c0 seq 4278 osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 start -32> 2014-06-09 16:22:12.250230 7f86a5e11700 10 _do_transaction on 0xbf7dc80 -31> 2014-06-09 16:22:12.250227 7f86a460e700 15 osd.26 pg_epoch: 243 pg[0.629( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:153 -30> 2014-06-09 16:22:12.250234 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -29> 2014-06-09 16:22:12.250236 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x2e0e9a0] -28> 2014-06-09 16:22:12.250238 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -27> 2014-06-09 16:22:12.250239 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -26> 2014-06-09 16:22:12.250243 7f8691765700 10 op_submit_start 4326 -25> 2014-06-09 16:22:12.250245 7f8691765700 5 queue_transactions (trailing journal) 4326 -24> 2014-06-09 16:22:12.250246 7f8691765700 5 queue_op 0x325266c0 seq 4326 osr(1.438 0x384d710) 828 bytes (queue has 50 ops and 4237120 bytes) -23> 2014-06-09 16:22:12.250249 7f8691765700 10 op_submit_finish 4326 -22> 2014-06-09 16:22:12.250252 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 243/191 242/242/223) -21> 2014-06-09 16:22:12.250288 7f8691765700 10 log is not dirty -20> 2014-06-09 16:22:12.250292 7f8691765700 5 queue_transactions existing osr(3.1b5 0x3ed0a80)/0x3ed0a80 -19> 2014-06-09 16:22:12.250295 7f8691765700 2 waiting 51 > 50 ops || 4237948 > 104857600 -18> 2014-06-09 16:22:12.250299 7f868df5e700 2 waiting 51 > 50 ops || 4237948 > 104857600 -17> 2014-06-09 16:22:12.250303 7f8690f64700 2 waiting 51 > 50 ops || 4237946 > 104857600 -16> 2014-06-09 16:22:12.250307 7f8691f66700 2 waiting 51 > 50 ops || 4237981 > 104857600 -15> 2014-06-09 16:22:12.250312 7f868f761700 2 waiting 51 > 50 ops || 4237944 > 104857600 -14> 2014-06-09 16:22:12.250318 7f868af58700 2 waiting 51 > 50 ops || 4237981 > 104857600 -13> 2014-06-09 16:22:12.250323 7f868a757700 2 waiting 51 > 50 ops || 4237948 > 104857600 -12> 2014-06-09 16:22:12.250327 7f868ef60700 2 waiting 51 > 50 ops || 4237946 > 104857600 -11> 2014-06-09 16:22:12.250331 7f8692767700 2 waiting 51 > 50 ops || 4237948 > 104857600 -10> 2014-06-09 16:22:12.250335 7f868d75d700 2 waiting 51 > 50 ops || 4237981 > 104857600 -9> 2014-06-09 16:22:12.250339 7f868e75f700 2 waiting 51 > 50 ops || 4237948 > 104857600 -8> 2014-06-09 16:22:12.250340 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -7> 2014-06-09 16:22:12.250342 7f8692f68700 2 waiting 51 > 50 ops || 4237948 > 104857600 -6> 2014-06-09 16:22:12.250346 7f8690763700 2 waiting 51 > 50 ops || 4237948 > 104857600 -5> 2014-06-09 16:22:12.250350 7f868cf5c700 2 waiting 51 > 50 ops || 4237948 > 104857600 -4> 2014-06-09 16:22:12.250355 7f8689f56700 2 waiting 51 > 50 ops || 4237981 > 104857600 -3> 2014-06-09 16:22:12.250360 7f8689755700 2 waiting 51 > 50 ops || 4237946 > 104857600 -2> 2014-06-09 16:22:12.250361 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4278.0.0 <= header.spos 150443.0.0 -1> 2014-06-09 16:22:12.250364 7f86a5e11700 5 submit_transaction r = 0 0> 2014-06-09 16:22:12.250365 7f868bf5a700 2 waiting 51 > 50 ops || 4237948 > 104857600 --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 20/20 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 20/20 osd 20/20 optracker 0/ 5 objclass 20/20 filestore 20/20 keyvaluestore 1/ 3 journal 0/ 5 ms 20/20 mon 0/10 monc 1/ 5 paxos 20/20 tp 1/ 5 auth 1/ 5 crypto 20/20 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /log/osd.26.log --- end dump of recent events --- 2014-06-09 16:22:12.250368 7f868b759700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250374 7f868ff62700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250387 7f86a5e11700 10 _do_op 0xc88a9c0 seq 4278 r = 0, finisher 0x3050b320 0 2014-06-09 16:22:12.250393 7f86a5e11700 10 _finish_op 0xc88a9c0 seq 4278 osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 2014-06-09 16:22:12.250394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdbaa40] 2014-06-09 16:22:12.250397 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdbaa40] 2014-06-09 16:22:12.250398 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.250399 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.250403 7f8691765700 10 op_submit_start 4327 2014-06-09 16:22:12.250405 7f8691765700 5 queue_transactions (trailing journal) 4327 2014-06-09 16:22:12.250405 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050b320] 2014-06-09 16:22:12.250406 7f8691765700 5 queue_op 0x32526660 seq 4327 osr(3.1b5 0x3ed0a80) 828 bytes (queue has 50 ops and 4237120 bytes) 2014-06-09 16:22:12.250409 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050b320] 2014-06-09 16:22:12.250410 7f868df5e700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250411 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.250412 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.250414 7f8690f64700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250415 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9238c0 (1 active) 2014-06-09 16:22:12.250418 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287200 (2 active) 2014-06-09 16:22:12.250418 7f8691f66700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250423 7f8691765700 10 op_submit_finish 4327 2014-06-09 16:22:12.250423 7f868f761700 2 waiting 51 > 50 ops || 4237944 > 104857600 2014-06-09 16:22:12.250428 7f868af58700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250433 7f86a5e11700 5 _do_op 0x24104300 seq 4279 osr(1.302 0x384d010)/0x384d010 start 2014-06-09 16:22:12.250436 7f86a5e11700 10 _do_transaction on 0xbdbd980 2014-06-09 16:22:12.250426 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 243/203 242/242/202) 2014-06-09 16:22:12.250441 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.250459 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.250463 7f8691765700 5 queue_transactions existing osr(3.1b1 0x3ed0a70)/0x3ed0a70 2014-06-09 16:22:12.250465 7f8691765700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250469 7f868a757700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250473 7f868ef60700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250477 7f8692767700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250480 7f868d75d700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250483 7f868e75f700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250482 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.250487 7f8692f68700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250490 7f8690763700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250494 7f868cf5c700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250499 7f8689f56700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250505 7f8689755700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250496 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4279.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.250508 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.250509 7f868bf5a700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250512 7f868b759700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250517 7f868ff62700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250527 7f86a5e11700 10 _do_op 0x24104300 seq 4279 r = 0, finisher 0x2e0eb60 0 2014-06-09 16:22:12.250533 7f86a5e11700 10 _finish_op 0x24104300 seq 4279 osr(1.302 0x384d010)/0x384d010 2014-06-09 16:22:12.250534 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc83bc0] 2014-06-09 16:22:12.250536 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc83bc0] 2014-06-09 16:22:12.250537 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.250538 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.250542 7f868df5e700 10 op_submit_start 4328 2014-06-09 16:22:12.250543 7f868df5e700 5 queue_transactions (trailing journal) 4328 2014-06-09 16:22:12.250544 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0eb60] 2014-06-09 16:22:12.250544 7f868df5e700 5 queue_op 0xcca6540 seq 4328 osr(2.101 0x2ed0730) 828 bytes (queue has 50 ops and 4237120 bytes) 2014-06-09 16:22:12.250548 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0eb60] 2014-06-09 16:22:12.250549 7f8690f64700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250550 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.250551 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.250552 7f8691f66700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250557 7f868f761700 2 waiting 51 > 50 ops || 4237944 > 104857600 2014-06-09 16:22:12.250562 7f868af58700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250554 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287200 (1 active) 2014-06-09 16:22:12.250566 7f8691765700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250568 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b440 (2 active) 2014-06-09 16:22:12.250571 7f868a757700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250572 7f86a5e11700 5 _do_op 0x30032c60 seq 4280 osr(1.465 0x384d650)/0x384d650 start 2014-06-09 16:22:12.250575 7f86a5e11700 10 _do_transaction on 0xbf7ce40 2014-06-09 16:22:12.250575 7f868df5e700 10 op_submit_finish 4328 2014-06-09 16:22:12.250580 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.250579 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.61f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) 2014-06-09 16:22:12.250610 7f868df5e700 10 log is not dirty 2014-06-09 16:22:12.250614 7f868df5e700 5 queue_transactions existing osr(0.61f 0x2bd7d30)/0x2bd7d30 2014-06-09 16:22:12.250617 7f868df5e700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250620 7f868ef60700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250619 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.250624 7f8692767700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250628 7f868d75d700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250631 7f868e75f700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250634 7f8692f68700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250633 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4280.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.250637 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.250638 7f8690763700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250641 7f868cf5c700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250647 7f8689f56700 2 waiting 51 > 50 ops || 4237981 > 104857600 2014-06-09 16:22:12.250653 7f8689755700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250657 7f868bf5a700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250658 7f86a5e11700 10 _do_op 0x30032c60 seq 4280 r = 0, finisher 0xbd720e0 0 2014-06-09 16:22:12.250661 7f868b759700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250665 7f868ff62700 2 waiting 51 > 50 ops || 4237948 > 104857600 2014-06-09 16:22:12.250671 7f86a5e11700 10 _finish_op 0x30032c60 seq 4280 osr(1.465 0x384d650)/0x384d650 2014-06-09 16:22:12.250673 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd64580] 2014-06-09 16:22:12.250675 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd64580] 2014-06-09 16:22:12.250676 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.250677 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.250681 7f8690f64700 10 op_submit_start 4329 2014-06-09 16:22:12.250683 7f8690f64700 5 queue_transactions (trailing journal) 4329 2014-06-09 16:22:12.250683 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd720e0] 2014-06-09 16:22:12.250684 7f8690f64700 5 queue_op 0x32527140 seq 4329 osr(1.a6 0x384c750) 826 bytes (queue has 50 ops and 4237118 bytes) 2014-06-09 16:22:12.250687 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd720e0] 2014-06-09 16:22:12.250688 7f8691f66700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.250689 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.250690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.250692 7f868f761700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250697 7f868af58700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.250693 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b440 (1 active) 2014-06-09 16:22:12.250702 7f8691765700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250702 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac5e60 (2 active) 2014-06-09 16:22:12.250706 7f86a5e11700 5 _do_op 0x1e08b7a0 seq 4281 osr(2.4af 0x2ed1790)/0x2ed1790 start 2014-06-09 16:22:12.250707 7f8690f64700 10 op_submit_finish 4329 2014-06-09 16:22:12.250709 7f86a5e11700 10 _do_transaction on 0xc262900 2014-06-09 16:22:12.250714 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.250710 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.408( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) 2014-06-09 16:22:12.250739 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.250743 7f8690f64700 5 queue_transactions existing osr(1.408 0x384d770)/0x384d770 2014-06-09 16:22:12.250745 7f8690f64700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250749 7f868a757700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250753 7f868df5e700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250753 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.250757 7f868ef60700 2 waiting 51 > 50 ops || 4237944 > 104857600 2014-06-09 16:22:12.250760 7f8692767700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250764 7f868d75d700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.250768 7f868e75f700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250767 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4281.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.250771 7f8692f68700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250772 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.250775 7f8690763700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250778 7f868cf5c700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250784 7f8689f56700 2 waiting 51 > 50 ops || 4237979 > 104857600 2014-06-09 16:22:12.250789 7f8689755700 2 waiting 51 > 50 ops || 4237944 > 104857600 2014-06-09 16:22:12.250792 7f86a5e11700 10 _do_op 0x1e08b7a0 seq 4281 r = 0, finisher 0x31434800 0 2014-06-09 16:22:12.250794 7f868bf5a700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250798 7f86a5e11700 10 _finish_op 0x1e08b7a0 seq 4281 osr(2.4af 0x2ed1790)/0x2ed1790 2014-06-09 16:22:12.250799 7f868b759700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250803 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324811c0] 2014-06-09 16:22:12.250803 7f868ff62700 2 waiting 51 > 50 ops || 4237946 > 104857600 2014-06-09 16:22:12.250805 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324811c0] 2014-06-09 16:22:12.250806 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.250807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.250815 7f868f761700 10 op_submit_start 4330 2014-06-09 16:22:12.250816 7f868f761700 5 queue_transactions (trailing journal) 4330 2014-06-09 16:22:12.250817 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31434800] 2014-06-09 16:22:12.250817 7f868f761700 5 queue_op 0xbdc99e0 seq 4330 osr(0.f 0x2bd60e0) 824 bytes (queue has 50 ops and 4237114 bytes) 2014-06-09 16:22:12.250822 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31434800] 2014-06-09 16:22:12.250822 7f868af58700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.250823 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.250824 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.250826 7f8691765700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250828 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac5e60 (1 active) 2014-06-09 16:22:12.250830 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa900 (2 active) 2014-06-09 16:22:12.250830 7f8690f64700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250836 7f868f761700 10 op_submit_finish 4330 2014-06-09 16:22:12.250835 7f86a5e11700 5 _do_op 0x324f4f60 seq 4282 osr(0.530 0x2bd78f0)/0x2bd78f0 start 2014-06-09 16:22:12.250837 7f86a5e11700 10 _do_transaction on 0xc227800 2014-06-09 16:22:12.250843 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.250839 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) 2014-06-09 16:22:12.250867 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.250871 7f868f761700 5 queue_transactions existing osr(0.f 0x2bd60e0)/0x2bd60e0 2014-06-09 16:22:12.250873 7f868f761700 2 waiting 51 > 50 ops || 4237938 > 104857600 2014-06-09 16:22:12.250876 7f868a757700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250880 7f868df5e700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250882 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.250884 7f868ef60700 2 waiting 51 > 50 ops || 4237940 > 104857600 2014-06-09 16:22:12.250887 7f8692767700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250891 7f868d75d700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.250894 7f868e75f700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250898 7f8692f68700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250896 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4282.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.250900 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.250901 7f8690763700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250905 7f868cf5c700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250910 7f8689f56700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.250917 7f8689755700 2 waiting 51 > 50 ops || 4237940 > 104857600 2014-06-09 16:22:12.250922 7f868bf5a700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250925 7f868b759700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250930 7f868ff62700 2 waiting 51 > 50 ops || 4237942 > 104857600 2014-06-09 16:22:12.250925 7f86a5e11700 10 _do_op 0x324f4f60 seq 4282 r = 0, finisher 0xbd70560 0 2014-06-09 16:22:12.250935 7f8691f66700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.250938 7f86a5e11700 10 _finish_op 0x324f4f60 seq 4282 osr(0.530 0x2bd78f0)/0x2bd78f0 2014-06-09 16:22:12.250939 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31435ba0] 2014-06-09 16:22:12.250942 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31435ba0] 2014-06-09 16:22:12.250944 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.250944 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.250949 7f868af58700 10 op_submit_start 4331 2014-06-09 16:22:12.250950 7f868af58700 5 queue_transactions (trailing journal) 4331 2014-06-09 16:22:12.250950 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd70560] 2014-06-09 16:22:12.250951 7f868af58700 5 queue_op 0xc64b380 seq 4331 osr(2.67f 0x2ed1f50) 861 bytes (queue has 50 ops and 4237147 bytes) 2014-06-09 16:22:12.250955 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd70560] 2014-06-09 16:22:12.250955 7f8691765700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.250956 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.250957 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.250959 7f8690f64700 2 waiting 51 > 50 ops || 4237975 > 104857600 2014-06-09 16:22:12.250960 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa900 (1 active) 2014-06-09 16:22:12.250963 7f868f761700 2 waiting 51 > 50 ops || 4237971 > 104857600 2014-06-09 16:22:12.250963 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc360 (2 active) 2014-06-09 16:22:12.250967 7f86a5e11700 5 _do_op 0xc2d4840 seq 4283 osr(2.523 0x2ed1610)/0x2ed1610 start 2014-06-09 16:22:12.250968 7f868af58700 10 op_submit_finish 4331 2014-06-09 16:22:12.250970 7f86a5e11700 10 _do_transaction on 0xc26e3c0 2014-06-09 16:22:12.250974 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.250985 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.250987 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.083118 7 0.000453 2014-06-09 16:22:12.250998 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.251008 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.251017 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 2014-06-09 16:22:12.251025 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.251026 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.251034 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.251040 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4283.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.251045 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.251041 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.251058 7f86a5e11700 10 _do_op 0xc2d4840 seq 4283 r = 0, finisher 0x3050aa60 0 2014-06-09 16:22:12.251051 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) 2014-06-09 16:22:12.251064 7f86a5e11700 10 _finish_op 0xc2d4840 seq 4283 osr(2.523 0x2ed1610)/0x2ed1610 2014-06-09 16:22:12.251061 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) 2014-06-09 16:22:12.251070 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.251075 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050aa60] 2014-06-09 16:22:12.251079 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050aa60] 2014-06-09 16:22:12.251081 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.251082 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.251082 7f868af58700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 2014-06-09 16:22:12.251085 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc360 (1 active) 2014-06-09 16:22:12.251087 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6537a0 (2 active) 2014-06-09 16:22:12.251089 7f86a5e11700 5 _do_op 0xc2ea540 seq 4284 osr(0.298 0x2bd6c60)/0x2bd6c60 start 2014-06-09 16:22:12.251092 7f86a5e11700 10 _do_transaction on 0xc2aad80 2014-06-09 16:22:12.251098 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 2014-06-09 16:22:12.251123 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.251128 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.251132 7f868af58700 5 queue_transactions existing osr(1.3e1 0x384d450)/0x384d450 2014-06-09 16:22:12.251134 7f868af58700 10 op_submit_start 4332 2014-06-09 16:22:12.251135 7f868af58700 5 queue_transactions (trailing journal) 4332 2014-06-09 16:22:12.251136 7f868af58700 5 queue_op 0x300618c0 seq 4332 osr(1.3e1 0x384d450) 861 bytes (queue has 50 ops and 4237147 bytes) 2014-06-09 16:22:12.251139 7f868af58700 10 op_submit_finish 4332 2014-06-09 16:22:12.251141 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.251167 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.251162 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.113537 7 0.000350 2014-06-09 16:22:12.251172 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.251181 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4284.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.251185 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.251182 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.251191 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.251197 7f86a5e11700 10 _do_op 0xc2ea540 seq 4284 r = 0, finisher 0xc5a7800 0 2014-06-09 16:22:12.251201 7f86a5e11700 10 _finish_op 0xc2ea540 seq 4284 osr(0.298 0x2bd6c60)/0x2bd6c60 2014-06-09 16:22:12.251200 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.251211 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a7800] 2014-06-09 16:22:12.251208 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.251215 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a7800] 2014-06-09 16:22:12.251216 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.251218 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.251215 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.251221 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6537a0 (1 active) 2014-06-09 16:22:12.251223 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7b00 (2 active) 2014-06-09 16:22:12.251225 7f86a5e11700 5 _do_op 0x325398c0 seq 4285 osr(3.4c4 0x3ed1420)/0x3ed1420 start 2014-06-09 16:22:12.251228 7f86a5e11700 10 _do_transaction on 0xc12f440 2014-06-09 16:22:12.251224 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) 2014-06-09 16:22:12.251233 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.251235 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) 2014-06-09 16:22:12.251243 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) 2014-06-09 16:22:12.251252 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) 2014-06-09 16:22:12.251260 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.251271 7f868af58700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 244 2014-06-09 16:22:12.251272 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.251286 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 2014-06-09 16:22:12.251316 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.251314 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4285.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.251320 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.251320 7f868af58700 5 queue_transactions existing osr(2.3bc 0x2ed1050)/0x2ed1050 2014-06-09 16:22:12.251323 7f868af58700 10 op_submit_start 4333 2014-06-09 16:22:12.251323 7f868af58700 5 queue_transactions (trailing journal) 4333 2014-06-09 16:22:12.251324 7f868af58700 5 queue_op 0x230b8c00 seq 4333 osr(2.3bc 0x2ed1050) 861 bytes (queue has 50 ops and 4237180 bytes) 2014-06-09 16:22:12.251327 7f868af58700 10 op_submit_finish 4333 2014-06-09 16:22:12.251333 7f86a5e11700 10 _do_op 0x325398c0 seq 4285 r = 0, finisher 0x3050a160 0 2014-06-09 16:22:12.251336 7f86a5e11700 10 _finish_op 0x325398c0 seq 4285 osr(3.4c4 0x3ed1420)/0x3ed1420 2014-06-09 16:22:12.251346 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a160] 2014-06-09 16:22:12.251329 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.251350 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a160] 2014-06-09 16:22:12.251352 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.251353 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.251356 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7b00 (1 active) 2014-06-09 16:22:12.251358 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4d80 (2 active) 2014-06-09 16:22:12.251351 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.062839 7 0.000326 2014-06-09 16:22:12.251361 7f86a5e11700 5 _do_op 0xc88aa20 seq 4286 osr(2.482 0x2ed14f0)/0x2ed14f0 start 2014-06-09 16:22:12.251363 7f86a5e11700 10 _do_transaction on 0xc1463c0 2014-06-09 16:22:12.251362 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.251368 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.251371 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.251379 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000017 0 0.000000 2014-06-09 16:22:12.251388 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.251396 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.251403 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.251408 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.251412 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.251421 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4286.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.251426 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.251421 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.251430 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.251436 7f86a5e11700 10 _do_op 0xc88aa20 seq 4286 r = 0, finisher 0xbd5e880 0 2014-06-09 16:22:12.251439 7f86a5e11700 10 _finish_op 0xc88aa20 seq 4286 osr(2.482 0x2ed14f0)/0x2ed14f0 2014-06-09 16:22:12.251441 7f868af58700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 244 2014-06-09 16:22:12.251449 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5e880] 2014-06-09 16:22:12.251453 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5e880] 2014-06-09 16:22:12.251455 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.251456 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.251459 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4d80 (1 active) 2014-06-09 16:22:12.251455 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.251461 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114900 (2 active) 2014-06-09 16:22:12.251463 7f86a5e11700 5 _do_op 0xc2eb680 seq 4287 osr(3.206 0x3ed0cf0)/0x3ed0cf0 start 2014-06-09 16:22:12.251466 7f86a5e11700 10 _do_transaction on 0x3145200 2014-06-09 16:22:12.251472 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.251482 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.251485 7f868af58700 5 queue_transactions existing osr(1.3fa 0x384d790)/0x384d790 2014-06-09 16:22:12.251487 7f868af58700 10 op_submit_start 4334 2014-06-09 16:22:12.251488 7f868af58700 5 queue_transactions (trailing journal) 4334 2014-06-09 16:22:12.251489 7f868af58700 5 queue_op 0x309a34a0 seq 4334 osr(1.3fa 0x384d790) 861 bytes (queue has 49 ops and 4236385 bytes) 2014-06-09 16:22:12.251491 7f868af58700 10 op_submit_finish 4334 2014-06-09 16:22:12.251493 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.251504 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.084117 7 0.003356 2014-06-09 16:22:12.251511 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.251514 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.251522 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.251527 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4287.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.251531 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.251531 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.251541 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.251551 7f86a5e11700 10 _do_op 0xc2eb680 seq 4287 r = 0, finisher 0xbe318c0 0 2014-06-09 16:22:12.251548 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.251555 7f86a5e11700 10 _finish_op 0xc2eb680 seq 4287 osr(3.206 0x3ed0cf0)/0x3ed0cf0 2014-06-09 16:22:12.251555 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.251566 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbe318c0] 2014-06-09 16:22:12.251564 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.251575 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbe318c0] 2014-06-09 16:22:12.251577 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.251578 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.251574 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.251581 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114900 (1 active) 2014-06-09 16:22:12.251584 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc5a0 (2 active) 2014-06-09 16:22:12.251586 7f86a5e11700 5 _do_op 0x159975c0 seq 4288 osr(1.5cb 0x384dd70)/0x384dd70 start 2014-06-09 16:22:12.251589 7f86a5e11700 10 _do_transaction on 0xbcb1680 2014-06-09 16:22:12.251583 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.251594 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.251591 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.251600 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.251630 7f868c75b700 10 op_submit_start 4335 2014-06-09 16:22:12.251632 7f868c75b700 5 queue_transactions (trailing journal) 4335 2014-06-09 16:22:12.251634 7f868c75b700 5 queue_op 0x2ea52b40 seq 4335 osr(3.67d 0x3ed1d70) 219486 bytes (queue has 49 ops and 4455010 bytes) 2014-06-09 16:22:12.251637 7f868c75b700 10 op_submit_finish 4335 2014-06-09 16:22:12.251647 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.251662 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4288.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.251642 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 243'685 (0'0,243'685] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 luod=214'684 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc20ea80 243'685 rep_tid=33 committed?=0 applied?=0 lock=0 op=osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad 2014-06-09 16:22:12.251666 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.251669 7f868c75b700 10 osd.26 244 dequeue_op 0x257c4900 finish 2014-06-09 16:22:12.251673 7f8691765700 10 op_submit_start 4336 2014-06-09 16:22:12.251675 7f8691765700 5 queue_transactions (trailing journal) 4336 2014-06-09 16:22:12.251676 7f8691765700 5 queue_op 0x32526600 seq 4336 osr(3.1b1 0x3ed0a70) 828 bytes (queue has 50 ops and 4455838 bytes) 2014-06-09 16:22:12.251677 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) 2014-06-09 16:22:12.251679 7f8691765700 10 op_submit_finish 4336 2014-06-09 16:22:12.251689 7f86a5e11700 10 _do_op 0x159975c0 seq 4288 r = 0, finisher 0xbcc8140 0 2014-06-09 16:22:12.251693 7f86a5e11700 10 _finish_op 0x159975c0 seq 4288 osr(1.5cb 0x384dd70)/0x384dd70 2014-06-09 16:22:12.251683 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.251702 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f170 (20 active) 2014-06-09 16:22:12.251703 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbcc8140] 2014-06-09 16:22:12.251707 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbcc8140] 2014-06-09 16:22:12.251708 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.251709 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.251700 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] check_local 2014-06-09 16:22:12.251712 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc5a0 (1 active) 2014-06-09 16:22:12.251706 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.725( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) 2014-06-09 16:22:12.251715 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad680 (2 active) 2014-06-09 16:22:12.251717 7f86a5e11700 5 _do_op 0x324e7860 seq 4289 osr(3.2df 0x3ed0e40)/0x3ed0e40 start 2014-06-09 16:22:12.251711 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 2014-06-09 16:22:12.251720 7f86a5e11700 10 _do_transaction on 0xbf29ec0 2014-06-09 16:22:12.251725 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.251720 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing 2014-06-09 16:22:12.251736 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.251731 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.251740 7f868c75b700 5 queue_transactions existing osr(0.725 0x384c3d0)/0x384c3d0 2014-06-09 16:22:12.251742 7f868c75b700 10 op_submit_start 4337 2014-06-09 16:22:12.251743 7f868c75b700 5 queue_transactions (trailing journal) 4337 2014-06-09 16:22:12.251745 7f868c75b700 5 queue_op 0x2fcaf500 seq 4337 osr(0.725 0x384c3d0) 828 bytes (queue has 50 ops and 4455838 bytes) 2014-06-09 16:22:12.251747 7f868c75b700 10 op_submit_finish 4337 2014-06-09 16:22:12.251749 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.725( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) 2014-06-09 16:22:12.251749 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1019 2014-06-09 16:22:12.251764 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.251760 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear 2014-06-09 16:22:12.251774 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.251777 7f868c75b700 5 queue_transactions existing osr(0.725 0x384c3d0)/0x384c3d0 2014-06-09 16:22:12.251770 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] agent_stop 2014-06-09 16:22:12.251779 7f868c75b700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251778 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4289.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.251781 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.251782 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.251783 7f868df5e700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251785 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.763( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) 2014-06-09 16:22:12.251795 7f86a5e11700 10 _do_op 0x324e7860 seq 4289 r = 0, finisher 0xc168c60 0 2014-06-09 16:22:12.251798 7f86a5e11700 10 _finish_op 0x324e7860 seq 4289 osr(3.2df 0x3ed0e40)/0x3ed0e40 2014-06-09 16:22:12.251814 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.251818 7f8691765700 5 queue_transactions existing osr(0.763 0x384c350)/0x384c350 2014-06-09 16:22:12.251820 7f8691765700 10 op_submit_start 4338 2014-06-09 16:22:12.251821 7f8691765700 5 queue_transactions (trailing journal) 4338 2014-06-09 16:22:12.251822 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc168c60] 2014-06-09 16:22:12.251822 7f8691765700 5 queue_op 0x24104180 seq 4338 osr(0.763 0x384c350) 828 bytes (queue has 50 ops and 4455838 bytes) 2014-06-09 16:22:12.251826 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc168c60] 2014-06-09 16:22:12.251827 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.251828 7f868ef60700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.251828 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.251833 7f868d75d700 2 waiting 51 > 50 ops || 4456699 > 104857600 2014-06-09 16:22:12.251832 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad680 (1 active) 2014-06-09 16:22:12.251835 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc5a0 (2 active) 2014-06-09 16:22:12.251836 7f868e75f700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251841 7f8692767700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251841 7f8691765700 10 op_submit_finish 4338 2014-06-09 16:22:12.251845 7f8690763700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251849 7f8692f68700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251853 7f868cf5c700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251844 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 243/243 242/242/223) 2014-06-09 16:22:12.251857 7f8689755700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.251861 7f868bf5a700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251865 7f868b759700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251869 7f868a757700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251872 7f8691f66700 2 waiting 51 > 50 ops || 4456699 > 104857600 2014-06-09 16:22:12.251876 7f8690f64700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251879 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.251880 7f868f761700 2 waiting 51 > 50 ops || 4456662 > 104857600 2014-06-09 16:22:12.251884 7f8691765700 5 queue_transactions existing osr(3.1b5 0x3ed0a80)/0x3ed0a80 2014-06-09 16:22:12.251884 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3196c640,0x1322d4e0,0x3062b2e0,0xbd5e900,0x1322d3a0,0x281b76a0,0x3196c660] 2014-06-09 16:22:12.251887 7f8691765700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251893 7f8689f56700 2 waiting 51 > 50 ops || 4456699 > 104857600 2014-06-09 16:22:12.251888 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.251899 7f868ff62700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251898 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.251903 7f86a5e11700 5 _do_op 0xbf7f320 seq 4290 osr(2.566 0x2ed18f0)/0x2ed18f0 start 2014-06-09 16:22:12.251906 7f86a5e11700 10 _do_transaction on 0xc3f09c0 2014-06-09 16:22:12.251906 7f868df5e700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251911 7f868c75b700 2 waiting 51 > 50 ops || 4456666 > 104857600 2014-06-09 16:22:12.251906 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 2014-06-09 16:22:12.251912 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.251915 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] _finish_recovery 2014-06-09 16:22:12.251924 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] purge_strays 2014-06-09 16:22:12.251933 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] publish_stats_to_osd 243:978 2014-06-09 16:22:12.251942 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3196c640,0x1322d4e0,0x3062b2e0,0xbd5e900,0x1322d3a0,0x281b76a0,0x3196c660] 2014-06-09 16:22:12.251945 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.251946 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.251611 7f868af58700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 2014-06-09 16:22:12.251952 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.251963 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 2014-06-09 16:22:12.251990 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.251994 7f868af58700 5 queue_transactions existing osr(1.36f 0x384d290)/0x384d290 2014-06-09 16:22:12.251996 7f868af58700 2 waiting 51 > 50 ops || 4456699 > 104857600 2014-06-09 16:22:12.251996 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4290.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.252001 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.252013 7f86a5e11700 10 _do_op 0xbf7f320 seq 4290 r = 0, finisher 0xcdb3ea0 0 2014-06-09 16:22:12.252019 7f86a5e11700 10 _finish_op 0xbf7f320 seq 4290 osr(2.566 0x2ed18f0)/0x2ed18f0 2014-06-09 16:22:12.252021 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd68c20] 2014-06-09 16:22:12.252023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd68c20] 2014-06-09 16:22:12.252025 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.252026 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.252030 7f868ef60700 10 op_submit_start 4339 2014-06-09 16:22:12.252032 7f868ef60700 5 queue_transactions (trailing journal) 4339 2014-06-09 16:22:12.252033 7f868ef60700 5 queue_op 0xcdab680 seq 4339 osr(2.f2 0x2ed07e0) 826 bytes (queue has 50 ops and 4455836 bytes) 2014-06-09 16:22:12.252037 7f868d75d700 2 waiting 51 > 50 ops || 4456697 > 104857600 2014-06-09 16:22:12.252041 7f868e75f700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252032 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcdb3ea0] 2014-06-09 16:22:12.252045 7f8692767700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252046 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcdb3ea0] 2014-06-09 16:22:12.252048 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.252048 7f8690763700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252050 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.252052 7f8692f68700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252055 7f868cf5c700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252059 7f8689755700 2 waiting 51 > 50 ops || 4456662 > 104857600 2014-06-09 16:22:12.252062 7f868bf5a700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252066 7f868b759700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252066 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc5a0 (1 active) 2014-06-09 16:22:12.252069 7f868a757700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252069 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a360 (2 active) 2014-06-09 16:22:12.252073 7f8691f66700 2 waiting 51 > 50 ops || 4456697 > 104857600 2014-06-09 16:22:12.252074 7f86a5e11700 5 _do_op 0xcc866c0 seq 4291 osr(0.1da 0x2bd6920)/0x2bd6920 start 2014-06-09 16:22:12.252076 7f86a5e11700 10 _do_transaction on 0xc140b40 2014-06-09 16:22:12.252077 7f868ef60700 10 op_submit_finish 4339 2014-06-09 16:22:12.252091 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.252082 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.252095 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.208596 7 0.000433 2014-06-09 16:22:12.252105 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.252114 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.252123 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.252132 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.252132 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.252140 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.252147 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4291.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.252151 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.252147 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.252157 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.252167 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.252175 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.252186 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 2014-06-09 16:22:12.252201 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:165 2014-06-09 16:22:12.252229 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.252233 7f868ef60700 5 queue_transactions existing osr(2.5e 0x2ed0590)/0x2ed0590 2014-06-09 16:22:12.252236 7f868ef60700 2 waiting 51 > 50 ops || 4456694 > 104857600 2014-06-09 16:22:12.252239 7f8690f64700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252244 7f868f761700 2 waiting 51 > 50 ops || 4456660 > 104857600 2014-06-09 16:22:12.252249 7f8691765700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252256 7f8689f56700 2 waiting 51 > 50 ops || 4456697 > 104857600 2014-06-09 16:22:12.252261 7f868ff62700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252266 7f868df5e700 2 waiting 51 > 50 ops || 4456664 > 104857600 2014-06-09 16:22:12.252281 7f86a5e11700 10 _do_op 0xcc866c0 seq 4291 r = 0, finisher 0xbd62040 0 2014-06-09 16:22:12.252289 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062b600] 2014-06-09 16:22:12.252292 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062b600] 2014-06-09 16:22:12.252293 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.252294 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.252288 7f86a5e11700 10 _finish_op 0xcc866c0 seq 4291 osr(0.1da 0x2bd6920)/0x2bd6920 2014-06-09 16:22:12.252302 7f868d75d700 10 op_submit_start 4340 2014-06-09 16:22:12.252304 7f868d75d700 5 queue_transactions (trailing journal) 4340 2014-06-09 16:22:12.252305 7f868d75d700 5 queue_op 0xc931c20 seq 4340 osr(2.446 0x2ed1550) 861 bytes (queue has 50 ops and 4455869 bytes) 2014-06-09 16:22:12.252307 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd62040] 2014-06-09 16:22:12.252310 7f868af58700 2 waiting 51 > 50 ops || 4456730 > 104857600 2014-06-09 16:22:12.252313 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd62040] 2014-06-09 16:22:12.252314 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.252314 7f868e75f700 2 waiting 51 > 50 ops || 4456697 > 104857600 2014-06-09 16:22:12.252316 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.252319 7f8692767700 2 waiting 51 > 50 ops || 4456697 > 104857600 2014-06-09 16:22:12.252319 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a360 (1 active) 2014-06-09 16:22:12.252322 7f8690763700 2 waiting 51 > 50 ops || 4456697 > 104857600 2014-06-09 16:22:12.252322 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1200 (2 active) 2014-06-09 16:22:12.252327 7f8692f68700 2 waiting 51 > 50 ops || 4456697 > 104857600 2014-06-09 16:22:12.252328 7f86a5e11700 5 _do_op 0x324f4de0 seq 4292 osr(2.26a 0x2ed0dd0)/0x2ed0dd0 start 2014-06-09 16:22:12.252331 7f86a5e11700 10 _do_transaction on 0xc0e8fc0 2014-06-09 16:22:12.252331 7f868d75d700 10 op_submit_finish 4340 2014-06-09 16:22:12.252336 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.252335 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.252349 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] exit Started/Primary/Active/Activating 1.072232 7 0.000490 2014-06-09 16:22:12.252361 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.252370 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] needs_recovery is recovered 2014-06-09 16:22:12.252377 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.252380 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 2014-06-09 16:22:12.252392 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4292.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.252396 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.252391 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.252400 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] finish_recovery 2014-06-09 16:22:12.252408 7f86a5e11700 10 _do_op 0x324f4de0 seq 4292 r = 0, finisher 0xcd33ca0 0 2014-06-09 16:22:12.252408 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] clear_recovery_state 2014-06-09 16:22:12.252414 7f86a5e11700 10 _finish_op 0x324f4de0 seq 4292 osr(2.26a 0x2ed0dd0)/0x2ed0dd0 2014-06-09 16:22:12.252425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd33ca0] 2014-06-09 16:22:12.252418 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.252430 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd33ca0] 2014-06-09 16:22:12.252432 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.252433 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.252436 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1200 (1 active) 2014-06-09 16:22:12.252429 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'635 mlcod 214'635 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.252438 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496240 (2 active) 2014-06-09 16:22:12.252441 7f86a5e11700 5 _do_op 0xc64aea0 seq 4293 osr(3.380 0x3ed1020)/0x3ed1020 start 2014-06-09 16:22:12.252444 7f86a5e11700 10 _do_transaction on 0x344db00 2014-06-09 16:22:12.252439 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 214'635 mlcod 214'635 active+clean] share_pg_info 2014-06-09 16:22:12.252450 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.252452 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 2014-06-09 16:22:12.252466 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 214'635 mlcod 214'635 active+clean] publish_stats_to_osd 243:1051 2014-06-09 16:22:12.252489 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.252496 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.252499 7f868d75d700 5 queue_transactions existing osr(3.44d 0x3ed1180)/0x3ed1180 2014-06-09 16:22:12.252502 7f868d75d700 10 op_submit_start 4341 2014-06-09 16:22:12.252502 7f868d75d700 5 queue_transactions (trailing journal) 4341 2014-06-09 16:22:12.252503 7f868d75d700 5 queue_op 0xc2d4a20 seq 4341 osr(3.44d 0x3ed1180) 861 bytes (queue has 50 ops and 4455902 bytes) 2014-06-09 16:22:12.252506 7f868d75d700 10 op_submit_finish 4341 2014-06-09 16:22:12.252505 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4293.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.252509 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.252508 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.252522 7f86a5e11700 10 _do_op 0xc64aea0 seq 4293 r = 0, finisher 0x33a74a0 0 2014-06-09 16:22:12.252525 7f86a5e11700 10 _finish_op 0xc64aea0 seq 4293 osr(3.380 0x3ed1020)/0x3ed1020 2014-06-09 16:22:12.252520 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.116585 7 0.000333 2014-06-09 16:22:12.252535 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a74a0] 2014-06-09 16:22:12.252530 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.252539 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a74a0] 2014-06-09 16:22:12.252541 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.252542 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.252545 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496240 (1 active) 2014-06-09 16:22:12.252540 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.252547 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407200 (2 active) 2014-06-09 16:22:12.252549 7f86a5e11700 5 _do_op 0x11fce300 seq 4294 osr(0.648 0x2bd7bd0)/0x2bd7bd0 start 2014-06-09 16:22:12.252552 7f86a5e11700 10 _do_transaction on 0xbe88180 2014-06-09 16:22:12.252549 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.252557 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.252558 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.252565 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.252572 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.252581 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-217 up [26,2](26) acting [26,2](26) maybe_went_rw) 2014-06-09 16:22:12.252596 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.252591 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 pi=218-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(218-222 up [26](26) acting [26](26)) 2014-06-09 16:22:12.252600 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 pi=223-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(223-235 up [26,2](26) acting [26,2](26)) 2014-06-09 16:22:12.252610 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4294.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.252614 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.252609 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [2](2) acting [2](2)) 2014-06-09 16:22:12.252618 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.252624 7f86a5e11700 10 _do_op 0x11fce300 seq 4294 r = 0, finisher 0x31435740 0 2014-06-09 16:22:12.252628 7f86a5e11700 10 _finish_op 0x11fce300 seq 4294 osr(0.648 0x2bd7bd0)/0x2bd7bd0 2014-06-09 16:22:12.252629 7f868d75d700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 244 2014-06-09 16:22:12.252638 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31435740] 2014-06-09 16:22:12.252641 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31435740] 2014-06-09 16:22:12.252643 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.252644 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.252647 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407200 (1 active) 2014-06-09 16:22:12.252643 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:182 2014-06-09 16:22:12.252650 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca48ea0 (2 active) 2014-06-09 16:22:12.252652 7f86a5e11700 5 _do_op 0x32539860 seq 4295 osr(3.113 0x3ed0800)/0x3ed0800 start 2014-06-09 16:22:12.252655 7f86a5e11700 10 _do_transaction on 0xc262180 2014-06-09 16:22:12.252660 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.252672 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.252672 7f868c75b700 10 op_submit_start 4342 2014-06-09 16:22:12.252674 7f868c75b700 5 queue_transactions (trailing journal) 4342 2014-06-09 16:22:12.252676 7f868d75d700 5 queue_transactions existing osr(1.246 0x384cff0)/0x384cff0 2014-06-09 16:22:12.252676 7f868c75b700 5 queue_op 0x3248a840 seq 4342 osr(0.725 0x384c3d0) 828 bytes (queue has 49 ops and 4455041 bytes) 2014-06-09 16:22:12.252679 7f868c75b700 10 op_submit_finish 4342 2014-06-09 16:22:12.252684 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.5f5( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) 2014-06-09 16:22:12.252699 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.252700 7f8689755700 2 waiting 51 > 50 ops || 4456728 > 104857600 2014-06-09 16:22:12.252708 7f868bf5a700 2 waiting 51 > 50 ops || 4456730 > 104857600 2014-06-09 16:22:12.252713 7f868b759700 2 waiting 51 > 50 ops || 4456730 > 104857600 2014-06-09 16:22:12.252715 7f868c75b700 10 log is not dirty 2014-06-09 16:22:12.252713 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4295.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.252716 7f868cf5c700 2 waiting 51 > 50 ops || 4456730 > 104857600 2014-06-09 16:22:12.252717 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.252720 7f868d75d700 10 op_submit_start 4343 2014-06-09 16:22:12.252720 7f868c75b700 5 queue_transactions existing osr(0.5f5 0x2bd7990)/0x2bd7990 2014-06-09 16:22:12.252721 7f868d75d700 5 queue_transactions (trailing journal) 4343 2014-06-09 16:22:12.252722 7f868c75b700 2 waiting 51 > 50 ops || 4456730 > 104857600 2014-06-09 16:22:12.252722 7f868d75d700 5 queue_op 0xbf7ede0 seq 4343 osr(1.246 0x384cff0) 861 bytes (queue has 50 ops and 4455902 bytes) 2014-06-09 16:22:12.252725 7f868d75d700 10 op_submit_finish 4343 2014-06-09 16:22:12.252730 7f86a5e11700 10 _do_op 0x32539860 seq 4295 r = 0, finisher 0x327f3400 0 2014-06-09 16:22:12.252733 7f86a5e11700 10 _finish_op 0x32539860 seq 4295 osr(3.113 0x3ed0800)/0x3ed0800 2014-06-09 16:22:12.252728 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.252745 7f868ef60700 10 op_submit_start 4344 2014-06-09 16:22:12.252747 7f868ef60700 5 queue_transactions (trailing journal) 4344 2014-06-09 16:22:12.252742 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.252748 7f868ef60700 5 queue_op 0x309a2180 seq 4344 osr(2.5e 0x2ed0590) 858 bytes (queue has 50 ops and 4455932 bytes) 2014-06-09 16:22:12.252753 7f8690f64700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252751 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x14053400,0x1b0d6e00,0x1b0d6d00,0x1b0d6c00 2014-06-09 16:22:12.252759 7f868f761700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.252763 7f8691765700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252768 7f8689f56700 2 waiting 51 > 50 ops || 4456793 > 104857600 2014-06-09 16:22:12.252772 7f868ff62700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252777 7f868df5e700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252772 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.252782 7f868e75f700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252787 7f8692767700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252782 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.252791 7f8690763700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252796 7f8692f68700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252791 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.252801 7f868bf5a700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252805 7f868b759700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252810 7f868cf5c700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252804 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1117 2014-06-09 16:22:12.252814 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.252821 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x327f3400] 2014-06-09 16:22:12.252825 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x327f3400] 2014-06-09 16:22:12.252827 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.252828 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.252822 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.252831 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca48ea0 (1 active) 2014-06-09 16:22:12.252833 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.252833 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c0900 (2 active) 2014-06-09 16:22:12.252839 7f868ef60700 10 op_submit_finish 4344 2014-06-09 16:22:12.252838 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.252842 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.252850 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.252857 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.252864 7f868c75b700 2 waiting 51 > 50 ops || 4456760 > 104857600 2014-06-09 16:22:12.252856 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.209936 7 0.006586 2014-06-09 16:22:12.252865 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.252868 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.252873 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.252880 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.252879 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.252890 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 2014-06-09 16:22:12.252892 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:157 2014-06-09 16:22:12.252900 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.252901 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.252907 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.252916 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.252910 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.252918 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.252920 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.252931 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.252928 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) 2014-06-09 16:22:12.252939 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops 2014-06-09 16:22:12.252940 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) 2014-06-09 16:22:12.252946 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.252950 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) 2014-06-09 16:22:12.252954 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill 2014-06-09 16:22:12.252961 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery 2014-06-09 16:22:12.252960 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) 2014-06-09 16:22:12.252970 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.252973 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:120 2014-06-09 16:22:12.252983 7f868ef60700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 2014-06-09 16:22:12.252981 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.252988 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.252996 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.252999 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:207 2014-06-09 16:22:12.253001 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.253016 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] check_local 2014-06-09 16:22:12.253033 7f868ef60700 10 log is not dirty 2014-06-09 16:22:12.253026 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 0x306b800 2014-06-09 16:22:12.253038 7f868ef60700 5 queue_transactions existing osr(2.4e3 0x2ed16d0)/0x2ed16d0 2014-06-09 16:22:12.253040 7f868ef60700 2 waiting 51 > 50 ops || 4456793 > 104857600 2014-06-09 16:22:12.253039 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing 2014-06-09 16:22:12.253050 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.253064 7f8689755700 2 waiting 51 > 50 ops || 4456758 > 104857600 2014-06-09 16:22:12.253071 7f86a5e11700 5 _do_op 0xc88bf80 seq 4296 osr(3.48d 0x3ed14e0)/0x3ed14e0 start 2014-06-09 16:22:12.253065 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1037 2014-06-09 16:22:12.253075 7f86a5e11700 10 _do_transaction on 0xc3f3a40 2014-06-09 16:22:12.253080 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.253075 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear 2014-06-09 16:22:12.253085 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] agent_stop 2014-06-09 16:22:12.253095 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.253099 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated 2014-06-09 16:22:12.253112 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local 2014-06-09 16:22:12.253120 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16bd6800,0x306b400 2014-06-09 16:22:12.253140 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.253135 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.1 has 2 missing 2014-06-09 16:22:12.253144 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery 2014-06-09 16:22:12.253155 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4296.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.253160 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.253159 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:936 2014-06-09 16:22:12.253168 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear 2014-06-09 16:22:12.253177 7f86a5e11700 10 _do_op 0xc88bf80 seq 4296 r = 0, finisher 0xbd5ea00 0 2014-06-09 16:22:12.253181 7f86a5e11700 10 _finish_op 0xc88bf80 seq 4296 osr(3.48d 0x3ed14e0)/0x3ed14e0 2014-06-09 16:22:12.253175 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop 2014-06-09 16:22:12.253184 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.253193 7f868f761700 10 op_submit_start 4345 2014-06-09 16:22:12.253194 7f868f761700 5 queue_transactions (trailing journal) 4345 2014-06-09 16:22:12.253188 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.253196 7f868f761700 5 queue_op 0xbdc91a0 seq 4345 osr(0.f 0x2bd60e0) 824 bytes (queue has 50 ops and 4455928 bytes) 2014-06-09 16:22:12.253201 7f8691765700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253206 7f8689f56700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253200 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.059548 7 0.000426 2014-06-09 16:22:12.253210 7f868ff62700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253215 7f868df5e700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253210 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.253219 7f868e75f700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253224 7f8692767700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253219 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.253228 7f8690763700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253233 7f8692f68700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253228 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.253237 7f868bf5a700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253242 7f868b759700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253238 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.253246 7f868cf5c700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253246 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.253257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ea00] 2014-06-09 16:22:12.253253 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.253261 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ea00] 2014-06-09 16:22:12.253263 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.253264 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.253267 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c0900 (1 active) 2014-06-09 16:22:12.253269 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b7a0 (2 active) 2014-06-09 16:22:12.253262 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) 2014-06-09 16:22:12.253275 7f868f761700 10 op_submit_finish 4345 2014-06-09 16:22:12.253273 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) 2014-06-09 16:22:12.253282 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.253278 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1c7( v 214'650 lc 214'649 (0'0,214'650] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) 2014-06-09 16:22:12.253293 7f868d75d700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 2014-06-09 16:22:12.253294 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed 2014-06-09 16:22:12.253306 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed 2014-06-09 16:22:12.253308 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:164 2014-06-09 16:22:12.253315 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info 2014-06-09 16:22:12.253329 7f868f761700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 244 2014-06-09 16:22:12.253337 7f868d75d700 10 log is not dirty 2014-06-09 16:22:12.253340 7f868d75d700 5 queue_transactions existing osr(0.1ca 0x2bd6980)/0x2bd6980 2014-06-09 16:22:12.253343 7f868d75d700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253347 7f8691f66700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253352 7f868a757700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253345 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1101 2014-06-09 16:22:12.253357 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfce200,0x18742660,0xbd6cd60,0xbd75ea0,0x324a8a20] 2014-06-09 16:22:12.253362 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.253366 7f868c75b700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253361 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.253375 7f8690f64700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253366 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 243/191 242/242/223) 2014-06-09 16:22:12.253372 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.253382 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1106 2014-06-09 16:22:12.253392 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfce200,0x18742660,0xbd6cd60,0xbd75ea0,0x324a8a20] 2014-06-09 16:22:12.253394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.253395 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.253398 7f868af58700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253401 7f868f761700 10 log is not dirty 2014-06-09 16:22:12.253405 7f868f761700 5 queue_transactions existing osr(3.366 0x3ed1080)/0x3ed1080 2014-06-09 16:22:12.253409 7f868f761700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253433 7f8689755700 2 waiting 51 > 50 ops || 4456754 > 104857600 2014-06-09 16:22:12.253440 7f868ef60700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253444 7f86a5e11700 5 _do_op 0xc2d4c60 seq 4297 osr(2.2da 0x2ed0c10)/0x2ed0c10 start 2014-06-09 16:22:12.253448 7f86a5e11700 10 _do_transaction on 0xc3d7980 2014-06-09 16:22:12.253454 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.253496 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.253512 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4297.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.253517 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.253531 7f86a5e11700 10 _do_op 0xc2d4c60 seq 4297 r = 0, finisher 0xbd6cac0 0 2014-06-09 16:22:12.253538 7f86a5e11700 10 _finish_op 0xc2d4c60 seq 4297 osr(2.2da 0x2ed0c10)/0x2ed0c10 2014-06-09 16:22:12.253539 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5a880] 2014-06-09 16:22:12.253550 7f8689f56700 10 op_submit_start 4346 2014-06-09 16:22:12.253543 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery 2014-06-09 16:22:12.253551 7f8689f56700 5 queue_transactions (trailing journal) 4346 2014-06-09 16:22:12.253553 7f8689f56700 5 queue_op 0xc2d4120 seq 4346 osr(0.5e4 0x2bd79d0) 861 bytes (queue has 50 ops and 4455928 bytes) 2014-06-09 16:22:12.253551 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays 2014-06-09 16:22:12.253558 7f868ff62700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253563 7f868df5e700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253560 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 2014-06-09 16:22:12.253567 7f868e75f700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253568 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5a880] 2014-06-09 16:22:12.253570 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.253571 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.253571 7f8692767700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253576 7f8690763700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253581 7f8692f68700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253585 7f868bf5a700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253589 7f868b759700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253594 7f868cf5c700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253599 7f868d75d700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253604 7f8691f66700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253604 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6cac0] 2014-06-09 16:22:12.253608 7f868a757700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6cac0] 2014-06-09 16:22:12.253614 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.253615 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.253618 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b7a0 (1 active) 2014-06-09 16:22:12.253620 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb680 (2 active) 2014-06-09 16:22:12.253625 7f8689f56700 10 op_submit_finish 4346 2014-06-09 16:22:12.253630 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.253643 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.048759 7 0.000470 2014-06-09 16:22:12.253656 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.253666 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.253676 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 2014-06-09 16:22:12.253687 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.253695 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.253703 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.253714 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) 2014-06-09 16:22:12.253726 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) 2014-06-09 16:22:12.253736 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.253748 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 244 2014-06-09 16:22:12.253763 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 2014-06-09 16:22:12.253797 7f8689f56700 10 log is not dirty 2014-06-09 16:22:12.253801 7f8689f56700 5 queue_transactions existing osr(1.116 0x384c8f0)/0x384c8f0 2014-06-09 16:22:12.253804 7f8689f56700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253808 7f86a5e11700 5 _do_op 0xbf7f020 seq 4298 osr(1.35c 0x384d2b0)/0x384d2b0 start 2014-06-09 16:22:12.253811 7f86a5e11700 10 _do_transaction on 0x39da3c0 2014-06-09 16:22:12.253816 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.253858 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.253872 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4298.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.253877 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.253889 7f86a5e11700 10 _do_op 0xbf7f020 seq 4298 r = 0, finisher 0x13964600 0 2014-06-09 16:22:12.253895 7f86a5e11700 10 _finish_op 0xbf7f020 seq 4298 osr(1.35c 0x384d2b0)/0x384d2b0 2014-06-09 16:22:12.253897 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074f40] 2014-06-09 16:22:12.253900 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074f40] 2014-06-09 16:22:12.253904 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.253905 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.253908 7f8690f64700 10 op_submit_start 4347 2014-06-09 16:22:12.253909 7f868af58700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253910 7f8690f64700 5 queue_transactions (trailing journal) 4347 2014-06-09 16:22:12.253911 7f8690f64700 5 queue_op 0x325270e0 seq 4347 osr(1.408 0x384d770) 828 bytes (queue has 50 ops and 4455928 bytes) 2014-06-09 16:22:12.253917 7f868f761700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253921 7f8689755700 2 waiting 51 > 50 ops || 4456754 > 104857600 2014-06-09 16:22:12.253926 7f868ef60700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253930 7f8691765700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253935 7f868df5e700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253939 7f868e75f700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253944 7f8692767700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253948 7f8690763700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253952 7f8692f68700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253956 7f868bf5a700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253960 7f868b759700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253965 7f868cf5c700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253969 7f868d75d700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253975 7f8691f66700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253975 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13964600] 2014-06-09 16:22:12.253978 7f868a757700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253979 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13964600] 2014-06-09 16:22:12.253980 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.253981 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.253984 7f8689f56700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.253990 7f868ff62700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.253993 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb680 (1 active) 2014-06-09 16:22:12.253996 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c240 (2 active) 2014-06-09 16:22:12.254001 7f8690f64700 10 op_submit_finish 4347 2014-06-09 16:22:12.254005 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 243/243 242/242/200) 2014-06-09 16:22:12.254041 7f8690f64700 10 log is not dirty 2014-06-09 16:22:12.254045 7f8690f64700 5 queue_transactions existing osr(3.5c6 0x3ed17f0)/0x3ed17f0 2014-06-09 16:22:12.254048 7f8690f64700 2 waiting 51 > 50 ops || 4456756 > 104857600 2014-06-09 16:22:12.254052 7f86a5e11700 5 _do_op 0xc2d56e0 seq 4299 osr(3.2a9 0x3ed0b40)/0x3ed0b40 start 2014-06-09 16:22:12.254055 7f86a5e11700 10 _do_transaction on 0x30d0c00 2014-06-09 16:22:12.254060 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.254162 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.254177 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4299.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.254181 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.254194 7f86a5e11700 10 _do_op 0xc2d56e0 seq 4299 r = 0, finisher 0x3205d880 0 2014-06-09 16:22:12.254200 7f86a5e11700 10 _finish_op 0xc2d56e0 seq 4299 osr(3.2a9 0x3ed0b40)/0x3ed0b40 2014-06-09 16:22:12.254201 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2780] 2014-06-09 16:22:12.254204 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2780] 2014-06-09 16:22:12.254206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.254206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.254212 7f868af58700 10 op_submit_start 4348 2014-06-09 16:22:12.254213 7f868af58700 5 queue_transactions (trailing journal) 4348 2014-06-09 16:22:12.254214 7f868f761700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254215 7f868af58700 5 queue_op 0x324e7980 seq 4348 osr(1.36f 0x384d290) 861 bytes (queue has 50 ops and 4455961 bytes) 2014-06-09 16:22:12.254219 7f8689755700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254224 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205d880] 2014-06-09 16:22:12.254228 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205d880] 2014-06-09 16:22:12.254230 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.254231 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.254234 7f868ef60700 2 waiting 51 > 50 ops || 4456822 > 104857600 2014-06-09 16:22:12.254238 7f8691765700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254242 7f868df5e700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254247 7f868e75f700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254251 7f8692767700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254256 7f8690763700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254260 7f8692f68700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254264 7f868bf5a700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254269 7f868b759700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254273 7f868cf5c700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254278 7f868d75d700 2 waiting 51 > 50 ops || 4456822 > 104857600 2014-06-09 16:22:12.254278 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c240 (1 active) 2014-06-09 16:22:12.254282 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f560 (2 active) 2014-06-09 16:22:12.254284 7f8691f66700 2 waiting 51 > 50 ops || 4456822 > 104857600 2014-06-09 16:22:12.254286 7f86a5e11700 5 _do_op 0xcca7560 seq 4300 osr(3.400 0x3ed1260)/0x3ed1260 start 2014-06-09 16:22:12.254288 7f868a757700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254289 7f86a5e11700 10 _do_transaction on 0xbf92e40 2014-06-09 16:22:12.254293 7f868af58700 10 op_submit_finish 4348 2014-06-09 16:22:12.254293 7f8689f56700 2 waiting 51 > 50 ops || 4456822 > 104857600 2014-06-09 16:22:12.254298 7f868ff62700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254303 7f8690f64700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.254297 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered 2014-06-09 16:22:12.254310 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.254309 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.044023 7 0.000367 2014-06-09 16:22:12.254320 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered 2014-06-09 16:22:12.254329 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered 2014-06-09 16:22:12.254338 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 2014-06-09 16:22:12.254347 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean 2014-06-09 16:22:12.254354 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery 2014-06-09 16:22:12.254361 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.254361 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state 2014-06-09 16:22:12.254375 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4300.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.254371 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) 2014-06-09 16:22:12.254379 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.254381 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) 2014-06-09 16:22:12.254390 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info 2014-06-09 16:22:12.254396 7f86a5e11700 10 _do_op 0xcca7560 seq 4300 r = 0, finisher 0x32078220 0 2014-06-09 16:22:12.254403 7f86a5e11700 10 _finish_op 0xcca7560 seq 4300 osr(3.400 0x3ed1260)/0x3ed1260 2014-06-09 16:22:12.254403 7f868af58700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 2014-06-09 16:22:12.254414 7f8689755700 10 op_submit_start 4349 2014-06-09 16:22:12.254416 7f8689755700 5 queue_transactions (trailing journal) 4349 2014-06-09 16:22:12.254418 7f8689755700 5 queue_op 0x33a3bc0 seq 4349 osr(1.fa 0x384c970) 826 bytes (queue has 50 ops and 4455959 bytes) 2014-06-09 16:22:12.254423 7f868ef60700 2 waiting 51 > 50 ops || 4456820 > 104857600 2014-06-09 16:22:12.254417 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 2014-06-09 16:22:12.254428 7f8691765700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254432 7f868df5e700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254437 7f868e75f700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254441 7f8692767700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254446 7f8690763700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254447 7f868af58700 10 log is not dirty 2014-06-09 16:22:12.254450 7f8692f68700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254451 7f868af58700 5 queue_transactions existing osr(0.732 0x384c3e0)/0x384c3e0 2014-06-09 16:22:12.254453 7f868af58700 2 waiting 51 > 50 ops || 4456820 > 104857600 2014-06-09 16:22:12.254459 7f868b759700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254466 7f868cf5c700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254472 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32078220] 2014-06-09 16:22:12.254477 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32078220] 2014-06-09 16:22:12.254477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32079c00] 2014-06-09 16:22:12.254479 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.254480 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.254482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32079c00] 2014-06-09 16:22:12.254484 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.254483 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f560 (1 active) 2014-06-09 16:22:12.254485 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.254486 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62ec60 (2 active) 2014-06-09 16:22:12.254488 7f868d75d700 2 waiting 51 > 50 ops || 4456820 > 104857600 2014-06-09 16:22:12.254491 7f8689755700 10 op_submit_finish 4349 2014-06-09 16:22:12.254493 7f8691f66700 2 waiting 51 > 50 ops || 4456820 > 104857600 2014-06-09 16:22:12.254496 7f868a757700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254495 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 243/243 242/242/212) 2014-06-09 16:22:12.254533 7f8689755700 10 log is not dirty 2014-06-09 16:22:12.254538 7f8689755700 5 queue_transactions existing osr(3.375 0x3ed1060)/0x3ed1060 2014-06-09 16:22:12.254540 7f8689755700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254545 7f8689f56700 2 waiting 51 > 50 ops || 4456820 > 104857600 2014-06-09 16:22:12.254550 7f868ff62700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254554 7f8690f64700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254559 7f868f761700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254563 7f868bf5a700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.254567 7f86a5e11700 5 _do_op 0x3280a000 seq 4301 osr(3.9f 0x3ed0600)/0x3ed0600 start 2014-06-09 16:22:12.254570 7f86a5e11700 10 _do_transaction on 0x39dbb00 2014-06-09 16:22:12.254576 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.254617 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 2014-06-09 16:22:12.254632 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4301.0.0 <= header.spos 150443.0.0 2014-06-09 16:22:12.255632 7f868c75b700 2 waiting 51 > 50 ops || 4456787 > 104857600 2014-06-09 16:22:12.323646 7f86a5e11700 5 submit_transaction r = 0 2014-06-09 16:22:12.323677 7f86a5e11700 10 _do_op 0x3280a000 seq 4301 r = 0, finisher 0xbd698a0 0 2014-06-09 16:22:12.323675 7f8674ab4700 10 osd.26 244 new session 0x1078b400 con=0x1abd8420 addr=172.16.96.220:0/1123804 2014-06-09 16:22:12.323689 7f86a5e11700 10 _finish_op 0x3280a000 seq 4301 osr(3.9f 0x3ed0600)/0x3ed0600 2014-06-09 16:22:12.323691 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e9b20] 2014-06-09 16:22:12.323697 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e9b20] 2014-06-09 16:22:12.323699 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty 2014-06-09 16:22:12.323700 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping 2014-06-09 16:22:12.323707 7f8691765700 10 op_submit_start 4350 2014-06-09 16:22:12.323708 7f8691765700 5 queue_transactions (trailing journal) 4350 2014-06-09 16:22:12.323711 7f8691765700 5 queue_op 0x325265a0 seq 4350 osr(3.1b5 0x3ed0a80) 828 bytes (queue has 50 ops and 4455961 bytes) 2014-06-09 16:22:12.323717 7f868df5e700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.323724 7f868e75f700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.323729 7f8692767700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.323735 7f8690763700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.323740 7f8692f68700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.323748 7f868af58700 2 waiting 51 > 50 ops || 4456822 > 104857600 2014-06-09 16:22:12.323751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd698a0] 2014-06-09 16:22:12.323759 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd698a0] 2014-06-09 16:22:12.323761 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty 2014-06-09 16:22:12.323762 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping 2014-06-09 16:22:12.323766 7f868b759700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.323771 7f868cf5c700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.323776 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62ec60 (1 active) 2014-06-09 16:22:12.323779 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338240 (2 active) 2014-06-09 16:22:12.323783 7f8691765700 10 op_submit_finish 4350 2014-06-09 16:22:12.323789 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 243/203 242/242/202) 2014-06-09 16:22:12.323835 7f8691765700 10 log is not dirty 2014-06-09 16:22:12.323845 7f8691765700 5 queue_transactions existing osr(3.11f 0x3ed07e0)/0x3ed07e0 2014-06-09 16:22:12.323848 7f8691765700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.323852 7f86a5e11700 5 _do_op 0x2e21380 seq 4302 osr(3.224 0x3ed0c80)/0x3ed0c80 start 2014-06-09 16:22:12.323855 7f86a5e11700 10 _do_transaction on 0xc2be300 2014-06-09 16:22:12.323863 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 2014-06-09 16:22:12.325506 7f86a5610700 -1 *** Caught signal (Aborted) ** in thread 7f86a5610700 ceph version 0.80-820-g5d606cd (5d606cd0d00698699c91a378a1bd9f71cc8a77c9) 1: /usr/local/bin/ceph-osd() [0xa95f40] 2: (()+0xfcb0) [0x7f86aed35cb0] 3: (gsignal()+0x35) [0x7f86ad21b425] 4: (abort()+0x17b) [0x7f86ad21eb8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f86adb16e2d] 6: (()+0x5ef26) [0x7f86adb14f26] 7: (()+0x5ef53) [0x7f86adb14f53] 8: (()+0x5f17e) [0x7f86adb1517e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0xb70e6d] 10: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x750) [0x9e6f20] 11: (KeyValueStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9e8d1e] 12: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x9a) [0x9e8e2a] 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb62a3a] 14: (ThreadPool::WorkThread::entry()+0x10) [0xb63c90] 15: (()+0x7e9a) [0x7f86aed2de9a] 16: (clone()+0x6d) [0x7f86ad2d8ccd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2014-06-09 16:22:12.327623 7f868d75d700 2 waiting 51 > 50 ops || 4456822 > 104857600 2014-06-09 16:22:12.327635 7f8691f66700 2 waiting 51 > 50 ops || 4456822 > 104857600 2014-06-09 16:22:12.327644 7f868a757700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.327651 7f8689755700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.327662 7f8689f56700 2 waiting 51 > 50 ops || 4456822 > 104857600 2014-06-09 16:22:12.327669 7f868ff62700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.327677 7f8690f64700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.327684 7f868f761700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.327690 7f868bf5a700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.327696 7f868c75b700 2 waiting 51 > 50 ops || 4456789 > 104857600 2014-06-09 16:22:12.327702 7f868ef60700 2 waiting 51 > 50 ops || 4456822 > 104857600 --- begin dump of recent events --- -1048> 2014-06-09 16:22:12.250368 7f868b759700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1047> 2014-06-09 16:22:12.250374 7f868ff62700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1046> 2014-06-09 16:22:12.250387 7f86a5e11700 10 _do_op 0xc88a9c0 seq 4278 r = 0, finisher 0x3050b320 0 -1045> 2014-06-09 16:22:12.250393 7f86a5e11700 10 _finish_op 0xc88a9c0 seq 4278 osr(3.1a0 0x3ed0ab0)/0x3ed0ab0 -1044> 2014-06-09 16:22:12.250394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbdbaa40] -1043> 2014-06-09 16:22:12.250397 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbdbaa40] -1042> 2014-06-09 16:22:12.250398 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -1041> 2014-06-09 16:22:12.250399 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -1040> 2014-06-09 16:22:12.250403 7f8691765700 10 op_submit_start 4327 -1039> 2014-06-09 16:22:12.250405 7f8691765700 5 queue_transactions (trailing journal) 4327 -1038> 2014-06-09 16:22:12.250405 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050b320] -1037> 2014-06-09 16:22:12.250406 7f8691765700 5 queue_op 0x32526660 seq 4327 osr(3.1b5 0x3ed0a80) 828 bytes (queue has 50 ops and 4237120 bytes) -1036> 2014-06-09 16:22:12.250409 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050b320] -1035> 2014-06-09 16:22:12.250410 7f868df5e700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1034> 2014-06-09 16:22:12.250411 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -1033> 2014-06-09 16:22:12.250412 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -1032> 2014-06-09 16:22:12.250414 7f8690f64700 2 waiting 51 > 50 ops || 4237946 > 104857600 -1031> 2014-06-09 16:22:12.250415 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9238c0 (1 active) -1030> 2014-06-09 16:22:12.250418 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc287200 (2 active) -1029> 2014-06-09 16:22:12.250418 7f8691f66700 2 waiting 51 > 50 ops || 4237981 > 104857600 -1028> 2014-06-09 16:22:12.250423 7f8691765700 10 op_submit_finish 4327 -1027> 2014-06-09 16:22:12.250423 7f868f761700 2 waiting 51 > 50 ops || 4237944 > 104857600 -1026> 2014-06-09 16:22:12.250428 7f868af58700 2 waiting 51 > 50 ops || 4237981 > 104857600 -1025> 2014-06-09 16:22:12.250433 7f86a5e11700 5 _do_op 0x24104300 seq 4279 osr(1.302 0x384d010)/0x384d010 start -1024> 2014-06-09 16:22:12.250436 7f86a5e11700 10 _do_transaction on 0xbdbd980 -1023> 2014-06-09 16:22:12.250426 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.1b1( v 213'653 (0'0,213'653] local-les=243 n=122 ec=118 les/c 243/203 242/242/202) -1022> 2014-06-09 16:22:12.250441 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -1021> 2014-06-09 16:22:12.250459 7f8691765700 10 log is not dirty -1020> 2014-06-09 16:22:12.250463 7f8691765700 5 queue_transactions existing osr(3.1b1 0x3ed0a70)/0x3ed0a70 -1019> 2014-06-09 16:22:12.250465 7f8691765700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1018> 2014-06-09 16:22:12.250469 7f868a757700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1017> 2014-06-09 16:22:12.250473 7f868ef60700 2 waiting 51 > 50 ops || 4237946 > 104857600 -1016> 2014-06-09 16:22:12.250477 7f8692767700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1015> 2014-06-09 16:22:12.250480 7f868d75d700 2 waiting 51 > 50 ops || 4237981 > 104857600 -1014> 2014-06-09 16:22:12.250483 7f868e75f700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1013> 2014-06-09 16:22:12.250482 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -1012> 2014-06-09 16:22:12.250487 7f8692f68700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1011> 2014-06-09 16:22:12.250490 7f8690763700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1010> 2014-06-09 16:22:12.250494 7f868cf5c700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1009> 2014-06-09 16:22:12.250499 7f8689f56700 2 waiting 51 > 50 ops || 4237981 > 104857600 -1008> 2014-06-09 16:22:12.250505 7f8689755700 2 waiting 51 > 50 ops || 4237946 > 104857600 -1007> 2014-06-09 16:22:12.250496 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4279.0.0 <= header.spos 150443.0.0 -1006> 2014-06-09 16:22:12.250508 7f86a5e11700 5 submit_transaction r = 0 -1005> 2014-06-09 16:22:12.250509 7f868bf5a700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1004> 2014-06-09 16:22:12.250512 7f868b759700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1003> 2014-06-09 16:22:12.250517 7f868ff62700 2 waiting 51 > 50 ops || 4237948 > 104857600 -1002> 2014-06-09 16:22:12.250527 7f86a5e11700 10 _do_op 0x24104300 seq 4279 r = 0, finisher 0x2e0eb60 0 -1001> 2014-06-09 16:22:12.250533 7f86a5e11700 10 _finish_op 0x24104300 seq 4279 osr(1.302 0x384d010)/0x384d010 -1000> 2014-06-09 16:22:12.250534 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xcc83bc0] -999> 2014-06-09 16:22:12.250536 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xcc83bc0] -998> 2014-06-09 16:22:12.250537 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -997> 2014-06-09 16:22:12.250538 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -996> 2014-06-09 16:22:12.250542 7f868df5e700 10 op_submit_start 4328 -995> 2014-06-09 16:22:12.250543 7f868df5e700 5 queue_transactions (trailing journal) 4328 -994> 2014-06-09 16:22:12.250544 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x2e0eb60] -993> 2014-06-09 16:22:12.250544 7f868df5e700 5 queue_op 0xcca6540 seq 4328 osr(2.101 0x2ed0730) 828 bytes (queue has 50 ops and 4237120 bytes) -992> 2014-06-09 16:22:12.250548 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x2e0eb60] -991> 2014-06-09 16:22:12.250549 7f8690f64700 2 waiting 51 > 50 ops || 4237946 > 104857600 -990> 2014-06-09 16:22:12.250550 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -989> 2014-06-09 16:22:12.250551 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -988> 2014-06-09 16:22:12.250552 7f8691f66700 2 waiting 51 > 50 ops || 4237981 > 104857600 -987> 2014-06-09 16:22:12.250557 7f868f761700 2 waiting 51 > 50 ops || 4237944 > 104857600 -986> 2014-06-09 16:22:12.250562 7f868af58700 2 waiting 51 > 50 ops || 4237981 > 104857600 -985> 2014-06-09 16:22:12.250554 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc287200 (1 active) -984> 2014-06-09 16:22:12.250566 7f8691765700 2 waiting 51 > 50 ops || 4237948 > 104857600 -983> 2014-06-09 16:22:12.250568 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77b440 (2 active) -982> 2014-06-09 16:22:12.250571 7f868a757700 2 waiting 51 > 50 ops || 4237948 > 104857600 -981> 2014-06-09 16:22:12.250572 7f86a5e11700 5 _do_op 0x30032c60 seq 4280 osr(1.465 0x384d650)/0x384d650 start -980> 2014-06-09 16:22:12.250575 7f86a5e11700 10 _do_transaction on 0xbf7ce40 -979> 2014-06-09 16:22:12.250575 7f868df5e700 10 op_submit_finish 4328 -978> 2014-06-09 16:22:12.250580 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -977> 2014-06-09 16:22:12.250579 7f868df5e700 10 osd.26 pg_epoch: 243 pg[0.61f( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/202) [4,26] r=1 lpr=242 pi=10-241/23 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 0.61f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/202) -976> 2014-06-09 16:22:12.250610 7f868df5e700 10 log is not dirty -975> 2014-06-09 16:22:12.250614 7f868df5e700 5 queue_transactions existing osr(0.61f 0x2bd7d30)/0x2bd7d30 -974> 2014-06-09 16:22:12.250617 7f868df5e700 2 waiting 51 > 50 ops || 4237948 > 104857600 -973> 2014-06-09 16:22:12.250620 7f868ef60700 2 waiting 51 > 50 ops || 4237946 > 104857600 -972> 2014-06-09 16:22:12.250619 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -971> 2014-06-09 16:22:12.250624 7f8692767700 2 waiting 51 > 50 ops || 4237948 > 104857600 -970> 2014-06-09 16:22:12.250628 7f868d75d700 2 waiting 51 > 50 ops || 4237981 > 104857600 -969> 2014-06-09 16:22:12.250631 7f868e75f700 2 waiting 51 > 50 ops || 4237948 > 104857600 -968> 2014-06-09 16:22:12.250634 7f8692f68700 2 waiting 51 > 50 ops || 4237948 > 104857600 -967> 2014-06-09 16:22:12.250633 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4280.0.0 <= header.spos 150443.0.0 -966> 2014-06-09 16:22:12.250637 7f86a5e11700 5 submit_transaction r = 0 -965> 2014-06-09 16:22:12.250638 7f8690763700 2 waiting 51 > 50 ops || 4237948 > 104857600 -964> 2014-06-09 16:22:12.250641 7f868cf5c700 2 waiting 51 > 50 ops || 4237948 > 104857600 -963> 2014-06-09 16:22:12.250647 7f8689f56700 2 waiting 51 > 50 ops || 4237981 > 104857600 -962> 2014-06-09 16:22:12.250653 7f8689755700 2 waiting 51 > 50 ops || 4237946 > 104857600 -961> 2014-06-09 16:22:12.250657 7f868bf5a700 2 waiting 51 > 50 ops || 4237948 > 104857600 -960> 2014-06-09 16:22:12.250658 7f86a5e11700 10 _do_op 0x30032c60 seq 4280 r = 0, finisher 0xbd720e0 0 -959> 2014-06-09 16:22:12.250661 7f868b759700 2 waiting 51 > 50 ops || 4237948 > 104857600 -958> 2014-06-09 16:22:12.250665 7f868ff62700 2 waiting 51 > 50 ops || 4237948 > 104857600 -957> 2014-06-09 16:22:12.250671 7f86a5e11700 10 _finish_op 0x30032c60 seq 4280 osr(1.465 0x384d650)/0x384d650 -956> 2014-06-09 16:22:12.250673 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd64580] -955> 2014-06-09 16:22:12.250675 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd64580] -954> 2014-06-09 16:22:12.250676 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -953> 2014-06-09 16:22:12.250677 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -952> 2014-06-09 16:22:12.250681 7f8690f64700 10 op_submit_start 4329 -951> 2014-06-09 16:22:12.250683 7f8690f64700 5 queue_transactions (trailing journal) 4329 -950> 2014-06-09 16:22:12.250683 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd720e0] -949> 2014-06-09 16:22:12.250684 7f8690f64700 5 queue_op 0x32527140 seq 4329 osr(1.a6 0x384c750) 826 bytes (queue has 50 ops and 4237118 bytes) -948> 2014-06-09 16:22:12.250687 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd720e0] -947> 2014-06-09 16:22:12.250688 7f8691f66700 2 waiting 51 > 50 ops || 4237979 > 104857600 -946> 2014-06-09 16:22:12.250689 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -945> 2014-06-09 16:22:12.250690 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -944> 2014-06-09 16:22:12.250692 7f868f761700 2 waiting 51 > 50 ops || 4237942 > 104857600 -943> 2014-06-09 16:22:12.250697 7f868af58700 2 waiting 51 > 50 ops || 4237979 > 104857600 -942> 2014-06-09 16:22:12.250693 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77b440 (1 active) -941> 2014-06-09 16:22:12.250702 7f8691765700 2 waiting 51 > 50 ops || 4237946 > 104857600 -940> 2014-06-09 16:22:12.250702 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac5e60 (2 active) -939> 2014-06-09 16:22:12.250706 7f86a5e11700 5 _do_op 0x1e08b7a0 seq 4281 osr(2.4af 0x2ed1790)/0x2ed1790 start -938> 2014-06-09 16:22:12.250707 7f8690f64700 10 op_submit_finish 4329 -937> 2014-06-09 16:22:12.250709 7f86a5e11700 10 _do_transaction on 0xc262900 -936> 2014-06-09 16:22:12.250714 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -935> 2014-06-09 16:22:12.250710 7f8690f64700 10 osd.26 pg_epoch: 243 pg[1.408( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/205) [5,26] r=1 lpr=242 pi=1-241/24 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 5 info: 1.408( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/205) -934> 2014-06-09 16:22:12.250739 7f8690f64700 10 log is not dirty -933> 2014-06-09 16:22:12.250743 7f8690f64700 5 queue_transactions existing osr(1.408 0x384d770)/0x384d770 -932> 2014-06-09 16:22:12.250745 7f8690f64700 2 waiting 51 > 50 ops || 4237946 > 104857600 -931> 2014-06-09 16:22:12.250749 7f868a757700 2 waiting 51 > 50 ops || 4237946 > 104857600 -930> 2014-06-09 16:22:12.250753 7f868df5e700 2 waiting 51 > 50 ops || 4237946 > 104857600 -929> 2014-06-09 16:22:12.250753 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -928> 2014-06-09 16:22:12.250757 7f868ef60700 2 waiting 51 > 50 ops || 4237944 > 104857600 -927> 2014-06-09 16:22:12.250760 7f8692767700 2 waiting 51 > 50 ops || 4237946 > 104857600 -926> 2014-06-09 16:22:12.250764 7f868d75d700 2 waiting 51 > 50 ops || 4237979 > 104857600 -925> 2014-06-09 16:22:12.250768 7f868e75f700 2 waiting 51 > 50 ops || 4237946 > 104857600 -924> 2014-06-09 16:22:12.250767 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4281.0.0 <= header.spos 150443.0.0 -923> 2014-06-09 16:22:12.250771 7f8692f68700 2 waiting 51 > 50 ops || 4237946 > 104857600 -922> 2014-06-09 16:22:12.250772 7f86a5e11700 5 submit_transaction r = 0 -921> 2014-06-09 16:22:12.250775 7f8690763700 2 waiting 51 > 50 ops || 4237946 > 104857600 -920> 2014-06-09 16:22:12.250778 7f868cf5c700 2 waiting 51 > 50 ops || 4237946 > 104857600 -919> 2014-06-09 16:22:12.250784 7f8689f56700 2 waiting 51 > 50 ops || 4237979 > 104857600 -918> 2014-06-09 16:22:12.250789 7f8689755700 2 waiting 51 > 50 ops || 4237944 > 104857600 -917> 2014-06-09 16:22:12.250792 7f86a5e11700 10 _do_op 0x1e08b7a0 seq 4281 r = 0, finisher 0x31434800 0 -916> 2014-06-09 16:22:12.250794 7f868bf5a700 2 waiting 51 > 50 ops || 4237946 > 104857600 -915> 2014-06-09 16:22:12.250798 7f86a5e11700 10 _finish_op 0x1e08b7a0 seq 4281 osr(2.4af 0x2ed1790)/0x2ed1790 -914> 2014-06-09 16:22:12.250799 7f868b759700 2 waiting 51 > 50 ops || 4237946 > 104857600 -913> 2014-06-09 16:22:12.250803 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x324811c0] -912> 2014-06-09 16:22:12.250803 7f868ff62700 2 waiting 51 > 50 ops || 4237946 > 104857600 -911> 2014-06-09 16:22:12.250805 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x324811c0] -910> 2014-06-09 16:22:12.250806 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -909> 2014-06-09 16:22:12.250807 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -908> 2014-06-09 16:22:12.250815 7f868f761700 10 op_submit_start 4330 -907> 2014-06-09 16:22:12.250816 7f868f761700 5 queue_transactions (trailing journal) 4330 -906> 2014-06-09 16:22:12.250817 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31434800] -905> 2014-06-09 16:22:12.250817 7f868f761700 5 queue_op 0xbdc99e0 seq 4330 osr(0.f 0x2bd60e0) 824 bytes (queue has 50 ops and 4237114 bytes) -904> 2014-06-09 16:22:12.250822 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31434800] -903> 2014-06-09 16:22:12.250822 7f868af58700 2 waiting 51 > 50 ops || 4237975 > 104857600 -902> 2014-06-09 16:22:12.250823 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -901> 2014-06-09 16:22:12.250824 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -900> 2014-06-09 16:22:12.250826 7f8691765700 2 waiting 51 > 50 ops || 4237942 > 104857600 -899> 2014-06-09 16:22:12.250828 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac5e60 (1 active) -898> 2014-06-09 16:22:12.250830 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fa900 (2 active) -897> 2014-06-09 16:22:12.250830 7f8690f64700 2 waiting 51 > 50 ops || 4237942 > 104857600 -896> 2014-06-09 16:22:12.250836 7f868f761700 10 op_submit_finish 4330 -895> 2014-06-09 16:22:12.250835 7f86a5e11700 5 _do_op 0x324f4f60 seq 4282 osr(0.530 0x2bd78f0)/0x2bd78f0 start -894> 2014-06-09 16:22:12.250837 7f86a5e11700 10 _do_transaction on 0xc227800 -893> 2014-06-09 16:22:12.250843 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -892> 2014-06-09 16:22:12.250839 7f868f761700 10 osd.26 pg_epoch: 243 pg[0.f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/223) -891> 2014-06-09 16:22:12.250867 7f868f761700 10 log is not dirty -890> 2014-06-09 16:22:12.250871 7f868f761700 5 queue_transactions existing osr(0.f 0x2bd60e0)/0x2bd60e0 -889> 2014-06-09 16:22:12.250873 7f868f761700 2 waiting 51 > 50 ops || 4237938 > 104857600 -888> 2014-06-09 16:22:12.250876 7f868a757700 2 waiting 51 > 50 ops || 4237942 > 104857600 -887> 2014-06-09 16:22:12.250880 7f868df5e700 2 waiting 51 > 50 ops || 4237942 > 104857600 -886> 2014-06-09 16:22:12.250882 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -885> 2014-06-09 16:22:12.250884 7f868ef60700 2 waiting 51 > 50 ops || 4237940 > 104857600 -884> 2014-06-09 16:22:12.250887 7f8692767700 2 waiting 51 > 50 ops || 4237942 > 104857600 -883> 2014-06-09 16:22:12.250891 7f868d75d700 2 waiting 51 > 50 ops || 4237975 > 104857600 -882> 2014-06-09 16:22:12.250894 7f868e75f700 2 waiting 51 > 50 ops || 4237942 > 104857600 -881> 2014-06-09 16:22:12.250898 7f8692f68700 2 waiting 51 > 50 ops || 4237942 > 104857600 -880> 2014-06-09 16:22:12.250896 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4282.0.0 <= header.spos 150443.0.0 -879> 2014-06-09 16:22:12.250900 7f86a5e11700 5 submit_transaction r = 0 -878> 2014-06-09 16:22:12.250901 7f8690763700 2 waiting 51 > 50 ops || 4237942 > 104857600 -877> 2014-06-09 16:22:12.250905 7f868cf5c700 2 waiting 51 > 50 ops || 4237942 > 104857600 -876> 2014-06-09 16:22:12.250910 7f8689f56700 2 waiting 51 > 50 ops || 4237975 > 104857600 -875> 2014-06-09 16:22:12.250917 7f8689755700 2 waiting 51 > 50 ops || 4237940 > 104857600 -874> 2014-06-09 16:22:12.250922 7f868bf5a700 2 waiting 51 > 50 ops || 4237942 > 104857600 -873> 2014-06-09 16:22:12.250925 7f868b759700 2 waiting 51 > 50 ops || 4237942 > 104857600 -872> 2014-06-09 16:22:12.250930 7f868ff62700 2 waiting 51 > 50 ops || 4237942 > 104857600 -871> 2014-06-09 16:22:12.250925 7f86a5e11700 10 _do_op 0x324f4f60 seq 4282 r = 0, finisher 0xbd70560 0 -870> 2014-06-09 16:22:12.250935 7f8691f66700 2 waiting 51 > 50 ops || 4237975 > 104857600 -869> 2014-06-09 16:22:12.250938 7f86a5e11700 10 _finish_op 0x324f4f60 seq 4282 osr(0.530 0x2bd78f0)/0x2bd78f0 -868> 2014-06-09 16:22:12.250939 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x31435ba0] -867> 2014-06-09 16:22:12.250942 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x31435ba0] -866> 2014-06-09 16:22:12.250944 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -865> 2014-06-09 16:22:12.250944 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -864> 2014-06-09 16:22:12.250949 7f868af58700 10 op_submit_start 4331 -863> 2014-06-09 16:22:12.250950 7f868af58700 5 queue_transactions (trailing journal) 4331 -862> 2014-06-09 16:22:12.250950 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd70560] -861> 2014-06-09 16:22:12.250951 7f868af58700 5 queue_op 0xc64b380 seq 4331 osr(2.67f 0x2ed1f50) 861 bytes (queue has 50 ops and 4237147 bytes) -860> 2014-06-09 16:22:12.250955 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd70560] -859> 2014-06-09 16:22:12.250955 7f8691765700 2 waiting 51 > 50 ops || 4237975 > 104857600 -858> 2014-06-09 16:22:12.250956 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -857> 2014-06-09 16:22:12.250957 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -856> 2014-06-09 16:22:12.250959 7f8690f64700 2 waiting 51 > 50 ops || 4237975 > 104857600 -855> 2014-06-09 16:22:12.250960 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fa900 (1 active) -854> 2014-06-09 16:22:12.250963 7f868f761700 2 waiting 51 > 50 ops || 4237971 > 104857600 -853> 2014-06-09 16:22:12.250963 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc0fc360 (2 active) -852> 2014-06-09 16:22:12.250967 7f86a5e11700 5 _do_op 0xc2d4840 seq 4283 osr(2.523 0x2ed1610)/0x2ed1610 start -851> 2014-06-09 16:22:12.250968 7f868af58700 10 op_submit_finish 4331 -850> 2014-06-09 16:22:12.250970 7f86a5e11700 10 _do_transaction on 0xc26e3c0 -849> 2014-06-09 16:22:12.250974 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -848> 2014-06-09 16:22:12.250985 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -847> 2014-06-09 16:22:12.250987 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.083118 7 0.000453 -846> 2014-06-09 16:22:12.250998 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -845> 2014-06-09 16:22:12.251008 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -844> 2014-06-09 16:22:12.251017 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000019 0 0.000000 -843> 2014-06-09 16:22:12.251025 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -842> 2014-06-09 16:22:12.251026 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -841> 2014-06-09 16:22:12.251034 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -840> 2014-06-09 16:22:12.251040 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4283.0.0 <= header.spos 150443.0.0 -839> 2014-06-09 16:22:12.251045 7f86a5e11700 5 submit_transaction r = 0 -838> 2014-06-09 16:22:12.251041 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/212 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -837> 2014-06-09 16:22:12.251058 7f86a5e11700 10 _do_op 0xc2d4840 seq 4283 r = 0, finisher 0x3050aa60 0 -836> 2014-06-09 16:22:12.251051 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=211-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(211-235 up [26,8](26) acting [26,8](26) maybe_went_rw) -835> 2014-06-09 16:22:12.251064 7f86a5e11700 10 _finish_op 0xc2d4840 seq 4283 osr(2.523 0x2ed1610)/0x2ed1610 -834> 2014-06-09 16:22:12.251061 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [8](8) acting [8](8)) -833> 2014-06-09 16:22:12.251070 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -832> 2014-06-09 16:22:12.251075 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050aa60] -831> 2014-06-09 16:22:12.251079 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050aa60] -830> 2014-06-09 16:22:12.251081 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -829> 2014-06-09 16:22:12.251082 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -828> 2014-06-09 16:22:12.251082 7f868af58700 20 osd.26 243 share_map_peer 0xbeb9ce0 already has epoch 244 -827> 2014-06-09 16:22:12.251085 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc0fc360 (1 active) -826> 2014-06-09 16:22:12.251087 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6537a0 (2 active) -825> 2014-06-09 16:22:12.251089 7f86a5e11700 5 _do_op 0xc2ea540 seq 4284 osr(0.298 0x2bd6c60)/0x2bd6c60 start -824> 2014-06-09 16:22:12.251085 7f868af58700 1 -- 172.16.96.202:6850/30904 --> osd.8 172.16.96.201:6859/53035 -- pg_info(1 pgs e243:1.3e1) v4 -- ?+0 0x1c038e00 -823> 2014-06-09 16:22:12.251092 7f86a5e11700 10 _do_transaction on 0xc2aad80 -822> 2014-06-09 16:22:12.251098 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.3e1( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,8] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:152 -821> 2014-06-09 16:22:12.251123 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -820> 2014-06-09 16:22:12.251128 7f868af58700 10 log is not dirty -819> 2014-06-09 16:22:12.251132 7f868af58700 5 queue_transactions existing osr(1.3e1 0x384d450)/0x384d450 -818> 2014-06-09 16:22:12.251134 7f868af58700 10 op_submit_start 4332 -817> 2014-06-09 16:22:12.251135 7f868af58700 5 queue_transactions (trailing journal) 4332 -816> 2014-06-09 16:22:12.251136 7f868af58700 5 queue_op 0x300618c0 seq 4332 osr(1.3e1 0x384d450) 861 bytes (queue has 50 ops and 4237147 bytes) -815> 2014-06-09 16:22:12.251139 7f868af58700 10 op_submit_finish 4332 -814> 2014-06-09 16:22:12.251141 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -813> 2014-06-09 16:22:12.251167 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -812> 2014-06-09 16:22:12.251162 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.113537 7 0.000350 -811> 2014-06-09 16:22:12.251172 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -810> 2014-06-09 16:22:12.251181 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4284.0.0 <= header.spos 150443.0.0 -809> 2014-06-09 16:22:12.251185 7f86a5e11700 5 submit_transaction r = 0 -808> 2014-06-09 16:22:12.251182 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -807> 2014-06-09 16:22:12.251191 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -806> 2014-06-09 16:22:12.251197 7f86a5e11700 10 _do_op 0xc2ea540 seq 4284 r = 0, finisher 0xc5a7800 0 -805> 2014-06-09 16:22:12.251201 7f86a5e11700 10 _finish_op 0xc2ea540 seq 4284 osr(0.298 0x2bd6c60)/0x2bd6c60 -804> 2014-06-09 16:22:12.251200 7f868af58700 5 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -803> 2014-06-09 16:22:12.251211 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc5a7800] -802> 2014-06-09 16:22:12.251208 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -801> 2014-06-09 16:22:12.251215 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc5a7800] -800> 2014-06-09 16:22:12.251216 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -799> 2014-06-09 16:22:12.251218 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -798> 2014-06-09 16:22:12.251215 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -797> 2014-06-09 16:22:12.251221 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6537a0 (1 active) -796> 2014-06-09 16:22:12.251223 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6f7b00 (2 active) -795> 2014-06-09 16:22:12.251225 7f86a5e11700 5 _do_op 0x325398c0 seq 4285 osr(3.4c4 0x3ed1420)/0x3ed1420 start -794> 2014-06-09 16:22:12.251228 7f86a5e11700 10 _do_transaction on 0xc12f440 -793> 2014-06-09 16:22:12.251224 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-215 up [26,1](26) acting [26,1](26) maybe_went_rw) -792> 2014-06-09 16:22:12.251233 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -791> 2014-06-09 16:22:12.251235 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=216-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(216-221 up [26](26) acting [26](26)) -790> 2014-06-09 16:22:12.251243 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=222-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(222-235 up [26,1](26) acting [26,1](26)) -789> 2014-06-09 16:22:12.251252 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [1](1) acting [1](1)) -788> 2014-06-09 16:22:12.251260 7f868af58700 10 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -787> 2014-06-09 16:22:12.251271 7f868af58700 20 osd.26 243 share_map_peer 0xc13cdc0 already has epoch 244 -786> 2014-06-09 16:22:12.251272 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -785> 2014-06-09 16:22:12.251273 7f868af58700 1 -- 172.16.96.202:6850/30904 --> osd.1 172.16.96.201:6811/57862 -- pg_info(1 pgs e243:2.3bc) v4 -- ?+0 0x2d740e00 -784> 2014-06-09 16:22:12.251286 7f868af58700 15 osd.26 pg_epoch: 243 pg[2.3bc( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:176 -783> 2014-06-09 16:22:12.251316 7f868af58700 10 log is not dirty -782> 2014-06-09 16:22:12.251314 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4285.0.0 <= header.spos 150443.0.0 -781> 2014-06-09 16:22:12.251320 7f86a5e11700 5 submit_transaction r = 0 -780> 2014-06-09 16:22:12.251320 7f868af58700 5 queue_transactions existing osr(2.3bc 0x2ed1050)/0x2ed1050 -779> 2014-06-09 16:22:12.251323 7f868af58700 10 op_submit_start 4333 -778> 2014-06-09 16:22:12.251323 7f868af58700 5 queue_transactions (trailing journal) 4333 -777> 2014-06-09 16:22:12.251324 7f868af58700 5 queue_op 0x230b8c00 seq 4333 osr(2.3bc 0x2ed1050) 861 bytes (queue has 50 ops and 4237180 bytes) -776> 2014-06-09 16:22:12.251327 7f868af58700 10 op_submit_finish 4333 -775> 2014-06-09 16:22:12.251333 7f86a5e11700 10 _do_op 0x325398c0 seq 4285 r = 0, finisher 0x3050a160 0 -774> 2014-06-09 16:22:12.251336 7f86a5e11700 10 _finish_op 0x325398c0 seq 4285 osr(3.4c4 0x3ed1420)/0x3ed1420 -773> 2014-06-09 16:22:12.251346 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3050a160] -772> 2014-06-09 16:22:12.251329 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -771> 2014-06-09 16:22:12.251350 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3050a160] -770> 2014-06-09 16:22:12.251352 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -769> 2014-06-09 16:22:12.251353 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -768> 2014-06-09 16:22:12.251356 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6f7b00 (1 active) -767> 2014-06-09 16:22:12.251358 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xcac4d80 (2 active) -766> 2014-06-09 16:22:12.251351 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.062839 7 0.000326 -765> 2014-06-09 16:22:12.251361 7f86a5e11700 5 _do_op 0xc88aa20 seq 4286 osr(2.482 0x2ed14f0)/0x2ed14f0 start -764> 2014-06-09 16:22:12.251363 7f86a5e11700 10 _do_transaction on 0xc1463c0 -763> 2014-06-09 16:22:12.251362 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -762> 2014-06-09 16:22:12.251368 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -761> 2014-06-09 16:22:12.251371 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -760> 2014-06-09 16:22:12.251379 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000017 0 0.000000 -759> 2014-06-09 16:22:12.251388 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -758> 2014-06-09 16:22:12.251396 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -757> 2014-06-09 16:22:12.251403 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -756> 2014-06-09 16:22:12.251408 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -755> 2014-06-09 16:22:12.251412 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) -754> 2014-06-09 16:22:12.251421 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4286.0.0 <= header.spos 150443.0.0 -753> 2014-06-09 16:22:12.251426 7f86a5e11700 5 submit_transaction r = 0 -752> 2014-06-09 16:22:12.251421 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) -751> 2014-06-09 16:22:12.251430 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -750> 2014-06-09 16:22:12.251436 7f86a5e11700 10 _do_op 0xc88aa20 seq 4286 r = 0, finisher 0xbd5e880 0 -749> 2014-06-09 16:22:12.251439 7f86a5e11700 10 _finish_op 0xc88aa20 seq 4286 osr(2.482 0x2ed14f0)/0x2ed14f0 -748> 2014-06-09 16:22:12.251441 7f868af58700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 244 -747> 2014-06-09 16:22:12.251443 7f868af58700 1 -- 172.16.96.202:6850/30904 --> osd.7 172.16.96.201:6851/52636 -- pg_info(1 pgs e243:1.3fa) v4 -- ?+0 0x1ac94e00 -746> 2014-06-09 16:22:12.251449 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5e880] -745> 2014-06-09 16:22:12.251453 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5e880] -744> 2014-06-09 16:22:12.251455 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -743> 2014-06-09 16:22:12.251456 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -742> 2014-06-09 16:22:12.251459 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xcac4d80 (1 active) -741> 2014-06-09 16:22:12.251455 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.3fa( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 -740> 2014-06-09 16:22:12.251461 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc114900 (2 active) -739> 2014-06-09 16:22:12.251463 7f86a5e11700 5 _do_op 0xc2eb680 seq 4287 osr(3.206 0x3ed0cf0)/0x3ed0cf0 start -738> 2014-06-09 16:22:12.251466 7f86a5e11700 10 _do_transaction on 0x3145200 -737> 2014-06-09 16:22:12.251472 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -736> 2014-06-09 16:22:12.251482 7f868af58700 10 log is not dirty -735> 2014-06-09 16:22:12.251485 7f868af58700 5 queue_transactions existing osr(1.3fa 0x384d790)/0x384d790 -734> 2014-06-09 16:22:12.251487 7f868af58700 10 op_submit_start 4334 -733> 2014-06-09 16:22:12.251488 7f868af58700 5 queue_transactions (trailing journal) 4334 -732> 2014-06-09 16:22:12.251489 7f868af58700 5 queue_op 0x309a34a0 seq 4334 osr(1.3fa 0x384d790) 861 bytes (queue has 49 ops and 4236385 bytes) -731> 2014-06-09 16:22:12.251491 7f868af58700 10 op_submit_finish 4334 -730> 2014-06-09 16:22:12.251493 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -729> 2014-06-09 16:22:12.251504 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.084117 7 0.003356 -728> 2014-06-09 16:22:12.251511 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -727> 2014-06-09 16:22:12.251514 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -726> 2014-06-09 16:22:12.251522 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -725> 2014-06-09 16:22:12.251527 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4287.0.0 <= header.spos 150443.0.0 -724> 2014-06-09 16:22:12.251531 7f86a5e11700 5 submit_transaction r = 0 -723> 2014-06-09 16:22:12.251531 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -722> 2014-06-09 16:22:12.251541 7f868af58700 5 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -721> 2014-06-09 16:22:12.251551 7f86a5e11700 10 _do_op 0xc2eb680 seq 4287 r = 0, finisher 0xbe318c0 0 -720> 2014-06-09 16:22:12.251548 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -719> 2014-06-09 16:22:12.251555 7f86a5e11700 10 _finish_op 0xc2eb680 seq 4287 osr(3.206 0x3ed0cf0)/0x3ed0cf0 -718> 2014-06-09 16:22:12.251555 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -717> 2014-06-09 16:22:12.251566 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbe318c0] -716> 2014-06-09 16:22:12.251564 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -715> 2014-06-09 16:22:12.251575 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbe318c0] -714> 2014-06-09 16:22:12.251577 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -713> 2014-06-09 16:22:12.251578 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -712> 2014-06-09 16:22:12.251574 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -711> 2014-06-09 16:22:12.251581 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc114900 (1 active) -710> 2014-06-09 16:22:12.251584 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6fc5a0 (2 active) -709> 2014-06-09 16:22:12.251586 7f86a5e11700 5 _do_op 0x159975c0 seq 4288 osr(1.5cb 0x384dd70)/0x384dd70 start -708> 2014-06-09 16:22:12.251589 7f86a5e11700 10 _do_transaction on 0xbcb1680 -707> 2014-06-09 16:22:12.251583 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -706> 2014-06-09 16:22:12.251594 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -705> 2014-06-09 16:22:12.251591 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -704> 2014-06-09 16:22:12.251600 7f868af58700 10 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -703> 2014-06-09 16:22:12.251630 7f868c75b700 10 op_submit_start 4335 -702> 2014-06-09 16:22:12.251632 7f868c75b700 5 queue_transactions (trailing journal) 4335 -701> 2014-06-09 16:22:12.251634 7f868c75b700 5 queue_op 0x2ea52b40 seq 4335 osr(3.67d 0x3ed1d70) 219486 bytes (queue has 49 ops and 4455010 bytes) -700> 2014-06-09 16:22:12.251637 7f868c75b700 10 op_submit_finish 4335 -699> 2014-06-09 16:22:12.251647 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -698> 2014-06-09 16:22:12.251662 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4288.0.0 <= header.spos 150443.0.0 -697> 2014-06-09 16:22:12.251642 7f868c75b700 10 osd.26 pg_epoch: 243 pg[3.67d( v 243'685 (0'0,243'685] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 luod=214'684 crt=0'0 lcod 0'0 mlcod 0'0 active] eval_repop repgather(0xc20ea80 243'685 rep_tid=33 committed?=0 applied?=0 lock=0 op=osd_op(client.4576.0:36701 rbd_data.11806b8b4567.000000000000189d [set-alloc-hint object_size 4194304 write_size 4194304,write 0~217088] 3.119d067d RETRY=3 ack+ondisk+retry+write e242) v4) wants=ad -696> 2014-06-09 16:22:12.251666 7f86a5e11700 5 submit_transaction r = 0 -695> 2014-06-09 16:22:12.251669 7f868c75b700 10 osd.26 244 dequeue_op 0x257c4900 finish -694> 2014-06-09 16:22:12.251673 7f8691765700 10 op_submit_start 4336 -693> 2014-06-09 16:22:12.251675 7f8691765700 5 queue_transactions (trailing journal) 4336 -692> 2014-06-09 16:22:12.251676 7f8691765700 5 queue_op 0x32526600 seq 4336 osr(3.1b1 0x3ed0a70) 828 bytes (queue has 50 ops and 4455838 bytes) -691> 2014-06-09 16:22:12.251677 7f868c75b700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (19 active) -690> 2014-06-09 16:22:12.251679 7f8691765700 10 op_submit_finish 4336 -689> 2014-06-09 16:22:12.251689 7f86a5e11700 10 _do_op 0x159975c0 seq 4288 r = 0, finisher 0xbcc8140 0 -688> 2014-06-09 16:22:12.251693 7f86a5e11700 10 _finish_op 0x159975c0 seq 4288 osr(1.5cb 0x384dd70)/0x384dd70 -687> 2014-06-09 16:22:12.251683 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -686> 2014-06-09 16:22:12.251702 7f868c75b700 12 OSD::op_tp worker wq OSD::PeeringWQ start processing 0xcd8f170 (20 active) -685> 2014-06-09 16:22:12.251703 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbcc8140] -684> 2014-06-09 16:22:12.251707 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbcc8140] -683> 2014-06-09 16:22:12.251708 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -682> 2014-06-09 16:22:12.251709 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -681> 2014-06-09 16:22:12.251700 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] check_local -680> 2014-06-09 16:22:12.251712 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6fc5a0 (1 active) -679> 2014-06-09 16:22:12.251706 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 210/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.725( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) -678> 2014-06-09 16:22:12.251715 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc4ad680 (2 active) -677> 2014-06-09 16:22:12.251717 7f86a5e11700 5 _do_op 0x324e7860 seq 4289 osr(3.2df 0x3ed0e40)/0x3ed0e40 start -676> 2014-06-09 16:22:12.251711 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] requeue_ops -675> 2014-06-09 16:22:12.251720 7f86a5e11700 10 _do_transaction on 0xbf29ec0 -674> 2014-06-09 16:22:12.251725 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -673> 2014-06-09 16:22:12.251720 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing -672> 2014-06-09 16:22:12.251736 7f868c75b700 10 log is not dirty -671> 2014-06-09 16:22:12.251731 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery -670> 2014-06-09 16:22:12.251740 7f868c75b700 5 queue_transactions existing osr(0.725 0x384c3d0)/0x384c3d0 -669> 2014-06-09 16:22:12.251742 7f868c75b700 10 op_submit_start 4337 -668> 2014-06-09 16:22:12.251743 7f868c75b700 5 queue_transactions (trailing journal) 4337 -667> 2014-06-09 16:22:12.251745 7f868c75b700 5 queue_op 0x2fcaf500 seq 4337 osr(0.725 0x384c3d0) 828 bytes (queue has 50 ops and 4455838 bytes) -666> 2014-06-09 16:22:12.251747 7f868c75b700 10 op_submit_finish 4337 -665> 2014-06-09 16:22:12.251749 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.725( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/209) [7,26] r=1 lpr=242 pi=1-241/21 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 7 info: 0.725( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/209) -664> 2014-06-09 16:22:12.251749 7f8691765700 15 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1019 -663> 2014-06-09 16:22:12.251764 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -662> 2014-06-09 16:22:12.251760 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear -661> 2014-06-09 16:22:12.251774 7f868c75b700 10 log is not dirty -660> 2014-06-09 16:22:12.251777 7f868c75b700 5 queue_transactions existing osr(0.725 0x384c3d0)/0x384c3d0 -659> 2014-06-09 16:22:12.251770 7f8691765700 20 osd.26 pg_epoch: 243 pg[3.6e( v 214'612 lc 214'611 (0'0,214'612] local-les=243 n=117 ec=118 les/c 243/201 242/242/242) [26,3] r=0 lpr=242 pi=200-241/2 crt=214'612 lcod 0'0 mlcod 0'0 active m=1] agent_stop -658> 2014-06-09 16:22:12.251779 7f868c75b700 2 waiting 51 > 50 ops || 4456666 > 104857600 -657> 2014-06-09 16:22:12.251778 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4289.0.0 <= header.spos 150443.0.0 -656> 2014-06-09 16:22:12.251781 7f8691765700 10 log is not dirty -655> 2014-06-09 16:22:12.251782 7f86a5e11700 5 submit_transaction r = 0 -654> 2014-06-09 16:22:12.251783 7f868df5e700 2 waiting 51 > 50 ops || 4456666 > 104857600 -653> 2014-06-09 16:22:12.251785 7f8691765700 10 osd.26 pg_epoch: 243 pg[0.763( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 0.763( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/223) -652> 2014-06-09 16:22:12.251795 7f86a5e11700 10 _do_op 0x324e7860 seq 4289 r = 0, finisher 0xc168c60 0 -651> 2014-06-09 16:22:12.251798 7f86a5e11700 10 _finish_op 0x324e7860 seq 4289 osr(3.2df 0x3ed0e40)/0x3ed0e40 -650> 2014-06-09 16:22:12.251814 7f8691765700 10 log is not dirty -649> 2014-06-09 16:22:12.251818 7f8691765700 5 queue_transactions existing osr(0.763 0x384c350)/0x384c350 -648> 2014-06-09 16:22:12.251820 7f8691765700 10 op_submit_start 4338 -647> 2014-06-09 16:22:12.251821 7f8691765700 5 queue_transactions (trailing journal) 4338 -646> 2014-06-09 16:22:12.251822 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xc168c60] -645> 2014-06-09 16:22:12.251822 7f8691765700 5 queue_op 0x24104180 seq 4338 osr(0.763 0x384c350) 828 bytes (queue has 50 ops and 4455838 bytes) -644> 2014-06-09 16:22:12.251826 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xc168c60] -643> 2014-06-09 16:22:12.251827 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -642> 2014-06-09 16:22:12.251828 7f868ef60700 2 waiting 51 > 50 ops || 4456664 > 104857600 -641> 2014-06-09 16:22:12.251828 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -640> 2014-06-09 16:22:12.251833 7f868d75d700 2 waiting 51 > 50 ops || 4456699 > 104857600 -639> 2014-06-09 16:22:12.251832 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc4ad680 (1 active) -638> 2014-06-09 16:22:12.251835 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc6dc5a0 (2 active) -637> 2014-06-09 16:22:12.251836 7f868e75f700 2 waiting 51 > 50 ops || 4456666 > 104857600 -636> 2014-06-09 16:22:12.251841 7f8692767700 2 waiting 51 > 50 ops || 4456666 > 104857600 -635> 2014-06-09 16:22:12.251841 7f8691765700 10 op_submit_finish 4338 -634> 2014-06-09 16:22:12.251845 7f8690763700 2 waiting 51 > 50 ops || 4456666 > 104857600 -633> 2014-06-09 16:22:12.251849 7f8692f68700 2 waiting 51 > 50 ops || 4456666 > 104857600 -632> 2014-06-09 16:22:12.251853 7f868cf5c700 2 waiting 51 > 50 ops || 4456666 > 104857600 -631> 2014-06-09 16:22:12.251844 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 243/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1b5( v 213'636 (0'0,213'636] local-les=243 n=124 ec=118 les/c 243/243 242/242/223) -630> 2014-06-09 16:22:12.251857 7f8689755700 2 waiting 51 > 50 ops || 4456664 > 104857600 -629> 2014-06-09 16:22:12.251861 7f868bf5a700 2 waiting 51 > 50 ops || 4456666 > 104857600 -628> 2014-06-09 16:22:12.251865 7f868b759700 2 waiting 51 > 50 ops || 4456666 > 104857600 -627> 2014-06-09 16:22:12.251869 7f868a757700 2 waiting 51 > 50 ops || 4456666 > 104857600 -626> 2014-06-09 16:22:12.251872 7f8691f66700 2 waiting 51 > 50 ops || 4456699 > 104857600 -625> 2014-06-09 16:22:12.251876 7f8690f64700 2 waiting 51 > 50 ops || 4456666 > 104857600 -624> 2014-06-09 16:22:12.251879 7f8691765700 10 log is not dirty -623> 2014-06-09 16:22:12.251880 7f868f761700 2 waiting 51 > 50 ops || 4456662 > 104857600 -622> 2014-06-09 16:22:12.251884 7f8691765700 5 queue_transactions existing osr(3.1b5 0x3ed0a80)/0x3ed0a80 -621> 2014-06-09 16:22:12.251884 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3196c640,0x1322d4e0,0x3062b2e0,0xbd5e900,0x1322d3a0,0x281b76a0,0x3196c660] -620> 2014-06-09 16:22:12.251887 7f8691765700 2 waiting 51 > 50 ops || 4456666 > 104857600 -619> 2014-06-09 16:22:12.251893 7f8689f56700 2 waiting 51 > 50 ops || 4456699 > 104857600 -618> 2014-06-09 16:22:12.251888 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -617> 2014-06-09 16:22:12.251899 7f868ff62700 2 waiting 51 > 50 ops || 4456666 > 104857600 -616> 2014-06-09 16:22:12.251898 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -615> 2014-06-09 16:22:12.251903 7f86a5e11700 5 _do_op 0xbf7f320 seq 4290 osr(2.566 0x2ed18f0)/0x2ed18f0 start -614> 2014-06-09 16:22:12.251906 7f86a5e11700 10 _do_transaction on 0xc3f09c0 -613> 2014-06-09 16:22:12.251906 7f868df5e700 2 waiting 51 > 50 ops || 4456666 > 104857600 -612> 2014-06-09 16:22:12.251911 7f868c75b700 2 waiting 51 > 50 ops || 4456666 > 104857600 -611> 2014-06-09 16:22:12.251906 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.523( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:131 -610> 2014-06-09 16:22:12.251912 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -609> 2014-06-09 16:22:12.251915 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] _finish_recovery -608> 2014-06-09 16:22:12.251924 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] purge_strays -607> 2014-06-09 16:22:12.251933 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.206( v 243'597 (0'0,243'597] local-les=243 n=114 ec=118 les/c 243/243 242/242/242) [26,3] r=0 lpr=242 crt=0'0 lcod 214'596 mlcod 214'596 active+clean] publish_stats_to_osd 243:978 -606> 2014-06-09 16:22:12.251942 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3196c640,0x1322d4e0,0x3062b2e0,0xbd5e900,0x1322d3a0,0x281b76a0,0x3196c660] -605> 2014-06-09 16:22:12.251945 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -604> 2014-06-09 16:22:12.251946 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -603> 2014-06-09 16:22:12.251611 7f868af58700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 -602> 2014-06-09 16:22:12.251952 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -601> 2014-06-09 16:22:12.251950 7f868af58700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:1.36f) v4 -- ?+0 0x1ac95340 -600> 2014-06-09 16:22:12.251963 7f868af58700 15 osd.26 pg_epoch: 243 pg[1.36f( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:191 -599> 2014-06-09 16:22:12.251990 7f868af58700 10 log is not dirty -598> 2014-06-09 16:22:12.251994 7f868af58700 5 queue_transactions existing osr(1.36f 0x384d290)/0x384d290 -597> 2014-06-09 16:22:12.251996 7f868af58700 2 waiting 51 > 50 ops || 4456699 > 104857600 -596> 2014-06-09 16:22:12.251996 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4290.0.0 <= header.spos 150443.0.0 -595> 2014-06-09 16:22:12.252001 7f86a5e11700 5 submit_transaction r = 0 -594> 2014-06-09 16:22:12.252013 7f86a5e11700 10 _do_op 0xbf7f320 seq 4290 r = 0, finisher 0xcdb3ea0 0 -593> 2014-06-09 16:22:12.252019 7f86a5e11700 10 _finish_op 0xbf7f320 seq 4290 osr(2.566 0x2ed18f0)/0x2ed18f0 -592> 2014-06-09 16:22:12.252021 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd68c20] -591> 2014-06-09 16:22:12.252023 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd68c20] -590> 2014-06-09 16:22:12.252025 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -589> 2014-06-09 16:22:12.252026 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -588> 2014-06-09 16:22:12.252030 7f868ef60700 10 op_submit_start 4339 -587> 2014-06-09 16:22:12.252032 7f868ef60700 5 queue_transactions (trailing journal) 4339 -586> 2014-06-09 16:22:12.252033 7f868ef60700 5 queue_op 0xcdab680 seq 4339 osr(2.f2 0x2ed07e0) 826 bytes (queue has 50 ops and 4455836 bytes) -585> 2014-06-09 16:22:12.252037 7f868d75d700 2 waiting 51 > 50 ops || 4456697 > 104857600 -584> 2014-06-09 16:22:12.252041 7f868e75f700 2 waiting 51 > 50 ops || 4456664 > 104857600 -583> 2014-06-09 16:22:12.252032 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcdb3ea0] -582> 2014-06-09 16:22:12.252045 7f8692767700 2 waiting 51 > 50 ops || 4456664 > 104857600 -581> 2014-06-09 16:22:12.252046 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcdb3ea0] -580> 2014-06-09 16:22:12.252048 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -579> 2014-06-09 16:22:12.252048 7f8690763700 2 waiting 51 > 50 ops || 4456664 > 104857600 -578> 2014-06-09 16:22:12.252050 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -577> 2014-06-09 16:22:12.252052 7f8692f68700 2 waiting 51 > 50 ops || 4456664 > 104857600 -576> 2014-06-09 16:22:12.252055 7f868cf5c700 2 waiting 51 > 50 ops || 4456664 > 104857600 -575> 2014-06-09 16:22:12.252059 7f8689755700 2 waiting 51 > 50 ops || 4456662 > 104857600 -574> 2014-06-09 16:22:12.252062 7f868bf5a700 2 waiting 51 > 50 ops || 4456664 > 104857600 -573> 2014-06-09 16:22:12.252066 7f868b759700 2 waiting 51 > 50 ops || 4456664 > 104857600 -572> 2014-06-09 16:22:12.252066 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc6dc5a0 (1 active) -571> 2014-06-09 16:22:12.252069 7f868a757700 2 waiting 51 > 50 ops || 4456664 > 104857600 -570> 2014-06-09 16:22:12.252069 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc77a360 (2 active) -569> 2014-06-09 16:22:12.252073 7f8691f66700 2 waiting 51 > 50 ops || 4456697 > 104857600 -568> 2014-06-09 16:22:12.252074 7f86a5e11700 5 _do_op 0xcc866c0 seq 4291 osr(0.1da 0x2bd6920)/0x2bd6920 start -567> 2014-06-09 16:22:12.252076 7f86a5e11700 10 _do_transaction on 0xc140b40 -566> 2014-06-09 16:22:12.252077 7f868ef60700 10 op_submit_finish 4339 -565> 2014-06-09 16:22:12.252091 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -564> 2014-06-09 16:22:12.252082 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -563> 2014-06-09 16:22:12.252095 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.208596 7 0.000433 -562> 2014-06-09 16:22:12.252105 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -561> 2014-06-09 16:22:12.252114 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -560> 2014-06-09 16:22:12.252123 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -559> 2014-06-09 16:22:12.252132 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -558> 2014-06-09 16:22:12.252132 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -557> 2014-06-09 16:22:12.252140 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -556> 2014-06-09 16:22:12.252147 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4291.0.0 <= header.spos 150443.0.0 -555> 2014-06-09 16:22:12.252151 7f86a5e11700 5 submit_transaction r = 0 -554> 2014-06-09 16:22:12.252147 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -553> 2014-06-09 16:22:12.252157 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -552> 2014-06-09 16:22:12.252167 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -551> 2014-06-09 16:22:12.252175 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -550> 2014-06-09 16:22:12.252186 7f868ef60700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 -549> 2014-06-09 16:22:12.252189 7f868ef60700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:2.5e) v4 -- ?+0 0x1804a8c0 -548> 2014-06-09 16:22:12.252201 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.5e( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:165 -547> 2014-06-09 16:22:12.252229 7f868ef60700 10 log is not dirty -546> 2014-06-09 16:22:12.252233 7f868ef60700 5 queue_transactions existing osr(2.5e 0x2ed0590)/0x2ed0590 -545> 2014-06-09 16:22:12.252236 7f868ef60700 2 waiting 51 > 50 ops || 4456694 > 104857600 -544> 2014-06-09 16:22:12.252239 7f8690f64700 2 waiting 51 > 50 ops || 4456664 > 104857600 -543> 2014-06-09 16:22:12.252244 7f868f761700 2 waiting 51 > 50 ops || 4456660 > 104857600 -542> 2014-06-09 16:22:12.252249 7f8691765700 2 waiting 51 > 50 ops || 4456664 > 104857600 -541> 2014-06-09 16:22:12.252256 7f8689f56700 2 waiting 51 > 50 ops || 4456697 > 104857600 -540> 2014-06-09 16:22:12.252261 7f868ff62700 2 waiting 51 > 50 ops || 4456664 > 104857600 -539> 2014-06-09 16:22:12.252266 7f868df5e700 2 waiting 51 > 50 ops || 4456664 > 104857600 -538> 2014-06-09 16:22:12.252281 7f86a5e11700 10 _do_op 0xcc866c0 seq 4291 r = 0, finisher 0xbd62040 0 -537> 2014-06-09 16:22:12.252289 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x3062b600] -536> 2014-06-09 16:22:12.252292 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x3062b600] -535> 2014-06-09 16:22:12.252293 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -534> 2014-06-09 16:22:12.252294 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -533> 2014-06-09 16:22:12.252288 7f86a5e11700 10 _finish_op 0xcc866c0 seq 4291 osr(0.1da 0x2bd6920)/0x2bd6920 -532> 2014-06-09 16:22:12.252302 7f868d75d700 10 op_submit_start 4340 -531> 2014-06-09 16:22:12.252304 7f868d75d700 5 queue_transactions (trailing journal) 4340 -530> 2014-06-09 16:22:12.252305 7f868d75d700 5 queue_op 0xc931c20 seq 4340 osr(2.446 0x2ed1550) 861 bytes (queue has 50 ops and 4455869 bytes) -529> 2014-06-09 16:22:12.252307 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd62040] -528> 2014-06-09 16:22:12.252310 7f868af58700 2 waiting 51 > 50 ops || 4456730 > 104857600 -527> 2014-06-09 16:22:12.252313 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd62040] -526> 2014-06-09 16:22:12.252314 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -525> 2014-06-09 16:22:12.252314 7f868e75f700 2 waiting 51 > 50 ops || 4456697 > 104857600 -524> 2014-06-09 16:22:12.252316 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -523> 2014-06-09 16:22:12.252319 7f8692767700 2 waiting 51 > 50 ops || 4456697 > 104857600 -522> 2014-06-09 16:22:12.252319 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc77a360 (1 active) -521> 2014-06-09 16:22:12.252322 7f8690763700 2 waiting 51 > 50 ops || 4456697 > 104857600 -520> 2014-06-09 16:22:12.252322 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3d1200 (2 active) -519> 2014-06-09 16:22:12.252327 7f8692f68700 2 waiting 51 > 50 ops || 4456697 > 104857600 -518> 2014-06-09 16:22:12.252328 7f86a5e11700 5 _do_op 0x324f4de0 seq 4292 osr(2.26a 0x2ed0dd0)/0x2ed0dd0 start -517> 2014-06-09 16:22:12.252331 7f86a5e11700 10 _do_transaction on 0xc0e8fc0 -516> 2014-06-09 16:22:12.252331 7f868d75d700 10 op_submit_finish 4340 -515> 2014-06-09 16:22:12.252336 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -514> 2014-06-09 16:22:12.252335 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -513> 2014-06-09 16:22:12.252349 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] exit Started/Primary/Active/Activating 1.072232 7 0.000490 -512> 2014-06-09 16:22:12.252361 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] enter Started/Primary/Active/Recovered -511> 2014-06-09 16:22:12.252370 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] needs_recovery is recovered -510> 2014-06-09 16:22:12.252377 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -509> 2014-06-09 16:22:12.252380 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] exit Started/Primary/Active/Recovered 0.000020 0 0.000000 -508> 2014-06-09 16:22:12.252392 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4292.0.0 <= header.spos 150443.0.0 -507> 2014-06-09 16:22:12.252396 7f86a5e11700 5 submit_transaction r = 0 -506> 2014-06-09 16:22:12.252391 7f868d75d700 5 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] enter Started/Primary/Active/Clean -505> 2014-06-09 16:22:12.252400 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] finish_recovery -504> 2014-06-09 16:22:12.252408 7f86a5e11700 10 _do_op 0x324f4de0 seq 4292 r = 0, finisher 0xcd33ca0 0 -503> 2014-06-09 16:22:12.252408 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active] clear_recovery_state -502> 2014-06-09 16:22:12.252414 7f86a5e11700 10 _finish_op 0x324f4de0 seq 4292 osr(2.26a 0x2ed0dd0)/0x2ed0dd0 -501> 2014-06-09 16:22:12.252425 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xcd33ca0] -500> 2014-06-09 16:22:12.252418 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 lcod 214'635 mlcod 214'635 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -499> 2014-06-09 16:22:12.252430 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xcd33ca0] -498> 2014-06-09 16:22:12.252432 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -497> 2014-06-09 16:22:12.252433 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -496> 2014-06-09 16:22:12.252436 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3d1200 (1 active) -495> 2014-06-09 16:22:12.252429 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 lcod 214'635 mlcod 214'635 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -494> 2014-06-09 16:22:12.252438 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x3496240 (2 active) -493> 2014-06-09 16:22:12.252441 7f86a5e11700 5 _do_op 0xc64aea0 seq 4293 osr(3.380 0x3ed1020)/0x3ed1020 start -492> 2014-06-09 16:22:12.252444 7f86a5e11700 10 _do_transaction on 0x344db00 -491> 2014-06-09 16:22:12.252439 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 214'635 mlcod 214'635 active+clean] share_pg_info -490> 2014-06-09 16:22:12.252450 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -489> 2014-06-09 16:22:12.252452 7f868d75d700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 -488> 2014-06-09 16:22:12.252454 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:3.44d) v4 -- ?+0 0xbfb0fc0 -487> 2014-06-09 16:22:12.252466 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.44d( v 243'636 (0'0,243'636] local-les=243 n=123 ec=118 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 lcod 214'635 mlcod 214'635 active+clean] publish_stats_to_osd 243:1051 -486> 2014-06-09 16:22:12.252489 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -485> 2014-06-09 16:22:12.252496 7f868d75d700 10 log is not dirty -484> 2014-06-09 16:22:12.252499 7f868d75d700 5 queue_transactions existing osr(3.44d 0x3ed1180)/0x3ed1180 -483> 2014-06-09 16:22:12.252502 7f868d75d700 10 op_submit_start 4341 -482> 2014-06-09 16:22:12.252502 7f868d75d700 5 queue_transactions (trailing journal) 4341 -481> 2014-06-09 16:22:12.252503 7f868d75d700 5 queue_op 0xc2d4a20 seq 4341 osr(3.44d 0x3ed1180) 861 bytes (queue has 50 ops and 4455902 bytes) -480> 2014-06-09 16:22:12.252506 7f868d75d700 10 op_submit_finish 4341 -479> 2014-06-09 16:22:12.252505 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4293.0.0 <= header.spos 150443.0.0 -478> 2014-06-09 16:22:12.252509 7f86a5e11700 5 submit_transaction r = 0 -477> 2014-06-09 16:22:12.252508 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -476> 2014-06-09 16:22:12.252522 7f86a5e11700 10 _do_op 0xc64aea0 seq 4293 r = 0, finisher 0x33a74a0 0 -475> 2014-06-09 16:22:12.252525 7f86a5e11700 10 _finish_op 0xc64aea0 seq 4293 osr(3.380 0x3ed1020)/0x3ed1020 -474> 2014-06-09 16:22:12.252520 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.116585 7 0.000333 -473> 2014-06-09 16:22:12.252535 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x33a74a0] -472> 2014-06-09 16:22:12.252530 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -471> 2014-06-09 16:22:12.252539 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x33a74a0] -470> 2014-06-09 16:22:12.252541 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -469> 2014-06-09 16:22:12.252542 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -468> 2014-06-09 16:22:12.252545 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0x3496240 (1 active) -467> 2014-06-09 16:22:12.252540 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -466> 2014-06-09 16:22:12.252547 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc407200 (2 active) -465> 2014-06-09 16:22:12.252549 7f86a5e11700 5 _do_op 0x11fce300 seq 4294 osr(0.648 0x2bd7bd0)/0x2bd7bd0 start -464> 2014-06-09 16:22:12.252552 7f86a5e11700 10 _do_transaction on 0xbe88180 -463> 2014-06-09 16:22:12.252549 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -462> 2014-06-09 16:22:12.252557 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -461> 2014-06-09 16:22:12.252558 7f868d75d700 5 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -460> 2014-06-09 16:22:12.252565 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -459> 2014-06-09 16:22:12.252572 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -458> 2014-06-09 16:22:12.252581 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-217 up [26,2](26) acting [26,2](26) maybe_went_rw) -457> 2014-06-09 16:22:12.252596 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -456> 2014-06-09 16:22:12.252591 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 pi=218-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(218-222 up [26](26) acting [26](26)) -455> 2014-06-09 16:22:12.252600 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 pi=223-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(223-235 up [26,2](26) acting [26,2](26)) -454> 2014-06-09 16:22:12.252610 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4294.0.0 <= header.spos 150443.0.0 -453> 2014-06-09 16:22:12.252614 7f86a5e11700 5 submit_transaction r = 0 -452> 2014-06-09 16:22:12.252609 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [2](2) acting [2](2)) -451> 2014-06-09 16:22:12.252618 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -450> 2014-06-09 16:22:12.252624 7f86a5e11700 10 _do_op 0x11fce300 seq 4294 r = 0, finisher 0x31435740 0 -449> 2014-06-09 16:22:12.252628 7f86a5e11700 10 _finish_op 0x11fce300 seq 4294 osr(0.648 0x2bd7bd0)/0x2bd7bd0 -448> 2014-06-09 16:22:12.252629 7f868d75d700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 244 -447> 2014-06-09 16:22:12.252631 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.2 172.16.96.201:6828/58271 -- pg_info(1 pgs e243:1.246) v4 -- ?+0 0x22012fc0 -446> 2014-06-09 16:22:12.252638 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x31435740] -445> 2014-06-09 16:22:12.252641 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x31435740] -444> 2014-06-09 16:22:12.252643 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -443> 2014-06-09 16:22:12.252644 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -442> 2014-06-09 16:22:12.252647 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc407200 (1 active) -441> 2014-06-09 16:22:12.252643 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.246( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,2] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:182 -440> 2014-06-09 16:22:12.252650 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xca48ea0 (2 active) -439> 2014-06-09 16:22:12.252652 7f86a5e11700 5 _do_op 0x32539860 seq 4295 osr(3.113 0x3ed0800)/0x3ed0800 start -438> 2014-06-09 16:22:12.252655 7f86a5e11700 10 _do_transaction on 0xc262180 -437> 2014-06-09 16:22:12.252660 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -436> 2014-06-09 16:22:12.252672 7f868d75d700 10 log is not dirty -435> 2014-06-09 16:22:12.252672 7f868c75b700 10 op_submit_start 4342 -434> 2014-06-09 16:22:12.252674 7f868c75b700 5 queue_transactions (trailing journal) 4342 -433> 2014-06-09 16:22:12.252676 7f868d75d700 5 queue_transactions existing osr(1.246 0x384cff0)/0x384cff0 -432> 2014-06-09 16:22:12.252676 7f868c75b700 5 queue_op 0x3248a840 seq 4342 osr(0.725 0x384c3d0) 828 bytes (queue has 49 ops and 4455041 bytes) -431> 2014-06-09 16:22:12.252679 7f868c75b700 10 op_submit_finish 4342 -430> 2014-06-09 16:22:12.252684 7f868c75b700 10 osd.26 pg_epoch: 243 pg[0.5f5( empty local-les=243 n=0 ec=1 les/c 191/191 242/242/222) [1,26] r=1 lpr=242 pi=1-241/22 crt=0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 1 info: 0.5f5( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/222) -429> 2014-06-09 16:22:12.252699 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -428> 2014-06-09 16:22:12.252700 7f8689755700 2 waiting 51 > 50 ops || 4456728 > 104857600 -427> 2014-06-09 16:22:12.252708 7f868bf5a700 2 waiting 51 > 50 ops || 4456730 > 104857600 -426> 2014-06-09 16:22:12.252713 7f868b759700 2 waiting 51 > 50 ops || 4456730 > 104857600 -425> 2014-06-09 16:22:12.252715 7f868c75b700 10 log is not dirty -424> 2014-06-09 16:22:12.252713 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4295.0.0 <= header.spos 150443.0.0 -423> 2014-06-09 16:22:12.252716 7f868cf5c700 2 waiting 51 > 50 ops || 4456730 > 104857600 -422> 2014-06-09 16:22:12.252717 7f86a5e11700 5 submit_transaction r = 0 -421> 2014-06-09 16:22:12.252720 7f868d75d700 10 op_submit_start 4343 -420> 2014-06-09 16:22:12.252720 7f868c75b700 5 queue_transactions existing osr(0.5f5 0x2bd7990)/0x2bd7990 -419> 2014-06-09 16:22:12.252721 7f868d75d700 5 queue_transactions (trailing journal) 4343 -418> 2014-06-09 16:22:12.252722 7f868c75b700 2 waiting 51 > 50 ops || 4456730 > 104857600 -417> 2014-06-09 16:22:12.252722 7f868d75d700 5 queue_op 0xbf7ede0 seq 4343 osr(1.246 0x384cff0) 861 bytes (queue has 50 ops and 4455902 bytes) -416> 2014-06-09 16:22:12.252725 7f868d75d700 10 op_submit_finish 4343 -415> 2014-06-09 16:22:12.252730 7f86a5e11700 10 _do_op 0x32539860 seq 4295 r = 0, finisher 0x327f3400 0 -414> 2014-06-09 16:22:12.252733 7f86a5e11700 10 _finish_op 0x32539860 seq 4295 osr(3.113 0x3ed0800)/0x3ed0800 -413> 2014-06-09 16:22:12.252728 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -412> 2014-06-09 16:22:12.252745 7f868ef60700 10 op_submit_start 4344 -411> 2014-06-09 16:22:12.252747 7f868ef60700 5 queue_transactions (trailing journal) 4344 -410> 2014-06-09 16:22:12.252742 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -409> 2014-06-09 16:22:12.252748 7f868ef60700 5 queue_op 0x309a2180 seq 4344 osr(2.5e 0x2ed0590) 858 bytes (queue has 50 ops and 4455932 bytes) -408> 2014-06-09 16:22:12.252753 7f8690f64700 2 waiting 51 > 50 ops || 4456760 > 104857600 -407> 2014-06-09 16:22:12.252751 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x14053400,0x1b0d6e00,0x1b0d6d00,0x1b0d6c00 -406> 2014-06-09 16:22:12.252759 7f868f761700 2 waiting 51 > 50 ops || 4456756 > 104857600 -405> 2014-06-09 16:22:12.252763 7f8691765700 2 waiting 51 > 50 ops || 4456760 > 104857600 -404> 2014-06-09 16:22:12.252768 7f8689f56700 2 waiting 51 > 50 ops || 4456793 > 104857600 -403> 2014-06-09 16:22:12.252772 7f868ff62700 2 waiting 51 > 50 ops || 4456760 > 104857600 -402> 2014-06-09 16:22:12.252777 7f868df5e700 2 waiting 51 > 50 ops || 4456760 > 104857600 -401> 2014-06-09 16:22:12.252772 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery is recovered -400> 2014-06-09 16:22:12.252782 7f868e75f700 2 waiting 51 > 50 ops || 4456760 > 104857600 -399> 2014-06-09 16:22:12.252787 7f8692767700 2 waiting 51 > 50 ops || 4456760 > 104857600 -398> 2014-06-09 16:22:12.252782 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_backfill does not need backfill -397> 2014-06-09 16:22:12.252791 7f8690763700 2 waiting 51 > 50 ops || 4456760 > 104857600 -396> 2014-06-09 16:22:12.252796 7f8692f68700 2 waiting 51 > 50 ops || 4456760 > 104857600 -395> 2014-06-09 16:22:12.252791 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate all replicas clean, no recovery -394> 2014-06-09 16:22:12.252801 7f868bf5a700 2 waiting 51 > 50 ops || 4456760 > 104857600 -393> 2014-06-09 16:22:12.252805 7f868b759700 2 waiting 51 > 50 ops || 4456760 > 104857600 -392> 2014-06-09 16:22:12.252810 7f868cf5c700 2 waiting 51 > 50 ops || 4456760 > 104857600 -391> 2014-06-09 16:22:12.252804 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:1117 -390> 2014-06-09 16:22:12.252814 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -389> 2014-06-09 16:22:12.252821 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x327f3400] -388> 2014-06-09 16:22:12.252825 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x327f3400] -387> 2014-06-09 16:22:12.252827 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -386> 2014-06-09 16:22:12.252828 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -385> 2014-06-09 16:22:12.252822 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.443( v 213'673 (0'0,213'673] local-les=243 n=125 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -384> 2014-06-09 16:22:12.252831 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xca48ea0 (1 active) -383> 2014-06-09 16:22:12.252833 7f868d75d700 10 log is not dirty -382> 2014-06-09 16:22:12.252833 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc9c0900 (2 active) -381> 2014-06-09 16:22:12.252839 7f868ef60700 10 op_submit_finish 4344 -380> 2014-06-09 16:22:12.252838 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -379> 2014-06-09 16:22:12.252842 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -378> 2014-06-09 16:22:12.252850 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -377> 2014-06-09 16:22:12.252857 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -376> 2014-06-09 16:22:12.252864 7f868c75b700 2 waiting 51 > 50 ops || 4456760 > 104857600 -375> 2014-06-09 16:22:12.252856 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.209936 7 0.006586 -374> 2014-06-09 16:22:12.252865 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -373> 2014-06-09 16:22:12.252868 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -372> 2014-06-09 16:22:12.252873 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -371> 2014-06-09 16:22:12.252880 7f868d75d700 10 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -370> 2014-06-09 16:22:12.252879 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -369> 2014-06-09 16:22:12.252890 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000022 0 0.000000 -368> 2014-06-09 16:22:12.252892 7f868d75d700 15 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:157 -367> 2014-06-09 16:22:12.252900 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -366> 2014-06-09 16:22:12.252901 7f868ef60700 5 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -365> 2014-06-09 16:22:12.252907 7f868d75d700 20 osd.26 pg_epoch: 243 pg[2.2f7( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -364> 2014-06-09 16:22:12.252916 7f868d75d700 10 log is not dirty -363> 2014-06-09 16:22:12.252910 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] finish_recovery -362> 2014-06-09 16:22:12.252918 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/191 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active] clear_recovery_state -361> 2014-06-09 16:22:12.252920 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -360> 2014-06-09 16:22:12.252931 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] check_local -359> 2014-06-09 16:22:12.252928 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=190-241/4 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(190-213 up [26,0](26) acting [26,0](26) maybe_went_rw) -358> 2014-06-09 16:22:12.252939 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] requeue_ops -357> 2014-06-09 16:22:12.252940 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=214-241/3 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(214-219 up [26](26) acting [26](26)) -356> 2014-06-09 16:22:12.252946 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -355> 2014-06-09 16:22:12.252950 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=220-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(220-235 up [26,0](26) acting [26,0](26)) -354> 2014-06-09 16:22:12.252954 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] needs_backfill does not need backfill -353> 2014-06-09 16:22:12.252961 7f868d75d700 10 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] activate all replicas clean, no recovery -352> 2014-06-09 16:22:12.252960 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [0](0) acting [0](0)) -351> 2014-06-09 16:22:12.252970 7f868ef60700 10 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -350> 2014-06-09 16:22:12.252973 7f868d75d700 15 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] publish_stats_to_osd 243:120 -349> 2014-06-09 16:22:12.252983 7f868ef60700 20 osd.26 243 share_map_peer 0xbf6e6e0 already has epoch 244 -348> 2014-06-09 16:22:12.252981 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] hit_set_clear -347> 2014-06-09 16:22:12.252986 7f868ef60700 1 -- 172.16.96.202:6850/30904 --> osd.0 172.16.96.201:6801/57551 -- pg_info(1 pgs e243:2.4e3) v4 -- ?+0 0x30c76380 -346> 2014-06-09 16:22:12.252988 7f868d75d700 20 osd.26 pg_epoch: 243 pg[1.76c( empty local-les=243 n=0 ec=1 les/c 243/203 242/242/242) [26,4] r=0 lpr=242 pi=202-241/2 crt=0'0 mlcod 0'0 active] agent_stop -345> 2014-06-09 16:22:12.252996 7f868d75d700 10 log is not dirty -344> 2014-06-09 16:22:12.252999 7f868ef60700 15 osd.26 pg_epoch: 243 pg[2.4e3( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,0] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:207 -343> 2014-06-09 16:22:12.253001 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 inactive m=1] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -342> 2014-06-09 16:22:12.253016 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] check_local -341> 2014-06-09 16:22:12.253033 7f868ef60700 10 log is not dirty -340> 2014-06-09 16:22:12.253026 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] requeue_ops 0x306b800 -339> 2014-06-09 16:22:12.253038 7f868ef60700 5 queue_transactions existing osr(2.4e3 0x2ed16d0)/0x2ed16d0 -338> 2014-06-09 16:22:12.253040 7f868ef60700 2 waiting 51 > 50 ops || 4456793 > 104857600 -337> 2014-06-09 16:22:12.253039 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] needs_recovery primary has 1 missing -336> 2014-06-09 16:22:12.253050 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] activate not all replicas are up-to-date, queueing recovery -335> 2014-06-09 16:22:12.253064 7f8689755700 2 waiting 51 > 50 ops || 4456758 > 104857600 -334> 2014-06-09 16:22:12.253071 7f86a5e11700 5 _do_op 0xc88bf80 seq 4296 osr(3.48d 0x3ed14e0)/0x3ed14e0 start -333> 2014-06-09 16:22:12.253065 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] publish_stats_to_osd 243:1037 -332> 2014-06-09 16:22:12.253075 7f86a5e11700 10 _do_transaction on 0xc3f3a40 -331> 2014-06-09 16:22:12.253080 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -330> 2014-06-09 16:22:12.253075 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] hit_set_clear -329> 2014-06-09 16:22:12.253085 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.730( v 214'648 lc 214'647 (0'0,214'648] local-les=243 n=124 ec=118 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=214'648 lcod 0'0 mlcod 0'0 active m=1] agent_stop -328> 2014-06-09 16:22:12.253095 7f868d75d700 10 log is not dirty -327> 2014-06-09 16:22:12.253099 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasActivated -326> 2014-06-09 16:22:12.253112 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] check_local -325> 2014-06-09 16:22:12.253120 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] requeue_ops 0x16bd6800,0x306b400 -324> 2014-06-09 16:22:12.253140 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -323> 2014-06-09 16:22:12.253135 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] needs_recovery osd.1 has 2 missing -322> 2014-06-09 16:22:12.253144 7f868d75d700 10 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] activate not all replicas are up-to-date, queueing recovery -321> 2014-06-09 16:22:12.253155 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4296.0.0 <= header.spos 150443.0.0 -320> 2014-06-09 16:22:12.253160 7f86a5e11700 5 submit_transaction r = 0 -319> 2014-06-09 16:22:12.253159 7f868d75d700 15 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] publish_stats_to_osd 243:936 -318> 2014-06-09 16:22:12.253168 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] hit_set_clear -317> 2014-06-09 16:22:12.253177 7f86a5e11700 10 _do_op 0xc88bf80 seq 4296 r = 0, finisher 0xbd5ea00 0 -316> 2014-06-09 16:22:12.253181 7f86a5e11700 10 _finish_op 0xc88bf80 seq 4296 osr(3.48d 0x3ed14e0)/0x3ed14e0 -315> 2014-06-09 16:22:12.253175 7f868d75d700 20 osd.26 pg_epoch: 243 pg[3.55d( v 214'593 (0'0,214'593] local-les=243 n=114 ec=118 les/c 243/191 242/242/242) [26,1] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 active] agent_stop -314> 2014-06-09 16:22:12.253184 7f868d75d700 10 log is not dirty -313> 2014-06-09 16:22:12.253193 7f868f761700 10 op_submit_start 4345 -312> 2014-06-09 16:22:12.253194 7f868f761700 5 queue_transactions (trailing journal) 4345 -311> 2014-06-09 16:22:12.253188 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -310> 2014-06-09 16:22:12.253196 7f868f761700 5 queue_op 0xbdc91a0 seq 4345 osr(0.f 0x2bd60e0) 824 bytes (queue has 50 ops and 4455928 bytes) -309> 2014-06-09 16:22:12.253201 7f8691765700 2 waiting 51 > 50 ops || 4456756 > 104857600 -308> 2014-06-09 16:22:12.253206 7f8689f56700 2 waiting 51 > 50 ops || 4456789 > 104857600 -307> 2014-06-09 16:22:12.253200 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.059548 7 0.000426 -306> 2014-06-09 16:22:12.253210 7f868ff62700 2 waiting 51 > 50 ops || 4456756 > 104857600 -305> 2014-06-09 16:22:12.253215 7f868df5e700 2 waiting 51 > 50 ops || 4456756 > 104857600 -304> 2014-06-09 16:22:12.253210 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -303> 2014-06-09 16:22:12.253219 7f868e75f700 2 waiting 51 > 50 ops || 4456756 > 104857600 -302> 2014-06-09 16:22:12.253224 7f8692767700 2 waiting 51 > 50 ops || 4456756 > 104857600 -301> 2014-06-09 16:22:12.253219 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -300> 2014-06-09 16:22:12.253228 7f8690763700 2 waiting 51 > 50 ops || 4456756 > 104857600 -299> 2014-06-09 16:22:12.253233 7f8692f68700 2 waiting 51 > 50 ops || 4456756 > 104857600 -298> 2014-06-09 16:22:12.253228 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -297> 2014-06-09 16:22:12.253237 7f868bf5a700 2 waiting 51 > 50 ops || 4456756 > 104857600 -296> 2014-06-09 16:22:12.253242 7f868b759700 2 waiting 51 > 50 ops || 4456756 > 104857600 -295> 2014-06-09 16:22:12.253238 7f868d75d700 5 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -294> 2014-06-09 16:22:12.253246 7f868cf5c700 2 waiting 51 > 50 ops || 4456756 > 104857600 -293> 2014-06-09 16:22:12.253246 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -292> 2014-06-09 16:22:12.253257 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd5ea00] -291> 2014-06-09 16:22:12.253253 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/206 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -290> 2014-06-09 16:22:12.253261 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd5ea00] -289> 2014-06-09 16:22:12.253263 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -288> 2014-06-09 16:22:12.253264 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -287> 2014-06-09 16:22:12.253267 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc9c0900 (1 active) -286> 2014-06-09 16:22:12.253269 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89b7a0 (2 active) -285> 2014-06-09 16:22:12.253262 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=205-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(205-235 up [26,5](26) acting [26,5](26) maybe_went_rw) -284> 2014-06-09 16:22:12.253275 7f868f761700 10 op_submit_finish 4345 -283> 2014-06-09 16:22:12.253273 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [5](5) acting [5](5)) -282> 2014-06-09 16:22:12.253282 7f868d75d700 10 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -281> 2014-06-09 16:22:12.253278 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.1c7( v 214'650 lc 214'649 (0'0,214'650] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) -280> 2014-06-09 16:22:12.253293 7f868d75d700 20 osd.26 243 share_map_peer 0xbf7a840 already has epoch 244 -279> 2014-06-09 16:22:12.253296 7f868d75d700 1 -- 172.16.96.202:6850/30904 --> osd.5 172.16.96.201:6837/51639 -- pg_info(1 pgs e243:0.1ca) v4 -- ?+0 0x1ffed340 -278> 2014-06-09 16:22:12.253294 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] state: peer osd.2 activated and committed -277> 2014-06-09 16:22:12.253306 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 191/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] all_activated_and_committed -276> 2014-06-09 16:22:12.253308 7f868d75d700 15 osd.26 pg_epoch: 243 pg[0.1ca( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:164 -275> 2014-06-09 16:22:12.253315 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] share_pg_info -274> 2014-06-09 16:22:12.253329 7f868f761700 20 osd.26 243 share_map_peer 0xbf7b080 already has epoch 244 -273> 2014-06-09 16:22:12.253337 7f868d75d700 10 log is not dirty -272> 2014-06-09 16:22:12.253332 7f868f761700 1 -- 172.16.96.202:6850/30904 --> osd.2 172.16.96.201:6828/58271 -- pg_info(1 pgs e243:3.1c7) v4 -- ?+0 0x25fff340 -271> 2014-06-09 16:22:12.253340 7f868d75d700 5 queue_transactions existing osr(0.1ca 0x2bd6980)/0x2bd6980 -270> 2014-06-09 16:22:12.253343 7f868d75d700 2 waiting 51 > 50 ops || 4456789 > 104857600 -269> 2014-06-09 16:22:12.253347 7f8691f66700 2 waiting 51 > 50 ops || 4456789 > 104857600 -268> 2014-06-09 16:22:12.253352 7f868a757700 2 waiting 51 > 50 ops || 4456756 > 104857600 -267> 2014-06-09 16:22:12.253345 7f868f761700 15 osd.26 pg_epoch: 243 pg[3.1c7( v 214'650 (0'0,214'650] local-les=243 n=132 ec=118 les/c 243/191 242/242/242) [26,2] r=0 lpr=242 pi=190-241/4 crt=0'0 lcod 0'0 mlcod 0'0 inactive] publish_stats_to_osd 243:1101 -266> 2014-06-09 16:22:12.253357 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbfce200,0x18742660,0xbd6cd60,0xbd75ea0,0x324a8a20] -265> 2014-06-09 16:22:12.253362 7f868f761700 10 log is not dirty -264> 2014-06-09 16:22:12.253366 7f868c75b700 2 waiting 51 > 50 ops || 4456756 > 104857600 -263> 2014-06-09 16:22:12.253361 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] _finish_recovery -262> 2014-06-09 16:22:12.253375 7f8690f64700 2 waiting 51 > 50 ops || 4456756 > 104857600 -261> 2014-06-09 16:22:12.253366 7f868f761700 10 osd.26 pg_epoch: 243 pg[3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 191/191 242/242/223) [2,26] r=1 lpr=242 pi=118-241/12 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 2 info: 3.366( v 213'611 (0'0,213'611] local-les=243 n=119 ec=118 les/c 243/191 242/242/223) -260> 2014-06-09 16:22:12.253372 7f86a460e700 10 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] purge_strays -259> 2014-06-09 16:22:12.253382 7f86a460e700 15 osd.26 pg_epoch: 243 pg[3.380( v 213'672 (0'0,213'672] local-les=243 n=131 ec=118 les/c 243/243 242/242/242) [26,5] r=0 lpr=242 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:1106 -258> 2014-06-09 16:22:12.253392 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbfce200,0x18742660,0xbd6cd60,0xbd75ea0,0x324a8a20] -257> 2014-06-09 16:22:12.253394 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -256> 2014-06-09 16:22:12.253395 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -255> 2014-06-09 16:22:12.253398 7f868af58700 2 waiting 51 > 50 ops || 4456789 > 104857600 -254> 2014-06-09 16:22:12.253401 7f868f761700 10 log is not dirty -253> 2014-06-09 16:22:12.253405 7f868f761700 5 queue_transactions existing osr(3.366 0x3ed1080)/0x3ed1080 -252> 2014-06-09 16:22:12.253409 7f868f761700 2 waiting 51 > 50 ops || 4456756 > 104857600 -251> 2014-06-09 16:22:12.253433 7f8689755700 2 waiting 51 > 50 ops || 4456754 > 104857600 -250> 2014-06-09 16:22:12.253440 7f868ef60700 2 waiting 51 > 50 ops || 4456789 > 104857600 -249> 2014-06-09 16:22:12.253444 7f86a5e11700 5 _do_op 0xc2d4c60 seq 4297 osr(2.2da 0x2ed0c10)/0x2ed0c10 start -248> 2014-06-09 16:22:12.253448 7f86a5e11700 10 _do_transaction on 0xc3d7980 -247> 2014-06-09 16:22:12.253454 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -246> 2014-06-09 16:22:12.253496 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -245> 2014-06-09 16:22:12.253512 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4297.0.0 <= header.spos 150443.0.0 -244> 2014-06-09 16:22:12.253517 7f86a5e11700 5 submit_transaction r = 0 -243> 2014-06-09 16:22:12.253531 7f86a5e11700 10 _do_op 0xc2d4c60 seq 4297 r = 0, finisher 0xbd6cac0 0 -242> 2014-06-09 16:22:12.253538 7f86a5e11700 10 _finish_op 0xc2d4c60 seq 4297 osr(2.2da 0x2ed0c10)/0x2ed0c10 -241> 2014-06-09 16:22:12.253539 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xbd5a880] -240> 2014-06-09 16:22:12.253550 7f8689f56700 10 op_submit_start 4346 -239> 2014-06-09 16:22:12.253543 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] _finish_recovery -238> 2014-06-09 16:22:12.253551 7f8689f56700 5 queue_transactions (trailing journal) 4346 -237> 2014-06-09 16:22:12.253553 7f8689f56700 5 queue_op 0xc2d4120 seq 4346 osr(0.5e4 0x2bd79d0) 861 bytes (queue has 50 ops and 4455928 bytes) -236> 2014-06-09 16:22:12.253551 7f86a460e700 10 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] purge_strays -235> 2014-06-09 16:22:12.253558 7f868ff62700 2 waiting 51 > 50 ops || 4456756 > 104857600 -234> 2014-06-09 16:22:12.253563 7f868df5e700 2 waiting 51 > 50 ops || 4456756 > 104857600 -233> 2014-06-09 16:22:12.253560 7f86a460e700 15 osd.26 pg_epoch: 243 pg[2.2da( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,1] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:177 -232> 2014-06-09 16:22:12.253567 7f868e75f700 2 waiting 51 > 50 ops || 4456756 > 104857600 -231> 2014-06-09 16:22:12.253568 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xbd5a880] -230> 2014-06-09 16:22:12.253570 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -229> 2014-06-09 16:22:12.253571 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -228> 2014-06-09 16:22:12.253571 7f8692767700 2 waiting 51 > 50 ops || 4456756 > 104857600 -227> 2014-06-09 16:22:12.253576 7f8690763700 2 waiting 51 > 50 ops || 4456756 > 104857600 -226> 2014-06-09 16:22:12.253581 7f8692f68700 2 waiting 51 > 50 ops || 4456756 > 104857600 -225> 2014-06-09 16:22:12.253585 7f868bf5a700 2 waiting 51 > 50 ops || 4456756 > 104857600 -224> 2014-06-09 16:22:12.253589 7f868b759700 2 waiting 51 > 50 ops || 4456756 > 104857600 -223> 2014-06-09 16:22:12.253594 7f868cf5c700 2 waiting 51 > 50 ops || 4456756 > 104857600 -222> 2014-06-09 16:22:12.253599 7f868d75d700 2 waiting 51 > 50 ops || 4456789 > 104857600 -221> 2014-06-09 16:22:12.253604 7f8691f66700 2 waiting 51 > 50 ops || 4456789 > 104857600 -220> 2014-06-09 16:22:12.253604 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd6cac0] -219> 2014-06-09 16:22:12.253608 7f868a757700 2 waiting 51 > 50 ops || 4456756 > 104857600 -218> 2014-06-09 16:22:12.253612 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd6cac0] -217> 2014-06-09 16:22:12.253614 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -216> 2014-06-09 16:22:12.253615 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -215> 2014-06-09 16:22:12.253618 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89b7a0 (1 active) -214> 2014-06-09 16:22:12.253620 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc3fb680 (2 active) -213> 2014-06-09 16:22:12.253625 7f8689f56700 10 op_submit_finish 4346 -212> 2014-06-09 16:22:12.253630 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -211> 2014-06-09 16:22:12.253643 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.048759 7 0.000470 -210> 2014-06-09 16:22:12.253656 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -209> 2014-06-09 16:22:12.253666 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -208> 2014-06-09 16:22:12.253676 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000021 0 0.000000 -207> 2014-06-09 16:22:12.253687 7f8689f56700 5 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -206> 2014-06-09 16:22:12.253695 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -205> 2014-06-09 16:22:12.253703 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/210 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -204> 2014-06-09 16:22:12.253714 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=209-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(209-235 up [26,7](26) acting [26,7](26) maybe_went_rw) -203> 2014-06-09 16:22:12.253726 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [7](7) acting [7](7)) -202> 2014-06-09 16:22:12.253736 7f8689f56700 10 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -201> 2014-06-09 16:22:12.253748 7f8689f56700 20 osd.26 243 share_map_peer 0xbfa6580 already has epoch 244 -200> 2014-06-09 16:22:12.253751 7f8689f56700 1 -- 172.16.96.202:6850/30904 --> osd.7 172.16.96.201:6851/52636 -- pg_info(1 pgs e243:1.116) v4 -- ?+0 0x240e8c40 -199> 2014-06-09 16:22:12.253763 7f8689f56700 15 osd.26 pg_epoch: 243 pg[1.116( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,7] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:145 -198> 2014-06-09 16:22:12.253797 7f8689f56700 10 log is not dirty -197> 2014-06-09 16:22:12.253801 7f8689f56700 5 queue_transactions existing osr(1.116 0x384c8f0)/0x384c8f0 -196> 2014-06-09 16:22:12.253804 7f8689f56700 2 waiting 51 > 50 ops || 4456789 > 104857600 -195> 2014-06-09 16:22:12.253808 7f86a5e11700 5 _do_op 0xbf7f020 seq 4298 osr(1.35c 0x384d2b0)/0x384d2b0 start -194> 2014-06-09 16:22:12.253811 7f86a5e11700 10 _do_transaction on 0x39da3c0 -193> 2014-06-09 16:22:12.253816 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -192> 2014-06-09 16:22:12.253858 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -191> 2014-06-09 16:22:12.253872 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4298.0.0 <= header.spos 150443.0.0 -190> 2014-06-09 16:22:12.253877 7f86a5e11700 5 submit_transaction r = 0 -189> 2014-06-09 16:22:12.253889 7f86a5e11700 10 _do_op 0xbf7f020 seq 4298 r = 0, finisher 0x13964600 0 -188> 2014-06-09 16:22:12.253895 7f86a5e11700 10 _finish_op 0xbf7f020 seq 4298 osr(1.35c 0x384d2b0)/0x384d2b0 -187> 2014-06-09 16:22:12.253897 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32074f40] -186> 2014-06-09 16:22:12.253900 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32074f40] -185> 2014-06-09 16:22:12.253904 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -184> 2014-06-09 16:22:12.253905 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -183> 2014-06-09 16:22:12.253908 7f8690f64700 10 op_submit_start 4347 -182> 2014-06-09 16:22:12.253909 7f868af58700 2 waiting 51 > 50 ops || 4456789 > 104857600 -181> 2014-06-09 16:22:12.253910 7f8690f64700 5 queue_transactions (trailing journal) 4347 -180> 2014-06-09 16:22:12.253911 7f8690f64700 5 queue_op 0x325270e0 seq 4347 osr(1.408 0x384d770) 828 bytes (queue has 50 ops and 4455928 bytes) -179> 2014-06-09 16:22:12.253917 7f868f761700 2 waiting 51 > 50 ops || 4456756 > 104857600 -178> 2014-06-09 16:22:12.253921 7f8689755700 2 waiting 51 > 50 ops || 4456754 > 104857600 -177> 2014-06-09 16:22:12.253926 7f868ef60700 2 waiting 51 > 50 ops || 4456789 > 104857600 -176> 2014-06-09 16:22:12.253930 7f8691765700 2 waiting 51 > 50 ops || 4456756 > 104857600 -175> 2014-06-09 16:22:12.253935 7f868df5e700 2 waiting 51 > 50 ops || 4456756 > 104857600 -174> 2014-06-09 16:22:12.253939 7f868e75f700 2 waiting 51 > 50 ops || 4456756 > 104857600 -173> 2014-06-09 16:22:12.253944 7f8692767700 2 waiting 51 > 50 ops || 4456756 > 104857600 -172> 2014-06-09 16:22:12.253948 7f8690763700 2 waiting 51 > 50 ops || 4456756 > 104857600 -171> 2014-06-09 16:22:12.253952 7f8692f68700 2 waiting 51 > 50 ops || 4456756 > 104857600 -170> 2014-06-09 16:22:12.253956 7f868bf5a700 2 waiting 51 > 50 ops || 4456756 > 104857600 -169> 2014-06-09 16:22:12.253960 7f868b759700 2 waiting 51 > 50 ops || 4456756 > 104857600 -168> 2014-06-09 16:22:12.253965 7f868cf5c700 2 waiting 51 > 50 ops || 4456756 > 104857600 -167> 2014-06-09 16:22:12.253969 7f868d75d700 2 waiting 51 > 50 ops || 4456789 > 104857600 -166> 2014-06-09 16:22:12.253975 7f8691f66700 2 waiting 51 > 50 ops || 4456789 > 104857600 -165> 2014-06-09 16:22:12.253975 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x13964600] -164> 2014-06-09 16:22:12.253978 7f868a757700 2 waiting 51 > 50 ops || 4456756 > 104857600 -163> 2014-06-09 16:22:12.253979 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x13964600] -162> 2014-06-09 16:22:12.253980 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -161> 2014-06-09 16:22:12.253981 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -160> 2014-06-09 16:22:12.253984 7f8689f56700 2 waiting 51 > 50 ops || 4456789 > 104857600 -159> 2014-06-09 16:22:12.253990 7f868ff62700 2 waiting 51 > 50 ops || 4456756 > 104857600 -158> 2014-06-09 16:22:12.253993 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc3fb680 (1 active) -157> 2014-06-09 16:22:12.253996 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc52c240 (2 active) -156> 2014-06-09 16:22:12.254001 7f8690f64700 10 op_submit_finish 4347 -155> 2014-06-09 16:22:12.254005 7f8690f64700 10 osd.26 pg_epoch: 243 pg[3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 243/201 242/242/200) [3,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 3 info: 3.5c6( v 213'693 (0'0,213'693] local-les=243 n=133 ec=118 les/c 243/243 242/242/200) -154> 2014-06-09 16:22:12.254041 7f8690f64700 10 log is not dirty -153> 2014-06-09 16:22:12.254045 7f8690f64700 5 queue_transactions existing osr(3.5c6 0x3ed17f0)/0x3ed17f0 -152> 2014-06-09 16:22:12.254048 7f8690f64700 2 waiting 51 > 50 ops || 4456756 > 104857600 -151> 2014-06-09 16:22:12.254052 7f86a5e11700 5 _do_op 0xc2d56e0 seq 4299 osr(3.2a9 0x3ed0b40)/0x3ed0b40 start -150> 2014-06-09 16:22:12.254055 7f86a5e11700 10 _do_transaction on 0x30d0c00 -149> 2014-06-09 16:22:12.254060 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -148> 2014-06-09 16:22:12.254162 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -147> 2014-06-09 16:22:12.254177 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4299.0.0 <= header.spos 150443.0.0 -146> 2014-06-09 16:22:12.254181 7f86a5e11700 5 submit_transaction r = 0 -145> 2014-06-09 16:22:12.254194 7f86a5e11700 10 _do_op 0xc2d56e0 seq 4299 r = 0, finisher 0x3205d880 0 -144> 2014-06-09 16:22:12.254200 7f86a5e11700 10 _finish_op 0xc2d56e0 seq 4299 osr(3.2a9 0x3ed0b40)/0x3ed0b40 -143> 2014-06-09 16:22:12.254201 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x318c2780] -142> 2014-06-09 16:22:12.254204 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x318c2780] -141> 2014-06-09 16:22:12.254206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -140> 2014-06-09 16:22:12.254206 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -139> 2014-06-09 16:22:12.254212 7f868af58700 10 op_submit_start 4348 -138> 2014-06-09 16:22:12.254213 7f868af58700 5 queue_transactions (trailing journal) 4348 -137> 2014-06-09 16:22:12.254214 7f868f761700 2 waiting 51 > 50 ops || 4456789 > 104857600 -136> 2014-06-09 16:22:12.254215 7f868af58700 5 queue_op 0x324e7980 seq 4348 osr(1.36f 0x384d290) 861 bytes (queue has 50 ops and 4455961 bytes) -135> 2014-06-09 16:22:12.254219 7f8689755700 2 waiting 51 > 50 ops || 4456787 > 104857600 -134> 2014-06-09 16:22:12.254224 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x3205d880] -133> 2014-06-09 16:22:12.254228 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x3205d880] -132> 2014-06-09 16:22:12.254230 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -131> 2014-06-09 16:22:12.254231 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -130> 2014-06-09 16:22:12.254234 7f868ef60700 2 waiting 51 > 50 ops || 4456822 > 104857600 -129> 2014-06-09 16:22:12.254238 7f8691765700 2 waiting 51 > 50 ops || 4456789 > 104857600 -128> 2014-06-09 16:22:12.254242 7f868df5e700 2 waiting 51 > 50 ops || 4456789 > 104857600 -127> 2014-06-09 16:22:12.254247 7f868e75f700 2 waiting 51 > 50 ops || 4456789 > 104857600 -126> 2014-06-09 16:22:12.254251 7f8692767700 2 waiting 51 > 50 ops || 4456789 > 104857600 -125> 2014-06-09 16:22:12.254256 7f8690763700 2 waiting 51 > 50 ops || 4456789 > 104857600 -124> 2014-06-09 16:22:12.254260 7f8692f68700 2 waiting 51 > 50 ops || 4456789 > 104857600 -123> 2014-06-09 16:22:12.254264 7f868bf5a700 2 waiting 51 > 50 ops || 4456789 > 104857600 -122> 2014-06-09 16:22:12.254269 7f868b759700 2 waiting 51 > 50 ops || 4456789 > 104857600 -121> 2014-06-09 16:22:12.254273 7f868cf5c700 2 waiting 51 > 50 ops || 4456789 > 104857600 -120> 2014-06-09 16:22:12.254278 7f868d75d700 2 waiting 51 > 50 ops || 4456822 > 104857600 -119> 2014-06-09 16:22:12.254278 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc52c240 (1 active) -118> 2014-06-09 16:22:12.254282 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc89f560 (2 active) -117> 2014-06-09 16:22:12.254284 7f8691f66700 2 waiting 51 > 50 ops || 4456822 > 104857600 -116> 2014-06-09 16:22:12.254286 7f86a5e11700 5 _do_op 0xcca7560 seq 4300 osr(3.400 0x3ed1260)/0x3ed1260 start -115> 2014-06-09 16:22:12.254288 7f868a757700 2 waiting 51 > 50 ops || 4456789 > 104857600 -114> 2014-06-09 16:22:12.254289 7f86a5e11700 10 _do_transaction on 0xbf92e40 -113> 2014-06-09 16:22:12.254293 7f868af58700 10 op_submit_finish 4348 -112> 2014-06-09 16:22:12.254293 7f8689f56700 2 waiting 51 > 50 ops || 4456822 > 104857600 -111> 2014-06-09 16:22:12.254298 7f868ff62700 2 waiting 51 > 50 ops || 4456789 > 104857600 -110> 2014-06-09 16:22:12.254303 7f8690f64700 2 waiting 51 > 50 ops || 4456789 > 104857600 -109> 2014-06-09 16:22:12.254297 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 AllReplicasRecovered -108> 2014-06-09 16:22:12.254310 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -107> 2014-06-09 16:22:12.254309 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Activating 1.044023 7 0.000367 -106> 2014-06-09 16:22:12.254320 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Recovered -105> 2014-06-09 16:22:12.254329 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] needs_recovery is recovered -104> 2014-06-09 16:22:12.254338 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] exit Started/Primary/Active/Recovered 0.000018 0 0.000000 -103> 2014-06-09 16:22:12.254347 7f868af58700 5 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] enter Started/Primary/Active/Clean -102> 2014-06-09 16:22:12.254354 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] finish_recovery -101> 2014-06-09 16:22:12.254361 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -100> 2014-06-09 16:22:12.254361 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/213 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active] clear_recovery_state -99> 2014-06-09 16:22:12.254375 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4300.0.0 <= header.spos 150443.0.0 -98> 2014-06-09 16:22:12.254371 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=212-241/2 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(212-235 up [26,9](26) acting [26,9](26) maybe_went_rw) -97> 2014-06-09 16:22:12.254379 7f86a5e11700 5 submit_transaction r = 0 -96> 2014-06-09 16:22:12.254381 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 pi=236-241/1 crt=0'0 mlcod 0'0 active+clean] trim_past_intervals: trimming interval(236-241 up [9](9) acting [9](9)) -95> 2014-06-09 16:22:12.254390 7f868af58700 10 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] share_pg_info -94> 2014-06-09 16:22:12.254396 7f86a5e11700 10 _do_op 0xcca7560 seq 4300 r = 0, finisher 0x32078220 0 -93> 2014-06-09 16:22:12.254403 7f86a5e11700 10 _finish_op 0xcca7560 seq 4300 osr(3.400 0x3ed1260)/0x3ed1260 -92> 2014-06-09 16:22:12.254403 7f868af58700 20 osd.26 243 share_map_peer 0xbeb8f20 already has epoch 244 -91> 2014-06-09 16:22:12.254406 7f868af58700 1 -- 172.16.96.202:6850/30904 --> osd.9 172.16.96.201:6865/53518 -- pg_info(1 pgs e243:0.732) v4 -- ?+0 0x29ec4c40 -90> 2014-06-09 16:22:12.254414 7f8689755700 10 op_submit_start 4349 -89> 2014-06-09 16:22:12.254416 7f8689755700 5 queue_transactions (trailing journal) 4349 -88> 2014-06-09 16:22:12.254418 7f8689755700 5 queue_op 0x33a3bc0 seq 4349 osr(1.fa 0x384c970) 826 bytes (queue has 50 ops and 4455959 bytes) -87> 2014-06-09 16:22:12.254423 7f868ef60700 2 waiting 51 > 50 ops || 4456820 > 104857600 -86> 2014-06-09 16:22:12.254417 7f868af58700 15 osd.26 pg_epoch: 243 pg[0.732( empty local-les=243 n=0 ec=1 les/c 243/243 242/242/242) [26,9] r=0 lpr=242 crt=0'0 mlcod 0'0 active+clean] publish_stats_to_osd 243:192 -85> 2014-06-09 16:22:12.254428 7f8691765700 2 waiting 51 > 50 ops || 4456787 > 104857600 -84> 2014-06-09 16:22:12.254432 7f868df5e700 2 waiting 51 > 50 ops || 4456787 > 104857600 -83> 2014-06-09 16:22:12.254437 7f868e75f700 2 waiting 51 > 50 ops || 4456787 > 104857600 -82> 2014-06-09 16:22:12.254441 7f8692767700 2 waiting 51 > 50 ops || 4456787 > 104857600 -81> 2014-06-09 16:22:12.254446 7f8690763700 2 waiting 51 > 50 ops || 4456787 > 104857600 -80> 2014-06-09 16:22:12.254447 7f868af58700 10 log is not dirty -79> 2014-06-09 16:22:12.254450 7f8692f68700 2 waiting 51 > 50 ops || 4456787 > 104857600 -78> 2014-06-09 16:22:12.254451 7f868af58700 5 queue_transactions existing osr(0.732 0x384c3e0)/0x384c3e0 -77> 2014-06-09 16:22:12.254453 7f868af58700 2 waiting 51 > 50 ops || 4456820 > 104857600 -76> 2014-06-09 16:22:12.254459 7f868b759700 2 waiting 51 > 50 ops || 4456787 > 104857600 -75> 2014-06-09 16:22:12.254466 7f868cf5c700 2 waiting 51 > 50 ops || 4456787 > 104857600 -74> 2014-06-09 16:22:12.254472 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0x32078220] -73> 2014-06-09 16:22:12.254477 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0x32078220] -72> 2014-06-09 16:22:12.254477 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0x32079c00] -71> 2014-06-09 16:22:12.254479 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -70> 2014-06-09 16:22:12.254480 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -69> 2014-06-09 16:22:12.254482 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0x32079c00] -68> 2014-06-09 16:22:12.254484 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -67> 2014-06-09 16:22:12.254483 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc89f560 (1 active) -66> 2014-06-09 16:22:12.254485 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -65> 2014-06-09 16:22:12.254486 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc62ec60 (2 active) -64> 2014-06-09 16:22:12.254488 7f868d75d700 2 waiting 51 > 50 ops || 4456820 > 104857600 -63> 2014-06-09 16:22:12.254491 7f8689755700 10 op_submit_finish 4349 -62> 2014-06-09 16:22:12.254493 7f8691f66700 2 waiting 51 > 50 ops || 4456820 > 104857600 -61> 2014-06-09 16:22:12.254496 7f868a757700 2 waiting 51 > 50 ops || 4456787 > 104857600 -60> 2014-06-09 16:22:12.254495 7f8689755700 10 osd.26 pg_epoch: 243 pg[3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 243/213 242/242/212) [9,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 9 info: 3.375( v 213'571 (0'0,213'571] local-les=243 n=117 ec=118 les/c 243/243 242/242/212) -59> 2014-06-09 16:22:12.254533 7f8689755700 10 log is not dirty -58> 2014-06-09 16:22:12.254538 7f8689755700 5 queue_transactions existing osr(3.375 0x3ed1060)/0x3ed1060 -57> 2014-06-09 16:22:12.254540 7f8689755700 2 waiting 51 > 50 ops || 4456787 > 104857600 -56> 2014-06-09 16:22:12.254545 7f8689f56700 2 waiting 51 > 50 ops || 4456820 > 104857600 -55> 2014-06-09 16:22:12.254550 7f868ff62700 2 waiting 51 > 50 ops || 4456787 > 104857600 -54> 2014-06-09 16:22:12.254554 7f8690f64700 2 waiting 51 > 50 ops || 4456787 > 104857600 -53> 2014-06-09 16:22:12.254559 7f868f761700 2 waiting 51 > 50 ops || 4456787 > 104857600 -52> 2014-06-09 16:22:12.254563 7f868bf5a700 2 waiting 51 > 50 ops || 4456787 > 104857600 -51> 2014-06-09 16:22:12.254567 7f86a5e11700 5 _do_op 0x3280a000 seq 4301 osr(3.9f 0x3ed0600)/0x3ed0600 start -50> 2014-06-09 16:22:12.254570 7f86a5e11700 10 _do_transaction on 0x39dbb00 -49> 2014-06-09 16:22:12.254576 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -48> 2014-06-09 16:22:12.254617 7f86a5e11700 10 lookup_strip_header done cid meta oid 16ef7597/infos/head//-1 -47> 2014-06-09 16:22:12.254632 7f86a5e11700 10 cid: oid: 16ef7597/infos/head//-1 skipping op, spos 4301.0.0 <= header.spos 150443.0.0 -46> 2014-06-09 16:22:12.255632 7f868c75b700 2 waiting 51 > 50 ops || 4456787 > 104857600 -45> 2014-06-09 16:22:12.311698 7f8674ab4700 1 -- 172.16.96.202:6830/30904 >> :/0 pipe(0x32071180 sd=94 :6830 s=0 pgs=0 cs=0 l=0 c=0x1abd8420).accept sd=94 172.16.96.220:45918/0 -44> 2014-06-09 16:22:12.323646 7f86a5e11700 5 submit_transaction r = 0 -43> 2014-06-09 16:22:12.323677 7f86a5e11700 10 _do_op 0x3280a000 seq 4301 r = 0, finisher 0xbd698a0 0 -42> 2014-06-09 16:22:12.323675 7f8674ab4700 10 osd.26 244 new session 0x1078b400 con=0x1abd8420 addr=172.16.96.220:0/1123804 -41> 2014-06-09 16:22:12.323689 7f86a5e11700 10 _finish_op 0x3280a000 seq 4301 osr(3.9f 0x3ed0600)/0x3ed0600 -40> 2014-06-09 16:22:12.323691 7f86a460e700 10 finisher(0x2c100f8) finisher_thread doing [0xc6e9b20] -39> 2014-06-09 16:22:12.323697 7f86a460e700 10 finisher(0x2c100f8) finisher_thread done with [0xc6e9b20] -38> 2014-06-09 16:22:12.323699 7f86a460e700 10 finisher(0x2c100f8) finisher_thread empty -37> 2014-06-09 16:22:12.323700 7f86a460e700 10 finisher(0x2c100f8) finisher_thread sleeping -36> 2014-06-09 16:22:12.323707 7f8691765700 10 op_submit_start 4350 -35> 2014-06-09 16:22:12.323708 7f8691765700 5 queue_transactions (trailing journal) 4350 -34> 2014-06-09 16:22:12.323711 7f8691765700 5 queue_op 0x325265a0 seq 4350 osr(3.1b5 0x3ed0a80) 828 bytes (queue has 50 ops and 4455961 bytes) -33> 2014-06-09 16:22:12.323717 7f868df5e700 2 waiting 51 > 50 ops || 4456789 > 104857600 -32> 2014-06-09 16:22:12.323724 7f868e75f700 2 waiting 51 > 50 ops || 4456789 > 104857600 -31> 2014-06-09 16:22:12.323729 7f8692767700 2 waiting 51 > 50 ops || 4456789 > 104857600 -30> 2014-06-09 16:22:12.323735 7f8690763700 2 waiting 51 > 50 ops || 4456789 > 104857600 -29> 2014-06-09 16:22:12.323740 7f8692f68700 2 waiting 51 > 50 ops || 4456789 > 104857600 -28> 2014-06-09 16:22:12.323748 7f868af58700 2 waiting 51 > 50 ops || 4456822 > 104857600 -27> 2014-06-09 16:22:12.323751 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread doing [0xbd698a0] -26> 2014-06-09 16:22:12.323759 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread done with [0xbd698a0] -25> 2014-06-09 16:22:12.323761 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread empty -24> 2014-06-09 16:22:12.323762 7f86a4e0f700 10 finisher(0x2c10388) finisher_thread sleeping -23> 2014-06-09 16:22:12.323766 7f868b759700 2 waiting 51 > 50 ops || 4456789 > 104857600 -22> 2014-06-09 16:22:12.323771 7f868cf5c700 2 waiting 51 > 50 ops || 4456789 > 104857600 -21> 2014-06-09 16:22:12.323776 7f86a5e11700 15 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ done processing 0xc62ec60 (1 active) -20> 2014-06-09 16:22:12.323779 7f86a5e11700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0xc338240 (2 active) -19> 2014-06-09 16:22:12.323783 7f8691765700 10 op_submit_finish 4350 -18> 2014-06-09 16:22:12.323789 7f8691765700 10 osd.26 pg_epoch: 243 pg[3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 203/203 242/242/202) [4,26] r=1 lpr=242 pi=118-241/10 luod=0'0 crt=0'0 lcod 0'0 active] handle_peering_event: epoch_sent: 243 epoch_requested: 243 MInfoRec from 4 info: 3.11f( v 213'644 (0'0,213'644] local-les=243 n=127 ec=118 les/c 243/203 242/242/202) -17> 2014-06-09 16:22:12.323835 7f8691765700 10 log is not dirty -16> 2014-06-09 16:22:12.323845 7f8691765700 5 queue_transactions existing osr(3.11f 0x3ed07e0)/0x3ed07e0 -15> 2014-06-09 16:22:12.323848 7f8691765700 2 waiting 51 > 50 ops || 4456789 > 104857600 -14> 2014-06-09 16:22:12.323852 7f86a5e11700 5 _do_op 0x2e21380 seq 4302 osr(3.224 0x3ed0c80)/0x3ed0c80 start -13> 2014-06-09 16:22:12.323855 7f86a5e11700 10 _do_transaction on 0xc2be300 -12> 2014-06-09 16:22:12.323863 7f86a5e11700 15 _omap_setkeys meta/16ef7597/infos/head//-1 -11> 2014-06-09 16:22:12.325506 7f86a5610700 -1 *** Caught signal (Aborted) ** in thread 7f86a5610700 ceph version 0.80-820-g5d606cd (5d606cd0d00698699c91a378a1bd9f71cc8a77c9) 1: /usr/local/bin/ceph-osd() [0xa95f40] 2: (()+0xfcb0) [0x7f86aed35cb0] 3: (gsignal()+0x35) [0x7f86ad21b425] 4: (abort()+0x17b) [0x7f86ad21eb8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f86adb16e2d] 6: (()+0x5ef26) [0x7f86adb14f26] 7: (()+0x5ef53) [0x7f86adb14f53] 8: (()+0x5f17e) [0x7f86adb1517e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0xb70e6d] 10: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x750) [0x9e6f20] 11: (KeyValueStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9e8d1e] 12: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x9a) [0x9e8e2a] 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb62a3a] 14: (ThreadPool::WorkThread::entry()+0x10) [0xb63c90] 15: (()+0x7e9a) [0x7f86aed2de9a] 16: (clone()+0x6d) [0x7f86ad2d8ccd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. -10> 2014-06-09 16:22:12.327623 7f868d75d700 2 waiting 51 > 50 ops || 4456822 > 104857600 -9> 2014-06-09 16:22:12.327635 7f8691f66700 2 waiting 51 > 50 ops || 4456822 > 104857600 -8> 2014-06-09 16:22:12.327644 7f868a757700 2 waiting 51 > 50 ops || 4456789 > 104857600 -7> 2014-06-09 16:22:12.327651 7f8689755700 2 waiting 51 > 50 ops || 4456789 > 104857600 -6> 2014-06-09 16:22:12.327662 7f8689f56700 2 waiting 51 > 50 ops || 4456822 > 104857600 -5> 2014-06-09 16:22:12.327669 7f868ff62700 2 waiting 51 > 50 ops || 4456789 > 104857600 -4> 2014-06-09 16:22:12.327677 7f8690f64700 2 waiting 51 > 50 ops || 4456789 > 104857600 -3> 2014-06-09 16:22:12.327684 7f868f761700 2 waiting 51 > 50 ops || 4456789 > 104857600 -2> 2014-06-09 16:22:12.327690 7f868bf5a700 2 waiting 51 > 50 ops || 4456789 > 104857600 -1> 2014-06-09 16:22:12.327696 7f868c75b700 2 waiting 51 > 50 ops || 4456789 > 104857600 0> 2014-06-09 16:22:12.327702 7f868ef60700 2 waiting 51 > 50 ops || 4456822 > 104857600 --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 20/20 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 20/20 osd 20/20 optracker 0/ 5 objclass 20/20 filestore 20/20 keyvaluestore 1/ 3 journal 0/ 5 ms 20/20 mon 0/10 monc 1/ 5 paxos 20/20 tp 1/ 5 auth 1/ 5 crypto 20/20 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /log/osd.26.log --- end dump of recent events --- 2014-06-09 16:22:12.330235 7f8674ab4700 5 -- op tracker -- , seq: 3684, time: 2014-06-09 16:22:12.324001, event: header_read, op: osd_op(client.4544.0:33845 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.330249 7f8674ab4700 5 -- op tracker -- , seq: 3684, time: 2014-06-09 16:22:12.324006, event: throttled, op: osd_op(client.4544.0:33845 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.330259 7f8674ab4700 5 -- op tracker -- , seq: 3684, time: 2014-06-09 16:22:12.330190, event: all_read, op: osd_op(client.4544.0:33845 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.330272 7f8674ab4700 5 -- op tracker -- , seq: 3684, time: 0.000000, event: dispatched, op: osd_op(client.4544.0:33845 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.330283 7f8674ab4700 20 osd.26 244 should_share_map client.4544 172.16.96.220:0/1123804 244 2014-06-09 16:22:12.330312 7f8674ab4700 15 osd.26 244 enqueue_op 0x1b0b7200 prio 63 cost 1032192 latency 0.006311 osd_op(client.4544.0:33845 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 1794048~1032192] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) v4 2014-06-09 16:22:12.333264 7f8674ab4700 5 -- op tracker -- , seq: 3685, time: 2014-06-09 16:22:12.330334, event: header_read, op: osd_op(client.4544.0:33846 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~1368064] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.333277 7f8674ab4700 5 -- op tracker -- , seq: 3685, time: 2014-06-09 16:22:12.330336, event: throttled, op: osd_op(client.4544.0:33846 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~1368064] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.333287 7f8674ab4700 5 -- op tracker -- , seq: 3685, time: 2014-06-09 16:22:12.333222, event: all_read, op: osd_op(client.4544.0:33846 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~1368064] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.333297 7f8674ab4700 5 -- op tracker -- , seq: 3685, time: 0.000000, event: dispatched, op: osd_op(client.4544.0:33846 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~1368064] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.333306 7f8674ab4700 20 osd.26 244 should_share_map client.4544 172.16.96.220:0/1123804 244 2014-06-09 16:22:12.333332 7f8674ab4700 15 osd.26 244 enqueue_op 0x2f097a00 prio 63 cost 1368064 latency 0.002997 osd_op(client.4544.0:33846 rbd_data.11606b8b4567.00000000000017de [set-alloc-hint object_size 4194304 write_size 4194304,write 2826240~1368064] 3.5e115e61 RETRY=3 ack+ondisk+retry+write e244) v4 2014-06-09 16:22:12.333481 7f8674ab4700 5 -- op tracker -- , seq: 3686, time: 2014-06-09 16:22:12.333374, event: header_read, op: osd_op(client.4544.0:34117 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.333493 7f8674ab4700 5 -- op tracker -- , seq: 3686, time: 2014-06-09 16:22:12.333376, event: throttled, op: osd_op(client.4544.0:34117 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.333503 7f8674ab4700 5 -- op tracker -- , seq: 3686, time: 2014-06-09 16:22:12.333457, event: all_read, op: osd_op(client.4544.0:34117 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.333517 7f8674ab4700 5 -- op tracker -- , seq: 3686, time: 0.000000, event: dispatched, op: osd_op(client.4544.0:34117 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.333524 7f8674ab4700 20 osd.26 244 should_share_map client.4544 172.16.96.220:0/1123804 244 2014-06-09 16:22:12.333540 7f8674ab4700 15 osd.26 244 enqueue_op 0x2f097900 prio 63 cost 57344 latency 0.000165 osd_op(client.4544.0:34117 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~57344] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) v4 2014-06-09 16:22:12.334181 7f8674ab4700 5 -- op tracker -- , seq: 3687, time: 2014-06-09 16:22:12.333573, event: header_read, op: osd_op(client.4544.0:34118 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 57344~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.334201 7f8674ab4700 5 -- op tracker -- , seq: 3687, time: 2014-06-09 16:22:12.333574, event: throttled, op: osd_op(client.4544.0:34118 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 57344~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.334211 7f8674ab4700 5 -- op tracker -- , seq: 3687, time: 2014-06-09 16:22:12.334154, event: all_read, op: osd_op(client.4544.0:34118 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 57344~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.334221 7f8674ab4700 5 -- op tracker -- , seq: 3687, time: 0.000000, event: dispatched, op: osd_op(client.4544.0:34118 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 57344~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.334227 7f8674ab4700 20 osd.26 244 should_share_map client.4544 172.16.96.220:0/1123804 244 2014-06-09 16:22:12.334244 7f8674ab4700 15 osd.26 244 enqueue_op 0x22008d00 prio 63 cost 516096 latency 0.000671 osd_op(client.4544.0:34118 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 57344~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) v4 2014-06-09 16:22:12.335158 7f8674ab4700 5 -- op tracker -- , seq: 3688, time: 2014-06-09 16:22:12.334261, event: header_read, op: osd_op(client.4544.0:34119 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 573440~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.335169 7f8674ab4700 5 -- op tracker -- , seq: 3688, time: 2014-06-09 16:22:12.334262, event: throttled, op: osd_op(client.4544.0:34119 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 573440~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.335182 7f8674ab4700 5 -- op tracker -- , seq: 3688, time: 2014-06-09 16:22:12.335131, event: all_read, op: osd_op(client.4544.0:34119 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 573440~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.335192 7f8674ab4700 5 -- op tracker -- , seq: 3688, time: 0.000000, event: dispatched, op: osd_op(client.4544.0:34119 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 573440~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.335199 7f8674ab4700 20 osd.26 244 should_share_map client.4544 172.16.96.220:0/1123804 244 2014-06-09 16:22:12.335216 7f8674ab4700 15 osd.26 244 enqueue_op 0xdfa7100 prio 63 cost 516096 latency 0.000955 osd_op(client.4544.0:34119 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 573440~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) v4 2014-06-09 16:22:12.335655 7f86749b3700 10 osd.26 244 new session 0x1078a200 con=0x1abd8580 addr=172.16.96.220:0/1123832 2014-06-09 16:22:12.336103 7f8674ab4700 5 -- op tracker -- , seq: 3689, time: 2014-06-09 16:22:12.335234, event: header_read, op: osd_op(client.4544.0:34120 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 1089536~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244) 2014-06-09 16:22:12.336114 7f8674ab4700 5 -- op tracker -- , seq: 3689, time: 2014-06-09 16:22:12.335235, event: throttled, op: osd_op(client.4544.0:34120 rbd_data.11606b8b4567.0000000000001808 [set-alloc-hint object_size 4194304 write_size 4194304,write 1089536~516096] 3.eb9de0d3 RETRY=3 ack+ondisk+retry+write e244)